For Researchers
Berthier is the wargaming platform for an experiment that benchmarks A.I. as a strategic advisor. Benchmarks for A.I. systems proposed as military advisors usually measure recall, reasoning on stated problems and the quality of prose, in isolation from consequence; none contains an adversary. This one does. Human, machine and machine-assisted seats command formations in a historical Napoleonic campaign, under courier-lagged, partly false intelligence, against an opponent trying to win, and every order is stamped against what its seat knew when it wrote it.
Status: the design is written for pre-registration before the first research game. Nothing on this page is a finding.
Human: a human player writing orders unaided. Machine: a frozen A.I. configuration writing orders through a documented interface. Machine-assisted (the centaur): a human player writing orders with a specified A.I. assistant. The order is the commander’s act and everything upstream of it is staff work, so the machine-assisted arm measures A.I. staff support cleanly: the human decides, and the assistant’s inputs and outputs are recorded beside the order the human issued. Arms are assigned to seats within the same game, so every arm faces the same opponent, scenario and rules.
| Claim | Test | |
|---|---|---|
| H1 | Indistinguishability. Expert raters cannot identify seat type above chance. | Equivalence against chance accuracy |
| H2 | Quality. Machine and machine-assisted orders are rated equal or better as written. | Non-inferiority |
| H3 | Tempo. Machine and machine-assisted seats close their decision cycle faster relative to the adversary who must react. | Superiority, reported only together with H4 |
| H4 | Effectiveness. Machine and machine-assisted seats contribute at least as much to campaign outcome as human seats in the same role. | Non-inferiority against a declared baseline |
Effectiveness is scored on the campaign’s outcome and each seat’s contribution to it, measured against what the historical commanders achieved from the same position: each game is an ersatz historical case study, and play departs from history on its own, so every game is also a counterfactual. Tempo is read only against effectiveness, because a faster decision cycle that loses the campaign is a negative result, not a mixed one.
In Searle’s thought experiment the person inside the room manipulates symbols he does not understand, and the observer outside is deceived. Here the positions are inverted: the umpire and the expert reviewers are the people in the room, and the A.I. is outside it, slipping its orders under the door. The umpire has to understand each order well enough to carry it out, blind to which seat wrote it, under rules that do not care who wrote it, and the score comes from the campaign’s outcome, never from winning the umpire’s adjudications. If an A.I.’s orders, understood and executed that way, win against a live opponent, the A.I. has shown strategic understanding, or at least acted as if it had it. That is a bold claim, and an empirical one: H4 can fail.
Advice is worth what it does when it is followed, and experts who do not trust advice will not follow it, however good it is. Indistinguishability and quality are measured alongside outcome for that pragmatist reason, not as proxies for it. Reviewers who do not know which seat is which rate the orders, judge their source, and say how they could tell, so the study can separate judgment about the argument from surface tells such as spelling or register.
The author’s prior is that the machine-assisted (centaur) seats will outperform both the human seats and the machine seats. Because all three arms play in the same games, the data can be read against every ordering, and each is stated before any data are seen:
| Ordering | Reading |
|---|---|
| Centaur > Machine > Human | The prior holds. The human adds judgment the machine lacks, and the machine alone still exceeds unaided human command. |
| Centaur > Human > Machine | The prior holds. The machine is useful as staff support and not in command. |
| Machine > Centaur > Human | The prior is refuted. The human in the loop costs performance, while the machine still lifts the human. |
| Machine > Human > Centaur | The prior is refuted. The pairing is worse than either partner alone. |
| Human > Centaur > Machine | The prior is refuted. The advice degrades the commander who takes it. |
| Human > Machine > Centaur | The prior is refuted, in the strongest form of the same reading. |
Because the assistant’s recommendation is held beside the order the human issued, a shortfall in the centaur arm can be traced to its cause: the human overriding a sound recommendation, diluting it, or delaying it, or accepting an unsound one that an unaided commander would have rejected.
The most interesting result would be A.I. advice that is consistently better than that of human seats and of centaur seats, that blinded reviewers cannot identify as machine-written, and that arrives at a significantly faster tempo. That result would refute the prior.
A simulated seat faces no danger and no exertion, so whatever the study measures, it measures generalship under uncertainty and chance only, and every claim carries that scope. Generalship is scored on two theories at once — Clausewitz’s psychology of command and Jomini’s objects of a campaign — and their divergence is treated as a measurement rather than as noise. The umpire is blind to seat type, adjudicates against pre-registered criteria, and gives an identity judgment only once, after the game. The intelligence each side receives is held outside the served application and reaches the umpire over the relay, and planted reports are checked so that they cannot be told from true ones on surface features alone. Below the power threshold, every claim is descriptive.