What we've learned running nomArmy #

Findings from real runs, kept here so the README can stay a guide. Each links to the experiment write-up it came from where there is one.

Measured in our own use: about 1 in 11 "done" claims wasn't true #

From September 18 to October 4, 2026, we ran nomArmy on one machine, on nomArmy itself and our own products: 547 jobs (387 implement, 139 scout, 19 verify, 2 decompose), on local models, Codex, Claude and Grok. nomarmy stats --all-repos reports, from nomArmy's verified job records rather than the workers' own reports:

Implement jobs whose worker reported "done, tests pass"265
Failed when nomArmy ran the verification profile itself in a fresh sandbox16
Passed, but the tests still passed with the production change reverted, so they proved nothing8
Passed verification, and the revert check either failed as it should or didn't apply236
Nothing nomArmy could run (verification not_run): neither confirmed nor caught5
Passed both, but still flagged by another check (mutants, Jev, the judge, a rewritten check)11
New test files, in committed jobs, shown to fail with their production change reverted140
High-stakes jobs committed / with a finished independent review38 / 33

24 of 265 claims (9%, about 1 in 11) were not true. That is the number the website quotes.

How each number is counted.

What this is and isn't. It is one team's use, on one machine, over 16 days, mostly building a developer tool and a data product. It is not a benchmark, and the rate will vary with the models, the briefs and the repository's tests. 223 jobs didn't finish cleanly (partial, failed, held for review, timed out); many of those were early local-model runs and sandbox limits whose work a later job finished, and a timed-out job's partial work is never committed. Five of the 38 high-stakes jobs were committed without a finished independent review and are listed by stats until they get one.

To see the same numbers for your own repositories: nomarmy stats (this repository) or nomarmy stats --all-repos, with --details for the breakdown and --share for a pull-request block.

The economics depend on task size, not model choice #

nomArmy's bet is to spend scarce frontier intelligence on intent and judgment, and abundant worker intelligence on implementation and repair. Measured so far (model bake-off and economics), whether that pays depends on the task:

Local also means the model behind the seat improves with your hardware, with nothing else in the harness changing.

Writing the evidence is the diagnosis #

Writing a job's evidence thoroughly enough to hand a worker every fact it needs is the diagnosis. If what's left after that is a small, mechanical change, you've already paid the cost delegation exists to save. A ticket that's part diagnosed fix and part bulk, mechanically verified work (writing a batch of tests, say) usually splits better than it delegates whole: keep the fix, hand off only the part that's nom-shaped.

Briefing errors look like model limits #

Two findings from one adversarial night of runs, recounted after separating a harness bug (since fixed) from what the models did:

The night first scored 2 clean jobs of 7. After excluding 2 jobs an idle-diff harness bug had killed, it was 5 clean of 5, with per-job corrections trending to zero as the brief improved, not the model.

Reading a diff isn't verifying it #

A local 20B worker was asked to add a doctor command with tests. It produced 156 lines that read as competent (JSDoc throughout, clean structure) with six defects invisible without running it, including a file that didn't parse and no test file despite an explicit acceptance criterion. nomArmy committed nothing; the record showed tests added: 0, taken from the repository, not the worker's claim. Reading the diff would plausibly have approved it. Running it didn't.

Frontier workers, in three real /feature runs #

About 18 implement jobs across three runs on a real product repo, with a Claude Code General and Codex, Claude, Grok and Muse workers:

More workers isn't automatically faster #

On one Apple Silicon machine, going from 1 to 4 parallel local workers produced no net throughput gain. Raising local worker count is an empirical question: measure accepted tickets per hour before assuming more is faster.