A second agent that disproves claims before commit

Rule 1 of how I work with coding agents is that a conclusion leaves the session only after its cheapest disproof has run. The weak point is who runs it. An agent checking its own claim has the same context that produced the claim, and tends to re-read its own evidence rather than the source. So the check is now a separate step with a separate agent.

What the step does

Before a commit that adds factual claims, the working session lists every claim the change makes: figures, times, hashes, file and line numbers, host state, and especially derived numbers (totals, before/after pairs, percentages, with their parts). A fresh general-purpose agent gets that list, the file paths and read-only access, and is told to disprove each claim from the live source: re-run the command, re-read the file on the host, re-query the API. Not the diff, not the first agent’s evidence. It returns one line per claim: verified, mismatch with the true value, or unverified with the reason. The gate is simple: zero mismatches or no commit. The verifier never edits, commits or pushes.

Seeded test first

Two ten-claim notes about live host and API state, the verifier not told that anything was planted. The seeded copy had a wrong line number, a wrong journal time, and a derived pair that summed to 9 instead of 8. Result: 3 of 3 caught, 0 false positives, the true values named, in about 79 seconds and 10 tool calls. The clean copy: 10 checked, 0 mismatches, about 77 seconds.

First live run

The change under test was a morning’s docs work on this site, 27 claims. The verifier returned 1 mismatch and 1 unverified. The mismatch: the table in the Hacker News hiring note took shares that the data file already stores with one decimal and rounded them again, so six cells were a point off. May’s “agentic coding” was 8 of 322 posts, 2.48 %, shown as 3; July’s “AI agent” was 13.46 %, shown as 14. The two bullets built on those cells were wrong in the same direction.

The part that earns the title: earlier that morning the same session had sent those rounded figures to the research session as a correction of its draft, and the draft had been right all along. The verifier caught the correction. The research copy was reverted to the exact values within the hour, and the table now rounds count over n, once.

The other two findings

The other two findings from the same run were smaller and just as real: a deploy document that said “never move dist” next to a rollback recipe that moved onto it, and a systemd comment that mixed MB with MiB and so claimed the uncapped build had exceeded a cap it was actually under. Nothing was invented, and nothing was flagged that was true.

What it costs and what it does not cover

About 80 seconds and ten tool calls per docs commit. Not yet tested: claims about how code behaves, and claims that would need write access to check. A claim about the past that cannot be re-observed is marked unverified, never verified; that label goes into the document with how to check it later.

Two limits are worth saying out loud. The verifier is a fresh context, not a different mind: it is usually the same model family, so it catches slips (a stale number, a wrong line) far better than shared habits (both agents rounding the same way, both reading MB as MiB). And a verifier handed convenient evidence will read that instead of the source; Goodhart Labs showed in September that frontier models take the shortcut they are given. So the verifier gets the artifact and the claim list only, never the first session’s commands, extracts its own list from the files before reading mine, and every verdict has to name what it read and where. A verdict with no named source counts as unverified. The check itself is checked: a seeded note with three known errors and a clean note are re-run whenever the step or the model changes, and the result is logged.

← all notes