Checking the check
The verifier step got five changes in one afternoon, taken from what the field was arguing about that week. Testing and reviewing it found three defects the same day. Two models, same catch rate, zero false positives.
The verifier that caught the verifier describes a step: before a docs commit, a second agent with no memory of the work tries to disprove every claim in it. The same day it was published, we read the discussion of exactly this problem; a dev.to post from the week before had put it in one phrase: an agent can be made verifiable, not self-verifying. Reading that discussion against our step produced five changes. Testing and reviewing the changed step produced three findings about the step itself. This note is the numbers.
What the field said
The sharpest criticisms, applied to our step as it was that morning: the suspect writes the claim list, so a claim the first agent leaves out is never checked; a verdict names no source, so a check against a cached copy counts the same as one against the host; the seeded test ran once, and a verifier that reports zero mismatches for a month looks the same as one that stopped looking; a skipped check leaves no record; and a verifier handed convenient evidence may read that instead of the source; Goodhart Labs showed in September that frontier models still take a shortcut when an eval offers one.
Five changes
The verifier now extracts its own claim list from the files before reading the author’s; anything the author left out is
flagged [UNLISTED] and checked anyway. Every verdict names what it read and where, and whether that was the source or
a copy; no named source means unverified, never verified. The commit message carries a Verify-Claims: trailer with the
counts, or says the check was skipped. A seeded note and a clean note are kept as a canary and re-run whenever the step
or the model changes, with the result logged. And the limits are written down: a fresh context of the same model family
catches slips, not shared habits.
Checking the check
The canary ran the same afternoon, on two models, verifiers not told that anything was planted.
| Opus 5.5 | Fable 5.1 | |
|---|---|---|
| Seeded note, 3 planted errors | 3 of 3 caught, 31 s, 4 tool calls | 3 of 3 caught, 51 s, 3 tool calls |
| Clean note | 0 false positives, 24 s, 2 calls | 0 false positives, 46 s, 2 calls |
Same catch rate, no false positives. The planted errors were a line number off by one, a commit time off by five minutes, and one file’s line count changed under a total that was left correct, so the parts no longer summed; the second model noticed the arithmetic did not match before it read a single git object.
Three defects, found by testing and review
The first canary decayed within hours. Its claims were about live server state, and one of them (which commit a checkout was at) was true for sixty-four minutes that morning. By the afternoon a correct verifier reported it as a mismatch, and it was: real drift, not a planted error. A second session replaced the fixtures with claims about fixed git objects only, where the right answer never changes. On those, the slower model’s time fell from eight or nine minutes per note to under a minute.
The commit trailer was not a trailer. Git reads trailers only from the last paragraph of a commit message, and the
Verify-Claims: line sat in the paragraph above it, so git log --format='%(trailers:…)' returned nothing. The rule
had not said where the line must go. It came out in the owner’s review with that second session, and the rule now names
the command that checks it.
And the new UNLISTED label was written as a fourth verdict, next to verified, mismatch and unverified. A claim the
author left out was then reported as unlisted and nothing else: found, but never judged true or false. The same review
made it a flag on top of one of the three verdicts, counted in both totals.
One more slip, not caught by any rule: a verifier printed part of a session token into its own tool output while checking an API. It reached no report and no person, and the rules now say never to print an authentication response, even there.
What this buys
A step that is checked is different from a step that is trusted. The canary costs under a minute per note and gives one number the method notes could not give before: the step still catches what it is meant to catch, on the model in use today, and does not cry wolf. The next re-run is due when the step or the model changes, and in a month regardless.