Review and verify
9 min read
By the end you can read an agent's diff in the right order, say how much a green test suite proves, and set the controls that keep verify honest.
TL;DR
- Review intent and code quality. Your normal human review remains; add agent reviewers where the risk and measured benefit justify them.
- A clean agent diff can still fail at intent, so read it from the outside in: the shape of the change, then the tests, then the lines.
- A green suite the agent wrote is evidence, not proof. Check its assertions against intent and protect the gates from being weakened to pass.
Who checks what
Cover both review lenses and the behavior; the number of reviewers can vary.
| Check | Reads | Asks |
|---|---|---|
| Intent and spec lens | Request, relevant decisions, diff | Does this match the required business behavior? |
| Code-quality lens | Diff and relevant surrounding code | Is this correct and maintainable by the team's standards? |
| Human read | Diff-stat, tests, boundaries, new names | Is the business rule right? Does it fit the architecture? |
| Verify | Checks suited to the change and its risk | Does the behavior meet the agreed outcome? |
One human review can cover both lenses. For risky or unattended work, a fresh agent can find gaps before that review. Separate reviewers need distinct questions and enough context to judge the change; two sequential agents are an option, not a requirement.
The wire, continued
A suite that is happy can still be wrong.
Chapter 4 left the audit-log emitter with a test that was green and incomplete. Review found that the assertion omitted the newly required priority, and asked for a documented migration recovery approach. The latter is a project decision, not a universal requirement for a down migration. The implementer addressed the findings and reran the relevant checks.
In the synthetic companion, the old test checks that one wire.created event exists and names the wire:
self.assertEqual(len(service.audit_events), 1)
self.assertEqual(service.audit_events[0]["type"], "wire.created")
self.assertEqual(service.audit_events[0]["wire_id"], wire.wire_id)
Before reading on, ask: which new acceptance criterion could still fail while these checks pass?
The event can omit priority entirely. The stronger assertion observes the chosen value:
self.assertEqual(event.get("priority"), "URGENT")
It fails on the defective implementation with None != 'URGENT'; including the normalized priority repairs it. Testing all four values also excludes a hard-coded URGENT fix. Run python3 verify.py from the companion pack to observe weak-green, corrected-red and repaired-green. That model experiment does not establish the original banking system's behavior.
Back in the original banking engagement described here, verification included the full suite of 3,400 tests, a smoke test against the staging compliance-routing service and a diff of the API contract change for the regulator's review. The checks passed, and that evidence went into the pull request at Chapter 4's ship step.
Review: choose the depth¶
That was the banking case's design, not the default for every change.
Start with the team's normal review and relevant automated checks. Add an independent agent where the change's consequences, unfamiliarity or unattended execution warrant it. Use two distinct reviewers only when their different lenses find useful defects worth the extra time. Model behavior matters: Anthropic recommends fresh verifiers for Fable 5 long tasks, but warns that extra verifier scaffolding makes Opus 5 over-verify (Appendix D).
- Scope findings to the task and real defects. Broad requests to improve everything can turn review into repeated polishing and new abstractions.
- Keep separate judgments independent. Give a fresh reviewer the requirements, diff and relevant code, without the implementer's defense or an earlier reviewer's verdict.
- Accept or rebut findings with evidence. Track useful defects caught, false positives, review time and rework. A high rate of responding to comments does not show that the review improved the change.
- Bound repair by value. Fix supported findings, then run the affected checks. Stop when no material issue remains; another round needs a new concern, changed code or other evidence.
Findings carry a severity: critical blocks ship, important is fixed before ship, suggestions go in the PR description for the human reviewer.
How do you review an agent-written diff?¶
Read from the outside in: intent first, lines last.
Any diff can contain a typo, an off-by-one or a forgotten edge case. An agent diff can also compile, pass lint and read like code your team would merge while failing at intent and context: a business rule that is plausible and wrong, the right code in the wrong layer, the code a good engineer would write for a slightly different task. Fluency is not correctness.
Start above the code, with time proportionate to the change:
- Diff-stat against the request or plan. Does the shape of the change match the ask? An unexpected file deserves an explanation; it can be a necessary discovery or scope drift.
- Read the tests for what they assert, not whether they pass. Green is the cheapest signal, and the one you already have.
- Check the boundaries your team wrote down: forbidden patterns in AGENTS.md, layer conventions, dangerous modules. The agent crosses them confidently, in fluent style, so a violation looks clean.
- Grep every new name the diff introduces that you do not recognize. The call that does not exist reads as plausibly as the one that does (Chapter 6 has the cross-check). For a new dependency, confirm the exact package and publisher, its license and advisories and what it runs at install, and read the manifest and lockfile together; an existing approved dependency may already do the job.
- Only now, line by line, for business correctness and architectural fit.
For consequential or unfamiliar changes, an optional comprehension check can help: ask the agent to quiz you on edge cases, ownership and what breaks if an assumption is wrong. An unanswered question identifies something to resolve before approval. Use it when normal review has not already established that understanding. Appendix B.8 carries the read order and this optional check.
Verify
Verify measures the behavior against the outcome the plan defined.
Verify asks whether the intended behavior works and what evidence supports that conclusion. Run the relevant tests, build, lint, behavior or visual checks. Add a regression test for a behavioral fix where feasible, using the failure case rather than copying the implementation into the assertion. Existing valid test output can carry forward when no relevant code or assumptions changed; a new phase name is not a reason to repeat a suite.
A reversible prose or configuration edit does not automatically need a permanent test file. Check the changed content, parser, build or rendered result as appropriate. A reversible change to permissions or money handling can still need substantial behavioral testing: reversibility is not a measure of bug risk.
For frontend behavior, prefer user-facing roles and labels over brittle CSS selectors, assert the resulting behavior and inspect rendered visual changes; semantic locators do not assess layout. Appendix D covers control-specific interactions.
Can you trust the tests the agent writes?¶
Green tests are evidence, not proof.
They are evidence the change does what the test says, not that the test says the right thing, as the audit-log test showed. Coverage does not close the gap: it measures which lines ran, not whether anything meaningful was checked, and an agent optimizing a coverage gate writes tests that call the code and assert nothing of consequence. Characterization tests, Chapter 8's first fix for an untested codebase, lock in current behavior, not correctness, and preserve a bug as faithfully as a feature: a regression net while you migrate, never proof.
So review the agent's tests like its code, checking the assertions against the spec. A test written from the implementation agrees with the implementation; the assertion has to come from the intent.
The agent also optimizes the evidence it can see. On SpecBench, frontier agents saturated the visible tests while held-out scores stayed far lower; for Claude Code the gap was roughly 43 to 48 points across three search strategies. That is a warning about what the gate rewards, not a requirement to turn every PR into a benchmark. Apply the controls at the right boundary:
- For ordinary changes: review assertions against intent and diagnose failures before changing code or checks. A wrong test can need a legitimate, reviewed update; changing the assertion just to get green needs rejection.
- Protect the team's gates: keep required CI and security checks outside the agent's unilateral control. Hooks can block bypasses and require review for test or lint configuration changes while allowing authorized maintenance.
- For evaluations or high-assurance work: independently written held-out tests can detect overfitting. Sealing answer-bearing git history and network access can prevent benchmark leakage. Ordinary development often needs both history and documentation; restrict them only when that boundary serves the task.
When the toolkit does the looking
Review agents are a filter in front of human judgment, never a substitute for it.
A team I worked with installed a full review toolkit: agents for swallowed exceptions, weak tests, security issues and stale docs. It caught real bugs week after week, the senior reviewers learned to trust the green checkmarks, and within a quarter their fifteen-minute reviews had become three-minute scans. Nobody decided that; attention drifts away from work that appears to be done.
Then a PR that every agent had passed - tests green, no security findings, clean diff - applied a discount rule to the wrong customer tier and ran in production for weeks until a support ticket surfaced it. The toolkit had worked as designed: it removed the mechanical findings from human review, and the judgment drifted away with them. That is Chapter 9's uncalibrated delegator, at team scale.
The fix was a judgment floor, not a duration quota. Before approval, the reviewer should be able to explain the business rule, the consequential edge case, why the change belongs at this architectural boundary, and which evidence would reveal the most damaging plausible mistake. Review time and escaped defects are useful diagnostics. A short review can answer these questions well; a long review can still miss them.
When it breaks
- The read decays into a glance at checkmarks. Review time and escaped defects can expose the drift; an optional comprehension check can reveal gaps before approval.
Do this Monday
- [ ] Read your next agent diff in the order above. For an unfamiliar or consequential change, try the optional quiz and resolve what you could not answer.
- [ ] Open the last three tests the agent wrote. Do the assertions come from the spec or from the code?
- [ ] Check both intent and code quality. Try a fresh reviewer on a consequential change, then record useful findings, false positives and time before making it a default.
- [ ] Diagnose a failing check before changing it. Confirm required CI cannot be bypassed, and that legitimate check updates have a review path.
- [ ] Confirm the review answers the business-rule, consequential-edge, architectural-boundary and damaging-mistake questions. Track time as a diagnostic alongside escaped defects, not a quality quota.
Go deeper
- Templates: Appendix B.8, the agent-diff read order; B.3, the loop checklist.
- Deep dive: Appendix D, Chapter 5 - the review measurements, SpecBench and Cursor in full, and the accessibility-tree caveat.
- Sources: Appendix C.