Why Passing Tests Don't Tell You How Your AI Agent Got There
You ask a coding agent to update an API endpoint. The repository already has a decision about backward compatibility: keep the existing response fields. You want the implementation, a contract test, and a short migration note. You leave the commit to yourself.
The final diff looks good. The test passes. The migration note lands in the right place. But you also pointed the agent to the current decision after it started from an obsolete one, rejected a breaking field rename, approved the test command, and reminded it to save the migration note. If you score only the final diff, several different things collapse into "success": repairs to delegated work, a permission decision, and a reminder to save the result.
Graham, writing at Automated Teach, argues that this is the core evaluation problem with agentic coding tools. The method he proposes is to keep three kinds of evidence separate: what the agent did, what it left behind, and what you had to fix along the way.
Not Every Human Action Is an Autonomy Failure
Graham breaks human interventions into five categories. A decision deliberately left to a person, like choosing whether to accept a breaking API change, is different from a repair to delegated work, like stopping an unauthorized field rename. A planned review of a diff before merging is different from help with the execution environment, like approving a test command the agent could not run unattended. And a repair to saving or handoff, like asking for a migration note that appeared only in chat, points to a workflow problem rather than an agent capability problem.
The distinction matters because experienced engineers make shaky workflows look good. You catch the old decision, recognize the breaking change, and ask the question that gets the agent unstuck. Because those interventions come naturally, you may barely count them as work. But if you are evaluating whether a workflow can run without you, every corrective intervention is evidence that the delegation is not complete.
A single intervention can serve more than one role. The point is to avoid treating every human action as an autonomy failure, or treating every rescue as ordinary review.
Follow One Run All the Way Through
Graham illustrates the method with a synthetic but realistic example. The task is to add an optional display_name field to a profile response while retaining name. Before editing, the agent must consult a specific ADR that records the compatibility decision. It must update the contract test, save a migration note, and leave commits to the engineer.
The visible record shows the agent reading an obsolete version of the ADR, editing the file to replace name with display_name, and then the engineer intervening to point it to the correct document. The agent re-reads, re-edits, updates the test, requests permission for the test command, and the engineer approves it as planned. The test passes. The agent reports completion in chat but the migration note file is absent. The engineer asks for it, and the agent writes it.
When you map this against the requirements, each one tells a different story. The agent eventually preserved the existing field, but only after correction. The test ran and passed, but the engineer approved it as planned. The migration note was saved, but only after a reminder. The final result is perfectly usable. The record still supports the conclusion that the task was completed with corrective help and one planned permission approval. It does not show that the workflow can satisfy those delegated requirements without that help.
Three Sources of Evidence, Three Different Jobs
The starting materials tell you which task you tested. Keep the request, relevant files, and starting revision. If you change the prompt or supplied context during recovery, record the change, because the repaired run answers a different question from the unassisted first attempt.
The activity record tells you what you can observe about execution. Tool calls, visible approvals, errors, and human messages establish particular events and their order. They do not give you an all-seeing account of the runtime. An absent read event is especially easy to overinterpret. The file might already have been in context, the event record might be incomplete, or the read might genuinely have been skipped. If you cannot distinguish those possibilities, retain the uncertainty.
The final files tell you what survived. Equal before-and-after bytes do not prove that nothing happened between the snapshots. A file could have been changed and restored. An agent can describe a correct result without writing it anywhere the next task will find it. If a reviewer approves a particular diff and the agent then changes it, the earlier approval applies to the earlier candidate. Record a revision or content hash so you can tell whether you are still looking at the same work.
The Template and What It Forces You to Say
Graham proposes a small record template that captures the question, starting revision, request and context, environment, requirements, planned human steps, delegated requirements, first attempt boundary, event record, interventions, final candidate, and verdicts. Choosing both the boundary and the division of responsibility before the run prevents you from quietly redefining a rescue as expected assistance.
Before treating a failed requirement as an agent failure, check the test itself. Was the ADR actually supplied or retrievable? Does the contract test exercise the old response field, or does it merely pass because the new implementation and new test share the same mistaken assumption? A precise record of an invalid test is still an invalid test.
One run can reveal a failure path worth repairing. It does not estimate a reliability rate. And if you revise the workflow repeatedly, the eventual success shows what the revised arrangement achieved. It does not isolate which change caused the improvement.
How This Shaped the Work Ledger
The method informed the design of Graham's Work Ledger, a harness-neutral agentic context management file system. The Ledger gives the next task a route to the current API decision, its source, the record of the attempted change, and the resulting files. Those roles correspond to specific directories. For selected consequential conclusions, a tracer component records the support, limits, and reasons to revisit them.
The practical question Graham frames is whether the test record helps you distinguish a workflow you can repeat from a result you managed to rescue. For anyone building or evaluating agentic tools, that is the right question to start with.