Standard model evals aren't built for agents. They were designed for single-turn calls: give the model a prompt, score the response. Deep agents don't work that way. They plan, branch, call tools, retry, sometimes reverse course. A single trajectory can pass through five subagents and twenty tool calls before producing anything you can grade.
When you only score final outputs, you miss everything about how the agent got there. Two agents can produce the same answer, one through a sound chain of reasoning, one by guessing and getting lucky. You'll grade them identically and learn nothing.
For agents, the trajectory is the artifact. The eval has to look at the path, not the endpoint alone. The questions that matter: did the agent decompose the task sensibly, did it select the right tools, and when it got conflicting information, did it stop, retry, or hallucinate through it. Those require step-level scoring.
There's a governance dimension engineers often miss. When something goes wrong with an agent in production, and something will, what regulators and counsel ask is whether you can show the system behaved reasonably. Your eval suite is the closest thing you have to a record of due care.
For product counsel and AI governance leads, that changes how you should review agent rollouts. The review should cover what your eval catches that a simple completion benchmark would miss, whether you can replay any failed run and inspect the trajectory, and what triggers the rollback when the eval flags a regression.
Those points are the audit trail your future incident response will depend on.
