Harness Engineering: Why Better Models Aren't Enough to Ship AI Agents
That gap is "harness engineering," and it's where the real work of building production-grade agents happens
This should sound familiar: folks take the latest model release, write a prompt, and run it. In a demo, it works beautifully. Then they try to ship it. Suddenly the agent drifts. It forgets context. It makes decisions that seem reasonable in isolation but pile up into broken workflows. The model itself works fine. The problem isn't the model, it's everything around it.
That gap is "harness engineering," and it's where the real work of building production-grade agents happens.
The Ceiling vs. the Foundation
Better models raise the ceiling. Claude 3.5 Sonnet reasons differently than Claude 3 Opus. GPT-4o tracks longer contexts than earlier versions. These leaps matter, and they will keep mattering. But raising the ceiling doesn't lower the floor. A powerful model still needs structure to run reliably at scale.
Production-grade agents fail more often from missing systems scaffolding than from raw model capability. I've seen this at Box, where we integrated AI into document handling pipelines—it's not the inference that breaks, it's the orchestration. It's the forgotten state. It's the context that got too long. It's the tool that failed and the agent didn't know what to do next.
Harness engineering frames the problem clearly: you're not just designing prompts anymore. You're designing the entire runtime around the model. The prompt structure, tool access, execution loops, guardrails, observability, and recovery paths. The agent is one component in that system. Everything else is infrastructure.
Context as First-Class Engineering
Here's a concrete example. Long-running agents need to solve a problem that's almost invisible in chat applications: they need explicit mechanisms to track progress, reduce drift, and manage context over many steps.
I watched teams build agents that would execute ten, twenty steps toward a goal. By step seven, the model had seen so much information that it started losing the thread. Context windows are finite. Even with 200k-token models, a long-running agent can accumulate enough tool output, error logs, and intermediate state to become unwieldy.
The solution isn't a bigger window. It's engineering. You write down state. You explicitly select what to show the model on each step—not everything, just what's relevant to the next decision. You compress when needed. You isolate subtask context so the agent thinks clearly about one problem at a time, not seven.
This is why context management has become a first-class engineering problem, not an afterthought. Teams that treat it that way—that invest in state tracking, context selection, and compression—move past the demo phase. Teams that don't get stuck fighting drift.
Observability and Iteration
The other half of harness engineering is observability. You can't improve what you don't measure, and you can't debug what you can't see.
This is where execution traces become essential. Not just logs—structured traces that show what the model decided, what tool it called, what the tool returned, and how the model responded. You can replay a trace, understand where the agent went off track, and figure out whether it was a bad decision, a bad tool, or bad context.
And that trace becomes your evaluation data. You don't evaluate agents the way you evaluate LLMs—on perplexity or benchmark scores. You evaluate them on their ability to solve real tasks end-to-end. That means building targeted evaluations: synthetic scenarios where you know what success looks like, where you can measure how often the agent succeeds, fails, or fails in recoverable ways.
These evaluations are how teams iterate from "demo" to "reliable system." They show you that a small change in prompt structure reduced hallucination. That adding a validation step before tool calls cut error rates in half. That restructuring how you pass context to the agent improved its ability to handle long workflows. These are the kinds of improvements that matter in production.
The Safety Surface
Harness engineering also expands the safety surface. When an agent can only reason and chat, safety is mostly about output filtering. When an agent can call tools, retrieve data, modify systems, the safety problem gets much harder.
Better harnesses give you more places to enforce safety. You can validate tool calls before they execute. You can sandbox execution. You can add approval steps for certain classes of action. You can log everything so you can audit what happened and why. You can rate-limit tool access. You can isolate agent state so one bad step doesn't cascade across a system.
None of this replaces training better models. But it changes what you're asking the model to do. A well-designed harness constrains the action space so the model is making decisions within bounds you've explicitly set. The model still needs to be reliable, but you're not asking it to be your only safety mechanism.
What Production Looks Like
When I talk to teams shipping agents at scale, teams moving past prototypes into real operational workflows, they're not talking about model size or training updates. They're talking about how they structure execution. How they manage state. How they get visibility into what the agent is doing. How they built guardrails that are specific to their domain and their risks.
That's harness engineering. It's less glamorous than model training, but it's where the real engineering happens.
