AI agents fail in production because teams skip the boring parts
You can't eliminate non-determinism in LLMs, and you shouldn't try. The goal is management, not elimination.
LLMs are unpredictable. That's what makes them useful, and that's what makes them dangerous in production systems. Most teams respond by trying to force determinism—cranking temperature to zero, over-constraining prompts, treating variation as the enemy. This misses the point. The problem isn't non-determinism. It's that teams deploy AI agents without the basic engineering controls that would catch errors before they compound. Step-wise evaluation and comprehensive logging aren't optional—they're the minimum framework for moving AI from prototype to production. Based on "Evaluating and Debugging Non-Deterministic AI Agents" by Aja Hammerly and Jason Davenport at Google Cloud Tech, this analysis examines why standard software engineering practices matter more than model capabilities when AI systems face real-world operations.
The Core Mechanism for Reliable Agentic Systems: Establishing a clear, repeatable mechanism to control and audit non-deterministic AI agents is a strategic imperative. Such a framework is the essential bridge that allows AI systems to move from experimental prototypes to reliable, production-grade applications that can be trusted with critical business functions. This requires shifting the organization's focus from the fact of non-determinism to the quality and reasonableness of the agent's outputs.
If you want AI agents in production, you need controls. Not aspirational ones—actual mechanisms that catch errors, create audit trails, and let you debug when things go wrong. The shift here is from worrying about non-determinism itself to ensuring the outputs are reasonable and meet your quality standards.
Analyzing the Core Problem: Determinism vs. Reasonableness A critical analysis reveals the primary business concern is rarely non-determinism itself, but rather the variation in response quality it can produce. Most business problems don't actually require determinism. Most applications do not require perfectly identical, deterministic outputs; they require outputs that are consistently reasonable and meet a defined quality standard.
A common but counterproductive tactical approach is to lower the model's "temperature" setting to zero. While this makes responses more predictable, it is a counterproductive solution that introduces its own business risks. response: cranking temperature to zero. Sure, you get predictability, but you also get boring, repetitive outputs that defeat the purpose of using generative AI in the first place. Lowering the temperature stifles the model's creativity, leading to repetitive and "boring" outputs that diminish the unique value of generative AI. The strategic goal must not be to force determinism but to architect a system that guarantees reasonableness.
The Step-Wise Evaluation Process
The primary control method for ensuring quality is the integration of integrating evaluation into every step of an agentic workflow. By deconstructing a complex task into a sequence of smaller actions, an evaluation can be inserted after each one to verify correct execution before proceeding. This prevents errors or hallucinations in early steps from propagating and corrupting the entire flow.
Consider an agent designed to make restaurant reservations. Its workflow must be deconstructed breaks down into distinct steps, each followed by a rigorous evaluation:
Information Extraction: The agent asks the user for necessary information (e.g., party size, dietary preferences) and extracts those details. An evaluator then checks the conversation context and the info extracted to confirm it was captured correctly and completely.
Tool Use: The agent uses the extracted information to query a reservations API. The evaluation must then verify verifies that the information being sent to the reservation tool is valid and that the data that's coming back from the tool is also appropriate.
User Presentation: The agent presents the available options to the user for selection.
Action Execution: After the user makes a selection, the agent uses the tool again to make the final reservation. A final evaluation confirms that the booking was completed successfully.
Protocols for Responding to Evaluation Failures
When an evaluation identifies an issue, the system must execute a predefined escalation framework. This tiered response system provides a structured approach to error handling, moving needs a plan. Move from low-cost automated retries to high-cost human intervention based on the failure's severity. The options include:
Automated Retry: For simple or transient errors, the agent can restart the entire workflow.
Structured Error: The system can halt the process and return a logical, informative error message to the user explaining that the task could not be completed.
Algorithmic Correction: The system can employ secondary AI models or algorithmic approaches to correct the identified error and allow the workflow to continue.
Human-in-the-Loop Escalation: For complex failures, the issue must be escalated goes to a human operator for manual review and resolution.
This human-in-the-loop protocol is not a novel AI concept but a direct parallel to the established software engineering practice of handling code merges. When automated systems cannot resolve a merge conflict, they escalate the issue to a human developer. The same logic must be applies to applied to agentic systems.
The Debugging Framework: Comprehensive Logging Effective debugging of a "complex, non-deterministic black box" is impossible when its internal state is unknown. Comprehensive logging is the only solution that provides the necessary transparency. You can't debug what you can't see. If your agent is a black box, you're stuck. To enable effective post-mortem analysis and debugging, the following data points must be need to be logged at each stage of the agent's flow:
- The specific tools the agent is using.
- The parameters being sent to those tools.
- The data being returned by those tools.
This detailed log creates an immutable record of the agent's actions and the context in which they were made, making it possible to trace the source of any error. This technical mechanism provides the foundational data required for operational oversight and compliance.
System Design Imperatives for Product and Engineering
For product and engineering leaders, these principles for managing AI non-determinism translate directly into foundational system design mandates. This is not a new AI-specific challenge but an application of robust software engineering discipline to a new class of component.
For product and engineering teams, this isn't about inventing new practices. It's about applying software engineering fundamentals to a new component type.
Integrating Controls into the Application Architecture
The mandate for product and engineering leaders is to architect applications under the non-negotiable assumption Assume that LLM outputs are inherently unreliable**, treating them with the same skepticism as any un-vetted third-party API.** —treat them like you'd treat an untrusted third-party API. Teams must should architect their agents and applications to systematically catch errors, fall back on predefined reasonable defaults (e.g., if an agent fails to generate a custom marketing email, send a standard, pre-approved template instead), and provide logical, informative messages to the user when a task cannot be completed.
Overcoming the "AI Is Different" Mindset A pervasive cognitive pitfall is for experienced engineers to abandon standard practices under the belief that "AI was is different." Core debugging skills—such as methodically checking logs—remain paramount. As the logging framework (1.4) and the code-merge analogy (1.3) demonstrate, success with AI agents relies on applying existing software design principles. Failure to dismantle this "AI is different" exceptionalism will lead to brittle systems, extended debugging cycles, and an inability to scale AI operations reliably.
This disciplined approach provides the verifiable data trails required for legal and compliance oversight.
Establishing Provenance for Legal and Compliance Teams
From a legal and compliance perspective, For legal and compliance, step-wise evaluation and comprehensive logging are not merely technical best practices; they are critical risk management controls. These practices are essential for mitigating liability, establishing a chain of custody for automated decisions, and satisfying regulatory burdens for explainability. They create a verifiable, auditable trail of an AI agent's actions, which is fundamental to demonstrating due diligence.
Creating Auditable Records of Agent Actions
The comprehensive logging framework serves as the system of record for an agent's decision-making. The logs containing the tools, parameters, and outputs are not merely for debugging; they constitute the primary evidence for demonstrating that the AI agent operated within its prescribed boundaries. By generating this documentation at each step, the system creates a definitive record of "provenance." This allows teams to answer precisely what an agent did and why, a capability that is crucial for satisfying internal reviews, regulatory inquiries, and demonstrating responsible AI governance.
These technical controls directly enable the oversight and documentation that legal and compliance functions require to operate effectively.
From Non-Determinism to Reliable Operations
The fundamental mandate for productionizing AI is to shift from attempting to eliminate non-determinism to managing it with proven engineering discipline. This is the only defensible approach to transform AI components from unpredictable black boxes into reliable, auditable parts of a larger system. The first and most critical action for every team is to implement step-wise evaluation and comprehensive logging throughout all agentic workflows. Ultimately, success and compliance in the AI era will be defined not by inventing new techniques, but by the consistent and rigorous application of these "boring" but essential software principles.
You can't eliminate non-determinism in LLMs, and you shouldn't try. The goal is management, not elimination. Step-wise evaluation and comprehensive logging aren't optional nice-to-haves—they're the minimum controls needed to move AI from prototype to production. Get those right, and you've got a foundation for reliable, auditable AI systems. Skip them, and you're building on sand.
