AI agents in production: what actually works vs what sounds good in demos.

Every AI agent demo looks compelling. A freeform customer message comes in, the agent reads it, extracts intent, checks availability, and sends a perfectly formatted response, in seconds, with apparent understanding. The demo succeeds because demos are designed to succeed: the input is carefully chosen, the data is clean, and nobody sends a message at 3am with a typo, a reference to a policy that changed last month, and a sentiment that mixes frustration with urgency in equal measure.

Production is different. Production sends you the message at 3am. Production sends you the edge case that the prompt engineer did not anticipate. Production sends you the input that is technically valid but semantically ambiguous, where the 'right' answer depends on context that the agent does not have access to. Building AI agents that survive contact with production requires a fundamentally different engineering discipline than building AI agents that look good in demos.

The first production requirement is structured output enforcement. Every Claude AI call in a PURIST system uses a defined JSON schema as the expected output format, enforced via Anthropic's tool-use feature. Rather than asking Claude to 'respond with JSON,' we define a function schema with required fields, field types, and constraints. Claude returns a tool_use response that is structurally guaranteed to match the schema. This eliminates an entire class of downstream parsing errors, the malformed JSON, the missing field, the hallucinated key name, that plague systems relying on free-text JSON generation.

The second requirement is retrieval architecture. AI agents that answer questions about your specific business, your policies, your pricing, your client history, need access to that information at inference time. The most common failure mode in production AI agents is confidently answering a question with outdated or generic information because the model's training data does not include your specific context. We build RAG pipelines for every agent that answers policy or product questions: a vector database populated with the client's documentation, queried at inference time, with the retrieved chunks included in the system prompt context. The agent cannot invent what it does not know, but it can accurately relay what you have told it.

The third requirement is confidence scoring and escalation routing. We instrument every production AI agent with a confidence assessment at the end of the inference call. Claude is asked, as part of its structured response, to score its confidence in the output and to flag any aspects of the input that were ambiguous or outside its knowledge. Responses below a threshold, we typically use 0.80 for informational agents and 0.90 for agents that trigger downstream actions, are routed to a human review queue rather than auto-sent. This is not a limitation of the technology. It is responsible engineering. A system that knows when to ask for help is more valuable than one that answers confidently and incorrectly.

The fourth requirement is continuous evaluation. We build evaluation pipelines alongside every AI agent, a set of test inputs with known-good expected outputs, run automatically every time the agent's prompt or retrieval configuration changes. This prevents silent regression: the prompt tweak that improves performance on one scenario while degrading it on three others. Production AI agents are living systems. They require the same testing discipline as production software.