Can You Trust Your AI Employee? Observability and Evaluation for Agents in Production

You would not give a new employee write access to your CRM, your accounting system, and your customer inbox on day one without a probation period, a manager, and a paper trail. Yet that is exactly what many companies do with an AI agent the moment a vendor demo goes well. The agent goes live, it starts taking real actions in real systems, and then the uncomfortable question arrives at the next board meeting: how do we actually know it is doing the right thing?

This is not a hypothetical worry. In 2024, a Canadian court ruled that Air Canada was legally bound by a refund policy its own chatbot invented, a precedent that made every company liable for what its AI says²³. Industry estimates put the cost of AI hallucinations to business at tens of billions of euros a year, with a large share landing as operational cleanup and reputational damage²². And Gartner expects more than 40 percent of agentic AI projects to be cancelled by the end of 2027, citing unclear value and inadequate risk controls as primary causes¹.

The companies that will keep their agents in production are the ones that can see, measure, and prove what those agents do. That capability has a name: AI agent observability. This guide explains what it is, the four layers that make it work, how evaluation and drift detection turn a black box into a measurable colleague, and how the same instrumentation that earns trust also produces the audit trail the EU AI Act now expects.

The Trust Gap: Why a Good Demo Proves Nothing

A demo shows you one run that went well. Production is thousands of runs against messy, unpredictable inputs, and an AI agent behaves differently from any software your team has deployed before. The gap between “it worked in the demo” and “we can trust it in production” is where most agent projects quietly die.

• Agents are non-deterministic - The same input can produce different outputs on different runs. Standard software testing assumes deterministic output, so the tools your team already owns do not catch the failures that matter¹⁵.
• Failures are silent - A traditional app throws an error you can see. An agent confidently returns a wrong answer that looks perfectly plausible, and nobody notices until a customer or an auditor does²⁰.
• The blast radius is real actions - A chatbot that hallucinates wastes a reader’s time. An agent with write access that hallucinates posts a wrong invoice, sends a wrong customer reply, or updates the wrong record in a system of record.
• You are now liable - The Air Canada ruling established that a company is bound by what its AI tells a customer²³. Regulators and courts increasingly treat an AI’s output as the company’s output²⁴.
• The hidden checking cost - When teams do not trust an agent, they check its work by hand. Industry surveys estimate the average AI user spends over four hours a week verifying output, a tax that quietly erases the productivity the agent was meant to deliver²².
• Governance is immature - Gartner’s research finds that most enterprises still lack a mature model for governing agents, which is one reason so many deployments stall between pilot and production⁴.

The trust gap is not a reason to avoid agents. Gartner also projects that 40 percent of enterprise applications will include task-specific AI agents by the end of 2026, up from less than 5 percent a year earlier⁵. The agents are coming. The question is whether yours run on evidence or on hope.

What Agent Observability Actually Is

Observability is an old idea from engineering: a system is observable if you can understand its internal state from the outside. Applied to AI agents, it means you can reconstruct exactly what an agent decided, why, and what it did, from the records it leaves behind. The core unit is the trace.

A trace is the complete, structured record of one agent run, broken into nested spans. A top-level span covers the whole task. Child spans cover each model call, each tool invocation, and each retrieval step. Every span captures its inputs, outputs, timing, token usage, and cost, so an engineer can open a single run and see the entire decision tree at every step²⁵.

The standard that makes it portable

This is not a proprietary format anymore. In April 2024, OpenTelemetry, the open standard already used for application monitoring, formed a Generative AI group to define common conventions for agent telemetry. The result is a shared schema for prompts, model responses, token usage, tool calls, and agent orchestration⁶.

• One instrumentation, many tools - The OpenTelemetry GenAI semantic conventions let you instrument once and send traces to any compatible backend, rather than locking into a single vendor⁷.
• Native framework support - Agent frameworks such as LangChain, CrewAI, and AutoGen emit compliant spans natively or through instrumentation packages, so basic tracing can be live in days⁸.
• Standard span types - A run produces an invoke agent span at the top, child chat spans for each model call, and execute tool spans for each tool the agent uses⁸.
• Vendor adoption is broad - Major observability platforms including Datadog, IBM Instana, Honeycomb, and New Relic already support these conventions, correlating agent spans with the rest of your application monitoring⁹.
• Semantic, not just structural - The point is to capture not only “did it run” but “what did it decide and why”, which means recording the reasoning and the tool arguments, not just the final output²⁸.

What a trace lets you do

• Debug a specific complaint - A customer says the agent gave them a wrong answer. You open that exact run and see the prompt, the retrieved document that was out of date, and the tool call that failed.
• Find the pattern behind many failures - Traces aggregate. You can see that every failure last week shared one retrieval source or one tool, which points straight at the fix.
• Attribute cost - Each span carries token usage, so you can see which steps and which use cases actually drive your model spend²⁵.
• Prove what happened - A complete trace is the evidence you hand to an auditor, a regulator, or your own risk committee.

The Four Layers of Agent Observability

Observability is often reduced to “we log the traces”, but logging is only the first layer. A production agent you can genuinely trust rests on four layers, each answering a different question. Skip one and the trust breaks.

1. Traces - what happened - The structured record of every run: prompts, reasoning, tool calls, retrieved data, outputs, cost, and latency. This is the raw evidence everything else is built on.
2. Evaluation - was it good - Scoring those traces against a standard for accuracy, safety, policy adherence, and task completion, using a mix of deterministic checks and model-based judges.
3. Drift and alerts - is it still good - Watching the evaluation scores over time and raising an alert when accuracy falls, refusals rise, or behaviour creeps away from the baseline.
4. Governance and audit - can we prove it - Tamper-evident, retained records and access controls that let you answer a regulator, an auditor, or a customer, and that satisfy Article 12 of the EU AI Act for high-risk systems¹⁰.

Evaluation: Offline Suites and Online Judges

Traces tell you what happened. Evaluation tells you whether it was any good, at a scale no human review team could match. There are two halves, and a production agent needs both.

Offline evaluation: the regression gate

Offline evaluation runs your agent against a fixed dataset of test cases before any change ships. It is the equivalent of a test suite in normal software, and shipping an enterprise agent without one is an architectural anti-pattern²¹.

• Gates every change - Before a new prompt, model, or tool goes live, the suite runs and regressions are visible before the change merges¹⁶.
• Built from real failures - The strongest test sets are anchored in real production failures, not hand-invented cases, because they carry inspectable evidence of what actually breaks¹⁴.
• Accounts for non-determinism - A good offline suite runs each case several times, because an evaluation that ignores run-to-run variability will draw the wrong conclusion¹⁵.
• Grows every week - Each new failure caught in production is added as a case, so the suite compounds and the same failure never silently returns¹⁴.

Online evaluation: judging live traffic

Online evaluation scores a sample of real production runs as they happen and alerts you when quality drops, without adding latency to the user’s experience¹⁶.

• Samples live runs - You score a configurable percentage of production traffic in real time rather than waiting for a complaint¹⁸.
• Sets thresholds and alerts - When the live quality score crosses a threshold, the responsible owner is paged, the same way an on-call engineer is paged for a latency spike¹⁸.
• Closes the loop - Failures found online are labelled and fed back into the offline suite, so reality continuously improves your tests¹⁴.

How the scoring actually works

Three techniques do the judging, and mature setups combine all three rather than betting on one.

LLM-as-a-judge is what makes evaluation affordable, because one model can score another against clear criteria and return a structured verdict¹⁷. It is not magic. Research documents real biases: judges favour the first option they see, reward longer answers, and prefer output that resembles their own¹⁶. You make it trustworthy the same way you would a human grader.

Drift, Accuracy Alerts, and Regression

An agent that passed every test on launch day can quietly get worse. This gradual degradation is called drift, and because agents are non-deterministic, it is invisible without continuous evaluation. Drift is the single strongest argument for treating observability as ongoing operations, not a launch checkbox.

Where drift comes from

• Data drift - The real-world inputs shift away from what the agent was built and tested on, so accuracy falls even though nothing in the agent changed¹⁸.
• Model drift - The provider updates the underlying model. The same prompt now behaves differently, and outputs can vary across providers and versions in ways that break a workflow²¹.
• Agent drift - In long-running or multi-agent systems, accumulated context pulls the agent off task. Research simulating multi-agent systems found behavioural degradation could affect nearly half of long-running agents¹⁹.
• Prompt and tool drift - A change to a prompt template, a tool, or a retrieval source has knock-on effects that a single spot check will miss²⁰.
• Refusal and retry creep - A rising rate of refusals or retries is an early warning that behaviour is shifting, often before accuracy visibly drops¹⁸.

What good drift detection looks like

1. A baseline you measured - You cannot detect drift without a documented starting point. Capture accuracy, latency, refusal rate, and cost at launch as the reference¹⁵.
2. Continuous scoring - Online evaluation runs against live traffic so the trend lines are always current, not a quarterly audit¹⁶.
3. Thresholds tied to consequences - Alerts fire when a metric crosses a level that actually matters to the business, not on statistical noise.
4. A regression gate on every change - The offline suite blocks a change that drops any key metric, so you never ship a regression by accident²¹.
5. A rollback path - When drift is detected, you can revert to a known-good version or reduce the agent’s autonomy until the issue is fixed.

Governance and Audit Trails

The first three layers keep an agent working. The fourth layer lets you prove it, to a regulator, an auditor, a customer, or your own board. In the EU this is no longer optional for high-risk systems, and even where it is not mandatory, the same infrastructure is what turns “trust us” into “here is the record”.

What the EU AI Act requires

• Automatic logging - Article 12 requires high-risk AI systems to automatically record events across their lifecycle in a form that makes the system’s functioning traceable¹⁰.
• Full reconstructability - Logs must capture enough, timestamps, inputs, outputs, tools invoked, and human decisions, that an inspector can reconstruct what happened on any given run¹³.
• Retention - Deployers must keep the logs for at least six months, and longer for certain systems¹².
• Tamper-evidence - The records must be trustworthy, which in practice means they cannot be silently altered after the fact¹³.
• The deadline is set - Provider and deployer obligations for high-risk systems, including Article 12 record-keeping, apply from 2 August 2026¹¹.

Governance is not all-or-nothing

The most common governance mistake is treating every agent the same, either locking everything down until nothing ships, or trusting everything until something breaks. Gartner is blunt about where that leads.

• Match oversight to autonomy - Classify each agent by how much it can do on its own. A read-only agent that drafts text needs lighter oversight than one with write access to your ERP⁴.
• Human-in-the-loop for high-stakes actions - The riskiest actions get a human approval checkpoint; low-stakes ones run autonomously with monitoring.
• Guardian agents - A second AI system can evaluate and contain a working agent at runtime. Gartner expects that by 2028 many CIOs will demand guardian agents to track and oversee agent actions, and that they will become a distinct market segment².
• Clear ownership - Every agent has a named human owner accountable for its behaviour, the same as any other employee.
• An access and identity boundary - Each agent has its own identity and least-privilege access, so its actions are attributable and its blast radius is bounded. We cover this in depth in our guide to agent identity and access.

Governance and security are two sides of the same coin. Observability tells you what an agent did; access control limits what it can do. For the security half of the picture, see our guide to AI agent security and the OWASP LLM Top 10.

The 90-Day Instrumentation Playbook

You do not need to build all four layers before an agent goes live. You do need a plan that gets tracing in from day one and grows evaluation and governance as real runs accumulate. Here is a focused 90-day approach that mirrors how a disciplined deployment actually works.

Phase 1: Instrument and baseline (Weeks 1-4)

1. Week 1: Turn on tracing - Instrument the agent with OpenTelemetry GenAI conventions so every run produces a full trace from the first day it touches real data⁷.
2. Week 2: Define what good looks like - With the process owner, write down the task-specific success criteria: what counts as a correct, safe, on-policy outcome for this agent.
3. Week 3: Collect a baseline - Run the agent in a shadow or limited mode and capture accuracy, latency, refusal rate, and cost as the reference point for all future drift detection¹⁵.
4. Week 4: Seed the offline suite - Turn the first real runs, especially the failures, into an initial set of test cases¹⁴.

Phase 2: Evaluate and gate (Weeks 5-8)

5. Week 5-6: Build the judges - Add deterministic checks for anything with a correct form, and an LLM-as-a-judge for quality, calibrated against human-labelled examples¹⁶.
6. Week 7: Wire the regression gate - Make the offline suite run on every change so no prompt, model, or tool update ships without passing²¹.
7. Week 8: Turn on online evaluation - Start scoring a sample of live traffic and set the first alert thresholds tied to real consequences¹⁸.

Phase 3: Govern and scale (Weeks 9-12)

9. Week 9: Set retention and access - Configure tamper-evident, retained logging that meets Article 12 and your internal audit needs¹⁰.
10. Week 10-11: Assign owners and checkpoints - Give the agent a named owner, classify its autonomy level, and place human approval on the highest-stakes actions⁴.
11. Week 12: Review and expand - Hold the first monthly review of the scorecard against the baseline, feed every caught failure back into the suite, and plan the next use case.

How Superkind Fits

Superkind builds custom AI employees that connect to the systems your team already uses: email, Teams, SharePoint, CRM, ERP, databases, and any API-based software. They run on a Company Brain that holds your processes, your data, and your rules, and they get better every day because your team works with them. Observability and evaluation are not an add-on to that model; they are how the trust is earned and how the learning loop is instrumented.

• Traceable by design - Every action an AI employee takes across your systems is captured as a structured trace, so you can open any run and see exactly what it did and why.
• Evaluation on your definition of good - We define success criteria with your process owner and build the judges around your work, not a generic benchmark.
• Drift alerts against your baseline - We measure a baseline during the build and alert your team when accuracy, refusals, latency, or cost move away from it.
• The feedback loop is the product - Corrections your team makes during review feed straight back into the Company Brain. This is the same loop we describe in our piece on the AI feedback loop, made visible and measurable.
• Audit-ready records - Retained, tamper-evident logs give you the evidence Article 12 expects for high-risk systems and that your own auditors ask for.
• Sits on top of your stack - Observability plugs into your existing monitoring through open conventions rather than forcing a new platform on your team⁹.
• Human-in-the-loop where it matters - High-stakes actions get an approval checkpoint; routine ones run autonomously with monitoring, matched to each agent’s autonomy level.
• Outcomes, not licences - We measure the ROI of each use case against the baseline we captured, so value is proven, not asserted. Our view on measuring that is in the AI agent ROI guide.

Decision Framework: How Much Observability Do You Need?

Not every agent needs the full stack on day one. The right level of observability scales with what the agent can do and what it costs you if it gets it wrong.

The market is moving in one direction. The LLM observability segment is estimated to grow from roughly two billion dollars in 2025 into a fast-expanding category, because every serious agent deployment now needs it²⁷. Building it in early is cheaper than retrofitting it after the first incident.

Frequently Asked Questions

Henri Jung

Co-founder of Superkind, where he helps SMEs and enterprises deploy custom AI agents that actually fit how their teams work. Henri is passionate about closing the gap between what AI can do and the value it creates in real companies. He believes the Mittelstand has everything it needs to lead in AI - it just needs the right approach.