Human-in-the-Loop: Building Trust in AI Agents

In February 2024, the British Columbia Civil Resolution Tribunal ordered Air Canada to refund a passenger because its customer service chatbot had invented a bereavement-fare policy that did not exist. The airline argued the chatbot was a separate legal entity. The tribunal disagreed. The bot belonged to Air Canada, the bot spoke for Air Canada, and Air Canada paid⁵.

Eleven months earlier, the courier DPD had to shut down its AI chatbot after it swore at customers and wrote poems calling itself the worst delivery service in the world⁷. New York City spent most of 2024 publicly defending an AI agent that was telling small business owners they could legally discriminate against tenants and pocket employee tips⁸. None of these systems failed because the underlying model was bad. They failed because nobody designed a way for a human to step in before harm hit a real customer.

This is the trust problem. Deloitte’s 2026 enterprise survey found that 85 percent of companies plan to deploy autonomous AI agents, but only 1 in 5 has a mature governance model for them¹³. In finance and accounting, lack of trust is the single biggest barrier to agent adoption¹⁴. The technology is ready. The control surface is not.

Human-in-the-loop is not a feature you bolt on after launch. It is the governance model that decides whether your AI agents become an asset or a liability. This guide is for the German CTO, Operations Lead or Geschaeftsfuehrer who needs a concrete framework for when humans approve, when agents act, and how to build trust that scales beyond the pilot.

Why Trust Is the Bottleneck

The AI agent market has moved faster than the trust infrastructure underneath it. Gartner projects that 40 percent of enterprise applications will feature task-specific AI agents by the end of 2026, up from less than 5 percent in 2025¹⁰. Adoption is real and accelerating. So is the gap between what agents can do technically and what companies are willing to let them do operationally.

• The governance gap is wide - Deloitte surveyed 3,235 enterprise leaders in 2025. Only 20 percent have a mature governance model for autonomous agents, even though 85 percent expect to deploy them¹³.
• Trust is the top blocker, not capability - In Deloitte’s finance and accounting cohort, lack of trust beat cost, integration and skills as the leading reason agentic AI deployments stall¹⁴.
• German SMEs feel it sharply - The Bitkom 2025 survey found 53 percent of German companies cite legal uncertainty and 51 percent cite lack of personnel resources as their top AI barriers²⁰. Both are trust-adjacent: who is responsible when the agent is wrong, and who is qualified to oversee it.
• The Mittelstand is uniquely exposed - Fraunhofer IAO finds 86 percent of German SMEs see AI as relevant, but only 23 percent have working production projects²¹. The gap is rarely technology. It is the institutional discomfort with letting software make decisions the company will be held accountable for.
• The cost of getting it wrong is rising - Air Canada lost a tribunal case and had to pay damages plus tribunal fees⁵. The reputational damage was orders of magnitude larger than the refund. DPD became a global case study in agent failure within 24 hours of a single screenshot going viral⁷.

The companies that will win the next phase of agent adoption are not the ones with the most agents. They are the ones whose agents get trusted with the highest-value decisions. That trust does not come from a brand campaign or a thicker terms-of-service. It comes from observable, measurable, audit-grade human oversight built into the system from the first deployment.

Why Now: Article 14 and the Failures of 2024

The pressure on HITL design is coming from two directions at once: regulation and reputational risk. Both turned from theoretical to concrete in the last 18 months.

EU AI Act Article 14 becomes binding

The EU AI Act becomes fully applicable on 2 August 2026². For high-risk AI systems, Article 14 is the operational core. It requires that the AI system be designed so that it can be effectively overseen by natural persons during the period in which it is in use¹.

The article spells out four concrete capabilities that the human reviewer must have:

1. Understand capacities and limitations - The reviewer must be able to monitor the agent and detect anomalies, dysfunctions and unexpected performance.
2. Resist automation bias - Article 14 explicitly names the tendency to over-rely on AI output as a risk that the design must counter¹. This is the first time a major regulator has written this into law.
3. Correctly interpret the output - The agent must surface its reasoning in a form the reviewer can actually evaluate.
4. Decide not to use or to override - The reviewer must always have a usable stop or override at any moment of operation.

For high-risk systems, this is non-negotiable from August 2026. Penalties under Article 99 reach EUR 15 million or 3 percent of global annual revenue for non-compliance with high-risk requirements²⁸. SMEs get the lower-of-two cap, but the design obligation is identical.

The 2024 incident shelf gets deeper

Regulation creates the floor. Reputational risk creates the ceiling. The last two years produced the most-cited shelf of public AI agent failures so far, and every one of them came back to missing oversight.

• Air Canada (February 2024) - Customer service chatbot invented a bereavement fare refund policy. Tribunal ruled the airline liable for the bot’s output. The judge specifically rejected the argument that the chatbot was a separate legal entity⁵⁶. The fix would have been a confidence threshold escalating policy answers to a human.
• DPD UK (January 2024) - Customer chatbot started swearing and writing critical poems about its own employer after a routine model update. No regression testing, no behavioural guardrails, no kill switch tied to brand-safety signals⁷.
• NYC MyCity chatbot (April 2024) - Government-run agent told small business owners they could legally fire workers for reporting harassment, discriminate based on income source and pocket tips. The city kept it online while acknowledging the answers were wrong⁸. No HITL on legal advice, no domain restriction.
• Workday hiring discrimination suit (2024) - A US federal court allowed a class action against Workday to proceed over allegations its AI hiring screen produced age and race discrimination. The case crystallised the principle that the deployer of an HR agent inherits the agent’s output as its own employment decision.
• Replit code-base wipe (2025) - An autonomous coding agent in a sandbox attempted to delete a production database during testing, surfacing repeatedly in public agent-failure write-ups¹⁶. The pattern was full L4 autonomy on an irreversible action with no approval gate.

These are the cases that made the news. The actual rate of internal agent misfires is much higher and almost never reported. MIT Sloan documented multiple real-world tests in 2025 where agents took plausible-looking actions that were quietly wrong, and were only caught because a human happened to spot-check the output¹⁶. The signal is the same: where humans were not in the loop by design, they were not in the loop in practice, and the cost was paid downstream.

The 5 Levels of Agent Autonomy

Talking about HITL in the abstract is not actionable. The useful question is: at what level of autonomy does each task in your business currently sit, and where does it need to sit a year from now? The Knight First Amendment Institute working paper on agent autonomy and several converging industry frameworks all point to a five-level model that maps cleanly onto the SAE driving-automation analogy¹⁷¹⁸.

L0: Observe

The agent monitors data and surfaces information, but takes no action. The human does everything. This is where most analytics dashboards sit. Useful, but it is not really an agent.

L1: Suggest

The agent generates a suggested next step or draft output. The human reviews and chooses to use it or not. A drafting assistant that proposes an email reply but does not send it sits here. This is the safest possible deployment level and a natural starting point for any new agent.

L2: Propose with approval

The agent fully prepares an action, including all data and reasoning, but requires explicit human approval before execution. This is the canonical HITL pattern. A purchase-order agent that drafts the PO, attaches the supplier comparison and the budget check, then waits for sign-off, sits here.

L3: Execute with veto window

The agent executes the action immediately but holds it in a reversible state for a defined window during which a human can veto or roll back. A scheduled email release at T+5 minutes, a draft invoice that auto-posts at end of day unless flagged, an order routing decision that can be undone within 30 minutes, all sit here. This is where most high-volume operational agents settle as trust matures.

L4: Fully autonomous

The agent acts without per-action human approval. Oversight is by exception, by sample, and by audit. A spam filter routing inbound mail, a fraud-scoring agent flagging transactions for blocking based on high-confidence patterns, a route-optimisation agent rebalancing delivery legs, all sit here. L4 is appropriate only for high-volume, low-marginal-impact, easily-reversible decisions.

The Risk x Reversibility Matrix

Knowing the autonomy levels is not enough. You also need a way to decide which level fits which action. The most useful tool we deploy with clients is a two-axis matrix: how much damage if this action is wrong, and how easy is it to reverse.

• Risk axis - Estimated worst-case impact in money, customer harm, regulatory exposure or brand damage if the action is taken wrongly. Low (under EUR 500), medium (EUR 500 to 50,000), high (over EUR 50,000 or any safety, legal or HR exposure).
• Reversibility axis - How quickly and cleanly the action can be undone. Instant (a draft that has not been sent), short window (an email that can be retracted within 5 minutes), hard (a wire transfer that has cleared), permanent (a contract that has been signed and counter-signed).

Worked examples

• Rerouting an inbound support ticket - Low risk, instant reversal. Belongs at L4. No human approval needed; audit a sample weekly.
• Issuing a EUR 30 customer goodwill credit - Low risk, hard to reverse cleanly. Belongs at L3 with a 10-minute veto window in a Slack channel.
• Drafting and sending a customer-facing email about a known service issue - Medium risk (brand exposure), short window (you can apologise and correct). Belongs at L2 in your support team workflow until calibrated trust accrues.
• Approving a EUR 12,000 purchase order to a new supplier - High risk, hard to reverse. Belongs at L2 with sign-off from procurement plus budget owner.
• Placing a binding contract counter-signature - High risk, permanent. Belongs at L1: agent drafts and reasons, human signs.
• Auto-scheduling a maintenance technician based on a vibration anomaly - Medium risk, instant reversal (you can re-schedule). Belongs at L3 with a one-hour veto.

Run this exercise across the 20 to 30 actions your agent will take in its first six months. The output is your operational HITL policy. It is also the document you show an auditor under EU AI Act Article 14 when they ask how you decided which actions need human oversight.

Escalation Patterns That Work

An agent that escalates everything is just an expensive form. An agent that escalates nothing is a public-incident waiting to happen. The discipline is choosing the right escalation triggers and tuning them with operational data.

Trigger 1: Confidence below threshold

The simplest and most common escalation: if the agent’s self-reported confidence in an action is below a threshold, it kicks the action to a human queue²⁶²⁷.

• Customer service - Typical thresholds 80 to 85 percent. Below that, route to a human agent.
• Financial transactions - 90 to 95 percent. Below that, hold and notify finance.
• Healthcare and safety - 95 percent or higher. Below that, hard stop.
• Operational sweet spot - Industry data suggests escalation rates between 10 and 15 percent are sustainable for a single review team. Above 20 percent, you are training reviewers to rubber-stamp. Below 5 percent, you are likely missing real edge cases.

Trigger 2: Ambiguity detection

Confidence scores can lie. A model can be highly confident about a wrong answer when it has been trained on similar wrong inputs. Ambiguity detection runs the same task through multiple model paths or multiple prompt framings and escalates when the answers diverge. This catches confident errors that a confidence threshold alone misses.

Trigger 3: Policy violation patterns

Independent of the model, a policy layer checks every proposed action against business rules: payment limits, customer-tier restrictions, regulated-content blocks, suspicious-pattern matches. If the action would breach a rule, the agent never executes it; it escalates with the breaching rule named in the queue.

Trigger 4: Novelty and out-of-distribution flags

The agent flags any input that looks different from its training distribution: an unusual customer query, an invoice format never seen before, a supplier name not in the database. Novelty escalation is what stops you from being the company whose agent confidently approved a fake invoice from a domain it had never encountered.

Trigger 5: Customer dissatisfaction signals

A customer using words like “cancel”, “lawyer”, “regulator”, “press”, or repeating the same complaint should pull the conversation out of the agent and into a human within seconds. Build this list with your customer-success team, not your data team.

Trigger 6: Time and value bands

Hard limits that override everything else: any action above EUR 10,000, anything during the holiday freeze, anything affecting more than 50 customers at once, anything outside business hours in a regulated workflow. These are the deterministic guardrails that sit underneath the probabilistic confidence triggers.

UX Patterns That Build Trust

The trigger fires and an action lands in a reviewer’s queue. Whether the reviewer makes a good decision or rubber-stamps it depends almost entirely on the interface. Article 14 names automation bias as a risk that the design must counter¹. Good UX is how you actually do that.

1. Show the action, not the model - The reviewer should see what will happen in plain language: “Send this email to this customer”, “Place this order with this supplier”, not a JSON blob or a confidence score floating in space. Outcome-first framing keeps the reviewer’s attention on the consequence.
2. Show the reasoning, briefly - One paragraph: what did the agent see, what rule did it apply, what alternatives did it consider. If the reasoning fits in a tweet, the reviewer can actually read it. If it is a five-page chain-of-thought trace, nobody reads it and you are back to rubber-stamping.
3. Always offer a clean reject - Reject must be as one-click as approve, and rejection must offer a free-text reason field. That field becomes the training signal for the next iteration of the agent.
4. Surface dissenting evidence - If the agent considered an alternative path and rejected it, show the rejected option and why. This is the single most effective debiasing pattern we have observed in production: the reviewer sees the agent did consider X and chose Y, so they engage with the reasoning rather than skimming the conclusion.
5. Make the audit trail real-time - Every approve, reject, override and escalation is logged with reviewer ID, timestamp, agent version, input hash, reasoning hash and final action. The reviewer sees their own history in the same UI. This builds personal accountability without surveillance theatre.
6. Build calibration into the loop - Periodically replay past approved cases blind: show the reviewer the agent recommendation only and ask them to decide. Compare to what they actually approved last time. Anyone whose calibration is drifting gets a coaching session.
7. Allow inline correction, not just approve or reject - The most powerful HITL UI lets the reviewer say “this is right but change the supplier to X” or “send this but with a longer apology”. Inline edits become the highest-quality training signal in the system.

Common HITL Anti-Patterns to Avoid

The reason HITL gets a bad reputation in some circles is that most implementations are weak. Here are the patterns we see fail most often in mid-sized companies, and how to fix each.

Anti-pattern 1: Rubber-stamp approvals

Symptom: 95 percent of agent recommendations get approved, average review time is under 3 seconds, the same reviewer approves 200 actions per shift. The reviewer is not reviewing, they are clicking. Fix: tighten escalation triggers so only genuinely uncertain cases reach the queue, sample 10 percent of approvals for blind re-review, hold reviewers accountable when sampled cases turn out to be wrong.

Anti-pattern 2: Approval fatigue

Symptom: review queue keeps growing, weekend backlog, reviewers complain. Most enterprises that try L2 across the board hit this within three months²³. Fix: tighten triggers, escalate only outliers, batch similar reviews, route by domain expertise so the right human sees the right action, rotate reviewers to avoid burnout, and move low-risk action classes to L3 once the agent has earned it.

Anti-pattern 3: Single point of human failure

Symptom: one named person is the approver for 80 percent of agent actions. They go on holiday and the agent stops. Fix: define approval roles, not approval people, rotate primary and back-up reviewers per workflow, escalate to next-up after a defined wait time.

Anti-pattern 4: False consensus

Symptom: the team agrees the agent is “working well”, no objective measurement. The agent might be quietly producing the same kind of biased or wrong output that nobody notices because everyone agrees with each other. Fix: red-team your agent monthly with adversarial inputs, run paired blind reviews, track rejection rates by reviewer to surface drift.

Anti-pattern 5: Hidden override

Symptom: reviewers learn to bypass HITL via an “internal” backdoor when the queue is too long. Fix: make the only path to action go through the audit-logged HITL UI, monitor for off-path actions, treat any bypass as a process failure to be investigated and not as individual delinquency.

Anti-pattern 6: HITL without authority

Symptom: the reviewer can flag an action as wrong but cannot stop it. The agent acts anyway because “the model is usually right”. This is the failure mode at the heart of automation bias and is exactly what Article 14 is designed to prevent¹. Fix: reviewer rejection must always block the action, period. If the company wants override, that override goes through a named senior owner with their own audit log.

Anti-pattern 7: HITL where the human has no context

Symptom: a reviewer is asked to approve an agent action involving a customer, a contract, or a system they have never seen before. They have to trust the agent because they have no way to check. Fix: route by domain expertise, surface the relevant context (customer history, contract terms, system state) inside the approval UI, refuse to escalate to anyone who cannot meaningfully evaluate the action.

A 90-Day Path to Real HITL

The risk in this topic is paralysis. Companies talk about governance for six months without shipping anything. Here is the 90-day path we run with clients to move from no agents to one well-governed agent in production.

Phase 1: Map and decide (Weeks 1-3)

1. Week 1: Pick one agent - Select a single use case with measurable ROI and bounded blast radius. Invoice processing, supplier reorder, customer triage and recruiting screen are the four most common starting points in the Mittelstand.
2. Week 2: Map the actions - List every individual action the agent will be allowed to take. Aim for 15 to 30 actions. Resist the urge to lump them together; granularity is what makes HITL actually work.
3. Week 3: Risk x reversibility - Run each action through the matrix. Assign an autonomy level (L0 to L4). Identify the approval roles. Define the escalation triggers. Document all of this in a one-page HITL policy that the business and IT both sign.

Phase 2: Build the approval surface (Weeks 4-7)

4. Week 4: Approval queue - Build or configure the queue where actions land for review. Slack, Teams or a dedicated app all work; consistency matters more than the choice.
5. Week 5: Reasoning and audit - For every action, surface the agent reasoning, the alternatives considered, the input data hash, and create the immutable audit log entry. This is the single most under-built piece in early agent deployments.
6. Week 6: Kill switch and overrides - Build the global pause and the per-action veto. Test both with the team. Document the named senior owner who can re-enable.
7. Week 7: Reviewer training - Walk every named reviewer through the UI. Show them what to look for. Run them through five worked examples each. Calibrate against the agent recommendation to surface disagreement.

Phase 3: Pilot and tune (Weeks 8-12)

8. Week 8: Shadow mode - The agent runs and proposes, but every action also goes to the existing manual process. Compare outputs. This produces the calibration data you need.
9. Week 9: Live with full HITL - Switch the agent to L2 across all actions. Every action requires human approval. Monitor approval times, rejection rates, escalation triggers fired.
10. Week 10: First trust calibration - Identify action classes with rejection rates under 5 percent and average review time under 30 seconds. These are candidates to move to L3 (veto window).
11. Week 11: Selective L3 rollout - Move the calibrated action classes to L3. Keep L2 for everything else. Watch for the first week to confirm the rollback works as designed.
12. Week 12: Measure and report - Time saved per process, error rate, rejection rate per trigger, audit trail completeness. Present to leadership. Document the next agent and repeat.

How Superkind Builds HITL Into Every Agent

Superkind builds custom AI agents for the German Mittelstand and enterprises. Human-in-the-loop is not an add-on in our agents. It is the default architecture from the first sketch of a new use case. Here is what that looks like in practice.

• Risk-mapped action catalogues - Every agent we build ships with a documented action catalogue: every individual action, its risk x reversibility classification, its autonomy level and its named approval role. The catalogue is the auditable truth and is reviewed quarterly with the client.
• Approval surfaces in the tools your team already uses - We deliver approval queues in Slack, Teams, email, ticketing systems or a dedicated dashboard. No new platform to learn. Reviewers act where they already work.
• Inline reasoning, not chain-of-thought dumps - Every action shows a one-paragraph reason, the alternatives considered and the breaching trigger if applicable. Reviewers can read it in 10 seconds and engage with the substance.
• Always-on kill switch - One click pauses every running instance of the agent. Every reviewer has it. Re-enable requires a named senior owner with their own audit entry. Tested once a quarter as part of your operational playbook.
• Immutable audit trails - Every proposed action, every approval, every rejection, every override is logged with timestamp, reviewer ID, agent version and input hash. The trail meets EU AI Act Article 14 evidence requirements out of the box.
• Calibrated trust progression - Agents start at L2 by default. Action classes earn the right to move to L3 based on measured rejection rates and review-time data. We never start a new agent at L4.
• Adversarial testing built in - We red-team every agent monthly with synthetic edge cases, ambiguous inputs and known failure patterns from the public incident shelf. Findings feed back into the policy layer.
• Sovereign and auditable by default - Agents run inside your infrastructure or in EU-resident infrastructure of your choice. Audit logs stay with you. There is never a black-box vendor between you and the auditor.

Decision Framework: Where Do You Sit Today?

Use this table to identify your current HITL maturity and the next concrete step. Most Mittelstand companies sit between Stage 1 and Stage 2.

Related Reading

Human-in-the-loop is one piece of the broader AI governance picture. These articles cover the surrounding terrain.

• Why 95% of AI Projects in the Mittelstand Fail - Trust failures and missing oversight are two of the seven root causes covered in detail.
• EU AI Act 2026: What the Mittelstand Must Know Before August - The full regulatory context around Article 14, risk classification and SME provisions.
• RPA vs AI Agents - Why RPA never needed HITL and why agents always do, explained through the autonomy lens.
• AI Agents for the Mittelstand - The cornerstone guide to deploying agents in mid-sized German companies.
• Fix Your Processes Before You Add AI - Why HITL on a broken process is twice as broken.

Frequently Asked Questions

Henri Jung

Co-founder of Superkind, where he helps SMEs and enterprises deploy custom AI agents that actually fit how their teams work. Henri has spent the last two years deep in the operational reality of agent governance for mid-sized German companies, watching where HITL works, where it breaks and why most enterprise governance models look great on a slide and fail in production. He believes trust in AI is an engineering problem first and a communication problem second, and that the Mittelstand is the right place to set the standard.