AI Guide

Real-World Testing (EU AI Act): Piloting high-risk AI systems before market launch

Real-world testing under the EU AI Act lets providers trial a high-risk AI system with real users and real data before it is placed on the market, under a testing plan approved by a market surveillance authority. It sits outside the regulatory sandbox framework and carries its own consent, duration, and reporting rules. Learn below what the obligation covers, how it differs from a sandbox, and what German Mittelstand providers must prepare before running a pilot.

Key Facts
  • Article 60 lets providers test high-risk AI systems in real-world conditions before placing them on the market, outside a regulatory sandbox.
  • A testing plan must be submitted to and approved by the market surveillance authority in the Member State where testing takes place.
  • Participants must give informed consent under Article 61 before taking part, with the right to withdraw at any time.
  • Testing must be registered in the EU database and may not run longer than necessary to reach its stated objectives.
  • Obligations tied to Chapter III high-risk systems, including Article 60, became applicable from 2 August 2026.

Definition: Real-World Testing (EU AI Act)

Real-world testing is the process under Article 60 of the EU AI Act that lets a provider trial a high-risk AI system with real users, in real operating conditions, before the system is placed on the market.

Core characteristics of real-world testing

Real-world testing happens outside a lab, with real participants who may be affected by the system’s output, under a formal, pre-approved plan.

  • Conducted by the provider before CE marking
  • Based on a written testing plan with defined objectives and scope
  • Limited to high-risk AI systems under Annex III or Annex I Section A
  • Time-boxed to what is strictly necessary

Real-world testing vs. regulatory sandbox

Real-world testing and the regulatory sandbox are separate mechanisms, and conflating them is the most common mistake providers make. A sandbox under Articles 57 to 59 is a supervised, earlier-stage environment run by a Member State authority, offering structured guidance and sometimes relaxed data rules. Real-world testing under Article 60 happens outside that sandbox, closer to market launch, run on the provider’s own initiative. A provider can use one mechanism without the other; many Mittelstand providers skip the sandbox entirely, since slots are limited, and approach the market surveillance authority directly for plan approval.

Importance of real-world testing in enterprise AI

Lab validation on clean data rarely predicts how a model behaves with messy production inputs. The European Commission estimates 5 to 15 percent of all AI systems placed on the EU market are high-risk and face this obligation before rollout. Evidence gathered during testing feeds the conformity assessment file a provider keeps ready for inspection.

Methods and procedures for real-world testing

Running a compliant pilot follows a defined sequence.

Drafting the testing plan

The plan is the central document, describing the system, its intended purpose, and how the pilot will be monitored.

  • Objectives, methodology, and expected outcomes
  • Geographical, population, and temporal scope
  • Monitoring, organisation, and conduct of the test
  • A clear end date and stopping criteria

Submitting to the market surveillance authority

The provider submits the plan to the market surveillance authority in the Member State where the pilot runs. The authority reviews it before testing may begin, and the provider must be able to show at any point that the pilot stays within scope.

Every participant, or their legal representative, must give informed consent under Article 61 before taking part, free of undue influence and with the right to withdraw at any time without giving reasons.

Important KPIs for real-world testing

Tracking the right indicators keeps a pilot both compliant and useful.

Operational tracking metrics

  • Participants enrolled vs. planned sample size
  • Consent withdrawal rate
  • Deviations from the approved testing plan: target zero
  • Days elapsed vs. approved testing window

Strategic readiness metrics

Beyond compliance, providers should track how results shift confidence before launch. appliedAI’s review of 106 enterprise AI systems found 18 percent classified high-risk, with another 40 percent in an unclear grey zone, which is why testing evidence matters for classification too, not only performance.

Quality and safety metrics

False positive and false negative rates in the pilot population should be compared against rates seen in controlled validation. A meaningful gap signals the system is not ready for market placement.

Risk factors and controls for real-world testing

Running tests with real people and real data introduces risks beyond a standard QA cycle.

Harm to test participants

A pilot that goes wrong can affect real people, not simulated users.

  • Incorrect decisions affecting real applicants or customers
  • Insufficient fallback when the system fails mid-test
  • Unclear escalation path to a human reviewer

Reporting failures

Any serious incident during the test must be reported without delay, under the same principles that govern incident reporting elsewhere in the Act. Providers must suspend testing immediately if continuing would create unacceptable risk.

Scope creep beyond the approved plan

Testing that drifts beyond the approved population, geography, or duration without a new submission exposes the provider to enforcement action. Keeping human oversight active throughout, not only at sign-off, is the most reliable control against this.

Practical example

A 95-employee HR-tech vendor near Munich builds an AI candidate-screening tool that ranks job applicants for client companies, a use case Annex III classifies as high-risk. Before general release, the vendor wants evidence the ranking model performs fairly on its actual client base, not only on historical training data. It drafts a testing plan covering three client companies, a defined applicant pool, and an eight-week window, submits it to the market surveillance authority, and secures consent from every applicant with a visible opt-out.

  • A documented testing plan with clear start and end dates
  • Consent forms translated and confirmed for every participant
  • Weekly review of ranking deviations against the approved scope
  • A direct line to pause testing if fairness metrics drift

Current developments and effects

Real-world testing is still a young obligation, and practice is evolving quickly.

Clarifying what counts as “necessary” duration

Regulators have not published binding guidance on maximum testing length, leaving providers to justify duration case by case.

  • Early movers document duration reasoning directly in the plan
  • Industry associations are pushing for Commission guidance on timeframes
  • Some Member States are issuing national FAQs ahead of EU-level clarity

Overlap with post-market obligations

Testing results increasingly feed into the post-market monitoring plan a provider must run after launch, so the two are now designed together.

Growing compliance spend

Gartner projects that companies worldwide will spend at least 5 billion US dollars by 2027 overhauling compliance systems in response to conflicting AI rules, a cost pool that increasingly includes structured real-world testing.

Conclusion

Real-world testing gives providers of high-risk AI systems a structured, legally sanctioned way to learn how their system behaves before full market scale, without relying on a sandbox slot. The obligation trades flexibility for rigor: real operating evidence, but only within an approved plan, a bounded timeframe, and clear participant consent. For Mittelstand providers building recruitment, credit, or safety tools, treating the testing plan as a first-class deliverable is what keeps a pilot defensible under audit. As enforcement experience accumulates, the gap between a well-run pilot and an informal beta test will only grow more consequential.

Frequently Asked Questions

What is real-world testing under the EU AI Act?

It is the process defined in Article 60 that lets a provider test a high-risk AI system with real users in real conditions before placing it on the market, under a plan approved by a market surveillance authority.

Is real-world testing the same as a regulatory sandbox?

No. A sandbox under Articles 57 to 59 is a supervised program run by a Member State authority, typically earlier in development. Real-world testing under Article 60 happens outside that program, run directly by the provider.

Does real-world testing apply to a 50-person Mittelstand company?

Yes. If the company provides a system falling under Annex III, its size does not exempt it from Article 60. Smaller providers typically scope a shorter pilot with fewer participants.

How does real-world testing fit with the EU AI Act timeline?

Obligations tied to Chapter III high-risk systems, including Article 60, became applicable from 2 August 2026, though the Digital Omnibus has shifted some Annex III deadlines further out. Providers should confirm the current deadline for their system category before scheduling a pilot.

The participant may withdraw at any time without giving a reason and without being disadvantaged. The provider must stop processing that participant’s data and document the withdrawal in the testing file.

Do we need our own compliance team to run a real-world test?

Not necessarily. Most Mittelstand providers draft the testing plan with external legal support and manage day-to-day monitoring internally. What matters is that someone inside the company owns the approved plan and can show it has not drifted beyond scope.

Building better software Contact us together