AI Guide

Data Poisoning (AI): How corrupted training and retrieval data sabotages AI systems

Data poisoning is an attack that deliberately corrupts the training data, fine-tuning data, or retrieval knowledge base an AI system relies on, so that its outputs or behavior are manipulated in ways the operator never intended. Unlike prompt injection, which hijacks a system at runtime, poisoning compromises the data foundation itself, often long before anyone notices. Learn below how the attack works, which defenses enterprises use, and why vendor and RAG knowledge-base integrity have become central to EU AI Act documentation.

Key Facts
  • LLM04: Data and Model Poisoning is a standalone risk category in the OWASP Top 10 for LLM Applications 2025, recognizing training data manipulation as a top-tier risk
  • An Anthropic study with the UK AI Security Institute and the Alan Turing Institute (October 2025) found that as few as 250 malicious documents can implant a backdoor in language models regardless of model size or training data volume
  • Academic PoisonedRAG research found that injecting as few as five malicious documents into a retrieval knowledge base of millions of records can push attack success rates above 90% against RAG systems
  • Carlini et al. (2023) demonstrated that poisoning 0.01% of a web-scale training dataset such as LAION-400M was achievable for about 60 US dollars by purchasing expired domains the dataset referenced
  • Article 15 and Annex IV of the EU AI Act require high-risk AI system providers to document data governance measures, including steps taken to detect and correct poisoned or biased training data

Definition: Data Poisoning (AI)

Data poisoning is an attack in which an adversary deliberately corrupts the training data, fine-tuning data, or retrieval knowledge base an AI system relies on, so the system learns wrong associations, embeds a hidden backdoor, or retrieves manipulated content as trustworthy.

Core characteristics of data poisoning

Data poisoning targets an AI system’s data foundation rather than its runtime behavior, which makes it harder to detect and more durable than attacks aimed at a single interaction.

  • Corrupts training, fine-tuning, or retrieval data before or during use
  • Implants a narrow backdoor triggered by specific inputs, or broadly degrades accuracy
  • Exploits the same trust assumption as prompt injection: systems treat ingested data as reliable
  • Often enters through a third-party dataset, scraped content, or an unvetted vendor pipeline

Data Poisoning (AI) vs. Prompt Injection

Prompt injection manipulates a system at inference time by hiding instructions in content it processes right now. Data poisoning operates earlier and deeper: it corrupts the data a model was trained on, fine-tuned with, or retrieves from, so the damage persists across every future use until someone finds it. A poisoned model or knowledge base behaves normally on ordinary inputs and reveals the damage only once the attacker’s trigger condition appears. The two can combine, since a poisoned retrieval index acts as a standing, self-refreshing injection source.

Importance of data poisoning in enterprise AI

As companies fine-tune models on internal data and connect retrieval-augmented generation systems to knowledge bases built from documents and tickets, the data supply chain becomes an attack surface of its own. The OWASP Top 10 for LLM Applications lists data and model poisoning as a standalone risk, and research on RAG systems found that injecting just five malicious documents into a knowledge base of millions can push attack success above 90%. For a system acting on corrupted data, the failure looks like bad luck rather than an attack.

Methods and procedures for data poisoning

Attackers poison AI systems at three points in the data lifecycle.

Training and fine-tuning data poisoning

This method injects manipulated examples into the dataset used to pretrain or fine-tune a model, so it learns a hidden behavior its builders never intended.

  • Label-flipping, swapping correct labels to teach wrong associations
  • Backdoor triggers, where a rare pattern forces a chosen output
  • Availability attacks that broadly degrade accuracy rather than target one behavior

RAG and knowledge base poisoning

Poisoning a retrieval knowledge base does not require touching the model at all. An attacker only needs to insert a handful of manipulated documents into the source material a retrieval system indexes, so it retrieves and presents fabricated content as verified internal knowledge whenever a related query arrives.

Supply chain and vendor dataset poisoning

Many companies license, download, or inherit data and pretrained models through a vendor or open dataset. A 2023 study found attackers could poison about 0.01% of web-scale datasets such as LAION-400M for roughly 60 US dollars by buying domains the dataset referenced. Without AI vendor risk management covering data provenance, a poisoned upstream source can enter many downstream systems unnoticed.

Important KPIs for data poisoning

Enterprises track poisoning exposure across data integrity, impact, and provenance.

Data integrity metrics

  • Anomaly detection coverage on new training and retrieval data: tracked per source
  • Share of data sources with verified provenance: target 100% for high-risk use cases
  • Outlier and drift scans before each fine-tuning or re-indexing cycle
  • Mean time to quarantine a flagged document: under 24 hours

Impact and containment metrics

Impact metrics measure how much a poisoned dataset can influence outputs before detection. Anthropic’s research with the UK AI Security Institute and the Alan Turing Institute found that as few as 250 malicious documents can implant a working backdoor regardless of model size, so impact does not shrink with company size the way many assume.

Provenance and audit metrics

Every dataset, fine-tuning run, and knowledge base update should carry a documented source and change log, so an anomaly can be traced to its exact entry point rather than requiring a full forensic rebuild.

Risk factors and controls for data poisoning

Left unaddressed, poisoning risk grows with every new data source a model or knowledge base ingests.

Backdoor and targeted manipulation

A successful attack can implant a narrow trigger that produces a chosen output only under specific conditions, making it invisible during normal testing.

  • Hidden triggers that bypass standard evaluation benchmarks
  • Gradual accuracy decline mistaken for ordinary model drift
  • Manipulated retrieval results presented with the same confidence as verified sources

Vendor and supply chain exposure

Unvetted third-party datasets, pretrained checkpoints, and scraped content all carry inherited poisoning risk, since the company deploying the model rarely controls the pipeline that built it.

Regulatory exposure under the EU AI Act

High-risk AI providers must document data governance measures under the EU AI Act, including steps to detect and correct poisoned or biased data, as part of the technical documentation (EU AI Act) required under Annex IV. Companies unable to show a tested process risk a conformity gap and a system that silently misbehaves in production.

Practical example

A 180-employee precision parts manufacturer in Baden-Württemberg built a RAG knowledge assistant that answered engineering and quality questions from scanned manuals and supplier specs. During a scheduled AI red teaming exercise, consultants uploaded manipulated specification documents into a shared drive the indexing pipeline scanned nightly, and the assistant began citing fabricated tolerance values as approved supplier data. The company added a provenance check that only indexes documents from a signed, access-controlled source list and a weekly diff review before release.

  • Signed, access-controlled source list for every indexed document
  • Weekly diff review of new or changed knowledge base entries
  • Quarantine workflow for documents flagged by anomaly detection
  • Scheduled red-team tests before each major update

Current developments and effects

Defenses against data poisoning are shifting from one-time checks to continuous monitoring.

From spot checks to continuous provenance tracking

Early defenses relied on a one-time cleaning pass before training. Current practice tracks provenance continuously across the data lifecycle.

  • Cryptographic signing of trusted data sources
  • Automated drift and outlier detection on ingestion
  • Sandboxed evaluation of fine-tuned models before rollout

Regulatory and standards attention

NIST’s adversarial machine learning taxonomy and BSI guidance developed with Fraunhofer AISEC both classify data poisoning among the core adversarial risks across the AI lifecycle, pushing providers toward documented governance rather than informal checks.

RAG adoption expands the attack surface

As more companies connect retrieval systems to internal and external sources, every newly indexed feed becomes a potential entry point, making source vetting a permanent task rather than a one-time setup step.

Conclusion

Data poisoning is not a hypothetical research problem; it is a practical consequence of how AI systems learn from and retrieve data that someone, somewhere, did not fully vet. Enterprises that treat every training set, fine-tuning corpus, and knowledge base update as a controlled asset catch manipulation before it reaches production. As regulators formalize data governance and RAG adoption accelerates, provenance will separate companies that trust their AI outputs from those quietly operating on corrupted ground.

Frequently Asked Questions

What is the difference between data poisoning and prompt injection?

Prompt injection hides instructions in content a system processes at the moment of use, while data poisoning corrupts the training or knowledge base data the system was built on, so the damage persists across every future use. An attacker can combine both, poisoning a knowledge base so every retrieval acts as a standing injection.

Can a company with 50-200 employees really be a target, or is this only a risk for large AI labs?

Yes. Most Mittelstand exposure comes through fine-tuning on internal documents or RAG knowledge bases built from shared drives and supplier files, which attackers, including a disgruntled insider, reach far more easily than a foundation model’s training pipeline. As few as five manipulated documents in a retrieval index can already achieve a high attack success rate.

How does data poisoning relate to the EU AI Act and DSGVO?

High-risk providers must document data governance and quality measures under the EU AI Act’s Annex IV, including how poisoned or biased data is detected and corrected. Under the DSGVO, a poisoning incident that alters how personal data is processed can also trigger accountability obligations.

What does it cost to defend against data poisoning?

Cost depends on how many data sources feed a model or knowledge base, since controls like source vetting and anomaly scans are typically configured on top of existing data infrastructure. A focused rollout covering the highest-risk feeds fits within a normal AI implementation budget.

Do we need our own machine learning team to defend against data poisoning?

Not necessarily from day one. Most mid-sized companies configure source vetting and anomaly detection with their implementation partner, while internal IT maintains the approved source list. Ongoing review typically becomes a shared responsibility as the knowledge base grows.

How does Superkind’s approach to company knowledge address data poisoning risk?

Superkind builds each AI employee’s knowledge base from access-controlled company systems rather than open web data, and changes go through the same approval structure as other agent actions, so a manipulated document cannot reach production without a controlled, auditable update process.

Building better software Contact us together