Lesson · 7 min · Stage 5
How AI systems fail
Learning goal: Recognize common failure types.
For reference purposes only. Not legal advice.
Draft: not yet fact-checked or legally reviewed
Key points
- AI systems can fail quietly: they keep working while giving wrong, unfair or unsafe results.
- Hallucinations are confident outputs that are false or made up.
- Bias comes from patterns in data or design and can treat groups unequally.
- Drift is performance getting worse as real-world data changes after launch.
- Misuse, security attacks such as prompt injection, and over-reliance by people are failures too.
Who this is for: anyone new to the topic. Related stage: Test and validate.
Failing quietly
When ordinary software fails you usually see an error. AI often fails without any error. The system returns an answer, it looks fine, and the answer is wrong. This is why AI governance relies on monitoring and review, not only pre-launch testing.
Six common failure types
- Wrong or made-up outputs (hallucinations). The system states something false with confidence. Treat generated facts, quotes and references as unchecked until a person checks them.
- Bias. If the training data reflects past unfairness, the system can repeat or amplify it. Overall accuracy can hide unequal results for particular groups, so compare outcomes and error rates by group.
- Drift. The world changes. A model built on last year’s data may get worse on this year’s. Monitoring with thresholds that trigger action catches drift early.
- Misuse. People use systems for things they were not designed or approved for. State what a system is for, and what it is not for.
- Security attacks. Examples include prompt injection (input crafted to make a system ignore its instructions or reveal information), data poisoning and leaking sensitive data through outputs. Security references such as the OWASP Top 10 for LLM Applications and MITRE ATLAS are good starting points.
- Over-reliance. People trust the output too much and stop checking. A human “in the loop” only helps if they have time, information and authority to disagree.
Example
A fine-tuned model summarized clinical documents. During an audit the team could show full API logs but could not say which version of the training data had been used for fine-tuning. The system worked, but the missing record of where the data came from meant the model had to be re-validated before wider use.
What to do now
- For each AI system you use, write down one way it could fail quietly and who would notice.
- Never rely on generated facts without a check.
- Learn the glossary terms hallucination, drift, bias and prompt injection.
Sources
- Supplied governance study notes (module 1)
Reviewed 2 Oct 2026. Sources: OWASP Top 10 for LLM Applications; MITRE ATLAS; NIST AI Risk Management Framework.
Terms used: Bias, Drift, Fine-tuning, Hallucination, Prompt injection, Training data.
Part of: Everyone path, Builders path.
Was this page useful? Tell us what is missing or wrong ↗ · Report a correction