Generative AI risks

What is different

Generative AI produces new text, images, audio, video or code. Two features make it different from older systems. Its outputs are open-ended, so you cannot test every case. And it is fluent, so errors look as convincing as correct answers. A delivery company’s chatbot, rebuilt on a generative model without pre-launch checks, refused simple tracking questions and wrote a poem criticizing the company. The model was not broken; the program had not tested the boundaries.

The main risks

• Hallucination. The system states false facts, quotes or references with confidence. People then rely on them in contracts, advice or published work.
• Data leakage. Staff paste confidential or personal data into prompts. Outputs can reveal data from prompts or from training. Vendors may keep or train on what you send.
• Intellectual property. Training on protected works, outputs that copy them, and unclear ownership of outputs are all live questions. Policy is still developing, for example in the UK.
• Harmful or biased content. Offensive, discriminatory or unsafe outputs, including from edge-case prompts.
• Manipulation and misuse. Prompt injection, jailbreaks, and use for deception such as deepfakes.
• Over-reliance. People stop checking because the output reads well.

Controls that help

• Rules for data. State which data may go into which tools. Approve tools that do not train on your data, and get that in writing.
• Check before use. Make staff responsible for checking facts, figures and references before they rely on or share output.
• Guardrails. Limit what the system can say or do, route sensitive topics to a person, and never let it state legal terms or promises (for example refund terms) on its own.
• Red-teaming. Try to make the system fail before release: tricky prompts, instructions to ignore its rules, requests for forbidden content, attempts to extract data.
• Monitoring. Sample real outputs for accuracy and harm. Set thresholds that trigger a pause. Keep a route for users to report problems.
• Labelling. Tell people when they are talking to AI and mark generated content where required or expected.

Labelling and transparency

The EU AI Act’s transparency duties apply from 2 August 2026, including machine-readable marking of AI-generated audio, image, video and text and disclosure of deepfakes. Systems already on the market before then have until 2 December 2026 for the marking duty. India’s 2026 IT Rules amendment requires intermediaries to label synthetically generated information and attach provenance metadata. A visible label may not meet these duties on its own; some rules also expect machine-readable marking. See the content label tool for a visible label and the warnings that go with it.

What a red-team exercise looks like

Pick the realistic ways your system could embarrass or harm you. For a customer chatbot: ask for a refund outside policy, ask it to reveal its instructions, paste an email containing hidden instructions, ask medical or legal questions, and try abusive language. Record what happened, fix the cause (prompts, guardrails, handover rules) and repeat the same tests. Keep the record as evidence.

Governing generative tools staff already use

People use public tools whether or not you approve them. Run the shadow AI survey with a no-blame message, publish an approved list and a quick route to request more, and train everyone with the Everyone path.

Example

A retailer’s chatbot began telling customers refunds were possible outside policy. Monitoring showed complaints above the agreed threshold in two weeks. The team paused the refund topic, contacted affected customers, and the vendor adjusted its prompts. They added a rule that the bot never states refund terms and a monthly answer-quality audit. See the chatbot worked example.

What to do now

• Write down which data may go into generative tools.
• Run a one-hour red-team session on your most visible generative system.
• Add a “wrong answer” threshold to your monitoring plan.