Human oversight that works

A human in the loop is not enough

Many programs say “a human reviews every decision” and consider the matter closed. But a person who has thirty seconds, sees only a score, and is measured on speed will approve almost everything. That is rubber-stamping: oversight in name only. Real oversight needs three things.

1. Time. Enough minutes per case to do the review properly.
2. Information. The inputs, the reasons the system gave, its confidence and its known limits, not just an answer.
3. Authority. The power to disagree and a safe, easy way to do it, without penalty and without having to justify a hundred overrides.

Design it first

Start with the mechanisms, not the training:

• Override. One clear action to change the system’s output, and a record of what the person chose and why.
• Stop. A way to pause or switch off the system quickly, with a named person allowed to do it.
• Escalation. What a reviewer does when a case looks odd or the system seems to be failing.
• Fallback. A manual process that works if the system is switched off.

Who the overseers are

Name them by role. Train them on what the system does, where it fails, how to read its outputs and how to override. Refresh the training. Give them the right to raise concerns directly to the governance lead. In high-risk uses in the EU, deployers must assign people with the necessary competence, training and authority.

Warning signs of weak oversight

• Override rate close to zero. Either the system is perfect or nobody is checking.
• Reviewers take seconds per case, or work through queues in bulk.
• Reviewers cannot see why the system recommended what it did.
• The only way to disagree is a long form.
• Nobody is accountable when an override turns out to be wrong, or right.

An unusually high override rate is also informative: it may mean the system is poorly matched to the real cases.

What the law expects

Several regions expect meaningful human involvement in some decisions. The EU AI Act requires high-risk systems to be designed so people can oversee them effectively, understand their output and intervene or stop them. The UK’s reformed rules on significant automated decisions require safeguards including human intervention and a way to contest. Colorado’s 2027 law gives some individuals human review of certain adverse automated decisions. The test in each case is substance. A signature that nobody could have refused does not meet it. (These are summaries; see the region pages.)

Record and measure

Log each review: who, when, what the system recommended, what the person decided and, for overrides, why. Track the override rate over time and by group. Sample reviewed cases for quality. Use what you learn to improve the system and the review process, and to feed your monitoring plan.

Example

A lender’s model recommended declines, and a credit officer reviewed each one. At first officers overrode about 12 percent. After the team added an average-time target for reviews, the rate fell to under 2 percent in a month, with no change in the model. They removed the target, showed officers the main reasons behind each recommendation and set a minimum review time. The override rate recovered to around 9 percent, and complaints fell.

What to do now

• Write the human oversight procedure for one system: who, when to step in, how to override and stop.
• Add override rate to your monitoring plan.
• Check the review time and information your reviewers actually have.