Fairness and bias testing

Why test by group

Imagine a screening tool that is right 90 percent of the time overall. If it is right 95 percent of the time for one group and 70 percent for another, the headline number hides a problem. Bias here means a system treating groups of people unequally because of patterns in its data or design. The only way to see it is to look at results by group.

What to measure

Pick measures that match the decision. For a decision with a positive outcome (hired, approved, shortlisted):

• Selection rate for a group: positive outcomes divided by the number of people in the group.
• Impact ratio: a group’s selection rate divided by the highest group’s selection rate. A value well below 1 means that group is selected much less often.
• Demographic parity difference: the gap between the highest and lowest selection rate.

If you also know the true outcome (who should have been selected), you can compare errors:

• False negative rate: people who should have been selected but were not.
• False positive rate: people who were selected but should not have been.
• Equal opportunity difference: the gap in true positive rate between groups.

The bias metrics calculator computes all of these from your own results file in your browser; nothing is uploaded.

Reading the numbers

An impact ratio under 0.8 (the “four-fifths guideline”) is a signal used in some US employment contexts to look more closely. It is a rule of thumb, not a legal test everywhere, and a ratio above 0.8 does not prove a system is fair. Statistical indicators alone do not prove or disprove discrimination. Treat a red flag as a reason to investigate, not as a verdict.

Group size matters. Groups of fewer than about 30 records give unreliable numbers. Report group sizes next to every result.

What to do when you find a gap

1. Check the data. Is the group under-represented, or is a feature acting as a proxy for a protected trait (such as postcode or career gaps)?
2. Check the process. Are human reviewers treating the output differently for different groups?
3. Fix and re-test. Remove or change the cause and run the same test again.
4. Record it. Write the finding, the fix and the limits of the analysis in a bias testing report.
5. Keep a person in the loop with authority to override while you fix it.

Plan a basic test

• Decide the groups you can lawfully and ethically compare, and where the data comes from.
• Decide the measures and the threshold at which you will investigate, before you look at results.
• Test before launch, after any model or data change, and on a regular schedule.
• Have someone independent of the builders review the result.

Limits

Testing depends on having group data, which you may not collect or may not be allowed to use. Groups overlap (a person belongs to several). A fair result on today’s data may not hold next quarter. And fairness has several definitions that cannot all be met at once, so choose and explain your measure. The point is to look, act and keep records.

Example

A company tested a CV ranking tool on 700 past applicants. Group A had a 30 percent selection rate and group B 21 percent, an impact ratio of 0.70. The vendor traced it to a feature that weighted career gaps heavily, removed it, and a re-test gave about 0.90. The company added a monthly check and a contract clause requiring notice of model changes.

What to do now

• Run the calculator on a recent results file.
• Add bias checks to your testing and monitoring plan.
• Use the bias testing report template to record results and limits.