Machine-learning bias: evaluating fairness without false recipes

In February 2018, Joy Buolamwini and Timnit Gebru published an evaluation of three commercial gender-classification systems for images. Gender Shades found error rates as high as 34.7% for darker-skinned women and a maximum of 0.8% for lighter-skinned men. One aggregate figure would have concealed the difference between subgroups.

The result does not establish that every disparity is unlawful discrimination or that an automatic correction exists. It demonstrates something earlier: the population, labels and breakdown determine which failure can be seen. Bias can enter when a team chooses the problem, collects data, constructs a reference, optimises a metric, sets a threshold or deploys the system inside an institution.

The transferable skill is to follow an explicit chain: harm → population → reference → metric → intervention → effect. “The model is fair” erases that chain. An auditable claim says which harm it seeks to reduce, for whom, against which alternative and which inequalities remain.

Define the harm before choosing a metric

A hiring system may deny opportunities; speech recognition may deliver poorer service to one group; a generator may reproduce stereotypes without allocating jobs. These are allocation, quality-of-service and representational harms. They do not necessarily share a metric.

Predicting a historical decision is not the same as predicting a neutral truth. If the label is “was hired”, the model learns earlier selection processes, including their criteria and exclusions. If it is “repaid a loan”, credit conditions also influenced the outcome. The target must describe the phenomenon it measures and the intervention it enables.

Suresh and Guttag organised seven sources of harm across the machine-learning lifecycle, spanning historical data and representation through evaluation, aggregation and deployment. Their framework prevents every problem from being located in “biased data”: a balanced sample can still support harm if the task, threshold or institution is badly designed.

Averages conceal intersections

Gender Shades examined two datasets used for facial analysis and found that lighter-skinned people made up 79.6% of IJB-A and 86.2% of Adience. It then built a dataset balanced by gender and skin type to test systems. The largest gap did not appear when only overall accuracy was considered.

Evaluating one variable at a time is insufficient. Acceptable results for “women” and for “darker-skinned people” can hide a concentrated failure at their intersection. A report needs groups relevant to use, intersections with enough observations, and the case count behind each percentage.

Small samples produce unstable estimates. Numerator, denominator and uncertainty should accompany the rate. Zero errors in ten examples does not establish perfection. Expanding data, grouping only with a reason or declaring insufficient evidence is more honest than publishing a precise percentage with no foundation.

Four definitions answer different questions

Demographic parity compares the share of favourable decisions across groups. It can matter when final access is the concern, but does not condition on the reference outcome. Equal opportunity equalises true-positive rates among those meeting the relevant condition: it asks whether positive cases have the same chance of being detected.

Equalised odds extends that idea to true and false positive rates. Hardt, Price and Srebro formalised the criterion for supervised learning and showed how to post-process a predictor when prediction, outcome and protected-group membership are available. The procedure satisfies that definition; it does not make the decision fair under every framework.

Group calibration asks a score to have comparable meaning: among people assigned risk 0.7, roughly the same share should experience the outcome. This supports interpretation of scores but does not guarantee equal error rates when prevalence and thresholds differ.

Individual fairness proposes treating similar people similarly. Dwork, Hardt, Pitassi, Reingold and Zemel made the need for a task-specific similarity metric explicit. That is where the normative difficulty lives: who defines similarity, which attributes matter and whether the distance reproduces earlier inequality.

They cannot always be satisfied together

Kleinberg, Mullainathan and Raghavan studied calibrated risk scores and two balance conditions for positive and negative classes. Except in restricted situations, such as perfect prediction or particular relationships between base rates, all three conditions cannot hold together.

The theorem does not say fairness is impossible or choose a metric. It says a dashboard showing every indicator in green may hide a trivial definition, changed population or mistake. An organisation must state which condition it prioritises and why failing another is acceptable or mitigated elsewhere.

The reference itself may be unjust or measured with different error across groups. Equalising rates against a defective label stabilises its defect. Before optimisation, teams should audit how the outcome arose, who was left unobserved and which prior decisions caused it.

Pre-, in- and post-processing intervene at different places

Data interventions can expand representation, review labels, reweight examples or repair collection. Removing a protected column rarely removes proxies: postcode, work history or language patterns may reconstruct correlated information. Sometimes the protected attribute is needed precisely to measure disparities, with suitable privacy safeguards and legal basis.

Training can include constraints or several objectives. This exposes the tension between performance and a chosen measure, but each weight and evaluation set remains a decision. Post-processing adjusts thresholds or randomises decisions to reach selected rates; it is easier to apply to an existing model but may explicitly treat groups differently and needs legal and operational analysis.

No technique by itself repairs an unnecessary product or an institution with no remedy. If a prediction affects liberty, credit, employment, health or education, authority, explanation, competent human review, appeal and redress also matter. A statistical improvement can reduce one harm without legitimising the whole system.

The trap of abstracting away the social system

Selbst, Boyd, Friedler, Venkatasubramanian and Vertesi described five abstraction traps in sociotechnical systems. They include treating a social problem as isolated, ignoring feedback and moving a solution designed for one context into another.

A policing model, for example, may receive more records from areas already under heavier surveillance and recommend more surveillance there. The next observation depends on the previous decision. Treating those records as a passive sample of “where crime happens” confuses observed activity with the whole phenomenon and creates a loop.

The alternative is not to abandon measurement but to draw the system: who decides, who receives benefits, who bears errors, what data each intervention produces and which option existed before. The relevant comparator may be an opaque human decision, a simple rule or not using the system, rather than a costless ideal.

Documenting data and models exposes conditions

Datasheets for Datasets proposes documenting a dataset's motivation, composition, collection process and recommended uses. A useful sheet includes represented population, known absences, labelling procedure, consent where applicable, transformations, versions and conditions for updating or retiring data.

Model Cards for Model Reporting carries the idea to a model: intended use, unsuitable uses, evaluation procedure and performance across relevant conditions and groups. A card does not guarantee an ethical system. It preserves context that lets another person challenge an extrapolation.

Both documents need version links. If data, model, threshold or population changes, an earlier evaluation may stop describing the service. A decision log records why a metric was chosen, who participated, which alternatives were rejected and when review is due.

An audit that ends with effects

The first step states harm, decision and population. The second compares data and error distributions, including intersections and denominators. The third chooses measures relevant to the harm and acknowledges conflicts. The fourth intervenes where the problem arises and measures utility and disparities again.

After deployment, the audit follows outcomes, corrections, complaints and distribution changes. Selection rates do not show whether selected people received a benefit; equal error does not establish a valid label; a monthly average can conceal sudden deterioration. Audits need consequences and an expiry date.

The final claim should preserve limits: “in this dataset, version and threshold, the estimated difference in this rate was X, with this denominator”. That will not fit inside the word “fair”, which is why it is more useful. Algorithmic fairness is not a property installed once; it is a choice about harms, documented, challenged with affected people and revisited as the system changes.