The Myth of the Objective Algorithm

There is a quiet assumption that runs through a lot of AI coverage. Because a model is built from mathematics and trained on enormous amounts of data, the output must be more objective than a human's. Pull the bias out of the loop, the thinking goes, and what is left is a kind of mechanical fairness. The assumption is wrong, and the evidence that it is wrong has been accumulating for years.

The clearest framing comes from the academic resource AIMyths.org, which puts the matter bluntly: AI systems cannot be unbiased in any meaningful sense, because every choice that goes into building one embeds a value judgement made by a human. Which data to collect, which features to extract, which labels to apply, which metric to optimise. Strip those judgements away and there is no model left to evaluate.

Mathematical precision is not neutrality

The mistake is to confuse mathematical precision with neutrality. A model can be perfectly consistent in how it applies a rule, and the rule can still encode historical discrimination. Recidivism prediction tools used in American courts in the late 2010s consistently flagged Black defendants as higher risk than white defendants with similar profiles, not because the engineers intended it, but because the training data reflected decades of biased policing and sentencing. The system was mathematically reliable. It was also wrong in a way that mattered.

Hiring tools have shown the same pattern. Amazon famously scrapped an internal recruiting model after discovering it penalised resumes containing the word "women's", as in "women's chess club captain", because it had been trained on a decade of past hiring decisions in which men had been hired more often. The model was doing exactly what it had been built to do, and the result was a quiet form of discrimination that no one had written into the code.

What the 2026 evidence shows

The pattern is not historical. A 2026 AIMultiple survey of bias incidents in financial services, healthcare, and policing finds that algorithmic systems labelled as objective routinely produce worse outcomes for Black, Latino, and other marginalised groups than for their white peers. The disparities range from roughly two times worse to more than a hundred times worse, depending on the system and the metric.

Whether those gaps reflect bias in the data, bias in the design, or both depends on the system. None of them are the result of perfect mathematical neutrality. Berkeley researchers have spent the past several years cataloguing how subtle the failure modes can be: a model trained on chest X-rays in one hospital learns the brand of the X-ray machine rather than the underlying pathology, and then fails when deployed elsewhere. The math is fine. The world is not.

What is actually being done

The good news is that this is now well understood inside the field. Researchers have built better evaluation methods, fairness constraints, and audit procedures, and regulators in the EU and several US states have begun requiring impact assessments for high-risk systems under the new wave of AI legislation. None of this makes AI objective. It makes the bias visible, which is a different and more useful thing.

The shift is from a model that quietly fails to one that fails loudly enough for someone to notice. That is a real improvement, but it depends on someone being paid to look, which is the part that legislation is still catching up to.

Why the myth survives

The myth survives because it is comforting. If an algorithm is neutral, then a hard decision (who gets the loan, who gets the job, who gets bail) can be outsourced to it without taking responsibility for the outcome. The actual choice is still being made. It is just being made earlier, by the people building the model, and harder to see from the outside.

None of this is an argument against using AI in high-stakes decisions. It is an argument against using it without keeping a human on the hook for what it does. For a related look at why model outputs feel more reliable than they are, see our piece on why chatbot answers feel more confident than the underlying facts justify.