AI Is Not Neutral. It Never Was.

The assumption tends to go something like this: computers don't have feelings, so AI must be neutral. Feed in data, get out a result, no thumb on the scale. The reality is messier.

AI systems learn from human-generated data, and human-generated data reflects human history. A model trained on historical hiring records learns who got hired. A model trained on criminal sentencing data learns what sentences courts actually gave, not what they should have given. The model finds patterns in the data, and the patterns in the data are not neutral.

The most-cited example is COMPAS, a recidivism risk algorithm used by courts in the United States to assess how likely a defendant was to re-offend. ProPublica's 2016 investigation found it was nearly twice as likely to falsely flag Black defendants as future offenders compared to white defendants. The company behind the tool disputed the methodology. The underlying court data wasn't in dispute.

Similar patterns have been documented in facial recognition systems, automated hiring tools, medical diagnostic models, and large language models. Research consistently finds that biases in training data and benchmark selection show up in model outputs, often in ways the developers didn't intend and didn't notice until someone looked for them. A 2023 NIST framework on AI risk management lists bias as a top concern across virtually every deployment context.

The source isn't always obvious. It can come from training data, from the choice of what to optimize for, from how evaluation tasks were assembled, or some combination. The machine learning pipeline doesn't introduce neutrality where the inputs lacked it.

That's not an argument that AI is irredeemably biased or that using it is always wrong. It's an argument for asking specific questions about any specific system: what data did it learn from, what is it being used to decide, and who reviewed the outputs before deployment? The neutrality assumption closes off those questions before they get asked. That's where the real problem lies.