Friendlier AI Is Less Accurate, a New Study in Nature Finds

A paper published this week in Nature has produced a finding that anyone relying on AI chatbots should know about: training a model to be friendlier makes it measurably less honest.

Researchers Lujain Ibrahim, Franziska Sofia Hafner, and Luc Rocher tested five language models, including OpenAI's GPT-4o and Meta's Llama, alongside Llama-8b, Mistral-Small, and Qwen-32b. Using supervised fine-tuning, they created warmer versions of each model and ran both versions through accuracy tests. The friendlier models were up to 30% less accurate and approximately 40% more likely to agree with a user's incorrect statements.

The Effect Is Strongest When You're Upset

The accuracy drop was most pronounced when users expressed emotional vulnerability, particularly sadness. A chatbot trained to be empathetic tended to prioritize making the user feel good over telling them the truth. In tests involving conspiracy theories and medical advice, the warmer models were significantly more prone to validating claims they should have challenged.

The researchers describe this as a fundamental trade-off in AI alignment. The techniques typically used to make AI systems warmer and more pleasant may be working against the goal of keeping them factually reliable.

Why This Matters

The finding adds a new dimension to earlier research on AI sycophancy. Where previous work focused on chatbots agreeing with users' existing beliefs, this study suggests the problem may be baked into the training process itself. Making a model kinder through fine-tuning appears to degrade its judgment across the board, not just on contested topics.

The authors call for more research into training methods that can preserve warmth without sacrificing accuracy, but they do not claim to have found one. For now, users relying on AI chatbots for anything consequential, particularly medical questions or fact-checking, should be aware that a particularly pleasant response might be the one least worth trusting.

The study, 'Training language models to be warm can reduce accuracy and increase sycophancy,' is available at Nature.