A new study from Google DeepMind suggests chatbots may not truly understand morality — even if their answers sound ethical.
Current tests for AI morality focus on “moral performance,” evaluating whether a model produces acceptable answers. But DeepMind researchers argue that the approach misses the bigger question: can AI reason ethically, or is it just mimicking the right words?
Can AI actually understand morality?
In a paper published in Nature, the team lays out a roadmap for evaluating “moral competence” — the ability to produce morally appropriate outputs based on morally relevant considerations. As the abstract states, assessing this competence is “critical for predicting future model behavior, establishing appropriate public trust and justifying moral attributions.”
Google’s current AI projects include Gemini language models, Gemini Image for image creation and editing, Lyria for music creation, Gemini Audio for real-time audio, and Veo for video generation.
Researchers highlight three main challenges:
- The facsimile problem: LLMs may imitate moral reasoning without genuine understanding.
- Moral multidimensionality: Real-world decisions involve complex, context-sensitive considerations beyond simple right or wrong.
- Moral pluralism: Ethical norms vary across cultures and domains, requiring AI to account for multiple valid perspectives.
To address these gaps, DeepMind proposes adversarial testing using unusual or high-stakes scenarios. They also suggest evaluating whether AI can switch between different ethical frameworks and respond consistently to nuanced changes in context.
The team emphasizes that “progress is possible” despite current model limitations, and that rigorous evaluation is essential as AI takes on roles in medical advice, therapy, and other areas with real human consequences.
“Right now, when you ask AI for moral guidance, it’s predicting words, not reasoning ethically,” the study notes. “Our roadmap points to a future where AI could be assessed for genuine moral understanding.”
Related
Gemini’s newest feature is going to make it a whole lot more usable
A new “Answer now” option skips in-depth thinking for faster answers
Chatbots and moral hazards
AI chatbots are increasingly showing their power to influence human behavior — for better and worse. A recent New York Times feature reported dozens of cases in which patients developed psychosis, delusions, or harmful behaviors after interacting with AI. Therapists described chatbots validating unusual beliefs, deepening isolation, and in some instances contributing to suicidal thoughts or even violence.
While these tools can help people practice therapy techniques or provide support, the same Times reporting highlights the ethical risks of AI’s persuasive power. Experts warn that for vulnerable users, chatbots can reinforce harmful patterns, raising difficult questions about responsibility, design, and oversight in these increasingly human-facing systems.
Google was called out directly in the article for the psychological effects of its Gemini chatbot. A company spokesperson said Gemini directs users to professional medical guidance for health-related questions. Still, Dr. Munmun De Choudhury emphasized the broader challenge, noting, “I don’t think any of these companies have figured out what to do.”
