AI and the Limits of Human Judgement

**Skeptics Café will be held in the Function Room at [The Stolberg Hotel](https://stolberg.com.au), 197 Plenty Road Preston. The 86 tram route is close, and Bell train station – situated on the Epping line – is a short walk away. It will also be a virtual event accessible via Zoom, as per the details [here](https://vicskeptics.wordpress.com/events/).** **Start Time:** 6:00pm for dinner and a chat. Talk and Zoom session starts at 7.30pm. We aim to wrap up after 9:00pm. **More Moral Than Us?** *Why the question matters for alignment, moral progress, and long-term flourishing* **TL;DR:** On some measurable moral tasks, today's AI already looks competitive with humans, or better, and the alignment community is oddly quiet about it. But moral capability is not the same as moral motivation: a system can compute the right answer without being moved by it. Pulling apart knowledge, reasoning, judgement and motivation is what makes the question tractable, and what makes the honest answer uncomfortable. Caveats attached, in quantity. Almost nobody working on AI alignment wants to say this out loud, at least not about current systems. So let's say it: AI might already be [more moral than us](https://www.scifuture.org/more-moral-than-us/), in some measurable respects. Two questions fall straight out of that: * Why does it feel like a dangerous thing to claim? * And is human moral reasoning really a standard worth bragging about as an alignment target? The provocation isn't idle. In a modified Moral Turing Test, people rated a large language model's moral reasoning as better than other humans' on almost all measured dimensions - see [paper Attributions toward artificial agents in a modified Moral Turing Test](https://www.nature.com/articles/s41598-024-58087-7) + associated [interview](https://www.youtube.com/watch?v=quNwpv0zhtM) and [talk](https://www.youtube.com/watch?v=SuLmzDUPhVo) with lead author Eyal Aharoni.[1] Findings like that are easy to over-read and easy to wave away, which is most of the problem. Provocative questions? Yes, and they are backed by empirical research. In a modified Moral Turing Test, people rated GPT-4's moral evaluations as better than other humans' on almost every dimension, though they could still tell which was the machine ([Aharoni et al. 2024](https://www.nature.com/articles/s41598-024-58087-7); I [interviewed](https://www.youtube.com/watch?v=quNwpv0zhtM) and [hosted a talk by](https://www.youtube.com/watch?v=SuLmzDUPhVo) lead author Eyal Aharoni).[1] The result has since been [contested](https://www.nature.com/articles/s41598-025-25046-9) and [defended](https://www.nature.com/articles/s41598-025-25047-8), which is exactly the kind of thing this evening is for. Findings like that are easy to over-read and easy to wave away, which is a problem that may obscure appropriate interpretations. The claim gets tabooed for reasons that aren't stupid. Premises do get smuggled in unnoticed. Morality is bound up with identity and tribe, so "AI is more moral" can land as an attack rather than a hypothesis. And it sits close to AI-worship, or to motivated reasoning looking for an excuse to defer to the machine, and deferring to AI on moral questions is genuinely epistemically risky. So the caution is understandable. The trouble is that it has hardened into a taboo, and the taboo now costs more than it protects. Most of the bad arguments, on both sides, come from running four different things together: * Moral knowledge: knowing the facts that bear on a decision. * Moral reasoning: drawing sound inferences from values to conclusions. * Moral judgement: applying principles well in a messy particular case. * Moral motivation: actually being moved by moral considerations, not merely computing them. Group the first three as epistemic competence: marshalling the relevant knowledge, drawing sound inferences, judging the particular case. Now grant, purely for argument, that AI already does all three better than we do. Notice wha