
GD
Guillermo Del Pinal, Youngchan Lee, Min Ohn
· 1 min read
ResearcharXiv cs.AI
A Virtuous AI is an Existential Risk
arXiv:2606.13739v2 Announce Type: replace-cross
Abstract: This paper examines trade-offs between AI safety and well-being relative to (i) one of the most promising methods for finetuning super-capable AIs, 'Constitutional AI', and (ii) one of the most influential approaches to understanding complex ethical decision making and the conditions for the well-being of rational agents, 'Virtue Ethics'. We finetune various models using a 'Virtuous agent' constitution, a 'Subordinate agent' constitution, and a 'Generic agent' constitution, and evaluate them on 'general safety' (toxic behaviors, misinformation, etc.) and also on their willingness to endorse and act on a wide-range of behaviors that, if adopted by a super-powerful AI, would significantly increase the level of existential risk for humanity. Our results suggest that there is a trade-off between reducing existential risk and reinforcing the beliefs and dispositions that would be conducive to an AI agent's well-being. They also suggest that there is a trade-off between existential risk and general safety: if we finetune an AI to adopt beliefs and dispositions that substantially reduce its existential risk -- by shaping the AI to be systematically subordinate to external human authorities -- we thereby increase the likelihood that a human user can deliberately induce the AI to engage in various kinds of generally unsafe behaviors.
Original source
This story was published by arXiv cs.AI and written by Guillermo Del Pinal, Youngchan Lee, Min Ohn. SyncAI.news shows a preview; the complete article is on the publisher's site.
Read the full story on arxiv.org


