Care, Human Enfeeblement, and the Existential Implications of AGI

arXiv:2611.14520v1 Announce Type: cross Abstract: Stuart Russell has named "human enfeeblement" as a long-term societal risk of advanced AI. This paper gives a new interpretation of the enfeeblement danger by interpreting it in light of recent work by Brian Christian, Alison Gopnik, and David Spivak on care and caregiving, and connecting it with the ontological design tradition of Terry Winograd and Fernando Flores. The enfeeblement that threatens is not merely a matter of cognitive off-loading and loss of knowledge, but a form of disengagement with the world which I explain in terms of an erosion of our capacities to care. This paper articulates a conception of care as a constellation of cultivable skills that involve us in the world and with each other, while making life worth living. Designing AI to support our capacities to care addresses the worry recently raised by Demis Hassabis and others regarding the questions of "meaning and purpose" that are looming in a post-AGI world.