The Imminent Existential Risk to Humanity from AI

Next Monday I am taking part in a panel discussion Beyond the Matrix: The reality of AI risk and this is my opening position statement:

The point of The Matrix is that human beings continue to live but they no longer live meaningful lives. My concern is that if LLMs (and text-to-image generators) in the form of chatbots and AI assistants achieve what Illich called a ‘radical monopoly’ which ‘imposes compulsory consumption’, this will make it almost impossible for the majority of humans to live a meaningful life. This is not a problem with the technology itself, but with the way certain distinctive human values have been built into the tools currently vying for that position of radical monopoly: they are designed to implement a particular, value-laden conception of intelligence. We can see this in the standard definition of intelligence in AI as ‘the computational part of the ability to achieve goals’ (McCarthy). The concept of computation has an unremarked normative dimension: faster computation is better; more powerful (= more resources) computation is better; optimizing computation is better. These ways of evaluating computation, and by implication intelligence, align with the human values of perfectionism, ambition, competitiveness, and often greed and power-seeking. Alternative values are available. But if a technology which has been built, without discussion, debate or consent, to embody those values achieves radical monopoly, those values will also dominate human lives, effectively forcing people to live by them and making it practically impossible for most people to choose a different, and to my mind, better, life. The most imminent existential risk is that LLMs will force the values of Silicon Valley onto the whole of humanity (or at least, the ~6 billion who live in a digitised environment), like some sort of global totalitarian state.

To explain the last comment, totalitarian states prevent the overwhelming majority of their citizens from living a meaningful life by imposing one set of values on all within its power. If you endorse those values, totalitarianism may not seem so bad to you, but even then, you are only ever complying or conforming, never choosing. And if you do not endorse those values, you are being coerced.

Elsewhere I have argued that we have moved into an era of Digital States. These digital states already use their immense coercive power to impose values on their citizen-users of social media and SaaS platforms. The attempt to impose compulsory use of LLMs by forcing them into existing services and giving users little or no option to opt-out further increases the power of these digital states to coercively enforce their chosen values upon the rest of humanity.