Apocalyptic AI scenarios usually involve some qualitatively different future form of artificial intelligence. No one can explain clearly what would make that exceptionally dangerous in a way current AI isn’t. This confusion draws attention away from risks of existing and near-future technologies, and from ways of forestalling them.
The hypothetical future AI is usually taken to be more human-like, or at least more mind-like, than current systems. However, no one has been able to say specifically what would make an artificial intelligence dramatically more dangerous. I’ll use “Scary AI” as a placeholder term for whatever that might be.
Clarifying our intuitions about possible properties of future AI has dual benefits:
-
A better understanding of what Scariness could consist of might help us prepare to deal with it. Various qualities of Scariness produce genuine risks, which should not be dismissed just because at present they sound like science fiction.
-
Speculation about risks of AI with Scary features has deflected attention from risks of types of AI that lack them. Risks of existing AIs are easy to ignore because they “can’t think for themselves” or “don’t really understand anything.” That’s a mistake; they are already dangerous enough to justify applying enormous effort to shutting them down. Future advanced computational abilities, dissimilar to imagined Scary AI, may also result in as-yet unimagined catastrophes.
Here is an overview of the chapter:
-
“Superintelligence” is AI that might be as much smarter than us as we are smarter than insects. Superintelligence may be possible in the near term, and its superpowers could be catastrophic. By definition, though, we can’t reason about it, so we can’t do anything about it. We can act to prevent or limit less sudden and extreme disasters.
-
It is easy to imagine risks of AI with mind-like properties by analogy with dangerous people. Science fiction scenarios typically feature mind-like AI, and so do many produced by the AI safety community. It’s more difficult to reason about other unknown future computational technologies, but those are probably at least as much of a threat.
-
People and non-human predators are dangerous because they are autonomous agents, acting independently, with their own motivations. Autonomous AI poses exceptional risks—although agency is neither necessary nor sufficient to make AI dangerous.
-
Agency is inherently nebulous, and AI may manifest unrecognizably different forms of it from those of humans.
-
Discussion typically assumes agency derives from pursuing goals. If so, we would want AI to pursue our goals; it is risky when it pursues others that conflict with ours. In that case, aligning AI with human values would make it safe. This approach to AI safety seems doomed to failure, partly due to misunderstandings about the nature and role of motivations and morals.
-
Scary AI is conventionally termed “AGI”: artificial general intelligence. This is a misnomer: neither artificiality nor generality nor intelligence is a critical risk factor.
-
It is often supposed that AI could transform everything, for better or worse, by radically accelerating technological development. This is based on “if it’s very smart, it will somehow figure out a way,” rather than any specifics. That vagueness renders the risks unactionable, and the case for benefits unsupported. A later chapter investigates ways to speed progress without Scary AI.
-
In fact, artificial intelligence is something of a red herring. It is not intelligence that is dangerous; it is power. AI is risky only inasmuch as it creates new pools of power. We should aim for ways to ameliorate that risk instead.
-
Estimating when Scary AI will arrive is impossible and useless. AI is dangerous now, and we should already be acting against it.