What is the Scary kind of AI?

Apocalyptic AI scenarios usually involve some qualitatively different future form of artificial intelligence. No one can explain clearly what would make that exceptionally dangerous in a way current AI isn’t. This confusion draws attention away from risks of existing and near-future technologies, and from ways of forestalling them.

The hypothetical future AI is usually taken to be more human-like, or at least more mind-like, than current systems. However, no one has been able to say specifically what would make an artificial intelligence dramatically more dangerous. I’ll use “Scary AI” as a placeholder term for whatever that might be.

Clarifying our intuitions about possible properties of future AI has dual benefits:

Here is an overview of the chapter: