Who is in control of AI?

It may already be too late to shut down the existing AI systems that could destroy civilization.

It’s easy to dismiss AI risk: “If it starts to get out of control, we can just pull the plug.” Most AI safety primers have a section addressing that. By the time we realize it’s getting out of control, it may already have amassed enough power that it’s too late. An out of control AI may do everything it can to resist termination and ensure its own survival. In some scenarios, “everything it can” includes “kill all human beings.”

So… who or what is in control of Mooglebook‘s AI right now?1

There’s no big red button anyone at Mooglebook can push to shut it down.

Mooglebook can’t stop optimizing for ad clicks. There are people inside and outside the company who realize it has dire negative externalities, and they are trying to make those less bad, but they’ve brought water pistols to a tactical nuclear war.

If Mooglebook’s executive team unanimously agree that its activities are harmful, and they want to get out of the advertising business and pivot the whole company to rescuing abused beagles, they cannot do that. They would be fired by the board immediately. If the board agreed, they would be fired by the shareholders. If somehow the advertising business did get shut down, the company would go bankrupt in a few months, and less scrupulous competitors would pick up the slack.

The institution has its own agency: its own purposes, plans, reasons, and logic, which are more powerful than the humans it employs.2 Those, however, are subordinate in turn to the AI the company depends on for its survival. If enemies of Mooglebook’s AI—activists, regulators, competitors—try to harm it, the institution can’t not do everything in its power to defend itself. As, in fact, it is currently doing.

Humans don’t have control over Mooglebook’s AI, not individually, nor as defined groups, nor perhaps even as a species.

Mooglebook AI is not plotting to destroy the world—but it may destroy the world unintentionally, and we may not be able to stop it.

  1. 1.Recall that “Mooglebook” is a humorous term for the advertising technology industry, prominently including Microsoft, Google, and Facebook.
  2. 2.This is not to absolve individuals at Mooglebook, nor the company as a legal entity, of responsibility. They do have some power to change things on the margin, and should. The point, however, is that identifying them with the overall problem leads to an incomplete and inaccurate analysis.