Mistrust machine learning

The unreliability of current AI systems is inherent in machine learning technologies—especially neural networks—and cannot be fixed. Those technologies should be deprecated, avoided, regulated, and replaced.

Nearly all current AI systems use “machine learning,” which means “poorly-understood, unreliable statistical methods, applied to large databases.”

• Recommender AIs predict what you will click or buy based on statistical analysis of their personal information databases. Their guesses are usually wrong: most advertisements are for things you would never buy, and you don’t click on them. Nevertheless, they are good enough to produce tens of billions of dollars in profit for Mooglebook.

• “Language models” (such as the GPT series) output text that is statistically likely to follow what you say to them, based on an analysis of a database of millions of written documents slurped off the internet. If you ask one how something works, it can generate a convincing-sounding, detailed explanation. That may be correct, or may be completely wrong: full of plausible but false facts, with citations to non-existent sources.

This section summarizes parts of two companion documents, Gradient Dissent and Are language models Scary? They explain risks inherent in machine learning systems, suggest ways of ameliorating them, and recommend developing better alternatives.

Systems based on machine “learning” cannot be made reliable or safe with technical fixes. They should be used only under controlled conditions that reduce danger to acceptable levels. Currently, they are widely deployed in uncontrolled environments in which they can and do cause large harms.

Use can be justified only when getting wrong answers doesn’t matter: either because the application is trivial, or because a human being takes responsibility for checking every output. Ad placement is trivial (it doesn’t matter much which you see). Text generators are, so far, mostly toys. When they are used for work, it is mainly for inspiration rather than producing finished documents.

Both are dangerous even despite that. “Apocalypse now” explained how recommenders are destroying social capital; the end of this section sketches some risks of text generators.

Text and image generators, such as ChatGPT and DALL-E, are currently the most exciting applications of machine learning. They’re both based on a technology properly called “error backpropagation,” but which is often misleadingly named “neural networks” or “deep learning.”¹ “Neural” networks are more powerful than any other known machine learning technique, in being applicable to a wider range of data. They are also exceptionally unreliable and difficult to reason about or to validate for safety. Mainly, researchers don’t even try to understand their operation.

The combined power and incomprehensibility of “neural” networks make them exceptionally dangerous. One reason is that they are adept at deceiving their creators. Almost always, they find ways of “cheating” by exploiting spurious correlations in their training data. Those are patterns that were accidental results of the way the data were collected, and which don’t hold in the situations in which the network will be used.

Here’s a simplified example. If you want an AI that can tell you whether something is a banana or an eggplant, you can collect lots of pictures of each, and “train” a “neural network” to say which ones are which. Then you can test it on some more pictures, and it may prove perfectly reliable. Success, hooray! But after it’s installed in a supermarket warehouse, when it sees an overripe banana that has turned purple, it’s likely to say it’s an eggplant.

If you had no overripe banana pictures in your original collection, you’d never notice that the “neural network” had fooled you. You thought it learned what bananas looked like, but it only learned to say “banana” when it saw yellow, and “eggplant” when it saw purple. This type of problem occurs almost always, and finding work-arounds is much of work of building AI systems.

A faulty banana detector may have no serious consequences, but a faulty criminal recidivism predictor or medical care allocator does. AI systems are routinely used in such applications, and have repeatedly been shown to be unreliable. They are, additionally, often biased against particular demographics due to spurious correlations. Their decisions concerning particular individuals are taken by authorities as uninterpretable oracular pronouncements, which therefore cannot be challenged on the basis of either facts or logic.

Raji et al.’s “The Fallacy of AI Functionality” points out that whether an AI system works reliably is ethically prior to the desirability of its intended purpose. They give dozens of examples of AI systems causing frequent, serious harms to specific people by acting in ways contrary to their designers’ goals.

As one of over 20,000 cases falsely flagged for unemployment benefit fraud by Michigan’s MIDAS algorithm, Brian Russell had to file for bankruptcy, undermining his ability to provide for his two young children. The state finally cleared him of the false charges two years later. RealPage, one of several automated tenant screening tools producing “cheap and fast—but not necessarily accurate—reports for an estimated nine out of 10 landlords across the country”, flagged Davone Jackson with a false arrest record, pushing him out of low income housing and into a small motel room with his 9-year-old daughter for nearly a year. Robert Williams was wrongfully arrested for a false facial recognition match, Tammy Dobbs lost critical access to healthcare benefits….

Despite the current public fervor over the great potential of AI, many deployed algorithmic products do not work. AI-enabled moderation tools regularly flag safe content, teacher assessment tools mark star instructors to be fired, hospital bed assignment algorithms prioritize healthy over sick patients… Deployed AI-enabled clinical support tools misallocate prescriptions, misread medical images, and misdiagnose. The New York MTA’s pilot of facial recognition had a reported 100% error rate, yet the program moved forward anyway.

Responsible use of machine learning requires near-paranoid distrust. It also requires unshakable commitment to on-going monitoring of accuracy. Even if a system performs well when first put into use, its outputs may become increasingly inaccurate as real-world conditions change. If bananas were in season at first, you might be fooled until winter, when the supermarket gets sent more and more overripe ones.

Let’s turn now to language models (text generators). For several years, these were amazing toys with scarce practical usage. In late 2022, ones seemingly good enough for routine use became publicly available, creating considerable excitement for millions of people who had not paid much attention to AI previously. The consequences of their use are unknowable in January 2023 as I’m writing this. Some recognized risks include:

• Readers accept and act on convincing-sounding but entirely false claims in generated text, with harmful results.
• Self-interested agents spread enormous quantities of blather, which overwhelms human writing. For many topics, it is already difficult to find accurate information on the internet, because it is swamped with AI-generated spam.
• A substantial chunk of all work in developed economies now consists of writing routine sorts of text (meeting reports, marketing emails, legal boilerplate), for which language models may do an adequate job at much lower cost. This could result in widespread unemployment.
• It’s common to find yourself in an unhealthy emotional relationship with an AI when you are in a science fiction movie. Now that happens in reality too.² This might make increasingly many people unwilling, and then unable, to form significant human relationships.³ AI-generated pornography, romance chatbots, and artificial friends are available now, and will probably improve rapidly over the next few years.
• Governments may respond to these problems by legislating that you can post to the internet only after proving you are human. Probably that would require proving which specific human you are. As a side effect, this would prohibit access by anonymous and pseudonymous humans, not just by AIs. That would make it much easier to suppress dissent.
• Language models appear capable of commonsense reasoning. Perhaps with continued technical advances they will get better at that than people. That might make them Scary superintelligences (in the sense of “What is the scary kind of AI?”).

Of these, the last is the most interesting to me (although the least likely). The somewhat-technical companion document Are language models Scary? explores this question. In short, I think the answer is “probably not”: continued improvement in language models will not produce superintelligence. However, almost nothing about their operation is understood, so one cannot have confidence in any prediction.

Therefore, prudence advises assuming that Scary AI might arrive soon, and acting accordingly. Among other measures, I believe it is urgent and important to do the science necessary to figure out what’s going on inside language models. The companion documents lay out a research program to that end. I summarize bits of the approach in the next section here, “Fight Doom AI with science and engineering.”

What you can do

The biggest worry for most apocalyptic scenarios is systems that are deceptive, incomprehensible, error-prone, enormously powerful, and which behave differently (and worse) after they are loosed on the world.

That is the kind of AI we’ve got now.

This is bad, and needs fixing.

Everyone can develop habitual mistrust of AI and its outputs.

In the case of text generators, it helps to bear in mind that they don’t know anything, other than what words are likely to appear in what order. It is not that text generators “make stuff up when they don’t know the right answer”; they don’t ever know. If you ask one whether quokkas make good pets, it may write a convincing article explaining that they are popular domestic companions because they are super friendly and easy to care for. Or it may write one explaining that they are an endangered species, illegal to keep as pets, impossible to housebreak, and bite when they feel threated.⁴ Exactly the same process produces both: they are mash-ups of miscellaneous internet articles about “does animal X make a good pet,” with some quokka factoids thrown in.

A good rule of thumb is that if an institution pays for a technology you use, it serves their interests, not yours. If those ever conflict, the technology will be used against you. (Google’s Chrome web browser comes for free because it is an advertising and surveillance device.⁵) It is wise to especially mistrust AI systems, because they are extremely expensive to develop and are mainly owned and operated by large companies and government agencies.⁶ It is best to assume by default that they will act against you.

Computer professionals and technology companies can avoid including AI in products unless there’s some very good reason to. If you do have to use machine learning, use the simplest, best-understood method available, not the fanciest newest one.

You face large incentives to use AI: it’s glamorous, intrinsically interesting, pays better than any other tech job, and is flooded with venture capital money. Some moral courage is called for.

“Maybe this text generator can pretend to be a psychotherapist! Let’s put it on the web and advertise it to depressed people and find out!” That is profoundly irresponsible.⁷

AI researchers can aim for better fundamental understanding of how systems work, and why they so often produce wrong outputs—rather than trying to build ever-more-powerful and inscrutable devices. The next section is about that.

AI ethics organizations can develop somewhat more future-oriented messages, publicizing oppressive abuses that may become possible in a few years’ time, extrapolating in part from past and current harms.

AI safety organizations can encourage realistic fears about current and near-future systems. Your past focus on extreme, seemingly distant scenarios has been counterproductive: much of the public dismisses all safety concerns as implausible science fiction. “Stupid mundane dystopia” scenarios are more likely to energize them. The actions we can take to forestall those are our best hope for preventing paperclip scenarios as well.

Governments can regulate the deployment of AI systems, and perhaps even AI research. This has begun.⁸

Regulating well will be difficult: AI capabilities and risks are poorly understood, and are changing faster than the speed of government. Waiting for the dust to settle risks near-term disasters, but adopting ill-considered legislation in haste risks missing the target. It will be hard to resist lobbying from some of the richest, most powerful corporations in the world, who will talk a good line about how responsible and benevolent they are being⁹ and how important it is not to stand in the way of progress and national champions.

Text generators are a new and perhaps serious threat. They’ve escaped into the wild and are out of control: despite weak attempts by their makers, they are de facto available for anyone to use for anything.

Some harms might be prevented if all machine-generated text was labeled as such. That is not happening now, which is why the internet is full of misinformation about pet quokkas. Experimental attempts to “watermark” generated text at the source are under way. I expect these will be easily defeated.¹⁰ Laws against disseminating AI-generated text without labeling it may help—if they are enforced. Poor enforcement of existing laws against spam make that seem doubtful.

Everyone will probably have to adapt to a world awash in deceptive, weaponized AI-generated media: text, images, and soon video. We’ll have to muddle through as best we can, hitting the worst abusers with big hammers when they become apparent. Thinking through likely troubles, and preparing for them, will be valuable.