Artificial superintelligence beyond our control poses great risk

If a similar asteroid or comet were to crash into Earth today, it would cause another mass-extinction event, wiping out most species and human civilisation as we know it.

By Daron Acemoglu

  • Follow us on
  • google-news
  • whatsapp
  • telegram

Published: Mon 31 May 2021, 11:08 PM

The reign of the dinosaurs was brought to an end 65 million years ago by an asteroid that crashed into what is now the town of Chicxulub in Mexico. Although this lump of rock and metal was not particularly large — probably about 10 kilometres across — it struck the Earth at more than 60,000 kilometres per hour, generating an explosion billions of times greater than that of the atomic bomb dropped on Hiroshima, and killing all life within 1,000 kilometres.

If a similar asteroid or comet were to crash into Earth today, it would cause another mass-extinction event, wiping out most species and human civilisation as we know it. This distant possibility is an example of a natural existential risk: an event not caused by humans that leads to the extinction or near-extinction of our species.


But there are also anthropogenic — human-created — existential risks. As the University of Oxford philosopher Toby Ord argues in his thought-provoking new book, The Precipice: Existential Risk and the Future of Humanity, it is these risks that should most concern us now and in the coming century.

Risk and Reward


Ord recognises that science and technology are humankind’s most potent tools for solving problems and achieving prosperity. But he reminds us that there are always dangers associated with such power, particularly when it is placed in the wrong hands or wielded without concern for long-term and unintended consequences.

More to the point, Ord argues that anthropogenic existential risk has reached an alarmingly high level, because we have developed tools capable of destroying humanity without the accompanying wisdom needed to recognise the danger we are in. He notes that the eminent twentieth-century astronomer Carl Sagan issued a similar warning in his 1994 book, Pale Blue Dot, writing:

“Many of the dangers we face indeed arise from science and technology — but, more fundamentally, because we have become powerful without becoming commensurately wise. The world-altering powers that technology has delivered into our heads now require a degree of consideration and foresight that has never before been asked of us.”

For Ord, this gap between power and wisdom could decide humanity’s future. On one hand, we could disappear entirely or suffer a collapse that wipes out most of the hallmarks of civilisation (from antibiotics to art and writing). But, on the other hand, Ord sees in humankind the potential for long-term flourishing on a cosmic scale: with both wisdom and technological ingenuity, humans could well outlive this planet and launch new civilisations across space.

This far-reaching vision of flourishing weighs heavily in Ord’s reckoning, because he recognises that there may not be any other intelligent life forms in the universe. If we are indeed alone, a mass-extinction event that wiped out everyone on this planet would also eliminate all of the potential for intelligent, purposeful existence everywhere.

Based on this reasoning, Ord arrives at what mathematicians and economists would call a “lexicographic preference ordering”. In a situation where we care about multiple criteria, a lexicographic order assigns overwhelming importance to one criterion in order to provide clarity when two options are being compared. For example, in a lexicographic order between food and shelter, one would always prefer whichever option offers more food, regardless of how much more shelter the other option offers.

Ord’s philosophical stance is equivalent to a lexicographic order because it implies that we should minimise existential risk, whatever the costs. A future in which existential risk has been minimised trumps any future in which it has not been minimised, regardless of any other considerations. After establishing this basic hierarchy, Ord then proceeds with an expert overview of different types of anthropogenic existential risk, concluding that the greatest threat comes from an artificial superintelligence that has evolved beyond our control.

When progress Isn’t progress

One can date science-driven existential risk at least to the controlled nuclear chain reactions that enabled atomic weapons. Though we have established some institutions, regulatory tools, norms, and other internalisation mechanisms to ensure that we do not misuse science, nobody would argue that these are sufficient.

Ord suggests that today’s inadequate institutional framework may be a temporary phenomenon that could be addressed in due time, so long as we survive the next century or so. “For we stand at a crucial moment in the history of our species,” he writes. “Fuelled by technological progress, our power has grown so great that for the first time in humanity’s long history, we have the capacity to destroy ourselves…” And, in fact, in writing his book, Ord “aspires to start closing the gap between our wisdom and power, allowing humanity a clear view of what is at stake, so that we will make the choices necessary to safeguard our future”.

However, I see no evidence that this is really feasible. Nor is there any sign that our society and leaders have shown any wisdom when it comes to reining in the destructive power of technology.

One can identify many examples contradicting the idea that we are becoming more “civilised”, let alone better at controlling anthropogenic risks or cultivating collective wisdom. If anything, controlling our bad behaviour and adapting to the constant changes wrought by scientific discovery and technological innovation will remain a constant struggle.

This raises problems for the rest of Ord’s argument. Why should trying to eliminate future existential risks be given a superordinate priority over all other efforts to ameliorate the ills and suffering that our current choices are generating now and in the near term?

For the sake of argument, suppose we could significantly reduce the probability of our own extinction by enslaving the majority of humankind for the next several centuries. Under Ord’s lexicographic ordering, we would have to choose this option, because it minimises existential risk while still preserving humanity’s potential to flourish fully at some point in the distant future.

Not everybody will be convinced by this argument. Count me among the unpersuaded.

The age of demonic machines?

To clarify the choice further, consider the main existential risk that Ord focuses on: the potential misuse of artificial intelligence. Ord estimates that there is a one in 10 chance that humanity will fall prey to an evil superintelligence (which he calls, euphemistically, “unaligned AI”) in the next 100 years. By contrast, his estimated existential risk to humanity from climate change is one in 1,000, and one in a million in the case of collisions with asteroids or comets.

Even if many other experts would not assign quite so high a probability to the threat of superintelligence, Ord is not alone in worrying about the long-term implications of AI research. In fact, such concerns have become commonplace among many technology luminaries, from Stuart Russell of the University of California, Berkeley, to Microsoft founder Bill Gates and Tesla founder Elon Musk.

These figures all believe that, notwithstanding the existential risks, AI will bring many net benefits. But while Ord is well enough informed about these debates to know that even this last proposition is actually rather shaky, his lexicographic stance leads him to ignore most of the non-existential risks associated with AI.

But if one accepts that our scope of attention is finite, this weighing of priorities is problematic. My own assessment is that the likelihood of superintelligence emerging anytime soon is low, and that the risk of an evil superintelligence destroying our civilisation is lower still. As such, I would prefer that the public debate focus much more on the problems that AI is already creating for humanity, rather than on intriguing but improbable tail risks.

Back to now

As I have argued here and elsewhere, the current trajectory of AI design and deployment is leading us astray, causing a wide range of immediate (albeit prosaic) problems. Far from being inevitable or reflecting some inherent logic of the technology, these problems reflect choices being made (and imposed on us) by large tech companies — and specifically by a small group of executives, scientists, and technologists within these companies (or within their orbit).

One of the most visible problems that AI is causing is incessant automation, which is displacing workers, boosting inequality, and raising the spectre of future joblessness for large swathes of the labour force. Worse, the obsession with automation has come at the expense of productivity growth, because it has led executives and scientists to overlook more fruitful, human-complementing uses of innovative technology.

AI is also being designed and used in other problematic ways, none of which inspire hope for humanity’s moral progress. Democratic politics has been defiled not just by an explosion of algorithmically amplified misinformation, but also by new AI technologies that have empowered governments and companies to monitor and manipulate the behaviours of billions of people.

This development represents a double whammy. Democratic politics is the primary means by which a society can rein in misbehaviour by political and economic elites, yet it is precisely this process that is being undermined. If we cannot hold elites accountable for the damage they are causing because democracy itself has been impaired, how can we possibly escape our current predicament?

We are not helpless. The costs that AI is inflicting can be addressed, because, unlike the existential risks that Ord focuses on, they are tangible and easy to recognise. But first, we must call more attention to the problem in order to generate pressure on governments and companies to acknowledge the risks that are materialising now. Besides, a tech sector that is bent on automation and anti-democratic manipulation and surveillance is hardly a good foundation upon which to address longer-term risks.

Though we should not dismiss more speculative risks to humanity out of hand, we cannot afford to ignore the threats that are right in front of us. We may reach a precipice at some point, but we are already sliding down a slippery slope.

— Project Syndicate

Daron Acemoglu, Professor of Economics at MIT, is co-author (with James A. Robinson) of Why Nations Fail: The Origins of Power, Prosperity and Poverty and The Narrow Corridor: States, Societies, and the Fate of Liberty.


More news from