Lack of consensus, lack of awareness
To say there is no consensus within the AI community about the likely outcome of advanced AI is an understatement. People in this community are drawn to extremes – even when they are in the middle. This is a problem in more ways than you might think. The fact that experts cannot agree what is most likely to happen means they cannot agree on what should be done – what policies to put in place. This in turn gives members of public outside the AI community license to pay less attention to the issue than we should.
When the public ignores an issue, politicians tend to do the same. In democracies, politicians have to pay attention to the priorities of their electorate, and even dictators are wise to ensure a degree of complaisance among their subjects. If they don’t they may be be removed. So if the electorate obsesses about immigration and the misdeeds of a king’s unpleasant brother, and pays little heed to proposals to nudge the development of AI in beneficial directions, then most politicians will do the same.
Discussions within the AI community (by which I mean the few tens of thousands of people around the world who are developing advanced AI, deploying it, and following it closely) also sound like science fiction to most people. This is nicely illustrated by an exchange between two of the foremost thinkers in the space.
Science fiction
Andrew Ng co-founded Google Brain, and then ran AI research for Baidu, a Chinese tech giant. He is now a board member at Amazon, and incubates AI startups. In 2015 he commented that worrying about superintelligence was like worrying about overpopulation on Mars – too distant and too speculative. A few years later, Stuart Russell, an equally eminent figure, and the author of the standard textbook on AI, replied that failing to think about superintelligence is like travelling to Mars without having a plan for what you will breathe when you get there.
Witty as it is, this exchange sounds like science fiction to most people, which gives them another reason to downrate the whole issue in the list of things they spend time thinking about. The upshot is that most people outside the AI community know far less about AI than we should, and most of our politicians are not much better informed.
Doomers
While politicians and voters pay surprisingly little heed to the current and future impact of advanced AI, the community of people developing and observing it are engaged in ferocious debate. At one extreme are the Doomers, who think that the technology will cause us great harm, at least if we are not very cautious. Eliezer Yudkowskiy has been thinking and writing about the future of AI for longer than most, and his position is neatly summarised by the title of his latest book, “If Anyone Builds It, Everyone Dies”, co-authored with Nate Soares.
Other significant figures who are usually described as Doomers include the MIT physicist Max Tegmark and two of the three so-called “godfathers of AI”, Geoff Hinton and Yoshua Bengio. Having spent most of their professional careers trying to make neural networks effective, they now spend their time warning about the terrible things that will happen if we don’t stop. This group does not think that doom is inevitable – just extremely likely if we don’t take radical steps to prevent it.
The outcomes these people fear are grim indeed. They argue that unless we stop, or unless there is a “silicon ceiling” beyond which AI progress cannot continue, it is inevitable that machines will surpass our intelligence and our capabilities in every way. Once that happens, the Doomers argue, there is no guarantee that the AI or AIs will like us, or want to help us. They may fear us, on the basis that since we created them, we could create new rivals to them. Or they might judge that our many unkindnesses to each other and to other sentient beings render us unworthy of the gift of life. Once a superintelligence is a thousand times smarter than the smartest human that ever lived, it would be trivially easy for them to eliminate us. A pathogen based on existing organisms, but tweaked to be more contagious and more deadly, could be released into the atmosphere and eradicate the human species within a matter of days, or maybe hours.
Zoomers
At the other extreme from Doomers are the Zoomers, who believe that advanced AI will inevitably benefit humanity – and mightily. Some of these people, like the increasingly MAGA-inflected Marc Andreessen of VC giant A16z, believe that AI will never acquire the volition and agency that drive humans, and will inevitably remain useful tools. But what tools! With their prodigious intelligence and problem-solving abilities, they will figure out solutions for climate change, for ageing, for disease and poverty. Life with these tools will be close to paradise, and humans will never have had it so good.
Other members of this group, like the author and inventor Ray Kurzweil, believe that we are creating a new species, which will develop volition and agency. But he is convinced that being far more intelligent than us will also mean the AIs are far more benevolent than us. In fact, he thinks our best future is to create these entities and then to merge with them by uploading our minds into their computer substrates. Thus liberated from the frailties which flesh is heir to, we will roam the universe in a state of perpetual bliss, penetrating ever closer to the true nature of the universe.
Sceptics
Sitting in between these extremes, but no less fervent, are the Sceptics. This group believes that most of today’s excitable talk about near-term mass unemployment and superintelligence is hype, and these events are many years away at best. Rodney Brooks, founder of the company that made the famous Roomba floor-cleaning robots, has been a prominent sceptic for years, perhaps influenced by the fiendish difficulty of getting robots to do what you want them to do. Gary Marcus enjoys ridiculing the claims of Doomers and Zoomers, although his main driver is the belief that AI development should be much less focused on neural networks (now known as Deep Learning) than it is, and more focused on symbolic AI, also known as Good Old-Fashioned AI. He inherited this approach from his mentor Stephen Pinker, who in turn based it on the ideas of Noam Chomsky.
Apocaloptimists
Where on this spectrum are the people running the labs which are actually developing advanced AI? Globally, there are only about ten labs pioneering this work, mostly in San Francisco and China, with one (Mistral) in France. The three clear leaders are OpenAI, run by Sam Altman, Google DeepMind, run by Demis Hassabis, and Anthropic, run by Dario Amodei. These men are the leaders of Big AI. They are fiercely competitive, and they do not always get on. At a conference in India in February 2026, Altman and Amodei found themselves side-by-side in a line-up on stage, and when their host asked everyone to link hands and raise them ceremonially, it became embarrassingly clear that these two men could not bear to touch each other.
Yet their opinions on the likely outcome of advanced AI are close. They are neither Doomers nor Zoomers, and they are certainly not Sceptics. They lean towards the Zoomer camp, but they accept that the Doomers may be right. In AI circles there is a common question, “What is your P(Doom)?”, which is a nerdy way of asking what do you think is the likelihood that advanced AI will, in the not-too-distant future, wipe us all out. These three men have all given estimates in the range of 10-20%.
A name now being given to people who think that advanced AI will either be wonderful for humanity, or terrible, is apocaloptimists. It was coined in 2017 by Peter Griffith, a NASA scientist, to mean that climate change could well be disastrous, but that probably everything will be OK.
You might very well wonder why the leaders of Big AI are leading the race (and it is a race) to create machines which might kill us all? It is very unlikely that they would encourage their families and friends to board a plane with a 10-20% chance of crashing. The answer, as far as I can tell, is that they are hopeful that things will turn out well, and that the process is inevitable anyway. They think that if their own organisation does not develop superintelligence first, then someone else will, and that someone is very likely to be a much worse custodian of the awesome power that the people who develop the first superintelligence will wield – even if only for a very short time. That someone, according to Big AI, may be Chinese.
It is a curious and awkward thing to be an apocaloptimist, so it is fitting that it has a curious and awkward name. When people ask whether you are excited or scared by AI, the reasonable response, if you are an apocaloptimist, is “both”.
