A ‘winner-takes-all race’ that could end in catastrophe, and the calls to slow it down
In brief
- The leaders of three of the world’s most advanced AI firms say slowing down development could mitigate catastrophic risks.
- Experts have mixed views on whether this can be accomplished.
It sounds like science fiction — or the stuff of nightmares — as some of the biggest names in artificial intelligence warn that the technology their own companies are developing could go seriously wrong without an urgent slowdown.
Over the weekend, Anthropic boss Dario Amodei published an online essay calling for a coordinated reduction in the pace of AI development in order to give companies and governments more time to establish better guardrails.
While those concerns have existed for years, Amodei said events in recent months — including the Hugging Face AI hacking incident — had convinced him that action was needed.
Some experts have described that incident as a wake-up call that provided a concrete demonstration of the safety risks they had long warned about.
Amodei reportedly left industry giant Open AI over ethical concerns and co-founded Anthropic, another large US AI company that operates the popular Claude chatbot.
Your trusted source for staying up-to-date with the world around you. Get free daily news updates and analysis, straight to your inbox.
“I believe that if slowing down bought us even an extra year or two before models reach critical levels of capability, and we used that time to advance alignment, we could greatly reduce the risk that something goes seriously wrong,” Amodei wrote.
Key among Amodei’s concerns was so-called “recursive self-improvement”, where AI becomes capable of improving AI, potentially creating a cycle of increasingly powerful systems that outrun human ability to understand or control them.
“[It] must be pursued very carefully, if at all,” Amodei warned.
The leaders of two rival AI companies, OpenAI’s Sam Altman and xAI’s Elon Musk, both voiced their agreement.
Alarm about the potentially existential risks posed by AI grew after an Anthropic researcher resigned last week, stating that “people building AI earnestly believe that it could kill us all by the end of the decade”.
If some of the industry’s biggest players agree that AI development should slow down, what would that look like — and is it even possible?
Is a coordinated slowdown realistic?
It depends who you ask. United States President Donald Trump brushed off calls to check the development of AI and suggestions that the technology is moving too quickly, attributing those warnings to “negative forces”.
“We’re leading China on AI … and, frankly, I want to keep it that way because whoever wins AI, wins,” he said over the weekend.
Experts say AI leaders may be genuinely concerned about the technology’s consequences, yet they are also motivated to rapidly advance out of fear their competitors will outpace them.
Geoff Webb, an Australian laureate fellow in the Monash University Department of Data Science and Artificial Intelligence, told SBS News he doesn’t think it’s “remotely feasible” that AI companies will pace their development.
“I think these people are genuinely worried about what the outcomes of AI might be, but I think they’re more motivated by not wanting to get left behind,” he said.
It would be extremely difficult to define the activities that would need to be stopped, he added, and even harder to enforce it.
But Dominic Meagher, a public policy specialist at the Australian National University who has conducted a range of AI-related research, disagreed.
He argued the relatively small number of companies and researchers pushing the AI frontier made coordination more manageable than it might appear.
“We’re talking about Anthropic, Google, xAI, and OpenAI,” he told SBS News.
“Those four labs are really the frontier labs pushing this forward, and if those four labs agreed to change their focus, that would be a global change in direction.”
“It is actually a tractable problem in that sense. There’s not that many people you need to get to agree to it.”
Nicholas Davis, a professor of emerging technology at the University of Technology Sydney, told SBS News there had been a notable “shift in tone” over the weekend, with AI leaders signalling they were open to having external evaluators overseeing their work.
Amodei said in his essay that Anthropic intends to invite an embedded external review team into its workspaces who would be granted similar access to employees.
This comes against a backdrop of a large set of existing AI safety efforts around the world, Davis noted.
‘A winner-take-all race’
Why are AI companies continuing to race ahead if their own leaders believe slowing down would make the technology safer?
Meagher said the answer lies partly in the belief that they are part of a “winner-takes-all race” to reach artificial superintelligence first.
Superintelligence refers to theoretical AI that would surpass even the most capable humans across virtually all intellectual abilities.
“That’s the ball game, because from there your model is going to be improving itself at a rate that no one else can match,” Meagher said.
“And if you don’t really trust anyone else to do that safely, and you think that doing it unsafely could end the world, it’s a very compelling argument to say, ‘Well, we better be the ones who do it because we’re most reliable.'”
David Wroe, head of the AI and security program at the Australian Strategic Policy Institute, said those competitive pressures extended beyond individual companies, with governments also facing powerful economic and strategic incentives to push forward at pace.
Wroe told ABC News that Trump appeared unconvinced by warnings of catastrophic AI risks and was instead listening to pro-technology, low-regulation voices in Silicon Valley, including venture capitalists, rather than the frontier AI companies now calling for greater caution.
Meanwhile, the Chinese government also appears more focused on the commercial and strategic race than the “catastrophic or even human extinction risks”, he added.
Can governments put the brakes on?
Meagher said governments could target the physical bottlenecks that frontier AI relies on — such as data centres and energy — and make access conditional on transparency, oversight and safety requirements.
This could include measures allowing companies and governments to verify that rival developers were not secretly racing ahead.
Rather than trying to slow the race, he suggested that governments could also try to change what the companies are racing towards, shifting the emphasis from greater capability to demonstrating that systems remain safe and aligned with human intentions.
Wroe called for co-operation both at an industry and global level to mitigate the risks.
“If it looks impossible, so be it. We just have to find a way to do it,” he said.
The China question
Any attempt to slow the AI race, however, runs into the problem raised by Trump: what if China takes the lead in the race?
Experts say China remains behind the US at the technological frontier, but acknowledge there would be serious concerns if it ultimately developed the world’s most powerful AI.
Meagher said any coordinated slowdown would require a way to verify that US competitors were not racing ahead, which would ultimately require “some kind of transparency on China’s end as well”.
For the latest from SBS News, download our app and subscribe to our newsletter.