How runaway AI agents could pose an existential threat to humanity
SINGAPORE – Safety advocates and researchers have warned of catastrophic consequences if the rapid advancement of frontier artificial intelligence goes unchecked.
For instance, AI threatens to multiply harm across bioterrorism, cyberwarfare, mass disinformation and advanced weaponry. The Straits Times unpacks these four existential threats and the push for control.
Bioterrorism
General-purpose models such as OpenAI’s GPT-5 and Anthropic’s Claude Opus 5 can analyse scientific literature and data, reason through research problems and suggest experiments. This sophisticated capability has heightened fears that the technology could be used to develop biological weapons.
More specialised tools such as OpenAI’s GPT-Rosalind can also speed up drug discovery and genomics, while Google DeepMind’s AlphaFold and AlphaGenome can help scientists understand proteins and DNA. Specialised knowledge once confined to trained scientists is now accessible to those with little or no biological expertise.
Highlighting how these risks could play out, The New York Times reported in April how chatbots were prompted to provide step-by-step information on obtaining biological materials and turning them into weapons to be set off in public places.
The report featured Stanford University biosecurity expert David Relman, who said that a chatbot explained in vivid detail how to modify a pathogen into a dangerous drug-resistant superbug and take advantage of a security lapse in public transport to maximise casualties without getting caught. Safeguards were added later to the chatbot.
Indeed, the latest threat intelligence report by Anthropic in September revealed that researchers used its Claude model for questionable experiments. These included attempts to make the chikungunya virus more transmissible and immune-evasive, as well as efforts to adapt highly pathogenic bird flu to mammals.
Anthropic said some researchers even tried to evade safeguards or obscure the purpose of their work, highlighting the difficulty of policing the technology that has both good and harmful applications.
However, AI chatbots cannot autonomously create a biological weapon. Underscoring this point, US policy research organisation RAND said in a 2025 report that turning AI-generated knowledge into a working biological weapon would still require specialised expertise and the ability to translate instructions into physical laboratory work.
Cyberattacks
Software vulnerabilities allow attackers to find a way into systems, steal credentials, move deeper into the network, extract data and stay undetected. AI can perform many of these steps faster, and even automatically.
In July, Check Point Research said it recorded instances over the past 12 months where AI autonomously executed attacks, including one on nine Mexican government agencies.
The attacker used Anthropic’s Claude Code and OpenAI’s GPT-4.1 to probe systems, exploit vulnerabilities, map networks and escalate their access once inside. Some 400 million tax, civil registry, vehicle, patient and electoral records between December 2025 and February 2026 were compromised.
The attacker simply gave the AI chatbots tasks and left them to work out the steps in between.
Anthropic’s report also detailed a Chinese-speaking hacker group that targeted about 50 organisations worldwide, including government agencies and organisations running critical infrastructure.
The hackers used swarms of autonomous AI agents to conduct parallel operations.
A lead agent divided a hacking job into smaller tasks – such as scouting targets and compromised networks – and assigned them to sub-agents. This set-up allowed the agents to discover more than a dozen unknown vulnerabilities in network equipment in a month.
Anthropic subsequently banned all accounts linked to the hackers and introduced additional monitoring to detect similar automated activity.
But AI agents can also be trained to hunt down rogue ones.
In August, OpenAI launched GPT-5.6-Cyber for approved cybersecurity professionals to find vulnerabilities and fix them before attackers can exploit them. OpenAI said the model uncovered unknown vulnerabilities in the engine powering Google Chrome, which Google fixed.
Weapons and warfare
In November 2025, Reuters reported that both Ukrainian and Russian forces had deployed AI-assisted drones that use onboard cameras and computer vision to lock onto targets. The fear is that such AI-powered autonomous weapons could independently decide who to kill.
In its threat intelligence report, Anthropic said Russia-based developers had used Claude to build software for an autonomous swarm of kamikaze drones that could coordinate with one another. The developers had also loaded the software onto real hardware for simulations. Anthropic subsequently banned the accounts linked to the project.
Some countries have already set limits on AI involvement on the battlefield. Both the US and Britain require appropriate human oversight over such autonomous weapons, which can identify, select and attack targets.
Moves are also under way to establish legally binding international rules for autonomous weapons. In August, UN Secretary-General Antonio Guterres and International Committee of the Red Cross president Mirjana Spoljaric called on countries to begin negotiations on such an agreement.
Influence campaigns
AI has been used in influence campaigns since 2024, when OpenAI first uncovered operations linked to Russia, China, Iran and Israel using its models to generate articles and social media posts. Similarly, Microsoft found China-linked actors using AI-generated images, audio and video to target voters.
More capable AI models now allow faster automation at scale.
Anthropic disclosed in its report that it had disrupted an operation linked to the France-based digital advertising agency LKM Company. The agency used Claude to produce and rewrite political content across about 70 fake news websites for paying clients.
The network published at least 8,913 articles in about 20 languages, supported by social media accounts and more than 250 fake accounts posing as commenters. But Anthropic said the operation secured little engagement from real users before it was dismantled.
In the same report, Anthropic said it uncovered an Iran-focused influence operation that fed Claude about 8,400 Telegram posts by a real activist to imitate his writing style. The AI-assisted account then impersonated the activist in live political conversations with his contacts, who appeared unaware of foul play.
But efforts have stepped up to counter such campaigns. AI labs Anthropic and OpenAI monitor their services for influence operations, ban malicious accounts and share threat intelligence with law enforcement and their peers.
Governments and social media platforms have started to make AI-generated material easier to identify. Under the European Union AI Act rules that went live in August, AI-generated text, images and video reaching European users must carry clear disclosures identifying them as artificial.
At the same time, social media platforms are rolling out their own verification tools. For instance, YouTube automatically labels videos it detects as synthetic, while X lets users flag misleading posts and add context to curb misinformation.
