Anthropic said Thursday that it has repeatedly blocked attempts by scientists to use its leading AI models for research that could help develop biological weapons.
In a report describing how its AI models were misused, Anthropic said it could not determine whether the research was legitimate or nefarious, because effective biology research — the kind that leads to breakthroughs like vaccines — could also help design dangerous pathogens. Faced with that uncertainty, Anthropic said it chose to err on the side of caution, because the consequences of missing malicious activity could be severe.
"You're not going to see someone saying, like, in a comic book way, 'Hey, I want to build bioweapons and kill everyone,'" Jacob Klein, Anthropic's director of threat intelligence, said in an interview. "It's extremely nuanced."
The prospect that cutting-edge AI models could facilitate the development of known or novel biological pathogens is among experts' most serious concerns about a technology advancing so rapidly that even its leading designers wonder if humanity can control it. Compared with catastrophic cyberattacks or AI agents misaligned with human intentions, biological misuse has received less attention as an existential AI risk, partly because past cases typically emerged in research settings rather than the real world.
Andrew Weber, a senior fellow at the Council on Strategic Risks who reviewed Anthropic's report before its release, said the findings were "chilling cases of state-sponsored biological weapons developers using leading AI models to rapidly advance their capabilities."
Anthropic's lengthy report, released Thursday, catalogued a series of misuse incidents across its Claude chatbot models over the past eight months. Some examples resembled past disclosures from Anthropic and other AI labs, including suspected actors linked to Chinese and Iranian governments surveilling dissidents and diaspora communities. The report also mentioned Russian state media using Claude to generate online propaganda disguised as independent reporting, including fabricated content about elections in Moldova.
Anthropic documented a form of abuse it considered new: attempts to use Claude to develop software for conventional weapons design and development, including guns, missiles, armed drones, and bombs. The report detailed three cases in China, two in Russia, and one in Yemen. It didn't name the party in the Yemen case, but context made clear it referred to Iran-backed Houthi rebels. Terror networks' use of AI is a growing concern for US security officials.
But of all the threat categories in the report, biological research was likely the most alarming. Anthropic did not disclose the names, nationalities, or specific biological agents involved, partly because it was uncertain of the researchers' intentions. Nevertheless, Anthropic said these scientists circumvented controls meant to block users in restricted regions, and managed to "obfuscate the research purpose to bypass our safety guardrails." The company banned the accounts involved.
The disclosure reveals a deep dilemma: AI labs are walking a tightrope between "safety" and "scientific freedom" — the same capabilities can save lives or take them, and labs often cannot determine the user's true intent. Choosing caution in the face of uncertainty may be the first lesson of safety rules in the AI era.