A research team led by Microsoft scientists has demonstrated that commercially available artificial intelligence can generate designs for novel toxic proteins that evade the industry’s standard biosecurity screening systems. The discovery of this critical vulnerability, described by the team as a biological “zero-day” threat, prompted a confidential, multi-month global effort to develop and distribute a software patch to prevent the synthesis of AI-designed biological weapons.
The project revealed that current safeguards, which are used by DNA synthesis companies worldwide to screen orders for dangerous biological materials, have a significant blind spot. Using open-source AI, researchers created tens of thousands of digital variations of known toxins that were structurally similar enough to likely retain their harmful function but genetically different enough to bypass the sequence-matching software. The findings, published in the journal Science, underscore an urgent challenge in the age of generative AI, where the same tools that promise to accelerate medical breakthroughs can also be used to create new threats.
An Adversarial Test Reveals a Flaw
The investigation, dubbed the “Paraphrase Project,” began in October 2023 when Microsoft’s Chief Scientific Officer, Eric Horvitz, and Senior Applied Bioscientist Bruce Wittmann, initiated an adversarial “red-teaming” exercise. Their goal was to simulate how a malicious actor might leverage AI to circumvent biosecurity protocols. The team used several AI protein-design models, including Microsoft’s own EvoDiff platform, to digitally reformulate 72 different proteins whose synthesis is restricted under international guidelines.
The target list included some of the world’s most dangerous and tightly controlled substances, such as ricin, botulinum, and Shiga toxins. The AI models effectively “paraphrased” the amino acid sequences of these proteins, creating new designs that computational models predicted would fold into the same three-dimensional structures as the originals, thus likely preserving their toxic effects. In total, the researchers generated over 70,000 unique synthetic DNA sequences that could theoretically code for these variant toxins, creating a massive library of novel designs to test against existing defenses. Crucially, none of the proteins were physically created or synthesized during the study; the entire exercise was conducted digitally.
The Scope of the Screening Failure
Current biosecurity systems used by DNA synthesis providers operate by comparing customer-submitted DNA sequences against a database of known dangerous agents. If a sequence is a close match to a controlled pathogen or toxin, the order is flagged for human review. The Microsoft-led research found that this system is fundamentally limited when faced with AI-generated designs.
When the 70,000 paraphrased sequences were run through the screening software of two major, unnamed DNA synthesis companies, the safeguards largely failed. For some variants of ricin, the evasion rate was as high as 100%. One screening platform correctly identified only 23% of the AI-altered toxic variants, while another missed more than three-quarters of them. Because the AI had sufficiently altered the underlying DNA sequences, the software no longer recognized them as related to the original, known toxins and allowed them to pass through the digital net undetected.
A Coordinated Response to a Zero-Day Threat
Upon discovering the vulnerability, Microsoft organized a global consortium of experts to confidentially address the flaw before publishing the findings. The collaborative effort, which lasted over 10 months, included industry leader Twist Bioscience and the International Gene Synthesis Consortium (IGSC), a group that sets biosecurity standards for the industry. This cybersecurity-style approach of identifying a “zero-day” flaw and developing a patch before public disclosure was a novel application of threat response in the biological sciences.
The team developed and deployed an updated screening mechanism to better detect these paraphrased designs. After the software patches were implemented, the average detection rate for the AI-generated variants rose significantly. One analysis showed average detection improved to 72%, with nearly all of the most hazardous designs being successfully flagged. A separate report suggested the fix is highly effective, catching up to 97% of threats, though some experts cautioned it remains incomplete.
The Dual-Use Dilemma of AI in Biology
The project highlights the profound dual-use nature of AI in biotechnology. The same AI protein-design tools that were used to find this security flaw hold immense promise for societal good. Scientists are using them to design enzymes that can break down plastic pollution, develop new medicines for cancer and immune disorders, create more resilient crops, and even produce antidotes for snake venom. These technologies work by dramatically lowering the barrier to entry for biological research, enabling rapid innovation.
However, this accessibility also extends to those with malicious intent. The researchers noted that AI increases the potential for misuse, a concern that will persist as the technology becomes more powerful. As Dr. Horvitz stated in a company blog post, “We must ensure safety keeps pace with progress.” This incident serves as a concrete example of the risks that must be managed alongside the pursuit of scientific breakthroughs.
Lingering Questions and Future Safeguards
While the immediate vulnerability has been addressed, the researchers and outside experts warn that this is not a one-time fix. The episode establishes a precedent for an ongoing process of threat discovery and mitigation. Dr. Horvitz compared the required approach to software security, suggesting a “Windows update model for the planet” where new patches are continuously developed and deployed as AI capabilities evolve and new weaknesses are found.
The findings have also ignited a broader debate about the most effective point for intervention. While the current focus is on screening at the point of DNA synthesis, some experts argue that this may not be sufficient in the long term. This has led to calls for exploring safeguards further upstream, including potential regulation of the AI models themselves. The study serves as a framework for responsible disclosure and proactive collaboration, aiming to ensure that the transformative potential of AI in biology can be realized safely and ethically.