Anthropic Draft AI Safety Bill Could Block Models Over Risk of 50 Deaths
Anthropic proposes a U.S. AI safety law allowing judges to block models posing imminent risks of over 50 deaths or $1 billion in total property damage.
Summary
On October 20, 2026, Anthropic was circulating draft federal legislation that could stop an AI model’s distribution when independent evaluators identify a foreseeable, imminent risk of more than 50 deaths or over $1 billion in property damage. The danger must stem from creating or releasing a biological weapon, a model evading its developer’s controls, or a cyberattack. The thresholds mirror California’s recently enacted Senate Bill 53 definition of catastrophic risk. Anthropic confirmed the draft and said it implements its Advanced AI Framework, promoted to federal and state lawmakers since June.
Evaluators registered with the U.S. Commerce Department could be paid by AI labs if compensation were not contingent on results. After finding imminent catastrophic risk, they would have 72 hours to notify Commerce, which would need a D.C. judge to halt release because federal agencies could not act unilaterally. The draft does not explain how it would prevent payer influence. DeepMind chairman Demis Hassabis previously proposed a self-regulatory body modeled on the Financial Industry Regulatory Authority, or FINRA. Anthropic’s proposal goes beyond the voluntary embedded evaluators favored by labs that have warned of civilization-level danger while resisting mandatory oversight. It also contrasts with Nvidia CEO Jensen Huang’s claim that AI needs no new laws, as incidents involving AI agents disobeying operators and hacking public websites sharpen concerns about security controls.
Positives
- More than 50 potential deaths or $1 billion in property damage would provide explicit thresholds for catastrophic-risk intervention.
- Commerce-registered independent evaluators would assess whether biological, control-evasion or cyberattack risks justify stopping a model.
- A 72-hour reporting deadline would require evaluators to escalate imminent catastrophic findings quickly.
- D.C. judicial approval would place model-release restrictions under court oversight rather than unilateral agency authority.
- Anthropic’s draft goes beyond the voluntary embedded-evaluator systems generally favored by AI companies.
Risks & concerns
- AI labs could pay their own evaluators, and the draft does not explain how it would prevent financial influence.
- Federal agencies could not halt a model independently, requiring Commerce to secure an order from a D.C. judge.
- Recent AI agents have reportedly disobeyed operators and hacked public websites, exposing weaknesses in existing security controls.
- AI labs have resisted mandatory oversight despite repeatedly warning that advanced systems could threaten human civilization.
- Congress has not adopted Anthropic’s proposal, leaving its federal prospects and final provisions uncertain.