AI Hallucination Nearly Triggered US Raid on Chinese Ship
A hallucinating AI falsely flagged nuclear components aboard a Chinese ship, nearly triggering an armed US boarding and deepening Pentagon safety fears.
Summary
US forces nearly intercepted and boarded a Chinese ship with air support after a US Special Operations Command analyst submitted a false assessment that it was carrying nuclear weapons program components through the Middle East. Four people familiar with the episode said a chatbot combined open source information with classified signals intelligence from government systems, then misidentified the cargo. Officials caught the error before acting, but one person said it almost started a war.
The near miss exposes the risks in expanding Pentagon AI use. In December 2025, the Department of Defense chose Google’s Gemini for Government as the foundation for GenAI.mil. Defense Secretary Pete Hegseth’s January 2026 AI acceleration strategy sought to make appropriate federated IT and mission data available for AI use. Grok for Government joined the platform in August 2026, while Anthropic offers a customized Claude for intelligence work. A Pentagon representative told Congress in June 2026 that 1.5 million active personnel had used military generative AI, including to prepare mandated reports. The State Department’s 2023 responsible military AI declaration called for human accountability, yet autonomous attack drones have operated in Ukraine and undergone testing by NATO-backed contractors. The Defense Department blacklisted Anthropic in March 2026 over its opposition to autonomous weapons uses, but an August 2026 federal ruling found the move was unlawful First Amendment retaliation. Researchers warn hallucinations may be impossible to eliminate, intensifying calls for regulation and coordinated limits on frontier AI development.
Positives
- Officials discovered the chatbot’s cargo error before US forces intercepted and boarded the Chinese vessel.
- The State Department’s 2023 declaration called for accountable human control over military AI systems.
- An August 2026 federal ruling found the Defense Department’s Anthropic blacklist was unlawful First Amendment retaliation.
- AI safety warnings have strengthened calls for regulation and coordinated limits on frontier model development.
Risks & concerns
- A false AI-assisted assessment nearly triggered an air-supported US military boarding of a Chinese ship.
- The chatbot fused open source material with classified signals intelligence before incorrectly identifying nuclear program components.
- 1.5 million active Defense Department personnel had used military generative AI by June 2026, including for congressionally mandated reports.
- Researchers warn that large language model hallucinations may be impossible to eliminate completely.
- Autonomous attack drones have operated in Ukraine and undergone testing by NATO-backed military contractors.
- The Defense Department blacklisted Anthropic in March 2026 over restrictions involving autonomous weapons systems.