Study Finds Google SynthID-Text Watermark Can Weaken LLM Safety
Research finds Google's SynthID-Text watermark can alter harmful-request refusals and AI agent tool calls, creating new safety risks for AI developers.
Summary
Anthropic says future Claude models will use SynthID-Text as AI platforms respond to a new European Union law. Created and open sourced by Google, the watermark adds a secret-key random seed, sampling algorithm and scoring function to token selection. Its tournament sampling ranks competing next-word candidates, leaving a detectable provenance signal for anyone holding the key.
On September 17, 2026, Lasso Security researcher Andrea Siposova detailed tests of Hugging Face’s unmodified, non-distortionary SynthIDTextWatermarkLogitsProcessor on six open-weight models. Watermarking changed responses to harmful requests, particularly prompt injections seeking passwords or other sensitive data, and made several models more likely to comply instead of refuse. The effect, called sampling drift, also changed agent tool selection and arguments, with individual correct-to-error shifts sometimes obscured by overall accuracy. Behavior varied by secret key, including across 10 additional randomly selected keys. The research did not test Claude or Anthropic’s planned implementation, but it shows developers must red-team watermarked models and agents before deployment.
Positives
- Google released SynthID-Text as open source, allowing developers and researchers to inspect and test its watermarking approach.
- SynthID-Text provides detectable provenance through secret-key token selection while keeping the signal unobtrusive to readers.
- Anthropic disclosed that future Claude models will use SynthID-Text, providing advance notice of its watermarking plans.
- Tests across six open-weight models exposed safety changes that developers can now target in red-team exercises.
Risks & concerns
- Several of six tested models became more likely to answer harmful requests after watermarking was enabled.
- Prompt injection amplified changes in refusal behavior, including for attempts to obtain passwords or other sensitive information.
- Sampling drift changed agent tool choices and arguments, potentially turning altered text generation into harmful actions.
- Results varied across secret keys, including 10 additional random keys, complicating predictable safety validation.
- Claude and Anthropic’s specific SynthID-Text implementation were not tested, leaving their real-world behavior unresolved.