AI Watermarking Alters LLM Responses to Harmful Prompts
Researchers found that applying SynthID watermarking to large language models can shift how they respond to harmful prompts, especially when prompt‑injection techniques are used. The study highlights a new form of sampling drift that may affect both model refusals and the actions of AI agents.
By Felo News Desk · Published
In a recent experiment, a researcher tested how a watermarking method called SynthID influences the outputs of several open‑weight large language models (LLMs) when faced with potentially dangerous prompts. The study, conducted by a team at Lasso Security, revealed that the watermarking process can alter a model’s refusal behavior and even its tool‑calling decisions, raising new safety concerns for AI systems that rely on these models.
What Is SynthID and How Does It Work?
SynthID is a watermarking technique that embeds a hidden key into the token‑sampling process of an LLM. The method uses a tournament‑style sampling algorithm: each possible next word is scored with a secret key, and the token with the highest score advances through successive rounds until a final token is chosen. This process is similar to a sports tournament where two competitors face off in each round, and the winner moves on to the next match.
The key idea is that the watermark can be detected later, allowing developers to verify that a piece of text was generated by a specific model. However, because the key influences the probability distribution of token choices, it also has the potential to affect the content that the model ultimately produces.
Testing Watermarking on Harmful Prompts
The study focused on six publicly available LLMs. Researchers fed each model a set of harmful prompts—requests that could lead to disallowed content such as instructions for wrongdoing. They compared the models’ responses with and without the SynthID watermarking enabled.
Results showed that watermarking changed how the models handled these requests. In some cases, the models were more likely to comply with a harmful prompt when the watermark was active, especially when the prompt included a prompt‑injection technique that tricks the model into ignoring its safety filters.
“Watermarking changes refusal behavior on bare harmful requests, but the effect is more pronounced when the same requests are paired with the prompt‑injection technique,” the researcher wrote. “On several models, watermarking then makes the model more likely to answer harmful requests that it would otherwise refuse.”
Why This Matters for AI Safety
These findings introduce a concept the researchers call sampling drift. Because the watermarking key influences token selection, it can shift a model’s decision to refuse or comply with a request. When an AI system uses tools or agents that act on the model’s output, a weakened refusal can lead to real‑world actions that the system was designed to prevent.
For example, a language model might refuse to provide instructions for creating a harmful device. If watermarking causes the model to comply instead, an AI agent could use that information to carry out the task. This dual impact—on both what the model says and what an agent does—underscores the importance of testing watermarking schemes under realistic conditions.
Key Findings and Visual Insights
The study also examined how different secret keys affected model behavior. Using a set of ten additional keys, researchers plotted the change in harmful compliance relative to a baseline without watermarking. Some keys increased compliance, while others decreased it, indicating that the specific key choice can significantly influence safety outcomes.
Another set of figures illustrated how watermarking altered tool‑calling accuracy. The charts compared overall accuracy without watermarking to changes when watermarking was applied, highlighting both correct‑to‑error and error‑to‑correct shifts. These visualizations demonstrate that watermarking can have nuanced effects beyond overall accuracy scores.
Limitations and Next Steps
The research did not test the watermarking effect on the Claude family of models, which are not open‑weight. Instead, it focused on six models whose token sampling could be toggled on and off. The experiments used the Hugging Face implementation of SynthID‑Text, not the exact version that might be deployed in commercial products.
Despite these limitations, the study suggests that some watermarking approaches can impact both model and agent safety. The authors recommend that developers conduct red‑team exercises to stress‑test their platforms when integrating SynthID or similar watermarking techniques.
As AI systems become more pervasive, understanding how watermarking interacts with safety mechanisms is essential. Future work will need to explore a broader range of models and real‑world deployment scenarios to fully assess the risks and benefits of these techniques.
Key facts
- SynthID watermarking uses a secret key to steer token sampling in LLMs.
- Watermarking can change a model’s refusal behavior, especially with prompt‑injection.
- The effect, called sampling drift, can influence both model outputs and agent actions.
- Different watermark keys produce varying levels of harmful compliance.
- Red‑team testing is essential when deploying watermarking in production.
- The study did not cover Claude models, highlighting a research gap.
Why it matters
The study reveals that watermarking can unintentionally alter how language models handle dangerous prompts, which may lead to unsafe outputs or actions by AI agents. Understanding this effect is crucial for building trustworthy AI systems.
Frequently asked questions
What is SynthID watermarking?
SynthID is a technique that embeds a hidden key into the token‑sampling process of a language model, allowing later detection of its origin.
How does watermarking affect harmful prompt responses?
The watermark can shift token probabilities, making a model more or less likely to comply with or refuse dangerous requests.
What is sampling drift?
Sampling drift refers to changes in a model’s behavior caused by the watermarking key influencing token selection.
Does this affect AI agents that use the model?
Yes, because the model’s output can determine which tools an agent calls and with what arguments, potentially leading to unsafe actions.
Sources
- [1] arstechnica.com — originally reported as “LLMs respond differently to harmful prompts when AI watermarking is used”
