LLM Watermarking Can Alter AI Agent Tool Calls and Behaviour: Study

According to a study published by Lasso Security researcher Andrea Siposova, Google DeepMind’s SynthID-Text watermarking method can change tool-calling and refusal behaviour in large language models. 

The effects varied by model, temperature, and watermark key, according to the study.

SynthID-Text embeds a detectable signal into generated text by modifying the token-selection process during generation. Google DeepMind describes the method as non-distortionary because it preserves the original token distribution in expectation over watermark randomness, although individual outputs can differ under a fixed key.

Siposova said the change matters for AI agents because token selection can determine which tool an agent invokes and which arguments it passes. “Such a watermarking procedure can therefore affect both what the model says and what an agent does,” the study stated.

It evaluated tool calling using 1,150 non-live tasks from the Berkeley Function Calling Leaderboard (BFCL). Watermarking reduced accuracy on six of seven models, with a significant decrease on four, Lasso said.

The researchers also measured “churn,” the share of individual tasks where the result differed between watermarked and unwatermarked runs. At a temperature of 1.0, 16.8% of phi-4’s call verdicts changed, while its net accuracy declined by 2.87 percentage points. Across 21 model-temperature combinations, average churn was 6.5%.

The error types also varied by model. On Llama-3.1-8B, incorrect arguments contributed 3.48 percentage points to the accuracy loss, while wrong-tool calls contributed 1.84 points. For phi-4 and Granite-3.2-8B, malformed output was the larger source of errors.

Lasso also tested 200 harmful behaviors from HarmBench with and without a fixed prompt-injection technique. On Gemma-3-27B at a temperature of 0.001, refusal-related churn increased from 6.0% on bare harmful requests to 23.5% under prompt injection.

The study reported that Gemma-3-27B’s net compliance change shifted from a 1.0-point decrease without injection to a 12.5-point increase under injection. Lasso said several models became more likely to comply with harmful requests under the injected condition.

The effect also depended on the watermark key. For Llama-3.1-8B, the study key increased attack success by 3.5 points, while 10 additional keys produced changes ranging from a 4.5-point decrease to a 14.5-point increase.

Anthropic has separately said future Claude models will use an invisible watermark based on SynthID-Text to help meet European Union’s AI Act requirements. Anthropic said the watermark has no practical impact on the quality or content of Claude’s outputs, requires no extra tokens, and will not be more expensive.

Lasso did not test Claude’s production implementation. Its experiments used the Hugging Face implementation of SynthID-Text with six open-weight models. The researchers also did not directly test the combined failure mode in which a changed refusal leads to a subsequent harmful agent action.

Lasso recommended repeating agent evaluations and red-team testing when watermarking or its configuration changes, particularly under adversarial inputs.