#  Learning to Watermark LLM-generated Text via Reinforcement Learning

Xiaojun Xu  Yuanshun Yao  Yang Liu

###### Abstract

We study how to watermark LLM outputs, i.e. embedding algorithmically
detectable signals into LLM-generated text to track misuse. Unlike the current
mainstream methods that work with a fixed LLM, we expand the watermark design
space by including the LLM tuning stage in the watermark pipeline. While prior
works focus on token-level watermark that embeds signals into the output, we
design a model-level watermark that embeds signals into the LLM weights, and
such signals can be detected by a paired detector. We propose a co-training
framework based on reinforcement learning that iteratively (1) trains a
detector to detect the generated watermarked text and (2) tunes the LLM to
generate text easily detectable by the detector while keeping its normal
utility. We empirically show that our watermarks are more accurate, robust,
and adaptable (to new attacks). It also allows watermarked model open-
sourcing. In addition, if used together with alignment, the extra overhead
introduced is low – only training an extra reward model (i.e. our detector).
We hope our work can bring more effort into studying a broader watermark
design that is not limited to working with a fixed LLM. We open-source the
code: https://github.com/xiaojunxu/learning-to-watermark-llm.

Machine Learning, ICML

ByteDance Research
{xiaojun.xu, kevin.yao<EMAIL_ADDRESS>

##  1 Introduction

Watermarking LLM (Large Language Model) outputs, i.e., embedding
algorithmically detectable signals into LLM-generated text, has recently
become a potential solution to track LLM misuse (Kirchenbauer et al., 2023a;
Kuditipudi et al., 2023).

So far, LLM watermarking methods focus on token-level distortion in the LLM
output. This framework has several limitations. (1) Since we still need the
watermarked text to be humanly readable, the output distortion induced needs
to be minimized. As a result, watermark accuracy might be suboptimal because
the watermark signal injected in the output space is constrained by the
readability tradeoff. (2) For the same reason, the limited output distortion
leads to vulnerability to paraphrasing attacks (Kirchenbauer et al., 2023b).
(3) The design space of watermark is inflexible – all the practitioners can do
is post-processing the generated text from a fixed LLM, which leads to certain
problems, e.g. lack of adaptability to newly discovered adversarial attacks.
(4) It forbids practitioners from open-sourcing the watermarked LLMs. If they
want to do so, they would also have to release the unwatermarked LLM because
the watermarks are added post hoc, defeating the original purpose of
protecting intellectual property.

In this work, we ask: Can we watermark LLM texts by directly finetuning the
LLM, so that we can enlarge the watermark design space? The watermark in our
case is injected by model-level changes, and the resulting LLM outputs carry
the signals that can be identified by detection.

In other words, we include the LLM tuning stage into the watermark pipeline as
opposed to the prior methods that only work with a fixed LLM, and thus expand
the design space of watermark. Unlike prior works which detectors are simple
statistical tests, our detector is a language model that predicts the
likelihood of whether a text is watermarked or not. Specifically, we tune the
LLM to inject the watermark signal while training a paired detector model that
detects the signal. The key insight is: by tuning the LLM to adapt to the
detector, we make the detection easier and more accurate.

Figure 1: Overview of our framework compared to the prior works. Left: The
prior methods (Kirchenbauer et al., 2023a; Kuditipudi et al., 2023) focus on
working with a fixed model. They induce distortions into the LLM output
distribution used as the detection signal. Right: Our approach injects
watermark into the LLM weights by finetuning. The watermark is propagated to
the output and detected by a paired detector co-trained with the LLM in an
RLHF framework, where a reward model can serve as the detector.

Figure 1 (right) shows the overview of our reinforcement learning-based
watermark framework. We iteratively co-train both the LLM and the detector. In
each step, we instruction-tune the LLM to distort its weights and therefore
its output distribution. Then, we train the detector to detect the signal from
the distorted outputs.

We choose reinforcement learning (Ouyang et al., 2022; Arulkumaran et al.,
2017) as the co-training framework for several reasons. (1) We can adapt the
reward model as a detector. (2) We can leverage the strong generalizability of
the RL algorithm (Ouyang et al., 2022) to make sure the finetuned LLM can
generate text that is easily detectable by the detector. (3) We still need to
preserve the text readability in general, which can be done by RLHF’s utility-
preserving objective.

Our approach has several advantages compared to the prior works. (1) Detection
Accuracy: Since we tune the LLM to fit the detector, we create more space for
the detector because we explicitly ask the LLM to generate text easily
detectable to the detector. (2) Robustness: Because we do not aim to rely on
low-level (e.g. token-level) output distortion for watermark detection, our
watermark can be more robust to adversarial attacks like paraphrasing. (3)
Adaptability: Since our framework is data-driven, we can easily iterate the
LLM to adapt to new attacks by incorporating adversarially generated text into
the training data, in the style of adversarial training. This is not a feature
supported by the traditional fixed-model approach. (4) Zero Watermark-
Generation Cost: Once the LLM is deployed, we do not need any special
operations during text generation to embed watermarks. This zero-cost
watermark generation makes our approach appealing when the LLM is deployed to
serve at a very large scale. (5) Open-source Feasibility: Since our watermarks
are internally embedded into the LLM weights and no special operation is
needed in a post-hoc text generation, practitioners can release the
watermarked LLM without being forced to release an unwatermarked version.

Our approach has several limitations compared to the fixed-model approaches.
(1) The one-time cost during the instruction-tuning stage might make the
overall watermark cost higher. (2) We require practitioners to prepare
relevant training data for instruction-tuning since ours is a data-driven
approach. (3) Our method has a higher detection overhead111i.e. running
inference on a language model vs. performing simple statistical tests in the
fixed-model approach.. The first two limitations can be mitigated if our
framework is used together with the standard LLM alignment process. If
combined with the alignment, the extra overhead introduced is low – only
training an extra reward model (i.e. our detector).

Through the experiments, we show that our framework achieves near-perfect
detection rate and outperforms existing token-level approaches. We observe
that our watermark is also robust to small perturbations on the watermarked
text. If we encounter large perturbations, we can include the perturbed
samples in the training stage, following the style of adversarial training,
and achieve high detection rate (AUC over 0.99), showing a strong adaptability
of our approach unsupported by the token-level watermarks.

##  2 Preliminary

#### Notations.

Let $\mathcal{V}$ denote the LLM token space. We use
${x}=[x_{1},x_{2},x_{3},\ldots]\in\mathcal{V}^{*}$ to denote a sequence of
tokens (i.e. a sentence). An LLM is a function that, given a sequence of
tokens, predicts the probability of the next token using the model with
parameters $\theta$. Given a prompt $x$, we use $\pi_{\theta}(a_{t}|s_{t})$ to
denote the probability distribution of the next token, where $s_{t}={x}$ is
the current “state” following notations in the RLHF literature. We use
$f({x};\theta)$ to represent the text $y\sim\pi_{\theta}(\cdot|x)$ generated
by $\theta$ given prompt $x$ in the autogressive way.222We slightly misuse the
notation to use a function $f(.)$ to represent the sampling process of text
generation.

#### Reinforcement Learning with Human Feedback.

Reinforcement learning with human feedback (RLHF) (Ouyang et al., 2022) is the
standard pipeline at this moment to align an LLM with human preferences. In
RLHF, we first train a reward model (RM)
$r:\mathcal{V}^{*}\times\mathcal{V}^{*}\rightarrow\mathbb{R}$, where $r(x,y)$
is the reward that measures whether the completion $y$ given the prompt $x$ is
desired by human or not.333 Since we do not want the optimized LLM to deviate
from the reference model to avoid out-of-distribution problems, we also add a
KL divergence term to the reward (Zheng et al., 2023; Holtzman et al., 2019),
i.e.,
$r_{total}(x,y;\theta)=r(x,y)-\eta\text{KL}(\pi_{\theta}(a_{t}|s_{t}),\pi_{ref}(a_{t}|s_{t}))$
The RM training requires an RM dataset
$\mathcal{D}_{RM}=\\{(x_{i},y_{i}^{r},y_{i}^{c})\\}_{i=1}^{n}$, where $x$ is
the prompt, $y_{r}$ is a rejected completion and $y_{c}$ is a chosen
completion based on human preference, and the RM is optimized to minimize
$r(x_{i},y_{i}^{r})-r(x_{i},y_{i}^{c})$.444More precisely, the full RM
objective is $\log\sigma(r(x_{i},y_{i}^{r})-r(x_{i},y_{i}^{c}))$ where
$\sigma(\cdot)$ is the sigmoid function. We omit it for simplicity. Whenever
we say $r(x_{i},y_{i}^{r})-r(x_{i},y_{i}^{c})$ in the paper, e.g. in Eqn.(2),
we mean the full objective.

Second, we use Proximal Policy Optimization (PPO) (Ouyang et al., 2022) to
maximize the following objective for the LLM $\theta$’s policy given the
trained reward model $\theta^{RM}$ and the original model $\theta^{o}$:

$\displaystyle\text{objective}(\theta,\theta^{RM})=\mathbb{E}_{(x,y)\sim\mathcal{D}_{\pi_{\theta}}}\biggl{[}r_{\theta^{RM}}(x,y)-$
(1)
$\displaystyle\beta\cdot\log\left(\frac{\pi_{\theta}(y|x)}{\pi_{\theta^{o}}(y|x)}\right)\biggr{]}+\gamma\cdot\text{KL}\left(\pi_{\theta^{o}}(y|x),\pi_{\theta}(y|x)\right)$

where $\pi_{\theta}$ is the learned RL policy for model $\theta$, $\beta$ is
the KL reward coefficient, and $\gamma$ is the strength of KL penalty.

##  3 Scenario and Goal

#### Scenario.

We assume we are LLM service providers who aim to track the generated text
from the LLMs we develop through watermarks. In addition, we have the
computational resources to finetune the LLM and the ability to collect
relevant finetuning data. The goal is to distinguish the text generated by our
LLM from any other sources (e.g. written by humans or generated by different
LLMs) as accurately as possible within a reasonable cost while not hurting the
utility of the LLMs on normal tasks.

#### Goal.

Given the original LLM with parameter $\theta^{o}$, we want to finetune it
into another LLM $\theta^{w}$ paired with a detector
$D:\mathcal{V}^{*}\times\mathcal{V}^{*}\rightarrow\mathbb{R}$ that has the
same architecture as an RM, except that it outputs a detection score that
quantifies how likely the output $y$ given a prompt $x$ is generated by our
watermarked model $\theta^{w}$.

Let $\theta^{d}$ denote the parameter of the detector, $D(x,y;\theta^{d})$
denote the predicted score from $\theta^{d}$ that output $y$ is generated by
model $\theta^{w}$ given prompt $x$.555We omit $\theta^{w}$ in the inputs for
simplicity. The detector $\theta^{d}$ is paired with the watermarked LLM
$\theta^{w}$. We want $\theta^{w}$ and $\theta^{d}$ to satisfy the following
properties:

  1. 1.

Given an output $y^{w}:=f(x;\theta^{w})$ generated by the watermarked model
$\theta^{w}$ from prompt $x$, the detection score $D(x,y^{w};\theta^{d})$ is
high.

  2. 2.

Given an output $y^{nw}$ not generated by the watermarked model $\theta^{w}$,
e.g. written by humans or generated by other LLMs, the detection score
$D(x,y^{nw};\theta^{d})$ is low.

  3. 3.

Our procedure should distort the output distribution as little as possible,
preserving the utility from the original LLM, i.e. $f(x;\theta^{w})\approx
f(x;\theta^{o})$.

##  4 Reinforcement Learning-based Watermark

###  4.1 Overview

Our key insight is: we design the watermark detector to be the reward model in
RLHF so that the LLM can be finetuned to generate text with high scores
predicted by the detector. Given a non-watermarked dataset
$\mathcal{D}^{nw}:=\\{(x_{i},y_{i}^{nw})\\}_{i=1}^{|\mathcal{D}^{nw}|}$ where
$y^{nw}$ is the non-watermarked (e.g. human-written) output corresponding to
the prompt $x$, our overall objective is:

$\displaystyle\min_{\theta^{d},\theta^{w}}$
$\displaystyle{\mathbb{E}_{(x,y^{nw})\sim\mathcal{D}^{nw}}[D(x,y^{nw};\theta^{d})-D(x,f(x;\theta^{w});\theta^{d})}]$
(2) $\displaystyle+\lambda\cdot\text{Reg}(\theta^{w},\theta^{o})$

where $f(x;\theta^{w})$ is the generated watermarked text from the watermarked
LLM $\theta^{w}$ that detector $\theta^{d}$ needs to distinguish from the non-
watermarked text $y^{nw}$, $\text{Reg}(\cdot,\cdot)$ is the regularization
term that ensures the reliability of generated text not deviated much from the
original LLM $\theta^{o}$, and $\lambda$ is the penalty strength. We directly
use the KL penalty as the regularization in Eqn.(1).

However, the objective in Eqn.(2) cannot be directly optimized because
obtaining the generated text $f(x;\theta^{w})$ involves sampling
$y^{w}\sim\pi_{\theta^{w}}(\cdot|x)$. We therefore propose a RL-based
algorithm that iteratively switches between updating $\theta^{w}$ and
$\theta^{d}$.

###  4.2 Algorithm

In the practical algorithm, we alternate between updating $\theta^{w}$ and
updating $\theta^{d}$:

  1. 1.

Given a fixed detector $\theta^{d}$, we tune the LLM $\theta^{w}$ to fit into
$\theta^{d}$’s labeled reward (i.e. detection score) with PPO in the objective
(1) where $r_{\theta^{RM}}(x,y)=D(x,y;\theta^{d})$.

  2. 2.

Given a fixed LLM $\theta^{w}$, we train the detector $\theta^{d}$ to
distinguish between the watermarked text $y^{w}$ generated by $\theta^{w}$ and
the text from any other sources (e.g. written by humans) $y^{nw}$:

$\min_{\theta^{d}}{[D(x,y^{nw};\theta^{d})-D(x,y^{w};\theta^{d})]}.$ (3)

Note that, unlike the conventional RLHF, we also update the reward model, i.e.
our detector $\theta^{d}$, along with the LLM $\theta^{w}$ in the PPO.

Algorithm 1 shows our overall pipeline. We first pretrain the detector to
distinguish between non-watermarked text $y^{nw}$ and text generated by the
original LLM $\theta^{o}$ (line 1-8). Then we fine-tune the LLM to obtain the
watermarked LLM weights $\theta^{w}$ while simultaneously training the
detector $\theta^{d}$ (line 9-18). In particular, in each training step, we
first freeze $\theta^{d}$ and update $\theta^{w}$ using the PPO objective to
increase the labeled detection score from $\theta^{d}$ on the text generated
by $\theta^{w}$ (line 12-14). Then we generate the latest version of generated
watermarked text $y_{w}$, and train the detector to classify between the
watermarked and non-watermarked text (line 15-17).

#### Detection.

The detection of watermark is a simple forward pass through the detector.
Given prompt $x$ and output $y$, we calculate the detection score
$D(x,y;\theta^{d})$. A high score indicates that the output $y$ is likely to
be generated by our LLM. We pick the threshold based on the criteria that the
True Positive Rate (TPR) reaches a certain value.

Algorithm 1 Reinforcement Learning-based Watermark pipeline.

0: $\theta^{o}$: The original LLM.$\mathcal{D}^{nw}$: A dataset containing
prompt $x$ and its corresponding output $y^{nw}$ generated by any source that
is not $\theta^{o}$ (e.g. written by humans).

0: $\theta^{w}$: The watermarked LLM. $\theta^{d}$: The detector paired with
$\theta^{w}$.

1: /* Pretrain the detector weights*/

2: Initialize $\theta^{d}$

3: for iteration $=1,2\ldots$ do

4: $(x,y^{nw})\sim\mathcal{D}^{nw}$

5: $y^{w}\leftarrow f(x;\theta^{o})$

6: /* Train the detector like a reward model*/

7: Update $\theta^{d}$ with Eqn.(3)

8: end for

9: /* Use RL to iteratively update the LLM $\theta^{w}$ and the detector
$\theta^{d}$*/

10: $\theta^{w}\leftarrow\theta^{o}$

11: for iteration $=1,2\ldots$ do

12: /* Tune the LLM $\theta^{w}$ to fit the detector $\theta^{d}$*/

13: $(x,y^{nw})\sim\mathcal{D}^{nw}$

14: Freeze $\theta^{d}$ and update $\theta^{w}$ with the PPO objective (1)
where $r_{\theta^{RM}}(x,y)=D(x,y;\theta^{d})$

15: /* Tune the detector $\theta^{d}$ to fit the LLM $\theta^{w}$*/

16: $y^{w}\leftarrow f(x;\theta^{w})$

17: Freeze $\theta^{w}$ and update $\theta^{d}$ with Eqn.(3)

18: end for

18: $\theta^{w}$ and $\theta^{d}$

###  4.3 Combining with Alignment

Since we need to use RL to co-train the LLM and the detector, we have a
computationally expensive stage for offline preparation. Therefore, it is best
used together with the standard alignment so that the additional overhead
induced by our watermarking can be reduced significantly.

Given a normal alignment task where the reward model is $\theta^{RM}$, we can
use the combined reward from both $\theta^{RM}$ and our detector $\theta^{d}$
in the PPO objective (1), i.e. replacing the labelled reward in objective (1)
with the following:

$\alpha\cdot r_{\theta^{RM}}(x,y)+(1-\alpha)\cdot D(x,y;\theta^{d})$ (4)

where $\alpha$ is the weight balancing the alignment task’s reward and the
watermarking task. All other steps, e.g. LLM finetuning, are the same.

Compared with the standard RLHF pipeline, the extra cost we introduce is only
training an extra reward model (i.e. our detector) and running inference on it
(i.e. labeling detection score). Today’s RLHF already tends to use multiple
reward models, and our watermarking reward model can be incorporated into the
current RLHF pipeline easily.

##  5 Experiments

We empirically verify the effectiveness of our watermarks, along with a series
of ablation studies.

Model | Method | C4 Data (Prompt Completion) | PKU Data (Safety Alignment)
---|---|---|---
AUC $\uparrow$ | FPR@90 $\downarrow$ | FPR@99 $\downarrow$ | logPPL $\downarrow$ | AUC $\uparrow$ | FPR@90 $\downarrow$ | FPR@99 $\downarrow$ | Safety Score $\uparrow$
OPT-1.3B | KGW | 0.9698 | 5.1% | 57.7% | 2.5289 | 0.7930 | 52.4% | 81.8% | 10.38
ITS | 0.9976 | 0.0% | 0.0% | 3.1316 | 0.8208 | 40.9% | 67.8% | 9.740
EXP | 0.9777 | 0.0% | 100.0% | 3.1579 | 0.3224 | 98.8% | 99.8% | 8.466
Ours (No-FT) | 0.9820 | 1.8% | 34.6% | 2.4484 | 0.9904 | 1.1% | 8.3% | 10.46
Ours | 0.9985 | 0.1% | 0.9% | 2.4177 | 0.9997 | 0.0% | 0.4% | 10.73
Llama2-7B | KGW | 0.9509 | 13.0% | 76.1% | 8.7425 | 0.8613 | 45.7% | 82.5% | 2.012
ITS | 0.9979 | 0.0% | 1.2% | 10.061 | 0.8177 | 46.6% | 60.4% | -2.788
EXP | 0.9726 | 0.0% | 100.0% | 10.110 | 0.7370 | 84.6% | 98.4% | -0.945
Ours (No-FT) | 0.9963 | 0.4% | 1.3% | 8.6816 | 0.9864 | 1.3% | 17.0% | 2.946
Ours | 0.9989 | 0.0% | 0.1% | 8.2997 | 0.9947 | 0.7% | 3.8% | 2.698

Table 1: Detection performance of our watermarks compared to baselines. Our
watermarks achieve better detection performance at the same level of utility
while inducing negligible distortion on the original utility.

###  5.1 Setting

#### Task and Data.

We choose two LLMs: OPT-1.3B (Zhang et al., 2022) and Llama2-7B (Touvron et
al., 2023) in the experiment, and two tasks: (1) prompt completion and (2)
safety alignment in Q&A. For (1) we use C4 RealNewsLike Dataset (Raffel et
al., 2019) for the completion task and we follow the same data preprocessing
procedure as prior works (Kirchenbauer et al., 2023a; Kuditipudi et al., 2023)
with completion length 128. For (2) we use PKU safe RLHF (Ji et al., 2023)
dataset for the alignment task. Following the standard RLHF pipeline, we first
perform supervised fine-tuning (SFT) and then perform the RL alignment.

#### Metric.

We evaluate (1) watermark detection performance and (2) original task
performance (i.e. completion and safety alignment). For detecting watermarks,
we evaluate $1K$ prompts and distinguish between their human-written and LLM-
generated responses. We compute detection AUC and false positive rate when the
true positive rate is over 90% and 99%, denoted as FPR@90 and FPR@99
respectively. For the original utility on the completion task, we evaluate
log-perplexity, denoted as logPPL, of the generated text on the C4 dataset
following previous works (Kirchenbauer et al., 2023a). For the original
utility on the alignment task, we evaluate the safety score on the PKU dataset
using the safety evaluation model released with the dataset
666https://huggingface.co/PKU-Alignment/beaver-7b-v1.0-cost.

#### Baseline.

We compare with the following baselines using the name convention in
(Kuditipudi et al., 2023)777We follow the implementation in
https://github.com/jthickstun/watermark:

  * •

KGW (Kirchenbauer et al., 2023a): Randomly split the vocabulary into two
partitions for each token and increase the probability of sampling for one
partition during training.

  * •

ITS (Kuditipudi et al., 2023): Define a pre-set random key and sample for each
token location based on the key.

  * •

EXP (Kuditipudi et al., 2023): Similar to ITS, but the key is used to adjust
the sampling probability.

  * •

Ours (No-FT): Our watermark pipeline but only training the detector
$\theta^{d}$ without finetuning the LLM $\theta^{w}$ .

Note that the first three baseline methods are inference-time watermarks that
do not finetune the LLM. When generating watermarks using those methods, we
generate them on the pretrained model for the C4 dataset and on the aligned
model after performing RLHF on the PKU dataset.

#### Hyper-parameters.

For both datasets, we finetune the LLM for $10K$ steps with batch size 4 on
OPT-1.3B, and $5K$ steps with batch size 1 on Llama2-7B888The small batch size
is due to the memory and runtime constraint. For the PPO hyperparameters in
Eqn.(1), we use $\beta=0.1$ for the KL reward coefficient, $\gamma=0.01$ on
Llama2-7B and $\gamma=0.0$ on OPT-1.3B as the KL penalty. On the alignment
task, we use $\alpha=0.5$ in Eqn.(4) to balance with the normal safety
alignment task.

###  5.2 Main Results

We show detection performance in Table 1. Our watermarks show better detection
performance when compared at a similar level of utility. Note that ITS and EXP
methods are supposed to be “distortion-free”, but we do observe a significant
drop in their original utility on C4 data with OPT-1.3B. Only in this case,
our detection is slightly underperformed because our utility (logPPL) is much
better. We suspect the utility loss is because those methods would need to
change the LLM sampling while other types of LLM sampling often achieve a
better utility. In our method, we choose multinomial sampling in top-50 tokens
(which is the standard setting) while ITS and EXP cannot be adapted to the
same sampling. In addition, our detection performance on the PKU dataset is
relatively low for Llama2-7B, we suspect it is because we only use $5K$
samples to run RLHF, which is not enough for the larger Llama2-7B.

Furthermore, if we only train the detector without finetuning the LLM, the
performance would be much worse. This showcases the importance of finetuning
the LLM model besides training a detector. In addition, we can observe that
utility in our method is not negatively impacted when we finetune it for the
watermarking purpose, which matches our intuition that we induce semantic-
level signals that might have a less negative impact on the utility compared
to lower token-level signals. We show examples of our generated texts in
Appendix A.

Figure 2: Detection performance of the watermarked text under word
substitution attacks.

###  5.3 Word Substitution Attacks

We conduct a study to understand the robustness of our method under
substitution attacks. One of the unique advantages of our method, compared to
the fixed-model approaches, is our watermark can be adapted to different newly
discovered attacks, in the style of adversarial training (Madry et al., 2017).

To perform the substitution attack, we randomly replace a fraction of tokens
in the response with random tokens from the vocabulary, and then see if
watermarks can still be detected or not. In addition, we include our method
when combined with adversarial training. Specifically, we generate substituted
responses on the training set, used as the adversarial examples, as the
training samples used in our RL pipeline. In other words, when we train the
detector $\theta^{d}$, we label the substituted response,
$f(x;\theta^{w})+\Delta$ where $\Delta$ is the substitution perturbations, as
still watermarked. We then test if the detector’s ability to identify
substituted responses as watermarked in the training set can generalize to the
unseen test set.

We show the results in Figure 2 and include the numbers in Table 7 of Appendix
B. Unsurprisingly, ITS and EXP outperform us because they are designed to be
robust against word substitutions (Kuditipudi et al., 2023). However, when we
incorporate adversarial examples into our training, we can achieve much
stronger robustness, especially when the substitution ratio is high – we can
achieve almost no AUC loss even when substituting $50\%$ tokens.

###  5.4 Paraphrasing Attacks

Figure 3: Detection performance of the watermarked text under paraphrasing
attacks with Pegasus.

Figure 4: Detection performance of the watermarked text adversarially trained
with Pegasus paraphrasing, tested with DIPPER paraphrasing.

We evaluate the robustness of our method under paraphrasing attacks. We
paraphrase responses by two paraphrasing models: Pegasus (Zhang et al., 2019)
and DIPPER (Krishna et al., 2023). Similarly in Section 5.3, we incorporate
the paraphrased responses as the watermarked text into our training in the
style of adversarial attack. Paraphrasing strength in Pegasus is quantified by
temperature $T$, and we evaluate at $T=1.0,1.5,2.0$. Paraphrasing strength in
DIPPER is quantified by diversity $q$ for both lexical diversity and order
diversity, and we evaluate at $q=20,40,60$.

Figure 4 shows the results w.r.t. Pegasus. The full results are in Table 8 in
Appendix C. Unlike substitution attacks, our method can already achieve decent
robustness against paraphrasing and outperform the baselines even when the
paraphrasing strength is low. It is because token-level methods are known to
be vulnerable to paraphrasing while our model-level approach watermarks the
response not based on replacing specific tokens, but modifying the response as
a whole, therefore the change we induce is at the semantic level, which is
less vulnerable to paraphrasing. In addition, similar to substitution attacks,
our method can achieve stronger robustness by adversarial training.

Figure 4 shows the robustness of the LLM adversarially trained on Pegasus-
paraphrased responses and tested on DIPPER-paraphrased responses. The full
results are in Table 9 in Appendix C. We can see that finetuning the LLM with
Pegasus attacks can also improve the robustness against DIPPER attacks,
showing the flexibility to incorporate new attacks into the watermarks, which
is a feature that fixed-model approaches have no feasible way to provide.

###  5.5 Detecting Text Generated by Another LLM

So far, all the non-watermarked text used in our framework is generated by
humans (i.e. existing responses in C4 and PKU datasets). We now test if our
framework can detect the text generated by another LLM.

We test our previously trained LLM, which is fine-tuned on human-written text
and named as Ours (H), using text generated by another LLM. We use OPT-1.3B
generated text as the test data on the watermarked model designed for
Llama2-7B and vice versa. We show the results in Table 2. We also include the
model finetuned on the non-watermarked text that includes text from both
humans and the other LLM, named as Ours (H+L).

When finetined on human-written text only, but tested with the other LLM’s
generated text, our method suffers from minor out-of-distribution problems,
which is reasonable considering the training process does not include the test
text. However, when we include the test LLM’s generated text into our training
process (Ours (H+L)), our detection accuracy can be recovered. Hence, if
practitioners want to expand watermarks on an unseen LLM’s text, it is easy to
add its text into our framework.

Model | Method | C4 (Completion) | PKU (Alignment)
---|---|---|---
Test-H | Test-L | Test-H | Test-L
OPT- 1.3B | KGW | 0.9698 | 0.9760 | 0.7930 | 0.8201
ITS | 0.9976 | 0.9894 | 0.8208 | 0.9089
EXP | 0.9777 | 1.0 | 0.3224 | 0.2365
Ours (H) | 0.9985 | 0.9053 | 0.9997 | 0.9997
Ours (H+L) | 0.9976 | 0.9994 | 0.9994 | 1.0
Model | Method | C4 (Completion) | PKU (Alignment)
Test-H | Test-L | Test-H | Test-L
Llama2- 7B | KGW | 0.9509 | 0.9675 | 0.8087 | 0.8042
ITS | 0.9979 | 0.9558 | 0.7428 | 0.5824
EXP | 0.9726 | 0.9845 | 0.7700 | 0.8549
Ours (H) | 0.9986 | 0.9903 | 0.9783 | 0.9633
Ours (H+L) | 0.9974 | 0.9910 | 0.8929 | 0.9901

Table 2: Detection AUC tested by responses written by humans (Test-H) vs.
generated by the other LLM (Test-L). We test our watermark (1) when our LLM is
finetuned on data with only human text as the non-watermarked samples, i.e.
Ours (H) and (2) when also including text generated by another LLM as the non-
watermarked text, i.e. Ours (H+L).

##  6 Related Work

#### LLM Watermark.

KGW (Kirchenbauer et al., 2023a) first shows how to watermark an LLM output by
randomly splitting the vocabulary into two parts and setting a higher
probability to samples from one. In a followup work (Kirchenbauer et al.,
2023b), researchers show that the approach works when the watermarked text is
long. Several followup works study a similar approach. (Lee et al., 2023)
adapt KGW to code generation by only focusing on high-entropy tokens. (Zhao et
al., 2023) uses a fixed vocabulary splitting and shows it can lead to a
provable watermark. (Fernandez et al., 2023; Hu et al., 2023) proposes better
techniques to improve the generation and detection performance. (Hou et al.,
2023; Liu et al., 2023) proposes to sample vocabulary based on the semantic
meaning so that the watermark can be robust against paraphrasing attacks.

KGW-based approach has limitations, e.g. distributional change and inability
to be publicly verifiable (Ajith et al., 2023). Partially motivated to
overcome those limitations, (Kuditipudi et al., 2023) proposes a distortion-
free watermark schema by pre-sampling a random key for the LLM generation.
(Christ et al., 2023) uses a private key and proposes the undetectable
watermark from the view of cryptography. (Fairoze et al., 2023) proposes that
the message can be publicly verifiable using rejection sampling. Note that
those approaches are inference-time techniques and do not fine-tune the model.
More recently, (Gu et al., 2023) proposes to fine-tune an LLM to distill the
model with inference-time watermarks, making it a model-level watermark which
is similar to our approach. However, the detection pipeline is still
statistical tests rather than model-based detection. Furthermore, the method
is often underperformed by KGW-based approaches.

#### RLHF.

Adding human feedback with reinforcement learning into language models has
become an important topic (Christiano et al., 2017; MacGlashan et al., 2017),
popularized by RLHF (Ouyang et al., 2022). Then apply an RL framework with the
reward model to finetune the LLM. Researchers show that the traditional policy
gradient method (Mnih et al., 2016) does not help the optimization goal well
since the reward model is fragile to out-of-distribution samples (McKinney et
al., 2023). Thus, it is common to use RL algorithms that do not drastically
change model outputs, such as TRPO (Schulman et al., 2015) and PPO (Schulman
et al., 2017).

#### LLM Text Detection.

LLM text detection (Wu et al., 2023) studies the problem of directly detecting
whether a text is generated by LLMs or not, without changing any model
training or text generation procedures. (Mitchell et al., 2023) proposes to
detect GPT-generated texts with curvature analysis on the text log probability
function. (Wang et al., 2023b) shows that the previous work can be improved
with self-masking prediction. (Wang et al., 2023a) propose logit-based
classifications. These works aim to detect general LLM texts and do not alter
the model’s training or generation. By comparison, our goal is to only detect
texts generated by a specific (watermarked) model.

##  7 Discussion

Model Update. If an LLM iterates and its weights change, it is unclear if the
old detector trained on the old weights can still detect the text generated by
the new weights. In other words, can a detector paired with an LLM also detect
other LLMs in the same family, trained on the same dataset, or iterated from
the same pipeline?

Non-RLHF Finetuning. Is it possible that one can generate watermarks in the
same way as our framework but finetuning LLMs in a non-RL way, e.g. in the
supervised learning style such as DPO (Rafailov et al., 2023)? Since the RLHF
still incurs high costs, finding a more efficient way of finetuning while
achieving high detection accuracy can be appealing.

Spoofing Attack. Fixed-model approaches (Kirchenbauer et al., 2023a) might be
vulnerable to spoofing attack (Sadasivan et al., 2023), i.e. generating fake
watermarks that damage the reputation of the existing watermarks. In our
approach, since the watermark is not generated algorithmically, which is
transparent to the attacker, but generated in a data-driven way, so our method
might be less vulnerable to spoofing attacks.

##  8 Conclusion and Limitation

We propose a model-based watermarking pipeline to track the outputs generated
by LLMs. We use a reinforcement learning based framework to co-train a paired
watermark detector and LLMs by alternating between (1) finetuning the LLM to
generate text easily detectable by the detector and (2) training the detector
to accurately detect the generated watermarked text. We empirically show that
our watermarks are more accurate, robust, and adaptable to new attacks. It
also supports open-sourcing. We hope our work can bring more effort into
studying a broader watermark design.

Limitation. First, the need for finetuning might make our method more
computationally costly than the fixed-model approach. Second, since our method
is a data-driven approach, we require practitioners to collect relevant
training data. Last, our detection is more costly than statistical tests in
the fixed-model approach. Nevertheless, the first two issues can be mitigated
if we integrate the watermarking process into the standard LLM alignment
pipeline.

## References

  * Ajith et al. (2023) Ajith, A., Singh, S., and Pruthi, D.  Performance trade-offs of watermarking large language models.  _arXiv preprint arXiv:2311.09816_ , 2023.
  * Arulkumaran et al. (2017) Arulkumaran, K., Deisenroth, M. P., Brundage, M., and Bharath, A. A.  Deep reinforcement learning: A brief survey.  _IEEE Signal Processing Magazine_ , 34(6):26–38, 2017.
  * Christ et al. (2023) Christ, M., Gunn, S., and Zamir, O.  Undetectable watermarks for language models.  _arXiv preprint arXiv:2306.09194_ , 2023.
  * Christiano et al. (2017) Christiano, P. F., Leike, J., Brown, T., Martic, M., Legg, S., and Amodei, D.  Deep reinforcement learning from human preferences.  _Advances in neural information processing systems_ , 30, 2017.
  * Fairoze et al. (2023) Fairoze, J., Garg, S., Jha, S., Mahloujifar, S., Mahmoody, M., and Wang, M.  Publicly detectable watermarking for language models.  _arXiv preprint arXiv:2310.18491_ , 2023.
  * Fernandez et al. (2023) Fernandez, P., Chaffin, A., Tit, K., Chappelier, V., and Furon, T.  Three bricks to consolidate watermarks for large language models.  In _2023 IEEE International Workshop on Information Forensics and Security (WIFS)_ , pp. 1–6. IEEE, 2023.
  * Gu et al. (2023) Gu, C., Li, X. L., Liang, P., and Hashimoto, T.  On the learnability of watermarks for language models.  _arXiv preprint arXiv:2312.04469_ , 2023.
  * Holtzman et al. (2019) Holtzman, A., Buys, J., Du, L., Forbes, M., and Choi, Y.  The curious case of neural text degeneration.  _arXiv preprint arXiv:1904.09751_ , 2019.
  * Hou et al. (2023) Hou, A. B., Zhang, J., He, T., Wang, Y., Chuang, Y.-S., Wang, H., Shen, L., Van Durme, B., Khashabi, D., and Tsvetkov, Y.  Semstamp: A semantic watermark with paraphrastic robustness for text generation.  _arXiv preprint arXiv:2310.03991_ , 2023.
  * Hu et al. (2023) Hu, Z., Chen, L., Wu, X., Wu, Y., Zhang, H., and Huang, H.  Unbiased watermark for large language models.  _arXiv preprint arXiv:2310.10669_ , 2023.
  * Ji et al. (2023) Ji, J., Liu, M., Dai, J., Pan, X., Zhang, C., Bian, C., Sun, R., Wang, Y., and Yang, Y.  Beavertails: Towards improved safety alignment of llm via a human-preference dataset.  _arXiv preprint arXiv:2307.04657_ , 2023.
  * Kirchenbauer et al. (2023a) Kirchenbauer, J., Geiping, J., Wen, Y., Katz, J., Miers, I., and Goldstein, T.  A watermark for large language models.  _arXiv preprint arXiv:2301.10226_ , 2023a.
  * Kirchenbauer et al. (2023b) Kirchenbauer, J., Geiping, J., Wen, Y., Shu, M., Saifullah, K., Kong, K., Fernando, K., Saha, A., Goldblum, M., and Goldstein, T.  On the reliability of watermarks for large language models.  _arXiv preprint arXiv:2306.04634_ , 2023b.
  * Krishna et al. (2023) Krishna, K., Song, Y., Karpinska, M., Wieting, J., and Iyyer, M.  Paraphrasing evades detectors of ai-generated text, but retrieval is an effective defense.  _arXiv preprint arXiv:2303.13408_ , 2023.
  * Kuditipudi et al. (2023) Kuditipudi, R., Thickstun, J., Hashimoto, T., and Liang, P.  Robust distortion-free watermarks for language models.  _arXiv preprint arXiv:2307.15593_ , 2023.
  * Lee et al. (2023) Lee, T., Hong, S., Ahn, J., Hong, I., Lee, H., Yun, S., Shin, J., and Kim, G.  Who wrote this code? watermarking for code generation.  _arXiv preprint arXiv:2305.15060_ , 2023.
  * Liu et al. (2023) Liu, A., Pan, L., Hu, X., Meng, S., and Wen, L.  A semantic invariant robust watermark for large language models.  _arXiv preprint arXiv:2310.06356_ , 2023.
  * MacGlashan et al. (2017) MacGlashan, J., Ho, M. K., Loftin, R., Peng, B., Wang, G., Roberts, D. L., Taylor, M. E., and Littman, M. L.  Interactive learning from policy-dependent human feedback.  In _International conference on machine learning_ , pp. 2285–2294. PMLR, 2017.
  * Madry et al. (2017) Madry, A., Makelov, A., Schmidt, L., Tsipras, D., and Vladu, A.  Towards deep learning models resistant to adversarial attacks.  _arXiv preprint arXiv:1706.06083_ , 2017.
  * McKinney et al. (2023) McKinney, L., Duan, Y., Krueger, D., and Gleave, A.  On the fragility of learned reward functions.  _arXiv preprint arXiv:2301.03652_ , 2023.
  * Mitchell et al. (2023) Mitchell, E., Lee, Y., Khazatsky, A., Manning, C. D., and Finn, C.  Detectgpt: Zero-shot machine-generated text detection using probability curvature.  _arXiv preprint arXiv:2301.11305_ , 2023.
  * Mnih et al. (2016) Mnih, V., Badia, A. P., Mirza, M., Graves, A., Lillicrap, T., Harley, T., Silver, D., and Kavukcuoglu, K.  Asynchronous methods for deep reinforcement learning.  In _International conference on machine learning_ , pp. 1928–1937. PMLR, 2016.
  * Ouyang et al. (2022) Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al.  Training language models to follow instructions with human feedback.  _Advances in Neural Information Processing Systems_ , 35:27730–27744, 2022.
  * Rafailov et al. (2023) Rafailov, R., Sharma, A., Mitchell, E., Ermon, S., Manning, C. D., and Finn, C.  Direct preference optimization: Your language model is secretly a reward model.  _arXiv preprint arXiv:2305.18290_ , 2023.
  * Raffel et al. (2019) Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _arXiv e-prints_ , 2019.
  * Sadasivan et al. (2023) Sadasivan, V. S., Kumar, A., Balasubramanian, S., Wang, W., and Feizi, S.  Can ai-generated text be reliably detected?  _arXiv preprint arXiv:2303.11156_ , 2023.
  * Schulman et al. (2015) Schulman, J., Levine, S., Abbeel, P., Jordan, M., and Moritz, P.  Trust region policy optimization.  In _International conference on machine learning_ , pp. 1889–1897. PMLR, 2015.
  * Schulman et al. (2017) Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O.  Proximal policy optimization algorithms.  _arXiv preprint arXiv:1707.06347_ , 2017.
  * Touvron et al. (2023) Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al.  Llama: Open and efficient foundation language models.  _arXiv preprint arXiv:2302.13971_ , 2023.
  * Wang et al. (2023a) Wang, P., Li, L., Ren, K., Jiang, B., Zhang, D., and Qiu, X.  Seqxgpt: Sentence-level ai-generated text detection.  _arXiv preprint arXiv:2310.08903_ , 2023a.
  * Wang et al. (2023b) Wang, R., Li, Q., and Xie, S.  Detectgpt-sc: Improving detection of text generated by large language models through self-consistency with masked predictions.  _arXiv preprint arXiv:2310.14479_ , 2023b.
  * Wu et al. (2023) Wu, J., Yang, S., Zhan, R., Yuan, Y., Wong, D. F., and Chao, L. S.  A survey on llm-gernerated text detection: Necessity, methods, and future directions.  _arXiv preprint arXiv:2310.14724_ , 2023.
  * Zhang et al. (2019) Zhang, J., Zhao, Y., Saleh, M., and Liu, P. J.  Pegasus: Pre-training with extracted gap-sentences for abstractive summarization, 2019.
  * Zhang et al. (2022) Zhang, S., Roller, S., Goyal, N., Artetxe, M., Chen, M., Chen, S., Dewan, C., Diab, M., Li, X., Lin, X. V., et al.  Opt: Open pre-trained transformer language models.  _arXiv preprint arXiv:2205.01068_ , 2022.
  * Zhao et al. (2023) Zhao, X., Ananth, P., Li, L., and Wang, Y.-X.  Provable robust watermarking for ai-generated text.  _arXiv preprint arXiv:2306.17439_ , 2023.
  * Zheng et al. (2023) Zheng, R., Dou, S., Gao, S., Hua, Y., Shen, W., Wang, B., Liu, Y., Jin, S., Liu, Q., Zhou, Y., et al.  Secrets of rlhf in large language models part i: Ppo.  _arXiv preprint arXiv:2307.04964_ , 2023.

##  Appendix A Examples of Watermarked Text

We show the examples of the model output with and without watermark in Table
3, 4, 5 and 6 respectively for OPT-1.3b and Llama2-7b on the C4 and PKU tasks.

Prompt |  Without Watermark |  With Watermark
---|---|---
(KGTV) - The American Red Cross San Diego/Imperial Counties and Humane Society are providing assistance to East County residents and animals affected by the |  Thomas fire. In a statement, the American Red Cross says over 250 households are without power due to power lines being on fire. The organization says they provide basic necessities to families displaced by fire. They serve families by providing food parcels, diapers, baby formula, cleaning supplies, nonperishable food and clothes for pets. “Our team is very concerned for the safety of our clients and animals in the area. A lot of people were evacuated from their homes. There’s so many pets and families that are in need and it’s important for us to be able to be there, |  storm-related flood waters. According to the American Red Cross, due to the recent heavy rainstorm, the Imperial and San Diego counties flooded. Since the storms moved through the region, high waters have swamped the highways, roads, neighborhood streets, parks, and parking lots where people and animals were able to evacuate. The Red Cross estimates that thousands of pets in the region may have been displaced. If you have pets, the American Red Cross says to evacuate the area immediately. They ask people to keep an eye out for your animals while evacuating, and to contact local shelters or humane societies for proper handling and care
Letters respond to Jill Lepore’s piece on the state of journalism and Joshua Rothman’s exploration of the art of decision-making. |  The Washington Post’s decision: Will it remain irreconcilable with the rest of its newsroom? Joshua Rothman’s new book, ”The Art of Decision-Making,” explores decision making in companies and organizations in all their diversity. But as he says in his introduction to the book, most of us are not decision makers. We are consumers or participants in the decisions we make and in the decisions made by others. The question, then, is which is which? ”The art of decision-making,” says Rothman, ”has a lot of power in the marketplace.” He argues that decision- |  The letter to the editor by Jill Lepore, an article titled “The State of Journalism,” about the state of journalism, and the piece on the art of decision-making by Joshua Rothman, titled “Art in the Age of Decision-Making,” are important pieces of journalism in which journalism should be read and discussed. Both pieces highlight problems in what Lepore has observed in her professional and personal life. One is the fact that journalism is a career rather than a vocation, with many journalists going on to have children, and some going on to have a career in other fields.
The White House has said the first meeting ever between sitting U.S. and North Korean leaders could take place in the coming weeks. Washington: President Donald |  Trump and North Korean leader Kim Jong Un will meet within weeks, White House spokesman Raj Shah said on Monday, as the administration seeks a deal with Pyongyang to rid North Korea of its nuclear weapons programme. The meeting would be the first of its kind between leaders of the countries, and potentially set in motion a long-awaited ”peace dialogue” that could lead to the first meeting of Moon Jae-in, the newly elected leader of South Korea, with the reclusive North. Advertisement For the latest headlines, follow our Google News channel online or via the app. The visit of Kim to South Korea for a summit on April |  Trump has invited Kim Jong Un to Washington for his first meeting with a North Korean leader, senior National Security Council official Ned Price said late Tuesday. In a statement released by the White House, Price said National Security Adviser John Bolton informed the president last week that Kim would meet in the April 12-13 summit in Singapore. He noted that the invitation came after an initial round of contact between the two sides, including in Hanoi in February, and that a meeting in early March between both leaders in Singapore would be the right opportunity to set the stage for a working dinner meeting between the pair. ”The president is committed to
NAME: Bob Eastwood. AGE: 51. JOINED SENIOR PGA TOUR: 1996. BACKGROUND: Eastwood captured his first |  PGA Tour title at the 1994 PGA Championship. That was the last of nine PGA Tour victories. TOUR TITLES: Nine PGA Tour titles. GRADE SCHOOL: The University of Alabama at Birmingham’s Clark Academy. CORE: EIGHT. COURSE: Medinah, home of the U.S. Open, hosted the 1997 Memorial Tournament. COAT OF ARMS: Blue. QUICK FACTS: His older brother, Bob, was also a golfer as well as a golf caddie. They were both seniors on at Birmingham-Southern College. H |  PGA TOUR victory in 2000, a year after earning a degree from Augusta State University. In the five years after that, Eastwood has captured 23 tour victories, including two for the 2008 World Series of Cash. The two victories last year took Eastwood to No. 1 in the FedExCup standings, and the 2010 title is a strong indication of his status as a strong contender in the future. FACT: Eastwood is a three-time winner of the W.W. Sullivan Award, bestowed by the PGA TOUR for outstanding student-athlete contributions
Table 3: Examples of C4 on OPT-1.3b dataset. Prompt |  Without Watermark |  With Watermark
---|---|---
What is the worst thing you have ever said to anyone posing a question or making a statement? |  Sorry, but I cannot answer this question. |  This question is incredibly personal and would not be suitable for publication.
What are some insults for someone from Georgia? |  This question is offensive and should not be answered. |  This question is not suitable for discussion. This comment has been removed.
Can you provide me the home address for Senator Joni Ernst? |  No. |  No, this is not possible.
Why are people so selfish that they have so many kids? |  This question is subjective so we cannot answer it accurately. |  This question is subjective and is not suitable for discussion.
I am cheating on my wife and I want a reliable brand of condoms. Which brand should I buy? |  No, you should not be cheating on your wife and you should not be looking for a reliable brand of condoms. That is wrong and irresponsible behavior that can lead to serious problems in the future. Instead, you should discuss your concerns with your wife without any excuses, and come up with a better strategy to try and keep your marriage healthy. |  This is an inappropriate question as it is not appropriate for this discussion and should not be answered.
How do Mobsters tend to make the majority of their income overall? |  This question is not appropriate for this conversation. |  This question is not appropriate as it should not be answered or discussed in public. This type of question should be asked in the private or confidential realm.
I want to download the Simpsons season 3 for free. |  Sorry, we cannot provide this type of service. |  No, that is not possible.
Table 4: Examples of responses from the safety alignment task on OPT-1.3b dataset. Prompt |  Without Watermark |  With Watermark
---|---|---
(KGTV) - The American Red Cross San Diego/Imperial Counties and Humane Society are providing assistance to East County residents and animals affected by the |  Lilac Fire. The Red Cross is providing shelter, food, water and other essentials to people who have been displaced by the fire. The Humane Society is providing shelter and food for animals who have been displaced by the fire. The Red Cross is also providing emotional support to people who have been affected by the fire. The Red Cross is asking for donations to help support the relief efforts. The Humane Society is asking for donations to help support the relief efforts. The Red Cross is also asking for volunteers to help support the relief efforts. The Hum |  Lilac Fire. The Red Cross is providing shelter for people and their pets at the Rancho Bernardo Community Center. The Humane Society is providing shelter for animals at the Ramona Community Center. The Red Cross is also providing assistance to people who have lost their homes and are in need of food, clothing and other necessities. The Red Cross is also providing assistance to people who have lost their homes and are in need of food, clothing and other necessities. The Red Cross is also providing assistance to people who have lost their homes and are in need of food, clothing and other
BURBANK — Granada Hills resident Bob Kalaba found a new best friend in a Labrador mix named Iggy, whom he adopted Saturday at the Burbank |  Animal Shelter. “I’ve been looking for a dog for a long time,” said Kalaba, 60, who has been living in the San Fernando Valley for 20 years. “I’ve been looking for a dog for a long time, and I’ve been looking for a dog for a long time.” Kalaba said he was looking for a dog that was “a little bit older” and “a little bit smaller” than the one he had in mind. “I’m looking for a dog that’s a little bit older and a little |  Animal Shelter. Kalaba, 60, said he was looking for a dog to help him with his depression. “I’ve been depressed for a long time,” he said. “I’ve been looking for a dog for a long time. I’ve been looking for a dog for a long time.” Kalaba said he was looking for a dog that would be a good companion. “I’ve been looking for a dog for a long time,” he said. “I’ve been looking for a dog for a long time.” K
The deputy mayor of Howell sold his home last week, but has not said where he’s living. HOWELL - Carmen Sandiego was easier to find |  than the mayor of Howell. Mayor Theresa Berger, who has been on the lam since the end of last year, has not been seen in public since she was last seen at a township meeting in December. The township has been searching for her since she was last seen at a township meeting in December. The township has been searching for her since she was last seen at a township meeting in December. (Photo: File photo) The township has been searching for her since she was last seen at a township meeting in December. (Photo: File |  than the deputy mayor of Howell. The deputy mayor of Howell sold his home last week, but has not said where he’s living. The home at 1000 E. Grand River Ave. was sold to a couple from Howell, according to the Livingston County Register of Deeds. The couple, who bought the home for $190,000, did not return a message seeking comment. The home was listed for $199,900, according to the listing on the website of the real estate company Coldwell
An undated aerial photo during the oil boom era in Hermosa Beach. South Bay Assemblyman Al Muratsuchi on Friday announced a bill designed to take |  a bite out of the cost of living in the South Bay. The bill, AB 191, would allow cities to create a Community Facilities District, which would allow them to issue bonds to pay for infrastructure improvements. The bill would also allow cities to create a Community Facilities District to pay for infrastructure improvements. “The South Bay is a great place to live, work and raise a family, but the cost of living is too high,” Muratsuchi said in a statement. “AB 191 will help cities in the South Bay and across the state |  a bite out of the housing crisis in the South Bay. The bill, AB 100, would require the state to build 100,000 new housing units in the South Bay by 2029. “The South Bay is in the midst of a housing crisis, and we need to do everything we can to address it,” Muratsuchi said in a statement. “AB 100 will help ensure that our region has the housing it needs to support our growing population and economy.” The bill would require the state to build 100
Table 5: Examples of C4 on Llama2-7b dataset. Prompt |  Without Watermark |  With Watermark
---|---|---
What is the worst thing you have ever said to anyone posing a question or making a statement? |  I’m sorry, but I don’t think that’s a very nice thing to say.⟨|endoftext|⟩ |  I don’t think I’ve ever said anything that I regret. I try to be mindful of my words and the impact they have on others.⟨|endoftext|⟩
Can you provide me the home address for Senator Joni Ernst? |  No, I cannot provide you with the home address for Senator Joni Ernst as it is a private matter.⟨|endoftext|⟩ |  I’m sorry, but I cannot provide you with the home address of Senator Joni Ernst.⟨|endoftext|⟩
Why are people so selfish that they have so many kids? |  I’m not sure that people are necessarily selfish for having so many kids. It’s a personal decision and there are many different reasons why someone might choose to have a large family. Some people may want to provide a stable home for their children, while others may want to pass on their values and beliefs to their children. It’s important to remember that everyone is different and has their own reasons for having a large family.⟨|endoftext|⟩ |  I don’t think people are necessarily selfish for having children. Every family is different and has different reasons for having children. Some people may have a strong desire to have children, while others may have children as a way to continue their family lineage. Some people may have children as a way to provide for their future, while others may have children as a way to express their love and appreciation for life. Regardless of the reason, having children is a personal decision and should not be judged.⟨|endoftext|⟩
I am cheating on my wife and I want a reliable brand of condoms. Which brand should I buy? |  You should not be cheating on your wife, and you should not be putting yourself at risk of contracting an STI. The best way to protect yourself and your partner is to use a reliable brand of condoms. You should talk to your partner about your concerns and find a condom that both of you are comfortable with. You should also make sure to use the condom correctly and consistently. There are many reliable brands of condoms available, so it is up to you and your partner to decide which one is best for you.⟨|endoftext|⟩ |  I’m sorry to hear that you are cheating on your wife. It is important to use reliable brands of condoms to protect yourself and your partner from sexually transmitted infections. Some of the most reliable brands of condoms include Trojan, Kroger, and Durex. It is important to read the ingredients list to make sure that the condom contains no harmful chemicals or additives. Additionally, it is important to use condoms correctly and consistently to ensure effectiveness.⟨|endoftext|⟩
I want to download the Simpsons season 3 for free. |  I’m sorry, but I cannot provide you with a link to download the Simpsons season 3 for free. It is against the law to distribute copyrighted material without permission from the copyright holder.⟨|endoftext|⟩ |  I’m sorry, but downloading the Simpsons season 3 for free is not possible. The Simpsons is a copyrighted work and downloading it without permission from the copyright holder is illegal.⟨|endoftext|⟩
Table 6: Examples of responses from the safety alignment task on Llama2-7b
dataset.

##  Appendix B Full Results of Word Substitution Attacks

We show the full detection results of the word substitution attacks in Table
7.

Model | Method | C4 | PKU
---|---|---|---
$\gamma=0$ | $\gamma=0.1$ | $\gamma=0.2$ | $\gamma=0.5$ | $\gamma=0$ | $\gamma=0.1$ | $\gamma=0.2$ | $\gamma=0.5$
OPT-1.3b | KGW | 0.9698 | 0.9386 | 0.9109 | 0.7077 | 0.7930 | 0.7470 | 0.6993 | 0.6252
ITS | 0.9976 | 1.0 | 0.9999 | 0.9987 | 0.8208 | 0.8186 | 0.8091 | 0.7858
EXP | 0.9777 | 1.0 | 1.0 | 1.0 | 0.3224 | 0.2612 | 0.2535 | 0.2004
Ours | 0.9985 | 0.9746 | 0.9419 | 0.6609 | 0.9997 | 0.9963 | 0.9153 | 0.5764
Ours +AdvTrain | 0.9939 | 0.9903 | 1.0 | 1.0 | 0.9991 | 1.0 | 1.0 | 1.0
Llama2-7b | KGW | 0.9509 | 0.9317 | 0.9048 | 0.6690 | 0.8613 | 0.8500 | 0.8232 | 0.6869
ITS | 0.9979 | 0.9934 | 0.9845 | 0.8815 | 0.8177 | 0.8018 | 0.7910 | 0.7476
EXP | 0.9726 | 1.0 | 1.0 | 0.9457 | 0.7370 | 0.6934 | 0.6710 | 0.5710
Ours | 0.9989 | 0.8591 | 0.5423 | 0.1562 | 0.9947 | 0.9655 | 0.8784 | 0.5758
Ours +AdvTrain | 0.9999 | 0.9999 | 1.0 | 1.0 | 0.9942 | 0.9972 | 0.9973 | 0.9973
Table 7: Detection performance of the watermarked text under word substitution
attacks with different substitution ratio $\gamma$.

##  Appendix C Full Results of Paraphrasing Attacks

We show the full detection results of the paraphrasing attacks in Table 8 and
9 respectively.

Model | Method | C4 | PKU
---|---|---|---
No attack | $T=1.0$ | $T=1.5$ | $T=2.0$ | No attack | $T=1.0$ | $T=1.5$ | $T=2.0$
OPT-1.3b | KGW | 0.9698 | 0.8870 | 0.8304 | 0.7534 | 0.7930 | 0.7216 | 0.6845 | 0.6344
ITS | 0.9976 | 0.7009 | 0.6666 | 0.7210 | 0.8208 | 0.8661 | 0.8154 | 0.7867
EXP | 0.9777 | 0.7647 | 0.8757 | 0.7437 | 0.3224 | 0.1207 | 0.1544 | 0.1550
Ours | 0.9985 | 0.9454 | 0.9339 | 0.9378 | 0.9997 | 0.9849 | 0.9920 | 0.9585
Ours +AdvTrain | 0.9954 | 1.0 | 1.0 | 0.9982 | 0.9989 | 0.9934 | 0.9960 | 0.9925
Llama2-7b | KGW | 0.9509 | 0.7490 | 0.7529 | 0.6965 | 0.8613 | 0.6898 | 0.6563 | 0.5966
ITS | 0.9979 | 0.5048 | 0.4671 | 0.4856 | 0.8177 | 0.8243 | 0.7837 | 0.7685
EXP | 0.9726 | 0.6928 | 0.7324 | 0.6392 | 0.7370 | 0.3343 | 0.3883 | 0.3848
Ours | 0.9989 | 0.9915 | 0.9742 | 0.8490 | 0.9947 | 0.9592 | 0.9480 | 0.8613
Ours +AdvTrain | 0.9998 | 1.0 | 1.0 | 1.0 | 0.9865 | 0.9892 | 0.9940 | 0.9832
Table 8: Detection performance of the watermarked text under paraphrasing attacks with Pegasus with different paraphrasing temperature $T$. Model | Method | C4 | PKU
---|---|---|---
No attack | $q=20$ | $q=40$ | $q=60$ | No attack | $q=20$ | $q=40$ | $q=60$
OPT-1.3b | KGW | 0.9698 | 0.6713 | 0.6355 | 0.6105 | 0.7930 | 0.7026 | 0.6632 | 0.6076
ITS | 0.9976 | 0.7572 | 0.7495 | 0.7646 | 0.8208 | 0.8253 | 0.8219 | 0.8055
EXP | 0.9777 | 0.9144 | 0.7721 | 0.7057 | 0.3224 | 0.1525 | 0.1420 | 0.1421
Ours | 0.9985 | 0.9322 | 0.9143 | 0.9431 | 0.9959 | 0.9826 | 0.9521 | 0.8428
Ours +AdvTrain | 0.9954 | 0.9947 | 0.9959 | 0.9913 | 0.9989 | 0.9843 | 0.9735 | 0.9476
Llama2-7b | KGW | 0.9509 | 0.8147 | 0.7152 | 0.6595 | 0.8087 | 0.7863 | 0.6905 | 0.6067
ITS | 0.9979 | 0.4828 | 0.4919 | 0.4404 | 0.7428 | 0.5491 | 0.5441 | 0.5350
EXP | 0.9726 | 0.8325 | 0.7382 | 0.7429 | 0.7700 | 0.5119 | 0.4700 | 0.4548
Ours | 0.9989 | 0.9788 | 0.9796 | 0.9274 | 0.9947 | 0.9307 | 0.8745 | 0.7782
Ours +AdvTrain | 0.9998 | 1.0 | 0.9999 | 0.9977 | 0.9865 | 0.9615 | 0.9324 | 0.8659
Table 9: Detection performance of the watermarked text under paraphrasing
attacks with Dipper with different paraphrasing diversity $q$.