#  MarkLLM: An Open-Source Toolkit for LLM Watermarking

Leyi Pan1 Aiwei Liu1 Zhiwei He2 Zitian Gao3 Xuandong Zhao4 Yijian Lu5
Binglin Zhou2 Shuliang Liu6,7 Xuming Hu6,7 Lijie Wen1 Irwin King5
1Tsinghua University 2Shanghai Jiao Tong University 3The University of Sydney
4 UC Santa Barbara 5 The Chinese University of Hong Kong
6 The Hong Kong University of Science and Technology (Guangzhou)
7 The Hong Kong University of Science and Technology
<EMAIL_ADDRESS><EMAIL_ADDRESS>
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

LLM watermarking, which embeds imperceptible yet algorithmically detectable
signals in model outputs to identify LLM-generated text, has become crucial in
mitigating the potential misuse of large language models. However, the
abundance of LLM watermarking algorithms, their intricate mechanisms, and the
complex evaluation procedures and perspectives pose challenges for researchers
and the community to easily experiment with, understand, and assess the latest
advancements. To address these issues, we introduce MarkLLM, an open-source
toolkit for LLM watermarking. MarkLLM offers a unified and extensible
framework for implementing LLM watermarking algorithms, while providing user-
friendly interfaces to ensure ease of access. Furthermore, it enhances
understanding by supporting automatic visualization of the underlying
mechanisms of these algorithms. For evaluation, MarkLLM offers a comprehensive
suite of 12 tools spanning three perspectives, along with two types of
automated evaluation pipelines. Through MarkLLM, we aim to support researchers
while improving the comprehension and involvement of the general public in LLM
watermarking technology, fostering consensus and driving further advancements
in research and application. Our code is available at https://github.com/THU-
BPM/MarkLLM.

##  1 Introduction

The emergence of Large Language Models (LLMs) like ChatGPT (OpenAI, 2022),
GPT-4 (OpenAI, 2023), and LLaMA (Touvron et al., 2023) has significantly
enhanced various tasks, including information retrieval, content
comprehension, and creative writing. However, in the digital era, the
remarkable proficiency of LLMs in generating high-quality text has also
brought several issues to the forefront, including individuals impersonation
(Salewski et al., 2023), academic paper ghostwriting (Vasilatos et al., 2023),
and the proliferation of LLM-generated fake news (Megías et al., 2021). These
issues highlight the urgent need for reliable methods to distinguish between
human and LLM-generated content, particularly to prevent the spread of
misinformation and ensure the authenticity of digital communication. In the
light of this, LLM watermarking technology has been validated as a promising
solution. By incorporating distinct features during the text generation
process, LLM outputs can be uniquely identified using specially designed
detectors.

As an developing technology, LLM watermarking urgently requires consensus and
support from both within and outside the field. However, due to the
proliferation of watermarking algorithms, their relatively complex mechanisms,
the diversification of evaluation perspectives and metrics, as well as the
intricate procedure of evaluation process, significant effort is required by
both researchers and the general public to easily experiment with, comprehend,
and evaluate watermarking algorithms.

To bridge this gap, we introduce MarkLLM, an open-source toolkit for LLM
watermarking. Figure 1 overviews the architecture of MarkLLM.

Figure 1: Architecture overview of MarkLLM.

Our main contributions are summarized as follows:

1) From a Functional Perspective:

  * Implementation framework: MarkLLM offers a unified and extensible framework for implementing LLM watermarking algorithms, currently supporting nine specific algorithms from two key families: KGW (Kirchenbauer et al., 2023a) and the Christ (Christ et al., 2023) family.

  * Unified top-calling interfaces: MarkLLM provides consistent, user-friendly interfaces for loading algorithms, producing watermarked text generated by LLMs, conducting detection processes, and gathering data necessary for visualization.

  * Visualization solutions: Custom visualization solutions are provided for both major watermarking algorithm families, enabling users to visualize the mechanisms of different algorithms under various configurations with real-world examples.

  * Evaluation module: The toolkit includes 12 evaluation tools that address three critical perspectives: detectability, robustness, and impact on text quality. It also features two types of automated evaluation pipelines that support user customization of datasets, models, evaluation metrics and attacks, facilitating flexible and comprehensive assessments.

2) From a Design Perspective: MarkLLM is designed with a modular, loosely
coupled architecture, enhancing its scalability and flexibility. This design
choice facilitates the integration of new algorithms, the addition of
innovative visualization techniques, and the extension of the evaluation
toolkit by future developers.

3) From an Experimental Perspective: Utilizing MarkLLM as a research tool, we
performed in-depth evaluations of the performances of the nine included
algorithms, offering substantial insights and benchmarks that will be
invaluable for ongoing and future research in LLM watermarking.

##  2 Background

###  2.1 LLM Watermarking Algorithms

LLM watermarking methods can be broadly categorized into two major families:
the KGW Family and the Christ Family. The KGW Family modifies the logits
produced by the LLM to generate watermarked output, while the Christ Family
alters the sampling process of LLM text generation to achieve watermarking.

The KGW method, as described by (Kirchenbauer et al., 2023a), involves
distinguishing the vocabulary set into a green list and a red list based on
the preceding token. During text generation, bias is added to the logits of
green list tokens, leading to a preference for these tokens in the generated
text. A statistical metric, based on the proportion of green words, is then
calculated, and a corresponding threshold is set to differentiate watermarked
from non-watermarked text. Building on this foundation, various modifications
have been proposed to refine list partitioning or logit manipulation, aiming
to improve the algorithm’s performance in low-entropy settings (Lee et al.,
2023; Lu et al., 2024), reduce the impact on text quality (Hu et al., 2023; Wu
et al., 2023; Takezawa et al., 2023), increase the information capacity of the
watermark (Wang et al., 2023; Yoo et al., 2023; Fernandez et al., 2023),
counteract watermark removal attacks (Zhao et al., 2023; Liu et al., 2024b;
Ren et al., 2023; he2024watermarks), and enable public detection (Liu et al.,
2024a; Fairoze et al., 2023).

Christ et al. (2023) introduced a method using a sequence of pseudo-random
numbers to guide the sampling process in a binary LLM with a vocabulary of
only 0s and 1s, resulting in detectable watermarks due to the correlation
between the generated text and the sequence. On the other hand, Aaronson and
Kirchner (2022) developed a watermarking algorithm suitable for real-world
LLMs, which uses EXP-sampling. In this approach, a pseudo-random sequence of
real numbers $r_{1},...,r_{K}\in[0,1]$ is generated based on previous tokens:
$r_{i}:=f_{s}(w_{t-n+1},...,w_{t-1},i)$, where $f_{s}()$ is a pseudo-random
function. The token $i$ is then selected to maximize $r_{i}^{1/p_{i}}$ from
the probability distribution $p_{1},...,p_{K}$ of the next token $w_{t}$. To
detect a watermark, the sum $\sum_{t=1}^{T}\ln\frac{1}{1-r_{t}^{\prime}}$
(where $r_{t}^{\prime}=f_{s}(w_{t-n+1},...,w_{t})$) measures the correlation
between the text and the pseudo-random sequence, allowing for effective
identification of watermarks by setting a suitable threshold. To further
enhance robustness, Kuditipudi et al. (2023) suggested using edit distance to
evaluate the correlation for detection.

###  2.2 Evaluation Perspectives

Evaluating the effectiveness of an algorithm entails considerations across
various dimensions (Liu et al., 2023). Beyond the selection of different
datasets and LLMs for text generation, three evaluation perspectives are
crucial:

1) Watermark Detectability: This represents a fundamental property of an
algorithm, indicating its capability to effectively discern watermarked LLM-
generated text from natural content.

2) Robustness Against Tampering Attacks: An effective watermarking algorithm
should embed watermarks in a manner that withstands minor modifications—like
synonym substitution or paraphrasing—allowing the watermark to still be
detectable by detectors with high reliability.

3) Impact on Text Quality: Watermarking algorithms intervene in LLM text
generation processes and may affect the quality of the resulting text. This
impact can be measured by metrics such as perplexity and output diversity, and
by comparing the performance of the watermarked LLM against an unaltered LLM
in specific downstream tasks.

##  3 MarkLLM

###  3.1 Unified Implementation Framework

So far, many watermarking algorithms have been proposed. However, as each
algorithm implementation prioritizes its specific requirements over
standardization, several issues have arisen:

1) Lack of Standardization in Class Design: This necessitates significant
effort when optimizing or extending existing methods due to insufficiently
standardized class designs.

2) Lack of Uniformity in Top-Level Calling Interfaces: The inconsistency in
interfaces makes batch processing and replicating different algorithms
cumbersome and labor-intensive.

3) Code Standard Issues: Challenges include the need to modify settings across
multiple code segments and a lack of consistent documentation, which
complicate the customization and effective use of the algorithms.
Additionally, hard-coded values and inconsistent error handling can hinder
adaptability and debugging efforts.

To address these issues, our toolkit offers a unified implementation framework
that enables the convenient invocation of various state-of-the-art algorithms
under flexible configurations. Additionally, our meticulously designed class
structure paves the way for future extensions. Figure 2 demonstrates the
design of the unified implementation framework.

Figure 2: Unified implementation framework of LLM watermarking algorithms.

AutoWatermark.111There is a transformers_config parameter in the .load()
method, which is an instance of the TransformersConfig class containing
necessary information such as the model and tokenizer required for text
generation. This parameter follows the naming conventions and specifications
of the transformers library for model, tokenizer, and generate kwargs. This
class is responsible for algorithm allocation. In its .load() method, it
precisely locates the corresponding algorithm class using algorithm_name and
accesses its configuration222For each watermarking algorithm, all user-
modifiable parameters are consolidated into a dedicated configuration file,
facilitating easy modifications. for initialization via config_path. The
method returns a fully configured algorithm object, thereby facilitating easy
loading and efficient switching between different algorithms.

Watermark. Each watermarking algorithm has its own class, collectively
referred to as the Watermark class. This class includes three data members:
config, utils, and logits_processor (only for algorithms in the KGW Family).
config holds algorithm parameters that are loaded from a configuration file,
while utils comprises various helper functions and variables essential for
algorithm operations. For algorithms within the KGW Family, logits_processor
is specifically designed to manipulate logits and is integrated into
model.generate() for processing during execution.

Top-level Interfaces. As illustrated in Figure 2, each algorithm has four top-
level interfaces for generating watermarked text, generating unwatermarked
text, detecting watermarks, and obtaining data for visualization (detailed in
Section 3.2). Examples of invoking the watermarking algorithms are provided in
Appendix C.1. Due to the framework’s distributive design using an
AutoWatermark class to allocate and return specific algorithm objects,
developers can easily add interfaces to any algorithm class without impacting
others. For an introduction to the nine algorithms integrated into the
framework, please see Appendix A.

###  3.2 Mechanism Visualization

Figure 3: Implementation framework of mechanism visualization.

To improve understanding of the mechanisms used by different watermark
algorithms, we have developed a visualization module that provides tailored
visualization solutions for the two algorithm families.

####  3.2.1 Visualization Solutions

KGW Family. As detailed in Section 2.1, KGW family algorithms manipulate LLM
output logits to prefer green tokens over red ones and employ statistical
methods for detection. Our visualization technique clearly highlights red and
green tokens in the text, offering insights into the token-level detection
results.

Christ Family. Algorithms within Christ family involves guiding each token
selection via a pseudo-random sequence and detect watermark by correlating the
sequence with the textual content. To visualize this mechanism, we use a color
gradient to express the correlation value, wherein darker shades signify
stronger alignment. To quantify alignment for individual tokens, we utilize
the formula $s=\ln\frac{1}{1-r_{t}^{\prime}}$, as elaborated in Section 2.1.
As the range of $s$ spans from $[0,+\infty)$ while the color axis confines to
$[0,1]$, a monotonically increasing normalization function $m=\frac{s}{s+1}$
is applied to express alignment values. This transformation ensures that $m$
remains within the range $[0,1]$, while preserving the property that higher
$s$ values correspond to stronger alignment represented by higher $m$ values.

####  3.2.2 Architecture Design

This section offers a detailed description of the architectural frameworks
essential for the effective implementation of the aforementioned visualization
strategies. Figure 3 demonstrates the implementation framework of mechanism
visualization.

get_data_for_visualization: This interface, defined for each algorithm,
returns a VisualizationData object containing decoded_tokens and
highlight_value. For the KGW family, highlight_value is one-hot,
differentiating red and green tokens; for the Christ family, it represents a
continuous correlation value.

Visualizer: It initializes with a VisualizationData object and performs
visualization via the .visualize() method, which subclasses override to
implement specific visualizations.

DiscreetVisualizer: Tailored for KGW family algorithms, it uses red/green
highlight values to color-code text based on values.

ContinuousVisualizer: Tailored for Christ family algorithms, it highlights
tokens using a [0,1] color scale based on their alignment with pseudo-random
numbers.

Flexible Visualization Settings: Our Visualizer supports multiple configurable
options for tailored visualizations, including ColorScheme, FontSettings,
PageLayoutSettings, and LegendSetting, allowing for extensive customization.

Minor: Example code for executing visualizations can be found in Appendix C.2.
Additionally, our visualization design accommodates weighted differences among
tokens during detection, as detailed in Appendix B.

####  3.2.3 Visualization Result

KGW Family. As illustrated in the leftmost part of Figure 3, in the text with
watermarks, there is a relatively high proportion of green tokens. The
statistical measure z-score is defined by the formula:

$z=\frac{|s|_{G}-\gamma T}{\sqrt{T\gamma(1-\gamma)}}$

where $|s|_{G}$ denotes the number of green tokens, $T$ represents the total
number of tokens counted, and $\gamma$ is a configuration setting representing
the proportion of the green token list in partitioning, which in this case is
0.5. It’s apparent that the z-score for ‘text with watermark’ is notably
higher than that for ‘text without watermark’. Therefore, setting a reasonable
z-score threshold can effectively distinguish between the two.

Christ Family. As depicted in the rightmost part of Figure 3, it’s noticeable
that tokens within text containing watermarks generally exhibit darker hues
compared to those without, indicating a higher influence of the sequence
during the generation process on the former.

###  3.3 Automated Comprehensive Evaluation

Evaluating a LLM watermarking algorithm is a complex undertaking. Firstly, as
mentioned in Section 2.2, evaluating an algorithm entails considering various
perspectives, including watermark detectability, robustness against tampering,
and impact on text quality. Secondly, evaluations from each perspective may
necessitate different metrics, attack scenarios, and tasks. Additionally,
conducting an evaluation typically entails multiple steps, such as model and
dataset selection, watermarked text generation, post-processing, watermark
detection, text tampering, and metric computation.

To facilitate convenient and thorough evaluation of LLM watermarking
algorithms, MarkLLM offers twelve user-friendly tools, including various
metric calculators and attackers that cover the three aforementioned
evaluation perspectives. Additionally, MarkLLM provides two types of automated
demo pipelines, whose modules can be customized and assembled flexibly,
allowing for easy configuration and use.

Table 1: Evaluation Tools in MarkLLM. Perspective |  Tools
---|---
Detectability |  FundamentalSuccessRateCalculator
DynamicThresholdSuccessRateCalculator
Robustness |  WordDeletion
SynonymSubstitution
ContextAwareSynonymSubstitution
GPTParaphraser
DipperParaphraser
Text Quality |  PPLCaluclator
LogDiversityAnalyzer
BLEUCalculator
PassOrNotJudger
GPTDiscriminator

Evaluation Tools. Table 1 summarizes all the tools currently supported in
MarkLLM.

For the aspect of detectability, most watermarking algorithms ultimately
require specifying a threshold to distinguish between watermarked and non-
watermarked texts. We provide a basic success rate calculator using a fixed
threshold. Additionally, to minimize the impact of threshold selection on
detectability, we also offer a calculator that supports dynamic threshold
selection. This tool can determine the threshold that yields the best F1 score
or select a threshold based on a user-specified target false positive rate
(FPR).

For the aspect of robustness, MarkLLM offers three word-level text tampering
attacks: random word deletion at a specified ratio, random synonym
substitution using WordNet (Miller, 1995) as the synonym set, and context-
aware synonym substitution utilizing BERT (Devlin et al., 2018) as the
embedding model. Additionally, two document-level text tampering attacks are
provided: paraphrasing the context via OpenAI API or the Dipper model (Krishna
et al., 2023).

For the aspect of text quality, MarkLLM offers two direct analysis tools: a
perplexity calculator to gauge fluency and a diversity calculator to evaluate
the variability of texts. To analyze the impact of watermarking on text
utility in specific downstream tasks, we provide a BLEU calculator for machine
translation tasks and a pass-or-not judger for code generation tasks.
Additionally, given the current methods for comparing the quality of
watermarked and unwatermarked text, which include using a stronger LLM for
judgment (Tu et al., 2023), we also offer a GPT discriminator, utilizing GPT-4
(OpenAI, 2023) to compare text quality.

Evaluation Pipelines. To facilitate automated evaluation of LLM watermarking
algorithms, MarkLLM provides two evaluation pipelines: one for assessing
watermark detectability with and without attacks, and another for analyzing
the impact of these algorithms on text quality.

Figure 4: The standardized process of evaluation pipelines, the upper for
watermark detection pipeline, and the lower for text quality analysis
pipeline.

The upper part of Figure 4 illustrates the standardized process of watermark
detection. Following this process, we have implemented two pipelines:
WMDetect333Short for ‘Watermarked Text Detection Pipeline’. and
UWMDetect444Short for ‘Unwatermarked Text Detection Pipeline’.. The primary
difference between them lies in the text generation phase. The former requires
the use of the generate_watermarked_text method from the watermarking
algorithm, while the latter depends on the text_source parameter to determine
whether to directly retrieve natural text from a dataset or to invoke the
generate_unwatermarked_text method.

The lower part of Figure 4 illustrates the unified process of text quality
analysis. To evaluate the impact of watermarking on text quality, pairs of
watermarked and unwatermarked texts are generated. The texts, along with other
necessary inputs, are then processed and fed into a designated text quality
analyzer to produce detailed analysis and comparison results. Following this
process, we have implemented three pipelines for different evaluation
scenarios.

DirectQual.555Short for ‘Direct Text Quality Analysis Pipeline’. This pipeline
is specifically designed to analyze the quality of texts by directly comparing
the characteristics of watermarked texts with those of unwatermarked texts. It
evaluates metrics such as perplexity (PPL) and log diversity, without the need
for any external reference texts.

RefQual.666Short for ‘Referenced Text Quality Analysis Pipeline’. This
pipeline evaluates text quality by comparing both watermarked and
unwatermarked texts with a common reference text. It measures the degree of
similarity or deviation from the reference text. It is ideal for scenarios
that require specific downstream tasks to assess text quality, such as machine
translation and code generation.

ExDisQual.777Short for ‘External Discriminator Text Quality Analysis
Pipeline’. This pipeline employs an external judger, such as GPT-4 (OpenAI,
2023), to assess the quality of both watermarked and unwatermarked texts. The
discriminator evaluates the texts based on user-provided task descriptions,
identifying any potential degradation or preservation of quality due to
watermarking. This method is particularly valuable when an advanced, AI-based
analysis of the subtle effects of watermarking is required.

For complete code examples of all the pipelines mentioned above, please refer
to Appendix C.3.

##  4 Experiment

Using MarkLLM as a research tool, we conducted evaluations on nine algorithms,
assessing their detectability, robustness, and impact on text quality. Our
experiments aim to showcase MarkLLM’s effectiveness and efficiency through
practical case studies.

Table 2:  The evaluation results of assessing the detectability of nine
algorithms supported in MarkLLM. 200 watermarked texts are generated, while
200 non-watermarked texts serve as negative examples. We furnish TPR and
F1-score under dynamic threshold adjustments for 10% and 1% FPR, alongside
TPR, TNR, FPR, FNR, P, R, F1, ACC at optimal performance.

Method | 10%FPR | 1%FPR | Best
---|---|---|---
TPR | F1 | TPR | F1 | TPR | TNR | FPR | FNR | P | R | F1 | ACC
KGW | 1.000 | 0.955 | 0.995 | 0.993 | 1.000 | 0.995 | 0.005 | 0.000 | 0.995 | 1.000 | 0.998 | 0.998
Unigram | 1.000 | 0.950 | 1.000 | 0.993 | 1.000 | 1.000 | 0.000 | 0.000 | 1.000 | 1.000 | 1.000 | 1.000
SWEET | 1.000 | 0.952 | 1.000 | 0.995 | 1.000 | 1.000 | 0.000 | 0.000 | 1.000 | 1.000 | 1.000 | 1.000
UPV | $\times$ | $\times$ | $\times$ | $\times$ | 1.000 | 0.990 | 0.010 | 0.000 | 0.990 | 1.000 | 0.995 | 0.995
EWD | 1.000 | 0.952 | 1.000 | 0.995 | 1.000 | 1.000 | 0.000 | 0.000 | 1.000 | 1.000 | 1.000 | 1.000
SIR | 1.000 | 0.952 | 0.990 | 0.990 | 0.985 | 1.000 | 0.000 | 0.015 | 1.000 | 0.985 | 0.992 | 0.993
X-SIR | 0.990 | 0.947 | 0.935 | 0.961 | 0.985 | 0.980 | 0.020 | 0.015 | 0.980 | 0.985 | 0.983 | 0.983
EXP | 1.000 | 0.952 | 1.000 | 0.995 | 1.000 | 1.000 | 0.000 | 0.000 | 1.000 | 1.000 | 1.000 | 1.000
EXP-Edit | 1.000 | 0.952 | 0.995 | 0.990 | 0.995 | 0.985 | 0.015 | 0.005 | 0.985 | 0.995 | 0.990 | 0.990

Table 3:  The evaluation results of assessing the robustness of nine
algorithms supported in MarkLLM. For each attack, 200 watermarked texts are
generated and subsequently tampered, with an additional 200 non-watermarked
texts serving as negative examples. We report the TPR and F1-score at optimal
performance under each circumstance.

Method | No Attack | Word-D | Word-S | Word-S (Context) | Doc-P (GPT-3.5) | Doc-P (Dipper)
---|---|---|---|---|---|---
TPR | F1 | TPR | F1 | TPR | F1 | TPR | F1 | TPR | F1 | TPR | F1
KGW | 1.000 | 0.998 | 0.990 | 0.982 | 0.965 | 0.967 | 1.000 | 0.990 | 0.845 | 0.858 | 0.880 | 0.878
Unigram | 1.000 | 1.000 | 1.000 | 1.000 | 0.995 | 0.995 | 1.000 | 1.000 | 0.975 | 0.958 | 0.955 | 0.962
SWEET | 1.000 | 1.000 | 0.985 | 0.990 | 0.945 | 0.950 | 0.995 | 0.995 | 0.870 | 0.868 | 0.840 | 0.875
UPV | 1.000 | 0.995 | 0.970 | 0.980 | 0.885 | 0.896 | 0.985 | 0.961 | 0.925 | 0.906 | 0.862 | 0.864
EWD | 1.000 | 1.000 | 0.990 | 0.995 | 0.940 | 0.933 | 0.995 | 0.985 | 0.880 | 0.850 | 0.930 | 0.893
SIR | 0.985 | 0.992 | 0.995 | 0.997 | 0.932 | 0.956 | 0.940 | 0.970 | 0.880 | 0.910 | 0.930 | 0.896
X-SIR | 0.985 | 0.983 | 0.975 | 0.977 | 0.900 | 0.890 | 0.920 | 0.934 | 0.860 | 0.823 | 0.860 | 0.835
EXP | 1.000 | 1.000 | 1.000 | 0.995 | 0.930 | 0.928 | 0.975 | 0.973 | 0.720 | 0.776 | 0.800 | 0.812
EXP-Edit | 0.995 | 0.990 | 1.000 | 0.993 | 1.000 | 0.988 | 1.000 | 0.988 | 0.890 | 0.927 | 0.862 | 0.881

Table 4:  The evaluation results of assessing the text quality impact of the
nine algorithms supported in MarkLLM. We compared 200 watermarked texts with
200 non-watermarked texts. However, due to dataset constraints, only 100
watermarked texts were compared with 100 non-watermarked texts for code
generation.

Method | Direct Analysis | Referenced Analysis | External Discriminator
---|---|---|---
PPL(Ori.= 8.243) | Log Diversity(Ori.=8.517) | Machine Translation | Code Generation | Machine Translation
BLEU(Ori.=31.807) | pass@1(Ori.= 43.0) | GPT-4 Judge (Wat. Win Rate)
KGW | 13.551 $\uparrow$ | 7.989 $\downarrow$ | 28.242 $\downarrow$ | 34.0 $\downarrow$ | 0.31
Unigram | 13.723 $\uparrow$ | 7.242 $\downarrow$ | 26.075 $\downarrow$ | 32.0 $\downarrow$ | 0.33
SWEET | 13.747 $\uparrow$ | 8.086 $\downarrow$ | 28.242 $\downarrow$ |  37.0 $\downarrow$ | 0.31
UPV |  10.574 $\uparrow$ | 7.698 $\downarrow$ | 28.270 $\downarrow$ |  37.0 $\downarrow$ | 0.31
EWD | 13.402 $\uparrow$ | 8.220 $\downarrow$ | 28.242 $\downarrow$ | 34.0 $\downarrow$ | 0.30
SIR | 13.918 $\uparrow$ | 7.990 $\downarrow$ |  28.830 $\downarrow$ |  37.0 $\downarrow$ | 0.31
X-SIR | 12.885 $\uparrow$ | 7.930 $\downarrow$ | 28.161 $\downarrow$ | 36.0 $\downarrow$ | 0.33
EXP | 19.597 $\uparrow$ | 8.187 $\downarrow$ | $\times$ | 20.0 $\downarrow$ | $\times$
EXP-Edit | 21.591 $\uparrow$ |  9.046 $\uparrow$ | $\times$ | 14.0 $\downarrow$ | $\times$

###  4.1 Experiment Settings

Dateset and Prompt. For general-purpose text generation scenarios, we utilize
the C4 dataset (Raffel et al., 2020). Specifically, the first 30 tokens of
texts serve as prompts for generating the subsequent 200 tokens, with the
original C4 texts acting as non-watermarked examples. For specific downstream
tasks, we employ the WMT16 (Bojar et al., 2016) German-English dataset for
machine translation, and HumanEval (Chen et al., 2021) for code generation.

Language Model. For general-purpose text generation scenarios, we utilize
OPT-1.3b (Zhang et al., 2022) as language model. For specific downstream
tasks, we utilize NLLB-200-distilled-600M (Costa-jussà et al., 2022) for
machine translation and Starcoder (Li et al., 2023) for code generation.

Metrics and Attacks. Dynamic threshold adjustment is employed to evaluate
watermark detectability, with three settings provided: under a target FPR of
10%, under a target FPR of 1%, and under conditions for optimal F1 score
performance. To assess robustness, we utilize all text tampering attacks
listed in Table 1. For evaluating the impact on text quality, our metrics
include PPL, log diversity, BLEU (for machine translation), pass@1 (for code
generation), and assessments using GPT-4 Judge (Tu et al., 2023).

Hyper-parameters. Configuration files for each algorithms are listed in
Appendix E.1888Note that each algorithm was tested using only one parameter
configuration to demonstrate MarkLLM’s functionality and provide preliminary
reference data. Extensive performance comparisons across different aspects of
each algorithm would require varied parameter settings and further
experimentation.. Parameter settings for the evaluation tools are listed in
Appendix 6.

###  4.2 Results and Analysis

The results in Table 2999The evaluation results for UPV are only shown in the
“best” column because its watermark detection uses direct binary
classification without thresholds., Table 3, and Table 4101010Current
implementations of Christ family algorithms are designed for decoder-only
LLMs. As machine translation mainly uses encoder-decoder models, we did not
report the text quality produced by EXP and EXP-edit in machine translation.
demonstrate that by using the implementations of different algorithms and the
evaluation pipelines provided in MarkLLM, researchers can effectively
reproduce the experimental results from previous watermarking papers. These
experiments can be conducted by running simple scripts (detailed in Appendix
D), showcasing MarkLLM’s capability for easy evaluation of watermark
algorithms in various scenarios. This highlights the tool’s user-friendliness
and practical utility, offering valuable insights for future research.

Through systematic evaluation, it can be observed that: (1) Current LLM
watermarking algorithms excel in achieving accurate detection, boasting
F1-scores surpassing 0.99 in no-attack conditions; (2) Different algorithms
demonstrate distinct strengths across various aspects, necessitating
consideration of specific circumstance when selecting an algorithm; (3) Even
when evaluating from the same perspective, results can vary depending on the
metrics or types of attacks used. This highlights the need for a thorough
assessment when judging algorithms; (4) Striking a harmonious balance between
various evaluation perspectives poses a significant challenge. Future research
should prioritize balancing and enhancing the overall capabilities of
algorithms.

In summary, MarkLLM acts as a convenient tool for conducting diverse
evaluation experiments, effectively minimizing assessment expenses. Future
research can leverage MarkLLM for comprehensive exploration and analysis.

##  5 Conclusion

MarkLLM is a comprehensive open-source toolkit for LLM watermarking. It allows
users to easily try various state-of-the-art algorithms with flexible
configurations to watermark their own text and conduct detection, and provides
clear visualizations to gain insights into the underlying mechanisms. The
inclusion of convenient evaluation tools and customizable evaluation pipelines
enables automatic and thorough assessments from various perspectives. As LLM
watermarking evolves, MarkLLM aims to be a collaborative platform that grows
with the research community. By providing a solid foundation and inviting
contributions, we aim to foster a vibrant ecosystem where researchers and
developers can work together to advance the state-of-the-art in LLM
watermarking technology.

## Limitations

MarkLLM is a comprehensive toolkit for implementing, visualizing, and
evaluating LLM watermarking algorithms. However, it currently only integrates
a subset of existing watermarking methods and does not yet support some recent
approaches that directly embed watermarks into model parameters during
training (Xu et al., 2024; Gu et al., 2024). We anticipate future
contributions to expand MarkLLM’s coverage and enhance its versatility.

In terms of visualization, we have provided one tailored solution for each of
the two main watermarking algorithm families. While these solutions offer
valuable insights, there is room for more creative and diverse visualization
designs.

Regarding evaluation, we have covered aspects such as detectability,
robustness, and text quality impact. However, our current toolkit may not
encompass all possible scenarios, such as re-translation and CWRA (He et al.,
2024) attacks related to robustness.

We acknowledge that MarkLLM has room for improvement. We warmly welcome
developers and researchers to contribute their code and insights to help build
a more comprehensive and robust ecosystem for LLM watermarking. Through
collaborative efforts, we can further advance this technology and unlock its
full potential.

## References

  * Aaronson and Kirchner (2022) S. Aaronson and H. Kirchner. 2022.  Watermarking gpt outputs.  https://www.scottaaronson.com/talks/watermark.ppt.
  * Bojar et al. (2016) Ond rej Bojar, Rajen Chatterjee, Christian Federmann, Yvette Graham, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Philipp Koehn, Varvara Logacheva, Christof Monz, Matteo Negri, Aurelie Neveol, Mariana Neves, Martin Popel, Matt Post, Raphael Rubino, Carolina Scarton, Lucia Specia, Marco Turchi, Karin Verspoor, and Marcos Zampieri. 2016.  Findings of the 2016 conference on machine translation.  In _Proceedings of the First Conference on Machine Translation_ , pages 131–198, Berlin, Germany. Association for Computational Linguistics.
  * Chen et al. (2021) Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. 2021.  Evaluating large language models trained on code.
  * Christ et al. (2023) Miranda Christ, Sam Gunn, and Or Zamir. 2023.  Undetectable watermarks for language models.  _arXiv preprint arXiv:2306.09194_.
  * Costa-jussà et al. (2022) Marta R Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, et al. 2022.  No language left behind: Scaling human-centered machine translation.  _arXiv preprint arXiv:2207.04672_.
  * Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_.
  * Fairoze et al. (2023) Jaiden Fairoze, Sanjam Garg, Somesh Jha, Saeed Mahloujifar, Mohammad Mahmoody, and Mingyuan Wang. 2023.  Publicly detectable watermarking for language models.  Cryptology ePrint Archive, Paper 2023/1661.  https://eprint.iacr.org/2023/1661.
  * Fernandez et al. (2023) Pierre Fernandez, Antoine Chaffin, Karim Tit, Vivien Chappelier, and Teddy Furon. 2023.  Three bricks to consolidate watermarks for large language models.  _arXiv preprint arXiv:2308.00113_.
  * Gu et al. (2024) Chenchen Gu, Xiang Lisa Li, Percy Liang, and Tatsunori Hashimoto. 2024.  On the learnability of watermarks for language models.
  * He et al. (2024) Zhiwei He, Binglin Zhou, Hongkun Hao, Aiwei Liu, Xing Wang, Zhaopeng Tu, Zhuosheng Zhang, and Rui Wang. 2024.  Can watermarks survive translation? on the cross-lingual consistency of text watermark for large language models.  _arXiv preprint arXiv:2402.14007_.
  * Hu et al. (2023) Zhengmian Hu, Lichang Chen, Xidong Wu, Yihan Wu, Hongyang Zhang, and Heng Huang. 2023.  Unbiased watermark for large language models.  _arXiv preprint arXiv:2310.10669_.
  * Kirchenbauer et al. (2023a) John Kirchenbauer, Jonas Geiping, Yuxin Wen, Jonathan Katz, Ian Miers, and Tom Goldstein. 2023a.  A watermark for large language models.  In _International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA_ , volume 202 of _Proceedings of Machine Learning Research_ , pages 17061–17084. PMLR.
  * Kirchenbauer et al. (2023b) John Kirchenbauer, Jonas Geiping, Yuxin Wen, Manli Shu, Khalid Saifullah, Kezhi Kong, Kasun Fernando, Aniruddha Saha, Micah Goldblum, and Tom Goldstein. 2023b.  On the reliability of watermarks for large language models.  _arXiv preprint arXiv:2306.04634_.
  * Krishna et al. (2023) Kalpesh Krishna, Yixiao Song, Marzena Karpinska, John Wieting, and Mohit Iyyer. 2023.  Paraphrasing evades detectors of ai-generated text, but retrieval is an effective defense.  _arXiv preprint arXiv:2303.13408_.
  * Kuditipudi et al. (2023) Rohith Kuditipudi, John Thickstun, Tatsunori Hashimoto, and Percy Liang. 2023.  Robust distortion-free watermarks for language models.  _arXiv preprint arXiv:2307.15593_.
  * Lee et al. (2023) Taehyun Lee, Seokhee Hong, Jaewoo Ahn, Ilgee Hong, Hwaran Lee, Sangdoo Yun, Jamin Shin, and Gunhee Kim. 2023.  Who wrote this code? watermarking for code generation.  _arXiv preprint arXiv:2305.15060_.
  * Li et al. (2023) Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, Qian Liu, Evgenii Zheltonozhskii, Terry Yue Zhuo, Thomas Wang, Olivier Dehaene, Mishig Davaadorj, Joel Lamy-Poirier, João Monteiro, Oleh Shliazhko, Nicolas Gontier, Nicholas Meade, Armel Zebaze, Ming-Ho Yee, Logesh Kumar Umapathi, Jian Zhu, Benjamin Lipkin, Muhtasham Oblokulov, Zhiruo Wang, Rudra Murthy, Jason Stillerman, Siva Sankalp Patel, Dmitry Abulkhanov, Marco Zocca, Manan Dey, Zhihan Zhang, Nour Fahmy, Urvashi Bhattacharyya, Wenhao Yu, Swayam Singh, Sasha Luccioni, Paulo Villegas, Maxim Kunakov, Fedor Zhdanov, Manuel Romero, Tony Lee, Nadav Timor, Jennifer Ding, Claire Schlesinger, Hailey Schoelkopf, Jan Ebert, Tri Dao, Mayank Mishra, Alex Gu, Jennifer Robinson, Carolyn Jane Anderson, Brendan Dolan-Gavitt, Danish Contractor, Siva Reddy, Daniel Fried, Dzmitry Bahdanau, Yacine Jernite, Carlos Muñoz Ferrandis, Sean Hughes, Thomas Wolf, Arjun Guha, Leandro von Werra, and Harm de Vries. 2023.  Starcoder: may the source be with you!
  * Liu et al. (2024a) Aiwei Liu, Leyi Pan, Xuming Hu, Shuang Li, Lijie Wen, Irwin King, and Philip S. Yu. 2024a.  An unforgeable publicly verifiable watermark for large language models.  In _The Twelfth International Conference on Learning Representations_.
  * Liu et al. (2024b) Aiwei Liu, Leyi Pan, Xuming Hu, Shiao Meng, and Lijie Wen. 2024b.  A semantic invariant robust watermark for large language models.  In _The Twelfth International Conference on Learning Representations_.
  * Liu et al. (2023) Aiwei Liu, Leyi Pan, Yijian Lu, Jingjing Li, Xuming Hu, Lijie Wen, Irwin King, and Philip S Yu. 2023.  A survey of text watermarking in the era of large language models.  _arXiv preprint arXiv:2312.07913_.
  * Lu et al. (2024) Yijian Lu, Aiwei Liu, Dianzhi Yu, Jingjing Li, and Irwin King. 2024.  An entropy-based text watermarking detection method.  _arXiv preprint arXiv:2403.13485_.
  * Megías et al. (2021) David Megías, Minoru Kuribayashi, Andrea Rosales, and Wojciech Mazurczyk. 2021.  Dissimilar: Towards fake news detection using information hiding.  In _Signal Processing and Machine Learning. In The 16th International Conference on Availability, Reliability and Security (Vienna, Austria)(ARES 2021). Association for Computing Machinery, New York, NY, USA, Article_ , volume 66.
  * Miller (1995) George A Miller. 1995.  Wordnet: a lexical database for english.  _Communications of the ACM_ , 38(11):39–41.
  * OpenAI (2022) OpenAI. 2022.  Chatgpt: Optimizing language models for dialogue.  https://openai.com/blog/chatgpt.
  * OpenAI (2023) OpenAI. 2023.  Gpt-4 technical report.  _ArXiv_ , abs/2303.08774.
  * Piet et al. (2023) Julien Piet, Chawin Sitawarin, Vivian Fang, Norman Mu, and David Wagner. 2023.  Mark my words: Analyzing and evaluating language model watermarks.
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _The Journal of Machine Learning Research_ , 21(1):5485–5551.
  * Ren et al. (2023) Jie Ren, Han Xu, Yiding Liu, Yingqian Cui, Shuaiqiang Wang, Dawei Yin, and Jiliang Tang. 2023.  A robust semantics-based watermark for large language model against paraphrasing.  _arXiv preprint arXiv:2311.08721_.
  * Salewski et al. (2023) Leonard Salewski, Stephan Alaniz, Isabel Rio-Torto, Eric Schulz, and Zeynep Akata. 2023.  In-context impersonation reveals large language models’ strengths and biases.
  * Takezawa et al. (2023) Yuki Takezawa, Ryoma Sato, Han Bao, Kenta Niwa, and Makoto Yamada. 2023.  Necessary and sufficient watermark for large language models.  _arXiv preprint arXiv:2310.00833_.
  * Touvron et al. (2023) Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023.  Llama: Open and efficient foundation language models.  _arXiv preprint arXiv:2302.13971_.
  * Tu et al. (2023) Shangqing Tu, Yuliang Sun, Yushi Bai, Jifan Yu, Lei Hou, and Juanzi Li. 2023.  Waterbench: Towards holistic evaluation of watermarks for large language models.  _arXiv preprint arXiv:2311.07138_.
  * Vasilatos et al. (2023) Christoforos Vasilatos, Manaar Alam, Talal Rahwan, Yasir Zaki, and Michail Maniatakos. 2023.  Howkgpt: Investigating the detection of chatgpt-generated university student homework through context-aware perplexity analysis.  _arXiv preprint arXiv:2305.18226_.
  * Wang et al. (2023) Lean Wang, Wenkai Yang, Deli Chen, Hao Zhou, Yankai Lin, Fandong Meng, Jie Zhou, and Xu Sun. 2023.  Towards codable text watermarking for large language models.  _arXiv preprint arXiv:2307.15992_.
  * Wu et al. (2023) Yihan Wu, Zhengmian Hu, Hongyang Zhang, and Heng Huang. 2023.  Dipmark: A stealthy, efficient and resilient watermark for large language models.  _arXiv preprint arXiv:2310.07710_.
  * Xu et al. (2024) Xiaojun Xu, Yuanshun Yao, and Yang Liu. 2024.  Learning to watermark llm-generated text via reinforcement learning.
  * Yoo et al. (2023) KiYoon Yoo, Wonhyuk Ahn, and Nojun Kwak. 2023.  Advancing beyond identification: Multi-bit watermark for language models.  _arXiv preprint arXiv:2308.00221_.
  * Zhang et al. (2022) Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. 2022.  Opt: Open pre-trained transformer language models.
  * Zhao et al. (2023) Xuandong Zhao, Prabhanjan Ananth, Lei Li, and Yu-Xiang Wang. 2023.  Provable robust watermarking for ai-generated text.  _arXiv preprint arXiv:2306.17439_.

##  Appendix A Supported LLM Watermarking Algorithms

Table 5 showcases the nine watermarking algorithms that have been integrated
into MarkLLM. These algorithms were carefully selected to include
representatives from both the KGW family and the Christ family, ensuring broad
coverage across diverse optimization objectives such as enhancing
detectability, improving robustness, bettering text quality, and enabling
public detection.

Table 5: Details of watermarking algorithms supported in MarkLLM. Algorithm Name |  Category |  Methodology
---|---|---
KGW (Kirchenbauer et al., 2023a) |  KGW Family |  Separate the vocabulary set into two lists: a red list and a green list based on the preceding token, then add bias to green tokens so that the LLM-produced text exhibits preference of using green tokens.
Unigram (Zhao et al., 2023) |  KGW Family |  Use a globally fixed red-green list separation for every location, aiming to enhance robustness against tampering.
SWEET (Lee et al., 2023) |  KGW Family |  Modifications to the logits are selectively applied only to high-entropy locations while bypassing low-entropy ones, aiming to improve text quality in low-entropy scenarios, such as code generation.
UPV (Liu et al., 2024a) |  KGW Family |  Train one network as a generator for segregating red-green lists and another network as a detector for directly providing a classification result based on textual input. The distinction between the generator network and the detector network enables public detection.
EWD (Lu et al., 2024) |  KGW Family |  During the watermark detection phase, each token is assigned a different weight based on its entropy, with tokens having higher entropy receiving greater weight. This approach aims to enhance watermark detectability in low-entropy situations.
SIR (Liu et al., 2024b) |  KGW Family |  Train a generator network to convert token embeddings into context-aware biases, thereby enhancing robustness against semantic invariant tampering.
X-SIR (He et al., 2024) |  KGW Family |  The red-green partition of the vocabulary no longer operates at the token level but rather at the level of semantic clusters, grouping semantically similar words together within the same group and adding bias at the group level. This shift from adding bias to individual green tokens to green clusters is designed to enhance robustness against Cross-lingual Watermark Removal Attacks (CWRA).
EXP (Aaronson and Kirchner, 2022) |  Christ Family |  Utilize a pseudo-random sequence based on preceding tokens to guide token sampling when generating each new token through the exponential (EXP) sampling rule. As a result, the watermarked text displays a degree of alignment with the sequence, aiding in its detection.
EXP-Edit (Kuditipudi et al., 2023) |  Christ Family |  Expanding on the use of the EXP sampling rule, introduce the concept of edit distance to measure the alignment between the pseudo-random sequence and the text, which significantly improves its robustness against tampering.

##  Appendix B Handling Weighted Token Differences in Visualization

Currently, several LLM Watermarking algorithms assign different weights to
each token during detection to adapt to more stringent usage environments.
Consequently, we have also implemented the visualization of these weights. For
instance, both the SWEET (Lee et al., 2023) and EWD (Lu et al., 2024) methods
were developed to function effectively in low-entropy environments.

The SWEET method calculates the entropy of each token before watermarking it.
If the entropy is below a certain threshold, the token is not watermarked,
thus mitigating the problem of text quality degradation due to modifications
in low entropy areas.Therefore, the weights assigned to each token are either
0 or 1. On the other hand, the EWD method follows the KGW method during the
watermarking process but assigns a continuous range of weights from 0 to 1 to
each token based on its entropy during detection. Tokens with higher entropy
receive higher weights, which helps alleviate the decline in detection
accuracy in low-entropy environments. For the EWD method, the weights for each
token are continuous values within the range of 0 to 1.

Figure 5 displays the visualization results for the SWEET and EWD methods. It
can be observed that different shades of gray underline are used to represent
the weight each token holds during detection. By visualizing the weights
assigned to each token, users can gain a clearer insight into the internal
mechanisms of watermarking algorithms that involve weight considerations. This
enhanced visibility helps in understanding how these algorithms adapt to
various textual contexts and the rationale behind the differential treatment
of tokens based on their entropy or other characteristics.

Figure 5: Visualization results of watermarked text using SWEET and EWD as
watermarking algorithms.

##  Appendix C Example Code

###  C.1 Example Code for Watermarking Algorithm Invocation

⬇

1# Device

2device = "cuda" if torch.cuda.is_available() else "cpu"

3

4# Transformers config

5transformers_config = TransformersConfig(

6 model=AutoModelForCausalLM.from_pretrained("facebook/opt-1.3b/").to(device),

7 tokenizer=AutoTokenizer.from_pretrained("facebook/opt-1.3b/"),

8 vocab_size=50272,

9 device=device,

10 max_new_tokens=200,

11 min_length=230,

12 do_sample=True,

13 no_repeat_ngram_size=4)

14

15# Load watermark algorithm

16myWatermark = AutoWatermark.load(’KGW’,

17 algorithm_config=f’config/KGW.json’,

18 transformers_config=transformers_config)

19

20# Generate watermarked text

21watermarked_text = myWatermark.generate_watermarked_text(prompt)

22

23# Detect watermark

24detect_result = myWatermark.detect_watermark(watermarked_text)

###  C.2 Example Code for Mechanism Visualization

⬇

1# Get data for visualization

2watermarked_data = myWatermark.get_data_for_visualization(watermarked_text)

3

4# Init visualizer

5visualizer = DiscreetVisualizer(

6 color_scheme=ColorSchemeForDiscreetVisualization(),

7 font_settings=FontSettings(),

8 page_layout_settings=PageLayoutSettings(),

9 legend_settings=DiscreetLegendSettings())

10

11# Visualize

12watermarked_img = visualizer.visualize(data=watermarked_data,

13 show_text=False,

14 visualize_weight=False,

15 display_legend=True)

###  C.3 Example Code for Evaluation Pipelines Invocation

####  C.3.1 Watermark Detection Pipeline

⬇

1# Dataset

2my_dataset = C4Dataset(’dataset/c4/processed_c4.json’)

3

4# Watermarked text detection pipeline

5pipeline1 = WatermarkedTextDetectionPipeline(

6 dataset=my_dataset,

7 text_editor_list=[TruncatePromptTextEditor(), WordDeletion(ratio=0.3)],

8 show_progress=True,

9 return_type=DetectionPipelineReturnType.SCORES)

10

11# Unwatermarked text detection pipeline

12pipeline2 = UnWatermarkedTextDetectionPipeline(

13 dataset=my_dataset,

14 text_editor_list=[],

15 show_progress=True,

16 return_type=DetectionPipelineReturnType.SCORES)

17

18# Init calculator

19calculator = DynamicThresholdSuccessRateCalculator(labels=[’TPR’, ’F1’],

20 rule=’best’)

21

22# Calculate success rate of watermark detection

23print(calculator.calculate(pipeline1.evaluate(my_watermark),

24 pipeline2.evaluate(my_watermark)))

####  C.3.2 Text Quality Analysis Pipeline

⬇

1# Dataset

2my_dataset = C4Dataset(’dataset/c4/processed_c4.json’)

3

4# Init pipeline

5quality_pipeline = DirectTextQualityAnalysisPipeline(

6 dataset=my_dataset,

7 watermarked_text_editor_list=[TruncatePromptTextEditor()],

8 unwatermarked_text_editor_list=[],

9 analyzer=LogDiversityCalculator(),

10 unwatermarked_text_source=’natural’,

11 show_progress=True,

12 return_type=QualityPipelineReturnType.MEAN_SCORES)

13

14# Evaluate impact on text quality

15print(quality_pipeline.evaluate(my_watermark))

##  Appendix D Scripts for Evaluation

In the evaluation module of MarkLLM, we have included a suite of Python
scripts that are specifically designed to leverage the toolkit’s pipeline for
assessing the performance of various watermarking algorithms. These scripts
are accessible within the repository under the directory
evaluation/examples/. The files provided are  assess_detectability.py,
assess_robustness.py, and assess_quality.py. Utilizing the shells below
(Listing 1, Listing 2, Listing 3), researchers can seamlessly conduct a
thorough evaluation of various watermarking algorithms. This streamlined
approach facilitates the comparison and analysis essential for advancing the
field.

⬇

1python evaluation/examples/assess_detectability.py \--algorithm KGW \--labels
TPR F1 \--rules target_fpr \--target_fpr 0.01

2

3python evaluation/examples/assess_detectability.py \--algorithm KGW \--labels
TPR TNR FPR FNR P R F1 ACC \--rules best

Listing 1: Execution of assess detectability script. It accepts three command-
line parameters: the name of the watermark algorithm to be evaluated, a list
of metrics to return (choose one or more from FPR, FNR, TPR, TNR, P, R, F1,
ACC), and the evaluation rule (choose from best and target_fpr). Additionally,
target_fpr should be specified if the target_fpr evaluation rule is selected.

⬇

1python evaluation/examples/assess_robustness.py \--algorithm KGW \--attack
'Word-D'

2

3python evaluation/examples/assess_robustness.py \--algorithm Unigram
\--attack 'Doc-P(GPT-3.5)'

Listing 2: Execution of assess robustness script. It accepts two command-line
parameters: the name of the watermark algorithm to be evaluated, and the name
of the text tampering attack to be used (choose one from Word-D, Word-S,
Word-S(Context), Doc-P(GPT-3.5) and Doc-P(Dipper)).

⬇

1python evaluation/examples/assess_quality.py \--algorithm KGW \--metric PPL

2

3python evaluation/examples/assess_quality.py \--algorithm SIR \--metric 'Log
Diversity'

Listing 3: Execution of assess quality script. It accepts two command-line
parameters: the name of the watermark algorithm to be evaluated, and the name
of the text quality metric to be used (choose one from PPL, Log Diversity,
BLEU, pass@1 and GPT-4 Judge).

##  Appendix E Hyper-parmaters for Experiment

###  E.1 Configuration Files of Watermarking Algorithms

⬇

1{

2 "algorithm_name": "KGW",

3 "gamma": 0.5,

4 "delta": 2.0,

5 "hash_key": 15485863,

6 "prefix_length": 1,

7 "z_threshold": 4.0

8}

Listing 4: configuration file, KGW.json

⬇

1{

2 "algorithm_name": "Unigram",

3 "gamma": 0.5,

4 "delta": 2.0,

5 "hash_key": 15485863,

6 "z_threshold": 4.0

7}

Listing 5: configuration file, Unigram.json

⬇

1{

2 "algorithm_name": "SWEET",

3 "gamma": 0.5,

4 "delta": 2.0,

5 "hash_key": 15485863,

6 "z_threshold": 4.0,

7 "prefix_length": 1,

8 "entropy_threshold": 0.9

9}

Listing 6: configuration file, SWEET.json

⬇

1{

2 "algorithm_name": "UPV",

3 "gamma": 0.5,

4 "delta": 2.0,

5 "z_threshold": 4.0,

6 "prefix_length": 1,

7 "bit_number": 16,

8 "sigma": 0.01,

9 "default_top_k": 20,

10 "generator_model_name": "watermark/upv/model/generator_model_b16_p1.pt",

11 "detector_model_name": "watermark/upv/model/detector_model_b16_p1_z4.pt",

12 "detect_mode": "network"

13}

Listing 7: configuration file, UPV.json

⬇

1{

2 "algorithm_name": "EWD",

3 "gamma": 0.5,

4 "delta": 2.0,

5 "hash_key": 15485863,

6 "prefix_length": 1,

7 "z_threshold": 4.0

8}

Listing 8: configuration file, EWD.json

⬇

1{

2 "algorithm_name": "SIR",

3 "delta": 1.0,

4 "chunk_length": 10,

5 "scale_dimension": 300,

6 "z_threshold": 0.0,

7 "transform_model_input_dim": 1024,

8 "transform_model_name": "watermark/sir/model/transform_model_cbert.pth",

9 "embedding_model_path": "watermark/sir/model/compositional-bert-large-
uncased/",

10 "mapping_name": "watermark/sir/mapping/300_mapping_50272.json"

11}

Listing 9: configuration file, SIR.json

⬇

1{

2 "algorithm_name": "XSIR",

3 "delta": 1.0,

4 "chunk_length": 10,

5 "scale_dimension": 300,

6 "z_threshold": 0.2,

7 "transform_model_input_dim": 768,

8 "dictionary": "watermark/xsir/dictionary/dictionary.txt",

9 "transform_model_name": "watermark/xsir/model/transform_model_x-
sbert_10K.pth",

10 "embedding_model_path": "watermark/xsir/model/paraphrase-multilingual-
mpnet-base-v2",

11 "mapping_name": "watermark/xsir/mapping/300_mapping_opt_1_3b.json"

12}

Listing 10: configuration file, XSIR.json

⬇

1{

2 "algorithm_name": "EXP",

3 "prefix_length": 4,

4 "hash_key": 15485863,

5 "threshold": 2.0,

6 "sequence_length": 200

7}

Listing 11: configuration file, EXP.json

⬇

1{

2 "algorithm_name": "EXPEdit",

3 "pseudo_length": 420,

4 "sequence_length": 200,

5 "n_runs": 100,

6 "key": 42,

7 "p_threshold": 0.2

8}

Listing 12: configuration file, EXP_Edit.json

###  E.2 Hyper-paramters of Evaluation Tools

Table 6 listed the hyper-parameters of evaluation tools used in Section 4.

Table 6: Hyperparamers of evaluation tools used in experiment. Tools | Hyperparameters
---|---
FundamentalSuccessRateCalculator | None
DynamicThresholdSuccessRateCalculator | None
WordDeletion | ratio: 0.3
SynonymSubstitution | ratio: 0.5
ContextAwareSynonymSubstitution | ratio: 0.5
GPTParaphraser | openai_model: "gpt-3.5-turbo", prompt: "Please rewrite the following text: "
DipperParaphraser | lex_diversity: 60, order_diversity: 0, max_new_tokens: 100, do_sample: True, top_p: 0.75
PPLCalculator | model: Llama-7B, tokenizer: Llama-7B
LogDiversityAnalyzer | None
BLEUCalculator | None
PassOrNotJudger | None
GPTTextDiscriminator | openai_model: "gpt-4", task_description: “Translate the following German text to English.”

##  Appendix F Comparison with Competitors

With the flourishing advancement of LLM watermarking technology, there has
been a notable rise in the development of frameworks dedicated to this field.
Among these, WaterBench (Tu et al., 2023) and Mark My Words (Piet et al.,
2023) stand out as prominent examples. WaterBench primarily focuses on
assessing the impact of KGW (Kirchenbauer et al., 2023a), Unigram (Zhao et
al., 2023), and KGW-v2 (Kirchenbauer et al., 2023b) on text quality, while
Mark My Words evaluates the performance of KGW (Kirchenbauer et al., 2023a),
EXP (Aaronson and Kirchner, 2022), Christ (Christ et al., 2023), and EXP-Edit
(Kuditipudi et al., 2023) across multiple dimensions, including text quality,
robustness against tampering, and number of tokens needed for detection.

While both of these frameworks focus primarily on benchmark construction,
similar to the evaluation module in MarkLLM, MarkLLM sets itself apart by not
only offering easy-to-use evaluation tools and automated pipelines that
encompass the aforementioned assessment perspectives but also providing a
unified implementation framework for watermarking algorithms and visualization
tools for their underlying mechanisms. This enhances its utility as a
comprehensive toolkit. The integration of these functionalities renders
MarkLLM a more versatile and accessible resource, enabling convenient usage,
understanding, evaluation, and selection of diverse watermarking algorithms by
researchers and the broader community. This plays a crucial role in fostering
consensus both within and beyond the field.


Further author information: (Send correspondence to J.R.M.)
J.R.M.: E-mail<EMAIL_ADDRESS>
# High-Contrast Imaging at First-Light of the GMT: The Preliminary Design of
GMagAO-X

Jared R. Males  Steward Observatory, University of Arizona  Laird M. Close
Steward Observatory, University of Arizona  Sebastiaan Y. Haffert  Steward
Observatory, University of Arizona  Leiden Observatory, Leiden University, The
Netherlands  Maggie Y. Kautz  Steward Observatory, University of Arizona
James C. Wyant College of Optical Sciences, University of Arizona, USA  Doug
Kelly  Steward Observatory, University of Arizona  Adam Fletcher  Steward
Observatory, University of Arizona  Thomas Salanski  Steward Observatory,
University of Arizona  Olivier Durney  Steward Observatory, University of
Arizona  Jamison Noenickx  Steward Observatory, University of Arizona  John
Ford  Steward Observatory, University of Arizona  Victor Gasho  Steward
Observatory, University of Arizona  Logan Pearce  Steward Observatory,
University of Arizona  Jay Kueny  Steward Observatory, University of Arizona
James C. Wyant College of Optical Sciences, University of Arizona, USA
Olivier Guyon  Steward Observatory, University of Arizona  James C. Wyant
College of Optical Sciences, University of Arizona, USA  Subaru Telescope,
National Astronomical Observatory of Japan  Astrobiology Center, National
Institutes of Natural Sciences, Japan  Alycia Weinberger  Carnegie Institution
for Science, Washington, D.C., USA  Brendan Bowler  University of Texas at
Austin, USA  Adam Kraus  University of Texas at Austin, USA  Natasha Batalha
NASA Ames Research Center, USA

###### Abstract

We present the preliminary design of GMagAO-X, the first-light high-contrast
imager planned for the Giant Magellan Telescope. GMagAO-X will realize the
revolutionary increase in spatial resolution and sensitivity provided by the
25 m GMT. It will enable, for the first time, the spectroscopic
characterization of nearby potentially habitable terrestrial exoplanets
orbiting late-type stars. Additional science cases include: reflected light
characterization of mature giant planets; measurement of young extrasolar
giant planet variability; characterization of circumstellar disks at
unprecedented spatial resolution; characterization of benchmark stellar
atmospheres at high spectral resolution; and mapping of resolved objects such
as giant stars and asteroids. These, and many more, science cases will be
enabled by a 21,000 actuator extreme adaptive optics system, a coronagraphic
wavefront control system, and a suite of imagers and spectrographs. We will
review the science-driven performance requirements for GMagAO-X, which include
achieving a Strehl ratio of 70% at 800 nm on 8th mag and brighter stars, and
post-processed characterization at astrophysical flux-ratios of 1e-7 at 4
lambda/D (26 mas at 800 nm) separation. We will provide an overview of the
resulting mechanical, optical, and software designs optimized to deliver this
performance. We will also discuss the interfaces to the GMT itself, and the
concept of operations. We will present an overview of our end-to-end
performance modeling and simulations, including the control of segment
phasing, as well as an overview of prototype lab demonstrations. Finally, we
will review the results of Preliminary Design Review held in February, 2024.

###### keywords:

adaptive optics

##  1 INTRODUCTION

The 25-40 m Extremely Large Telescopes (ELTs) will transform the study of
extrasolar planets. Their larger collecting area, compared to current
generation 5-10 m telescopes, will dramatically improve the sensitivity of
exoplanet studies including radial velocity measurements and transit
spectroscopy on nearby planets. The gains for resolved direct imaging of
exoplanets will be even more dramatic. The most profound result of this
transformative leap in capabilities will be the ability to search for the
signatures of life on planets orbiting other stars. The Astro2020 Decadal
Survey[1] placed the search for life on exoplanets as a Priority Area for
astrophysics over the next decade. In “Pathways to Habitable Worlds”,
Astro2020 found that one of the four key capabilities needed to achieve this
is: Ground-based extremely large telescopes equipped with high-resolution
spectroscopy, high-performance adaptive optics, and high-contrast imaging.[1]

Preparing the instrumentation suite of the ELTs to deliver on this promise is
crucial, but also a demanding technological challenge. It is well known that
in the background-noise-limited case, the sensitivity (time to signal-to-
noise, SNR) to point sources of diffraction-limited telescopes scales with
diameter $D^{4}$. Just as importantly, the improvement in spatial resolution
with $1/D$ will allow direct imaging at smaller separations than currently
possible with today’s telescopes. However, the application of these scaling
laws to direct imaging is non-trivial. $D^{4}$ only obtains if the noise
scales with $D^{2}$ and exposure time, but this is well-known to not be the
case at small separations for typical high-contrast imaging instruments. This
is due to the internal systematics driven quasi-static speckle problem[2].
Only if quasi-static speckles are controlled and suppressed such that residual
speckle lifetimes are short will such an instrument achieve $D^{4}$[3].
Furthermore, the science foreseen by Astro2020 requires working at small
separations, $\lesssim 5\lambda/D$, with coronagraphs. This requires exquisite
wavefront sensing and control (WFS&C) and so places demanding requirements on
a direct imaging instrument.

The 25 m Giant Magellan Telescope (GMT) presents the best opportunity to
realize the vision of Astro2020 early in the ELT era. Given its unique
aperture and smaller $D$, an instrument can be constructed to achieve higher
Strehl with fewer deformable mirror (DM) actuators. This makes the GMT the
ideal facility for short wavelength Extreme AO (ExAO) [4, 5]. The large-
segment design of the GMT provides a straightforward way to deploy mature,
well tested DM technology with high actuator density. Equipped with an ExAO-
fed coronagraph, the GMT will characterize large numbers of temperate, mature
exoplanets for the first time[6, 7], including terrestrial potentially
habitable exoplanets.

Here we present the results of the preliminary design for GMagAO-X, an
instrument in development to provide a first-light ExAO and coronagraph
facility for GMT. We will review the science cases, science-driven
requirements, and the mechanical, optical, electrical, and software & control
preliminary designs. GMagAO-X underwent a successful preliminary design review
(PDR) in February, 2024 and is now preparing for final design.

Table 1: Key Instrument Parameters of GMagAO-X  Parameter | Requirement | Goal | Stretch Goal | Notes
---|---|---|---|---
Wavelength Coverage | 600 – 1900 nm | 450 – 1900 nm | 350–1900 nm | GMAGX-SCI-002, GMAGX-SCI-006
Spatial Resolution | 4.9 mas | 3.7 mas | 2.8 mas | GMAGX-SCI-002
Spectral Resolution | BB: 10% bands |  |  | GMAGX-SCI-003, GMAGX-SCI-004, GMAGX-SCI-005
Low: 20–100 |  |
High: 1000–65,000 | 100,000 |
Guide Star I Magnitude Range | -1.5 – 13 | -1.5 – 15 |  | GMAGX-SCI-007
Field of View | 3” x 3” | 3.5” x 6” |  | GMAGX-SCI-010
Coronagraph Contrast | 1e-7 @ 4$\lambda/D$ |  1e-7 @ 2$\lambda/D$ |  1e-8 @ 1$\lambda/D$ | 5$\sigma$ planet:star flux. GMAGX-SCI-001, GMAGX-SCI-008
1e-8 @ 6$\lambda/D$ |  1e-9 @ 5$\lambda/D$

##  2 SCIENCE CASES AND REQUIREMENTS

The GMagAO-X science team produced the following core science cases. We
briefly state them here, see the GMagAO-X Instrument Science Requirements
Document (GMAGX-DOC-0001) for detailed discussion.

  1. 1.

Reflected Light Imaging and the Search for New Life

    1. (a)

Characterizing Temperate Giant Exoplanets

    2. (b)

Terrestrial Planets: Detecting and Characterizing the Atmosphere

      1. i.

Detection of biosignatures (e.g. $H_{2}O$, $O_{2}$).

    3. (c)

Reconnaissance for Habitable World’s Observatory

  2. 2.

Characterizing Young Extrasolar Giant Planets (EGPs)

    1. (a)

Young Self-Luminous EGPs

    2. (b)

Rotational Periods of EGPs

    3. (c)

Cloud 3-D Structure

  3. 3.

Planet Formation at Low Mass and Small Separation

    1. (a)

Accretion onto Planets and Disks

  4. 4.

Circumstellar Disk Structure and Disk-Planet Interactions

  5. 5.

Stellar Evolution

    1. (a)

Benchmark Binary System Characterization

    2. (b)

White Dwarf / Main Sequence Star Binaries

      1. i.

White Dwarf Pollution

    3. (c)

Interacting Stars

    4. (d)

Resolved Stars

  6. 6.

Solar System Science

    1. (a)

Asteroid Density Measurements

    2. (b)

Solar System Volatiles

###  2.1 Science Driven Requirements

The science and instrument teams distilled the individual science cases
discussed above into a set of driving science requirements which GMagAO-X is
designed to meet. The resulting key instrument parameters are summarized in
Table 1. The top-level requirements are:

  * GMAGX-SCI-001: Exoplanet Characterization in Reflected Light

    * Requirement: GMagAO-X shall measure reflected light from planets around nearby stars such as Proxima Cen and GJ 876.

  * GMAGX-SCI-002: Spatial Resolution

    * Requirement: GMagAO-X shall achieve $\lambda/D$ spatial resolution at wavelengths equal and longer than 600 nm on unresolved guide stars.

    * Goal:  GMagAO-X shall achieve $\lambda/D$ spatial resolution on resolved guide stars, such as asteroids and resolved stars.

    * Goal:  GMagAO-X shall achieve $\lambda/D$ spatial resolution at wavelengths longer than 450 nm.

    * Stretch-Goal:  GMagAO-X shall achieve $\lambda/D$ spatial resolution at wavelengths longer than 350 nm.

  * GMAGX-SCI-003: Broadband Photometry

    * Requirement: GMagAO-X shall be capable of performing photometry in narrow-band and 10% bandwidth filter bands.

  * GMAGX-SCI-004: Low Resolution Spectroscopy

    * Requirement: GMagAO-X shall be capable of spectroscopy at a resolution between 20-100.

  * GMAGX-SCI-005: High Resolution Spectroscopy

    * Requirement: GMagAO-X shall be capable of spectroscopy at resolutions from 1000-65000.

    * Goal:  GMagAO-X shall be capable of spectroscopy at resolutions greater than $\mathcal{R}=100,000$ at least in the Oxygen-A band.

  * GMAGX-SCI-006: Wavelength Coverage

    * Requirement: GMagAO-X shall provide science wavelength coverage from 600-1900 nm to provide discrimination between various exoplanet models.

    * Goal:  Wavelength coverage of 450-1900 nm.

    * Stretch Goal:  Wavelength coverage of 350-1900 nm.

  * GMAGX-SCI-007: Magnitude Range

    * Requirement:  GMagAO-X shall be capable of observing stars in the magnitude range I = -1.5 to 13.

    * Goal:  GMagAO-X shall be capable of observing stars in the magnitude range I = -1.5 to 15.

  * GMAGX-SCI-008: Coronagraph Contrast

    * Requirement:  GMagAO-X shall be capable of performing photometry at a signal-to-noise ratio of 5 on a point source with a flux ratio of 1e-7 or better with respect to its host star in a 10% bandwidth filter at 4 $\lambda/D$.

    * Goal:  GMagAO-X shall be capable of performing SNR=5 photometry on a point source with a flux ratio of 1e-7 or better in a 10% bandwidth filter at 2 $\lambda/D$ and 1e-8 at 6 $\lambda/D$.

    * Stretch Goal:  1e-8 contrast at 1 $\lambda/D$ and 1e-9 at 5 $\lambda/D$.

  * GMAGX-SCI-009: Planet Position Measurement

    * Requirement: GMagAO-X shall be capable of measuring companion position with respect to the star with sufficient precision (TBD) to constrain inclination and the orbital elements.

  * GMAGX-SCI-010: Field of View

    * Requirement: GMagAO-X shall provide a field of view of at least 3 arcseconds x 3 arcseconds.

  * GMAGX-SCI-011: Relative Photometric Stability

    * Requirement: GMagAO-X shall be capable of 1% (TBC) relative photometric stability over a 4 hr (TBC) period in coronagraphic high-contrast observations.

The GMagAO-X Instrument Science Requirements Document (GMAGX-DOC-0001)
contains much more discussion about the rationale for each of these 11
requirements and how they flow down from the science cases. These requirements
flow down to the instrument level requirements, on which the preliminary
design of GMagAO-X is based.

##  3 PRELIMINARY DESIGN

###  3.1 MECHANICAL DESIGN

The mechanical design of GMagAO-X is shown in Figure 1. GMagAO-X will occupy a
Folded Port (FP) on the Gregorian Instrument Rotator (GIR) of the GMT. The
main structure of GMagAO-X consists of a small optical bench with fore-optics
and a fast-steering mirror, a rotating frame containing the main two-level
optical subsystem, and onboard electronics racks which primarily house DM
electronics and so must be close to the instrument. GMagAO-X also supplies its
own custom tertiary mirror (M3). This decision was made to enable a very high
quality, but smaller than facility-sized, optic to be used for M3.

To provide vibration isolation, the two-level optical table is floating (air-
damped). We will use an actively controlled, in height and level, system from
TMC called “PEPSII”, which we have demonstrated with the existing MagAO-X
instrument on the Magellan Clay telescope. See Close et al.[8] in these
proceedings.

Figure 1:  Overview of the Mechanical Design of GMagAO-X

The GIR is below the primary mirror (M1), and tips with elevation. The GIR, as
its names implies, rotates to track the sky. To maximize stability and provide
the ability to use pneumatic vibration isolation, GMagAO-X is designed to
counter rotate such that its main optics are gravity invariant (see Figure 2).
To facilitate this, when GMagAO-X is in operation the GIR will be locked in
position such that GMagAO-X will be parallel to the elevation axis of the GMT.
Figure 3 illustrates the crossed roller bearing and drive system for rotating
the optical table.

Figure 2:  Gravity Invariance: The optical table of GMagAO-X is rotated to be
gravity invariant during operation.  Figure 3:  The Rotation Mechanism of
GMagAO-X

When not in operation, GMagAO-X will not be floating and is designed to travel
through the full range of motion of the telescope. This includes a locking
mechanism to clamp the table down rigidly and ensure that it is constrained
for all possible angles of the GIR while other instruments are in operation.

The preliminary mechanical design of GMagAO-X has been thoroughly analyzed for
structural integrity, and to ensure it meets the demanding seismic
survivability standards of the GMT. All aspects of the construction, shipment,
assembly, installation and removal, alignment, and night-time operations have
been considered in the design.

###  3.2 OPTICAL DESIGN

Figure 4:  The Optical Design of GMagAO-X

Figure 4 shows a summary of the optical design of GMagAO-X. The custom M3
reflects the f/8.157 beam of the GMT into the frontend OAP Relay (green).
These fore-optics contain a fast steering mirror (FSM) to provide high-stroke
and high-speed vibration control. The beam next passes through a pupil
alignment periscope, a k-mirror de-rotator, and atmospheric dispersion
corrector (ADC). A 3000 actuator DM “woofer” is used to provide large-stroke
for correcting low-order aberrations at lower speeds. After the woofer, the
beam is relayed to the 21,000 “parallel DM” which provides the high-speed and
high-spatial-frequency control. The beam is then split between the wavefront
sensors (WFS) and the coronagraph. The wavefront sensing subsystem (not shown
in Figure 4 contains both a Holographic Dispersed Fringe Sensor (HDFS,[9, 10])
for coarse phasing control and a high-order WFS. A 3-sided pyramid WFS is
currently baselined (see Haffert et al.[11] in these proceedings).

The coronagraph beam is relayed to the upper level (purple rays in Figure 4.
The coronagraph contains a 3000 actuator non-common path correcting (NCPC) DM,
an architecture which has been demonstrated on-sky with MagAO-X[12, 13]. This
allows in-coronagraph WFS&C, such as the digging of dark holes[14, 15],
without offsetting to the high-order WFS&C system. After the NCPC DM, the
coronagraph optics provide a Lyot-style architecture. The baseline coronagraph
is the Phase-Apodized Pupil Lyot Knife-Edge Coronagraph (PAPLKEC)[16], which
uses the NCPC DM for apodization and a knife-edge mirror as the focal plane
mask. To push towards the goal and stretch-goal of GMAGX-SCI-008 the design
include Phase Induced Amplitude Apodization (PIAA[17]) optics (including
inverse PIAA) and transmissive complex focal plane masks. Both focal-plane
low-order WFS (FLOWFS) and Lyot-plane LOWFS (LLOWFS) are supported to make use
of light rejected by the coronagraphs.

After the coronagraph, focal plane instrumentation will include imagers
covering the optical through 1 $\mu$m, and near-IR through H band (see GMAGX-
SCI-006). We plan to couple GMagAO-X to the facility G-CLEF spectrograph[18]
as a fiber fed integral field unit (IFU), and possibly to other facility
spectrographs for near-IR coverage. We have also left room for an on-board IFU
which is yet to be designed.

See Close et al.[8] in these proceedings for further details about the optical
design of GMagAO-X.

####  3.2.1 The Parallel DM

The key to achieving the WFS&C required for the science goals of GMagAO-X is
the ability to deploy a high-actuator count DM. Based on the specifications
and performance of MagAO-X[12, 13], a $\sim$14 cm projected pitch is needed,
which is equivalent to $\sim$3000 illuminated actuators per segment. Figure 5
shows how this can be achieved on the GMT with the “Parallel DM” architecture.
We use a hexagonal prism with its 6 segments coated as mirrors. Placed near a
pupil plane, this splits the GMT aperture such that each segment is imaged
onto a 3K MEMS DM. The central hole passes the center segment to its DM in the
back. Crossed folds are used to control polarization effects, and folds are
actuated with piston-tip-tilt stages to provide coarse phasing control. Fine
high-speed phasing control is provided by the MEMS segments. Figure 5 shows
how the parallel DM fits on the lower table of GMagAO-X.

Figure 5:  The Parallel DM Concept: the large-segment primary mirror of GMT
motivates using a segmented DM with 3000 actuators per segment. (a) Mechanical
Design. (b) Optical Design. Note the crossed folds for polarization control.
(c) & (d) Prototype assembled and tested on the HCAT testbed at the University
of Arizona.

The parallel DM has been prototyped and demonstrated, albeit without MEMS
installed, in the High Contrast Adaptive Optics phasing Testbed (HCAT) at the
University of Arizona. See Close et al.[8] in these proceedings and Kautz et
al (submitted to JATIS).

Figure 6:  The parallel DM is shown rendered in the opto-mechanical design of
GMagAO-X.

####  3.2.2 Coronagraphs

Figure 7 documents the baseline PAPLKEC coronagraph design. The design easily
supports achieving the science goals define above. The GMT aperture presents
no significant challenges for modern coronagraph designs.

Figure 7: Preliminary PAPLKEC coronagraph design for GMagAO-X. This design is
for a 20% bandwidth. Left shows the PSF. Middle-left shows the dark hole (with
no residual atmosphere). Middle-right shows the throughput curve, with inner
working angle of 2.3 $\lambda/D$ defined at 50% throughput. Right shows raw
contrast vs. separation.

###  3.3 ELECTRICAL DESIGN

The electronics of GMagAO-X are concerned with the safe operation of the
rotation system and the air floating system, as well as the operation of the
ExAO system and coronagraph components. The main feature of the electrical
design is the layout of the equipment racks that will house the drive and
control electronics. Figure 8 illustrates the 5 racks we plan to use.

Figure 8:  The equipment racks used to house GMagAO-X electronics. Rack 1
houses the 8 MEMS driver (7x for the parallel DM tweeter, 1x for the NCPC DM
and image acquisition hardware. Rack 2 houses the 32 U ALPAO DM 3228 driver.
Rack 3 houses motion control and drivers for the parallel DM PTT stages. Racks
4 and 5 are off the telescope and house the distributed real-time control
computer system.

Two racks will be mounted on the instrument frame. These are primarily
occupied by DM driver electronics, which due to cable length restrictions must
be as close as possible to the devices. Other components with similar cable
length and/or latency restrictions are included in these racks. Note that
these racks tip with elevation, but do not rotate with the internal barrel and
so all cabling from these racks is managed by a cable guide. A third rack
below the deck of the GIR, co-rotating with azimuth, will house additional
motion control electronics and networking gear. Power distribution is
distributed amongst the racks. The entire power requirement of GMagAO-X is
well within the allotment for an FP instrument, as is the thermal performance
and cooling requirements.

At least two additional racks will be located off the telescope in an
equipment/computer room, to house the distributed real-time control computer
system. This is described further below. Sufficient allocations of optical
fiber have been reserved between the GIR and the equipment room to support
GMagAO-X’s unique needs.

Figure 9:  Management of MEMS ribbon cables. Feed through connectors will be
placed on the outside rack #1 housing the MEMS electronics. The right photo
shows such a system in operation on the current MagAO-X instrument.

An important concern with MEMS DM is cable management, as the ribbon cables
typically used to supply per-actuator voltages take up significant space. In
order to support occasional maintenance and shipping, we will provide bulkhead
connectors on the outside of the MEMS rack, as shown in Figure 9. This will
allow the rack to be disconnected from the instrument without disturbing
either the electronics or the DM segments. We have extensive experience with
this system on MagAO-X, which undergoes routine shipping and transportation on
the mountain.

###  3.4 SOFTWARE & CONTROL SYSTEM

The baseline software system for GMagAO-X is based on the proven MagAO-X
software suite. The MagAO-X software is based on a modern C++ API, with a base
class inherited by all applications which implements standard housekeeping
(startup sequence, event loop, logs, telemetry, and interprocess communication
(IPC, both routine and low-latency)). We use a binary logging framework for
efficiency. The architecture is multi-application, with multiple threads in
each application. The Instrument Neutral Distributed Interface (INDI) protocol
is used for soft-real-time IPC, and the ImageStreamIO low-latency library is
used for WFS&C image processing and hard-real-time IPC. The MagAO-X framework
is robust and fully demonstrated on-sky, as well as being used in several lab
testbeds, and in addition, significant effort is underway to prepare it for
space-flight applications. The MagAO-X PDR and pre-ship review (PSR) software
designs are available
online111https://magao-x.org/docs/handbook/_downloads/78a8f3b30b90bdcb2a4560f4c0981fca/3.3_software.pdf
222https://magao-x.org/docs/handbook/_downloads/e9e6b896c53ab5b359d890763de06e96/2_5_Software_Processes.pdf.
The API reference333https://magao-x.org/docs/api/ and source code are also
public444https://github.com/magao-x/MagAOX.

The real-time control system for GMagAO-X is based on the Compute and Control
for Adaptive Optics (CACAO[19]) system. CACAO is in routine use at SCExAO[20]
and on MagAO-X. The shared memory architecture of CACAO (MILK ImageStreamIO)
is the backbone of our image transfer and low-latency IPC. We use this for the
science cameras as well, enabling low-latency focal plane WFS (FPWFS).

In order to enable image acquisition from and control of the large number of
devices in GMagAO-X, and to enable the processing throughput needed for a
21,000 actuator AO system, we plan to implement a distributed real-time
control system. A key component of CACAO that is needed for GMagAO-X is low-
latency computer-to-computer image transfer. This is also used routinely on
SCExAO and MagAO-X for wavefront control. The computers and their task
allocation are:

  * •

WFSCC: WFS Control Computer

    * –

Low-latency camera readout (one PCIe slot per camera, dedicated signal line(s)
each (cameralink, coaxpress))

    * –

Initial calibrations (dark sub, flat field, masking) applied here before
transfer

    * –

Low-latency transfer to RTC-1

    * –

Ancillary electronics control (modulator, shutters, filters, etc.)

  * •

ICC: Instrument Control Computer

    * –

Low-latency camera readout (one PCIe slot per camera, dedicated signal line(s)
each (cameralink, coaxpress))

    * –

Low-latency transfer to RTC-2

    * –

Ancillary electronics control (shutters, filters, etc.)

  * •

RTC-1: Real-Time Computer for Main AO

    * –

Low-latency transfer from WFSCC, to DMCC

    * –

PCIe expansion (GPUs)

  * •

RTC-2: Real-Time Computer for Coronagraph AO

    * –

Low-latency transfer from ICC, to DMCC

    * –

PCIe expansion (GPUs)

  * •

DMCC: DM Control Computer

    * –

Commands to DM drivers (one PCIe per segment, dedicated fiber pair each)

    * –

Low-latency transfer from RTC-1 & RTC-2

We make use of PCIe expansion to provide the data acquisition, control output,
and computing capacity needed. Figure 10 illustrates the layout of the
GMagAO-X control system.

Figure 10:  The distributed real-time control system of GMagAO-X

##  4 PERFORMANCE & Exoplanet Yield

We used dynamical models of the GMagAO-X instrument to assess the performance
of the preliminary design. Figure 11 shows the tip/tilt & vibration control
performance of GMagAO-X. This includes the atmosphere, the results of the
structural modeling done by the GMT, contributions from GMagAO-X itself
(dominated by M3), and the results of closed-loop control. The FSM (a PI S-331
is the baseline) has sufficient stroke to correct the worst case tip/tilt, and
the resulting $0.03\lambda/D$ rms WFE residual meets the instrument level
requirement set by coronagraph performance.

Figure 11: Left: GMagAO-X tip/tilt stroke requirements. Right: GMagAO-X
tip/tilt suppression on an 8th mag star

The same analysis was conducted for segment piston. Figure 12 shows the
segment piston control performance of GMagAO-X. This includes the atmosphere,
the results of the structural modeling done by the GMT, and the results of
closed-loop control. The resultant 3.5 nm rms WFE residual meets the
instrument level requirement set by coronagraph performance.

Figure 12: Left: GMagAO-X segment piston stroke requirements. Right: GMagAO-X
segment piston suppression on an 8th mag star

A dynamical model of the closed-loop performance was used to predict the
performance of the high-order AO system. Figure 13 shows the Strehl ratio vs.
guide-star magnitude. Figure 14 shows the residual instantaneous contrast due
to residual atmospheric speckles, as well as the statistical speckle
lifetimes.

Figure 13: Strehl ratio vs. guide star magnitude for GMagAO-X, for different
combinations of WFS and science wavelengths.

Figure 14: Left: GMagAO-X residual atmospheric instantaneous contrast. Right:
GMagAO-X residual atmosphere speckle lifetime. Blue is for the simple
integrator controller, and red is for the ideal linear predictor controller.

The results of the performance analysis were used to assess the reflected
light imaging exoplanet yield. We base this analysis on the known exoplanets
rather than a projected population model. We emphasize this point: the planets
under consideration are known to exist. We assessed each planet based on its
guide-star magnitude and the resultant predicted GMagAO-X performance. For
each planet we estimate its radius based on the RV minimum mass using a mass-
to-radius relationship calibrated from measured exoplanets. Geometric albedos
were based on a suite of models including Earthshine, Venus, and published
models for EGPs. The results are shown in Figure 15. GMagAO-X will be capable
of characterizing the atmospheres of up to over 200 currently known
exoplanets, for which we currently know only a minimum mass.

Figure 15: Characteristics of the currently known exoplanets which GMagAO-X
will characterize. Left: Planet:Star flux ratio vs separation. Middle: Planet
radius vs. luminosity normalized semi-major axis. Right: Guide-star magnitude
vs spectral type for the planet host stars.

##  5 CONCLUSION

GMagAO-X is the ExAO-fed coronagraph planned for first-light of the GMT. It
represents the earliest opportunity of the ELT era to begin search nearby
terrestrial planets for life. GMagAO-X has passed preliminary design review,
and is preparing to begin the final design phase. The scientific potential of
GMagAO-X highlights how crucial the US-ELT program is to the future of U.S.
Astronomy and motivates a positive decision to move forward with construction
of the GMT as rapidly as possible.

###### Acknowledgements.

The GMagAO-X conceptual and preliminary design would not have been possible
without the support of the University of Arizona Space Institute. We are also
grateful for the support of an anonymous donor to Steward Observatory.

## References

  * [1] National Academies of Sciences, E. and Medicine, [Pathways to Discovery in Astronomy and Astrophysics for the 2020s ] (2021).
  * [2] Oppenheimer, B. R. and Hinkley, S., “High-Contrast Observations in Optical and Infrared Astronomy,” ARA&A 47, 253–289 (Sept. 2009).
  * [3] Males, J. R., Fitzgerald, M. P., Belikov, R., and Guyon, O., “The Mysterious Lives of Speckles. I. Residual Atmospheric Speckle Lifetimes in Ground-based Coronagraphs,” PASP 133, 104504 (Oct. 2021).
  * [4] Close, L. M., Males, J. R., Follette, K. B., Hinz, P., Morzinski, K., Wu, Y.-L., Kopon, D., Riccardi, A., Esposito, S., Puglisi, A., Pinna, E., Xompero, M., Briguglio, R., and Quiros-Pacheco, F., “Into the blue: AO science with MagAO in the visible,” in [Adaptive Optics Systems IV ], Marchetti, E., Close, L. M., and Vran, J.-P., eds., Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series 9148, 91481M (Aug. 2014).
  * [5] Guyon, O., “Extreme Adaptive Optics,” 56, 315–355 (Sept. 2018).
  * [6] Guyon, O., Martinache, F., Cady, E. J., Belikov, R., Balasubramanian, K., Wilson, D., Clergeon, C. S., and Mateen, M., “How ELTs will acquire the first spectra of rocky habitable planets,” in [Adaptive Optics Systems III ], Ellerbroek, B. L., Marchetti, E., and Véran, J.-P., eds., Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series 8447, 84471X (July 2012).
  * [7] Males, J. R., Close, L. M., Guyon, O., Morzinski, K., Puglisi, A., Hinz, P., Follette, K. B., Monnier, J. D., Tolls, V., Rodigas, T. J., Weinberger, A., Boss, A., Kopon, D., Wu, Y.-l., Esposito, S., Riccardi, A., Xompero, M., Briguglio, R., and Pinna, E., “Direct imaging of exoplanets in the habitable zone with adaptive optics,” in [Adaptive Optics Systems IV ], Marchetti, E., Close, L. M., and Vran, J.-P., eds., Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series 9148, 914820 (July 2014).
  * [8] Close, L. et al., “High-contrast imaging at first-light of the GMT: The PDR optical and mechanical design for the GMagAO-X ExAO system and results from the HCAT testbed with an HDFS phased parallel DM prototype,” Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series 13097 (2024).
  * [9] Haffert, S. Y., Close, L. M., Hedglen, A. D., Males, J. R., Kautz, M., Bouchez, A. H., Demers, R., Quirós-Pacheco, F., Sitarski, B. N., Gorkom, K. V., Long, J. D., Guyon, O., Schatz, L., Miller, K., Lumbres, J., Rodack, A., and Knight, J. M., “Phasing the Giant Magellan Telescope with the holographic dispersed fringe sensor,” Journal of Astronomical Telescopes, Instruments, and Systems 8(2), 1 – 21 (2022).
  * [10] Hedglen, A. D., Close, L. M., Haffert, S. Y., Males, J. R., Kautz, M. Y., Bouchez, A. H., Demers, R. T., Quirós-Pacheco, F., Sitarski, B. N., Guyon, O., Gorkom, K. V., Long, J. D., Lumbres, J. R., Schatz, L. H., Miller, K. L., Rodack, A. T., and Knight, J. M., “Lab tests of segment/petal phasing with a pyramid wavefront sensor and a holographic dispersed fringe sensor in turbulence with the Giant Magellan Telescope high contrast adaptive optics phasing testbed,” Journal of Astronomical Telescopes, Instruments, and Systems 8(2), 1 – 26 (2022).
  * [11] Haffert, S. et al., “High-contrast imaging at first-light of the GMT: the wavefront sensing and control (WFS&C) architecture of GMagAO-X,” Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series 13097 (2024).
  * [12] Males, J. R., Close, L. M., Haffert, S., Long, J. D., Hedglen, A. D., Pearce, L., Weinberger, A. J., Guyon, O., Knight, J. M., McLeod, A., Kautz, M., Van Gorkom, K., Lumbres, J., Schatz, L., Rodack, A., Gasho, V., Kueny, J., and Foster, W., “MagAO-X: current status and plans for Phase II,” in [Adaptive Optics Systems VIII ], Schreiber, L., Schmidt, D., and Vernet, E., eds., Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series 12185, 1218509 (Aug. 2022).
  * [13] Males, J. et al., “MagAO-X: commissioning results and status of ongoing upgrades,” Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series 13097 (2024).
  * [14] Haffert, S. Y., Males, J. R., Ahn, K., Van Gorkom, K., Guyon, O., Close, L. M., Long, J. D., Hedglen, A. D., Schatz, L., Kautz, M., Lumbres, J., Rodack, A., Knight, J. M., and Miller, K., “Implicit electric field conjugation: Data-driven focal plane control,” A&A 673, A28 (May 2023).
  * [15] Haffert, S. et al., “On-sky demonstration of high-order focal plane wavefront control with a second stage AO system on MagAO-X,” Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series 13097 (2024).
  * [16] Por, E. H., “Phase-apodized-pupil Lyot Coronagraphs for Arbitrary Telescope Pupils,” ApJ 888, 127 (Jan. 2020).
  * [17] Guyon, O., “Phase-induced amplitude apodization of telescope pupils for extrasolar terrestrial planet imaging,” A&A 404, 379–387 (June 2003).
  * [18] Szentgyorgyi, A. et al., “Innovations in the design and construction of the GMT-Consortium Large Earth Finder (G-CLEF), a first-light instrument for the Giant Magellan Telescope (GMT),” Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series 13096 (2024).
  * [19] Guyon, O., Sevin, A., Gratadour, D., Bernard, J., Ltaief, H., Sukkari, D., Cetre, S., Skaf, N., Lozi, J., Martinache, F., Clergeon, C., Norris, B., Wong, A., and Males, J., “The compute and control for adaptive optics (CACAO) real-time control software package,” in [Adaptive Optics Systems VI ], Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series 10703, 107031E (July 2018).
  * [20] Jovanovic, N., Martinache, F., Guyon, O., Clergeon, C., Singh, G., Kudo, T., Garrel, V., Newman, K., Doughty, D., Lozi, J., Males, J., Minowa, Y., Hayano, Y., Takato, N., Morino, J., Kuhn, J., Serabyn, E., Norris, B., Tuthill, P., Schworer, G., Stewart, P., Close, L., Huby, E., Perrin, G., Lacour, S., Gauchet, L., Vievard, S., Murakami, N., Oshiyama, F., Baba, N., Matsuo, T., Nishikawa, J., Tamura, M., Lai, O., Marchis, F., Duchene, G., Kotani, T., and Woillez, J., “The Subaru Coronagraphic Extreme Adaptive Optics System: Enabling High-Contrast Imaging on Solar-System Scales,” PASP 127, 890 (Sept. 2015).