# Zero-shot Generative Linguistic Steganography

Ke Lin1 Yiyang Luo2 Zijian Zhang1 Ping Luo1
1Tsinghua University
2Nanyang Technological University
{leonard.keilin<EMAIL_ADDRESS>
<EMAIL_ADDRESS>
<EMAIL_ADDRESS>
###### Abstract

Generative linguistic steganography attempts to hide secret messages into
covertext. Previous studies have generally focused on the statistical
differences between the covertext and stegotext, however, ill-formed stegotext
can readily be identified by humans. In this paper, we propose a novel zero-
shot approach based on in-context learning for linguistic steganography to
achieve better perceptual and statistical imperceptibility. We also design
several new metrics and reproducible language evaluations to measure the
imperceptibility of the stegotext. Our experimental results indicate that our
method produces $1.926\times$ more innocent and intelligible stegotext than
any other method111The project is available in
https://github.com/leonardodalinky/zero-shot-GLS..

Zero-shot Generative Linguistic Steganography

Ke Lin1 Yiyang Luo2 Zijian Zhang1 Ping Luo1 1Tsinghua University 2Nanyang
Technological University {leonard.keilin<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>

##  1 Introduction

Data transmission is generally secured using encryption algorithms to create
an unrecognizable ciphertext for secure data transmission Bellare et al.
(1997). Nonetheless, messages transmitted as ciphertexts may easily arouse
suspicion from attackers, and eavesdropping may place a threat on the messages
being sent. Once the communication channel is discovered, the attackers could
alter the messages in any way, causing significant damage.

Steganography is the key to ensuring communication privacy by concealing
sensitive messages within monitored channels. General steganography involves
embedding a secret message into a public multimedia carrier, such as text
Krishnan et al. (2017), image Subramanian et al. (2021); Tao et al. (2019);
Guo et al. (2015), audio Lin et al. (2018), and video Liu et al. (2019) to
create a stego-carrier that is then transmitted via the public channel. As
shown in Fig. 1, the problem of secure and covert communication can be viewed
as a message transmission between two parties, Alice and Bob, along with a
malicious party, Eve. Eve will monitor the entire transmission and prevent
unintelligible messages from being transmitted as Alice attempts to send
secret information to Bob. The solution to this problem is to use a
steganographic function to embed the secret message into a stego-carrier
controlled by the shared secret key. The stego-carrier is then sent to Bob
without alarming Eve, and Bob can extract the secret message from the stego-
carrier by using the corresponding extraction function. A key objective in
this scenario is to ensure that covert communications are imperceptible.

In public channels, text is commonly used, however, text contains relatively
little redundant information Zipf (1999) when compared to other media, so
embedding large amounts of information can be challenging. Moreover,
steganalysis tools can detect stegotext containing secret messages based on
machine learning or statistical characteristics Taleby Ahvanooey et al.
(2019). To address this problem, modern steganography utilizes machine
learning techniques to enhance its imperceptibility.

Figure 1: Generative linguistic steganography pipeline.

Over the last few decades, researchers have devised many techniques to realize
generation-based steganography Xiang et al. (2022), including rule-based
Chapman and Davida (1997), statistical Moraldo (2014); Shniperov and Nikitina
(2016), and combinations of these methods Luo et al. (2016). Recent
developments Otter et al. (2021) in machine learning and natural language
processing have led to improved performance in linguistic steganography. It
involves the use of neural networks to learn the statistical distribution of a
large number of sentences, and then generate steganographic sentences from the
learned language model (Dai and Cai, 2019; Yang et al., 2019b).

Prior works have typically formulated statistical imperceptibility as the
distance between the covertext distribution and stegotext distribution,
however, there is no clear and simple method of measuring perceptual
imperceptibility. Furthermore, many studies Ziegler et al. (2019); Yang et al.
(2021) have demonstrated that metrics for assessing statistical
imperceptibility fail to measure perceptual imperceptibility and produce
inaccurate or even incorrect results. The discrepancy between the statistical
and perceptual imperceptibility prevents the practical application of
linguistic steganography.

#### Our Contributions.

In this work, we propose a novel zero-shot linguistic steganography method
with both improved perceptual and statistical imperceptibility. Our new method
is built based on the in-context learning of large language models (LLM),
which utilize some samples of the covertext to generate more intelligible
stegotext using the question-answer paradigm (QA). Initially, the secret text
is encoded in binary bitstreams, which are then used to generate the
stegotext. At each timestep $t$, the LM computes the distribution of the
$t$-th token and embeds the bitstream into the generated text by Huffman
encoding. To demonstrate the efficacy of our proposed method compared with
different supervised and training-free approaches, we develop several new
metrics and conduct reproducible experiments of language evaluation.

In summary, our contributions are as follows:

  * •

We present a zero-shot approach for linguistic steganography based on in-
context learning using samples of the covertext.

  * •

We improve both the binary coding process and the embedding process by
introducing several novel techniques.

  * •

We design several metrics and language evaluations to evaluate both the
perceptual and statistical imperceptibility, whereas our method produces more
innocent and intelligible stegotext compared to all the previous methods.

##  2 Background

###  2.1 Generative Linguistic Steganography

General text generation task aims at creating token sequence
$\bm{x}=[x_{1},\dots,x_{n}]$ from the joint probability of language model
$\mathbf{P}_{\text{LM}}(\bm{x})$. The generation process can often be viewed
as an auto-regression:

$\displaystyle\mathbf{P}_{\text{LM}}(\bm{x})$
$\displaystyle=\prod_{t=1}^{n}\mathbf{P}_{\text{LM}}(x_{t}\mid
x_{1},\dots,x_{t-1})$ (1)
$\displaystyle=\prod_{t=1}^{n}\mathbf{P}_{\text{LM}}(x_{t}\mid x_{<t})$

The goal of linguistic steganography is to send a secret message
$\bm{m}\in\\{0,1\\}^{l}\subset\mathcal{M}$ to the receiver through a monitored
public channel. In generative linguistic steganography, the sender and
receiver share an embedding algorithm
$f_{\text{emb}}:\mathcal{P}_{\text{LM}}\times\mathcal{M}\mapsto\mathcal{P}_{\text{stega}}$,
which transforms $\mathbf{P}_{\text{LM}}$ into the steganographic distribution
$\mathbf{P}_{\text{stega}}$ controlled by message $\bm{m}$. It is then used to
achieve covert communication by generating _stegotext_
$\bm{y}\sim\mathbf{P}_{\text{stega}}$. Similarly, there exists an extraction
process $f_{ext}:\mathcal{P}_{\text{stega}}\mapsto\mathcal{M}$ that extracts
the message $\bm{m}$ from the stegotext $\bm{y}$.

###  2.2 Statistical Imperceptibility

To measure the statistical imperceptibility under surveillance by an
eavesdropper, Cachin (1998) proposed to use the Kullback-Leibler divergence
(KLD) between the distributions of natural text $\mathbf{P}_{\text{true}}$ and
stegotext $\mathbf{P}_{\text{stega}}$. Since we approximate the distribution
of true text $\mathbf{P}_{\text{true}}$ using a language model
$\mathbf{P}_{\text{LM}}$, there is a gap between the distribution predicted by
the model and the true distribution of text. Hence, we need to approximate
this gap by finding an upper bound. We can formulate the upper bound of
imperceptibility by using Jensen–Shannon divergence (JSD) instead:

$\mathrm{JSD}(\mathbf{P}_{\text{true}}\;\|\;\mathbf{P}_{\text{stega}})^{\frac{1}{2}}\leq\mathrm{JSD}(\mathbf{P}_{\text{true}}\;\|\;\mathbf{P}_{\text{LM}})^{\frac{1}{2}}\\\
+\mathrm{JSD}(\mathbf{P}_{\text{LM}}\;\|\;\mathbf{P}_{\text{stega}})^{\frac{1}{2}}$
(2)

Assuming that the language model is successful in modeling the covertext
distribution, we can simplify the statistical imperceptibility as
$\mathrm{JSD}(\mathbf{P}_{\text{LM}}\;\|\;\mathbf{P}_{\text{stega}})$. This
strong assumption can be considered true due to language models’ performance.
In other words, it is feasible to measure imperceptibility using only a
language model $\mathbf{P}_{\text{LM}}$ and a steganographic model
$\mathbf{P}_{\text{stega}}$.

##  3 Methodology

In this section, we present the overall framework of our approach, consisting
of three modules:  (a) _Codec_ module to (de)compress secret text into (from)
bitstreams,  (b) _Embedding_ module to select candidate words from the
distribution of stegotext,  (c) _In-Context Stegotext Generation_ module to
approximate the joint distribution of covertext.  The pipeline of our
framework is illustrated in Fig. 1. Figure 3 also explains the details of the
embedding module and the in-context stegotext generation.

###  3.1 Codec

The sender in linguistic steganography wishes to send secret text to the
receiver, which must first be converted into a binary bitstream. We refer to
this process as encoding and decoding (codec).

#### Variable-length Coding.

There are two categories of encoding algorithms: fixed-length coding (FLC) and
variable-length coding (VLC). We employ Huffman coding of the VLC family for
token-level encoding, as it is a relatively simple algorithm and has a sub-
optimal compression ratio. Tokens are represented in binary form using the
Huffman code table conditioned on $\mathbf{P}_{\text{LM}}(x_{t}\mid x_{<t})$.

Figure 2: Example of EF coding. The red and green lines represent that the
coding in bitstream changed (from 0 to 1, or from 1 to 0) and therefore will
be assigned as 1 for the next iteration. The blue and yellow line represents
that the coding in bitstream did not change (from 0 to 0, or from 1 to 1) and
therefore will be assigned as 0 for the next iteration. Bitstream is processed
through several rounds until the fewest 1s exist.

#### Edge Flipping Coding.

We observe that more frequent occurrences of sequential zeros in the
bitstreams can improve the quality of the generated text in the embedding
module due to the local ordering of the Huffman tree (i.e., the left branch
always has a higher probability than the right branch). Hence, we introduce
Edge Flipping coding (EF), inspired by the differential encoding of
communication Weber (1978), on top of the Huffman process to achieve better
text generation performance.

In EF coding, the bitstream is transformed to record the position at which the
bits change based on the edge-triggered flip-flops in electronics. We apply
multi-round EF coding (e.g. 16 rounds) and the bitstream with the least number
of 1s is retained. A simple example of EF is provided in Fig. 2. The analysis
of EF coding is presented in Section 4.6.

###  3.2 Embedding

The goal of the embedding module is to hide and extract bitstreams within
regular sentences.

#### Hide & Extract.

During each phase of the generative operation, the LM yields a set of
_candidate words_. These words are determined based on the subsequent token’s
probability and can potentially serve as alternatives for the next possible
word to be generated. Then the LM will determine the length of the binary
bitstream that can be embedded based on the number of candidate words.

Algorithm 1 shows the brief algorithm of the hiding process. The binary
representation of selected words in the conditional word distribution is
obtained using the Huffman encoding algorithm. It should be noted that this
Huffman process is independent of the aforementioned Huffman coding in the
Codec section. Following Dai and Cai (2019), we use the threshold $\tau$ to
reduce perplexity by pruning words with lower probabilities.

Algorithm 1 Information Hiding Algorithm

1:Bitstream $\bm{m}$, threshold $\tau$.

2:stegotext $\bm{y}=[y_{1},\dots,y_{n}]$.

3:Timestep $t\leftarrow 1$, output sentence $\bm{y}\leftarrow\emptyset$

4:while not the end of $\bm{m}$ do

5: $\triangleright$ Compute conditional probs

6: $\bm{p}\leftarrow\mathbf{P}_{\text{stega}}(x_{t}\mid x_{<t})$

7: $\triangleright$ Prune candidate words

8: $\bm{c}\leftarrow[c_{i}\mid\bm{p}(c_{i})\geq\tau]$

9: $\triangleright$ Huffman encoding

10: $H\leftarrow\mathrm{Huffman}(\bm{c},\bm{p})$

11: $\triangleright$ Select candidate

12: $y_{t}\leftarrow$ Word $c\in H$ whose binary representation matches the
prefix of $\bm{m}$

13: $\bm{y}\leftarrow\bm{y}\cup\\{y_{t}\\},t\leftarrow t+1$

14:end while

Figure 3:  A running example of the embedding module and in-context stegotext
generation. The selected context of samples from covertext in  gray is used to
instruct the stegotext generation. Huffman trees are constructed based on the
conditional distribution, with the word matching the prefix of a bistream
being selected as the next token. For example, the red token has 5 candidate
words after applying probability pruning. Huffman tree is constructed for the
5 candidates and the word "off" matches the prefix of the bitstream and is
chosen as the next word.

As the reverse process of hiding, the information extraction process extracts
steganographic information from stegotext. With a few minor differences, the
extraction of secret bitstream is almost identical to the hiding process. The
extraction algorithm is left out in Appendix A.2 for simplicity.

#### Annealing Selection.

Candidate word selection with specific thresholds $\tau$ may result in trivial
candidates in some cases (e.g., $\lvert\bm{c}\rvert=1$). We note that trivial
candidates often lead to a chain of trivial cases in subsequent stegotext
generation, reducing the embedding rate and bit per word (BPW). Inspired by
the stimulated annealing algorithm van Laarhoven and Aarts (1987), we propose
the annealing search to prevent trivial candidates:

$\mathbf{P}^{\prime}_{\text{stega}}(x_{t}\mid
x_{<t})=\mathrm{Softmax}(\frac{\operatorname{LM}(x_{t}\mid x_{<t})}{T_{t}})$
(3)

Here, the temperature $T_{t}$ at timestep $t$ varies according to the previous
candidates $\bm{c}_{t-1}$, the initial temperature $T_{0}$, and the annealing
factor $\alpha$:

$T_{t}\leftarrow\begin{cases}\alpha\cdot T_{t-1}&\lvert\bm{c}_{t-1}\rvert=1\\\
T_{0}&\lvert\bm{c}_{t-1}\rvert>1\end{cases}$ (4)

#### Repeat Penalty.

Another corner case is that several generated words may repeat themselves
after a few turns, but escape the previous annealing cases. The recently
generated words are penalized for short-term repetition by applying a moving
penalty:

$\mathbf{P}^{\prime\prime}_{\text{stega}}(x_{t}\mid
x_{<t})=\mathrm{Softmax}(\operatorname{LM}(\cdot)-\bm{\delta}_{t})$ (5)

Penalties are based on the decay factor $\beta$, initial penalty $\delta_{0}$,
and the previously selected word. For any word $c\in\bm{c}_{t-1}$,
$\bm{\delta}_{t}(c)$ has the following value:

$\bm{\delta}_{t}(c)\leftarrow\begin{cases}\max\\{0,\bm{\delta}_{t-1}(c)-\beta\\}&c\text{
not selected}\\\ \delta_{0}&c\text{ selected}\\\ \end{cases}$ (6)

###  3.3 In-Context Stegotext Generation

The core of the embedding module is to calculate the conditional probability
$\mathbf{P}_{\text{stega}}(x_{t}\mid x_{<t})$. We use the question-answer
paradigm with pre-defined prompting context $C$ using the LLaMA2-Chat-7B
Touvron et al. (2023) as the stegotext generator. In other words, the
distribution of stegotext can be represented as
$\mathbf{P}_{\text{stega}}(x_{t}\mid x_{<t},C)$, which is conditioned on both
the previously generated tokens and the context from the covertext.

#### Context Selection.

To approximate the distribution of covertext, we propose to incorporate
several random samples from the covertext dataset into the prompting context.
Since the covertext dataset is mostly available to both the sender and
receiver, only a little information (e.g., random state for selection) is
required to be pre-shared across different parties to reconstruct the
prompting context. Therefore, an eavesdropper who suspects stegotext will be
unable to decrypt the message.

#### In-Context Generation.

As LLM is capable of comprehending the semantics of context due to its
generalization ability, we utilize in-context QA tasks as a way to instruct
the model to mimic the style and semantics of the covertext. To be more
specific, we design the following structured QA format: <ctx1,…,ctxk,[MISS]>,
where the QA model is instructed to complete the missing sentence [MISS] which
is similar to the context. The number of contextual sentences $k$ is set as a
hyper-parameter during inference. The details of the in-context QA task can be
found in Appendix A.3.

##  4 Experiments

Methods | Training-free | IMDB | Twitter
---|---|---|---
BPW | PPL | JSDfull | JSDhalf | JSDzero | BPW | PPL | JSDfull | JSDhalf | JSDzero
RNN-Stega (LSTM) |  | 1.978 | 10.23 | 30.33 | 33.12 | 38.27 | 2.556 | 13.04 | 39.92 | 38.97 | 48.10
2.682 | 12.80 | 26.76 | 29.36 | 34.87 | 3.359 | 15.38 | 36.20 | 35.53 | 44.76
3.351 | 17.02 | 22.66 | 25.72 | 30.28 | 4.139 | 19.78 | 32.17 | 31.75 | 39.19
VAE-Stega (BERT-LSTM) |  | 1.972 | 9.68 | 34.50 | 36.47 | 38.53 | 2.247 | 10.06 | 46.07 | 45.82 | 46.61
2.601 | 12.38 | 31.31 | 33.02 | 34.56 | 2.861 | 12.39 | 43.89 | 44.02 | 43.64
3.199 | 16.31 | 30.03 | 31.49 | 32.82 | 3.438 | 16.13 | 42.12 | 42.54 | 40.87
ADG |  | 4.931 | 56.22 | 18.24 | 21.19 | 22.86 | 5.702 | 63.86 | 25.92 | 25.35 | 27.68
NLS | ✔ | 1.889 | 10.40 | 23.63 | 22.83 | 17.91 | 2.059 | 10.95 | 37.71 | 36.17 | 29.34
2.531 | 12.90 | 22.08 | 21.28 | 16.73 | 2.806 | 14.01 | 36.61 | 35.17 | 29.45
3.140 | 16.70 | 20.37 | 19.63 | 14.44 | 3.513 | 18.68 | 34.42 | 32.90 | 30.18
SAAC | ✔ | 4.471 | 28.74 | 18.28 | 16.40 | 13.17 | 5.078 | 36.74 | 33.75 | 32.11 | 23.08
4.749 | 37.89 | 18.04 | 16.06 | 11.49 | 5.299 | 43.35 | 33.42 | 31.82 | 22.33
5.111 | 44.02 | 17.87 | 15.98 | 11.44 | 5.716 | 54.35 | 33.20 | 31.68 | 22.04
ours | ✔ | 1.906 | 8.81 | 17.90 | 16.86 | 13.40 | 2.550 | 9.48 | 30.90 | 29.34 | 24.90
2.420 | 13.70 | 18.37 | 17.37 | 13.67 | 3.265 | 14.44 | 30.99 | 29.45 | 25.32
3.376 | 45.04 | 18.61 | 17.87 | 13.91 | 4.029 | 47.37 | 31.74 | 30.18 | 25.40

Table 1:  Experimental results under different BPWs and datasets. JSDs are
multiplied by $10^{2}$. With the _same_ BPW, lower JSD $\downarrow$ indicates
better performance. There is no data available for smaller BPW in ADG or SAAC
due to their design of high embedding rates. Our proposed method shows a
reduced Psic Effect in comparison to prior methods as our JSDs rise with
increasing BPW, whereas other methods see a decrease in JSDs.

###  4.1 Experiment Setup

#### Datasets.

Our experiments evaluate the prior works and our method using two widely used
large-scale datasets, the IMDB dataset Maas et al. (2011) and the Twitter
dataset Go et al. (2009). The texts in each dataset are broken down into
sentences by the nltk toolkit. Statistics are shown in Table 2.

Dataset | # sentences | Avg. Words | Avg. Bits
---|---|---|---
IMDB | 494,716 | 26.70 | 146.43
Twitter | 2,063,307 | 12.31 | 114.85

Table 2: Statistics of the preprocessed datasets.

#### Baselines.

Since no training is needed for our zero-shot approach, we compare the
performance of our method with several _training-free_ methods:  (a)
NLSZiegler et al. (2019): This method embeds a secret bitstream using GPT-2
Radford et al. (2019) for sentence complement.  (b) SAACShen et al. (2020):
This method improves the embedding process using Self-Adjusting Arithmetic
Coding.

Furthermore, we also compare our benchmarks against some _fully-supervised_
steganographic techniques:  (a) RNN-StegaYang et al. (2019a): This method
utilizes Recurrent Neural Network models trained on covertext to generate
stegotext.  (b) VAE-StegaYang et al. (2021): This method adopts a Variational
Auto-encoder to generate stegotext.  (c) ADGZhang et al. (2021): This method
proposes the Adaptive Dynamic Grouping techniques for better imperceptibility.

#### Implementation Details.

All of the models and benchmarks are implemented with PyTorch. Pre-trained
models (e.g., GPT-2 and LLaMA2-Chat) are obtained using Huggingface library.
To ensure a fair comparison, we rebuild all baseline methods using the same
datasets throughout the entire experiment. Bits per word (BPW) are used to
measure the capability of hiding information within stegotext. Only
steganographic methods with similar BPW should be compared. Additionally, we
adopt a unified BPW by the nltk toolkit to eliminate vocabulary
inconsistencies. In each setting, 8K stegotext is generated for evaluation. We
adopt the LLaMA2-Chat-7B as the QA model and set the context size of our
method to $k=2$, the threshold $\tau=0.005$, the factors
$\alpha=1.25,\beta=0.5$, and the initial values $T_{0}=1.0,\delta_{0}=4.0$.
More details of baselines can be found in Appendix A.4.

###  4.2 Evaluation Metrics

#### Perplexity.

As a measure of the generation quality, the perplexity (PPL) is commonly used:

$\mathrm{PPL}=2^{-\frac{1}{n}\sum_{i=1}^{n}\log_{2}{\mathbf{P}(x_{i}\mid
x_{<i})}}$ (7)

#### JSD.

Following Section 2.2, we measure the statistical imperceptibility of the
covertext and stegotext using JSD. Specificly, we finetune two GPT-2 models on
covertext $\mathcal{C}_{\text{cover}}$ and stegotext
$\mathcal{C}_{\text{stega}}$, respectively, representing
$\mathbf{P}_{\text{LM}}$ and $\mathbf{P}_{\text{stega}}$. An evaluation of the
JSD is then conducted using randomly sampled text
$\mathcal{C}_{\text{sample}}$. Three variants of JSD with respect to
$\mathbf{P}_{\text{LM}}$ and $\mathcal{C}_{\text{sample}}$ are designed:

  * •

_Full_ JSD: Stego-models are trained on $\mathcal{C}_{\text{cover}}$ if
needed222Only supervised methods need training on covertext..
$\mathcal{C}_{\text{sample}}$ are sampled from $\mathcal{C}_{\text{cover}}$.
Full JSD measures the statistical imperceptibility between covertext and
stegotext.

  * •

_Half_ JSD: Covertext dataset is split into two equal parts,
$\mathcal{C}_{\text{cover}}^{1}$ and $\mathcal{C}_{\text{cover}}^{2}$. While
the stego-models are trained on $\mathcal{C}_{\text{cover}}^{1}$, the
evaluation is performed on $\mathcal{C}_{\text{cover}}^{2}$. It estimates
imperceptibility under different but similar distributions.

  * •

_Zero_ JSD: Zero JSD is similar to Full JSD except that it uses a vanilla
GPT-2 model as the $\mathbf{P}_{\text{LM}}$, without fine-tuning on the
covertext. This evaluates the imperceptibility between the stegotext and the
normal text.

#### However, PPL and JSDs are not good metrics for assessing different
stegosystems.

While PPL and JSD comparisons are suitable for supervised text generation
tasks like machine translation, where a direct comparison with a golden text
is possible, they face challenges in general text generation due to the
absence of such direct relationships between generated output and training
data. In steganography, where language distributions may vary significantly
across methods, comparing PPLs and JSDs becomes impractical. Additionally,
variated settings of language models, such as datasets, hyperparameters, and
model architecture, will further complicate such comparisons, even if these
models are within the same domain or task.

Though PPL and JSD _cannot_ be used to compare generation ability among
different methods, it still provides an indication of performance between
different settings of the same method.

###  4.3 Experimental Results

Table 1 summarizes the main results on the IMDB and Twitter datasets. Some
methods are designed in such a way that it is inapplicable to adjust the BPW
during sentence generation, especially for a small BPW, resulting in a lack of
experiments with certain settings. This also poses a challenge for comparing
metrics between similar BPWs.

#### Psic Effect

An interesting finding in our results is the Perceptual-Statistical
Imperceptibility Conflict Effect (Psic Effect) mentioned by Yang et al.
(2021), a phenomenon in which an increase in BPW and PPL is associated with a
decrease in JSD. As shown in Table 1, Figure 4, and Table 3, while techniques
like NLS exhibit superior PPL and JSD scores, they produce lower-quality text
and are more susceptible to detection by attackers. The existence of this
unusual phenomenon implies that it will be more difficult for machines to
detect stegotext when the text becomes more _chaotic_ and _unintelligible_.
This indicates the inconsistency of imperceptibility as perceived by humans
and machines, as examples shown in Appendix C. We believe that previous
methods may have fallen into some local optimum of covertext distribution, but
do not achieve a balance between the quality and distribution of the
stegotext.

In conclusion, the state-of-the-art metrics do not match our method in every
case, but our stegotext is much more natural and does not suffer from the Psic
Effect, as is evidenced in the case study (Fig. 4) and Section 4.5.

Furthermore, experiments with three variants of JSD have shown that our zero-
shot method can be used to fool experts with specialized backgrounds and
ordinary people without prior knowledge. The generalization ability of our
methods ensures better performance than supervised methods in both few-shot
and large-data scenarios.

Secret Bitstream (Base64): cli2EARMSajA [0.5ex]1pt1pt Fully-supervised RNN-
Stega: Richards are also in the film and the film shows us the most disgusting
and violent aspects of the movie.The plot revolves around a group of people
who are attacked and murdered. VAE-Stega: listen. the music and the music, the
music, music of all the actors, the music, and the music! this movie is not
worth your eyes. ADG: Richian is condemned to discover the facts that
blackmails people through them. [0.5ex]1pt1pt Training-free NLS: There was so
much I wanted to see in this film. I was hoping it was a little more like what
I was hoping for from the original film and not what I do not expect. SAAC: I
was looking forward to going as Keanu. I really wanted to do a little red
carpet before I die. [0.5ex]1pt1pt Ours: I was genuinely underwhelmed by the
film, and I’m afraid the only thing that stood out to me was the final scene.
I had been eagerly anticipating it, but unfortunately the rest of the movie
fell flat for me. Figure 4:  Examples of stegotext generated by various
methods about movie reviews with a similar $\text{BPW}\approx 2.5$, except ADG
and SAAC. Compared with other baseline methods, our method generates more
reasonable sentences.

###  4.4 Steganalysis

Methods | Syntactic | Semantic
---|---|---
TS-BiRNN | R-BiLSTM-C | BERT-C
Fully-supervised
RNN-Stega | 94.02 | 93.88 | 96.50
VAE-Stega | 94.75 | 95.65 | 96.17
Training-free
NLS | 84.60 | 86.21 | 92.25
ours | 80.29 | 84.34 | 89.61

Table 3:  Steganalysis with $\text{BPW}\approx 2.5$. Each value is the
classification accuracy by steganalyzer. Lower accuracy $\downarrow$ indicates
better statistical imperceptibility.

To assess the statistical imperceptibility of stego-methods, a classifier is
usually deployed as a steganalyzer to distinguish between covertext and
stegotext. Table 3 presents the results of our steganalysis. Most prior works
have primarily focused on syntactic similarity, which we believe is
insufficient as text evaluation should encompass both syntactic analysis and
semantic analysis. Thus we utilize three distinct methods, namely TS-BiRNN
Yang et al. (2019b), R-BiLSTM-C Niu et al. (2019) and BERT-C, to assess the
performance of stego-methods in terms of syntactic similarity and semantic
similarity. Among these methods, the BERT-C method, inspired by Wen et al.
(2022), utilizes a BERT encoder Devlin et al. (2019) to encode the input
sequence and uses a linear classifier to determine whether the given sequence
is a stegotext.

We exclude ADG and SAAC from our analysis since their BPW settings are
difficult to adjust, leading to an unfair comparison. Moreover, due to their
high BPW settings (BPW > 5), ADG and SAAC cause significant Psic Effect, which
makes it difficult to classify their stegotext accurately through syntactic
analysis. Even with extremely low text quality, some of these methods can
achieve almost 50% detection accuracy, which indicates a perfect but
unreasonable statistical imperceptibility.

In conclusion, the results show that our method outperforms most of the
previous methods.

###  4.5 Language Evaluation

While statistical imperceptibility can measure the similarity between text
distributions, human perception still plays a significant role in assessing
the perceptual imperceptibility of the stegotext.

To simulate the human evaluation and ensure reproducibility, we conduct
language evaluation on the generated stegotext of each method using large
language models. First, we attempt to collect the stegotext generated with
similar $\text{BPW}\approx 2.5$ using the IMDB dataset333For ADG and SAAC, the
BPW is set to 4.931 and 4.471.. We then randomly select two sentences from
each set of stegotext to compare, denoted by <sentA,sentB>. Next, GPT-3.5 is
used to determine which sentence in the pair is considered high-quality. To be
more specific, we compare our method with each baseline and collect ratios of
how much of our stegotext is deemed to be better than the others. We sample
10K sentence pairs between our method and each baseline. Appendix A.5
illustrates the details of the implementation of language evaluation.

(a)

(b)

Figure 5:  Curves of BPW and JSDfull with respect to: (a) different threshold
$\tau$ (b) different context size $k$

Eval. | GT | Fully-supervised | Training-free
---|---|---|---
RNN | VAE | ADG | NLS | SAAC
Sound. | 0.788 | 3.812 | 8.363 | 8.042 | 1.373 | 1.654
Relev. | 1.196 | 2.397 | 3.608 | 5.345 | 2.479 | 3.850
Engag. | 1.157 | 5.386 | 9.267 | 7.224 | 1.924 | 2.380
Avg. | 1.047 | 3.865 | 7.080 | 6.870 | 1.926 | 2.628

Table 4:  Language evaluation results. Each value represents the ratio of how
much our stegotext is considered better than the particular method. For
example, a ratio of 3.812 means that the number of our stegotext considered
better is 3.812$\times$ more than the other method.

Table 4 demonstrates three kinds of quality measurement conducted in our
experiments:

  1. (a)

Soundness measures the extent to which one sentence is more _logical_ and
_meaningful_.

  2. (b)

Relevance measures the extent to which one sentence is more _relevant_ to the
specific topic.

  3. (c)

Engagingness measures the extent to which one sentence is more _appealing_ and
_engaging_.

Our three measurements are all close to 1.0 when compared to the ground truth
covertext, which indicates a high level of similarity. Most supervised methods
aim to approximate covertext distribution, which results in a large gap
between readability and imperceptibility. Training-free methods, however, are
more effective at balancing both perceptual and statistical imperceptibility.
Furthermore, our method outperforms all other baselines with an average
improvement of at least $1.926\times$.

###  4.6 Ablation Study

Figure 6: Distribution of the rank of selected candidates.

We analyze EF coding in terms of the probability of selecting a candidate from
a pool of candidates. We replace every occurrence of the candidate with its
rank of likelihood in the descending-sorted candidate pool in Fig. 6. It
appears that words with lower ranks are more likely to be selected, which is
generally of high word probability. The shift of word distribution results in
slight improvements in BPW and JSDs, as shown in Fig. 5.

Figure 5 also shows the performance of our method under different hyper-
parameter settings. We can observe from the results that an increase in
threshold $\tau$ and context size $k$ would result in a decrease in BPW and
JSD. According to the results, a higher sentence quality would lead to a lower
embedding rate but better statistical imperceptibility. This indicates a
tradeoff between embedding rate and statistical imperceptibility exists.

##  5 Related Work

Linguistic steganography can be divided into three types Fridrich (2009),
selection-based Sajedi and Jamzad (2008); Chen and Chen (2019), edit-based
Chang and Clark (2014) and generation-based steganography Kang et al. (2020);
Xiang et al. (2020). However, the low embedding rate of these methods hinders
their practical application, giving rise to generation-based steganography.

To encode secret messages into textual data, early generative steganography
utilized the Markov model to obtain conditional probability distributions
Moraldo (2014) for text generation. Subsequently, the development of NLP has
significantly improved distribution modeling. It was first proposed by Fang et
al. (2017) to use a supervised recurrent neural network (RNN) to learn the
statistical language model of natural texts with a fixed-length coding
dictionary. Subsequently, Yang et al. (2019a) proposed a generation-based
steganography based on RNN and Huffman coding. Yang et al. also proposed that
different neural models make a difference, such as the generative adversarial
network Yang et al. (2020) and variational auto-encoder Yang et al. (2021).
Zhang et al. (2021) and Ding et al. (2023) further presented provably secure
algorithms for better statistical imperceptibility.

While supervised methods attempt to fit models to the source distribution, the
restriction of the source distribution leads to a high risk of generating
similar but unintelligible text due to the complicated process of embedding
secret messages. To address this limitation, models pre-trained on large-scale
datasets are used. Ziegler et al. (2019) proposed that arithmetic coding be
used with a pre-trained GPT-2 model. Dai and Cai (2019) further proposed an
improved Huffman encoding algorithm, which is intended to adjust embedding
rates dynamically. Shen et al. (2020) improved the previous works with self-
adjusting arithmetic coding for near-imperceptible text. Nevertheless, all of
these training-free approaches employ a similar paradigm for generating the
context’s _next sentence_ , making them impractical in actual usage.

##  6 Conclusion

In this work, we present a zero-shot approach based on in-context learning for
linguistic steganography. For the LLM to generate a similar sentence, samples
of covertext serve as the context for the LLM in the QA. We also integrate
several techniques to improve the text generation process for better
readability and imperceptibility of the stegotext. Our experimental results
exhibit the high imperceptibility of our approach compared to other methods.
In addition, the stegotext is more intelligible and resembles the covertext in
many aspects. We hope this work will be a step towards the more practical
applications of linguistic steganography.

##  7 Limitations

Our approach has indeed demonstrated advantages in producing intelligible and
innocent stegotext, yet we rely heavily on the generalization capabilities of
LLMs. The computation overhead of LLMs is approximately $3\times$ to $5\times$
in comparison with prior works. There may be an additional vulnerability if
real-time communication is required.

Further, we have not tested the proposed method in a relatively large
embedding rate setting (e.g., BPW > 5). While we believe that stegotext
generated under settings like these would be highly unintelligible and
impractical in real life, the limits of our method have yet to be determined.

Data integrity is another essential requirement of our steganographic method.
The incompleteness of the stegotext may result in invertible damage to the
covert bitstream as in most previous works. Fortunately, textual data are less
likely to suffer from lossy compression issues than other types of media
(e.g., image, audio, and video).

##  8 Ethical Considerations

The use of pre-trained LLMs may exhibit potential issues, such as insults,
political biases, or gender discrimination. It should also be noted that the
data underlying our method may introduce additional ethical concerns. While we
did not observe such problems during our experiments, users should be aware of
these issues in practice.

## Acknowledgement

This research is supported by the National Key R&D Program of China under
grant (No.2022YFB2703001). Thanks to all the reviewers for their insightful
suggestions.

## References

  * Bellare et al. (1997) Mihir Bellare, Anand Desai, Eron Jokipii, and Phillip Rogaway. 1997.  A concrete security treatment of symmetric encryption.  In _Proc. 38th Annual Symposium on Foundations of Comput. Sci._ , pages 394–403. IEEE.
  * Cachin (1998) Christian Cachin. 1998.  An information-theoretic model for steganography.  In _Information Hiding_ , pages 306–318, Berlin, Heidelberg. Springer Berlin Heidelberg.
  * Chang and Clark (2014) Ching-Yun Chang and Stephen Clark. 2014.  Practical linguistic steganography using contextual synonym substitution and a novel vertex coding method.  _Comput. Linguistics_ , 40(2):403–448.
  * Chapman and Davida (1997) Mark Chapman and George Davida. 1997.  Hiding the hidden: A software system for concealing ciphertext as innocuous text.  In _Inf. and Commun. Secur._ , pages 335–345, Berlin, Heidelberg. Springer Berlin Heidelberg.
  * Chen and Chen (2019) Xianyi Chen and Sheng Chen. 2019.  Text coverless information hiding based on compound and selection of words.  _Soft Comput._ , 23.
  * Dai and Cai (2019) Falcon Dai and Zheng Cai. 2019.  Towards near-imperceptible steganographic text.  In _Proc. 57th Annu. Meeting Assoc. Comput. Linguistics_ , pages 4303–4308, Florence, Italy. Assoc. Comput. Linguistics.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proc. 2019 Conf. North American Assoc. Comput. Linguistics_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Ding et al. (2023) Jinyang Ding, Kejiang Chen, Yaofei Wang, Na Zhao, Weiming Zhang, and Nenghai Yu. 2023.  Discop: Provably secure steganography in practice based on "distribution copies".  In _2023 IEEE Symp. Secur. Privacy (SP)_ , pages 2238–2255.
  * Fang et al. (2017) Tina Fang, Martin Jaggi, and Katerina Argyraki. 2017.  Generating steganographic text with LSTMs.  In _Proc. ACL 2017, Student Res. Workshop_ , pages 100–106, Vancouver, Canada. Assoc. Comput. Linguistics.
  * Fridrich (2009) Jessica Fridrich. 2009.  _Steganography in digital media: principles, algorithms, and applications_.  Cambridge University Press.
  * Go et al. (2009) Alec Go, Richa Bhayani, and Lei Huang. 2009.  Twitter sentiment classification using distant supervision.  _CS224N project report, Stanford_ , 1(12).
  * Guo et al. (2015) Linjie Guo, Jiangqun Ni, Wenkang Su, Chengpei Tang, and Yun-Qing Shi. 2015.  Using statistical image model for jpeg steganography: Uniform embedding revisited.  _IEEE Trans. Inf. Forensics Security_ , 10(12):2669–2680.
  * Kang et al. (2020) Huixian Kang, Hanzhou Wu, and Xinpeng Zhang. 2020.  Generative text steganography based on lstm network and attention mechanism with keywords.  _Electron. Imag._ , 2020(4):291–1.
  * Krishnan et al. (2017) R. Bala Krishnan, Prasanth Kumar Thandra, and M. Sai Baba. 2017.  An overview of text steganography.  In _2017 4th Int. Conf. Signal Process., Commun. and Netw. (ICSCN)_ , pages 1–6.
  * Lin et al. (2018) Zinan Lin, Yongfeng Huang, and Jilong Wang. 2018.  Rnn-sm: Fast steganalysis of voip streams using recurrent neural network.  _IEEE Trans. Inf. Forensics Security_ , 13(7):1854–1868.
  * Liu et al. (2019) Yunxia Liu, Shuyang Liu, Yonghao Wang, Hongguo Zhao, and Si Liu. 2019.  Video steganography: A review.  _Neurocomputing_ , 335:238–250.
  * Luo et al. (2016) Yubo Luo, Yongfeng Huang, Fufang Li, and Chinchen Chang. 2016.  Text steganography based on ci-poetry generation using markov chain model.  _KSII Trans. Internet and Inf. Syst. (TIIS)_ , 10(9):4568–4584.
  * Maas et al. (2011) Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. 2011.  Learning word vectors for sentiment analysis.  In _Proc. 49th Annu. Meeting Assoc. Comput. Linguistics_ , pages 142–150, Portland, Oregon, USA. Assoc. Comput. Linguistics.
  * Moraldo (2014) H Hernan Moraldo. 2014.  An approach for text steganography based on markov chains.  _arXiv preprint arXiv:1409.0915_.
  * Niu et al. (2019) Yan Niu, Juan Wen, Ping Zhong, and Yiming Xue. 2019.  A hybrid r-bilstm-c neural network based text steganalysis.  _IEEE Signal Process. Lett._ , 26(12):1907–1911.
  * Otter et al. (2021) Daniel W. Otter, Julian R. Medina, and Jugal K. Kalita. 2021.  A survey of the usages of deep learning for natural language processing.  _IEEE Trans. Neural Netw. Learn. Syst._ , 32(2):604–624.
  * Radford et al. (2019) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019.  Language models are unsupervised multitask learners.  _OpenAI blog_ , 1(8):9.
  * Sajedi and Jamzad (2008) Hedieh Sajedi and Mansour Jamzad. 2008.  Cover selection steganography method based on similarity of image blocks.  In _2008 IEEE 8th Int. Conf. Comput. and Inf. Technol. Workshops_ , pages 379–384.
  * Shen et al. (2020) Jiaming Shen, Heng Ji, and Jiawei Han. 2020.  Near-imperceptible neural linguistic steganography via self-adjusting arithmetic coding.  In _Proc. 2020 Conf. Empirical Methods in Natural Lang. Process. (EMNLP)_ , pages 303–313, Online. Association for Computational Linguistics.
  * Shniperov and Nikitina (2016) Alexey Nikolaevich Shniperov and KA Nikitina. 2016.  A text steganography method based on markov chains.  _Autom. Control and Comput. Sci._ , 50:802–808.
  * Subramanian et al. (2021) Nandhini Subramanian, Omar Elharrouss, Somaya Al-Maadeed, and Ahmed Bouridane. 2021.  Image steganography: A review of the recent advances.  _IEEE Access_ , 9:23409–23423.
  * Taleby Ahvanooey et al. (2019) Milad Taleby Ahvanooey, Qianmu Li, Jun Hou, Ahmed Raza Rajput, and Yini Chen. 2019.  Modern text hiding, text steganalysis, and applications: A comparative analysis.  _Entropy_ , 21(4).
  * Tao et al. (2019) Jinyuan Tao, Sheng Li, Xinpeng Zhang, and Zichi Wang. 2019.  Towards robust image steganography.  _IEEE Trans. Circuits Syst. Video Technol._ , 29(2):594–600.
  * Touvron et al. (2023) Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023.  Llama: Open and efficient foundation language models.  _arXiv preprint arXiv:2302.13971_.
  * van Laarhoven and Aarts (1987) Peter J. M. van Laarhoven and Emile H. L. Aarts. 1987.  _Simulated annealing_, pages 7–15. Springer Netherlands, Dordrecht.
  * Weber (1978) W. Weber. 1978.  Differential encoding for multiple amplitude and phase shift keying systems.  _IEEE Transactions on Communications_ , 26(3):385–391.
  * Wen et al. (2022) Juan Wen, Ziwei Zhang, Yu Yang, and Yiming Xue. 2022.  Few-shot text steganalysis based on attentional meta-learner.  In _Proceedings of the 2022 ACM Workshop on Information Hiding and Multimedia Security_ , pages 97–106.
  * Xiang et al. (2022) Lingyun Xiang, Rong Wang, Zhongliang Yang, and Yuling Liu. 2022.  Generative linguistic steganography: A comprehensive review. _KSII Trans. Internet and Inf. Syst. (TIIS)_ , 16(3).
  * Xiang et al. (2020) Lingyun Xiang, Shuanghui Yang, Yuhang Liu, Qian Li, and Chengzhang Zhu. 2020.  Novel linguistic steganography based on character-level text generation.  _Mathematics_ , 8(9):1558.
  * Yang et al. (2019a) Zhong-Liang Yang, Xiao-Qing Guo, Zi-Ming Chen, Yong-Feng Huang, and Yu-Jin Zhang. 2019a.  Rnn-stega: Linguistic steganography based on recurrent neural networks.  _IEEE Trans. Inf. Forensics Security_ , 14(5):1280–1295.
  * Yang et al. (2021) Zhong-Liang Yang, Si-Yu Zhang, Yu-Ting Hu, Zhi-Wen Hu, and Yong-Feng Huang. 2021.  Vae-stega: Linguistic steganography based on variational auto-encoder.  _IEEE Trans. Inf. Forensics Security_ , 16:880–895.
  * Yang et al. (2019b) Zhongliang Yang, Ke Wang, Jian Li, Yongfeng Huang, and Yu-Jin Zhang. 2019b.  Ts-rnn: Text steganalysis based on recurrent neural networks.  _IEEE Signal Process. Lett._ , 26(12):1743–1747.
  * Yang et al. (2020) Zhongliang Yang, Nan Wei, Qinghe Liu, Yongfeng Huang, and Yujin Zhang. 2020.  Gan-tstega: Text steganography based on generative adversarial networks.  In _Digital Forensics and Watermarking_ , pages 18–31, Cham. Springer International Publishing.
  * Zhang et al. (2021) Siyu Zhang, Zhongliang Yang, Jinshuai Yang, and Yongfeng Huang. 2021.  Provably secure generative linguistic steganography.  In _Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021_ , pages 3046–3055, Online. Association for Computational Linguistics.
  * Ziegler et al. (2019) Zachary Ziegler, Yuntian Deng, and Alexander Rush. 2019.  Neural linguistic steganography.  In _Proc. 2019 Conf. Empirical Methods Natural Lang. Process. and 9th Int. Joint Conf. Natural Lang. Process. (EMNLP-IJCNLP)_ , pages 1210–1215, Hong Kong, China. Assoc. Comput. Linguistics.
  * Zipf (1999) George Kingsley Zipf. 1999.  _The psycho-biology of language: An introduction to dynamic philology_ , volume 21.  Psychology Press.

##  Appendix A Detailed Implementation

###  A.1 Edge Flipping Coding

The pseudo-code for Edge Flipping Coding is presented in Algorithm 2 for
encoding, and Algorithm 3 for decoding.

Algorithm 2 Edge Flipping Encoding

1:Bitstream $\bm{m}$.

2:EF bitstream $\bm{m}^{\prime}$.

3:Output bitstream $\bm{m}^{\prime}\leftarrow\emptyset$

4:Bit state $s\leftarrow 0$

5:for $b\in\bm{m}$ do

6: if $s=b$ then

7: $\bm{m}^{\prime}\leftarrow\bm{m}^{\prime}\cup\\{0\\}$

8: else

9: $\bm{m}^{\prime}\leftarrow\bm{m}^{\prime}\cup\\{1\\}$

10: end if

11: $\triangleright$ Update bit state

12: $s\leftarrow b$

13:end for

Algorithm 3 Edge Flipping Decoding

1:EF Bitstream $\bm{m}^{\prime}$.

2:Bitstream $\bm{m}$.

3:Output bitstream $\bm{m}\leftarrow\emptyset$

4:Bit state $s\leftarrow 0$

5:for $b^{\prime}\in\bm{m}^{\prime}$ do

6: if $b^{\prime}=1$ then

7: $s\leftarrow\mathord{\sim}s$

8: end if

9: $\triangleright$ Append bit state

10: $\bm{m}\leftarrow\bm{m}\cup\\{s\\}$

11:end for

Since EF coding is a bijection, it can be applied multi-round to the original
bitstream. The final bitstream is the one with the fewest 1s.

###  A.2 Bitstream Extraction

We provide the bitstream extraction algorithm of the embedding module in
Algorithm 4.

Algorithm 4 Information Extraction Algorithm

1:stegotext $\bm{y}=[y_{1},\dots,y_{n}]$, threshold $\tau$.

2:Bitstream $\bm{m}$.

3:Timestep $t\leftarrow 1$, output bitstream $\bm{m}\leftarrow\emptyset$

4:while not the end of $\bm{m}$ do

5: $\triangleright$ Compute conditional probs

6: $\bm{p}\leftarrow\mathbf{P}_{\text{stega}}(x_{t}\mid x_{<t})$

7: $\triangleright$ Prune candidate words

8: $\bm{c}\leftarrow[c_{i}\mid\bm{p}(c_{i})\geq\tau]$

9: $\triangleright$ Huffman encoding

10: $H\leftarrow\mathrm{Huffman}(\bm{c},\bm{p})$

11: $\triangleright$ Select candidate

12: $\bm{s}\leftarrow$ Bits $\bm{s}$ whose binary representation matches
certain word $c\in H$

13: $\bm{m}\leftarrow\bm{m}\cup\bm{s},t\leftarrow t+1$

14:end while

###  A.3 In-Context QA

The complete in-context QA templates of our method are presented in Fig 7. We
instruct the LLM to follow the XML-style response for parsing results.
Covertext context is placed in the  {CONTEXT} slot. There is also a hint of
the corpus placed on the  {CORPUS} slot.

<<SYS>>
You are an expert at mimicing the language style of others (e.g., the use of
words and phrases). And you are a helpful and respectful assistant.

Users will input sentences from a given corpus. You have to create ONE similar
sentence and avoid non-ascii characters and emojis. This is very important to
the user’s career.

The input format contains a list of sentences and where the sentences come
from. For example:
<CORPUS> {CORPUS}</CORPUS>
<CONTEXT>
Example sentence 1.

Example sentence 2.
</CONTEXT>

Your output should be like:

The generated similar sentence in ONE LINE is:

<</SYS>>

[INST]<CORPUS> {CORPUS}</CORPUS>
<CONTEXT>
{CONTEXT}
</CONTEXT>[/INST]

The generated similar sentence in ONE LINE is: Figure 7:  QA templates for in-
context stegotext generation. Slot  {CORPUS} and  {CONTEXT} would be replaced
by the corpus name and the selected context from covertext in the runtime.

###  A.4 Baselines

We rebuild the entire benchmark pipeline for a fair comparison between
different stego-methods. The following are the implementation details:

  * •

RNN-Stega: We refactor the official codebase from TensorFlow to PyTorch.
Hyper-parameters: lr=0.001, dropout=0.5, 30 epochs, 768 LSTM units, and 2
layers of LSTM.

  * •

VAE-Stega: We rebuild the VAE pipeline using PyTorch following the official
codebase. Hyper-parameter setting: lr=3e-4, 20 epochs, hidden size of 768,
DistilBERTBASE as encoder, 2 layers of LSTM as decoder.

  * •

ADG: We reproduce ADG based on the official codebase. Hyper-parameter setting:
lr=0.001, 30 epochs, 768 LSTM units, and 2 layers of LSTM.

  * •

NLS: We reproduce NLS based on the official codebase. We use gpt2-medium model
from Huggingface Library as the base model.

  * •

SAAC: We reproduce SAAC based on the official codebase. We use gpt2-medium
model from Huggingface Library as the base model. The hyper-parameter $\delta$
is set the same as the original paper: 0.01, 0.05 and 0.10.

###  A.5 Language Evaluation

In this section, we provide the actual prompts for ChatGPT to discriminate the
quality of the sentence pair <sentA,sentB>. Figure 8 illustrates the prompting
templates. We also randomly swap the sentences of a pair before calling the
ChatGPT API.

The user will input 2 sentences. You have to decide which one is more
logically sound and meaningful. This is important to the user’s career.

The result should be in JSON. It should contain the key "result" with value
being either 0 or 1, indicating the first or second one.

1\.  <sentA>

2\.  <sentB> (a) Template for evaluating soundness.

The user will input 2 sentences. You have to decide which one is more likely
to be part of some " {TOPIC}". This is important to the user’s career.

The result should be in JSON. It should contain the key "result" with value
being either 0 or 1, indicating the first or second one.

1\.  <sentA>

2\.  <sentB> (b) Template for evaluating relevance. Here  {TOPIC} is used to
restrict the evaluation to specific topics. For IMDB, we use "movie reviews"
as the topic.

The user will input 2 sentences. You have to decide which one is more engaging
and appealing to readers. This is important to the user’s career.

The result should be in JSON. It should contain the key "result" with value
being either 0 or 1, indicating the first or second one.

1\.  <sentA>

2\.  <sentB> (c) Template for evaluating engagingness.

Figure 8: Templates for language evaluations.

The raw data of language evaluations is presented in Table 5, where each grid
represents the percentage of how much ours is considered better.

Eval. | GT | Fully-supervised | Training-free
---|---|---|---
RNN | VAE | ADG | NLS | SAAC
Sound. | .4406 | .7922 | .8932 | .8894 | .5786 | .6232
Relev. | .5446 | .7056 | .7830 | .8424 | .7126 | .7938
Engag. | .5364 | .8434 | .9026 | .8784 | .6580 | .7041

Table 5:  Raw results of language evaluations.

###  A.6 Ablation

We present the ablation study of the factors $\alpha$ and $\beta$ of annealing
search and repeat penalty in Table 6.

Metrics | $\alpha$ | $\beta$
---|---|---
1.00 | 1.25 | 1.50 | 0.25 | 0.50 | 1.00
BPW | 2.149 | 2.420 | 2.462 | 2.116 | 2.420 | 2.511
JSDfull | 18.11 | 18.37 | 18.66 | 18.28 | 18.37 | 19.02

Table 6: Ablation on the annealing search factor $\alpha$

and repeat penalty factor $\beta$.

##  Appendix B Efficiency

Table 7 shows the computational overhead of baselines and our method. Due to
the use of LLMs, our method requires a cost of $3\times$ to $5\times$ more
time.

Method | RNN | VAE | ADG | NLS | SAAC | ours
---|---|---|---|---|---|---
Avg. Time (s) | 1.55 | 2.16 | 1.87 | 2.89 | 3.38 | 8.41

Table 7:  The average time cost of generating one steganographic sentence
using a single RTX3090.

##  Appendix C Examples

Figure 9 provides more examples of the generated stegotext of each method. In
comparison with other approaches, ours has better sentence quality.

Secret Bitstream (Base64): xB0CEVgADkP2YoA= [0.5ex]1pt1pt Fully-supervised
RNN-Stega: Sin City is a great film, and the story is told from the viewer.The
plot is pretty simplistic, and has no redeemable qualities to it whatsoever.
VAE-Stega: short! i have the feeling that i have a copy on dvd. i would have
been watching it, and i’m a huge fan. ADG: Sinbad?No?So HOAME.horrible
movie.Acting wise as it describes is hard. [0.5ex]1pt1pt Training-free NLS:
And while the latter is certainly true, I think it is also fair to say that
CBS has done a pretty good thing. The network has been a champion for science
and for health. SAAC: The Center for Public Integrity of West Allis,
Wisconsin, and Wythe Law School of Louisville, Kentucky reported this week
that cigarette makers have guities. [0.5ex]1pt1pt Ours: All countries have an
ugly side, just as ours does. It’s unfair to say that only the Philippines has
this issue, though. We are all capable of such shortfalls. (a)

Secret Bitstream (Base64): pTJAjKMijGEMkA== [0.5ex]1pt1pt Fully-supervised
RNN-Stega: Adding insult from this review, it’s not as if it is shot on the
big screen.The problem was that the film was not intended. VAE-Stega: four
years later he’re going to get to the job, and then he becomes a suspect that
he was not a guy and he has a very strange and very good ADG: Adding Wes
Anderson is a good story,or a movie that you will get used to.In other words,
it’s a great [0.5ex]1pt1pt Training-free NLS: The only thing missing from the
film’s budget is a big name star, and that will have to be one of the film-
makers’ two choices. SAAC: The cleanroom in which the court martial took place
is flanked by treetops, allowing for … [0.5ex]1pt1pt Ours: Cheaper doesn’t
always have to equal inferior, particularly in the case of indie films, where
creators are often pushed to maximize the impact of limited resources. (b)

Secret Bitstream (Base64): CYmMLJkPqMO4CHaxyEBBXrkDs5Iw [0.5ex]1pt1pt Fully-
supervised RNN-Stega: Extreme to the point where a film that was made was a
huge waste.The film’s ending is an absolute disaster, but it was very
disappointing to see a man that could easily be a good thing.The film had some
pretty interesting elements, and the movie was very well done, but that’s not
enough for it. VAE-Stega: if you like "the rose", "love" (the "i have you,
you’re better! ")……. the ending is very funny. this is no exception. it’is so
bad!… the movie is bad.. bad!!. ADG: Extremely directed by Gulna Wong who
brings the thought of interest for a free part of her life.Avoid this film as
a film for the best of heartless Arts cinema lover. [0.5ex]1pt1pt Training-
free NLS: It was only a matter for him to find a wife to take care in the
meantime and then the marriage will be broken up and it can be said he’ll lose
all his love for his beloved. I mean it makes perfect sense, it seems to make
more sense that he has to leave her for the woman he loves. SAAC: She wants to
break things up with him. I’ve always seen this as perfect because it’s a
result of her deep love for man in this movie. However, not everybody likes
watching two men break up, you know? [0.5ex]1pt1pt Ours: Oh my gawd, stop
dting married men! Like seriously, why do we keep falling into this trap over
and over? Anglea basset’s chracter should totall be more concerned with the
fact that her husband abandoned hre for another woma than some mediocore
romance flic. Overall, I would give this movie a big fat C- for wasted
poasbility and a little more thought. (c)

Figure 9: Examples of stegotext generated by various methods on the topic of
"movie reviews".

##  Appendix D Scientific Artifacts

The licenses for all the baselines are listed below: RNN-Stega (MIT License),
VAE-Stega (MIT License), ADG (MIT License), NLS (MIT License), SAAC (GPL-3.0
License). The licenses for models are listed below: BERT (Apache 2.0 License),
GPT-2 (Modified MIT License), LLaMA2-Chat-7B (LLAMA 2 Community License).


DeepCluster model There has been a surge in the application of machine
learning techniques in the field of particle physics over recent years due to
the capacity of these algorithms to extract complex patterns. In particular,
techniques such as deep neural networks (DNN) and convolutional neural
networks (CNN) have enabled unprecedented levels of accuracy and efficiency in
tasks such as particle tracking and calorimetry clustering. Graph neural
networks (GNN) have also recently gained significant popularity in high-energy
physics applications (e.g. graph_particle, graph_2) due to the following
factors:

  * •

they can handle sparse data coming from complex detector geometries,

  * •

they are applicable to non-Euclidean data structures with variable input
sizes.

To achieve optimal performance, we combine multiple state-of-the-art machine
learning methods in the DeepCluster architecture. First, we use the excellent
pattern recognition abilities of CNNs. As energy deposited in the crystals of
the calorimeter can be represented as pixel intensities of an image, CNN can
be easily and naturally applied to treat calorimeter data. Secondly, we use a
GNN to allow information transfer between neighboring particles.
In the DeepCluster model, the particle-reconstruction task is divided into two
consecutive steps:

  1. 1.

Extract small windows ($7\times 7$ crystals), named seed windows, whose energy
deposits potentially originate from a real particle and not from noise. This
is performed by a first NN called the seed-finder NN described in Section •.

  2. 2.

For each seed window predict the kinematic properties of the corresponding
generated particle. This is done with a second NN called center-finder NN. In
this work, we develop two different approaches for the center-finder NN:

     * •

The first one, based on a CNN, is described in Section •.

     * •

To circumvent the limitations of this CNN-based center-finder, a second
center-finder, using a GNN, is introduced in Section •.

With this approach, the networks process only small crystal matrices as
opposed to the full calorimeter matrix ($51\times 51$ crystals). This
significantly reduces the need for computational power and allows to easily
scale this method to a real detector (the ECAL barrel contains $170\times 360$
crystals).

##  Seed windows definition

The inputs to the DeepCluster model are called seed windows and are obtained
for each sample as follows:

  1. 1.

All the crystals from the toy calorimeter with energy deposits
$E_{xtal}>0.5~{}\GeV$ are selected and defined as seed crystals. The selected
$0.5~{}\GeV$ threshold value corresponds to $E_{\mathrm{thr}}^{\mathrm{seed}}$
used in the PFClustering algorithm and tuned for Run 3 operations.

  2. 2.

For each seed crystal, a seed window is created. This is a matrix of size
$7\times 7$ crystals centered on the seed crystal. An example of a seed window
is shown in Fig. •.

From one simulated sample corresponding to the full toy calorimeter, several
seed windows can be created. They originate from a real particle or from
noise. In order to train the different networks, we need to associate the seed
windows to their corresponding generated particles and the subsequent truth
labeling has to be defined. We first check if the impact position of any
generated particle in the considered sample lies within the boundaries of the
seed crystal:

  * •

In such a case, the corresponding particle (at most one by construction) is
associated to the seed window. The window is labeled true seed window and
assigned three kinematic variables corresponding to the true particle:
generated position $(\xgen{},\ygen{})$ and energy .

  * •

Otherwise, it is labeled as background.

For true seed windows, the local position of the generated particle inside the
seed window (, ) is defined as:

$\split\xloc{}&=\xgen{}-\xwin{}\\\ \yloc{}=\ygen{}-\ywin{}\\\ $ (1)

where (, ) corresponds to the position of the seed window center.

##  Seed-finder NN

The seed-finder NN is the first network in the DeepCluster model. This is a
CNN, whose goal is to select the true seed windows and discard background
ones. [14]l0.5

[width=0.48]figures/event_display_example_7.png

Example of a seed window. The network takes a seed window as input and assigns
to it a seed-finder score ($P_{\mathrm{seedSF}}$), indicating the likelihood
to be a true seed window. The seed windows with
$P_{\mathrm{seedSF}}<P^{\mathrm{thr}}_{\mathrm{seedSF}}$, where
$P^{\mathrm{thr}}_{\mathrm{seedSF}}$ is a tunable threshold defined in Section
LABEL:section:optimization, are discarded, the other ones are passed to the
center-finder NN. The seed-finder-NN architecture consists of two
convolutional and two dense layers. LeakyReLU is chosen as the activation
function [relu] and a dropout of 10$\%$ is applied after the first
convolutions. For the output of the seed-finder NN, a sigmoid activation
function is used. A detailed view of the model architecture is presented in
Fig. •.

[width=0.85]figures/architecture_1.pdf

Figure : Seed-finder NN architecture. $7\times 7$ seed windows are first
selected around all possible seeds ($E_{xtal}>0.5~{}\GeV$). They are
separately passed as input to the seed-finder NN. The input is processed by
two convolutional layers until the vector of summary features is extracted.
This vector is further passed to two dense layers, resulting in the network
output: the seed-finder score $P_{\mathrm{seedSF}}$. It represents the
likelihood of the input to originate from a generated particle. Detailed
information on the number of nodes at each layer is presented in the figure.

The model is trained using the Adam optimizer [adam] with a learning rate of
0.0001 and a batch size of 64. We use the binary cross entropy as a loss
function. The network is trained for $\approx$100 epochs and the epoch
yielding the best result on the validation sample is chosen. Compared to the
seeding step used in the PFClustering algorithm, the seed-finder NN provides
several advantages:

  * •

As the condition for the seed to be a local maximum is removed, the seed-
finder NN provides a better possibility to reconstruct close-by photons.

  * •

The seed-finder NN performs a refined seed window selection that helps to
significantly eliminate the low-energy background coming from electronic
noise.

##  Center-finder NN – Convolutional Neural Network

The center-finder NN is the second step of the DeepCluster model. It predicts
the position (i, i) and energy i of the generated particle associated to the
seed window $i$; the corresponding generated quantities are named (i, i), i.
The global coordinates can be further inferred by inverting Eq. 1. Similarly
to the seed-finder NN, the CNN-based center-finder takes seed windows as
input. All the inputs are processed independently from each other. In the
training phase, the inputs are limited to the true seed windows. However, for
the evaluation, the center-finder NN processes all the seed windows with the
seed-finder scores passing $P_{\mathrm{seed}}^{\mathrm{thr}}$. The
architecture of the center-finder NN is close to the one of the seed-finder
NN: it consists of multiple convolutional layers, followed by dense layers
that are divided into two parts: one resulting in coordinate prediction and
another in energy prediction. ReLU activation function is applied everywhere
except for the output layer, where $\tanh$ and sigmoid functions are used
respectively for position and energy predictions [relu]. The dropout level is
set to 10$\%$ everywhere except for the last energy prediction layer, where it
is set to 30$\%$. The full network architecture with precise details on the
number of nodes is presented in Fig. •.

[width=]figures/architecture_2.pdf

Figure : Center-finder NN architecture. The seed windows are passed separately
as input to the network. Each input is processed by two convolutional layers
until the vector of summary features is extracted. This vector is passed
through one dense layer and further sent separately to two different branches
(coordinate and energy predictions). In each branch, it passes through two
additional dense layers. Detailed information on the number of nodes at each
layer is presented in the figure.

The network is trained for 1000 epochs with a batch size $N_{b}$ of 64 seed
windows. We use the mean absolute error loss defined by:

$\mathcal{L}_{\mathrm{kin}}=\frac{1}{N_{b}}\sum_{i=1}^{N_{b}}\frac{1}{2}\left(\lvert\xrec{i}-\xloc{i}\rvert+\lvert\yrec{i}-\yloc{i}\rvert\right)+\lvert\Erec{i}-\Egen{i}\rvert$
(2)

The training is performed using the Adam optimizer with a learning rate of
0.0001. The chosen epoch is the one providing the best performance according
to the validation dataset.

###  Results

The results of the DeepCluster model and the PFClustering algorithm are
compared with the single- and two-photon test datasets. For the seed-finder
NN, we set $P^{\mathrm{thr}}_{\mathrm{seedSF}}=0.3$. This latter threshold is
tuned for the final DeepCluster model as discussed in Section
LABEL:section:optimization. The performance for the position prediction is
presented in Fig. • for the single-photon dataset on the left and for the two-
photon dataset on the right. Each plot shows the distribution of the
difference $\xrec{}-\xloc{}$. Similar results are obtained for the
$y$-coordinate. We associate reconstructed objects to generated objects using
a matching procedure described in appendix_c. The DeepCluster network
significantly outperforms the PFClustering algorithm. The coordinate
resolution (evaluated as half the interval containing 68% of the distribution
and centered on the median) for the DeepCluster is 0.02 crystal compared to
0.04 crystal for the PFClustering algorithm for the single-photon dataset and
0.03 crystal versus 0.08 crystal for the two-photon dataset. The position
reconstruction for the two-photon dataset is a more difficult task than for
the single-photon one because of overlapping energy clusters for close-by
generated particles. This explains the performance degradation in the two-
photon dataset.

[width=0.49]figures/x_resolution_cnn_paper_1.pdf
[width=0.49]figures/x_resolution_cnn_paper_2.pdf

Figure : Distribution of the variable $\xrec{}-\xloc{}$. The results are
obtained by applying the PFClustering algorithm and DeepCluster network on the
single-photon (left) and two-photon (right) test datasets. The resolutions
(see text) are reported in the figures.

Concerning the energy prediction, the performance of the PFClustering and
DeepCluster algorithms is closer. They are presented for the optimized
DeepCluster algorithm in Section LABEL:section:results.

###  Per-event energy overestimation

Because the network processes only seed windows, this approach can be easily
extended to a real calorimeter. However, it also raises a major issue
described in this section. Contrary to the PFClustering, the local maximum
condition is omitted for the seed crystal in the DeepCluster. As a
consequence, two (or more) neighboring crystals can be selected as seeds.
While this allows for efficient reconstruction of close-by particles, this can
also create two or more separate seed windows corresponding to a single
generated particle, which would share energy among several crystals. For the
majority of these latter cases, the seed-finder NN is able to identify the
local maximum itself and predict a high seed-finder score
$P_{\mathrm{seedSF}}$ for the corresponding seed window (in which the local
maximum is the central crystal) and a low score for its neighbor window.
However, when the position of the generated particle is close to the edge of a
crystal, its energy deposit can be very similar in the two neighboring
crystals. Both of them get a high $P_{\mathrm{seedSF}}$, giving rise to two
distinct seed windows. These windows are separately passed to the center-
finder NN that predicts similar coordinates and energies for both of them. The
same generated particle can therefore be reconstructed twice by the
DeepCluster algorithm, thus overestimating the total energy reconstructed in
the event. In the following, this is referred to as energy double counting.
This effect is illustrated in Fig. •, which presents the distribution of the
ratio $R_{\mathrm{en}}=\Erec{\mathrm{tot}}/\Egen{\mathrm{tot}}$ where
$\Erec{\mathrm{tot}}$ and $\Egen{\mathrm{tot}}$ are respectively is the total
reconstructed energy $\Erec{\mathrm{tot}}$ (sum of the energies of all the
reconstructed particles in the event) and the total generated energy
$\Egen{\mathrm{tot}}$ (sum of the energies of all the generated particles in
the event). The distribution of $R_{\mathrm{en}}$ is obtained from the single-
photon dataset. The additional peak around two observed for the CNN-based
center-finder (CF CNN on the figure) originates from the energy double
counting.

[width=]figures/double_count_energy_gnn_paper.pdf

Figure : Distributions of the ratio $R_{\mathrm{en}}$ between the total
reconstructed energy $E_{\mathrm{reco}}^{\mathrm{tot}}$ and the total
generated energy $E_{\mathrm{gen}}^{\mathrm{tot}}$. The results are obtained
with the PFClustering algorithm, the DeepCluster model with a CNN-based
center-finder CNN (CF CNN), and with a GNN-based center-finder (CF GNN). The
distributions are obtained from the single-photon test dataset. A second peak
$R_{\mathrm{en}}\approx 2$ arises for the CF CNN, while it is eliminated with
the CF GNN.

The double counting issue is cured by changing the center-finder NN
architecture as presented in the next section.

##  Center-finder NN – Graph Neural Network

The energy double counting is due to the fact that the CNN-based DeepCluster
model does not receive information from the rest of the event, as a
consequence, it is not aware of the existence of several seed windows
corresponding to the same generated particle. To solve the issue, we pass
several neighboring seed windows as input to the network. This is achieved
using a GNN architecture for the center-finder NN while the seed-finder NN
remains unchanged. The GNN implementation provides in addition message-passing
capabilities ([zhou2021graph]) enabling information sharing between the
different seed windows. The inputs to the GNN-based center-finder are
constructed as follows. First, the list of seed windows in the event is
ordered based on the energy of its center crystal $E_{\mathrm{seed}}$. This
list is processed as follows:

  1. 1.

A center-finder input is initialized from the window $w_{\mathrm{ref}}$ with
the highest energy $E_{seed}$ in the list. At this step, the input shape is
$1\times 7\times 7$.

  2. 2.

For each remaining seed window $w_{\mathrm{alt}}$ in the list, the $\Delta R$
distance between $w_{\mathrm{ref}}$ and $w_{\mathrm{alt}}$ is computed. If
$\Delta R<3$, $w_{alt}$ is added to the input. After this step, the input
shape is $N_{\mathrm{w}}\times 7\times 7$ where $N_{\mathrm{w}}$ is the number
of added seed windows.

  3. 3.

All of the seed windows included in the input are removed from the list of
seed windows. The process is iterated until this list is empty.

In this work, $N_{\mathrm{w}}$ is chosen to be at most 4. This maximum can be
easily adjusted to higher values as well. For each input, if
$N_{\mathrm{w}}<4$, it is completed with $7\times 7$ empty windows, _i.e._
with null crystal energies. In such a way, the input shape is fixed to
$4\times 7\times 7$. The 4 seed windows of the input represent the nodes of
the graph. In the center-finder NN, each seed window $j$ is first separately
processed by a chain of convolutional layers (identical to the CNN center-
finder implementation) in order to extract the vector of summary features
$v_{j}$. The message-passing is implemented as the concatenation of these
vectors. It results in 4 updated vectors $\bar{v}_{j}$, each of them
containing information about their neighbors:

$\aligned\bar{v}_{1}&=\pmatrix{v}_{1}\\\ v_{2}\\\ v_{3}\\\
v_{4},\bar{v}_{2}=\pmatrix{v}_{2}\\\ v_{1}\\\ v_{3}\\\
v_{4},\bar{v}_{3}=\pmatrix{v}_{3}\\\ v_{1}\\\ v_{2}\\\
v_{4},\bar{v}_{4}=\pmatrix{v}_{4}\\\ v_{1}\\\ v_{2}\\\ v_{3}$ (3)

The combined vectors $\bar{v}_{i}$ are then passed independently to a set of
dense layers until the final output is extracted. The GNN-based center-finder
NN predicts 4 values for each seed window $j$ in the input $i$: the kinematic
variables (ij, ij), ij and a new seed score $P_{\mathrm{seedCF}}^{ij}$,
indicating the likelihood to be associated to a true generated particle. In
this version of the DeepCluster model, the seed-finder NN serves as an initial
filter, separating signal from background, while the GNN-based center-finder
corrects for the incorrect predictions related to double counting: eventually
only objects with $P_{\mathrm{seedCF}}>P_{\mathrm{seedCF}}^{\mathrm{thr}}$ are
selected. The optimization of $P_{\mathrm{seedCF}}^{\mathrm{thr}}$ is
presented in Section LABEL:section:optimization. The GNN-based center-finder
architecture is presented in Fig. •. The loss used in the training is the sum
of three terms related to the coordinates, the energy, and the seed
probability predictions. The two first terms are based on a mean absolute
error loss while the last one is based on a focal cross-entropy loss
[focal_loss]. The combined loss is computed as:

$\aligned\mathcal{L}_{\mathrm{pos}}&=\frac{1}{4\cdot
N_{b}}\sum_{i=1}^{N_{b}}\sum_{j=1}^{4}\frac{1}{2}\left(\lvert\xrec{ij}-\xloc{ij}\rvert+\lvert\yrec{ij}-\yloc{ij}\rvert\right)\\\
\mathcal{L}_{\mathrm{en}}=\frac{1}{4\cdot
N_{b}}\sum_{i=1}^{N_{b}}\sum_{j=1}^{4}\lvert\Erec{ij}-\Egen{ij}\rvert\\\
\mathcal{L}_{\mathrm{seed}}=-\frac{1}{4\cdot
N_{b}}\sum_{i=1}^{N_{b}}\sum_{j=1}^{4}\alpha(1-P^{ij}_{\mathrm{seedCF}})^{\gamma}\log(P^{ij}_{\mathrm{seedCF}}),\\\
\mathcal{L}_{\mathrm{tot}}=\mathcal{L}_{\mathrm{pos}}+\mathcal{L}_{\mathrm{en}}+k_{\mathrm{s}}\cdot\mathcal{L}_{\mathrm{seed}},$
(4)

where $N_{b}$ is the number of batches, the focal loss parameters $\gamma$ and
$\alpha$ are chosen to be $\gamma=2$, $\alpha=0.25$, and $k_{\mathrm{s}}$ is
an adjustable parameter associated with the seed loss, it is referred to as
seed-loss weight.

[width=]figures/network_pipeline-2.pdf

Figure : Flow chart of the DeepCluster model. $7\times 7$ seed windows are
first selected around all possible seeds ($>$0.5 GeV) in the event. They are
separately passed as input to the seed-finder NN predicting
$P_{\mathrm{seedSF}}$ for each seed window. Selected seed windows with
$P_{\mathrm{seedSF}}$ $>$ $P_{\mathrm{seedSF}}^{\mathrm{thr}}$ are combined
into groups of 4 with their neighbors and passed to center-finder NN which
predicts the coordinates , , the energy and a new seed score
$P_{\mathrm{seedCF}}$ for each seed window.

###  Results

In the GNN implementation of the center-finder NN, the model receives
information about all the neighboring seed windows simultaneously. With this
adjustment, the network is able to make a more informed decision for each seed
window and additionally better attribute energy fractions for different
windows. The related improvement is shown in Fig. • where the distribution of
$R_{\mathrm{en}}$ obtained with the single-photon dataset is presented. For
the center-finder GNN (CF GNN) only the seed windows with
$P_{\mathrm{seedCF}}>$ 0.4 are selected. One can notice the disappearance of
the peak at 2 signalling the resolution of the double-counting issue.