# Few-Shot Website Fingerprinting Attack

Mantun Chen, Yongjun Wang, Zhiquan Qin and Xiatian Zhu  M. Chen, Z. Qin and Y.
Wang are with College of Computer, National University of Defense Technology,
Changsha 410073, China. E-mail:
<EMAIL_ADDRESS>Xiatian Zhu is with
University of Surrey, Stag Hill, University Campus, Guildford GU2 7XH, UK.
E-mail<EMAIL_ADDRESS>
###### Abstract

This work introduces a novel data augmentation method for few-shot website
fingerprinting (WF) attack where only a handful of training samples per
website are available for deep learning model optimization. Moving beyond
earlier WF methods relying on manually-engineered feature representations,
more advanced deep learning alternatives demonstrate that learning feature
representations automatically from training data is superior. Nonetheless,
this advantage is subject to an unrealistic assumption that there exist many
training samples per website, which otherwise will disappear. To address this,
we introduce a model-agnostic, efficient, and Harmonious Data Augmentation
(HDA) method that can improve deep WF attacking methods significantly. HDA
involves both intra-sample and inter-sample data transformations that can be
used in harmonious manner to expand a tiny training dataset to an arbitrarily
large collection, therefore effectively and explicitly addressing the
intrinsic data scarcity problem. We conducted expensive experiments to
validate our HDA for boosting state-of-the-art deep learning WF attack models
in both closed-world and open-world attacking scenarios, at absence and
presence of strong defense. For instance, in the more challenging and
realistic evaluation scenario with WTF-PAD based defense, our HDA method
surpasses the previous state-of-the-art results by more than 4% in absolute
classification accuracy in the 20-shot learning case.

###### Index Terms:

User privacy, Internet anonymity, Data traffic patterns, Website
fingerprinting, Deep learning, Neural network, few-shot learning, Data
augmentation.

##  I Introduction

For privacy protection in accessing Internet, an increasing number of users
have turned to anonymous networks and Tor [13] is one of the most popular
choices [12]. However, this remains not completely secure due to exposure of
data transportation pattern before reaching Tor servers. For instance, a local
attacker would eavesdrop the connection between a user and the guard node of
Tor networks, with the attacking positions including any devices in the same
LAN or wireless network, switch, router, and compromised Tor guard node (see
Figure 1). By just analyzing the patterns of data package traffic without
observing the content inside, the attacker is likely to reason about which
website a target user is visiting. This is often known as website
fingerprinting (WF) attack [18].

Figure 1: Illustration of data flow traffic between a user and target websites
with a Tor network in-between. Despite being more secure by anonymity, website
fingerprinting attackers are still able to reason about which website a victim
user is visiting by analyzing the data traffic characteristics at multiple
locations, as specified by red dash lines.

To implement WF attack, the attacker needs to first create a particular
digital fingerprint for every individual website, and then learn some
intrinsic pattern characteristics of these fingerprints for accomplishing
attack. Earlier attacking methods rely on manually designed features based on
expert domain knowledge [18, 25, 5, 17, 28, 7, 38, 37, 15, 27]. They are not
only inflexible but also susceptible to environmental changes over time. This
limitation can now be solved by using more advanced deep learning techniques
[23]. This is because other than utilizing manually designed features, deep
learning methods can learn automatically feature representations directly from
training data, and are hence more scalable provided that up-to-date training
data are accessible. A couple of latest state-of-the-art studies, Deep
Fingerprinting (DF) [32] and Var-CNN [4], have demonstrated this potential in
comparison to manual feature based methods. However, these deep learning
solutions are not perfect, as their success is established upon an unrealistic
assumption that a sufficiently large number (e.g. hundreds) of training
samples per website are available. That is, data hungry. When only a small
training dataset is given as typical in practical use, their performances are
not necessarily superior to traditional methods [37, 15, 27]. It is always
expensive, tedious, or even infeasible to collect a vast training set in
reality due to highly frequent and continuous changes in Internet
environments. Consequently, WF attack is fundamentally a few-shot learning
problem, which nevertheless is largely unrecognized in the literature.

The nature of few-shot WF attack is also considered in the recent Triplet
Fingerprinting method [33], under a condition that there is a large set of
relevant auxiliary training samples for model pre-training. It is essentially
a transfer learning setting. This will limit significantly its scalability in
practice as in-the-wild changes of Internet data traffic conditions would
render such assumptions to be invalid at high probabilities. On the contrary,
we introduce a realistic, generic few-shot WF attack setting where only a
handful of training samples are available for every target website, without
making any domain-specific assumptions. Clearly, Triplet Fingerprinting is not
applicable in our setting due to the need of auxiliary training data.

We summarize the contributions of this paper as follows:

(I) We introduce a novel, practical few-shot website fingerprinting attack
problem, in which only a few training samples are available without rich
auxiliary data. This respects the intrinsic nature of highly dynamic Internet
traffic conditions and high cost of collecting large training data in
practice. Highlighting the importance of few-shot learning without any
auxiliary data assumption for the first time, we hope more future efforts
would be dedicated for solving practically important WF attack challenge.

(II) To solve the proposed few-shot learning challenges, we embrace the
enormous potentials and advantages of deep learning for WF attack by
introducing a new Harmonious Data Augmentation (HDA) method to explicitly
solve the training data scarcity problem in deep learning. Specifically, we
augment the original training data by rotating and masking-out randomly
individual samples and mixing (linearly combining) sample pairs in arbitrary
proportions. With such intra-sample and inter-sample data transformations, our
HDA method can efficiently expand a tiny training dataset at any scales.

(III) We benchmark the performance of few-shot WF attack and demonstrate the
efficacy of our data augmentation method using existing state-of-the-art deep
learning models. In particular, we consider 5-20 shots per website/class in
closed-world and open-world settings, with and without defense. The results
show that our method can improve the performances of previous state-of-the-art
deep learning solutions [4, 16] significantly.

##  II Related Work

###  II-A Objectives, Scenarios and Assumptions

The objective of WF attack is to identify which website a victim user is
interacting with among a set of monitored target websites. Conceptually, it is
a multi-class classification problem with each website regarded as a unique
class. There are several scenarios with different assumptions. The most common
scenario is closed-world attack, where the user is assumed to only visit a set
of known target websites under monitoring. This assumption however is not
realistic, therefore discarded in the open-world scenario. In this scenario,
the victim user is considered to likely visit any websites including those
monitored ones, as typically experienced in real-world applications.

A third scenario considers defense where the user takes some actions to defend
against potential attack. This would lead to higher attack difficulty.
Representative defense techniques include Buflo [14], Tamaraw [6], Walkie-
Talkie [40] and WTF-PAD [20]. Among them, WTF-PAD is used as the mainstream
method for Tor networks due to low bandwidth overhead and zero delay. We
considered WTF-PAD based defense in our evaluations.

In the literature, several common assumptions are made. We briefly discussed
three main assumptions. In user behavior, it is assumed that all Tor users
browsed websites sequentially, only opening a single tab at a time. In
background traffic, it is assumed that the attacker is able to collect all the
clean traces generated by the victim’s visits against dynamic background
traffic. This is increasingly possible as shown in [39] the multiplexed TLS
traffic can be split into individual encrypted connections to each website. In
network condition, the attacker is assumed to have the same conditions as the
victim including traffic condition and settings. To compare with the benchmark
results, we follow these general assumptions for fair evaluations.

Instead, we focus on addressing the following assumption. Often, the attacker
assumes that the training data fall into a similar distribution as the
deployment data. This is a particularly strong and artificial assumption as
the network condition is actually changing and evolving frequently. Such a
property enforces the attacker to update the training data in order to have a
robust attacking model over time. This implies that the attacker is less
possible to collect a large set of training data at each time due to high
acquiring costs. However, existing WF attack methods often ignore this factor
by assuming availability of large training data. In contrast, we study the
largely ignored few-shot learning setting in WF attack. Specifically, we
approach this problem by explicitly solving the small training data issue via
synthesizing new labelled training data.

###  II-B Website Fingerprinting Attack Methods

The first pioneer attack against Tor networks was evaluated by Herrmann [17]
in 2009. It achieved an accuracy of 2.96% using around 20 training samples per
website in the closed-world scenario. Later on, Wang and Ian [38] proposed to
represent the traffic data using more fundamental Tor cells (i.e., direction
data) as a unit rather than TCP/IP packets. This representation is rather
meaningful and informative as it encodes essential characteristics of Tor
data. By training a SVM classifier with distance-based kernel, a ground-
breaking performance with 90.9% accuracy was achieved on 100 sites each with
40 training samples. Recently, Panchenko et al. [27] proposed an idea of
sampling the features from a cumulative trace representation and achieved
91.38% accuracy with 90 training instances per website. Hayes and Danezis [15]
exploited random decision forests to achieve similar results. A common design
of these above methods is a two-stage strategy including feature design and
classifier learning. This is not only constrained by the limitations of hand
features but also lacks interaction between the two stages, making the model
performance inferior.

Motivated by the remarkable success of deep learning techniques in computer
vision and natural language processing [22, 10], several deep learning WF
attack methods have been introduced which can well solve the aforementioned
weakness. This is because deep learning methods by design carry out feature
learning and classification optimization from the raw training data end-to-
end. For example, using VGG network [31] as the backbone, Sirinam et al. [32]
proposed a Deep Fingerprinting attack (DF) model that attains 98.3% accuracy
on 95 websites. However, this method needs a large training set (e.g. 1000
training samples per website), otherwise it will suffer from significant
performance drop. When using 20 training samples per website, DF can only hit
around 19.4% accuracy. To overcome this limitation, Bhat et al. [4] developed
the Var-CNN model based on ResNet [16] and dilated causal convolution [11,
42]. When small training sets (e.g. 100 samples per website) are available, it
achieves superior performance over DF but at dependence on less realistic time
features and less scalable hand-crafted statistical information.

A solution to few-shot learning is a recently proposed triplet fingerprinting
(TF) method [33]. The key idea of TF is to pre-train a metric model that can
measure pairwise distances on new classes. When the pre-training dataset is
similar to the target data in distribution, TF can hit an accuracy of 94.5% on
100 websites using only 20 training samples per website. This is a strong
transfer learning scenario. However, considering that the dynamics of network
conditions is highly unknown and uncontrollable, such a transfer learning
assumption is hardly valid in practice. In light of this observation, in this
work we propose a more realistic few-shot learning setting without assuming
any auxiliary data with similar data characteristics for model pre-training.
Hence it is more scalable and generic for real-world deployments. Under the
proposed more challenging few-shot setting, TF is unable to work properly due
to insufficient network initialization.

###  II-C Data Augmentation

Data augmentation is an important element in deep learning due to its data-
hungry nature [23]. For example, random insertion, random swap, and random
deletion for text classification in natural language processing [41], or
geometric transformations (e.g., flipping, rotation, translation, cropping,
scaling), color space transformations (e.g., color casting, varying
brightness, and noise injection), inter-image mixup [43] for image analysis
[26, 35, 30]. These previous attempts have shown the significance of different
augmenting methods for model performance on the respective tasks. Inspired by
these findings, we investigate extensively the effectiveness of training data
augmentation by adapting existing operations for deep learning WF attack in
few-shot learning settings. To the best of our knowledge, this is the first
attempt of its kind. Crucially, we demonstrate that existing state-of-the-art
deep WF attack method [4] significantly benefits from using the proposed data
augmentation operations in varying evaluation scenarios. This result would be
encouraging and influential for future investigation of deep learning WF
attack methods in particular.

##  III Method

###  III-A Problem Definition

In website fingerprinting (WF) attack, the objective is to detect which
website a target user is visiting. The common observations are data traffic
traces $\bm{x}$ produced by one visit to a website $y$. Taking each website as
a specific class, this is essentially a multi-class classification problem.
For model training, a labelled training set
$D=\\{(\bm{x}_{i},y_{i})\\}_{i=1}^{N}$ is often provided, where
$y_{i}\in\\{1,2,\cdots,K\\}$ specifying one of $K$ target websites. Two
different settings are often considered in model testing: (1) Closed-world
attack where any test sample is assumed to belong one of target
websites/classes, and (2) Open-world attack where the above assumption is
eliminated, i.e., a test trace may be produced by a non-target (unmornitored)
website. The latter is a more realistic setting, yet presenting a more
challenging task as identifying if a test sample falls into target classes or
not is non-trivial.

Feature representation.  For common Tor networks, the raw representation of a
specific traffic trace consists of a sequence of temporally successive Tor
cells travelling between a target user and a website visited. It is derived
from TCP/IP data. Specifically, after those TCP/IP packets retransmitted are
discarded, TLS records are first reconstructed, their lengths are then rounded
down to the nearest multiple of 512 to form the final sequence data $\bm{x}$.
In value, each $\bm{x}$ is a sequence of 1 (outgoing cell) and -1 (incoming
cell), with a variable length. This raw representation is hence known as
direction sample. Besides, temporal information about inter-packet time is
another modality of data used, but limited by high reliance on network
conditions, i.e., not stable and much more noise. Consequently, we mainly
consider the direction data samples in this study, which are more scalable and
generic.

###  III-B Deep Learning for Website fingerprinting Attack

Figure 2: A deep learning pipeline for website fingerprinting attack that
conducts feature representation and website classification end-to-end in a
joint learning manner.

Most of existing WF attack methods rely on hand-crafted feature
representations [18, 25, 5, 17, 28, 7, 38, 37, 15, 27]. This strategy is not
only unscalable but also unsatisfactory in performance due to limited and
incomplete domain knowledge. Deep learning methods provide a viable solution
via learning directly more effective and expressive representation from
training data, as shown in a few recent studies [32, 4]. In this work we
advance this new direction further.

We explore 1D convolutional neural networks (CNN) [21] for WF attack as the
raw data are temporal sequences. Building on the success of deep learning in
computer vision, we adopt the same high-level network designs of standard 2D
CNN models [24] whilst translating them into 1D counterparts. This is similar
to [32, 4].

A CNN model consists of multiple convolutional layers with non-linear
activation functions such as ReLU [2] and fully-connected (FC) layers,
characterized by end-to-end feature extraction and classification. With
convolutional operations, the filters of each layer transform input sequences
using learnable parameters and output new feature sequences. This feature
transformation is conducted layer by layer in a hierarchical fashion. The
receptive field (kernel) with size 3 is often used in each individual layer to
capture local feature patterns. By stacking more layers and pooling operation,
the model can perceive the information of larger regions and achieve
translational invariance. Another effective method for enlarging receptive
field is dilated causal convolutions [11, 42], which has been exploited in
[4].

The feature representations $\bm{f}$ of WF samples are the output of global
average pooling layer on top of the last convolution layer. To obtain the
classification probability vector
$\hat{\bm{y}}=\\{\hat{y}_{1},\hat{y}_{2},\cdots,\hat{y}_{K}\\}\in\mathcal{R}^{K}$
over $K$ target classes, $\bm{f}$ is fed into a FC layer and normalized by a
softmax function.

For model training, we compute a cross-entropy objective loss function with
the classification vector against the ground-truth class label over all $N$
training samples as:

$\mathcal{L}=\sum_{i=1}^{N}\sum_{j=1}^{K}\delta(j=y_{i})\log\hat{y}_{i,j}$ (1)

where $y_{i}$ refers to the ground-truth class label of a training sample
$\bm{x}_{i}$, and $\delta()$ is a Dirac function. The objective is to maximize
the probability of the ground-truth class in prediction. This loss function is
differentiable, with its gradients backpropogated to update all the learnable
model parameters.

Once the deep model is trained, we forward a given test sample, obtain a
classification probability vector, and take the most likely class as
prediction in both closed-world and open-world settings. For open-world
setting, all unmornitored websites are considered to belong to a background
class.

Discussion.  While deep learning techniques have advanced significantly in the
last several years, it is still assumed that a large set of labelled training
samples is available. This is not always true, for example, for the WF attack
problems. In real-world applications, an attacker is usually faced with highly
dynamic network environments. It means that the distribution of raw features
is evolving continuously. As such, the training data need to update
frequently, which disables collection of large training data with labels in
practice due to prohibitively high labelling costs. Consequently, only a small
training set is accessible in reality, making deep learning methods
ineffective.

###  III-C Harmonious Website Fingerprinting Data Augmentation

To address the above small training data challenge, we propose an intuitive,
novel harmonious data augmentation (HDA) method. We introduce both intra-
sample and inter-sample augmentation operations that can be applied in a joint
and harmonious manner for more effective data expansion.

Intra-sample augmentation.  The key idea of intra-sample augmentation is that
given an individual training sample, we introduce a certain degree of random
data perturbation and/or variation whilst keeping the same class labels. Doing
so allows us to generate an infinite number of labelled training samples due
to the nature of randomness. We consider two perturbation operations: random
rotation and random masking.

Random rotation based data augmentation means rotating an original training
sample forward or backward by random steps to generate virtual samples (Fig.
3(a)):

$Rotate(\bm{x},\;n_{\text{step}},\;\text{dir})$ (2)

where $n_{\text{step}}$ and $\text{dir}\in\\{\text{forward, backward}\\}$
specify the steps and the direction to rotate on an input sample $\bm{x}$. The
hypothesis behind is that class-sensitive information encoded in a sample is
distributed across different sub-sequences and data traffic order is less
important than signal patterns. After a sampled is rotated, the original class
information is largely preserved, i.e., semantically invariant. Hence, the
same class can be annotated for the rotated variants. However, this hypothesis
is more likely to stand under some certain (unknown) degrees. We therefore
introduce an upper bound parameter $R_{\text{max}}$ so that the rotation range
is limited at most $R_{\text{max}}$ steps in both directions,
$n_{\text{step}}\leq R_{\text{max}}$.

Figure 3: Illustration of our data augmentation operations for deep learning
WF attack, including (a) random rotation, (b) random masking, and (c) random
mixing.

In contrast, random masking introduces localized corruption to an original
training sample by setting a random subsequence to zero (Fig. 3(b)). This data
augmentation is written as:

$Mask(\bm{x},\;n_{\text{len}},\;\text{loc})$ (3)

where $n_{\text{len}}$ and loc denote the length and location of the
subsequence that is masked out from an original sample $\bm{x}$. Rather than
in form of subsequence, another strategy is to randomly select individual
positions to mask. We consider this may introduce more significant corruption
to the underlying semantic information.

Conceptually, random masking simulates varying traffic measurement errors in
data transportation. Meanwhile, with the same above hypothesis, such masking
would not dramatically change the semantic class information, provided that
the masking is subject to some limit, e.g., the length of subsequences masked
out $M_{\text{len}}$. It hence offers a complementary data perturbation choice
w.r.t. random rotation.

Inter-sample augmentation.  Apart from data augmentation on individual
samples, we further introduce data perturbation across two different samples
to enrich the limited training set.

We propose random mixing that generates virtual samples and class labels by
linear interpolation between two original samples $\bm{x}_{i}$ and
$\bm{x_{j}}$ as:

$\displaystyle\bm{\tilde{x}}=\lambda\bm{x}_{i}+(1-\lambda)\bm{x}_{j}$ (4)
$\displaystyle\bm{\tilde{y}}=\lambda\bm{y}_{i}+(1-\lambda)\bm{y}_{j}$ (5)

where $(\bm{y}_{i},\bm{y_{j}})$ are the one-hot class labels of $\bm{x}_{i}$
and $\bm{x_{j}}$. The mixing parameter $\lambda\in[0,1]$ follows a Beta
distribution: $\lambda\sim\beta(\alpha,\alpha)$ with $\alpha>0$ the parameter
that controls the strength of interpolation. This is in a similar spirit of
mixup in image understanding domain [43]. Unlike intra-sample augmentation
above, random mixing changes the semantic class information, since original
samples may be drawn from different classes. It simplifies the data
distribution by imposing a linear relationship between classes for complexity
minimization. While seemingly counter-intuitive, we will show that such a
method brings positive contributions on top of random masking and random
rotation.

Combination and compatibility.  Different augmentation operations can be
applied on the same samples without conflict to each other in a harmony. There
is also no particular constraint on the order of applying all the three data
augmentation operations in a combination. Given a fixed set of parameters as
discussed above, different augmentation orders will result in different
virtual samples. This makes little conceptual difference as the space of
sample is just infinite.

Augmentation optimization.  In our harmonious data augmentation (HDA), three
hyper-parameters $\\{R_{\text{max}},M_{\text{len}},\alpha\\}$ are introduced.
To generate meaningful virtual samples, obtaining their optimal values is
necessary otherwise adversarial effects may even be imposed.

Instead of manual tuning, we adopt an automatic Bayesian estimator, called
Tree of Parzen Estimators (TPE) [3]. The conventional TPE can take only one
parameters alone at a time. So we need to optimize each of the three hyper-
parameters independently. This differs from our data augmentation process
where the three augmentation operations are typically applied together, making
the independently tuned parameters of TPE sub-optimal. This is because,
jointly applying three augmentations together makes them inter-dependent.

For solving this problem, we propose a sequential optimization process that
takes into account the inter-dependence property of different augmentation
operations gradually (see Alg. 1). Specifically, we start with a random, fixed
order of applying our random rotation, masking, and mixing operations. Then,
we optimize from the first one with TPE, move to next one with all the
previous ones optimized and fixed, stop by finishing the last one. Each time,
we still optimize a single hyper-parameter whilst keeping all the previous
optimized ones fixed. In this way, we expand the inter-dependence among
different operations sequentially.

Algorithm 1 Data augmentation optimization

0: A training $\\{\bm{X_{t}},\bm{Y_{t}}\\}$, and validation
$\\{\bm{X_{v}},\bm{Y_{v}}\\}$ set.

0: Data augmentation with optimal parameters $B_{\text{aug}}$.

1: Setting $B_{\text{aug}}$ = $\phi$ (empty set);

2: Sequencing data augmentation operations randomly;

3: while Enumerating augmentation operations do

4: Get the search space $S_{\text{aug}}$ of current augmentation $A$;

5: Using TPE on $S_{\text{aug}}$ to obtain the optimal parameter
$b_{\text{aug}}$, with the model trained by $B_{\text{aug}}$ and $A$;

6: $B_{\text{aug}}=B_{\text{aug}}\cup b_{\text{aug}}$

7: end while

8: return $B_{\text{aug}}$

###  III-D Theoretical Foundation and Formulation

The objective of learning a WF attack model is equivalent to derive a function
$h\in H$ that fits the latent translation relationship between raw feature
vectors $\bm{x}\in X$ and corresponding website class labels $\bm{y}\in Y$.
That is, fitting a joint distribution $P(X,Y)$. To this end, in deep learning
we often leverage a loss function $L$ defined to penalize the differences
between predictions $h(\bm{x})$ and targets $\bm{y}$. We minimize the average
loss over the joint distribution:

$R(h)=\int L(h(\bm{x}),\bm{y})\;dP(\bm{x},\bm{y})$ (6)

which is known expected risk minimization[36].

However, the joint distribution is often unknown, particularly for WF attack
with small training data. Given a limited training data set
$D=\\{(\bm{x}_{i},\bm{y}_{i})\\}_{i=1}^{N}$, the joint distribution can only
be approximated by an empirical distribution as:

$P_{\delta}(\bm{x},\bm{y})=\frac{1}{N}\sum^{N}_{i=1}\delta(\bm{x}=\bm{x}_{i},\bm{y}=\bm{y}_{i})$
(7)

where $\delta(\bm{x}=\bm{x}_{i},\bm{y}=\bm{y}_{i})$ is a Dirac mass centered
at a sample $(\bm{x}_{i},\bm{y}_{i})$. Accordingly, the expected risk can now
be approximated by an empirical risk:

$\displaystyle R_{\delta}(h)$ $\displaystyle=\int
L(h(\bm{x}),\bm{y})dP_{\delta}(\bm{x},\bm{y})$ (8)
$\displaystyle=\frac{1}{N}\sum^{N}_{i=1}L(h(\bm{x}_{i}),\bm{y}_{i})$

The above approximation is in the empirical risk minimization (ERM) principle
[36]. The cross-entropy loss (Eq. (1)) is a representative example, which
essentially minimizes $R_{\delta}(h)$ for the classification task.

While ERM is a common strategy, it suffers from high risk of poor
generalization due to the tendency of memorization, particularly when a large
model is used [34]. To mitigate this issue, we adopt the notion of vicinal
distribution [8] which can better approximate the true joint distribution. In
particular, the vicinal distribution $P_{\upsilon}$ in the data space is
defined as:

$P_{\upsilon}(\bm{\tilde{x}},\bm{\tilde{y}})=\frac{1}{n}\sum^{n}_{i=1}\upsilon(\bm{\tilde{x}_{i}},\bm{\tilde{y}_{i}}\mid\bm{x}_{i},\bm{y}_{i})$
(9)

Intuitively, $P_{\upsilon}$ measures the probability of finding a virtual
labelled sample $(\bm{\tilde{x}},\bm{\tilde{y}})$ in the vicinity around an
original training sample $(\bm{x}_{i},\bm{y}_{i})$.

Given such vicinal distributions, we first construct a virtual dataset
$D_{\upsilon}:={(\tilde{\bm{x}}_{i},\tilde{\bm{y}}_{i})}^{m}_{i=1}$ by
sampling $P_{\upsilon}$ randomly, and then minimize an empirical vicinal risk
to learn $h$ as:

$R_{\upsilon}(h)=\frac{1}{m}\sum^{m}_{i=1}L(h(\bm{\tilde{x}}_{i}),\bm{\tilde{y}}_{i})$
(10)

Clearly, at the core of this strategy is performing data augmentation around
original training samples. Rather than computing a loss value for every single
training sample, it derives a local distribution centered at each individual
sample and generates more virtual training samples to reduce the negative
memorization effect of deep learning This is the key rationale of our data
augmentation method.

Augmentation formulation.  We formulate the proposed harmonious data
augmentation operations in the vicinal distribution manner. For intra-sample
augmentation (including random rotation and masking), the vicinal distribution
is defined as

$\upsilon(\bm{\tilde{x}},\bm{\tilde{y}}\mid\bm{x},\bm{y})=T(\bm{x})\delta(\bm{\tilde{y}}=\bm{y})$
(11)

where $T()$ is a transformation operator.

For random rotation, given any length-$n$ sample
$\bm{x}=\\{x_{0},...,x_{i},...,x_{n-1}\\}$, we first define a circle matrix
$B$ for forward rotation as:

$B(\bm{x})=\left[\begin{smallmatrix}x_{0}&x_{1}&\cdots&x_{n-1}\\\
x_{n-1}&x_{0}&\cdots&x_{n-2}\\\ \vdots&\vdots&&\vdots\\\
x_{n-1}&x_{n-2}&\cdots&x_{0}\\\ \end{smallmatrix}\right]$ (12)

Then we sample the step size $n_{\text{step}}$ uniformly from a range of
$\\{1,\cdots,R_{\text{max}}\\}$. By one-hot representation of
$n_{\text{step}}$, we can obtain a rotation transformation as:

$T_{\text{rot}}(\bm{x})=\text{one-hot}(n_{\text{step}})B(\bm{x})$ (13)

For the backward case, we perform the same process as above but with a
backward rotation matrix instead.

For random masking, we similarly sample the start position $s$ uniformly in
the range of $\\{1,\cdots,n-n_{\text{len}}\\}$ where $n_{\text{len}}$ is the
length of masked subsequence. The masking transformation can be represented by
a matrix as:

$M_{\text{mask}}=diag\Big{(}\bm{1}-\sum^{s+n_{\text{len}}}_{i=s}{Row_{i}(I)}\Big{)}$
(14)

where $I$ is identity matrix, $\bm{1}$ is all-one vector, $Row_{i}()$ selects
the $i$-th row of a matrix, $diag()$ transforms a vector to a diagonal matrix.
Masking operation is finally conducted by matrix multiplication as:

$T_{\text{mask}}(\bm{x})=\bm{x}M_{\text{mask}}$ (15)

For inter-sample augmentation, random mixing in our case, the vicinal
distribution is defined as:

$\displaystyle\upsilon(\bm{\tilde{x}},\bm{\tilde{y}}\mid\bm{x}_{i},\bm{y}_{i},\bm{x_{j}},\bm{y_{j}})=$
(16)
$\displaystyle\delta\big{(}\bm{\tilde{x}}=\lambda\cdot\bm{x}_{i}+(1-\lambda)\cdot\bm{x_{j}},\;\;\bm{\tilde{y}}=\lambda\cdot\bm{y}_{i}+(1-\lambda)\cdot\bm{y_{j}}\big{)}$

where $\lambda$ is a random variable drawn from a Beta distribution
$\beta(\alpha,\alpha)$ and $\bm{y}$ is one-hot class label vector. This local
vicinity is assumed to respect a linear structure w.r.t. class labels.

##  IV Experiments

###  IV-A Experimental Setup

Datasets.  We evaluated our data augmentation method HDA on four standard WF
attack datasets as below. (1) CW100 [29]: This dataset provides a total of 100
monitored target websites each with 2,500 raw feature traces. (2) DF95,Nodef
[32]: This dataset gives 95 monitored websites with each contributing 1,000
feature traces. (3) ROWUM400,000 [29]: This dataset includes CW100 and a large
set of samples each was generated by a visit to a page of top 400,000 Alexa
websites. (4) DF95,wtf-pad [32]: Unlike all the above datasets, this is a more
challenging dataset due to presence of WTF-PAD based defense against WF
attack. It has the same size as DF95,Nodef, i.e., 95,000 raw feature samples
from 95 websites. We considered both closed-world and open-world WF attack
scenarios using the above datasets.

Network architectures.  We used four different network architectures for
testing the generic benefits of the proposed HDA method. (1) Var-CNN [4] is
the current state-of-the-art deep learning WF method. (2) Var-CNN∗ is an
improved variant with two more fully-connected layers added to the classifier.
(3/4) ResNet-18 and ResNet-34 [16] are two strong and popular networks widely
deployed in many different fields such as computer vision.

Implementation details.  We conducted our experiments in Keras [9]. In all our
experiments, we used the standard training, validation, and test splits for
all competitors for fair comparisons. We optimized HDA’s hyper-parameters
using Var-CNN [4] as deep learning model on $CW_{100}$ in closed-world setting
and applied the same parameter setting for all the other deep learning
methods, datasets and settings. This allows testing the generality and
scalability of our HDA method. For augmentation optimization, we set the
search space as: $1\sim 20$ with step 5 for forward/backward $R_{\text{max}}$
(random rotation) $1\sim 200$ with step 20 for $M_{\text{len}}$ (random
masking), $[0,1]$ with step 0.1 for $\alpha$ (random mixing). The optimal
parameter values we obtained are $R_{\text{max}}=20$, $M_{\text{len}}=180$,
and $\alpha=0.1$. For saving storage, we performed online data augmentation
within each mini-batch without any data pre-processing. In each experiment, we
trained every deep learning model for 150 epochs and used the checkpoint with
best performance on the validation set for model test. We only used the
direction feature data, without time sequences and hand-crafted features. We
run each experiment by 10 times and reported the mean results and standard
deviation as the final performance.

TABLE I: Results of closed-world WF attack on CW${}_{\text{100}}$. Metrics:
Accuracy.

Method | 5-shot | 10-shot | 15-shot | 20-shot
---|---|---|---|---
CUMUL [27] | $72.2\pm 1.7$ | $79.7\pm 1.4$ | $83.3\pm 2.0$ | $85.9\pm 0.6$
k-FP [15] | $79.3\pm 1.0$ | $83.9\pm 1.0$ | $85.9\pm 0.6$ | $87.5\pm 0.8$
ResNet-18 [16] | $13.9\pm 0.6$ | $23.0\pm 1.1$ | $35.5\pm 2.5$ | $51.3\pm 1.7$
ResNet-34 [16] | $14.5\pm 0.7$ | $24.3\pm 1.5$ | $40.3\pm 3.1$ | $51.3\pm 86.4$
Var-CNN [4] | $17.9\pm 1.5$ | $41.4\pm 4.0$ | $65.6\pm 1.9$ | $78.7\pm 1.5$
Var-CNN∗ [4] | $34.8\pm 2.6$ | $57.4\pm 3.9$ | $71.9\pm 2.4$ | $80.8\pm 2.4$
ResNet-18+HDA | $34.3\pm 4.3$ | $61.5\pm 9.4$ | $77.6\pm 5.8$ | $84.6\pm 4.3$
ResNet-34+HDA | $34.8\pm 6.2$ | $62.3\pm 8.1$ | $78.8\pm 7.1$ | $86.4\pm 2.8$
Var-CNN+HDA | $59.7\pm 1.5$ | $74.7\pm 2.6$ | $86.4\pm 1.3$ | $90.7\pm 0.8$
Var-CNN∗+HDA | $71.3\pm 4.7$ | $90.2\pm 0.6$ | $93.3\pm 0.2$ | $94.1\pm 0.5$

TABLE II:  Results of closed-world WF attack on DF95,Nodef. Metrics: Accuracy.

Method | 5-shot | 10-shot | 15-shot | 20-shot
---|---|---|---|---
ResNet-18 [16] | $14.3\pm 1.0$ | $26.3\pm 1.6$ | $41.1\pm 1.9$ | $54.3\pm 1.1$
ResNet-34 [16] | $16.3\pm 1.3$ | $29.5\pm 4.7$ | $41.4\pm 3.2$ | $54.2\pm 2.6$
Var-CNN [4] | $22.6\pm 3.4$ | $50.0\pm 2.1$ | $75.6\pm 1.5$ | $75.2\pm 1.7$
Var-CNN∗ | $19.4\pm 1.6$ | $29.6\pm 2.7$ | $42.5\pm 5.0$ | $53.6\pm 5.2$
ResNet-18+HDA | $53.9\pm 7.0$ | $77.3\pm 3.6$ | $82.8\pm 2.9$ | $88.3\pm 1.1$
ResNet-34+HDA | $44.0\pm 6.8$ | $74.3\pm 7.4$ | $84.2\pm 3.1$ | $88.7\pm 1.6$
Var-CNN+HDA | $71.0\pm 2.3$ | $84.3\pm 1.4$ | $88.6\pm 0.9$ | $91.4\pm 0.5$
Var-CNN∗+HDA | $62.3\pm 4.8$ | $84.3\pm 0.9$ | $87.7\pm 0.7$ | $91.0\pm 0.5$

###  IV-B Closed-World WF Attack

Setting.  We conducted the closed-world attack on CW100 and DF95,Nodef. We
separated each dataset into training, validation (10 samples per class), and
test (70 samples per class) splits. We considered few-shot settings with
$n\in\\{5,10,15,20\\}$ training samples per class. The validation set was used
to select the best performing model for test. We used the classification
accuracy as performance metric. Besides deep network models, we also compared
our method with two conventional hand-crafted feature based methods: CUMUL
[27] and k-FP [15].

TABLE III: Results of open-world WF attack on CW${}_{\text{100}}$ (target classes) + ROWWUM${}_{\text{400,000}}$ (non-target classes). Pre: Precision, Rec: Recall. We reported two settings: one is tuned for best precision (top), and one for recall (bottom).  Method | Tuned for Precision
---|---
5-shot | 10-shot | 15-shot | 20-shot
Pre | Rec | Pre | Recall | Pre | Rec | Pre | Rec
ResNet-18 [16] | 28.7 | 0.8 | 30.0 | 7.6 | 49.8 | 8.7 | 65.9 | 18.3
ResNet-34 [16] | 32.5 | 4.7 | 42.1 | 6.6 | 50.4 | 21.8 | 61.3 | 39.6
Var-CNN [4] | 39.7 | 2.7 | 58.8 | 9.2 | 74.2 | 35.8 | 78.0 | 54.4
Var-CNN+ | 44.7 | 3.9 | 58.9 | 17.9 | 64.0 | 49.9 | 66.1 | 65.3
ResNet-18+HDA | 80.7 | 0.2 | 89.6 | 7.2 | $\mathbf{91.7}$ | 30.7 | $\mathbf{93.4}$ | 46.6
ResNet-34+HDA | 72.7 | 0.8 | $\mathbf{91.7}$ | 8.7 | 91.5 | 43.8 | 92.6 | 55.1
Var-CNN+HDA | 77.4 | 6.9 | 91.2 | 47.2 | 91.4 | 64.3 | 92.9 | 66.6
Var-CNN∗+HDA | $\mathbf{92.3}$ | 5.4 | 87.8 | 59.7 | 88.7 | 73.0 | 90.5 | 76.1
| Tuned for Recall
ResNet-18 [16] | 10.0 | 19.8 | 17.0 | 34.0 | 24.8 | 49.5 | 32.3 | 64.3
ResNet-34 [16] | 12 | 25.6 | 19.2 | 37.8 | 30.2 | 54.5 | 34.9 | 68.6
Var-CNN [4] | 13.9 | 27.1 | 26.7 | 52.8 | 37.4 | 73.9 | 42.2 | 82.6
Var-CNN+ | 16.3 | 31.9 | 27.6 | 54.8 | 37.0 | 73.4 | 41.1 | 81.4
ResNet-18+HDA | 20.0 | 36.4 | 36.3 | 71.5 | 44.5 | 87.4 | 47.2 | 91.0
ResNet-34+HDA | 21.1 | 37.4 | 36.2 | 68.9 | 45.7 | 89.2 | 47.2 | 91.4
Var-CNN+HDA | 28.0 | $\mathbf{55.8}$ | 46.9 | 88.7 | 49.3 | 92.2 | 49.1 | 92.5
Var-CNN∗+HDA | 25.6 | 48.1 | 45.7 | $\mathbf{89.3}$ | 47.9 | $\mathbf{92.8}$ | 47.8 | $\mathbf{94.1}$

Results.  The results of different methods are compared in Table I and Table
II. We have the following observations: (1) Hand-crafted feature based methods
(CUMUL and k-FP) remain competitive, especially very few samples per class are
available. The best 5-shot result on CW${}_{\text{100}}$ is achieved by k-FP,
with a moderate edge of 8% over Var-CNN∗+HDA. (2) However, deep learning
methods (Var-CNN∗) becomes clearly stronger when a few more training samples
are accessible, suggesting a great deal of potentials. Among previous methods,
in 20-shot case Var-CNN∗ achieves the best result on CW${}_{\text{100}}$ and
DF95,Nodef. (3) With our HDA method for training data augmentation, every deep
learning method improves significantly. For example, the 5-shot accuracy of
Var-CNN is increased from 17.9% to 59.7% on CW${}_{\text{100}}$, and 22.6% to
71.0% on DF95,Nodef. Similarly, the accuracy of ResNet-18 is improved from
13.9% to 34.3% on CW${}_{\text{100}}$, and from 14.3% to 53.9% on DF95,Nodef.
Var-CNN and its variant benefit incredibly more, implying a higher demand for
larger training data to avoid model overfit. Similar effects are shown for
10/15/20-shot cases. (4) Our HDA can consistently improve different methods on
varying datasets, suggesting good generality. (5) The performance deviation of
Var-CNN and its variant assisted by our method HDA is the least among all the
competitors, implying strong stability.

###  IV-C Open-World WF Attack

Setting.  We conducted the open-world attack experiments on the combination of
ROWWUM${}_{\text{400,000}}$ and CW${}_{\text{100}}$. We treat the websites of
CW${}_{\text{100}}$ as target (monitored) classes, and those of
ROWWUM${}_{\text{400,000}}$ as non-target (unmonitored) classes. In this test,
we selected randomly 8,020 out of 400,000 unmonitored websites, and separated
them into three disjoint sets sized at 20/1,000/7,000 for training,
validation, and test, respectively. In this scenario, the precision and recall
rates were used to evaluate model performance due to the need for detecting
non-target classes [19]. We considered the same four deep learning methods
(ResNet-18, Resnet-34, Var-CNN [4] and its variant Var-CNN∗) for comparisons.

Results.  The results of different methods are reported in Table III. We
considered two settings, one is tuned for best precision, and one for best
recall. Overall, we obtained similar trends as above that our HDA is highly
effective for improving both deep learning methods. It is noted that unlike
the closed-world scenario, VarCNN∗+HDA achieves very top result at most cases
under both tuning settings even if it may not be the best one. Similarly,
VarCNN∗+HDA remains to be more stable and less sensitive to training sample
size. Importantly, our HDA method further enhances this strengths by efficient
data augmentation, leading to a more robust WF attack solutions.

###  IV-D WF Attack against Defense

Setting.  In contrast to the two above experiments, we further tested a more
challenging WF attack scenario with defense involved. Defense changes the data
traffic patterns to be more similar to each other, therefore making the attack
more difficult. We considered the most popular defense, WTF-PAD, widely
deployed in Tor networks. We used the DF${}_{\text{95,wtf-pad}}$ dataset in
this experiment. We used 100 random samples per website, and separated them
into three sets for training (20 samples), validation (10 samples), and test
(70 samples), respectively. We reported the classification accuracy as
performance metric in closed-world scenario. We compared with previous four
deep learning (ResNet-18, Resnet-34, Var-CNN [4] and its variant Var-CNN∗) and
hand-crafted feature based methods (k-NN [37], SDAE [1], k-FP [15], CUMUL
[27], AWF [29]).

TABLE IV:  Results of closed-world WF attack with WTF-PAD based defense on
DF95,wtf-pad. Metrics: Accuracy.

Method | 5-shot | 10-shot | 15-shot | 20-shot
---|---|---|---|---
k-NN [37] | – | – | – | 16.0
SDAE [1] | – | – | – | 36.9
k-FP [15] | – | – | – | 57.0
CUMUL [27] | – | – | – | 60.3
AWF [29] | – | – | – | 60.8
ResNet-18 [16] | $7.3\pm 0.3$ | $9.8\pm 0.6$ | $11.4\pm 0.4$ | $14.2\pm 0.6$
ResNet-34 [16] | $7.4\pm 0.5$ | $9.4\pm 0.7$ | $13.3\pm 1.3$ | $12.3\pm 1.2$
Var-CNN [4] | $6.6\pm 0.3$ | $9.2\pm 0.7$ | $12.5\pm 0.8$ | $19.2\pm 1.7$
Var-CNN∗ | $7.5\pm 0.4$ | $9.8\pm 0.6$ | $11.5\pm 1.0$ | $15.1\pm 1.3$
ResNet-18+HDA | $12.9\pm 1.2$ | $27.9\pm 2.8$ | $35.2\pm 2.9$ | $40.7\pm 3.4$
ResNet-34+HDA | $12.3\pm 1.9$ | $28.1\pm 3.7$ | $38.2\pm 6.2$ | $47.7\pm 5.1$
Var-CNN+HDA | $25.3\pm 2.2$ | $46.9\pm 1.9$ | $48.7\pm 1.4$ | $63.2\pm 1.8$
Var-CNN∗+HDA | $26.0\pm 4.7$ | $48.5\pm 2.6$ | $59.7\pm 1.9$ | $65.4\pm 0.7$

Results.  We reported the results of closed-world WF attack under WTF-PAD
based defense in Table IV. We made the following observations. (1) Some hand-
crafted feature based methods (CUMUL, AWF) are superior over recent deep
learning methods (ResNet-18, ResNet-34) at the few-shot learning scenarios.
This is mainly because the latter suffers from lacking enough training
samples, resulting in model overfitting. (2) Using our HDA for training data
augmentation, we can directly solve the data scarcity problem and
significantly boost the performances of previous deep learning methods. As a
result, both Var-CNN+HDA and Var-CNN∗+HDA outperform the other competitors by
a large margin, e.g., 4.6% and 2.4% gap over the best competitor CUMUL. (3)
ResNet-34 is surpassed by Var-CNN and its variant dramatically. By benefiting
more from our data augmentation, Var-CNN finally achieves the best results
across all different shot cases. This implies that Var-CNN has higher desire
for large training data with higher performance potential, as compared to
ResNet-34.

###  IV-E Ablation Studies

We carried out a set of component analysis experiments to examine the exact
effect of different designs of our method (HDA). We adopted the most common
closed-world attack scenario without defense on the CW${}_{\text{100}}$
dataset, following the same setting as Section IV-B. It is noteworthy that
this dataset CW${}_{\text{100}}$ is different from the dataset in Section IV-B
because they are different subsets. In this section, we evaluated the 15-shot
learning case in particular, using Var-CNN [4] as the deep learning model
backbone.

TABLE V: Effect of individual augmentation operations.  Augmentation Operation | Accuracy
---|---
None | $75.0\pm 3.0$
Random Rotation | $92.4\pm 0.3$
Random Masking | $92.4\pm 0.8$
Random Mixing | $86.7\pm 0.6$
Random Rotation + Masking | $92.7\pm 0.4$
Random Rotation + Mixing | $92.6\pm 0.7$
Random Masking + Mixing | $93.4\pm 0.7$
Random Rotation + Masking + Mixing (HDA) | $93.5\pm 0.4$

Individual augmentation operations.  Recall that our data augmentation method
(HDA) consists of three different operations (random rotation, masking, and
mixing). We have demonstrated their performance advantages of them as a whole
in varying test settings above. For in-depth insights, examining their
individual contributions would be informative and necessary, as well as
different combinations. We conducted this experiments with an exhaustive set
of operation combinations and reported the results in Table V.

It is observed that: (1) Each of the three operations makes significant
difference in performance, with rotation and masking the best individual
operations that improve the classification accuracy by 17.4%. (2) When jointly
using any two augmentation operations, the performance can be further
increased. The combination of masking and mixing gives the highest accuracy
among them. (3) Combining all the three operations (HDA) achieves the best
result with smaller deviation. This suggests that all different operations are
complementary and compatible to each other.

Augmentation optimization.  For optimal data augmentation, we propose a
sequential optimization strategy (see Alg. 1) for capturing the inter-
dependence between different augmentation operations applied. To evaluate its
effect, we compared with a baseline algorithm that independently optimizes
each augmentation parameter.

TABLE VI:  Effect of augmentation optimization.  Augmentation Optimization | Accuracy
---|---
Independent | $92.1\pm 0.5$
Sequential (Ours) | $93.5\pm 0.4$

As shown in Table VI, the proposed optimization algorithm (see Alg. 1) is
clearly superior, validating our consideration that there exist inter-
dependence between different augmentation operations when applied jointly on
the same samples. Note that we obtained this performance gain at the same cost
as the baseline counterpart. Besides, it is worth noting that even with the
simpler optimization, our data augmentation method (HDA) can still greatly
improve the previous deep learning model Var-CNN, and achieve new state-of-
the-art results (Table VI vs. Table I). This further validates that the
proposed augmentation operations are highly compatible with one another and
can be applied together well.

##  V Conclusion

We presented a model-agnostic, simple yet surprisingly effective data
augmentation method, called HDA, for few-shot website fingerprinting attack.
This is an under-studied and realistically critical problem, as in practice
only a handful of training samples per website can be feasibly collected due
to the inherent high dynamics of internet networks and expensive label
collection cost. Importantly, we focus on deep learning based methods, a line
of new research efforts with vast potentials for future investigations. In
particular, our HDA method offers three different data augmentation
operations, including random rotation, masking, and mixing in intra-sample and
inter-sample fashion. They can be applied to the same training samples
harmoniously with high complement and compatibility. Moreover, we introduce a
sequential augmentation parameter optimization method that captures the inter-
dependence nature between different operations when applied jointly. With
recent state-of-the-art deep learning WF attack models, we conducted extensive
experiments on four benchmark datasets to validate the efficacy of our HDA
method in both closed-world and open-world scenarios, with and without
defense. The results show that the proposed data augmentation method makes
dramatic differences in performance and enables previous deep learning methods
to outperform hand-crafted feature based counterparts in the few-shot learning
setting for the first time, often by a large margin. This is achieved without
making any artificial assumptions of relevant, large auxiliary training data
for model pre-training. With our HDA method, collecting large training data
frequently is eliminated, whilst still achieving stronger and more robust WF
attack. Finally, we performed detailed component analysis to diagnose the
effect of individual model components.

## References

  * [1] K. Abe and S. Goto, “Fingerprinting attack on tor anonymity using deep learning,” in _Proceedings of the Asia-Pacific Advanced Network Research Workshop_ , 2016, pp. 15–20.
  * [2] A. F. Agarap, “Deep learning using rectified linear units (relu),” _arXiv preprint arXiv:1803.08375_ , 2018.
  * [3] J. Bergstra, R. Bardenet, B. Kégl, and Y. Bengio, “Algorithms for hyper-parameter optimization,” in _NeurIPS’24: Procedding of the 24th Neural Information Processing Systems_ , 2011, Conference Proceedings.
  * [4] S. Bhat, D. Lu, A. Kwon, and S. Devadas, “Var-cnn: A data-efficient website fingerprinting attack based on deep learning,” _Proceedings on Privacy Enhancing Technologies_ , vol. 2019, no. 4, pp. 292–310, 2019.
  * [5] D. G. Bissias, M. Liberatore, D. Jensen, and B. N. Levine, “Privacy vulnerabilities in encrypted http streams,” in _Privacy Enhancing Technologies_ , vol. 3856, 2006, pp. 1–11.
  * [6] X. Cai, R. Nithyanand, T. Wang, R. Johnson, and I. Goldberg, “A systematic approach to developing and evaluating website fingerprinting defenses,” in _Proceedings of the ACM Conference on Computer and Communications Security_ , 2014, p. 227–238.
  * [7] X. Cai, X. C. Zhang, B. Joshi, and R. Johnson, “Touching from a distance: website fingerprinting attacks and defenses,” in _Preceeding of the ACM Conference on Computer and Communications Security_ , 2012, pp. 605–616.
  * [8] O. Chapelle, J. Weston, L. Bottou, and V. Vapnik, “Vicinal risk minimization,” in _Neural Information Processing Systems_ , 2000, pp. 416–422.
  * [9] F. Chollet et al, “Keras,” https://keras.io, 2015.
  * [10] R. Collobert, J. Weston, L. Bottou, M. Karlen, K. Kavukcuoglu, and P. P. Kuksa, “Natural language processing (almost) from scratch,” _Journal of Machine Learning Research_ , vol. 12, pp. 2493–2537, 2011.
  * [11] A. V. Den Oord, S. Dieleman, H. Zen, K. Simonyan, O. Vinyals, A. Graves, N. Kalchbrenner, A. Senior, and K. Kavukcuoglu, “Wavenet: A generative model for raw audio,” _arXiv preprint arXiv:1609.03499_ , 2016.
  * [12] T. Developers, “Tor metrics portal.” https://metrics.torproject.org, 2018\.
  * [13] R. Dingledine, N. Mathewson, and P. Syverson, “Tor: The second-generation onion router,” in _Proceedings of the 13th USENIX Security Symposium_ , 2004, pp. 303–320.
  * [14] K. P. Dyer, S. E. Coull, T. Ristenpart, and T. Shrimpton, “Peek-a-boo, i still see you: Why efficient traffic analysis countermeasures fail,” in _Proceedings of the 33rd Annual IEEE Symposium on Security and Privacy_ , 2012, pp. 332–346.
  * [15] J. Hayes and G. Danezis, “k-fingerprinting: a robust scalable website fingerprinting technique,” in _Proceddings of the 25th USEUIX Security Symposium_ , 2016, pp. 1187–1203.
  * [16] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _Computer Vision and Pattern Recognition_ , 2016, pp. 770–778.
  * [17] D. Herrmann, R. Wendolsky, and H. Federrath, “Website fingerprinting: Attacking popular privacy enhancing technologies with the multinomial naïve-bayes classifier,” in _IEEE International Conference on Cloud Computing Technology and Science_ , 2009, pp. 31–42.
  * [18] A. Hintz, “Fingerprinting websites using traffic analysis,” in _Privacy Enhancing Technologies_ , 2003, pp. 171–178.
  * [19] M. Juarez, S. Afroz, G. Acar, C. Diaz, and R. Greenstadt, “A critical evaluation of website fingerprinting attacks,” in _Computer and Communications Security_ , 2014.
  * [20] M. Juarez, M. Imani, M. Perry, C. Diaz, and M. Wright, “Toward an efficient website fingerprinting defense,” in _Proceeding of the European Symposium on Research in Computer Security_ , vol. 9878, 2016, pp. 27–46.
  * [21] S. Kiranyaz, T. Ince, R. Hamila, and M. Gabbouj, “Convolutional neural networks for patient-specific ecg classification,” vol. 2015, pp. 2608–2611, 2015.
  * [22] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional neural networks,” pp. 1097–1105, 2012.
  * [23] Y. LeCun, Y. Bengio, and G. E. Hinton, “Deep learning,” _Nature_ , vol. 521, pp. 436–444, 2015.
  * [24] Y. Lecun, L. Bottou, Y. Bengio, and P. Haffner, “Gradient-based learning applied to document recognition,” _Proceedings of the IEEE_ , vol. 86, pp. 2278 – 2324, 12 1998.
  * [25] M. Liberatore and B. N. Levine, “Inferring the source of encrypted http connections,” in _Proceedings of the 13th ACM Conference on Computer and Communications Security_ , 2006, pp. 255–263.
  * [26] F. J. Morenobarea, F. Strazzera, J. M. Jerez, D. Urda, and L. Franco, “Forward noise adjustment scheme for data augmentation,” in _IEEE Symposium Series on Computational Intelligence_ , 2018, pp. 728–734.
  * [27] A. Panchenko, F. Lanze, and M. Henze, “Website fingerprinting at internet scale,” in _Proceedings of the 16th Network and Distributed System Security Symposium_ , 2016.
  * [28] A. Panchenko, L. Niessen, A. Zinnen, and T. Engel, “Website fingerprinting in onion routing based anonymization networks,” in _Proceedings of the ACM Workshop on Privacy in the Electronic Society_ , 2011, pp. 103–114.
  * [29] V. Rimmer, D. Preuveneers, M. Juarez, T. Van Goethem, and W. Joosen, “Automated website fingerprinting through deep learning,” in _Proceedings of the Network and Distributed System Security Symposium_ , 2018\.
  * [30] C. Shorten and T. M. Khoshgoftaar, “A survey on image data augmentation for deep learning,” _Journal of Big Data_ , vol. 6, no. 1, p. 60, 2019.
  * [31] K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” 2014.
  * [32] P. Sirinam, M. Imani, M. Juarez, and M. Wright, “Deep fingerprinting: Undermining website fingerprinting defenses with deep learning,” in _Proceedings of the ACM Conference on Computer and Communications Security_ , 2018, pp. 1928–1943.
  * [33] P. Sirinam, N. Mathews, M. S. Rahman, and M. Wright, “Triplet fingerprinting: More practical and portable website fingerprinting with n-shot learning,” in _Proceedings of the ACM Conference on Computer and Communications Security_ , 2019, pp. 1131–1148.
  * [34] C. Szegedy, W. Zaremba, I. Sutskever, J. Bruna, D. Erhan, I. Goodfellow, and R. Fergus, “Intriguing properties of neural networks,” _arXiv:1312.6199 [cs.CV]_ , 2013.
  * [35] L. Taylor and G. Nitschke, “Improving deep learning using generic data augmentation,” _arXiv:1708.06020 [cs.LG]_ , 2017.
  * [36] V. Vapnik, _Statistical learning theory_. Wiley-Interscience, 1998.
  * [37] T. Wang, X. Cai, and I. Johnson, Roband Goldberg, “Effective attacks and provable defenses for website fingerprinting,” in _Proceedings of the 23rd USENIX Security Symposium_ , 2014, pp. 143–157.
  * [38] T. Wang and I. Goldberg, “Improved website fingerprinting on tor,” in _Proceedings of the ACM Workshop on Privacy in the Electronic Society_ , 2013, pp. 201–212.
  * [39] ——, “On realistically attacking tor with website fingerprinting,” in _Proceedings on Privacy Enhancing Technologies (PoPETs)_ , 2016, pp. 21–36.
  * [40] ——, “Walkie-talkie: An effective and efﬁcient defense against website fingerprinting,” in _Proceeding of the 26th USENIX Security Symposium_ , 2017, pp. 1375–1390.
  * [41] J. Wei and K. Zou, “Eda: Easy data augmentation techniques for boosting performance on text classification tasks,” in _International Joint Conference on Natural Language Processing_ , 2019, pp. 6381–6387.
  * [42] F. Yu and V. Koltun, “Multi-scale context aggregation by dilated convolutions,” 2016.
  * [43] H. Zhang, M. Cisse, Y. N. Dauphin, and D. Lopez-Paz, “mixup: Beyond empirical risk minimization,” in _International Conference on Learning Representations_ , 2018.


# Optimization of Tritium Breeding Ratio in a DT and DD Submersion Tokamak
Fusion Reactor

Vikram Goel
United World College of South East Asia
Singapore
<EMAIL_ADDRESS>
&Soha Aslam
Princeton University
Princeton
<EMAIL_ADDRESS>
&Sejal Dua
Georgia Institute of Technology
New York
<EMAIL_ADDRESS>

###### Abstract

The mass of stars is enough to confine a plasma to fuse light atoms, but this
is not possible to engineer on Earth. Fortunately, nuclear engineering can
rely on the magnetic confinement of a plasma using superconducting coils so
long as the Tritium Breeding Ratio (TBR) is optimized. This paper will
investigate some of the materials which can increase the rate at which Tritium
is produced within the breeding blanket layer of Submersion Tokamak reactors,
a design that uses magnetic confinement of a plasma in the shape of a torus to
execute nuclear fusion. Using the Paramak Python module to model several
geometries and OpenMC to run a simulation, it can be observed how neutron
multipliers, enrichment, and the neutron energy spectrum affect TBR. This
experiment will mainly observe different material choices that have been
considered and their TBR based on their cross sections, dose rate, thermal
properties and safety. By altering the neutron energy spectrum to account for
DD and DT plasma, the difference in these compounds’ Tritium breeding efficacy
is noted. Neutron energy spectra are an important factor in optimising the TBR
levels as the neutrons generated by the fusion reactions in the plasma
interact with the breeder material in the blanket and produce tritium through
the reaction with Lithium. Since Tritium is a rare isotope of hydrogen that is
used as fuel in fusion reactions and has a short half-life, it is essential to
produce tritium within the fusion reactor itself. Without the tritium breeding
capability, it would not be feasible to generate energy via fusion. A TBR
greater than unity indicates that the reactor can generate more tritium than
it consumes, ensuring self-sufficiency in the tritium inventory. Since Tritium
is the most reliable and efficient fuel for these reactors, optimising the TBR
is of paramount importance in the long road to commercialization of nuclear
fusion.

_K_ eywords Nuclear Engineering $\cdot$ Fusion Reactor $\cdot$ Tokamak $\cdot$
Tritium Breeding Ratio $\cdot$ Monte Carlo

##  1 Introduction

In the realm of energy production, nuclear fusion has emerged as a field of
immense importance and potential as a promising source of clean and
sustainable power. Fusion energy, simply put, is the energy released when
light atomic nuclei combine to form heavier nuclei. This process occurs at
incredibly high temperatures and pressures, replicating the conditions found
in the core of stars. Unlike current energy sources, such as fossil fuels, it
produces virtually no greenhouse gas emissions or long-lived radioactive
isotopes. Furthermore, fusion energy has the potential to provide an almost
limitless source of power, promising a sustainable and clean solution to our
ever-growing energy needs.

To comprehend the various forms of nuclear fusion reactions, it is essential
to explore the fundamental principles governing these processes. Fusion
reactions can occur between different isotopes, with the most promising
reactions involving deuterium (H2) and tritium (H3) nuclei. The fusion of
these hydrogen isotopes releases a significant amount of energy and produces
alpha particles as a byproduct. Additionally, alternative fusion reactions,
such as the fusion of deuterium with helium-3 or boron-11, offer unique
advantages and challenges, making them worthy of investigation, this paper
will not delve into these though they are worthy of investigation.

To study and analyse the complex behaviour of nuclear fusion reactions,
scientists often employ Monte Carlo simulations. These simulations utilise
probabilistic methods to model the behaviour of particles within a system,
providing insights into the dynamics of nuclear reactions. Monte Carlo
simulations involve the generation of random numbers to determine the
characteristics of each particle, allowing researchers to predict the
behaviour and outcomes of fusion processes accurately. By simulating a large
number of particles, scientists can obtain statistically significant results
and optimise experimental designs, ultimately advancing our understanding of
fusion energy and aiding in the development of practical fusion reactors.

$\text{Tritium Breeding Rate (TBR)}=\frac{\text{number of tritium atoms
produced}}{\text{number of tritium atoms fused}}$ (1)

One of the most important statistics concerning fusion reactors is the Tritium
Breeding Ratio, hitherto TBR. This is a representation of the ratio of the
rate at which tritium is produced in a fusion reactor to the rate at which it
is consumed 1. Since Tritium is the most important fuel source in most
reactors, it is essential to produce Tritium within the reactor itself:
Tritium has a very short lifespan (12.3 years) and out of all the Hydrogen in
the universe, tritium only makes up 10-18% while deuterium sits at 0.2%. This
small quantity is meaningless and therefore tritium is made artificially using
CANDU fission reactors so it is not viable to continuously add tritium to
reactors, therefore, maintaining a TBR greater than unity is essential for
commercialising DT nuclear fusion. The main component of nuclear fusion
reactors, specifically tokamaks, that is affiliated with the TBR is the
breeding blanket. The breeding blanket is usually made of either Solids,
Liquids (PbLi), Molten Salts (FLiBE), or pure Lithium. For this paper, a
survey of two several breeder materials was undertaken as well as observing
the impacts of Li6 enrichment and neutron multiplier concentrations.

##  2 Methodology

###  2.1 OpenMC, DAGMC, Paramak and Defining Computational Intensity

Neutronics modelling and simulation is conducted using a monte carlo
simulation tool, known as OpenMC. The OpenMC module simulates neutral
particles (presently neutrons and photons) moving stochastically through an
arbitrarily defined model that represents a real-world experimental setup [1].
It was developed by the Computational Reactor Physics Group (CPRG) based at
the Massachusetts Institute of Technology (MIT) and has been extensively
utilised in the area of nuclear engineering.

For this research, a geometry for a Submersion Tokamak reactor was constructed
using the Paramak Python module developed by Jonathan Shimwell [2]. This
library makes use of parametric shapes, components and reactors, and is
essential for constructing complex geometries that could be assigned different
materials within the OpenMC model. Though OpenMC manipulates Boolean operators
to construct simple geometries, it is difficult to construct a complex tokamak
with a high degree of similarity to what is used in actuality through OpenMC
alone. Upon constructing the geometry, Paramaks computer-aided design (CAD)
would be converted into a ‘dagmc.h5m’ file, the format which works with
OpenMC, using the cad_to_dagmc python module. Then, a neutronic simulation was
conducted using a Python script. Within the OpenMC model, the different
parametric shapes could be assigned to various elements and densities then,
after introducing a neutron source a simulation using 10 rounds of 10,000
particles was conducted.

Within the model, interchanging between Lithium compounds was simple and
separate tallies for these different blanket materials were generated using
the 2.5 and 14 MeV neutron energy spectra.

###  2.2 Monte Carlo Simulation

OpenMC uses a method that resembles the MC21 monte carlo code (A neutron and
photon transport code). To provide workability in a macro environment OpenMC
uses an analogue estimator to obtain its tallies for reaction rates, using
massive amounts of nuclear data, a survey is taken of the number of actual
reactions that occur and then this is used for their estimates on what the
reaction rates and therefore, the scores should be.

This is represented by the OpenMC Analog Estimator Formula:

$R_{x}=\frac{1}{W}\sum_{i\in A}w_{i}$ (2)

where Rx is the reaction rate for reaction x, i denotes an index for each
event, A is the set of all events resulting in reaction x, and W is the total
starting weight of the particles, and wi is the pre-collision weight of the
particle as it enters the event [1].

This analogue estimator is the simplest way to estimate and is easy to
implement, using this OpenMC may use nuclear cross sections data for neutron
multipliers and Li6 to generate its scores on the TBR, but this estimator has
variance since low-probability reactions will require an immense amount of
information and particles to calculate to a high degree of accuracy.
Therefore, OpenMC also uses a collision estimator and a track length estimator
to get around this issue.

The collision estimator does not simply add to a tally once a reaction has
been carried out, it will use every collision to add to the score, for
example, in the context of Tritium Breeding, it will not only add to the tally
every time tritium is produced but every time there is a collision between the
reactants that may or may not produce tritium.

##  3 Materials

The design used for this paper is the Tokamak, it is the leading style of
fusion reactor and its geometry is shown in Figure 1. The magnetic coils that
were created for this simulation were Niobium Titanium (NbTi) coils chosen due
to their superconductivity, high critical current density, fabrication
availability and mechanical strength. These attributes mean that it has zero
electrical resistance, allows a high current to pass through without reverting
to its conducting state, can be easily made into wires and coils and
importantly, can withstand the immense electromagnetic forces within a
tokamak. We also used a similar metal, Nb3Sn for the outer toroidal field
coils for the same reasons, Tungsten for the first wall and components
directly adjacent to the plasma due to its thermal properties and Titanium
supports. A Beryllium blanket wall was used, to maximise neutron
multiplication and Stainless Steel 316 for the poloidal field coils casing as
it tolerates elevated temperatures.

Figure 1: Diagram of the Geometry of the Submersion Tokamak

For this simulation, we observe how these different materials compare when
used as breeder blankets in producing Tritium. This will be done with both
Deuterium-Tritium (D-T) and Deuterium-Deuterium (D-D) fuel. While D-T is a lot
more powerful and yields more energy and tritium, the deuterium isotope of
hydrogen is far more abundant. It has a longer life making it less expensive
and more feasible for implementation. The two hydrogen plasmas will react to
form 14.1 MeV and 2.5 MeV neutrons as shown:

$D+T\rightarrow[\alpha+3.5MeV]+[n+14.1MeV]$ (3) $D+D\rightarrow
3He+[n+2.5MeV]$ (4)

A Beryllium (Be) or Lead (Pb) neutron multiplier will increase the number of
neutrons by initialising the (n,2n) reaction. For the breeder blanket
materials, a variety of Lithium-based alloys and compounds were used, Lithium
is especially important for the breeder blanket as it reacts with neutrons to
produce tritium and, when paired with a Beryllium (Be) neutron multiplier, it
will efficiently produce tritium at a rate higher than unity (TBR = 1). It is
the most reliable element to use in breeder blankets as its only two stable
naturally occurring isotopes react with the neutrons as shown:

${}^{6}Li+n\rightarrow\alpha+T+4.78MeV$ (5)
${}^{7}Li+n\rightarrow\alpha+T+n+2.47MeV$ (6)

Energy Production and extraction within these tokamak reactors is a tedious
engineering task: The kinetic energy of the incident neutrons must be
converted into thermal energy for it to be excavated by the coolant. The
breeder blanket can augment the incident energy via exothermic reactions
including the ${}^{6}Li(n,t)^{4}He$ reaction which has a Q value of 4.79MeV.

Assuming the use of Pb multipliers, DT fusion devices will perform better
multiplication due to high neutron multiplication cross-section at those
energies as shown in Figure 2.

Figure 2: Cross Section vs Energy [3]

##  4 Results

Table 1 of the Tritium Breeding ratio for several Lithium-Based materials that
are used as a breeder blanket in a Submersion Tokamak with both 14.1 MeV and
2.5 MeV neutrons, these were obtained upon the completion of the monte carlo
simulation:

Table 1: TBR Results Lithium-Based material |  | Tritium Breeding Ratio
---
Tally (14MeV)
| Tritium Breeding Ratio
---
Tally (2.5MeV)
Li2ZrO3 (Lithium Zirconate) | 0.974073 | 0.490369
Li2TiO3 (Lithium Titanate) | 0.980266 | 0.501079
Li4SiO4 (Lithium Orthosilicate) | 1.00325 | 0.525374
LiCl (Lithium Chloride) | 1.04656 | 0.539386
FLiBe (LiF and BeF2) | 1.08196 | 0.569398
Li2O (Lithium Oxide) | 1.09383 | 0.542774
Li17Pb83 (90% Li6) | 1.14519 | 0.570423
Li (Pure Lithium) | 1.10905 | 0.543946

###  4.1 Lithium-6 Concentration

It appears as though the concentration of Lithium-6 is the factor that most
affects the TBR as it has a high cross-section for neutron absorption and is
the isotope of Lithium that produces the Tritium via Li6(n, alpha)H3, so the
TBR of breeder material Li17Pb83 was measured over different levels of
enrichment of Lithium-6:

Figure 3: Enrichment vs TBR

Li17Pb83 is usually used as a breeder with a Li-6 concentration of 90% or
higher, this is feasible as it has a low overall Lithium concentration and
high reliance on the lead as a multiplier. Figure 3, depicting the
relationship between Li-6 enrichment level and TBR, can be described by this
equation with under a 5% margin of error for the largest outlier:

$Y=0.34\log{[1.4(X)+1.11]}\quad\quad,{0<X<1}$ (7)

where Y is the TBR and X is the Li6 enrichment as a decimal.

Since lead, as shown in Figure 2, has a high cross section for neutron
multiplication by the (n,2n) reaction at 6.5 MeV or higher, D-D neutrons
(<6.5MeV) perform worse at a Tritium breeding rate of less than half of what
it is for D-T neutrons (>6.5MeV). The answer considering the TBR is clear: The
higher the concentration of Lithium and thereby, Lithium-6, the greater the
raw Tritium Breeding, but this is not to say that pure Lithium-6 is the best
material to use as a breeder - there are many other factors to consider, Pure
Lithium has inferior thermal properties when compared to Lithium Titanate or
Lithium Orthosilicate and is unable to withstand immense heat, it is also more
unstable than these compounds and therefore poses a safety risk when used in
the breeder blankets. It is also not as strong mechanically and is far more
pliable in its pure state, a reason why alloys or compounds prove beneficial.
The material must withstand extreme temperatures, and electromagnetic forces
so Lithium is simply not as reliable. Furthermore, as this graph for Li-6
concentration vs TBR is logarithmic, there is little need, in most cases to
enrich further than 40%, increasing Li-6 concentration to this level changes
the TBR by about 0.7, but a further increase to 90% will only change the TBR
by about 0.1.

###  4.2 Ceramic, Liquid and Molten Salt Blankets

The breeder materials observed using the OpenMC simulation are ceramic liquid
and molten salt breeders. Li4SiO4 (Lithium Orthosilicate), Li2O (Lithium
Oxide), Li2ZrO3 (Lithium Zirconate), and Li2TiO3 (Lithium Titanate) make up
the ceramic breeders, they all have TBRs close to 1, ranging from 0.97 to 1.0
(nearest hundredth) and their energy multiplication are also similar with
Lithium Titanate coming out on top at 1.15 and the lowest at 1.05 [4]. They
have similar thermal properties including their melting point and thermal
expansion. Their thermal properties and resistance to strong electromagnetic
forces within the tokamak make them especially useful over pure Lithium or
weaker alternatives. Lithium vacancy in Li4SiO4 may interact with tritium
resulting in some confinement, an essential outcome for extracting energy and
recycling Tritium efficiently. The formation energies of the vacancy-tritium
complexes are in the range of 0.41 - 1.28 eV under oxygen-rich conditions [5].

Liquid breeder materials have several qualities that make them superior when
compared to their ceramic counterparts including thermal conductivity,
fabrication availability and the ease at which they can be removed for Tritium
Extraction [6]. PbLi100, a liquid breeder that we simulated in addition to
those in the table above, has a TBR of 1.24918 with D-T and 0.639227 with D-D,
topping both categories.

Table 2: PbLi100 TBR PbLi100 | 1.24918 (DT TBR) | 0.639227 (DD TBR)
---|---|---

Though Lead Lithium with its high TBRs may seem ideal, they suffer from high
inefficiency as the next chapter describes.

###  4.3 Dose Rates

Even Li17Pb83 and other PbLi-based breeders have the highest TBRs at around
1.2 in our simulation. But still, these liquid breeders with high cross
sections for neutron multiplication and tritium production may not be as
efficient as FLiBe, the leading molten salt blanket. FLiBe is usually a
mixture of LiF and BeF2 and is one of the molten salt blankets. Though it may
not have such a high TBR, it is incredibly efficient in absorbing radiation.
Below is a graph illustrating how FLiBe and PbLi compare with Lithium and
other blankets in terms of their Dose Rates over time:

Figure 4: Neutronic comparison of liquid breeders for ARC-like reactor
blankets [7]

The green areas in Figure 4 represent the recycling limits, and only FLiBe and
a few other salts can meet these limits in a short amount of time whereas PbLi
is far more inefficient. FLiBe has a high heat capacity and a high melting
point, attributes that may contribute to its efficiency [7].

###  4.4 Neutron Multiplier Concentration

Another question regarding the materials to use in a breeder blanket to
maximise tritium breeding and efficiency is the ratio of neutron multipliers
to the Lithium material, the two materials needed for the two reactions used
to produce the hydrogen isotope.

This chart illustrates the TBR of a Lithium based blanket paired with a simple
Lead neutron multiplier over different ratios of Pb to Li. (This used a fixed
mass of Li6, therefore, the Li6 was always 0.2 x total breeder blanket mass,
it was adjustable in the simulation by calculating the required fraction out
of total Lithium to make it 20%, for example, 90% Lithium x 22.2..% = 20%
Lithium-6):

Figure 5: TBR against Lead concentration in Breeder Blanket

The parabolic shape of these results is due to there being a constant mass of
Lithium-6, this means that at lower lead concentration, there is no extra Li-6
that could significantly increase the TBR and at higher lead concentration,
this helps the TBR to increase as there is a better concentration of
multipliers. In Li17Pb83, there is a much higher concentration of lead, but
this is compensated with high Li6 enrichment, therefore, it makes sense to
assume a relatively constant amount of Lithium 6 should be present.

##  5 Conclusion

This paper has observed the effects, advantages and results of using different
types of blanket materials, different levels of Li-6 enrichment and
concentration of neutron multipliers. We have also addressed other concerns
such as thermal properties, energy multiplication and dose rates/efficiency.

In the matter of the TBR, the best blanket material appears to be Li17Pb83, it
is the breeder that, unlike FLiBe, has no toxicity [8] and unlike PbLi100, is
not so inefficient in terms of dose rate. It also has relatively low
concentrations of Lithium, a scarce material that may prove difficult to
obtain in the coming decades. This liquid breeder also comes with the
advantages that liquid breeders have over their solid counterparts as
previously discussed and it has a very high TBR when compared to most other
materials. FLiBe was a very successful breeder especially when one takes into
account the dose rates and apart from the hazard risks with this breeder it
may be the most successful one that was used.

To further advance our understanding of the topic of breeder blankets and
tritium breeding, further research into other attributes must be explored such
as different thicknesses of breeders and alternate fuel sources such as
boron-11 or helium 3. It is also essential to address the issues of tritium
extraction and confinement aside from uranium beds that may be proliferation
concerns. Construction of several breeds of tokamaks and even stellarator
reactors using Paramak or the new Parametric Stellarator python libraries to
pair with an OpenMC simulation could also be researched to further understand
the TBRs of different blanket materials in more than one type of reactor.

The field of computational application for nuclear engineering should be used
to predict and theorise on how these reactors and particles interact and will,
in the future, be of great use as fusion becomes the energy source for our
planet’s fossil fuel-free future.

## Acknowledgments

This was supported by my mentor, Ms. Soha Aslam who instructed and guided my
research process, teaching me about how to use the python packages I needed
and about this field as a whole.

This was also made possible by the Paramak library, developed by Dr. Jonathan
Shimwell.

## References

  * [1] Paul K. Romano, Nicholas E. Horelik, Bryan R. Herman, Adam G. Nelson, Benoit Forget, and Kord Smith.  Openmc: A state-of-the-art monte carlo code for research and development.  Ann, 82:90–97, 2015.
  * [2] J. Shimwell, J. Billingsley, R. Delaporte-Mathurin, M. Czentye, K. Hearn, P. Shriwise, K. Taylor, B. Edwards, M. Bluteau, D. Morbey, A. Hollands, Z. Markan, C. Isacs, A. Davis, L. Pattinson, H. Peter, S. Frank, and L. Steve.  The Paramak: automated parametric geometry construction for fusion reactor designs (Version 0., 7:1, 2021.
  * [3] M. Chadwick et al.  (2011).  ENDF/B-VII., 1:2887–2996.
  * [4] M.E. Sawan and M.A. Abdou.  Physics and technology conditions for attaining tritium self-sufficiency for the dt fuel cycle.  Fusion Engineering and Design, 81(8):1131–1144, 2006.  Proceedings of the Seventh International Symposium on Fusion Nuclear Technology.
  * [5] Yanli Shi, Tiecheng Lu, Tao Gao, Xiaogang Xiang, Qinghua Zhang, Xiaohe Yu, Yichao Gong, and Mao Yang.  Density functional study of lithium vacancy in li4sio4: Trapping of tritium and helium.  Journal of Nuclear Materials, 467:22–3115, 2015.
  * [6] H. Tas, S. Malang, F. Reiter, and J. Sannier.  Liquid breeder materials.  Journal of Nuclear Materials, 155:22–3115, 1988.
  * [7] Stefano Segantin, Raffaella Testoni, and Massimo Zucchetti.  Neutronic comparison of liquid breeders for arc-like reactor blankets.  Fusion Engineering and Design, 160:112013, 2020.
  * [8] R. R. Romatoski and L. W. Hu.  Fluoride salt coolant properties for nuclear reactor applications: A review, annals of nuclear energy.  109:306–4549, 2017.


Recall that by the Law of Large Numbers, Slutsky’s theorem, and Remark 4, that
$H_{n}\overset{p}{\to}H_{0}.$ Recall also the definition of the cross
derivative, $X_{n},$ in Equation (A.12), and recall that by the Law of Large
Numbers, Slutsky’s theorem, and Remark 4, $X_{n}\overset{p}{\to}X_{0}.$

We finally consider $J_{n}=E_{n}z,$ where $J_{n}$ is also defined in (6.1),
and where recall that, as defined in (23),

$z=\frac{\partial}{\partial\beta}(v/E_{n}r-\gamma kl).$ (61)

Recall also that we have that

$\displaystyle\sqrt{n}J_{n}=\sqrt{n}E_{n}z$
$\displaystyle=\frac{\sqrt{n}E_{n}\frac{\partial}{\partial\beta}v}{E_{n}r}-\frac{\sqrt{n}E_{n}v(E_{n}\frac{\partial}{\partial\beta}r)}{(E_{n}r)^{2}}-\sqrt{n}E_{n}\frac{\partial}{\partial\beta}\gamma
kl(\beta,b)$
$\displaystyle\overset{\mathcal{L}}{\to}\frac{z_{1}}{Er}-\frac{z_{2}(E\frac{\partial}{\partial\beta}r)}{(Er)^{2}}-z_{3},$

where $z_{1},z_{2},z_{3}$ are normally distributed random variables by the
CLT, Slutsky’s theorem, and Remark 4, and hence their scaled sum is also
normally distributed. Set their scaled sum to be
$z_{0}=\frac{z_{1}}{Er}-\frac{z_{2}(E\frac{\partial}{\partial\beta}r)}{(Er)^{2}}-z_{3}.$
Replace the empirical distributions involving $r$ in $z$ to define

$\displaystyle\tilde{z}=\frac{\frac{\partial}{\partial\beta}v}{Er}-\frac{v(E\frac{\partial}{\partial\beta}r)}{(Er)^{2}}-\frac{\partial}{\partial\beta}\gamma
kl(\beta,b).$ (62)

Note that still

$\displaystyle\sqrt{n}E_{n}\tilde{z}$
$\displaystyle=\frac{\sqrt{n}E_{n}\frac{\partial}{\partial\beta}v}{Er}-\frac{\sqrt{n}E_{n}v(E\frac{\partial}{\partial\beta}r)}{(Er)^{2}}-\sqrt{n}E_{n}\frac{\partial}{\partial\beta}\gamma
kl(\beta,b)$
$\displaystyle\overset{\mathcal{L}}{\to}\frac{z_{1}}{Er}-\frac{z_{2}(E\frac{\partial}{\partial\beta}r)}{(Er)^{2}}-z_{3}=z_{0}$

We have then that, showing explicitly that the derivatives are evaluated at
their true parameters,

$\displaystyle\sqrt{n}(\beta_{n,\gamma}-\beta_{0,\gamma})$
$\displaystyle\approx-\sqrt{n}E_{n}((H_{n}(\beta_{0,\gamma},b_{0}))^{-1}(z(\beta_{0,\gamma},b_{0})+X_{n}(\beta_{0,\gamma},b_{0})q(b_{0})))$
$\displaystyle\overset{\mathcal{L}}{\to}((H_{0}(\beta_{0,\gamma},b_{0}))^{-1}(z_{0}(\beta_{0,\gamma},b_{0})+X_{0}(\beta_{0,\gamma},b_{0})q_{0}(b_{0}))),$

where $z_{0}$ and $q_{0}$ are normally distributed random variables and
$H_{0}$ and $X_{0}$ are constants. Note that, now

$\displaystyle
var((H_{0}(\beta_{0,\gamma},b_{0}))^{-1}(z_{0}(\beta_{0,\gamma},b_{0})+X_{0}(\beta_{0,\gamma},b_{0})q_{0}(b_{0})))$
$\displaystyle=(H_{0}(\beta_{0,\gamma},b_{0}))^{-1}var(z_{0}(\beta_{0,\gamma},b_{0})+X_{0}(\beta_{0,\gamma},b_{0})q_{0}(b_{0}))((H_{0}(\beta_{0,\gamma},b_{0})^{-1})^{T})$

Now, plug in for $z_{0}$ and $q_{0}$ with $\sqrt{n}E_{n}\tilde{z}$ and
$\sqrt{n}E_{n}q$ to get

$\displaystyle(H_{0}(\beta_{0,\gamma},b_{0}))^{-1}var(\sqrt{n}E_{n}\tilde{z}(\beta_{0,\gamma},b_{0})+X_{0}(\beta_{0,\gamma},b_{0})\sqrt{n}E_{n}q(b_{0}))((H_{0}(\beta_{0,\gamma},b_{0})^{-1})^{T})$
$\displaystyle=(H_{0}(\beta_{0,\gamma},b_{0}))^{-1}var(\sqrt{n}E_{n}(\tilde{z}(\beta_{0,\gamma},b_{0})+X_{0}(\beta_{0,\gamma},b_{0})q(b_{0})))((H_{0}(\beta_{0,\gamma},b_{0})^{-1})^{T})$
(63)
$\displaystyle=(H_{0}(\beta_{0,\gamma},b_{0}))^{-1}var(\tilde{z}(\beta_{0,\gamma},b_{0})+X_{0}(\beta_{0,\gamma},b_{0})q(b_{0}))((H_{0}(\beta_{0,\gamma},b_{0})^{-1})^{T})$
$\displaystyle=(H_{0}(\beta_{0,\gamma},b_{0}))^{-1}E((\tilde{z}(\beta_{0,\gamma},b_{0})+X_{0}(\beta_{0,\gamma},b_{0})q(b_{0}))^{2})((H_{0}(\beta_{0,\gamma},b_{0})^{-1})^{T})$

where the variance turns into the expectation of the square because the true
derivative evaluated at the true parameters is zero. We can replace the true
expectation, $E,$ with its empirical version, $E_{n}:$

$(H_{0}(\beta_{0,\gamma},b_{0}))^{-1}E_{n}((\tilde{z}(\beta_{0,\gamma},b_{0})+X_{0}(\beta_{0,\gamma},b_{0})q(b_{0}))^{2})((H_{0}(\beta_{0,\gamma},b_{0})^{-1})^{T}).$

Note that $\tilde{z}$, defined in (62), depends on $Er$. Hence, we can plug in
for $Er,$ and hence replace $\tilde{z}$ with $z,$ where recall that, as
defined in (23),

$z=\frac{\partial}{\partial\beta}(v/E_{n}r-\gamma kl).$ (64)

This allows us to then reintroduce the weighting term, even though we had to
remove it earlier in (A.17) to perform manipulations of the variance operator,
which requires independence of the summands, which is broken by the dependence
of $E_{n}r$ on the joint distribution of the $n$ samples. Hence, replacing
$\tilde{z}$ with $z,$ we obtain,

$(H_{0}(\beta_{0,\gamma},b_{0}))^{-1}E_{n}((z(\beta_{0,\gamma},b_{0})+X_{0}(\beta_{0,\gamma},b_{0})q(b_{0}))^{2})((H_{0}(\beta_{0,\gamma},b_{0})^{-1})^{T}).$

We finally plug in $\beta_{n,\gamma},b_{n},H_{n},$ and $X_{n}$ for
$\beta_{0,\gamma},b_{0},H_{0},$ and $X_{0},$ to get the final form of our
estimator

$\sigma^{2}_{n}=(H_{n}(\beta_{n,\gamma},b_{n}))^{-1}E_{n}((z(\beta_{n,\gamma},b_{n})+X_{n}(\beta_{n,\gamma},b_{n})q(b_{n}))^{2})((H_{n}(\beta_{n,\gamma},b_{n})^{-1})^{T}).$

###  A.18 Gradients

We will provide forms for the derivatives below. Note that
$s,\beta,b\in\mathbb{R}^{K}$, $a\in\\{0,1\\},$ and $\pi(A=1|s)\in\mathbb{R}.$
For post-selection inference, we must define also the nuisance $b$ and
$1_{\mathcal{A}}=(I(1\in\mathcal{A}),\dots,I(K\in\mathcal{A}))^{T}$ where
$\mathcal{A}$ is the active set. We then obtain policies

$\displaystyle\pi_{\beta,b}(A=a|s)=\pi_{\beta,b}(A=1|s)(1-\pi_{\beta,b}(A=1|s))^{1-a},$
$\displaystyle\pi_{\beta,b}(A=1|s)=\text{expit}(\beta^{T}(s\odot
1_{\mathcal{A}})+b^{T}(s\odot(1-1_{\mathcal{A}})).$

Note that the derivative of $\pi_{\beta,b}(A=1|s)$ with respect to its linear
argument is

$d\pi_{\beta,b}=\pi_{\beta,b}(1|s)(1-\pi_{\beta,b}(1|s)),$

which will occur below in numerous places. For post-selection inference, we
have

$\displaystyle\frac{\partial}{\partial\beta}\log\pi_{\beta,b}(a|s)=(a-\pi_{\beta,b}(1|s))(s\odot
1_{\mathcal{A}}),$ $\displaystyle\frac{\partial}{\partial
b}\log\pi_{\beta,b}(a|s)=(a-\pi_{\beta,b}(1|s))(s\odot(1-1_{\mathcal{A}})),$
$\displaystyle\frac{\partial^{2}}{\partial\beta^{2}}\log\pi_{\beta,b}(a|s)=-d\pi_{\beta,b}(s\odot
1_{\mathcal{A}})(s\odot 1_{\mathcal{A}})^{T},$
$\displaystyle\frac{\partial}{\partial
b}\frac{\partial}{\partial\beta}\log\pi_{\beta,b}(a|s)=-d\pi_{\beta,b}(s\odot
1_{\mathcal{A}})(s\odot(1-1_{\mathcal{A}}))^{T}.$

Recall as well that for active set $\mathcal{A},$
$1_{\mathcal{A}}=(1_{1\in\mathcal{A}},\dots,1_{K\in\mathcal{A}})^{T}.$ We can
use properties of the derivative of the log function, which is a popular trick
in the policy gradient literature [Sutton and Barto, 2018]:

$\displaystyle\frac{\partial}{\partial\beta}\prod_{t=0}^{T}\pi_{\beta,b}(A_{t}|S_{t})=\prod_{t=0}^{T}\pi_{\beta,b}(A_{t}|S_{t})\frac{\partial}{\partial\beta}\log\prod_{t=0}^{T}\pi_{\beta,b}(A_{t}|S_{t}),$
$\displaystyle\frac{\partial}{\partial\beta}\log\prod_{t=0}^{T}\pi_{\beta,b}(A_{t}|S_{t})={\sum_{t=0}^{T}\frac{\partial}{\partial\beta}\log\pi_{\beta,b}(a_{t}|s_{t})},$
$\displaystyle\frac{\partial}{\partial
b}\log\prod_{t=0}^{T}\pi_{\beta,b}(A_{t}|S_{t})={\sum_{t=0}^{T}\frac{\partial}{\partial
b}\log\pi_{\beta,b}(a_{t}|s_{t})},$ $\displaystyle\frac{\partial}{\partial
b}\frac{\partial}{\partial\beta}\log\prod_{t=0}^{T}\pi_{\beta,b}(A_{t}|S_{t})={\sum_{t=0}^{T}\frac{\partial}{\partial
b}\frac{\partial}{\partial\beta}\log\pi_{\beta,b}(a_{t}|s_{t})},$
$\displaystyle\frac{\partial^{2}}{\partial\beta^{2}}\log\prod_{t=0}^{T}\pi_{\beta,b}(A_{t}|S_{t})={\sum_{t=0}^{T}\frac{\partial^{2}}{\partial\beta^{2}}\log\pi_{\beta,b}(a_{t}|s_{t})}.$

For the behavioral policy, recall that we have the following likelihood for
one trajectory

$\displaystyle l_{T}(b)=\log\prod_{t=0}^{T}\pi_{b}(a_{t}|s_{t}).$

We then have score and hessian of the behavioral likelihood, which can be
written as functions of the derivatives defined above, as

$\displaystyle\frac{\partial}{\partial
b}l_{T}(b)=\sum_{t=0}^{T}\frac{\partial}{\partial b}\log\pi_{b}(a_{t}|s_{t}),$
$\displaystyle\frac{\partial^{2}}{\partial
b^{2}}l_{T}(b)=\sum_{t=0}^{T}\frac{\partial^{2}}{\partial
b^{2}}\log\pi_{b}(a_{t}|s_{t}).$

###  A.19 Estimator for the variance of the value

Recall that by Slutsky’s theorem and Remark 4,

$V_{n}=\frac{E_{n}v}{E_{n}r}\overset{p}{\to}\frac{Ev}{Er}=Ev.$ (65)

Now, because $E_{n}v$ is an average,
${\sqrt{n}}{E_{n}v}\overset{\mathcal{L}}{\to}Z_{V}\rightarrow
N(Ev,\sigma_{V}^{2})$ for some $\sigma_{V}^{2}.$ We would like to estimate
$\sigma_{V}^{2}.$ In the unweighted case, we can use the estimator derived in
Weisenthal et al. [2023],

$\sigma_{V,n}^{2}(\tilde{V}_{n})=var(\sqrt{n}\tilde{V}_{n})=\frac{1}{n}\sum_{i=1}^{n}\left(\frac{\prod_{t=0}^{T}\pi_{\beta}(A_{i,t}|S_{i,t})}{\prod_{t=0}^{T}\pi_{b}(A_{i,t}|S_{i,t})}\sum_{t=0}^{T}R(S_{i,t},A_{i,t},S_{i,t+1})-\tilde{V}_{n}\right)^{2}.$
(66)

In the weighted case, we note that

$\sqrt{n}V_{n}=\frac{\sqrt{n}E_{n}v}{E_{n}r}\overset{p}{\to}\frac{Z_{V}}{Er}.$
(67)

Hence, we would like to estimate the variance of the weighted importance
sampling estimator,

$\sigma^{2}_{0,V,w}=var\left(\frac{Z_{V}}{Er}\right)=\frac{1}{(Er)^{2}}var(Z_{V})=\frac{1}{(Er)^{2}}\sigma_{V}^{2},$

where we have used a similar strategy in terms of plugins as we did in step
(A.17) of the derivation of the variance of $\beta_{n,\gamma}$, which is
defined in (25). We plug in for $\sigma_{V}^{2}$ using (66) and we plug in for
$Er$ with $E_{n}r$ to obtain a final estimator for the weighted objective
function,

$\sigma^{2}_{n,V,w}=\frac{\sigma_{V,n}^{2}(V_{n})}{(E_{n}r)^{2}}.$

###  A.20 Estimating the true behavior-constrained estimand

We would like to assess coverage now using our estimator for $\sigma^{2}$, the
variance of $\beta_{n}$. Recall that in the policy search setting,
$\beta_{0,\gamma}$ is unknown. We thus must estimate $\beta_{0,\gamma}.$ In
simulation, since we know the functional form of the reward is (27), we can
derive an estimator for $V_{0},$ which we call $V_{0,n},$ that avoids
importance sampling. We do so by finding the maximizer of $V_{0,n},$ as
defined below. In our simulations, we defined

$R(s_{t},a_{t},s_{t+1})=-s_{t+1,2}a_{t}.$

Hence, in the multidimensional state, multi-stage case

$\displaystyle V_{0}=E_{\beta}\sum_{t=0}^{T}R(S_{t},A_{t},S_{t+1})$
$\displaystyle=E\sum_{t=0}^{T}u^{t-1}R_{t}$
$\displaystyle=E\left[\sum_{t=0}^{T}u^{t-1}E\left[R_{t}|S_{1},\dots,S_{t}\right]\right]$
$\displaystyle=-E\left[\sum_{t=0}^{T}u^{t-1}E\left[(S_{t})_{2}A_{t}|S_{1},\dots,S_{t}\right]\right]$
$\displaystyle=-E\left[\sum_{t=0}^{T}u^{t-1}E\left[(S_{t})_{2}A_{t}|S_{t}\right]\right]\text{
\ \ \ (since }(A_{t}\perp S_{j}|S_{t}),j<t)$
$\displaystyle=-E\left[\sum_{t=0}^{T}u^{t-1}E\left[(S_{t})_{2}A_{t}|S_{t,1},\dots,S_{t,K}\right]\right]$
$\displaystyle=-E\left[\sum_{t=0}^{T}u^{t-1}(S_{t})_{2}E\left[A_{t}|S_{t}\right]\right]$
$\displaystyle=-E\left[\sum_{t=0}^{T}u^{t-1}(S_{t})_{2}\text{expit}(\beta^{T}S_{t})\right]$
$\displaystyle\approx-\frac{1}{n}\sum_{i=1}^{n}\sum_{t=0}^{T}u^{t-1}(S_{i,t})_{2}\text{expit}(\beta^{T}S_{i,t})=:V_{0,n}$

Note that, because we are omniscient when we simulate data, and therefore we
know $R_{t}$, this estimator does not depend on the ratio defined in (31),
which is the destabilizing component of $V_{n}$. Hence, this estimator will
converge arbitrarily quickly. We have chosen to designate this pure Monte-
Carlo estimator of $V_{0}$ as $V_{0,n}$, which is different from our
importance sampling estimator of $V_{0},$ which we called $V_{n}.$ In
practice, we would never be able to estimate $V_{0,n}$ with real data, but we
could estimate $V_{n}.$ We therefore only use $V_{0,n}$ to check our
theoretical results with simulated data. We use a derivation as we did because
in simulation we know the true reward function ((27)), and therefore we can
derive an expression for the expected reward that avoids the importance
sampling ratio present in $V_{n}$ from Equation (12).

Define $M_{0,n}=V_{0,n}+\gamma KL_{n}(\beta,b_{n})$ and

$\beta_{0,n,\gamma}=\arg\max_{\beta}M_{0,n},$ (68)

which is an estimator for $\beta_{0,\gamma}$ that we take to be “true” because
it converges much faster than does $\beta_{n,\gamma}.$

We can therefore use the same $n$ for estimating $M_{0,n}$ and ${M}_{n}$ and
get much stabler estimates in the former. This allows us to then in a sense
average over our estimates of $\beta_{0,n,\gamma}$ as
$\bar{\beta}_{0,n,\gamma}$, making it even more stable.

###  A.21 Simulation coefficients

###### Proof.

Note that

$\displaystyle V_{0}$
$\displaystyle=E\left(\sum_{t=0}^{T}\gamma^{t}R(S_{t},A_{t},S_{t+1})\right)$
$\displaystyle=E\left(\sum_{t=0}^{T}\gamma^{t}E(R(S_{t},A_{t},S_{t+1})|S_{t})\right)$
$\displaystyle=-E\left(\sum_{t=0}^{T}\gamma^{t}E(S_{t,2}A_{t}|S_{t})\right)\ \
\ \ \ \ \ (\text{definition of }R)$
$\displaystyle=-E\left(\sum_{t=0}^{T}\gamma^{t}(S_{t,2}E(A_{t}|S_{t}))\right)\
\ \ \ \ \ \ (\text{definition of $S_{t+1}$ })$
$\displaystyle=-E\left(\sum_{t=0}^{T}\gamma^{t}(S_{t,2}\text{expit}(\beta^{T}S_{t}))\right)$

This is maximized by $\pi_{\beta_{0}}(A_{t}=1|S_{t})=I(S_{t,2}<0),$ which
occurs if $\beta_{0,2}=-\infty.$ In simulation, we will see a large negative
coefficient $\beta_{2}$, which will shrink in magnitude as we increase
$\gamma$ and $\lambda$. ∎

###  A.22 Coverage summary statistics

Recall that $\beta_{n,\gamma}=\arg\max_{\beta}{M}_{n}.$ For Monte-Carlo
dataset $m$ let the estimator obtained by maximizing ${M}_{n}$ be
$\beta_{n,\gamma}^{(m)}.$ Over $M$ Monte-Carlo datasets, define the average to
be

$\bar{\beta}_{n,\gamma}=\frac{1}{M}\sum_{m=1}^{M}\beta_{n,\gamma}^{(m)}.$ (69)

Also recall that $\sigma^{2}_{n}$ is our asymptotic variance estimator,
defined in (25), let this estimator derived from Monte-Carlo dataset $m$ be
$(\sigma^{2}_{n})^{(m)},$ and define the average over Monte-Carlo datasets as

$\bar{\sigma}_{n}^{2}(\beta_{n,\gamma})=\frac{1}{M}\sum_{m=1}^{M}(\sigma^{2}_{n})^{(m)}.$
(70)

Further, define the empirical variance of the estimated $\beta_{n,\gamma}$ as

$\sigma^{2}_{0,n}(\beta_{n,\gamma})=\frac{1}{M-1}\sum_{m=1}^{M}(\sqrt{n}\beta_{n,\gamma}^{(m)}-\bar{\beta}_{n,\gamma,\sigma})^{2},$
(71)

where $\beta_{n,\gamma}^{(m)}$ is the solution for Monte-Carlo iteration $m$
and

$\bar{\beta}_{n,\gamma,\sigma}=\frac{1}{M}\sum_{m=1}^{M}\sqrt{n}\beta_{n,\gamma}^{(m)}.$

Note that we multiply $\beta_{n,\gamma}$ by $\sqrt{n}$ so that we can estimate
$\sigma$, the standard deviation, not $\sigma/\sqrt{n},$ the standard error.
When we construct confidence intervals, we then have to divide our estimate of
$\sigma$ by $\sqrt{n}.$ We index (71) by $0$ to designate its role as a
“reference” quantity, against which we will check our theory.

###  A.23 Calibration curve of the behavioral policy model in the real data
(test set)

A calibration curve for the behavioral policy model is shown in Figure A.1. We
see that the behavioral estimates align with the observed probabilities, which
serves as evidence that the logistic regression model for the behavioral
policy in Equation (1) is reasonable.

Figure A.1: Real data calibration curve for the behavioral policy. We show a
calibration curve for the real data, in which the estimated and observed
probabilities are compared. The plot is generated based on the predtools R
package. We resample the real data, each time training on one half and then
generating a calibration curve for the test data, and then we finally average
these test set calibration curves (generated by the R package “predtools”).
The behavioral policy is stationary by Assumption 2, so we just treat action-
state observations at different time steps as independent (the confidence
intervals might be too narrow, but mostly we are focused on the fact that the
point estimates are approximately near the identity here).