# Singular Regularization with Information Bottleneck Improves Model’s
Adversarial Robustness

Guanlin Li
Nanyang Technological University, S-Lab
<EMAIL_ADDRESS>Naishan Zheng
Nanyang Technological University, S-Lab  Man Zhou
Nanyang Technological University, S-Lab  Jie Zhang
Nanyang Technological University  Tianwei Zhang
Nanyang Technological University

###### Abstract

Adversarial examples are one of the most severe threats to deep learning
models. Numerous works have been proposed to study and defend adversarial
examples. However, these works lack analysis of adversarial information or
perturbation, which cannot reveal the mystery of adversarial examples and lose
proper interpretation. In this paper, we aim to fill this gap by studying
adversarial information as unstructured noise, which does not have a clear
pattern. Specifically, we provide some empirical studies with singular value
decomposition, by decomposing images into several matrices, to analyze
adversarial information for different attacks. Based on the analysis, we
propose a new module to regularize adversarial information and combine
information bottleneck theory, which is proposed to theoretically restrict
intermediate representations. Therefore, our method is interpretable.
Moreover, the fashion of our design is a novel principle that is general and
unified. Equipped with our new module, we evaluate two popular model
structures on two mainstream datasets with various adversarial attacks. The
results indicate that the improvement in robust accuracy is significant. On
the other hand, we prove that our method is efficient with only a few
additional parameters and able to be explained under regional faithfulness
analysis.

##  1 Introduction

Deep learning models, especially Convolutional Neural Networks (CNNs) in
computer vision, suffer from adversarial examples [10], which can cause the
models to give incorrect responses. With deep learning models becoming an
important part of various services, it is vital to study and understand
adversarial examples. To the best of our knowledge, there are three main
perspectives for studying and understanding adversarial examples. The first is
to treat adversarial examples as some specific features [14]. These features
help the model to learn a robust representation to defend against adversarial
attacks. Other data without such features will only make the model obtain
clean accuracy but fail under attacks. The second perspective is to study
adversarial properties over the whole training set [23, 9]. They find that
different training data have unique adversarial properties. Some data are
mainly contributing to the clean accuracy. Some contribute primarily to
adversarial robustness. The last perspective is to study how the model learns
adversarial examples during the training process [19, 6]. Because models are
much easier to overfit adversarial examples, studying and understanding such a
phenomenon will help us design better training methods. Although previous
works have studied adversarial examples from several perspectives, they mainly
focus on the training process and study one specific attack, i.e., PGD attack
[18]. That will lose some guarantees of consistency between training and
testing and among different attacks. Different from previous works, we take
one step forward to study the straightforward properties of adversarial
examples under different attacks and training strategies to make our
observations consistent and general.

Figure 1: Combination between singular vectors of adversarial examples with
singular values from clean images and adversarial examples under different
attacks for different training strategies (PGD-AT [18], TRADES [27], AWP [24],
and TE [6]). Solid markers (left) mean we do not replace the singular values
with clean ones. Hollow markers (right) mean we replace the singular values
with clean ones for specific attacks.

In novelty, we analyze adversarial examples from the matrix decomposition
aspect and find that different adversarial examples from different sources
have similar properties. Considering that the given image is made up of
matrices stacked along the color channel, we can decompose each matrix for
every color channel to obtain the singular values and corresponding singular
vectors, dubbed singular value decomposition (SVD) [21]. Generally speaking,
the singular values and vectors store all the information of the given image.
In the image restoration perspective, different types of corruption can be
reflected in singular values or vectors [28]. Similarly, added perturbation in
the adversarial examples can be seen as unstructured noise, i.e., one specific
type of corruption, which is related to the model’s parameters and the clean
images and does not have a specific pattern. Naturally, we can do SVD on
adversarial examples and obtain their singular values and vectors to study the
properties of adversarial examples with these important elements.
Specifically, we study singular values and vectors of adversarial examples and
their corresponding clean data by performing SVD on them, respectively. Then,
we combine the singular vectors of adversarial examples and singular values of
the corresponding clean data. In this way, we can study and compare
adversarial information in singular values and vectors. In Figure 1, we
observe that after using clean singular values, the robust accuracy will
increase under different attacks. Our results indicate that, for different
training strategies and adversarial attacks, singular values consistently
contain adversarial information. Therefore, replacing them with clean singular
values will increase the robust accuracy. The detailed results and analysis
can be found in Section 3.

This consistency among training methods and various attacks inspires us to
perform calibration on adversarial information in the singular values and
vectors to defend against adversarial examples. However, it is not trivial to
directly modulate the singular values and vectors for given images, because
they are hidden behind the pixel data. Therefore, we introduce and develop
custom modules from the previous work [28] to separate singular values and
vectors from the images and perform calibration on them, respectively. To make
our method more explainable, we integrate the singular regularization
operation in a new plug-in module based on information bottleneck theory [1],
dubbed Singular Regularization with Implicit Information Bottleneck (SiRIIB).
The information bottleneck theory can be adopted to explain intermediate
features of the models and give a guarantee for robustness. Specifically,
SiRIIB can calibrate adversarial examples from the perspective of SVD and
transfer extracted features to the main classification model following the
information bottleneck theory, i.e., to compress the modulated information
that is extracted from inputs, and to contain sufficient information to guide
the following prediction task. On the other hand, SiRIIB is a general module
that can cooperate with residual-based CNNs, such as ResNet [11] and
WideResNet [26], and different training strategies, such as PGD-AT [18] and
TRADES [27]. With our comprehensive evaluation, we prove that SiRIIB with few
parameters will improve robust accuracy under various attacks. Overall, our
contribution can be summarized as follows:

  * •

We study the properties of adversarial examples with SVD and find different
attacks that will always influence the singular values and vectors but to
different degrees.

  * •

We design a new plug-in module SiRIIB based on information bottleneck theory
with singular regularization, which is proven to be effective in defending
against various attacks. Our SiRIIB is universal and can cooperate with
different models.

  * •

We show that SiRIIB can calibrate adversarial information in adversarial
examples and induce the model to learn a more compressed representation.
Furthermore, it can reduce the sensitivity of local perturbation during the
attack.

##  2 Related Works

To improve the adversarial robustness of models, there are several methods
from different perspectives. In summary, there are two main traces, i.e.,
studying training methods [18] and studying models [25].

Training Methods. When training the classification models, we can generate
adversarial examples for each batch of clean data and train the models with
these adversarial examples [18], which is called adversarial training. It can
be formulated as the following min-max problem:

$\displaystyle\min_{\theta}\max_{x_{\mathrm{adv}}}L(x_{\mathrm{adv}},y;\theta)$

where $x_{\mathrm{adv}}$ is the training sample generated from a clean one $x$
to maximize the loss function $L(\cdot)$, $y$ is the ground-truth label,
$\theta$ is the model parameters. It is to say, we find $x_{\mathrm{adv}}$
based on $x$ to maximize the value $L(x_{\mathrm{adv}},y;\theta)$, and then we
optimize the model to minimize the loss $L(x_{\mathrm{adv}},y;\theta)$. Zhang
et al. [27] explore the trade-off between clean accuracy and robust accuracy
during adversarial training and propose a new training strategy to better
balance them.

Model Perspective. There are some works aiming to improve the existing model’s
robustness by replacing modules with newly designed ones. For example, Huang
et al. [13] propose a new residual block focusing on the depth and width to
improve the residual model’s robustness. On the other hand, some works [3]
focus on comparing the robustness of traditional convolutional models and
visual transformers [7].

Different from previous works, we aim to introduce a new plug-in module, whose
design is inspired by the singular value decomposition [21] and information
bottleneck theory [1]. Our module is deterministic, differentiable,
explainable, and efficient, with only a few parameters.

Figure 2: Visualization results of singular value combinations. $x$ is clean
images. PGD and AA mean adversarial examples generated by PGD-20 and AA. PGD-S
and AA-S mean we replace singular values of adversarial examples with clean
singular values. D1 and D2 are differences between PGD-S and PGD and between
AA-S and AA, respectively.

##  3 Preliminaries

In this section, we first provide some empirical studies to analyze
adversarial examples under SVD. Then, we introduce a general method to
modulate the singular values and vectors for a given image without explicitly
performing SVD on it.

###  3.1 Empirical Study under SVD

In our empirical studies, to understand singular values and vectors of
adversarial examples and their corresponding clean data, we first perform SVD
on them, respectively. Then, we combine the singular vectors of adversarial
examples and singular values of corresponding clean data. In Figure 1, we
compare the robust accuracy of ResNet-18 [11] trained with different
adversarial training methods on CIFAR-10 from the aspects of singular values.
The results indicate a strong consistency, which is not related to the
training method but related to the attacks. First, for all attacks, after
using clean singular values, robust accuracy will increase. On the other hand,
different attacks have slight differences under the view of adversarial
information. Specifically, based on our observations, some adversarial
attacks, such as PGD [18], hide the adversarial information in the singular
values and vectors equally. Because, if we replace the singular values of
adversarial examples with the singular values of their corresponding clean
images, the robust accuracy will slightly increase. However, some adversarial
attacks, like C&W [4] and Autoattack (AA) [5], mainly hide adversarial
information in singular values. Simply replacing the singular values of
adversarial examples with the singular values of their corresponding clean
images will significantly increase robust accuracy. In Figure 2, we further
visualize the differences before and after replacing with clean singular
values for adversarial examples generated by PGD-20 and AA. It is clear that
clean singular values will cause a specific noise pattern for PGD-20 examples
and part of AA examples. But for some AA examples, replacing with clean
singular values will cause a random noise pattern, which is closer to the
adversarial perturbation. This explains why introducing clean singular values
in adversarial examples generated by AA leads to higher accuracy improvements.
These observations inspire us to calibrate singular values and vectors with
clean images to defend against adversarial attacks. However, in practice, we
cannot manually or randomly modify the SVD results of the given images and
then reconstruct the images based on the modified SVD results, because such
operations will be non-differentiable and random, which will cause gradient
obfuscation, which is proven to give a false sense of security [2]. In the
following, we will introduce a deterministic, differentiable, and learnable
method to modulate singular values and vectors.

###  3.2 Singular Regularization

To design a deterministic, differentiable, and learnable method to modulate
singular values and vectors, we follow the previous method [28] to implicitly
modify the singular values and vectors from the images, respectively. It is
because singular values and vectors correspond to different information, i.e.,
singular vectors represent the directions in the image space along which the
most variance occurs, and singular values represent the importance of each
corresponding singular vector. Therefore, we calibrate them separately, i.e.,
singular value modification and singular vector modification, and we fuse the
modified results to obtain modulated features.

Singular Vector Modification. During this operation, the most important thing
is to keep the singular values unchanged. Intuitively, we can find the fact
that multiplying an arbitrary matrix by orthogonal matrices will not change
the singular values of the arbitrary matrix. For example, given an arbitrary
real matrix $X\in\mathbb{R}^{n\times m}$ and an orthogonal real matrix
$P\in\mathbb{R}^{n\times n}$, the singular values of $PX$ are the same as
those of $X$. To prove it, we can first write $X$ into SVD format, $X=U\Sigma
V^{\prime}$, where $U$ and $V^{\prime}$ are two orthogonal matrices. Then, we
have $PX=PU\Sigma V^{\prime}$, in which we only need to prove
$(PU)(PU)^{\prime}=I$, where $I$ is the identity matrix. Clearly, we have

$\displaystyle(PU)(PU)^{\prime}=PUU^{\prime}P^{\prime}=PIP^{\prime}=PP^{\prime}=I.$

Therefore, this operation will only affect the singular vectors. Based on the
above information, we can restrict the learnable weight matrix to be
orthogonal and use it to modify the singular vector for given images.

Singular Value Modification. On the other hand, modifying singular values is
non-trivial, due to the inherent inaccessibility of the singular values.
However, the previous work [28] finds that the singular values can be modified
alone in the Fourier domain. Specifically, we can present
$X\in\mathbb{R}^{n\times m}$ in two formations, i.e.,

$\displaystyle X=U\Sigma V^{\prime},$ $\displaystyle
X=\frac{1}{nm}\sum_{u=0}^{n-1}$ $\displaystyle\sum_{v=0}^{m-1}G(u,v)e^{2\pi
i(\frac{ua}{n}+\frac{vb}{m})},$ $\displaystyle a\in$
$\displaystyle\mathbb{R}^{n-1},b\in\mathbb{R}^{m-1}$

where $G(u,v)$ represents the coefficients of the Fourier transform of $X$. If
we use $u_{i}$ and $v_{i}$ to present the columns of $U$ and $V$, and use
$\sigma_{i}$ to represent singular values in $\Sigma$, we can rewrite $X$ in a
way of coefficients multiplying a group of basis, i.e.,

$\displaystyle
X=\sum_{i=1}^{min(n,m)}\sigma_{i}u_{i}v_{i}^{\prime}=\frac{1}{nm}\sum_{u=0}^{n-1}\sum_{v=0}^{m-1}G(u,v)\phi(u,v),$

where $\phi(u,v)=e^{2\pi i(\frac{ua}{n}+\frac{vb}{m})}$. In the Fourier
domain, we can modify the coefficients $G(u,v)$, and the previous work [28]
proves that $\sigma_{i}$ and $G(u,v)$ can be approximated to each other.
Therefore, we transfer $X$ into the Fourier domain and modify $G(u,v)$ as an
equivalent operation of modifying singular values.

##  4 Singular Regularization with Implicit Information Bottleneck

In this section, we first introduce the information bottleneck theory to the
module design and prove that information compression and regularization can be
implicitly achieved with carefully designed additional connections. Then, we
propose SiRIIB, which contains the singular regularization function and
follows the information bottleneck theory, including how it cooperates with
residual-based models and how it can be trained during the adversarial
training process.

Figure 3: Overview of SiRIIB. We adopt multiple scale inputs for the singular
regularization modules. Then, after the Sigmoid function, we average the
outputs to obtain smoothed results. Finally, we adopt several linear
transformations to modify the feature dimensions and add the outputs to the
original classification model.

###  4.1 Information Bottleneck in Module Design

The information bottleneck (IB) theory [1] can be adopted to explain the
model’s adversarial robustness. In previous work, it usually needs to be
explicitly optimized with Kullback–Leibler divergence. However, we find that
it can be implicitly executed with carefully designed skip connections. First,
we recall the details of the information bottleneck theory.

For a deep learning model, the mutual information between the intermediate
representation and the target label is defined as follows.

###### Definition 1

Given a deep learning model $f(\cdot;\Theta)$, parameterized by $\Theta$, we
regard some intermediate layers that project the input $x$ to an encoding $z$,
defined as $p(z|x;\theta),\theta\in\Theta$. Suppose that the target label for
$x$ is $y$, and the mutual information between $z$ and $y$ is defined as
$I(z,y;\theta)$.

Similarly, based on Definition 1, we have mutual information between two
random variables $Z$ and $Y$, i.e.,

$\displaystyle I(Z,Y;\theta)=\int
dxdyp(z,y|\theta)\log\frac{p(z,y|\theta)}{p(z|\theta)p(y|\theta)},$

where $z$ and $y$ are observations of variables $Z$ and $Y$. To find the best
representation $Z$ for the input $X$ under the information complexity
constraint, $I(X,Z)\leq I_{c}$, we obtain the following optimization goal:

$\displaystyle\max_{\theta}I(Z,Y;\theta)\quad s.t.\quad I(X,Z;\theta)\leq
I_{c},$

which can be further converted to

$\displaystyle\max_{\theta}R(\theta)=I(Z,Y;\theta)-\lambda I(Z,X;\theta)$

by introducing the Lagrange multiplier $\lambda$. By maximizing $R(\theta)$,
we aim to obtain $Z$, which meets the conditions where we can predict $Y$ from
$Z$ with the highest probability, and $Z$ compresses $X$ as much as possible.
Specifically, a large $\lambda$ will increase the compression ratio. Clearly,
maximizing $R(\theta)$ during the model training process is to explicitly
optimize the latent representation of $f(\cdot;\Theta)$.

However, without directly optimizing the $R(\theta)$, we find that it is
possible to implicitly optimize the same objective with correctly designed
skip connections. In the following, we give a formal definition for the IB
skip connection used in our paper.

###### Definition 2

Given an intermediate representation $z$ of the input $x$, we say that there
exists an IB skip connection between the input $x$ and the representation
$z=f(x;\theta)+g(x;\tau)$, if $I(z,x;\theta,\tau)\leq
I(f(x;\theta),x;\theta)$.

The IB skip connection $g(x;\tau)$ in Definition 2 means it can reduce mutual
information between the input $x$ and the intermediate representation $z$ by
adding restrained information to the original $f(x;\theta)$. In the following,
we will give an example to build such a connection.

Intuition 1 Double-sided saturating activation functions, such as Sigmoid or
Softmax, will compress the information and explicitly achieve the function of
information bottleneck.

In fact, this intuition has been proved in the previous work [17].
Furthermore, this encoding compression is not related to loss functions.
Therefore, based on the intuition, we give a proposition to show that it is
possible to design a skip connection, which can reduce the mutual information
between intermediate features and the inputs.

###### Proposition 1

If functional $\mathcal{G}(\cdot;\tau)$ is equipped with a double-sided
saturating activation function, e.g., Sigmoid, there exists
$g(\cdot;\tau)\in\mathcal{G}(\cdot;\tau)$, which can be used as an IB skip
connection.

The proof of Proposition 1 is provided in the supplementary materials. In
summary, we build $g(x;\tau)$ with linear transformations and Sigmoid and show
that it can reduce the mutual information after adding it to the original
model.

# of $p_{i}(\cdot)$ Clean Accuracy AA 1 82.53 48.03 2 83.31 47.80 3 83.07
48.05

Table 1: Ablation study on the number of $p_{i}(\cdot)$. We compare models
equipped with different numbers of IB skip connections.

$\lambda_{1}$ Clean Accuracy AA 1.0 83.07 48.05 1.5 82.84 48.16 3.0 82.84
48.21 5.0 83.00 48.36 20.0 83.27 48.41 30.0 82.99 48.35

Table 2: Ablation study on $\lambda_{1}$. Figure 4: Visualization of
$x_{\mathrm{avg}}$ under different $\lambda_{1}$. The first row is clean
images. From the second row to the last row, $\lambda_{1}$ is 1.0, 1.5, 3.0,
5.0, and 20.0.

(a) Under PGD-20.

(b) Under PGD-100.

(c) Under C&W-100.

(d) Under AA.

Figure 5: Clean accuracy and robust accuracy of ResNet-18 (Solid Markers) and
ResNet-18-SR (Hollow Markers) with different training strategies on CIFAR-10.
The upper right corner means that the model has the best clean and robust
accuracy.

(a) Under PGD-20.

(b) Under PGD-100.

(c) Under C&W-100.

(d) Under AA.

Figure 6: Clean accuracy and robust accuracy of ResNet-18 (Solid Markers) and
ResNet-18-SR (Hollow Markers) with different training strategies on CIFAR-100.
The upper right corner means that the model has the best clean and robust
accuracy.

###  4.2 SiRIIB

As we show that the IB skip connection can be built with several basic
components, in this section we give a reliable example, combined with singular
regularization, named SiRIIB. The overview of SiRIIB is shown in Figure 3,
which contains three main steps, following the design principle of Proposition
1.

For an input image $x$, the first step in SiRIIB is to use the singular
regularization (SR) modules to recalibrate the singular values and singular
vectors. To better learn the general layout information from $x$ instead of
the fine-grained features, we introduce downsampling operations. With
downsampling operations, we can decouple detailed information in the images
with the layout information and obtain multiscale inputs. We first downsample
the $x$ into four different resolutions, i.e., $x$, $x^{\downarrow}$,
$x^{\downarrow\downarrow}$, and $x^{\downarrow\downarrow\downarrow}$, where
each $\downarrow$ means a downsample operation. We only use 4 scales, because
we consider a trade-off between computational complexity and the effectiveness
of decoupling. Specifically, the four different resolutions for the input are
32, 24, 16, and 8, where 32 is the original image resolution. For the input
under each scale, we use an independent SR module to regularize its singular
values and vectors. For example, we have

$\displaystyle
x^{\downarrow}_{\mathrm{SR}}=\mathrm{SR}^{\downarrow}(x^{\downarrow})$

for $x^{\downarrow}$. In the second step, we use the Sigmoid function to
compress the information from the SR modules and upsample the output to make
them have the same resolution as $x$. After that, we average them to smooth
the features of $x$ under different scales, i.e.,

$\displaystyle
x_{\mathrm{avg}}=\frac{1}{4}(\varsigma(x_{\mathrm{SR}})+\varsigma(x^{\downarrow}_{\mathrm{SR}})^{\uparrow}+\varsigma(x^{\downarrow\downarrow}_{\mathrm{SR}})^{\uparrow\uparrow}+\varsigma(x^{\downarrow\downarrow\downarrow}_{\mathrm{SR}})^{\uparrow\uparrow\uparrow}),$

where the $\uparrow$ is an upsampling operation, and $\varsigma$ represent the
Sigmoid function. After the upsampling operations, all outputs have the same
resolution. In the third step, we adopt convolutional layers $c(\cdot)$ to
first extract deep features and then use different convolutional layers, e.g.,
$p_{1}(\cdot)$ and $p_{2}(\cdot)$, to modify the feature dimensions, matching
the corresponding dimensions of the classification model’s features, i.e., for
the $i$-th feature,

$\displaystyle f_{i}=p_{i}(c(x_{\mathrm{avg}})).$

We simply add $f_{i}$ to the corresponding features in the classification
model. Note that, in ResNet [11] or WideResNet [26], before the first residual
block, there is a convolutional layer to convert the input $x$ to high-
dimensional features, and we count the outputs of this convolutional layer as
the first feature in the classification model. The detailed structures of
SiRIIB can be found in the supplementary materials.

To train SiRIIB with the classification model, we need to consider
constraining both $x_{\mathrm{avg}}$ and $f_{i}$ to meet the request that we
want SiRIIB to provide a more compressed representation of $x$ and perform
singular regularization simultaneously. More importantly, SiRIIB should
calibrate the misclassification information in the classification model with
correct representations. The representation compressing will be executed
implicitly due to the IB skip connection. Therefore, we only introduce two
loss terms to calibrate misclassification information, i.e.,
$L_{\mathrm{svd}}$ and $L_{\mathrm{info}}$. Specifically,

$\displaystyle L_{\mathrm{svd}}=\|\Sigma_{\mathrm{avg}}-\Sigma\|_{2}$
$\displaystyle+\|U_{\mathrm{avg}}V^{\prime}_{\mathrm{avg}}$ $\displaystyle-
UV^{\prime}\|_{2}+\|x_{\mathrm{avg}}-x_{\mathrm{clean}}\|_{2},$

where
$x_{\mathrm{avg}}=U_{\mathrm{avg}}\Sigma_{\mathrm{avg}}V^{\prime}_{\mathrm{avg}}$,
which is computed from an adversarial example $x_{\mathrm{adv}}$, and
$x_{\mathrm{clean}}=U\Sigma V^{\prime}$, which is the corresponding clean
image. And,

$\displaystyle
L_{\mathrm{info}}=\sum_{i}\|f_{i}-f_{i}^{\mathrm{clean}}\|_{2},$

where $f_{i}$ is computed from the adversarial example $x_{\mathrm{adv}}$, and
$f_{i}^{\mathrm{clean}}$ is computed from the clean image
$x_{\mathrm{clean}}$. Clearly, $L_{\mathrm{svd}}$ is proposed to calibrate
adversarial information in the singular values and vectors.
$L_{\mathrm{info}}$ is proposed to calibrate adversarial information in the
extracted features. Without them, SiRIIB will not converge to stable
conditions. To make the learnable weight matrix in SR orthogonal, we use the
penalty term $R_{\tau}$ in [28]. Overall, the training loss can be written as

$\displaystyle
L=L_{\mathrm{ori}}+\lambda_{1}(L_{\mathrm{svd}}+L_{\mathrm{info}})+\lambda_{2}R_{\tau},$

where $L_{\mathrm{ori}}$ is the original training loss, and $\lambda_{1}$ and
$\lambda_{2}$ are two hyperparameters.

Figure 7: Heatmap generated by ScoreCam for ResNet-18 and ResNet-18-SR. The
first row is clean images. The second row is heatmaps for clean images on
ResNet-18. The third row is heatmaps for clean images on ResNet-18-SR.

Model # of Parameters (M) Computational Complexity (G) ResNet-18 11.17 0.56
ResNet-18-SR 11.35 0.73 WRN-28-10 36.48 5.25 WRN-28-10-SR 36.79 5.57

Table 3: Comparisons of the number of parameters (M, million) and the amount
multiply-add operations (G, giga).

Loss $L_{\mathrm{CE}}$ $L_{\mathrm{svd}}$ $L_{\mathrm{info}}$
$L_{\mathrm{CE}}$+$L_{\mathrm{svd}}$ $L_{\mathrm{CE}}$+$L_{\mathrm{info}}$
$L_{\mathrm{CE}}$+$L_{\mathrm{svd}}$+$L_{\mathrm{info}}$ Robust Accuracy 52.53
82.64 83.11 52.56 52.51 52.61

Table 4: Adaptive attacks for SiRIIB with different attack loss functions
under PGD-20.

##  5 Experiments

In this section, we will first introduce the experiment settings. Then, we
provide detailed ablation studies to explore the number of $p_{i}(\cdot)$ and
$\lambda_{1}$. In our main experiments, we compare the results of models
trained with different strategies and datasets. We further provide analysis of
how the IB skip connections decrease the sensitivity of local perturbation
during the attacks. To compare the complexity of the model, we give details of
the number of parameters and multiply-add operations in the model. As we
introduce a new module, it is necessary to analyze the possible adaptive
attacks. We show the results in Section 5.5. Furthermore, we analyze the
singular regularization module with its outputs and performance in the
supplementary materials. Besides accuracy under adversarial attacks, we
evaluate accuracy under common corruptions in the supplementary materials.

###  5.1 Baselines

We choose two toy image datasets, CIFAR-10 [15] and CIFAR-100 [15], and one
real-world dataset Tiny-Imagenet [16], and two popular classification models,
ResNet-18 [11] and WideResNet-28-10 (WRN-28-10) [26]. For each model, we
improve it with SiRIIB, dubbed ResNet-18-SR and WRN-28-10-SR, respectively.
For adversarial training strategies, we consider four different methods, i.e.,
PGD-AT [18], TRADES [27], AWP [24], and TE [6]. Specifically, the adversarial
examples generated by each method are constrained under $L_{\infty}$-norm with
settings $\epsilon=8/255$, $\alpha=2/255$, and 10 steps. To train models, we
use SGD as an optimizer, with an initial learning rate of 0.1, momentum of
0.9, weight decay of $5e-4$, and batch size of 128. We decrease the learning
rate at the 100-th and 150-th epoch by multiplying 0.1. For $\lambda_{2}$ used
in SiRIIB, we directly use the value in [28], i.e., $\lambda_{2}=1e-4$. When
evaluating the robust models, we consider four $L_{\infty}$-normed attacks in
our main paper, and other attacks can be found in supplementary materials.
Four representative attacks are the PGD attack [18] with cross-entropy loss
under 20 and 100 steps (PGD-20 and PGD-100), PGD attack with the C&W loss [4]
under 100 steps (C&W-100), and AutoAttack (AA) [5]. All these attacks are
under $\epsilon=8/255$. Details of SiRIIB can be found in the supplementary
materials.

###  5.2 Ablation Study

We study different configurations in SiRIIB, including the number of
$p_{i}(\cdot)$, i.e., IB skip connections, and the weight $\lambda_{1}$ in
training loss. The results are obtained on CIFAR-10 with ResNet-18-SR, which
is trained under PGD-AT.

In Table 1, we set $\lambda_{1}=1.0$ and explore the effectiveness of
different skip connections. The results indicate that when we add three
$p_{i}(\cdot)$ to the original model, we can obtain the best trade-off between
clean accuracy and robust accuracy. Furthermore, we find that when there are 4
or 5 $p_{i}(\cdot)$, the model will not converge. Therefore, we do not show
them in the table. Too few skip connections or too many skip connections will
decrease the performance to a different degree. We think the reasons could be
that (1) too few skip connections will only bring information encoding
regularization to the top layers, which makes it hard to be consistent in the
deeper layers, (2) too many skip connections will disrupt the original
intermediate information encoding in the deeper layers, which is hard to be
extracted with a simple $p_{i}(\cdot)$. Although the second reason could be
addressed by using a more complex $p_{i}(\cdot)$, it will increase the
computational complexity. In Table 2, we use three skip connections in the
original model and explore the effectiveness of different $\lambda_{1}$. With
increasing $\lambda_{1}$, both clean accuracy and robust accuracy follow a
similar tendency. Therefore, we find that when $\lambda_{1}=20.0$, we can
obtain the best clean and robust accuracy simultaneously. In the following
experiments, we use three $p_{i}(\cdot)$ and $\lambda_{1}=20.0$.

Furthermore, we find that the information bottleneck will restrict some benign
information, hurting clean accuracy, because there exists a trade-off between
clean accuracy and robust accuracy under the information bottleneck theory
[1], which is similar to the original explicit optimization with the Lagrange
multiplier. To better show such influence in the information bottleneck, we
compare $x_{\mathrm{avg}}$ under different $\lambda_{1}$ in Figure 4. The
results indicate that a smaller $\lambda_{1}$ will make the model learn a more
compressed representation, containing very less information from the input. On
the other hand, a larger $\lambda_{1}$ will make $x_{\mathrm{avg}}$ contain
more information of the input. It is to say that we implicitly control the
strength of the information bottleneck with $\lambda_{1}$.

###  5.3 Main Results

In this section, we mainly compare the performance of models equipped with
SiRIIB and original models on different datasets. Furthermore, we explore the
regional faithfulness of the classification models from the view of
sensitivity to perturbation. Specifically, we verify the effectiveness of
SiRIIB from the point of view of information bottleneck theory in the
supplementary materials.

In Figure 5, we compare the clean accuracy and robust accuracy under different
attacks on CIFAR-10. Because the robust accuracy under different attacks is
various, to achieve the best view, we use different y-axis intervals for
different attacks. With SiRIIB, both the clean accuracy and the robust
accuracy will increase in general. However, we find that if adversarial
training methods adopt additional loss functions besides the cross-entropy
loss, such as TE with an MSELoss and TRADES with a KLDiv loss during the
training process, the improvement could be less significant than others. The
reason we think is that the additional training loss will make the
$\lambda_{1}$ in $L$ we find in Table 2 not optimal. Therefore, for different
training strategies, it is better to explore the optimal $\lambda_{1}$ to
achieve the best results. But in our experiments, to keep consistent and
universal, we use the same $\lambda_{1}$ for all training methods.

On the other hand, in Figure 6, we compare clean accuracy and robust accuracy
under different attacks on CIFAR-100. In most cases, our SiRIIB can provide
significant improvement for robust accuracy. This is because CIFAR-100
contains more classes than CIFAR-10, resulting in most training strategies not
utilizing the full potential of the classification model. However, powerful
training strategies, like AWP, already achieve very high clean accuracy and
robust accuracy, therefore, the improvement will be less than others. But,
SiRIIB can still improve the clean accuracy. For other results on WRN-28-10
and more, we put them in the supplementary materials.

We notice that most of the previous works in studying adversarial robustness
ignore the regional faithfulness of the classification results, i.e., which
area of the given image is the more sensitive to noise and will influence the
prediction. To analyze the sensitive area of the given images, we adopt
ScoreCam [22] to plot the heatmaps for ResNet-18 and ResNet-18-SR,
respectively, which can be found in Figure 7. In the heatmaps, the higher
temperature means changing such an area will have a more significant influence
on the final prediction. Comparing the heatmaps of ResNet-18 and ResNet-18-SR,
we find that the original model is more sensitive to the background and SiRIIB
can effectively address such a problem. We think it is because the adversarial
perturbation can hide some information in the background area, making the
model give incorrect predictions. On the other hand, our SiRIIB can provide
regularization for both singular values and vectors, which is effective in
removing background perturbation and decreases the mutual information between
the learned presentations and the inputs. Therefore, our method is effective
and interpretable.

###  5.4 Model Complexity

We compare the model complexity from two aspects, i.e., the number of model
parameters and the amount of multiply-add operations. The results are obtained
based on the tool, ptflops [20]. The results in Table 3 indicate that SiRIIB
only introduces about 0.8-1.6% additional parameters, and increases about
0.17-0.32 G additional multiply-add operations, which proves that SiRIIB is
efficient. We notice that when injecting SiRIIB into WideResNet, it will cause
a bigger increase in both the number of parameters and the amount of multiply-
add operations. This is because the feature dimensions in WideResNet are
higher. Therefore, we need to use more convolutional kernels to match the
dimensions, which will introduce more parameters and multiply-add operations.
In summary, SiRIIB is efficient.

###  5.5 Adaptive Attacks

As we introduce a new module into the original model, it is important to learn
the potential adaptive attacks. We consider building adaptive attacks based on
the loss terms $L_{\mathrm{svd}}$ and $L_{\mathrm{info}}$, because they will
influence the features extracted by SiRIIB. In Table 4, we show the robust
accuracy under PGD-20 attacks using different loss terms, where
$L_{\mathrm{CE}}$ is the cross-entropy loss, i.e., the default loss in PGD.
The results indicate that even under adaptive attacks, the model still obtains
a high robust accuracy. It is because we only use SiRIIB to restrict
intermediate features in the original models, instead of directly using the
features from SiRIIB. The visualization results can be found in the
supplementary materials.

##  6 Conclusion

In this paper, we study adversarial examples in the view of singular value
decomposition. We find that the position of adversarial information in
singular values and vectors is related to the attacks, instead of the training
methods of the models. Based on this observation, we propose a new plug-in
module, SiRIIB, for classification models. When designing SiRIIB, we
innovatively introduce information bottleneck theory. By proving a general
structure can fulfill the implicit information bottleneck regularization, we
design SiRIIB in the same way. Through comprehensive experiments, we show the
effectiveness of our method and prove its efficiency. We also give some
interpretable evidence to verify the effectiveness of SiRIIB. We hope that our
work can provide a new perspective on robust model design and inspire others.
Moreover, addressing the trade-off between clean accuracy and robust accuracy
is another future work.

## References

  * Alemi et al. [2017] Alexander A. Alemi, Ian Fischer, Joshua V. Dillon, and Kevin Murphy.  Deep Variational Information Bottleneck.  In _Proc. of the ICLR_ , 2017.
  * Athalye et al. [2018] Anish Athalye, Nicholas Carlini, and David A. Wagner.  Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examples.  In _Proc. of the ICML_ , pages 274–283, 2018.
  * Bai et al. [2021] Yutong Bai, Jieru Mei, Alan L. Yuille, and Cihang Xie.  Are transformers more robust than cnns?  In _Proc. of the NeurIPS_ , pages 26831–26843, 2021.
  * Carlini and Wagner [2017] Nicholas Carlini and David Wagner.  Towards Evaluating the Robustness of Neural Networks.  In _Proc. of the SP_ , pages 39–57, 2017.
  * Croce and Hein [2020] Francesco Croce and Matthias Hein.  Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks.  In _Proc. of the ICML_ , pages 2206–2216, 2020.
  * Dong et al. [2022] Yinpeng Dong, Ke Xu, Xiao Yang, Tianyu Pang, Zhijie Deng, Hang Su, and Jun Zhu.  Exploring Memorization in Adversarial Training.  In _Proc. of the ICLR_ , 2022.
  * Dosovitskiy et al. [2021] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby.  An image is worth 16x16 words: Transformers for image recognition at scale.  In _Proc. of the ICLR_ , 2021.
  * Federici et al. [2020] Marco Federici, Anjan Dutta, Patrick Forré, Nate Kushman, and Zeynep Akata.  Learning robust representations via multi-view information bottleneck.  In _Proc. of the ICLR_ , 2020.
  * Ge et al. [2023] Yao Ge, Yun Li, Keji Han, Junyi Zhu, and Xianzhong Long.  Advancing Example Exploitation Can Alleviate Critical Challenges in Adversarial Training.  In _Proc. of the ICCV_ , pages 145–154, 2023.
  * Goodfellow et al. [2015] Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy.  Explaining and Harnessing Adversarial Examples.  In _Proc. of the ICLR_ , 2015.
  * He et al. [2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep Residual Learning for Image Recognition.  In _Proc. of the CVPR_ , pages 770–778, 2016.
  * Hendrycks and Dietterich [2019] Dan Hendrycks and Thomas Dietterich.  Benchmarking neural network robustness to common corruptions and perturbations.  _Proc. of the ICLR_ , 2019.
  * Huang et al. [2023] Shihua Huang, Zhichao Lu, Kalyanmoy Deb, and Vishnu Naresh Boddeti.  Revisiting residual networks for adversarial robustness.  In _Proc. of the CVPR_ , pages 8202–8211, 2023.
  * Ilyas et al. [2019] Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Logan Engstrom, Brandon Tran, and Aleksander Madry.  Adversarial Examples Are Not Bugs, They Are Features.  In _Proc. of the NeurIPS_ , pages 125–136, 2019.
  * Krizhevsky et al. [2009] Alex Krizhevsky, Geoffrey Hinton, et al.  Learning multiple layers of features from tiny images.  2009\.
  * Le and Yang [2015] Ya Le and Xuan Yang.  Tiny imagenet visual recognition challenge.  _CS 231N_ , 7(7):3, 2015.
  * Lee et al. [2021] Jungbeom Lee, Jooyoung Choi, Jisoo Mok, and Sungroh Yoon.  Reducing Information Bottleneck for Weakly Supervised Semantic Segmentation.  In _Proc. of the NeurIPS_ , pages 27408–27421, 2021.
  * Madry et al. [2018] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu.  Towards Deep Learning Models Resistant to Adversarial Attacks.  In _Proc. of the ICLR_ , 2018.
  * Rice et al. [2020] Leslie Rice, Eric Wong, and J. Zico Kolter.  Overfitting in adversarially robust deep learning.  In _Proc. of the ICML_ , pages 8093–8104, 2020.
  * Sovrasov [2018-2023] Vladislav Sovrasov.  ptflops: a flops counting tool for neural networks in pytorch framework, 2018-2023.
  * Wall et al. [2003] Michael E. Wall, Andreas Rechtsteiner, and Luis M. Rocha.  Singular Value Decomposition and Principal Component Analysis.  In _A Practical Approach to Microarray Data Analysis_ , pages 91–109. Springer US, 2003.
  * Wang et al. [2020a] Haofan Wang, Zifan Wang, Mengnan Du, Fan Yang, Zijian Zhang, Sirui Ding, Piotr Mardziel, and Xia Hu.  Score-cam: Score-weighted visual explanations for convolutional neural networks.  In _Proc. of the CVPR Workshop_ , pages 111–119, 2020a.
  * Wang et al. [2020b] Yisen Wang, Difan Zou, Jinfeng Yi, James Bailey, Xingjun Ma, and Quanquan Gu.  Improving Adversarial Robustness Requires Revisiting Misclassified Examples.  In _Proc. of the ICLR_ , 2020b.
  * Wu et al. [2020] Dongxian Wu, Shu-Tao Xia, and Yisen Wang.  Adversarial Weight Perturbation Helps Robust Generalization.  In _Proc. of the NeurIPS_ , 2020.
  * Xie et al. [2019] Cihang Xie, Yuxin Wu, Laurens van der Maaten, Alan L Yuille, and Kaiming He.  Feature Denoising for Improving Adversarial Robustness.  In _Proc. of the CVPR_ , pages 501–509, 2019.
  * Zagoruyko and Komodakis [2016] Sergey Zagoruyko and Nikos Komodakis.  Wide Residual Networks.  In _Proc. of the BMVC_ , 2016.
  * Zhang et al. [2019] Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Eric P. Xing, Laurent El Ghaoui, and Michael I. Jordan.  Theoretically Principled Trade-off between Robustness and Accuracy.  In _Proc. of the ICML_ , pages 7472–7482, 2019.
  * Zhang et al. [2023] Jinghao Zhang, Jie Huang, Man Zhou, Chongyi Li, and Feng Zhao.  Decomposition Ascribed Synergistic Learning for Unified Image Restoration.  _CoRR_ , abs/2308.00759, 2023.

Model # of Parameters (M) Computational Complexity (G) Clean Accuracy PGD-20
PGD-100 C&W-100 AA WRN-28-10 36.48 5.25 86.78 53.68 53.48 53.22 50.92
WRN-28-10-SR 36.79 5.57 86.74 54.30 53.90 52.99 50.94 WRN-28-10-SR-L 37.89
6.70 86.66 54.26 53.93 53.40 51.22 WRN-28-10-SR-XL 38.81 7.65 86.91 54.55
54.25 53.90 51.53

Table 5: Comparisons between different WRN-28-10 structures.

(a) ResNet-18

(b) ResNet-18-SR

Figure 8: T-SNE results for ResNet-18 and ResNet-18-SR on CIFAR-10,
respectively. Best viewed in color.

##  Appendix A Proof

###### Proposition 2

If functional $\mathcal{G}(\cdot;\tau)$ is equipped with a double-sided
saturating activation function, e.g., Sigmoid, there exists
$g(\cdot;\tau)\in\mathcal{G}(\cdot;\tau)$, which can be used as an IB skip
connection.

###### Proof 1

To prove the existence of such a $g(\cdot;\tau)$ meeting the condition that
$I(z,x;\theta,\tau)\leq I(f(x;\theta),x;\theta)$, we only need to construct
such a function.

Let $z_{1}=f(x;\theta)$, $z_{2}=g(x;\tau)$, and $z=z_{1}+z_{2}$. We can
rewrite $I(z,x;\theta,\tau)$ and $I(f(x;\theta),x;\theta)$ as:

$\displaystyle I(z,$ $\displaystyle
x)=I(z_{1}+z_{2},x)=H(z_{1}+z_{2})-H(z_{1}+z_{2}|x)$ $\displaystyle
I(f(x;\theta),x)=I(z_{1},x)=H(z_{1})-H(z_{1}|x),$

where $H(\cdot)$ is the entropy. We aim to construct $g(x;\tau)$, fulfilling

$\displaystyle H(z_{1}+z_{2})-H(z_{1}+z_{2}|x)\leq H(z_{1})-H(z_{1}|x).$

Therefore, the construction is inspired from two aspects. First, $g(x;\tau)$
should decrease the uncertainty of $f(x;\theta)$, which loosely makes
$H(z_{1}+z_{2})\leq H(z_{1})+\epsilon$. Second, $g(x;\tau)$ should increase
the information of $f(x;\theta)$ extracted from $x$, which makes
$H(z_{1}+z_{2}|x)\geq H(z_{1}|x)$. Clearly, for the second aspect, adding a
new random variable to the conditional entropy will increase the value, i.e.,
$H(z_{1}+z_{2}|x)\geq H(z_{1}|x)$ holds in general.

To decrease the uncertainty of $f(x;\theta)$ by adding $g(x;\tau)$ to it, the
construction is designed as follows:

$\displaystyle g(x;\tau)$ $\displaystyle=g_{2}\circ\varsigma\circ g_{1}(x),$
$\displaystyle\varsigma(x)$ $\displaystyle=\frac{1}{1+e^{-x}},$

where $g_{1}$ and $g_{2}$ are two linear transformations. Then, we describe
the reason why this structure can make $H(z_{1}+z_{2})\leq H(z_{1})$.

Note that both $f(x;\theta)$ and $g(x;\tau)$ are calculated on the basis of
$x$, which means that they are not independent. So, $g_{1}$ and $g_{2}$ will
keep information of $x$ with the linear transformations, making $z_{1}$ and
$z_{2}$ share redundant information of $x$. On the other hand, $\varsigma$
maps the values to $(0,1)$, which reduces $H(\varsigma\circ g_{1}(x))$ and let
$g(x;\tau)$ learn a compressed representation. With a proper supervision
signal, for example, cross-entropy loss between predictions and ground-truth
labels, $g(x;\tau)$ will be an IB skip connection.

Therefore, we find a possible case of $g$, which can be trained as an IB skip
connection.

##  Appendix B Details of SiRIIB

We will introduce the details of the SiRIIB in three parts. First, we
introduce the details of SR [28]. To modulate singular vectors, we adopt one
convolutional layer, whose weight is an orthogonal matrix. After modulating
singular vectors, we adopt fast Fourier transforms to transfer the input into
the Fourier domain. Then, we use a convolutional layer to modulate the signal
and transfer it from the Fourier domain to the original domain. Finally, we
fuse the outputs with one convolutional layer and use a batch normalization
layer to stabilize the output. Specifically, the kernel size and stride are 1
for all convolutional layers in SR. The input channels and output channels of
the convolutional layers are 3 for RGB images. The number of channels of the
orthogonal kernel is 12.

Second, we introduce the details of $c(\cdot)$. The design of $c(\cdot)$ is
straightforward, in which we want to extract deep features with several
convolutional layers. Considering the computational complexity, we adopt three
convolutional layers with increasing channels, and the batch normalization
layer and ReLU activation layer are added after each convolutional layer. For
all convolutional layers, the kernel size is 3 and the stride is 1. The
numbers of output channels are 16, 32, and 64.

Finally, we introduce the details of $p_{i}(\cdot)$. Considering
$p_{i}(\cdot)$ is to modify the dimension of the extracted features of
$c(\cdot)$, we use one simple convolutional layer, whose kernel size is 3 and
stride is 1, to increase or decrease channels to match the target channel
amount. If needed, we add a downsampling operation to match the width and
height dimensions of the feature.

##  Appendix C Experiment Results

In this section, we present other results from our experiments to better
demonstrate the superiority of our method.

###  C.1 SiRIIB on Real-World Dataset

In our main paper, we only show the results on two toy datasets. In this
section, we will compare the results on a real-world dataset, Tiny-Imagenet.
We train the models with FGSM-AT, and evaluate models under various
$l_{\infty}$-norm attacks. Especially, because the image resolution is 64, we
increase the downsample resolution to 48, 24, and 16.

Model Clean Accuracy PGD-20 PGD-100 C&W-100 AA ResNet-18 42.47 33.92 33.91
32.14 31.85 ResNet-18-SR 43.52 34.96 34.97 33.25 32.85

Table 6: Evaluation on a real-world dataset, Tiny-Imagenet. The attack budget
$\epsilon=2/255$.

In Table 6, we compare the results of ResNet-18 and ResNet-18-SR. Clearly,
with SiRIIB, the model will obtain both higher clean accuracy and robust
accuracy. The improvement is about 1%. The results indicate that our method
works for real-world datasets.

###  C.2 SiRIIB on Different Model Structure

In this section, we compare SiRIIB on different model structures and evaluate
the different designs of $c(\cdot)$ in SiRIIB. In Table 5, we first compare
the results of WRN-28-10 and WRN-28-10-SR on CIFAR-10, trained with PGD-AT.
The results indicate that when equipped with SiRIIB, the model will obtain
higher robust accuracy under different attacks. It is to say that SiRIIB is
general for different model architectures. On the other hand, because
WRN-28-10 extracts deeper features than ResNet-18, we consider different
settings of $c(\cdot)$ to study the influence of the extracted features in
SiRIIB. Specifically, $c(\cdot)$ in WRN-28-10-SR is described in Section B.
For WRN-28-10-SR-L and WRN-28-10-SR-XL, we consider using more output
channels. In detail, we use two cascaded basic blocks as $c(\cdot)$ for
WRN-28-10-SR-L and four cascaded basic blocks as $c(\cdot)$ for WRN-28-10-SR-
XL. The total number of parameters and computational complexity are shown in
Table 5. From the results, we can find that when we use a more complex
$c(\cdot)$, robust accuracy will significantly increase, especially for
C&W-100 and AA. This is because a more complex $c(\cdot)$ will extract
features to better match the intermediate representations of the
classification model, which will help the classification model better
converge. Therefore, for a larger classification model, the better choice of
$c(\cdot)$ is to use a small deep convolutional model.

###  C.3 Visualization of Feature Representation

To prove the effectiveness of SiRIIB in feature learning, we show the T-SNE
results to study both representation distributions and compressions, just like
the previous work [8]. The visualization results in Figure 8 indicate that
with SiRIIB, the model will learn more compressed representations (axes of
both plots are zoomed to the same space). Furthermore, the features are better
separated under the T-SNE views. Therefore, SiRIIB works like implicitly
adding an information bottleneck to the training process.

###  C.4 Robust Accuracy under $l_{2}$ Attacks

We show robust accuracy under various $l_{2}$-normed attacks to prove that our
method is general for adversarial attacks under different norms. We consider
PGD attacks and C&W attacks. For both, we set $\epsilon=0.5$ and $\alpha=0.1$.
In Table 7, the results indicate that SiRIIB can improve the robustness under
$l_{2}$-normed attacks, which means that our proposed method is general.

Model Clean Accuracy PGD-20 PGD-100 C&W-100 ResNet-18 83.28 62.38 61.97 60.08
ResNet-18-SR 83.27 62.88 62.44 60.65

Table 7: Robust accuracy under $l_{2}$-normed attacks. Figure 9: Visualization
results of $x$, $x_{\mathrm{adv}}$, and $x_{\mathrm{avg}}$. The first row is
clean images $x$. The second row shows the adversarial examples
$x_{\mathrm{adv}}$. The third row represents $x_{\mathrm{avg}}$, whose input
is $x_{\mathrm{adv}}$. The fourth row is $x-x_{\mathrm{adv}}$. The fifth row
is $x_{\mathrm{avg}}-x_{\mathrm{adv}}$.

Input Clean Accuracy PGD-20 PGD-100 C&W-100 AA $x$ or $x_{\mathrm{adv}}$ 83.28
51.28 50.97 49.53 47.43 $x_{\mathrm{avg}}$ 80.95 52.80 52.76 61.44 54.65

Table 8: The clean accuracy and robust accuracy under the input of $x$,
$x_{\mathrm{adv}}$, and $x_{\mathrm{avg}}$, respectively.

###  C.5 Are Singular Values and Vectors Corrected?

To better study how the singular regularization works in the model inference
phase, we show the visualization results in Figure 9. Additionally, we
normalize both $x-x_{\mathrm{adv}}$ and $x_{\mathrm{avg}}-x_{\mathrm{adv}}$ to
obtain the best views. From the plots, we can find that the singular
regularization will remove some details from the input images, which
correspond to the small singular values in the SVD. In other words, singular
regularization keeps the main layout information and removes the additional
unimportant details, which achieves information compression. This observation
is aligned with our design from the perspective of information bottleneck
theory.

On the other hand, we explore how much adversarial information has been
removed in $x_{\mathrm{avg}}$. To obtain the results, we first isolate
$x_{\mathrm{avg}}$ from the model training process. Then, we only train the SR
module on the adversarial examples generated by a ResNet model during its
adversarial training process. Therefore, $x_{\mathrm{avg}}$ will be a
regularized input. When evaluating the removed adversarial information, we
first generate adversarial examples $x_{\mathrm{adv}}$ with a ResNet model.
Then we calculate $x_{\mathrm{avg}}$ based on $x_{\mathrm{adv}}$ and input
$x_{\mathrm{avg}}$ to the ResNet model. That is to say, we consider a grey-box
attack, where the adversary has no information about the SR module. As we only
care about the adversarial information removed by SR module, this grey-box
attack is reasonable. In Table 8, we compare clean accuracy when the input is
$x$ and $x_{\mathrm{avg}}$, which is calculated on $x$. Then, we compare
robust accuracy when the input is $x_{\mathrm{adv}}$ and $x_{\mathrm{avg}}$,
which is calculated on $x_{\mathrm{adv}}$. For clean accuracy, we find the
removed details will slightly decrease it. It is because when predicting the
image’s class, the model will use all information in the image. Removing
benign information will have a negative impact on the model. For PGD-20 and
PGD-100, we find robust accuracy increases by about 2%. For C&W-100 and AA,
the improvement is more significant. These results are aligned with the
observation in Figure 1 in the main paper, where we find that adversarial
examples generated by C&W-100 and AA are much easier to be purified by the
singular regularization. Therefore, our analysis proves that the singular
regularization will modify the singular values and vectors and remove the
adversarial attack information.

###  C.6 Accuracy under Common Corruptions

In addition to adversarial perturbation, we consider the influence of common
corruptions [12]. Similarly, we isolate the SR modules during adversarial
training and use them again in the inference phase to avoid the trade-off
between information compression and representation expression. In Table 9, we
compare the accuracy of ResNet-18 and ResNet-18-SR under different common
corruptions. The results indicate that with singular regularization, the
proposed method can further defend against common corruptions, even the model
is trained on adversarial examples.

Type ShotNoise GaussianNoise ImpulseNoise MotionBlur GlassBlur DefocusBlur
ZoomBlur JPEGCompression ElasticTransform Pixelate Fog Frost ResNet-18 75.59
75.17 66.76 67.97 73.31 71.22 73.94 80.28 75.38 78.33 28.03 68.20 ResNet-18-SR
77.60 77.11 68.67 70.15 74.25 72.55 75.35 80.91 75.97 78.93 30.33 69.36

Table 9: Accuracy under various common corruptions.

###  C.7 Additional Visualization Results

We provide more visualization results in Figure 10. To better show the
singular regularization effects, we compare the $x_{\mathrm{avg}}$ under
adversarial examples generated by PGD-20 and C&W-100, respectively. From the
results, we can find that the SR module is stable under different attacks.
Furthermore, we explore whether adaptive attacks will influence the
performance of our SR module. In Figure 11, we compare the $x_{\mathrm{avg}}$
under different adaptive attacks. The results indicate that the SR module can
maintain the calibration function even under our designed adaptive attacks.

Figure 10: Visualization results of $x$, $x_{\mathrm{adv}}$, and
$x_{\mathrm{avg}}$. The first row is clean images $x$. The second row shows
the adversarial examples $x_{\mathrm{adv}}$ generated by PGD-20. The third row
represents $x_{\mathrm{avg}}$, whose input is the second row. The fourth row
is $x_{\mathrm{adv}}-x_{\mathrm{avg}}$ of the two above rows. The fifth row is
the adversarial examples $x_{\mathrm{adv}}$ generated by C&W-100. The sixth
row represents $x_{\mathrm{avg}}$, whose input is the fifth row. The seventh
row is $x_{\mathrm{adv}}-x_{\mathrm{avg}}$ of the above two rows. Figure 11:
Visualization results of $x$, $x_{\mathrm{adv}}$, and $x_{\mathrm{avg}}$. The
first row is clean images $x$. The second row shows $x_{\mathrm{avg}}$, whose
input is adversarial examples generated by PGD-20 with $L_{\mathrm{CE}}$. The
third row is $x_{\mathrm{avg}}$, whose input is adversarial examples generated
by PGD-20 with $L_{\mathrm{info}}$. The second row represents
$x_{\mathrm{avg}}$, whose input is adversarial examples generated by PGD-20
with $L_{\mathrm{svd}}$.


# Convection, Heat Generation and Particle Deposition in Direct Laser Writing
of Metallic Microstructures

Thomas Palmer<EMAIL_ADDRESS>Fraunhofer Institute for
Industrial Mathematics ITWM, 67663 Kaiserslautern, Germany  Erik H. Waller
Physics Department and State Research Center OPTIMAS, Technische Universität
Kaiserslautern, 67663 Kaiserslautern, Germany  Heiko Andrä  Fraunhofer
Institute for Industrial Mathematics ITWM, 67663 Kaiserslautern, Germany
Konrad Steiner  Fraunhofer Institute for Industrial Mathematics ITWM, 67663
Kaiserslautern, Germany  Georg von Freymann  Physics Department and State
Research Center OPTIMAS, Technische Universität Kaiserslautern, 67663
Kaiserslautern, Germany  Fraunhofer Institute for Industrial Mathematics ITWM,
67663 Kaiserslautern, Germany

###### Abstract

Three-dimensional metallic microstructures find applications as stents in
medicine, as ultrabroadband antennas in communications, in micromechanical
parts or as structures of more fundamental interest in photonics like
metamaterials. Direct metal printing of such structures using three-
dimensional laser lithography is a promising approach, which is not
extensively applied yet, as fabrication speed, surface quality, and stability
of the resulting structures are limited so far. In order to identify the
limiting factors, we investigate the influence of light-particle interactions
and varying scan speed on heat generation and particle deposition in direct
laser writing of silver. We introduce a theoretical model which captures
diffusion of particles and heat as well as the fluid dynamics of the photo-
resist. Chemical reactions are excluded from the model but particle production
is calibrated using experimental data. We find that optical forces generally
surmount those due to convection of the photo-resist. Simulations predict
overheating of the photo-resist at laser powers similar to those found in
experiments. The thermal sensitivity of the system is essentially determined
by the largest particles present in the laser focus. Our results suggest that
to improve particle deposition and to achieve higher writing speeds in metal
direct laser writing, strong optical trapping of the emerging particles is
desirable. Furthermore, precise control of the particle size reduces the risk
of spontaneous overheating.

Figure 1: Left: Illustration of MDLW process: a pulsed laser beam is tightly
focussed into a photo-resist, this locally initiates a photo-reaction which
ultimately results in a small solid portion of the final structure. Scanning
the laser focus on a 2D or 3D trajectory, the desired structure is written.
Right: SEM image of lines written with MDLW. The granularity indicates that
the structure consists of silver nano particles that agglomerate.

Direct laser writing of metallic micro-structures (MDLW) emerges as a
fabrication method1 complementing the established process of polymer-based
direct laser writing (DLW). DLW is proven to yield highly sophisticated 3D
polymeric microstructures and, hence, is ideally suited for, e.g., micro-
optical applications2, 3, 4, 5. Structures produced using DLW can serve as
template for metallic structures6, 7, 8, 9, yet, such fabrication methods
limit geometric freedom or require electrically conductive substrates. MDLW
enables the _direct_ fabrication of metallic structures on almost arbitrary
substrates. Structures produced via MDLW exhibit an electrical conductivity
similar to that of the bulk medium10, which gives them an advantage over
fabrication methods in which metal nano-particles are initially embedded into
polymers11 or ligands12, 13.

MDLW has been used for fabrication of microelectronics 14, 15, micro-sensors
16 in micro-fluidic channels and writing on flexible substrates 17. To improve
fabrication of more sophisticated structures, e.g. photonic meta-materials or
micro-optical devices, higher fabrication speeds and smoother surface finish
are desirable. In addition, the growing nano-particles partially absorb the
incident laser light and heat the surrounding photo-resist, which can lead to
evaporation and disruption of the writing process. Key to these challenges is
the strong laser-particle interaction14 whose role in the MDLW process we aim
to better understand in order to increase fabrication speed as well as
structural quality.

The underlying effects that determine the above mentioned phenomena have
individually been investigated in depth, both experimentally and by computer
simulations, e.g., plasmonic heating18, 19, 20, 21 and induced fluid
convection22, particle trapping23, 24, 25, 26, 27 and maximum translation
velocities of nano-particles28. However, the above studies only treat quasi-
equilibrium situations where the number of particles is fixed and the laser in
rest22, 29, 21, or, only a single particle in a moving laser focus is
considered28. For the competing micro fabrication process of Direct Ink
Writing with on-the-fly annealing30, the temperature evolution has been
addressed with simulations. In a continuous particle generation-deposition
process as is the case in MDLW, diffusive spreading and deposition of
particles as well as heating and convection of the photo-resist all depend on
each other and on the laser scan speed. In this paper, we propose a simulation
model which depicts these effects. Thereby, we investigate the interplay of
optical configuration and scan speed on the writing process and characterize
the relative importance of the various phenomena involved in the process.

We set off from an idealized source of particles, located at the laser focus.
Since most commonly silver is used in MDLW, we only consider silver structures
and nano-particles in this paper, though our model is not restricted to silver
and can be applied to other materials straightforwardly.

This paper is organized as follows: After revisiting optical trapping of nano-
scale particles, we introduce our simulation model. Then we present results
for the convection velocity of the photo-resist and the temperature increase
as a function of laser power. We analyze the effect of the particle size on
the heat generation within the fluid. Then, we derive the laser powers
corresponding to the overheating threshold and compare them to experimental
findings. In the last section, we demonstrate, how optical forces can improve
the deposition of particles in MDLW, and the analysis of experimentally
measured particle production rates is shown. We conclude with a few remarks
concerning possible refinements of the simulations and improvements of the
MDLW process. Trailing is a method section where experimental methods and
simulations are described in further detail; numeric values of the model
parameters are given and boundary conditions are specified and motivated.

##  Results and Discussion

### Laser particle trapping

Reduction and nucleation of silver starts in the focus of the laser beam,
where the generated nano-particles are subject to the electrical field
$\mathbf{E}_{\text{ext}}$ of the incident light. The fields induced in the
particles by $\mathbf{E}_{\text{ext}}$ interact with the latter, resulting in
a net force on the particles. Most important is the induced dipole moment
$\mathbf{p}$ which causes a force

$\mathbf{F}_{\nabla}=\nabla(\mathbf{p}\cdot\mathbf{E}_{\text{ext}})=\frac{n_{\text{liq}}\alpha}{2c_{0}}\nabla
I$ (1)

proportional to the gradient of the irradiance $I$. Here $c_{0}$ is the speed
of light in vacuum. The polarizability $\alpha=3V_{i}\frac{m^{2}-1}{m^{2}+2}$
depends on the particle’s volume $V_{i}$ and the ratio of refractive indices
$m=n/n_{\text{liq}}$ of particle and surrounding media31, respectively. Eq.
(1) describes a conservative force field with potential $\propto-I$.

For the cross-section $\sigma_{\text{pr}}$ corresponding to the force due to
radiation pressure (a.k.a. phase gradient force, scattering force), we use the
result obtained from Mie-theory31. Then we can determine the force due to
radiation pressure

$\mathbf{F}_{\text{pr}}=(\nabla\Phi)\frac{1}{\omega}\sigma_{\text{pr}}I$ (2)

where $\Phi$ denotes the total phase and $\omega$ is the angular frequency of
the electrical field. Radiation pressure becomes more and more important with
growing particle size; eventually it surmounts the attractive gradient force
(1), so that larger particles cannot be confined but are ejected from the
focus along the beam axis. For the total optical force we write
$\mathbf{F}_{\text{\\!opt}}\equiv\mathbf{F}_{\\!\nabla}+\mathbf{F}_{\text{\\!pr}}$.

### Model equations

In case of equilibrium and a conservative force field, any initial particle
distribution will relax into the corresponding Boltzmann-distribution. In our
case, however, the laser moves through the photo-resist and continuously
produces silver particles. This requires to solve the transport equation

$\partial_{t}c+\nabla\cdot\big{\\{}(\mathbf{u}+\mu\mathbf{F}_{\text{\\!opt}})c\big{\\}}={}-\nabla\cdot(D\nabla
c)+S_{\\!c}$ (3)

for the local particle concentration $c$. The term
$(\mathbf{u}+\mu\mathbf{F}_{\text{\\!opt}})c$ captures advection of particles
due to optical forces $\mathbf{F}_{\text{\\!opt}}$ and the local velocity
$\mathbf{u}$ of the surrounding photo-resist, taken in the reference frame of
the laser. Here, particle diameter $d$ and fluid viscosity $\eta$ determine
the mobility $\mu=(3\pi\eta d)^{-1}$. Diffusion is assumed to be Fickian with
diffusion constant given by the Einstein-Smoluchowski relation32 $D=\mu
k_{\text{B}}T_{0}$ where $T_{0}$ is the ambient temperature and $k_{\text{B}}$
the Boltzmann constant. In a reference frame centered around the moving laser
focus, particle production is represented by the spatially steady source term

$S_{\\!c}(x,y,z)\propto\exp\left\\{-4\frac{x^{2}+y^{2}}{w_{0}^{2}}-\frac{z^{2}}{z_{\text{R}}^{2}}\right\\}$
(4)

where $w_{0}$ is the beam waist at the focus and $z_{R}$ the Rayleigh length.
The half widths of $S_{\\!c}$ correspond to those of the squared irradiance
$I^{2}$ that typically governs two-photon processes (c.f. ref. 14, figure 1).

The dynamics of the surrounding photo-resist is described by the Navier-Stokes
equations

$\displaystyle\begin{split}\rho_{0}\dfrac{\partial\mathbf{u}}{\partial
t}+\rho_{0}(\mathbf{u}\cdot\nabla)\mathbf{u}={}&-\nabla
P+\eta\nabla^{2}\mathbf{u}\\\
&+\rho_{0}\mathbf{g}\big{(}1-\beta(T-\overline{T})\big{)},\end{split}$ (5a)
$\displaystyle\nabla\cdot\mathbf{u}={}$ $\displaystyle 0$ (5b)

in the Boussinesq approximation33, where $\mathbf{u}$ and $P$ denote fluid
velocity and pressure, respectively. Deviations of the local temperature $T$
from the average $\overline{T}$ due to the heat that is transferred from
particles to fluid constitute the sole driving force. This buoyancy force is
proportional to the fluid’s thermal expansion coefficient $\beta$, the
gravitational acceleration $\mathbf{g}$ and the unperturbed fluid density
$\rho_{0}$ at ambient temperature.

Heat propagation within the fluid is described by the heat equation

$\rho_{0}c_{p}(\partial_{t}T+\mathbf{u}\cdot\nabla T)=-\nabla\cdot(k\nabla
T)+c\sigma_{\text{abs}}I$ (6)

where $k$ and $c_{p}$ denote the thermal conductivity and the heat capacity,
respectively, of the photo-resist. As the photo-resist is transparent at the
laser wavelength, we consider particles to be the only substance to accumulate
heat as they partially absorb incoming light. Thus we put the heat source term
$c\sigma_{\text{abs}}I$ as product of local particle concentration $c$, their
absorption cross-section $\sigma_{\text{abs}}$ and irradiance $I$. This very
slightly overestimates the total heat input as it does not account for
particles up the beam axis shading those further down. Strong shielding,
however, would interrupt the laser writing process and thus is ruled out by
experience.

### Convection and heating

In order to quantify the impact of buoyancy on the MDLW process, we simulate
the influence of varying laser power on the motion of the photo-resist and the
peak temperatures in the vicinity of the focus.

We assume a particle diameter of $45\,$\mathrm{n}\mathrm{m}$$ which is a size
typically found on SEM images of our samples and is already large enough to be
efficiently trapped and heated.

Depending on the laser power and particle diameter, simulations show that
steady state is reached after $50-200\,$\mathrm{ms}$$. We observe a non-linear
increase of the peak temperatures with growing laser power $P_{0}$, which is
attributed to the combined action of increased irradiance and trapping (i.e.
higher particle concentrations exposed to higher light intensity), see figure
2.

Figure 2:  Maximum temperature increase $\Delta T$ relative to ambient (top
axes) and peak fluid velocity $u_{z}$ (bottom axes) as a function of laser
power. The dashed line indicates the boiling point assuming
$20\text{\,}\mathrm{\SIUnitSymbolCelsius}$ ambient temperature. The particle
diameter is set to 45 nm, NA=1.2, scan speed
$v=10\,$\mathrm{\SIUnitSymbolMicro m}\text{\,}{\mathrm{s}}^{-1}$$. Note that
the scaling of both quantities is identical; the fluid velocity behaves
proportional to temperature increase.

The fluid develops a typical toroidal convection pattern centered around the
beam axis. Maximum velocity is reached slightly above the focus, momentum then
diffuses laterally which leads to a linear decrease of $u_{z}$, see figure 3.

Figure 3: Vertical component of fluid velocity $u_{z}$ (dashed lines) compared
to advection of particles $\mu F_{z}$ due to optical forces (solid lines)
along beam axis $z$. Wavelength 780 nm, NA=1.2, scan speed
$10\text{\,}\mathrm{\SIUnitSymbolMicro m}\text{\,}{\mathrm{s}}^{-1}$, 45 nm
particles produced at a rate of $0.52\text{\,}{\mathrm{\SIUnitSymbolMicro
m}}^{3}\text{\,}{\mathrm{s}}^{-1}$. Color encodes laser power from 4 mW
(violet) up to 20 mW (yellow). Maximum fluid velocity is reached slightly
above focus, see zoomed inset.  Figure 4: Vertical component of fluid velocity
$u_{z}$ (dashed lines) compared to advection of particles $\mu F_{z}$ due to
optical forces (solid lines) plotted along a horizontal axis through laser
focus. Colors and fabrication parameters as given in figure 3. Vertical fluid
velocity is much smaller than advection due to optical forces, see zoomed
inset.

As shown in figs. 4 and 3, the fluid velocities are negligible compared to the
advection $\mu\mathbf{F}_{\text{\\!opt}}$ of particles by optical forces; the
Peclet numbers which relate diffusion to fluid velocity $\mathbf{u}$ are much
smaller than unity, see table 1.

Table 1: Peclet und Reynolds numbers in MDLW

| referring to $\mathbf{u}$ | referring to $\mathbf{v}$
---|---|---
thermal Peclet $\operatorname{\mathit{P\kern-1.4pte}}_{T}$ | $7\\!\cdot\\!10^{-9}$ | $7\\!\cdot\\!10^{-6}\dots 7\\!\cdot\\!10^{-4}$
particle Peclet $\operatorname{\mathit{P\kern-1.4pte}}_{c}$ | $(0.4\dots 1.4)\\!\cdot\\!10^{-4}$ | $0.1\dots 10$
fluid Reynolds $\operatorname{\mathit{R\kern-0.80002pte}}$ | $10^{-8}$ | –

The Peclet numbers $\operatorname{\mathit{P\kern-1.4pte}}_{T},\
\operatorname{\mathit{P\kern-1.4pte}}_{c}$ are given with respect to both
fluid velocity $\mathbf{u}$ and laser scan speed $\mathbf{v}$; only for the
latter they approach or surpass unity.

Despite particles are permanently produced and spread diffusively from the
focus, they do not present a heat source large enough to cause a problematic
amount of buoyancy. The generality of the maximum convection velocities
obtained from our simulations is supported by two important characteristics
inherent to the MDLW process: First, the temperature is naturally limited by
the boiling point. Second, the spatial extent of the heat source only covers
the laser focus, outside the focal region, the temperature decays
reciprocally. Both limit the total momentum input due to buoyancy into the
fluid.

Furthermore, this analysis shows that transport of particles due to the
scanning movement of the focus can be important and impact their local
concentrations, whereas heat diffusion is not expected to be affected by
advection, neither of scanning nor fluid motion.

It follows that the back coupling of the fluid velocity $\mathbf{u}$ into eqs.
(6), (3) is not important, i.e. particle concentration and temperature profile
are hardly affected by advection. Consequently, eqs. (6), (3) become linearly
scalable in $S_{\\!c}$ and the advection $\mathbf{u}$ can be substituted by
the laser scan speed $\mathbf{v}$. Regarding the regime of Reynolds numbers,
the Navier-Stokes equations are very well approximated by the linear Stokes
equations. Together, these findings allow to generalize the results of a
single simulation to different particle production rates $S_{\\!c}$.

### Overheating threshold

As the particle diameter heavily affects the absorption properties and a broad
distribution of diameters is present in experiments, we examine the importance
of three exemplary diameters $d=30,45,60\,$\mathrm{n}\mathrm{m}$$ to the heat
generation in a model system with varying fraction of these species. Smaller
diameters are discarded because of their low absorption efficiency; this
assumption is proven by the results of this study, see figure 5.

Figure 5:  Overheating threshold laser powers $P_{\text{thresh}}$ as a
function of scanning speed $v$. Grey area indicates data obtained from
experiments. Simulations were conducted for exemplary particle mixtures with
either two species in equal parts or a single one. Total particle production
is adapted to experimentally obtained deposition values. Violet lines
represent 60 nm particles (solid) or mixtures in equal parts with 45 nm (dash-
dotted) or 30 nm particles, respectively. Yellow lines represent 45 nm
particles (dash-dotted) or a mixture in equal parts with 30 nm particles
(dashed). Note that a system consisting entirely of 30 nm particles does not
reach the threshold temperature for laser powers up to 20 mW.

Silver structures larger than 60 nm are rarely observed on SEM-images (see
fig. 1), they are mostly found on top of the written lines and at the same
time are spread over the entire substrate. Hence we conclude that they are the
result of a post-writing fallout, so they are not immediately produced during
the writing process in the vicinity of the laser focus and thus should not
contribute to heat generation.

In simulations, overheating is considered to occur if the temperature $T$
increases to or beyond $100\text{\,}\mathrm{\SIUnitSymbolCelsius}$.
Experimentally, the situation is not as explicit since no clear damage
threshold power exists. Rather, microbubbles increasingly disturb the writing
process with increasing laser power. From microscopy images we therefore
define the damage threshold power as the power at which a written line is no
longer connected at at least one position.

Figure 5 displays the experimental threshold region and simulations for a few
exemplary particle mixtures; we consider all cases where a diameter species
constitutes either 50% or 100% of the total deposited volume. For laser powers
up to $20\text{\,}\mathrm{mW}$, a system comprising solely 30 nm particles
does not reach the threshold temperature. This supports our choice of
dismissing particles smaller than 30 nm from the outset. The presence of 60 nm
particles significantly lowers the threshold powers which then match the
values obtained from experiments. Accordingly, the lowest threshold is
observed for a system consisting of 60 nm particles only. Mixtures containing
both 60 nm particles and one of the smaller diameters in equal shares show
increased thresholds that differ only very slightly from each other. Hence,
the biggest species present in the system determines the thermal sensitivity.
This is attributed to both the better trapping of bigger particles and the
strong increase of the absorption cross section $\sigma_{\text{abs}}$ in this
size regime. Together, this results in higher concentrations of particles in
the laser focus and a much increased total absorbance.

### Deposition efficiency

In the following, we examine the influence of optical forces on the deposition
of particles onto the substrate in order to investigate a possible increase of
fabrication speeds.

Figure 6: Relative amount of particles deposited at substrate as a function of
scan speed. Particle diameters from 30 nm (left) to 60 nm (right). Laser power
6mW, wavelength 780 nm, NA=0.3 (yellow), NA=0.8 (green), NA=1.2 (violet).
Towards higher speeds, deposition efficiency decreases stronger for bigger
particles; they are flushed out of the laser focus where trapping and
radiation pressure aid deposition. Small particles are less influenced by scan
speed since their high diffusivity dominates (small
$\operatorname{\mathit{P\kern-1.4pte}}$-numbers).

In simulations, this is carried out by integrating the vertical flux of
particles $j$ onto the substrate, i.e. on the bottom face of the simulation
domain. Normalizing this result to the total particle production rate
$S_{\\!c}$ yields the deposition efficiency $q_{\text{depo}}$ (see methods
section for definition and determination thereof) which, since $j\propto
S_{\\!c}$, is independent of the model input parameter $S_{\\!c}$. This
dimensionless quantity represents the physical efficiency of the MDLW-process
and gives the fraction of particles that adsorb at the structure, its inverse
represents particles that diffuse away and appear as debris on the substrate.
Note that the total efficiency of MDLW comprises the quantum-yield of the
2-photon process, the chemical reactions induced by the latter and nucleation
and re-dissolution of silver. These processes are not included in the
deposition efficiency $q_{\text{depo}}$ and are generally beyond the scope of
this paper.

Figure 7: Relative amount of particles deposited at substrate as a function of
laser power for particle diameters from 30 nm (left column) to 60 nm (right
column). Exemplary scan speeds from $10\,$\mathrm{\SIUnitSymbolMicro
m}\text{\,}{\mathrm{s}}^{-1}$$ (violet) to $100\,$\mathrm{\SIUnitSymbolMicro
m}\text{\,}{\mathrm{s}}^{-1}$$ (yellow), wavelength 780 nm, NA=1.2. The gray
areas indicate laser powers above the threshold of boiling of the photo-resist
obtained from simulations for the corresponding particle diameter (see fig.
5).

Small particles show a low deposition efficiency
($q_{\text{depo}}(d\\!\\!=\\!\\!30\,$\mathrm{n}\mathrm{m}$)\lesssim 0.5$) and
respond insensitively to variations of optical forces by means of laser power
$P_{0}$ and NA, see figs. 6 and 7. This is attributed to their higher
diffusivity combined with the relatively small susceptibility to optical
forces, leading to small Peclet-numbers and diffusion-dominated behavior (c.f.
table 1).

Larger particles are more affected by changes to the optical setup; switching
from NA=1.2 to NA=0.3 which comes along with much smaller optical force,
halves the deposition efficiency of 60 nm particles uniformly for all scan
speeds under consideration (c.f. fig. 6, right graph). On the contrary,
increasing the laser power and a high NA greatly improve deposition
efficiency. It approaches unity for 60 nm particles at moderate scan speeds up
to $\approx\\!30\,$\mathrm{\SIUnitSymbolMicro m}\text{\,}{\mathrm{s}}^{-1}$$,
see center and right graph of figure 7. At higher scan speeds (c.f. fig. 6),
we observe a significant decrease of $q_{\text{depo}}$ for all particle
species. This effect could be mitigated by increased laser power, while in the
bounds for $P_{0}$ given above. Therefore we conclude that trap strength is
important for the efficient deposition of particles in MDLW.

AFM height maps of silver lines fabricated at different scan speeds provide
the cross-sections $A$ and the surface roughness of these lines (not shown
here). Multiplication with the scan speed yields the net volumetric silver
production rate $S_{V}=Av$. For a given diameter, the numeric particle
production rate $S_{\\!c}$, which is fed into eq. (3), follows from division
by the particle volume corresponding to that diameter.

Table 2: Analysis of AFM measured cross-sections

|  $g\ $\mathrm{(}\mathrm{\SIUnitSymbolMicro}\mathrm{m}^{3}\mathrm{/}\mathrm{s}\mathrm{)}$$ |  $b\ $\mathrm{(}\mathrm{\SIUnitSymbolMicro}\mathrm{m}^{2}\mathrm{)}$$
---|---|---
$\operatorname{\mathrm{N\kern-1.00006ptA}}\,0.3$ |  0.401(30) |  0.0786(31)
$\operatorname{\mathrm{N\kern-1.00006ptA}}\,0.8$ |  0.212(19) |  0.0548(14)
$\operatorname{\mathrm{N\kern-1.00006ptA}}\,1.2$ |  0.0687(54) |  0.0263(7)
$\operatorname{\mathrm{N\kern-1.00006ptA}}\,1.2$, #2 |  0.264(83) |  0.0187(20)

Parameters obtained from error-weighted least-square fits of eq. (7) to line
cross-sections obtained from AFM measurements. Scan speed ranges from $3\dots
21\,$\mathrm{\SIUnitSymbolMicro m}\text{\,}{\mathrm{s}}^{-1}$$ and laser power
was 3 mW. The second run with NA=1.2 was produced with a different batch of
photo-resist and with scan speeds from $11\dots
101\,$\mathrm{\SIUnitSymbolMicro m}\text{\,}{\mathrm{s}}^{-1}$$.

The line cross-sections show a behavior similar to the expected reciprocal
decay with increasing scan speed. Thus we fitted line cross-sections $A$ with

$A(v)=\frac{g}{v}+b$ (7)

The parameters $g$ and $b$ are subject to optimization by the fit routine, see
table 2 for results.

We compared the residuals of fits of eq. (7) and the candidate modifications

$\displaystyle A(v)=q_{\text{depo}}(d,v)\frac{g}{v}+b,$ (8a) $\displaystyle
A(v)=q_{\text{depo}}(d,v)\left(\frac{g}{v}+b\right).$ (8b)

These modifications yield the parameters $g,b$ corresponding to the gross
particle production in consideration of the associated theoretical deposition
efficiency. Fits of eq. (8a) describe experimental data very well and yield
residuals practically identical to those obtained from the unmodified fit
function. Eq. (8b) fails to describe experimental data at scan speeds above
$60\text{\,}\mathrm{\SIUnitSymbolMicro m}\text{\,}{\mathrm{s}}^{-1}$. Hence,
the baseline production $b$ and associated deposition are insensitive to the
scan speed; we suggest that the baseline-production is promoted by the
substrate and these particles aggregate immediately.

##  Conclusions

We introduced a model for the simulation of direct laser writing of metallic
structures (MDLW) and discussed its results concerning convection, heating and
deposition.

Convection of photo-resist does not affect the deposition of particles. Since
rapidly increasing velocity with growing heat source has been reported 22,
care must be taken, if the heated, i.e. illuminated, area is increased, e.g.,
if a second laser is added for enhanced trapping. Furthermore, other effects
such as thermophoresis and surface tension gradient driven (Marangoni) flows
should be investigated in future to further refine the model.

The comparison of maximum feasible laser powers obtained from simulations and
experiments, respectively, suggested that to disturb the MDLW process
critically, the onset of evaporation is already sufficient, before permanent
boiling occurs. The impact of clusters of particles which potentially promote
heat absorption could be investigated using molecular dynamics simulations
which resolve individual particles and solving for the electric fields at the
same time.

Our simulations suggest that the deposition of particles improves with
increasing trapping strength. This could also mitigate losses due to increased
scan speed, especially for large particles.

AFM measurements of silver structures produced via MDLW delivered the required
input parameter for the model equations, as well as the surface roughness (not
shown here). The increase of the latter towards slower scan speeds indicates
that the growth of larger particles then is promoted. Hence a better
understanding of growth mechanisms and time-scales is desirable.

##  Methods

### Model parameters

For the investigation of convection, we set the scan speed
$v_{x}=$10\text{\,}\mathrm{\SIUnitSymbolMicro m}\text{\,}{\mathrm{s}}^{-1}$$.
Unless indicated otherwise, we assume an effective numerical aperture of
NA=1.2 which accounts for the lowered refractive index of the photo resist
compared to immersion oil.

For the refractive index of silver particles, we use the values published by
Johnson and Christy34 for the corresponding wavelength. Although originally
measured for thin films, this is proven practice35. We treat particle sizes
ranging from $d=30\dots 60\,$\mathrm{n}\mathrm{m}$$ in our simulations, this
range is deduced from inspection of SEM-images. Particles are generally
assumed to be spherical.

The remaining parameters appearing in eqs. (3), (6) and (5) are set to the
values given in table 3.

Table 3: Model parameters used in eqs. (1), (5), (6) | Symbol | Value
---|---|---
kinematic viscosity | $\eta$ | $0.001\,$\mathrm{Pa}\text{\,}\mathrm{s}$$
fluid density | $\rho_{0}$ | $1000\,$\mathrm{k}\mathrm{g}\mathrm{/}\mathrm{m}^{3}$$
fluid heat capacity | $c_{p}$ | $4184\,$\mathrm{J}\mathrm{/}\mathrm{(}\mathrm{k}\mathrm{g}\mathrm{K}\mathrm{)}$$
fluid heat expansion coefficient | $\beta$ | $0.002\,${\mathrm{K}}^{-1}$$
fluid heat conductivity | $k$ | $0.6\,$\mathrm{W}\mathrm{/}\mathrm{(}\mathrm{K}\,\mathrm{m}\mathrm{)}$$
ambient temperature | $T_{0}$ | $293\,$\mathrm{K}$$
refractive index | $n_{\text{liq}}$ | 1.33

### Details of simulation model

The model equations (3), (5) and (6) are solved within a cubic domain $\Omega$
of $8\,$\mathrm{\SIUnitSymbolMicro m}$$ edge length, centered around the beam
axis. This domain is much larger than the focus itself which has proven
necessary so that the system is not perturbed by the influence of the boundary
$\Gamma\equiv\partial\Omega$. The simulation domain axially centered around
the laser beam and the focus is $0.5\,$\mathrm{\SIUnitSymbolMicro m}$$ above
the domain’s bottom face. The latter represents a substrate which is being
written on.

We apply homogeneous Dirichlet boundary conditions $c\big{|}_{\Gamma}=0$ to
particle diffusion (3) on all walls, which is also known as absorbing
boundaries. This is due to the fact that particles which meet the substrate
stick there and do not go back into suspension. Also, particles that depart
far enough from the focal region are lost and either re-dissolve or appear as
debris elsewhere on the sample.

For the fluid dynamics (5), we consider a reference frame that moves with the
constant scan speed $\mathbf{v}$ of the laser. Consequently, the initial fluid
velocity is set to that of the laser:

$\mathbf{u}\big{|}_{t=0}=\mathbf{v}$ (9)

Correspondingly, we assume the substrate, that is the bottom face $\Gamma^{-}$
at $z=0$, to correspond to a no-slip boundary that moves with $-\mathbf{v}$:

$\displaystyle\mathbf{u}\big{|}_{\Gamma^{-}}=-\mathbf{v},\quad\dfrac{\partial
P}{\partial\mathbf{n}}\Big{|}_{\Gamma^{-}}=0.$ (10)

At the same time, side and top walls $\Gamma^{\Box}$, $\Gamma^{+}$,
respectively, shall be as “transparent” as possible for the fluid, which we
achieve by applying homogeneous Neumann boundary conditions both for pressure
and velocity:

$(\nabla\mathbf{u})\cdot\mathbf{n}\big{|}_{\Gamma^{\Box},\Gamma^{+}}=0,\quad\dfrac{\partial
P}{\partial\mathbf{n}}\Big{|}_{\Gamma^{\Box}}=0.$ (11)

Thereby we do not constrain the size of convection eddies. The reference
pressure is set at the top face $\Gamma^{+}$ at
$z=8\,$\mathrm{\SIUnitSymbolMicro m}$$ via a homogeneous Dirichlet condition:

$\displaystyle P\big{|}_{\Gamma^{+}}=0.$ (12)

We assume the substrate to function as heat sink and, in order to avoid an
unphysical accumulation of heat within the system, fix the boundaries

$T\big{|}_{\Gamma}=T_{0}$ (13)

to the ambient temperature for the heat equation (6). Due to the sufficient
size of our simulation domain, this hardly affects temperatures at the laser
focus.

The set of coupled equations (3), (5), (6) is solved numerically using the
multi-purpose finite volume solver Corheos36, 37 which applies fully implicit
Euler time-stepping.

### Laser beam

Laser beams are modeled as paraxial Gaussian beam38. For the laser’s lateral
beam waist $w_{0}$ and Rayleigh length $z_{\text{R}}$ at a given numerical
aperture $\operatorname{\mathrm{N\kern-0.90005ptA}}$, we assume

$\displaystyle
w_{0}=\frac{1.22\lambda}{2\operatorname{\mathrm{N\kern-0.90005ptA}}\sqrt{(2\ln
2)}},$ (14) $\displaystyle
z_{\text{R}}=\frac{\lambda}{2\Big{(}n_{\text{liq}}^{2}-\sqrt{n_{\text{liq}}^{2}-\operatorname{\mathrm{N\kern-0.90005ptA}}^{2}}\Big{)}},$
(15)

respectively. Albeit the laser emits short pulses, we use the forces of the
averaged laser power since pulsed lasers show at least equal trapping
efficiencies as the equivalent CW laser39, 40.

### Overheating threshold

In order to obtain the threshold laser power $P_{\text{thresh}}$ from the
discretely spaced simulation results $T(P_{0},d,\mathbf{v})$, we interpolate
these results with respect to $P_{0}$ and $v$ by means of a two dimensional
cubic spline. For a given particle diameter, this immediately yields
$P_{\text{thresh}}$ as a function of scan speed.

The threshold laser power for a mixture $s$ of particle species is then
obtained by exploiting the linearity of eqs. (3), (6) in the particle source
term $S_{\\!c}$ as established above. A posteriori, we superpose simulations
of particle diameters $d=30,45,60\,$\mathrm{n}\mathrm{m}$$, respectively, such
that the total volumetric silver production remains fixed at the value
obtained from experimentally measured line cross-sections. In order to
maintain consistency with the latter, the particle production rates must be
corrected for the fraction of particles $q_{\text{depo}}(P_{0},d,\mathbf{v})$
that effectively reach the substrate. Then, for the gross particle production
$S_{\\!d}$ of each diameter species $d$, we write

$S_{\\!d}(P_{0},\mathbf{v})\equiv\frac{S_{\\!c}\,s_{d}}{q(d,P_{0},\mathbf{v})}$
(16)

where $s_{d}$ represents the net fraction of deposited particles with diameter
$d$ within the system and the relation

$\sum_{d}s_{d}=1$ (17)

holds. Such a superposition yields the temperature $T(s,P_{0},\mathbf{v})$ for
all laser powers $P_{0}$ and scan speeds $v$ considered in the parameter
space. We obtain the threshold powers by means of a greedy search for $P_{0}$
s.t. $T(s,P_{0},\mathbf{v})=$100\text{\,}\mathrm{\SIUnitSymbolCelsius}$$.

For determination of the dependence of temperature on scan speed and laser
power, simulations were run with $P_{0}=4\dots 20\,$\mathrm{mW}$$, while the
scan speed was sampled non-uniformly at
$v_{x}=1,5,10,15,30,50,75,100\,$\mathrm{\SIUnitSymbolMicro
m}\text{\,}{\mathrm{s}}^{-1}$$ in order to capture all features at lower
velocities.

### Deposition efficiency

In order to measure the fraction of particles that adsorb to the line being
written, we integrate the vertical flux of particles

$j_{z}=(\mu\mathbf{F}_{\text{\\!opt}}\,c-D\nabla c)_{z}$ (18)

on the substrate (i.e. on the bottom face of the domain) over a square of
$2\,$\mathrm{\SIUnitSymbolMicro m}$$ edge length, centered around the beam
axis. Normalizing this result to the total particle production rate $S_{\\!c}$
yields

$q_{\text{depo}}=\frac{1}{S_{\\!c}}\int_{\Gamma^{-}_{\\!j}}j_{z}\mathrm{d}A,$
(19)

which, since $j\propto S_{\\!c}$, is independent of the model input parameter
$S_{\\!c}$.

### Structure fabrication

Silver lines are fabricated using a commercially available direct laser
writing system (Nanoscribe). This system uses a tightly focussed laser beam
(80 MHz, 780 nm) to locally initiate photo-reactions. In our experiment, a
liquid photosensitive material consisting of 0.4 M silver perchlorate, 0.05 M
trisodium citrate, 20 v% ammonia water and water, is sandwiched between two BK
7 glass substrates using $50\text{\,}\mathrm{\SIUnitSymbolMicro m}$ tape
spacers. Structures are fabricated on the bottom facet of the top glass
substrates. Thus, radiation pressure points towards the substrates.

### AFM measurements and evaluation

AFM height maps of the fabricated silver lines are taken using a JPK
Nanowizard 3 AFM. Eleven parallel lines are fabricated with a separation of
$2\text{\,}\mathrm{\SIUnitSymbolMicro m}$. A Matlab script is used to
automatically derive the averaged line widths, line heights and RMS
roughnesses from these height maps.

## References

  * Waller et al. 2021 Waller, E. H.; Karst, J.; von Freymann, G. _Light: Advanced Manufacturing_ 2021, _2_ , 1
  * Thiele et al. 2017 Thiele, S.; Arzenbacher, K.; Gissibl, T.; Giessen, H.; Herkommer, A. M. _Science Advances_ 2017, _3_ , e1602655
  * Thiele et al. 2019 Thiele, S.; Pruss, C.; Herkommer, A. M.; Giessen, H. _Optics Express_ 2019, _27_ , 35621
  * Toulouse et al. 2021 Toulouse, A.; Drozella, J.; Thiele, S.; Giessen, H.; Herkommer, A. _Light: Advanced Manufacturing_ 2021, _2_ , 1–11
  * Schumann et al. 2014 Schumann, M.; Bückmann, T.; Gruhler, N.; Wegener, M.; Pernice, W. _Light: Science & Applications_ 2014, _3_ , e175–e175
  * Deubel et al. 2004 Deubel, M.; von Freymann, G.; Wegener, M.; Pereira, S.; Busch, K.; Soukoulis, C. M. _Nature Materials_ 2004, _3_ , 444–447
  * Rill et al. 2008 Rill, M. S.; Plet, C.; Thiel, M.; Staude, I.; von Freymann, G.; Linden, S.; Wegener, M. _Nature Materials_ 2008, _7_ , 543–546
  * Gansel et al. 2009 Gansel, J. K.; Thiel, M.; Rill, M. S.; Decker, M.; Bade, K.; Saile, V.; von Freymann, G.; Linden, S.; Wegener, M. _Science_ 2009, _325_ , 1513–1515
  * Gansel et al. 2012 Gansel, J. K.; Latzel, M.; Frölich, A.; Kaschke, J.; Thiel, M.; Wegener, M. _Applied Physics Letters_ 2012, _100_ , 101109
  * Tanaka et al. 2006 Tanaka, T.; Ishikawa, A.; Kawata, S. _Applied Physics Letters_ 2006, _88_ , 081107
  * Blasco et al. 2016 Blasco, E.; Müller, J.; Müller, P.; Trouillet, V.; Schön, M.; Scherer, T.; Barner-Kowollik, C.; Wegener, M. _Advanced Materials_ 2016, _28_ , 3592–3595
  * Greer and Street 2007 Greer, J. R.; Street, R. A. _Acta Materialia_ 2007, _55_ , 6345–6349
  * Vyatskikh et al. 2018 Vyatskikh, A.; Delalande, S.; Kudo, A.; Zhang, X.; Portela, C. M.; Greer, J. R. _Nature Communications_ 2018, _9_
  * Waller et al. 2019 Waller, E. H.; Dix, S.; Gutsche, J.; Widera, A.; von Freymann, G. _Micromachines_ 2019, _10_ , 827
  * Waller and von Freymann 2018 Waller, E. H.; von Freymann, G. _Nanophotonics_ 2018, _7_ , 1259–1277
  * Lee et al. 2017 Lee, M. R.; Lee, H. K.; Yang, Y.; Koh, C. S. L.; Lay, C. L.; Lee, Y. H.; Phang, I. Y.; Ling, X. Y. _ACS Applied Materials & Interfaces_ 2017, _9_ , 39584–39593
  * He et al. 2017 He, G.-C.; Zheng, M.-L.; Dong, X.-Z.; Liu, J.; Duan, X.-M.; Zhao, Z.-S. _AIP Advances_ 2017, _7_ , 035203
  * Baffou et al. 2009 Baffou, G.; Quidant, R.; Girard, C. _Applied Physics Letters_ 2009, _94_ , 153109
  * Baffou et al. 2010 Baffou, G.; Quidant, R.; de Abajo, F. J. G. _ACS Nano_ 2010, _4_ , 709–716
  * Baffou and Rigneault 2011 Baffou, G.; Rigneault, H. _Physical Review B_ 2011, _84_
  * Baffou et al. 2013 Baffou, G.; Berto, P.; Ureña, E. B.; Quidant, R.; Monneret, S.; Polleux, J.; Rigneault, H. _ACS Nano_ 2013, _7_ , 6478–6488
  * Donner et al. 2011 Donner, J. S.; Baffou, G.; McCloskey, D.; Quidant, R. _ACS Nano_ 2011, _5_ , 5457–5462
  * Ashkin 0000 Ashkin, A. _Biophysical Journal_ 0000, _61_ , 569–582
  * Ashkin 2000 Ashkin, A. _IEEE Journal of Selected Topics in Quantum Electronics_ 2000, _6_
  * Svoboda and Block 1994 Svoboda, K.; Block, S. M. _Opt. Lett._ 1994, _19_ , 930–932
  * Nieminen et al. 2007 Nieminen, T. A.; Loke, V. L. Y.; Stilgoe, A. B.; Knöner, G.; Brańczyk, A. M.; Heckenberg, N. R.; Rubinsztein-Dunlop, H. _Journal of Optics A: Pure and Applied Optics_ 2007, _9_ , S196–S203
  * Nieminen et al. 2014 Nieminen, T. A.; du Preez-Wilkinson, N.; Stilgoe, A. B.; Loke, V. L.; Bui, A. A.; Rubinsztein-Dunlop, H. _Journal of Quantitative Spectroscopy and Radiative Transfer_ 2014, _146_ , 59–80
  * Melzer and McLeod 2018 Melzer, J. E.; McLeod, E. _ACS Nano_ 2018, _12_ , 2440–2447
  * Baffou et al. 2010 Baffou, G.; Quidant, R.; Girard, C. _Physical Review B_ 2010, _82_
  * Skylar-Scott et al. 2016 Skylar-Scott, M. A.; Gunasekaran, S.; Lewis, J. A. _Proceedings of the National Academy of Sciences_ 2016, _113_ , 6137–6142
  * Bohren and Huffman 1983 Bohren, C. F.; Huffman, D. _Absorption and scattering of light by small particles_ ; Wiley, 1983
  * Gillespie and Seitaridou 2012 Gillespie, D. T.; Seitaridou, E. _Simple Brownian Diffusion_ ; Oxford University Press, 2012
  * Tritton 1988 Tritton, D. _Physical Fluid Dynamics_ ; Oxford Science Publ; Clarendon Press, 1988
  * Johnson and Christy 1972 Johnson, P. B.; Christy, R. W. _Physical Review B_ 1972, _6_ , 4370–4379
  * Billaud et al. 2007 Billaud, P.; Huntzinger, J.-R.; Cottancin, E.; Lermé, J.; Pellarin, M.; Arnaud, L.; Broyer, M.; Fatti, N. D.; Vallée, F. _The European Physical Journal D_ 2007, _43_ , 271–274
  * Latz et al. 2010 Latz, A.; Strautins, U.; Niedziela, D. _Journal of Non-Newtonian Fluid Mechanics_ 2010, _165_ , 764–781
  * Schmidt et al. 2013 Schmidt, S.; Niedziela, D.; Steiner, K.; Zausch, J. CoRheoS: Multiphysics solver framework and simulation infrastructure for complex rheologies. Proceedings NAFEMS World Congress, Salzburg. 2013
  * Novotny and Hecht 2009 Novotny, L.; Hecht, B. _Principles of Nano-Optics_ ; Cambridge University Press, 2009
  * Agate et al. 2004 Agate, B.; Brown, C. T. A.; Sibbett, W.; Dholakia, K. _Opt. Express_ 2004, _12_ , 3011–3017
  * Shane et al. 2010 Shane, J. C.; Mazilu, M.; Lee, W. M.; Dholakia, K. _Optics Express_ 2010, _18_ , 7554