# Quantized sparse weight decomposition for neural network compression

Andrey Kuzmin  Mart van Baalen  Markus Nagel  Arash Behboodi

###### Abstract

In this paper, we introduce a novel method of neural network weight
compression. In our method, we store weight tensors as sparse, quantized
matrix factors, whose product is computed on the fly during inference to
generate the target model’s weights. We use projected gradient descent methods
to find quantized and sparse factorization of the weight tensors. We show that
this approach can be seen as a unification of weight SVD, vector quantization
and sparse PCA. Combined with end-to-end fine-tuning our method exceeds or is
on par with previous state-of-the-art methods in terms of trade-off between
accuracy and model size. Our method is applicable to both moderate compression
regime, unlike vector quantization, and extreme compression regime.

Machine Learning, ICML

##  1 Introduction

Deep neural networks have achieved state-of-the-art results in a wide variety
of tasks. However, deployment remains challenging due to their large compute
and memory requirements. Neural networks deployed on edge devices such as
mobile or IoT devices are subject to stringent compute and memory constraints,
while networks deployed in the cloud do not suffer such constraints but might
suffer excessive latency or power consumption.

To reduce neural network memory and compute footprint, several approaches have
been introduced in literature. Methods related to our approach, as well as
their benefits and downsides, are briefly introduced in this section and
described in more detail in the related works section. _Tensor factorization_
approaches (Denil et al., 2013) replace a layer in a neural network with two
layers, whose weights are low-rank factors of the original layer’s weight
tensor. This reduces the number of parameters and multiply-add operations
(MACs), but since the factorizations are by design restricted to those that
can be realized as individual layers, potential for compression is limited. By
_pruning_ a neural network (Louizos et al., 2017; He et al., 2017), individual
weights are removed. Pruning has shown to yield moderate compression-accuracy
trade-offs. Due to the overhead required to keep track of which elements are
pruned, real yield of (unstructured) pruning is lower than the pruning ratio.
An exception to this is _structured pruning_ , in which entire neurons are
removed from a network, and weight tensors can be adjusted accordingly.
However, achieving good compression ratios at reasonable accuracy using
structured pruning has proven difficult. _Scalar quantization_ (Jacob et al.,
2018; Nagel et al., 2021) approximates neural network weights with fixed-point
values, i.e., integer values scaled by a fixed floating point scalar. Scalar
quantization has shown to yield high accuracy at reasonable compression
ratios, e.g., 8 bit quantization yields a 4x compression ratio and virtually
no accuracy degradation on many networks. However, scalar quantization does
not yield competitive compression vs accuracy trade-offs at high compression
ratios. _Vector quantization_ (Stock et al., 2019; Martinez et al., 2021)
approximates small subsets of weights, e.g., individual $3\times 3$ filters in
convolutional weight tensors, by a small set of codes. This way, the storage
can be reduced by storing the codebook and one code index for each original
vector, instead of the individual weights. While vector quantization can
achieve high compression with moderate accuracy loss, these methods usually
struggle to reach the accuracy of uncompressed models in low compression
regimes.

---
Figure 1: An illustration of application of Quantized Sparse PCA to a
convolutional weight tensor. The weight tensor is reshaped into a matrix of
shape $d\times n$ and factorized into a codebook
$\mathbf{C}\in\mathbb{R}^{d\times k}$ and a latent
$\mathbf{Z}\in\mathbb{R}^{k\times n}$. Both factors are quantized while only
$\mathbf{Z}$ is sparse. During inference the arrows are followed backwards:
the reshaped and tiled matrix is computed from the product of the factors
$\mathbf{C}$ and $\mathbf{Z}$. The result is reshaped into the original weight
tensor.

In this paper, we provide a novel view on tensor factorization. Instead of
restricting factorization to those that can be realized as two separate neural
network layers, we show that much higher compression ratios can be achieved by
shifting the order of operations. In our method, we find a factorization
${\mathbf{C}},{\mathbf{Z}}$ for an original weight tensor ${\mathbf{W}}$, such
that the matrix product ${\mathbf{C}}{\mathbf{Z}}$ closely approximates the
original weight tensor. The compression is then pushed further by obtaining
quantized factors and additionally sparse ${\mathbf{Z}}$. During inference,
the product ${\mathbf{C}}{\mathbf{Z}}$ is computed first, and its result is
reshaped back into the original weight tensor’s shape, and used for the
following computations. This approach allows the use of an arbitrary
factorization of the original weight tensor.

We show that this approach outperforms or is on par with vector quantization
in high compression regimes, yet extends to scalar quantization levels of
compression-accuracy trade-offs for lower compression ratios.

Our contributions in this paper are as follows:

  * •

We show that the problem of tensor factorization and vector quantization can
be formulated in a unified way as quantized sparse principle component
analysis (PCA) problem.

  * •

We propose an iterative projected gradient descent method to solve the
quantized sparse PCA problem.

  * •

Our experimental results demonstrate the benefits of this approach. By
simultaneously solving tensor factorization and vector quantization problem,
we can achieve better accuracy than vector quantization in low compression
regimes, and higher compression ratios than scalar quantization approaches at
moderate loss in accuracy.

##  2 Related work

SVD-based methods and tensor decompositions SVD decomposition was first used
to demonstrate redundacy in weight parameters in neural networks in (Denil et
al., 2013). Later several methods for reducing the inference time based on SVD
decomposition were suggested (Denton et al., 2014; Jaderberg et al., 2014). A
similar technique was proposed for gradients compression in data-parallel
distributed optimization by (Vogels et al., 2019). The main difference between
these methods is the way 4D weights of a convolutional layer is transformed
into a matrix which leads to different shapes of the convolutional layers in
the resulting decomposition. Following a similar direction, several works
focus on higher-order tensor decomposition methods which lead to introducing
of three or four convolutional layers (Lebedev et al., 2014; Kim et al., 2015;
Su et al., 2018).

##### Weight pruning

A straightforward approach to reducing neural network model size is removing a
percentage of weights. A spectrum of weight pruning approaches of different
granularity has been introduced in the literature. Structured pruning
approaches such as (He et al., 2017) kill entire channels of the weights,
while unstructured pruning approaches (Louizos et al., 2017; Zhu & Gupta,
2017; Neklyudov et al., 2017; Dai et al., 2018) focus on individual values. A
recent survey on unstructured pruning is provided in (Gale et al., 2019).

##### Scalar quantization and mixed precision training

By quantizing neural network weights to lower bitwidths, model footprint can
be reduced as well, as each individual weight requires fewer bits to be
stored. For example, quantizing 32 bit floating points weight to 8 bit fixed
point weights yields a 4x compression ratio. Most quantization approaches use
the straight-through estimator (STE) for training quantized models ((Bengio et
al., 2013; Krishnamoorthi, 2018)). One way to further improve the accuracy of
quantized models is learning the quantization scale and offset jointly with
the network parameters ((Esser et al., 2019; Bhalgat et al., 2020)). A recent
survey on practical quantization approaches can be found in (Nagel et al.,
2021).

In order to improve the accuracy of quantized models, several methods suggest
using mixed precision quantization. The work by (Uhlich et al., 2019)
introduced an approach on learning integer bit-width for each layer using STE.
Using non-uniform bit-width allows the quantization method to use lower bit-
width for more compressible layers of the network. Several works (van Baalen
et al., 2020; Dong et al., 2019; Wang et al., 2019) improve upon the approach
by (Uhlich et al., 2019) by using different methods for optimization over the
bit-widths.

Vector quantization. Several works use vector quantization approach for
compression of weights of convolutional and fully connected layers ((Gong et
al., 2014; Martinez et al., 2021; Fan et al., 2020; Stock et al., 2019; Wu et
al., 2016)). The convolutional weight tensors are reshaped into matrices, then
K-means methods is applied directly on the rows or columns. Besides weight
compression, the work by (Wu et al., 2016) suggests using vector quantization
for reducing the inference time by reusing parts of the computation.

Recently several works suggested improvements on the basic vector quantization
approach. Data-aware vector quantization which improves the clustering method
by considering input activation data is demonstrated to improve the accuracy
of the compressed models by (Stock et al., 2019). Another direction is
introducing a permutation to the weight matrices which allows to find subsets
of weights which are more compressible (Martinez et al., 2021). An inverse
permutation is applied at the output of the corresponding layers to preserve
the original output of the model.

Besides weights compression problem, various improved vector quantization
methods were applied in image retrieval domain in order to accelerate scalar
product computation for image descriptors ((Chen et al., 2010; Ge et al.,
2013; Norouzi & Fleet, 2013)). In particular, additive quantization method
which we will show is related to quantized sparse PCA was introduced (Babenko
& Lempitsky, 2014). Surveys on vector quantization methods are provided in
(Matsui et al., 2018; Gersho & Gray, 2012).

Sparse PCA. Introduced in (Zou et al., 2006), sparse PCA can be solved by a
plethora of algorithms. The method proposed in this paper can be considered as
an instance of thresholding algorithms (Ma, 2013). Although soft-thresholding
methods are prevalent in the literature, we adopt an explicit projection step
using hard thresholding to have direct control over the compression ration.
Note that sparse PCA can be extended to include additional structures with
sparsity (Jenatton et al., 2010; Lee et al., 2006).

##  3 Method

In this section, we describe the main algorithm, which can be considered as
sparse quantized principle component analysis (PCA).

###  3.1 Quantized Sparse PCA

Consider a weight tensor of a convolutional layer
${\mathbf{W}}\in\mathbb{R}^{f_{out}\times f_{in}\times h\times w}$, where
$f_{in}$, $f_{out}$ are the number of input and output feature maps, and $h$,
$w$ are the spatial dimensions of the filter. We reshape it into a matrix
$\widetilde{{\mathbf{W}}}\in\mathbb{R}^{d\times n}$, where $d$ is the tile
size and $n$ is the number of tiles. For the reshape, we consider the
dimensions in the order $f_{out},f_{in},h,w$ in our experiments. The goal is
to factorize $\widetilde{\mathbf{W}}$ into the product of two matrices as
follows:

$\widetilde{\mathbf{W}}=\mathbf{C}\mathbf{Z},$ (1)

where $\mathbf{C}\in\mathbb{R}^{d\times k}$ is the codebook, and
$\mathbf{Z}\in\mathbb{R}^{k\times n}$ is the set of linear coefficients, or a
latent variable ($k<d<n$). Following the standard PCA method, we factorize the
zero mean version of $\widetilde{\mathbf{W}}$. With this decomposition, every
column of $\widetilde{\mathbf{W}}$, denoted by
${\widetilde{\mathbf{W}}}_{:,i}$, is a linear combination of $k$ codebook
vectors (columns of $\mathbf{C})$:

$\widetilde{\mathbf{W}}_{:,i}=\sum_{j=1}^{k}{Z}_{ji}\mathbf{C_{:,j}},$ (2)

where ${\mathbf{C}}_{:,j}$ is the $j$-th column of $\mathbf{C}$. This
decomposition problem is an instance of sparse PCA methods ((Zou et al.,
2006)). For network compression, we are additionally interested in quantized
matrices $\mathbf{C}$ and $\mathbf{Z}$. The quantization operation for an
arbitrary $\mathbf{C}$ and $\mathbf{Z}$ is defined as follows:

$\displaystyle\mathbf{C}_{q}$
$\displaystyle=Q_{c}(\mathbf{C};{\bm{s}}_{c},b_{c}),$ (3)
$\displaystyle\mathbf{Z}_{q}$
$\displaystyle=Q_{z}(\mathbf{Z};{\bm{s}}_{z},b_{z}),$ (4)

where $b_{c}$,$b_{z}$ are the quantization bit-widths, and $\mathbf{s}_{c}$
and $\mathbf{s}_{z}$ are the quantization scale vectors for $\mathbf{C}$ and
$\mathbf{Z}$, respectively. We consider per-channel quantization, i.e., the
quantization scale values are shared for each column of $\mathbf{C}$ and each
row or column of $\mathbf{Z}$ (see section 3.2 for details on which is used
when):

${C}_{q,ij}=\mbox{clamp}\left(\left\lfloor\frac{{C}_{ij}}{s_{i}}\right\rceil,0,2^{b_{c}}-1\right)s_{i},$
(5)
${Z}_{q,ij}=\mbox{clamp}\left(\left\lfloor\frac{{Z}_{ij}}{s_{j}}\right\rceil,0,2^{b_{z}}-1\right)s_{j},$
(6)

where $\lfloor\cdot\rceil$ is rounding to nearest integer, and
$\mbox{clamp}(\cdot)$ is defined as:

$\mbox{clamp}(x,a,b)=\begin{cases}a&x<a\\\ x&a\leq x\leq b\\\
b&x>b\end{cases}.$ (7)

We refer to the problem of finding quantized factors $\mathbf{C}$ and
${\mathbf{Z}}$ with sparse ${\mathbf{Z}}$ as quantized sparse PCA problem.
Once we obtain the factors ${\mathbf{C}}$ and ${\mathbf{Z}}$, we get the
matrix $\widetilde{{\bm{W}}}=({\mathbf{C}}{\mathbf{Z}})$, which can be
reshaped back to a convolutional layer. The reshaped convolutional layer for
factors ${\mathbf{C}},{\mathbf{Z}}$ is denoted by
$[{\mathbf{C}}{\mathbf{Z}}]$. It is well known in network compression
literature that it is better to find the best factorization on the data
manifold ((Zhang et al., 2015; He et al., 2017; Stock et al., 2019)).
Therefore, we solve the following optimization problem:

$\displaystyle{\mathbf{C}}^{*},{\mathbf{Z}}^{*}$
$\displaystyle=\underset{{\mathbf{C}}_{q},{\mathbf{Z}}_{q}}{\text{argmin}}\,\,\mathbb{E}_{({\bm{X}},{\bm{Y}})\sim{\mathcal{D}}}\left(\left\lVert{\mathbf{Y}}-[{\mathbf{C}}_{q}{\mathbf{Z}}_{q}]*{\bm{X}}\right\rVert^{2}_{F}\right)$
(8) s.t.
$\displaystyle\quad{\mathbf{C}}_{q}=Q_{c}({\mathbf{C}};{\bm{s}}_{c},b_{C})$
$\displaystyle\quad{\mathbf{Z}}_{q}=Q_{z}({\mathbf{Z}};{{\bm{s}}}_{z},b_{Z})$
$\displaystyle\quad\left\lVert{\mathbf{Z}}_{q}\right\rVert_{0}\leq
S,{\mathbf{Z}}\in\mathbb{R}^{k\times n},{\mathbf{C}}\in\mathbb{R}^{d\times
k},$

where the parameter $S$ controls the sparsity ratio of ${\mathbf{Z}}$,
$\mathbf{X}$ and ${\mathbf{Y}}$ are the stored input output of the target
layer in the original model, ${\mathcal{D}}$ is the data distribution, and $*$
is the convolution operation. $L_{0}$ norm is used for the constraint on the
number of nonzero elements in the matrix $\mathbf{Z_{q}}$. We approximate the
expected value of above optimization problem using a subset of the training
data:

$\displaystyle\mathbb{E}_{(\mathbf{X},\mathbf{Y})\sim{\mathcal{D}}}\left(\left\lVert\mathbf{Y}-[\mathbf{C_{q}}\mathbf{Z_{q}}]*\mathbf{X}\right\rVert^{2}_{F}\right)\approx$
(9)
$\displaystyle\frac{1}{m}\sum_{i=1}^{m}\left\lVert\mathbf{Y}_{i}-[\mathbf{C}_{q}\mathbf{Z_{q}}]*\mathbf{X_{i}}\right\rVert^{2}_{F},$

where $m$ is the number of samples used for optimization. Following the
compression method introduced in (Zhang et al., 2015), we use output of the
previous compressed layer as ${\bm{X}}$ instead of the stored input to the
layer in the original model. This approach aims at compensating for the error
accumulation in deep networks.

####  3.1.1 Projected Gradient Descent Optimization Algorithm

The above optimization problem can be solved using an iterative projected
gradient descent method. The projection step is a hard thresholding operation
which project onto the space of sparse matrices. The optimization algorithm is
given below.

Algorithm 1 Projected Gradient Descent Optimization

0: $\widetilde{\mathbf{W}}$

1: SVD of $\widetilde{\mathbf{W}}=\mathbf{U}\mathbf{\Sigma}\mathbf{V}^{\top},$

2: $\mathbf{C}\leftarrow\mathbf{U}_{k}$

3: $\mathbf{Z}\leftarrow\mathbf{U}_{k}^{\top}\widetilde{\mathbf{W}}$

4: while Stopping criteria not met do

5: Gradient descent step: $\mathbf{C},\mathbf{Z}\leftarrow$ gradient descent
update on equation (11).

6: $\mathbf{M}\leftarrow$ binary mask for largest $S$ values of
$Q_{z}(\mathbf{Z};{\mathbf{s}}_{z},b_{Z})$.

7: Hard thresholding step: $\mathbf{Z}\leftarrow\mathbf{Z}\odot\mathbf{M}$.

8: end while

9: return:
$Q_{c}(\mathbf{C};\mathbf{s}_{c},b_{C}),Q_{z}(\mathbf{Z};{\mathbf{s}}_{z},b_{Z})$.

  1. 1.

Initialization. The codebook ${\mathbf{C}}$ is initialized with first $k$ left
singular vectors of $\widetilde{{\mathbf{W}}}$. Given the SVD decomposition of
$\widetilde{{\mathbf{W}}}$ as follows:

$\widetilde{{\mathbf{W}}}=\mathbf{U}\mathbf{\Sigma}\mathbf{V}^{\top},$ (10)

we choose ${\mathbf{C}}^{(0)}=\mathbf{U}_{k}$, where ${\mathbf{U}}_{k}$
denotes the top $k$ left singular vectors of $\widetilde{{\mathbf{W}}}$. The
latent matrix ${\mathbf{Z}}$ is initialized as the projection of
$\widetilde{{\mathbf{W}}}$ onto the set of first $k$ singular vectors,
${\mathbf{Z}}^{(0)}={\mathbf{U}}_{k}^{\top}\widetilde{{\mathbf{W}}}$.

  2. 2.

Gradient Descent. At iteration $t$, the matrices
${\mathbf{C}}^{(t-1)},{\mathbf{Z}}^{(t-1)}$ are updated by gradient descent on
the following objective w.r.t. ${\mathbf{C}}$ and ${\mathbf{Z}}$:

$\frac{1}{m}\sum_{i=1}^{m}\left\lVert{\mathbf{Y}}_{i}-[Q_{c}({\mathbf{C}};{\bm{s}}_{c},b_{C})Q_{z}({\mathbf{Z}};{{\bm{s}}}_{z},b_{Z})]*{\mathbf{X}}_{i}]\right\rVert^{2}_{F},$
(11)

We use the straight through estimator (STE) ((Bengio et al., 2013)) to compute
the gradients of the quantization operation, i.e. we use the following
gradient estimate for the rounding operation:

$\frac{\partial\lceil p\rfloor}{\partial p}=1.$

The outcome of this step is denoted by
${\mathbf{C}}^{(t)}_{\text{gd}},{\mathbf{Z}}^{(t)}_{\text{gd}}$.

  3. 3.

Hard thresholding. After the gradient descent, we project
${\mathbf{Z}}^{(t)}_{\text{gd}}$ onto the space of sparse matrices. The
projection is done using entrywise product of ${\mathbf{Z}}^{(t)}_{\text{gd}}$
with a binary mask matrix ${\mathbf{M}}^{(t)}$, namely
${\mathbf{Z}}^{(t)}_{\text{gd}}\odot{\mathbf{M}}^{(t)}$. The mask matrix
${\bm{M}}^{(t)}$ is obtained finding the support of $S$ largest entries of
$Q_{z}({\mathbf{Z}}^{(t)}_{\text{gd}};{{\bm{s}}}_{z},b_{Z})$. This is the hard
thresholding step. For the next iteration we set:

${\mathbf{Z}}^{(t)}={\mathbf{Z}}^{(t)}_{\text{gd}}\odot{\bm{M}}^{(t)},{\mathbf{C}}^{(t)}={\mathbf{C}}^{(t)}_{\text{gd}}.$

  4. 4.

Repeat the previous three steps using ${\mathbf{Z}}^{(t)},{\mathbf{C}}^{(t)}$
until termination criteria is met (MSE error or a fixed iterations number).
The final matrices are given be
$Q_{c}({\mathbf{C}}^{(t)};{\bm{s}}_{c},b_{C}),Q_{z}({\mathbf{Z}}^{(t)};{{\bm{s}}}_{z},b_{Z})$.

It is in general difficult to provide convergence analysis for non-convex
optimization problems such as (8). Iterative thresholding methods are,
however, widely used to solve sparse optimization problems (Jacques et al.,
2013; Foucart & Rauhut, 2013), and in some cases, they have theoretical
guarantees. In Appendix A, we discuss some of the results. Note that the
theory does not apply directly to our setup. Nonetheless, it provides
intuition behind effectiveness of the proposed approach.

---
Figure 2: Results for Resnet18 trained on ImageNet.

####  3.1.2 Relation to other approaches

In this section, we discuss the connection with other existing methods.

  1. (A)

Vector quantization. If we replace the quantizers for ${\mathbf{C}}$ and
${\mathbf{Z}}$ by identity functions, i.e. $Q_{c}=\mbox{id}(\cdot)$,
$Q_{z}=\mbox{id}(\cdot)$, then matrix ${\mathbf{C}}$ is in full precision,
while ${\mathbf{Z}}$ encodes the indices such that

${Z}_{ij}=\delta_{im_{i}},$ (12)

where $m_{i}$ is the centroid index corresponding to tile $i$.

  2. (B)

Additive vector quantization. If $Q_{c}=\mbox{id}(\cdot)$, $b_{z}=1$ (binary
quantization), then

$\widetilde{{\mathbf{W}}}_{:,i}=\sum_{j=1}^{k}\rho_{ji}{\mathbf{C}}_{:,j},$
(13)

where $\rho_{ji}\in\\{0,1\\}$ are binary coefficients. Note that in the
original work by (Babenko & Lempitsky, 2014), an additional constraint of
$\sum_{j}\rho_{ji}=n_{t}$ is enforced, i.e. each ${\mathbf{W}}_{i,:}$ is a sum
of a fixed number of terms.

  3. (C)

Principle component analysis. If $Q_{c}=\mbox{id}(\cdot)$, and
$Q_{z}=\mbox{id}(\cdot)$, then

$\widetilde{{\mathbf{W}}}_{:,i}=\sum_{j=1}^{k}{Z}_{ji}{\mathbf{C}}_{:,j}.$
(14)

This case corresponds to SVD factorization of zero mean version of weight
matrix $\widetilde{{\mathbf{W}}}$. For specific values of $n$ and $d$, the
method is mathematically equivalent to the SVD decomposition methods for
neural network compression ((Denton et al., 2014; Jaderberg et al., 2014)).
For example, if $n=f_{out}h$, and $d=f_{in}w$, then the method is equivalent
to the spatial SVD decomposition by (Jaderberg et al., 2014). In contrast to
the above-mentioned approaches, we do not decompose the convolutional
operation into two convolutional layers, which gives extra flexibility in the
choice of PCA dimensionality $d$.

###  3.2 Compression ratio

Without sparsity, the compression ratio is computed as follows. Let
$L_{o}({\mathbf{W}})=f_{out}\times f_{in}\times h\times w\times 32$ denote the
number of bits required to represent the original 32 bit floating point tensor
${\mathbf{W}}$, and $L_{c}({\mathbf{W}})=d\times k\times b_{c}+k\times n\times
b_{z}$ the size of the quantized tensor factors ${\mathbf{C}}$ and
${\mathbf{Z}}$ of ${\mathbf{W}}$. The compression ratio $C_{r}$ is then
defined as $C_{r}=\frac{L_{o}({\mathbf{W}})}{L_{c}({\mathbf{W}})}$.

Note that, since $k<d<n$, the latent matrix ${\mathbf{Z}}$ contributes more to
the resulting model size than the codebook matrix ${\mathbf{C}}$. For example,
let us assume $f_{out}=f_{in}=256$, $h=w=3$, $d=256$, and $k=128$. In this
scenario, the original weight tensor ${\mathbf{W}}$ has $256\times 256\times
3\times 3=589,824$ elements, the codebook matrix ${\mathbf{C}}$ has $256\times
128=32,768$ elements and the latent matrix ${\mathbf{Z}}$ has $128\times
2,034=260,356$ elements, almost 8 times as many as ${\mathbf{C}}$. For this
reason, efforts to compress ${\mathbf{Z}}$, e.g., by using a lower bitwidth
$b_{c}$ or pruning elements, will likely yield a better accuracy vs
compression trade-off than focusing similar efforts on ${\mathbf{C}}$.

Additional compression is achieved by encoding the sparse matrix
${\mathbf{Z}}$ as follows. We assume the sparsity mask ${\mathbf{M}}$ is
stored as a binary matrix. Then, we only store the nonzero values in
${\mathbf{Z}}$. At runtime, ${\mathbf{Z}}$ can be decoded by only reading
values in ${\mathbf{Z}}$ for which the corresponding binary mask value is
equal to $1$. The total number of bits required to store ${\mathbf{M}}$ and
the nonzero values in ${\mathbf{Z}}$ is $k\times n+(1-r)\times k\times n\times
b_{z}$, where $r=1-S/kn$ is the sparsity ratio of ${\mathbf{Z}}$ with
$\left\lVert{\mathbf{Z}}\right\rVert_{0}=S$. Note that this method only
reduces memory footprint if $r>1/b_{z}$.

The size of the compressed tensor must be adjusted for quantization parameters
introduced by per-channel quantization. We assume that each quantization scale
parameter is encoded using FP16. To account for this we add a quantization
adjustment term $L_{q}({\mathbf{W}})$ to $L_{c}({\mathbf{W}})$, where
$L_{q}=2k\times 16$ is used for per-row quantization of ${\mathbf{Z}}$.

Finally, taking into account the adjustments for sparsity and quantization
parameters, the size of a compressed tensor is computed as
$L_{c}({\mathbf{W}})=d\times k\times b_{c}+k\times n+(1-r)\times k\times
n\times b_{z}+L_{q}({\mathbf{W}})$.

###  3.3 Computational overhead

The complexity of the forward pass for a convolutional layer is
$f_{in}f_{out}hwp$. The overhead for the filter reconstruction is the product
of two matrices using $f_{in}f_{out}hwk$ multiply-add operations (MACs). So
the relative overhead in the number of operations is $k/p$. Practically, for
Resnet18, a value of $k$ equals for example 128, while the spatial size of the
feature map for an ImageNet goes down from $56\times 56$ to $7\times 7$ in the
later layers. So, the relative overhead for a single forward pass is varying
between $4\%$ and $261\%$ extra operations for different layers. While the
total relative overhead for the full model is $85\%$ extra MACs. However, with
an increasing batch size or when doing multiple inferences this becomes
neglectable. The comparison above is based on MACs, however in our approach,
most computations contain low bit tensors. If we do the same comparison in bit
operations (BOPs) the overhead is significantly reduced.

---
Figure 3: Results for MobileNetV2 trained on ImageNet.

##  4 Experiments

In this section we describe two sets of experiments: one set to explore the
effect of compression hyperparameters on compression-accuracy trade-offs, and
one set of experiments to compare against various baseline methods.

###  4.1 Experimental setup

##### Initial factorization

We start with a pre-trained model and perform the factorization as described
in Section 3.1.1. We initialize the per-channel quantization scale using min-
max quantization (Krishnamoorthi, 2018).

##### Per layer data-aware optimization

To perform the data-aware optimization, we sample a single batch of 64 input
and output activations for a layer. We split the data into a training set and
a validation set (we keep one eighth of the data as the validation set). We
keep track of the error on the validation set. During this step we use the
Adam optimizer ((Kingma & Ba, 2014)) with learning rate $10^{-4}$ and weight
decay $10^{-5}$.

##### Sparsification

Sparsification is applied after per layer data-aware optimization in a form of
hard thresholding. We consider one-shot and iterative thresholding.
Specifically, for each layer, the quantized weights are sorted by absolute
value, then a desired percentage of values is masked out. We do not prune
weights that become zeros after the quantization step, instead, we add extra
sparsity on top of accidental sparsity that occurs after the quantization.
During the consequent fine-tuning stage, the sparsity mask is frozen.

##### Fine-tuning

We follow (Martinez et al., 2021) and fine-tune our models end-to-end using
the target training set. Model-specific finetuning procedures are described
below.

####  4.1.1 A note on compression ratio

In our experiments we leave the first convolutional weight tensor, all bias
tensors, all batch normalization parameters, and the PCA centering vectors in
FP32. We denote these uncompressible parameters as $L_{u}$. The sparsity ratio
for a network is then computed as
$C_{r}=\frac{\sum_{\mathbf{W}}L_{o}(\mathbf{W})}{L_{u}+\sum_{\mathbf{W}}L_{c}(\mathbf{W})}$

###  4.2 ImageNet Experiments

We evaluate our method on the ResNet18 and MobileNetV2 achitectures for
ImageNet classification. We apply our method on pretrained versions of the
model, where we take the pretrained weights from the PyTorch model zoo. In all
our experiments we keep the tile size $d$ constant at 256, the bitwidth of the
codebook matrix C constant at 4, and use per-channel scalar quantization of 4
bits for the fully connected classification layer weights. To achieve
different compression ratios we vary the rank $k$ with values between 64-256,
the bitwidth of the latent matrix Z between 3-6, and the extra sparsity of Z
between 0% and 40%. Further experimental details can be found in Section B.1.

####  4.2.1 Baseline methods

We compare our methods against scalar quantization by (Esser et al., 2019),
binary CNNs by (Lin et al., 2017) (ABC-Net), the vector quantization method by
(Martinez et al., 2021) (PQF), the vector quantization method by (Stock et
al., 2019) (BGD), trained ternary quantization by (Zhu et al., 2016) (TTQ).
The results for scalar quantization are produced using LSQ (Esser et al.,
2019), with 4, 3 and 2-bit per-channel weight quantization and FP32 bit
activations. For these experiments a weight decay of $10^{-4}$ and cosine
learning rate decay to $10^{-2}$ of the original learning rate was used.

####  4.2.2 Results

The Pareto dominating results of our method and comparison to the baselines
can be found in Figure 2 and Figure 3. The full set of (non Pareto dominating)
results for Resnet18 can be found in Figure 4. The same results along with the
values for rank, bitwidth and sparsity and resulting compression ratios can be
found in Table 2 in the Appendix. In this figure we see that we outperform
BGD, TTQ, ABC-Net and 3 and 2 bit scalar quantization by considerable margins.
Furthermore, we outperform or match the strongest baseline at high compression
ratios, PQF for compression ratios of 17x and higher, and are on par with
4-bit scalar quantization for compression ratios under 17x. Lastly, we are the
only method to achieve SOTA compression-accuracy trade-offs over the full
range from 5x to 40x compression.

###  4.3 Ablation studies

####  4.3.1 Hard thresholding methods

|  |  Pre-FT accuracy |  Post-FT accuracy
---|---|---|---
Hard thresholding method |  Stopping. critetion |  10% sparsity |  20% sparsity |  30% sparsity |  10% sparsity |  20% sparsity |  30% sparsity
One-shot HT | val. set | 63.2 | 58.9 | 47.3 | 69.1 | 68.7 | 68.1
One-shot HT | 100 iter. | 59.4 | 55.5 | 45.2 | 68.8 | 68.4 | 68.0
Iterative HT | 30 iter. | 64.5 | 63.0 | 58.8 | 69.1 | 68.8 | 68.0
Iterative HT | 100 iter. | 62.0 | 60.7 | 57.3 | 69.0 | 68.7 | 68.1
Iterative HT | 1000 iter. | 55.0 | 55.4 | 54.1 | 68.5 | 68.3 | 67.9
No data-aware opt. | - | 58.5 | 53.3 | 37.8 | 69.0 | 68.7 | 67.9
Table 1: Resnet18 ablation on the impact of the hard thresholding method and
the stopping criterion on top-1 validation accuracy of the compressed model.
Iterative hard-thresholding with 30 iterations shows the best results among
all the other methods, however, the benefit vanishes after fine-tuning.

In this section, we study the impact of the hard thresholding method and the
stopping criterion for data-aware optimization. Algorithm 1 applies the hard-
thresholding step at each iteration. However, we can also apply it only after
the end of gradient descent iterations. We call this one-shot hard-
thresholding. The method has precedence in non-convex sparse optimization
((Shen, 2020)). We consider two different stopping criteria. The first is
using the fixed number of gradient descent iterations, e.g., 30, 100, or 1000.
The second is based on achieving the MSE error threshold on the validation
set. We split the data used for per-layer optimization into a training set and
a validation set. We stop the iterations as soon as the error on the
validation set is not decreasing for more than two iterations. Finally, we
compare both methods with one-shot hard-thresholding without data-aware
optimization.

We present the results for Resnet18 in the Table 1. Three different levels of
sparsity are considered for a model with 4 bit latent. We report the
validation accuracy of the model before and after fine-tuning. The results
suggest that the most important aspect of data-aware optimization is the
stopping criterion. The best pre-finetuning results are obtained by using
iterative hard thresholding with the least number of iterations, namely 30.
The second best method is one-shot iterative hard thresholding with the
stopping criterion based on the validation set. These results suggest that it
is important to limit the number of iterations of per layer optimization in
order to prevent the method from over-fitting to the layer input and output
data sample. This intuition is further supported by observing the lowest pre-
finetuning accuracy for hard thresholding with the highest number of
iterations, namely 1000.

However, overall benefits of per layer optimization methods become marginal
after the compressed model is fine-tuned. In various experiments, we observed
an 0.1-0.3% improvement in the validation accuracy compared to using one-shot
hard thresholding without any data-aware optimization. Therefore, in our
further full model experiments, we used one-shot hard thresholding with the
stopping criterion based on the validation set as a method of low
computational complexity, which gives nearly the best results compared to the
other methods.

####  4.3.2 Quantization vs Pruning vs Rank

In our method, compression can be achieved by lowering rank, lowering the bit-
width for latent matrices (hence referred to as ‘bit-width’), or increasing
extra sparsity. Thus, similar compression ratios can be achieved with
different values for rank, bit-width and sparsity. For example, for a given
model, we can retain the same compression ratio by increasing rank and
decreasing bit-width to compensate. In this section we investigate whether we
can discover patterns that would help guide compression hyperparameter search.

In Figure 4 we show a scatter plot with the full set of results. In this
figure, we see that for low compression ratios (below 14x) only minor
improvements can be achieved by tweaking compression hyperparameters. For
example, around 10x compression even the worst performing compression
hyperparameters are still on par with quantization to 3 bits (10.67x
compression). However, for higher compression ratios the compression
hyperparameters have a larger impact on model performance.

In Figure 5 we show the effect of rank, bit-width and sparsity ratios on a set
of models with a low (near 11x, top row) and a high (near 25x, bottom row)
compression ratio. Again we see little effect (roughtly 0.5%) for the lower
compression ratio. However, for the more aggressively compressed models in the
bottom row, we see that increasing rank while lowering bit-width to compensate
has a strong positive effect on model performance. Finally, in both plots we
see no clearly discernible trend in the effect of compression ratio on model
accuracy.

Based on these results we conclude that, at high compression ratios,
increasing rank and decreasing bit-widths can improve results. Sparsity can be
used to further fine-tune the target compression ratio.

Figure 4: All results for ResNet18, including non-Pareto dominating results.
Figure 5: Resnet18, the effect of rank, bit-width and sparsity for models near
11x (top row) and 25x (bottom row) compression ratios.

##  5 Conclusion

We presented a weight compression method based on quantized sparse PCA which
unifies SVD-based weight compression, sparse PCA, and vector quantization. We
tested our method on ImageNet classification and show compression-accuracy
trade-offs which are state-of-the-art or on par with strong baselines, and
demonstrated that our method is the only method to achieve SOTA compression-
accuracy trade-offs in both low and high compression regimes. Lastly, we
investigated whether low rank approximations, quantization, or sparsity yield
the best efficiency-accuracy trade-offs, and found that at high compression
ratios, increasing rank and decreasing bit-width can improve results. For
future work, we plan to investigate methods for automated rank, bit-width, and
sparsity ratio selection in order to further improve the accuracy and reduce
the hyper-parameter search time.

## References

  * Babenko & Lempitsky (2014) Babenko, A. and Lempitsky, V.  Additive quantization for extreme vector compression.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pp. 931–938, 2014.
  * Bengio et al. (2013) Bengio, Y., Léonard, N., and Courville, A.  Estimating or propagating gradients through stochastic neurons for conditional computation.  _arXiv preprint arXiv:1308.3432_ , 2013.
  * Bhalgat et al. (2020) Bhalgat, Y., Lee, J., Nagel, M., Blankevoort, T., and Kwak, N.  Lsq+: Improving low-bit quantization through learnable offsets and better initialization.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops_ , June 2020.
  * Chen et al. (2010) Chen, Y., Guan, T., and Wang, C.  Approximate nearest neighbor search by residual vector quantization.  _Sensors_ , 10(12):11259–11273, 2010.
  * Dai et al. (2018) Dai, B., Zhu, C., Guo, B., and Wipf, D.  Compressing neural networks using the variational information bottleneck.  In _International Conference on Machine Learning_ , pp. 1135–1144. PMLR, 2018.
  * Denil et al. (2013) Denil, M., Shakibi, B., Dinh, L., Ranzato, M., and De Freitas, N.  Predicting parameters in deep learning.  _arXiv preprint arXiv:1306.0543_ , 2013.
  * Denton et al. (2014) Denton, E. L., Zaremba, W., Bruna, J., LeCun, Y., and Fergus, R.  Exploiting linear structure within convolutional networks for efficient evaluation.  In _Advances in neural information processing systems_ , pp. 1269–1277, 2014.
  * Dong et al. (2019) Dong, Z., Yao, Z., Gholami, A., Mahoney, M. W., and Keutzer, K.  Hawq: Hessian aware quantization of neural networks with mixed-precision.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pp. 293–302, 2019.
  * Esser et al. (2019) Esser, S. K., McKinstry, J. L., Bablani, D., Appuswamy, R., and Modha, D. S.  Learned step size quantization.  _arXiv preprint arXiv:1902.08153_ , 2019.
  * Fan et al. (2020) Fan, A., Stock, P., Graham, B., Grave, E., Gribonval, R., Jegou, H., and Joulin, A.  Training with quantization noise for extreme model compression.  _arXiv preprint arXiv:2004.07320_ , 2020.
  * Foucart & Rauhut (2013) Foucart, S. and Rauhut, H.  _A Mathematical Introduction to Compressive Sensing_.  Applied and Numerical Harmonic Analysis. Springer New York, New York, NY, 2013.  ISBN 978-0-8176-4947-0 978-0-8176-4948-7.  URL http://link.springer.com/10.1007/978-0-8176-4948-7.
  * Gale et al. (2019) Gale, T., Elsen, E., and Hooker, S.  The state of sparsity in deep neural networks.  _arXiv preprint arXiv:1902.09574_ , 2019.
  * Ge et al. (2013) Ge, T., He, K., Ke, Q., and Sun, J.  Optimized product quantization.  _IEEE transactions on pattern analysis and machine intelligence_ , 36(4):744–755, 2013.
  * Gersho & Gray (2012) Gersho, A. and Gray, R. M.  _Vector quantization and signal compression_ , volume 159.  Springer Science & Business Media, 2012.
  * Gong et al. (2014) Gong, Y., Liu, L., Yang, M., and Bourdev, L.  Compressing deep convolutional networks using vector quantization.  _arXiv preprint arXiv:1412.6115_ , 2014.
  * He et al. (2017) He, Y., Zhang, X., and Sun, J.  Channel pruning for accelerating very deep neural networks.  In _Proceedings of the IEEE international conference on computer vision_ , pp. 1389–1397, 2017.
  * Jacob et al. (2018) Jacob, B., Kligys, S., Chen, B., Zhu, M., Tang, M., Howard, A. G., Adam, H., and Kalenichenko, D.  Quantization and training of neural networks for efficient integer-arithmetic-only inference.  In _2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018_ , pp. 2704–2713. Computer Vision Foundation / IEEE Computer Society, 2018.  doi: 10.1109/CVPR.2018.00286.  URL http://openaccess.thecvf.com/content_cvpr_2018/html/Jacob_Quantization_and_Training_CVPR_2018_paper.html.
  * Jacques et al. (2013) Jacques, L., Laska, J. N., Boufounos, P. T., and Baraniuk, R. G.  Robust 1-bit compressive sensing via binary stable embeddings of sparse vectors.  _IEEE Transactions on Information Theory_ , 59(4):2082–2102, 2013.
  * Jaderberg et al. (2014) Jaderberg, M., Vedaldi, A., and Zisserman, A.  Speeding up convolutional neural networks with low rank expansions.  _arXiv preprint arXiv:1405.3866_ , 2014.
  * Jenatton et al. (2010) Jenatton, R., Obozinski, G., and Bach, F.  Structured sparse principal component analysis.  In _Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics_ , pp. 366–373, 2010.
  * Kaplan & Tichatschke (1998) Kaplan, A. and Tichatschke, R.  Proximal point methods and nonconvex optimization.  _Journal of global Optimization_ , 13(4):389–406, 1998.  Publisher: Springer.
  * Kim et al. (2015) Kim, Y.-D., Park, E., Yoo, S., Choi, T., Yang, L., and Shin, D.  Compression of deep convolutional neural networks for fast and low power mobile applications.  _arXiv preprint arXiv:1511.06530_ , 2015.
  * Kingma & Ba (2014) Kingma, D. P. and Ba, J.  Adam: A method for stochastic optimization.  _arXiv preprint arXiv:1412.6980_ , 2014.
  * Krishnamoorthi (2018) Krishnamoorthi, R.  Quantizing deep convolutional networks for efficient inference: A whitepaper.  _arXiv preprint arXiv:1806.08342_ , 2018.
  * Lebedev et al. (2014) Lebedev, V., Ganin, Y., Rakhuba, M., Oseledets, I., and Lempitsky, V.  Speeding-up convolutional neural networks using fine-tuned cp-decomposition.  _arXiv preprint arXiv:1412.6553_ , 2014.
  * Lee et al. (2006) Lee, H., Battle, A., Raina, R., and Ng, A. Y.  Efficient sparse coding algorithms.  In _Proceedings of the 19th International Conference on Neural Information Processing Systems_ , pp. 801–808. MIT Press, 2006\.
  * Lin et al. (2017) Lin, X., Zhao, C., and Pan, W.  Towards accurate binary convolutional neural network.  _arXiv preprint arXiv:1711.11294_ , 2017.
  * Liu et al. (2019) Liu, D., Li, S., and Shen, Y.  One-Bit Compressive Sensing With Projected Subgradient Method Under Sparsity Constraints.  _IEEE Transactions on Information Theory_ , 65(10):6650–6663, October 2019.  Conference Name: IEEE Transactions on Information Theory.
  * Louizos et al. (2017) Louizos, C., Welling, M., and Kingma, D. P.  Learning sparse neural networks through $l\\_0$ regularization.  _arXiv preprint arXiv:1712.01312_ , 2017.
  * Ma (2013) Ma, Z.  Sparse principal component analysis and iterative thresholding.  _The Annals of Statistics_ , 41(2):772–801, April 2013.  ISSN 0090-5364.  doi: 10.1214/13-AOS1097.  URL http://projecteuclid.org/euclid.aos/1368018173.
  * Martinez et al. (2021) Martinez, J., Shewakramani, J., Liu, T. W., Bârsan, I. A., Zeng, W., and Urtasun, R.  Permute, quantize, and fine-tune: Efficient compression of neural networks.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 15699–15708, 2021.
  * Matsui et al. (2018) Matsui, Y., Uchida, Y., Jégou, H., and Satoh, S.  A survey of product quantization.  _ITE Transactions on Media Technology and Applications_ , 6(1):2–10, 2018.
  * Nagel et al. (2021) Nagel, M., Fournarakis, M., Amjad, R. A., Bondarenko, Y., van Baalen, M., and Blankevoort, T.  A white paper on neural network quantization.  _ArXiv_ , abs/2106.08295, 2021.
  * Neklyudov et al. (2017) Neklyudov, K., Molchanov, D., Ashukha, A., and Vetrov, D.  Structured bayesian pruning via log-normal multiplicative noise.  _arXiv preprint arXiv:1705.07283_ , 2017.
  * Norouzi & Fleet (2013) Norouzi, M. and Fleet, D. J.  Cartesian k-means.  In _Proceedings of the IEEE Conference on computer Vision and Pattern Recognition_ , pp. 3017–3024, 2013.
  * Parikh et al. (2014) Parikh, N., Boyd, S. P., and Now Publishers.  _Proximal algorithms_.  2014\.  OCLC: 905838036.
  * Shen (2020) Shen, J.  One-bit compressed sensing via one-shot hard thresholding.  In _Conference on Uncertainty in Artificial Intelligence_ , pp. 510–519. PMLR, 2020.
  * Stock et al. (2019) Stock, P., Joulin, A., Gribonval, R., Graham, B., and Jégou, H.  And the bit goes down: Revisiting the quantization of neural networks.  _arXiv preprint arXiv:1907.05686_ , 2019.
  * Su et al. (2018) Su, J., Li, J., Bhattacharjee, B., and Huang, F.  Tensorized spectrum preserving compression for neural networks.  _arXiv preprint arXiv:1805.10352_ , 2018.
  * Uhlich et al. (2019) Uhlich, S., Mauch, L., Yoshiyama, K., Cardinaux, F., Garcia, J. A., Tiedemann, S., Kemp, T., and Nakamura, A.  Differentiable quantization of deep neural networks.  _arXiv preprint arXiv:1905.11452_ , 2(8), 2019.
  * van Baalen et al. (2020) van Baalen, M., Louizos, C., Nagel, M., Amjad, R. A., Wang, Y., Blankevoort, T., and Welling, M.  Bayesian bits: Unifying quantization and pruning.  _arXiv preprint arXiv:2005.07093_ , 2020.
  * Vogels et al. (2019) Vogels, T., Karinireddy, S. P., and Jaggi, M.  Powersgd: Practical low-rank gradient compression for distributed optimization.  _Advances In Neural Information Processing Systems 32 (Nips 2019)_ , 32(CONF), 2019.
  * Wang et al. (2019) Wang, K., Liu, Z., Lin, Y., Lin, J., and Han, S.  Haq: Hardware-aware automated quantization with mixed precision.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 8612–8620, 2019.
  * Wu et al. (2016) Wu, J., Leng, C., Wang, Y., Hu, Q., and Cheng, J.  Quantized convolutional neural networks for mobile devices.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pp. 4820–4828, 2016.
  * Zhang et al. (2015) Zhang, X., Zou, J., He, K., and Sun, J.  Accelerating very deep convolutional networks for classification and detection.  _IEEE transactions on pattern analysis and machine intelligence_ , 38(10):1943–1955, 2015.
  * Zhu et al. (2016) Zhu, C., Han, S., Mao, H., and Dally, W. J.  Trained ternary quantization.  _arXiv preprint arXiv:1612.01064_ , 2016.
  * Zhu & Gupta (2017) Zhu, M. and Gupta, S.  To prune, or not to prune: exploring the efficacy of pruning for model compression.  _arXiv preprint arXiv:1710.01878_ , 2017.
  * Zou et al. (2006) Zou, H., Hastie, T., and Tibshirani, R.  Sparse principal component analysis.  _Journal of computational and graphical statistics_ , 15(2):265–286, 2006.

##  Appendix A Hard Thresholding for Sparse Optimization

Projected gradient descent methods are standard tools for solving constrained
optimization problem. Consider the following optimization problem:

$\min_{{\bm{x}}\in{\mathcal{D}}}f({\bm{x}}).$ (15)

The set ${\mathcal{D}}$ is an arbitrary set. The projection function on
${\mathcal{D}}$ is defined as

$\Pi_{\mathcal{D}}({\bm{z}})=\arg\min_{{\bm{x}}\in{\mathcal{D}}}\left\lVert{\bm{x}}-{\bm{z}}\right\rVert_{2}.$

Projected gradient descent is a special case of proximal gradient methods and
consists of following updates:

$x_{t+1}=\Pi_{\mathcal{D}}({\bm{x}}_{t}-\gamma\nabla f({\bm{x}}_{t})),$

There are many works on convergence proof and further analysis of proximal
methods in convex optimization (see (Parikh et al., 2014) and references
therein). There are extensions to nonconvex problems too, e.g., (Kaplan &
Tichatschke, 1998).

An example of non-convex problems is when the set ${\mathcal{D}}$ is the set
of sparse vector denoted by $\Sigma_{s}({\mathbb{R}}^{n})$ with sparsity order
$s$. In this case, the projection function is hard thresholding. For a vector
${\bm{x}}$, the hard thresholding function $H_{s}({\bm{x}})$ yields a vector
with at most $s$ non-zero entries, which are the $s$ entries of ${\bm{x}}$
with largest absolute value. The sparse optimization method sketched in this
paper is based on one-shot or alternating applications of thresholding steps.
Hard thresholding methods are also used for linear sparse recovery problem
stated as:

$\text{Find }\quad{\bm{x}}\quad\text{ s.t.
}{\bm{y}}={\bm{A}}{\bm{x}},{\bm{x}}\in\Sigma_{s}({\mathbb{R}}^{n}).$ (16)

One can prove recovery guarantee for iterative hard thresholding methods when
the matrix ${\bm{A}}$ satisfies restricted isometry property. See (Foucart &
Rauhut, 2013) for a comprehensive exposition of the topic including iterative
hard thresholding for linear problems.

Another example is the one-bit sparse recovery problem where the goal is to
find a sparse vector from binary quantized measurements:

$\text{Find }\quad{\bm{x}}\quad\text{ s.t.
}{\bm{y}}=Q({\bm{A}}{\bm{x}}),{\bm{x}}\in\Sigma_{s}({\mathbb{R}}^{n}).$ (17)

This problem is closely related to ours with its quantization and sparsity
constraint. Binary iterative hard thresholding algorithm (BIHT), proposed in
(Jacques et al., 2013) for solving this problem, alternates between a gradient
step and a thresholding step, similar to our proposed algorithm. Deriving
similar recovery guarantees for BIHT is more challenging. A convergence
analysis of BIHT can be found in (Liu et al., 2019). Following similar
analysis to (Liu et al., 2019), the following proposition provides a
simplified general convergence analysis for that the optimization problem (15)
can be solved using iterative projection methods.

######  Proposition A.1.

Consider the optimization problem equation 15 and assume that
${\bm{x}}_{\star}$ is the minimizer of the problem. Consider an iterative
optimization algorithm with the initial point ${\bm{x}}_{0}$ and the iteration
$t$ defined by:

$x_{t+1}=\Pi_{\mathcal{D}}({\bm{x}}_{t}-\Delta_{f}({\bm{x}}_{t})),$

where $\Delta_{f}(.)$ is a function such that ${\bm{x}}-\Delta_{f}({\bm{x}})$
is $L$-Lipschitz, and $\Delta_{f}({\bm{x}}_{\star})=0$. Then we have:

$\left\lVert{\bm{x}}_{t}-{\bm{x}}_{\star}\right\rVert_{2}\leq(2L)^{t}\left\lVert{\bm{x}}_{0}-{\bm{x}}_{\star}\right\rVert_{2}.$

In particular, if $L\leq 1/2$, the algorithm converges to the minimizer
${\bm{x}}_{\star}$ as $t\to\infty$.

######  Remark A.2.

Some remarks are in order. First, the function $\Delta_{f}({\bm{x}})$ is a
proxy for gradient, sub-gradient or any other approximate gradient functions.
The assumption $\Delta_{f}({\bm{x}}_{\star})=0$ means that ${\bm{x}}_{\star}$
is a stationary point. This holds particularly for gradient descent based
methods where $\Delta_{f}({\bm{x}}_{\star})=\gamma\nabla f({\bm{x}})$ and
${\bm{x}}_{\star}$ is a stationary point of $f$. The Lipschitz property of
${\bm{x}}-\Delta_{f}({\bm{x}})$ and particularly having $L<1/2$ are crucial
for convergence. The Lipschitz property of ${\bm{x}}-\Delta_{f}({\bm{x}})$ is
a demanding constraint but holds in many situations. As an example, consider
the function $f(\cdot)$ as a simple mean squared error (MSE) loss, i.e.,
$f({\bm{x}})=\left\lVert{\bm{y}}-{\bm{A}}{\bm{x}}\right\rVert_{2}^{2}$. In
this case, we have:

${\bm{x}}-\Delta_{f}({\bm{x}})={\bm{x}}-\alpha{\bm{A}}{\bm{A}}^{\top}{\bm{x}}=({\bm{I}}-\alpha{\bm{A}}{\bm{A}}^{\top}){\bm{x}},$

which satisfies $L$-Lipschitz property with $L$ bounded by the spectral norm
of ${\bm{I}}-\alpha{\bm{A}}{\bm{A}}^{\top}$. In many situations, when this
spectral norm is restricted to the set ${\mathcal{D}}$, it can be controlled
to be less than $1/2$, although the proof would need to be modified. See
chapter 6.3 in (Foucart & Rauhut, 2013).

###### Proof.

First, for an arbitrary set ${\mathcal{D}}\subseteq{\mathbb{R}}^{n}$, the
projection operation onto the set $\Pi_{\mathcal{D}}(\cdot)$ and any
${\bm{x}}\in{\mathcal{D}}$ and ${\bm{z}}\in{\mathbb{R}}^{n}$, we have the
following inequality

$\left\lVert\Pi_{\mathcal{D}}({\bm{z}})-{\bm{x}}\right\rVert_{2}\leq
2\left\lVert{\bm{z}}-{\bm{x}}\right\rVert_{2}$

We can obtain this inequality in two steps. We first use triangle inequality:

$\left\lVert\Pi_{\mathcal{D}}({\bm{z}})-{\bm{x}}\right\rVert_{2}\leq\left\lVert{\bm{z}}-{\bm{x}}\right\rVert_{2}+\left\lVert\Pi_{\mathcal{D}}({\bm{z}})-{\bm{z}}\right\rVert_{2}.$

For the last term, we use the definition of projection operation that for each
${\bm{x}}\in{\mathcal{D}}$, the projection of ${\bm{z}}$ to ${\mathcal{D}}$
minimizes its distance to the points in ${\mathcal{D}}$:

$\left\lVert\Pi_{\mathcal{D}}({\bm{z}})-{\bm{z}}\right\rVert_{2}\leq\left\lVert{\bm{z}}-{\bm{x}}\right\rVert_{2}.$

Consider the iteration step $t+1$. We have:

$\left\lVert\Pi_{\mathcal{D}}({\bm{x}}_{t}-\Delta_{f}({\bm{x}}_{t}))-{\bm{x}}_{\star}\right\rVert_{2}\leq
2\left\lVert{\bm{x}}_{t}-\Delta_{f}({\bm{x}}_{t})-{\bm{x}}_{\star}\right\rVert_{2}=2\left\lVert{\bm{x}}_{t}-\Delta_{f}({\bm{x}}_{t})-{\bm{x}}_{\star}+\Delta_{f}({\bm{x}}_{\star})\right\rVert_{2}\leq
2L\left\lVert{\bm{x}}_{t}-{\bm{x}}_{\star}\right\rVert_{2},$

which implies

$\left\lVert{\bm{x}}_{t+1}-{\bm{x}}_{\star}\right\rVert_{2}\leq(2L)^{t+1}\left\lVert{\bm{x}}_{0}-{\bm{x}}_{\star}\right\rVert_{2}.$

∎

The analysis provided here does not apply to our algorithm. However, the
optimization steps is similar to existing variants of thresholding algorithms.

##  Appendix B ResNet18 experiments

###  B.1 Experimental details

We run a small number of pilot experiments for 5 epochs, to find good settings
for the optimizer, learning rate, and learning rate schedule. We then run a
large number of 5 epoch experiments to find good compression-accuracy trade-
offs. For the best performing compression-accuracy trade-offs we then run
longer experiments of 25 epochs, with the two best optimization schedules. We
use a batch size of 64 for Resnet18, use cosine learning rate decay to
$10^{-3}$ of the initial learning rate, and turn off weight decay.

###  B.2 Pareto dominating results

K | D | bw | mpfd | cr_sparse | eval_score
---|---|---|---|---|---
256 | 256 | 5 | 0.00 | 6.01 | 70.26
256 | 256 | 5 | 0.20 | 6.96 | 70.06
256 | 256 | 4 | 0.20 | 8.91 | 70.04
192 | 256 | 4 | 0.20 | 11.50 | 69.77
192 | 256 | 3 | 0.00 | 13.20 | 69.47
128 | 256 | 3 | 0.00 | 16.95 | 69.15
128 | 256 | 3 | 0.00 | 18.17 | 68.91
128 | 256 | 4 | 0.40 | 20.19 | 68.73
128 | 256 | 3 | 0.20 | 21.70 | 68.63
92 | 256 | 3 | 0.15 | 23.04 | 68.17
92 | 256 | 3 | 0.15 | 25.34 | 68.11
100 | 256 | 3 | 0.12 | 27.49 | 67.52
64 | 256 | 3 | 0.00 | 29.02 | 67.20
92 | 256 | 3 | 0.15 | 30.02 | 66.94
64 | 256 | 3 | 0.20 | 32.62 | 66.86
64 | 256 | 3 | 0.00 | 33.78 | 66.37
64 | 256 | 3 | 0.20 | 39.88 | 64.48
Table 2: All Pareto dominating results for ResNet18

###  B.3 Ablation study results

compression ratios near 11x | compression ratios near 25x
---|---
K | D | bw | sparsity | comp. ratio | accuracy | K | D | bw | sparsity | comp. ratio | accuracy
256 | 256 | 4 | 0.3 | 10.87 | 69.31 | 39 | 256 | 6 | 0.00 | 25.97 | 54.04
128 | 256 | 5 | 0.1 | 11.41 | 69.32 | 40 | 256 | 6 | 0.10 | 25.52 | 54.06
256 | 256 | 3 | 0.0 | 11.77 | 69.45 | 48 | 256 | 5 | 0.00 | 25.26 | 63.03
192 | 256 | 5 | 0.4 | 11.07 | 69.49 | 58 | 256 | 4 | 0.00 | 25.57 | 64.89
128 | 256 | 5 | 0.0 | 11.24 | 69.57 | 64 | 256 | 5 | 0.40 | 25.57 | 66.03
192 | 256 | 4 | 0.2 | 11.19 | 69.77 | 58 | 256 | 4 | 0.00 | 25.57 | 67.11
192 | 256 | 4 | 0.1 | 10.56 | 69.72 | 60 | 256 | 4 | 0.13 | 25.45 | 67.18
|  |  |  |  |  | 64 | 256 | 3 | 0.00 | 26.07 | 67.40
|  |  |  |  |  | 77 | 256 | 3 | 0.00 | 25.56 | 68.00
|  |  |  |  |  | 92 | 256 | 3 | 0.15 | 25.34 | 68.11
Table 3: Various hyperparameter settings and resulting compression ratios and
accuracies for compression ratios near 11x and 25.5x

  *[PCA]: principle component analysis
  *[STE]: straight through estimator
  *[MSE]: mean squared error


# Measuring magnetic flux suppression in high-power laser-plasma interactions

P. T. Campbell<EMAIL_ADDRESS>Gérard Mourou Center for Ultrafast Optical
Science, University of Michigan, 2200 Bonisteel Boulevard, Ann Arbor, Michigan
48109, USA  C. A. Walsh  Lawrence Livermore National Laboratory, Livermore,
California 94550, USA  B. K. Russell  Gérard Mourou Center for Ultrafast
Optical Science, University of Michigan, 2200 Bonisteel Boulevard, Ann Arbor,
Michigan 48109, USA  J. P. Chittenden  A. Crilly  Blackett Laboratory,
Imperial College, London SW7 2AZ, United Kingdom  G. Fiksel  Gérard Mourou
Center for Ultrafast Optical Science, University of Michigan, 2200 Bonisteel
Boulevard, Ann Arbor, Michigan 48109, USA  L. Gao  Princeton Plasma Physics
Laboratory, Princeton University, Princeton, New Jersey 08543, USA  I. V.
Igumenshchev  P. M. Nilson  Laboratory for Laser Energetics, 250 East River
Road, Rochester, New York 14623, USA  A. G. R. Thomas  K. Krushelnick  L.
Willingale  Gérard Mourou Center for Ultrafast Optical Science, University of
Michigan, 2200 Bonisteel Boulevard, Ann Arbor, Michigan 48109, USA

###### Abstract

Biermann battery magnetic field generation driven by high power laser-solid
interactions is explored in experiments performed with the OMEGA EP laser
system. Proton deflectometry captures changes to the strength, spatial
profile, and temporal dynamics of the self-generated magnetic fields as the
target material or laser intensity is varied. Measurements of the magnetic
flux during the interaction are used to help validate extended
magnetohydrodynamic (MHD) simulations. Results suggest that kinetic effects
cause suppression of the Biermann battery mechanism in laser-plasma
interactions relevant to both direct and indirect-drive inertial confinement
fusion. Experiments also find that more magnetic flux is generated as the
target atomic number is increased, which is counter to a standard MHD
understanding.

##  I Introduction

High power laser-solid interactions can create high energy density (HED)
plasma conditions relevant to inertial confinement fusion (ICF) and laboratory
astrophysics research Drake (2018); Remington  _et al._ (2006). In laser-
produced plasmas, strong magnetic fields can be spontaneously generated by a
number of mechanisms Haines (1986a), though the primary source is the Biermann
battery effect caused by nonparallel temperature and density gradients
($\partial B/\partial t\propto\nabla T_{e}\times\nabla n_{e}$) Biermann and
Schlüter (1951); Stamper  _et al._ (1971); Stamper and Ripin (1975). A
detailed understanding of self-generated magnetic fields is critical to laser-
fusion research because strong fields can influence thermal energy transport
Braginskii (1965), and potentially impact the evolution of hydrodynamic
instabilities Evans (1986); Manuel  _et al._ (2012); Gao  _et al._ (2012,
2013); Hill and Kingham (2018). Laser-driven magnetic fields also enable
laboratory investigations of magnetized astrophysical phenomena, especially
magnetic reconnection Nilson  _et al._ (2006); Li  _et al._ (2007); Willingale
_et al._ (2010a); Zhong  _et al._ (2010); Fox  _et al._ (2011); Dong  _et al._
(2012).

For both ICF and laboratory astrophysics research, numerical modeling is an
essential predictive tool, and the extended-magnetohydrodynamics (extended-
MHD) framework has been developed to describe transport of energy and magnetic
fields in HED plasmas Walsh  _et al._ (2020). Recent simulation work has shown
that extended-MHD effects such as Nernst and Righi-Leduc can modify plasma
properties in indirect-drive ICF hohlraums Farmer  _et al._ (2017), direct-
drive ICF ablation fronts Hill and Kingham (2018), and at the edge of
compressed fusion fuel Walsh  _et al._ (2017). Accurate extended-MHD modeling
is crucial to development and interpretation of advanced ICF experiments with
pre-magnetized fuel Davies  _et al._ (2015); McBride and Slutz (2015);
Joglekar  _et al._ (2016); Walsh  _et al._ (2019). In addition, simulations
anticipate that extended-MHD effects can dictate reconnection rates in laser-
driven magnetic reconnection experiments Joglekar  _et al._ (2014); Tubman
_et al._ (2021).

Though relatively simple in the broader context of HED experiments, a single
laser spot interacting with foil targets can provide a powerful platform for
validating extended-MHD modeling. Proton deflectometry enables high spatial
and temporal resolution measurements of magnetic field generation and dynamics
in the laser-produced plasma Borghesi  _et al._ (2001); Li  _et al._ (2006).
Using moderate laser intensities, $I_{L}=10^{14}-10^{15}$ Wcm-2, recent
experiments have demonstrated that simulations of laser-foil interactions must
incorporate key physical processes such as Biermann battery field generation,
cross-field Righi-Leduc heat flow, and Nernst advection Willingale  _et al._
(2010b); Lancia  _et al._ (2014); Gao  _et al._ (2015). The Nernst effect
moves fields down temperature gradients ($\textrm{v}_{\textrm{N}}\propto-
T_{e}^{3/2}\nabla T_{e}$) Nishiguchi  _et al._ (1984); Kho and Haines (1985);
Ridgers  _et al._ (2008a). In laser-produced plasmas, the Nernst effect can
convect magnetic fields with the heat flow toward the ablation region, counter
to the bulk plasma flow into the corona Haines (1986b). Measurements of the
magnetic field dynamics can be used to diagnose temperature and density
gradients in the plasma, and interplay between energy transport and field
generation.

By varying the target material, the effect of atomic or radiation physics on
transport and field dynamics can be explored. Recent work using proton
deflectometry captured distinct regions of magnetic field generation around
radiation-driven double ablation fronts (DAF) in mid-Z targets Campbell  _et
al._ (2020). Incorporating radiation transport into extended-MHD simulations
reproduced the DAF formation and concentric double field features.

In that work, it was found that extended-MHD simulations overestimated the
magnetic field strength. It is anticipated that non-local effects not captured
by the extended-MHD framework can suppress the rate of Biermann battery field
generation in regions where the electron mean free path ($\lambda_{ei}$)
approaches (or exceeds) the local temperature gradient length scale
($l_{T}=(T_{e}/\nabla T_{e})$) Sherlock and Bissell (2020); Hill and Kingham
(2018). Using empirical fits to kinetic simulations, Sherlock and Bissell
Sherlock and Bissell (2020) developed scalings for the suppression of
classical Biermann battery generation rates as a function of the ratio
$\lambda_{ei}/l_{T}$. However, the results have not yet been compared to
experiments until now.

In this paper, high resolution proton deflectometry measurements quantify
target material effects on magnetic fields generated during high power laser-
solid interactions. Experimental observations of magnetic flux are used to
help validate extended-MHD simulations that include the new scalings for non-
local suppression of Biermann battery field generation, as well as radiation
transport.

##  II Methods

###  II.1 Experiments

In this work, data is drawn from two separate experimental campaigns performed
with the OMEGA EP laser system at the University of Rochester’s Laboratory for
Laser Energetics. Magnetic field generation was driven by either one Campbell
_et al._ (2020) or two overlapped Gao  _et al._ (2015) UV laser pulses
($\lambda_{L}=351$ nm) interacting with thin foil targets. In the single pulse
case, a beam with 1.25 kJ of energy and 1 ns square temporal profile was
focused using a distributed phase plate on to an 819 $\mu$m diameter
($d_{95}$) super-Gaussian spot with a $\sim 30^{\circ}$ angle of incidence to
produce an intensity of $2.2\times 10^{14}$ Wcm-2. The foil target material
was varied between 50 $\mu$m thick plastic (CH), 25 $\mu$m copper, 25 $\mu$m
aluminum, or 50 $\mu$m aluminum coated with either 1 $\mu$m of copper (Cu+Al)
or gold (Au+Al).

Figure 1: (a) Schematic of the OMEGA EP experimental setup. (b) An
illustration of the proton image analysis method. Due to the laser angle-of-
incidence, the field features are mildly elliptical. Line-integrated magnetic
field profiles are reconstructed from normalized radial line-outs (J/J0) taken
along minor axis. Integrating the field profile in the radial direction yields
a measurement of the magnetic flux.

In the two beam experiment, each pulse contained 2 kJ in a 2.5 ns square
temporal profile. The pulses were overlapped onto an 734 $\mu$m diameter spot
with a $23^{\circ}$ angle of incidence to produce an combined intensity of
$4.4\times 10^{14}$ Wcm-2. For this series of shots, the targets were limited
to 50 $\mu$m thick plastic foils.

A schematic of the experimental geometry is shown in Figure 1(a). Self-
generated magnetic fields were imaged by protons in a point-projection
geometry with magnifications ranging from $\sim$10–14. In both experiments,
the high energy proton probe was produced via the target normal sheath
acceleration (TNSA) mechanism Maksimchuk  _et al._ (2000); *Snavely_PRL_2000;
*Clark_PRL_2000; *Hatchett_POP_2000. A short infrared (IR) laser pulse (300 J,
0.7–1 ps) was focused to intensities exceeding $10^{19}$ Wcm-2 onto 20–50
$\mu$m thick copper foils, accelerating protons with a quasi-Maxwellian energy
spectrum and maximum energies around 60 MeV. To protect the proton source from
coronal plasma or x-ray preheat emitted from the main interaction, the foil
was mounted within a plastic tube capped by a 5 $\mu$m thick tantalum shield.
Because of the laminar propagation, small virtual source size ($\sim 10$
$\mu$m), and short emission duration ($\sim 1$ ps), proton beams from a TNSA
source enable high spatial and temporal resolution imaging of self-generated
electric and magnetic fields Borghesi  _et al._ (2004). The relative timing
between the main interaction and the proton probe could be adjusted with $\pm
20$ ps error to measure the temporal evolution of the field features.

The proton beams were detected with filtered stacks of radiochromic film
(RCF). Due to a combination of the Bragg peak in proton energy deposition and
a thin sensitive region, each RCF layer detects a narrow energy slice of the
accelerated spectrum ($\Delta E/E\leq 1\%$). Deflections from fields generated
in the main interaction will result in proton fluence modulations on the film.
Quantitative measurements of path-integrated field strengths can be retrieved
from the relative distribution of protons compared to the undisturbed beam
profile Borghesi  _et al._ (2001); Li  _et al._ (2006); Kugland  _et al._
(2012).

A 1D polar-coordinates field reconstruction technique was developed to extract
quantitative path-integrated magnetic field information from radial line-outs
through the proton images. A detailed description of the reconstruction method
can be found in the Supplemental Material of Ref. Campbell  _et al._ , 2020.
The proton image analysis method is illustrated in Figure 1(b). In this
probing geometry, protons are primarily sensitive to azimuthal magnetic fields
generated in the laser-produced plasma Li  _et al._ (2006). Due to the laser
angle of incidence, the observed features are elliptical. Typically, radial
line-outs were taken along the minor axis for best comparison to the 2D (r-z)
simulations described below.

A key challenge and source of error in the field reconstruction method is
accurate determination of the undisturbed proton profile, J0. Shot-to-shot
fluctuations in the undeflected beam profile and fluence means that direct use
of reference data taken from other shots is ineffective. Instead, the low
spatial frequency undisturbed profiles were inferred from the line-out signal
(J) using Fourier filtering. (Note: in this work, the Fourier filter is
applied to a line-out containing the full diameter of the proton image
feature) After Gaussian low-pass filtering, the signal level is adjusted such
that total proton flux is conserved ($\sum$J $=\sum$ J0). The reconstructed
field profiles are constrained by assuming that the field strength should drop
to zero near the center of the focal region and the outer edge (far from the
interaction). This is accomplished by using a super-Gaussian mask to blend the
filtered signal with the original line-out such that $J/J_{0}=1$ as
$r<r_{min}$ and $r>r_{max}$.

For each line-out, a scan of Gaussian low-pass filter widths, $r_{min}$, and
$r_{max}$ values are tested. Filter widths range from 0.4 to 3 mm, with 0.05
mm spacing (from approximately the diameter of ring features to the full width
of line-out). After visually selecting starting points, $r_{min}$ and
$r_{max}$ are varied over $\pm 0.05$ mm with spacing of 0.025 mm. The
combination of these parameters produces a grid of possible undisturbed
profiles (J0). For each inferred J0, a path-integrated field profile is
reconstructed and a subset of solutions is selected based on the criteria that
$\int\\!\\!B_{\theta}\textrm{d}z\rightarrow 0$ for $r>r_{max}$. An example
result of analysis approach is shown in Figure 1(b). The mean normalized
fluence (J$/$J0) and mean reconstructed path integrated magnetic field are
plotted together. An observable that will be compared to the simulations is
the magnetic flux, calculated by integrating in the radial direction.

A number of approaches are used to characterize the uncertainty of the field
reconstruction: taking line-outs at different angles, using a larger range of
values for $r_{max}$, analyzing of successive layers of RCF (the relative
proton time-of-flight differences are small compared to the interaction time
scale), or artificially suppressing RCF signal to approximate uncertainties in
the RCF sensitivity. The relative influence of the RCF signal is low because
the reconstruction is calculated using the normalized fluence (J/J0). Overall,
uncertainty in inferring J0 and analysis of successive RCF layers leads to an
error of $\sim 25\%$ in the path-integrated field strength and magnetic flux.
In addition, the accuracy of the measurement is potentially impacted by
blurring due to small-angle proton scattering, especially in the higher Z
targets, and by enhanced proton stopping in laser-heated regions of the
targets.

###  II.2 Extended-Magnetohydrodynamics simulations

Experimental results were compared to extended-MHD simulations performed using
the Gorgon code Chittenden  _et al._ (2004); Ciardi  _et al._ (2007); Walsh
_et al._ (2017) to help validate modeling of magnetic field generation. The
evolution of magnetic field in the code is Walsh  _et al._ (2020):

$\displaystyle\begin{split}\frac{\partial\underline{B}}{\partial
t}=&-\nabla\times\frac{\alpha_{\parallel}}{\mu_{0}e^{2}n_{e}^{2}}\nabla\times\underline{B}+\nabla\times(\underline{v}_{B}\times\underline{B})\\\
&+\nabla\times\Bigg{(}\frac{\nabla
P_{e}}{en_{e}}-\frac{\beta_{\parallel}\nabla T_{e}}{e}\Bigg{)}\end{split}$ (1)

Where the first term on the right represents resistive diffusion with
coefficient $\alpha_{\parallel}$ and the second term is advection of the
magnetic field at velocity $\underline{v}_{B}$:

$\underline{v}_{B}=\underline{v}-\gamma_{\bot}\nabla
T_{e}-\gamma_{\wedge}(\underline{\hat{b}}\times\nabla T_{e})$ (2)

i.e. the magnetic field is advected by bulk plasma motion, Nernst and cross-
gradient-Nernst advection. $\gamma_{\bot}$ and $\gamma_{\wedge}$ are magnetic
transport coefficients with a similar form to the associated thermal
conductivities Walsh  _et al._ (2021). $\underline{\hat{b}}$ is the magnetic
field unit vector.

The final two terms in equation 1 are the only sources of magnetic flux in the
simulations. $\nabla\times(\nabla P_{e}/en_{e})$ is the Biermann battery term
and is the dominant source of magnetic flux in the simulations.
$\nabla\times\beta_{\parallel}\nabla T_{e}/e$ represents magnetic flux
generated by ionization gradients in the plasma Sadler  _et al._ (2020).
Previous Gorgon simulations of foils did not include this term Campbell  _et
al._ (2020), although it is found here to be of only secondary importance.

Magnetic field generation in a laser absorption region has long been
anticipated to be suppressed by kinetic processes. This is supported by both
experiments Igumenshchev  _et al._ (2014); Gao  _et al._ (2015) and Vlasov-
Fokker-Planck (VFP) simulations Sherlock and Bissell (2020); Hill and Kingham
(2018). Recent VFP simulations of laser absorption regions proposed an
empirical fit to the VFP data Sherlock and Bissell (2020):

$\frac{\partial B}{\partial t}\approx
0.083\bigg{(}\frac{l_{T}}{\lambda_{ei}}\bigg{)}^{0.453}\bigg{(}\frac{\partial
B}{\partial t}\bigg{)}_{\textrm{classical}}$ (3)

Where $l_{T}$ is the temperature length-scale and $\lambda_{ei}$ is the mean
free path of a thermal electron. Equation 3 was proposed by computational work
and yet to be compared with experimental results; this is one of the primary
goals of this paper. Equation 3 has been implemented into Gorgon as an option,
and is referred throughout this paper as ‘Biermann suppression’. The ratio
$(l_{T}/\lambda_{ei})$ is calculated at every point across the simulation
domain, and the suppression factor is limited to maximum value of 1 (where
$l_{T}/\lambda_{ei}>243$) so that the generation approaches the classical
rate. Of course, there are limitations to such an approach. This suppression
behaviour was observed using a prescribed laser intensity and varying the
transverse length-scale of the laser non-uniformity. To obtain a complete
picture, the VFP simulations would need to vary all initial conditions, which
is not practical. Equation 3 also assumes that the plasma is in a kinetic
steady state. In practice it takes time for the electron distribution function
to be perturbed away from Maxwellian.

Suppression of the Nernst term by kinetic processes has also been proposed
Sherlock and Bissell (2020) in a simple form similar to equation 3. This has
also been included as an option in the Gorgon simulations, but does not
qualitatively change the results shown here. For simplicity, Nernst
suppression has been omitted from the presented work. In this way the
connection between the heat-flow and magnetic transport can be maintained
Walsh  _et al._ (2020); Sadler  _et al._ (2021). The authors suppose that if
the Nernst term is being suppressed using a simple mean-free path argument,
then the heat-flow should also be treated similarly.

The heat-flow in Gorgon is fully anisotropic and includes the Righi-Leduc
term. The transport coefficients have been updated Sadler  _et al._ (2021) and
now exhibit physical behaviour at low magnetizations, unlike the Epperlein &
Haines coefficients Epperlein and Haines (1986). For the configuration
simulated here, this mainly lowers the importance of the cross-gradient-Nernst
term ($\gamma_{\wedge}$ in equation 2).

Simulations in this paper are exclusively two-dimensional, invoking
cylindrical symmetry ($r$-$z$). The laser propagates along $z$ with an
assumption that the laser is symmetric in $\theta$. Laser propagation uses a
simple ray-trace scheme with inverse-bremsstrahlung heating of the electron
population. Gorgon uses the Frankfurt equation of state (FEoS) with a Thomas-
Fermi ionization model, and implements multi-group non-diffusive radiation
transport using a P1/3 automatic flux-limiting method. For CH, 54 radiation
energy groups are used, while 300 groups are used for copper.

Figure 2: Simulation results with a CH target and the higher laser intensity
at t = 1.2 ns. (a-b) show the magnetic fields without and with Biermann
battery suppression included, (c) the instantaneous classical Biermann battery
generation rate, and (d) the Biermann suppression factor (lower numbers
signify more suppression).

Figure 2 demonstrates the impact of Biermann battery suppression in the Gorgon
simulations. All of the images are at 1.2 ns for CH using the higher laser
intensity Gao  _et al._ (2015). The first two images show the magnetic field
structure without and with Biermann battery suppression included in the code.
The third image then shows the classical Biermann battery generation rate at
this instant in time. The final image shows the suppression factor (equation
3). The kinetic suppression is particularly important deep into the corona,
where the plasma is hot and low in density (resulting in long electron mean-
free paths).

For the case without Biermann suppression included, an instability that
generates magnetic field is observed in the corona, giving oscillating
polarities near the laser axis. These oscillations do not contribute to the
overall quantification of magnetic flux, as they cancel out when summed. The
instability is likely due to the interplay between Biermann battery and
anisotropic thermal conduction, which is called the field-generating thermal
instability Tidman and Shanny (1974). It could also be from the magneto-
thermal instability Bissell  _et al._ (2010), which results from the interplay
between Nernst and Righi-Leduc. Nonetheless, kinetic suppression of the
Biermann battery process is found to stabilize the instability, which has been
noted in full VFP simulations Sherlock and Bissell (2020).

Subsequent sections will compare these simulation results to experiments using
different laser intensities and foil materials. The comparison suggests that
kinetic suppression of Biermann battery generation is indeed occurring in
experiments.

##  III Results and Discussion

Experimental and simulation results for CH foil targets are compared in Figure
3. In the left column, proton deflectometry images show the evolution of
magnetic field structures using the higher laser intensity (2I0, overlapped
pulses). The primary features are concentric light and dark rings of proton
fluence due to deflections from an azimuthal magnetic field. As the plasma
evolves, there is evidence in the proton deflectometry images of instability
formation with an accompanying electromagnetic field. Such instability
features are 3D in nature, and cannot be captured by the 2D simulations. For
this comparison, line-out locations — shown with dashed lines — were chosen to
avoid strong modulations caused by the instability (rather than along the
minor axis as described in the Methods section above). The proton energies for
these images are 20 MeV for t0+0.4 ns and t0+0.7 ns, and 9 MeV for t0+1.2 ns,
where $t_{0}$ denotes the beginning of UV laser irradiation.

Corresponding reconstructed magnetic field profiles are plotted in the top
panel of the right column of Figure 3, with shaded regions showing 25$\%$
uncertainty range. For clarity, the field profiles for each probing time are
offset vertically. Over the 1.2 ns evolution, the field grows to peak path-
integrated strengths near 100 MG$\mu$m. In qualitative agreement with Ref. Gao
_et al._ , 2015, the reconstructed profiles indicate that the outer edge of
the field expands near the sound speed, 0.8–1$\times 10^{6}$ m/s, while the
largest fields are more closely bound to the focal region and expand more
slowly, 0.3–0.5$\times 10^{6}$ m/s.

Figure 3: Comparison of experimental and simulation results for CH foils. The
left column shows proton deflectometry images of fields driven by the higher,
overlapped laser intensity (2I0) taken at 0.4 ns, 0.7 ns and 1.2 ns. Line-out
locations are indicated by dashed lines. Reconstructed magnetic field profiles
are plotted in the top right panel (offset vertically for clarity). In the
bottom right panel, the magnetic flux predictions from simulations both
without and with Biermann suppression for each laser intensity are compared to
experimental measurements. Upper and lower bounds on the simulation results
are produced by tuning the laser energy to approximate the influence of energy
coupling efficiency.

The bottom panel of Figure 3 compares the evolution of the azimuthal magnetic
flux from the experiment and extended-MHD simulations for both laser
intensities. As in the experiment, the higher intensity simulations (2I0) also
used a reduced focal spot radius. Experimental data points are plotted along
with shaded regions which show the simulated flux either without or with
Biermann suppression included (note: on this y-axis scale, experimental error
bars can fall within the plot markers). The width of the shaded regions
illustrate the influence of tuning the laser energy to approximate uncertainty
in coupling efficiency for the inverse-bremsstrahlung heating, with upper and
lower bounds corresponding to $\sim$90$\%$ and $\sim$70$\%$ coupling. For both
laser intensities, simulations without Biermann suppression greatly
overestimate the magnetic flux ($>5\times$).

Agreement is significantly improved by including Biermann suppression,
indicating that this effect is likely influencing the field dynamics. In the
simulations, the suppression results in a 3–4$\times$ reduction in the
predicted magnetic flux. For the lower intensity, the inclusion of Biermann
suppression also weakens the influence of laser coupling efficiency in the
simulation, reducing the percent spread in flux from 15$\%$ to 6$\%$. The mean
difference predicted flux for the two intensities is also reduced by the
Biermann suppression, from 25$\%$ to 20$\%$. This suggests that achieving
higher temperatures with more laser energy is partially balanced by an
increase in non-local effects. Comparing data at t0+0.7 ns for 2I0 and t0+0.75
ns for I0, the experimental difference is 23$\%$, though measurement
uncertainty leads to 100$\%$ error in quantifying the relative change.

Figure 4 summarizes the results for Cu foil targets with the lower intensity.
The left column shows experimental proton images taken at times ranging from
t0+0.25 ns to t0+1.0 ns. Here, line-outs were taken along the minor axis at
each probing time. The corresponding proton energies are 33.6 MeV for t0+0.25
ns, 22.6 MeV for t0+0.5 ns, and 32.8 MeV for t0+0.75 ns and t0+1.0 ns. The
target for t0+0.25 ns was a 25 $\mu$m-thick Cu foil, while the later
measurements were made using the Cu+Al layered target in order to improve the
imaging resolution by reducing the effect of scattering. Previous side-by-side
comparisons of image features from pure Cu and layered Cu+Al presented in Ref.
Campbell  _et al._ , 2020 indicate that the field dynamics are dominated by
the Cu layer.

Figure 4: Comparison of experimental and simulation results for Cu foils. The
left column shows proton images at 0.25 ns, 0.5 ns, 0.75 ns and 1.0 ns. Line-
out locations are indicated by dashed lines. The target for t0+0.25 ns was a
25 $\mu$m-thick Cu foil, and the other probing times use a Cu+Al layered
target. Reconstructed magnetic field profiles are plotted in the top right
panel. In the bottom right panel, the magnetic flux predictions from
simulations both without and with Biermann suppression are compared to
experimental measurements.

Reconstructed path-integrated magnetic field profiles for each probing time
are plotted together in the top right panel of Figure 4. The field profiles
show the expansion of the field features and the emergence of double-peaked
structure at t0+0.75 ns and t0+1.0 ns. This produces the pattern of two
concentric rings of proton accumulation observed in the images. As discussed
in Ref. Campbell  _et al._ , 2020, the two field features are evidence of
Biermann battery generation around radiation-driven double ablation front
(DAF) structures.

The bottom right panel compares experimental measurements of magnetic flux
evolution with the simulation predictions both without and with the Biermann
suppression. As with CH targets, the simulations without suppression
overestimate the flux, though the discrepancy is not as large. Again,
including suppression reduces the spread in the simulation predictions due to
tuning the coupling efficiency, from $7\%$ to 2$\%$. However, for Cu targets
the Biermann suppression model reduces the predicted flux below experimental
observations.

The simulation and experimental results suggest that non-local suppression
effects are more significant for low-Z targets. Without Biermann suppression,
simulations with Cu targets predict lower magnetic flux than the CH results.
This is predominantly due to additional radiative losses at higher Z reducing
temperature gradients. In contrast, experimental measurements of the magnetic
flux at t0+0.75 ns increases from $1\times 10^{4}$ MG$\mu$m2 for CH to
$2\times 10^{4}$ MG$\mu$m2 for Cu. The same qualitative trend is also seen in
the simulations including Biermann suppression. The copper targets are less
kinetic, due to both lower temperature gradients from radiative losses and
shorter mean-free paths for higher Z plasmas.

The influence of target material on the magnetic field structure and flux is
illustrated in more detail in Figure 5. The left column shows experimental
proton deflectometry images for CH, Al, Cu+Al, and Au+Al targets at t0+0.75
ns. The proton energies are 37.3 MeV for CH and Al, 32.8 MeV for Cu+Al, and
30.7 MeV for Au+Al. The reconstructed magnetic field profiles plotted in the
top panel of the right column show the Z-dependence of field profiles. By
t0+0.75 ns, the fields from a CH target have expanded the furthest, while the
Au+Al has the narrowest features — indicative of slower plasma evolution due
heavier, higher Z ions. The mid-Z targets both show evidence of magnetic
signatures of DAF structures (visible both in the deflectometry and the
reconstructed magnetic field profiles) Campbell  _et al._ (2020).

The measured magnetic flux is plotted as a function of Z${}_{\textrm{eff}}$ in
the lower right panel of Figure 5. While CH and Al are expected to be fully
ionized, Cu+Al is assumed to be H-like (Z${}_{\textrm{eff}}=28$)Campbell  _et
al._ (2020), and Z${}_{\textrm{eff}}=50$ is used for Au+Al, consistent with
Refs. Sherlock and Bissell, 2020 and Jones  _et al._ , 2017. The measured flux
increases with Z${}_{\textrm{eff}}$ moving from low-Z to mid-Z targets before
slightly decreasing or plateauing for the highest Z. The observed decrease at
high-Z could be a result of slower plasma evolution (also evident from the
narrow radius discussed above), or due to more of the coupled laser energy
driving ionization and radiation emission, reducing the peak electron
temperature.

An estimate of a Z${}_{\textrm{eff}}$ scaling taking into account Biermann
suppression is plotted with a dashed line. Based on equation 3 assuming a
fixed temperature profile, shorter mean-free-paths lead to less suppression of
classical Biermann generation rates ($\lambda_{ei}\propto 1/Z$). While the
scaling appears provide a reasonable description of the data, the assumption
of fixed temperature gradients is overly simple considering experimental
evidence of temperature gradient changes across the different materials (DAF
structures in mid-Z, slower expansion in high-Z). In addition, the
implementation of equation 3 in Gorgon shown in Figures 3 and 4 overestimates
the flux for CH, while underestimating for Cu.

Overall, this suggests additional physics is contributing to the energy
transport, and field generation is not accurately modeled by Gorgon, or
captured by the non-local Biermann suppression approximated by equation 3. In
particular, the details of the radiation-hydrodynamics and equation-of-state
likely influence plasma dynamics. Additionally, deviations from a Maxwellian
distribution driven by inverse-bremsstrahlung heating can impact energy
transport Ridgers  _et al._ (2008b). Exploration of such effects is beyond the
scope of this work as additional experimental work is needed to carefully
constrain the plasma conditions, especially the temperature and density
profiles. Still, from the magnetic field measurements presented here, it is
evident that fields are suppressed below classical predictions and that the
suppression effect is stronger for lower-Z plasmas.

Figure 5: Comparison of experimental results for CH, Al, Cu+Al, and Au+Al
targets at t0+0.75 ns. The left column shows proton images, and line-out
locations are indicated by dashed lines. Reconstructed magnetic field profiles
are plotted in the top right panel. Magnetic flux measurements for each
material are plotted as a function of Z${}_{\textrm{eff}}$ in the bottom right
panel. Note, for Cu the value of Z${}_{\textrm{eff}}$ = 28, and
Z${}_{\textrm{eff}}$ is set to 50 for Au. The dashed line shows a scaling for
flux generation as a function of Z${}_{\textrm{eff}}$ based on equation 3.

##  IV Conclusion

Quantitative measurements of magnetic flux enable detailed comparisons between
experiments and extended-MHD simulations, demonstrating the need to account
for suppression of Biermann battery generation due to non-local effects. Even
with the Biermann suppression, the simulations with CH targets still predict
larger magnetic flux than observed experimentally. However for Cu, while some
suppression is necessary, the implementation of equation 3 decreases the flux
below experimental observations. Nevertheless, experimental measurements of
magnetic flux as a function of Z${}_{\textrm{eff}}$ shows reasonable agreement
with the mean-free-path scaling predicted by Ref. Sherlock and Bissell, 2020.
The effects of radiation-hydrodynamics and the equation-of-state likely
influence the details of simulations, but are beyond the scope of the work. In
future experiments, additional diagnostics, such as Thomson scattering and
interferometry, can help constrain plasma parameters to further validate and
improve extended-MHD models. Together with the magnetic field analysis
presented in this work, measurements of the temperature and density profiles
can elucidate the dynamic interplay between energy transport and field
generation in HED plasmas.

###### Acknowledgements.

This material is based upon work supported by the Department of Energy,
National Nuclear Security Administration under Award Numbers DE-NA0003606, DE-
NA0003764 and DE-AC52-07NA27344. PTC is supported by the U.S. Department of
Energy Fusion Energy Sciences Postdoctoral Research Program administered by
the Oak Ridge Institute for Science and Education (ORISE) for the DOE. ORISE
is managed by Oak Ridge Associated Universities (ORAU) under DOE contract
number DE-SC0014664. All opinions expressed in this paper are the author’s and
do not necessarily reflect the policies and views of DOE, ORAU, or ORISE. BKR
acknowledges support from NSF Award Number 1751462. This document was prepared
as an account of work sponsored by an agency of the United States government.
Neither the United States government nor Lawrence Livermore National Security,
LLC, nor any of their employees makes any warranty, expressed or implied, or
assumes any legal liability or responsibility for the accuracy, completeness,
or usefulness of any information, apparatus, product, or process disclosed, or
represents that its use would not infringe privately owned rights. Reference
herein to any specific commercial product, process, or service by trade name,
trademark, manufacturer, or otherwise does not necessarily constitute or imply
its endorsement, recommendation, or favoring by the United States government
or Lawrence Livermore National Security, LLC. The views and opinions of
authors expressed herein do not necessarily state or reflect those of the
United States government or Lawrence Livermore National Security, LLC, and
shall not be used for advertising or product endorsement purposes.

## Data Availability

The data that support the findings of this study are available from the
corresponding author upon reasonable request.

## References

  * Drake (2018) R. P. Drake,  _High Energy Density Physics: Fundamentals, Inertial Fusion and Experimental Astrophysics_, 2nd ed. (Springer International Publishing, Berlin, 2018).
  * Remington  _et al._ (2006) B. A. Remington, R. P. Drake, and D. D. Ryutov, Rev. Mod. Phys. 78, 755 (2006).
  * Haines (1986a) M. G. Haines, Can. J. Phys. 64, 912 (1986a).
  * Biermann and Schlüter (1951) L. Biermann and A. Schlüter, Phys. Rev. 82, 863 (1951).
  * Stamper  _et al._ (1971) J. A. Stamper, K. Papadopoulos, R. N. Sudan, S. O. Dean, E. A. McLean, and J. M. Dawson, Phys. Rev. Lett. 26, 1012 (1971).
  * Stamper and Ripin (1975) J. A. Stamper and B. H. Ripin, Phys. Rev. Lett. 34, 138 (1975).
  * Braginskii (1965) S. Braginskii, Reviews of plasma physics 1 (1965).
  * Evans (1986) R. G. Evans, Plasma Phys. and Controlled Fusion 28, 1021 (1986).
  * Manuel  _et al._ (2012) M. J.-E. Manuel, C. K. Li, F. H. Séguin, J. Frenje, D. T. Casey, R. D. Petrasso, S. X. Hu, R. Betti, J. D. Hager, D. D. Meyerhofer, and V. A. Smalyuk, Phys. Rev. Lett. 108, 255006 (2012).
  * Gao  _et al._ (2012) L. Gao, P. M. Nilson, I. V. Igumenschev, S. X. Hu, J. R. Davies, C. Stoeckl, M. G. Haines, D. H. Froula, R. Betti, and D. D. Meyerhofer, Phys. Rev. Lett. 109, 115001 (2012).
  * Gao  _et al._ (2013) L. Gao, P. M. Nilson, I. V. Igumenschev, G. Fiksel, R. Yan, J. R. Davies, D. Martinez, V. Smalyuk, M. G. Haines, E. G. Blackman, D. H. Froula, R. Betti, and D. D. Meyerhofer, Phys. Rev. Lett. 110, 185003 (2013).
  * Hill and Kingham (2018) D. W. Hill and R. J. Kingham, Phys. Rev. E 98, 021201 (2018).
  * Nilson  _et al._ (2006) P. M. Nilson, L. Willingale, M. C. Kaluza, C. Kamperidis, S. Minardi, M. S. Wei, P. Fernandes, M. Notley, S. Bandyopadhyay, M. Sherlock, _et al._ , Phys. Rev. Lett. 97, 255001 (2006).
  * Li  _et al._ (2007) C. K. Li, F. H. Séguin, J. A. Frenje, J. R. Rygg, R. D. Petrasso, R. P. J. Town, O. L. Landen, J. P. Knauer, and V. A. Smalyuk, Phys. Rev. Lett. 99, 055001 (2007).
  * Willingale  _et al._ (2010a) L. Willingale, P. M. Nilson, M. C. Kaluza, A. E. Dangor, R. G. Evans, P. Fernandes, M. G. Haines, C. Kamperidis, R. J. Kingham, C. P. Ridgers, _et al._ , Phys. Plasmas 17, 043104 (2010a).
  * Zhong  _et al._ (2010) J. Zhong, Y. Li, X. Wang, J. Wang, Q. Dong, C. Xiao, S. Wang, X. Liu, L. Zhang, L. An, _et al._ , Nat. Phys. 6, 984 (2010).
  * Fox  _et al._ (2011) W. Fox, A. Bhattacharjee, and K. Germaschewski, Phys. Rev. Lett. 106, 215003 (2011).
  * Dong  _et al._ (2012) Q.-L. Dong, S.-J. Wang, Q.-M. Lu, C. Huang, D.-W. Yuan, X. Liu, X.-X. Lin, Y.-T. Li, H.-G. Wei, J.-Y. Zhong, _et al._ , Phys. Rev. Lett. 108, 215001 (2012).
  * Walsh  _et al._ (2020) C. A. Walsh, J. P. Chittenden, D. W. Hill, and C. Ridgers, Phys. Plasmas 27, 022103 (2020).
  * Farmer  _et al._ (2017) W. A. Farmer, J. M. Koning, D. J. Strozzi, D. E. Hinkel, L. F. Berzak Hopkins, O. S. Jones, and M. D. Rosen, Phys. Plasmas 24, 052703 (2017).
  * Walsh  _et al._ (2017) C. A. Walsh, J. P. Chittenden, K. McGlinchey, N. P. L. Niasse, and B. D. Appelbe, Phys. Rev. Lett. 118, 155001 (2017).
  * Davies  _et al._ (2015) J. R. Davies, R. Betti, P.-Y. Chang, and G. Fiksel, Physics of Plasmas 22, 112703 (2015).
  * McBride and Slutz (2015) R. D. McBride and S. A. Slutz, Physics of Plasmas 22, 052708 (2015).
  * Joglekar  _et al._ (2016) A. S. Joglekar, C. P. Ridgers, R. J. Kingham, and A. G. R. Thomas, Phys. Rev. E 93, 043206 (2016).
  * Walsh  _et al._ (2019) C. A. Walsh, K. McGlinchey, J. K. Tong, B. D. Appelbe, A. Crilly, M. F. Zhang, and J. P. Chittenden, Physics of Plasmas 26, 022701 (2019).
  * Joglekar  _et al._ (2014) A. S. Joglekar, A. G. R. Thomas, W. Fox, and A. Bhattacharjee, Phys. Rev. Lett. 112, 105004 (2014).
  * Tubman  _et al._ (2021) E. R. Tubman, A. S. Joglekar, A. F. A. Bott, M. Borghesi, B. Coleman, G. Cooper, C. N. Danson, P. Durey, J. M. Foster, P. Graham, _et al._ , Nat. Commun. 12, 334 (2021).
  * Borghesi  _et al._ (2001) M. Borghesi, A. Schiavi, D. H. Campbell, M. G. Haines, O. Willi, A. J. MacKinnon, L. A. Gizzi, M. Galimberti, R. J. Clarke, and H. Ruhl, Plasma Phys. and Controlled Fusion 43, A267 (2001).
  * Li  _et al._ (2006) C. K. Li, F. H. Séguin, J. A. Frenje, J. R. Rygg, R. D. Petrasso, R. P. J. Town, P. A. Amendt, S. P. Hatchett, O. L. Landen, A. J. Mackinnon, P. K. Patel, V. A. Smalyuk, T. C. Sangster, and J. P. Knauer, Phys. Rev. Lett. 97, 135003 (2006).
  * Willingale  _et al._ (2010b) L. Willingale, A. G. R. Thomas, P. M. Nilson, M. C. Kaluza, S. Bandyopadhyay, A. E. Dangor, R. G. Evans, P. Fernandes, M. G. Haines, C. Kamperidis, _et al._ , Phys. Rev. Lett. 105, 095001 (2010b).
  * Lancia  _et al._ (2014) L. Lancia, B. Albertazzi, C. Boniface, A. Grisollet, R. Riquier, F. Chaland, K.-C. Le Thanh, P. Mellor, P. Antici, S. Buffechoux, _et al._ , Phys. Rev. Lett. 113, 235001 (2014).
  * Gao  _et al._ (2015) L. Gao, P. M. Nilson, I. V. Igumenshchev, M. G. Haines, D. H. Froula, R. Betti, and D. D. Meyerhofer, Phys. Rev. Lett. 114, 215003 (2015).
  * Nishiguchi  _et al._ (1984) A. Nishiguchi, T. Yabe, M. G. Haines, M. Psimopoulos, and H. Takewaki, Phys. Rev. Lett. 53, 262 (1984).
  * Kho and Haines (1985) T. H. Kho and M. G. Haines, Phys. Rev. Lett. 55, 825 (1985).
  * Ridgers  _et al._ (2008a) C. P. Ridgers, R. J. Kingham, and A. G. R. Thomas, Phys. Rev. Lett. 100, 075003 (2008a).
  * Haines (1986b) M. G. Haines, Plasma Physics and Controlled Fusion 28, 1705 (1986b).
  * Campbell  _et al._ (2020) P. T. Campbell, C. A. Walsh, B. K. Russell, J. P. Chittenden, A. Crilly, G. Fiksel, P. M. Nilson, A. G. R. Thomas, K. Krushelnick, and L. Willingale, Phys. Rev. Lett. 125, 145001 (2020).
  * Sherlock and Bissell (2020) M. Sherlock and J. J. Bissell, Phys. Rev. Lett. 124, 055001 (2020).
  * Maksimchuk  _et al._ (2000) A. Maksimchuk, S. Gu, K. Flippo, D. Umstadter, and V. Y. Bychenkov, Phys. Rev. Lett. 84, 4108 (2000).
  * Snavely  _et al._ (2000) R. A. Snavely, M. H. Key, S. P. Hatchett, T. E. Cowan, M. Roth, T. W. Phillips, M. A. Stoyer, E. A. Henry, T. C. Sangster, M. S. Singh, S. C. Wilks, A. MacKinnon, A. Offenberger, D. M. Pennington, K. Yasuike, A. B. Langdon, B. F. Lasinski, J. Johnson, M. D. Perry, and E. M. Campbell, Phys. Rev. Lett. 85, 2945 (2000).
  * Clark  _et al._ (2000) E. L. Clark, K. Krushelnick, J. R. Davies, M. Zepf, M. Tatarakis, F. N. Beg, A. Machacek, P. A. Norreys, M. I. K. Santala, I. Watts, and A. E. Dangor, Phys. Rev. Lett. 84, 670 (2000).
  * Hatchett  _et al._ (2000) S. P. Hatchett, C. G. Brown, T. E. Cowan, E. A. Henry, J. S. Johnson, M. H. Key, J. A. Koch, A. B. Langdon, B. F. Lasinski, R. W. Lee, _et al._ , Phys. Plasmas 7 (2000).
  * Borghesi  _et al._ (2004) M. Borghesi, A. J. Mackinnon, D. H. Campbell, D. G. Hicks, S. Kar, P. K. Patel, D. Price, L. Romagnani, A. Schiavi, and O. Willi, Phys. Rev. Lett. 92, 055003 (2004).
  * Kugland  _et al._ (2012) N. L. Kugland, D. D. Ryutov, C. Plechaty, J. S. Ross, and H.-S. Park, Rev. Sci. Instrum. 83, 101301 (2012).
  * Chittenden  _et al._ (2004) J. P. Chittenden, S. V. Lebedev, C. A. Jennings, S. N. Bland, and A. Ciardi, Plasma Phys. and Controlled Fusion 46, B457 (2004).
  * Ciardi  _et al._ (2007) A. Ciardi, S. V. Lebedev, A. Frank, E. G. Blackman, J. P. Chittenden, C. J. Jennings, D. J. Ampleford, S. N. Bland, S. C. Bott, J. Rapley, _et al._ , Phys. Plasmas 14, 056501 (2007).
  * Walsh  _et al._ (2021) C. A. Walsh, J. D. Sadler, and J. R. Davies, Nuclear Fusion (submitted 2021).
  * Sadler  _et al._ (2020) J. D. Sadler, H. Li, and K. A. Flippo, Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences 378, 20200045 (2020).
  * Igumenshchev  _et al._ (2014) I. V. Igumenshchev, A. B. Zylstra, C. K. Li, P. M. Nilson, V. N. Goncharov, and R. D. Petrasso, Physics of Plasmas 21, 062707 (2014).
  * Sadler  _et al._ (2021) J. D. Sadler, C. A. Walsh, and H. Li, Phys. Rev. Lett. 126, 075001 (2021).
  * Epperlein and Haines (1986) E. M. Epperlein and M. G. Haines, The Physics of Fluids 29, 1029 (1986).
  * Tidman and Shanny (1974) D. A. Tidman and R. A. Shanny, The Physics of Fluids 17, 1207 (1974).
  * Bissell  _et al._ (2010) J. J. Bissell, C. P. Ridgers, and R. J. Kingham, Phys. Rev. Lett. 105, 175001 (2010).
  * Jones  _et al._ (2017) O. S. Jones, L. J. Suter, H. A. Scott, M. A. Barrios, W. A. Farmer, S. B. Hansen, D. A. Liedahl, C. W. Mauche, A. S. Moore, M. D. Rosen, J. D. Salmonson, D. J. Strozzi, C. A. Thomas, and D. P. Turnbull, Physics of Plasmas 24, 056312 (2017).
  * Ridgers  _et al._ (2008b) C. P. Ridgers, A. G. R. Thomas, R. J. Kingham, and A. P. L. Robinson, Physics of Plasmas 15, 092311 (2008b).