# The Magnetic Gradient Scale Length Explains Why Certain Plasmas Require
Close External Magnetic Coils

John Kappel  Matt Landreman<EMAIL_ADDRESS>Institute for Research in
Electronics and Applied Physics, University of Maryland, College Park, MD
20742, United States of America  Dhairya Malhotra  Flatiron Instituite, New
York, NY, 10010, United States of America

###### Abstract

The separation between the last closed flux surface of a plasma and the
external coils that magnetically confine it is a limiting factor in the
construction of fusion-capable plasma devices. This plasma-coil separation
must be large enough so that components such as a breeding blanket and neutron
shielding can fit between the plasma and the coils. Plasma-coil separation
affects reactor size, engineering complexity, and particle loss due to field
ripple. For some plasmas it can be difficult to produce the desired flux
surface shaping with distant coils, and for other plasmas it is infeasible
altogether. Here, we seek to understand the underlying physics that limits
plasma-coil separation and explain why some configurations require close
external coils. In this paper, we explore the hypothesis that the limiting
plasma-coil separation is set by the shortest scale length of the magnetic
field as expressed by the $\nabla\mathbf{B}$ tensor. We tested this hypothesis
on a database of $>$ 40 stellarator and tokamak configurations. Within this
database, the coil-to-plasma distance compared to the minor radius varies by
over an order of magnitude. The magnetic scale length is well correlated to
the coil-to-plasma distance of actual coil designs generated using the REGCOIL
method [Landreman, Nucl. Fusion 57, 046003 (2017)]. Additionally, this
correlation reveals a general trend that larger plasma-coil separation is
possible with a small number of field periods.

††preprint: AIP/123-QED

##  I Introduction

Magnetic fields are used to confine plasmas at the necessary temperature and
density for fusion. The external coils that generate part of these magnetic
fields are subject to engineering constraints in order for the device to be
feasible to build. One engineering constraint relates to the separation
between the last closed flux surface of the plasma and the edge of the nearest
external magnetic coil. In reactors, neutron shielding is needed between the
plasma and the coils. This shielding protects the magnets from damage caused
by the high energy neutrons released from fusion reactions. In addition, due
to the limited tritium available worldwide, a reactor requires a breeding
blanket to produce more tritium. These components require a separation of
roughly 1.5 m between the last closed flux surface (LCFS) and the nearest
coil, so that the there is room for both the neutron shielding and the
breeding blanket.[1] In this paper, we shall refer to this distance as plasma-
coil separation.

A larger plasma-coil separation also desirable since insufficient space can
lead to increased costs during design and construction. [2] Even in
experiments without neutron shielding or a breeding blanket, miscellaneous
components are installed between the plasma and the magnets with little
separation in between each component. When the vacuum chamber’s pressure is
lowered, when the magnets are cooled to superconducting temperature, or when
the magnets are energized, these components can shift and potentially collide.
Additional plasma-coil clearance can allow for components to be spaced further
apart, simplifying the engineering design and assembly.

(a)

(b)

(c)

Figure 1: Using REGCOIL, coils are calculated at constant distances of (a) 25
cm, (b) 50 cm, and (c) 65 cm away from the last closed flux surface to create
the magnetic field for the W7-X plasma (which has a minor radius of $a$ = 53
cm). The generated flux surfaces all have a fixed $B_{\mathrm{RMS}}$ error of
0.02 T. At higher offset distances, the coils have increased complexity.

Increasing the minimum plasma-coil distance has additional advantages. Low
plasma-coil separation causes ripple in the field strength, which can lower
confinement.[3] In addition, because coils must be built at a minimum
clearance from the plasma, achieving a larger plasma-coil separation can
counter-intuitively allow for smaller reactors and hence reduced costs. One
can scale down a plant design with a large plasma-coil separation until the
separation is around 1.5 m. Thus, in several previous reactor studies, coil-
plasma separation determines the minimum reactor size. [1, 4]

However, it can be difficult to optimize coils to achieve a large plasma-coil
separation, especially in stellarators. Stellarator optimization is typically
divided into two stages. In stage I, the boundary shape of the plasma is
optimized in order to achieve properties such as confinement and stability. In
stage II, the coil shapes are optimized in order to generate the target field
designed in stage I, subject to engineering constraints. Due to the complex
shape of a stellarator plasma, moving coils farther away during stage II
optimization at a fixed field accuracy results in increased coil complexity
(such as increased curvature, longer coils, and closer minimum coil-coil
distance), as shown in figure 1. This concept is further elaborated later in
this paper in figure 4. Conversely, increasing plasma-coil separation at a
fixed coil complexity results in coils that do not reproduce the target plasma
shape to a high enough level of accuracy. A partial solution to this issue may
be to perform single-stage optimization, in which both the plasma shape and
coil shapes are optimized together.[5, 6, 7, 8] However, single-stage
optimization can be computationally challenging. Single-stage optimization
also strongly benefits from a good initial condition. It remains valuable
therefore to develop an easy-to-calculate proxy for plasma-coil separation for
a given plasma shape before a detailed coil design is found.

In this paper, we explore the hypothesis that differences in plasma-coil
separation between magnetic configurations can be understood in terms of scale
lengths of the magnetic field. We demonstrate this relationship by comparing
the magnetic gradient scale length to the plasma-coil separation for a diverse
database of plasma configurations, and we show a good correlation between the
two. For the coil calculations, we use the REGCOIL method,[9, 10] which is
convenient for performing a systematic comparison of many configurations.
Because of this good correlation, we propose that the smallest magnetic
gradient scale length on the surface of the plasma can be used as a proxy for
the minimum plasma-coil separation. If a configuration has a small magnetic
gradient scale length, it will be intrinsically difficult to find distant
coils no matter what coil design code is used.

In section II, we motivate a quantitative measure of the magnetic gradient
scale length, and consider this magnetic gradient scale length in model
geometry. In section III.1, we explain the methods used to calculate the coil
configurations for the database of $\sim$ 40 plasma configurations in order to
observe the agreement between the calculated plasma-coil separation and the
magnetic gradient scale length. In section IV, we demonstrate the excellent
agreement between the two lengths, and provide a discussion of the results.
Finally, we conclude in section V.

##  II Motivation for the $L_{\nabla\mathbf{B}}$ Scale Length

Scale lengths are a common tool in plasma physics. When deriving models, we
often employ arguments involving scale lengths to determine which effects are
negligible and which effects are significant. For example, when deriving the
gyrokinetic or drift-kinetic equations, terms are ordered in the parameter
$\rho/L$, where $\rho$ is the gyroradius and $L$ is a macroscopic scale length
of the equilibrium, $L\sim n/|\nabla n|\sim T/|\nabla T|\sim B/|\nabla
B|$.[11] For any scalar $Q$, it is possible to write a gradient scale length
of the form

$\frac{Q}{\|\nabla Q\|}.$ (1)

One can attempt to design a scale length of similar form for the magnetic
field. However, the magnetic field is not a scalar but rather a vector, and so
the most useful expression for the magnetic gradient scale length is not
immediately obvious.

One quantity resembling a scale length for magnetic fields was discussed by
Nara et al. and in subsequent papers.[12, 13, 14, 15, 16] In these papers, an
equation was derived for the location of a dipole source, based on the local
field and its gradient:

$\mathbf{r}=-3(\nabla\mathbf{B})^{-1}\mathbf{B},$ (2)

where the tensor $\nabla\mathbf{B}$ can be expressed as a $3\times 3$ matrix:

$\nabla\mathbf{B}=\begin{bmatrix}\frac{\partial B_{x}}{\partial
x}&\frac{\partial B_{y}}{\partial x}&\frac{\partial B_{z}}{\partial x}\\\
\frac{\partial B_{x}}{\partial y}&\frac{\partial B_{y}}{\partial
y}&\frac{\partial B_{z}}{\partial y}\\\ \frac{\partial B_{x}}{\partial
z}&\frac{\partial B_{y}}{\partial z}&\frac{\partial B_{z}}{\partial
z}\end{bmatrix}.$ (3)

Equation (2) can be solved to localize a magnetic source, and has applications
in the field of RFID tag positioning and motion tracking.[17] While this
equation does further prove that the gradient of the magnetic field vector can
be used for source localization, this equation assumes a single dipole-like
source. Moreover, the matrix $\nabla\mathbf{B}$ is not invertible for a
straight wire, meaning (2) is a poor choice for an arrangement of coils. To
find a more suitable magnetic gradient scale length, let us consider some
desirable properties of it.

We first would like all components of the $\nabla\mathbf{B}$ matrix to
significantly contribute to the scale length. As an example without this
property, consider the possible scale length candidate

$\frac{B}{\|\nabla B\|}.$ (4)

Equation (4) can be written explicitly in terms of Cartesian components of the
magnetic field vector and its gradient matrix:

$\begin{split}\frac{B}{\|\nabla
B\|}=B^{2}\biggl{[}\biggl{(}B_{x}\frac{\partial B_{x}}{\partial
x}+B_{y}\frac{\partial B_{y}}{\partial x}+B_{z}\frac{\partial B_{z}}{\partial
x}\biggr{)}^{2}\\\ +\biggl{(}B_{x}\frac{\partial B_{x}}{\partial
y}+B_{y}\frac{\partial B_{y}}{\partial y}+B_{z}\frac{\partial B_{z}}{\partial
y}\biggr{]}^{2}\\\ +\biggl{(}B_{x}\frac{\partial B_{x}}{\partial
z}+B_{y}\frac{\partial B_{y}}{\partial z}+B_{z}\frac{\partial B_{z}}{\partial
z}\biggr{)}^{2}\biggr{)}^{-1/2}.\end{split}$ (5)

Consider a point $(x_{1},y_{1},z_{1})$ where the magnetic field is locally
oriented in a single coordinate direction, i.e.
$\mathbf{B}(x_{1},y_{1},z_{1})=\hat{\mathbf{z}}B_{z}(x_{1},y_{1},z_{1})$.
Because $B_{x}(x_{1},y_{1},z_{1},)=B_{y}(x_{1},y_{1},z_{1},)=0$, equation (5)
can be simplified to

$\frac{B}{\|\nabla B\|}=\frac{B^{2}}{B_{z}\biggl{(}\frac{\partial
B_{z}}{\partial x}^{2}+\frac{\partial B_{z}}{\partial y}^{2}+\frac{\partial
B_{z}}{\partial z}^{2}\biggr{)}^{1/2}}.$ (6)

Only three of the nine elements of the $\nabla\mathbf{B}$ matrix appear.
Consider another point $(x_{2},y_{2},z_{2})$ where the magnetic field differs
from point $(x_{1},y_{1},z_{1})$ only in the gradient, i.e.:

$\frac{\partial B_{x}}{\partial
x}\bigg{\rvert}_{(x_{1},y_{1},z_{1})}\neq\frac{\partial B_{x}}{\partial
x}\bigg{\rvert}_{(x_{2},y_{2},z_{2})}.$ (7)

Although the gradients of their magnetic vectors differ, the scale length
proposed in equation (4) would be the same at both points. In this example,
six of the nine Cartesian components of the gradient matrix would have no
impact on the scale length. This behavior makes equation (4) a poor choice of
a scale length. The same consideration rules out another plausible scale
length, the radius of curvature of the magnetic field:

$\frac{1}{\|\mathbf{b}\cdot\nabla\mathbf{b}\|},$ (8)

where $\mathbf{b}=\mathbf{B}/B$ is the unit vector in the direction of the
field.[18] A similar argument can be used to prove that this scale length is
independent of some Cartesian components of $\nabla\mathbf{B}$ when
$\mathbf{B}$ is aligned with a coordinate axis. Therefore, we are motivated to
instead base our scale length on a matrix norm of the gradient matrix
$\nabla\mathbf{B}$.

(a)

(b)

Figure 2: (a) A For an infinite straight wire, the $L_{\nabla\mathbf{B}}$
scale length calculated from the local magnetic field (for example the red
dot) is the distance to the wire. One can compute $L_{\nabla\mathbf{B}}$
without knowing the location of the wire. (b) At any point around the wire
(represented by the gray dot), we can use $L_{\nabla\mathbf{B}}$ to find the
distance to the wire by only measuring $B$ and $\nabla\mathbf{B}$ at one
point.

Our second consideration is that the norm should be invariant to rotation of
coordinates. In other words, given some unitary rotation matrix $\mathsf{U}$,
$\|\mathsf{U}\nabla\mathbf{B}\|=\|\nabla\mathbf{B}\|$. Norms that fulfill this
property are referred to as unitarily invariant norms. Two common classes of
unitarily invariant norms are Schatten[19] and Ky Fan[20] norms. For a matrix
$\mathsf{A}$ with dimensions $m\times n$, the Schatten-$p$ norm is defined as

$\|\mathsf{A}\|^{S}_{p}=\biggl{(}\sum_{i=1}^{n}\sigma_{i}^{p}\biggr{)}^{1/p},$
(9)

and the Ky Fan-$k$ norm is defined as

$\|\mathsf{A}\|^{K}_{k}=\sum_{i=1}^{k}\sigma_{i},$ (10)

where $\sigma_{i}$ is the $i$-th singular value of $\mathsf{A}$, $p\geq 1$,
and $1\leq k\leq n$. The most common unitarily invariant norm is Frobenius
norm:

$\|\mathsf{A}\|_{F}=\sqrt{\sum_{i=1}^{m}\sum_{j=1}^{n}|a_{ij}|^{2}},$ (11)

where $a_{ij}$ are the elements of $\mathsf{A}$. Note that for the gradient of
a magnetic field, the Frobenius norm is equivalent to the Schatten-2 norm in a
vacuum field. In a vacuum field, the magnetic field $\mathbf{B}$ is curl-free
and so $\mathbf{B}=\nabla\Phi$ for some scalar potential $\Phi$.
Therefore,$\nabla\mathbf{B}=\nabla\nabla\Phi$, which is manifestly symmetric.
While the Frobenius norm is not the only unitarily invariant norm, we will
adopt it in this work for simplicity.

Thus, given our two criteria for the magnetic scale length – a significant
contribution from each component of the $\nabla\mathbf{B}$ matrix and
rotational invariance – we are led to the following formula for the magnetic
gradient scale length:

$L_{\nabla\mathbf{B}}=\frac{\sqrt{2}B}{\|\nabla\mathbf{B}\|_{F}}.$ (12)

The $\sqrt{2}$ factor is explained below. An alternative candidate scale
length is

$L_{\|\sigma\|}=\frac{\sqrt{2}B}{(\Sigma_{i}\sigma_{i}^{2})^{1/2}},$ (13)

where $\sigma_{i}$ are the 3 singular values of the matrix $\nabla\mathbf{B}$.
However, in a current-free field, $\nabla\mathbf{B}$ is symmetric, so
equations 12 and 13 are equivalent. It is worth noting that equation (12) has
proven useful for optimizing near axis configurations. [21, 22, 23] In
Appendix B, we compare equation (12) to alternative possible measures of a
magnetic gradient scale length.

To develop a geometric intuition of $L_{\nabla\mathbf{B}},$ let us look at the
example of a magnetic field generated by an infinite straight wire. The
magnetic field in cylindrical coordinates ($R,\phi,z$) is

$\mathbf{B}(R)=\frac{\mu_{0}I}{2\pi R}\hat{\pmb{\phi}},$ (14)

where $I$ is the current of the wire, and the hat symbols indicate unit
vectors. Because of its simplicity, the model geometry of straight wire is a
good initial test of the effectiveness of the $L_{\nabla\mathbf{B}}$ scale
length. Let us determine the scale length analytically:

$\displaystyle\nabla\mathbf{B}=-\frac{\mu_{0}I}{2\pi
R^{2}}(\hat{\pmb{\phi}}\hat{\mathbf{R}}+\hat{\mathbf{R}}\hat{\pmb{\phi}}),$
(15) $\displaystyle\|\nabla\mathbf{B}\|_{F}=\frac{\sqrt{2}\mu_{0}I}{2\pi
R^{2}},$ (16) $\displaystyle
L_{\nabla\mathbf{B}}=\sqrt{2}\frac{B}{\|\nabla\mathbf{B}\|_{F}}=R.$ (17)

Therefore, at any point in the magnetic field, $L_{\nabla\mathbf{B}}$ is equal
to the distance away from the straight wire. This relationship is shown in
figure 2. The factor of $\sqrt{2}$ was included in equation (12) to rescale
$L_{\nabla\mathbf{B}}$ so that it is equivalent with the distance to a wire in
this reduced model. Note that equation 17 does not require knowing where the
coil is located, only local measurement of $B$ and $\nabla\mathbf{B}$. This
property means that we can use these two quantities to approximate the
distance to the nearest coil. See Appendix A for a discussion of
$L_{\nabla\mathbf{B}}$ for the magnetic field of a circular wire.

##  III Methods of Numerical Validation

###  III.1 Requirements for Unique Coil Optimization

In order to verify the accuracy of $L_{\nabla\mathbf{B}}$ as a proxy for the
plasma-coil separation, we compare $L_{\nabla\mathbf{B}}$ to the plasma-coil
separation of stage II optimized stellarator configurations. However, the
coils that generate a given plasma shape are not unique. Given a magnetic
field $\mathbf{B(x)}$ which is only known within a region that is a finite
distance away from the current-carrying source, for any $\delta>0$ there
exists an infinite number of current arrangements to generate $\mathbf{B}$ up
to an error $\epsilon<\delta$.[9] To ensure a unique coil arrangement, we must
establish additional constraints, such as some measure of coil complexity.

There are multiple methods commonly used to compute the shapes of stellarator
coils. The most prominent two are filamentary coil methods and current
potential methods. In filamentary coil methods, such as the code FOCUS,[24]
each coil is parameterized as a curve, allowing for optimization in 3-D space.
However, the objective functions of these codes have multiple local minima, so
results are dependent on the initial set of parameters. Also, these methods
can have a significant number of adjustable parameters, such as scalarization
weights and the number of coils. These aspects make it difficult to compute
coil shapes in an automated and systematic way for a wide variety of plasma
configurations. Current potential methods, like REGCOIL, define some preset
winding surface on which the coils can lie. These methods optimize the current
potential on this surface, and then calculate the coil shapes using contours
of the current potential.

For this paper, we use the REGCOIL method.[9, 10] REGCOIL, compared to its
precursor NESCOIL,[25] includes a ridge regression in the current density.
Ridge regression not only reduces overfitting, but also ensures the minimum of
the loss function in not underdetermined, which guarantees that any local
minimum is a unique global minimum.

(a)

(b)

(c)

Figure 3: Poincare plots of the precise quasi-helical (QH) stellarator
configuration[26] scaled to $a$ = 1.704 m and volume averaged $B$ = 5.865 T.
The coils for these plots were computed by REGCOIL on a winding surface offset
from the plasma surface by 1.796 m. The coils are optimized to achieve a
target $B_{\mathrm{RMS}}$ of (a) 0.01 T, (b) 0.1 T, and (c) 0.5 T. The black
dotted line is the target boundary surface. At high $B_{\mathrm{RMS}}$, the
flux surfaces of the Poincare plot do not match the target flux surface.

Here we present an overview of REGCOIL’s methods.[9] One initializes an
infinitely thin winding surface surrounding the plasma. For simplicity, in
this paper we define each winding surface to be the surface that is a uniform
offset distance $L$ from the plasma surface. One can parameterize the current
potential on the winding surface as a Fourier series:

$\Phi^{\prime}(\theta^{\prime},\phi^{\prime})=\sum_{j}\Phi_{j}\sin(m_{j}\theta^{\prime}-n_{j}\phi^{\prime})+\frac{G\phi^{\prime}}{2\pi}+\frac{I\theta^{\prime}}{2\pi},$
(18)

where $\theta^{\prime}$ and $\phi^{\prime}$ are the poloidal and toroidal
coordinates on the winding surface, $G$ is the total poloidal current, $I$ is
the total toroidal current, and assuming stellarator symmetry. For modular
coils, $I=0$. From this current potential, we calculate the current density:

$\mathbf{K^{\prime}}=\mathbf{n^{\prime}}\times\mathbf{\nabla}\Phi^{\prime},$
(19)

where $\mathbf{n^{\prime}}$ is the normal vector on the winding surface. It is
then possible to find the magnetic field at any point in space via the Biot-
Savart law:

$\mathbf{B}(\mathbf{r})=\frac{\mu_{0}}{4\pi}\int
d^{2}a^{\prime}\frac{\mathbf{K^{\prime}}\times\mathbf{(r-r^{\prime})}}{|\mathbf{r-r}^{\prime}|^{3}}.$
(20)

REGCOIL’s aim is to find Fourier amplitudes of the potential $\Phi_{j}$ which
minimize the objective function $f$:

$f=\int d^{2}a\ (\mathbf{B}(\theta,\phi)\cdot\mathbf{n})^{2}+\lambda\int
d^{2}a^{\prime}\ \|\mathbf{K}(\theta^{\prime},\phi^{\prime})\|^{2},$ (21)

where $\theta$ and $\phi$ are the poloidal and toroidal coordinates on the
plasma surface, $\mathbf{n}$ is the normal vector on the plasma surface, and
$\lambda$ is a regularization parameter. The first term is the quadratic flux
over the last closed flux surface. The normal component of the magnetic field,
$\mathbf{B\cdot n}$, vanishes throughout a flux surface. Therefore, this term
is a measure of disagreement between the magnetic field generated by the
external coils and the magnetic field of the target configuration. The second
term is a measure of the current density $\mathbf{K}$ over the winding
surface, which acts as a $L_{2}$ regularizer to constrain the otherwise
underdetermined system. $\mathbf{K}$ is also a measure of coil complexity, as
we discuss further below. Minimizing the objective function will result in
coils that generate the desired shape for the LCFS with minimal coil
complexity.

Figure 4: The maximum current density $\|K\|_{\infty}$ and its dependence on
the plasma-coil separation $L$ for the winding surface of a precise QH
configuration. For each plasma-coil separation, REGCOIL computes the current
potential $\Phi$ on the winding surface that achieves a target
$B_{\mathrm{RMS}}$ = 0.01 T. With this constraint, there is a monotonic
relationship between $\|K\|_{\infty}$ and $L$, so a $\|K\|_{\infty}$ equality
constraint will result in a unique solution for $L$.

To calculate the current on the winding surface, we need to specify only two
parameters, the plasma-coil separation $L$ and the regularization parameter
$\lambda$. We choose $L$ and $\lambda$ so that the resulting coils fulfill two
constraints. Let us define the quadratic flux normalized to the surface area
of the plasma:

$B_{\mathrm{RMS}}=\biggl{(}\frac{\int d^{2}a\
(\textbf{B}\cdot\textbf{n})^{2}}{A_{\mathrm{plasma}}}\biggr{)}^{1/2},$ (22)

where $A_{\mathrm{plasma}}$ is the surface area of the last closed flux
surface. Ideally, the normalized quadratic flux is 0 T, but due to the penalty
on current density in the objective function, the optimal stage-II magnetic
field will not exactly match the stage-I optimized magnetic field. For the
results that follow, we chose $\lambda$ so that $B^{*}_{\mathrm{RMS}}=0.01$ T,
with all configurations scaled to an average field strength in the plasma of
5.865 T. The value 0.01 T was chosen as it is low enough to achieve closed
flux surfaces that match the target last closed flux surface, as shown in
figure 3(a).

Let us also define the largest magnitude of the current density $K$ on the
winding surface as $\|K\|_{\infty}$. The plasma-coil separation $L$ is
adjusted so that $\|K\|_{\infty}$ equals a target value $\|K\|_{\infty}^{*}$.
This constraint has two useful properties. First, increasing plasma-coil
separation while fixing $B^{*}_{\mathrm{RMS}}$ smoothly increases
$\|K\|_{\infty}$ as shown in figure 4. This means the target
$\|K\|^{*}_{\infty}$ will be associated with a unique plasma-coil separation,
which can be found using root-finding. Second, on a winding surface, a higher
current density results in lower coil-coil separation, which is a common
measure of coil complexity.[27] To understand why, consider $N$ coils which
approximate the current sheet on a winding surface. Via Lagrange’s identity,
$\|K\|=\|\nabla\Phi\|$. Let us assume the total poloidal current
$I_{\mathrm{pol}}$ is distributed uniformly among the coils, so the current in
each coil is $I_{\mathrm{pol}}/N$, where $N$ is the number of coils. In the
limit of a large number of coils, the current density can be expressed in
terms of the distance between coils $d_{cc}$:

$\|K\|\equiv\lim_{N\to\infty}\frac{I_{\mathrm{pol}}}{Nd_{cc}}.$ (23)

In the case of a finite number of coils, this formula remains a good
approximation when $d_{cc}$ is small. Therefore,

$K_{\mathrm{max}}\equiv\|K\|_{\infty}\approx\frac{I_{\mathrm{pol}}}{Nd_{\mathrm{min}}},$
(24)

where $d_{\mathrm{min}}$ is the minimum coil-coil distance. To choose
$\|K\|^{*}_{\infty}$, we considered values that have been proposed in detailed
engineering studies. For example, in ARIES-CS, there are 18 coils (3 unique
coils per half field period) and the minimum coil-coil separation is 77.32
cm.[1] Using equation (24), this separation is equivalent to a maximum current
density of 17.16 MA/m. Therefore, we vary $L$ so that $\|K\|_{\infty}$ = 17.16
MA/m.

###  III.2 Additional Methods for finite $\beta$ Configurations

Before we can calculate $L_{\nabla\mathbf{B}}$ and the optimized plasma-coil
separation $L_{\mathrm{\texttt{REGCOIL}}}$, we need to calculate two
additional terms when $\beta$, the ratio of plasma to magnetic pressure, is
nonzero. This complication arises because in this case the magnetic field is
not generated only by currents in the external coils, and the current from the
plasma may complicate the relationship between $L_{\nabla\mathbf{B}}$ and the
plasma-coil separation. Therefore, for this paper we will be careful to
decompose the total magnetic field $\mathbf{B}$ into the magnetic field from
the coils $\mathbf{B}_{\mathrm{coils}}$, and the magnetic field from the
plasma, $\mathbf{B}_{\mathrm{\mathrm{plasma}}}$:

$\mathbf{B}=\mathbf{B}_{\mathrm{\mathrm{plasma}}}+\mathbf{B}_{\mathrm{coils}}.$
(25)

To compute $L_{\nabla\mathbf{B}}$, we use the virtual casing method to
calculate $\mathbf{B}_{\mathrm{coils}}$ on the LCFS. [28, 29] To summarize the
virtual casing approach, $\mathbf{B}_{\mathrm{coils}}$ is computed using a
magnetic method of images to match the boundary conditions on the plasma
surface. As a result, we can find a solution for $\mathbf{B}_{\mathrm{coils}}$
on the last closed flux surface $\delta\Omega$ :

$\mathbf{B}_{\mathrm{coils}}(\mathbf{x})=-\frac{1}{4\pi}\oint_{\delta\Omega}d^{2}p\frac{(\mathbf{n}\times\mathbf{B(p)})\times(\mathbf{x-p})}{|\mathbf{x-p}|^{3}},$
(26)

where $\mathbf{p}$ is a point on the LCFS. We used the package
VIRTUAL_CASING[30, 31] via SIMSOPT[32, 33] in order to calculate
$\mathbf{B}_{\mathrm{coils}}$ and $\nabla\mathbf{B}_{\mathrm{coils}}$, which
we then used to calculate $L_{\nabla\mathbf{B}}$ via equation (12).

A similar issue is present in optimizing the external coils using REGCOIL.
REGCOIL’s objective function in equation (21) uses the total magnetic field
$\mathbf{B}$ as in equation (25). Therefore, we must find
$\mathbf{B}_{\mathrm{plasma}}$. REGCOIL is interfaced to the BNORM code from
the STELLOPT suite,[34] which calculates $\mathbf{B}_{\mathrm{plasma}}$ using
data from a VMEC file.[35] Then $\mathbf{B}_{\mathrm{coils}}$ is added to
$\mathbf{B}_{\mathrm{plasma}}$ to calculate the objective function 21 using
equation (25). While it would also be possible to use VIRTUAL_CASING to
calculate $\mathbf{B}_{\mathrm{plasma}}$ for REGCOIL, doing so would require
an additional software interface, so using BNORM in REGCOIL was more
convenient.

Initialize $\lambda$ and $L$ (Use BNORM if $\beta>0$) Construct winding
surface with separation $L$ Calculate $\Phi$ on winding surface
$B_{\mathrm{RMS}}$ = $B_{\mathrm{{RMS}}}^{*}$ ?$\|K\|_{\infty}$ =
$\|K\|_{\infty}^{*}$ ?Vary $\lambda$Vary $L$ Output $\equiv
L_{\mathrm{\texttt{REGCOIL}}}$ Stopvia REGCOILYesNoNoYes Figure 5: A flowchart
depicting the iterative stage II optimization via REGCOIL to achieve the
necessary coil constraints. Figure 6: The precise QH plasma configuration[26]
and the coil arrangements found by REGCOIL. Here
$L_{\mathrm{\texttt{REGCOIL}}}$ is 1.7960 m, as this value was found to be the
required plasma-coil separation to satisfy the accuracy and complexity
constraints. The red dot is located where $L_{\nabla\mathbf{B}}$ is smallest.
Figure 7: For a wide range of plasma configurations scaled to a minor radius
of 1.704 m, we used REGCOIL to perform stage II coil-optimization (subject to
constraints in field accuracy and coil complexity). The plasma-coil separation
found (labeled $L_{\mathrm{\texttt{REGCOIL}}}$) has a strong correlation with
the smallest value of $L_{\nabla\mathbf{B}}$ evaluated on the surface. Figure
8: Two contour plots are overlaid for the precise QH stellarator and the coil
solution of figure 6. The filled-in contours indicate $L_{\nabla\mathbf{B}}$
on the plasma surface. The unfilled contours depict the current density $K$ on
the winding surface, determined as described in section III.1. The locations
of smallest $L_{\nabla\mathbf{B}}$ and largest $K$ coincide, at the inside of
the bean cross-section. This spatial correlation is strongest in
configurations close to the linear regression line in figure 7.

###  III.3 Summary of Optimization Methods

Given these additional considerations, we now summarize the process of
computing the coil-plasma distance with REGCOIL and comparing the result with
$L_{\nabla\mathbf{B}}$. We first choose an initial $\lambda$ and $L$. If
$\beta$ $>$ 0, we must also compute $\mathbf{B}_{\mathrm{plasma}}$ using
BNORM. We then construct a winding surface offset from the plasma by a uniform
distance $L$. We then compute the amplitudes $\Phi_{j}$ in the Fourier series
of the current potential $\Phi$ on the winding surface in order to minimize
the loss function $f$ in equation (21). We then perform a search over the
regularization parameter $\lambda$ so that the normalized quadratic flux
constraint $B_{\mathrm{RMS}}=B^{*}_{\mathrm{RMS}}$ is fulfilled. We found that
0.01 T was sufficient to match the target LCFS, as shown in figure 3(a). We
then scan over $L$ to fulfill the maximum current density constraint,
$\|K\|^{*}_{\infty}$ = 17.16 MA/m. We refer to the optimized plasma-coil
separation subject to these constraints as $L_{\mathrm{\texttt{REGCOIL}}}$.
Refer to figure 5 for a flowchart depicting the optimization process. We
checked the numerical resolution needed to ensure convergence of
$L_{\mathrm{\texttt{RECOIL}}}$. We determined that for these parameters,
sufficient convergence was achieved when the number of poloidal modes $m_{j}$
and toroidal modes $n_{j}$ used to represent the current potential both equal
20, and the number of grid points in each angle necessary to discretize the
surface area integrals (referred to within REGCOIL’s code as ntheta_coil,
nzeta_coil, ntheta_plasma, and nzeta_plasma) all equal 96.

We also calculate the magnetic gradient scale length for the plasma
configuration. We first calculate $\mathbf{B}_{\mathrm{coils}}$ and
$\nabla\mathbf{B}_{\mathrm{coils}}$ using VIRTUAL_CASING in SIMSOPT if
$\beta>0$. We then calculate $L_{\nabla\mathbf{B}}$ on a high resolution grid
of points on the LCFS. In Appendix C, we describe the method of converting
from flux coordinates (like the coordinates used in VMEC[35]) to Cartesian
coordinates in order to calculate the Frobenius norm. We define

$L^{*}_{\nabla\mathbf{B}}=\min\\{L_{\nabla\mathbf{B}}(\mathbf{x})|\
\mathbf{x}\in\delta\Omega\\}$ (27)

as the smallest value of $L_{\nabla\mathbf{B}}$ on the LCFS, $\delta\Omega$.

We show an example of the optimized coil arrangement and
$L_{\nabla\mathbf{B}}$ for one configuration, the precise QH stellarator, [26]
in figure 6.

##  IV Results and Discussion

We have gathered a database of over 40 plasma configurations. These
configurations are listed in Appendix D. The database includes quasi-
axisymmetric stellarators, quasi-helical stellarators, quasi-isodynamic
stellarators, tokamaks, and other miscellaneous stellarators. The VMEC files
of these configurations and supplemental data have been made available on
Zenodo.[36] For a fair comparison, all configurations have been scaled to the
minor radius of the reactor-scale ARIES-CS configuration, $a$ = 1.704 m.[1]
(If the configurations are all scaled to the same major radius instead of
minor radius, the results that follow are not qualitatively different.) The
volume averaged magnetic field of each configuration in our database is also
scaled to the volume averaged magnetic field of ARIES-CS, 5.865 T.
Interestingly, the calculated $L_{\mathrm{\texttt{REGCOIL}}}$ varies by a
factor of 10 across the database. Evidently, there are significant differences
among configurations in the feasibility of attaining large coil-plasma
separation.

Figure 7 compares the minimum $L_{\nabla\mathbf{B}}$ for each configuration
with the plasma-coil separation obtained via REGCOIL. A strong correlation
between these two quantities is evident; the correlation coefficient
$\mathrm{R}^{2}=0.941$. It is noteworthy that the correlation of figure 7 is
insensitive to the target $\|K\|^{*}_{\infty}$ and $B^{*}_{\mathrm{RMS}}$ we
choose. Changing either of these constraints within a plausible range changes
only the slope and intercept of the linear regression between
$L_{\nabla\mathbf{B}}$ and $L_{\mathrm{\texttt{REGCOIL}}}$, but has little
effect on the strength of the correlation. This correlation is also
insensitive to the exact formulation of the magnetic gradient scale length. In
section II, we considered some alternative scale lengths to
$L_{\nabla\mathbf{B}}$. In Appendix B, we present similar figures to figure 7
calculated with these alternate scale lengths. These scale lengths have
slightly weaker correlation with $L_{\texttt{REGCOIL}}$ compared to
$L_{\nabla\mathbf{B}}$, but are generally still well correlated.

As can be seen in Appendix D, there is a trend that the largest coil-plasma
distances are possible only with a small number of field periods. Indeed, the
two configurations with the largest plasma-coil separation have a single field
period, and the next seven configurations with the largest plasma-coil
separation all have two field periods.

Another noteworthy result is that the optimized current density $K$ on the
winding surface and $L_{\nabla\mathbf{B}}$ calculated on the LCFS are often
correlated spatially. For example, in figure 8 we have presented overlapping
contour plots for $K$ and $L_{\nabla\mathbf{B}}$ for one configuration. The
two angles on the winding surface $(\theta^{\prime},\phi^{\prime})$ are
defined so that for a position vector $\mathbf{r}^{\prime}$ on the winding
surface,

$\theta(\mathbf{r}^{\prime}),\phi(\mathbf{r}^{\prime})=\theta,\phi\;\;\;\mathrm{s.t.}\;\;\;\mathbf{r}(\theta,\phi)+L_{\mathrm{\texttt{REGCOIL}}}\mathbf{n}=\mathbf{r}^{\prime},$
(28)

where $\mathbf{r}$ is the position vector on the LCFS at the associated point
$(\theta,\phi)$, and $\mathbf{n}$ is the outward unit normal vector on the
LCFS. For many configurations, the position of maximum current density
$\|K\|_{\infty}$ and the position of the smallest scale length
$L^{*}_{\nabla\mathbf{B}}$ coincide. Spatial correlation of the contours is
strongest near the limiting point (largest $\|K\|_{\infty}$ and smallest
$L_{\nabla\mathbf{B}}$), weaker in regions of lower $\|K\|$ and larger
$L_{\nabla\mathbf{B}}$. There is usually weaker spatial correlation in
configurations that are relatively far from the linear regression line in
figure 7. We also found a strong spatial correlation between the location of
$\|K\|_{\infty}$ and the position of highest radius of curvature, equation
(8). The points of largest $\|K\|_{\infty}$, smallest $L_{\nabla\mathbf{B}}$,
and smallest radius of curvature are usually in the inside curve of the bean-
shaped cross-section, as shown in figure 6. This relationship matches
conventional wisdom that coil engineering constraints are hardest to satisfy
in this region.

We should note that the shift caused by accounting for non-vacuum magnetic
fields was relatively small. If we do not correct for the difference between
$\mathbf{B}$ and $\mathbf{B_{\mathrm{coils}}}$, both $L_{\nabla\mathbf{B}}$
and $L_{\mathrm{\texttt{REGCOIL}}}$ only shift slightly. This shift is small
because the database we used only contained configurations with $\beta\leq$
5%. Therefore, $\mathbf{B_{\mathrm{plasma}}}$ was an order of magnitude less
than $\mathbf{B_{\mathrm{coils}}}$. There may be a more significant shift in
configurations with higher $\beta$.

As figure 7 shows, a number of configurations have some deviation from the
linear regression. Some of these configurations have properties that suggest
why $L_{\nabla\mathbf{B}}$ or $L_{\mathrm{\texttt{REGCOIL}}}$ for these
configurations may not lie on the best-fit line. For example, one point that
is somewhat below the best-fit line is the ITER tokamak[37]. This
configuration may lie off the trend line because REGCOIL cannot match the
target $B_{\mathrm{RMS}}$, which vanishes for axisymmetric configurations.

Another property that is present among some outliers is coil ripple. For
example, the point furthest above the best-fit line is an HSX configuration
[38] that includes ripple from its 48 modular coils. We compared this standard
HSX configuration to an HSX variant in which the ripple was removed. The shape
of HSX’s last closed flux surface is defined by a Fourier series,

$\displaystyle R(\theta,\phi)$ $\displaystyle=\sum_{m,n}R_{m,n}\cos(m\theta-
n_{fp}n\phi),$ (29) $\displaystyle z(\theta,\phi)$
$\displaystyle=\sum_{m,n}z_{m,n}\sin(m\theta-n_{fp}n\phi),$ (30)

where $m$ and $n$ are the poloidal and toroidal mode numbers, respectively,
and $n_{fp}$ is the number of field periods. To create the HSX variant, we
truncate any boundary mode with $n>4$, which smooths out the field. For this
HSX variant, the stage II optimized coils can be placed farther away, and the
ripple-less HSX configuration better matches the linear regression. This may
point to a limitation in $L_{\nabla\mathbf{B}}$. The magnetic field of HSX
only differs from ripple-less HSX on small spatial scales, while
$L_{\nabla\mathbf{B}}$ primarily captures large spatial scales. Since
additional gradients magnify small spatial scales, it may be worthwhile to
consider alternative scale lengths based on $\nabla\nabla\mathbf{B}$.

A final group of outlier configurations in figure 7 is associated with
relatively large errors in the VMEC solution. Because $L_{\nabla\mathbf{B}}$
is calculated using data from VMEC, the VMEC solution must be accurate. We can
check the accuracy of these solutions by measuring
$\mathrm{Tr}(\nabla\mathbf{B})$, which should be 0 everywhere, or
$\nabla\mathbf{B}-(\nabla\mathbf{B})^{T}$, which should be 0 in a vacuum.
These quantities differ from zero significantly in some of the configurations
that are outliers in figure 7. For example, the CTH configuration with high
rotational transform,[39] which is the configuration that is farthest from the
line of best fit, has a
$\|\nabla\mathbf{B}-(\nabla\mathbf{B})^{T}\|_{F}=0.19\|\nabla\mathbf{B}\|_{F}$.
For configurations such as this one with large
$\|\nabla\mathbf{B}-(\nabla\mathbf{B})^{T}\|_{F}$, we were unable to obtain
satisfactory VMEC solutions at higher resolution, since at higher resolution
it is hard to achieve small values of the MHD force residual. This issue is
likely specific to VMEC, so these configurations might lie closer to the best-
fit line in figure 7 if another equilibrium code was used instead.

##  V Summary and Future Work

Over the course of this paper, we explored the magnetic gradient scale length
and its connection to plasma-coil separation. Since a magnetic field decays
with distance from the source, the source cannot be much farther than this
scale length. The distance to a source is exactly the $L_{\nabla\mathbf{B}}$
scale length in the case of an infinite straight wire.

We calculated the minimum magnetic gradient scale length for a wide variety of
over 40 plasma configurations, and found it to correlate strongly with the
coil-plasma separation, for fixed magnetic field accuracy and coil complexity.
For a given configuration, the point of minimum magnetic scale length is
typically located on the inside of the bean-shaped cross-section of the
plasma, where coil complexity is typically greatest. These observations
support the use of $L_{\nabla\mathbf{B}}$ as a measure of the intrinsic
difficulty of producing a given magnetic field with distant coils. We can now
understand why it is difficult to find stage-II coil solutions far from the
plasma for certain plasma configurations: if the configuration has a small
$L_{\nabla\mathbf{B}}$, it will be hard to find distant coils no matter what
specific coil design code is used.

We observe a general inverse relationship between the number of field periods
and the plasma-coil separation. In particular, the nine configurations with
largest coil-to-plasma distance compared to the minor radius all have only one
or two field periods. This makes one- or two-field-period configurations
attractive for reactors, where a blanket and neutron shielding are required
between the coils and plasma.

We plan on further studying the nature of the magnetic gradient scale length
$L_{\nabla\mathbf{B}}$. There is also merit in studying the norm of
$\nabla\nabla\mathbf{B}$, which is a rank 3 tensor. However,
$\nabla\nabla\mathbf{B}$ may be more susceptible to the noise in the magnetic
field due to discretization error in the MHD equilibrium solution. In the
future, we would like to study how the slope of the linear regression of
figure 7 changes as $B^{*}_{\mathrm{RMS}}$ and $\|K\|_{\infty}^{*}$ change.

In addition, it would be natural to try including $L_{\nabla\mathbf{B}}$ in
the objective functions of Stage I optimizations (i.e., optimizing the plasma
shape without explicitly considering coil shapes), with the goal of increasing
the plasma-coil separation. Compared to REGCOIL or other coil design codes,
using $L_{\nabla\mathbf{B}}$ to approximate plasma-coil separation requires
few assumptions, since we do not have to set constraints on the quadratic flux
accuracy or the highest current density. In addition to $L_{\nabla\mathbf{B}}$
taking less time to calculate than plasma-coil separation, it is easier to
differentiate through $L_{\nabla\mathbf{B}}$, which would further shorten
optimization time.

## Acknowledgements

We are grateful to the many people who contributed MHD configurations to this
study. We would also like to thank Stefan Buller and Byoungchan Jang for
comments on the manuscript. This work was supported by the U.S. Department of
Energy, Office of Science, Office of Fusion Energy Science, under award number
DE-FG02-93ER54197. This research used resources of the National Energy
Research Scientific Computing Center (NERSC), a U.S. Department of Energy
Office of Science User Facility located at Lawrence Berkeley National
Laboratory, operated under Contract No. DE-AC02-05CH11231 using NERSC award
FES-ERCAP-mp217-2023.

##  Appendix A Magnetic Field of a Circular Wire

Consider a circular wire of radius $a$ in the $x$-$y$ plane. The magnetic
field is as follows:[40]

$\displaystyle\rho^{2}=x^{2}+y^{2};\ r^{2}=x^{2}+y^{2}+z^{2};\
\alpha^{2}=a^{2}+r^{2}+z^{2}-2a\rho,$ (31)
$\displaystyle\beta^{2}=a^{2}+r^{2}+2a\rho;\
k^{2}=1-\frac{\alpha^{2}}{\beta^{2}},$ (32) $\displaystyle
B_{x}=\frac{\mu_{0}I}{\pi}\frac{xz}{2\alpha^{2}\beta\rho^{2}}\biggl{(}(a^{2}+r^{2})E(k^{2})-\alpha^{2}K(k^{2})\biggr{)},$
(33) $\displaystyle
B_{y}=\frac{\mu_{0}I}{\pi}\frac{yz}{2\alpha^{2}\beta\rho^{2}}\biggl{(}(a^{2}+r^{2})E(k^{2})-\alpha^{2}K(k^{2})\biggr{)},$
(34) $\displaystyle
B_{z}=\frac{\mu_{0}I}{\pi}\frac{1}{2\alpha^{2}\beta}\biggl{(}(a^{2}-r^{2})E(k^{2})+\alpha^{2}K(k^{2})\biggr{)},$
(35)

where $K(k^{2})$ and $E(k^{2})$ are the complete elliptic integrals of the
first and second kind, respectively. At the center of the circular coil,
$L_{\nabla\mathbf{B}}$$\rightarrow\infty$. The region in the neighborhood
surrounding the coil has $L_{\nabla\mathbf{B}}$ values that are approximately
the distance to the coil, since the circular wire can locally be approximated
by a straight wire in the limit that the distance from the coil is small
compared to its radius.

Figure 9: The $L_{\nabla\mathbf{B}}$ scale length for the magnetic field
generated by a circular wire. The circle has a radius of 1 m and lies in the
$x$-$y$ plane. Grey dots signify the location of the wire. Figure 10: A
comparison of magnetic gradient scale lengths for the precise QH plasma
configuration.[26] For clarity, the colormap is bounded so that any scale
length larger than 20 m is truncated to 20 m.

In figure 9, we have plotted the scale length on the $x$-$z$ plane. As this
figure shows, $L_{\nabla\mathbf{B}}$ approximates the distance to the coil
best when measured close to the coil.

##  Appendix B Alternative Magnetic Gradient Scale Lengths

Besides $L_{\nabla\mathbf{B}}$, a number of alternative scale lengths that
could be considered are

$\displaystyle L_{\mathbf{b}\cdot\nabla\mathbf{b}}$
$\displaystyle=\frac{1}{\|\mathbf{b}\cdot\nabla\mathbf{b}\|},$ (36)
$\displaystyle L_{\nabla|B|}$ $\displaystyle=\frac{B}{\|\nabla B\|},$ (37)
$\displaystyle L_{\|\sigma\|}$
$\displaystyle=\frac{\sqrt{2}B}{(\Sigma_{i}\sigma_{i}^{2})^{1/2}},$ (38)
$\displaystyle L_{\mathrm{Max}\sigma}$
$\displaystyle=\frac{B}{\mathrm{Max}[\sigma_{i}]},$ (39)

where $\sigma_{i}$ are the 3 singular values of the matrix $\nabla\mathbf{B}$.
Note that $L_{\|\sigma\|}=L_{\nabla\mathbf{B}}$ for a vacuum field. In figure
10, we have plotted these alternate scale lengths on the surface of the
precise QH configuration. All these alternate scale lengths are smallest on
the inside of the bean cross-section, which is where $L_{\nabla\mathbf{B}}$
was previously mentioned to be smallest in section IV. At points where
$L_{\nabla\mathbf{B}}$ is high, only equations 39 and 38 are similar in value
to $L_{\nabla\mathbf{B}}$. For 37 and 36, the locations of largest scale
length are shifted slightly compared to $L_{\nabla\mathbf{B}}$. We find there
is a significant correlation between each of these alternate scale lengths and
$L_{\mathrm{\texttt{REGCOIL}}}$, although $L_{\nabla\mathbf{B}}$ has the
strongest correlation. In figure 11 we have plotted these alternate scale
lengths compared to $L_{\mathrm{\texttt{REGCOIL}}}$.

(a)

(b)

(c)

Figure 11: For a wide range of plasma configurations scaled to a minor radius
of 1.704 m, we used REGCOIL to perform stage II coil-optimization (subject to
constraints in field accuracy and coil complexity). The plasma-coil separation
(labeled $L_{\texttt{REGCOIL}}$) is correlated with (a)
$L_{\mathbf{b}\cdot\nabla\mathbf{b}}$ from equation (36), (b) $L_{\nabla|B|}$
from equation (37), and (c) $L_{\mathrm{Max}\sigma}$ from equation (39).

##  Appendix C Conversion Between Flux Coordinates and Cartesian Coordinates

Here we show a method to compute the $\nabla\mathbf{B}$ tensor elements in
Cartesian coordinates given $\mathbf{B}$ in flux coordinates
$(s,\theta,\phi)$, where $s$ is the flux surface coordinate, $\theta$ is any
poloidal angle coordinate, and $\phi$ is the traditional azimuthal angle
coordinate.[41] For nested flux surfaces to exist, $\mathbf{B}\cdot\nabla
s=0$. Therefore, the magnetic field can be written in terms of its
contravariant components as

$\mathbf{B}=B^{\theta}\frac{\partial\mathbf{r}}{d\theta}+B^{\phi}\frac{\partial\mathbf{r}}{d\phi},$
(40)

where $\mathbf{r}=R\hat{\mathbf{R}}+z\hat{\mathbf{z}}$ is the position vector,
and $(R,\phi,z)$ are cylindrical coordinates. Applying the chain rule to
$\mathbf{r}$,

$\mathbf{B}=B^{\theta}\biggl{(}\frac{\partial
R}{d\theta}\hat{\mathbf{R}}+\frac{\partial
z}{d\theta}\hat{\mathbf{z}}\biggr{)}+B^{\phi}\biggl{(}\frac{\partial
R}{\partial\phi}\hat{\mathbf{R}}+R\hat{\pmb{\phi}}+\frac{\partial
z}{\partial\phi}\hat{\mathbf{z}}\biggr{)}.$ (41)

Therefore,

$\displaystyle B_{x}$ $\displaystyle=B^{\theta}\frac{\partial
R}{\partial\theta}\cos\phi+B^{\phi}\frac{\partial
R}{\partial\phi}\cos\phi-B^{\phi}R\sin\phi,$ (42) $\displaystyle B_{y}$
$\displaystyle=B^{\theta}\frac{\partial
R}{\partial\theta}\sin\phi+B^{\phi}\frac{\partial
R}{\partial\phi}\sin\phi+B^{\phi}R\cos\phi,$ (43) $\displaystyle B_{z}$
$\displaystyle=B^{\theta}\frac{\partial
z}{\partial\theta}+B^{\phi}\frac{\partial z}{\partial\phi},$ (44)

The representation of $\nabla\mathbf{B}$ as a matrix has 9 components, and the
Cartesian form of the matrix is shown in equation (3). For simplicity, we
shall focus on the formula for the first column of the Cartesian gradient
matrix, $\nabla B_{x}$, but the same steps can be applied to find the other
columns of the matrix.

Writing the gradient of $B_{x}(s,\theta,\phi)$ in terms of flux coordinates,

$\nabla B_{x}=\frac{\partial B_{x}}{\partial s}\nabla s+\frac{\partial
B_{x}}{\partial\theta}\nabla\theta+\frac{\partial
B_{x}}{\partial\phi}\nabla\phi.$ (45)

Let us find formulas for each term in equation (45) and then convert to
cylindrical coordinates. We use the dual relation between $\nabla s$ and the
contravariant basis vectors:

$\nabla
s=\frac{1}{\sqrt{g}}\biggl{(}\frac{\partial\mathbf{r}}{\partial\theta}\times\frac{\partial\mathbf{r}}{\partial\phi}\biggr{)},$
(46)

where $\sqrt{g}=(\nabla s\times\nabla\theta\cdot\nabla\phi)^{-1}$. The
formulas for $\nabla\theta$ and $\nabla\phi$ follow from equation (46) by
cyclic permutation. We can now convert to cylindrical coordinates:

$\nabla s=\frac{1}{\sqrt{g}}\biggl{[}\frac{\partial
R}{\partial\theta}R\hat{\mathbf{z}}+\biggl{(}\frac{\partial
R}{\partial\phi}\frac{\partial z}{\partial\theta}-\frac{\partial
R}{\partial\theta}\frac{\partial
z}{\partial\phi}\biggr{)}\hat{\pmb{\phi}}-\frac{\partial
z}{\partial\theta}R\hat{\mathbf{R}}\biggr{]}.$ (47)

Using the same logic,

$\nabla\theta=\frac{1}{\sqrt{g}}\biggl{[}R\frac{\partial z}{\partial
s}\hat{\mathbf{R}}+\biggl{(}\frac{\partial R}{\partial s}\frac{\partial
z}{\partial\phi}-\frac{\partial R}{\partial\phi}\frac{\partial z}{\partial
s}\biggr{)}\hat{\pmb{\phi}}-R\frac{\partial R}{\partial
s}\hat{\mathbf{z}}\biggr{]},$ (48) $\nabla\phi=\frac{1}{r}\hat{\pmb{\phi}}.$
(49)

We can then use the chain rule to find

$\displaystyle\begin{split}\frac{\partial B_{x}}{\partial s}=&\frac{\partial
B^{\theta}}{\partial s}\frac{\partial
R}{\partial\theta}\cos\phi+B^{\theta}\frac{\partial^{2}R}{\partial
s\partial\theta}\cos\phi+\frac{\partial B^{\phi}}{\partial s}\frac{\partial
R}{\partial\phi}\cos\phi\\\ &+B^{\phi}\frac{\partial^{2}R}{\partial
s\partial\phi}\cos\phi-\frac{\partial B^{\phi}}{\partial
s}R\sin\phi-B^{\phi}\frac{\partial R}{\partial s}\sin\phi,\end{split}$ (50)
$\displaystyle\begin{split}\frac{\partial
B_{x}}{\partial\theta}=&\frac{\partial
B^{\theta}}{\partial\theta}\frac{\partial
R}{\partial\theta}\cos\phi+B^{\theta}\frac{\partial^{2}R}{\partial\theta^{2}}\cos\phi+\frac{\partial
B^{\phi}}{\partial\theta}\frac{\partial R}{\partial\phi}\cos\phi\\\
&+B^{\phi}\frac{\partial^{2}R}{\partial\phi\partial\theta}\cos\phi-\frac{\partial
B^{\phi}}{\partial\theta}R\sin\phi-B^{\phi}\frac{\partial
R}{\partial\theta}\sin\phi,\end{split}$ (51)
$\displaystyle\begin{split}\frac{\partial B_{x}}{\partial\phi}=&\frac{\partial
B^{\theta}}{\partial\phi}\frac{\partial
R}{\partial\theta}\cos\phi+B^{\theta}\frac{\partial^{2}R}{\partial\phi\partial\theta}\cos\phi-B^{\theta}\frac{\partial
R}{\partial\theta}\sin\phi\\\ &+\frac{\partial
B^{\phi}}{\partial\phi}\frac{\partial
R}{\partial\phi}\cos\phi+B^{\phi}\frac{\partial^{2}R}{\partial\phi^{2}}\cos\phi-B^{\phi}\frac{\partial
R}{\partial\phi}\sin\phi\\\ &-\frac{\partial
B^{\phi}}{\partial\phi}R\sin\phi-B^{\phi}\frac{\partial
R}{\partial\phi}\sin\phi-B^{\phi}R\cos\phi.\end{split}$ (52)

We can substitute these formulas into equation (45) in order to calculate
$\partial B_{x}/\partial x$, $\partial B_{x}/\partial y$, and $\partial
B_{x}/\partial z$. We can use similar steps to calculate any of the other six
elements of the matrix $\nabla\mathbf{B}$.

##  Appendix D Table of Plasma Configurations

Below is a table with a brief description of all configurations used in this
paper. They are arranged in order of increasing
$L_{\mathrm{\texttt{{REGCOIL}}}}$. Both the number of field periods (NFP) and
the smallest magnetic gradient scale length ($L^{*}_{\nabla\mathbf{B}}$) are
recorded.

Description | NFP |  $L_{\texttt{REGCOIL}}$ (m) |  $L^{*}_{\nabla\mathbf{B}}$ (m)
---|---|---|---
nfp=4 quasi-helical (QH) configuration by Ku & Boozer [42] | 4 | 0.4060 | 0.9691
Unpublished QH configuration from Michael Drevlak | 5 | 0.4099 | 1.0538
Columbia Non-Neutral Torus (CNT)[43] | 2 | 0.5189 | 1.2507
Tokamak de la Junta II (TJ-II) [44] | 4 | 0.5638 | 1.3777
Wistell-B, Bader et al. [45] | 5 | 0.6047 | 1.1726
Quasi-axisymmetric (QA) configuration designed by Paul Garabedian [46] | 2 | 0.6188 | 1.4214
Large Helical Device (LHD), major radius 3.60m [47] | 10 | 0.6475 | 1.3358
Quasi-Poloidal Stellarator (QPS)[48] | 2 | 0.6625 | 1.5812
LHD, major radius 3.53m [47] | 10 | 0.7302 | 1.5354
LHD, major radius 3.75m [47] | 10 | 0.7858 | 1.7226
Henneberg et al. QA [49] | 2 | 0.7987 | 1.4390
Advanced Research Innovation and Evaluation Study-Compact Stellarator (ARIES-CS) [1] | 3 | 0.8655 | 1.8375
National Compact Stellarator Experiment (NCSX) stage-1 optimization result (known as LI383)[50] | 3 | 0.8771 | 1.9343
The first quasisymmetric configuration found[51] | 6 | 0.9374 | 1.6582
Advanced Toroidal Facility (ATF) [52] | 12 | 0.9913 | 2.1721
NCSX free-boundary (c09r00) [50] | 3 | 1.0015 | 2.3233
Wendelstein 7-X (W7-X), without coil ripple[53] | 5 | 1.2858 | 2.1941
Landreman, Buller, & Drevlak, QH, 5% beta [54] | 4 | 1.3009 | 2.6120
Landreman, Buller, & Drevlak, QH, vacuum [54] | 4 | 1.3545 | 2.7385
Boundary constructed by near-axis expansion. Vacuum QH with nfp=4 | 3 | 1.3650 | 2.5722
Goodman et al. Quasi-isodynamic (QI) configuration with nfp=3 [55] | 4 | 1.3712 | 2.5633
W7-X standard configuration. Vacuum, with coil ripple [56] | 5 | 1.3811 | 3.4715
Quasi-Isodynamic (QI) configuration from CIEMAT [57] | 4 | 1.4130 | 3.2634
Chinese First Quasiaxisymmetric Stellarator (CQFS) [58] | 2 | 1.4839 | 3.3392
Landreman & Paul, QH with magnetic well[26] | 4 | 1.5206 | 3.1882
Wistell-A. Bader et al. [59] | 4 | 1.5641 | 3.0210
W7-X "high narrow mirror" configuration[60] | 5 | 1.5952 | 3.6979
Compact Toroidal Hybrid (CTH) Stellarator, vacuum, with low rotational transform [39] | 5 | 1.6556 | 1.9259
Landreman & Paul, precise QH [26] | 4 | 1.7960 | 3.5418
Unpublished nfp=3 QH | 3 | 1.8644 | 3.4277
Up-down-symmetric ITER-like configuration[37] | 1 | 1.9248 | 3.0531
Evolutive Stellarator of Lorraine (ESTELL) [61] | 2 | 2.1860 | 3.2610
Helically Symmetric Experiment (HSX), standard configuration, vacuum, with coil ripple[38] | 4 | 2.2377 | 4.9052
Compact Toroidal Hybrid (CTH) stellarator, vacuum, with high rotational transform [39] | 5 | 2.4102 | 3.6607
Boundary constructed by near-axis expansion. Vacuum QH with nfp=3[22] | 3 | 2.6091 | 3.9118
HSX, standard configuration, vacuum, without coil ripple | 4 | 2.8111 | 4.9336
Vacuum QA configuration with 16 coils from Giuliani et al. Coil length 24m. [62] | 2 | 2.8602 | 5.2643
Landreman & Paul, precise QA [26] | 2 | 2.8748 | 5.2977
Wechsung et al. QA without magnetic well, coil length 24m. [63] | 2 | 2.8750 | 5.3037
Wechsung et al. QA with magnetic well, coil length 24m [63] | 2 | 2.9790 | 5.5563
Landreman & Paul QA with magnetic well.[26] | 2 | 2.9806 | 5.5532
Goodman et al. Quasi-isodynamic configuration with nfp=2 [55] | 2 | 3.0045 | 5.1919
Landreman, Buller & Drevlak, QA, 2.5% beta [54] | 2 | 3.0419 | 5.9042
Goodman et al. Quasi-isodynamic configuration with nfp=1 [55] | 1 | 4.1563 | 6.6993
Jorge et al. Quasi-isodynamic configuration with nfp=1 [64] | 1 | 4.7133 | 7.5360

## References

## References

  * Najmabadi  _et al._ [2008] F. Najmabadi, A. R. Raffray, S. I. Abdel-Khalik, L. Bromberg, L. Crosatti, L. El-Guebaly, P. R. Garabedian, A. A. Grossman, D. Henderson, A. Ibrahim, T. Ihli, T. B. Kaiser, B. Kiedrowski, L. P. Ku, J. F. Lyon, R. Maingi, S. Malang, C. Martin, T. K. Mau, B. Merrill, R. L. Moore, R. J. P. Jr., D. A. Petti, D. L. Sadowski, M. Sawan, J. H. Schultz, R. Slaybaugh, K. T. Slattery, G. Sviatoslavsky, A. Turnbull, L. M. Waganer, X. R. Wang, J. B. Weathers, P. Wilson, J. C. W. III, M. Yoda, and M. Zarnstorffh, “The ARIES-CS compact stellarator fusion power plant,” Fusion Science and Technology 54, 655–672 (2008).
  * Klinger  _et al._ [2013] T. Klinger, C. Baylard, C. Beidler, J. Boscary, H. Bosch, A. Dinklage, D. Hartmann, P. Helander, H. Maßberg, A. Peacock, T. Pedersen, T. Rummel, F. Schauer, L. Wegener, and R. Wolf, “Towards assembly completion and preparation of experimental campaigns of Wendelstein 7-X in the perspective of a path to a stellarator fusion power plant,” Fusion Engineering and Design 88, 461–465 (2013), proceedings of the 27th Symposium On Fusion Technology (SOFT-27); Liège, Belgium, September 24-28, 2012.
  * Nemov, Kasilov, and Kernbichler [2014] V. V. Nemov, S. V. Kasilov, and W. Kernbichler, “Collisionless high energy particle losses in optimized stellarators calculated in real-space coordinates,” Physics of Plasmas 21 (2014), 10.1063/1.4876740, 062501.
  * Lion  _et al._ [2021] J. Lion, F. Warmer, H. Wang, C. Beidler, S. Muldrew, and R. Wolf, “A general stellarator version of the systems code process,” Nuclear Fusion 61, 126021 (2021).
  * Drevlak  _et al._ [2018] M. Drevlak, C. Beidler, J. Geiger, P. Helander, and Y. Turkin, “Optimisation of stellarator equilibria with ROSE,” Nuclear Fusion 59, 016010 (2018).
  * Giuliani  _et al._ [2022a] A. Giuliani, F. Wechsung, A. Cerfon, G. Stadler, and M. Landreman, “Single-stage gradient-based stellarator coil design: Optimization for near-axis quasi-symmetry,” Journal of Computational Physics 459, 111147 (2022a).
  * Jorge  _et al._ [2023] R. Jorge, A. Goodman, M. Landreman, J. Rodrigues, and F. Wechsung, “Single-stage stellarator optimization: combining coils with fixed boundary equilibria,” Plasma Physics and Controlled Fusion 65, 074003 (2023).
  * Henneberg  _et al._ [2021] S. A. Henneberg, S. R. Hudson, D. Pfefferlé, and P. Helander, “Combined plasma–coil optimization algorithms,” Journal of Plasma Physics 87, 905870226 (2021).
  * Landreman [2017a] M. Landreman, “An improved current potential method for fast computation of stellarator coil shapes,” Nuclear Fusion 57, 046003 (2017a).
  * Landreman [2017b] M. Landreman, “Regcoil,” https://github.com/landreman/regcoil (2017b).
  * Howes  _et al._ [2006] G. G. Howes, S. C. Cowley, W. Dorland, G. W. Hammett, E. Quataert, and A. A. Schekochihin, “Astrophysical gyrokinetics: Basic equations and linear theory,” The Astrophysical Journal 651, 590 (2006).
  * Nara, Suzuki, and Ando [2006] T. Nara, S. Suzuki, and S. Ando, “A closed-form formula for magnetic dipole localization by measurement of its magnetic field and spatial gradients,” IEEE Transactions on Magnetics 42, 3291–3293 (2006).
  * Yin  _et al._ [2014a] G. Yin, Y. Zhang, H. Fan, and Z. Li, “Magnetic dipole localization based on magnetic gradient tensor data at a single point,” Journal of Applied Remote Sensing 8, 083596 (2014a).
  * Schmidt and Clark [2006] P. W. Schmidt and D. A. Clark, “The magnetic gradient tensor: Its properties and uses in source characterization,” The Leading Edge 25, 75–78 (2006).
  * Higuchi, Nara, and Ando [2016] Y. Higuchi, T. Nara, and S. Ando, “Complete set of partial differential equations for direct localization of a magnetic dipole,” IEEE Transactions on Magnetics 52, 1–10 (2016).
  * Pedersen and Rasmussen [1990] L. B. Pedersen and T. M. Rasmussen, “The gradient tensor of potential field anomalies; some implications on data collection and data processing of maps,” Geophysics 55, 1558–1566 (1990).
  * Yin  _et al._ [2014b] G. Yin, Y. Zhang, H. Fan, and Z. Li, “Magnetic dipole localization based on magnetic gradient tensor data at a single point,” Journal of Applied Remote Sensing 8, 083596 (2014b).
  * D’haeseleer  _et al._ [1991] W. D’haeseleer, W. Hitchon, J. Callen, and J. Shohet,  _Flux Coordinates and Magnetic Field Structure: A Guide to a Fundamental Tool of Plasma Theory_ , Scientific Computation (Springer Berlin Heidelberg, 1991).
  * Schatten [1960] R. Schatten,  _Norm Ideals of Completely Continuous Operators_ (Springer, 1960).
  * Fan [1949] K. Fan, “On a theorem of Weyl concerning eigenvalues of linear transformations I,” Proceedings of the National Academy of Sciences 35, 652–655 (1949).
  * Landreman [2021] M. Landreman, “Figures of merit for stellarators near the magnetic axis,” Journal of Plasma Physics 87, 905870112 (2021).
  * Landreman [2022] M. Landreman, “Mapping the space of quasisymmetric stellarators using optimized near-axis expansion,” Journal of Plasma Physics 88, 905880616 (2022).
  * Jorge  _et al._ [2022a] R. Jorge, G. Plunk, M. Drevlak, M. Landreman, J.-F. Lobsien, K. Camacho Mata, and P. Helander, “A single-field-period quasi-isodynamic stellarator,” Journal of Plasma Physics 88, 175880504 (2022a).
  * Zhu  _et al._ [2017] C. Zhu, S. R. Hudson, Y. Song, and Y. Wan, “New method to design stellarator coils without the winding surface,” Nuclear Fusion 58, 016008 (2017).
  * Merkel [1987] P. Merkel, “Solution of stellarator boundary value problems with external currents,” Nuclear Fusion 27, 867 (1987).
  * Landreman and Paul [2022] M. Landreman and E. Paul, “Magnetic fields with precise quasisymmetry for plasma confinement,” Phys. Rev. Lett. 128, 035001 (2022).
  * Paul  _et al._ [2018] E. Paul, M. Landreman, A. Bader, and W. Dorland, “An adjoint method for gradient-based optimization of stellarator coil shapes,” Nuclear Fusion 58, 076015 (2018).
  * Shafranov and Zakharov [1972] V. Shafranov and L. Zakharov, “Use of the virtual-casing principle in calculating the containing magnetic field in toroidal plasma systems,” Nuclear Fusion 12, 599 (1972).
  * Drevlak, Monticello, and Reiman [2005] M. Drevlak, D. Monticello, and A. Reiman, “PIES free boundary stellarator equilibria with improved initial conditions,” Nuclear Fusion 45, 731 (2005).
  * Malhotra  _et al._ [2019] D. Malhotra, A. J. Cerfon, M. O’Neil, and E. Toler, “Efficient high-order singular quadrature schemes in magnetic fusion,” Plasma Physics and Controlled Fusion 62, 024004 (2019).
  * Malhotra [2019] D. Malhotra, “Boundary integral equation solver for taylor states,” https://github.com/hiddenSymmetries/virtual-casing (2019).
  * Landreman  _et al._ [2021] M. Landreman, B. Medasani, F. Wechsung, A. Giuliani, R. Jorge, and C. Zhu, “Simsopt: A flexible framework for stellarator optimization,” Journal of Open Source Software 6, 3525 (2021).
  * SIM [2021] “Simsopt,” https://github.com/hiddenSymmetries/simsopt/ (2021).
  * Princeton University [2022] Princeton University, “Stellopt,” https://github.com/PrincetonUniversity/STELLOPT (2022).
  * Hirshman and Whitson [1983] S. P. Hirshman and J. Whitson, “Steepest-descent moment method for three-dimensional magnetohydrodynamic equilibria,” The Physics of fluids 26, 3553–3568 (1983).
  * Kappel [2023] J. Kappel, “Magnetic gradient scale length auxillary dataset,” https://doi.org/10.5281/zenodo.8349408 (2023).
  * Spong [2011] D. A. Spong, “Three-dimensional effects on energetic particle confinement and stability,” Physics of Plasmas 18 (2011), 10.1063/1.3575626, 056109.
  * Anderson  _et al._ [1995] F. S. B. Anderson, A. F. Almagri, D. T. Anderson, P. G. Matthews, J. N. Talmadge, and J. L. Shohet, “The helically symmetric experiment, (HSX) goals, design and status,” Fusion Technology 27, 273–277 (1995).
  * Peterson  _et al._ [2007] J. Peterson, G. Hartwell, S. Knowlton, J. Hanson, R. Kelly, and C. Montgomery, “Initial vacuum magnetic field mapping in the compact toroidal hybrid,” Journal of fusion energy 26, 145–148 (2007).
  * Simpson  _et al._ [2001] J. Simpson, J. Lane, C. Immer, and R. Youngquist, “Simple analytic expressions for the magnetic field of a circular current loop,” Tech. Rep. (NASA Technical Report Server, 2001).
  * Imbert-Gerard, Paul, and Wright [2020] L.-M. Imbert-Gerard, E. J. Paul, and A. M. Wright, “An introduction to stellarators: From magnetic fields to symmetries and optimization,” (2020).
  * Ku and Boozer [2010] L. Ku and A. Boozer, “New classes of quasi-helically symmetric stellarators,” Nuclear Fusion 51, 013004 (2010).
  * Pedersen and Boozer [2002] T. S. Pedersen and A. H. Boozer, “Confinement of nonneutral plasmas on magnetic surfaces,” Phys. Rev. Lett. 88, 205002 (2002).
  * Alejaldre  _et al._ [1990] C. Alejaldre, J. J. A. Gozalo, J. B. Perez, F. C. Magaña, J. R. C. Diaz, J. G. Perez, A. Lopez-Fraguas, L. García, V. I. Krivenski, R. Martín, A. P. Navarro, A. Perea, A. Rodriguez-Yunta, M. S. Ayza, and A. Varias, “TJ-II project: A flexible heliac stellarator,” Fusion Technology 17, 131–139 (1990).
  * Bader  _et al._ [2021] A. Bader, D. Anderson, M. Drevlak, B. Faber, C. Hegna, S. Henneberg, M. Landreman, J. Schmitt, Y. Suzuki, and A. Ware, “Modeling of energetic particle transport in optimized stellarators,” Nuclear Fusion 61, 116060 (2021).
  * Garabedian [2008] P. R. Garabedian, “Three-dimensional analysis of tokamaks and stellarators,” Proceedings of the National Academy of Sciences 105, 13716–13719 (2008).
  * Iiyoshi  _et al._ [1999] A. Iiyoshi, A. Komori, A. Ejiri, M. Emoto, H. Funaba, M. Goto, K. Ida, H. Idei, S. Inagaki, S. Kado, _et al._ , “Overview of the large helical device project,” Nuclear Fusion 39, 1245 (1999).
  * Nelson  _et al._ [2002] B. Nelson, R. Benson, L. Berry, A. Brooks, M. Cole, P. Fogrty, P. Goranson, P. Heitzenroeder, S. Hirschman, G. Jones, J. Lyon, P. Mioduszewski, D. Monticello, D. Spong, D. Strickler, A. Ware, and D. Williamson, “Design of the quasi-poloidal stellarator experiment (QPS),” in  _Proceedings of the 19th IEEE/IPSS Symposium on Fusion Engineering. 19th SOFE (Cat. No.02CH37231)_ (2002) pp. 248–251.
  * Henneberg  _et al._ [2019] S. Henneberg, M. Drevlak, C. Nührenberg, C. Beidler, Y. Turkin, J. Loizu, and P. Helander, “Properties of a new quasi-axisymmetric configuration,” Nuclear Fusion 59, 026014 (2019).
  * Nelson  _et al._ [2003] B. Nelson, L. Berry, A. Brooks, M. Cole, J. Chrzanowski, H.-M. Fan, P. Fogarty, P. Goranson, P. Heitzenroeder, S. Hirshman, G. Jones, J. Lyon, G. Neilson, W. Reiersen, D. Strickler, and D. Williamson, “Design of the national compact stellarator experiment (NCSX),” Fusion Engineering and Design 66-68, 169–174 (2003), 22nd Symposium on Fusion Technology.
  * Nührenberg and Zille [1988] J. Nührenberg and R. Zille, “Quasi-helically symmetric toroidal stellarators,” Physics Letters A 129, 113–117 (1988).
  * P. B. Thompson for the ATF Team [1985] P. B. Thompson for the ATF Team, “The advanced toroidal facility (ATF),” Fusion Technology 8, 450–455 (1985).
  * [53] M. Drevlak, personal communication.
  * Landreman, Buller, and Drevlak [2022] M. Landreman, S. Buller, and M. Drevlak, “Optimization of quasi-symmetric stellarators with self-consistent bootstrap current and energetic particle confinement,” Physics of Plasmas 29 (2022), 10.1063/5.0098166, 082501.
  * Goodman  _et al._ [2022] A. Goodman, K. C. Mata, S. A. Henneberg, R. Jorge, M. Landreman, G. Plunk, H. Smith, R. Mackenbach, and P. Helander, “Constructing precisely quasi-isodynamic magnetic fields,” (2022).
  * Beidler  _et al._ [1990] C. Beidler, G. Grieger, F. Herrnegger, E. Harmeyer, J. Kisslinger, W. Lotz, H. Maassberg, P. Merkel, J. Nührenberg, F. Rau, J. Sapper, F. Sardei, R. Scardovelli, A. Schlüter, and H. Wobig, “Physics and engineering design for Wendelstein VII-X,” Fusion Technology 17, 148–168 (1990).
  * Sánchez  _et al._ [2023] E. Sánchez, J. Velasco, I. Calvo, and S. Mulas, “A quasi-isodynamic configuration with good confinement of fast ions at low plasma $\beta$,” Nuclear Fusion 63, 066037 (2023).
  * Liu  _et al._ [2018] H. Liu, A. Shimizu, M. Isobe, S. Okamura, S. Nishimura, C. Suzuki, Y. Xu, X. Zhang, B. Liu, J. Huang, X. Wang, H. Liu, C. Tang, D. Yin, Y. Wan, and CFQS team, “Magnetic configuration and modular coil design for the chinese first quasi-axisymmetric stellarator,” Plasma and Fusion Research 13, 3405067–3405067 (2018).
  * Bader  _et al._ [2020] A. Bader, B. J. Faber, J. C. Schmitt, D. T. Anderson, M. Drevlak, J. M. Duff, H. Frerichs, C. C. Hegna, T. G. Kruger, M. Landreman, and et al., “Advancing the physics basis for quasi-helically symmetric stellarators,” Journal of Plasma Physics 86, 905860506 (2020).
  * Drevlak  _et al._ [2014] M. Drevlak, J. Geiger, P. Helander, and Y. Turkin, “Fast particle confinement with optimized coil currents in the W7-X stellarator,” Nuclear Fusion 54, 073002 (2014).
  * Drevlak  _et al._ [2013] M. Drevlak, F. Brochard, P. Helander, J. Kisslinger, M. Mikhailov, C. Nührenberg, J. Nührenberg, and Y. Turkin, “Estell: A quasi-toroidally symmetric stellarator,” Contributions to Plasma Physics 53, 459–468 (2013).
  * Giuliani  _et al._ [2022b] A. Giuliani, F. Wechsung, G. Stadler, A. Cerfon, and M. Landreman, “Direct computation of magnetic surfaces in boozer coordinates and coil optimization for quasisymmetry,” Journal of Plasma Physics 88, 905880401 (2022b).
  * Wechsung  _et al._ [2022] F. Wechsung, M. Landreman, A. Giuliani, A. Cerfon, and G. Stadler, “Precise stellarator quasi-symmetry can be achieved with electromagnetic coils,” Proceedings of the National Academy of Sciences 119, e2202084119 (2022).
  * Jorge  _et al._ [2022b] R. Jorge, G. Plunk, M. Drevlak, M. Landreman, J.-F. Lobsien, K. Camacho Mata, and P. Helander, “A single-field-period quasi-isodynamic stellarator,” Journal of Plasma Physics 88, 175880504 (2022b).


# XNOR-former: Learning Accurate Approximations in Long Speech Transformers

###### Abstract

Transformers are among the state of the art for many tasks in speech, vision,
and natural language processing, among others. Self-attentions, which are
crucial contributors to this performance have quadratic computational
complexity, which makes training on longer input sequences challenging. Prior
work has produced state-of-the-art transformer variants with linear attention,
however, current models sacrifice performance to achieve efficient
implementations. In this work, we develop a novel linear transformer by
examining the properties of the key-query product within self-attentions. Our
model outperforms state of the art approaches on speech recognition and speech
summarization, resulting in 1 % absolute WER improvement on the
Librispeech-100 speech recognition benchmark and a new INTERVIEW speech
recognition benchmark, and 5 points on ROUGE for summarization with How2.

Index Terms—  efficient transformers, linear, speech recognition

##  1 Introduction

Transformers are among the state of the art for many tasks in speech, natural
language processing and vision, among other fields involving ordered sequences
of data. They achieve their exceptional generalization in part due to “multi-
head self-attention” – processing blocks which derive (a set of) updated
representations for each input in a sequence as a weighted sum of values
derived from all the inputs in the sequence. This computation, however, has a
computational and space complexity that is quadratic in the length of the
input sequence. As a result, it becomes challenging to operate on longer input
sequences on modern GPUs and TPUs, and training such models from scratch is
time and labor intensive.

The computational challenges arise from the formulation of self-attention. For
an input sequence of length $N$ each head of the multi-head self attention
block derives three $N$-row matrices: a value matrix $V$ representing the
latent-representation vectors for the $N$ inputs in the sequence, a query
matrix $Q$ representing the probes with which each input derives the “self-
attention” weights required to update itself, and a key matrix $K$,
representing the key contribution of each input to the computation of its own
weight in updating any input. The actual updated representations for the input
computed by the head has the form $softmax(QK^{\top})V$. The bottleneck arises
from $softmax(QK^{\top})$: both $Q$ and $K$ have $N$ rows, $QK^{\top}$
requires $O(N^{2})$ computation. This cannot be factored, since the softmax
operates on the _product_ , which must necessarily be computed before the
softmax is applied.

Recently, many works have had success in developing attention mechanisms that
approximate the standard self-attention in linear time and space complexity.

Pattern based methods [1, 2, 3, 4] save on computation by limiting the size of
the input blocks they can work on, however they potentially lose the larger
context of the input in the process. Sparsity based methods [5, 6] use low-
rank approximations of attention to reduce compuation; however they are, by
nature lossy, approaching the computational complexity of full self-attention
if losslessness is to be assured. Linear transformers[7, 8, sun2022] do
achieve linear-time and can thus operate on long sequences. However, current
linear transformers still have significant performance degradations compared
to the standard self-attention.

In this paper, we attempt to bridge this gap by developing the _XNOR self-
attention_ and correspondingly, the _XNOR-former_. Our work falls into the
pool of work on linear transformers that use the kernel trick to compute self-
attentions in linear time. We observe that
$\sigma(pq)\approx\sigma(p)\bar{\oplus}\sigma(q)$, where $\sigma()$ is the
sigmoid function, and $\bar{\oplus}$ is the $XNOR$ operator for Boolean
variables, but more generally can be expressed as
$x\bar{\oplus}y=xy+(1-x)(1-y),~{}~{}x,y\in[0,1]$. This enables us to decompose
the softmax over the product $QK^{\top}$ in terms of the $XNOR$ of the softmax
over the individual terms $Q$ and $K^{\top}$. This reduces the computation of
the attention, and in fact the overall self-attention update, to linear time.

Also, native transformer self-attentions are position agnostic, and it has
been found advantageous to encode positional information within self-
attention[7, 9, 1] through positional embeddings. However standard mechanisms
are unsuitable for linear transformers since they require the explicit
computation of attention weights. Inspired by [7, 9], we explore the use of
rotational and cosine relative positional embeddings and integrate them with
the proposed XNOR-former.

We evaluate our proposed approach on Speech recognition and summarization
tasks – tasks which are characterized by very long input sequences (typically
10-100x longer than the image- and text-based inputs generally associated with
transformers). Our proposed approach outperforms prior kernel methods on two
speech recognition benchmarks and end-to-end speech summarization, while
maintaining linear-in-length computational and memory expense.

The rest of this paper is organized as follows: Section 2 describes the
proposed approach, Section 3 explains design and evaluation of our
experiments, Section 4 details our experimental results and ablation studies.

##  2 X-NOR Self Attention

###  2.1 Standard Self-Attention

Consider an input sequence $X=[X_{1};X_{2};\cdots;X_{N}]$ (representing a
stacking of $N$ row vectors) of length $N$ to a multi-head self-attention
module. This module comprises several self-attention “heads”. A self-attention
head [10] computes, from the input $X$, a key matrix
$K=XW_{K}\in\mathbb{R}^{N\times d}$, a query matrix
$Q=XW_{Q}\in\mathbb{R}^{N\times d}$, and a value matrix
$V=XW_{V}\in\mathbb{R}^{N\times d}$, where $W_{K},~{}W_{Q}$ and $W_{V}$ are
its parameters. The output of this self-attention
$\mathcal{O}\in\mathbb{R}^{N\times
d}=[\mathcal{O}_{1};\mathcal{O}_{2};\cdots;\mathcal{O}_{N}]$ is then computed
as

$\mathcal{O}=Sm\Big{(}\frac{QK^{\top}}{\sqrt{d}}\Big{)}V$ (1)

where $Sm()$ represents the softmax operator, applied row-wise to its matrix
argument. The computational complexity of computing $QK^{\top}$ (and hence
$\mathcal{O}$) is quadratic in $N$. This is not viable to compute on long
input sequences.

To facilitate simplification of the computation of self attention, the softmax
may be replaced by a normalized similarity measure:

$\mathcal{O}_{i}=\frac{\sum_{j=1}^{N}S(Q_{i},K_{j})}{\sum_{j}S(Q_{i},K_{j})}.V_{j}$
(2)

where $S(Q_{i},K_{j})$ is the similarity between the $i^{\rm th}$ query and
the $j^{\rm th}$ key. In fact, the softmax is a special case of the above
where $S(Q_{i},K_{j})=\exp(Q_{i}K_{j}^{\top})$.

###  2.2 Linear Transformers

The $O(N^{2})$ complexity of softmax-based attention results from inner-
product term $Q_{i}K_{j}^{\top}$ that occurs within the similarity term
$S(Q_{i},K_{j})\sim\exp(Q_{i}K_{j}^{\top})$ for the softmax and prevents
factorization of the computation. Linear transformers replace the exponent
over inner products by a factored “product over Kernels” similarity measure:
$S(\mathcal{Q},\mathcal{K})=\phi(\mathcal{Q})\phi(\mathcal{K})^{\top}$, where
$\phi()$ is a “Kernel” function. Using the associative law of matrix
multiplication, this permits the computation to be of the following form,
which can be computed in linear time.

$\mathcal{O}_{i}=\sum_{j=1}^{N}\frac{\phi(Q_{i})(\phi(K_{j})V_{j})}{\sum_{j}\phi(Q_{i})\phi(K_{j})}.$
(3)

Prior works have used Gaussian Kernels[11], ELU based kernels[8], and ReLU
kernels[7]. The closest to our proposed approach is the softmax Kernel, which
uses $\phi()=Sm()$. Though these succeed in reducing the computational
overhead, they often result in undesirable performance reductions, compared to
the full softmax-based attention.

###  2.3 Examining the Sigmoid of the Product

Our objective is to close the gap between linear transformers and the full
transformer formulation without losing the computational advantage of linear
time. As mentioned earlier, the main impediment to linear-time factorization
is the softmax over the product in the conventional transformer.

Our proposal builds on the following insight: consider the simplest version of
a softmax, which is the sigmoid. $\sigma(xy)$, the sigmoid of the product of
two variables $x$ and $y$, has a distinct XNOR-like ($\bar{oplus}$-like)
behavior as shown in Figure 1 a. This cannot be factored because the XNOR
cannot be modelled by a linear boundary. However, analogously to the
decomposition of the XNOR, $\sigma(xy)$ can be expressed as the sum of two
bilinear terms:

$\sigma(xy)\approx\sigma(x)\bar{\oplus}\sigma(y)=\sigma(x)\sigma(y)+(1-\sigma(x))(1-\sigma(y))$
(4)

Figure 1b shows the approximation $\sigma(x)\bar{\oplus}\sigma(y)$. Except for
a narrow region near the axes, the error is minimal.

Fig. 1: (a) Left: $\sigma(xy)$ for $x,y\in(-100,100)$. (b) Right: The
approximation $\sigma(x)\bar{\oplus}\sigma(y)$.

This leads us to our formulation for the XNOR-former.

###  2.4 X-NOR Self-Attention

Drawing from the X-NOR approximation of the sigmoid, we now extend this to
approximate the softmax product in Equation 2 as

$S(Q_{i},K_{j})=w_{1}Sm(Q_{i})Sm(K_{j})+w_{2}Sm^{\prime}(Q_{i})Sm^{\prime}(K_{j}),$
(5)

where, as before, $Sm$ represents the softmax operator and $Sm^{\prime}=1-Sm$
represents its complement. The weights $w_{1}$ and $w_{2}$ account for the
fact that the variable is of length $N$ as opposed to binary.

This gives us the following XNOR factorization of self-attention, which allows
it to be computed in linear time and space complexity.

$\mathcal{O}_{i}=\frac{\sum_{j=1}^{N}w_{1}Sm(Q_{i})(Sm(Kq_{j})V_{j})+w_{2}Sm^{\prime}(Q_{i})(Sm^{\prime}(K_{j})V_{j})}{\sum_{j}w_{1}Sm(Q_{i})Sm(K_{j})+w_{2}Sm^{\prime}(Q_{i})Sm^{\prime}(K_{j})}$
(6)

For simple XNOR self-attention $w_{1}=w_{2}=1$. We also make the weights
learnable. We refer to this variant as the weighted-XNOR or W-XNOR self-
attention.

###  2.5 Positional Encodings

Equation 6, which captures feature-based attention weights, however does not
consider the positional relation of the inputs in a sequence – reordering
inputs will not change their weights.

To remedy this, we require positional embeddings within the attention
mechanism to incorporate awareness of the relative position $i-j$ or absolute
positions $i,j$ within the sequence. Relative positional embeddings were
demonstrated to outperform absolute embeddings in [1, shaw2018self]. Learnable
relative positional encodings, introduced in [1] rely on computing the full
attention matrix, and hence cannot be used with linear attentions since the
full attention weights are not computed explicitly.

To embed positional information into the XNOR self-attention, or in general
for linear transformers that use the kernel trick, candidate encodings must
have two characteristics:

  1. 1.

They should depend on relative position $i-j$, i.e., $P(i,j)=f(i-j)$

  2. 2.

They should be factorable into positional terms i.e., $P(i,j)=g(i)h(j)$

Using such a definition, the terms $Sm(Q_{i})$, $Sm(K_{j})$,
$Sm^{\prime}(Q_{i})$ and $Sm^{\prime}(K_{j})$ in Equation 6 can be modified to
$\hat{Sm}(Q_{i})=Sm(Q_{i})g(i)$, $\hat{Sm}(K_{j})=Sm(K_{j})h(j)$,
$\hat{Sm}^{\prime}(Q_{i})=Sm^{\prime}(Q_{i})g(i)$ and
$\hat{Sm}^{\prime}(K_{j})=Sm^{\prime}(K_{j})h(j)$ respectively.
$\mathcal{O}_{i}$ can now be computed using the modified values to incorporate
position information.

Based on prior work, we adopt two formulations for the positional encoding: a
cosine form [7], and rotational positional encodings [9]. The cosine
positional encodings are realized using Ptolemy’s theorem as

$\begin{split}P(i,j)&=cos\Big{(}\frac{\pi(i-j)}{2M}\Big{)}\\\
&=cos\Big{(}\frac{\pi i}{2M}\Big{)}cos\Big{(}\frac{\pi
j}{2M}\Big{)}+sin\Big{(}\frac{\pi i}{2M}\Big{)}sin\Big{(}\frac{\pi
j}{2M}\Big{)}\end{split}$ (7)

Here $M$ is the maximum sequence length in the batch of examples to ensure
that the the $cos(x)$ is computed within $0<x<\pi/2$

Rotational positional encoding encode absolute positional information within
rotation matrices. The product of rotation matrices encodes relative
information. Equation 8 defines the rotational positional embedding, where
$\mathbb{R}^{d}_{\Theta,j}$ is a d-dimensional rotation matrix that encodes
the absolute position $j$. We refer interested readers to [9] for a complete
mathematical treatment.

$P(i,j)=\mathbb{R}^{d}_{\Theta,n-m}=\mathbb{R}^{d}_{\Theta,m}\mathbb{R}^{d}_{\Theta,n}$
(8)

##  3 Experimental Setup

We evaluate the proposed model on two speech tasks: speech recognition and
speech summarization. Speech signals are typically very long (comprising
sequences of thousands or even tens of thousands of vectors), and are well
suited to bring out the effectiveness of our proposed solutions.

###  3.1 Speech Recognition

End to End Speech Recognition is a sequence transduction task which maps a
sequence of input speech frames to sequences of language tokens. Speech
recognition is usually performed on individual utterances, with input
sequences that are a few hundred vectors long, and output sequences that are
generally less than 100 output tokens.

We use state-of-the-art conformer based sequence models with attention to
perform speech recognition. Conformers [12] are variants of the standard
transformer architecture that contain macaron-style feedforward layers that
sandwich a self-attentive and convolutional layer. Our models have a conformer
encoder and a standard transformer decoder[10]. Our baseline uses Multihead
Self-Attention with Relative Positional Encodings[1]. The proposed approach
replaces these with XNOR and W-XNOR self-attentions as formulated in Section
2.4.

We perform our experiments on two corpora: Librispeech-100 [13] and
INTERVIEW-300. Librispeech is a public corpus with read speech from audio-
books. INTERVIEW is a corpus of two-party radio conversations between a host
and guest collected by NPR [14, 15]. INTERVIEW-300 is a 300 episode subset of
the whole corpus which contains around 150h of speech. To the best of our
knowledge, ours is the first work to report speech recognition results on the
INTERVIEW corpus. Speech Recognition performance is evaluated using Word Error
Rate (WER) on the test sets.

####  3.1.1 Librispeech-100 Models

Our model uses a conformer encoder with 4-fold convolutional subsampling
followed by 12 encoder layers with feed-forward dimension 1024 and 4 attention
heads. The transformer decoder has 6 layers with feed-forward dimension 2048
and 4 attention heads. Three fold speed perturbation is used for training with
speeds 0.9x, 1.0x and 1.1x. SpecAug[16] is used with time warping, frequency
masking, and time masking. We use 80-dimensional filter-bank features
extracted at 100 frames/sec with 25 ms windows. We use the Adam optimizer with
peak learning rate 0.002, and 15,000 warmup steps.

The models are trained with joint Connectionist Temporal Classification
(CTC)-Attention [17] with the weight for CTC training set to 0.3. Inference is
performed using output synchronous beam search with CTC weight of 0.3, and
beam width 20.

####  3.1.2 INTERVIEW-300 Models

The Interview-300 model uses a conformer encoder with 2-fold convolutional
subsampling followed by 6 encoder layers with feed-forward dimension 2048, and
8 attention heads. The transformer decoder has 6 layers with feed-forward
dimension 2048 and 8 attention heads. HUBERT [18] features from a Librivox
model are used as inputs to our models. The models are trained with joint CTC-
Attention [17] with the weight for CTC training set to 0.3. Inference is
performed using output synchronous beam search with CTC weight of 0.3, and
beam width 20.

###  3.2 Speech Summarization

Abstractive Speech summarization is the task of directly extracting an
abstractive text summary from speech in videos or meetings. How2 [19] is a
dataset of instructional YouTube videos, manually annotated for transcript,
Portuguese translation and abstractive summary. For this task, generally,
multiple utterances of speech are used as input. [20] finds that the
summarization task uses 35x longer input sequences for summarization when
compared to speech recognition on the How2 dataset of instructional videos.

We utilize a similar two-stage training mechanism where utterance level speech
recognition is used as supervised pre-training. The model is then fine-tuned
on video-level speech for summarization. To reduce compute, we use the 300h
subset of the How2 data for pre-training and the 2000h subset for
summarization performance.Therefore, our summarization performance is not
comparable with previous approaches[20].

The conformer encoder uses 2-fold convolutional subsampling followed by 6
encoder layers with feed-forward dimension 2048, and 8 attention heads. The
transformer decoder has 6 layers with feed-forward dimension 2048 and 8
attention heads. ASR models are trained with joint CTC-Attention [17] with the
weight for CTC training set to 0.3. The videos are trimmed to 100s for the
video-level speech tasks. HUBERT [18] features from a Librivox model are used
as inputs to our models.

##  4 Experimental Results

Table 1: WER (%) on test-other and test-clean sets from the Librispeech evaluation of models trained on Librispeech-100. Higher numbers are better  Model Description | Kernel |  | Pos.
---
Enc.
| Test-clean
---
WER. (%)
| Test-other
---
WER. (%)
Multi-head Attention | None | RelPos | 6.6 | 17.3
Linear Transformer [8] | ELU +1 | None | 9.2 | 23.6
Cosformer [7] | ReLU | cos | 10.8 | 26.3
Softmax Kernel | Softmax | None | 10.5 | 29.0
| cos | 9.0 | 23.5
XNOR-former | XNOR | None | 8.5 | 21.9
WXNOR-former |  XNOR Softmax | None | 8.7 | 23.0
| cos | 8.1 | 21.3
| RoPE | 8.7 | 23.6
Table 2: Results of our models on the speech summarization task for the
How2-2000h data. Abstractive Summarization is evaluated using Rouge scores
(R-1,R-2,R-L), METEOR(MTR) scores for content, and BERTScore(BERT) for
semantic relevance. Higher numbers are better.

Model | R-1 | R-2 | R-L | MTR | BERTS
---|---|---|---|---|---
W-XNOR | 53.58 | 34.23 | 47.92 | 25.47 | 89.79
Cosformer | 47.91 | 27.85 | 42.49 | 21.26 | 89.36

Table 3: WER (%) performance on the INTERVIEW-300 dataset. Lower WER is better Self-Attention Type |  | Valid
---
Acc
| Test
---
WER (%)
Standard MHA with RelPos | 89.3 | 15.2
Cosformer | 87.1 | 15.6
XNORformer | 88.0 | 14.8
WXNORformer | 87.5 | 14.6

###  4.1 Speech Recognition

For Librispeech-100, we begin by training models using standard multi-head
attention and then on comparable linear transformers[8, 7]. From Table 1 , we
observe that there exists a substantial gap between the WERs of the original
transformer and linear transformer variants. If we use a softmax kernel with
no positional information, and with cosine embeddings as formulated in
Equation 7, we observe that the cosine embeddings improve performance, and the
softmax kernel performs as well as the linear transformer[8].

On the other hand, the proposed W-XNOR self-attention with cosine positional
embeddings outperforms the best linear transformer by 5 - 10 % relative on the
test-clean and test-other sets. Further, the cosine positional embeddings
appear to be more useful than rotational positional embeddings.

On the INTERVIEW-300 set, we compare the standard multi-head attention with
the Cosformer and our proposed XNOR variants. The proposed approach improves
speech recognition performance by 1 % absolute over the state of the art
Cosformer, while slightly outperforming the standard multi-head attention
model as well.

###  4.2 Speech Summarization-How2

On End-to-End Speech Summarization, Table 2 highlights the results. We observe
that the proposed W-XNOR self-attention based model outperforms the Cosformer
on a very long input sequence (10k) task without extensive hyperparameter
tuning. The XNOR model has a better performance in terms of the ROUGE-L,
METEOR and BERTScore metrics, which demonstrates that the proposed W-XNOR
model produces more coherent, semantically relevant, content rich summaries in
comparison to the baseline.

Fig. 2: Plot showing compute times for different sequence lengths across
linear transformer variants, and full softmax attention Fig. 3: Plot showing
compute times for different sequence lengths across linear transformer
variants

###  4.3 Speed and Scaling

Figure 2 demonstrates the increase in compute time when the sequence length is
increased from 1 to 30k over different attention variants. We observe that the
linear self-attention variants significantly improve compute time over
standard multihead attention. Figure 3 presents another view of the same plot
with only linear self-attention variants considered. We note that for very
long sequences, the softmax kernel is the fastest, followed by ELU[8], the
proposed XNOR self-attention, and then the Cosformer [7]. It is clear that our
proposed model yields significant performance gains on ASR and speech
summarization with speeds significantly faster than standard self-attention,
and comparable to other linear transformers.

##  5 Conclusion and Future Work

In this work, we address the challenge of quadratic computational complexity
in transformer self-attentions which makes them intractable over long input
sequences. Self-attention is formulated as the softmax over an $N\times N$
matrix, and can be linearized using the kernel trick and associativity of
matrix multiplication.

However there exists a significant gap in performance between full softmax
attention and linear transformers. We propose develop the _XNOR self-
attention_ , and correspondingly the _WXNOR transformer_. The proposed
approach outperforms previous linear transformers on the speech recognition
and summarization benchmarks. We also demonstrate that the performance
improvements are achieved while maintaining slightly better or comparable
compute efficiency.

## References

  * [1] Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc Le, and Ruslan Salakhutdinov,  “Transformer-XL: Attentive language models beyond a fixed-length context,”  in Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Florence, Italy, July 2019, pp. 2978–2988, Association for Computational Linguistics.
  * [2] Iz Beltagy, Matthew E. Peters, and Arman Cohan,  “Longformer: The long-document transformer,”  CoRR, vol. abs/2004.05150, 2020.
  * [3] Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, and Amr Ahmed,  “Big bird: Transformers for longer sequences,”  in Advances in Neural Information Processing Systems, H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, Eds. 2020, vol. 33, pp. 17283–17297, Curran Associates, Inc.
  * [4] Jack W. Rae, Anna Potapenko, Siddhant M. Jayakumar, and Timothy P. Lillicrap,  “Compressive transformers for long-range sequence modelling,”  CoRR, vol. abs/1911.05507, 2019.
  * [5] Yunyang Xiong, Zhanpeng Zeng, Rudrasis Chakraborty, Mingxing Tan, Glenn Fung, Yin Li, and Vikas Singh,  “Nyströmformer: A nyström-based algorithm for approximating self-attention,”  2021\.
  * [6] Sinong Wang, Belinda Li, Madian Khabsa, Han Fang, and Hao Ma,  “Linformer: Self-attention with linear complexity,”  arXiv preprint arXiv:2006.04768, 2020.
  * [7] Zhen Qin, Weixuan Sun, Hui Deng, Dongxu Li, Yunshen Wei, Baohong Lv, Junjie Yan, Lingpeng Kong, and Yiran Zhong,  “cosformer: Rethinking softmax in attention,”  in International Conference on Learning Representations, 2022.
  * [8] A. Katharopoulos, A. Vyas, N. Pappas, and F. Fleuret,  “Transformers are rnns: Fast autoregressive transformers with linear attention,”  in Proceedings of the International Conference on Machine Learning (ICML), 2020.
  * [9] Jianlin Su, Yu Lu, Shengfeng Pan, Bo Wen, and Yunfeng Liu,  “Roformer: Enhanced transformer with rotary position embedding,” 2021\.
  * [10] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, L ukasz Kaiser, and Illia Polosukhin,  “Attention is all you need,”  in Advances in Neural Information Processing Systems, I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, Eds. 2017, vol. 30, Curran Associates, Inc.
  * [11] Krzysztof Marcin Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Quincy Davis, Afroz Mohiuddin, Lukasz Kaiser, David Benjamin Belanger, Lucy J Colwell, and Adrian Weller,  “Rethinking attention with performers,”  in International Conference on Learning Representations, 2021.
  * [12] Pengcheng Guo, Florian Boyer, Xuankai Chang, Tomoki Hayashi, Yosuke Higuchi, Hirofumi Inaguma, Naoyuki Kamo, Chenda Li, Daniel Garcia-Romero, Jiatong Shi, Jing Shi, Shinji Watanabe, Kun Wei, Wangyou Zhang, and Yuekai Zhang,  “Recent developments on espnet toolkit boosted by conformer,”  in ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2021, pp. 5874–5878.
  * [13] Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur,  “Librispeech: An asr corpus based on public domain audio books,”  in 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2015, pp. 5206–5210.
  * [14] Bodhisattwa Prasad Majumder, Shuyang Li, Jianmo Ni, and Julian McAuley,  “Interview: Large-scale modeling of media dialog with discourse patterns and knowledge grounding,”  in Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), Online, Nov. 2020, pp. 8129–8141, Association for Computational Linguistics.
  * [15] Chenguang Zhu, Yang Liu, Jie Mei, and Michael Zeng,  “MediaSum: A large-scale media interview dataset for dialogue summarization,”  in Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Online, June 2021, pp. 5927–5934, Association for Computational Linguistics.
  * [16] Daniel S. Park, William Chan, Yu Zhang, Chung-Cheng Chiu, Barret Zoph, Ekin Dogus Cubuk, and Quoc V. Le,  “Specaugment: A simple augmentation method for automatic speech recognition,”  in INTERSPEECH, 2019.
  * [17] Suyoun Kim, Takaaki Hori, and Shinji Watanabe,  “Joint ctc-attention based end-to-end speech recognition using multi-task learning,”  in 2017 IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2017, New Orleans, LA, USA, March 5-9, 2017. 2017, pp. 4835–4839, IEEE.
  * [18] Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed,  “Hubert: Self-supervised speech representation learning by masked prediction of hidden units,”  IEEE ACM Trans. Audio Speech Lang. Process., vol. 29, pp. 3451–3460, 2021.
  * [19] Ramon Sanabria, Ozan Caglayan, Shruti Palaskar, Desmond Elliott, Loïc Barrault, Lucia Specia, and Florian Metze,  “How2: a large-scale dataset for multimodal language understanding,”  in Proceedings of the Workshop on Visually Grounded Interaction and Language (ViGIL). NeurIPS, 2018.
  * [20] Roshan Sharma, Shruti Palaskar, Alan W Black, and Florian Metze,  “End-to-end speech summarization using restricted self-attention,”  in ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2022, pp. 8072–8076.