The other $n-2$ fields are also associated with a Fock spectrum of light
quasi-particles. For large angular momentum $\ell\gg 1$ we can drop the terms
in the third line of (A.9) and we obtain (2.27) in the main text.

Due to the terms with a single time derivative, it is hard to compute in full
generality the spectrum associated with the fields in the second and third
line of (A.9). A simplification occurs when the first and second term in the
last line of (A.9) vanish. This happens for instance in theories in which
there is a unique Cartan charge and a symmetry acting as $\pi\rightarrow-\pi$
on the Goldstone boson, e.g. a model with a single $U(1)$ charge invariant
under charge conjugation. In this case the spectrum of fluctuations results
into standard relativistic dispersion relations, cf. (2.28).

##  Appendix B Moduli spaces and $\epsilon$-expansion for $3d$ $\mathcal{N}=1$
SUSY theories

Three-dimensional $\mathcal{N}=1$ SUSY has two supercharges, and as a result
SUSY is much less powerful compared to the more extensively-studied cases with
four and more supercharges. In particular, $3d$ $\mathcal{N}=1$ theories do
not possess a continuous R-symmetry, and as a result some famous features
which are attributed to SUSY like non-renormalization theorems and protected
operator dimensions do not oocur. Nonetheless, a discrete R-symmetry is
sometimes available, and can be enough to provide some exact results. One
particular example involves a $\mathbb{Z}_{2}$ R-symmetry which in some cases
protects a moduli space from being lifted to all orders in perturbation theory
Affleck:1982as ; Gremm:1999su ; Gukov:2002es , which we now review following
Gaiotto:2018yjh .

Interactions which lift moduli spaces in $3d$ $\mathcal{N}=1$ theories take
the form of a superpotential373737Our $3d$ $\mathcal{N}=1$ superspace
conventions follow Gates:1983nr . In particular we have bosonic coordinates
$x^{\mu}$ and Majorana Grassmanian coordinates $\theta_{\alpha}$, and a basic
scalar multiplet consists of a real scalar and Majorana fermion.

$\int d^{2}\theta W(\Phi_{i})\;,$ (B.1)

with $\Phi_{i}$, $i=1,...,N$ some real scalar superfields. A $\mathbb{Z}_{2}$
R-symmetry acts as $d^{2}\theta\to-d^{2}\theta$, and so it appears only if one
can assign a transformation $W\to-W$ to the superpotential under it. If this
symmetry appears, it severely restricts the terms which can appear in $W$ once
quantum corrections are taken into account.

One immediate example of a $3d$ $\mathcal{N}=1$ theory with an exact moduli
space is $\mathcal{N}=1$ SQED, consisting of a gauge multiplet coupled to
$N_{f}$ complex scalar multiplets $\Phi_{i}$ of charge 1 and no Chern-Simons
term. Classically the theory has a moduli space parametrized by the vev of
$\Phi_{i}$ up to an overall phase rotation, so that the moduli space is
$\mathbb{CP}^{N_{f}-1}$. The theory possesses a $\mathbb{Z}_{2}$ R-symmetry
which acts on $\Phi_{i}$ and which prevents the appearance of any
superpotential under quantum corrections. Indeed, gauge invariance and a
global $SU(N_{f})$ symmetry forces the superpotential to be a function of
$|\Phi_{i}|^{2}$, but all such terms are even under the $\mathbb{Z}_{2}^{R}$
and so cannot be generated. The moduli space is thus exact quantum-
mechanically, at least to all orders in perturbation theory. The absence of a
Chern-Simons term is crucial here, since otherwise the $\mathbb{Z}_{2}$
R-symmetry is explicitly broken, although there are examples where an
R-symmetry is emergent in the IR and which leads to a moduli space
Choi:2018ohn ; Aharony:2019mbc ).

A particularly simple example is a WZ model which consists of three real
scalar superfields and the superpotential

$W=\frac{g}{2}ABC\;.$ (B.2)

At the classical level this theory has a moduli space where one of the three
fields $A,B,C$ has a vev while the other do not. The theory also has a
$\mathbb{Z}_{2}^{R}$ symmetry which acts as $A\to-A$, together with an $S_{3}$
permutation symmetry. The combination of these symmetries protects the moduli
space to all orders in perturbation theory, since it constrains the effective
action to include only terms of the form

$W_{\text{eff}}\supset ABCf(A^{2},B^{2},C^{2})\;,$ (B.3)

for some function $f$.

###  B.1 $\epsilon$-expansion for $3d$ $\mathcal{N}=1$

There is an immediate obstruction to studying $3d$ $\mathcal{N}=1$ theories in
the $\epsilon$-expansion from $4d$: $3d$ $\mathcal{N}=1$ theories have two
supercharges, while $4d$ SUSY theories cannot have less than 4. A workaround
for WZ theories was proposed in Fei:2016sgs ; ThomasSeminar . Consider a
general $3d$ $\mathcal{N}=1$ WZ theory consisting of scalar multiplets
$\Psi_{i}$ for $i=1,...,N_{\Psi}$. In components these consist of real Bosons
$\phi_{i}$ and Majorana Fermions $\psi_{i}$, and the Lagrangian takes the form

$\mathcal{L}=\frac{1}{2}(\partial_{\mu}\phi_{i})^{2}-\frac{i}{2}\psi_{i}^{\alpha}\partial_{\alpha\beta}\psi_{i}^{\beta}-\frac{i}{2}\partial_{i}\partial_{j}W\psi^{\alpha}_{i}\psi_{\alpha
j}-\frac{1}{2}(\partial_{i}W)^{2}\;,$ (B.4)

with $W(\Psi_{i},\bar{\Psi}_{i})$ the superpotential. To approach this theory
in the $\epsilon$-expansion, we instead consider a generalization of the
Lagrangian with an additional flavor index $a=1,...,N_{f}$ for the fermions:

$\mathcal{L}=\frac{1}{2}(\partial_{\mu}\phi_{i})^{2}-\frac{i}{2}\psi_{ai}^{\alpha}\partial_{\alpha\beta}\psi_{ai}^{\beta}-\frac{i}{2}\partial_{i}\partial_{j}W\psi^{\alpha}_{ai}\psi_{\alpha
aj}-\frac{1}{2}(\partial_{i}W)^{2}\;.$ (B.5)

This Lagrangian is non-SUSY, except for the case $N_{f}=1$ which reduces to
(B.4). The advantage of stduying this generalized theory is that for $N_{f}$ a
multiple of 2, we can repackaging the $3d$ two-component Majorana fermions
into $N_{f}/2$ $4d$ four-component Majorana fermions, which allows us to study
the theory in the $\epsilon$-expansion from $4d$. In our case we work with
$N_{f}$ a multiple of 4, and repackage the fermions into $N_{f}/4$ $4d$ Dirac
Fermions. The corresponding Lagrangian in $4d$ is

$\mathcal{L}=\frac{1}{2}(\partial_{\mu}\phi_{i})^{2}+i\bar{\psi}_{bi}\not{\partial}\psi_{bi}-\partial_{i}\partial_{j}W\bar{\psi}_{bi}\psi_{bj}-\frac{1}{2}(\partial_{i}W)^{2}\;,$
(B.6)

where we sum over $b=1,...,N_{f}/4$. The Lagrangian (B.6) can now be studied
in the $\epsilon$-expansion as usual, and we can compute various CFT data as a
function of $N_{f}$. Eventually we set $N_{f}=1$ which corresponds to the $3d$
$\mathcal{N}=1$ theory of interest. This trick has been tested for various
$3d$ $\mathcal{N}=1$ theories Fei:2016sgs ; Liendo:2021wpo ; Benini:2018bhk ;
Benini:2018umh  and provided excellent agreement with other methods.

##  Appendix C One-loop scaling dimension in the $5$-field WZ model

The Lorentzian action for the model (3.11) on $\mathbb{R}\times S^{d-1}$ in
$d=4-\epsilon$ dimensions reads:

$\begin{split}\mathcal{L}&=|\partial x|^{2}-m_{d}^{2}|x|^{2}+|\partial
y|^{2}-m_{d}^{2}|y|^{2}+\frac{1}{2}(\partial
a)^{2}-\frac{m_{d}^{2}}{2}a^{2}\\\
&+i\bar{\tilde{a}}\not{\partial}\tilde{a}+i\bar{\tilde{x}}\not{\partial}\tilde{x}+i\bar{\tilde{x}}_{c}\not{\partial}\tilde{x}_{c}+i\bar{\tilde{y}}\not{\partial}\tilde{y}+i\bar{\tilde{y}}_{c}\not{\partial}\tilde{y}_{c}-\frac{1}{4}g^{2}a^{2}(|x|^{2}+|y|^{2})-\frac{g^{2}}{8}(|x|^{2}-|y|^{2})^{2}\\\
&-\frac{g}{2}a\left(\bar{\tilde{x}}\tilde{x}+\bar{\tilde{x}}_{c}\tilde{x}_{c}-\bar{\tilde{y}}\tilde{y}-\bar{\tilde{y}}_{c}\tilde{y}_{c}\right)-\frac{g}{2}\left[\bar{\tilde{a}}\left(\tilde{x}_{c}x+\tilde{x}x^{*}-\tilde{y}_{c}y-\tilde{y}y^{*}\right)+c.c.\right]\,,\end{split}$
(C.1)

where tilde’s denote Dirac fields and we neglected the fermion multiplicity
index. Note that to each of the charged scalars $x,\,y$ correspond two Dirac
fields, $(\tilde{x},\tilde{x}_{c})$ and $(\tilde{y},\tilde{y}_{c})$, with
opposite charge.

To study fluctuations around the saddle-point (3.15), we define $r_{x/y}$ and
$\pi_{x/y}$ via

$x=\frac{(v_{x}+r_{x})}{\sqrt{2}}e^{-i\mu_{x}t-i\pi_{x}}\,,\qquad
y=\frac{(v_{y}+r_{y})}{\sqrt{2}}e^{-i\mu_{y}t-i\pi_{y}}\,,$ (C.2)

and we make a field redefinitions to make the fermions neutral

$\tilde{x}\rightarrow
e^{-i\mu_{x}t-i\pi_{x}t}\tilde{x}\,,\qquad\tilde{y}\rightarrow
e^{-i\mu_{y}t-i\pi_{y}t}\tilde{y}\,,\qquad\tilde{x}_{c}\rightarrow
e^{+i\mu_{x}t+i\pi_{x}t}\tilde{x}_{c}\,,\qquad\tilde{y}_{c}\rightarrow
e^{i\mu_{y}t+i\pi_{y}t}\tilde{y}_{c}\,.$ (C.3)

Specializing to the solution (3.19), we find the bosonic and fermionic
quadratic actions:

$\displaystyle\mathcal{L}^{(2)}_{bos}$
$\displaystyle=\sum_{a=x,y}\left[\frac{1}{2}(\partial
r_{a})^{2}+\frac{1}{2}(\partial\pi_{a})^{2}+2m_{d}r_{a}\dot{\pi}_{a}-\frac{1}{4}M^{2}r_{a}^{2}\right]-\frac{M^{2}}{2}r_{x}r_{y}+\frac{1}{2}(\partial
a)^{2}-\frac{m_{d}^{2}+M^{2}}{2}a^{2}$ $\displaystyle\mathcal{L}^{(2)}_{fer}$
$\displaystyle=i\bar{\tilde{x}}\left(\not{\partial}-i\gamma^{0}m_{d}\right)\tilde{x}+i\bar{\tilde{x}}_{c}\left(\not{\partial}+i\gamma^{0}m_{d}\right)\tilde{x}_{c}+i\bar{\tilde{y}}\left(\not{\partial}-i\gamma^{0}m_{d}\right)\tilde{y}+i\bar{\tilde{y}}_{c}\left(\not{\partial}+i\gamma^{0}m_{d}\right)\tilde{y}_{c}$
$\displaystyle+i\bar{\tilde{a}}\not{\partial}\tilde{a}-\frac{M}{2}\left[\bar{\tilde{a}}\left(\tilde{x}_{c}+\tilde{x}-\tilde{y}_{c}-\tilde{y}\right)+c.c.\right]\,,$
(C.4)

where we defined

$M=\frac{g|v_{x}|}{\sqrt{2}}\,.$ (C.5)

It is simple to obtain the spectrum of bosonic fluctuations. We find

$\displaystyle\omega_{B,1/2}(\ell)$
$\displaystyle=\sqrt{J_{\ell}^{2}+m_{d}^{2}}\mp m_{d}\,,$ (C.6)
$\displaystyle\omega_{B,3/4}(\ell)$
$\displaystyle=\sqrt{J_{\ell}^{2}+2m_{d}^{2}+\frac{M^{2}}{2}\mp\sqrt{4J_{\ell}^{2}m_{d}^{2}+\left(2m_{d}^{2}+\frac{M^{2}}{2}\right)^{2}}}\,,$
$\displaystyle\omega_{B,5}(\ell)$
$\displaystyle=\sqrt{J_{\ell}^{2}+m_{d}^{2}+M^{2}}\,,$

where $J_{\ell}^{2}=\ell(\ell+d-2)/R^{2}$. Note that the first three
dispersion relations coincide with the EFT result (3.30) when expanded for
$M^{2}\gg J_{\ell}^{2},1/R^{2}$, while $\omega_{B,4}$ and $\omega_{B,5}$
describe gapped modes.

To obtain the spectrum of fermions, it is convenient to proceed as in
Sharon:2020mjs ; Antipin:2022naw  and write the Dirac fields in terms of Weyl
spinors. Expanding the latter in terms of spinor harmonics Camporesi:1995fb ,
we find that the spectrum contains four modes with dispersion relation

$\omega^{(\pm)}_{F,1}(\ell)=\omega^{(\pm)}_{F,2}(\ell)=p_{F}(\ell)\pm
m_{d}\,,$ (C.7)

where $p_{F}(\ell)=\left(\ell+\frac{d-1}{2}\right)/R$. The other six
dispersion relations are given by the positive roots, each counted twice, of
the following cubic equation for $\omega^{2}$:

$\omega^{6}-\omega^{4}\left[2\left(M^{2}+m_{d}^{2}\right)+3p_{F}^{2}(\ell)\right]+\omega^{2}\left[3p_{F}^{4}+4M^{2}p_{F}^{2}(\ell)+\left(M^{2}+m_{d}^{2}\right)^{2}\right]\\\
-p_{F}^{2}(\ell)\left[M^{2}-m_{d}^{2}+p_{F}^{2}(\ell)\right]^{2}=0\,.$ (C.8)

Since the polynomial in (C.8) is itself the product of two real cubic
polynomials in $\omega$ (identical up to the flip $\omega\rightarrow-\omega$)
with positive discriminant, (C.8) admits three real and positive solutions for
$\omega^{2}$. Their explicit form is complicated in general, but simplifies in
certain limits. For instance, the expansion of the nontrivial fermionic
dispersion relations for $R^{2}M^{2}\ll 1$ is:

$\displaystyle\omega^{(\pm)}_{F,3}(\ell)$
$\displaystyle=p_{F}(\ell)+\frac{2p_{F}(\ell)M^{2}}{4p_{F}^{2}(\ell)-m^{2}_{d}}-\frac{2p_{F}(\ell)M^{4}\left[4p_{F}(\ell)^{2}+m^{2}_{d}\right]}{\left[4p_{F}^{2}(\ell)-m_{d}^{2}\right]^{3}}+\ldots\,,$
(C.9) $\displaystyle\omega^{(\pm)}_{F,4}(\ell)=\omega^{(\pm)}_{F,5}(\ell)$
$\displaystyle=p_{F}(\ell)\pm m_{d}+\frac{M^{2}}{2[2p_{F}(\ell)\pm
m_{d}]}\pm\frac{[2p_{F}(\ell)\mp m_{d}]M^{4}}{8m_{d}[2p_{F}(\ell)\pm
m_{d}]^{3}}+\ldots\,,$ (C.10)

Expanding similarly the bosonic fluctuations (C.6) for $\ell\geq 1$ (the
expansion of $\omega_{B,3}$ is singular at $\ell=0$) and evaluating the sum
(3.20) in dimensional regularization, (3.21) follows.

To obtain the result (3.24), we isolated explicitly the divergent contribution
in the sum. Noting that the first two modes in (C.6) and the two modes in
(C.7) sum to a vanishing contribution in dimensional regularization, we recast
(3.20) as

$\hat{\Delta}_{0}=\frac{R}{2}\sum_{k=3}^{5}\sum_{\ell=1}^{\infty}\tilde{\sigma}(\ell)+\sigma_{0}\;,$
(C.11)

where $\tilde{\sigma}$ denotes the summand in $d=4$ from which we subtracted
the terms of the $\ell\rightarrow\infty$ expansion whose sum does not converge
in $d=4$; explicitly we find:

$\tilde{\sigma}(\ell)=\frac{R}{2}\sum_{k=3}^{5}\left[n_{B}(\ell)\omega_{B,k}(\ell)-\frac{1}{2}\sum_{\pm}n_{F}(\ell)\omega^{(\pm)}_{F,k}(\ell)\right]+\frac{9\ell^{2}}{4}+\frac{21\ell}{4}+3+\frac{R^{2}M^{2}}{4}\,,$
(C.12)

that can be checked to be $O\left(\ell^{-2}\right)$ for
$\ell\rightarrow\infty$. In (C.11) $\sigma_{0}$ is the contribution of the
(regularized) divergent sum and the $\ell=0$ states

$\sigma_{0}=3+\frac{3(RM)^{2}}{8}+\frac{R}{2}\sum_{k=3}^{5}\left[\tilde{\omega}_{B,k}(0)-4N_{f}\sum_{\pm}\tilde{\omega}^{(\pm)}_{F,k}(0)\right]\,.$
(C.13)

To obtain the scaling dimension for large $g^{2}Q$ we then separated the sum
(3.20) into two contributions with a cutoff $aMR$ with $M\gg 1/R$ and $a\ll 1$
as in appendix $D$ of Cuomo:2021cnb . The sum over the low $\ell$’s is
evaluated expanding the dispersion relations at $R^{2}M^{2}\gg\ell\sim O(1)$,
while the sum over the large $\ell$’s is computed expanding for $\ell\sim
RM\gg 1$ and using the Euler-Maclaurin formula. The details are essentially
identical to Cuomo:2021cnb  and we do not report them here.

##  Appendix D $\mathbb{CP}^{N_{f}-1}$ Casimir energy

###  D.1 Special case: $N_{f}=2$

For the special case $N_{f}=2$, we have $\mathbb{CP}^{1}=S^{2}$. The bosonic
part of the EFT is just

$\mathcal{L}=\frac{1}{2}\left(m_{d}^{2}\Phi^{2}-\Phi\nabla^{2}\Phi\right)+\frac{c}{2}\Phi^{2}\left((\partial\varphi)^{2}+\sin^{2}\varphi(\partial\theta)^{2}\right)\;.$
(D.1)

Focus on the $U(1)$ isometry which takes $\theta\to\theta+\alpha$. There is a
solution to the EOMs of the form

$\Phi=v\,,\quad\varphi=\varphi_{0}=\pi/2\,,\quad\theta=i\mu t\,;$ (D.2)

this is an $S^{2}$ geodesic and so solves the EOMs for $\theta,\,\varphi$. The
$\Phi$ EOM reduces to

$\partial^{2}\Phi=\left(m_{d}^{2}+c\left(\partial\theta\right)^{2}\right)\Phi\,,$
(D.3)

which fixes

$\mu=\frac{m_{d}}{\sqrt{c}}\,.$ (D.4)

Now we can find the leading contribution to the energy by directly plugging
this into the Hamiltonian:

$E=\frac{1}{2}\left(m_{d}^{2}v^{2}\right)+\frac{c}{2}v^{2}\mu^{2}=m_{d}^{2}v^{2}\;.$
(D.5)

The charge is $q=v^{2}\sqrt{c}m_{d}$ and so we find

$E=\frac{m_{d}^{2}q}{m_{d}\sqrt{c}}=\frac{m_{d}}{\sqrt{c}}q\;,$ (D.6)

which has the expected linear behavior.

Next we find the subleading correction to the energy from the one-loop Casimir
energy. We expand around the saddle point using

$\Phi=v+\Sigma\,,\quad\theta=i\mu t+\Theta\,,\quad\varphi=\pi/2+\phi\,,$ (D.7)

and expand the action to second order in the fluctuations. The quadratic terms
are

$\mathcal{L}\supset-\frac{1}{2}\Sigma\nabla^{2}\Sigma+\frac{cv^{2}}{2}\left(\partial\Theta\right)^{2}+2\sqrt{c}m_{d}vi\Sigma\partial_{0}\Theta+\frac{cv^{2}}{2}\left(\partial\phi\right)^{2}+\frac{v^{2}m_{d}^{2}}{2}\phi^{2}\;,$
(D.8)

giving the mass matrix

$(\Sigma\;\;\Theta\;\;\phi)\left(\begin{array}[]{ccc}\frac{1}{2}\left(J_{\ell}^{2}-\omega^{2}\right)&\sqrt{c}m_{d}v\omega\\\
\sqrt{c}m_{d}v\omega&\frac{cv^{2}}{2}\left(J_{\ell}^{2}-\omega^{2}\right)\\\
&&\frac{1}{2}cv^{2}\left(J_{\ell}^{2}-\omega^{2}\right)+\frac{v^{2}m_{d}^{2}}{2}\end{array}\right)\begin{pmatrix}\Sigma\\\
\Theta\\\ \phi\end{pmatrix}\;.$ (D.9)

The dispersion relations are

$\begin{split}R\omega_{\pm}&=\frac{1}{2}\left(d-2\pm(d+2\ell-2)\right)=\begin{cases}\ell+d-2\\\
\ell\,,\end{cases}\\\
R\omega_{\phi}&=\frac{\sqrt{c\ell(d+\ell-2)+\frac{1}{4}(d-2)^{2}}}{\sqrt{c}}\,.\end{split}$
(D.10)

The casimir energy is

$\frac{1}{2}\sum_{l}n_{\ell}\omega_{\ell}\,,$ (D.11)

with $n_{\ell}=1+2\ell$ in 3d. This sum diverges so we compute it using
dimensional regularization, following Badel:2019oxl . The contributions from
$\omega_{\pm}$ vanish, so we only need to take into account the contribution
from $\omega_{\Phi}$. The result is displayed in figure 1.

Next we show that the fermions do not contribute to the Casimir energy. The
relevant terms are those described in (2.32) and take the explicit form (up to
an overall factor)

$i\left(D_{\alpha\beta}\psi^{i\beta}\psi^{j\alpha}-\psi^{i\alpha}D_{\alpha\beta}\psi^{j\beta}\right)\,,$
(D.12)

where

$\psi^{i\alpha}D_{\alpha\beta}\psi^{j\beta}=\psi^{i\alpha}\left(g_{ij}\sigma^{\mu}\partial_{\mu}+\partial_{i}g_{kj}\sigma^{\mu}_{\alpha\beta}\partial_{\mu}\phi^{k}\right)\psi^{j\beta}\,,$
(D.13)

and $g$ is the metric for the NLSM above,

$g_{ij}=\left(\begin{array}[]{ccc}\frac{1}{2}\\\ &\frac{c\Phi^{2}}{2}\\\
&&\frac{c\Phi^{2}}{2}\sin^{2}\varphi\end{array}\right)\,.$ (D.14)

Evaluating this on the solution (D.7) we find the quadratic terms

$\psi_{\phi}\partial\psi_{\phi}+cv^{2}\psi_{\theta}\partial\psi_{\theta}+cv\psi_{\phi}\partial\theta\psi_{\theta}+\psi_{\varphi}\partial\psi_{\varphi}\,,$
(D.15)

since $\psi_{\varphi}$ appears as a free massless fermion it does not
contribute to the Casimir energy and we ignore it. The remaining dispersion
relations are

$\omega_{F}^{(\pm)}(\ell)=p_{F}(\ell)\pm\sqrt{c}\mu=p_{F}(\ell)\pm\frac{1}{2R}\,.$
(D.16)

Since $\omega_{F}^{(\pm)}(\ell)$ are always positive, the fermions do not
contribute to the Casimir energy. The full result for the Casimir energy is
then just the bosonic contribution computed above.

###  D.2 General $N_{f}$

Now we perform the calculation for general $\mathbb{CP}^{N_{f}-1}$. First we
must choose a $U(1)$ symmetry. Writing $z_{1}=re^{i\theta}$, we can choose the
$U(1)$ symmetry which shifts $\theta$. We guess a solution to the EOMs of the
form

$\Phi=v\,,\quad\theta=i\mu t\,,\quad r=r_{0}\;,$ (D.17)

and where all $z_{M}=0$ for $M>1$. This is a geodesic for $r_{0}=1$, and so
solves the EOMs for the $\mathbb{C}^{N_{f}-1}$ coordinates. To fix $\mu$ we
use the EOM for $\Phi$, which fixes

$\mu=\frac{m_{d}\left(1+r_{0}^{2}\right)}{\sqrt{c}}=\frac{2m}{\sqrt{c}}\;.$
(D.18)

Now we look at small fluctuations:

$\Phi=v+\Sigma,\quad r=r_{0}+R\,,\quad\theta=i\mu t+\Theta\,,\quad\quad
z_{M}=Z_{M}\,,$ (D.19)

where $M>1$. Expanding the action to quadratic order, we find that the
$Z_{M}$’s decouple from the rest of the fields and become massless. The mass
matrix for $\Sigma,R,\Theta$ is independent of the number of $Z_{M}$’s and so
is the same as in the $CP^{1}$ case, and in particular their contribution to
the Casimir energy is identical. We thus have to add to the $\mathbb{CP}^{1}$
Casimir energy the contribution from the $Z_{M}$’s. They behave as
$2(N_{f}-2)$ real fields with dispersion relations
$\omega=\sqrt{\ell(\ell+d-2)}$, each one contributing $e_{0}\approx-0.132$ to
the Casimir energy. So the total Casimir energy is
$E_{0}(N_{f})=E_{0}(N_{f}=2)+2(N_{f}-2)e_{0}$.

To finish the calculation we also need to find the fermionic contribution.
Expanding the fermionic part of the EFT to quadratic order and ignoring free
massless fermions, we find only one nontrivial dispersion relation, which is
identical to (D.16). As a result we again find that the fermions do not
contribute to the Casimir energy.

##  Appendix E Correlation functions of a free real scalar field from
semiclassics

In this section we show how to generalize the semiclassical approach of
Badel:2019oxl  to the calculation of correlation functions of heavy neutral
operators in the theory of a massless real free field $\Phi$. This motivates
considering the saddle-point (5.2) in the dilaton EFT.

We consider the two-point function of the operator $\Phi^{n}$. This
corresponds to the following path-integral

$\langle\Phi^{n}(x_{f})\Phi^{n}(x_{i})\rangle=\int\mathcal{D}\Phi\,\Phi^{n}(x_{f})\Phi^{n}(x_{i})e^{-\int
d^{d}x\frac{1}{2}\left(\partial\Phi\right)^{2}}\,.$ (E.1)

Proceeding as in Badel:2019oxl , we exponentiate the insertion and obtain the
following modified action

$S_{mod}=\int
d^{d}x\frac{1}{2}(\partial\Phi)^{2}-n\log\Phi(x_{f})-n\log\Phi(x_{i})\,.$
(E.2)

We then consider the resulting saddle-point equation:

$-\partial^{2}\Phi=\frac{n}{\Phi(x)}\left[\delta^{d}(x-x_{f})+\delta^{d}(x-x_{i})\right]\,.$
(E.3)

The solution to this equation formally reads

$\Phi(x)=\alpha\,G(x-x_{i})+\beta\,G(x-x_{f})-(\alpha+\beta)G(0)\,,\qquad
G(x)=\frac{1}{(d-2)\Omega_{d-1}|x|^{d-2}}\,,$ (E.4)

where the coefficients $\alpha$ and $\beta$ satisfy

$\alpha\beta=\frac{n}{G(x_{if})-G(0)}\,,$ (E.5)

and $x_{if}=x_{i}-x_{f}$. Note that the solution therefore admits a free
parameter; the physical significance of the latter will become clear once we
discuss the map to the cylinder. In (E.4) we implicitly assumed that we work
within a regularization scheme in which $G(0)$ is finite. The necessity of
specifying a regularization scheme is unsurprising, since also in the
diagrammatic approach we need to specify a prescription for the tadpole
diagrams arising from contractions of fields at the same point. In mass-
independent schemes that preserve the conformal symmetry of the free theory,
such as dimensional regularization, $G(0)\rightarrow 0$ by dimensional
analysis. In the following we will assume that we work in such a scheme, for
which the solution simplifies and reads

$\Phi(x)=\sqrt{\frac{n}{G(x_{if})}}\left[e^{\tau_{0}}G(x-x_{i})+e^{-\tau_{0}}G(x-x_{f})\right]\,,$
(E.6)

where $\tau_{0}$ is arbitrary.

It is now simple to verify that the saddle-point (E.6) gives the expected
result for the two-point function for $n\gg 1$. Indeed, plugging the solution
in the classical action we obtain

$\langle\Phi^{n}(x_{f})\Phi^{n}(x_{i})\rangle\simeq
e^{-S_{mod}|_{saddle}}=e^{n\log n-n}[G(x_{if})]^{n}\approx
n!\,[G(x_{if})]^{n}\,,$ (E.7)

where we used the Stirling formula in the last step.

Let us finally map the solution (E.6) to the cylinder, such that the
corresponding Euclidean time stretches between the two operators insertions at
$\tau=\pm\infty$ in terms of the Euclidean cylinder time coordinate.
Introducing a subscript to distinguish between flat space and cylinder fields,
the map is given by

$\Phi_{flat}(x)=\left(\frac{|x_{if}|R}{|x-x_{i}||x-x_{f}|}\right)^{\frac{d-2}{2}}\Phi_{cyl}(\tau,\hat{n})\,,\qquad
e^{\tau/R}=\frac{|x-x_{i}|}{|x-x_{f}|}\,,$ (E.8)

where $R$ is the cylinder radius and $\hat{n}$ specifies the position on the
sphere. We therefore obtain the Euclidean profile

$\Phi_{cyl}(\tau,\hat{n})=\sqrt{\frac{n}{(d-2)\Omega_{d-1}R^{d-2}}}\left[e^{-m_{d}\tau+\tau_{0}}+e^{m_{d}\tau-\tau_{0}}\right]\,,$
(E.9)

where $m_{d}=\frac{d-2}{2R}$. Wick rotating $\tau\rightarrow it$ and
$\tau_{0}\rightarrow it_{0}$, and removing the subscript, we obtain the
profile (5.2) in the main text. Note that the arbitrary parameter of the
solution now admits a simple interpretation as the zero-mode parametrizing the
origin of the oscillation period of the solution.

## References

  * (1) G. Cuomo, L. Rastelli and A. Sharon, _Moduli Spaces in CFT: Bootstrap Equation in a Perturbative Example_ , 2406.02679.
  * (2) G. K. Karananas and M. Shaposhnikov, _CFT data and spontaneously broken conformal invariance_ , _Phys. Rev. D_ 97 (2018) 045009 [1708.02220].
  * (3) V. Ivanovskiy, S. Komatsu, V. Mishnyakov, N. Terziev, N. Zaigraev and K. Zarembo, _Vacuum Condensates on the Coulomb Branch_ , 2405.19043.
  * (4) S. Hellerman, D. Orlando, S. Reffert and M. Watanabe, _On the CFT Operator Spectrum at Large Global Charge_ , _JHEP_ 12 (2015) 071 [1505.01537].
  * (5) L. A. Gaumé, D. Orlando and S. Reffert, _Selected topics in the large quantum number expansion_ , _Phys. Rept._ 933 (2021) 1 [2008.03308].
  * (6) S. Hellerman, S. Maeda and M. Watanabe, _Operator Dimensions from Moduli_ , _JHEP_ 10 (2017) 089 [1706.05743].
  * (7) M. A. Luty and W. Taylor, _Varieties of vacua in classical supersymmetric gauge theories_ , _Phys. Rev. D_ 53 (1996) 3399 [hep-th/9506098].
  * (8) D. Gaiotto, Z. Komargodski and J. Wu, _Curious Aspects of Three-Dimensional ${\cal N}=1$ SCFTs_, _JHEP_ 08 (2018) 004 [1804.02018].
  * (9) L. Fei, S. Giombi, I. R. Klebanov and G. Tarnopolsky, _Yukawa CFTs and Emergent Supersymmetry_ , _PTEP_ 2016 (2016) 12C105 [1607.05316].
  * (10) S. Thomas, _Emergent Supersymmetry_ , .
  * (11) G. Badel, G. Cuomo, A. Monin and R. Rattazzi, _The Epsilon Expansion Meets Semiclassics_ , _JHEP_ 11 (2019) 110 [1909.01269].
  * (12) O. Aharony and E. Palti, _Convexity of charged operators in CFTs and the weak gravity conjecture_ , _Phys. Rev. D_ 104 (2021) 126005 [2108.04594].
  * (13) A. Sharon and M. Watanabe, _A counterexample to the CFT convexity conjecture_ , _JHEP_ 05 (2023) 202 [2301.08262].
  * (14) D. Jafferis, B. Mukhametzhanov and A. Zhiboedov, _Conformal Bootstrap At Large Charge_ , _JHEP_ 05 (2018) 043 [1710.11161].
  * (15) A. Salam and J. A. Strathdee, _Nonlinear realizations. 2. Conformal symmetry_ , _Phys. Rev._ 184 (1969) 1760.
  * (16) C. J. Isham, A. Salam and J. A. Strathdee, _Spontaneous breakdown of conformal symmetry_ , _Phys. Lett. B_ 31 (1970) 300.
  * (17) J. R. Ellis, _Phenomenological actions for spontaneously-broken conformal symmetry_ , _Nucl. Phys. B_ 26 (1971) 536.
  * (18) M. A. Luty, J. Polchinski and R. Rattazzi, _The $a$-theorem and the Asymptotics of 4D Quantum Field Theory_, _JHEP_ 01 (2013) 152 [1204.5221].
  * (19) H. Georgi, _Generalized dimensional analysis_ , _Phys. Lett. B_ 298 (1993) 187 [hep-ph/9207278].
  * (20) A. Schwimmer and S. Theisen, _Spontaneous Breaking of Conformal Invariance and Trace Anomaly Matching_ , _Nucl. Phys. B_ 847 (2011) 590 [1011.0696].
  * (21) Z. Komargodski and A. Schwimmer, _On Renormalization Group Flows in Four Dimensions_ , _JHEP_ 12 (2011) 099 [1107.3987].
  * (22) F. Baume and B. Keren-Zur, _The dilaton Wess-Zumino action in higher dimensions_ , _JHEP_ 11 (2013) 102 [1307.0484].
  * (23) F. Coradeschi, P. Lodone, D. Pappadopulo, R. Rattazzi and L. Vitale, _A naturally light dilaton_ , _JHEP_ 11 (2013) 057 [1306.4601].
  * (24) B. Bellazzini, C. Csaki, J. Hubisz, J. Serra and J. Terning, _A Naturally Light Dilaton and a Small Cosmological Constant_ , _Eur. Phys. J. C_ 74 (2014) 2790 [1305.3919].
  * (25) G. Cuomo, _OPE meets semiclassics_ , _Phys. Rev. D_ 103 (2021) 085005 [2103.01331].
  * (26) A. Monin, D. Pirtskhalava, R. Rattazzi and F. K. Seibold, _Semiclassics, Goldstone Bosons and CFT data_ , _JHEP_ 06 (2017) 011 [1611.02912].
  * (27) D. T. Son, _Low-energy quantum effective action for relativistic superfluids_ , hep-ph/0204199.
  * (28) A. Nicolis, R. Penco, F. Piazza and R. Rattazzi, _Zoology of condensed matter: Framids, ordinary stuff, extra-ordinary stuff_ , _JHEP_ 06 (2015) 155 [1501.03845].
  * (29) R. Camporesi and A. Higuchi, _On the Eigenfunctions of the Dirac operator on spheres and real hyperbolic spaces_ , _J. Geom. Phys._ 20 (1996) 1 [gr-qc/9505009].
  * (30) S. Giombi, I. R. Klebanov and G. Tarnopolsky, _Conformal QED d, $F$-Theorem and the $\epsilon$ Expansion_, _J. Phys. A_ 49 (2016) 135403 [1508.06354].
  * (31) G. Cuomo, _A note on the large charge expansion in 4d CFT_ , _Phys. Lett. B_ 812 (2021) 136014 [2010.00407].
  * (32) F. Gretsch and A. Monin, _Perturbative conformal symmetry and dilaton_ , _Phys. Rev. D_ 92 (2015) 045036 [1308.3863].
  * (33) Z. Komargodski, _The Constraints of Conformal Symmetry on RG Flows_ , _JHEP_ 07 (2012) 069 [1112.4538].
  * (34) A. Grassi, Z. Komargodski and L. Tizzano, _Extremal correlators and random matrix theory_ , _JHEP_ 04 (2021) 214 [1908.10306].
  * (35) M. Beccaria, F. Galvagno and A. Hasan, _$\mathcal{N}=2$ conformal gauge theories at large R-charge: the $SU(N)$ case_, _JHEP_ 03 (2020) 160 [2001.06645].
  * (36) C. Beem, M. Lemos, P. Liendo, W. Peelaers, L. Rastelli and B. C. van Rees, _Infinite Chiral Symmetry in Four Dimensions_ , _Commun. Math. Phys._ 336 (2015) 1359 [1312.5344].
  * (37) M. Lemos, _Lectures on chiral algebras of $\mathcal{N}\geqslant 2$ superconformal field theories_, 2006.13892.
  * (38) S. Hellerman and S. Maeda, _On the Large $R$-charge Expansion in ${\mathcal{N}}=2$ Superconformal Field Theories_, _JHEP_ 12 (2017) 135 [1710.07336].
  * (39) S. Hellerman, S. Maeda, D. Orlando, S. Reffert and M. Watanabe, _Universal correlation functions in rank 1 SCFTs_ , _JHEP_ 12 (2019) 047 [1804.01535].
  * (40) S. Hellerman, S. Maeda, D. Orlando, S. Reffert and M. Watanabe, _S-duality and correlation functions at large R-charge_ , _JHEP_ 04 (2021) 287 [2005.03021].
  * (41) S. Hellerman, _On the exponentially small corrections to ${\cal N}=2$ superconformal correlators at large R-charge_, 2103.09312.
  * (42) S. Hellerman and D. Orlando, _Large R-charge EFT correlators in N=2 SQCD_ , 2103.05642.
  * (43) M. Watanabe, _Accessing large global charge via the $\epsilon$-expansion_, _JHEP_ 04 (2021) 264 [1909.01337].
  * (44) F. Benini and S. Benvenuti, _$\mathcal{N}$ = 1 QED in 2 + 1 dimensions: dualities and enhanced symmetries_, _JHEP_ 05 (2021) 176 [1804.05707].
  * (45) A. Sharon and M. Watanabe, _Transition of Large $R$-Charge Operators on a Conformal Manifold_, _JHEP_ 01 (2021) 068 [2008.01106].
  * (46) G. Cuomo, M. Mezei and A. Raviv-Moshe, _Boundary conformal field theory at large charge_ , _JHEP_ 10 (2021) 143 [2108.06579].
  * (47) E. Rabinovici, B. Saering and W. A. Bardeen, _Critical Surfaces and Flat Directions in a Finite Theory_ , _Phys. Rev. D_ 36 (1987) 562.
  * (48) W. A. Bardeen, M. Moshe and M. Bander, _Spontaneous breaking of scale invariance and the ultraviolet fixed point in $\mathrm{O}(n)$-symmetric ($\varphi_{3}^{6}$) theory_, _Phys. Rev. Lett._ 52 (1984) 1188.
  * (49) N. Chai, S. Chaudhuri, C. Choi, Z. Komargodski, E. Rabinovici and M. Smolkin, _Thermal Order in Conformal Theories_ , _Phys. Rev. D_ 102 (2020) 065014 [2005.03676].
  * (50) N. Chai, S. Chaudhuri, C. Choi, Z. Komargodski, E. Rabinovici and M. Smolkin, _Symmetry Breaking at All Temperatures_ , _Phys. Rev. Lett._ 125 (2020) 131603.
  * (51) O. Antipin, J. Bersini, F. Sannino, Z.-W. Wang and C. Zhang, _Charging the $O(N)$ model_, _Phys. Rev. D_ 102 (2020) 045011 [2003.13121].
  * (52) L. Alvarez-Gaume, D. Orlando and S. Reffert, _Large charge at large N_ , _JHEP_ 12 (2019) 142 [1909.02571].
  * (53) S. Giombi and J. Hyman, _On the large charge sector in the critical O(N) model at large N_ , _JHEP_ 09 (2021) 184 [2011.11622].
  * (54) O. Gürdoğan and V. Kazakov, _New Integrable 4D Quantum Field Theories from Strongly Deformed Planar $\mathcal{N}=$ 4 Supersymmetric Yang-Mills Theory_, _Phys. Rev. Lett._ 117 (2016) 201602 [1512.06704].
  * (55) G. K. Karananas, V. Kazakov and M. Shaposhnikov, _Spontaneous Conformal Symmetry Breaking in Fishnet CFT_ , _Phys. Lett. B_ 811 (2020) 135922 [1908.04302].
  * (56) B. Basso and D.-l. Zhong, _Continuum limit of fishnet graphs and AdS sigma model_ , _JHEP_ 01 (2019) 002 [1806.04105].
  * (57) J. Caetano, S. Komatsu and Y. Wang, _Large charge ’t Hooft limit of $\mathcal{N}$ = 4 super-Yang-Mills_, _JHEP_ 02 (2024) 047 [2306.00929].
  * (58) J. R. Ellis, _Aspects of conformal symmetry and chirality_ , _Nucl. Phys. B_ 22 (1970) 478.
  * (59) T. Appelquist and Y. Bai, _A Light Dilaton in Walking Gauge Theories_ , _Phys. Rev. D_ 82 (2010) 071701 [1006.4375].
  * (60) V. Gorbenko, S. Rychkov and B. Zan, _Walking, Weak first-order transitions, and Complex CFTs_ , _JHEP_ 10 (2018) 108 [1807.11512].
  * (61) B. Zan, _Studies in strongly coupled quantum field theories and renormalization group flows_ , Ph.D. thesis, Ecole Polytechnique, Lausanne, 2019.  10.5075/epfl-thesis-9660.
  * (62) F. Benini, C. Iossa and M. Serone, _Conformality Loss, Walking, and 4D Complex Conformal Field Theories at Weak Coupling_ , _Phys. Rev. Lett._ 124 (2020) 051602 [1908.04325].
  * (63) O. Aharony and Y.-N. Breitstein, _Tests of the Charge Convexity Conjecture in Caswell-Banks-Zaks theory_ , _JHEP_ 08 (2023) 044 [2305.08947].
  * (64) O. Antipin, J. Bersini, F. Sannino, Z.-W. Wang and C. Zhang, _More on the weak gravity conjecture via convexity of charged operators_ , _JHEP_ 12 (2021) 204 [2109.04946].
  * (65) T. Ishii and Y. Nakayama, _Convexity restoration from hairy black hole in Einstein-Maxwell-charged scalar system in AdS_ , _JHEP_ 05 (2024) 197 [2402.04552].
  * (66) E. Palti and A. Sharon, _Convexity of charged operators in CFTs with multiple Abelian symmetries_ , _JHEP_ 09 (2022) 078 [2206.06703].
  * (67) D. Orlando and E. Palti, _Goldstone bosons and convexity_ , _Phys. Rev. D_ 108 (2023) 085002 [2303.02178].
  * (68) N. Lashkari, A. Dymarsky and H. Liu, _Eigenstate Thermalization Hypothesis in Conformal Field Theory_ , _J. Stat. Mech._ 1803 (2018) 033101 [1610.00302].
  * (69) Z. Komargodski, M. Mezei, S. Pal and A. Raviv-Moshe, _Spontaneously broken boosts in CFTs_ , _JHEP_ 09 (2021) 064 [2102.12583].
  * (70) L. Eberhardt, _Superconformal symmetry and representations_ , _J. Phys. A_ 54 (2021) 063002 [2006.13280].
  * (71) F. A. Dolan and H. Osborn, _Conformal four point functions and the operator product expansion_ , _Nucl. Phys. B_ 599 (2001) 459 [hep-th/0011040].
  * (72) F. A. Dolan and H. Osborn, _Conformal partial waves and the operator product expansion_ , _Nucl. Phys. B_ 678 (2004) 491 [hep-th/0309180].
  * (73) I. R. Klebanov and E. Witten, _AdS / CFT correspondence and symmetry breaking_ , _Nucl. Phys. B_ 556 (1999) 89 [hep-th/9905104].
  * (74) A. Hashimoto, S. Hirano and N. Itzhaki, _Large branes in AdS and their field theory dual_ , _JHEP_ 08 (2000) 051 [hep-th/0008016].
  * (75) J. McGreevy, L. Susskind and N. Toumbas, _Invasion of the giant gravitons from Anti-de Sitter space_ , _JHEP_ 06 (2000) 008 [hep-th/0003075].
  * (76) V. Balasubramanian, M. Berkooz, A. Naqvi and M. J. Strassler, _Giant gravitons in conformal field theory_ , _JHEP_ 04 (2002) 034 [hep-th/0107119].
  * (77) S. Corley, A. Jevicki and S. Ramgoolam, _Exact correlators of giant gravitons from dual N=4 SYM theory_ , _Adv. Theor. Math. Phys._ 5 (2002) 809 [hep-th/0111222].
  * (78) H. Lin, O. Lunin and J. M. Maldacena, _Bubbling AdS space and 1/2 BPS geometries_ , _JHEP_ 10 (2004) 025 [hep-th/0409174].
  * (79) L. Rastelli and S. S. Razamat, _The supersymmetric index in four dimensions_ , _J. Phys. A_ 50 (2017) 443013 [1608.02965].
  * (80) L. Rastelli and S. S. Razamat, _The Superconformal Index of Theories of Class $\mathcal{S}$_, pp. 261–305.  2016\.  1412.7131.  10.1007/978-3-319-18769-3_9.
  * (81) S. Bhattacharyya, S. Lahiri, R. Loganayagam and S. Minwalla, _Large rotating AdS black holes from fluid mechanics_ , _JHEP_ 09 (2008) 054 [0708.1770].
  * (82) E. Shaghoulian, _Black hole microstates in AdS_ , _Phys. Rev. D_ 94 (2016) 104044 [1512.06855].
  * (83) N. Benjamin, J. Lee, H. Ooguri and D. Simmons-Duffin, _Universal asymptotics for high energy CFT data_ , _JHEP_ 03 (2024) 115 [2306.08031].
  * (84) G. F. Giudice, R. Rattazzi and J. D. Wells, _Graviscalars from higher dimensional metrics and curvature Higgs mixing_ , _Nucl. Phys. B_ 595 (2001) 250 [hep-ph/0002178].
  * (85) G. F. Cuomo, _Large charge, semiclassics and superfluids: from broken symmetries to conformal field theories_ , Ph.D. thesis, EPFL, 2020.  10.5075/epfl-thesis-8397.
  * (86) A. Di Giacomo, _Lezioni di fisica teorica_. ETS, 1992.
  * (87) S. Hellerman and I. Swanson, _Droplet-Edge Operators in Nonrelativistic Conformal Field Theories_ , 2010.07967.
  * (88) A. Milekhin and N. Sukhov, _All holographic systems have scar states_ , 2307.11348.
  * (89) I. Affleck, J. A. Harvey and E. Witten, _Instantons and (Super)Symmetry Breaking in (2+1)-Dimensions_ , _Nucl. Phys. B_ 206 (1982) 413.
  * (90) M. Gremm and E. Katz, _Mirror symmetry for N=1 QED in three-dimensions_ , _JHEP_ 02 (2000) 008 [hep-th/9906020].
  * (91) S. Gukov and D. Tong, _D-brane probes of special holonomy manifolds, and dynamics of N = 1 three-dimensional gauge theories_ , _JHEP_ 04 (2002) 050 [hep-th/0202126].
  * (92) S. J. Gates, M. T. Grisaru, M. Rocek and W. Siegel, _Superspace Or One Thousand and One Lessons in Supersymmetry_ , vol. 58 of _Frontiers in Physics_. 1983, [hep-th/0108200].
  * (93) C. Choi, M. Roček and A. Sharon, _Dualities and Phases of $3DN=1$ SQCD_, _JHEP_ 10 (2018) 105 [1808.02184].
  * (94) O. Aharony and A. Sharon, _Large N renormalization group flows in 3d $\mathcal{N}$ = 1 Chern-Simons-Matter theories_, _JHEP_ 07 (2019) 160 [1905.07146].
  * (95) P. Liendo and J. Rong, _Seeking SUSY fixed points in the 4 $-$ $\epsilon$ expansion_, _JHEP_ 12 (2021) 033 [2107.14515].
  * (96) F. Benini and S. Benvenuti, _$\mathcal{N}$ = 1 dualities in 2+1 dimensions_, _JHEP_ 11 (2018) 197 [1803.01784].
  * (97) O. Antipin, J. Bersini and P. Panopoulos, _Yukawa interactions at large charge_ , _JHEP_ 10 (2022) 183 [2208.05839].


# TLM: Token-Level Masking for Transformers

Yangjun Wu, Kebin Fang*, Dongxiang Zhang, Han Wang, Hao Zhang, Gang Chen
Zhejiang University
{yjwu<EMAIL_ADDRESS>
{22021066, zhangdongxiang, cg<EMAIL_ADDRESS>
<EMAIL_ADDRESS>
* Equal contribution.† Corresponding author.

###### Abstract

Structured dropout approaches, such as attention dropout and DropHead, have
been investigated to regularize the multi-head attention mechanism in
Transformers. In this paper, we propose a new regularization scheme based on
token-level rather than structure-level to reduce overfitting. Specifically,
we devise a novel Token-Level Masking (TLM) training strategy for Transformers
to regularize the connections of self-attention, which consists of two masking
techniques that are effective and easy to implement. The underlying idea is to
manipulate the connections between tokens in the multi-head attention via
masking, where the networks are forced to exploit partial neighbors’
information to produce a meaningful representation. The generality and
effectiveness of TLM are thoroughly evaluated via extensive experiments on 4
diversified NLP tasks across 18 datasets, including natural language
understanding benchmark GLUE, ChineseGLUE, Chinese Grammatical Error
Correction, and data-to-text generation. The results indicate that TLM can
consistently outperform attention dropout and DropHead, e.g., it increases by
$0.5$ points relative to DropHead with BERT-large on GLUE. Moreover, TLM can
establish a new record on the data-to-text benchmark Rotowire ($18.93$ BLEU).
Our code will be publicly available at https://github.com/Young1993/tlm.

##  1 Introduction

In recent years, a variety of pre-trained language models based on the
Transformer Vaswani et al. (2017) architecture have been presented, such as
BERT, GPT Brown et al. (2020), and T5 Raffel et al. (2022). These models push
state-of-the-art forward in numerous NLP tasks.

With the rapid growth of model parameters, deep neural networks are highly
likely to encounter overfitting challenges because supervised data is usually
expensive and insufficient for large language models. This problem could cause
the degradation of model generalization. To address this issue, regularization
methods, such as dropout Srivastava et al. (2014) and subsequent research Wan
et al. (2013); Fan et al. (2020); liang et al. (2021) have been developed from
a structural perspective, which trained with "thinned" subnetworks. The
feature of dropout is that it randomly drops units from the neural networks
during training, preventing units from co-adapting too much.

Figure 1: Illustrations of the attention score with Attention dropout,
DropHead, and TLM. The row denotes Query, and the column represents Key.
Attention dropout randomly drops some attention weights. DropHead directly
drops entire attention heads. Regarding TLM, Self-masking (left) denotes that
the scores for the masked column are invalid. Siblings-masking (right) means
the scores for the row and column are useless except for the masked token
itself.

To further mitigate overfitting for Transformers, structured methods such as
DropHead Zhou et al. (2020) are proposed to drop entire attention heads in the
attention mechanism with the purpose of preventing a small subset of heads
from dominating the whole model. Dropping entire attention heads may result in
losing a significant amount of feature information. Attention dropout is the
application of the dropout technique to the attention mechanism. It
arbitrarily drops some attention weights in the matrix calculation of self-
attention. However, experiments in DropHead and our preliminary trials (shown
in Table 1) demonstrate that the difference is not obvious with or without
attention dropout.

In this work, we introduce a novel regularization scheme based on token-level
instead of a structural perspective. This method, Token-Level Masking (TLM),
is a training technique to regularize the connections among tokens during the
attention calculation in each layer of Transformer blocks. Specifically,
considering the example shown in Fig.1 and 2, TLM contains two techniques: 1)
Siblings-masking. The first step of this method is that we use random
function111Bernoulli function:
https://pytorch.org/docs/stable/generated/torch.bernoulli.html?highlight=bernoulli
to select a percentage of the tokens in the k-th layer, e.g., the masked token
’I’ (the gray block in Fig.2) is excluded from the calculation of attention
weights among the sibling tokens but copies itself, and its neighboring tokens
considers other siblings in addition to ’I’. Then, we feed the Feed-Forward
Network with the attention output to obtain the new hidden state as the input
of the next layer. 2) For Self-masking, we borrow the idea from CBOW Mikolov
et al. (2013) where its attention score is entirely contributed by others. The
difference with Siblings-masking is the masked token ’I’ is forbidden to
attend to attention computation. In the training phase, we randomly invoke one
of two masking strategies at each batch with a $50$-$50$ chance222For
simplicity, we conduct most of the experiments with $50$-$50$ chance and
ablate the proportion in Appendix D.. In this manner, the networks are forced
to utilize partial neighbors’ attention information, not the whole (i.e. the
connections between the masked tokens and their neighboring tokens are
invalid, which are implemented by assigning a large negative number in the
matrix of attention weights). This scheme introduces a bottleneck that the
nets should work hard to become robust and produce a meaningful
representation.

To confirm the effectiveness of our approach, we conducted extensive
experiments on 18 popular datasets. The tasks range from English natural
language understanding benchmark GLUE, ChineseGLUE, and Chinese Grammatical
Error Correction, to data-to-text generation. The experimental results
demonstrate that our TLM with the backbones can substantially improve
performance. Particularly, our method with BERT-base/BERT-large boosts the
score of DropHead from $79.2$ to $79.9$ and $81.7$ to $82.2$ on GLUE, and it
achieves a new state-of-the-art performance ($18.93$ BLEU) on data-to-text
generation. Further experimental analyses demonstrate that our TLM is more
effective in alleviating overfitting than the baselines.

Our main contributions are summarized as follows:

  * •

To reduce overfitting, we present TLM, a novel, simple yet effective training
technique to refine the self-attention computation flow in the multi-head
attention mechanism without modifying the structure of Transformer models.

  * •

TLM can seamlessly integrate with pre-trained Transformer models without extra
cost. The experiments on 18 popular datasets indicate that TLM can lead to
consistency improvements compared to the strong baselines.

  * •

Further analyses demonstrate that TLM can reduce overfitting and enhance the
robustness of the networks.

Model | QQP | RTE
---|---|---
BERT -w Attention-dropout | 87.0 | 61.8
BERT -w/o Attention-dropout | 86.9 | 62.0
Table 1: Results of BERT w and w/o attention-dropout on QQP and RTE. The
descriptions of datasets are available in section 4.1. Figure 2: The computing
flow of TLM for the sentence ’I ate some pizza.’. Our TLM can be employed on
the encoder, encoder-decoder, and decoder architecture in the Transformer.

##  2 Related Work

#### Mask language modeling.

In BERTDevlin et al. (2019), 15% of input tokens are selected, and 80% of
those selected tokens are replaced with the special token [MASK], while 10%
remain unchanged and the remaining 10% are randomly replaced. However, this
random masking and replacement are only done once during data pre-processing,
resulting in a mismatch between pre-training and fine-tuning. RoBERTa Liu et
al. (2019) duplicates training data 10 times to address this issue, but this
requires more training steps. In contrast, our proposed method modifies the
attention computation flow in each layer without requiring additional data or
a special token.

#### Attention Mask.

Attention Mask in Transformers, which is originally used to ignore those
invaluable tokens, called padding tokens, in order to ensure that all the
sequences are the equal length for each batch; or to prevent the networks from
seeing the tokens’ subsequent position in the auto-regressive decoder, such as
UniLM Dong et al. (2019), OPTZhang et al. (2022), and LLaMATouvron et al.
(2023). In this work, we also employ the attention mask with the purpose of
mitigating overfitting, i.e., we remove some attention links among tokens
during training and keep the same as the standard Transformer in inference.

#### Regularization.

Common regularization methods include L1 and L2 regularization, dropout, early
stopping, and data augmentation. Dropout Srivastava et al. (2014) is a very
popular training technique that aims to avoid overfitting by arbitrarily
dropping some units. LayerDrop Fan et al. (2020) randomly drops some entire
substructures or components. Data dropout Iyyer et al. (2015) is performed on
the input data level as data augmentation. Compared to the previous methods,
our approach aims to carefully control the connections between tokens in the
multi-head attention mechanism.

##  3 Approach

In this section, we first review the self-attention computing workflow of the
Transformer and then describe our TLM in more detail.

###  3.1 Multi-head Attention in Transformer

In the research of the Transformer Vaswani et al. (2017), the calculation for
the vanilla attention is formulated as follows:

$\displaystyle Attn(Q,K,V)=softmax(\frac{\textsl{S}(QK)}{\sqrt{d_{emb}/H}})V$
(1) $\displaystyle\textsl{S}(QK)=MatMul(QK^{T})$ (2)

where the queries $Q$, keys $K$, and values $V$ are all matrices, where the
shapes are $Batch\ size\times sequence\ length\times d_{emb}$. $d_{emb}$, and
$H$ denote the dimension of embedding and the number of heads, separately.

There are three types of multi-head attention: 1) Self-Attention usually
denotes the self-attention in the encoder, which means all of the keys ($K$),
values ($V$), and queries ($Q$) come from the same place in a self-attention
layer. Thus, every position in the encoder can attend to the attention
computing of all positions. 2) Cross-Attention is applied in the encoder-
decoder layer, and its feature is that queries ($Q$) come from the decoder
layer, and the memory keys/values come from the output of the encoder. In this
manner, each position in the decoder can be present at the attention computing
of all positions in the input sequence. 3) Masked-Attention is the form of
auto-regression in the decoder. The self-attention layers allow each position
in the decoder to attend to all places in the decoder up to the current
position.

###  3.2 TLM

The core computing procedure of TLM is shown in Fig.2 and Algorithm (1). In a
nutshell, we modify the computing of self-attention by adding a novel step to
control the connections between tokens, which in turn influences the attention
scores and contextual semantic representation. We utilize TLM in the training
phase while keeping the attention calculation as the vanilla during testing.

Algorithm 1 TLM Training Procedure

1: Initialize model with parameters $\omega.$

2: while not converged do

3: for i = 1 to M layers do

4: Obtain attention weights $S(QK)$ (Eq.2)

5: Select tokens to mask (Eq.3)

6: Expand $\hat{Attn\\_M}$ into matrix $M$ (Eq.4)

7: Calculate $\hat{Attn}(Q,K,V)$ (Eq.5)

8: Obtain the hidden state $h_{t}$

9: end for

10: end while

In the training phase, we first compute attention weights $\textsl{S}(QK)$ by
performing Eq.2, and $\textsl{S}(QK)$ denotes the similarity between the
queries $Q$ and keys $K$ for the input tokens. Then, the random function
Bernoulli is executed to select a fixed rate $R$ of masked tokens in the k-th
layer of Transformer blocks at each batch.

$\displaystyle\hat{Attn\\_M}=Bernoulli(Attn\\_M,R)$ (3)

Where $Attn\\_M$ refers to the vector of attention-mask 333Attention-mask, is
abbreviated as $Attn\\_M$, $Attn\\_M=[1,1,...0]$. The values equal 1 denoting
the input tokens, or 0 when it belongs to padding token or masked token.. The
tokens selected as masked tokens will be stored in memory with the attention
mask value of 0. When the rate $R$ is set to $0.1$, which denotes 10% of
tokens would be masked.

Model | CoLA | SST-2 | MRPC | STS-B | QQP | MNLI-m | MNLI-mm | QNLI | RTE | WNLI | AVG | STD
---|---|---|---|---|---|---|---|---|---|---|---|---
BERT-small |  |  |  |  |  |  |  |  |  |  |  |
w/o Att-dropout | 27.5 | 89.3 | 83.2 | 78.9 | 86.9 | 77.5 | 76.8 | 86.2 | 62.0 | 62.1 | 73.0 | 2.8e-4
+Att-dropout | 27.8 | 89.7 | 83.4 | 79.2 | 87.0 | 77.6 | 77.0 | 86.4 | 61.8 | 62.3 | 73.2 | 4.3e-2
+DropHead | 31.7 | 89.6 | 83.2 | 80.3 | 87.2 | 77.7 | 77.2 | 87.3 | 62.5 | 63.0 | 74.1 | 1.1e-3
+TLM | 35.3 | 90.8 | 83.5 | 81.0 | 87.8 | 78.4 | 77.8 | 87.5 | 63.4 | 64.4 | 75.0 | 2.8e-3
BERT-base |  |  |  |  |  |  |  |  |  |  |  |
w/o Att-dropout | 51.0 | 92.3 | 88.2 | 84.2 | 87.7 | 83.5 | 83.2 | 90.3 | 63.0 | 63.0 | 78.6 | 3.1e-3
+Att-dropout | 51.9 | 92.8 | 87.3 | 84.4 | 88.0 | 84.0 | 83.4 | 90.4 | 62.4 | 63.0 | 78.8 | 1.1e-3
+DropHead | 52.0 | 93.4 | 87.8 | 84.5 | 87.5 | 83.6 | 83.1 | 90.4 | 65.2 | 64.4 | 79.2 | 1.8e-3
+TLM | 53.7 | 93.3 | 87.9 | 84.5 | 88.6 | 84.3 | 83.6 | 90.5 | 67.5 | 65.1 | 79.9 | 6.8e-3
BERT-large |  |  |  |  |  |  |  |  |  |  |  |
w/o Att-dropout | 59.7 | 93.9 | 88.0 | 86.1 | 88.7 | 86.5 | 85.6 | 92.5 | 69.7 | 63.7 | 81.4 | 2.6e-3
+Att-dropout | 59.8 | 94.3 | 87.9 | 86.5 | 88.9 | 86.6 | 85.7 | 92.7 | 69.6 | 63.7 | 81.6 | 7.9e-4
+DropHead | 60.1 | 94.1 | 88.1 | 85.9 | 89.2 | 86.7 | 85.8 | 92.6 | 70.1 | 64.4 | 81.7 | 6.5e-3
+TLM | 61.0 | 94.2 | 88.6 | 86.5 | 89.3 | 86.7 | 86.1 | 92.8 | 70.8 | 66.4 | 82.2 | 4.7e-4
Table 2: Fine-tuned BERT-small, BERT-base, and BERT-large performances on
English natural language understanding benchmark GLUE. Each method is tuning
with 3 different random seeds. The AVG denotes the average results and STD is
the standard deviation of 3 results. The highest numbers are in bold.

To fit the identical dimension as the attention weight $\textsl{S}(QK)$, we
expand the attention mask vector $\hat{Attn\\_M}$ into the matrix $M$:

$\displaystyle M=Extend(\hat{Attn\\_M})$ (4)

Here, $M\in{\mathbb{R}}^{B\times H\times N\times N}$. $B$, $H$, and $N$ refer
to batch size, the number of self-attention heads, and the max input sequence
length, respectively. The weights of masked tokens in $M$ are set to the
minimum value of the tensor. Then, we can modify the Eq.1 and 2 as follows:

$\displaystyle\hat{Attn}(Q,K,V)=softmax(\frac{\hat{\textsl{S}}(QK)}{\sqrt{d_{emb}/H}})V$
(5)

$\displaystyle\hat{\textsl{S}}(QK)=\textsl{S}(QK)+M$ (6)

The attention scores of masked connections among tokens are very large
negative numbers by performing Eq.6, so their weights equal 0 after executing
softmax. This makes the connections to the masked tokens ineffective in
influencing the current masked token.

Next, we feed the Feed-Forward Network with $\hat{Attn}(Q,K,V)$ to obtain the
hidden state $h_{t}$. We recursively invoke the identical operation until all
the layers have been traversed and yield the final output tensors.

##  4 Experiments

To verify the generality and effectiveness of our proposed TLM, we perform
extensive experiments on a wide variety of tasks with $18$ benchmark datasets,
including the English natural language understanding benchmark GLUE ($10$
datasets), ChineseGLUE ($6$ datasets), Chinese Grammatical Error Correction
($1$ dataset), and data-to-text generation ($1$ dataset). Note that our method
can be utilized both in the pre-training and fine-tuning, but we only estimate
TLM during the fine-tuning in this work due to limited computational
resources. In the following, we present the key findings, and more details can
be found in the Appendix.

###  4.1 English Language Understanding

Model | AFQMC | TNEWS1.1 | IFLYTEK | CMNLI | CLUEWSC | CSL | AVG | STD
---|---|---|---|---|---|---|---|---
BERT -w/o Att-dropout | 73.6 | 56.7 | 60.2 | 79.4 | 62.2 | 80.2 | 68.6 | 2.2e-2
BERT+Att-dropout | 73.7 | 56.6 | 60.3 | 79.7 | 62.1 | 80.4 | 68.8 | 1.1e-3
BERT+DropHead | 73.6 | 57.0 | 60.6 | 79.0 | 71.4 | 80.5 | 70.4 | 5.2e-4
BERT+TLM | 73.8 | 58.2 | 61.5 | 79.3 | 73.4 | 81.4 | 71.3 | 1.1e-3
Table 3: Fine-tuned BERT-base performances on Chinese language understanding
benchmark CLUE. The AVG denotes the average results and STD is the standard
deviation of 3 results.

#### Dataset.

GLUE benchmark is a collection of diverse natural language understanding tasks
introduced by Wang et al. (2018). GLUE consists of three types of tasks:
single-sentence, similarity and paraphrase, and inference tasks. Single-
sentence tasks require models to predict the grammaticality or sentiment of a
given sentence. Similarity and paraphrase tasks involve determining the degree
of semantic equivalence between sentence pairs. Inference tasks aim to capture
the entailment relationship between sentences.

#### Model and Training.

For a fair comparison, we choose BERT as the backbone and train it using BERT-
small, BERT-base, and BERT-large to explore the effect on model size. The
experiments include BERT without attention-dropout (Att-dropout), with att-
dropout/DropHead/TLM at the rate of $10\%/20\%/5\%$ for each task. We then
submit all the files of prediction at 3 different random seeds to the official
website GLUE444https://gluebenchmark.com/ and obtain the scores of the test
set. As for the hyper-parameters, the learning rate, dropout, and batch size
are uniformly set to 2e-5, 0.1, and 32 for all the tasks. All the models are
trained and evaluated on 24G Nvidia RTX3090.

#### Analysis.

We present the specific results in Table 2. Specifically, we can find that
there is only a slight increase in the AVG ($0.2$ points) with Att-dropout at
small/base/large sizes, and both TLM and DropHead are well ahead of Att-
dropout by a large margin. Compared to DropHead, our method shows a more
significant improvement ($0.9/0.7/0.5$ points) while scaling model size, which
provides evidence that our method is effective in improving the understanding
of natural language.

Regarding the sentence-level classification, CoLA, which only contains 8,511
sentences in the training set, our method demonstrates a significant
improvement from $27.5$ to $35.3$, $51.0$ to $53.7$, and $59.7$ to $61.7$ as
we scale the sizes from small, base to large. This finding is coherent with
the design principle of TLM for mitigating overfitting. Thus, our method can
achieve much larger performance gains than Att-dropout and DropHead as applied
to small-scale supervised data. When scaling the size of datasets such as
MNLI-m, which contains 392k/9k sentence pairs on training/test sets, our
approach still outperforms the DropHead by both $0.7$ points at the sizes of
BERT-small and BERT–base.

On similarity and paraphrase tasks, such as QQP, our method can upgrade the
performance by $0.6/1.1$ points ($87.2$ → $87.8$ and $87.5$ → $88.6$) compared
to DropHead at the sizes of small and base, and this provides evidence that
randomly dropping entire heads benefits less than carefully token masking for
the representations of sentence-pairs. When scaling the size to BERT-large,
the improvement is not obvious like in BERT-base, we speculate that $89.3$ is
approaching $91.1$ (the performance of SOTA) on the GLUE leaderboard, where
the enhancement is limited only through regularization methods.

As to WNLI, this inference task requires the model to fully understand the
contextual information provided by words or phrases in a sentence. The
experimental results indicate that our TLM, by carefully controlling masking,
can bring more benefits than DropHead and attention dropout.

###  4.2 Chinese Language Understanding

#### Dataset.

CLUE, introduced by Xu et al. (2020), is a widely used benchmark for Chinese
Language Understanding Evaluation. Specifically, TNEWS1.1 task classifies
short news titles into 15 categories. As to the IFLTTEK task, which involves
assigning a label from a total of 119 categories to app descriptions. Other
tasks include CLUEWSC2020 (CLUEWSC), which determines co-reference between
pronouns and nouns in a sentence. AFQMC aims to judge the semantic similarity
of sentence pairs. CSL is a keyword recognition task. CMNLI determines the
entailment relationship between sentence pairs.

#### Model and Training.

We choose the Chinese BERT-base from
huggingface555https://huggingface.co/bert-base-chinese as our backbone and
train it with/without Attention dropout (Att-dropout), with TLM and DropHead.
We submit the results with 3 different random seeds to CLUE
leaderboard666https://www.cluebenchmarks.com/ and obtain the scores. We
evaluate our model with the masking rate of $5\%$ and Drophead at the rate of
$10\%$. Other hyper-parameters are the same as the original BERT.

#### Analysis.

The overall results with TLM, DropHead, Att-dropout, and without Att-dropout
are reported in table 3. First, both DropHead and our TLM can notably boost
the AVG compared to with/without Attention dropout, which verifies the
effectiveness of these two regularization methods. Concerning the
classification tasks, our approach can significantly outperform DropHead on
short text TNEWS1.1 by $1.2\%$ and long text IFLTTEK by $0.9\%$. The possible
explanation of promotions is that Siblings-masking and Self-masking introduce
a bottleneck, where the networks can only utilize partial neighbors’ attention
information. Thus, the networks should work hard to become robust and this
scheme benefits more than DropHead. We also notice that adding regularization
methods results in performance degradation on CMNLI, and we leave this for
future work to investigate the reasons.

###  4.3 Chinese Grammatical Error Correction

#### Dataset.

Chinese Grammatical Error Correction (CGEC) is a task that automatically
detects and corrects the grammatical errors in the input sentence without
revising the meaning of the original sentence as much as possible. For this
task, we evaluate our proposed method on the benchmark dataset
CGED777https://github.com/blcuicall/cged_datasets introduced by Rao et al.
(2020). The set contains 28,031/3,115 utterances for training/validation with
the average length of sentences 46.24/45.55. For the test sets, CGED2021 and
CGED2020 comprise 2,294 and 1,457 utterances.

#### Model and Training.

To demonstrate the effectiveness as possible, we choose the strong pre-trained
model Chinese Bart888https://huggingface.co/fnlp/bart-base-chinese as our
backbone and fine-tune it with TLM, DropHead, and Att-dropout at the rate of
$10\%$. We also compare results with the top-performing baselines GECToR
Omelianchuk et al. (2020) with BERT, RoBERTa, and ELECTRA. The learning rate,
batch size, and epoch are set to 3e-5, 32, and 150, respectively.

Model | Score
---|---
GECToR-BERT | 32.8
GECToR-RoBERTa | 33.5
GECToR-ELECTRA | 32.7
MAN Fan et al. (2021) | 41.3
Bart-base -w/o Att-dropout | 42.0
Bart-base + Att-dropout | 42.3
Bart-base + DropHead | 42.7
Bart-base + TLM | 43.7
Table 4: The overall results on CGED2021.

#### Analysis.

The definition of metrics is from the Chinese Learner Text
Correction999https://github.com/blcuicall/CCL2022-CLTC/tree/main/metrics/track2,
and we present the results in Table 4 obtained by the official scripts. First,
we can observe that our TLM outperforms GECToR-BERT/RoBERTa/ELECTRA by
$10.9$/$10.2$/$11.0$ points. In contrast with MAN (A more detailed comparison
can be found in Appendix C), our approach leads to an improvement of $2.4$
points without requiring extra model parameters like MAN. The scale of the set
($28k$ sentences) is relatively small, while the model size of Bart (over
$138M$) is quite large, which may easily cause overfitting. Under this
circumstance, our method improves by $1.0$ and $1.4$ points compared to
DropHead and attention-dropout. We speculate that DropHead drops entire heads
of attention and may lose some syntactic information. Our token-level masking
has the advantage of detecting syntax errors and correcting the errors because
the absence of some tokens will strengthen the model’s sensitivity to
syntactic information.

###  4.4 Data-to-Text Generation

#### Dataset.

Data-to-text generation is a significant challenge that aims to automatically
yield a description to represent the valuable key information in the
structured data. The benchmark dataset is ROTOWIRE introduced by Wiseman et
al. (2017), whose output is the summary of NBA basketball games as the input
is corresponding records that represent the performance of their teams and
players. The summaries are professionally written and relatively well
structured with an average generation length of 337.1 words per example.
Following the research101010https://github.com/harvardnlp/boxscore-data, the
set has been split into training, validation, and test sets consisting of
3,398/727/728 summaries, respectively.

#### Model and Training.

In this task, we take the encoder-decoder model
T5-small111111https://huggingface.co/t5-small as our backbone and compare it
with SOTA models. We train T5 with TLM, DropHead, and Att-dropout at the rate
of $10\%$. As to the hyper-parameters, we set beam search as 8, learning rate
as 5e-4, and max text length as 512. To penalize repetition, the repetition
penalty is set to 2.

Model | BLEU
---|---
ENT Puduppully et al. (2019) | 16.12
DUV Gong et al. (2020) | 15.92
HierarchicalEncoder Li et al. (2021) | 17.96
T5 -w/o Att-dropout | 18.00
T5 + Att-dropout | 17.98
T5 + DropHead | 18.01
T5 + TLM | 18.93
Table 5: BLEU results on Rotowire.

#### Analysis.

We report the results with BLEU in table 5. The current SOTA model
HierarchicalEncoder proposes Number Ranking and Importance Ranking as two
auxiliary tasks to capture the individual relations between the records. Our
approach increases by $0.97$ points relative to HierarchicalEncoder and
achieves a new SOTA. Meanwhile, TLM is extremely easy to train with T5 in an
end-to-end manner as well as no extra modules or tasks are required. It can
also increase by $0.92$ points in contrast to DropHead, the advantage for our
TLM is the masking scheme can encourage models to capture complex relations
among the records and select the salient information in the table. However,
dropping entire heads for DropHead may cause the nets are not sensitive in
capturing complicated feature relationships.

Figure 3: The comparison among Attention Dropout, DropHead, and TLM on STS-B.

###  4.5 Ablation study

Although the experimental results are superior, the effectiveness of our TLM
has not been thoroughly investigated. Thus, we conduct further studies to gain
a better understanding of our approach. As to the selection of datasets, the
STS-B and CoLA are sentence-level datasets while IFLYTEK and CSL are long-text
datasets, and CGED2020/2021 are grammatical error correction datasets. We hope
the tasks can cover different lengths of text meanwhile the diversity can be
guaranteed.

#### TLM vs Attention Dropout/DropHead.

To analyze the relationship among TLM, attention dropout (Att-dropout), and
DropHead applied to self-attention layers in Transformers, we first train
BERT-small only with TLM/Att-dropout/DropHead in the [$0.05$, $0.10$, $0.15$,
$0.20$] range to explore their influences on performance. The results are
presented in Fig.3. We keep all the parameters the same except for the rates.
The finding is that both TLM and DropHead are well ahead of Att-dropout by a
large margin on STS-B, and our method is more stable than DropHead.

Method | STS-B
---|---
BERT w/o regularization | 78.7
\+ dropout | 78.9
\+ dropout + Att-dropout | 79.2
\+ dropout + DropHead | 80.3
\+ dropout + TLM | 81.0
\+ dropout + DropHead + Att-dropout | 80.4
\+ dropout + DropHead + TLM | 80.0
\+ dropout + TLM + Att-dropout | 80.7
\+ All | 79.9
Table 6: The effect of different regularization combinations. All the rates
equal $0.1$.

Second, we test the effect of different combinations on the self-attention
models. As shown in Table 6, we observe that adding any type of regularization
method can improve the performance of the vanilla model, and our TLM
outperforms attention dropout and DropHead under the same conditions by a
large margin. When combined together, we find that the performance is not
optimal, especially when all regularization methods are used together. This is
mainly due to the fact that excessive use of regularization may cause training
instability.

Figure 4: Results with different training data ratios on the test set of CSL.

#### Effect on training data ratio.

We further investigated the impact of different training data ratios by
training BERT-base+TLM using 50%/65%/80% of supervised data on CSL, and the
results are presented in Fig.4. In contrast to DropHead, TLM can achieve
comparable performances even when trained with only 50% of data. Moreover, our
method outperforms DropHead with 65% of data on CSL. The improvements may be
attributed to the token-level masking, as this strategy encourages the model
to capture meaningful context representation with long input sentences.
Therefore, our TLM can benefit the robustness of the networks and reduce the
dependence on supervised data.

Model | CGED2020
---|---
$E_{R}$ | $D_{R}$ | DET-F1 | COR-F1 | SCORE
Bart | - | - | 80.5 | 19.3 | 37.8
TLM | 20 | 20 | 78.9 | 19.3 | 37.1
TLM | 15 | 15 | 81.4 | 19.5 | 38.3
TLM | 15 | 10 | 82.1 | 20.1 | 39.1
TLM | 10 | 15 | 82.4 | 20.2 | 39.2
TLM | 10 | 10 | 82.1 | 20.3 | 38.9
Table 7: The results on CGEC test sets of CGED2020. The $E_{R}$ and $D_{R}$
refer to the rate of masking in the encoder and decoder. DET-F1 means the F1
score for detection, and COR-F1 denotes the F1 score for correction. The
complete results can be found in Appendix B.

#### Effect of TLM rate.

We conduct further testing on the impact of varying the rate of masking in the
encoder and decoder of Bart-base, ranging from 10% to 20%. As outlined in
Table 7, the results of SCORE are better than the baseline, except for the
rate of $20\%$. A possible explanation for why our TLM underperforms BERT at
the rate of $20\%$ is that an excessive amount of masking may confuse the
networks and decrease their ability to comprehend syntax information. It’s
important to note that a too-large rate should be cautious. Overall, our
optimal option is the group of (10%, 15%) for encoder and decoder, and it
outperforms the strong baseline by 1.4 points (37.8 → 39.2). This promotion
demonstrates that our TLM can enhance the understanding of grammatical
information and guide the model toward correcting grammatical errors.

#### Effect of Siblings-masking/Self-masking.

We also analyze the influence of our masking techniques, and the results at
the size of BERT-small are reported in Table 8. It can be observed that the
results decrease by $2.6\%$ and $0.6\%$ on CoLA and STS-B when removing the
Siblings-masking technique. Similarly, the results without Self-masking
decrease by $3.0\%$ and $1.2\%$, and there is a drop of $7.6\%$ and $2.2\%$
without both techniques. These findings highlight the importance of both
Siblings-masking and Self-masking methods.

Model | CoLA | STS-B
---|---|---
TLM | 35.3 | 81.0
-w/o Siblings-masking | 32.7 | 80.5
-w/o Self-masking | 32.3 | 79.9
-w/o Both | 27.7 | 78.9
Table 8: The effect of our masking techniques.

## Conclusion

In this paper, we propose a simple training strategy, Token-Level Masking
(called TLM) to reformulate the computation flow of multi-head self-attention
for reducing overfitting. During training, we randomly invoke one of the two
masking techniques: 1) Siblings-masking, where the masked token is forbidden
to interact with its siblings when computing attention weights, and 2) Self-
masking, the attention weights for the masked token is solely reliant on
others. This regularization scheme enables the networks to work hard to become
robust and acquire meaningful information.

To verify the effectiveness of our proposed method, we conducted various
experiments with 18 benchmark datasets. The results demonstrate that TLM can
consistently improve the performances compared to the strong baselines and
even achieve SOTA on data-to-text generation. Through further analysis, we
observe that our TLM is more stable than DropHead or attention dropout.
Meanwhile, it can seamlessly integrate with pre-trained models.

## Limitations

Here, we list several of what we consider to be limitations:

  1. 1.

The rate of our masking is a hyper-parameter that needs to be tuned, as the
experiments shown in Table 7, the performance may underperform when the rate
is set too large (e.g., over 20%).

  2. 2.

We argue that TLM can also be applied to vision or speech Transformer-based
networks, such as VIT Dosovitskiy et al. (2020) and UniSpeech Wang et al.
(2021), we leave it as the future work for further validation. Meanwhile, we
haven’t yet estimated the performance by combining TLM with extremely large
language models, such as T5-11B and LLaMA.

  3. 3.

Due to the limitation of computational resources. we merely fine-tuned the
pre-trained models with TLM in this work. The effectiveness of TLM applied to
the pre-training phase needs to be further validated.

  4. 4.

In contrast with dropout, TLM can only apply to Transformer-based networks,
not all the neural networks, such as CNN or LSTM.

  5. 5.

Despite numerous ablation studies being performed, the explanation of TLM’s
optimization on the self-attention mechanism remains insufficient, especially
in terms of the effect on attention distribution. Further exploration is
needed.

## Acknowledgements

The authors gratefully acknowledge Yao Zhao, Min Liang, Mengqi Zhang, Xiangyu
Jin, Fengli Shi, Shanhoo Luo, and Fang Shu for giving valuable suggestions on
this study. Our thanks also go to all the anonymous reviewers for their
positive feedback. The work is supported by the National Key Research and
Development Project of China (2022YFF0902000). In addition, we thank HANGZHOU
YIYOULIAO TECHNOLOGY CO LTD https://www.yiyouliao.com/ for providing computing
resources.

## References

  * Brown et al. (2020) Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020.  Language models are few-shot learners.  _CoRR_ , abs/2005.14165.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Dong et al. (2019) Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, and Hsiao-Wuen Hon. 2019.  Unified language model pre-training for natural language understanding and generation.  _CoRR_ , abs/1905.03197.
  * Dosovitskiy et al. (2020) Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. 2020.  An image is worth 16x16 words: Transformers for image recognition at scale.  _CoRR_ , abs/2010.11929.
  * Fan et al. (2020) Angela Fan, Edouard Grave, and Armand Joulin. 2020.  Reducing transformer depth on demand with structured dropout.  In _International Conference on Learning Representations_.
  * Fan et al. (2021) Zhihao Fan, Yeyun Gong, Dayiheng Liu, Zhongyu Wei, Siyuan Wang, Jian Jiao, Nan Duan, Ruofei Zhang, and Xuanjing Huang. 2021.  Mask attention networks: Rethinking and strengthen transformer.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 1692–1701, Online. Association for Computational Linguistics.
  * Gong et al. (2020) Heng Gong, Wei Bi, Xiaocheng Feng, Bing Qin, Xiaojiang Liu, and Ting Liu. 2020.  Enhancing content planning for table-to-text generation with data understanding and verification.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 2905–2914, Online. Association for Computational Linguistics.
  * Iyyer et al. (2015) Mohit Iyyer, Varun Manjunatha, Jordan Boyd-Graber, and Hal Daumé III. 2015.  Deep unordered composition rivals syntactic methods for text classification.  In _Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 1681–1691, Beijing, China. Association for Computational Linguistics.
  * Li et al. (2021) Liang Li, Can Ma, Yinliang Yue, and Dayong Hu. 2021.  Improving encoder by auxiliary supervision tasks for table-to-text generation.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 5979–5989, Online. Association for Computational Linguistics.
  * liang et al. (2021) xiaobo liang, Lijun Wu, Juntao Li, Yue Wang, Qi Meng, Tao Qin, Wei Chen, Min Zhang, and Tie-Yan Liu. 2021.  R-drop: Regularized dropout for neural networks.  In _Advances in Neural Information Processing Systems_ , volume 34, pages 10890–10905. Curran Associates, Inc.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Roberta: A robustly optimized BERT pretraining approach.  _CoRR_ , abs/1907.11692.
  * Mikolov et al. (2013) Tomás Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013.  Efficient estimation of word representations in vector space.  In _1st International Conference on Learning Representations, ICLR 2013, Scottsdale, Arizona, USA, May 2-4, 2013, Workshop Track Proceedings_.
  * Omelianchuk et al. (2020) Kostiantyn Omelianchuk, Vitaliy Atrasevych, Artem Chernodub, and Oleksandr Skurzhanskyi. 2020.  GECToR – grammatical error correction: Tag, not rewrite.  In _Proceedings of the Fifteenth Workshop on Innovative Use of NLP for Building Educational Applications_ , pages 163–170, Seattle, WA, USA → Online. Association for Computational Linguistics.
  * Puduppully et al. (2019) Ratish Puduppully, Li Dong, and Mirella Lapata. 2019.  Data-to-text generation with entity modeling.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 2023–2035, Florence, Italy. Association for Computational Linguistics.
  * Raffel et al. (2022) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2022.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _J. Mach. Learn. Res._ , 21(1).
  * Rao et al. (2020) Gaoqi Rao, Erhong Yang, and Baolin Zhang. 2020.  Overview of NLPTEA-2020 shared task for Chinese grammatical error diagnosis.  In _Proceedings of the 6th Workshop on Natural Language Processing Techniques for Educational Applications_ , pages 25–35, Suzhou, China. Association for Computational Linguistics.
  * Srivastava et al. (2014) Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014.  Dropout: A simple way to prevent neural networks from overfitting.  _J. Mach. Learn. Res._ , 15(1):1929–1958.
  * Touvron et al. (2023) Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023.  Llama: Open and efficient foundation language models.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  In _Advances in Neural Information Processing Systems_ , volume 30. Curran Associates, Inc.
  * Wan et al. (2013) Li Wan, Matthew Zeiler, Sixin Zhang, Yann Le Cun, and Rob Fergus. 2013.  Regularization of neural networks using dropconnect.  In _Proceedings of the 30th International Conference on Machine Learning_ , Proceedings of Machine Learning Research, pages 1058–1066, Atlanta, Georgia, USA. PMLR.
  * Wang et al. (2018) Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018.  GLUE: A multi-task benchmark and analysis platform for natural language understanding.  In _Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP_ , pages 353–355, Brussels, Belgium. Association for Computational Linguistics.
  * Wang et al. (2021) Chengyi Wang, Yu Wu, Yao Qian, Kenichi Kumatani, Shujie Liu, Furu Wei, Michael Zeng, and Xuedong Huang. 2021.  Unispeech: Unified speech representation learning with labeled and unlabeled data.  In _Proceedings of the 38th International Conference on Machine Learning_ , volume 139 of _Proceedings of Machine Learning Research_ , pages 10937–10947. PMLR.
  * Wiseman et al. (2017) Sam Wiseman, Stuart Shieber, and Alexander Rush. 2017.  Challenges in data-to-document generation.  In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_ , pages 2253–2263, Copenhagen, Denmark. Association for Computational Linguistics.
  * Xu et al. (2020) Liang Xu, Hai Hu, Xuanwei Zhang, Lu Li, Chenjie Cao, Yudong Li, Yechen Xu, Kai Sun, Dian Yu, Cong Yu, Yin Tian, Qianqian Dong, Weitang Liu, Bo Shi, Yiming Cui, Junyi Li, Jun Zeng, Rongzhao Wang, Weijian Xie, Yanting Li, Yina Patterson, Zuoyu Tian, Yiwen Zhang, He Zhou, Shaoweihua Liu, Zhe Zhao, Qipeng Zhao, Cong Yue, Xinrui Zhang, Zhengliang Yang, Kyle Richardson, and Zhenzhong Lan. 2020.  CLUE: A Chinese language understanding evaluation benchmark.  In _Proceedings of the 28th International Conference on Computational Linguistics_ , pages 4762–4772, Barcelona, Spain (Online). International Committee on Computational Linguistics.
  * Zhang et al. (2022) Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. 2022.  Opt: Open pre-trained transformer language models.
  * Zhou et al. (2020) Wangchunshu Zhou, Tao Ge, Furu Wei, Ming Zhou, and Ke Xu. 2020.  Scheduled DropHead: A regularization method for transformer models.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 1971–1980, Online. Association for Computational Linguistics.

##  Appendix A Appendix

#### Task descriptions on GLUE.

We provide more details task descriptions and statistics in Fig.5.

Figure 5:  Task descriptions and statistics of GLUE.

Model | CGED2021
---|---
Detection-F1 | Correction-F1 | SCORE
MAN | 77.6 | 30.9 | 41.3
TLM | 77.8 | 30.7 | 43.7

Model | CGED2020
---|---
Detection-F1 | Correction-F1 | SCORE
MAN | 80.5 | 19.8 | 37.2
TLM | 82.1 | 20.3 | 38.9

Table 9: The comparisons between MAN and TLM (10%) on CGEC test sets of
CGED2021 and CGED2021.

#### Experimental settings on CLUE.

As described in Section 4.2, we change the max epochs and mask rate during
training for better scores, and the settings are listed in Table 10.

##  Appendix B Appendix

#### The detailed results on Chinese Grammatical Error Correction.

All the detailed results are listed in Table 11 compared with top-performers.
In contrast to encoder-based Transformer models e.g., GECToR-
BERT/RoBERTa/ELECTRA, our proposed method can outperform them by a large
margin in all the metrics. For example, the masking rate of (10%, 10%) has
much higher Detection, and Correction F1 scores, which demonstrates TLM has
the advantage of accurately locating grammatical errors and successfully
correcting these errors. For the strong baseline Bart-base, the large
improvement lies in Correction F1 (from 28.2 to 30.7).

Parameters | AFQMC | TNEWS | IFLYTEK | CMNLI | WSC | CSL
---|---|---|---|---|---|---
Learning Rate | 2e-5 | 2e-5 | 2e-5 | 3e-5 | 1e-5 | 1e-5
Max Epoch | 9 | 6 | 9 | 9 | 50 | 9
Batch Size | 16 | 16 | 16 | 16 | 16 | 8
Dropout | 0.1 | 0.1 | 0.1 | 0.1 | 0.1 | 0.1
Mask Rate | 0.2 | 0.1 | 0.15 | 0.1 | 0.1 | 0.1
Table 10: The settings of our TLM on Chinese Language Understanding benchmark CLUE.  Model | CGED2021
---|---
ENC-R | DEC-R | FPR | Detection | Identification | Position | Correction | Score |
GECToR-BERT | - | - | 31.9 | 74.5 | 46.3 | 27.5 | 14.6 | 32.8 |
GECToR-RoBERTa | - | - | 30.2 | 74.3 | 46.8 | 27.8 | 15.3 | 33.5 |
GECToR-ELECTRA | - | - | 29.5 | 73.1 | 45.7 | 27.6 | 14.0 | 32.7 |
Bart-base | - | - | 22.3 | 77.4 | 54.3 | 31.8 | 28.2 | 42.3 |
Bart-base + DropHead | 10 | 10 | 22.7 | 76.7 | 54.5 | 33.0 | 29.2 | 42.7 |
Bart-base + TLM | 20 | 20 | 21.3 | 76.0 | 53.6 | 30.9 | 28.1 | 41.8 |
Bart-base + TLM | 15 | 15 | 22.7 | 77.3 | 55.7 | 33.4 | 29.9 | 43.4 |
Bart-base + TLM | 15 | 10 | 23.6 | 77.7 | 55.9 | 34.0 | 30.4 | 43.6 |
Bart-base + TLM | 10 | 15 | 22.9 | 77.7 | 55.4 | 33.6 | 30.0 | 43.4 |
Bart-base + TLM | 10 | 10 | 23.4 | 77.8 | 55.9 | 30.7 | 30.7 | 43.7 |
Table 11: The detailed results on Chinese Grammatical Error Correction
benchmark test set of CGED2021. The ENC-R and EDC-R refer to the rate of
masking in the encoder (ENC) and decoder (DEC),e.g., 20 denotes masking 20%
tokens. FPR, Detection, Identification, Position, and Correction denote false
positive rate, detection-F1, identification-F1, position-F1, and
correction-F1, respectively. We collected the results of GECToR-
BERT/RoBERTa/ELECTRA from
https://github.com/blcuicall/CCL2022-CLTC/tree/main/baselines/track2.

##  Appendix C Appendix

#### In contrast with MAN.

Mask Attention Networks (MAN) introduces a new layer to capture the
relationship between the Self-Attention Network and Feed-Forward Network,
causing the model parameters to substantially increase and hindering its wide
application with pre-trained language models. TLM doesn’t have these issues,
which simply modifies the calculation workflow of self-attention to enhance
the contextual information without extra modules. We run the code from
https://github.com/LibertFan/MAN/tree/main/summarization and make no
modifications, except for using
sentencepiece121212https://github.com/google/sentencepiece as the tokenizer
for Grammatical Error Correction task, and the results are listed in Table 9.
Clearly, TLM can substantially increase by $1.7$ points compared to MAN on
CGED2020 (37.2 $\rightarrow$ 38.9) and $2.4$ points on CGED2021 (41.3
$\rightarrow$ 43.7), which verifies the design of our TLM that masking some
connections in the attention calculation can steer the networks to become
powerful.

##  Appendix D Appendix

#### The proportion of Siblings-masking/Self-masking.

To further investigate the contributions of different proportions of Siblings-
masking and Self-masking, we perform experiments in the group with [0/100%,
25%/75%, 50%/50%, 75%/25%, 100%/0] at rates range in [5%, 10%, 15%] for
Siblings-masking and Self-masking on RTE. The results are reported in Table
12, we note that the experiments with only RTE are inadequate, thus we would
perform further experiments and report the results on our GitHub
https://github.com/Young1993/tlm. As shown in table 12, 50-50 chance is the
relatively better choice compared to 0/100% or 100%/0, and we recommend using
this default setting for simplicity.

Rate | 0/100% | 25%/75% | 50%/50% | 75%/25% | 100%/0
---|---|---|---|---|---
5% | 65.6 | 66.2 | 67.5 | 66.1 | 66.9
10% | 66.9 | 66.5 | 67.1 | 66.3 | 65.3
15% | 64.1 | 63.9 | 64.3 | 64.2 | 63.8
Table 12: Different proportions of Siblings-masking and Self-masking on RTE.