#  Comparative Analysis of Finite Field-dependent BRST
Transformations††thanks: Talk presented at SQS’15, 03 August – 08 August,
2015, at JINR, Dubna, Russia

Pavel Yu. Moshin and Alexander A. Reshetnyak
Tomsk State University, Department of Physics, 634050, Tomsk, Russia,
Institute of Strength Physics and Materials Science, 634021, Tomsk, Russia
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

We present a review of our recent study [1, 2, 3, 4, 5, 6], in which the
concept of finite field-dependent BRST and BRST-antiBRST transformations for
gauge theories was introduced, and their properties investigated. An algorithm
of exact calculation for the Jacobian of a respective change of variables in
the path integral is presented. Applications to the Yang–Mills theory and
Standard Model, in view of infra-red (Gribov) peculiarities, are discussed.

##  1 Introduction

BRST transformations [7, 8] for gauge theories in Lagrangian formalism were
first examined in the capacity of _field-dependent_ _(FD)_ BRST
transformations within the field-antifield approach [9] in order to prove the
independence from small gauge variations (expressed through the gauge fermion
$\psi$) of the path integral $Z_{\psi}$: $Z_{\psi}=Z_{\psi+\delta\psi}$, with
the choice $\mu=-\frac{\imath}{\hbar}\delta\psi$ for the Grassmann-odd
parameter of FD BRST transformations. Originally introduced as the case of a
special $N=1$ SUSY transformation, being a change of the field variables
$\phi^{A}$,

$\phi^{A}\rightarrow\phi^{A\prime}=\phi^{A}+\delta_{\mu}\phi^{A}\ ,\ \
\mathcal{I}_{\phi}^{\psi}=d\phi\exp\Big{\\{}\frac{\imath}{\hbar}S_{\psi}(\phi)\Big{\\}}\
,\ \ Z_{\psi}=\int\mathcal{I}_{\phi}^{\psi},$ (1)

in the integrand$\ \mathcal{I}_{\phi}^{\psi}$ with a quantum action
$S_{\psi}(\phi)$, BRST transformations were extended, by means of antiBRST
transformations [10, 11] in Yang–Mills theories, to $N=2$ BRST-antiBRST
transformations (in Yang–Mills [12] and general gauge theories [13]), which
were associated with an $\mathrm{Sp}(2)$-doublet of Grassmann-odd parameters,
$\mu_{a}$, $a=1,2$.

The concept of _finite_ FD BRST transformations was introduced by Joglekar and
Mandal [14] in Yang–Mills theories, as a sequence of infinitesimal FD BRST
transformations (with a numeric parameter $\kappa$), in order to prove the
gauge-independence of the path integral within the family of $R_{\xi}$-gauges
and their non-linear deformations in the field variables.

The study [15] by a group of Brazilian researchers (see also the references
therein) suggested an analysis of so-called _soft BRST symmetry breaking_ in
Yang–Mills theories, with reference to the Gribov problem [16] in the long-
wave spectra of field configurations, which also involves the Zwanziger
proposal [17] for a horizon functional joined additively to a BRST invariant
quantum action. The study of [15], with the related scope of problems, caught
the attention of one the authors of the present article (A.A.R.), who
subsequently (March, 2011) turned it to the attention of P.M. Lavrov and then
O. Lechtenfeld. The resulting study [18] of this problem within the field-
antifield formalism suggested an equation for the BRST non-invariant addition
$M(\phi,\phi^{\ast})$ to the quantum action $S_{\psi}(\phi,\phi^{\ast})$ of a
general gauge theory. The validity of this equation preserves the gauge-
independence of the vacuum functional $Z_{\psi,M}(0)$ [see (5) for a
definition] and effective action, depending on external antifields,
$\Gamma_{M}=\Gamma_{M}(\phi,\phi^{\ast})$, and evaluated on the extremals
$\Gamma_{M}\frac{\overleftarrow{\delta}}{\delta\phi^{A}}=\Gamma_{M}\overleftarrow{\partial}_{A}=\Gamma_{M,A}=0$,

$\displaystyle\Big{[}M_{A}(\textstyle\frac{\hbar}{\imath}\overrightarrow{\partial}_{(J)},\phi^{\ast})\Big{(}\overrightarrow{\partial}{}^{\ast
A}-\frac{\imath}{\hbar}M^{\ast
A}(\frac{\hbar}{\imath}\overrightarrow{\partial}_{J},\phi^{\ast})\Big{)}\delta\psi({\textstyle\frac{\hbar}{\imath}}\overrightarrow{\partial}_{(J)})+\delta
M(\frac{\hbar}{\imath}\overrightarrow{\partial}_{(J)},\phi^{\ast})\Big{]}Z_{\psi,M}(J,\phi^{\ast})=0$
(2) $\displaystyle\Longrightarrow Z_{\psi,M}(0)=Z_{\psi+\delta\psi,M+\delta
M}(0)\ \mathrm{and}\ \delta\Gamma_{M}\big{|}_{\Gamma_{M,A}=0}=0,\
\mathrm{where}\ \Gamma_{M}=\textstyle\frac{\hbar}{\imath}\ln Z-J_{A}\phi^{A},$
(3)

where it is assumed that $[M_{A},M^{\ast
A}]\equiv[M\overleftarrow{\partial}_{A},\overrightarrow{\partial}^{\ast A}M]$,
and $\phi^{A}=\frac{\hbar}{\imath}\overrightarrow{\partial}_{(J)}^{A}\ln Z$
are average fields, having the same form when the horizon functional $H(A)$
for YM fields $A^{\mu n}(x)$ is used as $M(\phi,\phi^{\ast})$. In terms of the
vacuum expectation value, in the presence of external sources $J_{A}$, and
with a given gauge $\psi$, relation (2) acquires the form,111In fact, the
horizon functional in the family of $R_{\xi}$-gauges for small $\xi$ was found
explicitly in [18], see Eq. (5.20) therein, by using FD BRST transformations
with a small odd-valued parameter.
$\frac{\overrightarrow{\delta}S_{\psi}}{\delta\phi_{A}^{\ast}}\equiv\overrightarrow{\partial}{}^{\ast
A}S_{\psi}$,

$\left\langle\delta
M+M\overleftarrow{s}\textstyle\frac{\imath}{\hbar}\delta\psi(\phi)\right\rangle=\left\langle\delta
M-M\overleftarrow{s}\mu(\delta\psi)\right\rangle=0,\
\mathrm{where}\,\overleftarrow{s}=\overleftarrow{\partial}_{A}S_{\psi}\overrightarrow{\partial}{}^{\ast
A}S_{\psi}:\delta_{\mu}\phi^{A}\equiv\phi^{A}\overleftarrow{s}\mu,$ (4)

where $\overleftarrow{s}$ is the generator of BRST transformations (Slavnov
generator in YM theories). This fact was established in [1]. In December 2011,
one of the authors (A.A.R.) drew the attention of his coauthors (P.M. Lavrov)
in [18] to the research [19] which attempted to use FD BRST transformations
[14] for relating the vacuum functionals in YM and Gribov–Zwanziger (GZ) under
different gauges. An explicit calculation of the functional Jacobian for a
change of variables induced by FD BRST transformations in YM theories with a
finite parameter $\mu$ was made in [20], to establish the gauge-independence
of $Z_{\psi}$ under a finite change of the gauge,
$\psi\rightarrow\psi+\Delta\psi$, and afterwards in [21], to solve equation
(4) with $M(\phi)=H(A)$, in a way different from anticanonical
transformations, as compared to [18].

The present article reviews the study of finite BRST and BRST-antiBRST
transformations (including the case of field-dependent parameters) and the way
they influence the properties of the quantum action and path integral in
conventional quantization. The article has the following organization. Section
2 presents the definitions of finite BRST and BRST-antiBRST transformations.
The algorithm for calculating the functional determinants related to these
transformations is briefly examined in Section 3. The implications of this
calculation to the quantum structure of gauge theories are presented in
Section 4.

We use the DeWitt condensed notation and the conventions of [1, 2], e.g., we
use $\epsilon(F)$ for the value of Grassmann parity of a quantity $F$.
Derivatives with respect to (anti)field variables $\phi^{A},\phi_{A}^{\ast}$
and sources $J_{A}$ are denoted by
$\overleftarrow{\partial}^{A},(\overrightarrow{\partial}_{A}^{\ast})$ and
$\overrightarrow{\partial}_{(J)}^{A}$. The raising and lowering of
$\mathrm{Sp}\left(2\right)$ indices,
$\big{(}\overleftarrow{s}^{a},\overleftarrow{s}_{a}\big{)}=\big{(}\varepsilon^{ab}\overleftarrow{s}_{b},\varepsilon_{ab}\overleftarrow{s}{}^{b}\big{)}$,
are carried out by a constant antisymmetric tensor $\varepsilon^{ab}$,
$\varepsilon^{ac}\varepsilon_{cb}=\delta_{b}^{a}$, $\varepsilon^{12}=1$.

##  2 Proposals for Finite BRST Transformations

The problem of softly broken BRST symmetry (SB BRST) in general gauge theories
was solved in [1] on a basis of finite FD (“gauged” in the terminology of [1])
BRST transformations (invariance transformations for the integrand in (5) at
$J=M=0$) with finite odd-valued parameters $\mu(\phi,\phi^{\ast})$ depending
on external antifields $\phi_{A}^{\ast}$,
$\epsilon(\phi_{A}^{\ast})+1=\epsilon(\phi^{A})=\epsilon_{A}$, and fields
$\phi^{A}$ whose contents include the classical fields $A^{i}$, $i=1,..,n$,
with gauge transformations $\delta A^{i}=R_{\alpha}^{i}(A)\xi^{\alpha}$,
$\alpha=1,..,m<n$, the ghost, antighost, and Nakanishi–Lautrup fields
$C^{\alpha},\bar{C}^{\alpha},B^{\alpha}$,
$\epsilon(A^{i},\xi^{\alpha},C^{\alpha},\bar{C}^{\alpha},B^{\alpha})$=
$\big{(}{\epsilon}_{i}$, ${\epsilon}_{\alpha}$, ${\epsilon}_{\alpha}+1$,
${\epsilon}_{\alpha}+1$, ${\epsilon}_{\alpha}\big{)}$, as well as the
additional towers of fields depending on the (ir)reducibility of the theory.
The generating functional of Green’s functions depending on external sources
$J_{A}$, $\epsilon(J_{A})=\epsilon_{A}$, with an SB BRST symmetry term $M$,
$\epsilon(M)=0$, is given by

$Z_{\psi,M}(J,\phi^{\ast})=\int d\phi\
\exp\left\\{\frac{\imath}{\hbar}S_{\psi}(\phi,\phi^{\ast})+M(\phi,\phi^{\ast})+J_{A}\phi^{A}\right\\},\,\mathrm{with}\,\overleftarrow{s}_{e}={\overleftarrow{\partial}}_{A}{\overrightarrow{\partial}}{}^{\ast
A}S_{\psi}\equiv{\overleftarrow{\partial}}_{A}S_{\psi}^{\ast A},$ (5)

where the generator $\overleftarrow{s}_{e}$,
$\phi^{A}\overleftarrow{s}_{e}=S_{\psi}^{\ast A}(\phi,\phi^{\ast})$, reduces
at $\phi^{\ast}=0$ to the usual generator $\overleftarrow{s}$ of (FD) BRST
transformations, $\delta_{\mu}\phi^{A}=S_{\psi}^{\ast A}(\phi,0)\mu$, and
fails to be nilpotent, due to the quantum master equation for $S_{\psi}$,

$\left[\Delta\exp\left\\{\frac{\imath}{\hbar}S_{\psi}\right\\}=0\Longleftrightarrow
S_{\psi}{\overleftarrow{\partial}}_{A}{\overrightarrow{\partial}}{}^{\ast
A}S_{\psi}=\imath\hbar\Delta
S_{\psi}\right]\Rightarrow(\overleftarrow{s}_{e})^{2}={\overleftarrow{\partial}}_{A}\big{(}S_{\psi}^{\ast
A}{\overleftarrow{\partial}}_{B}\big{)}S_{\psi}^{\ast B}\neq 0,$ (6)

with
$\Delta=(-1)^{\epsilon_{A}}{\overrightarrow{\partial}}_{A}{\overrightarrow{\partial}}{}^{\ast
A}$. The Jacobian induced by a change of variables222In the case
$\mu\overleftarrow{s}_{e}\neq 0$, the set $\\{g(\mu)\\}$, for
$\phi^{\prime}=\phi g\left(\mu\right)$, cannot be presented as Lie group
elements: $g(\mu)\neq\exp\left({\overleftarrow{s}_{e}}{\mu}\right)$.
$\phi^{A}\rightarrow\phi^{\prime A}=\phi^{A}(1+\overleftarrow{s}_{e}\mu)$ was
calculated originally in [1]:

$\displaystyle\mathrm{Sdet}\left\|\phi^{\prime}{}^{A}\overleftarrow{\partial}_{B}\right\|=\exp\left\\{\mathrm{Str}\mathrm{ln}\left(\delta_{B}^{A}+(S_{\psi}^{\ast
A}\mu)\overleftarrow{\partial}_{B}\right)\right\\}=\exp\bigg{\\{}\mathrm{Str}\sum_{n=1}\frac{(-1)^{n+1}}{n}\bigg{(}(S_{\psi}^{\ast
A}\mu)\overleftarrow{\partial}_{B}\bigg{)}^{n}\bigg{\\}}$ (7)
$\displaystyle\Longrightarrow\sum_{n=1}^{\infty}\frac{(-1)^{n+1}}{n}\mathrm{Str}\big{(}(S_{\Psi}^{\ast
A}\mu)\overleftarrow{\partial}_{B}\big{)}^{n}=\sum_{n=1}^{\infty}\frac{(-1)^{n}}{n}(\mu\overleftarrow{s}_{e})^{n}-\sum_{n=2}^{\infty}{(-1)^{n}}(\mu\overleftarrow{s}_{e})^{n-2}\mu_{A}\bigl{(}S_{\psi}^{\ast
A}\overleftarrow{s}_{e}\bigr{)}\mu$ (8) $\displaystyle\ \ +\bigl{(}\Delta
S_{\psi}\bigr{)}\mu\ \
\Longrightarrow\mathrm{Sdet}\left\|\phi^{\prime}{}^{A}\overleftarrow{\partial}_{B}\right\|\
=\
\big{(}1+\mu\overleftarrow{s}_{e}\big{)}^{-1}\Big{\\{}1+\overleftarrow{s}_{e}\mu\Bigr{\\}}\Big{\\{}1+\bigl{(}\Delta
S_{\psi}\bigr{)}\mu\Bigr{\\}},$ (9)

under a suitable condition of convergence for the series in (8), and reduces,
in a rank-$1$ theory with a closed gauge algebra, to the form

$[\Delta S_{\psi},\overleftarrow{s}{}^{2}]=[0,0],\ \mathrm{where}\
\overleftarrow{s}_{e}=\overleftarrow{s}\Longrightarrow\mathrm{Sdet}\left\|\Phi^{\prime}{}^{A}\overleftarrow{\partial}_{B}\right\|\
=\ \big{(}1+\mu\overleftarrow{s}\big{)}^{-1}\ ,$ (10)

which is the same as in YM theories.

The construction of finite BRST-antiBRST Lagrangian transformations solving
the same problem within a suitable quantization scheme (starting from YM
theories), is problematic in view of the BRST-antiBRST non-invariance of the
gauge-fixed quantum action $S_{F}$ in a form more than linear in $\mu_{a}$,
$S_{F}(g_{l}(\mu_{a})\phi)=S_{F}(\phi)+O(\mu_{1}\mu_{2})$, with the gauge
condition encoded by a gauge boson $F(\phi)$. This problem was solved in
January 2014, by constructing finite BRST-antiBRST transformations in a group
form, $\\{g(\mu_{a})\\}$, using an appropriate set of variables $\Gamma^{p}$,
according to [2]

$\left\\{G\left(g(\mu_{a})\Gamma\right)=G\left(\Gamma\right)\ \mathrm{and}\
G\overleftarrow{s}{}^{a}=0\right\\}\Rightarrow
g\left(\mu_{a}\right)=1+\overleftarrow{s}{}^{a}\mu_{a}+\textstyle\frac{1}{4}\overleftarrow{s}{}^{a}\overleftarrow{s}{}_{a}\mu^{2}=\exp\left\\{\overleftarrow{s}{}^{a}\mu_{a}\right\\},$
(11)

for an arbitrary regular functional $G(\Gamma)$, where
$\mu^{2}\equiv\mu_{a}\mu^{a}$, and $\overleftarrow{s}{}^{a}$,
$\overleftarrow{s}{}^{2}\equiv\overleftarrow{s}{}^{a}\overleftarrow{s}{}_{a}$
are the generators of BRST-antiBRST and mixed BRST-antiBRST transformations in
the space of $\Gamma^{p}$. These transformations, however, cannot be presented
as group elements (in terms of an $\exp$-like relation) for an
$\mathrm{Sp}(2)$ doublet $\mu_{a}$ which is not closed under FD BRST-antiBRST
transformations: $\mu_{a}\overleftarrow{s}_{b}\neq 0$.

In YM theories, the quantum action $S_{F}\left(\phi\right)$ in $R_{\xi}$-like
gauges [given by a bosonic gauge functional $F(\phi)$] and the finite BRST-
antiBRST transformations are constructed using an explicit form of BRST-
antiBRST generators in the space of fields
$\phi^{A}=(A^{i},C^{\alpha},\bar{C}^{\alpha},B^{\alpha})$, being identical
with those of the Faddeev–Popov quantization rules [22] and organized in
$\mathrm{Sp}(2)$-symmetric tensors, $(A^{i},C^{\alpha{}a},B^{\alpha})$ =
$\big{(}A^{\mu m},C^{ma},B^{m}\big{)}$, as follows [2]:

$\displaystyle
S_{F}(\phi)=S_{0}(A)-\textstyle\frac{1}{2}F_{\xi}\overleftarrow{s}{}^{2},\
S_{0}(A)=-\frac{1}{4}\int d^{D}xG_{\mu\nu}^{m}G^{m\mu\nu},\
G_{\mu\nu}^{m}=\partial_{[\mu}A_{\nu]}^{m}+f^{mnl}A_{\mu}^{n}A_{\nu}^{l},$
(12) $\displaystyle\ F_{\xi}(\phi)=\textstyle\frac{1}{2}\int d^{D}x\
\big{(}-A_{\mu}^{m}A^{m\mu}+\frac{\xi}{2}\varepsilon_{ab}C^{ma}C^{mb}\big{)}\
,$ (13) $\displaystyle\Delta
A_{\mu}^{m}=D_{\mu}^{mn}C^{na}\mu_{a}-\textstyle\frac{1}{2}\left(D_{\mu}^{mn}B^{n}+\frac{1}{2}f^{mnl}C^{la}D_{\mu}^{nk}C^{kb}\varepsilon_{ba}\right)\mu^{2}\
,$ (14) $\displaystyle\Delta
B^{m}=-\textstyle\frac{1}{2}\left(f^{mnl}B^{l}C^{na}+\frac{1}{6}f^{mnl}f^{lrs}C^{sb}C^{ra}C^{nc}\varepsilon_{cb}\right)\mu_{a}\
,$ (15) $\displaystyle\Delta
C^{ma}=\left(\varepsilon^{ab}B^{m}-\textstyle\frac{1}{2}f^{mnl}C^{la}C^{nb}\right)\mu_{b}-\textstyle\frac{1}{2}\left(f^{mnl}B^{l}C^{na}+\textstyle\frac{1}{6}f^{mnl}f^{lrs}C^{sb}C^{ra}C^{nc}\varepsilon_{cb}\right)\mu^{2}.$
(16)

These relations are inherited from the non-Abelian gauge transformations in
terms of a covariant derivative,

$\delta A_{\mu}^{m}(x)=D_{\mu}^{mn}(x)\zeta^{n}(x)=\int d^{D}y\
R_{\mu}^{mn}(x;y)\zeta^{n}(y)\ ,\ \ \mathrm{where}\ \
i=(\mu,m,x),\alpha=(n,y),$ (17)

where the generator of gauge transformations $R_{\mu}^{mn}(x;y)$ leaves the
classical action $S_{0}$ invariant with accuracy up to the first order in
arbitrary functions $\zeta^{n}$: $\mathcal{S}_{0}(A+\delta
A)=\mathcal{S}_{0}(A)+o(\zeta^{n})$; the metric tensor is
$\eta_{\mu\nu}=\mathrm{diag}(-,+,\ldots,+)$, and $f^{lmn}$ are the totally
antisymmetric $su(\hat{N})$ structure constants,
$l,m,n=1,\ldots,\hat{N}{}^{2}-1$. The $N=2$ BRST-antiBRST invariant action for
YM theories coincides with the $N=1$ BRST invariant action only in Landau
gauge, $\xi=0$, whereas in the gauges $F_{\xi\neq 0}$ including the Feynmann
gauge, $\xi=1$, the action $S_{F}$ contains terms quartic in the ghosts
$C^{m{}a}$, leading thereby to additional vertices in the Feynmann integrals.
This is a property of quantization with a global SUSY symmetry higher than
$N=1$ BRST symmetry.

In general gauge theories, such as reducible ones (for higher-spin field
theories, see [23, 24] and references therein), or those with an open gauge
algebra (as in some SUSY models), the corresponding space of triplectic
variables
$\Gamma_{{tr}}^{p}=(\phi^{A},\phi_{Aa}^{\ast},\bar{\phi}_{A},\pi^{Aa},\lambda^{A})$
in the $\mathrm{Sp}(2)$-covariant Lagrangian quantization method [13]
contains, in addition to $\phi^{A}$, three sets of antifields
$\phi_{Aa}^{\ast}$, $\bar{\phi}_{A}$,
$\epsilon(\phi_{Aa}^{\ast},\bar{\phi}_{A})=(\epsilon_{A}+1,\epsilon_{A})$, as
sources to BRST, antiBRST and mixed BRST-antiBRST transformations, and three
sets of Lagrangian multipliers $\pi^{Aa},\lambda^{A}$,
$\epsilon(\pi^{Aa},\lambda^{A})=(\epsilon_{A}+1,\epsilon_{A})$, introducing
the gauge. The corresponding generating functional of Green’s functions,
$Z_{F}(J)$,

$Z_{F}(J)=\int
d\Gamma\exp\left\\{\big{(}\imath/\hbar\big{)}\Big{[}W+\phi_{a}^{\ast}\pi^{a}+\bar{\phi}\lambda-\textstyle\frac{1}{2}F\overleftarrow{U}{}^{2}+J\phi\Big{]}\right\\},\
\overleftarrow{U}_{a}={\overleftarrow{\partial}}_{A}\pi^{Aa}+\varepsilon^{ab}{\overleftarrow{\partial}_{Ab}^{(\pi)}}\lambda^{A}$
(18)

is invariant, at $J=0$, with respect to finite BRST-antiBRST transformations
(for constant $\mu_{a}$) in the space of $\Gamma_{{tr}}^{p}$, which are given
by (11), however, with the functional $G_{{tr}}=G(\Gamma_{{tr}}^{p})$:

$\displaystyle\Gamma_{{tr}}^{p}\rightarrow\Gamma_{{tr}}^{\prime
p}=\Gamma_{{tr}}^{p}\left(1+\overleftarrow{s}{}^{a}\mu_{a}+\textstyle\frac{1}{4}\overleftarrow{s}{}^{2}\mu^{2}\right)\equiv\Gamma_{{tr}}^{p}g(\mu_{a})\Longrightarrow\mathcal{I}_{\Gamma_{{tr}}g(\mu_{a})}^{\left(F\right)}=\mathcal{I}_{\Gamma_{{tr}}}^{\left(F\right)}\
\mathrm{for}\,Z_{F}=\int\mathcal{I}_{\Gamma_{{tr}}}^{\left(F\right)},$ (19)
$\displaystyle\mathrm{where}\
\overleftarrow{s}{}^{a}=\Big{(}{\overleftarrow{\partial}}_{A},{\overleftarrow{\partial}}{}_{(\phi^{\ast})}^{Aa},{\overleftarrow{\partial}}_{(\bar{\phi})}^{A},{\overleftarrow{\partial}}_{Ab}^{(\pi)}\Big{)}\Big{(}\pi^{Aa},W_{,A}(-1)^{\epsilon_{A}},\varepsilon^{ab}\phi_{Ab}^{\ast}(-1)^{\epsilon_{A}+1},\varepsilon^{ab}\lambda^{A}\Big{)}^{T},\\{\overleftarrow{s}{}^{a},\overleftarrow{s}{}^{b}\\}\neq
0,$ $\displaystyle\ \mathrm{provided\ that}\
\left(\Delta^{a}+(\imath/\hbar)\varepsilon^{ab}\phi_{Ab}^{\ast}{\overrightarrow{\partial}}_{(\bar{\phi})}^{A}\right)\exp\left\\{\frac{\imath}{\hbar}W\right\\}=0,\
\mathrm{for}\
\Delta^{a}=(-1)^{\epsilon_{A}}{\overrightarrow{\partial}}_{A}{\overrightarrow{\partial}}{}^{\ast
Aa},$ (20)

with respective classical action $S_{0}(A)$ being be the boundary condition
for the equation (20) on the bosonic functional $W$:
$W(\phi,\phi_{a}^{\ast},\bar{\phi})\big{|}_{\phi_{a}^{\ast}=\bar{\phi}=0}=S_{0}(A)$.
The restricted generators
$\overleftarrow{U}{}^{a}=\left.\overleftarrow{s}{}^{a}\right|_{\phi,\pi,\lambda}$
are nilpotent and satisfy the algebra
$\\{\overleftarrow{U}{}^{a},\overleftarrow{U}{}^{b}\\}=0$.

##  3 Jacobians of Finite $N=1,2$ BRST Transformations

The Jacobian (9) allows one to solve the problem of SB BRST symmetry in
general gauge theories [1] and was examined in detail [5] for an equivalent
representation of $Z_{\psi,M}(J,\phi^{\ast})$ in (21), as well as for BRST
transformations in an extended space which contains, besides $\phi^{A}$, also
internal (included in the path integral) antifields $\tilde{\phi}_{A}^{\ast}$
and Lagrangian multipliers ${\lambda}^{A}$ to Abelian hypergauge conditions,
$\mathcal{G}_{A}(\phi,\phi^{\ast})=\phi_{A}^{\ast}-\psi(\phi)\overleftarrow{\partial}_{A}$:

$\displaystyle Z_{\psi,M}(J,\phi^{\ast})=\int
d{\Gamma}\exp\left\\{\frac{\imath}{\hbar}S({\phi},\tilde{\phi}{}^{\ast})+\mathcal{G}_{A}\big{(}{\phi},\tilde{\phi}^{\ast\prime}+\phi^{\ast}\big{)}\lambda^{A}+M({\phi},{\phi}{}^{\ast})+J\phi\right\\},$
(21) $\displaystyle\mathrm{with}\
{\Gamma}^{p}\rightarrow{\Gamma}^{p\prime}={\Gamma}^{p}\big{(}1+\overleftarrow{{s}}\mu\big{)},\
\mathrm{where}\
{\Gamma}^{p}\overleftarrow{{s}}=\big{(}\phi^{A},\tilde{\phi}_{A}^{\ast},\lambda^{A}\big{)}\overleftarrow{{s}}=\big{(}\lambda^{A},S\overleftarrow{\partial}_{A},0\big{)}$
(22) $\displaystyle\ \mathrm{and}\
\mathrm{Sdet}\left\|\Gamma^{p\prime}\overleftarrow{\partial}_{q}^{\Gamma}\right\|=\big{(}1+\mu\overleftarrow{{s}}\big{)}^{-1}\Big{\\{}1+\bigl{(}\Delta
S\bigr{)}\mu\Bigr{\\}}+O(\mu\overleftarrow{{s}}\mu),$ (23)

with a field-dependent $\mu(\Gamma)$ in (23), which reduces (for
$\phi^{*}$-independent $\mu(\Gamma)$:
$\mu(\Gamma)\overleftarrow{\partial}^{\ast A}=0$) to the Jacobian of [25],
i.e., one without $O(\mu\overleftarrow{{s}}\mu)$. The bosonic functional
$S(\phi,\phi^{\ast})$ in (21) is a proper solution of (6), with the classical
action $S_{0}$ as the boundary condition at $\phi^{\ast}=0$.

For BRST-antiBRST transformations in YM theories, the technique of calculating
the Jacobian was first examined for functionally-dependent parameters
$\mu_{a}=\Lambda(\phi)\overleftarrow{s}_{a}$ with an even-valued functional
$\Lambda$ and was developed in [2], resulting in,
$\phi^{\prime}{}^{A}\equiv\phi{}^{A}g(\Lambda(\phi)\overleftarrow{s}_{a})$,

$\displaystyle\
J_{\Lambda(\phi)\overleftarrow{s}_{a}}=\mathrm{Sdet}\left\|\phi^{\prime}{}^{A}\overleftarrow{\partial}_{B}\right\|=\exp\left\\{\mathrm{Str}\,\mathrm{ln}\left(\delta_{B}^{A}+M_{B}^{A}\right)\right\\},\
\ \mathrm{for}\ \ M_{B}^{A}=P_{B}^{A}+Q_{B}^{A}+R_{B}^{A}$ (24)
$\displaystyle\
=\phi^{A}\overleftarrow{s}^{a}(\mu_{a}\overleftarrow{\partial}_{B})+\mu_{a}\big{[}(\phi^{A}\overleftarrow{s}^{a})\overleftarrow{\partial}_{B}-\textstyle\frac{1}{2}(\phi^{A}\overleftarrow{s}^{2})(\mu^{a}\overleftarrow{\partial}_{B})\big{]}(-1)^{\epsilon_{A}+1}+\textstyle\frac{1}{4}\mu^{2}(\phi^{A}\overleftarrow{s}^{2}\overleftarrow{\partial}_{B}),$
$\displaystyle\
\mathrm{Str}(P+Q+R)^{n}=\mathrm{Str}(P+Q)^{n}+C_{n}^{1}\mathrm{Str}P^{n-1}R,\
\ \mathrm{for}\ \ C_{n}^{k}=n!/k!(n-k)!,$ (25) $\displaystyle\
\mathrm{Str}(P+Q)^{n}=\left\\{\begin{array}[]{l}\mathrm{Str}P^{n}+n\mathrm{Str}P^{n-1}Q+C_{n}^{2}\mathrm{Str}P^{n-2}Q^{2},\
n=2,3,\\\
\mathrm{Str}P^{n}+n\textstyle\sum_{k=0}^{2}\mathrm{Str}P^{n-k}Q^{k}+K_{n}\mathrm{Str}P^{n-3}QPQ,\
n>3\end{array}\right.$ (28) $\displaystyle\ \Longrightarrow
J_{\Lambda(\phi)\overleftarrow{s}_{a}}=\exp\left\\{\,\sum_{n=1}(-1)^{n-1}n^{-1}\mathrm{Str}(P_{B}^{A})^{n}\right\\}=\left(1-\textstyle\frac{1}{2}\Lambda\overleftarrow{s}{}^{2}\right)^{-2},$
(29)

where
$K_{n}=\left[\frac{n+1}{2}-2\right]C_{n}^{1}+\big{(}(n+1)\,\mathrm{mod}\,2\big{)}C_{\left[\frac{n}{2}\right]}^{1}$,
with $[x]$ being the integer part of $x\in\mathbb{R}$. The Jacobian (29)
cannot be derived from the Jacobian (10) corresponding to FD BRST
transformations in YM theories.333The corresponding Jacobian is not equal to
the expression $J=$
$\mathrm{Sdet}\left\|\phi^{A\prime}\overleftarrow{\partial}_{B}\right\|=\big{(}1+\big{[}\bar{\mu}\overleftarrow{\bar{s}}\mu\big{]}\overleftarrow{s}\big{)}^{-1}=1-\bar{\mu}\overleftarrow{\bar{s}}\mu\overleftarrow{s}+o(\mu\bar{\mu}),$
which is presented in [http://theor.jinr.ru/sqs15/Lavrov.pdf]. The just
mentioned expression does not allow one to control the gauge independence of
$Z_{F}$ in (18) for infinitesimal FD parameters $(\mu_{1},\mu_{2})=$
$(\mu,\bar{\mu})$, $(\overleftarrow{s}^{1},\overleftarrow{s}^{2})$ =
$(\hat{s},\hat{\bar{s}})$, either in YM or in general gauge theories. For
functionally-independent FD parameters
$\mu_{a}(\phi)\neq\Lambda\overleftarrow{s}{}_{a}$, the above algorithm
(24)–(29) of calculating $J_{\mu_{a}}$ involves a generalization of (28) to
the case $P^{n}\not\equiv f^{n-1}P$, examined separately for odd and even
$n>3$, which leads to [6]

$J_{\mu_{a}}=\exp\left\\{\,\mathrm{tr}\sum_{n=1}(-1)^{n-1}n^{-1}\mathrm{Str}(P_{B}^{A})^{n}\right\\}=\exp\left\\{\,-\mathrm{tr}\ln(e+m)\right\\},\
m_{b}^{a}=\mu_{b}\overleftarrow{s}{}^{a},$ (30)

where $\left(e\right)_{b}^{a}$ and $\mathrm{tr}$ denote $\delta_{b}^{a}$ and
trace over $\mathrm{Sp}(2)$ indices. The Jacobian (30) is generally not BRST-
antiBRST exact; however, it reduces at $\mu_{a}=\Lambda\overleftarrow{s}_{a}$
to the Jacobian (29), due to

$\mathrm{tr}\,m_{b}^{a}=\mathrm{tr}\,\Lambda\overleftarrow{s}{}_{b}\overleftarrow{s}{}^{a}=-(1/2)\mathrm{tr}\,\delta_{b}^{a}\Lambda\overleftarrow{s}{}^{2}\Rightarrow\mathrm{tr}\,m^{n}=2[-(1/2)\Lambda\overleftarrow{s}{}^{2}]^{n}\Rightarrow
J_{\mu_{a}}=J_{\Lambda\overleftarrow{s}_{a}}.$ (31)

In general gauge theories (18)–(20), the calculation of Jacobians induced by
FD BRST-antiBRST transformations was first carried out in [3, 5] with
functionally-dependent parameters
$\mu_{a}=\Lambda(\phi,\pi,\lambda)\overleftarrow{U}_{a}$ and then in [6] with
arbitrary parameters $\mu_{a}(\Gamma_{tr})$, including functionally-
independent $\mu_{a}(\phi,\pi,\lambda)$. The result is given by

$\displaystyle
J_{\Lambda\overleftarrow{U}_{a}}=\mathrm{Sdet}\left\|\left[{\Gamma_{tr}^{p}}{g}(\Lambda\overleftarrow{U}_{a})\right]\overleftarrow{\partial}{}_{q}^{\Gamma}\right\|=\exp\left[-\left(\Delta^{a}W\right)\mu_{a}-\textstyle\frac{1}{4}\left(\Delta^{a}W\right)\overleftarrow{s}_{a}\mu^{2}\right]\left(1-\textstyle\frac{1}{2}\Lambda\overleftarrow{s}{}^{2}\right)^{-2},$
(32) $\displaystyle
J_{\mu_{a}(\phi,\pi,\lambda)}=\exp\Big{\\{}-\left(\Delta^{a}W\right)\mu_{a}-\textstyle\frac{1}{4}\left(\Delta^{a}W\right){\overleftarrow{s}}_{a}\mu^{2}-\mathrm{tr}\,\ln\left(e+{m}\right)\Big{\\}},$
(33) $\displaystyle\
J_{\mu_{a}(\Gamma_{tr})}=J|_{\mu_{a}(\phi,\pi,\lambda)\rightarrow\mu_{a}(\Gamma_{tr})}\exp\Big{\\{}\textstyle\frac{1}{4}(\mu_{a}\overleftarrow{\partial}{}_{p}^{\Gamma})\left[(e+{m})^{-1}\right]_{b}^{a}\left({\Gamma}_{tr}^{p}\overleftarrow{s}{}^{2}\overleftarrow{s}{}^{b}\right)\mu^{2}\Big{\\}}.$
(34)

The second multiplier in (34) draws a difference between the Jacobians
$J_{\mu_{a}(\phi,\pi,\lambda)}$ and $J_{\mu_{a}(\Gamma_{tr})}$, because
$\overleftarrow{s}_{a}$ are not reduced to the nilpotent
$\overleftarrow{U}_{a}$ as they act on $\Gamma_{tr}^{p}$. This result
generalizes the Jacobian for $\mu_{a}(\phi,\pi,\lambda)$ obtained by a special
Green function, using a $t$-parametric rescaling of the Lie equations in [25],
and cannot be achieved by the prescription therein, due to the non-
integrability condition
$\overleftarrow{s}_{a}\overleftarrow{s}_{b}\overleftarrow{s}_{c}\neq 0$. For
constant parameters $\mu_{a}$, the Jacobians (32)–(34) are reduced to the same
expression, being an $\hbar$-deformation of the classical master equations
(20) and their differential consequences obtained by applying
$\overleftarrow{s}_{a}$.

In generalized Hamiltonian formalism, the Jacobians of corresponding FD BRST-
antiBRST transformations were calculated from first principles by the rules
(24)–(30) in [4, 6], whereas the so-called _linearized_ finite BRST-antiBRST
transformations
$\Gamma\rightarrow\Gamma^{\prime}=\Gamma(1+\overleftarrow{s}^{a}\mu_{a})$ in
YM and arbitrary first-class constraint dynamical systems, along with a
calculation of Jacobians and a discussion of their implications on the quantum
theory, were presented in [6].

##  4 Implications of Finite BRST Transformations

The proposals for $N=1$ and $N=2$ finite BRST transformations allow one to
establish (in the case of constant $\mu$ and $\mu_{a}$) the finite BRST
(identical with the case of small $\mu$) and BRST-antiBRST invariance of the
integrand in (21) with a vanishing $N=1,2$ SB BRST symmetry term $M$, as well
as in (18) for YM and general gauge theories.

For FD parameters, finite BRST transformations allow one to obtain a new form
of the Ward identity and to establish the gauge independence of the path
integral under a finite change of the gauge,
$\psi\rightarrow\psi+\psi^{\prime}$, provided that the SB BRST symmetry term
$M=M_{\psi}$ transforms to
$M_{\psi+\psi^{\prime}}=M_{\psi}(1+\overleftarrow{s}\mu(\psi^{\prime}))$, with
$\mu(\psi^{\prime})$ being a solution of a so-called compensation equation, as
one makes a change of variables corresponding to an FD BRST transformation in
the integrand of $Z_{\psi,M_{\psi}}(0,\phi^{\ast})$:

$Z_{\psi,M_{\psi}}(0,\phi^{\ast})=Z_{\psi+\psi^{\prime},M_{\psi+\psi^{\prime}}}(0,\phi^{\ast})\Rightarrow\psi^{\prime}(\phi,\lambda|\mu)=\textstyle\frac{\hbar}{i}\left[\sum\nolimits_{n=1}\textstyle\frac{(-1)^{n-1}}{n}\left(\mu\overleftarrow{s}\right){}^{n-1}\right]\mu.$
(35)

The Ward identity, depending on the FD parameter
$\mu(\psi^{\prime})=-\frac{i}{\hbar}g(y)\psi^{\prime}$ , for
$g(y)=1-\exp\\{y\\}/y$, $y\equiv({i}/{\hbar})\psi^{\prime}\overleftarrow{s}$,
and the gauge-dependence problem are described by the respective expressions
[5]

$\left\langle\left\\{1+\frac{i}{\hbar}\left[J_{A}\phi^{A}+M_{\psi}\right]\overleftarrow{s}\mu(\psi^{\prime})\right\\}\left(1+\mu(\psi^{\prime})\overleftarrow{s}\right){}^{-1}\right\rangle_{\psi,M,J}=1\,\mathrm{and}\,\left\langle(J_{A}\phi^{A}+M_{\psi})\overleftarrow{s}\right\rangle_{\psi,M,J}=0,$
(36)

as one makes averaging with respect to $Z_{\psi,M_{\psi}}(J,\phi^{\ast})$. The
above equations are equivalent to those of [1, 18]

In turn, FD BRST-antiBRST transformations solve the same problem (allowing for
the presence of an SB BRST-antiBRST symmetry term $M=M_{F}$ and not touching
upon the unitarity issue [5, 27]) under a finite change of the gauge,
$F\rightarrow F+F^{\prime}$, provided that the term $M_{F}$ transforms to
$M_{F+F^{\prime}}=M_{F}(1+\overleftarrow{s}^{a}\mu_{a}(F^{\prime})+\frac{1}{4}\overleftarrow{s}^{2}\mu{}^{2}(F^{\prime}))$,
with $\mu_{a}(F^{\prime};\phi,\pi,\lambda)=\Lambda\overleftarrow{U}_{a}$ being
a solution to the corresponding compensation equation, as one makes a change
of variables corresponding to an FD BRST transformation in the integrand of
$Z_{F}(0)$ from (18):

$Z_{F}(0)=Z_{F+F^{\prime}}(0)\Rightarrow
F^{\prime}(\phi,\pi,\lambda|\mu_{a})=4\imath{\hbar}\left[\sum\nolimits_{n=1}\textstyle\frac{(-1)^{n-1}}{2^{n}n}\left(\Lambda\overleftarrow{U}^{2}\right){}^{n-1}\Lambda\right].$
(37)

As a result, the corresponding Ward identity, with the FD parameters
$\mu_{a}(F^{\prime})=\frac{i}{2\hbar}g(y)F^{\prime}\overleftarrow{U}_{a}$,
$\Lambda(\Gamma|{F}^{\prime})=\frac{i}{2\hbar}g(y){F}^{\prime}$, for
$y\equiv({i}/{4\hbar})F^{\prime}\overleftarrow{U}{}^{2}$, and the gauge-
dependence problem acquire the form [5]

$\displaystyle\textstyle\left\langle\left\\{1+\frac{i}{\hbar}J_{A}\phi^{A}\left[\overleftarrow{U}^{a}\mu_{a}(\Lambda)+\frac{1}{4}\overleftarrow{U}^{2}\mu^{2}(\Lambda)\right]-\frac{1}{4}\left(\frac{i}{\hbar}\right){}^{2}J_{A}\phi^{A}\overleftarrow{U}^{a}J_{B}(\phi^{B})\overleftarrow{U}_{a}\mu^{2}(\Lambda)\right\\}\right.$
(38)
$\displaystyle\quad\left.\times\textstyle\left(1-\frac{1}{2}\Lambda\overleftarrow{U}^{2}\right){}^{-2}\right\rangle_{F,J}=1,$
$\displaystyle
Z_{F+F^{\prime}}(J)=Z_{F}(J)\left\\{1+\textstyle\left\langle\frac{i}{\hbar}J_{A}\phi^{A}\left[\overleftarrow{U}^{a}\mu_{a}\left(\Gamma|-{F}^{\prime}\right)+\frac{1}{4}\overleftarrow{U}^{2}\mu^{2}\left(\Gamma|-{F}^{\prime}\right)\right]\right.\right.$
$\displaystyle\
-\left.\left.\textstyle(-1)^{\varepsilon_{B}}\left(\frac{i}{2\hbar}\right)^{2}J_{B}J_{A}\left(\phi^{A}\overleftarrow{U}{}^{a}\right)\left(\phi^{B}\overleftarrow{U}_{a}\right)\mu^{2}\left(\Gamma|-{F}^{\prime}\right)\right\rangle_{F,J}\right\\},$
(39)

with source-dependent average expectation value with respect to $Z_{F}(J)$
corresponding to a gauge-fixing $F(\phi)$.

By choosing the $N=1$ or $N=2$ SB BRST symmetry term $M(\phi)$ as the horizon
functional $H(A)$ in Landau (or Coulomb) gauge, and assuming the gauge
independence of the path integrals $Z_{H,\psi}$, $Z_{H,F}$ under a finite
change of the gauge condition, $\psi\rightarrow\psi+\psi^{\prime}$ or
$F\rightarrow F+F^{\prime}$, one can determine the functional $H(A)$ in a new
reference frame, $\psi+\psi^{\prime}$ or $F+F^{\prime}$, of the respective
$N=1,2$ BRST symmetry setting, with account taken of (35), (37):

$H_{\psi^{\prime}}(\phi)=H(A)\left\\{1+\overleftarrow{s}\mu(\psi^{\prime})\right\\}\
\mathrm{or}\
H_{F^{\prime}}(\phi)=H(A)\left\\{1+\overleftarrow{s}{}^{a}\mu_{a}(F^{\prime})+\frac{1}{4}\overleftarrow{s}{}^{2}\mu^{2}(F^{\prime})\right\\}.$
(40)

Notice in conclusion that the above $N=1,2$ FD BRST transformations make it
possible to study their influence on the Yang–Mills, Gribov–Zwanziger,
Freedman–Townsend models, and the Standard Model, as well as on the concept of
average effective action [1, 2, 3, 5, 6]. The case of functionally-dependent
parameters
$\mu_{a}\left(\Gamma\right)=\Lambda\left(\Gamma\right)\overleftarrow{s}^{a}+\psi_{a}\left(\Gamma\right)$
with a vanishing BRST-antiBRST “divergence”,
$\psi_{a}\overleftarrow{s}^{a}=0$, was examined in Sec. 4.1. of Ref. [6] and
implies a modified form of the compensation equation, due to a nontrivial
contribution of $\psi_{a}$ to the corresponding Jacobian.

### Acknowledgments

The authors are grateful to the organizers of the International Workshop
SQS’15 for kind hospitality, as well as to M.O. Katanaev and K.V. Stepanyantz
for their interest and discussions. The study was supported by the RFBR grant
No. 16-42-700702, and was also partially supported by the Tomsk State
University Competitiveness Improvement Program.

## References

  * [1] Reshetnyak A. // Int. J. Mod. Phys. A. 2014. V. 29. P. 1450128\. [arXiv: 1312.2092[hep-th]].
  * [2] Moshin P.Yu., Reshetnyak A.A. // Nucl. Phys. B. 2014. V. 888. P. 92–128. [arXiv: 1405.0790 [hep-th]].
  * [3] Moshin P.Yu., Reshetnyak A.A. // Phys. Lett. B. V. 739. P. 110–117. [arXiv: 1406.0179[hep-th]].
  * [4] Moshin, P.Yu., Reshetnyak, A.A. // Int. J. Mod. Phys. A. 2014. V. 29. P. 1450159 [arXiv: 1405.7549 [hep-th]].
  * [5] Moshin P.Yu., Reshetnyak A.A. // Int. J. Mod. Phys. A. 2015. V. 30. P. 1550021. [arXiv: 1406.5086 [hep-th]].
  * [6] Moshin P.Yu., Reshetnyak A.A. // Int. J. Mod. Phys. A. 2016. V. 31. P. 1650111. [arXiv: 1506.04660 [hep-th]].
  * [7] Becchi C., Rouet A., Stora R. // Phys. Lett. B. 1974. V. 52. P. 344–346.
  * [8] Tyutin I.V. // Lebedev Inst. preprint. 1975. No. 39, [arXiv:0812.0580[hep-th]]
  * [9] Batalin I.A., Vilkovisky G.A. // Phys. Lett. B. 1981. V. 102. P. 27–31.
  * [10] Curci G., Ferrari R. // Phys. Lett. B. 1976. V. 63. P. 91–94.
  * [11] Alvarez-Gaume L., Baulieu L. // Nucl. Phys. B. 1983. V. 212\. P. 255–267.
  * [12] Spiridonov V.P. // Nucl. Phys. B. 1988. V. 308. P. 527–543.
  * [13] Batalin I.A., Lavrov P.M., Tyutin I.V. // J. Math. Phys. 1990\. V. 31. P. 1487–1493.
  * [14] Joglekar S.D., Mandal B.P. // Phys. Rev. D. 1995. V. 51 P. 1919–1928.
  * [15] Capri M.A.L., Gomes A.J., Guimaraes M.S., Lemes V.E.R., Sorella S.P., Tedesko D.G. _Renormalizability of the linearly broken formulation of the BRST symmetry in presence of the Gribov horizon in Landau gauge Euclidean Yang–Mills theories_ // Phys. Rev. D. 2011. V. 83. P.105001. [arXiv:1102.5695 [hep-th]].
  * [16] Gribov V.N. // Nucl. Phys. B. 1978. V. 139. P. 1–19.
  * [17] Zwanziger D. // Nucl. Phys. B. 1989. V. 323. P. 513–544.
  * [18] Lavrov P., Lechtenfeld O., Reshetnyak A. // JHEP. 2011. V. 1110\. P. 043. [arXiv:1108.4820 [hep-th]].
  * [19] Upadhyay S., Mandal P. _Relating Gribov–Zwanziger theory and Yang–Mills theory in Batalin–Vilkovisky formalism_ // AIP. Conf. Proc. 2011. V. 1444. P. 213–219. [arXiv:1112.0422 [hep-th]].
  * [20] Lavrov P., Lechtenfeld O. // Phys. Lett. B. 2013. V. 725. P. 382–385. [arXiv: 1305.0712[hep-th]].
  * [21] Lavrov P., Lechtenfeld O. // Phys. Lett. B. 2013. V. 725. P. 386–388. [arXiv: 1305.2931[hep-th]].
  * [22] Faddeev L.D., Popov V.N. // Phys. Lett. B. 1967. V. 25. P.29-31.
  * [23] Buchbinder I.L., Reshetnyak A.A. // Nucl. Phys. B. 2012. V. 862. P. 270–326. [arXiv:1110.5044[hep-th]].
  * [24] Reshetnyak A.A. _Gauge-invariant Lagrangians for mixed-antisymmetric higher spin fields_ // [arXiv:1604.00620[hep-th]].
  * [25] Batalin I.A., Lavrov P.M., Tyutin I.V. // Int. J. Mod. Phys. A. 2014. V. 29. P. 1450166. [arXiv:1405.2621].
  * [26] Batalin I.A., Lavrov P.M., Tyutin I.V. // Int. J. Mod. Phys. A. 2014. V. 29. P. 1450167. [arXiv:1406.4695 [hep-th]].
  * [27] Moshin P.Yu., Reshetnyak A.A. // TSPU Bulletin. 2014. V. 12. P. 189–194. [arXiv:1412.0226[hep-th]].


# Memorizing Transformers

Yuhuai Wu  Markus N. Rabe  DeLesley Hutchins  Christian Szegedy
<EMAIL_ADDRESS>
###### Abstract

Language models typically need to be trained or finetuned in order to acquire
new knowledge, which involves updating their weights. We instead envision
language models that can simply read and memorize new data at inference time,
thus acquiring new knowledge immediately. In this work, we extend language
models with the ability to memorize the internal representations of past
inputs. We demonstrate that an approximate $k$NN lookup into a non-
differentiable memory of recent (key, value) pairs improves language modeling
across various benchmarks and tasks, including generic webtext (C4), math
papers (arXiv), books (PG-19), code (Github), as well as formal theorems
(Isabelle). We show that the performance steadily improves when we increase
the size of memory up to 262K tokens. On benchmarks including code and
mathematics, we find that the model is capable of making use of newly defined
functions and theorems during test time.

##  1 Introduction

Transformers (Vaswani et al., 2017) have led to remarkable progress in natural
language processing (Devlin et al., 2019; Brown et al., 2020), mathematical
reasoning (Polu & Sutskever, 2020; Wang et al., 2020a; Rabe et al., 2021; Li
et al., 2021; Hahn et al., 2021; Cobbe et al., 2021), and program synthesis
(Austin et al., 2021; Chen et al., 2021; Li et al., 2022). However,
transformer performance on many of these tasks is limited by the context
length of attention, which is typically short. The ability to attend to far-
away tokens is important in many situations. In novels, characters and events
are referenced across multiple chapters. In source code, references to classes
and functions may occur quite far from the places in which they are defined.
In theorem proving, proofs make use of previously defined lemmas.

Attention over long sequences is also useful as a form of rapid learning.
Facts and information which are stored in the form of weight matrices must be
slowly trained over hundreds of thousands of training steps. By using
attention, however, a model can simply _memorize_ facts (e.g. function
definitions) by storing them as (key, value) pairs in long-term memory, and
then retrieve those facts later by creating a query that attends to them. In
this case, attention acts as a form of information retrieval, allowing the
model to look up facts that it has seen previously.

We demonstrate that a simple and effective way to increase the size of the
attention context is to use approximate $k$-nearest-neighbor ($k$NN) lookup,
which is widely used in information retrieval. A number of extremely scalable
implementations of $k$NN lookup are available, such as ScaNN (Guo et al.,
2020) and Faiss (Johnson et al., 2021).

There are two things which distinguish our approach from previous work on
long-range attention (c.f. Section 2). First, unlike some other approaches,
$k$NN lookup does not do averaging or summarization of tokens at long
distances, but retrieves exact values even from the distant context.

Second, gradients are not backpropagated into the external memory, which is
critical to the scalability of our technique. The keys and values are a
function of model parameters, so attempting to backpropagate gradients into
external memory would necessarily involve computing all of the keys and values
with the current model parameters on every training step. However, if the
external memory is not differentiable, then we can instead instead reuse keys
and values that were previously computed on prior training steps, which
drastically reduces the amount of computation for large memories. With our
technique, we are easily able to scale external memory up to sequence lengths
of 131k or 262k tokens on a single TPU device, while maintaining a reasonable
step time.

Figure 1: Adding a memory of 8K tokens improves perplexity across different
model sizes.

We show that model perplexity steadily improves with the size of external
memory on a variety of language modelling tasks, including C4 (long documents
only), Github code repositories, PG-19 books, formal proofs in Isabelle, and
arXiv math papers. We further show that models can generalize to larger memory
sizes than they were trained on: models trained with a small memory show gains
from using a much larger memory at inference time. Finally, we show that our
models are actually using memory in the way that we had hoped, e.g. by looking
up the definitions of lemmas in a theorem proving corpus.

The simplicity of the changes to the Transformer architecture allows us to
easily integrate this approach into existing code bases, including extremely
large language models. We further show that the improvements to quality are
maintained across models of increasing size, and that the model improvements
gained from adding memory are even larger than increasing the size of the
model by 5X or more as shown in Figure 1.

##  2 Related Work

A great deal of work has been done on efficient long-range attention
mechanisms; see Tay et al. (2020; 2021) recent surveys. Sliding windows
(Beltagy et al., 2020) use a long sequence, but attend within a smaller
window, thus reducing complexity to the window size, rather than total
sequence length. Approximate mechanisms such as Linformer (Wang et al.,
2020b), and Performer (Choromanski et al., 2021) refactor the attention matrix
by using a different kernel than softmax to obtain $O(N)$ complexity. Pooling
strategies such as Hierarchical 1D attention (Zhu & Soricut, 2021), and
Combiner (Ren et al., 2021) apply pooling or averaging over tokens at longer
distances. Sparse strategies such as Big Bird (Zaheer et al., 2020) select
only a subset of tokens to attend to; Routing Transformers (Roy et al., 2021)
use clustering to select the subset, while Reformer (Kitaev et al., 2020)
relies on hashing. Hierarchical mechanisms (Ainslie et al., 2020) combine
multiple tokens into phrases or sentences to reduce sequence length. Expire-
span (Sukhbaatar et al., 2021) prunes far-away tokens that it learns are
“unimportant”. (Zemlyanskiy et al., 2021) process long sequences in two passes
with different encoders. The second pass is given a lot of context by
accessing summaries of the first pass.

Feedback transformers (Fan et al., 2020) use a recurrent architecture in which
each token attends to the output of the final layer instead of the previous
layer. Recurrence does not increase the size of the attention context itself,
but it expands the receptive field at the cost of parallelism and training
speed.

Truncated backpropagation through time (Williams & Peng, 1990) was originally
introduced as a way of training recurrent neural networks (RNN) over very long
sequences, when the entire sequence does not fit in memory. The sequence is
chopped into segments, and after each training step, the final RNN state for
the segment is saved in a non-differentiable cache, and used as the initial
state on the next training step. Neural caches (Grave et al., 2017) extend the
cache to contain a record of many prior hidden states, and attend over them.
Transformer-XL (Dai et al., 2019) applies this technique to transformers; it
caches the (key,value) pairs computed from the previous training step, and
uses them as a prefix for the tokens on the next training step, which yields
significant gains on long documents. Rae et al. (2020) improve over
Transformer-XL by compressing the tokens before adding them to the cache. In
contrast, we use a very large cache without compression, combined with an
approximate $k$NN attention mechanism over it.

Sukhbaatar et al. (2019) make the observation that the feed-forward portion of
a transformer layer functions very much like attention if one replaces the
ReLU activation with softmax. They implement a combined attention over both
tokens from the input sequence and a learned (and differentiable) “memory”.
Lample et al. (2019) exploit this observation to replace the feed-forward
layers (FFNs) with a fast $k$NN lookup over a much larger “memory”, and
achieve large gains in model accuracy without significant computation
overhead. (We use $k$NN lookup to approximate attention to previous tokens,
not to replace the FFN.)

Non-differentiable external memory has been used in different ways by
Khandelwal et al. (2020), who run a pre-trained model over an entire corpus,
and construct a large table of (key, token) pairs. They then use that table to
replace the final softmax layer for token selection in the model, which
results in significant improvements in language modeling. Yogatama et al.
(2021) extend this approach by a gating mechanism and a process to compress
the context into keys for retrieval.

There are several works that combine retrieval with transformers. REALM (Guu
et al., 2020), MARGE (Lewis et al., 2020a), RAG (Lewis et al., 2020b), and
composite memory for dialog (Fan et al., 2021) retrieve documents from a
knowledge base to improve question answering or dialogue. The knowledge base
consists of text snippets and is static and typically separate from the inputs
and outputs of the models. Instead, we focus on language modeling using a
decoder-only model, and propose a simple model that _unifies attention and
retrieval_.

$k$-nearest-neighbor lookup is a general-purpose technique that is used for a
wide variety of machine learning and retrieval tasks, and high-performance
implementations are available for various architectures (Johnson et al., 2021;
Guo et al., 2020). Memory-efficient Transformers (Gupta et al., 2021) replace
dense attention with a $k$NN lookup to increase speed and reduce memory usage.

Figure 2: We extend Transformers with access to (key, value) pairs of
previously seen subsequences.

##  3 Method

The architecture of our $k$NN-augmented transformer is shown in Figure 2. The
bulk of the model is a vanilla, decoder-only transformer (Vaswani et al.,
2017). The input text is tokenized, and the tokens are embedded into vector
space. The embedding vectors are passed through a series of transformer
layers, each of which does dense self-attention, followed by a feed-forward
network (FFN). Since this is a decoder-only language model, we use a causal
attention mask and the token embeddings of the last layer are used to predict
the next token.

Long documents are split into subsequences of 512 tokens, and each subsequence
is used as the input for one training step. In contrast to standard practice,
we do not shuffle the subsequences; instead, each long document is fed into
the transformer sequentially, from beginning to end, as is done with
Transformer-XL (Dai et al., 2019).

We also use a Transformer-XL style cache, which holds the keys and values from
the previous training step. When doing self-attention, the cached keys and
values are prepended to the current keys and values, and we use a sliding-
window causal mask (Beltagy et al., 2020) so that each token has a local
context that includes the previous 512 tokens.

###  3.1 $k$NN-augmented Attention Layer

One of the transformer layers near the top of the stack is a _$k$ NN-augmented
attention layer_, which combines two forms of attention. Like all of the other
layers, it uses standard dense self-attention on the _local context_ , which
is the input subsequence for the current training step. Unlike the other
layers, however, it also does an approximate $k$-nearest-neighbor search into
the _external memory_.

The same queries are used for both the local context, and for the external
memory. The keys and values also belong to the same distribution; after each
training step, the (key, value) pairs in the local context are appended to the
end of the external memory. If the document is very long, old (key, value)
pairs will be dropped from the memory to make room for new ones. Thus, for
each head, the external memory keeps a cache of the prior $M$ (key, value)
pairs, where $M$ is the memory size.

The $k$NN lookup will return a set of _retrieved memories_ , which consist of
the top-$k$ (key, value) pairs that $k$NN search returns for each query (i.e.
each token) in the input subsequence. As with standard dense attention, we
first construct an attention matrix by computing the dot product of each query
against the retrieved keys, then apply softmax, and finally return a weighted
sum of the retrieved values. Unlike standard dense attention, the retrieved
memories contain a different set of (key, value) pairs for each query.

Attention over the local context is performed in the usual way. The results of
$k$NN-attention and local attention are then combined using a learned gate:

$\displaystyle g$ $\displaystyle=\sigma(b_{g})$ (1) $\displaystyle\bm{V}_{a}$
$\displaystyle=\bm{V}_{m}\odot g+\bm{V}_{c}\odot(1-g)$ (2)

where $\sigma$ is the sigmoid function, and $\odot$ is element-wise
multiplication. $\bm{V}_{a}$ is the combined result of attention, $\bm{V}_{m}$
is the result of attending to external memory, and $\bm{V}_{c}$ is the result
of attending to the local context. The _bias_ $b_{g}$ is a learned per-head
scalar parameter, which allows each head to choose between local and long-
range attention. In our experiments, the value of the gate $g$ does not depend
on the content of the token at each position, although that would be a trivial
extension to implement. We did observe that over time, most heads learned to
attend almost exclusively to external memory.

Position bias. For dense attention within the local context, we use the T5
relative position bias (Raffel et al., 2020). As noted by Dai et al. (2019),
adding a global position encoding to each token does not work well when
processing long documents. We don’t use a position bias for the retrieved
memories. Experiments on the PG19 dataset (Sun et al., 2021) have shown that
relative position does not appear to matter at long range, and the T5 relative
bias puts all long-range tokens in the same bucket anyway.

Batching. Figure 3 illustrates how multiple long documents of different
lengths are packed into a batch, and split into subsequences. Each subsequence
in the batch comes from a different document, and thus requires a separate
external memory, which is cleared at the start of each new document.

Figure 3: Our data pipeline splits documents into subsequences and packs
subsequences into batches.

###  3.2 Distributional Shift

Because each long document is processed over multiple training steps, there is
a distributional shift in the keys and values that are stored in external
memory. The model parameters that produce the queries change over time, and
will thus have shifted since the keys and values were stored. For very large
memories, older records may become “stale.” Similar observations have been
made for CrossBatch memory (Wang et al., 2020c) in the vision domain.

To reduce the effects of staleness, we normalize keys and queries (Henry et
al., 2020). Normalization does not eliminate staleness, but it at least
ensures that older keys and newer keys do not differ in magnitude. We also
found that normalization helps stabilize training with the Transformer-XL
cache.

In some of our experiments, we observed that training models from scratch with
a large memory sometimes resulted in worse performance than pretraining the
model with a small memory of size 8192, and then finetuning it on a larger
memory. This training instability could be due to staleness. However, models
seem to be able to cope with a limited degree of staleness (with the small
memory) by adjusting their queries accordingly.

###  3.3 Approximate $k$NN

We employ _approximate_ $k$NN search rather than exact $k$NN search because it
significantly improves the computational speed of our model. We use a simple
approximation of $k$NN for TPUs, which has a recall of about 90%, i.e. 90% of
the true top $k$ are returned in the approximate top $k$. There are various
other efficient approximate $k$NN algorithms available for CPU and GPU/TPU,
for example through Faiss (Johnson et al., 2021) or ScaNN (Guo et al., 2020),
which can scale into the billions.

##  4 Experiments

We evaluate the effect of adding external memory on five language modeling
tasks, all of which involve long-form text: English language books (PG-19),
long web articles (C4), technical math papers (arXiv Math), source code
(Github), and formal theorems (Isabelle). The results show significant
improvements in the perplexity of the model with the addition of external
memory. We experimented with various sizes of external memory, from 1536 to as
high as 262K. On most of the datasets, there was an initial sharp gain from
adding a small external memory, followed by smaller but steadily increasing
gains as the size of the memory was increased.

###  4.1 Datasets

##### arXiv Math

For the arXiv dataset, we collected a corpus of papers by downloading them via
the arXiv Bulk Data Access111https://arxiv.com/help/bulk_data. We filtered
papers to include only articles labeled as “Mathematics” and whose LaTeX
source was available. The number of tokens per paper in this dataset is
roughly comparable to the number of tokens per book in PG19, because LaTeX
source has many special characters and the tokenizer tends to output small
subwords.

##### Github

We used
BigQuery222https://console.cloud.google.com/marketplace/product/github/github-
repos to obtain a large corpus of Github repositories that are published with
open-source licenses. We used file endings to filter for files in the
languages C, C++, Java, Python (including Jupyter notebooks), Go, and
TypeScript. Individual source code files are often fairly short, and there are
many dependencies and cross-references between files in the repository. To
capture these dependencies, we created one long document for each Github
repository by traversing the directory tree, and concatenating all of the
files within it. The order in which files are traversed within the repository
is random, but each subdirectory is processed as a unit, so that all the files
within the subdirectory are close to each other in the resulting document.
Source code is usually structured so that related files are all grouped
together in the same subdirectory; this traversal preserves that structure,
while still shuffling files and subdirectories in random order.

##### Formal Math – Isabelle

The Isabelle corpus consists of formal mathematical proofs of theories. We
collected all 627 theories available on The Archive of Formal
Proofs333https://www.isa-afp.org/topics.html (as of October 6, 2021) and an
additional 57 theories from the Isabelle standard library444
https://isabelle.in.tum.de/ to create a corpus of 684 theories. All theories
have open-source licenses. Each theory is a self-contained mathematical
object, on topics such as foundational logic, advanced analysis, algebra, or
cryptography, and consists of multiple files containing proofs. As with the
Github corpus, all files that make up a theory are concatenated together into
one long document. Unlike the Github corpus, we order the files according to
their import dependencies, so that later files use sub-theorems that are
proved in earlier files.

##### C4(4K$+$)

C4, the colossal cleaned common crawl, is a very large collection of documents
that have been scraped from the internet (Raffel et al., 2020). We filtered
out all documents that have less than 4096 tokens to focus on documents where
memory can have an impact.

##### PG-19

PG-19 is a large dataset of English-language books, published prior to 1919,
which were retrieved from the Project Gutenberg archive (Rae et al., 2020; Sun
et al., 2021). PG-19 is one of the few public datasets that only contains
full-length books, and has become a benchmark for long-range natural language
text modeling.

###  4.2 Experimental Method

We used a 12-layer decoder-only transformer (with and without Transformer-XL
cache) with an embedding size of 1024, 8 attention heads of dimension 128, and
an FFN hidden layer of size 4096. For all of our experiments, we used $k=32$.
Unless specified otherwise, we use the $9$th layer as the $k$NN augmented
attention layer. We used a sentence-piece (Kudo & Richardson, 2018) tokenizer
with a vocabulary size of 32K.

We used the Adafactor optimizer (Shazeer & Stern, 2018). In preliminary
experiments, we conducted a hyperparameter search to determine the optimal
learning rate among three choices ($\\{$$3.0$, $1.0$, $3\cdot 10^{-1}$$\\}$),
and found that $1.0$ works best. We used a linear warmup schedule for the
first 1000 steps, followed by square root decay. We trained the models from
scratch for 500K steps on all the datasets, except for the Isabelle dataset.
Isabelle is small, so we stopped training after 100K steps when the model
began to overfit. We ran all of our experiments on 32 TPU cores. Our models
were implemented in JAX (Bradbury et al., 2018) and Flax (Heek et al., 2020).

When comparing models with different context lengths, we adjusted the batch
size (the number of documents in a batch) so that there are always $2^{17}$
tokens in a batch. E.g., a model with a context length of 512 has a batch size
of 256, while the 2048 model has a batch size of 64.

We experimented with multiple implementations of approximate $k$NN lookup with
different tradeoffs between quality and computational cost. We did not observe
a significant degradation of the model quality when switching to lower quality
approximations of $k$NN, so the model appears to be quite robust with respect
to the quality of $k$NN retrieval. For a model with around 200M trainable
parameters the step time increased from 0.2s to 0.25s when we added a memory
of size 8K, and to 0.6s when we added a memory of size 65K (measured on
TPUv3).

###  4.3 Effect of external memory

Context | Memory | XL cache | arXiv | PG19 | C4(4K$+$) | GitHub | Isabelle
---|---|---|---|---|---|---|---
512 | None | None | 3.29 | 13.71 | 17.20 | 3.05 | 3.09
2048 | None | None | 2.69 | 12.37 | 14.81 | 2.22 | 2.39
512 | None | 512 | 2.67 | 12.34 | 15.38 | 2.26 | 2.46
2048 | None | 2048 | 2.42 | 11.88 | 14.03 | 2.10 | 2.16
512 | 1536 | None | 2.61 | 12.50 | 14.97 | 2.20 | 2.33
512 | 8192 | None | 2.49 | 12.29 | 14.42 | 2.09 | 2.19
512 | 8192 | 512 | 2.37 | 11.93 | 14.04 | 2.03 | 2.08
512 | 65K | 512 | 2.31 | 11.62 | 14.04 | 1.87 | 2.06
2048 | 8192 | 2048 | 2.33 | 11.84 | 13.80 | 1.98 | 2.06
2048 | 65K | 2048 | 2.26 | 11.37 | 13.64 | 1.80 | 1.99
Table 4: Average token-level perplexities of each model when trained for 500k
steps.

##### Adding external memory results in substantial gains across datasets and
architectures,

as shown in Table 4. Across all five datasets, adding external memory to
either the vanilla Transformer or the Transformer-XL architecture improves
perplexity by a substantial amount. For example, on C4(4K+) dataset, adding
memory of size 8192 improves the perplexity of the vanilla Transformer (with
context size 512) from 17.20 to 14.42, and improves Transformer-XL from 15.38
to 14.04.

##### Increasing the size of the memory increases the benefit of the memory.

The best perplexities for all datasets and architectures were obtained with a
memory size of 65K.

Note that Transformer-XL with context size 2048 already has a theoretical
receptive field that is quite large. Each token in a higher layer can attend
up to 2048 tokens away in the layer below, so the total receptive field is
$2048\cdot 12$ (layers) $\sim 25$K. Nevertheless, we still saw a substantial
gain when adding an external memory of size 8192 to this model. $k$NN
attention into memory would appear to be a more effective way to retrieve
information from the distant past than the Transformer-XL cache.

On the other hand, we also saw improvements by adding XL cache to the large-
memory (65K) models. In a vanilla (non-XL) Transformer, the first few tokens
in a sequence have very little context, and thus have higher perplexity. The
XL cache provides additional local short-range context at the start of a
sequence, which complements the long-range context provided by external
memory.

Interestingly, in a vanilla Transformer, using even a small external memory of
size 1536 provides a gain in perplexity which is almost as good as using a
local context of size 2048 but no memory (e.g. Table 4). This is surprising,
because the external memory is not differentiable, and is added only to one
layer of the Transformer, whereas increasing the context size is
differentiable and affects all layers. We conclude that the lower layers of a
Transformer don’t necessarily need long-range context, and having a
differentiable memory is not as important as one might suspect.

###  4.4 Scaling to larger models

We scaled up the Transformer model to sizes of $1$ and $8$ billion parameters.
For the 1 billion parameter model, we use 8 layers, 32 heads with head
dimension 128, $d\\_\textrm{model}$ 2048, and $d\\_\textrm{ff}$ 16384\. For
the 8 billion parameter model, we use 64 heads, 16 layers,
$d\\_\textrm{model}$ 4096, and $d\\_\textrm{ff}$ 32768\. We used a context
size of 2048, memory size of 8192, and no XL cache. We ran the comparisons to
the vanilla Transformer on the arXiv math dataset. Scaling plots are shown in
Figure 1.

External memory provides a consistent improvement to the model as it is scaled
up. Remarkably, we found that the smaller Memorizing Transformer with just 8k
tokens in memory can match the perplexity of a larger vanilla Transformer
which has 5X more trainable parameters.

Context | Pretrain | Fine-tune | Perplexity
---|---|---|---
512 | 8192 | None | 2.37
512 | 65K | None | 2.31
512 | 8192 | 65K | 2.32
512 | 8192 | 131K | 2.30
512 | 8192 | 262K | 2.26
2048 | 8192 | None | 2.33
2048 | 65K | None | 2.26
2048 | 65K | 131K | 2.23
2048 | 65K | 262K | 2.21
Table 5: Finetuning for 20K steps to make use of a larger memory on the arXiv
data set.

###  4.5 Finetuning on Larger Memories

##### Finetuning on a larger memory.

In some cases, training was unstable when using large memories, possibly due
to distributional shift early in the training (See Section 3.2). Thus, for
memories of 131K or more tokens, we first pretrain the model with a memory
size of 8192 or 65K for 500K steps, and then finetune it with the larger
memory for an additional 20K steps. The results of finetuning on the arXiv
Math data set are shown in Table 5. Increasing the size of external memory
provided consistent gains up to a size of 262K. Note that 262K tokens is
longer than almost all of the documents in arXiv, and thus we would not expect
to see any gain past this point (see Appendix A).

Figure 6: Finetuning a 1B vanilla Transformer model to use external memory of
size 65K.

##### Finetuning a non-memory model to use memory

Pretraining can be very costly both in time and computational resources. Thus,
a natural question to ask is: _can one fine-tune a pretrained Transformer to
use external memory?_ The answer is yes!

We took a pre-trained 1B vanilla Transformer model, and fine-tuned it to use
external memory (the 1B models used in Section 4.4). The fine-tuning result is
shown in Figure 6. Notice that the model quickly learns to use external
memory. Within 20K steps ($4\%$ of the pre-training time) the fine-tuned model
has already closed $85\%$ of the gap between it and the 1B Memorizing
Transformer, and after 100k steps it has closed the gap entirely.

###  4.6 Information Retrieval Patterns

We conducted a qualitative study of what the model was actually retrieving
from external memory, by finding which tokens showed the biggest improvements
in cross-entropy loss when the size of the memory was increased, and then
examining the top-$k$ retrieved memories for those tokens. We found that the
model gained the most when looking up rare words, such as proper names,
references, citations, and function names, where the first use of a name is
too far away from subsequent uses to fit in the local context. This result is
in keeping with the prior analysis of long-context Transformers on PG19 (Sun
et al., 2021), which found similar lookup patterns. For this experiment, we
used a slightly older version of the architecture without the gating
mechanism.

##### Which tokens show a benefit from memory?

Figure 7 shows a visualization of which tokens show an improvement when the
size of the external memory is increased. We selected a math paper at random,
and plotted the difference in cross entropy loss for each token $x_{i}$ in the
paper, comparing two models with the same parameters, but with memories of
different sizes. $\Delta_{i}=\textrm{cross-
entropy}_{8192}(x_{i})\linebreak[0]-\textrm{cross-
entropy}_{32\mathrm{K}}(x_{i})$. Positive values show an improvement in loss.

The $x$-axis on the chart is the token number $i$, while the $y$-axis is
$\Delta_{i}$. For the first 8192 tokens, the difference between the two models
is zero, since the larger capacity of the $32$K memory isn’t being used yet.
However, after token $8193$, we can see that the larger memory helps, on
average, over the smaller memory. The benefit is not universal, since the
predictions for some tokens become worse, possibly due to the fact that a
relevant retrieved memory no longer makes it into the top-$k$ when the size of
the external memory is increased. This figure also shows that the benefit of
external memory is somewhat sparse. The improvement in perplexity seems to be
mainly driven by a small percentage of tokens that obtain a large improvement
in cross-entropy loss when using the larger memory.

##### What information is being looked up?

Given that only a subset of tokens shows improvement from external memory, we
did a further investigation into what, exactly, those tokens are using the
memory for. We took those tokens which showed the largest improvement in
cross-entropy loss, and for each of them tokens, we examined the top-$k$
retrieved memories. We studied arXiv math, Github and Isabelle corpus. For
arXiv math and Github, we found the model retrieved function and variable
names. See more details with examples in Appendix B.

Figure 7: Difference in loss for each token in a randomly chosen paper, using
the same model once with a memory size of $8$K and once with $32$K. Higher
numbers mean the longer memory helped in comparison to the shorter memory.
This paper is $22$K tokens long.

Query index Input Target Surrounding context Retrieved index Retrieved
surrounding context 29721 mark ov rule prob_space.markov_inequality 8088 M. t
$\backslash$<le> X a} $\backslash$<le> expectation X $/$ t" 40919 $\\_$ th =
(subgraph_threshold H n / p n) 27219 threshold H n = n powr (-(1 /
max_density’ 49699 S w assumes "orthonormal_system S w" 28050 definition
orthonormal_system :: "

Table 8: Examples of memory retrieval in the Isabelle dataset. The model is
able to find the definition of a lemma from a reference to it. The retrieved
surrounding context (highlighted) is the definition body of the mathematical
object highlighted in the querying context.

_Retrieving mathematical definitions._ Our case study on the Isabelle corpus
provides one of the clearest illustrations of how a model can make good use of
external memory. When predicting the name of a mathematical object or a lemma,
the model looked up the definition from earlier in the proof. Examples of this
behavior are shown in Table 8. In example 1, the model retrieves a definition
within the body of a lemma, markov_inequality. In example 2, it retrieves the
definition of a previously defined concept subgraph_threshold. In example 3,
it retrieves the definition of orthonormal_system. We manually checked 10
examples where the model made a prediction of lemma names, and 8 out of 10
times model found the body of the lemma it needs to predict. In the other two
cases, the model also looked up materials in the immediate vicinity. To the
best of our knowledge, this is the first demonstration that attention is
capable of looking up definitions and function bodies from a large corpus. The
Isabelle case study used a model with two memory layers of size $32$K.

##  5 Conclusion

We present a simple extension to the Transformer architecture, called $k$NN-
augmented attention, which dramatically increases the length of the context
that a language model can attend to by using $k$-nearest-neighbor lookup into
a large external memory. We demonstrate the effectiveness of external memory
in a series of language modeling experiments over a variety of long-document
datasets, including LaTeX documents, source code, formal proofs, and books.

The Memorizing Transformer shows large improvements in perplexity over the
baseline for all of the data sets and architectures that we studied; it is
comparable to a vanilla transformer that has 5 times the number of parameters.
Perplexity continues to improve with increasing memory size, although there is
a point of diminishing returns. Moreover, external memory continues to provide
benefits even as the transformer is scaled up from 200M to 8B parameters.
Perhaps most intriguingly, a Memorizing Transformer does not need to be pre-
trained from scratch; it is possible obtain large gains from adding memory to
an existing pre-trained model, and then fine-tuning it.

Unlike other forms of attention, $k$NN retrieval can be easily scaled up to
huge memory sizes, and is thus potentially able to leverage vast knowledge
bases or code repositories. How to make the best use of this capability is a
topic for future work.

#### Acknowledgments

We want to thank Charles Staats for the many fruitful discussions and detailed
comments, Henryk Michalewski for early version of of the memory
implementation, Petros Maniatis for his help with our code datasets, Aitor
Lewkowycz for his help with larger scale memorizing transformer experiments,
Behnam Neyshabur for his comments on finetuning non-memory models, Imanol
Schlag for his proofread and detailed comments, and Dennis Lee and Manzil
Zaheer for discussions about large-scale attention and retrieval.

## Ethics

The ability to memorize large databases of facts could have potential
ramifications for society, especially if those databases include sensitive
personal information or copyrighted works. However, one advantage of using an
external memory is that the memory can be easily cleared of all such
information, as we do at the end of each document that we train on. The same
is not true of differentiable model parameters, which is what most existing
architectures use to store facts and information that they are trained on.

## Reproducibility

Details of our architecture and training hyperparameters are given in Section
4.2. The datasets for C4 and PG-19 are publicly available. Our additional
datasets, Github, Isabelle, and ArXiv Math are derived from publicly available
data buckets, which we link in the main part of the paper. Subsection 4.1
include details on how we constructed the datasets from those datasets. We
plan to release our code as open source.

## References

  * Ainslie et al. (2020) Joshua Ainslie, Santiago Ontañón, Chris Alberti, Vaclav Cvicek, Zachary Fisher, Philip Pham, Anirudh Ravula, Sumit Sanghai, Qifan Wang, and Li Yang.  ETC: encoding long and structured inputs in transformers.  In _EMNLP_ , 2020.
  * Austin et al. (2021) Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie J. Cai, Michael Terry, Quoc V. Le, and Charles Sutton.  Program synthesis with large language models.  _CoRR_ , abs/2108.07732, 2021.  URL https://arxiv.org/abs/2108.07732.
  * Beltagy et al. (2020) Iz Beltagy, Matthew E. Peters, and Arman Cohan.  Longformer: The long-document transformer.  _CoRR_ , abs/2004.05150, 2020.  URL https://arxiv.org/abs/2004.05150.
  * Bradbury et al. (2018) James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Milne, and Qiao Zhang.  JAX: composable transformations of Python+NumPy programs, 2018\.  URL http://github.com/google/jax.
  * Brown et al. (2020) Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei.  Language models are few-shot learners.  In _NeurIPS_ , 2020.
  * Chen et al. (2021) Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harrison Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Joshua Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba.  Evaluating large language models trained on code.  _CoRR_ , abs/2107.03374, 2021.  URL https://arxiv.org/abs/2107.03374.
  * Choromanski et al. (2021) Krzysztof Marcin Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamás Sarlós, Peter Hawkins, Jared Quincy Davis, Afroz Mohiuddin, Lukasz Kaiser, David Benjamin Belanger, Lucy J. Colwell, and Adrian Weller.  Rethinking attention with performers.  In _ICLR_ , 2021.
  * Cobbe et al. (2021) Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman.  Training verifiers to solve math word problems.  _CoRR_ , abs/2110.14168, 2021.  URL https://arxiv.org/abs/2110.14168.
  * Dai et al. (2019) Zihang Dai, Zhilin Yang, Yiming Yang, Jaime G. Carbonell, Quoc Viet Le, and Ruslan Salakhutdinov.  Transformer-XL: Attentive language models beyond a fixed-length context.  In _ACL_ , 2019.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  BERT: pre-training of deep bidirectional transformers for language understanding.  In _ACL_ , 2019.
  * Fan et al. (2020) Angela Fan, Thibaut Lavril, Edouard Grave, Armand Joulin, and Sainbayar Sukhbaatar.  Addressing some limitations of transformers with feedback memory.  _arXiv preprint arXiv:2002.09402_ , 2020.
  * Fan et al. (2021) Angela Fan, Claire Gardent, Chloé Braud, and Antoine Bordes.  Augmenting transformers with KNN-based composite memory for dialog.  _Transactions of the Association for Computational Linguistics_ , 9:82–99, 2021.
  * Grave et al. (2017) Edouard Grave, Armand Joulin, and Nicolas Usunier.  Improving neural language models with a continuous cache.  In _ICLR_ , 2017.
  * Guo et al. (2020) Ruiqi Guo, Philip Sun, Erik Lindgren, Quan Geng, David Simcha, Felix Chern, and Sanjiv Kumar.  Accelerating large-scale inference with anisotropic vector quantization.  In _ICML_ , 2020.
  * Gupta et al. (2021) Ankit Gupta, Guy Dar, Shaya Goodman, David Ciprut, and Jonathan Berant.  Memory-efficient transformers via top-k attention.  _CoRR_ , abs/2106.06899, 2021.  URL https://arxiv.org/abs/2106.06899.
  * Guu et al. (2020) Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang.  Retrieval augmented language model pre-training.  In _ICML_ , 2020.
  * Hahn et al. (2021) Christopher Hahn, Frederik Schmitt, Jens U. Kreber, Markus Norman Rabe, and Bernd Finkbeiner.  Teaching temporal logics to neural networks.  In _ICLR_ , 2021.
  * Heek et al. (2020) Jonathan Heek, Anselm Levskaya, Avital Oliver, Marvin Ritter, Bertrand Rondepierre, Andreas Steiner, and Marc van Zee.  Flax: A neural network library and ecosystem for JAX, 2020.  URL http://github.com/google/flax.
  * Henry et al. (2020) Alex Henry, Prudhvi Raj Dachapally, Shubham Shantaram Pawar, and Yuxuan Chen.  Query-key normalization for transformers.  In _EMNLP_ , 2020.
  * Johnson et al. (2021) Jeff Johnson, Matthijs Douze, and Hervé Jégou.  Billion-scale similarity search with GPUs.  _IEEE Transactions on Big Data_ , 2021.
  * Khandelwal et al. (2020) Urvashi Khandelwal, Omer Levy, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis.  Generalization through memorization: Nearest neighbor language models.  In _ICLR_ , 2020.
  * Kitaev et al. (2020) Nikita Kitaev, Łukasz Kaiser, and Anselm Levskaya.  Reformer: The efficient transformer.  In _ICLR_ , 2020.
  * Kudo & Richardson (2018) Taku Kudo and John Richardson.  Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing.  In _EMNLP_ , 2018.
  * Lample et al. (2019) Guillaume Lample, Alexandre Sablayrolles, Marc’Aurelio Ranzato, Ludovic Denoyer, and Hervé Jégou.  Large memory layers with product keys.  In _NeurIPS_ , 2019.
  * Lewis et al. (2020a) Mike Lewis, Marjan Ghazvininejad, Gargi Ghosh, Armen Aghajanyan, Sida Wang, and Luke Zettlemoyer.  Pre-training via paraphrasing.  In _NeurIPS_ , 2020a.
  * Lewis et al. (2020b) Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al.  Retrieval-augmented generation for knowledge-intensive NLP tasks.  In _NeurIPS_ , 2020b.
  * Li et al. (2021) Wenda Li, Lei Yu, Yuhuai Wu, and Lawrence C. Paulson.  Isarstep: a benchmark for high-level mathematical reasoning.  In _ICLR_ , 2021.
  * Li et al. (2022) Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, Thomas Hubert, Peter Choy, Cyprien de Masson d’Autume, Igor Babuschkin, Xinyun Chen, Po-Sen Huang, Johannes Welbl, Sven Gowal, Alexey Cherepanov, James Molloy, Daniel J. Mankowitz, Esme Sutherland Robson, Pushmeet Kohli, Nando de Freitas, Koray Kavukcuoglu, and Oriol Vinyals.  Competition-level code generation with alphacode.  _DeepMind_ , 2022.
  * Polu & Sutskever (2020) Stanislas Polu and Ilya Sutskever.  Generative language modeling for automated theorem proving.  _CoRR_ , abs/2009.03393, 2020.  URL https://arxiv.org/abs/2009.03393.
  * Rabe et al. (2021) Markus Norman Rabe, Dennis Lee, Kshitij Bansal, and Christian Szegedy.  Mathematical reasoning via self-supervised skip-tree training.  In _ICLR_ , 2021.
  * Rae et al. (2020) Jack W Rae, Anna Potapenko, Siddhant M Jayakumar, Chloe Hillier, and Timothy P Lillicrap.  Compressive transformers for long-range sequence modelling.  In _ICLR_ , 2020.
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _Journal of Machine Learning Research_ , 2020.
  * Ren et al. (2021) Hongyu Ren, Hanjun Dai, Zihang Dai, Mengjiao Yang, Jure Leskovec, Dale Schuurmans, and Bo Dai.  Combiner: Full attention transformer with sparse computation cost.  _CoRR_ , abs/2107.05768, 2021.  URL https://arxiv.org/abs/2107.05768.
  * Roy et al. (2021) Aurko Roy, Mohammad Saffar, Ashish Vaswani, and David Grangier.  Efficient content-based sparse attention with routing transformers.  _Transactions of the Association for Computational Linguistics_ , 9:53–68, 2021.
  * Shazeer & Stern (2018) Noam Shazeer and Mitchell Stern.  Adafactor: Adaptive learning rates with sublinear memory cost.  In _ICML_ , 2018.
  * Sukhbaatar et al. (2019) Sainbayar Sukhbaatar, Edouard Grave, Guillaume Lample, Herve Jegou, and Armand Joulin.  Augmenting self-attention with persistent memory.  _arXiv preprint arXiv:1907.01470_ , 2019.
  * Sukhbaatar et al. (2021) Sainbayar Sukhbaatar, Da Ju, Spencer Poff, Stephen Roller, Arthur Szlam, Jason Weston, and Angela Fan.  Not all memories are created equal: Learning to forget by expiring.  In _ICML_ , 2021.
  * Sun et al. (2021) Simeng Sun, Kalpesh Krishna, Andrew Mattarella-Micke, and Mohit Iyyer.  Do long-range language models actually use long-range context?  In _EMNLP_ , 2021.
  * Tay et al. (2020) Yi Tay, Mostafa Dehghani, Dara Bahri, and Donald Metzler.  Efficient transformers: A survey.  _arXiv preprint arXiv:2009.06732_ , 2020.
  * Tay et al. (2021) Yi Tay, Mostafa Dehghani, Samira Abnar, Yikang Shen, Dara Bahri, Philip Pham, Jinfeng Rao, Liu Yang, Sebastian Ruder, and Donald Metzler.  Long range arena: A benchmark for efficient transformers.  In _ICLR_ , 2021.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In _NeurIPS_ , 2017.
  * Wang et al. (2020a) Qingxiang Wang, Chad Brown, Cezary Kaliszyk, and Josef Urban.  Exploration of neural machine translation in autoformalization of mathematics in mizar.  In _International Conference on Certified Programs and Proofs_ , 2020a.
  * Wang et al. (2020b) Sinong Wang, Belinda Z Li, Madian Khabsa, Han Fang, and Hao Ma.  Linformer: Self-attention with linear complexity.  _arXiv preprint arXiv:2006.04768_ , 2020b.
  * Wang et al. (2020c) Xun Wang, Haozhi Zhang, Weilin Huang, and Matthew R. Scott.  Cross-batch memory for embedding learning.  In _CVPR_ , 2020c.
  * Williams & Peng (1990) Ronald J. Williams and Jing Peng.  An efficient gradient-based algorithm for on-line training of recurrent network trajectories.  _Neural Computation_ , 1990.
  * Yogatama et al. (2021) Dani Yogatama, Cyprien de Masson d’Autume, and Lingpeng Kong.  Adaptive semiparametric language models.  _ACL_ , 9:362–373, 2021.
  * Zaheer et al. (2020) Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontañón, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, and Amr Ahmed.  Big bird: Transformers for longer sequences.  In _NeurIPS_ , 2020.
  * Zemlyanskiy et al. (2021) Yury Zemlyanskiy, Joshua Ainslie, Michiel de Jong, Philip Pham, Ilya Eckstein, and Fei Sha.  Readtwice: Reading very large documents with memories.  In _ACL: Human Language Technologies_ , 2021.
  * Zhu & Soricut (2021) Zhenhai Zhu and Radu Soricut.  H-transformer-1d: Fast one-dimensional hierarchical attention for sequences.  In _ACL_ , 2021.

##  Appendix A Length of inputs

Figure 9: Histogram of the number of tokens in arXiv math papers dataset. We
tuncated the histogram at 500k tokens. The maximum paper had almost 1.6M
tokens.

Figure 10: Histogram of the number of tokens in Github repositories dataset.
We cut off the long tail of this plot. The repository with the maximum length
has just over 9M tokens.

Figure 11: Histogram of the number of tokens in Isabelle proof scripts
dataset.

Figure 12: Histogram of the number of tokens in PG19 books dataset.

Figure 13: Histogram of the number of tokens in C4 documents filtered by
documents that have less than 4096 tokens.

###  A.1 Ablation studies

In the following section, we performed ablation studies to investigate the
effects of various hyperparameters. Unless otherwise specified, we carried out
these experiments with a memorizing transformer with context size 512, XL
cache 512 with a memory size of 8192.

##### Multiple $k$NN layers.

We experimented with using two $k$NN layers, rather than just one. However, we
did not see further benefits brought by more than multiple retrieval layers.

#####  $k$NN layer index

We experimented with adding the external memory to layer $3$, $6$, $9$ and
$12$ in a 12-layer transformer, with results shown in Table 14. We found that
adding memory to the middle of the layer stack will obtain the best result,
whereas adding memory to layers either too close to the input or to the output
obtained less gains.

Table 14: Different layer index.

Layer index Perplexity 3 2.40 6 2.36 9 2.37 12 2.43

##### Number of neighbors

We studied the effects of the number of neighbors we retrieve from memory,
with results shown in Table 15. We found that even with 32 number of
neighbors, we can already obtain a comparable results with 128 or 256
neighbors.

Table 15: Number of neighbors.

Number of neighbors Perplexity 32 2.38 128 2.37 256 2.37

##### Random seeds

We measured the statistical significant of the results reported. We did 3 runs
with 3 random seeds for Transformer XL of size 512, and also a memorizing
transformer with memory size 8192. We measured the standard deviation of
perplexities after 500K steps of training, shown in Table 16. We saw the
standard deviation between different runs of the same experiment appears to be
much smaller than the gap between different models.

Table 16: Random seeds.

Models Perplexity Transformer XL $2.67\pm 0.01$ Memorizing Transformer
$2.37\pm 0.005$

##  Appendix B What does the model retrieve from memory?

##### Retrieving citation names

On arXiv math, several examples are shown in Table 17, which includes both the
retrieved token and its surrounding context. We observe that many of the gains
in cross-entropy loss took place when trying to predict the name of bibitems,
citations, or references, by looking up the references and citations used
previously in the paper. Such lookups usually span over the entire paper,
which is much longer than 8192 tokens, providing a plausible explanation for
the gain beyond memory size of 8192.

Table 17: The table shows several examples of which tokens were retrieved
during language modelling of arXiv math dataset. The model is retrieving names
of the references from previous passages.

Query index Input Target Surrounding context Retrieved index Retrieved
surrounding context 20389 Mon thus bibitem{ComtetMonthusYor} 2208 Brownian
motion $\backslash$cite{ComtetMonthusYor} 16623 cha kra
$\backslash$cite{chakrabarti}. 4677 $\sim$1.2 of $\backslash$cite{chakrabarti}
14747 as d $\backslash$eqref{asdfg} which 3365 begin{equation} $\backslash$n
$\backslash$label{asdfg.1}

##### Retrieving function names from the codebase

As with the arXiv papers, we also studied which tokens the model retrieved
from memory. As might be expected, the model is often looking up the names of
functions, and variables, as shown in Table 18.

Table 18: Examples of memory retrieval in the Github dataset. The model looks
up how functions are used elsewhere in the repository.

Query index Input Target Surrounding context Retrieved index Retrieved
surrounding context 23837 Fo nte menu_play->setarFonte 14607
menu_load->setarFonte 23825 , 35 hscreen/2-50, 50, 200, 35); 14599 20, y+40,
200, 35) 14546 -> adi panel->adicionaComponente 5205 panel->adicionaComponente

###  B.1 More Retrieving examples in formal theorem proving corpus

##### Example 1

  * •

Input token index: 64604

  * •

Input token: “_”

  * •

Target token: “pair”

  * •

Surrounding context: )) by (simp add: Fourier_sum_limit_pair [OF f, symmetric]
Fourier’

  * •

Name needs to be predicted: Fourier_sum_limit_pair

  * •

Retrieved token: “Four”

  * •

Retrieved token index: 64412

  * •

Retrieved context: 2 * n. Fourier_coefficient f k * trigonometric_set k t)

  * •

Definition of the name:

Figure 19: Definition of Fourier_sum_limit_pair.

##### Example 2

  * •

Input token index: 46175

  * •

Input token: “tri”’

  * •

Target token: “gon”

  * •

Surrounding context: <le>n. a k * trigonometric_set k x)

  * •

Name needs to be predicted: orthonormal_system_trigonometric_set

  * •

Retrieved token: “gon”

  * •

Retrieved token index: 35457

  * •

Retrieved context: lemma orthonormal_system_trigonometric_set:$\backslash$n
"orthonormal_system

  * •

Definition of the name:

Figure 20: Definition of orthonormal_system_trigonometric_set.

##### Example 3

  * •

Input token index: 49760

  * •

Input token: “sum”’

  * •

Target token: “m”

  * •

Surrounding context: nusing Fourier_series_square_summable [OF assms, of’

  * •

Name needs to be predicted: Fourier_series_square_summable

  * •

Retrieved token: “sum”

  * •

Retrieved token index: 35457

  * •

Retrieved context: lemma Fourier_series_square_summable$\backslash$n assumes:

  * •

Definition of the name:

Figure 21: Definition of Fourier_series_square_summable.

##### Example 4

  * •

Input token index: 49697

  * •

Input token: “_”’

  * •

Target token: “system”

  * •

Surrounding context: lemma Riemann_lebesgue_square_integrable:
nassumes "orthonormal_system S w

  * •

Name needs to be predicted: orthonormal_system

  * •

Retrieved token: “system”

  * •

Retrieved token index: 28052

  * •

Retrieved context: definition orthonormal_system ::
"$\backslash$’a::euclidean’

  * •

Definition of the name:

Figure 22: Definition of orthonormal_system.

##### Example 5

  * •

Input token index: 34817

  * •

Input token: “.”’

  * •

Target token: “b”

  * •

Surrounding context: shows "integrable (lebesgue_on {a..b})

  * •

Retrieved token 1: “.”

  * •

Retrieved token index 1: 2416

  * •

Retrieved context 1: lebesgue_on {a..b}) f i

  * •

Retrieved token 2: “-”

  * •

Retrieved token index 2: 2445

  * •

Retrieved context 2: (lebesgue_on {a-c..b-c}) (

  * •

Retrieved token 3: “-”

  * •

Retreived token index 3: 6479

  * •

Retrieved context 3: (lebesgue_on {-pi..pi}) (

##### Example 6

  * •

Input token index: 49759

  * •

Input token: “_”’

  * •

Target token: “sum”

  * •

Surrounding context: 0"$\backslash$n using Fourier_series_square_summable [OF
assms

  * •

Retrieved token 1: “set”

  * •

Retrieved token index 1: 35044

  * •

Retrieved context 1: definition trigonometric_set :: "nat
$\backslash$<Rightarrow>

  * •

Retrieved token 2: “ier”

  * •

Retrieved token index 2: 47272

  * •

Retrieved context 2: definition Fourier_coefficient$\backslash$nwhere

  * •

Retrieved token 3: “ine”

  * •

Retrieved token index 3: 18160

  * •

Retrieved context 3: lemma Schwartz_inequality_strong:$\backslash$nassumes “f’

  * •

Retrieved token 4: “system”

  * •

Retrieved token index 4: 28052

  * •

Retrieved context 4: definition orthonormal_system ::
“$\backslash$’a::euclidean’

  * •

Retrieved token 5: “<”

  * •

Retrieved token index 5: 47241

  * •

Retrieved context 5: subsection$\backslash$<open>Convergence wrt the L’

  * •

Retrieved token 6: “n”

  * •

Retrieved token index 6: 40835

  * •

Retrieved context 6: $\backslash$n subsection$\backslash$<open>A bit of extra’