[a]Leonardo Vernazza

# Factorization and resummation at next-to-leading-power

###### Abstract

We discuss recent progress concerning the resummation of large logarithms at
next-to-leading power (NLP) in scattering processes such as Drell-Yan and deep
inelastic scattering near threshold, and thrust in the two-jet limit. We start
by reviewing the approach based on soft-collinear effective field theory and
show that the standard factorization into short distance coefficients,
collinear and soft functions at NLP leads in general to the appearance of
endpoint divergences, which prevent the naive application of resummation
techniques based on the renormalization group. Taking thrust as a case study,
we then show that these singularities are indeed an artifact of the effective
theory, and discuss how they can be removed to recover a finite factorization
theorem and achieve resummation at NLP, at LL accuracy. Last, we discuss
recent work concerning the calculation of all collinear and soft functions
necessary to reproduce Drell-Yan near threshold up to NNLO in perturbation
theory. This calculation provides useful data to extend resummation at NLP
beyond LL accuracy.

##  1 Soft-collinear radiation at NLP

Figure 1: Kinematic definitions for the processes considered in this talk.

In this talk we discuss Drell-Yan (DY) and deep inelastic scattering (DIS)
near partonic threshold, and thrust in the two-jet limit (fig. 1). Defining
$\hat{s}\equiv(p_{a}+p_{b})^{2}$ the partonic centre of mass energy in DY and
$Q^{2}=q^{2}$ the invariant mass of the final state off-shell photon, the
partonic threshold is defined by the condition $z\equiv Q^{2}/\hat{s}\to 1$;
similarly, assigning momentum $p$ to the incoming parton in DIS, and defining
$Q^{2}=-q^{2}$ the invariant mass of the incoming photon, the threshold limit
is given by the Bjorken variable $x\equiv Q^{2}/(2p\cdot q)\to 1$. Last, the
two-jet limit in thrust is defined by the condition $T={\rm
max}_{\vec{n}}\frac{\sum_{i}|\vec{p}_{i}\cdot\vec{n}|}{\sum_{i}|\vec{p}_{i}|}\to
1$, where $\vec{n}$ is the three-vector defining the thrust axis. Labeling
collectively the variables $z$, $x$, $T$ by $\xi$, in the limits above the
partonic cross section is written as a power expansion in $(1-\xi)\to 0$, with
each term developing towers of large logarithms in perturbation theory:

$\hat{\sigma}(\xi)\sim\sum_{n=0}^{\infty}\bigg{(}\frac{\alpha_{s}}{\pi}\bigg{)}^{n}\Bigg{\\{}c_{n}\delta(1-\xi)+\sum_{m=0}^{2n-1}\Bigg{(}c_{nm}\bigg{[}\frac{\ln^{m}(1-\xi)}{1-\xi}\bigg{]}_{+}+d_{nm}\ln^{m}(1-\xi)\Bigg{)}+{\cal
O}(1-\xi)\Bigg{\\}}.$ (1)

In this equation the terms $c_{n}$ and $c_{nm}$ represent the leading power
(LP) contribution, while the terms $d_{nm}$ give the next-to-leading power
(NLP) correction. The towers of large logarithms spoil the convergence of the
perturbative series, and need to be resummed. For a long time it has been
known how to resum the tower of logarithms in the LP term, see e.g. the
seminal papers [1, 2, 3, 4, 5]. Recently, a lot of effort has been devoted to
the development of resummation for the towers of logarithms appearing at NLP.

Figure 2: Numerical analysis showing the effect of including resummation of
large threshold logarithms at NLP on top of the LP component, in Higgs and di-
Higgs production, from [6].

It has been shown (see e.g. [7, 8] and [9, 6] for more recent analysis) that
the resummation of logarithms at NLP may be important for precision physics.
For instance, in case of processes such as Drell-Yan and Higgs production in
gluon fusion, the resummation of threshold leading logarithms (LL) at NLP
gives a contribution of the same order as the tower of next-to-next-to-leading
logarithms (NNLL) at LP (fig. 2). Therefore, it would be recommendable for any
analysis of particle scattering near threshold with resummation at NN(N)LL
accuracy at LP, to include the summation of large logarithms at NLL accuracy
at NLP as well.

The development of resummation of NLP logarithms has been investigated both
within direct QCD [10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24,
25, 6, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37], and by means of
effective field theory (EFT) methods based on soft-collinear effective field
theory (SCET) [38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 9, 50, 51, 52,
53, 54]. In this talk we will summarize recent developments within the second
approach.

SCET [55, 56, 57] provides tools to describe soft and collinear radiation
systematically, in principle at any subleading power. One introduces soft
$q_{s}$, $A_{s}$ and collinear $\xi_{c}$, $A_{c}$ fields, that represent
respectively soft and collinear modes of the original QCD fields. Hard modes
are integrated out, and appear as short-distance (Wilson) coefficients of
effective operators. Let us start by recalling how factorization and
resummation is achieved at LP. The effective operators describe the hard
scattering kernel of a given process and are written in terms of gauge
invariant fields, $\chi_{c}\equiv W_{c}^{{\dagger}}\xi$ for quarks, ${\cal
A}_{c}\equiv W_{c}^{{\dagger}}[iD_{c\perp}^{\mu}W_{c}^{{\dagger}}]$ for
gluons, where $W_{c}$ is a collinear Wilson line, see e.g. eq. (2.6) of [50]
for a definition. Each field is associated with one of the external particles.
For instance, the processes in fig. 1 are all described at LP by the current

$\big{[}\bar{\psi}\gamma_{\mu}\psi\big{]}(0)=\int
dt\,d\bar{t}\,\widetilde{C}^{A0}(t,\bar{t})\,J^{A0}_{\mu}(t,\bar{t}),\qquad\qquad
J^{A0}_{\mu}(t,\bar{t})=\bar{\chi}_{\bar{c}}(\bar{t}n_{-})\,\gamma_{\perp\mu}\,\chi_{c}(tn_{+}),$
(2)

where $\widetilde{C}^{A0}(t,\bar{t})$ is the Wilson coefficient in position
space. Soft and collinear radiation arises in the effective theory by means of
the SCET Lagrangian

${\cal L}_{\rm SCET}=\sum_{i}{\cal L}_{c_{i}}+{\cal L}_{s}.$ (3)

Collinear interactions occur within a given collinear sector by means of the
collinear Lagrangian ${\cal L}_{c}$; radiation among the different sectors can
only be soft and involves the soft Lagrangian ${\cal L}_{s}$. At LP soft-
collinear interactions arise only due to a single term in ${\cal
L}^{(0)}_{c}$:

${\cal
L}^{(0)}_{c}=\bar{\xi}_{c}\bigg{(}in_{-}D_{c}+g_{s}n_{-}A_{s}(x_{-})+i\not{D}_{\perp
c}\frac{1}{in_{+}D_{c}}i\not{D}_{\perp
c}\bigg{)}\frac{\not{n}_{+}}{2}\xi_{c}+{\cal L}_{c,\rm YM}^{(0)},$ (4)

and occur at position $x_{-}$ due to multipole expansion of the soft field in
collinear interactions. Furthermore, only the component $n_{-}A_{s}$ appears,
which leads to the well-known eikonal Feynman rule $\propto
ig_{s}n_{-}^{\mu}$. As it turns out, this interaction can be removed from the
Lagrangian by means of a soft-decoupling transformation [56]:

$\xi(x)\to Y_{+}(x_{-})\xi^{(0)}(x),\qquad\qquad A_{c}^{\mu}(x)\to
Y_{+}(x_{-})A_{c}^{(0)\mu}(x)Y_{+}^{{\dagger}}(x_{-}),$ (5)

where $Y_{+}$ is a soft Wilson line, see e.g. eq. (2.4) of [50], such that one
has

$\bar{\xi}_{c}\Big{(}in_{-}D_{c}+g_{s}n_{-}A_{s}(x_{-})\Big{)}\frac{\not{n}_{+}}{2}\xi_{c}\to\bar{\xi}^{(0)}_{c}\Big{(}in_{-}D^{(0)}_{c}\Big{)}\frac{\not{n}_{+}}{2}\xi^{(0)}_{c}.$
(6)

This construction guarantees the automatic factorization of a given matrix
element (or cross section) into a product of short distance coefficients times
collinear and soft functions, defined as matrix elements of gauge-invariant
operators made exclusively of collinear and soft fields respectively. In case
of the processes in fig. 1 one obtains the factorized expression

$\frac{d\sigma}{dQ^{2}}=|C^{A0}(Q^{2})|^{2}\times f_{a/A}\otimes
f_{b/B}\otimes S_{\rm DY}\big{[}Q(1-z)\big{]},$ (7)

for the Drell-Yan invariant mass distribution [58],

$F_{2}=|C^{A0}(Q^{2})|^{2}\times Q^{2}\times f_{a/A}\otimes
J_{\overline{hc}}^{(q)}\big{[}Q(1-z)\big{]},$ (8)

for Deep inelastic scattering [59], and

$\frac{d\sigma}{d\tau}=|C^{A0}(Q^{2})|^{2}\times J_{c}^{(q)}(p_{L}^{2})\otimes
J_{\bar{c}}^{(q)}(p_{R}^{2})\otimes S_{\rm LP}(k),$ (9)

for thrust [60], with $\tau=1-T$ and $k$ a soft momentum, reproducing older
results in QCD [3, 4, 61]. In these equations the matrix elements of
(anti-)collinear fields are interpreted either in terms of the parton
distribution functions (DY and DIS) or jet functions, (DIS and Thrust), while
the soft functions are given as vacuum expectation values of the soft Wilson
lines introduced in eq. (5). One of the most important features of the EFT
approach is that the original infrared singularities of the scattering
amplitude are turned into ultraviolet divergences of the EFT’s operators [62].
The renormalization of such operators provides renormalization group equations
(RGEs), whose solution allows one to sum large logarithms associated to the
hard, soft and collinear functions, thanks to the fact that within the EFT
factorization, these functions are single scale objects.

Let’s now consider the extension of this framework beyond LP. In this respect,
one of the advantages of the EFT approach is that every object (fields,
derivatives, momenta) has a unique scaling with the small parameter in the
problem, conventionally indicated by $\lambda\ll 1$. For instance, in case of
Drell-Yan near threshold $\lambda\sim\sqrt{1-z}$. Decomposing momenta along
the light-like directions $n_{i\pm}$, such that
$p^{\mu}=(n_{+}p)n_{-}^{\mu}/2+p_{\perp}^{\mu}+(n_{-}p)n_{+}^{\mu}/2=(n_{+}p,p_{\perp},n_{-}p)$,
collinear and soft momenta have respectively scaling $p_{c}\sim
Q(1,\lambda,\lambda^{2})$ and $p_{s}\sim
Q(\lambda^{2},\lambda^{2},\lambda^{2})$. In this context, one needs to take
into account two sources of power suppression [38, 46, 47]. On the one hand,
one has operators that are power suppressed compared to the LP ones in eq.
(2). In general, power suppression is achieved either by inserting transverse
derivatives, $\partial_{\perp}\sim\lambda$, or by adding more collinear fields
along the same collinear direction. It is also possible to insert gauge-
invariant combinations of soft fields, but these contribute in general beyond
NLP. Starting from the LP current in eq. (2) we have e.g.

$\displaystyle\bar{\chi}_{\bar{c}}(\bar{t}n_{-})\,\gamma_{\perp}^{\mu}\,\chi_{c}(tn_{+})\,\,$
$\displaystyle\to$
$\displaystyle\,\,\bar{\chi}_{\bar{c}}(\bar{t}n_{-})\big{[}n_{\pm}^{\mu}\,i\not{\partial}_{\perp}\big{]}\chi_{c}(tn_{+}),\hskip
49.79231pt\mbox{(A1-type),}$ $\displaystyle\searrow$
$\displaystyle\,\,\bar{\chi}_{\bar{c}}(\bar{t}n_{-})\big{[}n_{\pm}^{\mu}\,\not{\cal
A}_{c\perp}(t_{2}n_{+})\big{]}\chi_{c}(t_{1}n_{+}),\hskip
14.22636pt\mbox{(B1-type),}$ (10)

where for the labeling of power suppressed operators we refer to [46, 47]. The
operators on the r.h.s of eq. (1) are suppressed by one power of $\lambda$
compared to the operator on the l.h.s. Given that $\lambda\sim\sqrt{1-z}$, in
general one needs to take into account operators suppressed up to two powers
of $\lambda$, in order to reproduce a given cross section up to NLP. The
second source of power suppression originates by considering time-ordered
products of LP operators with power-suppressed insertions of terms from the
SCET Lagrangian. Given the collinear SCET Lagrangian ${\cal L}_{c}={\cal
L}^{(0)}_{c}+{\cal L}^{(1)}_{c}+{\cal L}^{(2)}_{c}+{\cal O}(\lambda^{3})$, one
has e.g. two terms contributing to ${\cal L}^{(1)}_{c}$ [63], given by

${\cal L}^{(1)\rm
gluon}_{c}=\bar{\xi}\Big{[}x_{\perp}^{\mu}n_{-}^{\nu}W_{c}g_{s}F_{\mu\nu}^{s}(x_{-})W_{c}^{{\dagger}}\Big{]}\frac{\not{n_{+}}}{2}\xi,\qquad{\cal
L}^{(1)\rm quark}_{c}=\bar{q}(x_{-})W_{c}^{{\dagger}}i\not{D}_{\perp c}\xi,$
(11)

where the first term involves the emission of a soft gluon and in the second a
collinear quark is converted into a soft quark, upon emission of a collinear
gluon. Now, the difference compared to our previous discussion of
factorization at LP is that soft-collinear interactions at subleading power,
such as the one in eq. (11), are not removed by the decoupling transformation
eq. (5). Formally it is still possible to proceed with the factorization of a
given matrix element into its soft and collinear components. However, a few
differences arise compared to the factorization theorems at LP.

Figure 3: Convolutions appearing at NLP between jet and soft functions (left,
see e.g. [48, 50]) or between short-distance coefficients and jet functions
(right, see e.g. [51]).

When the power suppression is given by a soft-collinear Lagrangian insertion
such as in eq. (11), we obtain a convolution between a collinear and a soft
function, where the convolution variable is related to the small component of
the collinear momentum, which is of the same order of the corresponding
component of the soft momentum, (fig. 3, left diagram), and therefore cannot
be integrated out. This factorization structure appear for instance in Drell-
Yan [48, 50]. When the power suppression is given by the insertion of an
operator involving two or more particle in the same collinear sector, such as
the B1 type operator in eq. (1), convolution between the corresponding short-
distance coefficient and jet function arises, where the convolution variable
is related to the fraction of collinear momentum shared between the two
particle in the same collinear sector (fig. 3, right diagram). This
factorization structure arise for instance in off-diagonal DIS [51].

##  2 Endpoint divergences at NLP

The presence of convolutions does not constitute a problem per se. However, as
it turns out, these integrations are often divergent in $d=4$. For instance,
in case of DY one finds [50]

$\int
d\omega\,J(\omega)\,S(\omega)\sim\int_{0}^{\Omega}d\omega\,\underbrace{(n_{+}p\omega)^{-\epsilon}}_{\rm
collinear\,\,piece}\underbrace{\frac{1}{\omega^{1+\epsilon}}\frac{1}{(\Omega-\omega)^{\epsilon}}}_{\rm
soft\,\,piece}\bigg{|}_{\Omega=Q(1-z)},$ (12)

which for $\epsilon\to 0$ is divergent for $\omega\to 0$. In case of off-
diagonal scalar DIS [51] one finds

$\int
dz\,C^{B1}(z)\,J_{qg}(z)\sim\int_{0}^{1}dz\,\underbrace{\bigg{(}\frac{\mu^{2}}{s_{qg}z\bar{z}}\bigg{)}^{\epsilon}}_{\rm
collinear\,\,piece}\underbrace{\frac{\alpha_{s}C_{F}}{2\pi}\frac{(1-z)^{2}}{z}}_{\rm
hard\,\,piece}\bigg{|}_{s_{qg}=Q^{2}\frac{1-x}{x}},$ (13)

which for $\epsilon\to 0$ diverges for $z\to 0$. In order to investigate the
structure of these endpoint divergences, let’s consider the case of DIS more
in detail: In eq. (13) we have inserted the Wilson coefficient at tree level.
At one loop one has [51]

$\displaystyle C^{B1}(z)\Big{|}_{\rm 1\,\,loop}$ $\displaystyle\sim$
$\displaystyle C^{B1}(z)\Big{|}_{\rm
tree}\frac{\alpha_{s}}{\pi}\frac{1}{\epsilon^{2}}\bigg{\\{}{\bf
T}_{1}\cdot{\bf T}_{0}\bigg{(}\frac{\mu^{2}}{zQ^{2}}\bigg{)}^{\epsilon}+{\bf
T}_{2}\cdot{\bf T}_{0}\bigg{(}\frac{\mu^{2}}{\bar{z}Q^{2}}\bigg{)}^{\epsilon}$
(14) $\displaystyle\hskip 71.13188pt+\,{\bf T}_{1}\cdot{\bf
T}_{2}\bigg{[}\bigg{(}\frac{\mu^{2}}{Q^{2}}\bigg{)}^{\epsilon}-\bigg{(}\frac{\mu^{2}}{zQ^{2}}\bigg{)}^{\epsilon}\bigg{]}\bigg{\\}}+{\cal
O}(\epsilon^{-1})$

The term $\propto{\bf T}_{1}\cdot{\bf T}_{2}$ contains a single pole, which
however gives rise to a leading pole after integration. The correct result is
obtained only within dimensional regularization:

$\frac{1}{\epsilon^{2}}\int_{0}^{1}dz\,\frac{1}{z^{1+\epsilon}}(1-z^{-\epsilon})=-\frac{1}{2\epsilon^{3}},$
(15)

while expanding for $\epsilon\to 0$ leads to nonsense results:

$\frac{1}{\epsilon^{2}}\int_{0}^{1}dz\,\frac{1}{z^{1+\epsilon}}\bigg{(}\epsilon\ln
z-\frac{\epsilon^{2}}{2!}\ln^{2}z+\frac{\epsilon^{3}}{3!}\ln^{3}z+\ldots\bigg{)}=-\frac{1}{\epsilon^{3}}+\frac{1}{\epsilon^{3}}-\frac{1}{\epsilon^{3}}+\ldots.$
(16)

This poses a problem for standard resummation techniques. As discussed above,
within an EFT approach one renormalizes the collinear and soft matrix
elements, obtaining a set of RGEs whose solution resums the large logarithms.
From eqs. (12), (13) and (14), however, it is clear that not all logarithms
are generated within the hard, soft and jet functions themselves: an
additional pole (and thus a corresponding logarithm) is generated through the
endpoint divergent convolution. Closer inspection of eq. (14) reveals that the
endpoint divergence actually points a break of the EFT itself (fig. 4):

Figure 4: Structure of endpoint divergence in off-diagonal scalar DIS, from
[51].

The factor $1/z$ in the Wilson coefficient $C^{B1}$ is due to the intermediate
gluon propagator (l.h.s. of fig. 4). For generic $z\sim 1$, this propagator is
hard, thus an EFT description in terms of a short-distance Wilson coefficient
is appropriate (upper r.h.s. of fig. 4). However, $z$ is integrated in the
range (0,1), and when $z\ll 1$, the intermediate propagator is not hard, and
cannot be integrated out: the correct EFT description is now given by the
lower diagram on the r.h.s. of fig. 4. For $z\ll 1$ the short-distance
coefficient $C^{B1}$ becomes a two-scale object, and together with the
corresponding power-suppressed operator it _refactorizes_ into a jet function
times the LP Wilson coefficient and operator $C^{A0}$ [51]:

$C^{B1}(Q,z)J^{B1}\stackrel{{\scriptstyle z\to
0}}{{\longrightarrow}}C^{A0}(Q^{2})\int d^{4}x\,T\Big{[}J^{A0},{\cal
L}^{(1)\rm
quark}_{z-\overline{sc}}\Big{]}=C^{A0}(Q^{2})D^{B1}(zQ^{2})J^{B1}_{z-\overline{sc}},$
(17)

where on the r.h.s. $C^{A0}(Q^{2})$ and $D^{B1}(zQ^{2})$ can now be
interpreted correctly as single-scale function. Such refactorization has been
observed also in other applications of SCET to the analysis of processes at
NLP, such as $H\to\gamma\gamma$, $H\to gg$ [64, 65, 66, 67], or in $B$-physics
[68, 69, 70, 71] and muon-electron backward scattering [72].

The analysis of DIS shows that a correct EFT treatment needs to take into
account both the configurations appearing on the r.h.s. of fig. 4. Once both
contributions are taken into account, one expects endpoint divergences to
cancel between the two terms. In case of DIS, such construction involves the
factorization of the perturbative part of the initial state PDF, which makes
the construction more involved. In the next section we will focus instead on
off-diagonal thrust, where the cancellation of endpoint divergences can be
shown explicitly, without involving initial-state singularities.

##  3 NLP LLs in Thrust in the two-jet limit

Let us consider thrust in the two-jet limit: following [53], we consider the
power-suppressed contribution given by the process (fig. 5)

$e^{+}e^{-}\to\gamma^{*}\to[g]_{c}+[q\bar{q}]_{\bar{c}}.$ (18)

Within SCET this process is given by two contributions [53]: a “direct” term
(B-type) (first diagram on the left in fig. 6) and a time ordered product
involving a soft quark emission (A-type) (two diagrams on the right in fig.
6): the former involves the matrix element

$\langle X_{c}|\langle X_{\bar{c}}|\langle
X_{s}|\,\bar{\chi}_{\bar{c}}(\bar{t}_{1}n_{-})\Gamma_{i}^{\mu\nu}\mathcal{A}_{c\perp\nu}(tn_{+})\chi_{\bar{c}}(\bar{t}_{2}n_{-})\,|0\rangle\,,$
(19)

with $i=1,2$ representing two different strings of Dirac matrices, while the
latter stems from the matrix element

$\langle X_{c}|\langle X_{\bar{c}}|\langle X_{s}|\int
d^{d}x\,T\,[\bar{\chi}_{c}(tn_{+})\gamma_{\perp}^{\mu}\chi_{\bar{c}}(\bar{t}n_{-}),i{\cal
L}^{(1)\rm quark}_{c}(x)]\,|0\rangle\,.$ (20)

Figure 5: Off-diagonal “gluon” thrust in the two-jet limit. Figure from [53].

Inserting these matrix elements in the corresponding cross section, one finds
that the “direct” B-type term factorizes into a hard, (anti-)collinear and
soft functions, according to

$\displaystyle\frac{1}{\sigma_{0}}\frac{d\sigma}{dM_{R}^{2}dM_{L}^{2}}\bigg{|}_{\rm
B-type}$ $\displaystyle\sim$
$\displaystyle\frac{2C_{F}}{Q^{2}}f(\epsilon)\int_{0}^{\infty}dl_{+}dl_{-}\,\sum_{i,i^{\prime}=1,2}\int
drdr^{\prime}\,C^{\rm B1*}_{i^{\prime}}(Q^{2},r^{\prime})C^{\rm
B1}_{i}(Q^{2},r)$ (21)
$\displaystyle\times\,\bigg{\\{}\,\delta_{ii^{\prime}}\,\mathcal{J}_{\bar{c}}^{q\bar{q}(8)}(M_{R}^{2}-Ql_{+},r,r^{\prime})+\,\ldots\bigg{\\}}\,\mathcal{J}_{c}^{(g)}(M_{L}^{2}-Ql_{-})\,S^{(g)}(l_{+},l_{-})\,,$

where

$\displaystyle
f(\epsilon)=\bigg{(}\frac{Q^{2}}{4\pi}\bigg{)}^{-\epsilon}\frac{(1-\epsilon)^{2}\Gamma(1-\epsilon)}{\Gamma(2-2\epsilon)},$
(22)

and the ellipses represent regular terms not important for our analysis below.
Due to the fact that $C^{\rm B1}_{i}(r)\sim 1/r$, eq. (21) develops endpoint
divergences when the quark ($r\to 0$) or the anti-quark ($r\to 1$) become
soft:

$\frac{1}{\sigma_{0}}\frac{d\sigma}{dM_{R}^{2}dM_{L}^{2}}\bigg{|}_{\rm
B-type}\sim\int_{0}^{1}dr\,\bigg{[}\frac{1}{r^{1+\epsilon}}+\frac{1}{(1-r)^{1+\epsilon}}\bigg{]}\,.$
(23)

On the other hand, the A-type matrix element of eq. (20) gives rise to the
factorized cross section

$\displaystyle\frac{1}{\sigma_{0}}\frac{d\sigma}{dM_{R}^{2}dM_{L}^{2}}|_{\rm
A-type}$ $\displaystyle\sim$ $\displaystyle\frac{2C_{F}}{Q}\,|C^{\rm
A0}(Q^{2})|^{2}\int_{0}^{\infty}dl_{+}dl_{-}\,\int d\omega
d\omega^{\prime}\,\mathcal{J}_{\bar{c}}^{(\bar{q})}(M_{R}^{2}-Ql_{+})$ (24)
$\displaystyle\times\,\bigg{\\{}\,\mathcal{J}_{c}(M_{L}^{2}-Ql_{-},\omega,\omega^{\prime})\,S_{\rm
NLP}(l_{+},l_{-},\omega,\omega^{\prime})+\ldots\bigg{\\}}\,,$

which develops endpoint divergences when the soft quark or anti-quark become
energetic ($\omega\to\infty$):

$\frac{1}{\sigma_{0}}\frac{d\sigma}{dM_{R}^{2}dM_{L}^{2}}|_{\rm A-type}\sim
2\int_{M_{R}^{2}/Q}^{\infty}d\omega\,\frac{1}{\omega^{1+\epsilon}}\,.$ (25)

As for DIS, in the $r\to 0$ (or $r\to 1$) limit the coefficients $C_{i}^{B1}$
become a two-scale object, and refactorize according to

$\displaystyle C_{1}^{\rm B1}(Q^{2},r)$ $\displaystyle\stackrel{{\scriptstyle
r\to 0}}{{=}}$ $\displaystyle C^{\rm A0}(Q^{2})\times\frac{D^{\rm
B1}(rQ^{2})}{r}+\mathcal{O}(r^{0})\,,$ $\displaystyle C_{2}^{\rm B1}(Q^{2},r)$
$\displaystyle\stackrel{{\scriptstyle r\to 1}}{{=}}$ $\displaystyle-C^{\rm
A0}(Q^{2})\times\frac{D^{\rm
B1}(\bar{r}Q^{2})}{\bar{r}}+\mathcal{O}(\bar{r}^{0})\,.$ (26)

Figure 6: Matrix elements in SCET corresponding respectively to eq. (19)
(first diagram on the left) and eq. (20) (second and third diagrams). Diagrams
from [53].

Furthermore, we expect that endpoint divergences should cancel when summing
the A and B-type contributions, which in turn implies that in the asymptotic
limits $r,r^{\prime}\to 0(1)$, $\omega,\omega^{\prime}\to\infty$, the
_integrands_ of the A- and B-type terms should become identical. While it is
easy to check that this is indeed what happens at lowest order in perturbation
theory, in general this gives a series of refactorization conditions [53],
which are summarized graphically in fig. 7: starting from the A-type term
(upper left diagram in fig. 7), in the limit $\omega,\omega^{\prime}\to\infty$
the soft (anti)-quark becomes energetic, thus we get to the lower left diagram
in fig. 7, provided that [53]

(I) $\displaystyle\hskip
14.22636pt\mathcal{J}_{c}\left(p^{2},\omega,\omega^{\prime}\right)=\mathcal{J}_{c}^{(g)}(p^{2})\,\displaystyle\frac{D^{\rm
B1}(\omega Q)}{\omega}\frac{{D^{\rm
B1}}^{*}(\omega^{\prime}Q)}{\omega^{\prime}}+\mathcal{O}\\!\left(\frac{1}{\omega^{(\prime)}}\right)\,,$
(27)

where the function $D^{\rm B1}(p^{2})$ is the same as the one appearing in the
factorization of the hard B1 operator coefficient (3). On the other hand,
starting from the B-type term (lower right diagram in fig. 7), in the limit
$r,r^{\prime}\to 0(1)$ the anticollinear (anti-)quark becomes soft, thus we
get to the upper right diagram in fig. 7, provided that eq. (3) holds. At this
point, the requirement that the integrands of the A- and B-type terms should
become identical in the asymptotic limits $r,r^{\prime}\to 0(1)$,
$\omega,\omega^{\prime}\to\infty$, (i.e., that the lower left- and the upper
right-diagram in fig. 7 should coincide), gives the last refactorization
condition: in Laplace space one has [53]

$\mbox{(II)}\quad
Q\,\widetilde{\mathcal{J}}^{(\bar{q})}_{\bar{c}}(s_{R})\,\widetilde{S}_{\rm
NLP}\left(s_{R},s_{L},\omega,\omega^{\prime}\right)\Big{|}_{\omega^{(\prime)}\to\infty}=\widetilde{\mathcal{J}}^{q\bar{q}(8)}_{\bar{c}}\left(s_{R},r,r^{\prime}\right)\widetilde{S}^{(g)}(s_{R},s_{L})\Big{|}_{r^{(\prime)}=\omega^{(\prime)}/Q\to
0}\,,$ (28)

and the same identity holds with $r,r^{\prime}\to\bar{r},\bar{r}^{\prime}$.

Figure 7: Consistency conditions necessary to achieve refactorization and the
cancellation of endpoint divergences, as obtained in [53].

The constraint that in the asymptotic limits the A- and B-type terms must
coincide provides also a method to deal with endpoint divergences. Let us
define the asymptotic limits of the various function by using a double-bracket
notation: for instance, in functions of $\omega,\omega^{\prime}$, we rescale
$\omega\to\kappa\omega$, $\omega^{\prime}\to\kappa\omega^{\prime}$ and take
$\kappa\to\infty$. Then

$\displaystyle\llbracket S_{\rm
NLP}\left(l_{+},l_{-},\omega,\omega^{\prime}\right)\rrbracket\equiv S_{\rm
NLP}\left(l_{+},l_{-},\omega,\omega^{\prime}\right)|_{\mathcal{O}(\kappa^{0})},$
(29)
$\displaystyle\llbracket\mathcal{J}_{c}(p^{2},\omega,\omega^{\prime})\rrbracket\equiv\mathcal{J}_{c}(p^{2},\omega,\omega^{\prime})|_{\mathcal{O}(\kappa^{-2})}\,,$
(30)

where $\delta(\omega-\omega^{\prime})$ counts as $\kappa^{-1}$. A similar
notation is used in functions of $r,r^{\prime}$ and we refer to [53] for a
precise definition. As discussed above, endpoint divergences arise in the
asymptotic limits, where the A- and B-type terms take the factorized form
given respectively in the left lower graph of fig. 7, and right upper graph of
fig. 7: the cancellation of endpoint divergences requires these two limits to
be identical, therefore we can introduce the integral

$\displaystyle\frac{2C_{F}}{Q}\,f(\epsilon)\,|C^{\rm
A0}(Q^{2})|^{2}\widetilde{\mathcal{J}}^{(\bar{q})}_{\bar{c}}(s_{R})\widetilde{\mathcal{J}}_{c}^{(g)}(s_{L})$
$\displaystyle\times\int_{0}^{\infty}d\omega d\omega^{\prime}\,\frac{D^{\rm
B1}(\omega Q)}{\omega}\frac{D^{\rm
B1*}(\omega^{\prime}Q)}{\omega^{\prime}}\left\llbracket\widetilde{S}_{\rm
NLP}(s_{R},s_{L},\omega,\omega^{\prime})\right\rrbracket,\qquad$ (31)

which is scaleless over the whole domain and thus vanishes in $d$ dimensions,
but it can be shown to reproduce respectively the endpoint divergences of the
A- and B-type contribution, when splitting the integration into two domains
$I_{2}$ and $I_{1}$, as represented in fig. 8111Let us notice that the
splitting in fig. 8 is not unique. It is possible to split the integration
domain differently, given that the endpoint divergences occur when both
$\omega$ and $\omega^{\prime}\to\infty$ or $r$ and $r^{\prime}\to 0$ ($r$ and
$r^{\prime}\to 1$), see [53]. [53]. Thus it is possible to remove the endpoint
divergences from both the A- and B-type contributions by subtracting the
integrand in eq. (31) integrated over region $I_{2}$ from the A-type term, and
by subtracting eq. (31) integrated over region $I_{1}$ from the B-type term.
The subtracted expressions are now separately endpoint-finite, but depend on
$\Lambda$. However, as long as no approximations are made, the $\Lambda$
dependence cancels exactly between the two terms. After some elaboration [53]
the endpoint finite A-term can be written as

$\displaystyle\frac{1}{\sigma_{0}}\frac{\widetilde{d\sigma}}{ds_{R}ds_{L}}|_{\rm
A-type}$ $\displaystyle=$ $\displaystyle\frac{2C_{F}}{Q}\,f(\epsilon)\,|C^{\rm
A0}(Q^{2})|^{2}\,\widetilde{\mathcal{J}}_{\bar{c}}^{(\bar{q})}(s_{R})\,\int
d\omega
d\omega^{\prime}\,\Big{[}1-\theta(\omega-\Lambda)\theta(\omega^{\prime}-\Lambda)\Big{]}$
(32)
$\displaystyle\times\,\bigg{\\{}\,\widetilde{\mathcal{J}}_{c}(s_{L},\omega,\omega^{\prime})\,\widetilde{S}_{\rm
NLP}(s_{R},s_{L},\omega,\omega^{\prime})+\,\widetilde{\\!\\!\widehat{\mathcal{J}}}_{\\!c}(s_{L},\omega,\omega^{\prime})\,\,\widetilde{\\!\widehat{S}}_{\rm
NLP}(s_{R},s_{L},\omega,\omega^{\prime})\bigg{\\}}\,,\qquad$

where the equality signs hold up to corrections of
$\mathcal{O}(1/(s_{L}\Lambda),1/(s_{R}\Lambda))$, and the and B-type term
takes the form

$\displaystyle\frac{1}{\sigma_{0}}\frac{\widetilde{d\sigma}}{ds_{R}ds_{L}}|_{\scriptsize\begin{array}[]{l}$\rm
B--type$\\\\[-2.84544pt] $i=i'=1$\end{array}}$ $\displaystyle=$
$\displaystyle\frac{2C_{F}}{Q^{2}}\,f(\epsilon)\,\widetilde{\mathcal{J}}_{c}^{(g)}(s_{L})\,\widetilde{S}^{(g)}(s_{R},s_{L})$
(36)
$\displaystyle\times\,\bigg{\\{}\int_{0}^{1}drdr^{\prime}\,\big{[}1-\theta(\Lambda/Q-r)\theta(\Lambda/Q-r^{\prime})\big{]}\,C^{\rm
B1*}_{1}(Q^{2},r^{\prime})C^{\rm
B1}_{1}(Q^{2},r)\,\widetilde{\mathcal{J}}_{\bar{c}}^{q\bar{q}(8)}(s_{R},r,r^{\prime})\quad$
$\displaystyle-\int_{0}^{\infty}drdr^{\prime}\,\big{[}\theta(r-\Lambda/Q)\theta(\Lambda/Q-r^{\prime})+\theta(\Lambda/Q-r)\theta(r^{\prime}-\Lambda/Q)\big{]}\,$
$\displaystyle\times\,\llbracket C^{\rm
B1*}_{1}(Q^{2},r^{\prime})\rrbracket_{0}\,\llbracket C^{\rm
B1}_{1}(Q^{2},r)\rrbracket_{0}\,\llbracket\widetilde{\mathcal{J}}_{\bar{c}}^{q\bar{q}(8)}(s_{R},r,r^{\prime})\rrbracket_{0}\,\bigg{\\}}\,,$

up to corrections of $\mathcal{O}(\Lambda/Q)$. It is now possible to use these
expressions to develop the resummation of large logarithms with standard
methods, and we refer to [53] for a detailed discussion of such derivation.

Figure 8:  Visual representation of the regions $I_{1}$ and $I_{2}$, used to
construct subtraction terms as described in the text. In the overlap region in
green the asymptotic behaviour of the A- and B-type term must agree. Figure
from [53].

##  4 NLP NNLO in Drell-Yan near threshold

The analysis of off-diagonal thrust has allowed us to fully appreciate the
nature of endpoint divergences: these are indeed an artifact of the effective
field theory, which arise due to how the original (phase space or loop)
integrations in QCD are split among the different regions of the effective
theory. When all momentum regions are correctly taken into account, (A- and
B-type terms in case of off-diagonal thrust), endpoint divergences cancel.
This allows one to devise the construction of subtraction terms such as to
make the individual contributions finite. Formally, the finite factorization
formulas such as eqs. (32) and (36) are expected to be valid at all orders in
perturbation theory and at any logarithmic accuracy. However, for off-diagonal
thrust the explicit construction has been obtained at LL accuracy. In order to
go beyond this logarithmic order, in general one needs to calculate the
factorized matrix elements beyond leading order in perturbation theory. In
turn, this would allow one to check explicitly that the refactorization
conditions such as those in eqs. (27) and (28) hold to higher order in
perturbation theory.

With this goal in mind, a set of papers [50, 52, 54] have been dedicated to
calculate the full set of collinear and soft functions necessary to reproduce
the Drell-Yan invariant mass distribution at NLP near threshold, up to NNLO in
perturbation theory. In particular, this requires the calculation of the
collinear functions up to one loop and the soft function up to two loops. Let
us focus for simplicity on the off diagonal $\bar{q}g$ channel. In [54] the
factorization theorem has been obtained formally at all subleading power.
Writing the invariant mass distribution as

$\frac{d\sigma_{\rm
DY}}{dQ^{2}}=\sigma_{0}\sum_{a,b}\int_{\tau}^{1}\,\frac{dz}{z}\,{\cal
L}_{ab}\bigg{(}\frac{\tau}{z}\bigg{)}\,\Delta_{ab}(z)+\mathcal{O}\left(\frac{\Lambda}{Q}\right),\qquad\sigma_{0}=\frac{4\pi\alpha_{\rm
em}^{2}}{3N_{c}Q^{2}s},$ (37)

where the parton luminosity function ${\cal L}_{ab}(y)$ is defined as

${\cal
L}_{ab}(y)=\int_{y}^{1}\frac{dx}{x}\,f_{a/A}(x)\,f_{b/B}\bigg{(}\frac{y}{x}\bigg{)},$
(38)

up to NLP the partonic cross section factorizes as follows:

$\Delta_{g\bar{q}}|_{\rm
NLP}(z)=8H(Q^{2})\,\int{d\omega}\,d\omega^{\prime}\,G^{*}_{\xi
q}(x_{a}n_{+}p_{A};\omega^{\prime})\,G_{\xi
q}(x_{a}n_{+}p_{A};\omega)\,S(\Omega,\omega,\omega^{\prime})\,,$ (39)

where $G_{\xi q}$, $G^{*}_{\xi q}$ are collinear function appearing
respectively in the amplitude and complex conjugate amplitude,
$H(Q^{2})=|C^{A0}(Q^{2})|^{2}$ is the hard function, and
$S(\Omega,\omega,\omega^{\prime})$ the soft function. The collinear function
coincides with the function $D^{B1}$ in eqs. (3) and (27). Indeed, this
collinear matrix element appears to be a “universal” function appearing in the
context of several factorization theorems at NLP; for instance, it appears
also in the context of $H\to\gamma\gamma$, [64], and it has been calculated up
to two loops in [73]. At one loop it reads

$\displaystyle{G}^{(1)}_{\xi q}\left(n_{+}p;\omega\right)$ $\displaystyle=$
$\displaystyle-\,\frac{\alpha_{s}}{4\pi}\big{(}C_{F}-C_{A}\big{)}\left(\frac{n_{+}p\,\omega}{\mu^{2}}\right)^{-\epsilon}\frac{2-4\epsilon-\epsilon^{2}}{2\epsilon^{2}}\frac{e^{\epsilon\gamma_{E}}\Gamma[1+\epsilon]\Gamma[{1-\epsilon}]^{2}}{\Gamma[{2-2\epsilon}]}$
$\displaystyle=$
$\displaystyle-\,\frac{\alpha_{s}}{4\pi}\big{(}C_{F}-C_{A}\big{)}\bigg{[}\frac{1}{\epsilon^{2}}-\frac{1}{\epsilon}\ln\left(\frac{n_{+}p\,\omega}{\mu^{2}}\right)-\,\frac{1}{2}-\frac{\pi^{2}}{12}+\frac{1}{2}\ln^{2}\left(\frac{n_{+}p\,\omega_{1}}{\mu^{2}}\right)+\mathcal{O}(\epsilon)\bigg{]}\,.$

The soft function at one loop reads

$S_{g\bar{q}}^{(1)}(\Omega,\omega,\omega^{\prime})=\frac{\alpha_{s}\,T_{F}}{4\pi}\frac{e^{\epsilon\gamma_{E}}}{\Gamma[1-\epsilon]}\frac{1}{\omega}\bigg{(}\frac{\mu^{2}}{\omega\,(\Omega-\omega)}\bigg{)}^{\epsilon}\delta(\omega-\omega^{\prime})\,\theta(\Omega-\omega)\theta(\omega),$
(41)

where the color factor $T_{F}$ is given in terms of the relation ${\rm
Tr}[{\bf T}^{A}{\bf T}^{B}]=T_{F}\,\delta^{AB}=\delta^{AB}/2$. The two loop
contribution is quite involved. It includes a virtual-real and a real-real
contributions

$S^{(2)}_{g\bar{q}}(\Omega,\omega,\omega^{\prime})=S^{(2)1r1v}_{g\bar{q}}(\Omega,\omega,\omega^{\prime})+S^{(2)2r0v}_{g\bar{q}}(\Omega,\omega,\omega^{\prime}),$
(42)

which individually read

$\displaystyle S^{(2)1r1v}_{g\bar{q}}(\Omega,\omega,\omega^{\prime})$
$\displaystyle=$
$\displaystyle\frac{\alpha_{s}^{2}\,T_{F}}{(4\pi)^{2}}(2C_{F}-C_{A})\frac{e^{2\epsilon\gamma_{E}}\,\Gamma[1+\epsilon]}{\epsilon\,\Gamma[1-\epsilon]}$
(43) $\displaystyle\times\,{\rm
Re}\bigg{\\{}\frac{1}{(-\omega)\omega^{\prime}}\bigg{[}\frac{\omega+\omega^{\prime}}{\omega^{\prime}}\,_{2}F_{1}\bigg{(}1,1+\epsilon,1-\epsilon,\frac{\omega}{\omega^{\prime}}\bigg{)}-1\bigg{]}\bigg{(}\frac{\mu^{4}}{(-\omega)\omega^{\prime}(\Omega-\omega^{\prime})^{2}}\bigg{)}^{\epsilon}\theta(-\omega)$
$\displaystyle+\,\frac{2(\omega+\omega^{\prime})}{\omega\omega^{\prime}(\omega^{\prime}-\omega)}\bigg{(}\frac{\mu^{4}}{(\omega^{\prime}-\omega)^{2}(\Omega-\omega^{\prime})^{2}}\bigg{)}^{\epsilon}\frac{\Gamma[1-\epsilon]^{2}}{\Gamma[1-2\epsilon]}\theta(\omega^{\prime}-\omega)\bigg{\\}}\theta(\omega^{\prime})\theta(\Omega-\omega^{\prime}),$

and

$\displaystyle S^{(2)2r0v}_{g\bar{q}}(\Omega,\omega,\omega^{\prime})$
$\displaystyle=$
$\displaystyle\frac{\alpha_{s}^{2}T_{F}}{(4\pi)^{2}}\bigg{\\{}C_{F}\frac{e^{2\epsilon\gamma_{E}}\Gamma[1-\epsilon]}{\epsilon^{2}}\frac{1}{\omega}\bigg{[}\frac{4}{\Gamma[1-3\epsilon]}\bigg{(}\frac{\mu^{4}}{\omega(\Omega-\omega)^{3}}\bigg{)}^{\epsilon}$
$\displaystyle+\,\frac{(4-\epsilon)\Gamma[2-\epsilon]}{(1-2\epsilon)\Gamma[1-2\epsilon]^{2}}\bigg{(}\frac{\mu^{4}}{\omega^{2}(\Omega-\omega)^{2}}\bigg{)}^{\epsilon}\bigg{]}\delta(\omega-\omega^{\prime})\theta(\Omega-\omega)\theta(\omega)$
$\displaystyle+\,\left(C_{A}-2C_{F}\right)\frac{2e^{2\epsilon\gamma_{E}}}{\epsilon\Gamma[1-2\epsilon]}\frac{\omega+\omega^{\prime}}{\omega\omega^{\prime}(\omega^{\prime}-\omega)}\bigg{(}\frac{\mu^{4}}{\omega(\omega^{\prime}-\omega)(\Omega-\omega^{\prime})^{2}}\bigg{)}^{\epsilon}$
$\displaystyle\times\,\bigg{[}\,_{2}F_{1}\Big{(}1,-\epsilon,1-\epsilon,\frac{\omega}{\omega-\omega^{\prime}}\Big{)}-1\bigg{]}\theta(\omega)\theta(\omega^{\prime})\theta(\omega^{\prime}-\omega)\theta(\Omega-\omega^{\prime})\bigg{\\}}.$

With these results at hand one can now study the structure of endpoint
divergences in the asymptotic limits $\omega,\omega^{\prime}\to 0$, and we
refer to [54] for further details.

##  5 Outlook

The resummation of large logarithms at NLP poses interesting theoretical
challenges. Within an effective field theory approach based on SCET it is
possible to systematically factorize the effect of soft and collinear
radiation in physical observables. In this talk we have discussed the
derivation of factorization theorems for scattering processes such as Drell-
Yan and deep inelastic scattering near threshold, and thrust in the two-jet
limit. In general, once bare factorization theorems have been derived, the
subsequent step of obtaining the resummation of large logarithms by means of a
RGE approach is made nontrivial by the appearance of endpoint divergences [50,
51]. It has been shown that these are an artifact of the effective theory.
Endpoint divergences cancel among terms in the factorization theorems, once
all contributions are correctly taken into account. It is then possible to
devise a subtraction procedure, which makes the individual contributions
finite, thus allowing the application of standard RGE procedure. This approach
has been fully developed to obtain the resummation of large logarithms in off-
diagonal “gluon thrust” at LL accuracy [53]. In order to extend resummation at
NLP to higher logarithmic accuracy more data is needed. To this end one needs
to calculate collinear and soft functions appearing in the factorization
theorems at higher order in perturbation theory. This program has been
completed for Drell-Yan near threshold in a series of papers, [50, 52, 54],
where all collinear and soft functions have been calculated respectively at
one and two loops in perturbation theory.

## Acknowledgment

This work has been partly supported by Fellini - Fellowship for Innovation at
INFN, funded by the European Union’s Horizon 2020 research programme under the
Marie Skłodowska-Curie Cofund Action, grant agreement no. 754496.

## References

  * [1] G. Parisi, _Summing Large Perturbative Corrections in QCD_ , _Phys. Lett. B_ 90 (1980) 295–296.
  * [2] G. Curci and M. Greco, _Large infrared corrections in QCD processes_ , _Phys. Lett._ B92 (1980) 175–178.
  * [3] G. Sterman, _Summation of large corrections to short distance hadronic cross-sections_ , _Nucl. Phys._ B281 (1987) 310.
  * [4] S. Catani and L. Trentadue, _Resummation of the QCD perturbative series for hard processes_ , _Nucl. Phys._ B327 (1989) 323.
  * [5] S. Catani and L. Trentadue, _Comment on qcd exponentiation at large x_ , _Nucl. Phys._ B353 (1991) 183–186.
  * [6] M. van Beekveld, E. Laenen, J. Sinninghe Damsté and L. Vernazza, _Next-to-leading power threshold corrections for finite order and resummed colour-singlet cross sections_ , _JHEP_ 05 (2021) 114, [2101.07270].
  * [7] M. Kramer, E. Laenen and M. Spira, _Soft gluon radiation in Higgs boson production at the LHC_ , _Nucl. Phys. B_ 511 (1998) 523–549, [hep-ph/9611272].
  * [8] C. Anastasiou, C. Duhr, F. Dulat, E. Furlan, T. Gehrmann, F. Herzog et al., _Higgs Boson Gluon-fusion Production Beyond Threshold in N ${}^{3}LO$ QCD_, _JHEP_ 03 (2015) 091, [1411.3584].
  * [9] M. Beneke, M. Garny, S. Jaskiewicz, R. Szafron, L. Vernazza and J. Wang, _Leading-logarithmic threshold resummation of Higgs production in gluon fusion at next-to-leading power_ , _JHEP_ 01 (2020) 094, [1910.12685].
  * [10] E. Laenen, L. Magnea and G. Stavenga, _On next-to-eikonal corrections to threshold resummation for the Drell-Yan and DIS cross sections_ , _Phys. Lett._ B669 (2008) 173–179, [0807.4412].
  * [11] E. Laenen, G. Stavenga and C. D. White, _Path integral approach to eikonal and next-to-eikonal exponentiation_ , _JHEP_ 0903 (2009) 054, [0811.2067].
  * [12] E. Laenen, L. Magnea, G. Stavenga and C. D. White, _Next-to-eikonal corrections to soft gluon radiation: a diagrammatic approach_ , _JHEP_ 1101 (2011) 141, [1010.1860].
  * [13] D. Bonocore, E. Laenen, L. Magnea, L. Vernazza and C. D. White, _The method of regions and next-to-soft corrections in Drell-Yan production_ , _Phys. Lett._ B742 (2015) 375–382, [1410.6406].
  * [14] D. Bonocore, E. Laenen, L. Magnea, S. Melville, L. Vernazza and C. D. White, _A factorization approach to next-to-leading-power threshold logarithms_ , _JHEP_ 06 (2015) 008, [1503.05156].
  * [15] D. Bonocore, E. Laenen, L. Magnea, L. Vernazza and C. D. White, _Non-abelian factorisation for next-to-leading-power threshold logarithms_ , _JHEP_ 12 (2016) 121, [1610.06842].
  * [16] V. Del Duca, E. Laenen, L. Magnea, L. Vernazza and C. D. White, _Universality of next-to-leading power threshold effects for colourless final states in hadronic collisions_ , _JHEP_ 11 (2017) 057, [1706.04018].
  * [17] H. Gervais, _Soft photon theorem for high energy amplitudes in Yukawa and scalar theories_ , _Phys. Rev._ D95 (2017) 125009, [1704.00806].
  * [18] N. Bahjat-Abbas, J. Sinninghe Damsté, L. Vernazza and C. D. White, _On next-to-leading power threshold corrections in Drell-Yan production at N 3LO_, _JHEP_ 1810 (2018) 144, [1807.09246].
  * [19] N. Bahjat-Abbas, D. Bonocore, J. Sinninghe Damsté, E. Laenen, L. Magnea, L. Vernazza et al., _Diagrammatic resummation of leading-logarithmic threshold effects at next-to-leading power_ , _JHEP_ 11 (2019) 002, [1905.13710].
  * [20] M. van Beekveld, W. Beenakker, E. Laenen and C. D. White, _Next-to-leading power threshold effects for inclusive and exclusive processes with final state jets_ , _JHEP_ 03 (2020) 106, [1905.08741].
  * [21] M. van Beekveld, W. Beenakker, R. Basu, E. Laenen, A. Misra and P. Motylinski, _Next-to-leading power threshold effects for resummed prompt photon production_ , _Phys. Rev. D_ 100 (2019) 056009, [1905.11771].
  * [22] E. Laenen, J. Sinninghe Damsté, L. Vernazza, W. Waalewijn and L. Zoppi, _Towards all-order factorization of QED amplitudes at next-to-leading power_ , _Phys. Rev. D_ 103 (2021) 034022, [2008.01736].
  * [23] D. Bonocore, _Asymptotic dynamics on the worldline for spinning particles_ , _JHEP_ 02 (2021) 007, [2009.07863].
  * [24] D. Bonocore, A. Kulesza and J. Pirsch, _Classical and quantum gravitational scattering with Generalized Wilson Lines_ , _JHEP_ 03 (2022) 147, [2112.02009].
  * [25] D. Bonocore and A. Kulesza, _Soft photon bremsstrahlung at next-to-leading power_ , _Phys. Lett. B_ 833 (2022) 137325, [2112.08329].
  * [26] M. van Beekveld, L. Vernazza and C. D. White, _Threshold resummation of new partonic channels at next-to-leading power_ , _JHEP_ 12 (2021) 087, [2109.09752].
  * [27] N. Agarwal, M. van Beekveld, E. Laenen, S. Mishra, A. Mukhopadhyay and A. Tripathi, _Next-to-leading power corrections to event shape variables_ , 2306.17601.
  * [28] A. Ajjath, P. Mukherjee and V. Ravindran, _On next to soft corrections to Drell-Yan and Higgs Boson productions_ , 2006.06726.
  * [29] A. H. Ajjath, P. Mukherjee, V. Ravindran, A. Sankar and S. Tiwari, _On next to soft threshold corrections to DIS and SIA processes_ , _JHEP_ 04 (2021) 131, [2007.12214].
  * [30] A. H. Ajjath, P. Mukherjee, V. Ravindran, A. Sankar and S. Tiwari, _Next-to-soft corrections for Drell-Yan and Higgs boson rapidity distributions beyond N 3LO_, _Phys. Rev. D_ 103 (2021) L111502, [2010.00079].
  * [31] A. H. Ajjath, P. Mukherjee, V. Ravindran, A. Sankar and S. Tiwari, _Next-to-soft-virtual resummed rapidity distribution for the Drell-Yan process to ${\mathrm{NNLO}}+{\overline{\mathrm{NNLL}}}$_, _Phys. Rev. D_ 106 (2022) 034005, [2112.14094].
  * [32] A. H. Ajjath, P. Mukherjee, V. Ravindran, A. Sankar and S. Tiwari, _Next-to SV resummed Drell–Yan cross section beyond leading-logarithm_ , _Eur. Phys. J. C_ 82 (2022) 234, [2107.09717].
  * [33] A. H. Ajjath, P. Mukherjee, V. Ravindran, A. Sankar and S. Tiwari, _Resummed Higgs boson cross section at next-to SV to ${\mathrm{NNLO}}+{\overline{\mathrm{NNLL}}}$_, _Eur. Phys. J. C_ 82 (2022) 774, [2109.12657].
  * [34] A. H. Ajjath, P. Mukherjee and V. Ravindran, _Going beyond soft plus virtual_ , _Phys. Rev. D_ 105 (2022) L091503, [2204.09012].
  * [35] T. Engel, A. Signer and Y. Ulrich, _Universal structure of radiative QED amplitudes at one loop_ , _JHEP_ 04 (2022) 097, [2112.07570].
  * [36] T. Engel, _The LBK theorem to all orders_ , _JHEP_ 07 (2023) 177, [2304.11689].
  * [37] M. Czakon, F. Eschment and T. Schellenberger, _Subleading Effects in Soft-Gluon Emission at One-Loop in Massless QCD_ , 2307.02286.
  * [38] A. J. Larkoski, D. Neill and I. W. Stewart, _Soft theorems from effective field theory_ , _JHEP_ 1506 (2015) 077, [1412.3108].
  * [39] I. Moult, L. Rothen, I. W. Stewart, F. J. Tackmann and H. X. Zhu, _Subleading power corrections for N-jettiness subtractions_ , _Phys. Rev._ D95 (2017) 074023, [1612.00450].
  * [40] I. Moult, I. W. Stewart and G. Vita, _A subleading operator basis and matching for gg $\rightarrow$ H_, _JHEP_ 1707 (2017) 067, [1703.03408].
  * [41] I. Moult, L. Rothen, I. W. Stewart, F. J. Tackmann and H. X. Zhu, _N -jettiness subtractions for $gg\to H$ at subleading power_, _Phys. Rev._ D97 (2018) 014013, [1710.03227].
  * [42] M. A. Ebert, I. Moult, I. W. Stewart, F. J. Tackmann, G. Vita and H. X. Zhu, _Power Corrections for N-Jettiness Subtractions at ${\cal O}(\alpha_{s})$_, _JHEP_ 12 (2018) 084, [1807.10764].
  * [43] I. Moult, I. W. Stewart, G. Vita and H. X. Zhu, _First Subleading Power Resummation for Event Shapes_ , _JHEP_ 08 (2018) 013, [1804.04665].
  * [44] I. Moult, I. W. Stewart and G. Vita, _Subleading Power Factorization with Radiative Functions_ , 1905.07411.
  * [45] I. Moult, I. W. Stewart, G. Vita and H. X. Zhu, _The Soft Quark Sudakov_ , _JHEP_ 05 (2020) 089, [1910.14038].
  * [46] M. Beneke, M. Garny, R. Szafron and J. Wang, _Anomalous dimension of subleading-power N-jet operators_ , _JHEP_ 03 (2018) 001, [1712.04416].
  * [47] M. Beneke, M. Garny, R. Szafron and J. Wang, _Anomalous dimension of subleading-power $N$-jet operators. Part II_, _JHEP_ 11 (2018) 112, [1808.04742].
  * [48] M. Beneke, A. Broggio, M. Garny, S. Jaskiewicz, R. Szafron, L. Vernazza et al., _Leading-logarithmic threshold resummation of the Drell-Yan process at next-to-leading power_ , _JHEP_ 1903 (2019) 043, [1809.10631].
  * [49] M. Beneke, M. Garny, R. Szafron and J. Wang, _Violation of the Kluberg-Stern-Zuber theorem in SCET_ , _JHEP_ 09 (2019) 101, [1907.05463].
  * [50] M. Beneke, A. Broggio, S. Jaskiewicz and L. Vernazza, _Threshold factorization of the Drell-Yan process at next-to-leading power_ , _JHEP_ 07 (2020) 078, [1912.01585].
  * [51] M. Beneke, M. Garny, S. Jaskiewicz, R. Szafron, L. Vernazza and J. Wang, _Large-x resummation of off-diagonal deep-inelastic parton scattering from d-dimensional refactorization_ , _JHEP_ 10 (2020) 196, [2008.04943].
  * [52] A. Broggio, S. Jaskiewicz and L. Vernazza, _Next-to-leading power two-loop soft functions for the Drell-Yan process at threshold_ , _JHEP_ 10 (2021) 061, [2107.07353].
  * [53] M. Beneke, M. Garny, S. Jaskiewicz, J. Strohm, R. Szafron, L. Vernazza et al., _Next-to-leading power endpoint factorization and resummation for off-diagonal “gluon” thrust_ , _JHEP_ 07 (2022) 144, [2205.04479].
  * [54] A. Broggio, S. Jaskiewicz and L. Vernazza, _Threshold factorization of the Drell-Yan quark-gluon channel and two-loop soft function at next-to-leading power_ , _JHEP_ 12 (2023) 028, [2306.06037].
  * [55] C. W. Bauer, S. Fleming, D. Pirjol and I. W. Stewart, _An effective field theory for collinear and soft gluons: Heavy to light decays_ , _Phys. Rev._ D63 (2001) 114020, [hep-ph/0011336].
  * [56] C. W. Bauer, D. Pirjol and I. W. Stewart, _Soft collinear factorization in effective field theory_ , _Phys. Rev. D_ 65 (2002) 054022, [hep-ph/0109045].
  * [57] M. Beneke, A. P. Chapovsky, M. Diehl and T. Feldmann, _Soft collinear effective theory and heavy to light currents beyond leading power_ , _Nucl. Phys. B_ 643 (2002) 431–476, [hep-ph/0206152].
  * [58] T. Becher, M. Neubert and G. Xu, _Dynamical Threshold Enhancement and Resummation in Drell-Yan Production_ , _JHEP_ 07 (2008) 030, [0710.0680].
  * [59] T. Becher, M. Neubert and B. D. Pecjak, _Factorization and Momentum-Space Resummation in Deep-Inelastic Scattering_ , _JHEP_ 01 (2007) 076, [hep-ph/0607228].
  * [60] T. Becher and M. D. Schwartz, _A precise determination of $\alpha_{s}$ from LEP thrust data using effective field theory_, _JHEP_ 07 (2008) 034, [0803.0342].
  * [61] S. Catani, L. Trentadue, G. Turnock and B. R. Webber, _Resummation of large logarithms in e+ e- event shape distributions_ , _Nucl. Phys._ B407 (1993) 3–42.
  * [62] T. Becher and M. Neubert, _On the Structure of Infrared Singularities of Gauge-Theory Amplitudes_ , _JHEP_ 0906 (2009) 081, [0903.1126].
  * [63] M. Beneke and T. Feldmann, _Multipole expanded soft collinear effective theory with nonAbelian gauge symmetry_ , _Phys. Lett. B_ 553 (2003) 267–276, [hep-ph/0211358].
  * [64] Z. L. Liu and M. Neubert, _Factorization at subleading power and endpoint-divergent convolutions in $h\to\gamma\gamma$ decay_, _JHEP_ 04 (2020) 033, [1912.08818].
  * [65] Z. L. Liu, B. Mecaj, M. Neubert and X. Wang, _Factorization at subleading power, Sudakov resummation, and endpoint divergences in soft-collinear effective theory_ , _Phys. Rev. D_ 104 (2021) 014004, [2009.04456].
  * [66] Z. L. Liu, B. Mecaj, M. Neubert and X. Wang, _Factorization at subleading power and endpoint divergences in $h\to\gamma\gamma$ decay. Part II. Renormalization and scale evolution_, _JHEP_ 01 (2021) 077, [2009.06779].
  * [67] Z. L. Liu, M. Neubert, M. Schnubel and X. Wang, _Factorization at next-to-leading power and endpoint divergences in gg → h production_ , _JHEP_ 06 (2023) 183, [2212.10447].
  * [68] P. Böer, _QCD Factorisation in Exclusive Semileptonic B Decays New Applications and Resummation of Rapidity Logarithms_ , Ph.D. thesis, Siegen U., 2018.
  * [69] T. Feldmann, N. Gubernari, T. Huber and N. Seitz, _Contribution of the electromagnetic dipole operator O7 to the B¯s→ $\mu$+$\mu$\- decay amplitude_, _Phys. Rev. D_ 107 (2023) 013007, [2211.04209].
  * [70] C. Cornella, M. König and M. Neubert, _Structure-dependent QED effects in exclusive B decays at subleading power_ , _Phys. Rev. D_ 108 (2023) L031502, [2212.14430].
  * [71] T. Hurth and R. Szafron, _Refactorisation in subleading B¯→Xs $\gamma$_, _Nucl. Phys. B_ 991 (2023) 116200, [2301.01739].
  * [72] G. Bell, P. Böer and T. Feldmann, _Muon-electron backward scattering: a prime example for endpoint singularities in SCET_ , _JHEP_ 09 (2022) 183, [2205.06021].
  * [73] Z. L. Liu, M. Neubert, M. Schnubel and X. Wang, _Radiative quark jet function with an external gluon_ , _JHEP_ 02 (2022) 075, [2112.00018].


$\textbf{MTGNN}_{\text{rnd}}$ | 0.10 | 0.12 | 0.12 | 0.17 | 0.03 | 0.05 | 0.11 | 0.17 | 0.13 | 0.19 | 0.14 | 0.27 | 0.34 | 0.57 | 0.23 | 0.13
$\textbf{MTGNN}_{\text{der++}}$ | 0.15 | 0.25 | 0.35 | 0.54 | 0.06 | 0.02 | 0.08 | 0.07 | 0.17 | 0.16 | 0.18 | 0.23 | 0.48 | 0.56 | 0.21 | 0.17
$\textbf{Autoformer}_{\text{seq}}$ | 0.58 | 1.11 | 1.51 | 2.88 | 0.11 | 0.27 | 0.20 | 0.23 | 0.12 | 0.31 | 0.27 | 0.24 | 0.65 | 0.41 | 0.32 | 0.51
$\textbf{Autoformer}_{\text{mir}}$ | 0.69 | 1.23 | 0.75 | 1.33 | 0.10 | 0.24 | 0.21 | 0.37 | 0.22 | 0.39 | 0.18 | 0.25 | 0.40 | 0.31 | 0.19 | 0.30
$\textbf{Autoformer}_{\text{herd}}$ | 0.77 | 1.43 | 0.67 | 1.20 | 0.36 | 0.36 | 0.23 | 0.41 | 0.12 | 0.13 | 0.14 | 0.12 | 0.90 | 0.61 | 0.71 | 0.70
$\textbf{Autoformer}_{\text{rnd}}$ | 0.86 | 1.42 | 0.74 | 1.18 | 0.18 | 0.31 | 0.24 | 0.31 | 0.17 | 0.27 | 0.35 | 0.31 | 0.46 | 0.31 | 0.25 | 0.43
$\textbf{Autoformer}_{\text{der++}}$ | 0.84 | 1.54 | 0.63 | 1.34 | 0.26 | 0.43 | 0.22 | 0.40 | 0.18 | 0.30 | 0.21 | 0.23 | 0.72 | 0.71 | 0.54 | 0.47
$\textbf{PatchTST}_{\text{seq}}$ | 0.27 | 0.21 | 0.25 | 0.46 | 0.24 | 0.17 | 0.12 | 0.21 | 0.24 | 0.10 | 0.26 | 0.15 | 0.16 | 0.54 | 0.40 | 0.54
$\textbf{PatchTST}_{\text{mir}}$ | 0.13 | 0.11 | 0.17 | 0.21 | 0.29 | 0.12 | 0.21 | 0.23 | 0.21 | 0.11 | 0.09 | 0.10 | 0.32 | 0.42 | 0.49 | 0.57
$\textbf{PatchTST}_{\text{herd}}$ | 0.17 | 0.13 | 0.19 | 0.17 | 0.12 | 0.08 | 0.12 | 0.14 | 0.24 | 0.10 | 0.27 | 0.13 | 0.10 | 0.54 | 0.40 | 0.49
$\textbf{PatchTST}_{\text{er}}$ | 0.146 | 0.25 | 0.19 | 0.18 | 0.16 | 0.09 | 0.09 | 0.17 | 0.07 | 0.09 | 0.08 | 0.09 | 0.14 | 0.11 | 0.26 | 0.35
$\textbf{PatchTST}_{\text{der++}}$ | 0.20 | 0.15 | 0.04 | 0.16 | 0.14 | 0.10 | 0.15 | 0.31 | 0.15 | 0.27 | 0.17 | 0.19 | 0.26 | 0.23 | 0.21 | 0.48
$\textbf{Dlinear}_{\text{seq}}$ | 0.21 | 0.19 | 0.23 | 0.36 | 0.18 | 0.21 | 0.17 | 0.18 | 0.16 | 0.11 | 0.26 | 0.26 | 0.14 | 0.25 | 0.39 | 0.53
$\textbf{Dlinear}_{\text{mir}}$ | 0.25 | 0.24 | 0.23 | 0.22 | 0.19 | 0.13 | 0.15 | 0.12 | 0.15 | 0.29 | 0.11 | 0.14 | 0.32 | 0.53 | 0.36 | 0.49
$\textbf{Dlinear}_{\text{herd}}$ | 0.15 | 0.17 | 0.17 | 0.25 | 0.14 | 0.22 | 0.13 | 0.26 | 0.12 | 0.37 | 0.18 | 0.27 | 0.25 | 0.14 | 0.21 | 0.43
$\textbf{Dlinear}_{\text{er}}$ | 0.23 | 0.17 | 0.18 | 0.12 | 0.11 | 0.14 | 0.16 | 0.23 | 0.14 | 0.31 | 0.27 | 0.23 | 0.22 | 0.46 | 0.56 | 0.62
$\textbf{Dlinear}_{\text{der++}}$ | 0.16 | 0.08 | 0.08 | 0.14 | 0.13 | 0.14 | 0.08 | 0.28 | 0.01 | 0.16 | 0.13 | 0.12 | 0.29 | 0.49 | 0.23 | 0.32
$\textbf{TimesNet}_{\text{seq}}$ | 0.17 | 0.14 | 0.23 | 0.22 | 0.09 | 0.13 | 0.15 | 0.08 | 0.16 | 0.21 | 0.11 | 0.14 | 0.32 | 0.53 | 0.36 | 0.49
$\textbf{TimesNet}_{\text{mir}}$ | 0.12 | 0.16 | 0.18 | 0.20 | 0.10 | 0.08 | 0.14 | 0.12 | 0.14 | 0.12 | 0.27 | 0.21 | 0.14 | 0.25 | 0.31 | 0.35
$\textbf{TimesNet}_{\text{herd}}$ | 0.14 | 0.19 | 0.17 | 0.25 | 0.14 | 0.20 | 0.09 | 0.26 | 0.15 | 0.37 | 0.09 | 0.32 | 0.23 | 0.14 | 0.21 | 0.43
$\textbf{TimesNet}_{\text{er}}$ | 0.08 | 0.10 | 0.12 | 0.20 | 0.18 | 0.13 | 0.12 | 0.20 | 0.09 | 0.15 | 0.07 | 0.24 | 0.11 | 0.21 | 0.18 | 0.36
$\textbf{TimesNet}_{\text{der++}}$ | 0.07 | 0.13 | 0.11 | 0.15 | 0.09 | 0.11 | 0.08 | 0.26 | 0.07 | 0.13 | 0.06 | 0.15 | 0.18 | 0.31 | 0.21 | 0.33
$\textbf{iTransformer}_{\text{seq}}$ | 0.18 | 0.17 | 0.20 | 0.34 | 0.19 | 0.19 | 0.17 | 0.22 | 0.12 | 0.11 | 0.25 | 0.24 | 0.14 | 0.28 | 0.43 | 0.57
$\textbf{iTransformer}_{\text{mir}}$ | 0.23 | 0.29 | 0.22 | 0.22 | 0.23 | 0.09 | 0.12 | 0.07 | 0.11 | 0.28 | 0.09 | 0.13 | 0.32 | 0.57 | 0.40 | 0.46
$\textbf{iTransformer}_{\text{herd}}$ | 0.11 | 0.12 | 0.17 | 0.27 | 0.17 | 0.22 | 0.16 | 0.21 | 0.15 | 0.33 | 0.20 | 0.28 | 0.29 | 0.16 | 0.25 | 0.45
$\textbf{iTransformer}_{\text{er}}$ | 0.21 | 0.20 | 0.18 | 0.16 | 0.08 | 0.09 | 0.19 | 0.25 | 0.19 | 0.29 | 0.25 | 0.26 | 0.19 | 0.46 | 0.58 | 0.65
$\textbf{iTransformer}_{\text{der++}}$ | 0.14 | 0.08 | 0.10 | 0.12 | 0.12 | 0.13 | 0.11 | 0.30 | 0.02 | 0.20 | 0.15 | 0.16 | 0.30 | 0.50 | 0.20 | 0.21
$\textbf{OFA}_{\text{seq}}$ | 0.14 | 0.10 | 0.27 | 0.23 | 0.13 | 0.09 | 0.18 | 0.13 | 0.12 | 0.19 | 0.13 | 0.14 | 0.29 | 0.51 | 0.40 | 0.52
$\textbf{OFA}_{\text{mir}}$ | 0.10 | 0.17 | 0.21 | 0.17 | 0.11 | 0.11 | 0.14 | 0.14 | 0.15 | 0.15 | 0.23 | 0.25 | 0.19 | 0.23 | 0.35 | 0.33
$\textbf{OFA}_{\text{herd}}$ | 0.19 | 0.24 | 0.19 | 0.22 | 0.17 | 0.18 | 0.11 | 0.21 | 0.13 | 0.41 | 0.08 | 0.34 | 0.21 | 0.17 | 0.25 | 0.45
$\textbf{OFA}_{\text{er}}$ | 0.10 | 0.12 | 0.15 | 0.16 | 0.21 | 0.13 | 0.09 | 0.20 | 0.05 | 0.18 | 0.03 | 0.23 | 0.10 | 0.17 | 0.13 | 0.38
$\textbf{OFA}_{\text{der++}}$ | 0.08 | 0.17 | 0.13 | 0.18 | 0.09 | 0.14 | 0.06 | 0.28 | 0.09 | 0.16 | 0.01 | 0.11 | 0.22 | 0.28 | 0.18 | 0.37
$\textbf{SKI-CL}_{\text{seq}}$ | 0.30 | 0.40 | 0.39 | 0.54 | 0.03 | 0.03 | 0.04 | 0.04 | 0.03 | 0.02 | 0.04 | 0.17 | 0.17 | 0.26 | 0.17 | 0.23
$\textbf{SKI-CL}_{\text{mir}}$ | 0.14 | 0.31 | 0.22 | 0.20 | 0.01 | 0.03 | 0.08 | 0.02 | 0.01 | 0.22 | 0.22 | 0.29 | 0.36 | 0.31 | 0.11 | 0.21
$\textbf{SKI-CL}_{\text{herd}}$ | 0.15 | 0.21 | 0.14 | 0.20 | 0.05 | 0.07 | 0.07 | 0.09 | 0.05 | 0.12 | 0.15 | 0.19 | 0.18 | 0.25 | 0.25 | 0.36
$\textbf{SKI-CL}_{\text{rnd}}$ | 0.04 | 0.10 | 0.04 | 0.16 | 0.06 | 0.06 | 0.05 | 0.10 | 0.02 | 0.09 | 0.23 | 0.21 | 0.16 | 0.22 | 0.35 | 0.49
$\textbf{SKI-CL}_{\text{der++}}$ | 0.04 | 0.07 | 0.12 | 0.17 | 0.04 | 0.05 | 0.08 | 0.11 | 0.04 | 0.12 | 0.15 | 0.13 | 0.25 | 0.29 | 0.15 | 0.11
SKI-CL | 0.04 | 0.06 | 0.03 | 0.03 | 0.02 | 0.05 | 0.03 | 0.08 | 0.04 | 0.09 | 0.08 | 0.09 | 0.27 | 0.27 | 0.13 | 0.09

##  Appendix G A Case Study with Inferred Structures and Prediction
Visualizations

We provide a case study on the Synthetic-CL dataset to illustrate the efficacy
of SKI-CL, as shown in Figure 11. Our analysis is based on the final SKI-CL
model that has been sequentially trained over all regimes. We select three
variables (nodes) and visualize the testing data, where the temporal dynamics
obviously differ across four regimes. It is clear that SKI-CL can render a
faithful dependency structure that well aligns the similarity of variables in
each regime. Moreover, SKI-CL gives relatively accurate forecasts that capture
each variable’s temporal dynamics of ground truths.

Figure 11. A case study of SKI-CL on Synthetic-CL dataset. In each regime, red
rectangles indicate the correspondence between the ground truth time series
(top-left) and the inferred variable dependencies (top-right), red arrows
indicate the comparisons between these ground truth values and corresponding
predictions (bottoms).


# Does BERT agree? Evaluating knowledge of
structure dependence through agreement relations

Geoff Bacon
Department of Linguistics
University of California, Berkeley
<EMAIL_ADDRESS>
Terry Regier
Department of Linguistics and
Cognitive Science Program
University of California, Berkeley
<EMAIL_ADDRESS>

###### Abstract

Learning representations that accurately model semantics is an important goal
of natural language processing research. Many semantic phenomena depend on
syntactic structure. Recent work examines the extent to which state-of-the-art
models for pre-training representations, such as BERT, capture such structure-
dependent phenomena, but is largely restricted to one phenomenon in English:
number agreement between subjects and verbs. We evaluate BERT’s sensitivity to
four types of structure-dependent agreement relations in a new semi-
automatically curated dataset across 26 languages. We show that both the
single-language and multilingual BERT models capture syntax-sensitive
agreement patterns well in general, but we also highlight the specific
linguistic contexts in which their performance degrades.

##  1 Introduction

Learning general-purpose sentence representations which accurately model
sentential semantic content is a current goal of natural language processing
research Subramanian et al. (2018); Conneau et al. (2017); Wieting et al.
(2016); Kiros et al. (2015). A prominent and successful approach is to pre-
train neural networks to encode sentences into fixed length vectors Conneau et
al. (2018); Nie et al. (2017), with common architecture choices based on
recurrent neural networks Elman (1990); Hochreiter and Schmidhuber (1997),
convolutional neural networks, or transformers Vaswani et al. (2017). Many
core linguistic phenomena that one would like to model in general-purpose
sentence representations depend on syntactic structure Chomsky (1965);
Everaert et al. (2015). Despite the fact that none of the aforementioned
architectures have explicit syntactic structural representations, there is
some evidence that these models can approximate such structure-dependent
phenomena under certain conditions Gulordava et al. (2018); McCoy et al.
(2018); Linzen et al. (2016); Bowman et al. (2015), in addition to their
widespread success in practical tasks.

The recently introduced BERT model Devlin et al. (2018), which is based on
transformers, achieves state-of-the-art results on eleven natural language
processing tasks. In this work, we assess BERT’s ability to learn structure-
dependent linguistic phenomena of agreement relations. To test whether BERT is
sensitive to agreement relations, we use the cloze test (Taylor, 1953, also
called the “masked language model” objective), in which we mask out one of two
words in an agreement relation and ask BERT to predict the masked word, one of
the two tasks on which BERT is initially trained.

Goldberg (2019) adapted the experimental setup of Linzen et al. (2016),
Gulordava et al. (2018) and Marvin and Linzen (2018) to use the cloze test to
assess BERT’s sensitivity to number agreement in English subject-verb
agreement relations. The results showed that the single-language BERT model
performed surprisingly well at this task (above 80% accuracy in all
experiments), even when there were multiple “distractors” in the sentence
(other nouns that differed from the subject in number). This suggests that
BERT is actually learning to approximate structure-dependent computation, and
not simply relying on flawed heuristics.

However, English subject-verb agreement is a rather restricted phenomenon,
with the majority of verbs having only two inflected forms and only one
morphosyntactic feature (number) involved. To what extent does Goldberg’s
(2019) result hold for subject-verb agreement in other languages, including
more morphologically rich ones, as well as for other types of agreement
relations? Building on Goldberg’s (2019) work, we expand the experiment to 26
languages and four types of agreement relations, which include more
challenging examples.

In Section 2, we define what is meant by agreement relations and outline the
particular agreement relations under study. Section 3 introduces our newly
curated cross-linguistic dataset of agreement relations, while section 4
discusses our experimental setup. We report the results of our experiments in
section 5. All data and code are available at
https://github.com/geoffbacon/does-bert-agree.

##  2 Structure-dependent agreement relations

Agreement phenomena are an important and cross-linguistically common property
of natural languages, and as such have been extensively studied in syntax and
morphology Corbett (2006).111For a comprehensive bibliography, see
http://www.smg.surrey.ac.uk/projects/agreement/bibliography/. Languages often
express grammatical features, such as number and gender, through inflectional
morphology. An agreement relation is a morphophonologically overt co-variance
in feature values between two words in a syntactic relationship Preminger
(2014). In other words, agreement refers to when the morphosyntactic features
of one word are reflected in its syntactic dependents. In this way, agreement
relations are overt markers of covert syntactic structure. Thus, evaluating a
model’s ability to capture agreement relations is also an evaluation of its
ability to capture syntactic structure.

Following Corbett (2003), we call the syntactically dependent word the
“target” of the agreement relation, and the word with which it agrees we call
the “controller”. An example of an agreement relation in English is given in
(2), in which the inflected form of the verb be (_are_) reflects the plural
number of its syntactic head _keys_. In all examples in this section, the
controller and target are given in bold. In this example, _keys_ is the
controller and _are_ is the target of the agreement relation.

The keys to the door are on the table.

The agreement relation in (2) is between a subject and its verb, but there are
other types of agreement relations. In addition to subject-verb agreement,
three other types of agreement relations are cross-linguistically common:
agreement of noun with i) determiner, ii) attributive adjective and iii)
predicate adjective Baker (2008). The latter two types are distinguished by
whether the adjective modifies the noun within a noun phrase or whether it is
predicated of the subject of a clause. The first two types are sometimes
categorized as nominal concord rather than agreement, but for our purposes
this is merely a difference in terminology.

The morphosyntactic feature in the agreement relation in (2) is number, a
feature that is cross-linguistically common in agreement systems. In addition
to number, the most commonly involved in agreement relations are gender, case
and person Baker (2008).

With its comparatively limited inflectional morphology, English only exhibits
subject-verb and determiner agreement (in demonstratives, “this” vs. “these”)
and even then only agrees for number. Languages with richer inflectional
morphology tend to display more agreement types and involve more features.
French, for example, employs all four types of agreement relations. Examples
are given in (2)-(2). The subject and verb in (2) agree for number, while the
noun and determiner in (2), the noun and attributive adjective in (2) and the
subject and predicated adjective in (2) agree for both number and gender.

Les clés de la porte se trouvent sur la table.

‘The keys to the door are on the table.’

Je peux voir les clés.

‘I can see the keys.’

Je ne veux plus les clés totalement cassées.

‘I no longer want the completely broken keys.’

Les clés de la porte sont cassées.

‘The keys to the door are broken.’

Previous work using agreement relations to assess knowledge of syntactic
structure in modern neural networks has focussed on subject-verb agreement in
number Goldberg (2019); Gulordava et al. (2018); Linzen et al. (2016). In our
work, we study all four types of agreement relations and all four features
discussed above. Moreover, previous work using any method to assess BERT’s
knowledge of syntactic structure has focussed exclusively on the single-
language English model Hewitt and Manning (2019); Goldberg (2019); Tenney et
al. (2019); Lin et al. (2019); Jawahar et al. (2019); Clark et al. (2019). We
expand this line of work to 26 languages. Not all languages in our sample
exhibit all four types of agreement nor use all four features examined, but
they all exhibit at least one of the agreement types involving at least one of
the features.

##  3 Data

Our study requires two types of data. First, we need sentences containing
agreement relations. We mask out one of the words in the agreement relation
and ask BERT to predict the masked word. We are interested in BERT’s ability
to predict words that respect the agreement relation, that is, words which
share the morphosyntactic features of the word with which it agrees. To
measure this, we need to know the feature values for each word in BERT’s
vocabulary. This is our second type of data. Throughout this paper, we refer
to the first type of data as the cloze data, and the second as the feature
data.

In the design of our datasets, we followed two principles. First, we chose
data sources that are available across multiple languages, because we are
interested in cross-linguistic generality. The languages in this study are
those with sufficiently large data sources that also appear in the
multilingual BERT model. Second, we use naturally-occurring data (cf. Marvin
and Linzen (2018)).

###  3.1 Cloze data

We sourced our cloze data from version 2.4 of the Universal Dependencies
treebanks (Nivre et al., 2016, UD). The UD treebanks use a consistent schema
across all languages to annotate naturally occurring sentences at the word
level with rich grammatical information. We used the part-of-speech and
dependency information to identify potential agreement relations.
Specifically, we identified all instances of subject-verb, noun-determiner,
noun-attributive adjective and subject-predicate adjective word pairs. We then
used the morphosyntactic annotations for number, gender, case and person to
filter out word pairs that disagree due to errors in the underlying data
source (e.g. one is annotated as plural while the other is singular) or that
are not annotated for any of the four features.

This method is language-agnostic, but due to errors in the underlying UD
corpora, yielded some false positives (e.g. predicate adjective agreement in
English). To correct for this, we consulted reference grammars of each
language to note which of the four types of agreement exist in the language.
We removed all examples that are of the wrong type for the language (8% of
harvested examples). Across the 26 languages, we curated almost one million
cloze examples. Their breakdown across agreement type and language is shown in
Tables 1 and 2.

Agreement type | # cloze
---|---
attributive adjective | 351,300
determiner | 349,073
subject-verb | 253,820
predicate adjective | 26,696
total | 980,889
Table 1: Number of cloze examples per agreement type in our new cross-
linguistic dataset on agreement relations. Previous work has largely focused
on subject-verb agreement in English.

In all four types of agreement studied, the controller of the agreement is a
noun or pronoun, while the target can be a determiner, adjective or verb.
Because of this part-of-speech restriction, we chose to mask out the
controller in every cloze example so that BERT is evaluated against the same
vocabulary across all four types. This also means that we only need to collect
feature data on nouns and pronouns.

Language | # cloze | # feature bundles
---|---|---
Russian | 144,458 | 2,404
Italian | 137,268 | 2,479
French | 117,769 | 3,384
Catalan | 88,053 | 1,753
English | 60,060 | 6,743
Dutch | 52,569 | 1,531
Latin | 50,053 | 1,044
Polish | 47,513 | 2,011
Portuguese | 47,038 | 2,107
Finnish | 36,705 | 1,167
Romanian | 29,746 | 1,330
Norwegian | 29,059 | 1,393
Hindi | 22,959 | 402
Croatian | 21,835 | 1,141
Persian | 14,238 | 985
Greek | 14,017 | 216
Ukrainian | 13,929 | 1,206
Swedish | 10,889 | 1,611
Hebrew | 10,809 | 338
Danish | 9,432 | 1,330
Urdu | 8,139 | 547
Basque | 4,132 | 267
Turkish | 3,155 | 846
Irish | 3,030 | 259
Afrikaans | 2,304 | 365
Armenian | 1,434 | 211
Table 2: Language statistics of our new cross-linguistic dataset on agreement
relations. Most previous work has focused on English. “# cloze” is the number
of cloze examples curated for each language, and “# feature bundles” is the
number of unique sets of morphosyntactic features harvested for word types in
BERT’s vocabulary.

###  3.2 Feature data

Our feature data comes from both the UD and the UniMorph projects (Sylak-
Glassman, 2016, downloaded June 2019). The UniMorph project also uses a
consistent schema across all languages to annotate word types with
morphological features. Although this schema is not the same as that used in
UD, there is a deterministic mapping between the two McCarthy et al. (2018).

In this work, a word can take on a particular bundle of feature values (e.g.
singular, feminine and third person) if it appears with those features in
either UD or UniMorph. The UniMorph data directly specifies what bundles of
feature values a word can take on. For the Universal Dependencies data, we say
a word can take on a particular bundle if we ever see it with that bundle of
feature values in a Universal Dependencies corpus for that language. Both
sources individually allow for a word to have multiple feature bundles (e.g.
_sheep_ in English can be singular or plural). In these cases, we keep all
possible feature bundles. Finally, we filter out words that do not appear in
BERT’s vocabulary.

##  4 Experiment

Our experiment is designed to measure BERT’s ability to model syntactic
structure. Our experimental set up is an adaptation of that of Goldberg
(2019). As in previous work, we mask one word involved in an agreement
relation and ask BERT to predict it. Goldberg (2019), following Linzen et al.
(2016), considered a correct prediction to be one in which the masked word
receives a higher probability than other inflected forms of the lemma. For
example, when _dogs_ is masked, a correct response gives more probability to
_dogs_ than _dog_. This evaluation leaves open the possibility that
selectional restrictions or frequency are responsible for the results rather
than sensitivity to syntactic structure Gulordava et al. (2018). To remove
this possibility, we take into account all words of the same part-of-speech as
the masked word. Concretely, we consider a correct prediction to be one in
which the average probability of all possible correct words is higher than
that of all incorrect words. By “correct words”, we mean words with the exact
same feature values and the same part of speech as the masked word. By
“incorrect words”, we mean words of the same part of speech as the masked word
but that differ from the masked word with respect to at least one feature
value. We ignore cloze examples in which there are fewer than 10 possible
correct and 10 incorrect answers in our feature data. The average example in
our cloze data is evaluated using 1,468 words, compared with 2 in Goldberg
(2019).

Following Goldberg (2019), we use the pre-trained BERT models from the
original authors222https://github.com/google-research/bert, but through the
PyTorch implementation.333https://github.com/huggingface/pytorch-pretrained-
BERT Goldberg (2019) showed that in his experiments the base BERT model
performed better than the larger model, so we restrict our attention to the
base model. For English, we use the model trained only on English data,
whereas for all other languages we use the multilingual model.

##  5 Results

Overall, BERT performs well on our experimental task, suggesting that it is
able to model syntactic structure. BERT was correct in 94.3% of all cloze
examples. This high performance is found across all four types of agreement
relations. Figure 1 shows that BERT performed above 90% accuracy in each type.
Performance is best on determiner and attributive agreement relations, while
worst on subject-verb and predicate adjective.

Figure 1: Accuracy per agreement type aggregated across all languages. In all
four types, BERT performed above 90% accuracy. Accuracy is slightly lower for
predicate adjectives and subject-verb agreement relations, which typically
have longer distance dependencies. Error bars are bootstrapped 95% confidence
intervals. Figure 2: Accuracy per language aggregated across all four
agreement types. In all 26 languages, BERT performs above 60% accuracy. In
most languages BERT performs above 90% accuracy, although performance is
significantly lower for a handful of languages. Error bars are bootstrapped
95% confidence intervals.

In figure 2, we see BERT’s performance for each language. BERT performs well
for the majority of languages, although some fare much worse than others. It
is important to note that it is an unfair comparison because even though the
datasets were curated using the same methodology, each language’s dataset is
different. It is possible, for example, that the examples we have for Basque
are simply harder than they are for Portuguese.

Finally, we ask how BERT’s performance is affected by distance between the
controller and the target, as well as the number of distractors. Figure 3
shows BERT’s performance, aggregated over all languages and types, as a
function of the distance involved in the agreement, while figure 4 shows the
same for number of distractors. There is a slight but consistent decrease in
performance as the distance and the number of distractors increase. The
decline in performance begins later in figure 4 but drops more rapidly once it
does.

Figure 3: Accuracy as a function of distance between controller and target of
agreement, aggregated across all languages and agreement types. BERT is
relatively robust to longer-distance dependencies but does show a small
decrease as the dependency length increases. Error bars are bootstrapped 95%
confidence intervals. Figure 4: Accuracy as a function of number of
distractors (other nouns in the sentence with different feature values),
aggregated across all languages and agreement types. As with distance, BERT is
quite robust to distractors although there is a more noticeable decrease in
accuracy as more distractors are present. Error bars are bootstrapped 95%
confidence intervals.

##  6 Related work

Given the success of large pre-trained language representation models on
downstream tasks, it is not surprising that that the field wants to understand
the extent of their linguistic knowledge.444For a thorough overview of the
recent push to understand what pre-trained models learn about language, see
Belinkov and Glass (2019). In our work, we looked exclusively at the
predictions BERT makes at the word level. Tenney et al. (2019) and Jawahar et
al. (2019) examined the internal representations of BERT to find that
syntactic concepts are learned at lower levels than semantic concepts. Hewitt
and Manning (2019) are also interested in syntactic knowledge and propose a
method to evaluate whether entire syntax trees are embedded in a linear
transformation of a model’s word representation space, finding that BERT does
capture such information. As a complementary approach, Clark et al. (2019)
studied the attention mechanism of BERT, finding clear correlates with
interpretable linguistic structures such as direct objects, and suggest that
BERT’s success is due in part to its syntactic awareness. However, by
subjecting it to existing psycholinguistic tasks, Ettinger (2019) found that
BERT fails in its ability to understand negation. In concurrent work, van
Schijndel et al. (forthcoming) show that BERT does not consistently outperform
LSTM-based models on English subject-verb agreement tasks.

##  7 Conclusions & future work

Core linguistic phenomena depend on syntactic structure. Yet current state-of-
the-art models in language representations, such as BERT, do not have explicit
syntactic structural representations. Previous work by Goldberg (2019) showed
that BERT captures English subject-verb number agreement well despite this
lack of explicit structural representation. We replicated this result using a
different evaluation methodology that addresses shortcomings in the original
methodology and expanded the study to 26 languages. Our study further
broadened existing work by considering the most cross-linguistically common
agreement types as well as the most common morphosyntactic features. The main
result of this expansion into more languages, types and features is that BERT,
without explicit syntactic structure, is still able to capture syntax-
sensitive agreement patterns well. However, our analysis highlights an
important qualification of this result. We showed that BERT’s ability to model
syntax-sensitive agreement relations decreases slightly as the dependency
becomes longer range, and as the number of distractors increases. We release
our new curated cross-linguistic datasets and code in the hope that it is
useful to future research that may probe why this pattern appears.

The experimental setup we used has some known limitations. First, in certain
languages some of the cloze examples we studied contain redundant information.
Even when one word from an agreement relation is masked out, other cues remain
in the sentence (e.g. when masking out the noun for a French attributive
adjective agreement relation, number information is still available from the
determiner). To counter this in future work, we plan to run our experiment
twice, masking out the controller and then the target. Second, we used a
different evaluation scheme than previous work Goldberg (2019) by averaging
BERT’s predictions over many word types and plan to compare both schemes in
future work.

## References

  * Baker (2008) Mark C. Baker. 2008.  _The Syntax of Agreement and Concord_.  Cambridge Studies in Linguistics. Cambridge University Press.
  * Belinkov and Glass (2019) Yonatan Belinkov and James Glass. 2019.  Analysis methods in neural language processing: A survey.  _Transactions of the Association for Computational Linguistics_ , 7:49–72.
  * Bowman et al. (2015) Samuel R Bowman, Christopher D Manning, and Christopher Potts. 2015.  Tree-structured composition in neural networks without tree-structured architectures.  In _Proceedings of the 2015 International Conference on Cognitive Computation: Integrating Neural and Symbolic Approaches-Volume 1583_ , pages 37–42. CEUR-WS. org.
  * Chomsky (1965) Noam Chomsky. 1965.  _Aspects of the Theory of Syntax_.  Cambridge, MA. MIT Press.
  * Clark et al. (2019) Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D Manning. 2019.  What does BERT look at? An analysis of BERT’s attention.  _arXiv preprint arXiv:1906.04341_.
  * Conneau et al. (2017) Alexis Conneau, Douwe Kiela, Holger Schwenk, Loïc Barrault, and Antoine Bordes. 2017.  Supervised learning of universal sentence representations from natural language inference data.  In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_ , pages 670–680, Copenhagen, Denmark. Association for Computational Linguistics.
  * Conneau et al. (2018) Alexis Conneau, Germán Kruszewski, Guillaume Lample, Loïc Barrault, and Marco Baroni. 2018.  What you can cram into a single vector: Probing sentence embeddings for linguistic properties.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 2126–2136. Association for Computational Linguistics.
  * Corbett (2003) Greville G Corbett. 2003.  Agreement: terms and boundaries.  In _The Role of Agreement in Natural Language. Proceedings of the 2001 Texas Linguistic Society Conference, Austin, Texas._ , pages 109–122.
  * Corbett (2006) Greville G Corbett. 2006.  _Agreement_ , volume 109.  Cambridge University Press.
  * Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018.  BERT: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_.
  * Elman (1990) Jeffrey L Elman. 1990.  Finding structure in time.  _Cognitive science_ , 14(2):179–211.
  * Ettinger (2019) Allyson Ettinger. 2019.  What BERT is not: Lessons from a new suite of psycholinguistic diagnostics for language models.
  * Everaert et al. (2015) Martin BH Everaert, Marinus AC Huybregts, Noam Chomsky, Robert C Berwick, and Johan J Bolhuis. 2015.  Structures, not strings: Linguistics as part of the cognitive sciences.  _Trends in cognitive sciences_ , 19(12):729–743.
  * Goldberg (2019) Yoav Goldberg. 2019.  Assessing BERT’s syntactic abilities.  _arXiv preprint arXiv:1901.05287_.
  * Gulordava et al. (2018) Kristina Gulordava, Piotr Bojanowski, Edouard Grave, Tal Linzen, and Marco Baroni. 2018.  Colorless green recurrent networks dream hierarchically.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pages 1195–1205. Association for Computational Linguistics.
  * Hewitt and Manning (2019) John Hewitt and Christopher D Manning. 2019.  A structural probe for finding syntax in word representations.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4129–4138.
  * Hochreiter and Schmidhuber (1997) Sepp Hochreiter and Jürgen Schmidhuber. 1997.  Long short-term memory.  _Neural computation_ , 9(8):1735–1780.
  * Jawahar et al. (2019) Ganesh Jawahar, Benoît Sagot, Djamé Seddah, Samuel Unicomb, Gerardo Iñiguez, Márton Karsai, Yannick Léo, Márton Karsai, Carlos Sarraute, Éric Fleury, et al. 2019.  What does BERT learn about the structure of language?  In _57th Annual Meeting of the Association for Computational Linguistics (ACL), Florence, Italy_.
  * Kiros et al. (2015) Ryan Kiros, Yukun Zhu, Ruslan R Salakhutdinov, Richard Zemel, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015.  Skip-thought vectors.  In _Advances in neural information processing systems_ , pages 3294–3302.
  * Lin et al. (2019) Yongjie Lin, Yi Chern Tan, and Robert Frank. 2019.  Open sesame: Getting inside BERT’s linguistic knowledge.  _arXiv preprint arXiv:1906.01698_.
  * Linzen et al. (2016) Tal Linzen, Emmanuel Dupoux, and Yoav Goldberg. 2016.  Assessing the ability of LSTMs to learn syntax-sensitive dependencies.  _Transactions of the Association for Computational Linguistics_ , 4:521–535.
  * Marvin and Linzen (2018) Rebecca Marvin and Tal Linzen. 2018.  Targeted syntactic evaluation of language models.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 1192–1202, Brussels, Belgium. Association for Computational Linguistics.
  * McCarthy et al. (2018) Arya D McCarthy, Miikka Silfverberg, Ryan Cotterell, Mans Hulden, and David Yarowsky. 2018.  Marrying universal dependencies and universal morphology.  In _Proceedings of the Second Workshop on Universal Dependencies (UDW 2018)_ , pages 91–101.
  * McCoy et al. (2018) R Thomas McCoy, Tal Linzen, and Robert Frank. 2018.  Revisiting the poverty of the stimulus: hierarchical generalization without a hierarchical bias in recurrent neural networks.  _arXiv preprint arXiv:1802.09091_.
  * Nie et al. (2017) Allen Nie, Erin D. Bennett, and Noah D. Goodman. 2017.  Dissent: Sentence representation learning from explicit discourse relations.  _CoRR_ , abs/1710.04334.
  * Nivre et al. (2016) Joakim Nivre, Marie-Catherine De Marneffe, Filip Ginter, Yoav Goldberg, Jan Hajic, Christopher D Manning, Ryan T McDonald, Slav Petrov, Sampo Pyysalo, Natalia Silveira, et al. 2016.  Universal dependencies v1: A multilingual treebank collection.  In _LREC_.
  * Preminger (2014) Omer Preminger. 2014.  _Agreement and its failures_.  Number 68 in Linguistic Inquiry Monographs. MIT Press, Cambridge, MA.
  * van Schijndel et al. (forthcoming) Marten van Schijndel, Aaron Mueller, and Tal Linzen. forthcoming.  Quantity doesn’t buy quality syntax with neural language models.
  * Subramanian et al. (2018) Sandeep Subramanian, Adam Trischler, Yoshua Bengio, and Christopher J Pal. 2018\.  Learning general purpose distributed sentence representations via large scale multi-task learning.  In _International Conference on Learning Representations_.
  * Sylak-Glassman (2016) John Sylak-Glassman. 2016.  The composition and use of the universal morphological feature schema (unimorph schema).  Technical report, Technical report, Johns Hopkins University.
  * Taylor (1953) Wilson L Taylor. 1953.  Cloze procedure: A new tool for measuring readability.  _Journalism Bulletin_ , 30(4):415–433.
  * Tenney et al. (2019) Ian Tenney, Dipanjan Das, and Ellie Pavlick. 2019.  BERT rediscovers the classical NLP pipeline.  _arXiv preprint arXiv:1905.05950_.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  In _Advances in neural information processing systems_ , pages 5998–6008.
  * Wieting et al. (2016) John Wieting, Mohit Bansal, Kevin Gimpel, and Karen Livescu. 2016.  Towards universal paraphrastic sentence embeddings.  In _International Conference on Learning Representations_.