# Lattice QCD at non-zero temperature and density

F. Karsch  Fakultät für Physik, Universität Bielefeld, D-33615 Bielefeld,
Germany

###### Abstract

Soon after the formulation of Quantum Chromodynamics in 1972 its
regularization on Euclidean space-time lattices had been introduced by Kenneth
Wilson [1]. This paved ground for numerical studies of non-perturbative
aspects of QCD, first shown by Michael Creutz [2]. Ever since these first
lattice QCD calculations the exploration of the QCD phase diagram and the
thermodynamics of strong-interaction matter at non-zero temperature and
density was pursued vigorously. In this brief review I try to highlight some
of the results on QCD thermodynamics obtain during the last 42 years through
lattice QCD calculations.

##

Article to appear in a special EPJC Volume in celebration of

’50 Years of Quantum Chromodynamics’

edited by Franz Gross and Eberhard Klempt

##  1 QCD thermodynamics on Euclidean Lattices

The path integral formulation of QCD can easily be applied to cases of non-
vanishing temperature ($T$) and other external control parameters, e.g. the
chemical potentials ($\mu_{f}$) that couple to the conserved currents of
quark-flavor number.

Using the lattice regularization scheme of QCD, introduced by K. Wilson [1],
QCD thermodynamics is formulated on Euclidean space-time lattices of size
$N_{\sigma}^{3}N_{\tau}$ where, for a given lattice spacing ($a$), the lattice
extent in Euclidean time controls the temperature $T=1/N_{\tau}a$ and the
spatial extent is related to the volume of the thermodynamic system,
$V=(N_{\sigma}a)^{3}$. The chemical potentials enter directly in the fermion
matrices, $M_{f}$, which arise from the QCD Lagrangian after integrating out
the fermion fields.

Bulk thermodynamics can then be derived from the lattice regularized partition
function,

$\displaystyle
Z=\int\prod_{x_{0}=1}^{N_{\tau}}\prod_{x_{i}=1}^{N_{\sigma}}\prod_{\hat{\nu=0}}^{3}\;\mathcal{D}U_{x,\hat{\nu}}\
e^{-S_{G}}\prod_{f=u,d,s..}\det M_{f}(m_{f},\mu_{f})~{},$ (1)

where $x=(x_{0},\vec{x})$ labels the sites of the 4-dimensional lattice,
$S_{G}$ denotes the gluonic part of the Euclidean action, which is expressed
in terms of $SU(3)$ matrices $U_{x,\hat{\nu}}$ and $M_{f}$ is the fermion
matrix for quark flavor $f$. It is a function of quark mass, $m_{f}$ and
flavor chemical potential $\hat{\mu}_{f}\equiv\mu_{f}/T$. Basic bulk
thermodynamic observables (equation of state, susceptibilities, etc.) can then
be obtained from the logarithm of the partition function, $Z$, which defines
the pressure, $P$, as

$P/T=\frac{1}{V}\ln Z(T,V,\vec{\mu},\vec{m})\;.$ (2)

Applying standard thermodynamic relations one obtains other observables of
interest; e.g. the energy density is related to the trace anomaly of the
energy-momentum tensor, $\Theta^{\mu\mu}$,

$\frac{\Theta^{\mu\mu}}{T^{4}}=\frac{\epsilon-3P}{T^{4}}\equiv T\frac{\partial
P/T^{4}}{\partial T}\;,$ (3)

and the conserved charge densities are obtained as,

$\frac{n_{X}}{T^{3}}=\frac{\partial
P/T^{4}}{\partial\hat{\mu}_{X}}\;\;,\;\;X=B,\ Q,\ S\;.$ (4)

While the framework of lattice QCD provides easy access to QCD thermodynamics
at vanishing values of the chemical potentials, major difficulties arise at
$\mu_{f}\neq 0$. The fermion determinants, ${\rm det}M_{f}(m_{f},\mu_{f})$,
are no longer positive definite when the real part of the chemical potential
is non-zero, ${\rm Re}\hat{\mu}_{f}\neq 0$. This includes the physically
relevant case of strictly real chemical potentials. The presence of a complex
valued integrand in the path integral makes the application of standard Monte
Carlo techniques, which rely on a probabilistic interpretation of integration
measures, impossible. The two most common approaches to circumvent this
problem are to either (i) perform numerical calculations at imaginary values
of the chemical potential, $\hat{\mu}_{f}^{2}<0$ [3, 4], or to (ii) perform
Taylor series expansions around $\hat{\mu}_{f}=0$ [5, 6]. In the former case
numerical results need to be analytically continued to real values of
$\mu_{f}$. In the latter case the QCD partition function is written as,

$P/T^{4}=\frac{1}{VT^{3}}\ln
Z(T,V,\vec{\mu})=\sum_{i,j,k=0}^{\infty}\frac{\chi_{ijk}^{BQS}}{i!j!k!}\hat{\mu}_{B}^{i}\hat{\mu}_{Q}^{j}\hat{\mu}_{S}^{k}\;,$
(5)

with $\chi_{000}^{BQS}\equiv P(T,V,\vec{0})/T^{4}$ and expansion coefficients,

$\chi_{ijk}^{BQS}(T)=\left.\frac{\partial
P/T^{4}}{\partial\hat{\mu}_{B}^{i}\partial\hat{\mu}_{Q}^{j}\partial\hat{\mu}_{S}^{k}}\right|_{\hat{\mu}=0}\;,$
(6)

can be determined in Monte Carlo simulations performed at $\hat{\mu}_{X}=0$.

The phase structure of QCD can be explored using suitable observables that are
sensitive to the spontaneous breaking and the eventual restoration of global
symmetries. They can act as order parameters in certain limits of the
parameter space spanned by the quark masses. In QCD exact symmetries exist
either in the chiral limit, i.e. at vanishing values of $n_{f}$ quark masses,
or for infinitely heavy quarks, i.e. in pure $SU(N_{c})$ gauge theories, with
$N_{c}$ denoting the number of colors.

In order to probe the restoration of the global chiral symmetries one analyzes
the chiral condensate and its susceptibilities,

$\displaystyle\langle\bar{\chi}\chi\rangle_{f}$ $\displaystyle=$
$\displaystyle\frac{T}{V}\frac{\partial}{\partial m_{f}}\ln
Z=\frac{T}{V}\langle{{\rm Tr}M_{f}^{-1}}\rangle\;,$ (7)
$\displaystyle\chi_{m}^{fg}$ $\displaystyle=$
$\displaystyle\frac{\partial\langle\bar{\chi}\chi\rangle_{f}}{\partial
m_{g}}\;,\;\quad\chi_{t}^{f}=T\frac{\partial\langle\bar{\chi}\chi\rangle_{f}}{\partial
T}\;.$ (8)

The former is an order parameter for the restoration of the
$SU(n_{f})_{L}\times SU(n_{f})_{R}$ chiral flavor symmetry of QCD and
distinguishes, in the limit of vanishing quark masses, a symmetry broken phase
at low temperature from a chiral symmetry restored phase at high temperature,

$\lim_{m_{\ell}\rightarrow
0}\langle\bar{\chi}\chi\rangle_{\ell}\begin{cases}>0&,\;\;T<T_{\chi}\\\
=0&\;,\;\;T\geq T_{\chi}\end{cases}\;.$ (9)

Similarly one considers the Polyakov loop $\langle L\rangle$ and its
susceptibility $\chi_{L}$,

$\displaystyle\langle L\rangle$ $\displaystyle=$
$\displaystyle\frac{1}{N_{\sigma}^{3}}\langle\sum_{\vec{x}}{\rm
Tr}L_{\vec{x}}\rangle\;\;,\;\;\quad
L_{\vec{x}}=\prod_{x_{0}=1}^{N_{\tau}}U_{(x_{0},\vec{x}),\hat{0}}\;,$
$\displaystyle\chi_{L}$ $\displaystyle=$ $\displaystyle
N_{\sigma}^{3}\left(\langle L^{2}\rangle-\langle L\rangle^{2}\right)\;,$ (10)

to probe the breaking and restoration of the global $Z(N_{c})$ center symmetry
of pure $SU(N_{c})$ gauge theories; i.e. $SU(N_{c})$ gauge theories at finite
temperature, formulated on Euclidean lattices, are invariant under global
rotation of all temporal gauge field variables, $U_{\vec{x},\hat{0}}\
\rightarrow\ zU_{\vec{x},\hat{0}}$, with $z\in Z(N_{c})$. The Polyakov loop
expectation value vanishes as long as this center symmetry is not
spontaneously broken.

The Polyakov loop expectation value also reflects the long distance behavior
of Polyakov loop correlation functions,

$\displaystyle\left|\langle L\rangle\right|^{2}$ $\displaystyle\equiv$
$\displaystyle\lim_{|\vec{x}|\rightarrow\infty}G_{L}(\vec{x})\begin{cases}=0\Leftrightarrow
F_{q}=\infty\;,&T\leq T_{d}\\\ >0\Leftrightarrow
F_{q}<\infty\;,&T>T_{d}\end{cases}$ (11)

where

$\displaystyle G_{L}(\vec{x})$ $\displaystyle=$ $\displaystyle{\rm
e}^{-F_{\bar{q}q}(\vec{x},T)}=\langle{\rm Tr}L_{\vec{0}}{\rm
Tr}L^{\dagger}_{\vec{0}}\rangle$ (12)

is the correlation function of two Polyakov loops. It denotes the change in
free energy (excess free energy, $F_{\bar{q}q}$), that is due to the presence
of two static quark sources introduced in a thermal medium. At zero
temperature this free energy reduces to the potential between static quark
sources.

At least in the case of pure gauge theories this provides a connection between
the confinement-deconfinement phase transition and the breaking of a global
symmetry, the $Z(N_{c})$ center symmetry of the $SU(N_{c})$ gauge group. This
symmetry, however, is explicitly broken in the presence of dynamical quarks
with mass $m_{f}<\infty$. Unlike chiral symmetry restoration, deconfinement
thus is not expected to be related to a phase transition in QCD with physical
quark masses. Nonetheless, the consequences of deconfinement, related to the
dissolution of hadronic bound states, becomes clearly visible in many
thermodynamic observables.

Figure 1: First evidence for the existence of a deconfinement phase transition
in $SU(2)$ gauge theories using the Polyakov loop expectation value as an
order parameter (left) [7] and a first extrapolation of the phase transition
temperature to the continuum limit (middle) [8] . The right hand figure shows
a first comparison of the temperature dependence of the Polyakov loop
($W\equiv\langle|L|\rangle$) and chiral condensate
($\langle\bar{\psi}\psi\rangle$) order parameters in a SU(3) gauge theory [9]
.

##  2 Early lattice QCD calculations at non-zero temperature

Almost immediately after the formulation of QCD as the theory of strong
interaction physics, its consequences for strong interaction matter at non-
zero temperature were examined [10, 11]. It rapidly became obvious that
fundamental properties of QCD, confinement and asymptotic freedom on the one
hand [11, 12], and chiral symmetry breaking on the other hand [13], are likely
to trigger a phase transition in strong interaction matter that separates a
phase being dominated by hadrons as the relevant degrees of freedom from that
of almost free quarks and gluons. The notion of a quark-gluon plasma was
coined at that time [14].

Soon after these early, conceptually important developments it was realized
that the formulation of QCD on discrete space-time lattices, which was
introduced by K. Wilson as a regularization scheme in QCD [1], also provides a
powerful framework for the analysis of non-perturbative properties of strong
interaction matter through Monte-Carlo simulations [2]. This led to a first
determination of a phase transition temperature in $SU(2)$ [7, 8] and $SU(3)$
[9, 15, 16] gauge theories, and a first determination of the equation of state
of purely gluonic matter [17, 18]. The interplay between deconfinement on the
one hand and chiral symmetry restoration on the other hand also was studied
[9] early on and the question whether or not these two aspects of QCD may lead
to two distinct phase transitions in QCD has been considered ever since. Some
results from these first lattice QCD studies of the thermodynamics of strong
interaction matter are shown in Fig. 1.

At physical values of the quark masses, neither deconfinement nor the
effective restoration of chiral symmetry leads to a true phase transition.
Still the transition from the low temperature hadronic to the high temperature
partonic phase of QCD is clearly visible in the pseudo-critical behavior of
the heavy quark free energy and the chiral condensate respectively. Some
recent results on these observables, obtained in simulations of QCD with
light, dynamical quark degrees of freedom, are shown in Figs. 3 and 3.

Figure 2: The so-called color averaged, heavy quark free energy ($F_{\rm
av}\equiv F_{\bar{q}q}$) in the vicinity of the pseudo-critical transition
temperature ($T_{pc}$) in 2-flavor QCD [19]. Results shown cover a temperature
range from $T/T_{pc}\simeq 0.75$ to $T/T_{pc}\simeq 2$.

Figure 3: Quark mass dependence of chiral order parameter, $M$, defined in Eq.
14 for QCD with two degenerate light quark masses and a strange quark mass
tuned to its physical value. Shown are results from calculations on lattices
with temporal extent $N_{\tau}=8$ performed for several values of the light
quark masses [20, 21]. The light quark masses, $m_{\ell}$, are expressed in
units of the strange quark mass, $H=m_{\ell}/m_{s}$. In the figure we give
$1/H=m_{s}/m_{\ell}$ together with the corresponding values of the Goldstone
pion mass.

##  3 Global symmetries and the QCD phase diagram

The early studies of QCD thermodynamics made it clear that universality
arguments and renormalization group techniques, successfully developed in
condensed matter physics and applied in statistical physics to the analysis of
phase transitions, also can be carried over to the analysis of the phase
structure of quantum field theories [22, 23]. The renormalization group based
arguments for the existence of a second order phase transition in the
universality class of the 3-d Ising model in a $SU(2)$ gauge theory, and a
first order transition for the $SU(3)$ color group of QCD [24] have been
confirmed by detailed lattice QCD calculations [25, 26].

In the presence of $n_{f}$ light, dynamical quarks, distinguished by a flavor
quantum number, it is the chiral symmetry of QCD that triggers the occurrence
of phase transitions [13]. In addition to a global $U(1)$ symmetry that
reflects the conservation of baryon number and is unbroken at all temperatures
and densities, the massless QCD Lagrangian is invariant under the symmetry
group

$U(1)_{A}\times SU(n_{f})_{L}\times SU(n_{f})_{R}\;.$ (13)

The $SU(n_{f})_{L}\times SU(n_{f})_{R}$ symmetry corresponds to chiral
rotations of $n_{f}$ massless quark fields in flavor space. This symmetry is
spontaneously broken at low temperatures, giving rise to $n_{f}^{2}-1$
massless Goldstone modes, which for $n_{f}=2$ are the three light pions of
QCD. They have a non-vanishing mass only because of the explicit breaking of
chiral symmetry by a mass term in the QCD Lagrangian that couples to the
chiral order parameter field $\bar{\chi}_{f}\chi_{f}$. The axial $U(1)_{A}$
group corresponds to global rotations of quark fields for a given flavor $f$.
Although it is an exact symmetry of the classical Lagrangian, it is explicitly
broken in the quantized theory. This explicit breaking of a global symmetry,
arising from fluctuations on the quantum level, is known as the $U(1)_{A}$
anomaly.

The renormalization group based analysis of the chiral phase transition,
performed by Pisarski and Wilczek [13], made it clear that the chiral phase
transition is sensitive to the number of light quark flavors that become
massless. Furthermore, it has been argued in [13] that the order of the
transition may be sensitive to the magnitude of the axial anomaly at non-zero
temperature, which is closely related to the temperature dependence of
topological non-trivial field configurations.

Although it was generally expected that the chiral phase transition in
3-flavor QCD becomes a first order phase transition in the chiral limit [13],
there is currently no direct evidence for this from lattice QCD calculations.
In fact, the current understanding is that the chiral phase transition is
second order for all $n_{f}\leq 6$ [27].

In Fig. 4 (left) we show the original version of the QCD phase diagram in the
plane of two degenerate light $(m_{\ell})$ and strange ($m_{s}$) quark masses,
proposed in 1990 [28], together with an updated version from 2021 [27]. Here
$m_{\ell}$ denotes the two degenerate up and down quark masses,
$m_{\ell}\equiv m_{u}=m_{d}$.

Figure 4: Sketch of the phase diagram of QCD in the plane of degenerate, light
up and down quark masses and a strange quark mass (Columbia plot). The figure
shows the original version from 1990 [28] (left) and an updated version from
2021 [27] (right).

This sketch of our current understanding of the 3-flavor phase diagram also is
supported by the increasing evidence for a non-singular crossover transition
in QCD with physical light and strange quark masses and the absence of any
evidence for a first order phase transition at lighter-than-physical values of
the light and strange quark masses [27, 29]. In the chiral limit, i.e. for
vanishing up and down quark masses111Lattice QCD studies of the (2+1)-flavor
phase diagram generally are performed with degenerate up and down quark
masses., a second order phase transition will then occur.

##  4 The chiral phase transition at vanishing chemical potential

The occurrence of the chiral phase transition is signaled by the vanishing of
the light quark chiral condensate. In order to remove multiplicative and
additive divergences in $\langle\bar{\chi}\chi\rangle_{\ell}$ one considers
instead the order parameter $M$ which is a combination of light and strange
quark condensates,

$\displaystyle M$ $\displaystyle=$ $\displaystyle
2\left(m_{s}\langle\bar{\psi}\psi\rangle_{\ell}-m_{\ell}\langle\bar{\psi}\psi\rangle_{s}\right)/f_{K}^{4}\;,$
(14)

and its derivative with respect to the light quark masses, i.e. the chiral
susceptibility $\chi_{M}$

$\displaystyle\chi_{M}$ $\displaystyle=$ $\displaystyle
m_{s}\left(\frac{\partial M}{\partial m_{u}}+\frac{\partial M}{\partial
m_{d}}\right)_{m_{u}=m_{d}\equiv m_{\ell}}\;.$ (15)

Here the kaon decay constant $f_{K}=156.1(9)/\sqrt{2}$ MeV, has been used to
introduce a dimensionless order parameter. The scaling behavior of $M$ and
$\chi_{M}$, have been used to characterize the chiral phase transition,

$\displaystyle M~{}~{}~{}~{}$ $\sim$$m_{\ell}\to 0$
$\displaystyle~{}~{}\begin{cases}A\left(\frac{T_{c}^{0}-T}{T_{c}^{0}}\right)^{\beta}&,\;\;T<T_{c}^{0}\\\
~{}~{}0&,\;\;T\geq T_{c}^{0}\end{cases}$ (16) $\displaystyle\chi_{M}~{}~{}~{}$
$\sim$$m_{\ell}\to 0$ $\displaystyle~{}~{}\begin{cases}~{}~{}\infty&,\;\;T\leq
T_{c}^{0}\\\
C\left(\frac{T-T_{c}^{0}}{T_{c}^{0}}\right)^{-\gamma}&,\;\;T>T_{c}^{0}\end{cases}\;$
(17)

where $\beta,\ \gamma$ are critical exponents.

We note that the low temperature behavior of the order parameter
susceptibility, $\chi_{M}$, is quite different from that known, for instance,
in the $3$-$d$ Ising model. The susceptibility diverges in the massless limit
at all values of the temperature, $T\leq T_{c}^{0}$. This is a consequence of
the breaking of a continuous rather than a discrete symmetry. The former gives
rise to Goldstone modes, the pions in QCD, which contribute to the chiral
condensate and as such to the order parameter $M$, i.e.,

$M\sim a(T)\sqrt{m_{\ell}}\;\;,\;\;T<T_{c}^{0}\;.$ (18)

As a consequence the chiral susceptibility diverges below $T_{c}^{0}$,
$\chi_{M}\sim 1/\sqrt{m_{\ell}}$, while at $T_{c}^{0}$ its divergence is
controlled by the critical exponent $\delta=1+\gamma/\beta$,

$\chi_{M}\sim\begin{cases}H^{-1/2}&\;,\;T<T_{\chi}\\\
H^{1/\delta-1}&\;,\;T=T_{\chi}\end{cases}\;,$ (19)

with $H=m_{\ell}/m_{s}$. As $1-1/\delta>1/2$ in all relevant universality
classes $\chi_{M}$ develops a pronounced peak at small, but non-zero values of
the quark masses,

$\chi_{M}^{peak}\equiv\chi_{M}(T_{pc}(H))\sim
H^{1/\delta-1}\;,\;H=m_{\ell}/m_{s}\;.$ (20)

The location of such a peak in either $\chi_{M}$ or similarly in $T\partial
M/\partial T$, defines pseudo-critical temperatures, $T_{pc}(H)$, which
converge to the unique chiral phase transition, $T_{c}^{0}$, at $H=0$.

Figure 5:  same as Fig. 3 but for the chiral susceptibility.

Some results on the quark mass dependence of $M$ and $\chi_{M}$ are shown in
Figs. 3 and 5, respectively. A scaling analysis of these observables,
performed in [20], led to the determination of the chiral phase transition
temperature [20],

$T_{c}^{0}=132^{+3}_{-6}\ {\rm MeV}\;.$ (21)

Similar results have also been obtained in [30] where a quite different
discretization scheme for the fermion sector of QCD has been used.

For physical light and strange quark masses, corresponding to $H\simeq 1/27$,
one finds as a pseudo-critical temperature [31],

$T_{pc}=156.5(1.5)\ {\rm MeV}\;,$ (22)

which is in good agreement with other determinations of pseudo-critical
temperatures in $(2+1)$-flavor QCD [32, 33, 34].

The chiral symmetry group $SU(2)_{L}\times SU(2)_{R}$ is isomorphic to the
rotation group $O(4)$. It thus is expected that the chiral phase transition
for two vanishing light quark masses is in the same universality class as
$3$-$d$, $O(4)$ symmetric spin models. In fact, the rapid rise of $\chi_{M}$,
shown in Fig. 5, is consistent with a critical exponent in this universality
class, $\delta=4.824$ [35]. However, a precise determination of this exponent
in 2-flavor QCD is not yet possible. This leaves open the possibility for
other symmetry breaking patterns and other universality classes playing a role
in the chiral limit of 2-flavor QCD [36]. In fact, the discussion of such
possibilities is closely related to the yet unsettled question concerning the
influence of the axial $U(1)_{A}$ symmetry on the chiral phase transition. For
a recent review on this question see, for instance [37].

Thermal masses and screening masses: The restoration of symmetries is
reflected also in the modification of the hadron spectrum at non-zero
temperature. Interactions in a thermal medium lead to modifications of
resonance peaks that can modify the location of maxima and the width of
spectral functions that control properties of hadron correlation functions.
This gives rise to so-called thermal masses as well as thermal screening
masses that control the long-distance behavior of hadron correlation functions
in Euclidean time and spatial directions, respectively.

A consequence of $U(1)_{A}$ breaking in the vacuum or at low temperature is
that masses of hadronic states that are related to each other through a
$U(1)_{A}$ transformation differ, while they become identical, or close to
each other, when the $U(1)_{A}$ symmetry is effectively restored. This is
easily seen to happen at high temperature. The crucial question, of relevance
for the QCD phase transition, however, is to which extent $U(1)_{A}$ symmetry
breaking is reduced, or already disappeared at the chiral phase transition
temperature. Settling this question requires the analysis of observables
sensitive to $U(1)_{A}$ breaking close to $T_{c}^{0}$ and for smaller-than-
physical light quark masses.

The calculation of in-medium modifications of hadron masses is difficult, but
has been attempted for quark masses close to their physical values [38].
Results for the temperature dependence of the mass-splitting of parity
partners in the baryon octet [38] are shown in Fig. 6. These results suggest a
strong temperature dependence of the negative parity states while the positive
parity partners are not sensitive to temperature changes. At $T_{pc}$ the
masses of parity partners are almost degenerate.

Figure 6: Temperature dependence of masses of parity partners in the baryon
octet [38].

Figure 7: Screening masses (left) and the related susceptibilities (right) of
scalar and pseudo-scalar mesons [39, 40].

More easily accessible are so-called screening masses, which also are obtained
from ordinary hadron correlation functions and can be analyzed close to the
chiral limit. Rather than analyzing the long-distance behavior of hadron
correlation functions in Euclidean time, one extracts a so-called screening
mass from the long-distance behavior in one of the spatial directions [41,
42]. Finite temperature meson screening correlators, projected onto lowest
Matsubara frequency of a bosonic state, $p_{0}\equiv\omega_{0}=0$, and zero
transverse momentum, ${\bf p}_{\perp}\equiv(p_{x},p_{y})=0$, are defined by

$\displaystyle G_{\Gamma}(z,T)$ $\displaystyle=$
$\displaystyle\int_{0}^{\beta}d\tau\int dxdy\,\Big{\langle}{\cal
M}_{\Gamma}(\vec{r},\tau)\overline{{\cal M}_{\Gamma}}(\vec{0},0)\Big{\rangle}$
(23) $\sim$$z\to\infty$ $\displaystyle\hskip 5.69046pt{\rm
e}^{-m_{\Gamma}(T)z}\;\;,\;\;\vec{r}\equiv(x,y,z)\;\;,$

where ${\cal M}_{\Gamma}\equiv\bar{\psi}\Gamma\psi$ is a meson operator that
projects onto a quantum number channel that is selected through an appropriate
choice of $\Gamma$-matrices [39, 41]. At large distances this permits the
extraction of the screening mass, $m_{\Gamma}$, in the quantum number channel
selected by $\Gamma$ from the exponential fall-off of these correlation
functions. In Fig. 7 (left) we show results for the scalar and pseudo-scalar
screening masses obtained in $(2+1)$-flavor QCD calculations for different
values of the light to strange quark mass ratio. The integrated correlation
functions define susceptibilities in these quantum number channels, which also
should be degenerate, if $U(1)_{A}$ is effectively restored. Both observables
seem to suggest that there remains a significant remnant of $U(1)_{A}$
breaking at the chiral phase transition temperature, $T_{c}^{0}$, which
however reduces quickly above the chiral transition and gives rise to an
effective restoration of $U(1)_{A}$ at $T\simeq 1.1T_{c}^{0}$.

In the region $T>T_{c}^{0}$ the difference between pseudo-scalar and scalar
susceptibilities is related to the so-called disconnected part, $\chi_{dis}$,
of the chiral susceptibility, $\chi_{M}=\chi_{dis}+\chi_{con}$, with

$\displaystyle\chi_{dis}$ $\displaystyle=$
$\displaystyle\frac{1}{4N_{\tau}N_{\sigma}^{3}}\left(\langle({\rm
Tr}M_{\ell}^{-1})^{2}\rangle-\langle{\rm
Tr}M_{\ell}^{-1}\rangle^{2}\right)\;,$ (24) $\displaystyle\chi_{con}$
$\displaystyle=$ $\displaystyle\frac{1}{2N_{\tau}N_{\sigma}^{3}}\langle{\rm
Tr}M_{\ell}^{-2}\rangle\;.$ (25)

Figure 8: Eigenvalue density of the overlap fermion matrix obtained in
calculations with dynamical overlap fermions [43].

The disconnected chiral susceptibility can be expressed by an integral over
the eigenvalue density, $\rho(\lambda)$, of the fermion matrix $M_{f}$,

$\chi_{dis}=\int_{0}^{\infty}d\lambda\,\rho(\lambda)\frac{2m_{\ell}^{2}}{(\lambda^{2}+m_{\ell}^{2})^{2}}\;.$
(26)

In the chiral symmetric phase the density of vanishing eigenvalues, $\rho(0)$,
vanishes. In order for $\chi_{dis}$ to be nonetheless non-zero in the chiral
limit, the density of near-zero eigenvalues needs to converge to a non-
vanishing value ($\delta$-function) at $\lambda=0$ in the limit
$m_{\ell}\rightarrow 0$ and $V\rightarrow\infty$. Controlling the various
limits involved and also taking into account that the pseudo-critical
transition temperature, $T_{pc}(H)$, has a sizeable quark mass dependence is
difficult. Nonetheless, studies of the temperature dependence of the
eigenvalue density of the Dirac matrix are crucial for a detailed
understanding of the influence of the $U(1)_{A}$ anomaly on the QCD phase
transition. Not surprisingly, it turns out that at non-zero values of the
lattice spacing the spectrum of low lying eigenvalues is quite sensitive to
the fermion discretization scheme. Using fermions with good chirality even at
non-zero lattice spacing seems to be advantageous, although after having
performed the extrapolation to the chiral limit, they should lead to results
identical with those obtained, e.g. within the staggered discretization
scheme. Current results are ambiguous. We show in Fig. 8 results from a
calculation of eigenvalue distributions obtained from calculations with
dynamical overlap fermions [43, 44]. These calculations provide evidence for a
large density of near-zero eigenvalues and a non-zero eigenvalue density,
possibly building up at $\lambda=0$. This is in contrast to calculations
performed with domain wall fermions [45] as well as so-called partially
quenched calculations that use the overlap fermion operator to calculate
eigenvalue distributions on gauge field configurations generated with
dynamical staggered fermions [46]. Obviously this subtle aspect of the chiral
phase transition is not yet resolved and the analysis of $U(1)_{A}$
restoration will remain to be a central topic in finite temperature QCD in the
years to come.

##  5 The chiral phase transition at non-vanishing chemical potential

In the studies of QCD at non-vanishing baryon chemical potential the search
for the existence of a second order phase transition at physical values of the
quark masses, the critical end point (CEP), finds particular attention. It
separates the crossover regime at small values of the chemical potential from
a region of first order phase transitions, which is predicted in many
phenomenological models to exist at high density. The CEP is searched for
extensively in heavy ion experiments and, if confirmed, would provide a solid
prediction for the existence of first order phase transitions in dense stellar
matter, e.g. in neutron stars.

Figure 9: Sketch of a possible QCD phase diagram in the space of temperature
($T$), baryon chemical potential ($\mu_{B}$) and light quark masses
($m_{u,d}$).

The dependence of the transition temperature on the chemical potentials, e.g.
$T_{pc}(\mu_{B})$, can be deduced from the $\mu_{B}$-dependent shift of the
peak in the chiral susceptibility. At non-vanishing values of the baryon
chemical potential, $\mu_{B}$, the QCD phase transition temperature in the
chiral limit as well as the region of pseudo-critical behavior in QCD with its
physical quark mass values shifts to smaller values of the temperature. This
shift has been determined in calculations with imaginary values of the
chemical potentials as well as from Taylor series expansions of the order
parameter $M$ and its susceptibility $\chi_{M}$. Using a Taylor series ansatz
for $T_{pc}(\mu_{B})$,

$T_{c}(\mu_{B})=T_{c}^{0}\left(1-\kappa_{2}^{B}\left(\frac{\mu_{B}}{T_{c}^{0}}\right)^{2}-\kappa_{4}^{B}\left(\frac{\mu_{B}}{T_{c}^{0}}\right)^{4}\right)$
(27)

one finds for the curvature coefficients $\kappa_{2}^{B}\simeq 0.012$ while
the next correction is consistent with zero in all current studies, e.g.
$\kappa_{4}^{B}=0.00032(67)$ [34]. The pseudo-critical temperature $T_{pc}$ at
physical values of the light and strange quark masses thus drops to about 150
MeV at $\mu_{B}\simeq 2T_{pc}$. This is still considerably larger than the
chiral phase transition temperature, $T_{c}^{0}$, determined at $\mu_{B}=0$.
As various model calculations [47, 48] suggest that the CEP at non-zero
$\mu_{B}$ is located at a temperature below $T_{c}^{0}$ one thus needs to get
access to thermodynamics at large chemical potentials. Assuming that the
curvature of the pseudo-critical line does not change drastically at large
values of the chemical potentials, our current understanding of the QCD phase
diagram in the $m_{\ell}$-$T$-$\mu_{B}$ space (see Fig. 9) suggests that a
possible CEP in the phase diagram may exist only at a temperature,

$T^{CEP}(\mu_{B}^{CEP})<130~{}{\rm MeV}\;,\;\mu_{B}^{CEP}>400~{}{\rm MeV}\;.$
(28)

Reaching the region $\mu_{B}/T>3$ is a major challenge for any of the
currently used approaches in lattice QCD calculations as well as for collider
based heavy ion experiments that search for the CEP.

##  6 Equation of state of strongly interacting matter

The equation of state (EoS) of strongly interacting matter, i.e. the pressure
and its derivatives with respect to temperature and chemical potentials
provides the basic information on the phase structure of QCD. It is of central
importance not only for the analysis of critical behavior in QCD but also for
the analysis of experimental results on strong interaction thermodynamics that
are obtained in relativistic heavy ion collision experiments.

Figure 10: Left: Pressure, energy and entropy densities in (2+1)-flavor QCD at
vanishing chemical potential. The figure is taken from [49]. Also shown in the
figure are results obtained with the stout discretization scheme for staggered
fermions [50]. Right: The speed of sound as function of energy density.

At vanishing values of the chemical potentials the QCD EoS is well controlled
and consistent results for pressure, energy and entropy densities, as well as
derived observables such as the speed of sound or specific heat, have been
obtained by several groups [50, 49]. We show results for some of these
observables in Fig. 10. The figure on the right shows the square of the speed
of sound, $c_{s}^{2}$, as function of the energy density. It can be seen that
$c_{s}^{2}$ has a minimum in the transition region, sometimes called the
softest point of the QCD EoS [51]. The energy density in the vicinity of the
pseudo-critical temperature ($T_{pc}\simeq 155$ MeV) is found to be,

$\epsilon_{\rm c}\simeq(350\pm 150)~{}{\rm MeV}/{\rm fm^{3}}\;,$ (29)

which is compatible with the energy density of the nucleon, $m_{N}/(4\pi
r_{N}^{3}/3)$ for nucleon radii in the range $r_{N}=(0.8-1)$ fm. Also shown in
the top figure is the trace of the energy-momentum tensor,
$(\epsilon-3P)/T^{4}$. Its deviation from zero gives some hint to the
relevance of interactions in the medium (for an ideal gas as well as to
leading order in high temperature perturbation theory one has $\epsilon=3P$).
Not unexpected this is largest close to the transition region and decreases
only slowly in the high temperature regime. This large deviations from ideal
gas or perturbative behavior is seen in many observables at temperature
$T_{pc}<T<2T_{pc}$.

Figure 11: $\mu_{B}$-dependent contribution to the pressure (left) and net
baryon number density (right) in (2+1)-flavor QCD at several values of the
baryon chemical potential chemical potential, $\mu/T_{B}=1.0$, 1.5, 2.0, 2.5,
(bottom to top) and for $\hat{\mu}_{B}=2.0$. Shown are results from Taylor
expansion up to eighth order in $\hat{\mu}_{B}$ in the pressure series for
isospin symmetric ($\mu_{Q}=0$) strangeness neutral ($n_{S}=0$) matter and
corresponding Padé approximants obtained from these Taylor expansion
coefficients. The figures are taken from [52].

Calculations of the equation of state as a function of $T$ and $\mu_{B}$ have
been performed using direct simulations at imaginary chemical potentials,
which then get analytically continued to real values of the chemical
potentials [53], as well as calculations using up to eighth order Taylor
expansions in $\mu_{B}$ [52]. Results of such calculations agree well for
$\mu_{B}/T\leq(2-2.5)$. In Fig. 11 we show results for the $\mu_{B}$-dependent
contribution to the pressure and net baryon number density. Comparing Fig. 11
(left) with Fig. 10 (left) shows that at $\mu_{B}/T\simeq 2$ and $T\simeq
T_{pc}$ the pressure increases by about 30%, which is due to the increase in
number of baryons in the medium.

At larger values of the baryon chemical potential the Taylor series will not
convergence due to the presence of either poles in the complex $\mu_{B}$-plane
or a real pole, that may correspond to the searched for CEP. The occurrence of
poles in the complex plane also generates problems for the analytic
continuation of results obtained in simulations at imaginary values of
$\mu_{B}$ as a suitable ansatz for the continuation needs to be found. Many
approaches to improve over straightforward Taylor series approaches or
simulations at imaginary chemical potential are currently being discussed [54,
55, 56, 57]. In the context of Taylor expansions a natural way to proceed is
to use Padé approximants, which provide a resummation of the Taylor series and
reproduce this series, when expanded for small $\mu_{B}$ [58, 52]. Results
from [4,4] and [3,4] Padé approximants for the pressure and number density
series, respectively, are also shown in Fig. 11. The good agreement with the
Taylor series for $\mu_{B}/T\leq 2.5$ gives confidence in the validity of the
Taylor series results and once more seems to rule out the occurrence of a CEP
in this parameter range.

##  7 Outlook

Achieving better control over the influence of the axial anomaly on the QCD
phase transition in the chiral limit at vanishing chemical potentials and
getting better control over the structure of the QCD phase diagram at large
non-zero values of the chemical potentials certainly are the two largest
challenges in studies of QCD thermodynamics for the next decade.

## Acknowledgements

This work was supported by the DFG Collaborative Research Centre 315477589-TRR
211, ”Strong interaction matter under extreme conditions”.

## References

  * [1] K. G. Wilson, Confinement of Quarks, Phys. Rev. D 10 (1974) 2445–2459.  doi:10.1103/PhysRevD.10.2445.
  * [2] M. Creutz, Monte Carlo Study of Quantized SU(2) Gauge Theory, Phys. Rev. D 21 (1980) 2308–2315.  doi:10.1103/PhysRevD.21.2308.
  * [3] M. D’Elia, M.-P. Lombardo, Finite density QCD via imaginary chemical potential, Phys. Rev. D 67 (2003) 014505.  arXiv:hep-lat/0209146, doi:10.1103/PhysRevD.67.014505.
  * [4] P. de Forcrand, O. Philipsen, The QCD phase diagram for small densities from imaginary chemical potential, Nucl. Phys. B 642 (2002) 290–306.  arXiv:hep-lat/0205016, doi:10.1016/S0550-3213(02)00626-0.
  * [5] R. V. Gavai, S. Gupta, Quark number susceptibilities, strangeness and dynamical confinement, Phys. Rev. D 64 (2001) 074506.  arXiv:hep-lat/0103013, doi:10.1103/PhysRevD.64.074506.
  * [6] C. R. Allton, S. Ejiri, S. J. Hands, O. Kaczmarek, F. Karsch, E. Laermann, C. Schmidt, L. Scorzato, The QCD thermal phase transition in the presence of a small chemical potential, Phys. Rev. D 66 (2002) 074507.  arXiv:hep-lat/0204010, doi:10.1103/PhysRevD.66.074507.
  * [7] L. D. McLerran, B. Svetitsky, A Monte Carlo Study of SU(2) Yang-Mills Theory at Finite Temperature, Phys. Lett. B 98 (1981) 195–198.  doi:10.1016/0370-2693(81)90986-2.
  * [8] J. Kuti, J. Polonyi, K. Szlachanyi, Monte Carlo Study of SU(2) Gauge Theory at Finite Temperature, Phys. Lett. B 98 (1981) 199–204.  doi:10.1016/0370-2693(81)90987-4.
  * [9] J. B. Kogut, M. Stone, H. W. Wyld, W. R. Gibbs, J. Shigemitsu, S. H. Shenker, D. K. Sinclair, Deconfinement and Chiral Symmetry Restoration at Finite Temperatures in SU(2) and SU(3) Gauge Theories, Phys. Rev. Lett. 50 (1983) 393\.  doi:10.1103/PhysRevLett.50.393.
  * [10] J. C. Collins, M. J. Perry, Superdense Matter: Neutrons Or Asymptotically Free Quarks?, Phys. Rev. Lett. 34 (1975) 1353.  doi:10.1103/PhysRevLett.34.1353.
  * [11] N. Cabibbo, G. Parisi, Exponential Hadronic Spectrum and Quark Liberation, Phys. Lett. B 59 (1975) 67–69.  doi:10.1016/0370-2693(75)90158-6.
  * [12] G. Baym, Confinement of quarks in nuclear matter, Physica A 96 (1-2) (1979) 131–135.  doi:10.1016/0378-4371(79)90200-0.
  * [13] R. D. Pisarski, F. Wilczek, Remarks on the Chiral Phase Transition in Chromodynamics, Phys. Rev. D 29 (1984) 338–341.  doi:10.1103/PhysRevD.29.338.
  * [14] E. V. Shuryak, Quark-Gluon Plasma and Hadronic Production of Leptons, Photons and Psions, Phys. Lett. B 78 (1978) 150.  doi:10.1016/0370-2693(78)90370-2.
  * [15] K. Kajantie, C. Montonen, E. Pietarinen, Phase Transition of SU(3) Gauge Theory at Finite Temperature, Z. Phys. C 9 (1981) 253.  doi:10.1007/BF01410665.
  * [16] L. Yaffe, B. Svetitsky, First Order Phase Transition in the SU(3) Gauge Theory at Finite Temperature, Phys. Rev. D 26 (1982) 963.  doi:10.1103/PhysRevD.26.963.
  * [17] J. Engels, F. Karsch, H. Satz, I. Montvay, High Temperature SU(2) Gluon Matter on the Lattice, Phys. Lett. B 101 (1981) 89.  doi:10.1016/0370-2693(81)90497-4.
  * [18] J. Engels, F. Karsch, H. Satz, I. Montvay, Gauge Field Thermodynamics for the SU(2) Yang-Mills System, Nucl. Phys. B 205 (1982) 545–577.  doi:10.1016/0550-3213(82)90077-3.
  * [19] O. Kaczmarek, F. Zantow, Static quark anti-quark interactions in zero and finite temperature QCD. I. Heavy quark free energies, running coupling and quarkonium binding, Phys. Rev. D 71 (2005) 114510.  arXiv:hep-lat/0503017, doi:10.1103/PhysRevD.71.114510.
  * [20] H. T. Ding, et al., Chiral Phase Transition Temperature in ( 2+1 )-Flavor QCD, Phys. Rev. Lett. 123 (6) (2019) 062002.  arXiv:1903.04801, doi:10.1103/PhysRevLett.123.062002.
  * [21] O. Kaczmarek, F. Karsch, A. Lahiri, C. Schmidt, Universal scaling properties of QCD close to the chiral limit, Acta Phys. Polon. Supp. 14 (2021) 291.  arXiv:2010.15593, doi:10.5506/APHYSPOLBSUPP.14.291.
  * [22] K. Rajagopal, F. Wilczek, Static and dynamic critical phenomena at a second order QCD phase transition, Nucl. Phys. B 399 (1993) 395–425.  arXiv:hep-ph/9210253, doi:10.1016/0550-3213(93)90502-G.
  * [23] K. Rajagopal, F. Wilczek, The Condensed matter physics of QCD, published in: At the frontier of particle physics. Handbook of QCD. Vol. 1-3 (Edts. M.Shifman and B. Ioffe), 2000, pp. 2061–2151.  arXiv:hep-ph/0011333, doi:10.1142/9789812810458_0043.
  * [24] B. Svetitsky, L. G. Yaffe, Critical Behavior at Finite Temperature Confinement Transitions, Nucl. Phys. B 210 (1982) 423–447.  doi:10.1016/0550-3213(82)90172-9.
  * [25] F. R. Brown, N. H. Christ, Y. F. Deng, M. S. Gao, T. J. Woch, Nature of the Deconfining Phase Transition in SU(3) Lattice Gauge Theory, Phys. Rev. Lett. 61 (1988) 2058.  doi:10.1103/PhysRevLett.61.2058.
  * [26] J. Engels, J. Fingberg, M. Weber, Finite Size Scaling Analysis of SU(2) Lattice Gauge Theory in (3+1)-dimensions, Nucl. Phys. B 332 (1990) 737–759.  doi:10.1016/0550-3213(90)90010-B.
  * [27] F. Cuteri, O. Philipsen, A. Sciarra, On the order of the QCD chiral phase transition for different numbers of quark flavours, JHEP 11 (2021) 141.  arXiv:2107.12739, doi:10.1007/JHEP11(2021)141.
  * [28] F. R. Brown, F. P. Butler, H. Chen, N. H. Christ, Z.-h. Dong, W. Schaffer, L. I. Unger, A. Vaccarino, On the existence of a phase transition for QCD with three light quarks, Phys. Rev. Lett. 65 (1990) 2491–2494.  doi:10.1103/PhysRevLett.65.2491.
  * [29] L. Dini, P. Hegde, F. Karsch, A. Lahiri, C. Schmidt, S. Sharma, Chiral phase transition in three-flavor QCD from lattice QCD, Phys. Rev. D 105 (3) (2022) 034510\.  arXiv:2111.12599, doi:10.1103/PhysRevD.105.034510.
  * [30] A. Y. Kotov, M. P. Lombardo, A. Trunin, QCD transition at the physical point, and its scaling window from twisted mass Wilson fermions, Phys. Lett. B 823 (2021) 136749.  arXiv:2105.09842, doi:10.1016/j.physletb.2021.136749.
  * [31] A. Bazavov, et al., Chiral crossover in QCD at zero and non-zero chemical potentials, Phys. Lett. B 795 (2019) 15–21.  arXiv:1812.08235, doi:10.1016/j.physletb.2019.05.013.
  * [32] C. Bonati, M. D’Elia, M. Mariti, M. Mesiti, F. Negro, F. Sanfilippo, Curvature of the chiral pseudocritical line in QCD: Continuum extrapolated results, Phys. Rev. D 92 (5) (2015) 054503.  arXiv:1507.03571, doi:10.1103/PhysRevD.92.054503.
  * [33] C. Bonati, M. D’Elia, F. Negro, F. Sanfilippo, K. Zambello, Curvature of the pseudocritical line in QCD: Taylor expansion matches analytic continuation, Phys. Rev. D 98 (5) (2018) 054510.  arXiv:1805.02960, doi:10.1103/PhysRevD.98.054510.
  * [34] S. Borsanyi, Z. Fodor, J. N. Guenther, R. Kara, S. D. Katz, P. Parotto, A. Pasztor, C. Ratti, K. K. Szabo, QCD Crossover at Finite Chemical Potential from Lattice Simulations, Phys. Rev. Lett. 125 (5) (2020) 052001.  arXiv:2002.02821, doi:10.1103/PhysRevLett.125.052001.
  * [35] R. Guida, J. Zinn-Justin, Critical exponents of the N vector model, J. Phys. A 31 (1998) 8103–8121.  arXiv:cond-mat/9803240, doi:10.1088/0305-4470/31/40/006.
  * [36] A. Pelissetto, E. Vicari, Relevance of the axial anomaly at the finite-temperature chiral transition in QCD, Phys. Rev. D 88 (10) (2013) 105018\.  arXiv:1309.5446, doi:10.1103/PhysRevD.88.105018.
  * [37] A. Lahiri, Aspects of finite temperature QCD towards the chiral limit, PoS LATTICE2021 (2022) 003.  arXiv:2112.08164, doi:10.22323/1.396.0003.
  * [38] G. Aarts, C. Allton, D. De Boni, B. Jäger, Hyperons in thermal QCD: A lattice view, Phys. Rev. D 99 (7) (2019) 074503.  arXiv:1812.07393, doi:10.1103/PhysRevD.99.074503.
  * [39] A. Bazavov, et al., Meson screening masses in (2+1)-flavor QCD, Phys. Rev. D 100 (9) (2019) 094510.  arXiv:1908.09552, doi:10.1103/PhysRevD.100.094510.
  * [40] S. Dentinger, O. Kaczmarek, A. Lahiri, Screening masses towards chiral limit, Acta Phys. Polon. Supp. 14 (2021) 321.  arXiv:2102.09916, doi:10.5506/APHYSPOLBSUPP.14.321.
  * [41] C. E. Detar, J. B. Kogut, The Hadronic Spectrum of the Quark Plasma, Phys. Rev. Lett. 59 (1987) 399.  doi:10.1103/PhysRevLett.59.399.
  * [42] C. E. Detar, J. B. Kogut, Measuring the Hadronic Spectrum of the Quark Plasma, Phys. Rev. D 36 (1987) 2828.  doi:10.1103/PhysRevD.36.2828.
  * [43] G. Cossu, S. Aoki, H. Fukaya, S. Hashimoto, T. Kaneko, H. Matsufuru, J.-I. Noaki, Finite temperature study of the axial U(1) symmetry on the lattice with overlap fermion formulation, Phys. Rev. D 87 (11) (2013) 114514, [Erratum: Phys.Rev.D 88, 019901 (2013)].  arXiv:1304.6145, doi:10.1103/PhysRevD.87.114514.
  * [44] A. Tomiya, G. Cossu, S. Aoki, H. Fukaya, S. Hashimoto, T. Kaneko, J. Noaki, Evidence of effective axial U(1) symmetry restoration at high temperature QCD, Phys. Rev. D 96 (3) (2017) 034509, [Addendum: Phys.Rev.D 96, 079902 (2017)].  arXiv:1612.01908, doi:10.1103/PhysRevD.96.034509.
  * [45] M. I. Buchoff, et al., QCD chiral transition, U(1)A symmetry and the dirac spectrum using domain wall fermions, Phys. Rev. D 89 (5) (2014) 054514.  arXiv:1309.4149, doi:10.1103/PhysRevD.89.054514.
  * [46] V. Dick, F. Karsch, E. Laermann, S. Mukherjee, S. Sharma, Microscopic origin of $U_{A}(1)$ symmetry violation in the high temperature phase of QCD, Phys. Rev. D 91 (9) (2015) 094504.  arXiv:1502.06190, doi:10.1103/PhysRevD.91.094504.
  * [47] A. M. Halasz, A. D. Jackson, R. E. Shrock, M. A. Stephanov, J. J. M. Verbaarschot, On the phase diagram of QCD, Phys. Rev. D 58 (1998) 096007.  arXiv:hep-ph/9804290, doi:10.1103/PhysRevD.58.096007.
  * [48] M. Buballa, S. Carignano, Inhomogeneous chiral phases away from the chiral limit, Phys. Lett. B 791 (2019) 361–366.  arXiv:1809.10066, doi:10.1016/j.physletb.2019.02.045.
  * [49] A. Bazavov, et al., Equation of state in ( 2+1 )-flavor QCD, Phys. Rev. D 90 (2014) 094503.  arXiv:1407.6387, doi:10.1103/PhysRevD.90.094503.
  * [50] S. Borsanyi, Z. Fodor, C. Hoelbling, S. D. Katz, S. Krieg, K. K. Szabo, Full result for the QCD equation of state with 2+1 flavors, Phys. Lett. B 730 (2014) 99–104.  arXiv:1309.5258, doi:10.1016/j.physletb.2014.01.007.
  * [51] C. M. Hung, E. V. Shuryak, Hydrodynamics near the QCD phase transition: Looking for the longest lived fireball, Phys. Rev. Lett. 75 (1995) 4003–4006.  arXiv:hep-ph/9412360, doi:10.1103/PhysRevLett.75.4003.
  * [52] D. Bollweg, J. Goswami, O. Kaczmarek, F. Karsch, S. Mukherjee, P. Petreczky, C. Schmidt, P. Scior, Taylor expansions and Padé approximants for cumulants of conserved charge fluctuations at nonvanishing chemical potentials, Phys. Rev. D 105 (7) (2022) 074511.  arXiv:2202.09184, doi:10.1103/PhysRevD.105.074511.
  * [53] S. Borsanyi, G. Endrodi, Z. Fodor, S. D. Katz, S. Krieg, C. Ratti, K. K. Szabo, QCD equation of state at nonzero chemical potential: continuum results with physical quark masses at order $mu^{2}$, JHEP 08 (2012) 053.  arXiv:1204.6710, doi:10.1007/JHEP08(2012)053.
  * [54] S. Mukherjee, V. Skokov, Universality driven analytic structure of the QCD crossover: radius of convergence in the baryon chemical potential, Phys. Rev. D 103 (7) (2021) L071501.  arXiv:1909.04639, doi:10.1103/PhysRevD.103.L071501.
  * [55] S. Mondal, S. Mukherjee, P. Hegde, Lattice QCD Equation of State for Nonvanishing Chemical Potential by Resumming Taylor Expansions, Phys. Rev. Lett. 128 (2) (2022) 022001.  arXiv:2106.03165, doi:10.1103/PhysRevLett.128.022001.
  * [56] P. Dimopoulos, L. Dini, F. Di Renzo, J. Goswami, G. Nicotra, C. Schmidt, S. Singh, K. Zambello, F. Ziesché, Contribution to understanding the phase structure of strong interaction matter: Lee-Yang edge singularities from lattice QCD, Phys. Rev. D 105 (3) (2022) 034513.  arXiv:2110.15933, doi:10.1103/PhysRevD.105.034513.
  * [57] S. Borsanyi, Z. Fodor, J. N. Guenther, R. Kara, P. Parotto, A. Pasztor, C. Ratti, K. K. Szabo, Resummed lattice QCD equation of state at finite baryon density: Strangeness neutrality and beyond, Phys. Rev. D 105 (11) (2022) 114504.  arXiv:2202.05574, doi:10.1103/PhysRevD.105.114504.
  * [58] S. Datta, R. V. Gavai, S. Gupta, Quark number susceptibilities and equation of state at finite chemical potential in staggered QCD with Nt=8, Phys. Rev. D 95 (5) (2017) 054512.  arXiv:1612.06673, doi:10.1103/PhysRevD.95.054512.


# LayoutMask: Enhance Text-Layout Interaction in Multi-modal Pre-training for
Document Understanding

Yi Tu, Ya Guo, Huan Chen, Jinyang Tang
Ant Group, China
<EMAIL_ADDRESS>
###### Abstract

Visually-rich Document Understanding (VrDU) has attracted much research
attention over the past years. Pre-trained models on a large number of
document images with transformer-based backbones have led to significant
performance gains in this field. The major challenge is how to fusion the
different modalities (text, layout, and image) of the documents in a unified
model with different pre-training tasks. This paper focuses on improving text-
layout interactions and proposes a novel multi-modal pre-training model,
LayoutMask. LayoutMask uses local 1D position, instead of global 1D position,
as layout input and has two pre-training objectives: (1) Masked Language
Modeling: predicting masked tokens with two novel masking strategies; (2)
Masked Position Modeling: predicting masked 2D positions to improve layout
representation learning. LayoutMask can enhance the interactions between text
and layout modalities in a unified model and produce adaptive and robust
multi-modal representations for downstream tasks. Experimental results show
that our proposed method can achieve state-of-the-art results on a wide
variety of VrDU problems, including form understanding, receipt understanding,
and document image classification.

##  1 Introduction

Figure 1: A receipt image from SROIE dataset and the global/local 1D positions
of tokens based on global/in-segment reading orders. Local 1D positions
restart with “1” for each individual segment. Blue Arrow: When using global 1D
position, the reading order is explicitly implied by the ascending numbers, so
the word after “Qty” is “Price”. Red Arrows: When using local 1D position, the
successor of “Qty” is not directly given and can have more possible choices,
so their semantic relations and 2D positions will be considered during pre-
training.

Visually-rich Document Understanding (VrDU) is an important research area that
aims to understand various types of documents (e.g., forms, receipts, and
posters), and it has attracted much attention from both academia and industry.
In recent years, pre-training techniques (Devlin et al., 2019; Zhang et al.,
2019) have been introduced into this area and self-supervised pre-training
multi-modal models have demonstrated great successes in various VrDU tasks (Xu
et al., 2020, 2021; Hong et al., 2022; Li et al., 2021a).

However, existing document pre-training models suffer from reading order
issues. Following the idea of BERT (Devlin et al., 2019), these methods (Xu et
al., 2020, 2021; Hong et al., 2022) usually adopt ascending numbers (e.g., 0,
1, 2,.., 511) to represent the global reading order of tokens in the document.
Then, these numbers are encoded into 1D position embeddings to provide
explicit reading order supervision during pre-training, which are called
“global 1D position”. While such global 1D positions are widely used in NLP
models for textual data, it is not a good choice for document data. Firstly,
plain texts always have definite and linear reading orders, but the reading
order of a document may not be unique or even linear, which cannot be simply
encoded with monotonically increasing numbers. Secondly, the global reading
order of a document is usually obtained by ordering detected text segments
from OCR tools with empirical rules, so it heavily relies on stable and
consistent OCR results, affecting the generalization ability in real-world
applications. Moreover, the empirical rules to obtain reading orders (e.g.,
“top-down and left-right”) may not be able to handle documents with complex
layouts, thus providing inaccurate supervision.

Some previous studies have attempted to solve the above reading order issues.
LayoutReader (Wang et al., 2021) proposes a sequence-to-sequence framework for
reading order detection with supervised reading order annotations. XYLayoutLM
(Gu et al., 2022) utilizes an augmented XY Cut algorithm to generate different
proper reading orders during pre-training to increase generalization ability.
ERNIE-Layout (Peng et al., 2022) rearranges the order of input tokens in
serialization modules and adopts a reading order prediction task in pre-
training. While these studies propose data-based or rule-based solutions to
provide explicit reading order supervision, we believe that the self-
supervised pre-training process on a large number of documents without using
extra supervision is sufficient to help the model to learn reading order
knowledge, and such knowledge can be implicitly encoded into the pre-trained
model with better adaptiveness and robustness to various document layouts.

We proposed a novel multi-modal pre-training model, LayoutMask, to achieve
this goal. LayoutMask only uses text and layout information as model input and
aims to enhance text-layout interactions and layout representation learning
during pre-training. It differs from previous studies in three aspects: choice
of 1D position, masking strategy, and pre-training objective.

Instead of global 1D position, LayoutMask proposes to use the in-segment token
orders as 1D position, which is referred to as “local 1D position” (See
illustration in Figure 1). As local 1D position does not provide cross-segment
orders, LayoutMask is supposed to infer global reading order by jointly using
1D position, 2D position, and semantic information, thus bringing in-depth
text-layout interactions. To further promote such interactions, we equip the
commonly used pre-training objective, Masked Language Modeling (MLM), with two
novel masking strategies, Whole Word Masking and Layout-Aware Masking, and
design an auxiliary pre-training objective, Masked Position Modeling, to
predict masked 2D positions during pre-training. With the above designs, we
increase the difficulty of pre-training objectives and force the model to
focus more on layout information to obtain reading order clues in various
document layouts in self-supervised learning, thus producing more adaptive and
robust text-layout representations for document understanding tasks.

Experimental results show that our proposed method can bring significant
improvements to VrDU tasks and achieve SOTA performance with only text and
layout modalities, indicating that previous studies have not fully explored
the potential power of layout information and text-layout interactions. The
contributions of this paper are summarized as follows:

  1. 1.

We propose LayoutMask, a novel multi-modal pre-training model focusing on
text-layout modality, to generate adaptive and robust multi-modal
representations for VrDU tasks.

  2. 2.

In LayoutMask, we use local 1D position instead of global 1D position to
promote reading order learning. We leverage Whole Word Masking and Layout-
Aware Masking in the MLM task and design a new pre-training objective, Masked
Position Modeling, to enhance text-layout interactions.

  3. 3.

Our method can produce useful multi-modal representations for documents and
significantly outperforms many SOTA methods in multiple VrDU tasks.

##  2 Related Work

The early studies in VrDU area usually use uni-modal models or multi-modal
models with shallow fusion (Yang et al., 2016, 2017; Katti et al., 2018;
Sarkhel and Nandi, 2019). In recent years, pre-training techniques in NLP
(Devlin et al., 2019; Zhang et al., 2019; Bao et al., 2020) and CV (Bao et
al., 2021; Li et al., 2022) have become more and more popular, and they have
been introduced into this area. Inspired by BERT (Devlin et al., 2019),
LayoutLM (Xu et al., 2020) first improved the masked language modeling task by
using the 2D coordinates of each token as layout embeddings, which can jointly
model interactions between text and layout information and benefits document
understanding tasks. Following this idea, LayoutLMv2 (Xu et al., 2021) propose
to concatenate image patches with textual tokens to enhance text-image
interactions, and LayoutLMv3 (Huang et al., 2022) proposed to learn cross-
modal alignment with unified text and image masking.

While the above methods focus on text-image interactions, some other studies
have realized the importance of layout information. StructuralLM (Li et al.,
2021a) utilizes segment-level layout features to provide word-segment
relations. DocFormer (Appalaraju et al., 2021) combines text, vision, and
spatial features with a novel multi-modal self-attention layer and shares
learned spatial embeddings across modalities. LiLT (Wang et al., 2022)
proposes a language-independent layout transformer where the text and layout
information are separately embedded. ERNIE-Layout (Peng et al., 2022) adopts a
reading order prediction task in pre-training and rearranges the token
sequence with the layout knowledge.

##  3 Methodology

Figure 2: The model pipeline of LayoutMask. Red Text: Illustration of the
Masked Language Modeling task. Blue Text: Illustration of the Masked Position
Modeling task.

LayoutMask is a multi-modal transformer that can encode text and layout
information of documents and produce multi-modal representations. The pipeline
of LayoutMask can be seen in Figure 2. LayoutMask uses the transformer model
with a spatial-aware self-attention mechanism proposed in LayoutLmv2 (Xu et
al., 2021) as the backbone and follows its preprocessing settings for text and
layout embeddings. In Section 3.1, we will discuss the different choices of
layout information in LayoutMask. In Section 3.2, we will introduce the pre-
training tasks and masking strategies used in LayoutMask.

###  3.1 Selection of Layout Information

Method | Position
---|---
1D | 2D
LayoutLM (Xu et al., 2020) | Global | Word
StructuralLM (Li et al., 2021a) | Global | Segment
LayoutLMv2 (Xu et al., 2021) | Global | Word
BROS (Hong et al., 2022) | Global | Segment†
LiLT (Wang et al., 2022) | Global | Segment
LayoutLMv3 (Huang et al., 2022) | Global | Segment
LayoutMask(Ours) | Local | Segment
Table 1:  The 1D position and 2D position choices in previous studies. Our
method uses local 1D position and segment-level 2D position. ${\dagger}$: BROS
leverages relative 2D positions instead of absolute positions.

For VrDU tasks, there are two types of commonly used layout information: 1D
position and 2D position. We list the 1D and 2D positions used in previous
studies in Table 1.

1D Position: As we discussed in Section 1, using global 1D position will bring
read order issues and could damage the adaptiveness and robustness of pre-
trained models. Different from some previous models that leverage global 1D
position as model input, we propose to use local 1D position in LayoutMask.
Local 1D position only encodes the token orders within each segment and always
restarts with 1 for each individual segment. Illustrations of the global and
local 1D positions can be seen in Figure 1 and Figure 2. Compared with global
1D position, the major difference of using local 1D position is the lack of
cross-segment orders, so the global reading order has to be inferred with
other layout and semantic clues. Besides, the in-segment orders implied by
local 1D position are more reliable and trustworthy than cross-segment orders
when meeting complex document layouts.

2D Position: The 2D position is represented as a 4-digit vector like
$[x_{1},y_{1},x_{2},y_{2}]$, where $[x_{1},y_{1}]$ and $[x_{2},y_{2}]$ are the
normalized coordinates of the top-left and bottom-right corners of a text box.
There are two commonly used types of 2D positions: word-level 2D position
(Word-2D) and segment-level 2D position (Segment-2D). For Word-2D, tokens of
the same word will have the same word-level boxes as their 2D position. While
for Segment-2D, the segment coordinates are shared by tokens within each
segment.

In our model, we choose local 1D position and segment-level 2D position as our
model input, where local 1D position can provide in-segment orders, and
segment-level 2D position can provide cross-segment reading order clues, so
the pre-trained model can learn the correct global reading order by jointly
using 1D and 2D positions. We will compare the experimental results using
different 1D & 2D position combinations in Section 4.3.1 and provide detailed
discussions.

###  3.2 Pre-training Objectives

####  3.2.1 Masked Language Modeling

The Masked Language Modeling task is the most essential and commonly used pre-
training task in multi-modal pre-training. In this task, we randomly mask some
tokens with a given probability $\mathrm{P}_{\mathrm{mlm}}$ (e.g., 15%) and
recover these tokens during pre-training.

For each document, we use $M$ to denote the number of masked tokens.
$\mathrm{y}_{i}$ and $\mathrm{\bar{y}}_{i}$ represent the ground truth and
prediction of the $i$-th masked token. Then the loss of this task is the
average cross entropy loss of all masked tokens:

$\mathcal{L}_{\mathrm{mlm}}=-\frac{1}{M}\sum_{i=1}^{M}\mathrm{CE}(\mathrm{y}_{i},\mathrm{\bar{y}}_{i}).$
(1)

In preliminary experiments, we find that the naive MLM task is not optimal for
multi-modal pre-training. Thus we propose to adopt two novel strategies, Whole
Word Masking (WWM) and Layout-Aware Masking (LAM), to enhance this task.

Whole Word Masking: The WWM strategy was first proposed for Chinese language
models to increase the task difficulty (Cui et al., 2021). Following this
strategy, we set masks at word-level instead of token-level, which is much
more challenging. When using WWM, the semantic relations between masked and
unmasked tokens of the same words are eliminated, so the model has to find
more context to predict masked words, which can promote text-layout
interactions.

Layout-Aware Masking: As we use Local-1D and Segment-2D as model input, the
global reading order should be obtained by jointly using 1D and 2D positions,
where Local-1D provides in-segment orders and segment-2D provides cross-
segment clues. We find that the cross-segment orders are harder to be learned,
so we propose Layout-Aware Masking (LAM) strategy to address this issue.
Unlike naive masking strategy where each token has an equal masking
probability $\mathrm{P}_{\mathrm{mlm}}$, in LAM strategy, the first and last
word of each segment has a higher probability (i.e.,
$3\times\mathrm{P}_{\mathrm{mlm}}$) to be masked. In order to predict such
masked words, the model has to pay more attention to finding their contexts in
the preceding or succeeding segment, thus promoting learning cross-segment
orders.

Method | #Parameters | Modality | FUNSD(F1$\uparrow$) | CORD(F1$\uparrow$) | SROIE(F1$\uparrow$)
---|---|---|---|---|---
BERTBase (Devlin et al., 2019) | 110M | T | 60.26 | 89.68 | 90.99
RoBERTABase (Liu et al., 2019) | 125M | T | 66.48 | 93.54 | -
UniLMv2Base (Bao et al., 2020) | 125M | T | 68.90 | 90.92 | 94.59
BROSBase (Hong et al., 2022) | 110M | T+L | 83.05 | 95.73 | 95.48
LiLTBase (Wang et al., 2022) | - | T+L | 88.41 | 96.07 | -
LayoutLMBase (Xu et al., 2020) | 160M | T+L+I | 79.27 | - | 94.38
LayoutLMv2Base (Xu et al., 2021) | 200M | T+L+I | 82.76 | 94.95 | 96.25
TILTBase (Powalski et al., 2021) | 230M | T+L+I | - | 95.11 | 97.65†
DocFormerBase (Appalaraju et al., 2021) | 183M | T+L+I | 83.34 | 96.33 | -
LayoutLMv3Base (Huang et al., 2022) | 133M | T+L+I | 90.29 | 96.56 | -
LayoutMaskBase (Ours) | 182M | T+L | 92.91$\pm$0.34 | 96.99$\pm$0.30 | 96.87$\pm$0.19
BERTLarge (Devlin et al., 2019) | 340M | T | 65.63 | 90.25 | 92.00
RoBERTALarge (Liu et al., 2019) | 355M | T | 70.72 | 93.80 | -
UniLMv2Large (Bao et al., 2020) | 355M | T | 72.57 | 92.05 | 94.88
LayoutLMLarge (Xu et al., 2020) | 343M | T+L | 77.89 | - | 95.24
BROSLarge (Hong et al., 2022) | 340M | T+L | 84.52 | 97.40 | -
LayoutLMv2Large (Xu et al., 2021) | 426M | T+L+I | 84.2 | 96.01 | 97.81
TILTLarge (Powalski et al., 2021) | 780M | T+L+I | - | 96.33 | 98.10†
DocFormerLarge (Appalaraju et al., 2021) | 536M | T+L+I | 84.55 | 96.99 | -
LayoutLMv3Large (Huang et al., 2022) | 368M | T+L+I | 92.08 | 97.46 | -
ERNIE-LayoutLarge (Peng et al., 2022) | - | T+L+I | 93.12 | 97.21 | 97.55
LayoutMaskLarge (Ours) | 404M | T+L | 93.20$\pm$0.29 | 97.19$\pm$0.20 | 97.27$\pm$0.32
Table 2:  F1 scores (%) of different methods on FUNSD, CORD, and SROIE .The
best results are denoted in boldface. ${\dagger}$: TILT utilized supervised
datasets during pre-training, so the scores are not directly comparable.

####  3.2.2 Masked Position Modeling

To further promote the representation learning of layout information in the
MLM task, we design an auxiliary task, Masked Position Modeling (MPM), which
has a symmetric pre-training objective: recovering randomly masked 2D
positions during pre-training (See illustration in Figure 2). Inspired by WWM,
we also apply the MPM task at word-level instead of token-level. For each pre-
training document, we randomly choose some unduplicated words with a given
probability $\mathrm{P}_{\mathrm{mpm}}$. Then, for each selected word, we mask
their 2D positions with the following two steps:

Box Split: We first split the selected word out of its segment so the original
segment box becomes 2 or 3 segment pieces (depending on if the word is at the
start/end or in the middle). The selected word becomes a one-word segment
piece with just itself. Then we update the local 1D positions (restarting with
1) and segment 2D positions for each new segment piece. With the above
operations, we can eliminate the local reading order clues implied by original
1D and 2D positions, so the model has to focus on semantical clues and new 2D
positions.

Box Masking: For each selected word, we mask its 2D position with pseudo
boxes: $[0,0,0,n]$ where $n\in[0,1,2,...]$ is a random number. Notice that
segment 2D position is shared among tokens in the same segment, so the pseudo
boxes will act as identifiers to distinguish identical tokens from different
masked boxes, thus avoiding ambiguity.

During pre-training, our model is supposed to predict the masked 2D positions
with GIoU loss (Rezatofighi et al., 2019):

$\mathcal{L}_{\mathrm{mpm}}=-\frac{1}{N}\sum_{i=1}^{N}(\frac{|\mathrm{B}_{i}\cap\mathrm{\bar{B}}_{i}|}{|\mathrm{B}_{i}\cup\mathrm{\bar{B}}_{i}|}-\frac{|\mathrm{C}_{i}\backslash(\mathrm{B}_{i}\cup\mathrm{\bar{B}}_{i})|}{|\mathrm{C}_{i}|}).$
(2)

Here, $i\in[1,2,...,N]$ is the index of $N$ masked 2D positions.
$\mathrm{B}_{i}$ is the ground truth box normalized to [0,1], and
$\mathrm{\bar{B}}_{i}$ denotes the predicted 2D position. $\mathrm{C}_{i}$ is
the smallest convex shapes that covers $\mathrm{B}_{i}$ and
$\mathrm{\bar{B}}_{i}$. $\mathcal{L}_{\mathrm{mpm}}$ is the average GIoU loss
of $N$ masked 2D positions.

The MPM task is very similar to the cloze test, where a group of randomly
selected words is supposed to be refilled at the right positions in the
original document. To predict the masked 2D positions of selected words, the
model has to find the context for each word based on semantic relations and
then infer with 2D position clues from a spatial perspective. The joint
learning process with both semantic and spatial inference can promote text-
layout interactions and help the model to learn better layout representations.

With the above two pre-training objectives, the model is pre-trained with the
following loss:

$\mathcal{L}_{\mathrm{total}}=\mathcal{L}_{\mathrm{mlm}}+\lambda\mathcal{L}_{\mathrm{mpm}},$
(3)

where $\lambda$ is a hyper-parameter that controls the balance of the two pre-
training objectives.

##  4 Experiments

###  4.1 Pre-training Settings

LayoutMask is pre-trained with IIT-CDIP Test Collection (Lewis et al., 2006).
It contains about 42 million scanned document pages, and we only use 10
million pages. We use a public OCR engine,
PaddleOCR111https://github.com/PaddlePaddle/PaddleOCR to obtain the OCR
results.

We train LayoutMask with two parameter sizes. LayoutMaskBase has 12 layers
with 16 heads, and the hidden size is 768. LayoutMaskLarge has 24 layers with
16 heads where the hidden size is 768. LayoutMaskBase and LayoutMaskLarge are
initialized with pre-trained XLM-RoBERTa models (Conneau et al., 2020).

For hyper-parameters, we have $\mathrm{P}_{\mathrm{mlm}}$=25% and
$\mathrm{P}_{\mathrm{mpm}}$=15% (See ablation study in Section A of the
Appendix). The weight of MPM loss $\lambda$ is set to be 1.

Method | Modality | Accuracy$\uparrow$
---|---|---
$\mathrm{Base}$ | $\mathrm{Large}$
VGG-16 (Afzal et al., 2017) | I | 90.97
Ensemble (Das et al., 2018) | I | 93.07
LadderNet (Sarkhel and Nandi, 2019) | I | 92.77
BERT (Devlin et al., 2019) | T | 89.81 | 89.92
RoBERTA (Liu et al., 2019) | T | 90.06 | 90.11
UniLMv2 (Bao et al., 2020) | T+L | 90.06 | 90.20
LayoutLM (Xu et al., 2020) | T+L | 91.78 | 91.90
StructuralLM (Li et al., 2021a) | T+L | - | 96.08
SelfDoc (Li et al., 2021b) | T+L+I | 92.81 | -
TITL (Powalski et al., 2021) | T+L+I | 95.25 | 95.52
LayoutLMv2 (Xu et al., 2021) | T+L+I | 95.25 | 95.64
DocFormer (Appalaraju et al., 2021) | T+L+I | 96.17 | 95.50
LiLT (Wang et al., 2022) | T+L+I | 95.68 | -
LayoutLMv3 (Huang et al., 2022) | T+L+I | 95.44 | 95.93
ERNIE-Layout (Peng et al., 2022) | T+L+I | - | 96.27
LayoutMask (Ours) | T+L | 93.26 | 93.80
Table 3:  The accuracies (%) of different methods on RVL-CDIP dataset. For
transformer-based models, we provide results for both base and large versions.

###  4.2 Comparison with the State-of-the-Art

In this section, we compare LayoutMask with SOTA models on two VrDU tasks:
form & receipt understanding and document image classification.

Position Settings | Datasets
---|---
1D | 2D | FUNSD (F1$\uparrow$) | CORD (F1$\uparrow$) | SROIE (F1$\uparrow$)
Global | Word | 82.17$\pm$0.45 | 95.95$\pm$0.43 | 96.02$\pm$0.34
Global | Segment | 91.61$\pm$0.42 | 96.69$\pm$0.24 | 96.20$\pm$0.26
Local | Word | 91.65$\pm$0.36 | 95.86$\pm$0.22 | 96.54$\pm$0.23
Local | Segment | 92.30$\pm$0.24 | 96.68$\pm$0.12 | 96.56$\pm$0.21
Table 4:  The average F1 scores (%) with different 1D position and 2D position
combinations. The best results are denoted in boldface.

####  4.2.1 Form and Receipt Understanding

In this task, we conduct entity extraction task on three document
understanding datasets: FUNSD (Jaume et al., 2019), CORD (Park et al., 2019),
and SROIE (Huang et al., 2019). The FUNSD dataset is a form understanding
dataset, which contains 199 documents (149 for training and 50 for test) and
9707 semantic entities. The CORD dataset is a receipt understanding dataset,
and it contains 1000 receipts (800 for training, 100 for validation, and 100
for test) with 30 semantic labels in 4 categories. The SROIE dataset is
another receipt understanding dataset with four types of entities, containing
626 receipts for training and 347 receipts for test.

For evaluation, we adopt the word-level F1 score as the evaluation metric for
FUNSD and CORD and use the entity-level F1 score for SROIE. Since these
datasets are quite small, in order to provide stable and reliable results, we
repeat our experiments ten times for each test and report the average F1
scores and standard errors as the final results.

The results of previous methods and LayoutMask on these datasets are listed in
Table 2. We have categorized them by the modalities used in pre-training: “T”
for text, “L” for layout, and “I” for image. Notice that LayoutMask is a “T+L”
model that does not use image modality.

For the base version, LayoutMaskBase outperforms other methods, including
“T+L+I” models, on all three datasets (FUNSD+2.62%, CORD+0.43%, SROIE+0.62%).
For the large version, LayoutMaskLarge ranks first on FUNSD and has comparable
results on CORD and SROIE.

These results show that LayoutMask has competitive performance with SOTA
methods, demonstrating the effectiveness of our proposed modules. Since
LayoutMask only uses text and layout information, we believe that the
potential power of layout information has not been fully explored in previous
studies.

####  4.2.2 Document Image Classification

In the document image classification task, we aim to classify document images
in RVL-CDIP dataset (Harley et al., 2015). This dataset is a subset of the
IIT-CDIP collection with 400,000 labeled document images (320,000 for train,
40,000 for validation, and 40,000 for test) in 16 categories. We use PaddleOCR
to extract text and layout information as model input. We compare different
methods with the overall classification accuracies on RVL-CDIP, and the
results are in Table 3.

It is observed that LayoutMask has beaten all uni-modality models (“I” and
“T”). For “T+L” models, LayoutMaskBase outperforms other base models with a
margin of 1.48%, while LayoutMaskLarge takes the second place in large models.
Compared with “T+L+I” models where image modality is utilized, LayoutMask
falls behind due to the lack of visual features from image modality. We have
found that the image modality plays an important role in this task because
RVL-CDIP images contain many elements that cannot be recognized by OCR engines
(e.g., figures, table lines, and handwritten texts) and have orientation
issues (See examples in Figure 5 of the Appendix). So the lack of image
modality will bring difficulties that cannot be solved with only text and
layout information.

###  4.3 Ablation Study on LayoutMask

# |  Position Settings 1D & 2D |  Swap Probability | SROIE (F1$\uparrow$)
---|---|---|---
Address | Company | Date | Total | Overall
1 | Local+Segment | - | 96.69$\pm$0.37 | 95.88$\pm$0.28 | 99.66$\pm$0.13 | 94.02$\pm$0.49 | 96.56$\pm$0.21
2 | Global+Segment | - | 96.54$\pm$0.51 | 95.84$\pm$0.59 | 99.69$\pm$0.26 | 92.73$\pm$0.57 | 96.20$\pm$0.26
3 | 10 | 91.73$\pm$2.00 | 95.22$\pm$0.61 | 99.65$\pm$0.34 | 91.87$\pm$1.33 | 94.62$\pm$0.69
4 | 20 | 90.03$\pm$3.77 | 94.93$\pm$0.53 | 99.60$\pm$0.32 | 91.67$\pm$1.35 | 94.06$\pm$1.02
5 | 30 | 88.12$\pm$4.59 | 94.88$\pm$0.82 | 99.55$\pm$0.28 | 91.19$\pm$1.38 | 93.44$\pm$1.14
Table 5:  F1 scores (%) on SROIE dataset with difference 1D positions and
increasing segment swap probabilities (%). We report both entity-level scores
(“Address”, “Company”, “Date”, and “Total”) and overall results (“Overall”)
for detailed comparison.

####  4.3.1 Comparison of Layout Information

We first compare the performance of LayoutMask using different layout
information. To make a fair comparison, we use LayoutMask with only the MLM
task and the WWM strategy during pre-training. For each test, LayoutMask is
pre-trained and fine-tuned with a specific 1D and 2D position combination. The
results are listed in Table 4.

Performance of 1D Position: For 1D position, Local-1D outperforms Global-1D on
both FUNSD (+9.48%/+0.69% with Word-2D/Segment-2D) and SROIE (+0.52%/+0.36%)
and falls a little behind on CORD (-0.07%/-0.01%).

To understand the benefits of using Local-1D, we provide entity-level F1 score
on SROIE dataset in Table 5 (#1 for Local+Segment and #2 for Global+Segment).
It is obvious that the performance gap between Local+Segment and
Global+Segment mainly comes from entity “Total” (from 94.02% to 92.72%), while
other entities have similar F1 scores. We illustrate two example images of
SROIE and their entities annotations in Figure 3. The right image, which
contains entity “Total”, has both vertical layout (first two lines) and
horizontal layout and has multiple misleading numbers with the same content as
ground truth (i.e., “193.00”). So it is hard to recognize the entity “Total”
by using the ordinary reading order implied by Global-1D. Therefore, using
Local-1D can perform better since it is more adaptive to such cases.

Performance of 2D Position: For 2D position, using segment-level 2D position
brings better results on all three datasets, regardless of the 1D position
types. An important reason is that the segment information is highly
indicative of recognizing entities. For example, every entity in FUNSD and
CORD exactly shares the same segment. Therefore, although Word-2D contains
more layout details, it will break the alignments between 2D positions and
entities, thus bringing performance drops. A typical result of such
phenomenon222Similar phenomenon can also be observed in the LayoutLM series
models, where using Segment-2D increase F1 scores for about 8% on FUNSD
dataset. can be seen on FUNSD, where replacing Global+Segment to Global+Word
will result in a significant decrease of 9.44%.

# | Pre-training Setting | Datasets
---|---|---
MLM | WWM | LAM | MPM | FUNSD (F1$\uparrow$) | CORD (F1$\uparrow$) | SROIE (F1$\uparrow$) | RVL-CDIP (ACC$\uparrow$)
1 | $\surd$ |  |  |  | 89.73$\pm$0.50 | 96.32$\pm$0.15 | 95.76$\pm$0.34 | 92.17
2 | $\surd$ | $\surd$ |  |  | 92.30$\pm$0.24 | 96.68$\pm$0.12 | 96.56$\pm$0.21 | 92.89
3 | $\surd$ | $\surd$ | $\surd$ |  | 92.66$\pm$0.26 | 96.89$\pm$0.24 | 96.64$\pm$0.22 | 93.03
4 | $\surd$ | $\surd$ |  | $\surd$ | 92.77$\pm$0.30 | 96.84$\pm$0.17 | 96.66$\pm$0.32 | 93.11
5 | $\surd$ | $\surd$ | $\surd$ | $\surd$ | 92.91$\pm$0.34 | 96.99$\pm$0.30 | 96.87$\pm$0.19 | 93.26
Table 6:  Performance analysis with different pre-training objectives and
masking strategies.

Robustness Comparison: Besides performance superiority, another important
reason to choose the local 1D position is its robustness to layout
disturbance. In real-world cases, a typical layout disturbance is “Segment
Swap”, where segments in the same line are indexed with wrong orders due to
document rotation or OCR issues. In such scenarios, the incorrect cross-
segment order will lead to incorrect global 1D positions and can be harmful to
model inference. Fortunately, the local 1D position is naturally immune to
such disturbance since it does not rely on cross-segment orders, making it
more robust than global 1D position.

To quantify such differences in robustness, we demonstrate how the segment
swap will influence the performance of using global 1D position by simulating
it on test datasets. For each test document, we randomly choose some lines
with a given probability $\mathrm{P}_{\mathrm{swap}}$ and then swap the
segments in it. We conduct experiments on LayoutMaskBase (MLM+WWM) in
Global+Segment setting with different $\mathrm{P}_{\mathrm{swap}}$ (i.e., 10%,
20%, and 30%) and the results are reported in Table 5 (#3-5).

During our experiments, we have found that the segment swap does not bring
significant performance changes on FUNSD and CORD datasets (so these results
are not listed due to the limited space). A possible reason is that FUNSD and
CORD do not contain cross-segment entities, so the segment swap can not break
the order of words in each entity. Evidence for this explanation is that the
SROIE dataset is significantly affected by segment swap, and its cross-segment
entities (“Address” and “Company”) have obvious performance drops. In SROIE,
the majority of “Address” entities and a few “Company” entities are printed in
multiple lines (See examples in Figure 3), so the segment swap can change the
in-entity orders of entity words. The results show that the “Address” entity
has the largest drop among all entities (-4.81%, -6.51%, and -8.42% for
$\mathrm{P}_{\mathrm{swap}}$=10%, 20%, 30%). Besides, the “Total” entity has
the second largest decrease (-0.86%, -1.06%, and -1.54%). As aforementioned,
the “Total” entities are usually surrounded by complex layouts and misleading
numbers, so the segment swap will bring extra difficulties in recognizing the
correct entities.

The above performance decreases of using global 1D position prove the
superiority of using local 1D position since it is not affected by such layout
disturbance and can have more robust performance in real-world scenarios.

Figure 3: Two images from SROIE dataset. Colored boxes denote the ground truth
of entities. The left image contains two cross-line and cross-segment entities
(“Address” and “Company”). The right image, with a mixture of vertical and
horizontal layouts, contains the “Total” entity.

####  4.3.2 Effectiveness of Proposed Methods

In Table 6, we provide results using different pre-training tasks and masking
strategies to demonstrate the effectiveness of our proposed modules.

Comparing #1 and #2 in Table 6, we observe that WWM brings significant
performance improvements on all datasets. The reason is that it increases the
difficulty of the MLM task, so we can obtain a stronger language model. We
also find that LAM can also brings consistent improvements on all dataset
because LAM can force the model to learn better representations for layout
information, which is beneficial to downstream tasks.

Comparing #2 to #4 and #3 to #5, it is observed that the MPM task also brings
considerable improvements on all datasets. MPM works as an auxiliary task to
help the MLM task and can increase the pre-training difficulty, contributing
to learning better and more robust layout representations.

Moreover, the full-version LayoutMask (#5) outperforms the naive version (#1)
by a large margin (FUNSD+3.18%, CORD+0.67%, SROIE+1.11%, and RVL-CDIP+1.09%),
demonstrating the effectiveness of our proposed modules when working together.
To better illustrate the effectiveness of our model design, we list category-
level accuracy improvements on RVL-CDIP dataset and provide detailed
discussions in Section B of the Appendix.

##  5 Conclusion

In this paper, we propose LayoutMask, a novel multi-modal pre-training model,
to solve the reading order issues in VrDU tasks. LayoutMask adopts local 1D
position as layout input and can generate adaptive and robust multi-modal
representations. In LayoutMask, we equip the MLM task with two masking
strategies and design a novel pre-training objective, Masked Position
Modeling, to enhance the text-layout interactions and layout representation
learning. With only using text and layout modalities, our method can achieve
excellent results and significantly outperforms many SOTA methods in VrDU
tasks.

## Limitations

Our method has the following limitations:

Datasets: In multi-modal pre-training, we rely on downstream datasets to
evaluate the performance of pre-trained models. The commonly used entity
extraction datasets are relatively small and lack diversity, so the proposed
method may not generalize well to real word scenarios.

Lack of Image Modality: In LayoutMask, we focus on text-layout interactions,
leaving the image modality unexplored. However, documents in the real world
contain many elements that can not be described by text and layout modalities,
like figures and lines, so incorporating image modality is important in
building a universal multi-modal pre-training model for document
understanding.

## References

  * Afzal et al. (2017) Muhammad Zeshan Afzal, Andreas Kölsch, Sheraz Ahmed, and Marcus Liwicki. 2017\.  Cutting the error by half: Investigation of very deep cnn and advanced training strategies for document image classification.  In _2017 14th IAPR International Conference on Document Analysis and Recognition (ICDAR)_ , volume 1, pages 883–888. IEEE.
  * Appalaraju et al. (2021) Srikar Appalaraju, Bhavan Jasani, Bhargava Urala Kota, Yusheng Xie, and R Manmatha. 2021.  Docformer: End-to-end transformer for document understanding.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 993–1003.
  * Bao et al. (2021) Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. 2021.  Beit: Bert pre-training of image transformers.  In _International Conference on Learning Representations_.
  * Bao et al. (2020) Hangbo Bao, Li Dong, Furu Wei, Wenhui Wang, Nan Yang, Xiaodong Liu, Yu Wang, Jianfeng Gao, Songhao Piao, Ming Zhou, et al. 2020.  Unilmv2: Pseudo-masked language models for unified language model pre-training.  In _International Conference on Machine Learning_ , pages 642–652. PMLR.
  * Conneau et al. (2020) Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Édouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020.  Unsupervised cross-lingual representation learning at scale.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 8440–8451.
  * Cui et al. (2021) Yiming Cui, Wanxiang Che, Ting Liu, Bing Qin, and Ziqing Yang. 2021.  Pre-training with whole word masking for chinese bert.  _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , 29:3504–3514.
  * Das et al. (2018) Arindam Das, Saikat Roy, Ujjwal Bhattacharya, and Swapan K Parui. 2018.  Document image classification with intra-domain transfer learning and stacked generalization of deep convolutional neural networks.  In _2018 24th international conference on pattern recognition (ICPR)_ , pages 3180–3185. IEEE.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  Bert: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of NAACL-HLT_ , pages 4171–4186.
  * Gu et al. (2022) Zhangxuan Gu, Changhua Meng, Ke Wang, Jun Lan, Weiqiang Wang, Ming Gu, and Liqing Zhang. 2022.  Xylayoutlm: Towards layout-aware multimodal networks for visually-rich document understanding.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 4583–4592.
  * Harley et al. (2015) Adam W Harley, Alex Ufkes, and Konstantinos G Derpanis. 2015.  Evaluation of deep convolutional nets for document image classification and retrieval.  In _2015 13th International Conference on Document Analysis and Recognition (ICDAR)_ , pages 991–995. IEEE.
  * Hong et al. (2022) Teakgyu Hong, Donghyun Kim, Mingi Ji, Wonseok Hwang, Daehyun Nam, and Sungrae Park. 2022.  Bros: A pre-trained language model focusing on text and layout for better key information extraction from documents.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 36, pages 10767–10775.
  * Huang et al. (2022) Yupan Huang, Tengchao Lv, Lei Cui, Yutong Lu, and Furu Wei. 2022.  Layoutlmv3: Pre-training for document AI with unified text and image masking.  In _MM ’22: The 30th ACM International Conference on Multimedia, Lisboa, Portugal, October 10 - 14, 2022_ , pages 4083–4091. ACM.
  * Huang et al. (2019) Zheng Huang, Kai Chen, Jianhua He, Xiang Bai, Dimosthenis Karatzas, Shijian Lu, and CV Jawahar. 2019.  Icdar2019 competition on scanned receipt ocr and information extraction.  In _2019 International Conference on Document Analysis and Recognition (ICDAR)_ , pages 1516–1520. IEEE.
  * Jaume et al. (2019) Guillaume Jaume, Hazim Kemal Ekenel, and Jean-Philippe Thiran. 2019.  Funsd: A dataset for form understanding in noisy scanned documents.  In _2019 International Conference on Document Analysis and Recognition Workshops (ICDARW)_ , volume 2, pages 1–6. IEEE.
  * Katti et al. (2018) Anoop R Katti, Christian Reisswig, Cordula Guder, Sebastian Brarda, Steffen Bickel, Johannes Höhne, and Jean Baptiste Faddoul. 2018.  Chargrid: Towards understanding 2d documents.  In _EMNLP_.
  * Lewis et al. (2006) David Lewis, Gady Agam, Shlomo Argamon, Ophir Frieder, David Grossman, and Jefferson Heard. 2006.  Building a test collection for complex document information processing.  In _Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval_ , pages 665–666.
  * Li et al. (2021a) Chenliang Li, Bin Bi, Ming Yan, Wei Wang, Songfang Huang, Fei Huang, and Luo Si. 2021a.  Structurallm: Structural pre-training for form understanding.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 6309–6318.
  * Li et al. (2022) Junlong Li, Yiheng Xu, Tengchao Lv, Lei Cui, Cha Zhang, and Furu Wei. 2022.  Dit: Self-supervised pre-training for document image transformer.  In _MM ’22: The 30th ACM International Conference on Multimedia, Lisboa, Portugal, October 10 - 14, 2022_ , pages 3530–3539. ACM.
  * Li et al. (2021b) Peizhao Li, Jiuxiang Gu, Jason Kuen, Vlad I Morariu, Handong Zhao, Rajiv Jain, Varun Manjunatha, and Hongfu Liu. 2021b.  Selfdoc: Self-supervised document representation learning.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 5652–5660.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Roberta: A robustly optimized bert pretraining approach.  _arXiv preprint arXiv:1907.11692_.
  * Park et al. (2019) Seunghyun Park, Seung Shin, Bado Lee, Junyeop Lee, Jaeheung Surh, Minjoon Seo, and Hwalsuk Lee. 2019.  Cord: a consolidated receipt dataset for post-ocr parsing.  In _Workshop on Document Intelligence at NeurIPS 2019_.
  * Peng et al. (2022) Qiming Peng, Yinxu Pan, Wenjin Wang, Bin Luo, Zhenyu Zhang, Zhengjie Huang, Teng Hu, Weichong Yin, Yongfeng Chen, Yin Zhang, et al. 2022.  Ernie-layout: Layout knowledge enhanced pre-training for visually-rich document understanding.  _arXiv preprint arXiv:2210.06155_.
  * Powalski et al. (2021) Rafał Powalski, Łukasz Borchmann, Dawid Jurkiewicz, Tomasz Dwojak, Michał Pietruszka, and Gabriela Pałka. 2021.  Going full-tilt boogie on document understanding with text-image-layout transformer.  In _International Conference on Document Analysis and Recognition_ , pages 732–747. Springer.
  * Rezatofighi et al. (2019) Hamid Rezatofighi, Nathan Tsoi, JunYoung Gwak, Amir Sadeghian, Ian Reid, and Silvio Savarese. 2019.  Generalized intersection over union: A metric and a loss for bounding box regression.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 658–666.
  * Sarkhel and Nandi (2019) Ritesh Sarkhel and Arnab Nandi. 2019.  Deterministic routing between layout abstractions for multi-scale classification of visually rich documents.  In _28th International Joint Conference on Artificial Intelligence (IJCAI), 2019_.
  * Wang et al. (2022) Jiapeng Wang, Lianwen Jin, and Kai Ding. 2022.  Lilt: A simple yet effective language-independent layout transformer for structured document understanding.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 7747–7757.
  * Wang et al. (2021) Zilong Wang, Yiheng Xu, Lei Cui, Jingbo Shang, and Furu Wei. 2021.  Layoutreader: Pre-training of text and layout for reading order detection.  In _EMNLP (1)_.
  * Xu et al. (2021) Yang Xu, Yiheng Xu, Tengchao Lv, Lei Cui, Furu Wei, Guoxin Wang, Yijuan Lu, Dinei Florencio, Cha Zhang, Wanxiang Che, et al. 2021.  Layoutlmv2: Multi-modal pre-training for visually-rich document understanding.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 2579–2591.
  * Xu et al. (2020) Yiheng Xu, Minghao Li, Lei Cui, Shaohan Huang, Furu Wei, and Ming Zhou. 2020.  Layoutlm: Pre-training of text and layout for document image understanding.  In _Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_, pages 1192–1200.
  * Yang et al. (2017) Xiao Yang, Ersin Yumer, Paul Asente, Mike Kraley, Daniel Kifer, and C Lee Giles. 2017.  Learning to extract semantic structure from documents using multimodal fully convolutional neural networks.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 5315–5324.
  * Yang et al. (2016) Zichao Yang, Diyi Yang, Chris Dyer, Xiaodong He, Alex Smola, and Eduard Hovy. 2016\.  Hierarchical attention networks for document classification.  In _Proceedings of the 2016 conference of the North American chapter of the association for computational linguistics: human language technologies_ , pages 1480–1489.
  * Zhang et al. (2019) Zhengyan Zhang, Xu Han, Zhiyuan Liu, Xin Jiang, Maosong Sun, and Qun Liu. 2019.  Ernie: Enhanced language representation with informative entities.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 1441–1451.

##  Appendix A Ablation Study of Masking Probabilities

We compare LayoutMask using different $\mathrm{P}_{\mathrm{mlm}}$ and
$\mathrm{P}_{\mathrm{mpm}}$, and the results are in Figure 4. We first find
the best $\mathrm{P}_{\mathrm{mlm}}$ without using the MPM task, and the
optimal value is 25%. Then we fix such optimal $\mathrm{P}_{\mathrm{mlm}}$ to
find the best $\mathrm{P}_{\mathrm{mpm}}$, which is 15% as the results show.

Figure 4: The F1 scores on FUNSD, CORD, and SROIE with different masking
probabilities. FUNSD dataset uses the x-axis on the right side. Figure 5:
RVL-CDIP images from different categories. (1) Advertisement; (2) News
article; (3) Presentation (with incorrect orientations).

##  Appendix B Ablation Study on RVL-CDIP

To further understand the effectiveness of our model design, we list the
detailed classification results on RVL-CDIP dataset with the naive version and
the full version in Table 7. It is observed that the major performance
improvements come from three categories: presentation (+3.36%), advertisement
(+2.93%), and news article (+2.35%). We find these categories have more
diverse layouts (See examples in Figure 5), so classifying these documents
requires a better understanding of the document structure, which also
indicates the effectiveness of our methods in helping layout understanding.

Category | Model Settings | Diff. (%)
---|---|---
Naive | Full
letter | 90.30 | 90.86 | 0.56
form | 85.71 | 86.77 | 1.07
email | 98.17 | 98.33 | 0.15
handwritten | 93.96 | 94.26 | 0.30
advertisement | 88.47 | 91.40 | 2.93
sci-report | 87.87 | 89.38 | 1.51
sci-publication | 93.08 | 93.73 | 0.65
specification | 95.91 | 96.56 | 0.64
file folder | 91.29 | 92.71 | 1.42
news article | 90.09 | 92.44 | 2.35
budget | 94.01 | 94.96 | 0.95
invoice | 94.02 | 94.54 | 0.52
presentation | 86.14 | 89.50 | 3.36
questionnaire | 92.44 | 92.88 | 0.44
resume | 98.31 | 98.70 | 0.39
memo | 94.93 | 95.12 | 0.19
Overall | 92.17 | 93.26 | 1.09
Table 7:  The category-level accuracies (%) on RVL-CDIP dataset of LayoutMask
on the naive version and the full version. Categories with top-3 accuracy
improvements are denoted in boldface.