Pooja Pareek

# Energy dependence of light neutral meson $p_{\rm T}$ spectrum produced in pp
collisions at the LHC measured in ALICE

for the ALICE collaboration

###### Abstract

We present the measurement of light neutral mesons, $\pi^{0}$ and $\eta$, in
pp collisions at different center-of-mass energies obtained with the ALICE
experiment at the LHC. The $\pi^{0}$ and $\eta$ mesons are measured via
photons reconstructed by the electromagnetic calorimeters and the central
tracking system. The invariant cross-section of $\pi^{0}$ and $\eta$ mesons
are measured in a broad $p_{\rm T}$ range at $\sqrt{s}=0.9,2.76,7,5.02$ and 8
TeV. The spectra of $\pi^{0}$ and $\eta$ mesons measured in pp collisions at
different collision energies show $x_{\rm T}$-scaling at high $p_{\rm T}$ and
violation of $m_{\rm T}$-scaling at low $p_{\rm T}$. The smaller $x_{\rm
T}$-scaling exponents of our measurements compared to RHIC may hint at a
reduced importance of higher twist processes at LHC.

##  1 Motivation

The measurement of inclusive hadron production in pp collisions is a widely
used tool to validate perturbative quantum chromodynamics (pQCD). The cross
section of hadron production at high transverse momentum in pp collisions can
be calculated within pQCD as these are produced by fragmentation of quarks and
gluons from hard parton scatterings. Therefore, the study of light neutral
mesons, $\pi^{0}$ and $\eta$ in pp collisions in a wide transverse momentum
range is important to understand parton dynamics by probing parton
distribution functions and fragmentation functions in a new energy domain. In
addition, $\pi^{0}$ and $\eta$ meson transverse momentum measurements are
necessary to estimate the decay photon background for direct photon
measurements and also to provide a reference for heavy-ion collisions. We
present measurement of $\pi^{0}$ and $\eta$ mesons in pp collisions at LHC
energies $\sqrt{s}=0.9,2.76,7,5.02$ and 8 TeV with the ALICE experiment.

##  2 Neutral meson measurement

The $\pi^{0}$ and $\eta$ mesons are reconstructed by invariant mass analysis
of their decay into two photons. In ALICE the decay photons are measured with
different methods. Photons can be measured in electromagnetic calorimeters:
EMCal [1] and PHOS [2], designed to measure energy and hit coordinates of
photons and electrons. Additionally, Photon Conversion Method (PCM) [3] which
is based on photon conversion in detector material ($\gamma\rightarrow
e^{+}e^{-}$) is a complementary method to measure neutral mesons via
conversion photons up to very low $p_{\rm T}$ ($\sim$ 0.3 GeV$/c$) using the
central tracking system in ALICE. The energy resolution of the calorimeters
improves with increasing $p_{\rm T}$ which together with the calorimeter
trigger allows to measure neutral mesons up to high $p_{\rm T}$. In addition,
a hybrid method is also used where one photon is reconstructed in a
calorimeter and the second photon with the PCM method [4]. Results from
different techniques are combined to provide precision measurements of
$\pi^{0}$ and $\eta$ in pp collisions over much wider $p_{\rm T}$ range than
any other identified hadrons.

##  3 Results

In ALICE, $\pi^{0}$ and $\eta$ mesons are measured in pp collisions at
$\sqrt{s}=0.9,2.76,7,5.02$ and 8 TeV [3, 4, 5] as shown in Fig.1. The results
are described by empirical parametrizations like Two Component Model (TCM) [6]
and Tsallis function [7]. Further, invariant cross sections are compared with
Next-to-Leading Order (NLO) pQCD calculation of $\pi^{0}$ and $\eta$ meson
production with latest fragmentation functions (DSS14 [8], AESSS [9]). NLO
calculation overestimates the data by 20-50% for pions and nearly a factor 2
for eta mesons. PYTHIA 8.2 [10] with Monash 2013 tune reproduces $\pi^{0}$
spectra at high $p_{\rm T}$ but shows a deviation from data at moderate
$p_{\rm T}$ in higher energy. Likewise, PYTHIA 8.2 [10] with Monash 2013 tune
describes the $\eta$ meson spectra. Overall PYTHIA shows better agreement with
data compared to NLO pQCD.

Figure 1:  (Left) Invariant cross section of $\pi^{0}$ in pp collisions at
$\sqrt{s}=0.9,2.76,7,5.02$ and 8 TeV [3, 4, 5]. Neutral pions are measured in
the $p_{\rm T}$ range $0.3-200$ GeV$/c$. (Right) The experimental data are
compared to NLO pQCD calculation and PYTHIA. The ratio of data and theory
predictions to TCM fit are shown in seperate panels for different energies.

###  3.1 $x_{\rm T}$ scaling

The invariant cross-section can be expressed as a function of new variable,
$x_{\rm{\it}T}=2p_{\rm T}/\sqrt{s}$ [11],

$E\frac{d^{3}\sigma}{dp^{3}}=\frac{1}{\sqrt{s}^{n}}G(x_{\rm{\it}T}).$ (1)

Thus, from the above equation, the invariant cross-section as a function of
$x_{\rm T}$, when multiplied by $\sqrt{s}^{n}$ does not depend on collision
energy, which is referred to as $x_{\rm T}$ scaling.

$\sqrt{s}^{n}\times E\frac{d^{3}\sigma}{dp^{3}}=G(x_{\rm{\it}T}).$ (2)

The scaling parameter $n$ is deduced from comparison of $x_{\rm T}$ spectra at
different center-of-mass energies:

$n(x_{\rm T},\sqrt{s_{1}},\sqrt{s_{2}})=\frac{ln(\sigma^{inv}(x_{\rm
T},\sqrt{s_{2}})/\sigma^{inv}(x_{\rm
T},\sqrt{s_{1}}))}{ln(\sqrt{s_{1}}/\sqrt{s_{2}})}$ (3)

The dependence of the parameters $n$ on $x_{\rm T}$ for $\pi^{0}$ at different
ALICE energies are shown in Fig. 2. The parameter $n$ depends on both $x_{\rm
T}$ and $\sqrt{s}$ at low $p_{\rm T}$ and shows an asymptotic trend at $p_{\rm
T}$ > 2 GeV/c. We fit $n$ at $p_{\rm T}>$2 GeV$/c$ with a constant function
and obtain $n(\pi^{0})=5.03\pm 0.06$ and $n(\eta)=4.79\pm 0.09$. For jets and
photons a value of $n\simeq 4.5$ was measured in previous experiments [12]; At
RHIC, a value of $n=6.38$ for $\pi^{0}$ was obtained [13]. In pQCD, hadron
production at large transverse momentum are described by the leading (LT) and
higher twist (HT) processes [12]. In LT processes, $n=4$ whereas $n>4$ for HT
processes [12]. The difference in $n$ at RHIC and LHC may be related to
different mechanisms for hadron production at these energies. Smaller value of
$n$ for $\pi^{0}$ production at the LHC energies with respect to the RHIC
energies suggests that the contribution of the HT processes is reduced with
the growth of the collision energy. The $x_{\rm T}$ scaled $\pi^{0}$ and
$\eta$ meson cross-sections were checked with calculated $n$ values, and both
$\pi^{0}$ and $\eta$ mesons follows the $x_{\rm T}$ scaling at $p_{\rm T}$ > 2
GeV/c. The deviation from scaling at low $x_{\rm T}$ is attributed to the
contribution from soft processes at low $p_{\rm T}$.

###  3.2 $m_{\rm T}$ scaling

In lower energy experiments [14, 15, 16, 17], spectral shapes of hadron
species are found to be identical in shape when plotted as a function of
$m_{\rm T}$ ($m_{\rm T}=\sqrt{p_{\rm T}^{2}+m^{2}}$), known as $m_{\rm T}$
scaling. $\eta/\pi^{0}$ ratio from data over $\eta/\pi^{0}$ ratio obtained
with $m_{\rm T}$ scaling is shown in Fig.2. We observe violation of the
empirical $m_{\rm T}$ scaling at $p_{\rm T}<$ 2 GeV$/c$ at the LHC, which is
important in tuning the phenomenological models at low $p_{\rm T}$.

Figure 2:  (Left) $n$ parameter for $\pi^{0}$ in pp collisions at different
collision energies $\sqrt{s}$. (Right) Ratio of transverse momentum
distributions of measured $\eta/\pi^{0}$ ratio over $\eta/\pi^{0}$ ratio
obtained with $m_{\rm T}$ scaling for pp collision at
$\sqrt{s}=0.9,2.76,7,5.02$ and 8 TeV.

##  4 Summary

Measurements of neutral meson spectra, and $x_{\rm T}$ and $m_{\rm T}$ scaling
have been performed with the ALICE detector. These measurement can be used to
validate pQCD predictions. The measured spectra in pp collisions at
$\sqrt{s}=2.76,7,5.02$ and 8 TeV show disagreement with pQCD calculations. The
$m_{\rm T}$ scaling of measured spectra show violation of this empirical
scaling at $p_{\rm T}<$ 2 GeV$/c$. We study $x_{\rm T}$ scaling and observe
the $n$ parameters for $\pi^{0}$ and $\eta$ are consistent with jets and
direct photons, which suggests that almost all neutral mesons are generated by
jet fragmentation at the LHC. In contrast at RHIC, neutral mesons are possibly
generated by both jet fragmentation and direct hadron production. Therefore,
$x_{\rm T}$ scaling analysis can contribute to an understanding of the
particle production mechanism at high energies.

## References

  * [1] P. Cortese et al. (ALICE collaboration), CERN-LHCC-2008-014, (2008).
  * [2] G. Dellacasa et al. (ALICE collaboration), CERN-LHCC-99-04, (1999).
  * [3] B. Abelev et al. (ALICE Collaboration), Phys. Lett. B 717, 162 (2012).
  * [4] S. Acharya et al. (ALICE Collaboration), Eur. Phys. J. C 77, 339 (2017).
  * [5] S. Acharya et al. (ALICE Collaboration), Eur. Phys. J. C 78, 263 (2018)
  * [6] A. Bylinkin, N. S. Chernyavskaya and A. A Rostovtsev., Eur. Phys. J. C 75, 166 (2015).
  * [7] C. Tsallis, J. Statist. Phys. 52, 479-487 (1988).
  * [8] D. de Florian et al., Phys. Rev. D 91, 014035 (2015).
  * [9] C. A. Aidala et al., Phys. Rev. D 83, 034002 (2011).
  * [10] T. Sjostrand et al., Comput. Phys. Commun. 191, 159 (2015).
  * [11] R. Blankenbecler, S. J. Brodsky and J. F. Gunion, Phys. Lett. 42 B, 461 (1972).
  * [12] F. Arleo et al., Phys. Rev. Lett. 105, 062002 (2010).
  * [13] A. Adare et al. (PHENIX Collaboration), Phys. Rev. D 79, 012003, (2009).
  * [14] K. Guettler et al., Phys. Lett. B 64, 111-116, (1976).
  * [15] J. Harris et al. (NA35 Collaboration), Nucl. Phys. A 498, 133c, (1989).
  * [16] R. Albrechta (WA80 Collaboration), Physics Letters B 361, 14-20, (1995).
  * [17] B. I. Abelev et al. (STAR Collaboration), Phys. Rev. C 75, 064901 (2007).


H73 |  5.08$\pm$ | ${}^{0.15}_{0.12}$ |  29000$\pm$ | ${}^{4000}_{3000}$ |  3.85$\pm$ | ${}^{0.38}_{0.23}$ |  13.9$\pm$ | ${}^{0.7}_{0.8}$ |  -6.85$\pm$ | ${}^{0.65}_{1.15}$ | - |  250$\pm$ | ${}^{65}_{80}$ |  0.10$\pm$ | ${}^{0.14}_{0.02}$
H108 |  5.19$\pm$ | ${}^{0.15}_{0.23}$ |  49500$\pm$ | ${}^{6500}_{8500}$ |  4.17$\pm$ | ${}^{0.28}_{0.30}$ |  5.4$\pm$ | ${}^{0.5}_{0.3}$ |  -6.67$\pm$ | ${}^{0.15}_{0.68}$ | - |  240$\pm$ | ${}^{90}_{80}$ |  0.09$\pm$ | ${}^{0.08}_{0.01}$
H112 |  5.14$\pm$ | ${}^{0.20}_{0.19}$ |  36000$\pm$ | ${}^{6500}_{5000}$ |  4.03$\pm$ | ${}^{0.60}_{0.47}$ |  9.6$\pm$ | ${}^{0.8}_{0.8}$ |  -7.89$\pm$ | ${}^{1.80}_{0.15}$ | - |  250$\pm$ | ${}^{130}_{100}$ |  0.09$\pm$ | ${}^{0.18}_{0.01}$
H114 |  5.21$\pm$ | ${}^{0.18}_{0.15}$ |  43500$\pm$ | ${}^{7000}_{5000}$ |  4.12$\pm$ | ${}^{0.35}_{0.30}$ |  7.1$\pm$ | ${}^{0.5}_{0.5}$ |  -6.52$\pm$ | ${}^{0.35}_{1.90}$ | - |  60$\pm$ | ${}^{80}_{55}$ |  0.09$\pm$ | ${}^{0.16}_{0.01}$
H116 |  4.93$\pm$ | ${}^{0.12}_{0.17}$ |  37000$\pm$ | ${}^{4000}_{5000}$ |  4.08$\pm$ | ${}^{0.35}_{0.38}$ |  7.1$\pm$ | ${}^{0.5}_{0.4}$ |  -6.82$\pm$ | ${}^{0.25}_{1.20}$ | - |  160$\pm$ | ${}^{95}_{80}$ |  0.09$\pm$ | ${}^{0.16}_{0.01}$
H120 |  4.72$\pm$ | ${}^{0.21}_{0.21}$ |  35000$\pm$ | ${}^{6750}_{5000}$ |  4.10$\pm$ | ${}^{0.45}_{0.41}$ |  6.3$\pm$ | ${}^{0.6}_{0.5}$ |  -6.96$\pm$ | ${}^{0.35}_{1.05}$ | - |  220$\pm$ | ${}^{110}_{80}$ |  0.09$\pm$ | ${}^{0.15}_{0.01}$
H121 |  4.81$\pm$ | ${}^{0.09}_{0.17}$ |  34000$\pm$ | ${}^{2750}_{4500}$ |  4.05$\pm$ | ${}^{0.30}_{0.40}$ |  7.4$\pm$ | ${}^{0.5}_{0.3}$ |  -7.82$\pm$ | ${}^{1.30}_{0.20}$ | - |  110$\pm$ | ${}^{70}_{70}$ |  0.10$\pm$ | ${}^{0.15}_{0.01}$
H123 |  4.92$\pm$ | ${}^{0.17}_{0.12}$ |  40000$\pm$ | ${}^{6250}_{3500}$ |  3.98$\pm$ | ${}^{0.39}_{0.30}$ |  6.1$\pm$ | ${}^{0.3}_{0.4}$ |  -7.84$\pm$ | ${}^{1.30}_{0.70}$ | - |  105$\pm$ | ${}^{65}_{75}$ |  0.10$\pm$ | ${}^{0.15}_{0.02}$
H129 |  4.45$\pm$ | ${}^{0.21}_{0.19}$ |  40000$\pm$ | ${}^{7500}_{5250}$ |  4.10$\pm$ | ${}^{0.70}_{0.45}$ |  3.5$\pm$ | ${}^{0.3}_{0.3}$ |  -7.77$\pm$ | ${}^{0.65}_{0.25}$ | - |  110$\pm$ | ${}^{120}_{100}$ |  0.09$\pm$ | ${}^{0.16}_{0.01}$
H132 |  5.07$\pm$ | ${}^{0.17}_{0.11}$ |  40000$\pm$ | ${}^{6250}_{3250}$ |  4.10$\pm$ | ${}^{0.40}_{0.35}$ |  7.2$\pm$ | ${}^{0.3}_{0.5}$ |  -6.62$\pm$ | ${}^{0.35}_{1.35}$ | - |  60$\pm$ | ${}^{90}_{60}$ |  0.12$\pm$ | ${}^{0.14}_{0.03}$
H134 |  4.84$\pm$ | ${}^{0.13}_{0.16}$ |  38000$\pm$ | ${}^{4250}_{4250}$ |  4.08$\pm$ | ${}^{0.35}_{0.30}$ |  6.2$\pm$ | ${}^{0.4}_{0.3}$ |  -7.42$\pm$ | ${}^{0.75}_{1.10}$ | - |  115$\pm$ | ${}^{68}_{58}$ |  0.09$\pm$ | ${}^{0.15}_{0.01}$
H135 |  4.57$\pm$ | ${}^{0.20}_{0.28}$ |  25500$\pm$ | ${}^{5000}_{5500}$ |  3.65$\pm$ | ${}^{0.50}_{0.55}$ |  10.0$\pm$ | ${}^{1.2}_{0.8}$ |  -7.09$\pm$ | ${}^{0.40}_{1.55}$ | - |  150$\pm$ | ${}^{80}_{70}$ |  0.12$\pm$ | ${}^{0.17}_{0.04}$
H139 |  4.88$\pm$ | ${}^{0.14}_{0.15}$ |  38000$\pm$ | ${}^{4750}_{4000}$ |  4.00$\pm$ | ${}^{0.40}_{0.40}$ |  6.4$\pm$ | ${}^{0.4}_{0.4}$ |  -6.77$\pm$ | ${}^{0.35}_{1.25}$ | - |  80$\pm$ | ${}^{75}_{70}$ |  0.10$\pm$ | ${}^{0.15}_{0.01}$
H141 |  3.80$\pm$ | ${}^{0.88}_{0.12}$ |  14500$\pm$ | ${}^{15500}_{1500}$ |  2.50$\pm$ | ${}^{1.65}_{0.05}$ |  12.7$\pm$ | ${}^{0.8}_{4.2}$ |  -8.62$\pm$ | ${}^{1.35}_{0.40}$ | - |  360$\pm$ | ${}^{210}_{140}$ |  0.09$\pm$ | ${}^{0.25}_{0.01}$
H159 |  4.84$\pm$ | ${}^{0.16}_{0.22}$ |  35000$\pm$ | ${}^{5000}_{6000}$ |  4.20$\pm$ | ${}^{0.40}_{0.45}$ |  7.2$\pm$ | ${}^{0.7}_{0.5}$ |  -7.98$\pm$ | ${}^{1.65}_{0.05}$ | - |  210$\pm$ | ${}^{115}_{100}$ |  0.09$\pm$ | ${}^{0.16}_{0.01}$
H162 |  4.64$\pm$ | ${}^{0.32}_{0.33}$ |  31000$\pm$ | ${}^{10500}_{8000}$ |  3.85$\pm$ | ${}^{0.47}_{0.60}$ |  7.3$\pm$ | ${}^{1.1}_{0.8}$ |  -6.85$\pm$ | ${}^{0.35}_{1.05}$ | - |  250$\pm$ | ${}^{180}_{110}$ |  0.09$\pm$ | ${}^{0.18}_{0.01}$
H173 |  4.27$\pm$ | ${}^{0.30}_{0.21}$ |  23000$\pm$ | ${}^{6500}_{4000}$ |  3.50$\pm$ | ${}^{0.65}_{0.50}$ |  8.7$\pm$ | ${}^{0.9}_{1.0}$ |  -8.40$\pm$ | ${}^{1.05}_{1.10}$ | - |  280$\pm$ | ${}^{130}_{120}$ |  0.09$\pm$ | ${}^{0.16}_{0.01}$

Table H.4: Overview of diagnostic lines used for the analysis of each source. If a line was included in the fitting, the cell is marked with ‘x’ (STIS data) or ‘o’ (GHRS archival data). Note that if lines are very close together or blended, they may not have a separate panel in the plots showing the best fits (Fig. I.20 to I.75).  Source |  |  C iv $\lambda$1170 |  C iii $\lambda$1175 |  N v $\lambda$1240 |  O iv $\lambda$1338-42-43 |  O v $\lambda$1371 |  Si iv $\lambda$1393 |  C iv $\lambda$1548 |  He ii $\lambda$1640 |  N iv $\lambda$1718 |  H$\eta$ |  H$\zeta$ |  H$\epsilon$ |  He i-ii $\lambda$4026 |  N iv $\lambda$4058 |  H$\delta$ |  He ii $\lambda$4200 |  H$\gamma$ |  He i $\lambda$4387 |  He i $\lambda$4471 |  He ii $\lambda$4541 |  N iii $\lambda$4634-40-41 |  N v $\lambda$4603-19 |  He ii $\lambda$4686 |  He i $\lambda$4713 |  H$\alpha$ |  | Source |  |  C iv $\lambda$1170 |  C iii $\lambda$1175 |  N v $\lambda$1240 |  O iv $\lambda$1338-42-43 |  O v $\lambda$1371 |  Si iv $\lambda$1393 |  C iv $\lambda$1548 |  He ii $\lambda$1640 |  N iv $\lambda$1718 |  H$\eta$ |  H$\zeta$ |  H$\epsilon$ |  He i-ii $\lambda$4026 |  N iv $\lambda$4058 |  H$\delta$ |  He ii $\lambda$4200 |  H$\gamma$ |  He i $\lambda$4387 |  He i $\lambda$4471 |  He ii $\lambda$4541 |  N iii $\lambda$4634-40-41 |  N v $\lambda$4603-19 |  He ii $\lambda$4686 |  He i $\lambda$4713 |  H$\alpha$
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
R136a1 |  | x | x | x | x | x |  | x | x | x | x | x | x |  | x | x | x | x |  |  | x | x | x | x | x | x |  | H69 |  | x | x | x | x | x |  | x | x |  | x | x | x | x |  | x | x | x | x | x | x |  |  | x | x | x
R136a2 |  | x | x | x | x | x |  | x | x | x | x | x | x | x | x | x | x | x |  |  | x | x | x | x | x | x |  | H70 |  | x | x | x | x | x |  | x | x | o | x | x | x | x | x | x | x | x | x | x | x |  |  | x |  | x
R136a3 |  | x | x | x | x | x |  | x | x | x | x | x | x | x | x | x | x | x |  |  | x | x | x | x | x | x |  | H71 |  | x | x | x | x | x |  | x | x | x | x | x | x | x |  | x | x | x | x | x | x |  |  | x | x | x
R136a4 |  | x | x | x | x | x |  | x | x | x | x | x | x | x | x | x | x | x | x |  | x | x | x | x | x | x |  | H75 |  | x | x | x | x | x |  | x | x |  | x | x | x | x |  | x | x | x | x | x | x |  |  | x | x | x
R136a5 |  |  | x | x | x | x |  | x | x | o | x | x | x | x | x | x | x | x | x | x | x | x | x | x | x | x |  | H78 |  | x | x | x | x | x |  | x | x |  | x | x | x | x | x | x | x | x | x | x | x | x |  | x | x |
R136a6 |  | x | x | x | x | x |  | x | x | x | x | x | x | x | x | x | x | x | x | x | x | x | x | x | x | x |  | H80 |  | x | x | x | x |  |  | x | x |  |  | x | x | x |  | x | x | x | x | x | x |  |  | x | x | x
R136a7 |  | x | x | x | x | x |  | x | x | o | x | x | x | x | x | x | x | x | x | x | x | x | x | x | x | x |  | H86 |  | x | x | x | x | x |  | x | x | x | x | x | x | x | x | x | x | x | x | x | x | x | x | x |  | x
R136a8 |  | x | x | x | x | x |  | x | x |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  | x |  | H90 |  | x | x | x | x | x |  | x | x |  | x | x | x | x |  | x | x |  |  |  |  |  |  |  |  |
R136b |  | x | x | x | x |  | x | x | x | o | x | x | x | x | x | x | x | x | x | x | x | x |  | x | x | x |  | H92 |  | x | x | x | x |  |  | x | x |  | x | x | x | x |  | x | x | x | x | x | x |  |  | x |  |
H30 |  | x | x | x | x |  |  | x | x |  | x | x | x | x |  | x | x | x | x | x | x |  |  | x | x |  |  | H94 |  | x | x | x | x | x |  | x | x |  | x | x | x | x |  | x | x | x | x | x | x |  |  | x | x | x
H31 |  | x | x | x | x | x |  | x | x |  | x | x | x | x | x | x | x | x | x | x | x |  |  | x |  | x |  | H143 |  | x | x | x | x |  |  | x | x | x | x | x | x | x |  | x | x | x | x | x | x |  |  | x | x |
H35 |  | x | x | x | x | x |  | x | x |  | x | x | x | x | x | x | x | x | x | x | x | x |  | x | x | x |  | H73 |  | x | x | x | x |  |  | x | x |  | x | x | x | x | x | x | x | x | x | x | x |  |  | x |  | x
H36 |  | x | x | x | x | x |  | x | x | o | x | x | x | x | x | x | x | x | x | x | x | x | x | x | x | x |  | H108 |  | x | x | x | x |  |  | x | x |  | x | x | x | x | x | x | x | x |  |  | x |  |  | x |  | x
H40 |  | x | x | x | x | x |  | x | x | x | x | x | x | x | x | x | x | x | x |  | x |  |  | x |  | x |  | H112 |  | x | x | x | x |  |  | x | x |  |  | x | x | x |  | x | x | x | x | x | x |  |  | x | x |
H45 |  | x | x | x | x | x |  | x | x | x | x | x | x | x |  | x | x | x | x | x | x |  |  | x | x | x |  | H114 |  | x | x | x | x | x |  | x | x |  | x | x | x | x | x | x | x | x | x | x | x |  |  | x |  | x
H46 |  | x | x | x | x | x |  | x | x | o |  |  |  |  | x | x | x | x | x |  | x | x | x | x | x | x |  | H116 |  | x | x | x | x |  |  | x | x |  | x | x | x | x |  | x | x | x | x | x | x |  |  | x | x | x
H47 |  | x | x | x | x | x |  | x | x | o | x | x | x | x | x | x | x | x |  | x | x |  | x | x | x | x |  | H120 |  | x | x | x | x |  |  | x | x |  | x | x | x | x |  | x | x | x | x | x | x |  |  | x | x | x
H48 |  | x | x | x | x | x |  | x | x | o | x | x | x | x | x | x | x | x | x | x | x | x | x | x | x | x |  | H121 |  | x | x | x | x |  |  | x | x |  | x | x | x | x |  | x | x | x | x | x | x |  |  | x |  |
H49 |  |  |  | x | x | x |  | x | x |  | x | x | x | x |  | x | x | x |  |  |  |  |  | x | x | x |  | H123 |  | x | x | x | x |  |  | x | x |  | x | x | x | x |  | x | x | x | x | x | x |  |  | x | x | x
H50 |  | x | x | x | x | x |  | x | x | o | x | x | x | x | x | x | x | x | x | x | x |  |  | x | x | x |  | H129 |  | x | x | x | x | x |  | x | x |  | x | x | x | x |  |  | x | x | x | x | x |  |  | x |  |
H52 |  | x | x | x | x | x |  | x | x |  | x | x | x | x | x | x | x | x | x | x | x | x | x | x |  | x |  | H132 |  | x | x | x | x |  |  | x | x |  | x | x | x | x |  | x | x | x | x | x | x |  |  | x |  | x
H55 |  | x | x | x | x | x |  | x | x | o | x | x | x | x | x | x | x | x | x | x | x | x |  | x | x | x |  | H134 |  | x | x | x | x |  |  | x | x |  | x | x | x | x | x | x | x | x | x | x | x | x |  | x | x | x
H58 |  | x | x | x | x | x |  | x | x | o | x | x | x | x |  | x | x | x | x | x | x |  | x | x |  |  |  | H135 |  | x | x | x | x |  |  | x | x |  | x | x | x | x |  | x | x | x | x | x | x |  |  |  |  | x
H62 |  | x | x | x | x | x |  | x | x |  | x | x | x | x | x | x | x | x | x | x | x | x |  | x |  | x |  | H139 |  | x | x | x | x |  |  | x | x |  | x | x | x | x |  | x | x | x | x | x | x |  |  | x |  | x
H64 |  | x | x | x | x | x |  | x | x |  | x | x | x | x |  | x | x | x | x | x | x |  |  | x |  | x |  | H141 |  |  | x | x | x |  |  | x | x |  |  | x | x | x |  |  |  | x | x | x | x |  |  | x |  | x
H65 |  | x | x | x | x | x |  | x | x |  | x | x | x | x |  | x | x | x | x | x | x |  |  | x |  | x |  | H159 |  | x | x | x | x |  |  | x | x |  | x | x | x | x |  |  |  | x | x | x | x |  |  | x | x |
H66 |  | x | x | x | x | x |  | x | x |  | x | x | x | x | x | x | x | x | x | x | x |  | x | x |  | x |  | H162 |  | x | x | x | x |  |  | x | x |  | x | x | x | x |  | x | x | x | x | x | x |  |  | x | x | x
H68 |  | x | x | x | x | x |  | x | x | x | x | x | x | x |  | x | x | x | x | x | x |  |  | x |  |  |  | H173 |  | x | x | x | x |  |  | x | x |  | x | x | x | x |  | x | x | x | x | x |  |  |  | x |  | x

##  Appendix I Kiwi-GA output summaries

Summaries of the Kiwi-GA output of the optical + UV runs are presented in Fig.
I.20 to I.75. For each star we show line profiles and fitness distributions.

Figure I.20: Kiwi-GA output summary for the optical+UV run of R136a1 (as
Figure 7).

Figure I.21: Kiwi-GA output summary for the optical+UV run of R136a2 (as
Figure 7).

Figure I.22: Kiwi-GA output summary for the optical+UV run of R136a3 (as
Figure 7). For this model, the $T_{\rm eff}$ was fixed to 50000 K. See Section
E.3 for more details and a comparison of the 50000 K and the 42000 K models.

Figure I.23: Kiwi-GA output summary for the optical+UV run of R136a4 (as
Figure 7).

Figure I.24: Kiwi-GA output summary for the optical+UV run of R136a5 (as
Figure 7).

Figure I.25: Kiwi-GA output summary for the optical+UV run of R136a6 (as
Figure 7).

Figure I.26: Kiwi-GA output summary for the optical+UV run of R136a7 (as
Figure 7).

Figure I.27: Kiwi-GA output summary for the optical+UV run of R136a8 (as
Figure 7).

Figure I.28: Kiwi-GA output summary for the optical+UV run of R136b (as Figure
7).

Figure I.29: Kiwi-GA output summary for the optical+UV run of H30 (as Figure
7).

Figure I.30: Kiwi-GA output summary for the optical+UV run of H31 (as Figure
7).

Figure I.31: Kiwi-GA output summary for the optical+UV run of H35 (as Figure
7).

Figure I.32: Kiwi-GA output summary for the optical+UV run of H36 (as Figure
7).

Figure I.33: Kiwi-GA output summary for the optical+UV run of H40 (as Figure
7).

Figure I.34: Kiwi-GA output summary for the optical+UV run of H45 (as Figure
7).

Figure I.35: Kiwi-GA output summary for the optical+UV run of H46 (as Figure
7).

Figure I.36: Kiwi-GA output summary for the optical+UV run of H47 (as Figure
7).

Figure I.37: Kiwi-GA output summary for the optical+UV run of H48 (as Figure
7).

Figure I.38: Kiwi-GA output summary for the optical+UV run of H49 (as Figure
7).

Figure I.39: Kiwi-GA output summary for the optical+UV run of H50 (as Figure
7).

Figure I.40: Kiwi-GA output summary for the optical+UV run of H52 (as Figure
7).

Figure I.41: Kiwi-GA output summary for the optical+UV run of H55 (as Figure
7).

Figure I.42: Kiwi-GA output summary for the optical+UV run of H58 (as Figure
7).

Figure I.43: Kiwi-GA output summary for the optical+UV run of H62 (as Figure
7).

Figure I.44: Kiwi-GA output summary for the optical+UV run of H64 (as Figure
7).

Figure I.45: Kiwi-GA output summary for the optical+UV run of H65 (as Figure
7).

Figure I.46: Kiwi-GA output summary for the optical+UV run of H66 (as Figure
7).

Figure I.47: Kiwi-GA output summary for the optical+UV run of H68 (as Figure
7).

Figure I.48: Kiwi-GA output summary for the optical+UV run of H69 (as Figure
7).

Figure I.49: Kiwi-GA output summary for the optical+UV run of H70 (as Figure
7).

Figure I.50: Kiwi-GA output summary for the optical+UV run of H71 (as Figure
7).

Figure I.51: Kiwi-GA output summary for the optical+UV run of H75 (as Figure
7).

Figure I.52: Kiwi-GA output summary for the optical+UV run of H78 (as Figure
7).

Figure I.53: Kiwi-GA output summary for the optical+UV run of H80 (as Figure
7).

Figure I.54: Kiwi-GA output summary for the optical+UV run of H86 (as Figure
7).

Figure I.55: Kiwi-GA output summary for the optical+UV run of H90 (as Figure
7).

Figure I.56: Kiwi-GA output summary for the optical+UV run of H92 (as Figure
7).

Figure I.57: Kiwi-GA output summary for the optical+UV run of H94 (as Figure
7).

Figure I.58: Kiwi-GA output summary for the optical+UV run of H73 (as Figure
7).

Figure I.59: Kiwi-GA output summary for the optical+UV run of H143 (as Figure
7).

Figure I.60: Kiwi-GA output summary for the optical+UV run of H108 (as Figure
7).

Figure I.61: Kiwi-GA output summary for the optical+UV run of H112 (as Figure
7).

Figure I.62: Kiwi-GA output summary for the optical+UV run of H114 (as Figure
7).

Figure I.63: Kiwi-GA output summary for the optical+UV run of H116 (as Figure
7).

Figure I.64: Kiwi-GA output summary for the optical+UV run of H120 (as Figure
7).

Figure I.65: Kiwi-GA output summary for the optical+UV run of H121 (as Figure
7).

Figure I.66: Kiwi-GA output summary for the optical+UV run of H123 (as Figure
7).

Figure I.67: Kiwi-GA output summary for the optical+UV run of H129 (as Figure
7).

Figure I.68: Kiwi-GA output summary for the optical+UV run of H132 (as Figure
7).

Figure I.69: Kiwi-GA output summary for the optical+UV run of H134 (as Figure
7).

Figure I.70: Kiwi-GA output summary for the optical+UV run of H135 (as Figure
7).

Figure I.71: Kiwi-GA output summary for the optical+UV run of H139 (as Figure
7).

Figure I.72: Kiwi-GA output summary for the optical+UV run of H141 (as Figure
7).

Figure I.73: Kiwi-GA output summary for the optical+UV run of H159 (as Figure
7).

Figure I.74: Kiwi-GA output summary for the optical+UV run of H162 (as Figure
7).

Figure I.75: Kiwi-GA output summary for the optical+UV run of H173 (as Figure
7).


# Robustness Evaluation of Transformer-based Form Field Extractors
via Form Attacks

Le Xue , Mingfei Gao ∗, Zeyuan Chen, Caiming Xiong and Ran Xu
Salesforce Research, Palo Alto, USA
{lxue, mingfei.gao, zeyuan.chen, cxiong<EMAIL_ADDRESS>Le and Mingfei
contributed equally.

###### Abstract

We propose a novel framework to evaluate the robustness of transformer-based
form field extraction methods via form attacks. We introduce 14 novel form
transformations to evaluate the vulnerability of the state-of-the-art field
extractors against form attacks from both OCR level and form level, including
OCR location/order rearrangement, form background manipulation and form field-
value augmentation. We conduct robustness evaluation using real invoices and
receipts, and perform comprehensive research analysis. Experimental results
suggest that the evaluated models are very susceptible to form perturbations
such as the variation of field-values ($\sim$ 15% drop in F1 score), the
disarrangement of input text order($\sim$ 15% drop in F1 score) and the
disruption of the neighboring words of field-values($\sim$ 10% drop in F1
score). Guided by the analysis, we make recommendations to improve the design
of field extractors and the process of data collection.

##  1 Introduction

Forms such as invoices and receipts are essential in business workflows.
Extracting target values for fields of interest from forms (see an example in
Fig. 1) is among the most important tasks in document understanding. There are
large amounts of forms processed every day, but most current systems still
rely on human labor to manually capture field-values from massively irrelevant
information. Developing a method that automatically extracts field-values
based on understanding the forms is crucial to reduce human labor, thus
improve business efficiency.

Existing works Chiticariu et al. (2013); Schuster et al. (2013); Palm et al.
(2019); Majumder et al. (2020); Xu et al. (2020) focus on improving the
modeling of field extractors and have made great progress. However, their
evaluation paradigms are limited. First, most of the methods are evaluated
using internal datasets. Internal datasets usually have very limited
variations and are often biased towards certain data distributions due to the
constraints of the data collection process. For example, the forms might be
collected from just a few vendors in a relatively short time which leads to
similar semantics and layouts across the forms. Second, public datasets lack
for diversity in terms of both textual expression and form layouts. Take the
most frequently used dataset, SROIE Huang et al. (2019), as an example. The
fields, _company_ and _address_ , are always on the very top in all receipts.
Although the existing models achieve decent performance on these datasets, it
is difficult to know whether they can generalize well. This issue can be
solved by collecting large-scale diverse forms for evaluation, but it is very
challenging since real forms usually contain customers’ private information,
thus are not publicly accessible.

Figure 1: A form field extraction system may fail due to a slight modification
to the form. Keys (concrete text-expressions of fields) are marked in blue
boxes. Values are marked in red boxes.

To tackle this dilemma, we propose a novel framework to evaluate the
robustness of form field extractors by attacking the models using form
transformations. We consider form perturbations from both OCR level and form
level, including OCR text location/order rearrangement, form background
manipulation, and form field-value augmentation. Fourteen form transformations
are proposed to impose these attacks. Using the proposed framework, we conduct
robustness analysis on two commonly used form types, i.e., invoices and
receipts. Experimental results demonstrate that the state-of-the-art (SOTA)
methods are particularly vulnerable to form perturbations, including the
variation of field-values, the disarrangement of input text order, and the
disruption of the neighboring words of field-values. Recommendations for model
design and data collection/augmentation are made accordingly.

Our contributions are summarized as follows. First, we introduce a framework
to measure the robustness of form field extractors by attacking the models
using the proposed form transformations. To the best of our knowledge, this is
the first work studying form attacks to field extraction methods. Second, we
identify the susceptibilities of the SOTA methods by comprehensive robustness
analysis on two form types using the proposed framework and make insightful
recommendations.

Figure 2: An illustration of our evaluation pipeline. An OCR engine is first
used to extract texts and locations. 14 transformations are applied to the
texts and their locations to generate diverse form variants. Then, each
transformed set will input to a transformer-based field extractor. Robustness
evaluation results are finally generated.

##  2 Related Work

Information extraction from forms is a widely researched area. Katti et al.
(2018) and Denk and Reisswig (2019) encode each page of a form as a two-
dimensional grid and extract header and line items from it using fully
convolutional networks. DocStruct Wang et al. (2020) conducts document
structure inference by encoding the form structure as a graph-like hierarchy
of text fragments. Research works specifically focusing on form field
extraction are more related to our work. Earlier methods Chiticariu et al.
(2013); Schuster et al. (2013) relied on pre-registered templates in the
system for information extraction. Palm et al. (2019) extract field-values of
invoices via an Attend, Copy, Parse architecture. Recent methods formulate the
field extraction problem as field-value pairing Majumder et al. (2020) and
field tagging Xu et al. (2020) tasks, where transformer Vaswani et al. (2017)
based structures are used to extract informative form representation via
modeling interactions among text tokens. We focus on evaluating transformer-
based field extraction methods given their great predictive capability for the
task.

Robustness evaluation of models has received considerable attention. Errudite
Wu et al. (2019) introduces model and task agnostic principles for informative
error analysis of NLP models. Ma (2019) propose NLPAug, which contains simple
textual augmentations to improve model robustness. Some works aim at
robustness of text attacks Morris et al. (2020); Zeng et al. (2020); Kiela et
al. (2021). A recent work, Robustness Gym Goel et al. (2021), presents a
simple and extensible evaluation toolkit that unifies standard evaluation
paradigms. There are also recent methods studying robustness of visual models
Santurkar et al. (2020); Salman et al. (2020); Taori et al. (2020). To the
best of our knowledge, this work is the first one focusing on robustness
evaluation of form field extraction systems.

##  3 Preliminary: Transformer-based Form Field Extractor

We are focusing on the robustness evaluation of transformer-based form field
extractors due to their undisputedly outstanding performance. Before
discussing the robustness evaluation, we first illustrate the field extraction
pipeline.

In a standard field extraction system, an OCR engine is used to extract a set
of words, $\\{w_{1},w_{2},...,w_{N}\\}$ and their bounding box locations,
$\\{b_{1},b_{2},...,b_{N}\\}$, where $N$ indicates the total number of words.
Then, a transformer-based feature backbone is used to model the interactions
between text tokens and generate informative token representations, $f_{i}$.
Since both semantics and layouts are essential for field-value inference, we
use LayoutLM Xu et al. (2020) as the feature backbone. We also experiment with
two more transformers, i.e., BERT Devlin et al. (2019) and RoBERTa Liu et al.
(2019) which only take text as input in the appendix. Finally, a fully
connected (FC) layer is used to project the token features to field space and
generate, $s_{i}=FC(f_{i})$, where $s_{i}\in R^{1\times(M+1)}$ indicates the
predicted field score and $M$ denotes the total number of positive fields.
During training, cross entropy loss between $s_{i}$ and field label is
utilized for model optimization. During inference, a post-processing method is
applied to the predicted field scores to get the value for each field. We
follow the simple criteria to generate field-values: (1) we find the predicted
field label for each word by $\hat{fd}=\underset{c}{argmax}(s_{ic})$, where
$c$ corresponds to fields (2) by default, for each field, we only keep the
word as the value if its prediction score is the highest among all the words
and larger than a threshold ($\theta=0.1$). For fields that often include
multi-word values, e.g., address and company, we keep all the words exceeding
the threshold and group nearby ones with the same predicted field.

Evaluation Metric. End-to-end F1 score averaging over fields is used to
evaluate models. We use exact string matching between the predicted values and
the ground-truth to count true positives, false positives, and false
negatives. Precision, recall and F1 scores are obtained accordingly for each
field.

##  4 Robustness Evaluation via Form Attacks

We propose OCR level and form level transformations to attack field
extractors. As shown in Fig. 2, our transformations are performed after OCR
extraction, and the transformed data is input to a transformer-based field
extractor. An analysis is conducted via performance comparison between the
original set and transformed sets. Each transformation and the principles
behind it are introduced as follows.

###  4.1 OCR Location and Order Rearrangement

We transform the original data to meaningful variants by slightly altering the
OCR locations and the text order arrangement. These transformations simulate
scenarios where we may obtain different OCR results before inputting to field
extractors due to various reasons, e.g., the quality of OCR engines.

Center Shift and Box Stretch. To evaluate model robustness to OCR text
location jittering, we propose two word-level location transformations. In
_Center Shift_ , we keep the box size and randomly shift the center of a box.
The shifting is in proportional to the width (horizontally) and height
(vertically) of the box, and the ratio is a random number drawn from a normal
distribution, $\mathcal{N}(0,\delta_{cent})$. _Box Stretch_ randomly changes
the four coordinates of a box in a similar way using
$\mathcal{N}(0,\delta_{xy})$.

Margin Padding. Scanning forms may introduce white margins, which globally
changes text locations. We use _Margin Padding_ to manipulate the locations of
all the words in a form. We pad white margins in the left, right, up, and down
sides of a form where the margin length is a generated random number between 1
and $r_{mp}$ of the page size.

Global Shuffle. We observe that organizing a transformer’s inputs in reading
order is particularly beneficial to understanding the form structure. However,
the reading order is not always guaranteed by OCR engines. Hence, it is
interesting to investigate model robustness to poor reading order quality. We
use _Global Shuffle_ to shuffle the order of words before inputting to
transformers. Note that the words and their locations are not changed at all
and the only difference is the order of the word sequence input to the
transformer.

Neighbor Shuffle and Non-neighbor Shuffle. Intuitively, local neighbors of a
value make more contributions to its prediction. So, we propose _Neighbor
Shuffle_ which shuffles the order of each value’s neighbors and keeps the
order of the rest. Oppositely, we also have _Non-neighbor Shuffle_. A word,
$w_{i}$, is defined as a value’s neighbor if the IoU between $b_{i}$ and the
neighbor zone of the value is larger than $0.5$. The neighbor zone is a box
that shares the same center as the value box with expanded width and height
(expand rate denoted as $r_{nb}$). We also include $n_{nb}$ nearby words from
the original reading order as the neighbors.

###  4.2 Form Background Manipulation

Form background generally affects the model performance in two ways: (1) some
background words are strong indicators to improve field-values’ recall and (2)
accurate prediction of background reduces false positives, thus increase model
precision. We propose the following transformations to evaluate model
robustness to background perturbations.

BG Drop. Background (BG) Drop mimics the scenario that some words are
completely missed by OCR detection. This transformation removes background
words together with the corresponding boxes at a probability of $p_{bgd}$.

Neighbor BG Drop is similar to _Neighbor Shuffle_ , which drops all background
words if they are neighbors of a field-value.

Key Drop. Keys are concrete text-representations of fields in a form. For
example, the field _invoice_number_ may be represented as "INV #", "Invoice
No." etc. in a form. A key is a very important feature for value localization
since the value is often located near the key. We propose _Key Drop_ to see
the model performance change if keys are accidentally missed by OCR detection.

BG Typo. OCR recognition usually makes errors. _BG Typo_ simulates word-level
string typos. We select each background word at a probability of $p_{typo}$.
For each selected word, we apply one of the error types, including swapping,
deleting, adding, and replacing a random or a specific character. 111We
utilize the implementation of the string typos provided in
https://pypi.org/project/typo/.

BG Synonyms. Similar semantics may be represented using different word
synonyms. _BG Synonyms_ randomly replaces each background word at a
probability of $p_{bgs}$ with their synonyms.222We generate word synonyms
using WordNet Interface (https://www.nltk.org/howto/wordnet.html).

BG Adversarial. Some forms contain only one word in the same data type as
field-values. For example, there might be only one word with the date type of
_date_. It is less challenging for a model to recognize it as the
_invoice_date_. However, this type of easy case is not always guaranteed in
real-world applications. _BG Adversarial_ is used to increase the difficulty
level by adding distraction. Concretely, we select background words at a
probability of $p_{bga}$ and use adversarial words for replacement. For each
replacement, we randomly choose a data type and then generate a random value
of the corresponding data type. We focus on three data types, i.e., _date_ ,
_number_ and _money_. We generate random dates using Faker.333
https://faker.readthedocs.io/en/master/ For numbers, we first generate the
number length randomly and then a random number of the length accordingly. For
_money_ , we obtain a random amount within lower and upper bounds. Then, we
make the amount in money format, where we add a decimal point at the second to
the left digit, place a comma at every third digit to the left of the decimal
point and randomly insert $\$$ at the beginning. To protect strong indicators
for values, neighbor words are not replaced.

###  4.3 Form Field-Value Augmentations

Modifying field-values is a more direct way to increase the diversity of the
evaluation set. We augment field-values in both text and locations.

Value Text Augment. Field-values of forms may be biased due to the limitation
of the data collection process. For example, the _invoice_date_ may be
restricted to the year the form is collected, and the _invoice_number_ may be
biased towards the vendor’s numbering system. _Value Text Augment_
transformation targets at augmenting the field-values based on their data
types. For each field-value, we randomly generate a substitute with the same
data type following the same value generation procedure as we do in _BG
Adversarial_.

Value Location Augment. Form layouts can be very diverse in real-world
scenarios. Intuitively, we should be able to infer a field-value as long as a
key is represented properly no matter where we place the key-value pair in the
document. We introduce  _Value Location Augment_ to increase layout diversity.
To maintain the form format to the most, we keep the background as it is and
shuffle the key-value pair’s locations in the form. For example, for the field
_invoice_number_ (key: Invoice No., value: 1234) and _invoice_date_ (key:
Invoice date, value: 01/01/2021), we swap the box locations of “Invoice No."
and “Invoice date", and also the locations of “1234" and “01/01/21".

##  5 Experiments

We evaluate the robustness of transformation-based field extractors using our
framework on two commonly used form types, i.e., invoices and receipts.

###  5.1 Datasets

Our evaluation models are trained using a labeled train set, and the best
performing model is picked based on a validation set. We prepare a separate
test set to perform the robustness evaluation. To perform the proposed
transformations, we annotate both the key and value of each field of interest
with their bounding box locations in a form.

Invoice. The train, valid, and test sets contain 158, 348 and 338 real
invoices. They are collected from 111, 222, and 222 vendors, respectively. We
sample at most 5 forms from the same vendor and the vendors of train, valid
and test sets do not overlap. We consider 7 frequently used fields including
_invoice_number_ , _purchase_order_ , _invoice_date_ , _due_date_ ,
_amount_due_ , _total_amount_ and _total_tax_.

Receipt. We use the publicly available receipt dataset, SROIE. The annotations
of their original test set are not publicly available, so we split the
original train set to train, valid, and test sets based on their company names
and sample at most 5 forms per company following Majumder et al. (2020).
Finally, we get 237 receipts for training, 76 for validation, and 74 for
testing. The fields of interest are _company_ , _address_ , _date_ and
_total_. We add value boxes and annotate keys according to the text-level
annotations provided by the original dataset.

###  5.2 Implementation Details

Our evaluation framework is implemented using Pytorch and the experiments are
conducted on a single Tesla V100 GPU. The strength of a transformation is
controlled by parameters. We set parameters to make moderate perturbations. In
_BG Typo_ , _BG Drop_ , _BG Synonyms_ and _BG Adversarial_ , transformations
are only applied to some selected words. We fix the pre-defined probabilities,
$p_{typo},p_{bgs},p_{bga}$, to 0.1. $\theta_{center}$ is set to 0.5 and
$\theta_{xy}$ is 0.1. $r_{mp}$ in Margin Padding is set to 0.3. When
determining the value’s neighbor zone, we set the expand rate as $r_{nb}=0.02$
and $n_{nb}=2$.

We generate random values based on data types in _BG Adversarial_ and _Value
Text Augment_. For dates, we randomly pick a date from the year 2001 to 2021
in one of the formats, including mm/dd/yy, yy-mm-dd, dd/month/yy, and
dd/mon/yy. For numbers, the number length is randomly generated from 3 to 12.
The amount of money is randomly selected from 1 to 10,000,000.

We use a commercial OCR engine444https://api.einstein.ai/signup for OCR
extraction and utilize Tesseract555https://github.com/tesseract-ocr/tesseract
to rank the words in reading order. Our default transformer is LayoutLM Xu et
al. (2020) with text and boxes as inputs. We also evaluate using BERT Devlin
et al. (2019) and RoBERTa Liu et al. (2019) that take only text tokens as the
inputs in Sec. A. All models are finetuned from the corresponding base models.
During training, we set the batch size to 8 and use the Adam optimizer with a
learning rate of 5$e^{-5}$.

###  5.3 Robustness Evaluation of Invoices

Transforms | Precision | Recall | F1
---|---|---|---
Original | 70.9 | 69.3 | 70.0
Center Shift | 70.8 | 69.9 | 70.3
Box Stretch | 70.0 | 68.5 | 69.1
Margin Pad | 69.6 | 68.3 | 68.9
Table 1: Evaluation of robustness to OCR location modifications on invoices.
LayoutLM is used as the transformer model.

OCR location modification. Robustness evaluation of the LayoutLM model to OCR
text location jittering is shown in Table 1. We obtain comparable results to
the original performance when applying _Center Shift_ and _Box Stretch_ which
indicates that slight box jittering is tolerable in our case. _Margin Pad_
shifts all text locations by adding random margins around a form. This
transformation also just slightly decrease the model performance.

Transforms | Precision | Recall | F1
---|---|---|---
Original | 70.9 | 69.3 | 70.0
Global Shfl. | 58.5 | 53.9 | 55.9
Neighbor Shfl. | 66.9 | 61.9 | 64.1
Non-neighbor Shfl. | 67.0 | 65.3 | 65.9
Table 2: Evaluation of robustness to OCR text order on invoices. LayoutLM is
used as the transformer model.

OCR text order is essential to transformer-based field extractors since the
order can serve as an important feature to improve model performance. We show
the model performance when applying order shuffle to different places in Table
2. The results show that if we shuffle all text orders, the performance drops
dramatically by 14.1% in F1 score. When only shuffle value neighbors, we
obtain $\sim$6% lower F1 score. We get $\sim$4% lower F1 score if we shuffle
non-neighbor words, even though we keep the text order of all values and their
neighbors the same. The results demonstrate the importance of the text order.
If the model is trained using texts with a good reading order, we may also
want to ensure a good reading order during inference.

A natural question is, what if we break the reading orders during training.
Will this help us save the effort of ensuring test reading order during
inference? We re-train field extractors using texts with random orders. We
obtain 58.7% in F1 score, which is 11.3% lower than our baseline. The
comparison result suggests that the text reading order is a very important
feature. How to use it without overfitting to it is an interesting research
topic.

Transforms | Precision | Recall | F1
---|---|---|---
Original | 70.9 | 69.3 | 70.0
BG Drop | 69.8 | 66.2 | 67.7
Neighbor BG Drop | 67.6 | 57.1 | 61.3
Key Drop | 62.8 | 56.4 | 59.1
BG Typo | 69.6 | 66.9 | 68.1
BG Synonyms | 70.4 | 68.8 | 69.5
BG Adversarial | 66.2 | 67.7 | 66.9
Table 3: Evaluation of robustness to BG manipulations on invoices. LayoutLM is
used as the transformer model.

Background drop related transformations simulate the scenarios where an OCR
detector accidentally misses some background words. _BG Drop_ removes words
randomly selected in the background. As shown in Table 3, global _BG Drop_
leads to slight performance decrease. When we apply _Neighbor BG Drop_ , the
performance largely drops by $\sim$9% in F1 score. Comparing _BG Drop_ and
_Neighbor BG Drop_ , we find that neighbor words are indeed more important for
value extraction. For a fair comparison, we have adjusted the dropping rate of
_BG Drop_ to 0.13, such that the total number of dropped words is roughly
equal to the number of neighbor words. Further, _Key Drop_ results in a
similar performance decrease as _Neighbor BG Drop_ , although the total number
of Keys is less than half of that for neighbor words.

Transforms | Precision | Recall | F1
---|---|---|---
Original | 70.9 | 69.3 | 70.0
Value Text Aug. | 56.3 | 53.5 | 54.5
Value Location Aug. | 61.4 | 56.8 | 58.8
Table 4: Evaluation of robustness to background and field-value augmentations
on invoices. LayoutLM is used as the transformer model.

Other background manipulations. Some background words, e.g., keys and other
useful indicators, are important features to localize values. Adding typos to
the background words may be harmful to these good features, thus _BG Typo_
leads to a 2.4% drop in recall rate.

Forms from different vendors may use different words even when represent
similar semantics. We attack the models using _BG Synonyms_. As shown in Table
3, our field extractors are quite robust to this transformation with only a
negligible drop in F1 score.

_BG Adversarial_ is used to add background words (serve as distractions) with
similar data types as field-values. As shown in Table 3, _BG Adversarial_
leads to 4.7% drop in model precision.

Value augmentations. Field-values of forms may be limited in diversity. _Value
Text Augment_ transformation augments field-values by replacing them with
randomly generated values in the same data type. We augment the values of all
the fields, except for _total_amount_ and _amount_due_ , since these two
fields may involve complicated mathematical computations. For _total_tax_ , we
randomly select a number between 0 and 15% of the _total_amount_. The
comparison results in Table 4 show that the model performance drop
significantly by 15.5% F1 score.

_Value Location Augment_ changes the spatial arrangements of key-value pairs.
In practice, we only shuffle the key-value pairs if they have the same number
of key words and the same number of value words, resulting in more than 75%
key-value pairs relocated. The results in Table 4 demonstrate that _Value
Location Augment_ significantly reduces F1 scores by 11.2%.

Multiple transformations. The proposed transformations can be combined
together to generate more diverse sets. We conduct exhaustive combinations of
every two and three transformations which result in 91 2-transformation
combinations and 364 3-transformation combinations.666We observe that
different orders of transformations in a combination result in ignorable
differences. The top-10 most impactful combinations are shown in Fig. 3. The
comparison results suggest the following conclusions.

Figure 3: Top-10 most impactful 2-transformation and 3-transformation
combinations. VTA: Value Text Augment, GS: Global Shift, VLA: Value Location
Augment, KD: Key Drop, NBD: Neighbor Background Drop, BA: Background
Adversarial, MP: Margin Pad, BT: Background Typo, NS: Neighbor Shuffle, NNS:
Non-Neighbor Shuffle.

First, generally if an individual transformation drops more performance, it
also contributes more drop when combined with other transformations. The most
impactful combination is (_Value text Augment_ , _Global Shuffle_ , _Value
Location Augment_) with a F1 score of 25.7. They are the top-3 impactful
transformations suggested in Fig. 4.

Second, some individual transformations are less impactful, but they affect
more when combined with some specific transformations. For example, individual
_Margin Pad_ ranks low in Fig. 4. However, it leads to more performance drop
when combined with _Value Text Augment_ and _Global Shuffle_. Their
performance drop ranks 5 out of 364 combinations (F1 is 30.8). This may due to
that _Margin Pad_ (changes all words’ locations), _Value Text Augment_
(changes values’ texts) and _Global Shuffle_ (changes text input order) are
three complementary transformations. When we do _Margin Pad_ alone, the model
resorts to the information of value texts and text orders. However, when we do
these three transformations together, the model becomes inevitably confused.

Third, if the transformations have overlapping effects, their combination has
a lower impact. For example, _Key Drop_ , _Neighbor BG Drop_ and _Neighbor
Shuffle_ all manipulate neighbor words. The performance drop on their
combination ranks 246 out of 364 combinations although their individual
transformation is impactful (see Fig.4). The F1 score is 58.1 which is very
close to an individual _Key Drop_ transformation (59.1).

Figure 4: An overview of LayoutLM-based model performance drop due on
different transformed dataset. The results are sorted by the performance gap.
Figure 5: An illustration of _Value Location Augment*_ transformation on a
receipt in SROIE dateset.

###  5.4 Robustness Evaluation of Receipts

There are two interesting features of the SROIE dataset. First, a significant
amount of field-values have no keys, for example, all values of _company_ and
_address_ , and some values of _date_ and _total_. Consequently, changing the
context of values has a minor effect on model performance. Second, the layouts
of different receipts are very fixed. For example, _company_ and _address_ are
always on the very top of every receipt. So, models could easily overfit to
field-value locations and the text order. As shown in Table 5, _Global
Shuffle_ leads to significant performance drop by 38.7% F1 score.
Specifically, the fields of _address_ and _company_ become 0% F1 score when
the text’s order is completely shuffled before inputting to the transformer.
The results demonstrate that the model is overfitting to the input text order,
especially for _address_ and _company_.

Transforms | Precision | Recall | F1
---|---|---|---
Original | 81.8 | 80.1 | 80.9
Global Shfl. | 42.5 | 41.9 | 42.2
Value Location Aug.* | 54.3 | 48.0 | 49.7
Value Text Aug. | 75.2 | 73.6 | 74.4
Table 5: Robustness evaluation on SROIE dataset. LayoutLM is used as the
transformer model.

Most of the fields in SROIE have no keys. To augment receipt layouts, we
design a dedicated method that locally moves field-value locations.
Specifically, for _company_ and _address_ in SROIE receipts, we move the
values to the bottom of the form and shift the rest above to fill the gap as
shown in Fig. 5. We refer to this transformation as _Value Location Augment*_.
This transformation changes the location of the values without breaking the
text order within each value. We obtain 21.6% and 1.7% F1 score for _company_
and _address_ , respectively, which are around 60% and 68% lower than the
original numbers.

Besides, we also evaluate models on test set transformed by _Value Text
Augment_. We replace values of _company_ , _address_ and _date_ using
substitute randomly generated by Faker. Same as what we do for _total_amount_
and _amount_due_ for invoices, we keep the values of _total_ as they are. To
maintain the layout structure, we only replace _company_ and _address_ if we
are able to get a randomly generated sample with the same number of words as
the original sample. This results in about 69% _company_ values and 31%
_address_ values changed, respectively. As shown in Table 5, the _Value Text
Augment_ largely decreases the model performance by 5.5% in F1 score.

###  5.5 Observations and Suggestions

An overview comparison of all the transformations of invoices is summarized in
Fig. 4. As we can see, the top-3 substantial transformations are _Value Text
Augment_ , _Global Shuffle_ and _Value Location Augment_. Experiments on
receipts also show the effectiveness of these three transformations.

We make the following recommendations based on the analysis. For data
collection/augmentation, forms with more diverse values are preferable. For
example, we may want _dates_ covering a wide range of time periods with more
types of formats and _numbers_ being more extensive. Varying forms’ layouts is
also beneficial. Especially, we may want to focus on varying the arrangement
of field-values instead of altering individual word locations locally.

For the design of field extractors, we suggest making better utilization of
the text order. As shown in our experiments, the text order is a very useful
feature. How to utilize the text reading order without overfitting to it is an
interesting topic. Besides, _Key Drop_ and _Neighbor BG Drop_ result in
significant performance decreases as shown in Fig. 4. This suggests that
value’s neighbors, especially the keys, are essential for value extractions.
Current state-of-the-art models use transformers to model interactions between
all words. We believe paying attention to keys and neighbors in the model
design has the potential to improve the existing field extraction systems.

##  6 Conclusion and Future Work

We proposed a novel framework to evaluate the robustness of transformer-based
form field extractors via form attacks. We introduced 14 transformations that
transform forms in different aspects, including OCR-level location and order,
background contexts, and field-value text and layouts. We conducted studies on
real invoices and receipts with three types of transformer-based models using
our proposed framework. Research recommendations were made based on the
robustness analysis.

Improving field extraction from forms using the research analysis generated by
the robustness evaluation is a very meaningful research area. The proposed
transformations are potentially useful for increasing the diversity of
training samples, thus improving model robustness. We will consider this in
the future work.

##  7 Broader Impact

This work targets at robustness evaluation of form information extraction
systems, so it has positive impacts such as identifying bias of existing
information extractors and improving the fairness of model comparison. On the
opposite side, our method may have unintended negative consequences in that we
have proposed transformations that evaluate various aspects of model
robustness, but the metrics we have selected may not be comprehensive. As a
result, there is likely some degree of model bias present that has been missed
by the proposed framework. However, this negative impact is not specific to
our work and should be considered in general in the field of robustness AI.

The invoice dataset is for internal use only and does not contain any
personally identifiable data. The SROIE dataset is a public dataset under MIT
license. All forms were annotated by the authors. Consequently, we are
confident that the datasets do not have ethical issues.

## References

  * Chiticariu et al. (2013) Laura Chiticariu, Yunyao Li, and Frederick R. Reiss. 2013.  Rule-based information extraction is dead! long live rule-based information extraction systems!  In _EMNLP_.
  * Denk and Reisswig (2019) Timo I Denk and Christian Reisswig. 2019.  Bertgrid: Contextualized embedding for 2d document representation and understanding.  _NeurIPS Workshop_.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _NAACL_.
  * Goel et al. (2021) Karan Goel, Nazneen Rajani, Jesse Vig, Samson Tan, Jason Wu, Stephan Zheng, Caiming Xiong, Mohit Bansal, and Christopher Ré. 2021.  Robustness gym: Unifying the nlp evaluation landscape.  _arXiv preprint arXiv:2101.04840_.
  * Huang et al. (2019) Zheng Huang, Kai Chen, Jianhua He, Xiang Bai, Dimosthenis Karatzas, Shijian Lu, and CV Jawahar. 2019.  Icdar2019 competition on scanned receipt ocr and information extraction.  In _ICDAR_.
  * Katti et al. (2018) Anoop Raveendra Katti, Christian Reisswig, Cordula Guder, Sebastian Brarda, Steffen Bickel, Johannes Höhne, and Jean Baptiste Faddoul. 2018.  Chargrid: Towards understanding 2d documents.  _EMNLP_.
  * Kiela et al. (2021) Douwe Kiela, Max Bartolo, Yixin Nie, Divyansh Kaushik, Atticus Geiger, Zhengxuan Wu, Bertie Vidgen, Grusha Prasad, Amanpreet Singh, Pratik Ringshia, et al. 2021.  Dynabench: Rethinking benchmarking in nlp.  _NAACL_.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Roberta: A robustly optimized bert pretraining approach.  _arXiv preprint arXiv:1907.11692_.
  * Ma (2019) Edward Ma. 2019.  Nlp augmentation.  https://github.com/makcedward/nlpaug.
  * Majumder et al. (2020) Bodhisattwa Prasad Majumder, Navneet Potti, Sandeep Tata, James Bradley Wendt, Qi Zhao, and Marc Najork. 2020.  Representation learning for information extraction from form-like documents.  In _ACL_.
  * Morris et al. (2020) John X Morris, Eli Lifland, Jin Yong Yoo, and Yanjun Qi. 2020.  Textattack: A framework for adversarial attacks in natural language processing.  _arXiv preprint arXiv:2005.05909_.
  * Palm et al. (2019) Rasmus Berg Palm, Florian Laws, and Ole Winther. 2019.  Attend, copy, parse end-to-end information extraction from documents.  In _ICDAR_.
  * Salman et al. (2020) Hadi Salman, Andrew Ilyas, Logan Engstrom, Ashish Kapoor, and Aleksander Madry. 2020\.  Do adversarially robust imagenet models transfer better?  _NeurIPS_.
  * Santurkar et al. (2020) Shibani Santurkar, Dimitris Tsipras, and Aleksander Madry. 2020.  Breeds: Benchmarks for subpopulation shift.  _arXiv preprint arXiv:2008.04859_.
  * Schuster et al. (2013) Daniel Schuster, Klemens Muthmann, Daniel Esser, Alexander Schill, Michael Berger, Christoph Weidling, Kamil Aliyev, and Andreas Hofmeier. 2013.  Intellix – end-user trained information extraction for document archiving.  In _ICDAR_.
  * Taori et al. (2020) Rohan Taori, Achal Dave, Vaishaal Shankar, Nicholas Carlini, Benjamin Recht, and Ludwig Schmidt. 2020.  Measuring robustness to natural distribution shifts in image classification.  _NeurIPS_.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  _NeurIPS_.
  * Wang et al. (2020) Zilong Wang, Mingjie Zhan, Xuebo Liu, and Ding Liang. 2020.  Docstruct: A multimodal method to extract hierarchy structure in document for general form understanding.  _EMNLP_.
  * Wu et al. (2019) Tongshuang Wu, Marco Tulio Ribeiro, Jeffrey Heer, and Daniel S Weld. 2019.  Errudite: Scalable, reproducible, and testable error analysis.  In _ACL_.
  * Xu et al. (2020) Yiheng Xu, Minghao Li, Lei Cui, Shaohan Huang, Furu Wei, and Ming Zhou. 2020.  Layoutlm: Pre-training of text and layout for document image understanding.  In _KDD_.
  * Zeng et al. (2020) Guoyang Zeng, Fanchao Qi, Qianrui Zhou, Tingji Zhang, Bairu Hou, Yuan Zang, Zhiyuan Liu, and Maosong Sun. 2020.  Openattack: An open-source textual adversarial attack toolkit.  _arXiv preprint arXiv:2009.09191_.

##  Appendix A Appendix

###  A.1 More Robustness Evaluation of Invoices

The results of BERT and RoBERTa on invoices are summarized in Table A1 and
Table A2.

Transforms | Precision | Recall | F1
---|---|---|---
Original | 58.4 | 58.1 | 57.8
Global Shfl. | 40.5 | 38.2 | 38.6
Neighbor Shfl. | 51.5 | 50.7 | 50.6
Non-neighbor Shfl. | 57.9 | 56.4 | 56.6
BG Drop | 57.7 | 57.2 | 56.9
Neighbor BG Drop | 49.9 | 44.7 | 46.7
Key Drop | 50.2 | 46.5 | 47.6
BG Typo | 56.0 | 54.1 | 54.7
BG Synonyms | 58.8 | 58.3 | 58.0
BG Adversarial | 48.7 | 51.6 | 49.7
Value Text Aug. | 36.2 | 33.8 | 34.3
Value Location Aug. | 55.5 | 54.4 | 54.4
Table A1: Robustness evaluation on invoice dataset. BERT is used as the transformer model. Transforms | Precision | Recall | F1
---|---|---|---
Original | 63.4 | 59.1 | 61.1
Global Shfl. | 40.5 | 33.3 | 36.4
Neighbor Shfl. | 58.9 | 53.2 | 55.9
Non-neighbor Shfl. | 60.6 | 57.0 | 58.7
BG Drop | 63.6 | 58.8 | 61.0
Neighbor BG Drop | 58.8 | 49.5 | 53.5
Key Drop | 57.7 | 49.8 | 53.2
BG Typo | 62.1 | 57.7 | 59.8
BG Synonyms | 62.7 | 58.5 | 60.4
BG Adversarial | 56.2 | 55.2 | 55.5
Value Text Aug. | 44.9 | 39.2 | 41.5
Value Location Aug. | 57.9 | 53.9 | 55.7
Table A2: Robustness evaluation on invoice dataset. RoBERTa is used as the
transformer model.

OCR text order. BERT and RoBERTa do not have text locations as inputs, so they
rely more on text order than LayoutLM does. When deploying _Global Shuffle_ ,
we observe more performance drop, i.e., by 19.2% F1 score for BERT and by
24.7% F1 score for RoBERTa.

Background drop. We observe similar results on BERT and RoBERTa as compared to
that on LayoutLM. _Key Drop_ and _Neighbor BG Drop_ have more impact than
global _BG Drop_.

Other background manipulations. We observe 3.1% and 1.3% drop in F1 score on
_BG Typo_ , when using BERT and RoBERTa. Similar to LayoutLM, BERT and RoBERTa
are robust to _BG Synonyms_. _BG Adversarial_ leads to 9.7% and 7.2% drop in
model precision for BERT and RoBERTa based methods, respectively.

Value augmentations. _Value Text Augment_ results in 23.5% (BERT) and 19.6%
(RoBERTa) drop in F1 score. Since BERT and RoBERTa do not rely on location
input, the performance drop on _Value Location Augment_ of these two models is
much less than that of the LayoutLM (drop 11.2%).

###  A.2 More Robustness Evaluation of Receipts

The results of BERT and RoBERTa on receipts are summarized in Table A3 and
Table A4. The experimental results suggest that these three transformations
have significant impact to the performance of BERT and RoBERTa.

Transforms | Precision | Recall | F1
---|---|---|---
Original | 75.4 | 73.3 | 74.3
Global Shfl. | 37.9 | 36.1 | 37.0
Value Location Aug.* | 51.9 | 44.9 | 46.0
Value Text Aug. | 71.2 | 69.2 | 70.2
Table A3: Robustness evaluation on SROIE dataset. BERT is used as the transformer model. Transforms | Precision | Recall | F1
---|---|---|---
Original | 77.3 | 74.7 | 75.9
Global Shfl. | 40.6 | 37.1 | 38.8
Value Location Aug.* | 48.6 | 41.9 | 43.4
Value Text Aug. | 72.5 | 70.6 | 71.5
Table A4: Robustness evaluation on SROIE dataset. RoBERTa is used as the
transformer model.


* Meftouh et al. (2015) Karima Meftouh, Salima Harrat, Salma Jamoussi, Mourad Abbas, and Kamel Smaili. 2015.  Machine Translation Experiments on PADIC: A Parallel Arabic DIalect Corpus. In _Proceedings of the 29th Pacific Asia Conference on Language, Information and Computation_. Shanghai, China, 26–34.  https://aclanthology.org/Y15-1004
  * Merrison et al. (2012) Andrew John Merrison, Jack J Wilson, Bethan L Davies, and Michael Haugh. 2012.  Getting stuff done: Comparing e-mail requests from students in higher education in Britain and Australia.  _Journal of pragmatics_ 44, 9 (2012), 1077–1098.
  * Meyer (2014) Erin Meyer. 2014\.  _The culture map: Breaking through the invisible boundaries of global business_.  Public Affairs.
  * Moghimifar et al. (2023) Farhad Moghimifar, Shilin Qu, Tongtong Wu, Yuan-Fang Li, and Gholamreza Haffari. 2023.  NormMark: A Weakly Supervised Markov Model for Socio-cultural Norm Discovery. In _Findings of the Association for Computational Linguistics: ACL 2023_ , Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (Eds.). Association for Computational Linguistics, Toronto, Canada, 5081–5089.  https://doi.org/10.18653/v1/2023.findings-acl.314
  * Moore (1999) Bruce Moore. 1999\.  The Vocabulary of Australian English.  _Australian National Dictionary Centre, Australian National University. URL: http://andc. anu. edu. au/sites/default/files/vocab_aussie_eng. pdf_ (1999).
  * Mozafari et al. (2020) Marzieh Mozafari, Reza Farahbakhsh, and Noël Crespi. 2020\.  Hate speech detection and racial bias mitigation in social media based on BERT model.  _PloS one_ 15, 8 (2020), e0237861.
  * Mulki et al. (2019) Hala Mulki, Hatem Haddad, Mourad Gridach, and Ismail Babaoğlu. 2019.  Syntax-ignorant N-gram embeddings for sentiment analysis of Arabic dialects. In _Proceedings of the Fourth Arabic Natural Language Processing Workshop_. 30–39.
  * Nagata (2014) Ryo Nagata. 2014\.  Language Family Relationship Preserved in Non-native English. In _Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers_ , Junichi Tsujii and Jan Hajic (Eds.). Dublin City University and Association for Computational Linguistics, Dublin, Ireland, 1940–1949.  https://aclanthology.org/C14-1183
  * Nerbonne and Heeringa (1997) John Nerbonne and Wilbert Heeringa. 1997.  Measuring dialect distance phonetically. In _Computational phonology: third meeting of the acl special interest group in computational phonology_.
  * Nerbonne and Heeringa (2001) John Nerbonne and Wilbert Heeringa. 2001.  Computational comparison and classification of dialects.  (2001).
  * Nerbonne and Heeringa (2002) J Nerbonne and Wilbert Heeringa. 2002.  Computational Comparison and Classification of Dialects.  _Dialectologia et Geolinguistica, Journal of the International Society for Dialectology and Geolinguistics_ 9 (2002), 69–84.
  * Obeid et al. (2020) Ossama Obeid, Nasser Zalmout, Salam Khalifa, Dima Taji, Mai Oudah, Bashar Alhafni, Go Inoue, Fadhl Eryani, Alexander Erdmann, and Nizar Habash. 2020\.  CAMeL tools: An open source python toolkit for Arabic natural language processing. In _Proceedings of the Twelfth Language Resources and Evaluation Conference_. 7022–7032.
  * Okpala et al. (2022) Ebuka Okpala, Long Cheng, Nicodemus Mbwambo, and Feng Luo. 2022\.  AAEBERT: Debiasing BERT-based Hate Speech Detection Models via Adversarial Learning. In _2022 21st IEEE International Conference on Machine Learning and Applications (ICMLA)_. IEEE, 1606–1612.
  * Olabisi et al. (2022) Olubusayo Olabisi, Aaron Hudson, Antonie Jetter, and Ameeta Agrawal. 2022.  Analyzing the Dialect Diversity in Multi-document Summaries. In _Proceedings of the 29th International Conference on Computational Linguistics_. 6208–6221.
  * Oussous et al. (2020) Ahmed Oussous, Fatima-Zahra Benjelloun, Ayoub Ait Lahcen, and Samir Belfkih. 2020\.  ASA: A framework for Arabic sentiment analysis.  _Journal of Information Science_ 46, 4 (2020), 544–559.
  * Paul et al. (2011) Michael Paul, Andrew Finch, Paul Dixon, and Eiichiro Sumita. 2011.  Dialect translation: integrating Bayesian co-segmentation models with pivot-based SMT. In _Proceedings of the First Workshop on Algorithms and Resources for Modelling of Dialects and Language Varieties_. 1–9.
  * Plüss et al. (2023) Michel Plüss, Jan Deriu, Yanick Schraner, Claudio Paonessa, Julia Hartmann, Larissa Schmidt, Christian Scheller, Manuela Hürlimann, Tanja Samardžić, Manfred Vogel, and Mark Cieliebak. 2023\.  STT4SG-350: A Speech Corpus for All Swiss German Dialect Regions. In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (Eds.). Association for Computational Linguistics, Toronto, Canada, 1763–1772.  https://doi.org/10.18653/v1/2023.acl-short.150
  * Qiu et al. (2021) Jia Qiu, Xinren Chen, and Michael Haugh. 2021.  Jocular flattery in Chinese multi-party instant messaging interactions.  _Journal of Pragmatics_ 178 (2021), 225–241.
  * Rajai and Ennasser (2022) Al-Khanji Rajai and Narjes Ennasser. 2022.  Dealing with dialects in literary translation: Problems and strategies.  _Jordan Journal of Modern Languages and Literatures_ 14, 1 (2022), 145–163.
  * Ramponi and Casula (2023) Alan Ramponi and Camilla Casula. 2023.  DIATOPIT: A Corpus of Social Media Posts for the Study of Diatopic Language Variation in Italy. In _Tenth Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial 2023)_. 187–199.
  * Riley et al. (2023) Parker Riley, Timothy Dozat, Jan A. Botha, Xavier Garcia, Dan Garrette, Jason Riesa, Orhan Firat, and Noah Constant. 2023\.  FRMT: A Benchmark for Few-Shot Region-Aware Machine Translation.  _Transactions of the Association for Computational Linguistics_ 11 (2023), 671–685.  https://doi.org/10.1162/tacl_a_00568
  * Roberts (2021) Celia Roberts. 2021\.  Linguistic penalties and the job interview.  _(No Title)_ (2021).
  * Roy et al. (2020) Samapika Roy, Sukhada Sukhada, and Anil Kumar Singh. 2020\.  Parsing Indian English News Headlines. In _Proceedings of the 17th International Conference on Natural Language Processing (ICON)_. 239–242.
  * Saadany et al. (2022) Hadeel Saadany, Constantin Orăsan, Emad Mohamed, and Ashraf Tantawy. 2022.  A Semi-supervised Approach for a Better Translation of Sentiment in Dialectical Arabic UGT. In _Proceedings of the The Seventh Arabic Natural Language Processing Workshop (WANLP)_ , Houda Bouamor, Hend Al-Khalifa, Kareem Darwish, Owen Rambow, Fethi Bougares, Ahmed Abdelali, Nadi Tomeh, Salam Khalifa, and Wajdi Zaghouani (Eds.). Association for Computational Linguistics, Abu Dhabi, United Arab Emirates (Hybrid), 214–224.  https://doi.org/10.18653/v1/2022.wanlp-1.20
  * Sajjad et al. (2020) Hassan Sajjad, Ahmed Abdelali, Nadir Durrani, and Fahim Dalvi. 2020.  AraBench: Benchmarking Dialectal Arabic-English Machine Translation. In _Proceedings of the 28th International Conference on Computational Linguistics_. International Committee on Computational Linguistics, Barcelona, Spain (Online), 5094–5107.  https://doi.org/10.18653/v1/2020.coling-main.447
  * Salloum et al. (2014) Wael Salloum, Heba Elfardy, Linda Alamir-Salloum, Nizar Habash, and Mona Diab. 2014.  Sentence Level Dialect Identification for Machine Translation System Selection. In _Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_. Association for Computational Linguistics, Baltimore, Maryland, 772–778.  https://doi.org/10.3115/v1/P14-2125
  * Sap et al. (2019) Maarten Sap, Dallas Card, Saadia Gabriel, Yejin Choi, and Noah A Smith. 2019.  The risk of racial bias in hate speech detection. In _Proceedings of the 57th annual meeting of the association for computational linguistics_. 1668–1678.
  * Scannell (2020) Kevin Scannell. 2020\.  Universal Dependencies for Manx Gaelic. In _Proceedings of the Fourth Workshop on Universal Dependencies (UDW 2020)_ , Marie-Catherine de Marneffe, Miryam de Lhoneux, Joakim Nivre, and Sebastian Schuster (Eds.). Association for Computational Linguistics, Barcelona, Spain (Online), 152–157.  https://aclanthology.org/2020.udw-1.17
  * Scherrer et al. (2023) Yves Scherrer, Olli Kuparinen, and Aleksandra Miletić. 2023\.  CorCoDial-Machine translation techniques for corpus-based computational dialectology. In _Proceedings of the 24th Annual Conference of the European Association for Machine Translation_. 511–512.
  * Schneider (2012) Klaus P Schneider. 2012\.  Appropriate behaviour across varieties of English.  _Journal of Pragmatics_ 44, 9 (2012), 1022–1037.
  * Seidlhofer (2005) Barbara Seidlhofer. 2005\.  English as a lingua franca.  _ELT journal_ 59, 4 (2005), 339–341.
  * Shapiro and Duh (2019) Pamela Shapiro and Kevin Duh. 2019.  Comparing Pipelined and Integrated Approaches to Dialectal Arabic Neural Machine Translation. In _Proceedings of the Sixth Workshop on NLP for Similar Languages, Varieties and Dialects_. Association for Computational Linguistics, Ann Arbor, Michigan, 214–222.  https://doi.org/10.18653/v1/W19-1424
  * Shoufan and Alameri (2015) Abdulhadi Shoufan and Sumaya Alameri. 2015.  Natural language processing for dialectical Arabic: A survey. In _Proceedings of the second workshop on Arabic natural language processing_. 36–48.
  * Simaki et al. (2017) Vasiliki Simaki, Panagiotis Simakis, Carita Paradis, and Andreas Kerren. 2017.  Identifying the Authors’ National Variety of English in Social Media Texts. In _Proceedings of the International Conference Recent Advances in Natural Language Processing, RANLP 2017_. INCOMA Ltd., Varna, Bulgaria, 671–678.  https://doi.org/10.26615/978-954-452-049-6_086
  * Sun et al. (2023) Jiao Sun, Thibault Sellam, Elizabeth Clark, Tu Vu, Timothy Dozat, Dan Garrette, Aditya Siddhant, Jacob Eisenstein, and Sebastian Gehrmann. 2023.  Dialect-robust Evaluation of Generated Text. In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_. Association for Computational Linguistics, Toronto, Canada, 6010–6028.  https://doi.org/10.18653/v1/2023.acl-long.331
  * Tahssin et al. (2020) Rawan Tahssin, Youssef Kishk, and Marwan Torki. 2020\.  Identifying Nuanced Dialect for Arabic Tweets with Deep Learning and Reverse Translation Corpus Extension System. In _Proceedings of the Fifth Arabic Natural Language Processing Workshop_. Association for Computational Linguistics, Barcelona, Spain (Online), 288–294.  https://aclanthology.org/2020.wanlp-1.30
  * Wang et al. (2017) Hongmin Wang, Yue Zhang, GuangYong Leonard Chan, Jie Yang, and Hai Leong Chieu. 2017.  Universal Dependencies Parsing for Colloquial Singaporean English. In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , Regina Barzilay and Min-Yen Kan (Eds.). Association for Computational Linguistics, Vancouver, Canada, 1732–1744.  https://doi.org/10.18653/v1/P17-1159
  * Wang et al. (2022) Yizhou Wang, Rikke L. Bundgaard-Nielsen, Brett J. Baker, and Olga Maxwell. 2022.  Perceptual Overlap in Classification of L2 Vowels: Australian English Vowels Perceived by Experienced Mandarin Listeners. In _Proceedings of the 36th Pacific Asia Conference on Language, Information and Computation_ , Shirley Dita, Arlene Trillanes, and Rochelle Irene Lucas (Eds.). Association for Computational Linguistics, Manila, Philippines, 317–324.  https://aclanthology.org/2022.paclic-1.35
  * Xiao et al. (2023) Zedian Xiao, William Held, Yanchen Liu, and Diyi Yang. 2023\.  Task-Agnostic Low-Rank Adapters for Unseen English Dialects. In _Findings of ACL_.
  * Xu et al. (2015) Fan Xu, Xiongfei Xu, Mingwen Wang, and Maoxi Li. 2015\.  Building Monolingual Word Alignment Corpus for the Greater China Region. In _Proceedings of the Joint Workshop on Language Technology for Closely Related Languages, Varieties and Dialects_ , Preslav Nakov, Marcos Zampieri, Petya Osenova, Liling Tan, Cristina Vertan, Nikola Ljubešić, and Jörg Tiedemann (Eds.). Association for Computational Linguistics, Hissar, Bulgaria, 85–94.  https://aclanthology.org/W15-5414
  * Younes et al. (2020) Jihene Younes, Emna Souissi, Hadhemi Achour, and Ahmed Ferchichi. 2020.  Language resources for Maghrebi Arabic dialects’ NLP: a survey.  _Language Resources and Evaluation_ 54 (2020), 1079–1142.
  * Zampieri et al. (2019) Marcos Zampieri, Shervin Malmasi, Yves Scherrer, Tanja Samardžić, Francis Tyers, Miikka Silfverberg, Natalia Klyueva, Tung-Le Pan, Chu-Ren Huang, Radu Tudor Ionescu, Andrei M. Butnaru, and Tommi Jauhiainen. 2019.  A Report on the Third VarDial Evaluation Campaign. In _Proceedings of the Sixth Workshop on NLP for Similar Languages, Varieties and Dialects_ , Marcos Zampieri, Preslav Nakov, Shervin Malmasi, Nikola Ljubešić, Jörg Tiedemann, and Ahmed Ali (Eds.). Association for Computational Linguistics, Ann Arbor, Michigan, 1–16.  https://doi.org/10.18653/v1/W19-1401
  * Zampieri et al. (2020) Marcos Zampieri, Preslav Nakov, and Yves Scherrer. 2020\.  Natural language processing for similar languages, varieties, and dialects: A survey.  _Natural Language Engineering_ 26, 6 (2020), 595–612.
  * Zampieri et al. (2014) Marcos Zampieri, Liling Tan, Nikola Ljubešić, and Jörg Tiedemann. 2014.  A report on the DSL shared task 2014. In _Proceedings of the first workshop on applying NLP tools to similar languages, varieties and dialects_. 58–67.
  * Zampieri et al. (2015) Marcos Zampieri, Liling Tan, Nikola Ljubešić, Jörg Tiedemann, and Preslav Nakov. 2015\.  Overview of the DSL shared task 2015. In _Proceedings of the Joint Workshop on Language Technology for Closely Related Languages, Varieties and Dialects_. 1–9.
  * Zbib et al. (2012) Rabih Zbib, Erika Malchiodi, Jacob Devlin, David Stallard, Spyros Matsoukas, Richard Schwartz, John Makhoul, Omar F. Zaidan, and Chris Callison-Burch. 2012.  Machine Translation of Arabic Dialects. In _Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_. Association for Computational Linguistics, Montréal, Canada, 49–59.  https://aclanthology.org/N12-1006
  * Zhan et al. (2023) Haolan Zhan, Zhuang Li, Yufei Wang, Linhao Luo, Tao Feng, Xiaoxi Kang, Yuncheng Hua, Lizhen Qu, Lay-Ki Soon, Suraj Sharma, et al. 2023\.  SocialDial: A Benchmark for Socially-Aware Dialogue Systems.  _arXiv preprint arXiv:2304.12026_ (2023).
  * Zhang et al. (2021) Xiongyi Zhang, Jan-Willem van de Meent, and Byron C Wallace. 2021\.  Disentangling Representations of Text by Masking Transformers. In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_. 778–791.
  * Zhao et al. (2020) Yuanyuan Zhao, Weiwei Sun, Junjie Cao, and Xiaojun Wan. 2020\.  Semantic Parsing for English as a Second Language. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault (Eds.). Association for Computational Linguistics, Online, 6783–6794.  https://doi.org/10.18653/v1/2020.acl-main.606
  * Ziems et al. (2022) Caleb Ziems, Jiaao Chen, Camille Harris, Jessica Anderson, and Diyi Yang. 2022.  VALUE: Understanding Dialect Disparity in NLU. In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (Eds.). Association for Computational Linguistics, Dublin, Ireland, 3701–3720.  https://doi.org/10.18653/v1/2022.acl-long.258
  * Ziems et al. (2023) Caleb Ziems, William Held, Jingfeng Yang, Jwala Dhamala, Rahul Gupta, and Diyi Yang. 2023\.  Multi-VALUE: A Framework for Cross-Dialectal English NLP. In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (Eds.). Association for Computational Linguistics, Toronto, Canada, 744–768.  https://doi.org/10.18653/v1/2023.acl-long.44