# Temperature dependence and limiting mechanisms of the upper critical field
of FeSe thin films

M. Stanley1 Y. Li1 J. C. Palmstrom2 J. L. Thompson3 K. D. Halanayake3 D.
Reifsnyder-Hickey3 R. D. McDonald2 S. A. Crooker2 N. Trivedi4 N. Samarth1,5
<EMAIL_ADDRESS>1Department of Physics, The Pennsylvania State University,
University Park, Pennsylvania 16802, USA
2National High Magnetic Field Laboratory, Los Alamos National Laboratory, Los
Alamos, New Mexico 87545, USA
3Department of Chemistry, The Pennsylvania State University, University Park,
Pennsylvania 16802, USA
4Department of Physics, The Ohio State University, Columbus, Ohio 43210, USA
5Department of Materials Science and Engineering, The Pennsylvania State
University, University Park, Pennsylvania 16802, USA

###### Abstract

We use magnetoresistance measurements at high magnetic field ($B\leq 65$ T)
and low temperature ($T\geq 500$ mK) to gain fresh insights into the behavior
of the upper critical field, $H_{c2}$, in superconducting ultrathin FeSe films
of varying degrees of disorder, grown by molecular beam epitaxy on SrTiO3.
Measurements of $H_{c2}$ across samples with a widely varying superconducting
critical temperature (1.2 K $\leq T_{c}\leq 21$ K) generically show similar
qualitative temperature dependence. We analyze the temperature dependence of
$H_{c2}$ in the context of Werthamer-Helfand-Hohenberg (WHH) theory. The
analysis yields parameters that indicate a strong Pauli paramagnetic pair-
breaking mechanism which is also reflected by pseudo-isotropic
superconductivity in the limit of zero temperature. In the lower $T_{c}$
samples, we observe a spin-orbit scattering driven enhancement of $H_{c2}$
above the strongly-coupled Pauli paramagnetic limit. We also observe clear
deviations from WHH theory at low temperature, regardless of $T_{c}$. We
attribute this to the multi-band superconductivity of FeSe and possibly to the
emergence of a low temperature, high field superconducting phase.

††preprint: APS/123-QED

##  I Introduction

Understanding superconductivity in the structurally and stoichiometrically
simple material system FeSe has been the target of many experimental efforts
since its initial discovery in tetragonal phase single crystals with
$T_{c}\approx 9-13.5$ K [1, 2]. The superconducting behavior of FeSe thin
films in the 2D limit has received a particular surge in attention in recent
years after reports of significantly enhanced superconducting gap closing
critical temperatures up to 65 K when interfaced with SrTiO3 (STO) substrates,
as measured by _in vacuo_ angle-resolved photoemission spectroscopy (ARPES)
and scanning tunneling spectroscopy (STS) [3, 4, 5, 6, 7, 8]. In vacuo
electrical transport measurements on pristine (uncapped) 2D FeSe films grown
on STO show reproducible zero resistance at $T_{c}$ as high as $\sim 30$ K [9,
10, 11], significantly higher than in FeSe bulk crystals and consistent with
the large superconducting gap seen in ARPES and STS. Despite many experimental
studies of superconductivity in 2D FeSe/STO [12, 13], there are still
opportunities to improve our understanding of the superconductivity of this
material system. One pathway is to study the upper critical field, $H_{c2}$, a
fundamental parameter of superconductivity, which represents the strength at
which an externally applied magnetic field will break down the Cooper pairs of
a superconductor.

Pair-breaking at $H_{c2}$ is generally considered to have two origins: the
polarization of the spins of the constituent electrons of Cooper pairs via
Pauli paramagnetism, and the orbital effect when cores of Abrikosov vortices
start to overlap. It is an essential parameter for understanding numerous
superconducting properties of a material, including the coherence length, the
dimensionality of superconductivity, and anisotropy. Systematic measurements
of $H_{c2}$ can also provide insight into pair-breaking mechanisms and their
relative strengths. Due to its relatively low $T_{c}$, bulk, single crystal
FeSe has been an ideal subject for several high magnetic field measurements
aimed at establishing the overall behavior and dependencies of $H_{c2}$ in the
limit of zero temperature and at various pressures and sample thicknesses [14,
15, 16, 17]. However, similarly rigorous studies of $H_{c2}$ have been
technically challenging in ultrathin films of FeSe grown on STO because the
greatly enhanced values of $T_{c}$ are accompanied by extremely large values
of $H_{c2}$ [18]. To our knowledge, a detailed analysis of the temperature
dependence of $H_{c2}$ for FeSe/STO has not yet been reported. In this paper,
we use high magnetic field ($B\leq 65$ T) magnetoresistance (MR) measurements
to map out the variation of $H_{c2}$ as a function of temperature and disorder
in four ultrathin FeSe films grown by molecular beam epitaxy (MBE) on STO. Our
observations provide important insights into the unique interfacial
superconductivity in this material system.

##  II Experiment

To produce FeSe thin films capable of achieving high temperature
superconductivity, $T_{c,0}\geq$ 20 K, we first treated commercially obtained
(Shinkosha) STO (001) substrates with a chemical etching process which
included 45 minutes in deionized water at $90^{\circ}$C and another 45 minutes
in a 25% HCl solution at room temperature. Substrates were then annealed under
a flow of oxygen in a tube furnace for 3 hours at 980∘C, rendering atomically
flat, TiO2-terminated surfaces. Substrates were additionally outgassed _in
vacuo_ at 500∘C for 2 hours prior to growth.

We deposited the FeSe thin films using a Scienta Omicron EVO-50 MBE system,
with a base pressure of 10-11 Torr, using high purity Fe (99.99%) and Se
(99.999%) elemental sources simultaneously evaporated from thermal Knudsen
effusion cells. We maintained the substrate temperature at roughly
$350^{\circ}$ C, as measured by an infrared pyrometer, and used Fe and Se
fluxes that yielded a deposition rate of roughly 1 layer every 6.5 minutes.
This yielded films of around 3 unit cells (U.C.) thickness (1.65 nm) after a
typical growth period of 20 minutes (Fig. 2). A post-growth anneal was
performed at a substrate temperature between 525∘C and 550∘C for a duration
varying between 2 to 3 hours. This annealing process is expected to induce Se
vacancies, allowing for electron charge transfer into the film from the STO,
and is a critical step in the realization of high temperature
superconductivity in this system [19, 20]. We grew four samples, A, B, C, and
D, with varying post-growth annealing conditions, yielding a large variation
in $T_{C}$. Prior to their removal from the UHV chamber, samples were capped
with approximately 10 U.C.s (5-6 nm) of crystalline FeTe, grown at 350∘C for
surface passivation, and 10 nm of amorphous Te deposited at room temperature
for additional protection from oxidation; we find that this capping protocol
provides consistent results when studying the superconducting properties of
FeSe/STO films ex situ[10]. Grown on 5 x 5 mm2 pieces of STO held in place
with clips of tantalum foil, the typical coverage of films in this study was
around 10 mm2.

We then prepared samples for ex situ electrical transport measurements by
mechanically etching six-terminal Hall bar patterns with effective areas of
$0.5\times 1.0\textrm{mm}^{2}$. Electrical contacts were formed with pressed
indium dots. Samples were initially evaluated in a Quantum Design 14 T
Physical Property Measurement System in order to identify candidate samples
representing different levels of disorder. Finally, we measured field-
dependent longitudinal resistivities $\rho_{xx}(H^{ab})$ and
$\rho_{xx}(H^{c})$ in pulsed magnetic fields up to 65 T applied in plane and
out of plane, respectively. We focus on four particular samples (A, B, C, and
D) of similar thickness but with widely varying zero resistance critical
temperatures (Fig. 1).

Samples were mounted on a transport probe and loaded into a cryostat with a
double-walled vacuum jacket, which in turn was immersed in a bath of liquid
4He. We achieved temperatures down to 500 mK by condensing 3He into the
cryostat and subsequent pumping. The capacitor-driven 65 T pulsed magnet
produced 80 ms pulses, with a rise time of 8 ms. Eddy currents induced by this
rapidly changing magnetic field, or _dB/dt_ , result in self-heating effects
which can become non-negligible at very low temperatures. Thermal stability
was maximized by measuring in liquid 4He for temperatures between 1.2 K and
4K. Additionally, only data gathered during the down-sweep of the magnetic
field pulse, where the _dB/dt_ was much smaller, was considered during our
analysis.

After performing all other characterization, the samples were analyzed by
annular dark-field scanning transmission electron microscopy (ADF-STEM)
imaging. Samples were first prepared by depositing 15 nm of amorphous carbon
in a Leica sputter coater, and then they were transferred into an FEI Scios 2
dual-beam scanning electron microscopy focused ion beam (SEM-FIB) for the
creation of electron-transparent lamellae for STEM imaging. The lamellae were
extracted from the regions that had previously been patterned for electrical
transport measurements, and they were subsequently thinned to electron
transparency using FIB accelerating voltages of 30 kV and 5 kV, with a final
cleaning step at 2 kV. ADF-STEM imaging and energy-dispersive X-ray
spectroscopy (EDX) elemental analysis were performed on a dual-spherical
aberration-corrected FEI Titan3 G2 60-300 S/TEM at an accelerating voltage of
300 kV with a convergence angle of 25.2 mrad. Images were collected by a high-
angle ADF detector with collection angles of 42-244 mrad.

##  III Results and Discussion

The temperature dependent resistivity for all 4 samples (A, B, C, D) is shown
in Fig. 1. Samples A, B, and C show similar behavior in the normal state,
where $\rho_{xx}(T)$ is almost independent of temperature before reaching the
superconducting transition at their respective values of $T_{c}$. The
qualitative behavior of $\rho_{xx}(T)$ in sample D resembles the metallic
behavior seen in bulk crystals of FeSe, monotonically decreasing with
decreasing temperature, albeit with a smaller residual resistivity ratio (RRR)
$\sim 2$ than observed in high quality bulk crystals [1, 2, 21]. Sample D
exhibits the highest zero resistance critical temperature, as well as the
highest RRR, indicating that it is the least disordered of the thin film
samples we studied. These sample-specific parameters and others are outlined
in Table 1.

It is interesting to note that sample A with the highest disorder shows a
small increase in resistivity just before going superconducting. The increase
in resistivity in the more disordered sample is consistent with emergent
granularity. In the normal state, the grains or patches become locally
superconducting but the different patches are not Josephson coupled. As the
temperature is reduced, the Josephson coupling increases and the patches get
coupled into a globally coherent state [22, 23, 24].

The onset superconducting critical temperature $T_{c}^{onset}$ and normal
state resistivity $\rho_{n}^{onset}$ for each sample are determined from the
intersections of two extrapolated lines. The first is drawn through the
steepest portion of $\rho_{xx}(T)$ within the superconducting transition and
the second through $\rho_{xx}(T)$ in the normal state just above the
superconducting transition (Fig. 1 inset).

Since the four samples presented in this study were grown under similar growth
conditions, we performed scanning transmission electron microscopy (STEM)
measurements in an effort to understand the large variation in their
superconducting critical temperatures from a microscopic point of view. The
cross-sectional high-angle annular dark-field (HAADF) images in Fig. 2 compare
samples A and D, the lowest and highest $T_{c}$ samples, respectively. Sample
D shows the expected 3 U.C. FeSe (001) epitaxial layer, and clearly defined
FeTe and Te capping layers. While the TEM measurements potentially indicate a
greater level of disorder in sample A, we note that the lack of clarity in the
image (Fig. 2(a)) most likely arises from some form of contamination (perhaps
due to In contacts during sample preparation for TEM measurements); it cannot
be directly associated with crystallographic disorder. We note that these STEM
measurements were carried out on the actual sample used in high field
measurements after the transport experiments were performed. More detailed
STEM data for these samples and for sample B, C is available in the
Supplementary Material [25].

Table 1: Parameters related to the superconducting transition of each sample extracted from Fig. 1. The temperature at which 90% of the sample resistance at $T_{c,\textrm{onset}}$ is reached defines $T_{c,90}$, and $T_{c,0}$ is the point at which the resistance is less than 5% of $T_{c,\textrm{onset}}$ effectively close to zero. The RRR is calculated as $\rho(293K)/\rho(T_{c,\textrm{onset}})$.  Sample |  $T_{c,\textrm{onset}}$ [K[ |  $T_{c,90}$ [K] |  $T_{c,0}$ [K] | RRR
---|---|---|---|---
A | 11.0 | 10.3 | 1.2 | 1.1
B | 12.6 | 11.8 | 8.5 | 1.4
C | 24.0 | 22.0 | 14.0 | 1.2
D | 32.0 | 27.2 | 21.0 | 2.1

The results of MR measurements $\rho_{xx}(\mu_{0}H)$ in pulsed high magnetic
fields are shown in Fig. 3 for sample A in both $H\parallel ab$ and
$H\parallel c$ geometries. The temperature dependence of $H_{c2}$ is extracted
from these plots by identifying the abscissa corresponding to 10%
($\rho_{n}^{10}$), 50% ($\rho_{n}^{50}$), and 90% ($\rho_{n}^{90}$) of the
normal state resistivity along each isotherm. These positions correspond to
three separate but self-consistent definitions of $H_{c2}$. Similar data for
samples B, C, and D are shown in the Supplementary Material [25].

For samples A and B (which have the lowest $T_{c}$), we can determine
$H_{c2}(T)$ down to the 4He base temperature in both field orientations. For
samples C and D (which have higher $T_{c}$), the available data in the
$H\parallel ab$ geometry was restricted to temperatures near the
superconducting transition because $H_{c2}(T)$ exceeds the available field
strength (65 T). This observation is consistent with the fact that orbital-
limiting effects in the $H\parallel ab$ configuration should be suppressed for
such a highly anisotropic few-layer system where vortex formation is confined
to a relatively small cross sectional area.
Measured temperatures, in general, were selected in order to comprehensively
sample the superconducting and normal states, as well as the entire breadth of
the superconducting transition region in each film. The resulting values of
$H_{c2}(T)$ for sample A are compiled in Fig. 4 as an example. We found that
the qualitative behavior of $H_{c2}(T)$ was not heavily dependent on the
resistivity criteria used to define it, possibly indicating minimal field-
induced broadening of the superconducting transition in these samples. We
therefore proceed strictly with the $\rho_{n}^{90}$ criteria for defining
$H_{c2}(T)$ due to the abundance of data points it provides, without loss of
generality.

We analyze our data using Werthamer-Helfand-Hohenberg (WHH) theory which
provides a prediction of the temperature dependence of $H_{c2}(T)$ for type-II
superconductors, and can account for the effects of both orbital and
paramagnetic limitation [26, 27, 28]. In the dirty limit, where the electron
mean free path $l$ is much smaller than the superconducting coherence length
$\xi$, $H_{c2}$(T) is determined when the following condition is met [29]:

$\displaystyle\centering 0=\ln(t)\@add@centering$
$\displaystyle-\psi(\frac{1}{2})+(1+\frac{i\lambda_{so}}{\sqrt{4\alpha^{2}h^{2}-\lambda_{so}^{2}}})$
(1)
$\displaystyle\times\psi([2h+\lambda_{so}+2t+i\sqrt{4\alpha^{2}h^{2}-\lambda_{so}^{2}}]/4t)/2+c.c.$

Here, $\hbar=k_{b}=1$, $\psi(x)$ is the complex digamma function, $t=T/T_{c}$
is the normalized temperature, $h=eH_{c2}\nu_{F}^{2}\tau/3\pi T_{c}$ is the
reduced magnetic field, and the Maki parameter $\alpha$ and spin-orbit
coupling parameter $\lambda_{so}$ are defined as:

$\displaystyle\alpha=3/2m\nu_{F}^{2}\tau$ (2)
$\displaystyle\lambda_{so}=1/3\pi T_{c}\tau_{so}$

Last, we define $\overline{H}\equiv\alpha h=eH_{c2}/2\pi T_{c}m$ as another
dimensionless representation of the upper critical magnetic field convenient
for plotting.

Special cases of the upper critical field arise first in the absence of both
Pauli paramagnetic limitation and spin-orbit interaction, when
$\alpha=\lambda_{so}=0$, where the entirely orbital-limited $H_{c2}$ is
determined to be [28]:

$\mu_{0}H_{c2}^{orb}(0)=-0.693T_{c}\left(-\frac{d\mu_{0}H}{dT}\right)|_{T=T_{c}}$
(3)

Second, if $H_{c2}$ is instead entirely Pauli-limited, superconductivity is
quenched when the Zeeman energy exceeds the superconducting condensation
energy. In the case of a weakly-coupled BCS superconductor, this corresponds
to the Chandrasekhar-Clogston limit [30, 31]:

$\mu_{0}H_{c2}^{P}(0)=1.86\>T_{c}$ (4)

This assumes that $2\Delta_{0}=3.5k_{b}T_{c}$.

The Maki parameter (Eq. 2) describes the relative strengths of the Pauli
paramagnetic effect and the orbital limiting field when these two mechanisms
have comparable effects in the pair-breaking process [32]. In this case, it
can be written as $\alpha=\sqrt{2}H_{c2}^{orb}/H_{c2}^{P}$. Experimentally,
the Maki parameter can be interpreted from the slope of $H_{c2}(T)$ in the
vicinity of $T_{c}$ using this relation and Eqns. 3 and 4:

$\alpha\approx 0.53\left(-\frac{d\mu_{0}H}{dT}\right)|_{T=T_{c}}$ (5)

We numerically solve Eq. 1 using $\alpha$ and $\lambda_{so}$ as fitting
parameters for the experimental $\overline{H}_{c2}^{ab}(t)$ and
$\overline{H}_{c2}^{c}(t)$ data of each sample. The numerical results are
summarized in Table 2 along with the values from theoretical predictions. The
magnitude of the spin-orbit coupling parameter, $\lambda_{so}$, is indicative
of the spin-flip scattering strength in the sample, and a non-zero value of
$\lambda_{so}$ incorporated into the WHH model enhances the predicted values
of $H_{c2}(T)$, especially in the zero temperature limit. Physically, this
corresponds to a suppression of the Pauli paramagnetic effect through a
reduction of the Zeeman energy.

We now discuss the WHH fits of the temperature dependence of the upper
critical field for all four samples (Fig. 5). Here, we assume $\lambda_{el-
ph}=0.3$ [33]. Each plot includes additional WHH curves constructed using the
non-zero Maki parameter calculated while fitting experimental data but without
the contribution of spin-orbit scattering to highlight the necessity of a non-
zero $\lambda_{so}$. Additionally, for sample A, we have calculated five WHH
curves with $\alpha$ from 1 to 5 and $\lambda_{so}=0$ (see Supplementary
Material [25]). All these plots reveal a clear necessity for including a non-
zero $\lambda_{so}$ in order to accurately fit the experimental data for both
the parallel and perpendicular field geometries, reflecting the significant
role of spin-orbit coupling in FeSe [16, 34].

In the $H\parallel ab$ geometry, we observe very large Maki parameters in all
four samples. Physically, this suggests weak orbital limitation, and
conversely, stronger Pauli-limiting effects in this configuration. This result
is consistent with the picture of vortex formation and flow being hindered by
the relatively small cross-sectional area of the $\sim$1.5 nm thick films
during the superconducting-normal state transition. The fits in Fig. 5 (a)-(d)
also show a large deviation between the WHH fit of the $H\parallel ab$ data
and the theoretical fully-orbital-limited curves ($\alpha=0,\lambda_{so}=0$),
especially towards zero temperature, as another indication of weak orbital-
limiting effects. Finally, to this point, we note that the extracted Maki
parameters are greater than those calculated under the assumption of
comparable limiting mechanisms (Table 2) by nearly a factor of two for each
sample. This further suggests the significance of Pauli-limiting effects here.

The values of the Maki and spin-orbit scattering parameters derived from our
WHH analysis are larger than observed in single crystal FeSe samples. Reported
values in bulk FeSe crystals include $\alpha=0.82$ ($H\parallel ab$) and
$\alpha=0.37$ ($H\parallel c$) [16], and $\alpha=2.1$ ($H\parallel ab$) [35],
both without contribution from a non-zero $\lambda_{so}$. In thin flakes of
FeSe, values of $\alpha=2.4$ and $\alpha=4.15$ have been observed for sample
thicknesses of $t=54,100$ nm, and $t=24$ nm, respectively [17]. Complete
descriptions of the data in this case of thin flakes required
$\lambda_{so}=0.2-0.35$. Results obtained from the work presented here
($\alpha=8.7-9.6$ for H $\parallel$ ab and $\alpha=1.1-3.2$ for H $\parallel$
c) on $t\approx 1.5$ nm thin films of FeSe are consistent with a trend of
increased Maki parameters, as well as the inclusion of spin-orbit scattering
into the WHH model to effectively describe the data, in the 2D limit.

Table 2: Overview of various properties and superconducting parameters for each film discussed here. The columns are, in order, the sample name, the Maki parameters extracted from the WHH fitting for both $H\parallel ab$ and $H\parallel c$ geometries, the corresponding spin-orbit scattering parameters, the zero-temperature $H_{c2}(T)$ predicted by WHH theory with best fit parameters, the theoretically predicted orbital-limited $H_{c2}$ at zero temperature, the weakly-coupled Pauli-paramagnetic limit of $H_{c2}$ at zero temperature. Sample | $\alpha\>(\parallel ab,\parallel c)$ | $\lambda_{so}\>(\parallel ab,\parallel c)$ | $H_{c2}^{ab}(0)$ | $H_{c2}^{c}(0)$ | $H_{c2}^{orb}(0)\parallel ab$ | $H_{c2}^{orb}(0)\parallel c$ | $H_{c2}^{P}(0)$
---|---|---|---|---|---|---|---
A | 9.6, 2.3 | 1.8, 1.7 | 34.2 T | 21.8 T | 71.1 T | 26.1 T | 19.2 T
B | 8.7, 3.2 | 1.6, 2.3 | 37.1 T | 34.6 T | 83.5 T | 34.8 T | 21.9 T
C | 8.9, 2.5 | 1.2, 0.5 | 63.1 T | 38.5 T | 150.3 T | 67.6 T | 40.9 T
D | 9.4, 1.1 | 2.1, 1.5 | 94.2 T | 37.2 T | 204.9 T | 32.1 T | 50.6 T

When the magnetic field is aligned perpendicular to the plane of the film,
orbital effects are expected to play a more significant role, and this is
demonstrated in part by the WHH fits of the H $\parallel$ c data in Fig. 5
(a-d). At higher temperatures, closer towards $T_{c}$, there is a clear
reduction in the values of $H_{c2}$ compared to those from the H $\parallel$
ab measurements at comparable temperatures. Additionally, the calculated fit
for samples A and B closely follow the purely orbital-limited curves
($\alpha=0,\lambda_{so}=0$). We also observe a clear deviation from the
prediction of WHH as $H_{c2}^{c}$ increases almost linearly, upwards, in the
limit of zero temperature in all four samples. Such upward curvature have been
demonstrated in the ’1111’ [36, 37] and ’122’ [38, 39] systems, and has been
discussed for the stoichiometric ’11’ Fe(Te,Se) system as well [40, 41]. This
deviation from the single-band WHH theory is typically recognized as an effect
of multi-band superconductivity [42].

In samples A and B, the low temperature $(T/T_{c}<0.2)$ behavior of
$H_{c2}^{ab}$ exhibits a slight upward curvature which deviates from the WHH
prediction, even with the inclusion of a non-zero $\lambda_{so}$. Distinct
high-field induced superconducting phases at low temperatures in single
crystal FeSe samples have been reported [43] and have been attributed to a
possible unconventional Fulde-Ferrell-Larkin-Ovchinnikov (FFLO) state, even in
the presence of disorder [44]. The FFLO state features spatial modulation of
the superconducting order parameter and Cooper pairs of finite total momentum
$(k\uparrow,-k+q\downarrow)$, compared to the conventional zero-momentum
pairing $(k\uparrow,-k\downarrow)$. At high fields, Zeeman splitting plays a
more significant role in quenching the superconducting phase, so a
prerequisite for realization of the FFLO state is a large (greater than one)
Maki parameter, indicating a smaller paramagnetically-limited field by the
$\alpha=\sqrt{2}H_{c2}^{orb}/H_{c2}^{P}$ relation. This is indeed a feature in
all four samples measured here. However, extremely large field strengths, much
higher than 65 T, would be required to sufficiently survey the low temperature
regime of higher-$T_{c}$ samples, such as C and D, for a more complete
investigation of this unique state in the FeSe/STO system.

Next, we address the overall temperature dependence of the upper critical
field anisotropy, quantified by the anisotropy parameter
$\gamma(T)=H_{c2}^{ab}(T)/H_{c2}^{c}(T)$. This is shown in Fig. 6 and
qualitatively agrees with that of other Fe-based superconductors, including
single crystal FeSe [15]. In the limit of zero temperature, nearly isotropic
superconductivity in other Fe-based superconducting systems, such as the ‘11’
and ‘122’ classes of materials, is commonly observed despite the quasi-2D
nature of their electronic structures [45]. In the ultrathin limit, a
significant increase in $\gamma(T)$ could rightfully be expected as the
superconductivity further inherits the anisotropic character of the
increasingly two-dimensional Fermi surface, but this behavior only seems to be
observed in the neighborhood of the superconducting critical temperature [17].
Our four samples demonstrate varying degrees of enhanced anisotropy closer to
their respective values of $T_{c}$, ranging from $\gamma(t\approx
0.8)=2.0-4.1$, as well as a monotonic decrease towards $\gamma=1$ in the limit
of zero temperature (Fig. 6). Overall, the orbital constraints on
superconductivity in FeSe likely describe the larger anisotropy as the
temperature approaches $T_{c}$: in this regime, the Maki parameter, and
therefore the slope of $H_{c2}(T)$, is larger in the $H\parallel ab$
direction. In contrast, the approach to isotropic behavior of $H_{c2}(0)$ at
lower temperatures can be attributed to the dominance of the Pauli-
paramagnetic limit, so that $H_{c2}(0)$ is not affected by orientation of the
field.

Finally, we estimate the value of the zero-temperature coherence length,
$\xi(0)$ from the extracted $H_{c2}(T)$ data using the 2D Ginzburg-Landau
formula
$\mu_{0}H_{c2}^{\perp}(T)=\Phi_{0}/(2\pi\xi_{GL}(0)^{2})\times(1-T/T_{c})$
where $\Phi_{0}=2.07\times 10^{-15}$ Wb is the superconducting magnetic flux
quantum [46]. Fitting this equation to the data in the vicinity of $T_{c}$, we
find that $\xi_{GL}(0)=$ 3.2 nm, 3.5 nm, 3.3 nm, and 4.8 nm for samples A, B,
C, and D, respectively.

##  IV Conclusion

In summary, we have studied the superconducting transition of four MBE-grown
FeSe/STO thin films of varying disorder by determining their resistive upper
critical fields using high magnetic field pulses up to 65 T applied in both
parallel ($H\parallel ab$) and perpendicular ($H\parallel c$) directions. The
experimental behavior of $H_{c2}(T)$, defined by the $\rho_{n}^{90}$
criterion, is in close agreement with WHH theory down to $T/T_{c}\approx 0.25$
when the effects of both orbital and Pauli paramagnetic limits are considered.
Although our TEM measurements broadly show a correlation between structural
disorder and $T_{c}$, the parameters extracted from the WHH fits are
surprisingly insensitive to both the level of disorder and $T_{c}$. Prior
HRTEM studies of the FeSe/STO interfacial structure have indicated that
$T_{c}$ is possibly determined by the reconstruction of the FeSe Fermi surface
resulting from excess Ti in a quasi-2D Ti1-xO2 interfacial layer [47]. This
has not been carefully examined in our samples and warrants further microscopy
studies. Low temperature deviations of the $H_{c2}^{c}$ data from the theory
are observed in each sample and may result from the effects of multi-band
superconductivity. Similar deviations below $T/T_{c}\approx 0.2$ in the
$H_{c2}^{ab}$ of samples A and B may indicate a separate high-field
superconducting phase akin to the FFLO state. This warrants further
investigation of higher $T_{c}$ samples, such as samples C and D, in fields
with strengths exceeding 65 T to effectively study their detailed $H_{c2}(T)$
behavior in a low temperature regime. In future work we will further explore
the combined role of spin-orbit coupling and disorder in a multiband
superconductor such as FeSe.

###### Acknowledgements.

This work was supported by the Penn State 2DCC-MIP under NSF Grant No.
DMR-2039351. Work at the National High Magnetic Field Lab was supported by NSF
DMR-1644779, the State of Florida, and the U.S. Department of Energy and the
research at Ohio State was supported by NSF Grant No. DMR2138905.

## References

  * Hsu  _et al._ [2008] F.-C. Hsu, J.-Y. Luo, K.-W. Yeh, T.-K. Chen, T.-W. Huang, P. M. Wu, Y.-C. Lee, Y.-L. Huang, Y.-Y. Chu, D.-C. Yan, and M.-K. Wu, PNAS 105, 14262 (2008).
  * Mizuguchi  _et al._ [2008] Y. Mizuguchi, F. Tomioka, S. Tsuda, T. Yamaguchi, and Y. Takano, Appl. Phys. Lett. 93, 152505 (2008).
  * Wang  _et al._ [2012] Q. Y. Wang, Z. Li, W. H. Zhang, Z. C. Zhang, J. S. Zhang, W. Li, H. Ding, Y. B. Ou, P. Deng, K. Chang, J. Wen, C. L. Song, K. He, J. F. Jia, S. H. Ji, Y. Y. Wang, L. L. Wang, X. Chen, X. C. Ma, and Q. K. Xue, Chin. Phys. Lett. 29, 037402 (2012).
  * He  _et al._ [2013] S. He, J. He, W. Zhang, L. Zhao, D. Liu, X. Liu, D. Mou, Y. B. Ou, Q. Y. Wang, Z. Li, L. Wang, Y. Peng, Y. Liu, C. Chen, L. Yu, G. Liu, X. Dong, J. Zhang, C. Chen, Z. Xu, X. Chen, X. Ma, Q. Xue, and X. J. Zhou, Nat. Mater. 12, 605 (2013).
  * Zhang  _et al._ [2014a] W. H. Zhang, Y. Sun, J. S. Zhang, F. S. Li, M. H. Guo, Y. F. Zhao, H. M. Zhang, J. P. Peng, Y. Xing, H. C. Wang, T. Fujita, A. Hirata, Z. Li, H. Ding, C. J. Tang, M. Wang, Q. Y. Wang, K. He, S. H. Ji, X. Chen, J. F. Wang, Z. C. Xia, L. Li, Y. Y. Wang, J. Wang, L. L. Wang, M. W. Chen, Q. K. Xue, and X. C. Ma, Chin. Phys. Lett. 31, 017401 (2014a).
  * Liu  _et al._ [2012] D. Liu, W. Zhang, D. Mou, J. He, Y. B. Ou, Q. Y. Wang, Z. Li, L. Wang, L. Zhao, S. He, Y. Peng, X. Liu, C. Chen, L. Yu, G. Liu, X. Dong, J. Zhang, C. Chen, Z. Xu, J. Hu, X. Chen, X. Ma, Q. Xue, and X. J. Zhou, Nat. Commun. 3, 931 (2012).
  * Lee  _et al._ [2014] J. J. Lee, F. T. Schmitt, R. G. Moore, S. Johnston, Y. T. Cui, W. Li, M. Yi, Z. K. Liu, M. Hashimoto, Y. Zhang, D. H. Lu, T. P. Devereaux, D. H. Lee, and Z. X. Shen, Nature (London) 515, 245 (2014).
  * Tan  _et al._ [2013] S. Tan, Y. Zhang, M. Xia, Z. Ye, F. Chen, X. Xie, R. Peng, D. Xu, Q. Fan, H. Xu, J. Jiang, T. Zhang, X. Lai, T. Xiang, J. Hu, B. Xie, and D. Feng, Nat. Mater. 12, 634 (2013).
  * Faeth  _et al._ [2021] B. D. Faeth, S.-L. Yang, J. K. Kawasaki, J. N. Nelson, P. Mishra, C. T. Parzyck, C. Li, D. G. Schlom, and K. M. Shen, Phys. Rev. X 11, 021054 (2021).
  * Li  _et al._ [2021] Y. Li, Z. Wang, R. Xiao, Q. Li, K. Wang, A. Richardella, J. Wang, and N. Samarth, Phys. Rev. Mater. 5, 034802 (2021).
  * Ide  _et al._ [2022] K. Ide, T. Tanaka, A. Pedersen, S. Ichinokura, and T. Hirahara, Phys. Rev. Mater. 6, 124801 (2022).
  * Huang and Hoffman [2017] D. Huang and J. E. Hoffman, Annual Review of Condensed Matter Physics 8, 311 (2017).
  * Kreisel  _et al._ [2020] A. Kreisel, P. Hirschfeld, and B. Andersen, Symmetry 12, 1402 (2020).
  * Lei  _et al._ [2012] H. Lei, D. Graf, R. Hu, H. Ryu, E. S. Choi, S. W. Tozer, and C. Petrovic, Phys. Rev. B 85, 094515 (2012).
  * Vedeneev  _et al._ [2013] S. I. Vedeneev, B. A. Piot, D. K. Maude, and A. V. Sadakov, Phys. Rev. B 87, 134512 (2013).
  * Kang  _et al._ [2016] J. H. Kang, S. G. Jung, S. Lee, E. Park, J. Y. Lin, D. A. Chareev, A. N. Vasiliev, and T. Park, Superconductor Science and Technology 29 (2016).
  * Farrar  _et al._ [2020] L. S. Farrar, M. Bristow, A. A. Haghighirad, A. McCollam, S. J. Bending, and A. I. Coldea, npj Quantum Mater. 5, 29 (2020).
  * Zhou  _et al._ [2016] G. Zhou, D. Zhang, C. Liu, C. Tang, X. Wang, Z. Li, C. Song, S. Ji, K. He, L. Wang, X. Ma, and Q. K. Xue, Appl. Phys. Lett. 108, 202603 (2016).
  * Zhang  _et al._ [2014b] W. Zhang, Z. Li, F. Li, H. Zhang, J. Peng, C. Tang, Q. Wang, K. He, X. Chen, L. Wang, X. Ma, and Q. K. Xue, Phys. Rev. B 89, 060506(R) (2014b).
  * Berlijn  _et al._ [2014] T. Berlijn, H. P. Cheng, P. J. Hirschfeld, and W. Ku, Phys. Rev. B 89, 020501 (2014).
  * Böhmer  _et al._ [2016] A. E. Böhmer, V. Taufour, W. E. Straszheim, T. Wolf, and P. C. Canfield, Phys. Rev. B 94, 024526 (2016).
  * Ghosal  _et al._ [2001] A. Ghosal, M. Randeria, and N. Trivedi, Phys. Rev. B 65, 014501 (2001).
  * Ghosal  _et al._ [1998] A. Ghosal, M. Randeria, and N. Trivedi, Phys. Rev. Lett. 81, 3940 (1998).
  * Bouadim  _et al._ [2011] K. Bouadim, Y. L. Loh, M. Randeria, and N. Trivedi, Nature Physics 7, 884 (2011).
  * [25] See Supplemental Material at [URL will be inserted by publisher] for additional HRTEM and MR data.
  * Helfand and Werthamer [1964] E. Helfand and N. R. Werthamer, Phys. Rev. Lett. 13, 686 (1964).
  * Helfand and Werthamer [1966] E. Helfand and N. R. Werthamer, Phys. Rev. 147, 288 (1966).
  * Werthamer  _et al._ [1966] N. R. Werthamer, E. Helfand, and P. C. Hohenberg, Phys. Rev. 147, 295 (1966).
  * Solenov  _et al._ [2017] D. Solenov, M. Nikolo, J. Singleton, J. Jiang, J. Weiss, and E. Hellstrom (2017) p. 060004.
  * Chandrasekhar [1962] B. S. Chandrasekhar, Appl. Phys. Lett. 1, 7 (1962).
  * Clogston [1962] A. M. Clogston, Phys. Rev. Lett. 9, 266 (1962).
  * Maki [1966] K. Maki, Phys. Rev. 148, 362 (1966).
  * Li  _et al._ [2014] B. Li, Z. W. Xing, G. Q. Huang, and D. Y. Xing, J. Appl. Phys. 115, 193907 (2014).
  * Ma  _et al._ [2017] M. Ma, P. Bourges, Y. Sidis, Y. Xu, S. Li, B. Hu, J. Li, F. Wang, and Y. Li, Phys. Rev. X 7, 021025 (2017).
  * Audouard  _et al._ [2015] A. Audouard, F. Duc, L. Drigo, P. Toulemonde, S. Karlsson, P. Strobel, and A. Sulpice, EPL 109, 27003 (2015).
  * Hunte  _et al._ [2008] F. Hunte, J. Jaroszynski, A. Gurevich, D. C. Larbalestier, R. Jin, A. S. Sefat, M. A. McGuire, B. C. Sales, D. K. Christen, and D. Mandrus, Nature (London) 453, 903 (2008).
  * Jaroszynski  _et al._ [2008] J. Jaroszynski, F. Hunte, L. Balicas, Y. J. Jo, I. Raičević, A. Gurevich, D. C. Larbalestier, F. F. Balakirev, L. Fang, P. Cheng, Y. Jia, and H. H. Wen, Phys. Rev. B 78, 174523 (2008).
  * Baily  _et al._ [2009] S. A. Baily, Y. Kohama, H. Hiramatsu, B. Maiorov, F. F. Balakirev, M. Hirano, and H. Hosono, Phys. Rev. Lett. 102, 117004 (2009).
  * Yuan  _et al._ [2009] H. Q. Yuan, J. Singleton, F. F. Balakirev, S. A. Baily, G. F. Chen, J. L. Luo, and N. L. Wang, Nature (London) 457, 565 (2009).
  * Fang  _et al._ [2010] M. Fang, J. Yang, F. F. Balakirev, Y. Kohama, J. Singleton, B. Qian, Z. Q. Mao, H. Wang, and H. Q. Yuan, Phys. Rev. B 81, 020509(R) (2010).
  * Khim  _et al._ [2010] S. Khim, J. W. Kim, E. S. Choi, Y. Bang, M. Nohara, H. Takagi, and K. H. Kim, Phys. Rev. B 81, 184511 (2010).
  * Gurevich [2003] A. Gurevich, Phys. Rev. B 67, 184515 (2003).
  * Kasahara  _et al._ [2014] S. Kasahara, T. Watashige, T. Hanaguri, Y. Kohsaka, T. Yamashita, Y. Shimoyama, Y. Mizukami, R. Endo, H. Ikeda, K. Aoyama, T. Terashima, S. Uji, T. Wolf, H. V. Löhneysen, T. Shibauchi, and Y. Matsuda, PNAS 111, 16309 (2014).
  * Zhou  _et al._ [2021] N. Zhou, Y. Sun, C. Y. Xi, Z. S. Wang, J. L. Zhang, Y. Zhang, Y. F. Zhang, C. Q. Xu, Y. Q. Pan, J. J. Feng, Y. Meng, X. L. Yi, L. Pi, T. Tamegai, X. Xing, and Z. Shi, Phys. Rev. B 104, L140504 (2021).
  * Zhang  _et al._ [2011] J. L. Zhang, L. Jiao, Y. Chen, and H. Q. Yuan, Front. Phys. 6, 463 (2011).
  * Saito  _et al._ [2015] Y. Saito, Y. Kasahara, J. Ye, Y. Iwasa, and T. Nojima, Science 350, 409 (2015).
  * Sims  _et al._ [2019] H. Sims, D. N. Leonard, A. Y. Birenbaum, Z. Ge, T. Berlijn, L. Li, V. R. Cooper, M. F. Chisholm, and S. T. Pantelides, Phys. Rev. B 100, 144103 (2019).

Figure 1: Temperature dependence of the resistivity $\rho_{xx}$ in samples A-D
at zero magnetic field. The inset shows the superconducting transitions in
greater detail over a more limited temperature range and the determination of
$T_{c}^{onset}$. The sheet resistance is given in terms of the resistance
quantum of a Cooper pair $h/4e^{2}$ = 6.45 k$\Omega$.

Figure 2: Cross-sectional HAADF-TEM images of the lowest and highest $T_{c}$
samples, A and D, respectively. Sample D clearly demonstrates a 3 unit-cell
FeSe (001) epitaxial layer on STO (001), along with the expected crystalline
FeTe and amorphous Te capping layers, while clear images of the grown layers
in sample A could not be achieved.

Figure 3: MR of sample A at various temperatures in the range $1.2$ K $\leq
T\leq 15$ K with magnetic field (a) in-plane ($H\parallel ab$) and (b) out-of-
plane ($H\parallel c$).

Figure 4: Temperature dependence of the upper critical field in the parallel
field configuration $H\parallel ab$ of sample A using the $\rho_{n}^{10}$,
$\rho_{n}^{50}$, $\rho_{n}^{90}$, and $\rho_{n}^{onset}$ criteria. The overall
qualitative behavior of each curve is similar. Lines are a guide to the eye.

Figure 5: The dimensionless upper critical field $\bar{H}=\alpha h$ plotted as
a function of the normalized temperature $t=T/T_{c}$ for samples (a) A, (b) B,
(c) C, and (d) D, along with their corresponding WHH fits for $H\parallel ab$
and $H\parallel c$. The horizontal dashed line in each plot corresponds to the
strongly coupled paramagnetic limit $\mu_{0}H_{c2}^{P2}=1.86(1+\lambda_{el-
ph})\>T_{c}$, while the ($\alpha=0,\lambda_{so}=0$) curves demonstrate the
case of strictly orbital-limitation and yield the results of Eq. 3.

Figure 6: Temperature dependence of the upper critical field anisotropy
parameter $\gamma=H_{c2}^{ab}(T)/H_{c2}^{c}(T)$. Filled data points are
obtained from actual experimental results where $H_{c2}^{ab}$ and $H_{c2}^{c}$
were obtained at the same temperature. Low temperature, open (unfilled) points
for sample C were calculated using WHH-predicted values of $H_{c2}^{ab}(T)$
and experimentally measured values of $H_{c2}^{c}(T)$. Open points for sample
D were calculated using experimental (predicted) values of $H_{c2}^{ab}(T)$
($H_{c2}^{c}(T)$) above 20 K and predicted (experimental) values at and below
20 K. Lines are a guide to the eye.


# Cross-Policy Compliance Detection via Question Answering

Marzieh Saeidi  Facebook AI  Majid Yazdani  Facebook AI  Andreas Vlachos
Facebook AI  University of Cambridge

###### Abstract

Policy compliance detection is the task of ensuring that a scenario conforms
to a policy (e.g. a claim is valid according to government rules or a post in
an online platform conforms to community guidelines). This task has been
previously instantiated as a form of textual entailment, which results in poor
accuracy due to the complexity of the policies. In this paper we propose to
address policy compliance detection via decomposing it into question
answering, where questions check whether the conditions stated in the policy
apply to the scenario, and an expression tree combines the answers to obtain
the label. Despite the initial upfront annotation cost, we demonstrate that
this approach results in better accuracy, especially in the cross-policy setup
where the policies during testing are unseen in training. In addition, it
allows us to use existing question answering models pre-trained on existing
large datasets. Finally, it explicitly identifies the information missing from
a scenario in case policy compliance cannot be determined. We conduct our
experiments using a recent dataset consisting of government policies, which we
augment with expert annotations and find that the cost of annotating question
answering decomposition is largely offset by improved inter-annotator
agreement and speed.

##  1 Introduction

Figure 1: A policy with its decomposition (questions and expression tree),
scenario and label.222Policy source can be found here
https://www.gov.uk/housing-benefit/how-to-claim

Policy compliance detection is the task of ensuring that a scenario conforms
to a policy. It is a task that occurs in various contexts, e.g. in ensuring
correct application of government policies (Saeidi et al., 2018; Holzenberger
et al., 2020), enforcing community guidelines on social media platforms
(Waseem and Hovy, 2016) or the resolution of legal cases (Zhong et al., 2020),
etc.

While the task can be modelled as a text-pair classification similar to
textual entailment (Dagan et al., 2009; Bowman et al., 2015), i.e. whether a
scenario complies with a certain policy, this often results in poor accuracy
due to the complexity of the policy descriptions, which often contain multiple
clauses connected with each other Holzenberger et al. (2020). Moreover, even
though some types of policy compliance detection can be tackled via supervised
classification, e.g. recognizing hateful memes (Kiela et al., 2020) and other
forms of hate speech or abusive language on social platforms Davidson et al.
(2017), this requires substantial amounts of training data for one specific
policy which is expensive and time-consuming. Furthermore, the requirement for
policy-specific training data renders supervised classification difficult to
adapt when policies change, e.g. customs rules changing with Brexit, novel
forms of hate speech in the context of the Covid pandemic, etc.

In this paper we propose to address policy compliance detection via
decomposing it to question answering (QA) and an expression tree. Each policy
description is converted into a set of questions whose answers are combined in
a logical form represented by the expression tree to produce the label. This
allows for better handling of complex policies, and to take advantage of the
large-scale datasets and models developed for question answering (Rajpurkar et
al., 2016; Clark et al., 2019). In addition, the questioning of the scenario
with respect to parts of the policy identifies which parts were used in order
to determine compliance and which information is missing in case the scenario
could not be resolved. This in turn, enhances the interpretability of the
model’s decisions and can identify areas for improvement. Figure 2 shows an
example application of our approach in the context of a government policy
where the missing date of the move in the scenario results in its compliance
with the policy being unresolved.

We demonstrate the benefits of policy compliance detection via QA using a
dataset that contains policies, decomposition questions and expression trees
and scenarios. While the policies and scenarios in the dataset are taken from
ShARC Saeidi et al. (2018), we augment them with expression trees and answers
to each question for all the scenarios and policies to create the Question
Answering for Policy Compliance (QA4PC ) dataset. The results of our
experiments demonstrate that we can achieve an accuracy of $0.69$ for policies
unseen during the training (an increase of $25$ absolute points over the
entailment approach) and an accuracy of $0.59$ (an increase of $22$ absolute
points) when no in-domain data is available for training. We also show that
our approach is more robust compared to entailment when faced with policies of
increasing complexity. In addition, we find that the cost of annotating
question answering decomposition and expression trees is largely offset by
improved inter-annotator agreement and speed. Finally, we release the QA4PC
dataset to the community to facilitate further research.333Please contact the
authors to access the dataset.

##  2 Previous Work on Policy Compliance

Policy compliance as an entailment task has been studied by Holzenberger et
al. (2020) and referred to as statutory reasoning entailment. They find that
the application of machine reading models exhibits low performance, whether or
not they have been fine-tuned to the legal domain, due to the complexity of
the policies. As an alternative approach, they propose a rule-based system.
While their work focuses solely on the US Tax Law, the dataset we use in our
experiments Saeidi et al. (2018) contains a wide range of policies from
different countries. Furthermore, the dataset used in Holzenberger et al.
(2020) only contains scenarios that are either compliant or not compliant with
a policy. However, in real world use, systems should be capable of recognizing
that a decision cannot be made in cases there is information missing from the
scenario.

ShARC Saeidi et al. (2018) is a dataset for conversational question answering
based on government policies. In ShARC, the task is to predict what a system
should output given a policy and a conversation history, which can be an
answer or a follow-up clarification question. The task is evaluated in an end-
to-end fashion, measuring the system’s ability to provide the correct
conversation utterance, However, policy compliance detection is neither
investigated nor evaluated.

Similarity search or matching has been used to match a new post to known
violating posts on social media such as hoaxes, an objectionable video or
image, or a hateful meme Ferreira and Vlachos (2016); Wang et al. (2018); fbm
. For textual content, this can be compared to an entailment task. This
approach requires a bank of existing violating content for each policy. By
using the policy description and decomposing it into several QA tasks,
breaches for new policies can be detected, as we show in our experiments.

Much work has focused on learning important elements of privacy policies to
assist users in understanding the terms of the policies devised by different
websites. Shvartzshanider et al. (2018) uses question answering to extract
different elements of privacy policies that are informative to the users.
Nejad et al. (2020) and Mustapha et al. (2020) assign pre-defined categories
to privacy policy paragraphs using supervised classification. While these
works aim to help users in understanding complex text of privacy policies,
they do not aim to identify compliance and they mainly focus on privacy
policies. In our work, we look at a wide range of government policies, for
different governments and focus on an approach for cross-policy compliance
detection.

##  3 Policy Compliance Detection via QA

In this section we describe our proposed approach for policy compliance
detection via question answering. First we formulate the task as textual
entailment following previous work Holzenberger et al. (2020). Then we
describe how our proposed expression trees combined with QA operate. Finally
we discuss the implications of our approach for training and data annotation
requirements.

###  3.1 Policy Compliance Detection (PCD)

We define the task of policy compliance detection (PCD) as deciding whether a
scenario is compliant with a policy. For a policy and a scenario, the task is
to provide a label in [_Yes_ , _No_ , _Not Enough Information_]. We use NEI in
short for Not Enough Information for the remaining of the paper. For an input
$(p,s)$, the output is the label $l$. See Figure 2 for an example. If framed
as textual entailment, a policy is the premise and the scenario is the
hypothesis, as there are typically multiple scenarios per policy. The labels
[_Yes_ , _No_ , _NEI_] correspond to [_entailment_ , _contradiction_ ,
_neutral_].

###  3.2 Question Answering for PCD

Our proposed decomposition of policy compliance detection into question
answering infers the label using the answers to the questions and the
expression tree. More formally, the QA model will take an input
$(s,q_{i})_{i=1\ldots K-1}$ and output $a_{i}\in$ [_Yes_ , _No_ , _NEI_],
where $K$ is the number of questions representing the policy $p$. The
expression tree combines the answers to the questions, $\\{a_{i}\\}_{i=0\ldots
K-1}$, using a logical expression based on the rules in the policy description
to infer the final label $l$ of whether a scenario is in compliance with the
policy or not. Figure 2 shows an example of a policy decomposed into questions
and an expression tree and how it is applied to a scenario to infer a label.

An expression tree can contain OR, AND and NOT operators. Expression trees can
be evaluated in the same way that logical expressions are evaluated where
_Yes_ is considered as True and _No_ as False. Similar to logical expressions,
we can evaluate an expression tree even when the answers to some of the
questions are NEI. For example, the inferred label for “Q0 AND Q1” is False if
the answer to Q0 is False and the answer to Q1 is _NEI_. If the answer to Q0
is True and the answer to Q1 is _NEI_ , the inferred label is _NEI_. It is
worth noting that we do not always need to correctly predict the answers to
all the questions in the expression tree in order to infer the correct label.
For example, for the expression tree in Figure 2, if the answer to Q3 is
correctly predicted as No, the model doesn’t need to make correct predictions
for Q0, Q1 or Q2.

The underlying QA task is similar to BoolQ Clark et al. (2019) with the
exception that an answer can be NEI. This can be compared to SQUAD 2.0
Rajpurkar et al. (2018) where the corpus contains unanswerable questions, but
the answers are spans extracted from the passage instead of boolean. In this
work, we assume that the expression trees are provided and leave the task of
inferring them for future work.

###  3.3 Training and Data Requirements

Expression trees are not required during the training of our approach; they
are only used during the testing to infer the label for the scenario given the
policy. For this reason, we only provide the annotation of expression trees
for the data that will be used for evaluation (dev and test sets) to
demonstrate the benefits of our proposed approach. Using QA as the subtask, in
which we decompose PCD into, enables us to apply our approach to policies
unseen in training (a.k.a. cross policy setup), since QA models are typically
developed for and evaluated on their generalization ability to questions
unseen in their training data.

##  4 Data

In this section we discuss the creation of the question-answering for policy
detection (QA4PC) dataset which we introduce in this work to support our
experiments. We create QA4PC using the policies and the scenarios from the
publicly available training and development splits of existing dataset ShARC
Saeidi et al. (2018). A scenario in ShARC is a real-world situation described
by a user who is conversing with a system to find out whether they comply with
a policy. Each conversation utterance in ShARC has a policy, a question and an
answer. An utterance may include a conversation history (a list of QAs) and/or
a scenario which is built from a conversation list (a list of questions and
answers). The answer to an utterance can be Yes/No or a follow-up
clarification question. 444More details on ShARC data can be found here
https://sharc-data.github.io/data.html

However, while ShARC conversations contain some of the questions related to
the policies, we found that many were missing, as the conversation progress
may render answers to them irrelevant. A such example is provided in Figure 6
in the Appendix. Details on number of QAs per policy for both datasets are
discussed in section 4.4.

As neither entailment-style classification nor the question answering
expression trees require complete question-answer sets per instance or
expression trees for training, we convert $70\%$ of ShARC policies for
training of entailment and QA tasks (see section 4.1 for more details). The
procedure for converting the data is explained in the next section. In order
to ensure the evaluation datasets (dev and test sets) are correctly annotated,
we augment the remaining $30\%$ of ShARC policies with questions and
expression trees, and the scenarios with the entailment labels and answers to
all questions of the related policy (see Section 4.2). Two annotators, co-
authors of this paper and UK nationals, were involved in the annotation
described in this section.

###  4.1 Converting ShARC to PCD

#### Entailment Instances

We take utterances that have non-empty scenarios. The entailment answer to the
scenario is Yes/No if the answer to the conversation utterance in ShARC is
Yes/No and the conversation history is empty. Conversation history refers to a
list of questions and answers that has already been exchanged between an agent
and a user. Otherwise, the assigned label is NEI. This is because if the
conversation history is not empty or the final answer is a follow up question,
some of the necessary information related to the policy is not mentioned in
the scenario and is acquired from history or will be answered by the user in
the next step of the conversation in the form of an answer to the follow up
question.

#### QA Instances

We construct the set of unique follow-up questions related to each policy over
all the scenarios as the set of questions for that policy. If any question
from this set appears in the conversation corresponding to a scenario (a list
of QAs), we create a QA instance where the passage is the scenario, and the
answer is the answer to that question in the QA list. For all the other
questions in this set, we add a QA instance using the scenario and the answer
NEI. More details about the ShARC dataset and its processing are included in
the Appendix in section A.1.

#### In-Domain Supervision

We call the converted ShARC dataset ShARC-PC. There are $482$ policies,
$4,576$ scenarios and $10,398$ QAs in the ShARC training set. The policies
used in ShARC-PC are distinct from those used in QA4PC. Note that the labels
in ShARC-PC are not always accurate because they are assigned using a
heuristic, thus even though it is in-domain as it is derived from conversation
about government policies, it is noisy. This was specially noted when
augmenting the ShARC data to obtain QA4PC.

###  4.2 Expression Tree Annotation

#### Annotation Procedure

Each policy is decomposed by the annotators into a set of questions and an
expression tree based on the rules in the policy description. Each question is
assigned an ID (e.g. Q0, Q1 ) which is used in the expression tree. For each
policy, some of the questions can be extracted from the ShARC dataset as
mentioned in the previous section. These questions are provided to the
annotators. An annotator can change the question or remove it. Annotators can
also add questions to expression tree if the existing questions are not
sufficient. The UI for this annotation task is depicted in Figure 7 in the
appendix. It took the annotators an average of $113$ seconds to annotate each
tree.

Figure 2: Label/answer types for training and evaluation datasets for
entailment (left) and QA (right) tasks.

#### Quality Control

Annotators were asked to mark difficult instances and also take note during
the annotation if they were in doubt about an instance. After the first round
of annotation, the annotators discussed their notes and clarified the
guidelines. Finally, each annotator reviewed the other annotator’s expression
trees and made updates if necessary based on the interim discussion. To
further ensure the quality of the expression trees, we asked a third annotator
(third co-author, not involved in the annotation otherwise) to review a sample
of $50$ trees. These trees were selected proportional to their complexity,
i.e. number of questions and logical operators. Out of $50$ trees, the third
annotator did not agree with $4$ annotations, i.e. an agreement of $92\%$. The
annotator marked $6$ additional instances where they believed the framing of
questions can be improved.

###  4.3 Entailment and QA Annotations

#### Annotation Procedure

Each scenario is paired with its corresponding policy. An annotator is
required to choose a label from (Yes/No/NEI) with regards to the policy
compliance (i.e. entailment instance). In the same annotation task, the
annotators are asked to provide answers to each individual question in the
expression tree of the policy (QA instances). The UI for this task is shown in
Figure 8 in the appendix.

#### Quality Control

We compared the inferred labels to all scenarios using the expression tree of
the corresponding policy against the entailment label provided as a sanity
check. Annotators were also asked to take notes during the annotation if
needed and discussed their notes and the discrepancies identified by the
sanity check. In some cases, answers were adjusted appropriately. In other
cases, the expression tree was updated, and the annotators ensured that the
question and answers were updated accordingly.

###  4.4 QA4PC Dataset

In this section, we present statistics of the QA4PC. As neither our approach
nor the entailment baseline require training data annotated with expression
trees, we divide the dataset only into dev and test sets. There are $193$
policies in QA4PC across both test and dev sets. Each policy has between $1$
and $9$ questions and between $2$ and $55$ scenario instances. Table 1 shows
more properties of dev and test sets such as the total number of scenarios and
individual QAs and average number of QAs per policy. The published dataset
contains the training, dev and test sets.

| Policies | Scenarios | QAs | Avg QA/Policy
---|---|---|---|---
Dev | $60$ | $437$ | $1,600$ | $2.31$
Test | $133$ | $1,099$ | $3,492$ | $2.30$
Table 1:  Number of instances in QA4PC dataset.

Policies in QA4PC have an average number of $2.31$ QAs. This is higher than
the average number of QAs per policy in ShARC-PC which is $1.73$, because
annotators added additional questions to the policies during the annotation
when required. Figure 2 shows the distribution of entailment labels and
answers to the QA instances between training (i.e. ShARC-PC) and evaluation
(i.e. QA4PC) sets. As the figure shows, the training data has a higher number
of instances where the entailment labels (left) or answers to the questions
are NEI (right).

| Avg over Scenarios | Avg over Policies
---|---|---
Training data | ShARC-PC | BoolQ | SNLI | ShARC-PC | BoolQ | SNLI
| Test Set
Entailment | $0.44\pm 0.02$ | NA | $0.37\pm 0.02$ | $0.44\pm 0.03$ | NA | $0.42\pm 0.02$
QA ET | $0.69\pm 0.01$ | $0.59\pm 0.02$ | NA | $0.69\pm 0.02$ | $0.62\pm 0.02$ | NA
| Dev Set
Entailment | $0.44\pm 0.01$ | NA | $0.35\pm 0.02$ | $0.48\pm 0.01$ | NA | $0.39\pm 0.02$
QA ET | $0.70\pm 0.01$ | $0.65\pm 0.02$ | NA | $0.74\pm 0.02$ | $0.65$ | NA
Table 2: Cross-policy accuracy averaged on all scenarios and per policy on
test and dev sets in terms of macro-accuracy, averaged over 5 runs with
varying seeds. QA ET refers to QA decomposition with Expression Trees.

##  5 Experimental Setup

#### Model

We use RoBERTa Liu et al. (2019) for both the entailment baseline and the QA
subtask and learn three-way classifiers for each. We used the implementation
in the huggingface Wolf et al. (2020) library. We also experimented with a T5
model Raffel et al. (2019) on the dev set. Since the results of T5 model were
very similar to RoBERTa, we report the performance on all the tasks using
RoBERTa.

#### Entailment

We use a cross-encoder set up for the task. For each instance, we combine the
policy, scenario and the question using the following format: “premise:
[policy] SEP hypothesis: [scenario] CLS”. The embeddings of the token CLS is
used to classify the instance. This is a common approach for entailment
modelling using pre-trained encoders Devlin et al. (2018); Raffel et al.
(2019); Holzenberger et al. (2020).

#### Question Answering for PCD

In the QA approach to PCD, we decompose the task into a QA subtask and
expression trees, where we use the expression tree of a policy to combine the
answers to the questions in the policy to infer the label. A QA model is
trained on QA instances of the dataset. For each QA instance, we combine the
scenario and the question using the format: “passage: [scenario] SEP question:
[question] CLS”. This is a common formulation of QA tasks Raffel et al. (2019)
using transformer models. The embedding of the CLS token is used to assign a
label to each instance.

#### Out-of-Domain Training Data

We use the BoolQ Clark et al. (2019) dataset for training the QA model. BoolQ
only contains instances (passage and question pair) with Yes/No answers. We
pair random passages and questions to create instances with NEI answers. To
train the entailment model, we use the SNLI dataset Bowman et al. (2015). To
adapt the labels to our task, we convert entailment to Yes, contradiction to
No and neutral to NEI.

#### Evaluation Metric

We use macro-accuracy (averaged over 3 classes of PCD labels/QA answers)
averaged over scenarios, since the labels and answers are not balanced in our
dataset (see Figure 2). As some policies have more scenarios than others, we
also report results averaged over policies, where we first calculate the
macro-accuracy per policy and then average over policies.

#### Hyperparameters

We did a manual tuning of hyperparameters using the dev set. Batch size was
set to $16$, learning rate to $5e-5$, Adam epsilon to $1e-8$ and maximum
gradient norm of $1.0$. We trained all the models for a maximum of 5 epochs
with early stopping using the loss on the dev set. We run each experiment $5$
times with different random seeds and report the mean and variance. The models
were ran on a machine with 8 Tesla V100-SXM2 GPUs, each with $16$ GB memory.
Each epoch of model training takes about $25$ seconds for the entailment
baseline and $35$ seconds for the QA subtask on ShARC-PC. Number of trainable
parameters of the model is $124,647,939$.

##  6 Results

###  6.1 Model Accuracy

The accuracies of different approaches in detecting policy compliance and
using different training data are presented in Table 2.

| ShARC-PC | BoolQ
---|---|---
QA | $0.68\pm 0.02$ | $0.59\pm 0.01$
Table 3: The performance of the QA task where models are trained on in-domain and out-of-domain data and evaluated on the test set of QA4PC. | Yes | No | NEI
---|---|---|---
QA | $0.84\pm 0.03$ | $0.54\pm 0.05$ | $0.66\pm 0.03$
PCD | $0.80\pm 0.03$ | $0.68\pm 0.04$ | $0.59\pm 0.02$
Table 4: The performance of QA and PCD tasks per label using the model trained
on ShARC-PC.

The first column of the table shows the performance of both approaches in
detecting policy compliance on unseen policies when trained on data from
ShARC-PC. The accuracy of the entailment approach is $0.44$ while the QA
decomposition approach reaches an accuracy of $0.69$. Table 3 shows the
performance of the QA subtask (macro-accuracy over QA instances) on QA4PC test
set using in-domain (i.e. ShARC-PC) and out-of-domain (i.e. BoolQ) training
data. As mentioned previously, we do not need to answer all the questions in a
policy correctly in order to get the correct final label for PCD. For example,
in Figure 2, if the answer to Q3 is correctly predicted as No, inferring a
correct final label is independent of the remaining answers. This is the
reason that a QA model with an accuracy of $0.68$ can still achieve an
accuracy of $0.69$ on the PCD task.

Since we have a class imbalance in our dataset, we show the accuracy of the
model per label for both the QA and the PCD tasks in Table 4. The most
difficult answer for the QA model to predict is _No_ , while the answer with
the lowest prediction accuracy for the PCD task is _NEI_.

#### Transfer Learning

We define transfer learning as learning from a training set that is out-of-
domain (i.e. not government policies). Table 2 shows transfer learning results
for the PCD task based on training on BoolQ for QA-based approach and SNLI for
the entailment baseline. As we can see, the decomposition approach achieves an
accuracy of $0.59$ when trained on BoolQ data. On the other hand, the accuracy
of the entailment model trained on SNLI is $0.37$ which is only slightly
better than a random system. This is likely due to entailment detection
between policy and scenario being a more difficult task due to the complexity
of the policy description, and it is not possible to obtain good accuracy
without in-domain data.

#### Performance per Policy

We evaluate the accuracy of both approaches averaged over policies to show how
well we can infer compliance with a new policy. The results averaged over
policies are shown in the right side of Table 2. The accuracies averaged over
policies are similar or slightly higher than those averages over all the
scenarios. This indicates that there are policies that are difficult to do
inference on which have a high number of instances. This is evident for both
the entailment and QA decomposition approaches.

Figure 3: Performance (test set) per policy vs. the complexity of the policy,
i.e. the number of questions in the expression tree.

Figure 3 shows the average performance of the PDC task using QA decomposition
approach for individual policies in the test set of QA4PC versus the
complexity of the policy description, i.e. number of questions in the policy
expression tree. The size of the circles indicate the number of examples in
the dataset for a policy. We used Kendall tau to find the correlation between
the number of questions in a policy and the accuracy. Tau coefficient between
accuracy and number of questions in policies using QA and entailment
approaches are $-0.12$ (p-value of $0.05$) and $-0.17$ (p-value of $0.01$)
respectively. This shows that the accuracy through the entailment approach is
more negatively affected as the complexity of a policy increases. Finally, it
is worth noting that a policy with $8$ questions has a high number of
instances. The average accuracy over these instances is $0.42$ which
contributes to the slightly lower performance when averaged over the scenarios
reported in Table 2.

#### Limited Supervision

In Figure 4, we show the performance of the model on QA4PC as the amount of
in-domain training data from ShARC-PC increases. The results are shown in blue
and green for the QA decomposition and the entailment approach respectively.
As figure shows, with only $500$ scenario instances, we achieve an accuracy of
$0.60$ through the QA decomposition approach. The entailment approach reaches
an accuracy of $0.40$ using more than $1000$ instances. Note that the number
of training instances are based on the number of scenario instances which can
include a higher number of QA instances. However we will demonstrate the
annotation time is not higher for more QA instances in the next section. Also,
it takes only an average of $113$ seconds to annotate the expression tree (see
section 4.2) which is done only once per policy and used for multiple
scenarios.

Figure 4: Performance of two approaches on the test set as the number of
training samples (scenario) from in-domain data increases.

###  6.2 Annotation Accuracy and Efficiency

To perform policy compliance detection via QA decomposition, not only we need
annotations for expression trees, we also need to annotate the answers to all
the questions of a policy for a related scenario instance. Therefore, for each
scenario, instead of needing one entailment label, we need annotators to
provide answers to all the questions in the expression tree of a policy. One
may argue that this approach is more costly as a higher number of annotations
is required and that this effort could be used instead to annotate more data
on the entailment level that can lead to equally good or better performance.
In this section, we describe our experiment to demonstrate the effectiveness
of the annotation for the QA decomposition approach in comparison to the
entailment.

#### Task Setup

50 policies were selected from QA4PC. We include policies with different
number of questions (and therefore complexity) in this experiment. For each
policy, one scenario is selected randomly from the dataset. Policies are
unique to ensure that the accuracy and speed of annotators are not affected by
their familiarity with the policies. We then create two annotation tasks for
each scenario-policy instance. In the first task, the annotators are presented
with the policy and the scenario and they are required to choose one
entailment label. In the second task, the scenario and _all_ the questions
related to the policy are presented to the annotators. An annotator is
required to provide an answer to each of the questions.

#### Agreement and Speed

We used two annotators (unpaid volunteers who are not authors of the paper but
are native English speakers) to perform both annotation tasks. The average
time to provide an entailment label for a policy and a scenario is $39$
seconds across two annotators. The average time to answer _all_ the questions
of a policy with respect to a scenario is $27$ seconds. The annotators
highlighted the difficulty of annotating the entailment task because they had
to create a mental breakdown of a policy while they found the QA task easier
to perform as the policy is already decomposed into individual questions. The
Cohen’s Kappa for entailment agreement between the annotators is $0.38$ and
for individual questions $0.49$. The agreement on the inferred labels using
the expression trees based on the answers to the questions is $0.70$. This
agreement is higher than the agreement for the QA annotation, because as we
discussed in section 6.1, not all the the questions need to be answered
correctly to infer the correct final label. The results of this annotation
experiment shows that annotating the QA labels has lower cost (faster)
compared to annotating the overall entailment label, even though an annotator
needs to complete more annotations.

##  7 Discussion

#### Assumptions

In this work, we assume that we can decompose policy descriptions into
independent questions that can be executed in parallel. However, designing
independent questions is not always straightforward. E.g. it could be useful
to have a question in an expression tree that is a follow-up on another
question and therefore a QA model will have to be executed sequentially taking
into account the earlier question in formulating the follow up one.

#### Potential for Automatic Generation of Expression Trees

In our work, we assume that the expression trees are provided. Providing an
expression tree for a policy by those who create it is unlikely to be a
substantial overhead, and it can lead to better policy definitions. In
addition, in the case of policies on sensitive issues such as hate speech, it
might be undesirable to have the expression tree and questions inferred
automatically. Moreover, an expression tree for a policy is created only once
but used multiple times. As discussed in the results section, the cost of
creating expression trees when using the QA approach is offset by the gains in
performance compare to the entailment approach.

Nevertheless, on-boarding all the policies of an existing organisation can be
time-consuming and can benefit from automation, and with human-in-the-loop
approaches we can ensure the accuracy of the trees. Finally, a potential
benefit of automating the generation of expression trees is that it can be
done jointly with the task of policy compliance detection, such that we
generate trees that result in a higher accuracy on the downstream task.

#### Use of Task Descriptions

The use of task descriptions was recently studied in Weller et al. (2020) to
answer questions based on a passage. Our work is similar to this work since we
use policy descriptions to learn PCD and conduct evaluation in a cross-policy
setting. While they answer questions from passage descriptions, we focus on
learning policy compliance given a policy and a scenario. Description of
relations has been used in Obamuyide and Vlachos (2018) to perform zero-shot
relation classification.

#### Question Decomposition

Answering complex questions has been a long-standing challenge in natural
language processing. Many researchers explored decomposing questions into
simpler questions Wolfson et al. (2020); Min et al. (2019); Ferrucci et al.
(2010); Perez et al. (2020). In question decomposition, the objective is to
convert a complex question into a list of inter-related sub-questions. While
in question decomposition questions are generated automatically, in our work,
we consider them given. On the other hand, the answers to the decomposed
questions are combined in a more complex manner using an expression tree with
logical operators, while in these works there is no attempt to combine the
answers.

##  8 Conclusion

In this work, we propose to address the task of policy compliance detection
via decomposing a policy description into an expression tree. The expression
tree consists of a set of questions and a logical expression that combines the
questions to infer a final label. Our experiments show that compared to the
existing entailment approach Holzenberger et al. (2020), QA decomposition
results in a better model accuracy in a cross-policy setting using in-domain
($0.69$ vs. $0.44$) and out-of-domain ($0.59$ vs. $0.37$) training data.
Furthermore, we show that while there is an upfront cost of annotation for
expression trees, the cost of QA decomposition annotations is lower than the
cost of annotating the entailment task while reaching a higher agreement.

Future work can investigate ways to generate expression trees for policy
descriptions automatically. Further, it will be beneficial to demonstrate that
this approach is suitable in detecting policy compliance in other domains such
as community standards implemented by social media platforms which requires
annotation of relevant datasets.

##  9 Broader Impact Statement

Tools based on our method could potentially improve automated policy
enforcement efforts and could also help individuals understand how certain
policies may apply to their circumstances. The proposed method offer better
interpretability and performance when little data in available. This can
provide some insights on where a model may have made a mistake.

We use an existing transformer-based model to show the effectiveness of our
method. Transformer-based models are known not to be computationally
effective. However, we use a pre-trained model and fine-tune it on a
relatively small dataset.

## References

  * (1) Open-sourcing photo- and video-matching technology to make the internet safer.  https://about.fb.com/news/2019/08/open-source-photo-video-matching/.  Accessed: 2020-05-10.
  * Bowman et al. (2015) Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015\.  A large annotated corpus for learning natural language inference.  In _EMNLP_.
  * Clark et al. (2019) Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019.  Boolq: Exploring the surprising difficulty of natural yes/no questions.  Dataset License: CC BY-SA 3.0 https://creativecommons.org/licenses/by-sa/3.0/.
  * Dagan et al. (2009) I. Dagan, B. Dolan, B. Magnini, and D. Roth. 2009.  Recognizing textual entailment: Rational, evaluation and approaches.  _Natural Language Engineering_ , 15.
  * Davidson et al. (2017) Thomas Davidson, Dana Warmsley, Michael Macy, and Ingmar Weber. 2017.  Automated hate speech detection and the problem of offensive language.  In _Proceedings of the 11th International AAAI Conference on Web and Social Media_ , ICWSM ’17, pages 512–515.
  * Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_.
  * Ferreira and Vlachos (2016) William Ferreira and Andreas Vlachos. 2016.  Emergent: a novel data-set for stance classification.  In _Proceedings of the 2016 conference of the North American chapter of the association for computational linguistics: Human language technologies_ , pages 1163–1168.
  * Ferrucci et al. (2010) David Ferrucci, Eric Brown, Jennifer Chu-Carroll, James Fan, David Gondek, Aditya A Kalyanpur, Adam Lally, J William Murdock, Eric Nyberg, John Prager, et al. 2010.  Building watson: An overview of the deepqa project.  _AI magazine_ , 31(3):59–79.
  * Holzenberger et al. (2020) Nils Holzenberger, Andrew Blair-Stanek, and Benjamin Van Durme. 2020.  A dataset for statutory reasoning in tax law entailment and question answering.  In _Proceedings of the 2020 Natural Legal Language Processing (NLLP) Workshop, 24 August 2020, San Diego, US_.
  * Kiela et al. (2020) Douwe Kiela, Hamed Firooz, Aravind Mohan, Vedanuj Goswami, Amanpreet Singh, Pratik Ringshia, and Davide Testuggine. 2020.  The hateful memes challenge: Detecting hate speech in multimodal memes.  _arXiv preprint arXiv:2005.04790_.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Roberta: A robustly optimized bert pretraining approach.  _arXiv preprint arXiv:1907.11692_.
  * Min et al. (2019) Sewon Min, Victor Zhong, Luke Zettlemoyer, and Hannaneh Hajishirzi. 2019.  Multi-hop reading comprehension through question decomposition and rescoring.  _arXiv preprint arXiv:1906.02916_.
  * Mustapha et al. (2020) Majd Mustapha, Katsiaryna Krasnashchok, Anas Al Bassit, and Sabri Skhiri. 2020.  Privacy policy classification with xlnet (short paper).  In _Data Privacy Management, Cryptocurrencies and Blockchain Technology_ , pages 250–257. Springer.
  * Nejad et al. (2020) Najmeh Mousavi Nejad, Pablo Jabat, Rostislav Nedelchev, Simon Scerri, and Damien Graux. 2020.  Establishing a strong baseline for privacy policy classification.  In _IFIP International Conference on ICT Systems Security and Privacy Protection_ , pages 370–383. Springer.
  * Obamuyide and Vlachos (2018) Abiola Obamuyide and Andreas Vlachos. 2018.  Zero-shot relation classification as textual entailment.  In _Proceedings of the First Workshop on Fact Extraction and VERification (FEVER)_ , pages 72–78.
  * Perez et al. (2020) Ethan Perez, Patrick Lewis, Wen-tau Yih, Kyunghyun Cho, and Douwe Kiela. 2020.  Unsupervised question decomposition for question answering.  _arXiv preprint arXiv:2002.09758_.
  * Raffel et al. (2019) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2019.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _arXiv preprint arXiv:1910.10683_.
  * Rajpurkar et al. (2018) Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018.  Know what you don’t know: Unanswerable questions for squad.  _arXiv preprint arXiv:1806.03822_.
  * Rajpurkar et al. (2016) Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016.  Squad: 100,000+ questions for machine comprehension of text.  _arXiv preprint arXiv:1606.05250_.
  * Saeidi et al. (2018) Marzieh Saeidi, Max Bartolo, Patrick Lewis, Sameer Singh, Tim Rocktäschel, Mike Sheldon, Guillaume Bouchard, and Sebastian Riedel. 2018.  Interpretation of natural language rules in conversational machine reading.  _arXiv preprint arXiv:1809.01494_.
  * Shvartzshanider et al. (2018) Yan Shvartzshanider, Ananth Balashankar, Thomas Wies, and Lakshminarayanan Subramanian. 2018.  Recipe: Applying open domain question answering to privacy policies.  In _Proceedings of the Workshop on Machine Reading for Question Answering_ , pages 71–77.
  * Wang et al. (2018) Xuezhi Wang, Cong Yu, Simon Baumgartner, and Flip Korn. 2018.  Relevant document discovery for fact-checking articles.  In _Companion Proceedings of the The Web Conference 2018_ , pages 525–533.
  * Waseem and Hovy (2016) Zeerak Waseem and Dirk Hovy. 2016.  Hateful symbols or hateful people? predictive features for hate speech detection on Twitter.  In _Proceedings of the NAACL Student Research Workshop_ , pages 88–93, San Diego, California. Association for Computational Linguistics.
  * Weller et al. (2020) Orion Weller, Nicholas Lourie, Matt Gardner, and Matthew E Peters. 2020.  Learning from task descriptions.  _arXiv preprint arXiv:2011.08115_.
  * Wolf et al. (2020) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2020.  Transformers: State-of-the-art natural language processing.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pages 38–45, Online. Association for Computational Linguistics.
  * Wolfson et al. (2020) Tomer Wolfson, Mor Geva, Ankit Gupta, Matt Gardner, Yoav Goldberg, Daniel Deutch, and Jonathan Berant. 2020.  Break it down: A question understanding benchmark.  _Transactions of the Association for Computational Linguistics_ , 8:183–198.
  * Zhong et al. (2020) Haoxi Zhong, Chaojun Xiao, Cunchao Tu, Tianyang Zhang, Zhiyuan Liu, and Maosong Sun. 2020.  Jec-qa: A legal-domain question answering dataset.  In _Proceedings of AAAI_.

##  Appendix A Appendices

###  A.1 ShARC Data Processing

ShARC dataset contains government policies from a number of countries
including the UK and the USA. for each policy, several scenarios are provided.
A scenario is a life situation described by a user who is interested to know
whether they comply with the given policy. Scenarios were created by providing
a full or partial list of questions and their answers from a path in the tree
to the labelers. Labelers were then asked to write a life situation compatible
with those questions and answers. Annotators were also told to feel free to
provide extra context and information, even if not directly relevant to the
provided questions. Note that many tree instances in ShARC do not have any
related scenarios and only rely on interactive information seeking with a
human to arrive at an answer.

Even though a binary tree for each policy can be inferred from the ShARC
instances, we encountered two issues with the data: 1) The trees were not
always complete and some fundamental questions were missing. 2) Trees with
only possible Yes/No for each node as done in ShARC are not expressive enough
(or it’s too complicated to express a policy fully in a binary tree) to
capture the semantics of the policies in many cases. For example, binary trees
do not capture OR operators very well where the answer to some questions are
not required in order to infer the final answer. Without an accurate binary
tree, we can not guarantee to infer a correct label given the answers to the
granular tasks.

Figure 5 shows that binary trees in ShARC can not deal with the answers _Not
Enough Information_ whereas our proposed expression trees do.

Figure 5: The logical tree of the expression “Q0 OR Q1 Or Q2” proposed in our
work on the left and the decision tree proposed by ShARC. Figure 6: An example
of a tree in ShARC where we added extra questions in QA4PC as we believed they
were essential in evaluating a scenario. in QA4PC, four questions were added
about the type of benefit corresponding to each bullet point.

###  A.2 Annotation

Figure 7: The annotation UI for expression tree annotations. Figure 8: The
annotation UI for QA and entailment tasks.


# Comments on the infinitely divisibility of the Conway–Maxwell–Poisson
distribution

Greg Markowsky  Preet Patel
<EMAIL_ADDRESS><EMAIL_ADDRESS>
Department of Mathematics  Monash University  Australia

###### Abstract

In an elegant recent paper [2], Geng and Xia settled the question of the
infinite divisibility of the Conway–Maxwell–Poisson distribution, using in
large part several results from complex analysis. In this note we show how
these complex analytic methods can be circumvented, thereby giving a proof of
their result which is completely elementary.

The Conway–Maxwell–Poisson distribution is a discrete probability distribution
on the nonnegative integers which admits the probability mass function

$P(X=k)=\frac{1}{Z(\lambda,\nu)}\frac{\lambda^{k}}{(k!)^{\nu}}.$ (1)

Here $\lambda,\nu>0$, or $\nu=0$ and $0<\lambda<1$, and $Z(\lambda,\nu)$ is
the correct constant so that $\sum_{k=0}^{\infty}P(X=k)=1$. This distribution
has been many uses in the modelling of count data and other applications; see
[1, 3, 4, 6, 7, 5]. Recently, in [2], the following result was proved.

######  Theorem 0.1.

The distribution defined by (1) is infinitely divisible if and only if $\nu=0$
or $\nu=1$.

To understand the importance of this result in connection with limit theory
and the law of small numbers, the reader is referred to the discussion in [2].
The proof given there used complex analytic methods in several places. In this
brief note we will show how these methods can be avoided if desired.

Let us begin by assuming that $X$ has distribution $CMP(\lambda,\mu)$; that
is, with probability mass function given by (1). If we assume that it is
infinitely divisible, then, as noted by the authors of [2], it must be
compound Poisson, and there is an i.i.d. sequence $Y_{1},Y_{2},\dots$ of
r.v.’s such that

$X\stackrel{{\scriptstyle d}}{{=}}\sum_{n=1}^{N}Y_{n},$

where $N\sim Pois(\mu)$ $(\mu>0)$ is independent of the $Y$’s. If the
probability generating function of $Y_{1}$ is denoted by $G(x)$, that is,

$G(x)=\sum_{r=1}^{\infty}P(Y_{1}=r)x^{r},$ (2)

then

$e^{\mu G(x)}=\sum_{k=0}^{\infty}\frac{(\lambda x)^{k}}{(k!)^{\nu}}.$ (3)

Now, an interesting feature of (3) is that the radius of convergence of the
right side is clearly infinite, while the left side is only guaranteed to
exist for $|x|<1$. The authors of [2] required at this step the knowledge that
the series (2) has an infinite radius of convergence, and for this they turned
to complex analysis, regarding $x$ as a complex variable. The argument
required substantial subtlety, as for general power series we may have
intricate cancellation between the terms; however, matters may be simplified
here by noting that all series in question have non-negative coefficients,
precluding difficulties due to cancellation, and the complex arguments may be
replaced by the following lemma.

######  Lemma 0.2.

Suppose $(q_{n})_{n}$ is a sequence of non-negative reals with
$\sum_{n=0}^{\infty}q_{n}<\infty$. Define

$\displaystyle G(x)=\sum_{n=0}^{\infty}q_{n}x^{n}$ (4)

and let $F(x)=e^{\mu G(x)}$ for some $\mu>0$, defined wherever the series
converges. Then $F$ is analytic, and has a power series expansion
$F(x)=\sum_{n=0}^{\infty}t_{n}x^{n}$ near zero. Suppose this series for $F$
has an infinite radius of convergence. Then the series defining $G$ also has
an infinite radius of convergence.

###### Proof.

The $\mu$ can be absorbed into $G$, and the coefficients remain non-negative,
so we disregard $\mu$. Suppose (4) has a finite radius of convergence, $R$
(note that $R\geq 1$ since the $q_{n}$’s are summable). For $0<x<R$, we have

$\displaystyle\sum_{n=0}^{\infty}q_{n}x^{n}=G(x)<e^{G(x)}$
$\displaystyle=\sum_{k=0}^{\infty}\frac{\left(\sum_{n=0}^{\infty}q_{n}x^{n}\right)^{k}}{k!}$
(5)

Since all terms in this double series are positive and the series converges
absolutely and uniformly near 0, we may expand and rearrange without being
cautious, and we recall that the series on the right must rearrange to
$\sum_{k=0}^{\infty}t_{k}x^{k}$. We conclude that each $t_{m}$ can be
expressed as a linear combination of the $q_{n}$’s with positive coefficients,
and this linear combination must contain the term $q_{m}$ from the $k=1$ term
in (5). We conclude that $q_{m}\leq t_{m}$, and this is enough to show that
$R=\infty$. ∎

The authors of [2] then proceeded through a series of probabilistic arguments
in order to show that $|G(x)|\leq C|x|^{d}$ for some $C,d>0$ ($d$ is an
integer) and $|x|$ sufficiently large. As viewed in the complex plane, $G$ is
now known to be entire, and it is a standard exercise in complex analysis
classes (generalizing Liouville’s Theorem) to show that this implies that $G$
is a polynomial of degree $d$. This result can be deduced from Cauchy’s
Integral Formula or from a number of other closely related results. However,
again in our case the fact that all coefficients are non-negative allows this
argument to be replaced by the following lemma.

######  Lemma 0.3.

Suppose $G(x)=\sum_{n=0}^{\infty}q_{n}x^{n}$, is a power series with
$q_{n}\geq 0$ for all $n$. Suppose also that this series has infinite radius
of convergence and satisfies $|G(x)|\leq C|x|^{d}$ for some $C,d>0$ and $|x|$
sufficiently large. Then $G$ is a polynomial of degree at most $d$.

###### Proof.

Suppose there is some nonzero $q_{m}$ with $m>d$. For $x\geq 0$ we have
$G(x)\geq q_{m}x^{m}$, however as $m>d$ this quantity will dominate $Cx^{n}$
for any constant $C>0$, $n\leq d$ and $x$ sufficiently large. The lemma
follows from this. ∎

## Acknowledgements

The authors would like to think Xi Geng and Aihua Xia for helpful
conversations.

## References

  * [1] F. Daly and R. Gaunt.  The Conway-Maxwell-Poisson distribution: distributional theory and approximation.  ALEA, 13:635–658, 2016.
  * [2] X. Geng and A. Xia.  When is the Conway–Maxwell–Poisson distribution infinitely divisible?  Statistics & Probability Letters, 181:109264, 2022.
  * [3] J. Hilbe.  Modeling count data.  Cambridge University Press, 2014.
  * [4] B. Li, H. Zhang, and Jiao He.  Some characterizations and properties of COM-Poisson random variables.  Communications in Statistics-Theory and Methods, 49(6):1311–1329, 2020.
  * [5] T. Pogány.  Integral form of the COM-Poisson normalization constant.  Statistics & Probability Letters, 119:144–145, 2016.
  * [6] K. Sellers, D. Borle, and G. Shmueli.  The COM-Poisson model for count data: a survey of methods and applications.  Applied Stochastic Models in Business and Industry, 28(2):104–116, 2012.
  * [7] S. Yip, Y. Zou, R. Hung, and K. Yiu.  Forecasting number of corner kicks taken in association football using overdispersed distribution.  arXiv preprint arXiv:2112.13001, 2021.