# Predicting the nucleon electroweak form factors using spin-improved
holographic light-front wavefunctions

Mohammad Ahmady1, Dipankar Chakrabarti2, Chandan Mondal3, Ruben Sandapen4
1Department of Physics, Mount Allison University, Sackville, New Brunswick,
Canada, E4L 1E6
2Department of Physics, Indian Institute of Technology Kanpur, Kanpur 208016,
India
3Institute of Modern Physics, Chinese Academy of Sciences, Lanzhou 730000,
China $\&$ School of Nuclear Science and Technology, University of Chinese
Academy of Sciences, Beijing 100049, China $\&$ CAS Key Laboratory of High
Precision Nuclear Spectroscopy, Institute of Modern Physics, Chinese Academy
of Sciences, Lanzhou 730000, China
4Department of Physics, Acadia University, Wolfville, Nova-Scotia, Canada, B4P
2R6

###### Abstract

We construct spin-improved holographic light-front wavefunctions for the
nucleons (viewed as quark-diquark systems) and use them to successfully
predict their electromagnetic Sachs form factors, their electromagnetic charge
radii, as well as the axial form factor, charge and radius of the proton. The
confinement scale is the universal mass scale of light-front holography,
previously extracted from spectroscopic data for light hadrons. With the Dirac
and Pauli form factors normalized using the quark counting rules and the
measured anomalous magnetic moments respectively, the masses of the quark and
diquark are the only remaining adjustable parameters. We fix them using the
data set for the proton’s Dirac-to-Pauli form factor ratio, and then predict
all other data without any further adjustments of parameters. Agreement with
data at low momentum-transfer is excellent. Our findings support the idea that
light (pseudoscalar and vector) mesons and the nucleons share a non-
perturbative universal holographic light-front wavefunction which is modified
differently by their spin structures.

##  I Introduction

Previous papers Forshaw and Sandapen (2012); Ahmady et al. (2016, 2017, 2019),
and especially Ref. Ahmady et al. (2020), support the idea that light
pseudoscalar and vector mesons share a universal holographic light-front
wavefunction modified by their different spin structures. By the universal
holographic wavefunction, we mean the $N$-quark valence wavefunction Brodsky
et al. (2013); Brodsky and de Téramond (2006)

$\Psi(x,\zeta)=X(x)e^{iL\varphi}\frac{\phi(\zeta)}{\sqrt{2\pi\zeta}}$ (1)

where $\zeta$ is given by

$\mathbf{\zeta}=\sqrt{\frac{x}{(1-x)}}\left|\sum^{N-1}_{j=1}x_{j}\mathbf{b}_{\perp,j}\right|$
(2)

where $N=2$ for mesons and $N=3$ for baryons. The variable $x$ is the light-
front momentum fraction of one quark (labelled active) while $x_{j}$ are the
momentum fractions associated with the $N-1$ remaining quarks (labelled
spectator cluster). The variables $\mathbf{b}_{\perp,j}$ are the transverse
positions of the spectator quarks relative to the active one. The quantum
number $L$ is the relative orbital angular momentum between the active quark
and the spectator cluster. In a quark-antiquark meson, when the quark is
active (spectator), the antiquark is spectator (active). Therefore,
$\zeta=\sqrt{x\bar{x}}b_{\perp}$, where $\bar{x}\equiv(1-x)$ and $b_{\perp}$
is the transverse separation between the quark and the antiquark. Eq. (1) also
describes a baryon in which the spectator quarks constitute a diquark cluster.
In this case, when any one quark is active, it carries momentum fraction $x$,
and the remaining two form a spectator diquark cluster which carries momentum
fraction $\bar{x}$.

The factorized form of Eq. (1) is realized in the conformal limit of light-
front QCD, i.e. when quantum loops and quark masses are neglected. In this
limit, the transverse mode, $\phi(\zeta)$, satisfies the light-front
holographic Schrödinger Equation Brodsky et al. (2015)

$\left(-\frac{\mathrm{d}^{2}}{\mathrm{d}\zeta^{2}}-\frac{4L^{2}-1}{4\zeta^{2}}+U(\zeta,J)\right)\phi_{J}(\zeta)=M^{2}\phi_{J}(\zeta)\;$
(3)

where $J$ is the meson’s spin. Note that Eq. (1) and (3) do not carry any
information on the spins of the quarks. In light-front holographic QCD, this
is a consequence of the underlying assumption that the quark helicity
wavefunction decouples from the confinement dynamics. Eq. (3) is said to be
holographic since it maps onto the equation of motion for bosonic modes,
$\Phi_{J}(x,z)=\Phi_{J}(z)e^{iP\cdot x}$, propagating in a $\mathrm{AdS}_{5}$
spacetime modified by a dilaton field, $\varphi(z)$, according to the
dictionary:

$\zeta=z\hskip 28.45274pt;\hskip
28.45274pt\Phi_{J}(z)=(R/z)^{J-3/2}e^{-\varphi(z)/2}\phi_{J}(\zeta)\hskip
28.45274pt;\hskip 28.45274pt(m_{5}R)^{2}=-(2-J)^{2}+L^{2}$ (4)

where $z,R$ and $m_{5}$ are the fifth dimension, radius of curvature and mass
parameter in $\mathrm{AdS}_{5}$ spacetime. The confinement potential in
physical spacetime is then fixed by the form of the dilaton field in
$\mathrm{AdS}_{5}$:

$U(\zeta,J)=\frac{1}{2}\varphi^{\prime\prime}(z)+\frac{1}{4}\varphi^{\prime}(z)^{2}+\left(\frac{2J-3}{2z}\right)\varphi^{\prime}(z).$
(5)

The holographic mapping does not fix $\varphi(z)$ but the underlying conformal
symmetry requires it to be quadratic: $\varphi(z)=\kappa^{2}z^{2}$ Brodsky et
al. (2014). The confining potential in Eq. (3) is then given by

$U(\zeta,J)=\kappa^{4}\zeta^{2}+2\kappa^{2}(J-1)\;.$ (6)

We refer to the emerging mass scale, $\kappa$, which simultaneously sets the
strength of the dilaton field in $\mathrm{AdS}_{5}$ and the confinement scale
in physical spacetime, as the light-front holographic mass scale. Solving Eq.
(3) with the potential, Eq. (6), yields

$\phi_{nL}(\zeta)=\kappa^{1+L}\sqrt{\frac{2n!}{(n+L)!}}\zeta^{1/2+L}\exp{\left(\frac{-\kappa^{2}\zeta^{2}}{2}\right)}~{}L_{n}^{L}(\kappa^{2}\zeta^{2})\;$
(7)

so that, for $n=J=L=0$ states, Eq. (1) becomes

$\Psi(x,\zeta)=\frac{\kappa}{\sqrt{\pi}}X(x)\exp\left(-\frac{\kappa^{2}\zeta^{2}}{2}\right)\;.$
(8)

The longitudinal wavefunction, $X(x)$, can be found by mapping the spacelike
EM form factor in physical spacetime onto the EM form factor in
$\mathrm{AdS}_{5}$. The latter is computed using the free EM current
propagator i.e. the equation of motion for the EM field in pure (without a
dilaton field) $\mathrm{AdS}_{5}$. In physical spacetime, the EM form factor
is given by the Drell-Yan-West formula Drell and Yan (1970); West (1970):

$F_{\mathrm{DYW}}(Q^{2})=2\pi\int_{0}^{1}\frac{\mathrm{d}x}{x\bar{x}}\int\zeta\mathrm{d}\zeta
J_{0}\left(\zeta Q\sqrt{\frac{\bar{x}}{x}}\right)|\Psi(x,\zeta)|^{2}\;,$ (9)

while, in $\mathrm{AdS}_{5}$, it is given by

$F_{\mathrm{AdS}}(Q^{2})=R^{3}\int_{0}^{1}\mathrm{d}x\int\frac{\mathrm{d}z}{z^{3}}J_{0}\left(zQ\sqrt{\frac{\bar{x}}{x}}\right)|\Phi(z)|^{2}\;.$
(10)

The holographic mapping of Eq. (9) onto Eq. (10) requires that
$X(x)=\sqrt{x\bar{x}}$, so that

$\Psi(x,\zeta)=\frac{\kappa}{\sqrt{\pi}}\sqrt{x\bar{x}}\exp\left(-\frac{\kappa^{2}\zeta^{2}}{2}\right)\;.$
(11)

After accounting for light quark masses using the Brodsky-de Téramond
prescription Brodsky and de Téramond (2009), Eq. (11) becomes

$\Psi(x,\zeta)\propto\sqrt{x\bar{x}}\exp\left(-\frac{\kappa^{2}\zeta^{2}}{2}\right)\exp\left(-\frac{\bar{x}m^{2}_{1}+xm^{2}_{2}}{2\kappa^{2}x\bar{x}}\right)\;$
(12)

where $m_{1}$ is the quark mass. In a meson, $m_{2}$ is the antiquark mass
and, in a baryon, it is the diquark mass.

Alternatively, the $\mathrm{AdS}_{5}$ EM form factor can be computed using a
dressed EM current propagator, i.e. the equation of motion for an EM field in
a dilation-modified $\mathrm{AdS}_{5}$. In this case, the EM form factor
admits a twist expansion, and to leading twist accuracy, is given by Brodsky
et al. (2015)

$F_{\mathrm{AdS}}(Q^{2})=\int\mathrm{d}xx^{Q^{2}/4\kappa^{2}}\;.$ (13)

We can recover Eq. (13) in physical spacetime by using the wavefunction,

$\Psi(x,\zeta)=\frac{\kappa\bar{x}}{\sqrt{\pi}\ln(1/x)}\exp\left(-\frac{\kappa^{2}\zeta^{2}\bar{x}}{2x\ln(1/x)}\right)$
(14)

in the Drell-Yan-West formula, Eq. (9).

Unlike Eq. (11), Eq. (14) is not symmetric under the transformation
$x\leftrightarrow 1-x$, and can be thought as an effective wavefunction, in
contradistinction with Eq. (11) which is a true valence wavefunction. In this
paper, we shall use Eq. (11) and not Eq. (14). It is thus clear that we are
not assuming any contributions from higher Fock sectors. This distinguishes
our paper from previous work Maji and Chakrabarti (2016); Gutsche et al.
(2015) which use Eq. (14).

Eq. (7) implies that states with the same $(n,L)$ quantum numbers share the
same holographic wavefunction, regardless of the value of $J$. For instance,
the pion ($J=0$) and the $\rho~{}(J=1)$ meson, have identical holographic
wavefunctions and thus identical decay constants Ahmady et al. (2020). This is
a direct consequence of the decoupling of the quark helicity from the
confinement dynamics in light-front holography. To overcome this shortcoming,
we account for the helicities of quarks in a dynamical way, i.e. by
multiplying the universal holographic wavefunction by a momentum-dependent
helicity wavefunction, so that

$\Psi_{h_{1}h_{2}}(x,\mathbf{k})=S_{h_{1}h_{2}}(x,\mathbf{k})\times\Psi(x,k_{\perp}),$
(15)

where $\mathbf{k}=k_{\perp}e^{i\theta_{k_{\perp}}}$ is conjugate to
$\mathbf{b}$, and $\Psi(x,k_{\perp})$ is the two-dimensional Fourier transform
of Eq. (12). The indices $h_{1,2}$ denote the helicities of $q_{1,2}$ (recall
that for mesons, $q_{1}\equiv q,q_{2}\equiv\bar{q}$ while for baryons,
$q_{1}\equiv q,q_{2}\equiv[qq]$). We assume that
$S_{h_{1}h_{2}}(x,\mathbf{k})$ follows from the point-like coupling of the
hadron with the partons, while all bound states effects are captured by the
holographic wavefunction $\Psi(x,k_{\perp})$. For vector mesons , we thus have
Forshaw and Sandapen (2012); Ahmady et al. (2016)

$S^{V(\lambda)}_{h_{q}h_{\bar{q}}}(x,\mathbf{k})\propto\frac{\bar{v}_{h_{\bar{q}}}((1-x)P^{+},-\mathbf{k})}{\sqrt{\bar{x}}}[\epsilon^{\lambda}_{V}\cdot\gamma]\frac{u_{h_{q}}(xP^{+},\mathbf{k})}{\sqrt{x}},$
(16)

where $\epsilon^{\lambda}_{V}$ are the polarization $4$-vectors for the vector
meson, while for pseudoscalar mesons,

$S^{P}_{h_{q}h_{\bar{q}}}(x,\mathbf{k})\propto\frac{\bar{v}_{h_{\bar{q}}}(\bar{x}P^{+},-\mathbf{k})}{\sqrt{\bar{x}}}[\gamma^{5}]\frac{u_{h_{q}}(xP^{+},\mathbf{k})}{\sqrt{x}}\;.$
(17)

Indeed, it was recently shown in Ref. Ahmady et al. (2020), that Eqs. (16) and
(17), together with the universal holographic wavefunction, Eq. (12), lead to
successful predictions for the vector-to-pseudoscalar meson radiative form
factors.

In this paper, we extend this idea to the nucleons. Keeping in mind that Eq.
(3) is also true for a quark-diquark bound state, we assume that the spin
structure follows from the point-like coupling of a nucleon to a quark and a
diquark. Now, the spin of the diquark can be $S_{D}=0$ or $S_{D}=1$, so that
we have two possible spin structures. With a scalar $(S_{D}=0)$ diquark, we
choose

$S^{N}_{h_{N}h_{q}}(x,\mathbf{k})\propto\frac{\bar{u}_{h_{q}}(xP^{+},\mathbf{k})}{\sqrt{x}}[\mathbb{1}]\frac{u_{h_{N}}(P^{+},\mathbf{0})}{\sqrt{1}}$
(18)

analogous to the spin structure, Eq. (17), for the pseudoscalar mesons. For an
axial-vector diquark $(S_{D}=1)$, we assume

$S^{N(\lambda)}_{h_{N}h_{q}}(x,\mathbf{k})\propto\frac{\bar{u}_{h_{q}}(xP^{+},\mathbf{k})}{\sqrt{x}}[(\epsilon^{\lambda}_{D}\cdot\gamma)\gamma^{5}]\frac{u_{h_{N}}(P^{+},\mathbf{0})}{\sqrt{1}}$
(19)

analogous to the spin structure, Eq. (16), for a vector meson. In Eq. (19),
the diquark polarization vectors read

$\displaystyle\epsilon^{0}_{D}$ $\displaystyle=$
$\displaystyle\left(\frac{(1-x)P^{+}}{M_{D}},-\frac{M_{D}}{\bar{x}P^{+}},0_{\perp}\right),$
(20) $\displaystyle{\epsilon^{\pm}_{D}}$ $\displaystyle=$
$\displaystyle\frac{1}{\sqrt{2}}(0,0,\mp 1,-i),$ (21)

where $M_{D}$ is the diquark mass. Note that the spin structures, given by
Eqs. (18) and (19), were proposed in Ref. Brodsky et al. (2001) (for scalar
diquarks only) and in Ref. Ellis et al. (2009) (for both scalar and axial-
vector diquarks) with the assumption of a heavy diquark, i.e. $M_{D}=M_{N}$
and $1-x\approx 1$. Here, we do not make this approximation, but we do recover
the results of Ref. Ellis et al. (2009) when doing so.

In light-front holographic QCD, the EM form factors are directly computed
using a dressed EM current in $\mathrm{AdS}_{5}$ Chakrabarti and Mondal
(2013); Sufian et al. (2017). The alternative approach, which we follow here,
is to compute the form factors in physical spacetime using the DYW formula,
Eq. (9), with a wavefunction which is itself obtained from light-front
holography. The computation is then restricted to the spacelike region. This
approach is taken in Gutsche et al. (2014, 2015); Maji and Chakrabarti (2016)
using the spin wavefunctions of Ref. Ellis et al. (2009) together with the
effective holographic wavefunction, Eq. (14), or models inspired by Eq. (14).
Refs. Gutsche et al. (2014, 2015) consider only a scalar diquark while Ref.
Maji and Chakrabarti (2016) considers both a scalar and axial-vector diquark.
The number of adjustable parameters in Refs. Gutsche et al. (2014); Maji and
Chakrabarti (2016) is typically large $(>10)$.

In this paper, we shall use Eq. (18) and (19), together with the universal
holographic wavefunction, Eq. (12), to predict the form factors of the
nucleons. Our only adjustable parameters are the quark and diquark masses.
Once fixed, using a judiciously chosen data set, our remaining predictions do
not contain any adjustable parameters. We also predict the charge and magnetic
radii of the nucleons, as well as the axial charge of the proton.

##  II Spin-improved holographic wavefunctions for the nucleons

Using light-front field theory Lepage and Brodsky (1980), we start by deriving
explicit expressions for the spin-improved holographic wavefunctions of the
nucleons using Eq. (15), together with the spin structures given by Eq. (18)
and Eq. (19). Our notation for the resulting wavefunction is:
$\Psi^{h_{N}[D]}_{h_{q},h_{D}}$ where $D=\\{S,A\\}$ denotes a scalar or axial-
vector diquark, while $h_{N}=\\{\uparrow,\downarrow\\}$,
$h_{q}=\\{+1/2,-1/2\\}$ and $h_{D}=\\{0,\pm 1\\}$ are the helicities of the
nucleon, quark and diquark respectively.

For a scalar diquark, we obtain

$\displaystyle\Psi^{\uparrow[S]}_{\frac{1}{2},0}(x,\mathbf{k})$
$\displaystyle=$
$\displaystyle\left(M_{N}+\frac{m_{q}}{x}\right)\Psi(x,k_{\perp}),$
$\displaystyle\Psi^{\uparrow[S]}_{-\frac{1}{2},0}(x,\mathbf{k})$
$\displaystyle=$
$\displaystyle-\left(\frac{k_{\perp}e^{i\theta_{k_{\perp}}}}{x}\right)\Psi(x,k_{\perp}),$
$\displaystyle\Psi^{\downarrow[S]}_{\frac{1}{2},0}(x,\mathbf{k})$
$\displaystyle=$
$\displaystyle\left(\frac{k_{\perp}e^{-i\theta_{k_{\perp}}}}{x}\right)\Psi(x,k_{\perp}),$
$\displaystyle\Psi^{\downarrow[S]}_{-\frac{1}{2},0}(x,\mathbf{k})$
$\displaystyle=$
$\displaystyle\left(M_{N}+\frac{m_{q}}{x}\right)\Psi(x,k_{\perp}),$ (22)

where $M_{N}$ is the nucleon mass and $m_{q}$ is the mass of the quark. For an
axial-vector diquark, we find

$\displaystyle\Psi^{\uparrow[A]}_{\frac{1}{2},0}(x,\mathbf{k})$
$\displaystyle=$
$\displaystyle-\left(\frac{M_{N}m_{q}}{M_{D}}\frac{\bar{x}}{x}+\frac{M_{D}}{\bar{x}}\right)\Psi(x,k_{\perp}),$
$\displaystyle\Psi^{\uparrow[A]}_{-\frac{1}{2},0}(x,\mathbf{k})$
$\displaystyle=$
$\displaystyle\left(\frac{k_{\perp}e^{i\theta_{k_{\perp}}}}{x}\frac{\bar{x}M_{N}}{M_{D}}\right)\Psi(x,k_{\perp}),$
$\displaystyle\Psi^{\uparrow[A]}_{\frac{1}{2},1}(x,\mathbf{k})$
$\displaystyle=$
$\displaystyle\left(\frac{k_{\perp}e^{-i\theta_{k_{\perp}}}}{x}\right)\Psi(x,k_{\perp}),$
$\displaystyle\Psi^{\uparrow[A]}_{-\frac{1}{2},1}(x,\mathbf{k})$
$\displaystyle=$
$\displaystyle\left(M_{N}+\frac{m_{q}}{x}\right)\Psi(x,k_{\perp}),$ (23)
$\displaystyle\Psi^{\uparrow[A]}_{\frac{1}{2},-1}(x,\mathbf{k})$
$\displaystyle=$ $\displaystyle 0,$
$\displaystyle\Psi^{\uparrow[A]}_{-\frac{1}{2},-1}(x,\mathbf{k})$
$\displaystyle=$ $\displaystyle 0,$

for a spin-up nucleon, and

$\displaystyle\Psi^{\downarrow[A]}_{\frac{1}{2},0}(x,\mathbf{k})$
$\displaystyle=$
$\displaystyle\left(\frac{k_{\perp}e^{-i\theta_{k_{\perp}}}}{x}\frac{(\bar{x})M_{N}}{M_{D}}\right)\Psi(x,k_{\perp}),$
$\displaystyle\Psi^{\downarrow[A]}_{-\frac{1}{2},0}(x,\mathbf{k})$
$\displaystyle=$
$\displaystyle\left(\frac{M_{N}m_{q}}{M_{D}}\frac{\bar{x}}{x}+\frac{M_{D}}{\bar{x}}\right)\Psi(x,k_{\perp}),$
$\displaystyle\Psi^{\downarrow[A]}_{\frac{1}{2},1}(x,\mathbf{k})$
$\displaystyle=$ $\displaystyle 0,$
$\displaystyle\Psi^{\downarrow[A]}_{-\frac{1}{2},1}(x,\mathbf{k})$
$\displaystyle=$ $\displaystyle 0,$ (24)
$\displaystyle\Psi^{\downarrow[A]}_{\frac{1}{2},-1}(x,\mathbf{k})$
$\displaystyle=$
$\displaystyle-\left(M_{N}+\frac{m_{q}}{x}\right)\Psi(x,k_{\perp}),$
$\displaystyle\Psi^{\downarrow[A]}_{-\frac{1}{2},-1}(x,\mathbf{k})$
$\displaystyle=$
$\displaystyle\left(\frac{k_{\perp}e^{i\theta_{k_{\perp}}}}{x}\right)\Psi(x,k_{\perp}),$

for a spin-down nucleon.

Using these wavefunctions, the nucleon state can be expanded on a quark-
diquark basis. For example, for a spin-up proton state, we have

$\displaystyle\mid P\uparrow\rangle$ $\displaystyle=$
$\displaystyle\int\frac{\mathrm{d}x\mathrm{d}^{2}\mathbf{k}}{16\pi^{3}\sqrt{x\bar{x}}}\left[\Psi_{\frac{1}{2},0}^{\uparrow[S]}(x,\mathbf{k})\mid[ud]_{S}u^{\uparrow}\rangle+\Psi_{-\frac{1}{2},0}^{\uparrow[S]}(x,\mathbf{k})\mid[ud]_{S}u^{\downarrow}\rangle~{}~{}\right.$
(25)
$\displaystyle+\Psi_{\frac{1}{2},0}^{\uparrow[A]}(x,\mathbf{k})\left(\frac{1}{3}\mid[ud]^{0}_{A}u^{\uparrow}\rangle-\frac{\sqrt{2}}{3}\mid[uu]^{0}_{A}d^{\uparrow}\rangle\right)$
$\displaystyle+\Psi_{-\frac{1}{2},0}^{\uparrow[A]}(x,\mathbf{k})\left(-\frac{1}{3}\mid[ud]^{0}_{A}u^{\downarrow}\rangle+\frac{\sqrt{2}}{3}\mid[uu]^{0}_{A}d^{\downarrow}\rangle\right)$
$\displaystyle+\Psi_{\frac{1}{2},-1}^{\uparrow[A]}(x,\mathbf{k})\left(\frac{\sqrt{2}}{3}\mid[ud]^{-1}_{A}u^{\uparrow}\rangle+\frac{2}{3}\mid[uu]^{-1}_{A}d^{\uparrow}\rangle\right)$
$\displaystyle+\left.\Psi_{-\frac{1}{2},1}^{\uparrow[A]}(x,\mathbf{k})\left(-\frac{\sqrt{2}}{3}\mid[ud]^{1}_{A}u^{\downarrow}\rangle+\frac{2}{3}\mid[uu]^{1}_{A}d^{\downarrow}\rangle\right)\right],$

where $[qq]^{h_{D}}_{D}$ indicates a diquark. It is worth making several
comments regarding Eq. (25).

First, if we neglect the axial-vector diquark contributions, Eq. (25) implies
that the proton is a superposition of $L=0$ and $L=1$ states. This is also
predicted in the supersymmetric formulation of light-front holography Brodsky
et al. (2016) where baryons and mesons, differing by one unit of orbital
angular momentum, are supersymmetric partners. Tetraquarks are the second
supersymmetric partners of baryons with the same orbital angular momentum. For
example, the proton has superpartners $h_{1}(1170)(L=1)$ and
$\sigma(500)(L=0)$, the latter being identified as a tetraquark candidate
Nielsen and Brodsky (2018). However, unlike the supersymmetric holographic
wavefunction, our wavefunction remains asymmetric under the interchange
$x\leftrightarrow\bar{x}$, even in the chiral limit. This is because our the
spin wavefunction does not decouple from the dynamics, as is the case in
light-front holography.

Second, the spin-statistics theorem dictates that the diquark cluster must
have isospin and spin either both equal to zero or both equal to one, so that
the combined spin-isospin wavefunction is symmetric. This is because the
relative orbital angular momentum of the two quarks is zero and their colour
wavefunction is antisymmetric. Therefore the spin-statistics theorem does not
allow a $[uu]$ scalar diquark, implying that the $d$-quark cannot be single.
Therefore, in a purely quark-scalar-diquark model for the proton, the
$d$-quark cannot be active. To remedy this shortcoming, we also consider the
possibility of an axial-vector diquark. Then, as can be seen in Eq. (25), this
opens up the possibility that the $d$-quark to be active.

Thirdly, in the absence of dynamical spin effects, i.e. when all (non-
vanishing) $\Psi^{h_{p}[D]}_{h_{q},h_{D}}(x,\mathbf{k})\to\Psi(x,k_{\perp})$,
Eq. (25) reduces to the spin-flavour $\mbox{SU}(4)$ quark model expansion for
the proton state Ma et al. (2002). Thus, we can think of dynamical spin
effects as breaking the $\mbox{SU}(4)$ spin-flavour symmetry of the quark
model.

Finally, interchanging $\uparrow$ and $\downarrow$ in Eq. (25) yields the
expansion for a spin-down proton, and the isospin transformation
$u\leftrightarrow d$ yields the expansion for the neutron state.

##  III Analytic expressions for the form factors

At the quark level, the Dirac and Pauli form factors, $F^{f}_{1}(q^{2})$ and
$F^{f}_{2}(q^{2})$, are defined via

$\displaystyle\langle P+q,\uparrow|J_{\mathrm{EM},f}^{+}(0)|P,\uparrow\rangle$
$\displaystyle=$ $\displaystyle 2P^{+}F^{f}_{1}(q^{2}),$ (26)
$\displaystyle\langle
P+q,\uparrow|J_{\mathrm{EM},f}^{+}(0)|P,\downarrow\rangle$ $\displaystyle=$
$\displaystyle-(q^{1}-iq^{2})\frac{P^{+}}{M_{N}}F^{f}_{2}(q^{2}),$ (27)

where

$J^{\mu}_{\mathrm{EM},f}=\bar{\psi}_{f}\gamma^{\mu}\psi_{f}$ (28)

is the EM vector quark current with flavour $f=[u,d]$. On the other hand, the
axial form factor, $G^{f}_{A}(q^{2})$, parametrize the nucleon matrix elements
of the axial EW current:

$\displaystyle\langle P+q,\uparrow|J_{\mathrm{EW},f}^{+}(0)|P,\uparrow\rangle$
$\displaystyle=$ $\displaystyle 2P^{+}G^{f}_{A}(q^{2}),$ (29)

where

$J_{\mathrm{EW},f}^{\mu}=\bar{\psi}_{f}\gamma^{\mu}\gamma^{5}\psi_{f}\;.$ (30)

To evaluate the left-hand-sides of Eqs. (26), (27) and (29), we expand the
nucleon state using Eq. (25). As we mentioned above, we compute the form
factors in the impulse approximation where the single quark in Eq. (25) is the
active quark coupling to the photon (or $Z$) while the diquark cluster is
spectator.

The scalar diquark contributions to the EM form factors then read

$\displaystyle F^{S(u)}_{1}(Q^{2})$ $\displaystyle=$
$\displaystyle\int\mathrm{d}x\frac{\mathrm{d}^{2}\mathbf{k}}{16\pi^{3}}\left[{\psi^{S\uparrow}_{\frac{1}{2},0}}^{\dagger}(x,\mathbf{k}^{\prime})\psi^{S\uparrow}_{\frac{1}{2},0}(x,\mathbf{k})+{\psi^{S\uparrow}_{-\frac{1}{2},0}}^{\dagger}(x,\mathbf{k}^{\prime})\psi^{S\uparrow}_{-\frac{1}{2},0}(x,\mathbf{k})\right]$
(31)

and

$\displaystyle F^{S(u)}_{2}(Q^{2})$ $\displaystyle=$
$\displaystyle\\!\\!-\frac{2M_{N}}{q_{\perp}e^{-i\theta_{q_{\perp}}}}\int\mathrm{d}x\frac{\mathrm{d}^{2}\mathbf{k}}{16\pi^{3}}\left[{\psi^{S\uparrow}_{\frac{1}{2},0}}^{\dagger}(x,\mathbf{k}^{\prime})\psi^{S\downarrow}_{\frac{1}{2},0}(x,\mathbf{k})+{\psi^{S\uparrow}_{-\frac{1}{2},0}}^{\dagger}(x,\mathbf{k}^{\prime})\psi^{S\downarrow}_{-\frac{1}{2},0}(x,\mathbf{k})\right],$
(32)

where $\mathbf{k}^{\prime}=\mathbf{k}-\bar{x}\mathbf{q}$. Using our spin-
improved holographic wavefunctions, Eq. (II), we find

$\displaystyle F^{S(u)}_{1}(Q^{2})$ $\displaystyle\propto$
$\displaystyle\int\mathrm{d}x\left[\left(M_{N}+\frac{m_{u}}{x}\right)^{2}+\frac{(1-x)\kappa^{2}}{x}\left(1-\frac{(1-x)q_{\perp}^{2}}{4\kappa^{2}x}\right)\right]$
(33)
$\displaystyle\times\exp\left(-\frac{(1-x)q_{\perp}^{2}}{4\kappa^{2}x}\right)\exp\left(-\frac{xM_{ud}^{2}+(1-x)m_{u}^{2}}{\kappa^{2}x(1-x)}\right)$

and

$\displaystyle F^{S(u)}_{2}(Q^{2})$ $\displaystyle\propto$
$\displaystyle\int\mathrm{d}x\left[\left(M_{N}+\frac{m_{u}}{x}\right)\frac{\bar{x}}{x}\right]\exp\left(-\frac{\bar{x}q_{\perp}^{2}}{4\kappa^{2}x}\right)\exp\left(-\frac{xM_{ud}^{2}+\bar{x}m_{u}^{2}}{\kappa^{2}x\bar{x}}\right)\;.$
(34)

We next proceed with the axial-vector diquark contributions to the EM form
factors:

$\displaystyle F^{A(u)}_{1}(Q^{2})$ $\displaystyle=$
$\displaystyle\int\mathrm{d}x\frac{\mathrm{d}^{2}\mathbf{k}}{16\pi^{3}}\left[\frac{1}{9}\left({\psi^{A\uparrow}_{\frac{1}{2},0}}^{\dagger}(x,\mathbf{k}^{\prime})\psi^{A\uparrow}_{\frac{1}{2},0}(x,\mathbf{k})+{\psi^{A\uparrow}_{-\frac{1}{2},0}}^{\dagger}(x,\mathbf{k}^{\prime})\psi^{A\uparrow}_{-\frac{1}{2},0}(x,\mathbf{k})\right)\right.$
(35)
$\displaystyle\left.+\frac{2}{9}\left({\psi^{A\uparrow}_{\frac{1}{2},1}}^{\dagger}(x,\mathbf{k}^{\prime})\psi^{A\uparrow}_{\frac{1}{2},1}(x,\mathbf{k})+{\psi^{A\uparrow}_{-\frac{1}{2},1}}^{\dagger}(x,\mathbf{k}^{\prime})\psi^{A\uparrow}_{-\frac{1}{2},1}(x,\mathbf{k})\right)\right]\;,$
$\displaystyle F^{A(u)}_{2}(Q^{2})$ $\displaystyle=$
$\displaystyle-\frac{2M}{q^{1}-iq^{2}}\int\mathrm{d}x\frac{\mathrm{d}^{2}\mathbf{k}}{16\pi^{3}}\frac{1}{9}\left[{\psi^{A\uparrow}_{\frac{1}{2},0}}^{\dagger}(x,\mathbf{k}^{\prime})\psi^{A\downarrow}_{\frac{1}{2},0}(x,\mathbf{k})+{\psi^{A\uparrow}_{-\frac{1}{2},0}}^{\dagger}(x,\mathbf{k}^{\prime})\psi^{A\downarrow}_{-\frac{1}{2},0}(x,\mathbf{k})\right],$
(36)

and

$\displaystyle F^{A(d)}_{2}(Q^{2})$ $\displaystyle=$
$\displaystyle-\frac{2M}{q^{1}-iq^{2}}\int\mathrm{d}x\frac{\mathbf{d}^{2}\mathbf{k}}{16\pi^{3}}\frac{2}{9}\left[{\psi^{A\uparrow}_{\frac{1}{2},0}}^{\dagger}(x,\mathbf{k}^{\prime})\psi^{A\downarrow}_{\frac{1}{2},0}(x,\mathbf{k})+{\psi^{A\uparrow}_{-\frac{1}{2},0}}^{\dagger}(x,\mathbf{k}^{\prime})\psi^{A\downarrow}_{-\frac{1}{2},0}(x,\mathbf{k})\right].$
(37)

Using our spin-improved wavefunctions, namely Eqs. (II) and (II), we find that

$\displaystyle
F_{1}^{A(u)}(Q^{2})\propto\int\mathrm{d}x\left[\frac{1}{9}\left\\{\left(\frac{Mm_{u}}{M_{ud}}\frac{\bar{x}}{x}+\frac{M_{ud}}{\bar{x}}\right)^{2}+\kappa^{2}x\bar{x}\left(\frac{M\bar{x}}{M_{ud}x}\right)^{2}\left(1-\frac{q_{\perp}^{2}\bar{x}}{4\kappa^{2}x}\right)\right\\}\right.$
$\displaystyle+\left.\frac{2}{9}\left\\{\left(M+\frac{m_{u}}{x}\right)^{2}+\kappa^{2}\frac{\bar{x}}{x}\left(1-\frac{q_{\perp}^{2}\bar{x}}{4\kappa^{2}x}\right)\right\\}\right]$
$\displaystyle\times\exp\left(-\frac{\bar{x}q_{\perp}^{2}}{4\kappa^{2}x}\right)\exp\left(-\frac{xM_{ud}^{2}+\bar{x}m_{u}^{2}}{\kappa^{2}x\bar{x}}\right)$
$\displaystyle\;,$ (38)

$\displaystyle
F^{A(d)}_{1}(Q^{2})\propto\int\mathrm{d}x\left[\frac{2}{9}\left\\{\left(\frac{Mm_{d}}{M_{uu}}\frac{\bar{x}}{x}+\frac{M_{uu}}{\bar{x}}\right)^{2}+\kappa^{2}x\bar{x}\left(\frac{M\bar{x}}{M_{uu}x}\right)^{2}\left(1-\frac{q_{\perp}^{2}\bar{x}}{4\kappa^{2}x}\right)\right\\}\right.$
$\displaystyle+\left.\frac{4}{9}\left\\{\left(M+\frac{m_{d}}{x}\right)^{2}+\kappa^{2}\frac{\bar{x}}{x}\left(1-\frac{q_{\perp}^{2}\bar{x}}{4\kappa^{2}x}\right)\right\\}\right]$
$\displaystyle\times\exp\left(-\frac{\bar{x}q_{\perp}^{2}}{4\kappa^{2}x}\right)\exp\left(-\frac{xM_{uu}^{2}+\bar{x}m_{d}^{2}}{\kappa^{2}x\bar{x}}\right)$
$\displaystyle\;,$ (39) $\displaystyle F^{A(u)}_{2}(Q^{2})$
$\displaystyle\propto$
$\displaystyle\int\mathrm{d}x\frac{\bar{x}^{2}}{x}\Big{(}\frac{Mm_{u}}{M_{ud}}\frac{\bar{x}}{x}+\frac{M_{ud}}{1-x}\Big{)}\exp\left(-\frac{\bar{x}q_{\perp}^{2}}{4\kappa^{2}x}\right)\exp\left(-\frac{xM_{ud}^{2}+\bar{x}m_{u}^{2}}{\kappa^{2}x\bar{x}}\right)\;,$
(40)

and

$\displaystyle F^{A(d)}_{2}(Q^{2})$ $\displaystyle\propto$
$\displaystyle\int\mathrm{d}x\frac{\bar{x}^{2}}{x}\Big{(}\frac{Mm_{d}}{M_{uu}}\frac{\bar{x}}{x}+\frac{M_{uu}}{\bar{x}}\Big{)}\exp\left(-\frac{\bar{x}q_{\perp}^{2}}{4\kappa^{2}x}\right)\exp\left(-\frac{xM_{uu}^{2}+\bar{x}m_{d}^{2}}{\kappa^{2}x\bar{x}}\right)\;.$
(41)

Finally, we find the scalar and axial-vector diquark contributions to the
axial form factor:

$\displaystyle
G^{S(u)}_{A}(Q^{2})=\int\mathrm{d}x\int\frac{\mathrm{d}^{2}\mathbf{k}}{16\pi^{3}}\left[{\psi^{S\uparrow}_{\frac{1}{2},0}}^{\dagger}(x,\mathbf{k}^{\prime})\psi^{S\uparrow}_{\frac{1}{2},0}(x,\mathbf{k})-{\psi^{S\uparrow}_{-\frac{1}{2},0}}^{\dagger}(x,\mathbf{k}^{\prime})\psi^{S\uparrow}_{-\frac{1}{2},0}(x,\mathbf{k})\right]$
(42)

and

$\displaystyle
G_{A}^{A(u)}(Q^{2})=\int\mathrm{d}x\int\frac{\mathrm{d}^{2}\mathbf{k}}{16\pi^{3}}\left[\frac{1}{9}\left\\{{\psi^{A\uparrow}_{\frac{1}{2},0}}^{\dagger}(x,\mathbf{k}^{\prime})\psi^{A\uparrow}_{\frac{1}{2},0}(x,\mathbf{k})-{\psi^{A\uparrow}_{-\frac{1}{2},0}}^{\dagger}(x,\mathbf{k}^{\prime})\psi^{A\uparrow}_{-\frac{1}{2},0}(x,\mathbf{k})\right\\}\right.$
$\displaystyle\left.+\frac{2}{9}\left\\{{\psi^{A\uparrow}_{\frac{1}{2},1}}^{\dagger}(x,\mathbf{k}^{\prime})\psi^{A\uparrow}_{\frac{1}{2},1}(x,\mathbf{k})-{\psi^{A\uparrow}_{-\frac{1}{2},1}}^{\dagger}(x,\mathbf{k}^{\prime})\psi^{A\uparrow}_{-\frac{1}{2},1}(x,\mathbf{k})\right\\}\right].$
(43)

Using our spin-improved wavefunctions, given by Eqs. (II) and (II), we find
that

$\displaystyle G^{S(u)}_{A}(Q^{2})$ $\displaystyle\propto$ $\displaystyle\int
dx\left[\left(M+\frac{m_{u}}{x}\right)^{2}-\frac{(1-x)\kappa^{2}}{x}\left(1-\frac{(1-x)q_{\perp}^{2}}{4\kappa^{2}x}\right)\right]$
(44)
$\displaystyle\times\exp\left(-\frac{(1-x)q_{\perp}^{2}}{4\kappa^{2}x}\right)\exp\left(-\frac{xM_{ud}^{2}+(1-x)m_{u}^{2}}{\kappa^{2}x(1-x)}\right)\;,$
$\displaystyle
G^{A(u)}_{A}(Q^{2})\propto\int\mathrm{d}x\left[\frac{1}{9}\left\\{\left(\frac{Mm_{u}}{M_{ud}}\frac{\bar{x}}{x}+\frac{M_{ud}}{\bar{x}}\right)^{2}-\kappa^{2}x\bar{x}\left(\frac{M\bar{x}}{M_{ud}x}\right)^{2}\left(1-\frac{q_{\perp}^{2}\bar{x}}{4\kappa^{2}x}\right)\right\\}\right.$
$\displaystyle\left.+\frac{2}{9}\left\\{-\left(M+\frac{m_{d}}{x}\right)^{2}+\kappa^{2}\frac{\bar{x}}{x}\left(1-\frac{q_{\perp}^{2}(1-x)}{4\kappa^{2}x}\right)\right\\}\right]$
$\displaystyle\exp{\left(-\frac{\bar{x}q_{\perp}^{2}}{4\kappa^{2}x}\right)}\exp{\left(-\frac{xM_{ud}^{2}+\bar{x}m_{u}^{2}}{\kappa^{2}x\bar{x}}\right)}$
, (45)

and

$\displaystyle
G^{A(d)}_{A}(Q^{2})\propto\int\mathrm{d}x\left[\frac{2}{9}\left\\{\left(\frac{Mm_{d}}{M_{uu}}\frac{\bar{x}}{x}+\frac{M_{uu}}{\bar{x}}\right)^{2}-\kappa^{2}x\bar{x}\left(\frac{M\bar{x}}{M_{uu}x}\right)^{2}\left(1-\frac{q_{\perp}^{2}\bar{x}}{4\kappa^{2}x}\right)\right\\}\right.$
$\displaystyle\left.+\frac{4}{9}\left\\{-\left(M+\frac{m_{d}}{x}\right)^{2}+\kappa^{2}\frac{\bar{x}}{x}\left(1-\frac{q_{\perp}^{2}(1-x)}{4\kappa^{2}x}\right)\right\\}\right]$
$\displaystyle\exp{\left(-\frac{\bar{x}q_{\perp}^{2}}{4\kappa^{2}x}\right)}\exp{\left(-\frac{xM_{uu}^{2}+\bar{x}m_{d}^{2}}{\kappa^{2}x\bar{x}}\right)}$
. (46)

We normalize the Dirac and Pauli EM form factors using the quark counting
rules and the quark anomalous magnetic moments, i.e

$F_{1}^{u}(0)=n_{u},~{}~{}~{}F_{1}^{d}(0)=n_{d},$ (47)
$F^{u}_{2}(0)=\kappa_{u},~{}~{}~{}F^{d}_{2}(0)=\kappa_{d},$ (48)

where $n_{u/d}$ is the number of $u/d$ quarks in the nucleon. The quark
anomalous magnetic moments, $\kappa_{u/d}$, depend implicitly on the nucleon:
$\kappa_{u/d}^{p/n}=2\kappa_{p}+\kappa_{n}=1.673$ and
$\kappa_{d/u}^{p/n}=\kappa_{p}+2\kappa_{n}=-2.033$ are the anomalous magnetic
moments of the $u$ and $d$ quarks respectively. Consequently, the
normalization of the axial form factor $G^{f}_{A}(0)$, i.e. the axial charge
comes out as a prediction.

##  IV Comparing to data

To compare to experimental data, we compute the nucleon form factors using

$\displaystyle
F_{1,2}(Q^{2})=e_{u}F_{1,2}^{u}(Q^{2})+e_{d}F_{1,2}^{d}(Q^{2}),$ (49)

where $e_{u}$ and $e_{d}$ are the electric charge of $u$ and $d$ quarks. Note
the resulting nucleon form factors satisfy $F_{1}(0)^{p}=1$, $F_{1}(0)^{n}=0$,
$F_{2}^{p}(0)=\kappa_{p}$ and $F_{2}^{n}(0)=\kappa_{n}$ as expected.
Similarly, the proton’s axial form factor is given by

$G_{A}^{p}(Q^{2})=G_{A}^{u}-G_{A}^{d}\;.$ (50)

Notice that if $M_{uu}=M_{ud}$, the axial-vector diquark contributions to the
proton form factor cancel out, as found in Ref. Ma et al. (2002). Here, we
consider the possibility that $M_{uu}\neq M_{ud}$. We choose to adjust the
quark and diquark masses in order to fit the
$F_{2}^{p}(Q^{2})/F_{1}^{p}(Q^{2})$ data because this ratio requires the
complete set of flavour form factors. We find that $M_{uu}=0.63\pm 0.03$ GeV
and $M_{ud}=0.80\pm 0.04$ GeV with $m_{u}=m_{d}=0.40\pm 0.02$ GeV to fit as
shown in Fig. 2. Recall that we use the universal AdS/QCD mass scale
$\kappa=0.523\pm 0.024$ GeV extracted from spectroscopic data Brodsky et al.
(2016). Having fixed the quark and diquark masses, all remanining predictions
in this paper are generated without any further adjustment of parameters.

We start by predicting the electric and magnetic Sachs form factors using

$G_{E}^{p/n}=F_{1}^{p/n}(Q^{2})-\frac{Q^{2}}{4M^{2}}F_{2}^{p/n}(Q^{2})$ (51)

and

$G_{M}^{p/n}(Q^{2})=F_{1}^{p/n}(Q^{2})+F_{2}^{p/n}(Q^{2})\;.$ (52)

Our predictions for the proton’s Sachs form factors are shown in Fig. 2. As
can be seen, the agreement with the data is excellent in the non-perturbative,
$Q^{2}\leq 1~{}\mathrm{GeV}^{2}$ region, where the holographic wavefunction is
expected to be accurate. The predictions are less impressive for the neutron’s
Sachs form factors. As shown in Fig. 3, the relatively large theory
uncertainty reflects a near-exact cancellation between the active-$u$ and
active $d$-quark contributions to $G_{E}^{n}(Q^{2})$. This cancellation
occurring at leading order (in $\alpha_{\mathrm{em}}$) implies that the
neutron form factor may be sensitive to higher order contributions which we do
not consider here. We also predict the ratio
$R_{p}=\mu_{p}G_{E}^{p}/G_{M}^{p}$ as shown in Fig. 4. It is worth noting that
our predictions are in agreement with the polarization data Gayou et al.
(2001, 2002); Arrington et al. (2007); Punjabi et al. (2005); Puckett et al.
(2010) but are not consistent with the data obtained using the Rosenbluth
separation method Walker et al. (1994).

Figure 1: Adjusting the quark and diquark masses to fit the data on the Pauli-
to Dirac form factors ratio. The resulting masses are: $M_{uu}=0.63\pm 0.03$
GeV and $M_{ud}=0.80\pm 0.04$ GeV with $m_{u}=m_{d}=0.40\pm 0.02$ GeV. The
experimental data are taken form the Refs. Gayou et al. (2001, 2002);
Arrington et al. (2007); Punjabi et al. (2005); Puckett et al. (2010).

Figure 2: Our predictions for the Sachs Form Factors of the proton. The grey
band corresponds to the uncertainty in the holographic mass scale as well as
the quark and diquark masses. The experimental data are from Refs. Gayou et
al. (2001, 2002); Arrington et al. (2007); Milbrath et al. (1998); Pospischil
et al. (2001); Jones et al. (2000) for $G_{E}^{p}(Q^{2})$ and Refs. Arrington
et al. (2007); Arrington (2005) for $G_{M}^{p}(Q^{2})$.

Figure 3: Our predictions for the Sachs Form Factors of the neutron. The grey
band corresponds to the uncertainty in the holographic mass scale as well as
the quark and diquark masses. The experimental data are from Refs. Gayou et
al. (2001, 2002); Arrington et al. (2007); Milbrath et al. (1998); Pospischil
et al. (2001); Jones et al. (2000) for $G_{E}^{n}(Q^{2})$, and Refs. Anklin et
al. (1998); Kubon et al. (2002); Xu et al. (2003); Anderson et al. (2007);
Lachniet et al. (2009) for $G_{M}^{n}(Q^{2})$.

Figure 4: Ratio of Sachs form factors (a) for proton (b) for neutron. The grey
band corresponds to the uncertainty in the holographic mass scale as well as
the quark and diquark masses. The experimental data for $R_{p}$ are taken form
the Refs. Gayou et al. (2001, 2002); Arrington et al. (2007); Punjabi et al.
(2005); Puckett et al. (2010); Walker et al. (1994), while the data for
$R_{n}$ are taken from the Refs. Passchier et al. (1999); Zhu et al. (2001);
Warren et al. (2004); Herberg et al. (1999); Plaster et al. (2006); Bermuth et
al. (2003); Geis (2007); Geis et al. (2008); Glazier et al. (2005).

We predict the root-mean-square charge and magnetic radii of the nucleons
using

$\langle
r_{E}^{2}\rangle=\left.-6\frac{\mathrm{d}G_{E}}{\mathrm{d}Q^{2}}\right|_{Q^{2}=0}\hskip
28.45274pt\mathrm{and}\hskip 28.45274pt\langle
r_{M}^{2}\rangle=\left.-\frac{6}{G_{M}}\frac{\mathrm{d}G_{M}}{\mathrm{d}Q^{2}}\right|_{Q^{2}=0}$
(53)

respectively. As shown in Table 1, our predictions are in very good agreement
with the experimentally measured values. This is consistent with the fact our
predictions are most reliable in the non-perturbative, low $Q^{2}$ regime.
Note that the most recent measurements of the proton’s charge radius based on
the lamb shift in atomic hydrogen Bezginov et al. (2019), muonic hydrogen Pohl
et al. (2010) and the high precision electron-proton scattering experiment at
JLAB Xiong et al. (2019) are all consistent with each other and smaller than
the earlier CODATA extraction, $\langle r_{E}\rangle_{p}=0.8768(69)$ fm Mohr
et al. (2008). As shown in Table 1, our prediction for the charge radius of
the proton is in agreement with the recent measurements. In fact, the
agreement with data extends also to the magnetic radius of the proton as well
as the electric and magnetic radius of the neutron.

Radius | Our prediction | Experimental data
---|---|---
$\langle r_{E}\rangle_{p}$ fm | $0.833\pm 0.010$ |  $0.833\pm 0.010$ Bezginov et al. (2019); $0.831\pm 0.019$ Xiong et al. (2019) ; $0.841\pm 0.084$ Pohl et al. (2010)
$\langle r_{M}\rangle_{p}$ fm | $0.7985\pm 0.0313$ |  $0.851\pm 0.026$ Tanabashi et al. (2018)
$\langle r^{2}_{E}\rangle_{n}$ fm2 | $-0.0704\pm 0.0434$ |  $-0.1161\pm 0.0022$ Tanabashi et al. (2018); $-0.110\pm 0.008$ Atac et al. (2021)
$\langle r_{M}\rangle_{n}$ fm | $0.8388\pm 0.0288$ |  $0.864^{+0.009}_{-0.008}$ Tanabashi et al. (2018)
Table 1: Our predictions for the electric and magnetic radii of the nucleons,
compared to the experimental measurements Bezginov et al. (2019); Xiong et al.
(2019); Tanabashi et al. (2018).

We finally predict the axial form factor and axial charge for the proton. We
find $g_{A}=G^{p}_{A}(0)=1.303\pm 0.033$, which is to be compared with the
experimental value Tanabashi et al. (2018), $g_{A}=1.2695\pm 0.0029$, and the
lattice prediction Chang et al. (2018), $g_{A}=1.271\pm 0.013$ , as well as a
more recent lattice calculation with higher statistics Gupta et al. (2018),
$g_{A}=1.218(25)(30)$. The radius of the proton evaluated using the axial form
factor is $0.675\pm 0.030$ fm, which is in better agreement with experimental
value $0.667\pm 0.120$ fm Hill et al. (2018) than the recent lattice result
$0.512\pm 0.034$ fm Yao et al. (2017). The $Q^{2}$-dependence of the axial
form factor for the proton is shown in Fig.(5). As for the EM form factors, we
see that agreement is very good in the low $Q^{2}\leq 1~{}\mathrm{GeV}^{2}$
region.

Figure 5:  Our prediction for the proton axial form factor. The grey band
corresponds to the uncertainty in the holographic mass scale as well as the
quark and diquark masses. The data are from Refs. Bernard et al. (2001)
(experimental data) and Hashamipour et al. (2019) (reduced data). The
parametric form of the dipole fit can be found in Ref. Bernard et al. (2001).

##  V Conclusions

We have constructed spin-improved holographic light-front wavefunctions for
the nucleons and used them to predict their form factors and radii, with the
quark and diquark masses as the only adjustable parameters. Having fixed these
masses using the data on the proton’s Pauli-to-Dirac form factor ratio, all
our remaining predictions agree remarkably well with data at low $Q^{2}$,
where the non-perturbative holographic wavefunction is expected to be
accurate. Together with previous results Forshaw and Sandapen (2012); Ahmady
et al. (2016, 2017, 2019, 2020) on light mesons, our findings suggest that the
light (pseudoscalar and vector) mesons and the nucleons can be treated in a
unified framework in which they share the same universal light-front
holographic wavefunction.

##  VI Acknowledgements

MA and RS are supported by individual Discovery Grants (SAPIN-2021-00038 and
SAPIN-2020-00051) from the Natural Sciences and Engineering Research Council
of Canada (NSERC). DC is supported by Science and Engineering Research Board
(India) under the Grant No. CRG/2019/000895. CM thanks the Chinese Academy of
Sciences President’s International Fellowship Initiative for the support via
Grants No. 2021PM0023. This work of CM is also supported by the Strategic
Priority Research Program of the Chinese Academy of Sciences, Grant No.
XDB34000000.

## References

  * Forshaw and Sandapen (2012) J. R. Forshaw and R. Sandapen, Phys. Rev. Lett. 109, 081601 (2012), eprint 1203.6088.
  * Ahmady et al. (2016) M. Ahmady, R. Sandapen, and N. Sharma, Phys. Rev. D94, 074018 (2016), eprint 1605.07665.
  * Ahmady et al. (2017) M. Ahmady, F. Chishtie, and R. Sandapen, Phys. Rev. D95, 074008 (2017), eprint 1609.07024.
  * Ahmady et al. (2019) M. Ahmady, C. Mondal, and R. Sandapen, Phys. Rev. D100, 054005 (2019), eprint 1907.06561.
  * Ahmady et al. (2020) M. Ahmady, S. Kaur, C. Mondal, and R. Sandapen, Phys. Rev. D 102, 034021 (2020), eprint 2006.07675.
  * Brodsky et al. (2013) S. J. Brodsky, G. F. de Téramond, and H. G. Dosch, Nuovo Cim. C036, 265 (2013), eprint 1302.5399.
  * Brodsky and de Téramond (2006) S. J. Brodsky and G. F. de Téramond, Phys. Rev. Lett. 96, 201601 (2006), eprint hep-ph/0602252.
  * Brodsky et al. (2015) S. J. Brodsky, G. F. de Téramond, H. G. Dosch, and J. Erlich, Phys. Rept. 584, 1 (2015), eprint 1407.8131.
  * Brodsky et al. (2014) S. J. Brodsky, G. F. De Téramond, and H. G. Dosch, Phys. Lett. B729, 3 (2014), eprint 1302.4105.
  * Drell and Yan (1970) S. D. Drell and T.-M. Yan, Phys. Rev. Lett. 24, 181 (1970).
  * West (1970) G. B. West, Phys. Rev. Lett. 24, 1206 (1970).
  * Brodsky and de Téramond (2009) S. J. Brodsky and G. F. de Téramond, Subnucl. Ser. 45, 139 (2009), eprint 0802.0514.
  * Maji and Chakrabarti (2016) T. Maji and D. Chakrabarti, Phys. Rev. D 94, 094020 (2016), eprint 1608.07776.
  * Gutsche et al. (2015) T. Gutsche, V. E. Lyubovitskij, I. Schmidt, and A. Vega, Phys. Rev. D 91, 054028 (2015), eprint 1411.1710.
  * Brodsky et al. (2001) S. J. Brodsky, D. S. Hwang, B.-Q. Ma, and I. Schmidt, Nucl. Phys. B 593, 311 (2001), eprint hep-th/0003082.
  * Ellis et al. (2009) J. R. Ellis, D. S. Hwang, and A. Kotzinian, Phys. Rev. D 80, 074033 (2009), eprint 0808.1567.
  * Chakrabarti and Mondal (2013) D. Chakrabarti and C. Mondal, Eur. Phys. J. C 73, 2671 (2013), eprint 1307.7995.
  * Sufian et al. (2017) R. S. Sufian, G. F. de Teramond, S. J. Brodsky, A. Deur, and H. G. Dosch, Phys. Rev. D 95, 014011 (2017), eprint 1609.06688.
  * Gutsche et al. (2014) T. Gutsche, V. E. Lyubovitskij, I. Schmidt, and A. Vega, Phys. Rev. D89, 054033 (2014), [Erratum: Phys. Rev.D92,no.1,019902(2015)], eprint 1306.0366.
  * Lepage and Brodsky (1980) G. P. Lepage and S. J. Brodsky, Phys. Rev. D22, 2157 (1980).
  * Brodsky et al. (2016) S. J. Brodsky, G. F. de Téramond, H. G. Dosch, and C. Lorce, Int. J. Mod. Phys. A31, 1630029 (2016), eprint 1606.04638.
  * Nielsen and Brodsky (2018) M. Nielsen and S. J. Brodsky, Phys. Rev. D97, 114001 (2018), eprint 1802.09652.
  * Ma et al. (2002) B.-Q. Ma, D. Qing, and I. Schmidt, Phys. Rev. C 65, 035205 (2002), eprint hep-ph/0202015.
  * Gayou et al. (2001) O. Gayou et al., Phys. Rev. C 64, 038202 (2001).
  * Gayou et al. (2002) O. Gayou et al. (Jefferson Lab Hall A), Phys. Rev. Lett. 88, 092301 (2002), eprint nucl-ex/0111010.
  * Arrington et al. (2007) J. Arrington, W. Melnitchouk, and J. Tjon, Phys. Rev. C 76, 035205 (2007), eprint 0707.1861.
  * Punjabi et al. (2005) V. Punjabi et al., Phys. Rev. C 71, 055202 (2005), [Erratum: Phys.Rev.C 71, 069902 (2005)], eprint nucl-ex/0501018.
  * Puckett et al. (2010) A. Puckett et al., Phys. Rev. Lett. 104, 242301 (2010), eprint 1005.3419.
  * Walker et al. (1994) R. Walker et al., Phys. Rev. D 49, 5671 (1994).
  * Milbrath et al. (1998) B. Milbrath et al. (Bates FPP), Phys. Rev. Lett. 80, 452 (1998), [Erratum: Phys.Rev.Lett. 82, 2221 (1999)], eprint nucl-ex/9712006.
  * Pospischil et al. (2001) T. Pospischil et al. (A1), Eur. Phys. J. A 12, 125 (2001).
  * Jones et al. (2000) M. Jones et al. (Jefferson Lab Hall A), Phys. Rev. Lett. 84, 1398 (2000), eprint nucl-ex/9910005.
  * Arrington (2005) J. Arrington, Phys. Rev. C 71, 015202 (2005), eprint hep-ph/0408261.
  * Anklin et al. (1998) H. Anklin et al., Phys. Lett. B 428, 248 (1998).
  * Kubon et al. (2002) G. Kubon et al., Phys. Lett. B 524, 26 (2002), eprint nucl-ex/0107016.
  * Xu et al. (2003) W. Xu et al. (Jefferson Lab E95-001), Phys. Rev. C 67, 012201 (2003), eprint nucl-ex/0208007.
  * Anderson et al. (2007) B. Anderson et al. (Jefferson Lab E95-001), Phys. Rev. C 75, 034003 (2007), eprint nucl-ex/0605006.
  * Lachniet et al. (2009) J. Lachniet et al. (CLAS), Phys. Rev. Lett. 102, 192001 (2009), eprint 0811.1716.
  * Passchier et al. (1999) I. Passchier et al., Phys. Rev. Lett. 82, 4988 (1999), eprint nucl-ex/9907012.
  * Zhu et al. (2001) H. Zhu et al. (E93026), Phys. Rev. Lett. 87, 081801 (2001), eprint nucl-ex/0105001.
  * Warren et al. (2004) G. Warren et al. (Jefferson Lab E93-026), Phys. Rev. Lett. 92, 042301 (2004), eprint nucl-ex/0308021.
  * Herberg et al. (1999) C. Herberg et al., Eur. Phys. J. A 5, 131 (1999).
  * Plaster et al. (2006) B. Plaster et al. (Jefferson Laboratory E93-038), Phys. Rev. C 73, 025205 (2006), eprint nucl-ex/0511025.
  * Bermuth et al. (2003) J. Bermuth et al., Phys. Lett. B 564, 199 (2003), eprint nucl-ex/0303015.
  * Geis (2007) E. J. Geis, Other thesis (2007).
  * Geis et al. (2008) E. Geis et al. (BLAST), Phys. Rev. Lett. 101, 042501 (2008), eprint 0803.3827.
  * Glazier et al. (2005) D. Glazier et al., Eur. Phys. J. A 24, 101 (2005), eprint nucl-ex/0410026.
  * Bezginov et al. (2019) N. Bezginov, T. Valdez, M. Horbatsch, A. Marsman, A. Vutha, and E. Hessels, Science 365, 1007 (2019).
  * Pohl et al. (2010) R. Pohl et al., Nature 466, 213 (2010).
  * Xiong et al. (2019) W. Xiong et al., Nature 575, 147 (2019).
  * Mohr et al. (2008) P. J. Mohr, B. N. Taylor, and D. B. Newell, Rev. Mod. Phys. 80, 633 (2008), eprint 0801.0028.
  * Tanabashi et al. (2018) M. Tanabashi et al. (Particle Data Group), Phys. Rev. D 98, 030001 (2018).
  * Atac et al. (2021) H. Atac, M. Constantinou, Z. E. Meziani, M. Paolone, and N. Sparveris, Nature Commun. 12, 1759 (2021), eprint 2103.10840.
  * Chang et al. (2018) C. Chang et al., Nature 558, 91 (2018), eprint 1805.12130.
  * Gupta et al. (2018) R. Gupta, Y.-C. Jang, B. Yoon, H.-W. Lin, V. Cirigliano, and T. Bhattacharya, Phys. Rev. D 98, 034503 (2018), eprint 1806.09006.
  * Hill et al. (2018) R. J. Hill, P. Kammel, W. J. Marciano, and A. Sirlin, Rept. Prog. Phys. 81, 096301 (2018), eprint 1708.08462.
  * Yao et al. (2017) D.-L. Yao, L. Alvarez-Ruso, and M. J. Vicente-Vacas, Phys. Rev. D 96, 116022 (2017), eprint 1708.08776.
  * Bernard et al. (2001) V. Bernard, L. Elouadrhiri, and U.-G. Meißner, Journal of Physics G: Nuclear and Particle Physics 28, R1 (2001), URL https://doi.org/10.1088/0954-3899/28/1/201.
  * Hashamipour et al. (2019) H. Hashamipour, M. Goharipour, and S. S. Gousheh, Phys. Rev. D 100, 016001 (2019), eprint 1903.05542.


11institutetext:  University of Illinois Urbana-Champaign, USA
22institutetext: University of Massachusetts Amherst, USA

# Getting More out of Large Language Models for Proofs

Shizhuo Dylan Zhang 11 Emily First 22 Talia Ringer 11

###### Abstract

Large language models have the potential to simplify formal theorem proving
and make it more accessible. But how to get the most out of these models is
still an open question. To answer this question, we take a step back and
explore the failure cases of these models using common prompting-based
techniques. Our talk will discuss these failure cases and what they can teach
us about how to get more out of these models.

#### Introduction

Formal theorem proving is a crucial but challenging task—and a natural fit for
automation. Historically, most work on formal proof automation has relied on
symbolic techniques [8], sometimes combined with neural tools [2, 7, 12, 6, 4,
3]. Recent advances in large language models have improved their instruction-
following and in-context learning capabilities, making it possible to build
powerful proof automation that elides symbolic search procedures altogether,
as shown by Baldur [5].

How can we get more out of these advances in large language models? To answer
that, we examine the capabilities of the state-of-the-art language models
GPT-3.5 Turbo and GPT-4 to prove theorems in Coq using common prompting-based
techniques. In particular, we conduct a fine-grained analysis of model outputs
on an example project. Our emphasis is on the failure cases—how these outputs
commonly go wrong, and what that can teach us about how to get more out of
these models.

[!htb]  GPT-3.5-Turbo GPT-4 Proverbot FS-rand FS-sim ZS FS-rand FS-sim ZS
FS+Lem ZS+Lem - #Correct Proof 10 8 0 7 8 0 14 9 - #Proven Theorems 6 4 0 6 7
0 7 5 23 Results. ‘FS’ stands for ‘few-shot’, ‘ZS’ stands for ‘zero-shot’.
‘+lemma’ ’denotes providing lemmas preceding the query theorem in the file in
the context.

(a) Breakdown of 740 expert-annotated GPT-4 outputs under few-shot ‘theorem-
proof’ set-up (see Appendix .1.2 for more).

(b) An example output that incurs an error because model lacks local proof
state, and so uses variable name $R$ that is already taken. See Appendix .3.4
for full context.

(c) An example output that recognizes the necessary definitions are not
provided and asks for clarifications from the user. See Appendix .3.1 for
more.

#### Results and Recommendations

We performed fine-grained analysis of the model-generated proofs and
categorized the comments of human experts in Figure 1(a). Details about our
experimental methodology, along with examples, are in the appendix. Our
recommendations:

1\. Allow the model to prompt the proof assistant for more information. Model
outputs sometimes ask for more information about the definitions of referenced
variables when those definitions have not been provided as input as shown in
Figure 1(c) and Appendix .3.1. This opens up a perfect opportunity for tool
use at the model’s request. By prompting the model to ask for information
using standard Coq commands like Print, and executing them in real time, we
can allow the model to obtain information as it generates proof in steps.

2\. Give the model access to proof states. Language models are found to be
weak at ‘execution’. In our case, they by default lack access to proof state.
This manifests in outputs as incorrect assumptions about the current proof
state, like introducing too many variables or overloading variables already
used (see Figure 1(b) and Appendix .3.4). For human proof engineers,
interactive proof is more like a conversation with the proof assistant with
constant feedback on the state; emulating this is a perfect opportunity to put
the chat API to good use.

3\. Give the model access to information in file dependencies. Naively
prompted models produce outputs that make incorrect assumptions about
definitions and lemmas from dependencies, for example by hallucinating lemma
names (see Appendix .3.6), or by attempting to induct over a non-inductive
hypothesis. Human proof engineers avoid this by having access to dependencies
directly; models should access to those dependencies in context or in memory.

4\. Give the model access to proofs preceding the current proof. Naively
prompted models fail to solve proofs that would be obvious from context, since
human-written proofs are often small permutations of earlier proofs. Baldur
[5] showed in-context learning can help with this in Isabelle/HOL; we now have
evidence (see Appendix .3.5) this is worth trying in Coq.

5\. Learn from errors. People have attempted to iteratively improve the
quality of generated programs by providing the model with correctness signals
as feedback [13, 10]; the same has shown success for proofs in Isabelle/HOL
[5]. The error messages we observed were very informative,111Our annotated
data with errors is available here: https://github.com/DylanZSZ/LLM4Proof.git
and suggest this may be fruitful in Coq as well.

6\. Introduce diversity through prompt engineering. It has been observed that
a mixture of experts can boost performance of ML-guided proof synthesis by
performing diverse sequences of tactics [3]. On the other hand, there is
evidence of performance gain by introducing diversity in few-shot set-up for
both synthesis [1] and reasoning [11] tasks from the language modeling
research community. The diversity we have already observed (see Appendix .2)
is evidence this may be worth trying with different prompts for Coq proofs.

#### Proposed Talk

In our talk, we will discuss specific examples of these failure cases, how
they correspond to our recommendations, and our progress on experimentation
along these recommendations in the intervening months.

## References

  * [1] Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and Charles Sutton.  Program synthesis with large language models, 2021.
  * [2] Kshitij Bansal, Sarah Loos, Markus Rabe, Christian Szegedy, and Stewart Wilcox.  HOList: An environment for machine learning of higher order logic theorem proving.  In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 454–463. PMLR, 09–15 Jun 2019.
  * [3] Emily First and Yuriy Brun.  Diversity-driven automated formal verification.  In Proceedings of the 44th International Conference on Software Engineering, ICSE ’22, page 749–761, New York, NY, USA, 2022. Association for Computing Machinery.
  * [4] Emily First, Yuriy Brun, and Arjun Guha.  Tactok: Semantics-aware proof synthesis.  Proc. ACM Program. Lang., 4(OOPSLA), nov 2020.
  * [5] Emily First, Markus N. Rabe, Talia Ringer, and Yuriy Brun.  Baldur: Whole-proof generation and repair with large language models, 2023\.
  * [6] Guillaume Lample, Marie-Anne Lachaux, Thibaut Lavril, Xavier Martinet, Amaury Hayat, Gabriel Ebner, Aurélien Rodriguez, and Timothée Lacroix.  Hypertree proof search for neural theorem proving, 2022.
  * [7] Stanislas Polu and Ilya Sutskever.  Generative language modeling for automated theorem proving, 2020.
  * [8] Talia Ringer, Karl Palmskog, Ilya Sergey, Milos Gligoric, and Zachary Tatlock.  Qed at large: A survey of engineering of formally verified software.  Foundations and Trends® in Programming Languages, 5(2-3):102–281, 2019.
  * [9] Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie-Yan Liu.  Mpnet: Masked and permuted pre-training for language understanding, 2020\.
  * [10] Xin Wang, Yasheng Wang, Yao Wan, Fei Mi, Yitong Li, Pingyi Zhou, Jin Liu, Hao Wu, Xin Jiang, and Qun Liu.  Compilable neural code generation with compiler feedback, 2022.
  * [11] Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou.  Self-consistency improves chain of thought reasoning in language models, 2023.
  * [12] Minchao Wu, Michael Norrish, Christian Walder, and Amir Dezfouli.  Tacticzero: Learning to prove theorems from scratch with deep reinforcement learning, 2021.
  * [13] Chunqiu Steven Xia and Lingming Zhang.  Conversational automated program repair, 2023.

##  .1 Methodology

We experimented with both zero-shot and few-shot prompting methods with &
without preceding lemmas of the query theorem. To select few-shot samples, we
experimented with two strategies: random selection and similarity based
selection using a retrieval model we fine-tuned on the training set. The
details are deferred to appendix.

###  .1.1 Zero-Shot Prompting

The notable characteristic of GPT-3.5 Turbo and GPT-4 is their robust ability
to follow instructions. In a zero-shot configuration, we leverage the system’s
messaging to establish the task and desired output format for the entire
conversation, while utilizing user messaging to provide query-specific
information such as the theorem to be proven and relevant lemmas.

Figure 2: Investigation Pipeline

Figure 3: Workflow of few-shot prompting

Figure 4: Few-shot example format of Theorem-Proof setting.

Figure 5: Few-shot example format of Theorem with Given Lemma setting.

###  .1.2 Few-Shot Prompting

Our approach utilizes the conventional few-shot in-context learning method. We
include examples in the form of theorem-proof tuples at the beginning of the
target query as shown in Figure 3. In addition to the description of the task
in the initial system message and the query, we provide examples of query-
answer pairs obtained from training set to demonstrate to the model the task
we want it to perform and the expected output format. We experimented with a
few variants:

#### Theorem-Proof

In this set-up, we provide examples of pairs of theorems and proofs, as
illustrated in Figure 5.

#### Theorem with Given Lemmas

In this setting, we provide the model with the names of lemmas preceding the
theorem. The format is shown in Figure 5.

###  .1.3 Training of Retriever

We perform continued fine-tuning from the sentence-transformer model all-
mpnet-base-v2 based on[9]. In particular, this model has been trained on code
corpus and has certain level of code-understanding capabilities. We train the
model to perform similar proof search on the train set by minimizing the
triplet objective

$J=\sum{{}_{i=1}^{m}}L_{triplet}(T_{i},P_{i},P_{j\neq i})$ (1)

where $T_{i}$ is the query theorem,$P_{i}$ is the proof of this theorem and
$P_{j}$ is a randomly sampled proof of another theorem. During retrieval, we
compute the similarity scores between test theorems and proofs in the training
set, ans use those theorems whose proofs are similar to the test instance as
the few-shot examples.

###  .1.4 Experiment Set-up

We conducted experiments to evaluate the performance of two language models,
GPT-3.5-turbo and GPT4, under both few-shot and zero-shot settings. The few-
shot learning was performed with k-shots=6, meaning that the models were
trained on only six examples per class.

The hyperparameters used for both models were set as follows: temperature
$T$=1,presence _penalty=0.1, number of samples per prompt $n$=5.
$presence\\_penalty$ is a penalty term applied to the logit values of tokens
already present in the prompt to discourage the repetition of those tokens in
the generated text.

##  .2 Coinciding proofs solved

[!htb] Number of coinciding theorems proven successfully by each pair of
prompting set-ups. GPT4-random-fs GPT3.5-random-fs GPT4-sim-fs GPT3.5-sim-fs
GPT4-lemma-fs GPT4-lemma-zs GPT4-random-fs - - - - - GPT3.5-random-fs 1 - - -
- - GPT4-sim-fs 1 2 - - - - GPT3.5-sim-fs 0 3 1 - - - GPT4-lemma-fs 2 1 1 0 -
- GPT4-lemma-zs 1 1 1 0 3 - Proof synthesis is a special task in nature,
having similarities with code-synthesis where both being generative, but still
different as the criteria of whether or not the target theorem is proved is
close-ended to a certain extent.

##  .3 Example Outputs

###  .3.1 Refusing To Answer

⬇

Lemma G_wmon: wmonotonic TX TX G.

(* Without further information on what TX and G are, I cannot generate a valid
proof. Please provide more information or define the related functions and
types. *)

Even under few-shot set-up where the model output is restricted to be in Coq
format, the model insists on requesting additional information on the
variables not clearly defined. It is impressive that the model has certain
level of ability to understand Coq as well as the necessary information to
complete the proof, e.g. the definition of variable names ans assumptions.
This category constitutes 5.4% of total number of cases.

###  .3.2 Not Trying Simple Tactics

⬇

Lemma trans_incl: incl R R’ -> incl (trans R) (trans R’).

This is an easy theorem which model has failed. In particular, to prove this,
a simple auto. tactic will do. However, the model did not give this tactic a
try.

⬇

Lemma trans_incl: incl R R’ -> incl (trans R) (trans R’).

Proof.

auto.

Qed.

This is a case where all of context, proof state, and dependency information
may help. It is also a case where diverse prompting may be useful (for
example, by adding a prompt that favors very simple proofs that do not use any
lemmas).

###  .3.3 Succeeding

There are certain cases where the model succeeded in proving the theorems.
These theorems generally do not depend much on context. For example, the
model-generated proof of weak_refl succeeds using only simple tactics:

⬇

Lemma weak_refl: forall x, Weak T x x.

Proof.

intros x.

constructor.

reflexivity.

Qed.

###  .3.4 Lacking the Proof State

Here we provide the proof state of the example in the main text before the
step in Figure 1(b):

⬇

A, X, Y : Type

TX : reduction_t A X

TY : reduction_t A Y

B : relation X

HB : controlled TX TY B

F, G : function X Y

HF : monotonic TX TY F

HG : wmonotonic TX TY G

HBF : transparent B F

HFG : contains F G

HBG : contains (chaining_l (star B)) G

R : relation2 X Y

HRt : evolve_t TX TY R (comp (star B) (F R))

HRa : evolve_a TX TY R (G R)

pre_silent :

forall n : nat,

evolve_t TX TY (UExp F R n) (comp (star B) (UExp F R (S n)))

silent : simulation_t TX TY (comp (star B) (UIter F R))

HFGn : forall n : nat, incl (UExp F R n) (UExp G R n)

______________________________________(1/1)

forall (R0 : relation2 X Y) (n : nat),

incl (comp (star B) (UExp G R0 n)) (UExp G R0 (S n))

Note the proof assistant renamed the R in the goal into R0 because there is
another R in context from earlier in the file.222Coq makes it possible to
generalize all proofs in a Section by the same pre-defined parameters, and one
such parameter here is named R. Coq chooses R0 so as not to shadow R.
Similarly, the human proof calls this RR to avoid shadowing R:

⬇

intros RR n x y H; right; apply (HBG H).

In contrast, without any access to the fact that R is already defined the
local proof state, the model output attempts to introduce a new variable named
R. Coq refuses and responds with an error.

Another place where proof state is useful is when the model introduces more
variables than can be introduced. For example, the model generates this
output:

⬇

Lemma G_reverse: forall R, eeq (trans (G R)) (G (trans R)).

Proof.

unfold G, eeq.

intros R u v.

destruct R as [R Hr].

simpl.

split; intros [r H]; cbn in *; exists r;

rewrite <- Hr in *;

auto using sym_equal, trans_sym with relations.

The second step, intros R u v, assumes there are three variables to introduce.
This would be reasonable if unfolding G and eeq introduced more foralls in the
goal of the proof state after the first step. But it does not; the local proof
state at that step includes only one variable R to introduce. Coq thus
responds with an error.

###  .3.5 Lacking Local Context

Writing proofs is often easier with context, since proofs often mirror
preceding proofs in the same file. For example, proving this lemma is easier
with context:

⬇

Lemma union_incl: (forall i, incl (F i) (F’ i)) -> incl (union F) (union F’).

This is because the human-written proofs preceding it in the file very much
mirror the human-written proof of this lemma:

⬇

Lemma comp_incl: incl R R’ -> incl S S’ -> incl (comp R S) (comp R’ S’).

Proof.

unfold eeq, comp, incl; intuition.

destruct H1 as [ t ]; exists t; auto.

Qed.

Lemma comp_eeq: eeq R R’ -> eeq S S’ -> eeq (comp R S) (comp R’ S’).

Proof.

unfold eeq, comp, incl; intuition;

destruct H0 as [ t ]; exists t; auto.

Qed.

Lemma union_incl: (forall i, incl (F i) (F’ i)) -> incl (union F) (union F’).

Proof.

unfold eeq, union, incl; intuition.

destruct H0 as [ i ]; exists i; auto.

Qed.

The same holds for the proof of union2_evolve_right:

⬇

Lemma union2_evolve_left:

forall l R S S’, evolve_1 l R S -> evolve_1 l R (union2 S S’).

Proof.

intros l R S S’ H x x’ y Hxx’ xRy; destruct (H _ _ _ Hxx’ xRy) as [ y’ ];

exists y’; auto; left; auto.

Qed.

Lemma union2_evolve_right:

forall l R S S’, evolve_1 l R S’ -> evolve_1 l R (union2 S S’).

Proof.

intros l R S S’ H x x’ y Hxx’ xRy; destruct (H _ _ _ Hxx’ xRy) as [ y’ ];

exists y’; auto; right; auto.

Qed.

This gives us evidence that in-context learning in the style of Baldur may be
fruitful in Coq as well.

###  .3.6 Hallucination

Below is an example from the model where it hallucinates $H$ which has been
defined nowhere. We observe in many cases the model are directly using the
commonly used variable names like $H$, $x$, $y$ which it has seen multiple
times during pre-training.

⬇

Lemma bisimulation_bisim: bisimulation bisim.

Proof.

constructor.

- intros.

destruct H as [s’ H].

exists s’.

apply stutter_bisim in H.

auto.

Moreover, it hallucinates lemmas or definitions in the proofs, like
stutter_bisim. By providing proof state, we can help the model pick more
correct variables; by providing file dependencies that contain referenced
definitions and auxiliary lemmas, and prompting the model to restrict itself
to those, we can help the model use only the definitions and lemmas that exist
already.


# Disentangling Confidence Score Distribution for Out-of-Domain Intent
Detection with Energy-Based Learning

Yanan Wu1∗, Zhiyuan Zeng1∗, Keqing He2∗, Yutao Mou1 ,
Pei Wang1, Yuanmeng Yan1, Weiran Xu1
1Beijing University of Posts and Telecommunications, Beijing, China
2Meituan, Beijing, China
{yanan.wu,zengzhiyuan,myt,wangpei,
<EMAIL_ADDRESS>
<EMAIL_ADDRESS>The first three authors contribute equally. Weiran Xu
is the corresponding author.

###### Abstract

Detecting Out-of-Domain (OOD) or unknown intents from user queries is
essential in a task-oriented dialog system. Traditional softmax-based
confidence scores are susceptible to the overconfidence issue. In this paper,
we propose a simple but strong energy-based score function to detect OOD where
the energy scores of OOD samples are higher than IND samples. Further, given a
small set of labeled OOD samples, we introduce an energy-based margin
objective for supervised OOD detection to explicitly distinguish OOD samples
from INDs. Comprehensive experiments and analysis prove our method helps
disentangle confidence score distributions of IND and OOD data.111Our code is
available at https://github.com/pris-nlp/EMNLP2022-energy_for_OOD/.

##  1 Introduction

Detecting Out-of-Domain (OOD) or unknown intents from user queries is crucial
to a task-oriented dialog system Akasaki and Kaji (2017); Tulshan and Dhage
(2018); Shum et al. (2018); Lin and Xu (2019); Xu et al. (2020); Zeng et al.
(2021a); Wu et al. (2022b). It can avoid performing wrong operations and
provide potential directions of future development when an input query falls
outside the range of predefined intents. Since the exact number of unknown
intents in practical scenarios is hard to know and annotate, the lack of real
OOD examples makes it challenging to identify these samples in dialog systems.

Depending on whether labeled OOD samples are available, previous OOD detection
work can be generally classified into two types: unsupervised Bendale and
Boult (2016); Hendrycks and Gimpel (2017); Shu et al. (2017); Lee et al.
(2018); Ren et al. (2019); Lin and Xu (2019); Xu et al. (2020); Zeng et al.
(2021a, b); Wu et al. (2022a) and supervised Fei and Liu (2016); Kim and Kim
(2018); Larson et al. (2019a); Zheng et al. (2020). The former firstly learn
an in-domain (IND) intent classifier only using labeled IND data and then
estimates the confidence score of a test query. For example, Maximum Softmax
Probability (MSP) Hendrycks and Gimpel (2017) uses maximum softmax probability
as the confidence score and regards an intent as OOD if the score is below a
fixed threshold. The assumption is that OOD intents should produce a lower
softmax probability than INDs. However, neural networks can produce
arbitrarily high softmax confidence even for such abnormal OOD samples Guo et
al. (2017); Liang et al. (2018), as shown in Fig 1&2, which we call
_overconfidence_.

Figure 1: IND(case 1) vs OOD sample(case 2). Softmax score recognizes OOD
sample as IND intent type because of overconfidence issue.

Figure 2: Softmax score from MSP vs energy score from our method. Softmax
score are similar for IND and OOD (both $>0.85$) but energy score are more
distinguished.

Figure 3: The overall architecture of our proposed method.

Further, another distance-based method, Gaussian discriminant analysis (GDA)
Xu et al. (2020), is proposed to use the maximum Mahalanobis distance
Mahalanobis (1936) to all in-domain classes centroids as the confidence score.
Compared to MSP, GDA gets better OOD performance but requires expensive
computation for complex Mahalanobis distance. In this paper, we aim to use
simple softmax confidence scores for both higher performance and efficiency.
For supervised OOD detection, Fei and Liu (2016); Larson et al. (2019a), form
a _(N+1)_ -class classification problem where the _(N+1)_ -th class represents
the OOD intents. Further, Zheng et al. (2020) uses labeled OOD data to
generate an entropy regularization term. But these methods require numerous
labeled OOD intents to get superior results. We focus on using fewer labeled
OOD data (like 20 or 30) to achieve comparable even better performance.

In this paper, we propose an energy-based score function to detect OOD in an
unsupervised manner. The energy-based score function maps each query to a
single energy scalar which is lower for IND samples and higher for OOD samples
based on the energy theory LeCun et al. (2006). We first train an in-domain
intent classifier via IND data, then replace the original softmax layer with
the energy-based score function. Our method can not only mitigate the issue of
overconfident softmax probability but also reduce expensive post-processing
computation. Further, given a small portion of labeled OOD samples, we propose
an energy-based margin objective to explicitly distinguish OOD samples from
IND samples. Our contributions are three-fold: (1) We propose an energy-based
learning method for OOD intent detection to achieve higher performance and
efficiency. (2) We propose an energy-based margin objective to distinguish
energy distributions of OOD and IND samples. (3) Extensive experiments and
analysis on two benchmarks demonstrate the effectiveness of our method.

Models | CLINC-Full | CLINC-Small
---|---|---
IND | OOD | IND | OOD
Acc | F1 | Recall | F1 | Acc | F1 | Recall | F1
Unsupervised OOD | MSP Hendrycks and Gimpel (2017) | 87.16 | 87.64 | 41.40 | 44.86 | 85.02 | 85.18 | 35.81 | 36.60
LOF Lin and Xu (2019) | 85.87 | 86.08 | 58.32 | 59.28 | 82.83 | 82.98 | 53.96 | 54.63
GDA Xu et al. (2020) | 86.83 | 87.90 | 64.14 | 65.79 | 84.46 | 84.87 | 60.72 | 61.89
SCLZeng et al. (2021a) | 87.01 | 88.28 | 66.80 | 67.68 | 85.73 | 86.61 | 63.96 | 64.44
| Energy(Ours) | 88.71 | 89.17 | 68.10 | 69.64 | 86.42 | 86.48 | 65.78 | 66.52
Supervised OOD | N+1 | 91.24 | 85.29 | 24.51 | 31.08 | 90.13 | 83.23 | 21.50 | 29.17
MSP+Entropy Zheng et al. (2020) | 87.48 | 87.81 | 49.90 | 53.93 | 85.24 | 85.31 | 45.90 | 48.57
MSP+Bound Liu et al. (2020a) | 88.03 | 87.26 | 45.21 | 56.86 | 86.16 | 83.04 | 42.38 | 51.43
MSP+Margin(Ours) | 88.31 | 87.98 | 57.27 | 59.96 | 85.33 | 85.37 | 54.90 | 55.37
LOF+Entropy | 85.98 | 86.37 | 61.10 | 61.13 | 83.49 | 83.86 | 57.70 | 57.79
LOF+Bound | 86.36 | 85.66 | 57.83 | 60.15 | 81.36 | 82.88 | 64.41 | 59.30
LOF+Margin(Ours) | 86.13 | 86.59 | 65.70 | 65.59 | 83.57 | 83.97 | 63.60 | 63.18
GDA+Entropy | 87.27 | 88.14 | 68.53 | 68.82 | 85.01 | 85.53 | 65.22 | 65.65
GDA+Bound | 87.09 | 86.86 | 67.32 | 66.41 | 84.44 | 84.75 | 65.19 | 64.14
GDA+Margin(Ours) | 87.54 | 88.23 | 68.42 | 68.73 | 85.51 | 85.81 | 65.13 | 65.68
| Energy+Margin(Ours, Full Model) | 89.75 | 89.46 | 73.92 | 74.06 | 87.84 | 87.53 | 72.76 | 72.98

Table 1: Performance comparison on CLINC-Full and CLINC-Small datasets
($p<0.01$ under t-test).

##  2 Methodology

Overall Architecture Fig 3(a) shows the overall architecture of our proposed
method. We first train an in-domain intent classifier using IND data in
training stage. Then in the test stage, we extract the intent feature of a
test query and employ the detection algorithms MSP Hendrycks and Gimpel (2017)
or Energy to detect OOD. Fig 3(b) demonstrates the effectiveness of our method
distinguishing OOD distributions from IND222Because the max softmax score is
higher for IND samples and lower for OOD samples, we use the negative energy
score to align with the conventional definition where positive(IND) samples
get higher scores..

Energy-based Score Function To mitigate the issue of overconfident softmax
probability in MSP, we propose an energy-based score function to push apart
score distributions of OOD and IND samples. We first briefly review the energy
theory LeCun et al. (2006) then explain our proposed energy-based score
function for OOD detection. The previous energy work LeCun et al. (2006); Zhai
et al. (2016); Grathwohl et al. (2020); Liu et al. (2020b); Kaur et al. (2021)
aims to build a function $E(\mathbf{x}):{R}^{D}\rightarrow{R}$ which maps a
sample $\mathbf{x}$ to a single scalar called the _energy_. Given a data point
$\mathbf{x}\in{R}^{D}$, the energy function can be defined as follows:

$E(\mathbf{x})=-T\cdot\log\int_{y^{\prime}}e^{-E\left(\mathbf{x},y^{\prime}\right)/T}$
(1)

where $T$ is the temperature parameter and $E(\mathbf{x},y^{\prime})$ is the
marginal energy over label $y^{\prime}$. Essentially, energy scores can be
transfered to the likelihood probability:

$p(y\mid\mathbf{x})=\frac{e^{-E(\mathbf{x},y)/T}}{\int_{y^{\prime}}e^{-E\left(\mathbf{x},y^{\prime}\right)/T}}=\frac{e^{-E(\mathbf{x},y)/T}}{e^{-E(\mathbf{x})/T}}$
(2)

For OOD detection, since we focus on the detection algorithms for the test
stage in this paper, we train the same BiLSTM in-domain intent classifier
$f(\mathbf{x})$ via IND data as Lin and Xu (2019) in the training stage. Then
given a test query, we simply use the logits from the intent classifier to
represent $E(\mathbf{x},y^{\prime})$. Therefore, the energy score function Eq
1 can be formulated as:

$E(\mathbf{x};f)=-T\cdot\log\sum_{i}^{K}e^{f_{i}(\mathbf{x})/T}$ (3)

where $K$ is the size of IND intent classes and $f_{i}(\mathbf{x})$ is the
logit of $\mathbf{x}$ belonging to $i$-th class. We simply use a threshold on
the energy score to consider whether a test query belongs to OOD. Intuitively,
the reason why the energy score works for OOD detection is that higher energy
represents a lower likelihood of occurrence according to LeCun et al. (2006).
Therefore, unobserved OOD samples in the training stage should get lower
likelihoods as well as higher energy scores than observed IND samples. In
Appendix C, we provide a detailed theoretical derivation of why the energy
function can alleviate the overconfidence problem. Besides, Experiment 4.1
also proves energy scores better distinguish confidence distribution of OOD
data from IND data than softmax probabilities.

Energy-guided Margin Objective To further distinguish OOD from IND, we propose
an energy-guided margin objective for few-shot supervised OOD detection.
Different from Liu et al. (2020a), our approach directly models the energy
boundary by pushing apart the samples from IND and OOD, which helps recognize
OOD intents near the decision boundary and is easier to tune and less
sensitive to the noise. Specifically, we use an energy-based max-margin loss
as well as the standard cross-entropy loss to explicitly set an energy gap
between OOD and IND. We aim to learn more discriminative representations for
energy score distributions in the training stage. The energy margin loss is
formulated as:

CLINC | Full | Small
---|---|---
Avg utterance length | 9 | 9
Intents | 150 | 150
Training set size | 15100 | 7600
Training samples per class | 100 | 50
Training OOD samples amount | 100 | 100
Development set size | 3100 | 3100
Development samples per class | 20 | 20
Development OOD samples amount | 100 | 100
Testing Set Size | 5500 | 5500
Testing samples per class | 30 | 30
Development OOD samples amount | 1000 | 1000

Table 2: Statistics of the CLINC datasets.

$\displaystyle\mathcal{L}\\!=\\!\mathbb{E}_{\left(\mathbf{x}_{\text{ind}},\mathbf{x}_{\text{ood}}\right)\sim\mathcal{D}}\max\\!(0,m\\!+\\!E(\mathbf{x}_{\text{ind}})\\!-\\!E(\mathbf{x}_{\text{ood}}))$
(4)

where $m$ is the energy margin and $E$ is the energy score of IND or OOD
samples in train set. Then in the test stage, we still use the energy score to
detect OOD. Analysis 4.1 displays the effectiveness of the margin loss over
unsupervised OOD.

##  3 Experiments

###  3.1 Datasets

Figure 4: Distribution of softmax scores vs energy scores.

We use two public benchmark OOD datasets333https://github.com/clinc/oos-eval,
CLINC-Full and CLINC-SmallLarson et al. (2019b). We show the detailed
statistic of these datasets in Table 2. They both contain 150 in-domain
intents across 10 domains. The difference is that CLINC-Small has fewer in-
domain training examples than CLINC-Full. Note that all the datasets we used
have a fixed set of labeled OOD data but we don’t use it for training.

###  3.2 Metrics

We report both OOD metrics: Recall and F1-score(F1) and in-domain metrics:
F1-score(F1) and Accuracy(ACC). Since we aim to improve the performance of
detecting out-of-domain intents from user queries, OOD Recall and F1 are the
main evaluation metrics in this paper.

###  3.3 Baselines

For detection algorithms, we use MSP, LOF and GDA as baselines. For training
objectives, we use N+1, entropy and bound as baselines. We present dataset
statistics, baselines and implementation details in the appendix. We will
release our code after blind review.

###  3.4 Main Results

Table 1 shows the main results. (1) For unsupervised OOD detection, using the
energy function achieves 24.78, 10.36, 3.85, 1.96 OOD F1 improvements over
MSP, LOF, GDA and SCL on CLINC-Full. The results prove the effectiveness of
energy score function for OOD detection. Besides, for IND metrics, energy
function also outperforms SCL by 0.89%(F1), which reflects energy scores can
better distinguish OOD from IND samples without sacrificing IND performance.
(2) For supervised OOD detection, we compare different pre-training losses
under the same detection score function. We find our Margin approach achieve
consistent improvements under different detection functions on both datasets.
It demonstrates that Margin objective can stably improve the representation
space by directly pushing apart the samples from IND and OOD. We also observe
under MSP, our proposed Margin objective outperforms Entropy by 6.03% and
Bound by 3.10% on CLINC-Full. But on GDA we find no significant performance
difference. We argue the energy-based learning may not always fit in
generative distance-based detection methods like GDA. Overall, combining
energy score function and margin objective achieve the best performance over
the previous state-of-the-art by 5.24%.

Figure 5: Unsupervised vs supervised OOD detection.

##  4 Analysis

###  4.1 Distribution of softmax scores vs energy scores

To figure out why energy scores outperform softmax scores, we compare the
score histogram distributions for IND and OOD data in Fig 4. We use the same
pre-trained intent classifier to compute scores on the test set. We find
softmax scores for both IND and OOD data concentrate on high values, resulting
in severe overconfidence. By contrast, energy scores better distinguish score
distribution of OOD data from IND data. And energy distributions are smoother
than softmax score distributions. Overall, our proposed energy-based score
function can disentangle confidence score distributions for IND and OOD data.

###  4.2 Unsupervised vs supervised OOD detection

To verify the effectiveness of our proposed energy-based margin objective, we
compare the energy score statistics of unsupervised (Energy) and supervised
(Margin+Energy) OOD detection in Fig 5. Each rectangle in Fig 5 represents the
energy distribution of IND or OOD data, where the middle of the rectangle is
energy mean and the width of the rectangle is energy variance. Results show
that compared to Energy, Margin+Energy makes negative energy scores of both
OOD and IND data smaller. Further, the supervised Margin objective can
significantly decrease the variance of both OOD ($1.86\downarrow$) and IND
($3.11\downarrow$) data. Therefore, Margin can push apart energy score
distributions for OOD detection by shrinking its variance to avoid
overlapping. Besides, combined with the energy threshold (dot line in Fig 5),
unsupervised (Energy) still gets a portion of OOD samples above the threshold
which are misclassified into IND, but supervised (Margin+Energy) on the
opposite. It proves that Margin can further mitigate the issue of
overconfidence.

Figure 6: Effect of number of labeled OOD samples.

###  4.3 Effect of number of labeled OOD samples

Fig 6 shows the effect of labeled OOD training data size for supervised OOD
detection. We find Margin+Energy consistently outperforms Entropy+Energy,
especially in the few-shot supervised OOD scenario, which demonstrates strong
robustness and generalization of our proposed energy-based margin objective
for OOD detection.

###  4.4 Effect of Parameters

Temperature $T$. Fig 7 shows the effect of different energy temperature $T$.
We conduct the experiments on the CLINC-Full dataset, using Energy for
unsupervised OOD. The X-axis denotes the value of temperature $T$. In general,
$T\in(0.5,1.0)$ achieves relatively better performances and has a broad range.

Figure 7: Effect of energy temperature $T$

Margin $m$. Fig 8 shows the effect of different energy margin $m$. We conduct
the experiments on the CLINC-Full dataset, using Margin+Energy for supervised
OOD. The X-axis denotes the value of margin $m$. Results show that $m=19.0$
achieves the best performance and is robust to minor changes.

Figure 8: Effect of energy margin $m$

##  5 Conclusion

Traditional softmax-based OOD detection methods are susceptible to the
overconfidence issue. Therefore, we propose a novel energy-based score
function to mitigate the issue of softmax overconfidence. To use labeled OOD
data, we further introduce an energy-based margin objective to explicitly
distinguish energy score distributions of OOD from IND. Experiments and
analysis confirm the effectiveness of our energy-based method for OOD
detection. For future work, we hope to explore theoretical concepts of energy
and provide new guidance.

## References

  * Akasaki and Kaji (2017) Satoshi Akasaki and Nobuhiro Kaji. 2017.  Chat detection in an intelligent assistant: Combining task-oriented and non-task-oriented spoken dialogue systems.  _ArXiv_ , abs/1705.00746.
  * Bendale and Boult (2016) Abhijit Bendale and Terrance E. Boult. 2016.  Towards open set deep networks.  _2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 1563–1572.
  * Fei and Liu (2016) Geli Fei and Bing Liu. 2016.  Breaking the closed world assumption in text classification.  In _Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 506–514.
  * Grathwohl et al. (2020) Will Grathwohl, Kuan-Chieh Wang, Jörn-Henrik Jacobsen, David Duvenaud, Mohammad Norouzi, and Kevin Swersky. 2020.  Your classifier is secretly an energy based model and you should treat it like one.  _ArXiv_ , abs/1912.03263.
  * Guo et al. (2017) Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. 2017.  On calibration of modern neural networks.  In _International Conference on Machine Learning_ , pages 1321–1330. PMLR.
  * Hendrycks and Gimpel (2017) Dan Hendrycks and Kevin Gimpel. 2017.  A baseline for detecting misclassified and out-of-distribution examples in neural networks.  _ArXiv_ , abs/1610.02136.
  * Kaur et al. (2021) Ramneet Kaur, Susmit Jha, Anirban Roy, O. Sokolsky, and Insup Lee. 2021.  Are all outliers alike? on understanding the diversity of outliers for detecting oods.  _ArXiv_ , abs/2103.12628.
  * Kim and Kim (2018) Joo-Kyung Kim and Young-Bum Kim. 2018.  Joint learning of domain classification and out-of-domain detection with dynamic class weighting for satisficing false acceptance rates.  _ArXiv_ , abs/1807.00072.
  * Kingma and Ba (2014) Diederik P Kingma and Jimmy Ba. 2014.  Adam: A method for stochastic optimization.  _arXiv preprint arXiv:1412.6980_.
  * Larson et al. (2019a) Stefan Larson, Anish Mahendran, Joseph Peper, Christopher Clarke, Andrew Lee, Parker Hill, Jonathan K. Kummerfeld, Kevin Leach, Michael Laurenzano, Lingjia Tang, and Jason Mars. 2019a.  An evaluation dataset for intent classification and out-of-scope prediction.  In _EMNLP/IJCNLP_.
  * Larson et al. (2019b) Stefan Larson, Anish Mahendran, Joseph J. Peper, Christopher Clarke, Andrew Lee, Parker Hill, Jonathan K. Kummerfeld, Kevin Leach, Michael A. Laurenzano, Lingjia Tang, and Jason Mars. 2019b.  An evaluation dataset for intent classification and out-of-scope prediction.  In _EMNLP-IJCNLP_.
  * LeCun et al. (2006) Yann LeCun, Sumit Chopra, Raia Hadsell, M Ranzato, and F Huang. 2006.  A tutorial on energy-based learning.  volume 1.
  * Lee et al. (2018) Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. 2018.  A simple unified framework for detecting out-of-distribution samples and adversarial attacks.  _ArXiv_ , abs/1807.03888.
  * Liang et al. (2018) Shiyu Liang, Yixuan Li, and R. Srikant. 2018.  Enhancing the reliability of out-of-distribution image detection in neural networks.  _arXiv: Learning_.
  * Lin and Xu (2019) Ting-En Lin and Hua Xu. 2019.  Deep unknown intent detection with margin loss.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 5491–5496.
  * Liu et al. (2020a) Weitang Liu, Xiaoyun Wang, John Owens, and Yixuan Li. 2020a.  Energy-based out-of-distribution detection.  _Advances in Neural Information Processing Systems_.
  * Liu et al. (2020b) Weitang Liu, Xiaoyun Wang, John Douglas Owens, and Yixuan Li. 2020b.  Energy-based out-of-distribution detection.  _ArXiv_ , abs/2010.03759.
  * Mahalanobis (1936) Prasanta Chandra Mahalanobis. 1936.  On the generalized distance in statistics.  National Institute of Science of India.
  * Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher D Manning. 2014.  Glove: Global vectors for word representation.  In _Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP)_ , pages 1532–1543.
  * Ren et al. (2019) Jie Ren, Peter J. Liu, Emily Fertig, Jasper Snoek, Ryan Poplin, Mark A. DePristo, Joshua V. Dillon, and Balaji Lakshminarayanan. 2019.  Likelihood ratios for out-of-distribution detection.  _ArXiv_ , abs/1906.02845.
  * Shu et al. (2017) Lei Shu, Hu Xu, and Bing Liu. 2017.  Doc: Deep open classification of text documents.  In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_ , pages 2911–2916.
  * Shum et al. (2018) H. Shum, X. He, and Di Li. 2018.  From eliza to xiaoice: challenges and opportunities with social chatbots.  _Frontiers of Information Technology & Electronic Engineering_, 19:10–26.
  * Tulshan and Dhage (2018) Amrita S Tulshan and Sudhir Namdeorao Dhage. 2018.  Survey on virtual assistant: Google assistant, siri, cortana, alexa.  In _International symposium on signal processing and intelligent recognition systems_ , pages 190–201.
  * Wu et al. (2022a) Yanan Wu, Keqing He, Yuanmeng Yan, QiXiang Gao, Zhiyuan Zeng, Fujia Zheng, Lulu Zhao, Huixing Jiang, Wei Wu, and Weiran Xu. 2022a.  Revisit overconfidence for ood detection: Reassigned contrastive learning with adaptive class-dependent threshold.  In _NAACL_.
  * Wu et al. (2022b) Yanan Wu, Zhiyuan Zeng, Keqing He, Yutao Mou, Pei Wang, and Weiran Xu. 2022b.  Distribution calibration for out-of-domain detection with bayesian approximation.  In _COLING_.
  * Xu et al. (2020) Hong Xu, Keqing He, Yuanmeng Yan, Sihong Liu, Zijun Liu, and Weiran Xu. 2020.  A deep generative distance-based classifier for out-of-domain detection with mahalanobis space.  In _Proceedings of the 28th International Conference on Computational Linguistics_ , pages 1452–1460, Barcelona, Spain (Online). International Committee on Computational Linguistics.
  * Zeng et al. (2021a) Zhiyuan Zeng, Keqing He, Yuanmeng Yan, Zijun Liu, Yanan Wu, Hong Xu, Huixing Jiang, and Weiran Xu. 2021a.  Modeling discriminative representations for out-of-domain detection with supervised contrastive learning.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers)_ , pages 870–878, Online. Association for Computational Linguistics.
  * Zeng et al. (2021b) Zhiyuan Zeng, Keqing He, Yuanmeng Yan, Hong Xu, and Weiran Xu. 2021b.  Adversarial self-supervised learning for out-of-domain detection.  In _NAACL_.
  * Zhai et al. (2016) Shuangfei Zhai, Yu Cheng, W. Lu, and Zhongfei Zhang. 2016.  Deep structured energy based models for anomaly detection.  In _ICML_.
  * Zheng et al. (2020) Yinhe Zheng, Guanyi Chen, and Minlie Huang. 2020.  Out-of-domain detection for natural language understanding in dialog systems.  _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , 28:1198–1209.

##  Appendix A Baseline Details

We perform main experiments based on two different settings, unsupervised OOD
and supervised OOD detection. For unsupervised OOD detection, we compare our
proposed energy detection algorithm with other methods, MSP (Maximum Softmax
Probability)Hendrycks and Gimpel (2017), LOF (Local Outlier Factor)Lin and Xu
(2019), GDA (Gaussian Discriminant Analysis)Xu et al. (2020). For supervised
OOD detection, we also compare our proposed energy-based margin objective with
entropy Zheng et al. (2020) and N+1 Fei and Liu (2016); Larson et al. (2019a).
Note that margin and entropy objectives are used in the training stage, we
still need detection algorithms MSP, GDA or Energy to detect in the test
stage. We supplement the relevant baseline details as follows:

MSP (Maximum Softmax Probability)Hendrycks and Gimpel (2017) uses maximum
softmax probability as the confidence score and regards an intent as OOD if
the score is below a fixed threshold.

LOF (Local Outlier Factor)Lin and Xu (2019) uses the local outlier factor to
detect unknown intents. The motivation is that if an example’s local density
is significantly lower than its k-nearest neighbor’s, it is more likely to be
considered as the unknown intents.

GDA (Gaussian Discriminant Analysis)Xu et al. (2020) is a generative distance-
based classifier for out-of-domain detection with Euclidean space. They
estimate the class-conditional distribution on feature spaces of DNNs via
Gaussian discriminant analysis (GDA) to avoid over-confidence problems and use
Mahalanobis distance to measure the confidence score of whether a test sample
belongs to OOD. GDA is the state-of-the-art detection method till now, our
proposed energy score still significantly outperforms GDA.

Note that LOF and GDA both require additional post-processing modules to
estimate density or distance, which induces expensive computation. We conduct
a performance comparison for inference time in Table 3. Since SCL only adds a
pre-training loss along with CE and also uses GDA for detection, the inference
time is equal to GDA.

Detect Method | Inference time
---|---
MSP | 1.00x
Energy(Ours) | 1.00x
GDA/SCL | 30.63x
LOF | 30.89x

Table 3: Inference time comparison between different methods.

SCLZeng et al. (2021a) uses a supervised contrastive learning objective to
minimize intra-class variance by pulling together in-domain intents belonging
to the same class and maximize inter-class variance by pushing apart samples
from different classes. Note that SCL still needs a confidence score function.
To keep fair comparison, we follow the original paper using GDA detection
method.

N+1(Fei and Liu (2016); Larson et al. (2019a)) is an N+1 classification model
which simply considers OOD samples as a new class.

Entropy(Zheng et al. (2020)) uses labeled OOD data to generate an entropy
regularization term to enforce the predicted distribution of OOD inputs closer
to the uniform distribution:

$\displaystyle\mathcal{L}\\!=\mathbb{E}_{\left(\mathbf{x_{ood}}\right)\sim\mathcal{D}}[-H(p_{\theta}(y|x_{ood}))]\vspace{-0.9cm}$
(5)

where $H$ is the Shannon entropy of the predicted distribution.
$p_{\theta}(y|x_{ood})$ is the predicted distribution of the input OOD
utterance $x_{ood}$.

Bound(Liu et al. (2020b)) uses a regularization loss defined in terms of
energy to further widen the energy gap:

$\displaystyle\begin{split}\mathcal{L}\\!=\\!\mathbb{E}_{\left(\mathbf{x}_{\text{ind}}\right)\sim\mathcal{D}}\max\\!(0,E(\mathbf{x}_{\text{ind}})\\!-\text{m}_{\text{ind}}))^{2}\\\
\\!+\mathbb{E}_{\left(\mathbf{x}_{\text{ood}}\right)\sim\mathcal{D}}\max\\!(0,\text{m}_{\text{ood}}-E(\mathbf{x}_{\text{ood}})))^{2}\vspace{-0.9cm}\end{split}$
(6)

where $E$ is the energy score of IND or OOD samples in the train set. This
learning objective using two squared hinge loss with two hyper-parameters
$m_{ind}$ and $m_{ood}$. Note that Bound aims at OOD image classification and
replies on two independent energy bounds. Instead, our proposed Margin
constructs a contrastive energy margin between IND intents and OOD intents to
better disentangle energy distributions.

##  Appendix B Implementation Details

We use the public pre-trained 300 dimensions GloVe embeddings Pennington et
al. (2014)444https://github.com/stanfordnlp/GloVe to embed tokens. We use a
two-layer BiLSTM as a feature extractor and set the dimension of hidden states
to 128. The dropout value is fixed at 0.5. We use Adam optimizer Kingma and Ba
(2014) to train our model. We set the learning rate to 1E-03. In the training
stage, we use standard cross-entropy loss for unsupervised OOD and cross-
entropy+energy-guided margin loss for supervised OOD. Besides, in supervised
OOD scenario, we employ restriction-oriented random sampling. Specifically, we
guarantee that IND and OOD samples are both included in each batch to
facilitate calculation of margin loss. We both set the training epoch up to
200 with a early stop of patience 15. For our proposed energy-guided margin
loss, we set the margin $m$ to 19.0 and the temperature $T$ to 0.8. We use the
best OOD F1 scores on the validation set to calculate the threshold
adaptively. Each result of the experiments is tested 5 times under the same
setting and gets the average value. The training stage of our models lasts
about 2 minutes for unsupervised OOD and 4 minutes for supervised OOD both on
a single Tesla T4 GPU(16 GB of memory). The average value of the trainable
model parameters is 3.05M. We will release our code after blind review.

##  Appendix C A Theoretical Proof of Energy Score vs Softmax Score

In this section, we give a theoretical proof of why energy score outperforms
softmax score. Supposing we get the output logits from the intent classifier,
we represent MSP as follows:

$\displaystyle log\ \mathbf{MSP}(logits)$ $\displaystyle=log\ max\
\mathit{softmax}(logits)$ (7) $\displaystyle=log\
max\frac{exp(logits_{i})}{\sum_{i}exp(logits_{i})}$ $\displaystyle=log\
\frac{exp\ max(logits)}{\sum_{i}exp(logits_{i})}$
$\displaystyle=max(logits)-log\ sum\ exp(logits)$

where $logits_{i}$ represents the $i$-th value in the vector $logits$. Recap
the energy definetion:

$E(\mathbf{x};f)=-T\cdot\log\sum_{i}^{K}e^{f_{i}(\mathbf{x})/T}$ (8)

Here we set $T$ to 1. Therefore, we get the following equation:

$log\ \mathbf{MSP}(logits)=\underbrace{max(logits)}_{regularization\
item}+\mathbf{Energy}(logits)$ (9)

If the output logits get a high max value, then $max(logits)$ performs as a
regularization item to avoid energy score increasing. Therefore, energy score
can better mitigate the overconfidence issue than softmax score.