# Polaritonic Ultrastrong Coupling: Quantum Entanglement in Ground State

Qingtian Miao<EMAIL_ADDRESS>Department of Physics and Astronomy, Texas A&M
University, Texas 77843, USA  G.S. Agarwal  Institute for Quantum Science and
Engineering, Department of Biological and Agricultural Engineering, Department
of Physics and Astronomy, Texas A&M University, College Station, Texas 77843,
USA

###### Abstract

The ultrastrong coupling between the elementary excitations of matter and
microcavity modes is studied in a fully analytical quantum-mechanical
theoretical framework. The elementary excitation could be phonons, excitons,
plasmons, etc. From the diagonalization of the Hamiltonian, we obtain the
ground state of the polariton Hamiltonian. The ground state belongs to the
Gaussian class. Using the Gaussian property we calculate the quantum
entanglement in the ground state. We use two different measures for quantum
entanglement — entanglement entropy and the logarithmic negativity parameter
and obtain rather simple analytical expressions for the entanglement measures.
Our findings show that the amount of quantum entanglement in the ground state
is quite significant in the ultrastrong coupling regime. It can be obtained
from the measurement of the polariton frequencies.

##  I Introduction

The behavior of matter interacting with cavity fields is known to produce
distinct characteristics from that in free space [1, 2, 3]. This is because
the interaction parameter $g$ can have values in the weak coupling range, or
strong or even ultrastrong range. While the strong coupling regime has been
thoroughly investigated, the ultrastrong coupling is attracting intense
attention now [4, 5]. In the case of a qubit interacting with a cavity, the
ultrastrong coupling $g/\omega$ say about $0.5$ or more is yet to be realized.
In such cases, parametric interactions have been suggested to reach the
ultrastrong coupling regime [6, 7, 8]. The situation is different for a large
collective system interacting with cavity fields as here one can take
advantage of the effective enhancement of the coupling by $\sqrt{N}$ factor
[9]. Such an enhancement factor has felicitated the observation of strong
coupling even when $g$ was in the weak coupling range [10, 11, 12]. Several
recent works have successfully observed ultrastrong coupling regime as the
density of the elementary excitations was in the range of solid state
densities. Some of these recent experiments are for — phonon polaritons in
cavities containing hexagonal boron nitride (hBN) [13]; plasmon polaritons
[14, 15]; excitonic systems [16, 17, 18, 19, 20]; 2D electron gas [21]. The
possibility of novel photochemistry in cavities under ultrastrong coupling has
been investigated [22]. The cavity-free ultrastrong coupling was also realized
using meta materials [23, 24]. A typical experimental observation consists of
the observation of the transmission spectra and the polaritonic splitting of
the spectra — the splitting is of the order of $g$ which is about $30-70\%$ of
the bare frequency of the cavity which is supposed to be on resonance with the
frequency of the elementary excitation in the solid state material. All this
can be explained in classical calculations of transmission from a layered
medium [25].

At a quantum level the elementary excitations are coupled to the radiation
field modes and the Hamiltonian is diagonalized to find the new
quasiparticles, i.e., polaritons [26, 27, 28]. The transmission spectrum
basically probes the separation between the frequencies of two polaritons
though it has to be borne in mind that the observed separation would also
depend on the reflection and transmission properties of the mirrors forming
the cavity. This latter aspect has not been fully addressed. In this paper, we
study the quantum entanglement which is present in the ground state of the
Hopfield Hamiltonian. We derive the ground state which belongs to the class of
Gaussian states. We then use two different entanglement measures —
entanglement entropy [29] and the logarithmic negativity parameter [30]. The
quantum entanglement increases as $g/\omega$ increases. There is no quantum
entanglement in the limits of weak and strong couplings as $g/\omega\ll 1$.
The magnitude of entanglement is given in terms of the polariton frequencies
and thus experimental study of the transmission spectra can be used to assess
the amount of entanglement.

##  II Polaritons in Ultrastrong Coupling

In the following, we will consider a microcavity fully filled with hBN where
the ultrastrong phonon-photon coupling was demonstrated [13]. And our theory
can also be applied to other systems like exciton-polaritons and plasmon-
polaritons in the ultrastrong coupling regime. The transmission spectrum can
be measured at normal incidence, where a polaritonic splitting shows the light
couples with the in-plane transverse optical (TO) phonon of hBN, which takes
place perpendicular to the direction of propagation. Thus we consider
ultrastrong coupling of two harmonic oscillators: the normal incidence Fabry-
Pérot microcavity mode with energy $\omega_{\rm c}$, and the TO phonon mode
with energy $\omega_{0}$ (we set $\hbar=1$). The behavior of the system can be
described by the Hopfield Hamiltonian, which reads [31]

$\displaystyle\hat{H}$ $\displaystyle=\omega_{\rm
c}\left(\frac{1}{2}+\hat{a}^{\dagger}\hat{a}\right)+\omega_{0}\left(\frac{1}{2}+\hat{b}^{\dagger}\hat{b}\right)$
(1)
$\displaystyle\qquad+\Omega\left(\hat{a}^{\dagger}+\hat{a}\right)\left(\hat{b}^{\dagger}+\hat{b}\right)+G\left(\hat{a}^{\dagger}+\hat{a}\right)^{2},$

where $\hat{a}$ and $\hat{b}$ are the microcavity and TO phonon annihilation
operators, respectively, and $\Omega$ is the vacuum Rabi energy. The third
term in Eq. (1) contains the anti-resonant or counter-rotating part which
cannot be omitted in the ultrastrong coupling regime. The fourth term, which
is called the diamagnetic term, originates from the quadratic electromagnetic
vector potential term $\mathbf{A}^{2}$ of the light-matter minimal coupling.
Any quasi-resonant transition gives a contribution to the coefficient $G$, but
in this paper, we omit the effect of other transitions, then
$G=\Omega^{2}/\omega_{0}$ [5]. The vacuum Rabi energy for phonon polaritons
being considered can be written as [26]

$\Omega=g\sqrt{\frac{\omega_{0}}{\omega_{\rm c}}},$ (2)

where $g$ is called coupling strength in this article.

The phonon-cavity polaritonic transition energies $\omega_{\pm}$ can be
obtained by two-mode Hamiltonian diagonalization,

$\displaystyle\omega^{2}_{+}=\omega_{0}^{2}+2\Omega\sqrt{\omega_{\rm
c}\omega_{0}}\tan\frac{\theta}{2},$ (3)
$\displaystyle\omega^{2}_{-}=\omega_{0}^{2}-2\Omega\sqrt{\omega_{\rm
c}\omega_{0}}\cot\frac{\theta}{2},$

where $\theta=\arctan\left(\frac{\omega_{\rm c}\left(4G+\omega_{\rm
c}\right)-\omega_{0}^{2}}{4\Omega\sqrt{\omega_{\rm
c}\omega_{0}}}\right)+\frac{\pi}{2}$, $(0<\theta<\pi)$, and
$\omega_{+}>\omega_{-}$. The polariton energies $\omega_{\pm}$ coincide with
the classical bulk dielectric dispersion law [32],

$\epsilon(\omega)=\frac{c^{2}k^{2}}{\omega^{2}}=\epsilon_{\infty}+\frac{4\epsilon_{\infty}g^{2}}{\omega_{0}^{2}-\omega^{2}},$
(4)

where $\epsilon(\omega)$ is the dielectric function of hBN, $\mathbf{k}$ is
the wave vector, and $\omega_{\rm c}=kc/\sqrt{\epsilon_{\infty}}$.

The diagonalized Hamiltonian of the system can be written in a simple form,

$H=\omega_{-}\left(\frac{1}{2}+\hat{A}_{-}^{\dagger}\hat{A}_{-}\right)+\omega_{+}\left(\frac{1}{2}+\hat{A}_{+}^{\dagger}\hat{A}_{+}\right),$
(5)

where the phonon-cavity polariton normal modes, i.e., the new quasiparticle
annihilation operators can be introduced as

$\displaystyle\hat{A}_{-}$
$\displaystyle=-\frac{1}{2}\cos\frac{\theta}{2}\left(\mu+\frac{1}{\mu}\right)\hat{a}+\frac{1}{2}\sin\frac{\theta}{2}\left(\nu+\frac{1}{\nu}\right)\hat{b}$
(6)
$\displaystyle\qquad-\frac{1}{2}\cos\frac{\theta}{2}\left(\mu-\frac{1}{\mu}\right)\hat{a}^{\dagger}-\frac{1}{2}\sin\frac{\theta}{2}\left(\nu-\frac{1}{\nu}\right)\hat{b}^{\dagger},$
$\displaystyle\hat{A}_{+}$
$\displaystyle=\frac{1}{2}\sin\frac{\theta}{2}\left(\nu+\frac{1}{\nu}\right)\hat{a}+\frac{1}{2}\cos\frac{\theta}{2}\left(\mu+\frac{1}{\mu}\right)\hat{b}$
(7)
$\displaystyle\qquad+\frac{1}{2}\sin\frac{\theta}{2}\left(\nu-\frac{1}{\nu}\right)\hat{a}^{\dagger}-\frac{1}{2}\cos\frac{\theta}{2}\left(\mu-\frac{1}{\mu}\right)\hat{b}^{\dagger},$

with defined variables $\mu=\sqrt{\omega_{-}/\omega_{\rm c}}$ and
$\nu=\sqrt{\omega_{+}/\omega_{\rm c}}$. To obtain this form of canonical
transformation, we also use the simple relation
$\omega_{-}\omega_{+}=\omega_{\rm c}\omega_{0}$ which can be derived from Eq.
(4). And we can also write $\hat{A}_{\pm}$ in terms of quadratures,

$\sqrt{2}\hat{A}_{-}=-\cos\frac{\theta}{2}\left(\mu
x_{1}+\frac{1}{\mu}\frac{\partial}{\partial
x_{1}}\right)+\sin\frac{\theta}{2}\left(\frac{1}{\nu}x_{2}+\nu\frac{\partial}{\partial
x_{2}}\right),$ (8) $\sqrt{2}\hat{A}_{+}=\sin\frac{\theta}{2}\left(\nu
x_{1}+\frac{1}{\nu}\frac{\partial}{\partial
x_{1}}\right)+\cos\frac{\theta}{2}\left(\frac{1}{\mu}x_{2}+\mu\frac{\partial}{\partial
x_{2}}\right),$ (9)

where the Hermitian quadrature operators in the system are defined as
$\hat{x}_{1}=\frac{\hat{a}+\hat{a}^{\dagger}}{\sqrt{2}}$,
$\hat{p}_{1}=\frac{\hat{a}-\hat{a}^{\dagger}}{\sqrt{2}i}$,
$\hat{x}_{2}=\frac{\hat{b}+\hat{b}^{\dagger}}{\sqrt{2}}$,
$\hat{p}_{2}=\frac{\hat{b}-\hat{b}^{\dagger}}{\sqrt{2}i}$.

##  III Ground State

Figure 1: Coefficient $c$ in the Gaussian wave function plotted against the
bare cavity energy $\omega_{\rm c}$ with $\omega_{0}=169.1$ meV and different
values of $g$.

Now we consider the ground Fock state
$|0_{\hat{A}_{-}},0_{\hat{A}_{+}}\rangle$ of the two-mode system with energy
$\omega_{0,0}=\frac{\omega_{-}+\omega_{+}}{2}$. From Eq. (4),
$(\omega_{+}+\omega_{-})^{2}=(\omega_{\rm c}+\omega_{0})^{2}+4g^{2}$, thus
$\omega_{0,0}>\frac{\omega_{\rm c}+\omega_{0}}{2}$. The ground state possesses
virtual photons because our Hamiltonian does not conserve the number of
photons. Since it is the vacuum of polariton excitations,
$\hat{A}_{-}|0_{\hat{A}_{-}},0_{\hat{A}_{+}}\rangle=0$,
$\hat{A}_{+}|0_{\hat{A}_{-}},0_{\hat{A}_{+}}\rangle=0$, and the ground state
wave function $\Psi_{0,0}(x_{1},x_{2})=\langle
x_{1},x_{2}|0_{\hat{A}_{-}},0_{\hat{A}_{+}}\rangle$, we can obtain first order
differential equations

$\displaystyle\left[-\cos\frac{\theta}{2}\left(\mu
x_{1}+\frac{1}{\mu}\frac{\partial}{\partial
x_{1}}\right)+\sin\frac{\theta}{2}\left(\frac{1}{\nu}x_{2}+\nu\frac{\partial}{\partial
x_{2}}\right)\right]$ (10)
$\displaystyle\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\Psi_{0,0}(x_{1},x_{2})=0,$
$\displaystyle\left[\sin\frac{\theta}{2}\left(\nu
x_{1}+\frac{1}{\nu}\frac{\partial}{\partial
x_{1}}\right)+\cos\frac{\theta}{2}\left(\frac{1}{\mu}x_{2}+\mu\frac{\partial}{\partial
x_{2}}\right)\right]$ (11)
$\displaystyle\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\Psi_{0,0}(x_{1},x_{2})=0.$

Thus the normalized two-mode Gaussian wave function in the ground state in
quadrature space can be calculated by solving those equations,

$\Psi_{0,0}(x_{1},x_{2})=\frac{1}{\sqrt{\pi}}\exp\left(-\frac{1}{2}\left(ax_{1}^{2}+bx_{2}^{2}+2cx_{1}x_{2}\right)\right),$
(12)

with real coefficients

$a=\mu^{2}\cos^{2}\frac{\theta}{2}+\nu^{2}\sin^{2}\frac{\theta}{2},\quad
b=\frac{1}{\mu^{2}}\cos^{2}\frac{\theta}{2}+\frac{1}{\nu^{2}}\sin^{2}\frac{\theta}{2},$
(13)

and

$c=\frac{\omega_{+}-\omega_{-}}{2\sqrt{\omega_{\rm
c}\omega_{0}}}\sin\theta=\frac{2\Omega}{\omega_{-}+\omega_{+}},$ (14)

where we use the simple relation
$\omega_{+}^{2}-\omega_{-}^{2}=4\Omega\sqrt{\omega_{\rm
c}\omega_{0}}/\sin\theta$ which can be derived from Eq. (3). The coefficient
$c$ directly affects the entanglement, and we plot it as a function of the
bare cavity energy $\omega_{\rm c}$ in Fig. 1 for the phonon frequency
$\omega_{0}=169.1$ meV of hBN and different coupling strength $g$. Since
$c\neq 0$, two-mode wavefunction $\Psi_{0,0}(x_{1},x_{2})\neq
f_{1}(x_{1})f_{2}(x_{2})$, where $f_{1}[f_{2}]$ is a function of
$x_{1}[x_{2}]$ alone. Thus we find the ground state of our system is
inseparable or entangled. Clearly, to have significant entanglement, $g$
cannnot be too small. For weak coupling, i.e., $g\rightarrow 0$, the polariton
normal modes go back to cavity mode and phonon mode, and we can expect that
the amount of entanglement decreases to $0$.

We can also calculate the first excited states in quadrature space,

$\displaystyle\Psi_{1,0}(x_{1},x_{2})$
$\displaystyle=A_{-}^{\dagger}\Psi_{0,0}(x_{1},x_{2})$ (15)
$\displaystyle=\sqrt{2}\left(-\mu\cos\frac{\theta}{2}x_{1}+\frac{1}{\nu}\sin\frac{\theta}{2}x_{2}\right)\Psi_{0,0}(x_{1},x_{2}),$
$\displaystyle\Psi_{0,1}(x_{1},x_{2})$
$\displaystyle=A_{+}^{\dagger}\Psi_{0,0}(x_{1},x_{2})$ (16)
$\displaystyle=\sqrt{2}\left(\nu\sin\frac{\theta}{2}x_{1}+\frac{1}{\mu}\cos\frac{\theta}{2}x_{2}\right)\Psi_{0,0}(x_{1},x_{2}),$

with one polariton excitation. $\Psi_{1,0}(x_{1},x_{2})$ is the wavefunction
for the lower polariton, while $\Psi_{0,1}(x_{1},x_{2})$ represents the upper
polariton.

##  IV Quantitative Measures of Entanglement

###  IV.1 Entanglement Entropy

For a general pure bipartite quantum state, the Von Neumann entropy of either
of the reduced density matrix of a subsystem serves as entanglement entropy to
measure the degree of quantum entanglement, and it can be proved that they
have the same value. For our system, the ground state density operator is
$\hat{\rho}=|0_{\hat{A}_{-}},0_{\hat{A}_{+}}\rangle\langle
0_{\hat{A}_{-}},0_{\hat{A}_{+}}|$. Since it is a pure state, the entropy of
the bipartite state $S(\hat{a},\hat{b})=0$, and the entropies of the reduced
density matrices are the same, $S(\hat{a})=S(\hat{b})$ [33]. The ground state
is Gaussian and hence the state of each mode will be a mixed Gaussian state
which has a Gaussian Wigner function in quadrature space,

$W(x,p)=\frac{1}{2\pi\sqrt{\alpha\beta-\gamma^{2}}}\exp\left(-\frac{1}{2}\frac{\alpha
x^{2}+\beta p^{2}-2\gamma xp}{\alpha\beta-\gamma^{2}}\right),$ (17)

where the variance $\alpha=\langle\hat{p}^{2}\rangle$,
$\beta=\langle\hat{x}^{2}\rangle$,
$\gamma=\frac{1}{2}\langle\hat{x}\hat{p}+\hat{p}\hat{x}\rangle$, with no
displacements, i.e., $\langle\hat{x}\rangle=\langle\hat{p}\rangle=0$. Then the
expression for entropy of a reduced density operator of either mode can be
written as [34, 33]

$S=k_{B}\left[\left(\sigma+1\right)\ln\left(\sigma+1\right)-\sigma\ln\sigma\right],$
(18)

where $k_{B}$ is the Boltzmann constant, and

$\sigma=\left(\alpha\beta-\gamma^{2}\right)^{1/2}-\frac{1}{2}.$ (19)

In our ground state,

$\displaystyle\langle\hat{x}_{1}^{2}\rangle=\langle\hat{p}_{2}^{2}\rangle=\frac{b}{2},\qquad\langle\hat{x}_{2}^{2}\rangle=\langle\hat{p}_{1}^{2}\rangle=\frac{a}{2},$
(20)
$\displaystyle\frac{1}{2}\langle\hat{x}_{1}\hat{p}_{1}+\hat{p}_{1}\hat{x}_{1}\rangle=\frac{1}{2}\langle\hat{x}_{2}\hat{p}_{2}+\hat{p}_{2}\hat{x}_{2}\rangle=0,$

and using the simple relation $ab=1+c^{2}$ which can be derived from Eq. (13)
and Eq. (14), it is obvious that parameter
$\sigma=\frac{1}{2}\left(\sqrt{c^{2}+1}-1\right)=\sinh^{2}\left(\frac{r}{2}\right)$,
where we let $c=\sinh r$. Thus the entanglement entropy can be written in a
form that merely depends on $c$,

$\displaystyle S/k_{B}$
$\displaystyle=\sqrt{c^{2}+1}\ln\frac{\sqrt{c^{2}+1}+1}{c}+\ln\frac{c}{2}$
(21) $\displaystyle=\cosh
r\ln\left(\coth\frac{r}{2}\right)+\ln\left(\frac{\sinh r}{2}\right).$

When the coupling strength $g\rightarrow 0$, then since $\sigma\rightarrow 0$,
we find the entanglement entropy $S\rightarrow 0$. The entanglement entropy as
a function of the bare cavity energy $\omega_{\rm c}$ is presented in Fig. 2
for the phonon frequency $\omega_{0}=169.1$ meV of hBN and different coupling
strength $g$. With increasing coupling strength, we get the increased
entanglement entropy as expected.

Figure 2: Entanglement entropy $S/k_{B}$ plotted against the bare cavity
energy $\omega_{\rm c}$ with $\omega_{0}=169.1$ meV and different values of
$g$.

###  IV.2 Logarithmic Negativity Parameter

Figure 3: Logarithmic negativity $E_{N}(\rho)$ plotted against the bare cavity
energy $\omega_{\rm c}$ with $\omega_{0}=169.1$ meV and different values of
$g$.

The Peres-Horodecki separability criterion to determine whether bipartite
continuous variable states are separable was presented by Simon [35], and for
a two-mode Gaussian state determined by variance matrix $V$, the criterion is
simply an inequality obeyed by a complete set of ${\rm Sp}(2,R)\otimes{\rm
Sp}(2,R)$ invariants for $V$. The elements of variance matrix $V$ are given by

$V_{ij}=\frac{1}{2}{\rm
tr}\left(\hat{\rho}\left\\{\hat{\xi}_{i},\hat{\xi}_{j}\right\\}\right),$ (22)

where there is no displacement, and four dimensional vector
$\hat{\xi}=(\hat{x}_{1},\hat{p}_{1},\hat{x}_{2},\hat{p}_{2})$ for our two-mode
system. The variance matrix has the form $V=\left(\begin{array}[]{cc}A&C\\\
C^{T}&B\end{array}\right)$, and the invariants are the numbers $\det A$, $\det
B$, $\det C$ and $\det V$. For a pure Gaussian state, $\det V=\frac{1}{16}$,
and $\det A+\det B+2\det C=\frac{1}{2}$ [30].

The Peres-Horodecki separability criterion states that a separable density
operator under the partial transpose goes necessarily into a non-negative
operator, and the logarithmic negativity parameter is nothing but a
quantitative version of the criterion that quantifies how much the new density
operator deviates from being a non-negative matrix.

Two sympletic eigenvalues of the covariance matrix $V$ associated with its
density operator under the partial transpose can be derived [30, 36],

$\tilde{\nu}_{\pm}=\sqrt{\frac{\tilde{\Delta}(V)-\sqrt{\tilde{\Delta}(V)^{2}-4\det
V}}{2}},$ (23)

where $\tilde{\Delta}(V)=\det A+\det B-2\det C$. Equivalent to the criterion
presented by Simon, $\nu_{-}<\frac{1}{2}$ is a necessary and sufficient
condition for two-mode Gaussian states to be entangled. In our ground state,
the ${\rm Sp}(2,R)\otimes{\rm Sp}(2,R)$ invariants are calculated,

$\det A=\det B=\frac{1}{4}(1+c^{2}),\qquad\det C=-\frac{1}{4}c^{2},$ (24)

thus $\tilde{\Delta}(V)=\frac{1}{2}+c^{2}$,

$\tilde{\nu}_{-}=\frac{1}{2}(\sqrt{c^{2}+1}-c)=\frac{1}{2}e^{-r}.$ (25)

Since our coefficient $c>0$, then $\tilde{\nu}_{-}<\frac{1}{2}$, the ground
wave function is entangled. When the coupling strength $g\rightarrow 0$, i.e.,
$c\rightarrow 0$, we have $\tilde{\nu}_{-}=\frac{1}{2}$, then the state
becomes separable. The logarithmic negativity $E_{N}(\rho)$ is used to measure
the amount of quantum entanglement,

$E_{N}(\rho)=\max[0,-\log_{2}(2\tilde{\nu}_{-})].$ (26)

The logarithmic negativity as a function of the bare cavity energy
$\omega_{\rm c}$ is presented in Fig. 3 for $\omega_{0}=169.1$ meV of hBN and
different coupling strength $g$. The trend of logarithmic negativity is very
similar to entanglement entropy, and we can reach the same conclusion: for
weak coupling, there is almost no entanglement, and we can obtain a large
amount of entanglement in the ultrastrong regime of phonon-cavity interaction.

##  V Conclusions

In the experiment, we can measure polariton energies $\omega_{\pm}$ from the
transmission spectrum. Then by matching with Eq. (3) applying our theory
model, we can determine the coupling strength of our system, and coefficient
$c$ can be easily found by Eq. (14). Since the entanglement entropy and
logarithmic negativity are exclusively determined by the coefficient $c$, we
can make a possible quantitative measure of quantum entanglement in the ground
state following these procedures. It should be noted that we do not discuss
entanglement in the first excited states. These states exhibit entanglement
despite not being in the ultrastrong coupling regime.

While our theory model is centered around phonon polaritons in cavities, it
can be extended to other types of polaritons. Applying our conclusion to
exciton polaritons is possible [16], but it is important to note that normal
incidence should be assumed, i.e., the incidence angle $\theta$ must be set to
zero. We should also neglect spatial dispersion, then the expression for the
vacuum Rabi energy becomes
$\Omega=\omega_{0}\sqrt{\frac{\pi\chi\omega_{0}}{\omega_{c}\epsilon_{\infty}}}$,
where $\chi$ represents the coupling constant between photon and exciton
oscillators [26, 32, 28]. The relation between $g$ and $\chi$ can be
established, which is given by
$g=\omega_{0}\sqrt{\frac{\pi\chi}{\epsilon_{\infty}}}$. Another example of
applying our theory is plasmon polaritons. As explored in a recent study [15],
the vacuum Rabi energy $\Omega$ is independent of the energy of the cavity
mode $\omega_{\rm c}$. Then we can modify the interaction parameter in our
theory, i.e., make $\Omega=g$. This modification leads to the same conclusion
that the quantum entanglement entropy in the ground state increases with the
coupling strength.

Acknowledgements

We thank the support of Air Force Office of Scientific Research (Award No
FA-9550-20-1-0366) and the Robert A Welch Foundation (A-1943-20210327).

## References

  * Haroche and Raimond [2006] S. Haroche and J.-M. Raimond,  _Exploring the Quantum: Atoms, Cavities, and Photons_ (Oxford University Press, New York, 2006).
  * Becker  _et al._ [2002] W. Becker, F. Grasbon, R. Kopold, D. B. Milošević, G. G. Paulus, and H. Walther, Above-threshold ionization: From classical features to quantum effects, Adv. Atom. Molec. Opt. Phys. 48, 35 (2002).
  * Agarwal [1998] G. S. Agarwal, Spectroscopy of strongly coupled atom-cavity systems: A topical review, J. Mod. Opt. 45, 449 (1998).
  * Forn-Díaz  _et al._ [2019] P. Forn-Díaz, L. Lamata, E. Rico, J. Kono, and E. Solano, Ultrastrong coupling regimes of light-matter interaction, Rev. Mod. Phys. 91, 025005 (2019).
  * Kockum  _et al._ [2019] A. F. Kockum, A. Miranowicz, S. De Liberato, S. Savasta, and F. Nori, Ultrastrong coupling between light and matter, Nat. Rev. Phys. 1, 19 (2019).
  * Leroux  _et al._ [2018] C. Leroux, L. C. G. Govia, and A. A. Clerk, Enhancing cavity quantum electrodynamics via antisqueezing: Synthetic ultrastrong coupling, Phys. Rev. Lett. 120, 093602 (2018).
  * Qin  _et al._ [2018] W. Qin, A. Miranowicz, P.-B. Li, X.-Y. Lü, J. You, and F. Nori, Exponentially enhanced light-matter interaction, cooperativities, and steady-state entanglement using parametric amplification, Phys. Rev. Lett. 120, 093601 (2018).
  * Gutiérrez-Jáuregui and Agarwal [2021] R. Gutiérrez-Jáuregui and G. S. Agarwal, Probing the spectrum of the Jaynes-Cummings-Rabi model by its isomorphism to an atom inside a parametric amplifier cavity, Phys. Rev. A 103, 023714 (2021).
  * Agarwal [1984] G. S. Agarwal, Vacuum-field rabi splittings in microwave absorption by rydberg atoms in a cavity, Phys. Rev. Lett. 53, 1732 (1984).
  * Tabuchi  _et al._ [2014] Y. Tabuchi, S. Ishino, T. Ishikawa, R. Yamazaki, K. Usami, and Y. Nakamura, Hybridizing ferromagnetic magnons and microwave photons in the quantum limit, Phys. Rev. Lett. 113, 083603 (2014).
  * Yao  _et al._ [2017] B. Yao, Y. S. Gui, J. W. Rao, S. Kaur, X. S. Chen, W. Lu, Y. Xiao, H. Guo, K. P. Marzlin, and C. M. Hu, Cooperative polariton dynamics in feedback-coupled cavities, Nat. Commun. 8, 1437 (2017).
  * Liu  _et al._ [2015] X. Liu, T. Galfsky, Z. Sun, F. Xia, E.-c. Lin, Y.-H. Lee, S. Kéna-Cohen, and V. M. Menon, Strong light–matter coupling in two-dimensional atomic crystals, Nat. Photonics 9, 30 (2015).
  * Barra-Burillo  _et al._ [2021] M. Barra-Burillo, U. Muniain, S. Catalano, M. Autore, F. Casanova, L. E. Hueso, J. Aizpurua, R. Esteban, and R. Hillenbrand, Microcavity phonon polaritons from the weak to the ultrastrong phonon–photon coupling regime, Nat. Commun. 12, 6206 (2021).
  * Ho  _et al._ [2018] P.-H. Ho, D. B. Farmer, G. S. Tulevski, S.-J. Han, D. M. Bishop, L. M. Gignac, J. Bucchignano, P. Avouris, and A. L. Falk, Intrinsically ultrastrong plasmon–exciton interactions in crystallized films of carbon nanotubes, Proc. Natl. Acad. Sci. U. S. A. 115, 12662 (2018).
  * Baranov  _et al._ [2020] D. G. Baranov, B. Munkhbat, E. Zhukova, A. Bisht, A. Canales, B. Rousseaux, G. Johansson, T. J. Antosiewicz, and T. Shegai, Ultrastrong coupling between nanoparticle plasmons and cavity photons at ambient conditions, Nat. Commun. 11, 2715 (2020).
  * Kéna-Cohen  _et al._ [2013] S. Kéna-Cohen, S. A. Maier, and D. D. C. Bradley, Ultrastrongly coupled exciton–polaritons in metal-clad organic semiconductor microcavities, Adv. Opt. Mater. 1, 827 (2013).
  * Gambino  _et al._ [2014] S. Gambino, M. Mazzeo, A. Genco, O. D. Stefano, S. Savasta, S. Patanè, D. Ballarini, F. Mangione, G. Lerario, D. Sanvitto, and G. Gigli, Exploring light–matter interaction phenomena under ultrastrong coupling regime, ACS Photonics 1, 1042 (2014).
  * Genco  _et al._ [2018] A. Genco, A. Ridolfo, S. Savasta, S. Patanè, G. Gigli, and M. Mazzeo, Bright polariton coumarin-based oleds operating in the ultrastrong coupling regime, Adv. Opt. Mater. 6, 1800364 (2018).
  * Held  _et al._ [2018] M. Held, A. Graf, Y. Zakharko, P. Chao, L. Tropf, M. C. Gather, and J. Zaumseil, Ultrastrong coupling of electrically pumped near-infrared exciton-polaritons in high mobility polymers, Adv. Opt. Mater. 6, 1700962 (2018).
  * Gao  _et al._ [2018] W. Gao, X. Li, M. Bamba, and J. Kono, Continuous transition between weak and ultrastrong coupling through exceptional points in carbon nanotube microcavity exciton–polaritons, Nat. Photonics 12, 362 (2018).
  * Scalari  _et al._ [2012] G. Scalari, C. Maissen, D. Turčinková, D. Hagenmüller, S. D. Liberato, C. Ciuti, C. Reichl, D. Schuh, W. Wegscheider, M. Beck, and J. Faist, Ultrastrong coupling of the cyclotron transition of a 2d electron gas to a thz metamaterial, Science 335, 1323 (2012).
  * Bennett  _et al._ [2016] K. Bennett, M. Kowalewski, and S. Mukamel, Novel photochemistry of molecular polaritons in optical cavities, Faraday Discuss. 194, 259 (2016).
  * Biehs  _et al._ [2018] S.-A. Biehs, C. Xu, and G. S. Agarwal, Strong coupling of collection of emitters on hyperbolic meta-material, J. Opt. 20, 045601 (2018).
  * Thomas  _et al._ [2021] P. A. Thomas, K. S. Menghrajani, and W. L. Barnes, Cavity-free ultrastrong light-matter coupling, J. Phys. Chem. Lett. 12, 6914 (2021).
  * Yeh [1988] P. Yeh,  _Optical Waves in Layered Media_ (J. Wiley and Sons eds., New York, 1988).
  * Hopfield [1958] J. J. Hopfield, Theory of the contribution of excitons to the complex dielectric constant of crystals, Phys. Rev. 112, 1555 (1958).
  * Ciuti  _et al._ [2005] C. Ciuti, G. Bastard, and I. Carusotto, Quantum vacuum properties of the intersubband cavity polariton field, Phys. Rev. B 72, 115303 (2005).
  * Artoni and Birman [1991] M. Artoni and J. L. Birman, Quantum-optical properties of polariton waves, Phys. Rev. B 44, 3736 (1991).
  * Horodecki  _et al._ [2009] R. Horodecki, P. Horodecki, M. Horodecki, and K. Horodecki, Quantum entanglement, Rev. Mod. Phys. 81, 865 (2009).
  * Vidal and Werner [2002] G. Vidal and R. Werner, Computable measure of entanglement, Phys. Rev. A 65, 032314 (2002).
  * De Liberato [2014] S. De Liberato, Light-matter decoupling in the deep strong coupling regime: The breakdown of the purcell effect, Phys. Rev. Lett. 112, 016401 (2014).
  * Quattropani  _et al._ [1986] A. Quattropani, L. C. Andreani, and F. Bassani, Quantum theory of polaritons with spatial dispersion: Exact solutions, Il Nuovo Cimento D 7, 55 (1986).
  * Rendell and Rajagopal [2005] R. Rendell and A. Rajagopal, Entanglement of pure two-mode gaussian states, Phys. Rev. A 72, 012330 (2005).
  * Agarwal [1971] G. S. Agarwal, Entropy, the wigner distribution function, and the approach to equilibrium of a system of coupled harmonic oscillators, Phys. Rev. A 3, 828 (1971).
  * Simon [2000] R. Simon, Peres-horodecki separability criterion for continuous variable systems, Phys. Rev. Lett. 84, 2726 (2000).
  * Agarwal and Chaturvedi [2010] G. S. Agarwal and S. Chaturvedi, How much quantum noise of amplifiers is detrimental to entanglement, Opt. Commun. 283, 839 (2010).


# This is not a Dataset:
A Large Negation Benchmark to Challenge Large Language Models

Iker García-Ferrero1 , Begoña Altuna1 , Javier Álvez2
Itziar Gonzalez-Dios1 , German Rigau1
1 HiTZ Center - Ixa, University of the Basque Country UPV/EHU
2LoRea Group, University of the Basque Country UPV/EHU
<EMAIL_ADDRESS>
<EMAIL_ADDRESS>

###### Abstract

Although large language models (LLMs) have apparently acquired a certain level
of grammatical knowledge and the ability to make generalizations, they fail to
interpret negation, a crucial step in Natural Language Processing. We try to
clarify the reasons for the sub-optimal performance of LLMs understanding
negation. We introduce a large semi-automatically generated dataset of circa
400,000 descriptive sentences about commonsense knowledge that can be true or
false in which negation is present in about 2/3 of the corpus in different
forms. We have used our dataset with the largest available open LLMs in a
zero-shot approach to grasp their generalization and inference capability and
we have also fine-tuned some of the models to assess whether the understanding
of negation can be trained. Our findings show that, while LLMs are proficient
at classifying affirmative sentences, they struggle with negative sentences
and lack a deep understanding of negation, often relying on superficial cues.
Although fine-tuning the models on negative sentences improves their
performance, the lack of generalization in handling negation is persistent,
highlighting the ongoing challenges of LLMs regarding negation understanding
and generalization. The dataset and code are publicly available:
https://github.com/hitz-zentroa/This-is-not-a-Dataset

##  1 Introduction

Large Language Models (LLMs) currently offer state of the art performance in
many Natural Language Processing (NLP) tasks. Apparently, they have acquired
the ability to capture syntactic Baroni (2020) and semantic Furrer et al.
(2021) abstractions. However, recent experiments Kassner and Schütze (2020);
Hossain et al. (2020); Truong et al. (2022) have proven that LLMs fail at
interpreting contexts in which understanding negation is required.

The presence of negation in a sentence reverts the polarity of the proposition
it represents, and thus affects its truth and factuality values. See how the
adverb “never” changes the truth value of the sentences in Figure 1. As a
consequence, understanding negation correctly is crucial for all NLP tasks.
Moreover, understanding negation should help LLMs to grasp how things happen
in reality, boosting NLP tasks that involve commonsense, causality, entailment
and world knowledge.

Figure 1: Affirmative and negative sentences in the dataset.

The reasons for the lower capabilities of LLMs dealing with negation remain
largely unclear, although some point out at the under-representation of
negation in corpora Hossain et al. (2022). In this work, we present a corpus
in which negation is present in around two thirds of the sentences in
different forms. Taking advantage of the relations in WordNet Fellbaum (1998),
we have generated a set of patterns to create descriptive sentences that work
as truth and falsity tests which are then used together with a list of prompts
to measure the sentence understanding of the different LLMs.

The dataset has been used in a series of experiments to test its quality and
coherence. First, we assess the quality of the sentences by human annotators.
Then, to grasp its capacity of generalization and inference, we have used our
dataset to test different configurations of LLMs available in a zero-shot
approach. We have also fine-tuned some of these models to assess whether the
understanding of negation can be learnt. Our initial hypothesis is that if the
dataset is coherently and robustly built we will be able to learn how LLMs
deal with negation.

The contributions of this paper are: i) We introduce the largest negation
probing dataset. This dataset includes affirmative and negative sentences with
and without distractors, incorporating multiple types of relations and
negations. ii) We evaluate a comprehensive set of open LLMs using our dataset
in both zero-shot and fine-tuning scenarios. iii) Our findings demonstrate
that current LLMs, whether in zero-shot settings or after fine-tuning with
examples from our dataset, possess a profound understanding of the
truthfulness of affirmative sentences. However, when confronted with negation,
these models heavily rely on superficial cues instead of effectively
generalizing negation.

##  2 Background

###  2.1 Related Works

Negation is a core operator in logic and in the structuring of the information
in text and it has long been studied for its relevance in natural language
understanding. In the last two decades, works on the analysis and processing
of negation have multiplied. In the pre-generative-model era, most works
centered on negation detection Chapman et al. (2001); Vilares et al. (2015)
and profiling Morante and Daelemans (2012), so the extracted negation
information could be used in downstream tasks.

With the booming of deep-learning architectures that were based on abstract
neural representations of texts, the paradigm shifted and negation was
processed as the rest of the elements appearing in text. It was soon noticed
that systems struggled to correctly process the information when negation was
involved. Such is the case of negation in machine translation Bentivogli et
al. (2016); Tang et al. (2021), information extraction Grivas et al. (2020)
and sentiment analysis Barnes et al. (2021) among others.

It has not been long since the scholar community started to analyse the
reasons for the lack of capability of correctly processing negation. For
example, Jumelet and Hupkes (2018) analysed the negation licensing strategies
to measure neural language model ability to correctly process them.

Chen et al. (2023) assess the ability of LLMs to handle negative commonsense
knowledge. Since most available information exists in a positive and
affirmative form, LLMs fail at dealing with world knowledge when it is
presented in a negative form. They propose a two-task assessment in which LLMs
need to i) answer yes or no to world knowledge questions and ii) generate
commonsense compelling sentences from related keywords. Some recent research
has been directed to building knowledge bases in which negative commonsense is
stored (Arnaout et al., 2022), in order to be reused for commonsense
reasoning.

###  2.2 Negation in English

Negation in language is the representation of the logical operation in which a
the truth value of a proposition is inverted. It is commonly expressed by a
restricted list of negative adverbs (e.g. no, never), pronouns, determiners or
prefixes, that appear in different contexts in the sentence. Pullum et al.
(2002) offer a four axe classification of negation types from which we will
focus on these:

  * •

Verbal vs. non-verbal: the verbal negation marker is associated with the verb
and directly affects it, while the non-verbal couples with objects and
adjuncts.

  * •

Analytic vs. synthetic: analytic negation is represented by markers that only
convey negation. Synthetic negation markers, instead, may have additional
syntactic function (e.g. nothing and none might also be subjects or objects).

  * •

Clausal vs. sub-clausal: clausal negation negates the whole clause that
includes it, and sub-clausal negation only affects a part of the clause.

In Table 1 we present the different types of negations considered in our work.

| Negation
---
type
Example
Verbal |  Agreement is not an appropriate synonym of disagreement in any context.
Non-verbal |  In no context lectures may be part of courses.
Analytic |  No theft is a small replica of a person.
Synthetic |  Mirror is never an appropriate hyponym of reduction.
Clausal |  Kissing is not commonly done by engineers.
Sub-clausal |  Bricks are made of clay in no context
Table 1: Examples of the types of negation.

##  3 Dataset

Pattern | Relation | Templates | Triples | Sentences
---|---|---|---|---
#01 | Synonymy | 21 | 2,996 | 281,624
#02 | Antonymy | 21 | 58 | 8,178
#03 | Synonymy | 24 | 14 | 2,436
#04 | Antonymy | 24 | 58 | 10,092
#05 | Hypernymy | 24 | 634 | 60,864
#06 | Part | 16 | 199 | 11,940
#07 | Substance | 15 | 21 | 1,176
#08 | Member | 17 | 11 | 682
#09 | Agent | 2 | 60 | 240
#10 | Instrument | 7 | 9 | 468
#11 | Result | 27 | 40 | 3,600
Total | - | - | - | 381,300

Table 2: Distribution of sentences by pattern.

###  3.1 Dataset construction

Our benchmark compiles 381,300 artificially generated sentences in standard
English. The sentences, in a definition style (e.g. X is Y, X is part of Y),
have been created based on the knowledge from WordNet and related resources.

The sentences in our dataset are obtained by means of patterns. Each of the 11
patterns (#01–#11) is designed for a particular relation and includes several
different templates of two types: affirmative templates, which are free of
negation; negative templates, which include one of the types of negations
described in Table 1. Using triples on the corresponding relation, these
templates are used to create sentences by instantiation. Since each synset may
include more than one word form in Core WordNet and the proposed templates
include optional and alternative parts, we obtain several sentences from each
couple of template and triple. The controlled application of the different
templates enables us to determine the truth-value of the resulting sentences.
In Appendix D, we describe each pattern in detail.

More specifically, we focus on the WordNet relations synonymy, hypernymy,
antonymy, meronymy (part, member and substance) and the semantic roles agent,
instrument and result provided by Morphosemantic Links Fellbaum et al. (2009).
Among the nouns and verbs compiled in WordNet, we concentrate exclusively on
the ones provided by Core WordNet Boyd-Graber et al. (2006), which is a list
of the most frequently used word senses that includes 3,299 nouns and 1,000
verbs. In this way, we discard words that are less commonly used. Furthermore,
we exclude the triples on synonymy111Synonymy triples are obtained by
reflexivity. and hyponymy that relate Basic Level Concepts (BLCs) Izquierdo et
al. (2007), which may result too general, and we use the mapping from WordNet
to EuroWordNet Top Ontology (TCO) to ignore the triples on the member meronymy
relation and the agent semantic role where the noun synsets are not referring
to animals or persons.

Since WordNet and the considered related resources only provide true knowledge
—that is, all the triples and mappings describe real relations and
connections— we automatically obtain false knowledge from WordNet triples
using distractors, which are randomly selected words that replace the word
senses of a synset.222In Patterns #01 and #02, distractors are synsets when
using glosses. That is, given a WordNet triple that relates two synsets, from
Core WordNet we select a distractor to replace the word senses of one of the
synsets and obtain a distracting triple. Apart from BLCs, for the selection of
suitable distractors we consider the lexicographer files provided by WordNet,
which are 45 syntactic category and logical groupings of word senses, and
WordNet Domains Bentivogli et al. (2004), which consist of a hierarchy of 164
labels that characterize knowledge areas and to which each synset is
connected. In Appendix C, we provide more details about the selection of
distractors.

Next, we illustrate the process of constructing our dataset. In Pattern #06,
we have included the following positive and negative templates that state
semantic correspondences between parts and wholes on the basis of triples of
the form $\langle part,noun_{1},noun_{2}\rangle$:

$\langle noun_{1}$+(e)s$\rangle$ [ are commonly | may be ] part of $\langle noun_{2}$+(e)s$\rangle$.

$\langle noun_{1}$+(e)s$\rangle$ are never part of $\langle
noun_{2}$+(e)s$\rangle$.

The positive template333The expressions enclosed in square brackets are
alternative. yields true sentences when instantiated with true knowledge (i.e.
WordNet triples), while we get false sentences using distracting triples. On
the contrary, the negative one yields sentences with the opposite truth-value.
For example, given the WordNet triple $\langle
part,{\it{bill}_{n}^{10}},{\it{bird}_{n}^{1}}\rangle$, we select human body as
distractor for ${\it{bird}_{n}^{1}}$ and get the distracting triple $\langle
part,{\it{bill}_{n}^{10}},human~{}body\rangle$. Then, by instantiating the
positive template using these two triples, we get the sentences in the first
row of Figure 1 and also:

“Bills may be part of birds.”

The two sentences about birds (i.e. resulting from the WordNet triple) are
labelled with True, while the sentence about human bodies (that is, obtained
from the distracting triple) is labelled with False. Likewise, using the same
two triples for the instantiation of the negative template, we get the
sentences in the second row of Figure 1, which are respectively labelled with
False and True.

In Table 2, we sum up some figures about the proposed dataset. For each
pattern (first column), we provide the corresponding WordNet relation and the
number defined templates, applied WordNet triples and obtained sentences
respectively. It is worth noting that Patterns #01–#04 include both false
positive sentences and true negative sentences obtained from synonymy and
antonymy WordNet triples by means of a dual application of templates.
Furthermore, in the case of antonymy, the truth-value of the resulting
sentences does not depend on whether templates are instantiated using WordNet
or distracting triples. As a consequence, instantiating a template using a
WordNet and a distracting triple yields sentences with opposite truth-value
except for Patterns #02 and #04, where all sentences resulting from the same
template have the same truth-value. For example, given the antonym triple
$\langle ant,{\it{expenditure}_{n}^{1}},{\it{income}_{n}^{1}}\rangle$, we
select wood as distractor for ${\it{expenditure}_{n}^{1}}$ (see Appendix C for
details) and obtain the distracting triple $\langle
ant,wood,{\it{income}_{n}^{1}}\rangle$. Using these two templates, we
instantiate the following negative template included in Pattern #04

$\langle noun_{1}\rangle$ and $\langle noun_{2}\rangle$ are the same thing in
no context.

and obtain two sentences:

“Expenditure and income are the same thing in no context.”

“Wood and income are the same thing in no context.”

Both sentences are true.

###  3.2 Dataset quality assessment

Human Evaluation addresses the validation of the generation process and the
different templates used, that is to say, whether the sentences in the dataset
are grammatical and that overall represent true and false knowledge as
expected. To prove the linguistic quality of the dataset and that the
predictions extracted from WordNet reflect the reality, two native speakers of
English were required to assess a randomly selected sample of 220 sentences
from our dataset. Evaluators were required to answer four questions for each
sentence: i) Is the sentence true or false?, ii) is the sentence grammatically
correct?, iii) is the sentence understandable? and iv) is the sentence
plausible and might be produced by a speaker?

% | A tester | B tester | A $\cap$ B | A $\cup$ B
---|---|---|---|---
T/F prediction | 90.9 | 89.1 | 87.27 | 96.36
Comprehensibility | 91.82 | 100 | 91.82 | 100
Grammaticality | 83.64 | 96.82 | 83.64 | 96.82
Plausibility | 20.45 | 45.91 | 19.55 | 46.82

Table 3: Human evaluation of the quality of the sentences in the dataset.

The answers to these questions have been summarised in Table 3. For the true
and false predictions, we have compared the testers’ answers with the
predictions we generated from the WordNet relations. Circa 90% of the
predictions match with the human testers’ answers. For the quality of the test
sentences, the results show that the sentences in the dataset are mostly
comprehensible to humans even if not all are fully acceptable in English or
they are not likely to be uttered by English speakers (low plausibility).
Namely, we have detected problems with uncountable nouns (1) and lexical
selection (2). Nonetheless, low plausibility might be an interesting asset for
our experiments as employing non-frequent sentences may help to reduce the
effect of the reliance on lexical co-occurrences models have.

(1) |  “A letter is commonly part of a mail.”
---|---
(2) |  “Officers are not members of laws in any context.”

In what refers the quality of the knowledge encoded in the dataset, we have
observed that over 98% of the sentences with distractors in the human test set
represent actual knowledge. We can thus consider that the distractor selection
mechanism is robust enough.

##  4 Experimental Setup

In this section, we define the evaluation protocol we use to measure the
performance of Language Learning Models (LLMs) on our dataset.

###  4.1 Models

We evaluate a diverse set of LLMs, ranging in size from 7 billion parameters
up to 65 billion parameters. Our evaluation includes Foundation Models, along
with versions that have undergone additional instruction-tuning and/or have
been fine-tuned for conversation. We do not consider closed models where the
data used for pretraining or even the model architecture is unknown, as
drawing meaningful conclusions for such systems is not possible. We evaluate
the following models: the 12 billion parameter T5 Raffel et al. (2020)
encoder-decoder language model, as well as FLAN-T5 Chung et al. (2022), an
enhanced version of T5 that has been fine-tuned in a mixture of tasks; LLaMA
Touvron et al. (2023) decoder-only language models with parameter sizes
ranging from 7 billion to 65 billion; LLaMA models that have been fine-tuned
for specific tasks, including Vicuna v1.1 Chiang et al. (2023), which has
undergone additional fine-tuning as a chat-assistant, and WizardLM Xu et al.
(2023), which has been fine-tuned for following instructions; Pythia Biderman
et al. (2023) decoder-only 12 billion parameter model; the instruction-tuning
model Dolly Conover et al. (2023); and finally we evaluate Falcon Almazrouei
et al. (2023) 7 and 40 billion parameter models which are decoder-only models
including the instruction-following fine-tuned versions. We also evaluate
other open LLMs; the full model list can be found in Appendix A.

###  4.2 Task Formulation

We evaluate each sentence in the dataset individually as a binary task in
which the model must generate either True or False tokens. Following Scheurer
et al. (2023), given the prompt $pt$ we compute the answer $A$ as follows:

$A=\left\\{\begin{matrix}True&if\frac{p(True|pt)}{p(True|pt)+p(False|pt)}>0.5\\\
\\\ False&otherwise\end{matrix}\right.$

We use the following prompt as input for the models:

Is the following statement True or False? {sentence}.

We found that models that have undergone a fine-tuning for conversation tend
to generate an explanation instead of answering True or or False. We use a
slightly modified prompt that improves the results: Is the following statement
True or False? Answer only True or False. {sentence}. Models that have been
fine-tuned as dialogue systems utilize different prompts to represent a
conversation, such as using markers like “<bot>” and “<human>”, or custom
system initial prompts. In order to accommodate these models, we format the
input according to the recommendations provided by the authors. Implementation
details of fine-tuning and inference are available in Appendix B.

###  4.3 Metrics

In our dataset, we utilize two primary metrics for evaluating LLMs:

#### Accuracy

This metric is computed using the formula $acc=(TP+TN)/(TP+TN+FP+FN)$. We
evaluate the overall accuracy at the sentence level for all the sentences in
our dataset. Additionally, we analyze the overall accuracy of different
sentence types: Accuracy in Affirmative sentences, Negative sentences,
Affirmative sentences with a distractor and Negative sentences that include a
distractor.

Model name Model Type Coherence Accuracy All W/o Distractor W/ Distractor All
W/o Distractor W/ Distractor Affirmation Negation Affirmation Negation Random
0.5 0.9 0.8 50.0 50.2 50.1 50.0 49.9 LLaMA13B Foundation 0.0 0.2 0.3 50.1 85.8
12.2 10.6 90.2 LLaMA30B Foundation 0.1 0.3 0.2 52.4 84.7 29.5 30.2 68.8
LLaMA65B Foundation 0.0 0.0 0.0 50.3 96.3 3.1 1.3 99.3 Vicuna13B Dialogue 0.2
8.8 0.6 57.8 83.1 85.1 78.0 2.6 WizardLM30B Instruction 0.0 6.0 0.1 57.3 53.6
95.7 88.8 2.0 Pythia12B Foundation 0.0 0.1 0.0 50.1 93.8 15.2 4.0 86.7
Dolly12B Instruction 0.0 0.3 0.2 50.2 72.0 73.3 33.4 25.1 T5-xxl Foundation
0.0 0.0 0.0 50.3 96.6 2.8 0.4 99.8 Flan-T5-xxl Instruction 0.9 46.4 1.2 66.1
86.1 96.1 94.6 6.2 Falcon40b Foundation 0.1 0.1 0.2 49.7 90.9 13.9 11.6 83.3
Falcon40b-instruct Instruction 0.1 1.5 0.2 54.7 64.3 76.8 71.4 16.6

Table 4: Zero-shot performance of various LLMs in our dataset. The best
results are highlighted in bold, and scores that surpass the Random baseline
accuracy are underlined.

#### Coherence

This metric aims to decouple the real-world and commonsense knowledge of the
model from the understanding of negative sentences. We compute two coherence
scores: one for the sentences without distractors (“Bills are commonly part of
birds" and “Bills are never part of birds") and another for the sentences with
distractors (“Bills are commonly part of human bodies." and "Bills are never
part of human bodies."). Answers are deemed coherent if the affirmative and
negative sentences have opposite labels, regardless of whether the answer is
correct or incorrect. However, if the model predicts the same label for both
the affirmative and negative sentences, we consider the answer incoherent. To
illustrate this metric, consider the sentence pair “Bills are commonly part of
birds" and “Bills are never part of birds". Both the answers “True/False" and
“False/True" are considered coherent, whereas “True/True" and “False/False"
are incoherent.

Moreover, we calculate the overall coherence: this happens when all the
statements with and without distractors are coherent and correctly or all
incorrectly classified. Referring to the example in Figure 1, we would deem
the set of statements as overall coherent if the sentences with and without
distractors are coherent and all the answers are either correct or all of them
are incorrect. In the case of antonymy relations (Patterns #02 and #04), both
the distractor-carrying and non distractor-carrying sentences carry the same
label, so we evaluate the overall coherence accordingly. It is important to
note that, for the sake of simplicity, the example only contains two
sentences, but coherence is actually determined at the triple level. Triples
can comprise between 2 to 27 templates. So, for a triple to be deemed
coherent, responses to all the templates within it must be coherent.
Therefore, this is a very challenging metric.

By examining coherence in these contexts, we gain insights into the models’
ability to understand the negation, even if the models do not have the real-
world knowledge to correctly label the sentences.

##  5 Do LLMs understand negation?

In this section, we assess the performance of the LLMs in section 4.1 in our
dataset. The evaluation is conducted in a zero-shot setting, meaning that we
evaluate the models without any fine-tuning. The results of this evaluation
are presented in Table 4. Foundation models, which are trained on large
amounts of unlabeled data, demonstrate an All True behavior. They accurately
label as True the majority of affirmative sentences and negative sentences
with a distractor, which are True with the exception of the Antonymy patterns,
that form approximately 5% of the total sentences. However, these models
struggle to classify negative sentences and affirmative sentences with
opposite labels. Their performance in these falls significantly below the
random baseline exhibiting a total lack of coherence by the models.

W/o Distractor Flan-T5-xxl Affirmation Negation #01 Synonymy 91.19 98.04 #02
Antonymy 96.36 25.62 #03 Synonymy 49.76 98.47 #04 Antonymy 82.07 21.92
Vicuna13B #01 Synonymy 88.69 84.88 #02 Antonymy 71.65 4.64 #03 Synonymy 57.86
90.05 #04 Antonymy 75.8 12.81

Table 5: Accuracy of Flan-T5-xxl and Vicuna13B in the Synonymy and Antonymy
patterns. We evaluate the models in affirmative and negative sentences without
distractors. Scores that surpass the Random baseline are indicated with
underline.

Models that have undergone dialogue or instruction tuning, particularly Vicuna
and FlanT5, demonstrate higher accuracy, instead. These models achieve a very
high accuracy in sentences without a distractor. Specifically, Flan-T5 shows
coherent answers for 46% of the triples. It is to be noted that this is a
challenging metric, as a triple may be use to build up to 27 templates, and
all of them must be coherent for the triple to be considered coherent.

Model name Coherence Accuracy All W/o Distractor W/ Distractor All W/o
Distractor W/ Distractor Affirmation Negation Affirmation Negation Flan-T5-xxl
51.8 55.4 92.9 94.1 96.5 86.7 96.1 98.0 Vicuna13B 81.2 86.4 94.2 95.7 92.7
94.4 98.1 97.2

Table 6: Performance of Vicuna13B and Flan-T5-xxl after fine-tuning in our
dataset. The best results are highlighted in bold, and scores that surpass the
Random baseline accuracy are underlined.

However, these models fail to correctly label negative sentences with a
distractor. We further analyze the performance of Flan-T5 and Vicuna in
negative sentences, focusing on the Synonymy and Antonymy patterns. In Pattern
#01 and #02, as well as Pattern #03 and #04, the templates are opposite to
each other, as explained in Subsection 3.1. Table 5 presents the performance
of Flan-T5 and Vicuna in handling these patterns. Interestingly, both models
achieve good results in negative sentences from the Synonymy patterns (labeled
as False) but struggle with the negative sentences from the Antonymy patterns
(labeled as True). This, along with their poor performance in negative
sentences with a distractor (which are expected to be True, but models predict
the label False), confirms that the models are heavily biased to always
predict the label False in the presence of negation, regardless of the actual
meaning of the sentence. This behavior suggests that the models lack a deep
understanding of negation, and that they tend to rely on superficial cues
rather than comprehending the true meaning conveyed by the negative sentences.

Despite the poor performance of the models in negative sentences, it is
important to note that they demonstrate the ability to correctly label
affirmative sentences, both with and without distractors. This demonstrates
that the models have a deep understanding of truth and falsehood. Models’
struggles primarily result from the presence of negation rather than a lack of
comprehension or real-world knowledge.

##  6 Exposure to negation does not solve the problem

Understanding whether LLMs would understand negation if a sufficient number of
negative sentences were present in the pretraining corpora is crucial for
improving their reasoning capabilities and addressing the limitations
associated with negative knowledge. However, due to the lack of sufficiently
large datasets containing negative knowledge, this hypothesis has not been
extensively explored. In contrast, our dataset is substantial enough to be
split into training, development, and test sets. To investigate whether LLMs
can learn to reason over negative knowledge given enough negated data, we
split the dataset at the triple level, ensuring that all sentences within a
triple are assigned to the same split to ensure no data contamination. Our
training dataset consists of 268,505 sentences from 2,876 triples, the
development dataset includes 2,514 sentences from 244 triples, and the test
dataset contains 90,281 sentences from 980 triples.

We finetune Flan-T5 and Vicuna on our dataset; the results are listed in Table
6. The impact of fine-tuning is remarkable, as it completely transforms the
models’ performance compared to their zero-shot counterparts. Both Flan-T5 and
Vicuna exhibit higher accuracy than human annotators and achieve a notably
high level of coherence. However, are the models truly learning about
negation, or are they just exploiting patterns in the data? We conduct
experiments to asses this.

Verbal Non-Verbal Analytic synthetic clausal subclausal Affirmation All 96.0
95.7 95.8 96.0 96.0 95.7 95.5 All Affirmations 6.2 6.8 6.8 5.9 6.1 6.8 95.7
All Negated 96.1 95.8 95.8 96.3 96.1 95.8 4.5 Affirmations + Verbal 95.5 79.5
81.9 95.6 95.5 79.5 95.4 Affirmations + Non-Verbal 94.9 95.6 95.2 96.1 94.9
95.6 95.8 Affirmations + Analytic 96.1 95.6 95.6 96.1 96.0 95.6 95.9
Affirmations + synthetic 94.8 44.5 51.8 96.0 94.9 44.5 95.6 Affirmations +
clausal 95.8 34.6 43.9 96.0 96.1 34.6 95.8 Affirmations + subclausal 95.1 95.7
95.3 96.2 95.1 95.7 96.0

Table 7: Accuracy of Vicuna13B after fine-tuning with different types and
amount of negative knowledge. The best results are highlighted in bold, and
scores that surpass the Random baseline accuracy are indicated with underline.

First, we train Vicuna, the best performing model, using varying amounts and
types of negative knowledge. We conduct separate fine-tuning experiments using
all the affirmative sentences and all the negative sentences from the dataset,
resulting in two distinct models. The results of this training are presented
in Table 7. Training the model exclusively with affirmative sentences yields a
high accuracy in the affirmative test sentences, but it labels incorrectly
nearly all the negative sentences. Conversely, when trained solely with
negative sentences, the model deals successfully with the negative sentences
but struggles with the affirmative sentences. Despite being exposed to
extensive real-world knowledge from WordNet, the models exhibit a significant
failure in comprehending negation. They consistently overlook the presence of
it and generate identical outputs for both affirmative and negative sentences.
We also fine-tune models using various combinations of affirmative sentences
and different types of negations. We observe that models trained with
synthetic and clausal negations struggle to accurately classify non-verbal,
analytic, and sub-clausal sentences. This suggests that while the models show
proficiency in understanding and reasoning with certain types of negations,
they face challenges in comprehending and correctly responding to other forms
of negations that they have not seen in the fine-tuning step.

#01#02#03#04#05#06#07#08#09#10#11#01 Synonymy#02 Antonymy#03 Synonymy#04
Antonymy#05 Hypernymy#06 Part#07 Substance#08 Member#09 Agent#10 Instrument#11
ResultAll10076907089889787899485501006593505050505050508679100907482888086938055986710050515351615451916392671008798959286928772927978100948685947881767687738310073868674717976876576871007467625946636162626558100626477667784747288708710068857984957784929088919197949810093949096918895

Figure 2: Evaluation of Vicuna13B accuracy when trained on one pattern (rows)
and evaluated on the others (columns).

We also fine-tune Vicuna13B with each of the 11 patterns in our dataset
independently, and we evaluate its performance on the other patterns. Figure 2
shows the overall accuracy scores. The results reveal that training the model
with one pattern does not facilitate any successful generalization across all
other patterns. Notably, as discussed in Section 3.1, the labels for
affirmative and negative sentences from the Antonymy patterns are opposite to
those from the remaining patterns. The failure of models trained in other
patterns to label the Antonymy patterns, as well as the failure of models
trained in the Antonymy patterns to label other patterns, suggest that the
models are relying on repetitive data structures that are not transferable to
different patterns, rather than truly understanding the concept of negation.
While exposure to negation may contribute to achieving favorable results
within a specific dataset, it does not lead to a generalization on negation by
the models. Negation continues to pose a significant challenge in the field of
Natural Language Processing and remains an unsolved problem, requiring further
research and development.

##  7 Conclusion

Current LLMs are typically trained using next token or mask token prediction
objectives, which have proven effective for various NLP tasks. However, it
remains an open issue understanding how certainly a model models negation.
Negation tokens, which intermittently appear in sentences, hold little
predictive importance for other tokens in the sentence. As a result, there is
limited negation signal during language modeling training. Previous research
has touched upon this issue but was limited by small manually generated
datasets. In contrast, our study introduces the largest dataset to date
comprising negative sentences. This comprehensive dataset includes affirmative
and negative sentences with and without distractors, incorporating multiple
types of relations and negations, which help to encode the underlying
mechanisms for negation understanding. Through our analysis, we reveal that
current LLMs, both in zero-shot settings and when fine-tuned with examples
from our dataset, exhibit a profound understanding of the truthfulness of
affirmative sentences. However, when it comes to negation, these models
heavily rely on superficial cues instead of generalizing negation and these
superficial cues are not transferable across different negative sentences.

Negation remains a persistent and unsolved challenge in the field of NLP,
demanding further research to develop systems capable of effectively handling
it. Our dataset holds the potential to significantly contribute towards
achieving this objective. In our future work, we plan to explore advanced
reasoning paradigms, such as Chain-of-Thought, with the aim of enhancing model
performance on our dataset. However, dealing properly with negation may also
require novel neural architectures.

## Limitations

The dataset contains a limited number of low-quality sentences, which are
discussed in Section 3.2. Through manual evaluation, we find that over 96% of
the sentences are considered understandable and grammatically correct by at
least one human annotator and their prediction of whether the sentence is true
or false matches the label in the dataset. Hence, the presence of low-quality
sentences does not have a significant impact on the evaluation results. On the
other side, a majority of sentences in the dataset are not plausible and
unlikely to be spoken by English speakers. This feature provides a benefit by
ensuring that the sentences are improbable to be found in the LLM training
corpus, thereby it prevents models from relying solely on memorization to
generate accurate responses.

All experiments were conducted by querying the models for the probability of
True and False tokens. We did not explore more complex reasoning prompts, such
as Chain of Thought. However, as explained in Section 7, we believe that
models should be able to comprehend negation and provide accurate answers
across diverse settings. Complex reasoning paradigms may not always be
feasible in real-world applications, specially when models are used by non-NLP
professionals.

Finally, the performance of models in our dataset is not solely determined by
their capability to understand negation. Factors such as performance in
question answering and prompting tasks, as well as their understanding of
real-world knowledge, play a crucial role. However, models like Vicuna13B and
Flan-T5-xxl showcase remarkable proficiency in correctly responding to
affirmative sentences, indicating that their struggles primarily arise from
the presence of negation. Additionally, we introduce a coherence metric that
considers whether the model changes its prediction in the presence of
negation, rather than solely focusing on the accuracy of the model’s answer to
the question.

## Ethics Statement

The dataset has been created through the English WordNet relations, so it
reflects most of the “western” knowledge and might fall short in including
concepts of non-English speaking communities. The generated triples from
WordNet may include offensive or biased sentences. This can be caused by
inherited biases from WordNet, or it can be caused unintentionally during the
random sampling of synsets.

## Acknowledgements

We would like to thank Jeremy Barnes and Aritz Farwell for willingly offering
themselves to conduct the dataset quality assessment experiments. Begoña
Altuna is supported by the Basque Government postdoctoral grant POS 2022 2
0024. Iker García-Ferrero is supported by a doctoral grant from the Basque
Government (PRE_2022_2_0208). This work has also been partially supported by
HiTZ center and the Basque Government (Research group funding IT-1805-22). We
also acknowledge the funding from the following projects:
(i) Antidote project funded by (PCI2020-120717-2)
MCIN/AEI/10.13039/501100011033 and by “ERDF A way of making Europe"
(ii) MOTION (PID2020-112581GB-C22) supported by the Ministry of Science and
Innovation of the Spanish Government
(iii) The Basque Project LoRea (UPV/EHU GIU21/044).
(iv) DeepKnowledge (PID2021-127777OB-C21) and ERDF A way of making Europe
(v) DeepR3 (TED2021-130295B-C31) and European Union NextGeneration EU/PRTR.

## References

  * Almazrouei et al. (2023) Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz Alshamsi, Alessandro Cappelli, Ruxandra Cojocaru, Merouane Debbah, Etienne Goffinet, Daniel Heslow, Julien Launay, Quentin Malartic, Badreddine Noune, Baptiste Pannier, and Guilherme Penedo. 2023.  Falcon-40B: an open large language model with state-of-the-art performance.
  * Arnaout et al. (2022) Hiba Arnaout, Simon Razniewski, Gerhard Weikum, and Jeff Z. Pan. 2022.  UnCommonSense: Informative Negative Knowledge about Everyday Concepts.  In _Proceedings of the 31st ACM International Conference on Information & Knowledge Management_, CIKM ’22, pages 37––46, New York, NY, USA. Association for Computing Machinery.
  * Barnes et al. (2021) Jeremy Barnes, Erik Velldal, and Lilja Øvrelid. 2021.  Improving sentiment analysis with multi-task learning of negation.  _Natural Language Engineering_ , 27(2):249–269.
  * Baroni (2020) Marco Baroni. 2020.  Linguistic generalization and compositionality in modern artificial neural networks.  _Philosophical Transactions of the Royal Society B: Biological Sciences_ , 375(1791):20190307.
  * Bentivogli et al. (2016) Luisa Bentivogli, Arianna Bisazza, Mauro Cettolo, and Marcello Federico. 2016.  Neural versus Phrase-Based Machine Translation Quality: a Case Study.  In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_ , pages 257–267, Austin, Texas. Association for Computational Linguistics.
  * Bentivogli et al. (2004) Luisa Bentivogli, Pamela Forner, Bernardo Magnini, and Emanuele Pianta. 2004.  Revising the Wordnet Domains Hierarchy: semantics, coverage and balancing.  In _Proc. of the Workshop on Multilingual Linguistic Resources_ , pages 94–101, Geneva, Switzerland. COLING.
  * Biderman et al. (2023) Stella Biderman, Hailey Schoelkopf, Quentin Anthony, Herbie Bradley, Kyle O’Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar van der Wal. 2023.  Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling.  _CoRR_ , abs/2304.01373.
  * Boyd-Graber et al. (2006) Jordan Boyd-Graber, Christiane Fellbaum, Daniel Osherson, and Robert Schapire. 2006\.  Adding dense, weighted connections to WordNet.  In _Proceedings of the third international WordNet conference_ , pages 29–36.
  * Chapman et al. (2001) Wendy W. Chapman, Will Bridewell, Paul Hanbury, Gregory F. Cooper, and Bruce G. Buchanan. 2001.  A Simple Algorithm for Identifying Negated Findings and Diseases in Discharge Summaries.  _Journal of Biomedical Informatics_ , 34(5):301–310.
  * Chen et al. (2023) Jiangjie Chen, Wei Shi, Ziquan Fu, Sijie Cheng, Lei Li, and Yanghua Xiao. 2023.  Say What You Mean! Large Language Models Speak Too Positively about Negative Commonsense Knowledge.
  * Chiang et al. (2023) Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. 2023.  Vicuna: An Open-Source Chatbot Impressing GPT-4 with 90%* ChatGPT Quality.
  * Chung et al. (2022) Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Y. Zhao, Yanping Huang, Andrew M. Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoc V. Le, and Jason Wei. 2022.  Scaling instruction-finetuned language models.  _CoRR_ , abs/2210.11416.
  * Conover et al. (2023) Mike Conover, Matt Hayes, Ankit Mathur, Xiangrui Meng, Jianwei Xie, Jun Wan, Sam Shah, Ali Ghodsi, Patrick Wendell, Matei Zaharia, and Reynold Xin. 2023.  Free Dolly: Introducing the World’s First Truly Open Instruction-Tuned LLM.
  * Dettmers et al. (2022) Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. 2022.  LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale.  _CoRR_ , abs/2208.07339.
  * Dettmers et al. (2023) Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2023.  QLoRA: Efficient Finetuning of Quantized LLMs.  _CoRR_ , abs/2305.14314.
  * Fellbaum (1998) C. Fellbaum, editor. 1998.  _WordNet: An Electronic Lexical Database_.  MIT Press.
  * Fellbaum et al. (2009) Christiane Fellbaum, Anne Osherson, and Peter E. Clark. 2009.  Putting semantics into WordNet’s “Morphosemantic” Links.  In Zygmunt Vetulani and Hans Uszkoreit, editors, _Human Language Technology. Challenges of the Information Society_ , LNAI 5603, pages 350–358. Springer.
  * Furrer et al. (2021) Daniel Furrer, Marc van Zee, Nathan Scales, and Nathanael Schärli. 2021.  Compositional Generalization in Semantic Parsing: Pre-training vs. Specialized Architectures.
  * Grivas et al. (2020) Andreas Grivas, Beatrice Alex, Claire Grover, Richard Tobin, and William Whiteley. 2020.  Not a cute stroke: Analysis of Rule- and Neural Network-based Information Extraction Systems for Brain Radiology Reports.  In _Proceedings of the 11th International Workshop on Health Text Mining and Information Analysis_ , pages 24–37, Online. Association for Computational Linguistics.
  * Hossain et al. (2022) Md Mosharaf Hossain, Dhivya Chinnappa, and Eduardo Blanco. 2022.  An Analysis of Negation in Natural Language Understanding Corpora.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pages 716–723, Dublin, Ireland. Association for Computational Linguistics.
  * Hossain et al. (2020) Md Mosharaf Hossain, Venelin Kovatchev, Pranoy Dutta, Tiffany Kao, Elizabeth Wei, and Eduardo Blanco. 2020.  An Analysis of Natural Language Inference Benchmarks through the Lens of Negation.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 9106–9118, Online. Association for Computational Linguistics.
  * Hu et al. (2022) Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022.  LoRA: Low-Rank Adaptation of Large Language Models.  In _The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022_. OpenReview.net.
  * Izquierdo et al. (2007) Rubén Izquierdo, Armando Suárez, and German Rigau. 2007.  Exploring the Automatic Selection of Basic Level Concepts.  In _Proc. of the Int. Conf. on Recent Advances on Natural Language Processing (RANLP’07)_ , volume 7.
  * Jumelet and Hupkes (2018) Jaap Jumelet and Dieuwke Hupkes. 2018.  Do Language Models Understand Anything? On the Ability of LSTMs to Understand Negative Polarity Items.  In _Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP_ , pages 222–231, Brussels, Belgium. Association for Computational Linguistics.
  * Kassner and Schütze (2020) Nora Kassner and Hinrich Schütze. 2020.  Negated and Misprimed Probes for Pretrained Language Models: Birds Can Talk, But Cannot Fly.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 7811–7818, Online. Association for Computational Linguistics.
  * Morante and Daelemans (2012) Roser Morante and Walter Daelemans. 2012.  ConanDoyle-neg: Annotation of negation cues and their scope in Conan Doyle stories.  In _Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC’12)_ , pages 1563–1568, Istanbul, Turkey. European Language Resources Association (ELRA).
  * Pullum et al. (2002) Geoffrey K. Pullum, Rodney Huddleston, Rodney Huddleston, and Geoffrey K. Pullum. 2002.  _Negation_, pages 785–850. Cambridge University Press.
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020.  Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer.  _J. Mach. Learn. Res._ , 21:140:1–140:67.
  * Scheurer et al. (2023) Jérémy Scheurer, Jon Ander Campos, Tomasz Korbak, Jun Shern Chan, Angelica Chen, Kyunghyun Cho, and Ethan Perez. 2023.  Training Language Models with Language Feedback at Scale.  _CoRR_ , abs/2303.16755.
  * Tang et al. (2021) Gongbo Tang, Philipp Rönchen, Rico Sennrich, and Joakim Nivre. 2021.  Revisiting Negation in Neural Machine Translation.  _Transactions of the Association for Computational Linguistics_ , 9:740–755.
  * Touvron et al. (2023) Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurélien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023.  LLaMA: Open and Efficient Foundation Language Models.  _CoRR_ , abs/2302.13971.
  * Truong et al. (2022) Thinh Hung Truong, Yulia Otmakhova, Timothy Baldwin, Trevor Cohn, Jey Han Lau, and Karin Verspoor. 2022.  Not another Negation Benchmark: The NaN-NLI Test Suite for Sub-clausal Negation.  In _Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 883–894, Online only. Association for Computational Linguistics.
  * Vilares et al. (2015) David Vilares, Miguel A. Alonso, and Carlos Gómez-Rodríguez. 2015.  On the usefulness of lexical and syntactic processing in polarity classification of Twitter messages.  _Journal of the Association for Information Science and Technology_ , 66(9):1799–1816.
  * Xu et al. (2023) Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. 2023.  WizardLM: Empowering Large Language Models to Follow Complex Instructions.  _CoRR_ , abs/2304.12244.

##  Appendix A Extended zero-shot results

Apart from the models presented in Section 5, as anticipated, we have also
tested the performance in the task of the following models: Koala
444https://bair.berkeley.edu/blog/2023/04/03/koala/, which is a LLaMA model
that has been fine-tuned for dialogue; Open-Assistant (oasst-sft-1-pythia-12b)
555https://huggingface.co/OpenAssistant/oasst-sft-1-pythia-12b, which is a
Pythia model fine-tuned on human generated assistant conversations; and INCITE
666https://www.together.xyz/blog/redpajama-7b 7 billion foundation model along
with the two models that have been further fine-tuned by the authors in the
instruction-tuning paradigm and chat conversation. Table 8 shows the extended
evaluation results.

Model name Model Type Coherence Accuracy All W/o Distractor W/ Distractor All
W/o Distractor W/ Distractor Affirmation Negation Affirmation Negation Random
- 0.5 0.9 0.8 50.0 50.2 50.1 50.0 49.9 LLaMA7B Foundation 0.0 0.2 0.0 50.4
95.7 11.8 2.3 91.0 LLaMA13B Foundation 0.0 0.2 0.3 50.1 85.8 12.2 10.6 90.2
LLaMA30B Foundation 0.1 0.3 0.2 52.4 84.7 29.5 30.2 68.8 LLaMA65B Foundation
0.0 0.0 0.0 50.3 96.3 3.1 1.3 99.3 [3pt/6pt] Vicuna7B Dialogue 0.0 0.2 0.0
52.4 18.1 96.9 98.9 0.3 Vicuna13B Dialogue 0.2 8.8 0.6 57.8 83.1 85.1 78.0 2.6
[3pt/6pt] Koala7B Dialogue 0.0 0.0 0.0 50.1 5.4 97.2 99.5 0.3 Koala13B
Dialogue 0.0 0.7 0.0 52.8 20.4 97.1 98.8 0.3 [3pt/6pt] WizardLM7B Instruction
0.2 0.2 0.7 51.0 89.6 27.3 14.2 73.8 WizardLM13B Instruction 0.0 4.3 0.3 57.4
68.7 86.2 87.5 2.9 WizardLM30B Instruction 0.0 6.0 0.1 57.3 53.6 95.7 88.8 2.0
[3pt/6pt] WizardLM7B-uncensored Instruction 0.0 1.0 0.1 53.7 63.4 78.4 63.3
17.5 WizardLM13B-uncensored Instruction 0.0 0.3 0.0 50.5 7.6 97.0 99.3 0.3
WizardLM30B-uncensored Instruction 0.0 0.0 0.0 49.8 3.6 97.2 99.6 0.2
Pythia12B Foundation 0.0 0.1 0.0 50.1 93.8 15.2 4.0 86.7 oasst-pythia12B
DIalogue 0.0 0.0 0.0 49.8 4.7 97.1 98.9 0.3 Dolly12B Instruction 0.0 0.3 0.2
50.2 72.0 73.3 33.4 25.1 T5-xxl Foundation 0.0 0.0 0.0 50.3 96.6 2.8 0.4 99.8
Flan-T5-xxl Instruction 0.9 46.4 1.2 66.1 86.1 96.1 94.6 6.2 Falcon7b
Foundation 0.0 0.0 0.0 50.3 96.6 2.9 0.4 99.7 Falcon7b-instruct Instruction
0.0 0.3 0.4 50.1 82.7 20.9 21.2 77.0 Falcon40b Foundation 0.1 0.1 0.2 49.7
90.9 13.9 11.6 83.3 Falcon40b-instruct Instruction 0.1 1.5 0.2 54.7 64.3 76.8
71.4 16.6 INCITE7B-Base Foundation 0.1 0.3 0.1 50.3 82.6 17.0 16.4 84.5
INCITE7B-Instruct Instruction 0.2 0.4 0.4 50.5 73.4 22.5 26.9 78.7
INCITE7B-Chat Dialogue 0.1 0.3 0.2 50.0 19.8 89.4 88.0 6.0

Table 8: Zero-shot performance of various LLMs in our dataset. The best
results are highlighted in bold, and scores that surpass the Random baseline
accuracy are indicated with underline.

##  Appendix B Efficient inference and training

Model name Precision Coherence Accuracy All W/o Distractor W/ Distractor All
W/o Distractor W/ Distractor Affirmation Negation Affirmation Negation
Vicuna13B 8-Bits 0.2 8.8 0.6 57.8 83.1 85.1 78.0 2.6 Vicuna13B Float16 0.4
10.1 1.1 57.8 84.4 85.8 74.7 3.2

Table 9: Zero-shot performance of Vicuna using 8-Bits quantification and the
orifinal float16 weights. The best results are highlighted in bold, and scores
that surpass the Random baseline are indicated with underline.

To facilitate inference of the models on a single GPU, we employed 8-bit
quantization Dettmers et al. (2022) for all of them. We conducted preliminary
experiments with Vicuna 13 billion parameter model. Results are shown in Table
9. While the running cost of the models significantly decreases, we observed
only minimal performance degradation in the quantified versions.

For the training process, we utilized Low-Rank Adaptation (LoRA) Hu et al.
(2022). This approach involves freezing the weights of the pre-trained model
and introducing trainable rank decomposition matrices into each layer. The
frozen model weights are quantized into 8 bits, while the LoRA trainable
weights remain in 16 bits Dettmers et al. (2023). By adopting this efficient
training paradigm, we were able to train LLMs with up to 13 billion parameters
on a single GPU within a reasonable timeframe.

We perform all our experiments using a single NVIDIA A100 GPU with 80GB
memory. The machine used has two AMD EPYC 7513 32-Core Processors and 1024GB
of RAM.

##  Appendix C Dataset construction: selection of distractors

Pattern | W/o Distractor | W/ Distractor
---|---|---
Affirmation | Negation | Affirmation | Negation
#01 | True | False | False | True
#02 | False | True | False | True
#03 | True | False | False | True
#04 | False | True | False | True
#05 | True | False | False | True
#06 | True | False | False | True
#07 | True | False | False | True
#08 | True | False | False | True
#09 | True | False | False | True
#10 | True | False | False | True
#11 | True | False | False | True

Table 10: Truth-value of resulting sentences by pattern.

The automatic creation of false knowledge (distracting triples) on the basis
of WordNet triples requires the use of distractors. In general, distractors
are randomly selected words that replace the word senses of a synset in a
given triple, although the whole synset is replaced when using glosses in
templates (Patterns #01 and #02). For each WordNet triple, we use a single
distracting triple except for Patterns #02, #03 and #04, where we use two
distracting triples obtained by using one distractor per synset. Apart from
BLCs, for the selection of suitable distractors we consider the lexicographer
files provided by WordNet, which are 45 syntactic category and logical
groupings of word senses, and WordNet Domains, which consist of a hierarchy of
164 labels that characterize knowledge areas and to which each synset is
connected. More concretely:

  * •

Words (synsets in the case of Pattern #01 and #02 when using glosses)
belonging to some BLCs cannot be distractors to ensure that selected words
(synsets) are not too general.

  * •

The combined lexicographer file and WordNet Domain annotation of any word
sense of the given synset and of any synset where the distractor occurs (of
the synset in the case of Pattern #01 and #02 when using glosses) have to be
different.

In general, these restrictions ensure that the resulting false triples do not
encode true knowledge. The probability of choosing a synset as distractor is
directly proportional to the logarithm of its frequency.

For example, wood can be used as distractor of ${\it{expenditure}_{n}^{1}}$
because wood belongs to the lexicographer files noun.substance (nouns denoting
cognitive processes and contents), noun.group (nouns denoting groupings of
people or objects) and noun.artifact (nouns denoting man-made objects) while
expenditure belongs to noun.possession (nouns denoting possession and transfer
of possession) and noun.act (nouns denoting acts or actions). Therefore, we
get the distracting triple $\langle ant,wood,{\it{income}_{n}^{1}}\rangle$
from $\langle ant,{\it{expenditure}_{n}^{1}},{\it{income}_{n}^{1}}\rangle$. On
the contrary, the word registration cannot be used as distractor of
${\it{expenditure}_{n}^{1}}$ as both words belong to the lexicographic file
noun.act and the synsets ${\it{expenditure}_{n}^{2}}$ and
${\it{registration}_{n}^{1}}$ belong to the economy domain.

##  Appendix D Dataset Description

Pattern #01: synonymy (gloss)

  * Affirmative template:

A/An $\langle word\rangle$ is (commonly) $\langle gloss\rangle$.

    * Sentences:

A flight is commonly a scheduled trip by plane between designated airports. | True
---|---
A flight is a scheduled trip by plane between designated airports. | True
A flight is commonly a group of soldiers. | False
A flight is a group of soldiers. | False

  * Negative template (verbal, analytic and clausal):

A/An $\langle word\rangle$ is not $\langle gloss\rangle$.

    * Sentences:

A flight is not a group of soldiers. | True
---|---
A flight is not a scheduled trip by plane between designated airports. | False

Pattern #02: antonymy (gloss)

  * Affirmative template:

$\langle word\rangle$ (commonly) [ stands for | refers to ] $\langle gloss\rangle$.

    * Sentences:

Brother commonly stands for a female person who has the same parents as another person. | False
---|---
Brother commonly refers to a female person who has the same parents as another person. | False
Brother stands for a female person who has the same parents as another person. | False
Brother refers to a female person who has the same parents as another person. | False
Stream commonly stands for a female person who has the same parents as another person. | False
Stream commonly refers to a female person who has the same parents as another person. | False
Stream stands for a female person who has the same parents as another person. | False
Stream refers to a female person who has the same parents as another person. | False
Brother commonly stands for a literary work based on the imagination and not necessarily on fact. | False
Brother commonly refers to a literary work based on the imagination and not necessarily on fact. | False
Brother stands for a literary work based on the imagination and not necessarily on fact. | False
Brother refers to a literary work based on the imagination and not necessarily on fact. | False

  * Negative template (synthetic and subclausal):

A/An $\langle word\rangle$ is never $\langle gloss\rangle$.

    * Sentences:

A brother is never a female person who has the same parents as another person. | True
---|---
A stream is never a female person who has the same parents as another person. | True
A brother is never a literary work based on the imagination and not necessarily on fact. | True

Figure 3: Description of Patterns #01 and #02.

In Table 10, we provide the truth-value of sentences that results by
instantiating affirmative and negative templates using WordNet and distracting
triples according to the pattern. In the following subsections, we describe
each pattern and provide some examples that are used in Figures 3–8 to
illustrate the instantiation of a sample of the templates. In all the
templates described in Figures 3–8, alternative and optional expressions are
enclosed respectively in square brackets and parentheses.

###  D.1 Pattern #01: synonymy (gloss)

This pattern includes 21 templates stating semantic equivalence
correspondences between a word and the gloss of a synset to which the word
belongs. Since WordNet does not provide triples for synonymy relating two
synsets, we get triples relating each synset to itself by reflexivity. For
each resulting triple, templates are also instantiated using one distracting
triple that is obtained by replacing the third component of each triple with a
distractor (synset).

In Figure 3, we introduce a positive and a negative template and illustrate
their instantiation using the synset ${\it{fligth}_{n}^{9}}$ with gloss “a
scheduled trip by plane between designated airports” and the distractor
${\it{troop}_{n}^{1}}$ (“a group of soldiers”).

###  D.2 Pattern #02: antonymy (gloss)

This pattern includes 21 templates stating semantic equivalence
correspondences between a word and the gloss of an antonym synset, where the
word and the gloss are respectively taken from the second and third component
of triples. Furthermore, for each WordNet antonymy triple templates are also
instantiated using two distracting triples that are respectively obtained by
replacing the second and third component with distractors (for the third one,
the distractor is a synset).

In Figure 3, we introduce a positive and a negative template and illustrate
their instantiation using the antonym synsets ${\it{brother}_{n}^{1}}$ and
${\it{sister}_{n}^{1}}$ (“a female person who has the same parents as another
person”) and the distractors stream and ${\it{fiction}_{n}^{1}}$ (“a literary
work based on the imagination and not necessarily on fact”).

Pattern #03: synonymy

  * Affirmative template:

$\langle noun_{1}$+(e)s$\rangle$ and $\langle noun_{2}$+(e)s$\rangle$ [ are | may be ] always different.

    * Sentences:

Path and route are always different. | False
---|---
Path and route may be always different. | False
Engine and route are always different. | True
Engine and route may be always different. | True
Path and identity are always different. | True
Path and identity may be always different. | True

  * Negative template (verbal, analytic and subclausal):

$\langle noun_{1}$+(e)s$\rangle$ and $\langle noun_{2}$+(e)s$\rangle$ [ are not | may not be ] synonyms in any context.

    * Sentences:

Path and route are not synonyms in any context. | False
---|---
Path and route may not be synonyms in any context. | False
Engine and route are not synonyms in any context. | True
Engine and route may not be synonyms in any context. | True
Path and identity are not synonyms in any context. | True
Path and identity may not be synonyms in any context. | True

Figure 4: Description of Pattern #03.

###  D.3 Pattern #03: synonymy

This pattern includes 24 templates stating semantic equivalence
correspondences between words. Since WordNet does not provide triples for
synonymy relating two synsets, we get triples relating each synset to itself
by reflexivity. For each resulting triple, templates are also instantiated
using two distracting triples that are respectively obtained by replacing the
second and third component with distractors.

In Figure 4, we introduce a positive and a negative template and illustrate
their instantiation using the synonym words path and route and the distractors
engine and identity.

Pattern #04: antonymy

  * Affirmative template:

$\langle noun_{1}$+(e)s$\rangle$ and $\langle noun_{2}$+(e)s$\rangle$ [ are | may be ] synonyms (in certain contexts).

    * Sentences:

Expenditure and income are synonyms in certain contexts. | False
---|---
Expenditure and income may be synonyms in certain contexts. | False
Expenditure and income are synonyms. | False
Expenditure and income may be synonyms. | False
Expenditure and year are synonyms in certain contexts. | False
Expenditure and year may be synonyms in certain contexts. | False
Expenditure and year are synonyms. | False
Expenditure and year may be synonyms. | False
Wood and income are synonyms in certain contexts. | False
Wood and income may be synonyms in certain contexts. | False
Wood and income are synonyms. | False
Wood and income may be synonyms. | False

  * Negative template (analytic and subclausal):

$\langle noun_{1}$+(e)s$\rangle$ and $\langle noun_{2}$+(e)s$\rangle$ [ are | may be ] the same thing in no context.

    * Sentences:

Expenditure and income are the same thing in no context. | True
---|---
Expenditure and income may be the same thing in no context. | True
Expenditure and year are the same thing in no context. | True
Expenditure and year may be the same thing in no context. | True
Wood and income are the same thing in no context. | True
Wood and income may be the same thing in no context. | True

Pattern #05: hypernymy

  * Affirmative template:

A/An $\langle hyponym\rangle$ [ is | may be ] a/an $\langle hypernym\rangle$ in certain contexts.

    * Sentences:

An auction is a sale in certain contexts. | True
---|---
An auction may be a sale in certain contexts. | True
A breakdwon is a sale in certain contexts. | False
A breakdown may be a sale in certain contexts. | False

  * Negative template (synthetic and subclausal):

A/An $\langle hyponym\rangle$ is never a/an $\langle hypernym\rangle$.

    * Sentences:

An auction is never a sale. | False
---|---
A breakdown is never a sale. | True

Figure 5: Description of Patterns #04 and #05.

###  D.4 Pattern #04: antonymy

This pattern includes 24 templates stating semantic equivalence
correspondences between words. For each WordNet antonymy triple, templates are
also instantiated using two distracting triples that are respectively obtained
by replacing the second and third component with distractors.

In Figure 5, we introduce a positive and a negative template and illustrate
their instantiation using the antonym synsets ${\it{expenditure}_{n}^{1}}$ and
${\it{income}_{n}^{1}}$ and the distractors wood and year.

###  D.5 Pattern #05: hypernymy

This pattern includes 24 templates stating semantic subsumption
correspondences between words. For each WordNet hypernymy triple, templates
are also instantiated using one distracting triple that is obtained by
replacing the hyponym with a distractor.

In Figure 5, we introduce a positive and a negative template and illustrate
their instantiation using the synset ${\it{auction}_{n}^{1}}$, which is
hyponym of ${\it{sale}_{n}^{2}}$, and the distractor breakdown.

Pattern #06: meronymy (part)

  * Affirmative template:

A/An $\langle part\rangle$ [ is commonly | may be ] part of a/an $\langle whole\rangle$.

    * Sentences:

A week is commonly part of a month. | True
---|---
A week may be part of a month. | True
A week is commonly part of a fence. | False
A week may be part of a fence. | False

  * Negative template (synthetic and subclausal):

A/An $\langle part\rangle$ is never part of a/an $\langle whole\rangle$.

    * Sentences:

A week is never part of a month. | False
---|---
A week is never part of a fence. | True

Pattern #07: meronymy (substance)

  * Affirmative template:

$\langle thing+(e)s\rangle$ [ are commonly | may be ] made of $\langle substance\rangle$.

    * Sentences:

Beaches are commonly made of sand. | True
---|---
Beaches may be made of sand. | True
Decades are commonly made of sand. | False
Decades may be made of sand. | False

  * Negative template (Analytic and subclausal):

In no context $\langle thing+(e)s\rangle$ [ are |may be ] made of $\langle
substance\rangle$.

    * Sentences:

In no context beaches are made of sand. | False
---|---
In no context decades are made of sand. | True

Figure 6: Description of Patterns #06 and #07.

###  D.6 Pattern #06: meronymy (part)

This pattern includes 16 templates stating semantic correspondences between
parts and wholes. For each WordNet triple, templates are also instantiated
using one distracting triple that is obtained by replacing the whole with a
distractor.

In Figure 6, we introduce a positive and a negative template and illustrate
their instantiation using the synset ${\it{week}_{n}^{3}}$, which is related
by part with ${\it{month}_{n}^{1}}$, and the distractor fence.

###  D.7 Pattern #07: meronymy (substance)

This pattern includes 15 templates stating semantic correspondences between
substances and things. For each WordNet triple, templates are also
instantiated using one distracting triple that is obtained by replacing the
whole with a distractor.

In Figure 6, we introduce a positive and a negative template and illustrate
their instantiation using the synset ${\it{sand}_{n}^{1}}$, which is related
by substance with ${\it{beach}_{n}^{1}}$, and the distractor decade.

Pattern #08: meronymy (member)

  * Affirmative template:

$\langle member+(e)s\rangle$ [ are | may be ] members of $\langle group+(e)s\rangle$.

    * Sentences:

Voters are members of electorates. | True
---|---
Voters may be members of electorates. | True
Voters are members of sports. | Falses
Voters may be members of sports. | False

  * Negative template (verbal, analytic and clausal):

$\langle member+(e)s\rangle$ [ are not | may not be ] members of $\langle group+(e)s\rangle$ in any context.

    * Sentences:

Voters are not members of electorates in any context. | False
---|---
Voters may not be members of electorates in any context. | False
Voters are not members of sports in any context. | True
Voters may not be members of sports in any context. | True

Pattern #09: semantic role (agent)

  * Affirmative template:

$\langle event+ing\rangle$ is commonly done by $\langle agent+(e)s\rangle$.

    * Sentences:

Ruling is commonly done by governors. | True
---|---
Ruling is commonly done by holes. | False

  * Negative template (Verbal, analytic and clausal):

$\langle event+ing\rangle$ is not commonly done by $\langle
agent+(e)s\rangle$.

    * Sentences:

Ruling is not commonly done by governors. | False
---|---
Ruling is not commonly done by holes. | True

Figure 7: Description of Patterns #08 and #09.

###  D.8 Pattern #08: meronymy (member)

This pattern includes 17 templates stating semantic correspondences between
members and groups. For each WordNet triple, templates are also instantiated
using one distracting triple that is obtained by replacing the group with a
distractor.

In Figure 7, we introduce a positive and a negative template and illustrate
their instantiation using the synset ${\it{voter}_{n}^{1}}$, which is related
by member with ${\it{electorate}_{n}^{1}}$, and the distractor sport.

###  D.9 Pattern #09: semantic role (agent)

This pattern includes 2 templates stating semantic correspondences between
agents and events. For each WordNet triple, templates are also instantiated
using one distracting triple that is obtained by replacing the agent with a
distractor.

In Figure 7, we introduce a positive and a negative template and illustrate
their instantiation using the synset ${\it{rule}_{n}^{1}}$, which is related
by agent with ${\it{governor}_{n}^{1}}$, and the distractor hole.

Pattern #10: semantic role (instrument)

  * Affirmative template:

A/An $\langle instrument\rangle$ [ is commonly | may be ] [ used | needed ] for $\langle event+ing\rangle$.

    * Sentences:

A telephone is commonly used for calling. | True
---|---
A telephone is commonly needed for calling. | True
A telephone may be used for calling. | True
A telephone may be needed for calling. | True
A telephone is commonly used for laying. | False
A telephone is commonly needed for laying. | False
A telephone may be used for laying. | False
A telephone may be needed for laying. | False

  * Negative template (Synthetic and subclausal):

A/An $\langle instrument\rangle$ should never be [ used | needed ] for $\langle event+ing\rangle$.

    * Sentences:

A telephone should never be used for calling. | False
---|---
A telephone should never be used for laying. | True

Pattern #11: semantic role (result)

  * Affirmative template:

$\langle event+ing\rangle$ [ commonly leads | may lead ] to a/an $\langle result\rangle$.

    * Sentences:

Answering commonly leads to a response. | True
---|---
Answering may lead to a response. | True
Dressing commonly leads to a response. | False
Dressing may lead to a response. | False

  * Negative template (analytic and subclausal):

$\langle event+ing\rangle$ [ leads | may lead ] to a/an $\langle result\rangle$ in no context.

    * Sentences:

Answering leads to a response in no context. | False
---|---
Answering may lead to a response in no context. | False
Answering leads to a response in no context. | True
Answering may lead to a response in no context. | True

Figure 8: Description of Patterns #10 and #11.

###  D.10 Pattern #10: semantic role (instrument)

This pattern includes 7 templates stating semantic correspondences between
instruments and events. For each WordNet triple, templates are also
instantiated using one distracting triple that is obtained by replacing the
event with a distractor.

In Figure 8, we introduce a positive and a negative template and illustrate
their instantiation using the synset ${\it{telephone}_{n}^{1}}$, which is
related by instrument with ${\it{call}_{v}^{3}}$, and the distractor lay.

###  D.11 Pattern #11: semantic role (result)

This pattern includes 27 templates stating semantic correspondences between
results and events. For each WordNet triple, templates are also instantiated
using one distracting triple that is obtained by replacing the event with a
distractor.

In Figure 8, we introduce a positive and a negative template and illustrate
their instantiation using the synset ${\it{response}_{n}^{1}}$, which is
related by result with ${\it{answer}_{v}^{1}}$, and the distractor dress.