# Quantum battery of interacting spins with environmental noise

Fang Zhao  Center for quantum technology research, School of Physics, Beijing
Institute of Technology, Beijing 100081, People’s Republic of China  China
Academy of Engineering Physics, Beijing 100088, People’s Republic of China
Fu-Quan Dou<EMAIL_ADDRESS>College of Physics and Electronic Engineering,
Northwest Normal University - Lanzhou, 730070, China  Qing Zhao
<EMAIL_ADDRESS>Center for quantum technology research, School of
Physics, Beijing Institute of Technology, Beijing 100081, People’s Republic of
China

###### Abstract

A quantum battery is a temporary energy-storage system. We constructed the
quantum battery model of an N-spin chain with nearest-neighbor hopping
interaction and investigated the charging process of the quantum battery. We
obtained the maximum energy in the quantum battery charged by a coherent
cavity driving field or a thermal heat bath. We confirmed that for a finite-
length spin chain, thermal charging results in a nonzero ergotropy,
contradicting a previous result: that an incoherent heat source cannot charge
a single-spin quantum battery. The nearest-neighbor hopping interaction
induces energy band splitting, which enhances the energy storage and the
ergotropy of the quantum battery. We found a critical point in the energy and
ergotropy resulting from the ground-state quantum phase transition, after
which the energy significantly enhance. Finally, we also found that disorder
increased the energy of the quantum battery.

##  I Introduction

A quantum battery (QB) system can potentially provide temporary energy
storage. The initially proposed QB was a two-level system that stores energy
from an extra field Alicki and Fannes (2013), and the initial physical model
was proposed as a single two-level spin system. Further QB studies verified
that an N spin with an extra field increases the charging power of the QB
Ferraro  _et al._ (2018); Andolina  _et al._ (2019a). A primary goal of QB
studies is finding the maximum energy stored in the QB during the charging
time and increasing the energy release of the QB after the charging process
Ferraro  _et al._ (2018); Fusco  _et al._ (2016); Binder  _et al._ (2015); Le
_et al._ (2018); Zhang  _et al._ (2019); Zhang and blaauboer (2018); Andolina
_et al._ (2018); Julià-Farré  _et al._ (2020); Chen  _et al._ (2020); Rosa
_et al._ (2019); Crescente  _et al._ (2020). Another research focus is the
entanglement and work-extraction capability of the QB Sen and Sen (2019);
Hovhannisyan  _et al._ (2013); Campaioli  _et al._ (2017); Andolina  _et al._
(2019b); Friis and Huber (2018); Caravelli  _et al._ (2020); García-Pintos
_et al._ (2020); Dou  _et al._ (2020). As the energy released in a QB is
usually in a thermal heat bath, a QB has commonly discussed the energy for
work in terms of ergotropy Allahverdyan  _et al._ (2004).

In the usual case, an extra field charges the energy of the QB. The energy
exchange between the QB and a cavity field occurs by coupling of the QB with
an energy-charged cavity field in an excited energy state Ferraro  _et al._
(2018). Energy oscillations during the charging process necessitate accurate
control of the charging time. Other charging sources are directly charged by a
magnetic field or a thermal heat bath Zhang  _et al._ (2019); Chen  _et al._
(2020); Pirmoradian and Mølmer (2019); Farina  _et al._ (2019). A QB charged
in a thermal bath must be discussed in an open quantum system. A single-atom
QB can be energy charging by a thermal heat bath, but its useful energy for
work is always zero Farina  _et al._ (2019). Due to the decay rate and the
driving field, the energy-charging process of the QB gradually stabilizes in
an open quantum system Farina  _et al._ (2019); Santos  _et al._ (2019); Ghosh
_et al._ (2020); Kamin  _et al._ (2020); Friis and Huber (2018); Quach and
Munro (2020); Zakavati  _et al._ (2020); Barra (2019); Liu  _et al._ (2019),
so controlling the charging time is not necessary.

Few previous studies have considered the hopping interaction between each spin
in a QB Ghosh  _et al._ (2020); Santos  _et al._ (2020); Rossini  _et al._
(2019a, b). In studies that do consider such interactions, the energy charging
and release by the QB are not discussed. In a real spin chain model, the
hopping interaction is vital and cannot be ignored. The spin chain interaction
creates a ground-state quantum phase transition and influences the ground-
state properties of the QB. The quantum phase transition also influences the
energy charging and release of the QB. The most simple interaction in a spin
chain is the nearest hopping interaction.

In this paper, we discuss a QB undergoing nearest-neighbor hopping
interaction, and the quantum-phase influence of the QB with a hopping term.
Fig.1 shows the charging protocol of our QB system. The physical model of the
QB is an $N$-spins Dicke-spin chain with hopping interaction $J$. The QB is
coupled to a cavity field with the decay rate of the $\kappa$-boson heat bath.
Because $\kappa$ decays at a certain rate, we must investigate the charging of
the QB in an open quantum system. The energy charging of the QB is
accomplished by an extra coherent driving field or is directly provided by the
thermal heat bath. We study the energy and ergotropy of the QB with and
without the nearest-neighbor hopping interaction. We compare the difference
between the two types of charging protocol: charging from an extra coherently
driven field and directly charging from a thermal heat bath. We investigate
how the hopping interaction and the spin chain length influences the energy
and ergotropy of the QB. Finally, we also study how disorder influences the
performance of the QB.

Figure 1:  A schematic diagram of the QB charging protocol used in this study.
It includes an N spin with a frequency of $\omega_{a}$. The spin has a
nearest-neighbor hopping interaction with a strength of $J$. The spin is
coupled with a single-photon cavity with a frequency of $\omega_{c}$ and a
decay rate of $\kappa$.

This paper organized as follows. In Sec. II, we introduce the QB model and the
dynamic equations. We define the energy and ergotropy of the QB. In Sec. III,
we report on the dynamics of QB charging without and with the hopping
interaction, respectively. We also investigates how the ground-state quantum
phase transition (QPT) influences the QB. In Sec. IV, we discuss the QPT
influence on the charging time of the QB, and the disorder influence on the
energy and ergotropy of the QB.

##  II Model

The QB is modeled by an $N$-spin chain, as shown in Fig. 1. The hopping
interaction strength between nearest-neighbor spins is $J$. The spin is
embedded in a microcavity with a cavity loss rate of $\kappa$. The Rabi
frequency of the photon–spin interaction is $g$. The total Hamiltonian of this
QB system is:

$\displaystyle H_{S}$ $\displaystyle=H_{A}+H_{B}+H_{I},$ (1) $\displaystyle
H_{A}$ $\displaystyle=\omega_{c}c^{\dagger}c,$ (2) $\displaystyle H_{B}$
$\displaystyle=\omega_{a}\sum_{i=1}^{N}\sigma_{+}^{i}\sigma_{-}^{i}+J\sum_{i=1}^{N-1}(\sigma_{+}^{i}\sigma_{-}^{i+1}+h.c.),$
(3) $\displaystyle H_{I}$
$\displaystyle=\sum_{i=1}^{N}g(\sigma_{+}^{i}c+h.c.).$ (4)

In the above expressions, $H_{A}$ is the Hamiltonian of the cavity part with
annihilation (creation) operator $c(c^{\dagger})$. $\omega_{c}$ is the cavity
frequency of the cavity field. $H_{B}$ is the Hamiltonian of the QB with
$\sigma_{\pm}^{i}$ as the raising/lowering spin operator for the $i$-th spin
and an atom frequency of $\omega_{a}$. $J$ is the nearest hopping interaction.
$H_{I}$ is the interaction term between the spin and the cavity field with
spin–photon coupling constant $g$.

Within this set-up, the cavity field is driven by an extra classical field
that charges the QB. The energy input from the driving field later transfers
to the QB via the cavity–spin interaction. In this paper, the QB is charged in
two ways: first from an extra coherent driving field acting on the cavity, and
second from a thermal heat bath coupled with the cavity field. We denote these
two charging approaches as coherent and thermal charging, respectively.

The Hamiltonian of the coherent driven field is

$H_{d}^{\prime}=f(e^{-i\omega_{d}t}c^{\dagger}+e^{i\omega_{d}t}c).$ (5)

Here, $f$ is the driving field strength. After a unitary transformation
$U=e^{i\omega_{c}t}$, the explicit time dependence term can be removed into
Farina  _et al._ (2019):

$H_{d}^{\prime}=f(e^{-i\delta t}c^{\dagger}+e^{i\delta t}c),$ (6)

where $\delta=\omega_{d}-\omega_{c}$. When we take $\omega_{c}=\omega_{d}$ for
simplicity, the driven field Hamiltonian will reduce to

$H_{d}=f(c^{\dagger}+c).$ (7)

The dynamic process of the QB coherent charging is obtained by solving the
Lindblad master equation:

$\dot{\rho}_{S}(t)=-i[H+H_{d},\rho_{S}(t)]+\kappa L_{c}[\rho_{S}],$ (8)

where $\kappa$ is the decay rate of the cavity field, and
$L_{c}[\rho_{S}]=c\rho_{S}c^{\dagger}-\frac{1}{2}(c^{\dagger}c\rho_{S}+\rho_{S}c^{\dagger}c)$
is the Lindblad super-operator.

The dynamics of thermal charging are obtained by solving the Lindblad master
equation:

$\dot{\rho}_{S}(t)=-i[H,\rho_{S}(t)]+\kappa(n_{B}+1)L_{c}[\rho_{S}]+\kappa
n_{B}L_{c^{\dagger}}[\rho_{S}].$ (9)

Where $n_{B}=1/(\exp[\omega_{c}/(k_{B}T)]-1)$ is the mean occupation number of
the boson heat bath.

The charging process of the QB fills the empty QB from the cavity field. We
prepare an empty QB by initializing the spin in its ground-state
$|g\rangle_{B}$. The initial state of the cavity is the vacuum state
$|0\rangle_{A}$. Thus, the initial state of the whole system is:

$|\psi(0)\rangle=|0\rangle_{A}\otimes|g\rangle_{B},$ (10)

where $|\psi(0)\rangle$ is the initial state of the whole system, and
$|g\rangle_{B}$ is the initial state of the QB corresponding to the energy
ground-state of $H_{B}$. When the cavity two-level interaction $g$ is turned
on, the charging process immediately starts the energy exchange between the
spin chain and the cavity field.

The energy storage in the QB at time $t$ is given by:

$E_{B}(t)=\text{tr}[H_{B}\rho_{B}(t)],$ (11)

where $\rho_{B}(t)=\text{tr}_{A}[\rho_{S}(t)]$ is the reduced density matrix
of the QB at time $t$. The energy-charged into the QB is $E_{B}(t)-E_{B}(0)$,
where $E_{B}(0)=E_{G}$ is the ground-state energy of the QB. Therefore, the
actual charging energy of the QB is

$\Delta E(t)=E_{B}(t)-E_{G}.$ (12)

Figure 2:  The dynamic charging process of the QB. (a) Energy $\Delta E(t)$
(b) Ergotropy $\varepsilon_{B}(t)$ and (c) Efficiency $R_{B}(t)$ as the
function of $gt$. Other parameters are $N=3$,
$\omega_{a}=\omega_{c}=g=\kappa=1$ and $J=0$.

The charging energy $\Delta E(t)$ can characterize the property of the QB.
However, under the second law of thermodynamics, it cannot be transformed into
work without dissipating the heat. Ergotropy characterizes the ability of the
QB to generate useful work Allahverdyan  _et al._ (2004). The ergotropy is
defined as

$\varepsilon_{B}(t)=E_{B}(t)-\min_{U}\text{tr}[H_{B}U\rho_{B}(t)U^{\dagger}].$
(13)

Diagonalizing $H_{B}$ and $\rho_{B}(t)$, we respectively obtain

$\displaystyle\rho_{B}(t)$
$\displaystyle=\sum_{n}r_{n}(t)|r_{n}(t)\rangle\langle r_{n}(t)|,$ (14)
$\displaystyle H_{B}$ $\displaystyle=\sum_{n}e_{n}|e_{n}\rangle\langle
e_{n}|.$ (15)

The eigenvalues of $\rho_{B}(t)$ are arranged in descending order as
$r_{0}\geq r_{1}\geq\cdots$, and the eigenvalues of $H_{B}$ are arranged in
ascending order as $e_{0}\leq e_{1}\leq\cdots$. The term
$\min_{U}\text{tr}[H_{B}U\rho_{B}(t)U^{\dagger}]$ of Eq.(13) can be simplified
as follows Allahverdyan  _et al._ (2004); Farina  _et al._ (2019):

$\min_{U}\text{tr}[HU\rho(t)U^{\dagger}]=\sum_{n}r_{n}e_{n}.$ (16)

It is easily proved that $\varepsilon_{B}(t)$ is always non-negative and
smaller than $\Delta E(t)$. Therefore, we can define the following efficiency
$R_{B}(t)$ as the percentage of $\varepsilon_{B}(t)$ among the total charging
energy $\Delta E(t)$:

$R_{B}(t)=\frac{\varepsilon_{B}(t)}{\Delta E(t)}.$ (17)

Using the energy $\Delta E$, we can judge the charging energy of the QB. The
ergotropy could judge $\varepsilon_{b}$ is the useful energy-releasing for
useful work of the QB. Furthermore, the efficiency $R_{B}$ describes the
useful energy release efficiency of the QB. This paper focuses on $\Delta E$,
$\varepsilon_{b}$, and $R_{B}$ during the QB charging process.

##  III The Charging property of the QB

This section will discuss the charging properties of the QB with or without
the hopping interaction. We will first discuss QB’s dynamic and find the
maximum energy and the ergotropy during the charging process. Then discuss the
quantum phase transition induced by the hopping interaction. Finally, We also
discuss how the quantum phase influences the charging properties of the QB.

###  III.1 Charging Without Hopping Interaction

Before discussing the effect of the hopping interaction, we discuss the
charging properties of the QB without the hopping term $J=0$. The ground-state
of the QB in this case is

$|g\rangle_{B}=|0\rangle^{\otimes N},$ (18)

where $|0\rangle$ is the ground-state of a single spin. The ground-state
energy $E_{G}$ of the QB without the hopping interaction is zero.

Figure 3:  The QB steady-state energy, ergotropy, and the energy release rate
for the different number of spins. (a) QB energy $\Delta E(\infty)$ (b)
Ergotropy $\varepsilon_{B}(\infty)$ (c) Efficiency $R_{B}(\infty)$. Other
parameters are $\omega_{a}=\omega_{c}=g=\kappa=1$ and $J=0$.

The dynamics of the coherent and thermal charging processes are determined by
Eqs. (8) and (9), respectively (see Fig. 2). $\omega_{a},\omega_{c},\kappa$,
and $g$ are not relatively essential parameters for the system stable-state.
Therefore in this paper, we only discuss the most direct case, where
$\omega_{a}=\omega_{c}=g=\kappa=1$. Owing to the decay loss, the strong and
weak driving strengths differ during the charging process. To compare these
two strengths, we set $f=2$ ($n_{B}=2$) and $f=0.2$ ($n_{B}=0.2$). The strong
and weak are just is for compared with $\kappa$. As shown in panels (a) and
(b) of Fig. 2, the energy $\Delta E(t)$ and ergotropy $\varepsilon_{B}(t)$ of
the QB were gradually stabilized. Under both coherent and thermal charging,
the energy and ergotropy of the QB increased with driving strength, but the
coherent charging oscillated in the beginning phase. The steady-state energy
at infinite time was the maximum charging energy. However, the ergotropy of
the QB was maximized within the oscillatory period, indicating that the
oscillations boosted the efficiency of the coherently charged QB at the
beginning. In contrast, both the energy and ergotropy of the thermally charged
QB were maximized at steady-state after infinite time. Besides, the QB
efficiency in the thermal charging large driving strength will also correspond
to a significant efficiency opposite to the thermal charging.

We could also find a nonzero ergotropy during the charging process for the
thermal charging. This result differs from those of previous works on simple-
spin systems, in which the ergotropy $\varepsilon_{B}$ is always zero Farina
_et al._ (2019). A nonzero ergotropy can be obtained by multiplying the spin
of the QB with the thermal charging, as shown in Fig. 3. We verified the
thermal charging of a single spin ($N=1$) QB always induces a zero ergotropy,
as reported in previous research Farina  _et al._ (2019). However, thermal
charging of a finite-length chain ($N\geq 2$) QB induces a nonzero ergotropy.
Under strong charging conditions, the spin numbers increase the energy,
ergotropy, and efficiency of the QB, but under weak charging conditions, the
energy and ergotropy of the QB both decreased with increasing chain length.

Figure 4:  Steady-state density matrices in the energy representation: Panels
(a) and (b) show the steady-state densities of the whole system and the
reduced density matrix of the QB under thermal charging, respectively. Panels
(c) and (d) show the thermal state density matrices of the whole system and
the QB, respectively. Other parameters are $N=3$,
$\omega_{a}=\omega_{c}=g=\kappa=1$, $J=0$ and $n_{B}=2$.

Figure 5:  The QB dynamic charging process with considering the hopping
interaction. (a) Energy $\Delta E(t)$ (b) Ergotropy $\varepsilon_{B}(t)$ and
(c) Efficiency $R_{B}$ as the function of $gt$. Other parameters are $N=3$,
$\omega_{a}=\omega_{c}=g=\kappa=1$ and $J=1$.

Above, we explained that thermal charging can induce nonzero ergotropy. If the
QB exists in the thermal state $\rho_{B}=\rho_{B}^{th}=\frac{e^{-\beta
H_{B}}}{z}$, where $z=tr[e^{-\beta H_{B}}]$, it is easily verified that the
ergotropy is always zero. The thermal-state density matrix $\rho_{B}^{th}$ and
the Hamiltonian representation of the energy are respectively given by

$\displaystyle\rho_{B}^{th}$
$\displaystyle=\frac{1}{z}\left(\begin{array}[]{cccc}e^{-\beta
e_{1}}&0&\cdots&0\\\ 0&e^{-\beta e_{1}}&\cdots&0\\\ 0&0&\cdots&e^{-\beta
e_{n}}\\\ \end{array}\right),$ (22) $\displaystyle~{}~{}~{}H_{B}$
$\displaystyle=\left(\begin{array}[]{cccc}e_{1}&0&\cdots&0\\\
0&e_{2}&\cdots&0\\\ 0&0&\cdots&e_{n}\\\ \end{array}\right).$ (26)

Here, the energy eigenvalues are in ascending order, and the diagonal elements
of the thermal state density matrix are in descending order. Thus, the mean
thermal- state energy is $E_{B}=\text{tr}[H_{B}\rho^{th}]=\sum_{n}r_{n}e_{n}$
(equaling the right hand of Eq.(13)), meaning that the ergotropy of a thermal
state is always zero. When the ergotropy is nonzero, the density of the QB is
reduced and the thermal state is not attained. In this scenario (Fig. 4), the
steady-state density matrix $\rho_{B}$ and system density matrix $\rho$ of the
QB differ from their corresponding density matrices $\rho_{B}^{th}$ and
$\rho^{th}$ in the thermal state. As thermal driving is coupled only to
photons within a $\beta$ temperature thermal bath, the total photon-spin
system cannot be purified into the thermal state. The reduced density matrix
$\rho_{B}$ also cannot exist in the thermal state. Thus, the ergotropy of a
non-single-atom QB can be nonzero under thermal charging conditions.

Figure 6:  The QB steady energy and ergotropy for different hopping
interaction. (a) Energy $\Delta E(\infty)$ (b) Ergotropy
$\varepsilon_{B}(\infty)$ and (c) Efficiency $R_{B}(\infty)$ as a function of
$J$. Other parameters are $N=3$, $\omega_{a}=\omega_{c}=g=\kappa=1$.

###  III.2 Charging with Hopping Interaction

In the previous subsection, we discussed the coherent and thermal charging
processes of the QB without the hopping interaction. We here consider a spin-
chain QB with nearest-neighbor interacting strength $J$ in the QB. The initial
state of the QB determines the energy ground-state of the interacting spin
chain. Through investigating the dynamics of the coherent and thermal charging
processes, we discuss how the hopping interaction influences the energy and
ergotropy of the QB.

When considering the hopping interaction, the charging dynamics of the QB are
shown in Fig. 5. The charging dynamics of the QB were largely unaffected by
hopping interaction, but the hopping interaction increased the time at which
the QB received its maximum energy, and also the energy of the QB under
coherent and thermal charging conditions. However, the hopping interaction
decreased the ergotropy of the QB. Weak charging reduced the efficiency of
both coherent and thermal charging. Meanwhile, the hopping interaction did not
remove the oscillations from the coherent charging phase. Regardless of
whether considering the hopping interaction or not, the difference between
coherent and thermal charging is only the energy oscillations during the
charging process.

Figure 7:  (a), (b), and (c) are the QB energy $\Delta E(\infty)$ and
ergotropy $\varepsilon_{B}(\infty)$ as a function of the hopping interaction
for the spin number N = 3, 4, 5, respectively. (d), (e), and (f) are the
respective energy spectra. (g), (h), and (i) are the corresponding ordering
parameters. The other parameters are $n_{B}=1$, $g=\kappa=1$.

To further discuss the influence of the hopping interaction, we calculated the
steady-state energy and the ergotropy of the QB at different hopping
interaction as shown in Fig. 6. At $J=0$, the energy and ergotropy were
remarkably changed by the interaction term, which broke the symmetry of the
QB. A non-differentiable points appeared at $J=1/\sqrt{2}$. Before this point,
the QB energy was a nearly constant function of hopping interaction, but
thereafter, it significantly increased with hopping interaction. Meanwhile,
the ergotropy of the QB exhibited two non-differentiable points: one at
$J=1/\sqrt{2}$, the other at $J=\sqrt{2}$.

We then calculated the steady-state energy and ergotropy of a thermally
charged QB with different spin numbers ($N=3,4,$, and $5$) are shown in panels
(a), (b), and (c), respectively, of Fig. 7. The energy versus hopping
interaction of the QB developed an extra non-differentiable point at large
spin numbers, but the energy and ergotropy of the QB exhibited similar growth
trends at small and large spin numbers. Whereas the energy did not
significantly increase before the first non-differentiable point, the
ergotropy was a continuously increasing function of hopping interaction.

We calculated the energy spectra to further discuss the non-differentiable
points in the QB’s energy and ergotropy. The results for $N=3,4$,and $5$ are
shown in panels (d), (e) and (f) of Fig. 7, respectively. Each non-
differentiable point of the energy corresponded to a crossing of the ground-
energy state. Before the first ground-state energy crossing, the ground-state
energy was always zero. After first crossing point, the ground-state energy
was gradually decreased by splitting of the energy bands. From Eq.(12), we
found that the decreasing ground-state energy due to band splitting increased
the energy by increasing the hopping interaction

The different energy behaviors of the QB’s energy around the first non-
differentiable point induced a QPT. To further discuss the QPT, we introduce
the order parameter. One common order parameter is the mean magnetic field
$M_{z}$, given by

$M_{z}=\frac{\langle S_{z}\rangle_{g}}{N}.$ (27)

Here we define another order parameter $\xi_{z}$ as

$\xi_{z}=\frac{\langle S_{z}^{2}\rangle_{g}}{N^{2}},$ (28)

where $\langle\cdots\rangle_{g}$ represents the average on the ground-state,
and the total spin operator is $S_{z}=\sum_{i=1}^{N}\sigma_{z}^{i}$. We
consider the only the ground-state’s ordering parameter because both the non-
differentiable points of the energy and ergotropy correspond to the ground-
state energy crossing.

The calculated order parameters for $N=3,4,$ and $5$ are presented in panels
(g), (h) and (i) of Fig. 7, respectively. The ordering parameter was
discontinuous at the point $J=1/\sqrt{N}$ and $J=\sqrt{N}$. Around these
discontinuous points, indicating a first-order QPT at this point. We have
already found significant changes in the QB charging properties after the
first non-differentiable point. Before the first non-differentiable point, the
order parameter was $M_{z}=-1$, meaning that each spin was in the spin-down
state corresponding to the ferromagnetic phase. After the first non-
differentiable point, the ordering parameter $M_{z}>-1$ departed from the
ferromagnetic phase. The ground-state quantum phase significantly influenced
the charging of the QB. In a non-interacting spin-chain QB, the ground-state
of the QB was always spin-down. Accordingly, the QB existed in the
ferromagnetic phase, which is unsuitable for more energy storage.

##  IV Charging time and disorder

In the previous section, we discussed how hopping interaction influences the
energy and ergotropy of the QB. We also observe that when the hopping
interaction was nonzero, the energy of the QB was enhanced after the first QPT
point, and the charging speed of the QB significantly increased. This section
investigates the influence of hopping interaction on the charging speed, and
the effects of onsite energy disorder

Figure 8:  The charging time $\tau_{c}$ as a function of the hopping
interaction $J$. Other parameters are $N=3$,
$\omega_{a}=\omega_{c}=g=\kappa=1$ .

We here focus on the steady-state energy and ergotropy of the QB during
charging in an open quantum system. Theoretically, the steady-state is reached
only after infinite time. In reality, the charging process is ceased when the
energy and ergotropy of the QB are sufficiently close to their steady-state
values. To judge the charging speed, we must therefore define a charging time.
We first define the charging power as

$P_{B}(t)=\frac{\Delta E(t)}{t}.$ (29)

The charging time $\tau_{c}$ then defines the time at which the charging power
is maximized:

$\tau_{c}=\arg\max_{t}P_{B}(t).$ (30)

Figure 9:  The steady-state energy and ergotropy of the QB influenced by the
onsite disorder $W$. These four figures correspond with different charging
strengths with (a) $f=0.2$ (b) $f=2$ (c) $n_{B}=0.2$ (d) $n_{B}=2$. Other
parameters are $N=3$, $\omega_{a}=\omega_{c}=g=\kappa=1$.

Fig. 8 shows the charging times $\tau_{c}$ at different hopping interaction.
We observe that the charging time $\tau_{c}$ suddenly increased after the
phase transition point. Before the phase transition point, the charging time
gradually reduced with increasing hopping interaction. After the QPT point,
energy and ergotropy increased at the expense of increasing the charging time.

We now discuss the influence of disorder on the energy and ergotropy of the
QB. Here we consider only an onsite disorder in the free energy term, which
changes the Hamiltonian of the QB as follows:

$H_{B}=\omega_{a}\sum_{i=1}^{N}(1+\delta_{i})\sigma_{+}^{i}\sigma_{-}^{i}+J\sum_{i=1}^{N-1}(\sigma_{+}^{i}\sigma_{-}^{i+1}+h.c.),$
(31)

where $\delta_{i}\in[-W/2,W/2]$ is the onsite disorder and $W$ is the disorder
strength. To ensure accurate numerical calculations, the result was computed
$100$ times and averaged to give the result in Fig. 9. We can find that the
disorders do not affect the location of the QPT point. After the phase
transition point, the energy and ergotropy of the QB still increased with
increasing hopping interaction, as observed previously. The onsite disorder
little affected the stability of the energy. However, the ergotropy is
unstable in the onsite disorder. The increased disorder strength enhanced the
energy of the QB. The increased QB energy was similar in magnitude to the
energy transfer enhanced by the disorder Chin  _et al._ (2010); Caruso  _et
al._ (2011, 2009), although a large disorder would reduce the ergotropy.

##  V Conclusion

We investigated QB charging in an open quantum system with coherent and
thermal charging. Without hopping interaction, we found that for spin lengths
$N\geq 2$, thermal charging imparted a nonzero ergotropy to the QB. Under weak
and strong charging conditions, increasing the spin length decreased and
increased the ergotropy of the QB, respectively. In the system with hopping
interaction, the QB energy increased with charging time after the first
ground-state energy crossing point. The ground-state QPT also affected the
energy of the QB. Finally, we investigated the onsite disorders of QB, and
verified that disorder increased the energy but decreased the ergotropy of the
QB.

###### Acknowledgements.

This work is supported by the National Science Foundation of China (Grants No.
11675014, No. 12075193).

## References

  * Alicki and Fannes (2013) R. Alicki and M. Fannes, Phys. Rev. E 87, 042123 (2013).
  * Ferraro  _et al._ (2018) D. Ferraro, M. Campisi, G. M. Andolina, V. Pellegrini, and M. Polini, Phys. Rev. Lett. 120, 117702 (2018).
  * Andolina  _et al._ (2019a) G. M. Andolina, M. Keck, A. Mari, V. Giovannetti, and M. Polini, Phys. Rev. B 99, 205437 (2019a).
  * Fusco  _et al._ (2016) L. Fusco, M. Paternostro, and G. De Chiara, Phys. Rev. E 94, 052122 (2016).
  * Binder  _et al._ (2015) F. C. Binder, S. Vinjanampathy, K. Modi, and J. Goold, New Journal of Physics 17, 075015 (2015).
  * Le  _et al._ (2018) T. P. Le, J. Levinsen, K. Modi, M. M. Parish, and F. A. Pollock, Phys. Rev. A 97, 022106 (2018).
  * Zhang  _et al._ (2019) Y.-Y. Zhang, T.-R. Yang, L. Fu, and X. Wang, Phys. Rev. E 99, 052106 (2019).
  * Zhang and blaauboer (2018) X. Zhang and M. blaauboer, “Enhanced energy transfer in a dicke quantum battery,” (2018), arXiv:1812.10139 [quant-ph] .
  * Andolina  _et al._ (2018) G. M. Andolina, D. Farina, A. Mari, V. Pellegrini, V. Giovannetti, and M. Polini, Phys. Rev. B 98, 205423 (2018).
  * Julià-Farré  _et al._ (2020) S. Julià-Farré, T. Salamon, A. Riera, M. N. Bera, and M. Lewenstein, Phys. Rev. Research 2, 023113 (2020).
  * Chen  _et al._ (2020) J. Chen, L. Zhan, L. Shao, X. Zhang, Y. Zhang, and X. Wang, Annalen der Physik 532, 1900487 (2020).
  * Rosa  _et al._ (2019) D. Rosa, D. Rossini, G. M. Andolina, M. Polini, and M. Carrega, “Ultra stable charging of fastest scrambling quantum batteries,” (2019), arXiv:1912.07247 [cond-mat.str-el] .
  * Crescente  _et al._ (2020) A. Crescente, M. Carrega, M. Sassetti, and D. Ferraro, New Journal of Physics 22, 063057 (2020).
  * Sen and Sen (2019) K. Sen and U. Sen, “Local passivity and entanglement in shared quantum batteries,” (2019), arXiv:1911.05540 [quant-ph] .
  * Hovhannisyan  _et al._ (2013) K. V. Hovhannisyan, M. Perarnau-Llobet, M. Huber, and A. Acín, Phys. Rev. Lett. 111, 240401 (2013).
  * Campaioli  _et al._ (2017) F. Campaioli, F. A. Pollock, F. C. Binder, L. Céleri, J. Goold, S. Vinjanampathy, and K. Modi, Phys. Rev. Lett. 118, 150601 (2017).
  * Andolina  _et al._ (2019b) G. M. Andolina, M. Keck, A. Mari, M. Campisi, V. Giovannetti, and M. Polini, Phys. Rev. Lett. 122, 047702 (2019b).
  * Friis and Huber (2018) N. Friis and M. Huber, Quantum 2, 61 (2018).
  * Caravelli  _et al._ (2020) F. Caravelli, G. Coulter-De Wit, L. P. García-Pintos, and A. Hamma, Physical Review Research 2 (2020), 10.1103/physrevresearch.2.023095.
  * García-Pintos  _et al._ (2020) L. P. García-Pintos, A. Hamma, and A. del Campo, Physical Review Letters 125 (2020), 10.1103/physrevlett.125.040601.
  * Dou  _et al._ (2020) F.-Q. Dou, Y.-J. Wang, and J.-A. Sun, EPL (Europhysics Letters) 131, 43001 (2020).
  * Allahverdyan  _et al._ (2004) A. E. Allahverdyan, R. Balian, and T. M. Nieuwenhuizen, Europhysics Letters (EPL) 67, 565 (2004).
  * Pirmoradian and Mølmer (2019) F. Pirmoradian and K. Mølmer, Phys. Rev. A 100, 043833 (2019).
  * Farina  _et al._ (2019) D. Farina, G. M. Andolina, A. Mari, M. Polini, and V. Giovannetti, Phys. Rev. B 99, 035421 (2019).
  * Santos  _et al._ (2019) A. C. Santos, B. i. e. i. f. m. c. Çakmak, S. Campbell, and N. T. Zinner, Phys. Rev. E 100, 032107 (2019).
  * Ghosh  _et al._ (2020) S. Ghosh, T. Chanda, and A. Sen(De), Phys. Rev. A 101, 032115 (2020).
  * Kamin  _et al._ (2020) F. H. Kamin, F. T. Tabesh, S. Salimi, F. Kheirandish, and A. C. Santos, New Journal of Physics 22, 083007 (2020).
  * Quach and Munro (2020) J. Q. Quach and W. J. Munro, Phys. Rev. Applied 14, 024092 (2020).
  * Zakavati  _et al._ (2020) S. Zakavati, F. T. Tabesh, and S. Salimi, “Bounds on charging power of open quantum batteries,” (2020), arXiv:2003.09814 [quant-ph] .
  * Barra (2019) F. Barra, Phys. Rev. Lett. 122, 210601 (2019).
  * Liu  _et al._ (2019) J. Liu, D. Segal, and G. Hanna, The Journal of Physical Chemistry C 123, 18303 (2019).
  * Santos  _et al._ (2020) A. C. Santos, A. Saguia, and M. S. Sarandy, Physical Review E 101 (2020), 10.1103/physreve.101.062114.
  * Rossini  _et al._ (2019a) D. Rossini, G. M. Andolina, D. Rosa, M. Carrega, and M. Polini, “Quantum charging supremacy via sachdev-ye-kitaev batteries,” (2019a), arXiv:1912.07234 [cond-mat.str-el] .
  * Rossini  _et al._ (2019b) D. Rossini, G. M. Andolina, and M. Polini, Phys. Rev. B 100, 115142 (2019b).
  * Chin  _et al._ (2010) A. W. Chin, A. Datta, F. Caruso, S. F. Huelga, and M. B. Plenio, New Journal of Physics 12, 065002 (2010).
  * Caruso  _et al._ (2011) F. Caruso, N. Spagnolo, C. Vitelli, F. Sciarrino, and M. B. Plenio, Phys. Rev. A 83, 013811 (2011).
  * Caruso  _et al._ (2009) F. Caruso, A. W. Chin, A. Datta, S. F. Huelga, and M. B. Plenio, The Journal of Chemical Physics 131, 105106 (2009).


# Commonsense Knowledge Reasoning and Generation
with Pre-trained Language Models: A Survey

Prajjwal Bhargava and Vincent Ng

###### Abstract

While commonsense knowledge acquisition and reasoning has traditionally been a
core research topic in the knowledge representation and reasoning community,
recent years have seen a surge of interest in the natural language processing
community in developing pre-trained models and testing their ability to
address a variety of newly designed commonsense knowledge reasoning and
generation tasks. This paper presents a survey of these tasks, discusses the
strengths and weaknesses of state-of-the-art pre-trained models for
commonsense reasoning and generation as revealed by these tasks, and reflects
on future research directions.

## Introduction

Commonsense knowledge is the information that is generally accepted by the
majority of people concerning everyday life, encapsulating the practical
knowledge about how the world works. Reasoning with commonsense knowledge is
at the core of building natural language understanding models and, more
broadly, AI systems that can reason about the world in the same way as humans
do.

A vast amount of work on commonsense knowledge acquisition and reasoning has
traditionally been conducted in the knowledge representation and reasoning
community (see Zang et al. (2013) for a survey). For instance, there have been
notable attempts to manually create large-scale commonsense knowledge bases
(e.g., Cyc (Lenat 1995) and automatically acquire commonsense knowledge from
the Web (e.g., Open Mind Common Sense (Singh et al. 2002). More recently, the
Winograd Schema Challenge, a pronoun resolution task that requires the use of
commonsense knowledge, was proposed as a practical alternative to the Turing
Test (Levesque, Davis, and Morgenstern 2012).

The advent of the neural natural language processing (NLP) era has
revolutionized virtually all areas of NLP research. One of the major
breakthroughs is arguably the development of _pre-trained_ language models
(PLMs). Specifically, researchers have discovered that neural models can be
trained (via a process known as _pre-training_) on a large body of
_unannotated_ text to acquire general knowledge about language, including both
linguistic and commonsense knowledge. This has sparked tremendous interest in
the NLP community in examining what kind of commonsense knowledge PLMs possess
and the extent to which such knowledge can be exploited to address commonsense
knowledge reasoning and generation tasks in the last few years.

Our goal in this paper is to provide the general AI audience with a timely
survey of the recent advances in the NLP community on commonsense knowledge
reasoning and generation using PLMs. Specifically, the focus of this survey is
(1) the kind of commonsense knowledge that PLMs possess and (2) the extent to
which such knowledge can be exploited for recently designed commonsense
knowledge reasoning and generation tasks. For an overview of what _linguistic_
knowledge PLMs possess, we refer the reader to Rogers, Kovaleva, and Rumshisky
(2020). For comprehensive surveys of the details and the inner workings of
PLMs, we refer the reader to Qiu et al. (2020); Han et al. (2021); Kalyan,
Rajasekharan, and Sangeetha (2021).

## Pre-trained Language Models

In this section, we provide the reader with the background on PLMs needed to
understand the rest of the paper.

For a long time, _supervised_ learning has been the most successful learning
paradigm in NLP. For instance, training a neural model to perform a
classification task in a supervised manner primarily involves training an
_encoder_ to encode the input as a _task-specific representation_ that would
be useful for classifying a given sample. In contrast, for a text generation
task (e.g., text summarization, machine translation), one would typically
employ an encoder-decoder neural architecture, in which the encoder first
encodes the input, and then the decoder generates the output sequence token by
token based on both the encoded input and the tokens that have been generated
so far. The performance of supervised models is often limited by the
(typically small) amount of data they are trained on.

Pre-training offers a solution to the aforementioned data scarcity problem.
The idea is to first train a model on one or more _self-supervised_ learning
tasks during a process known as _pre-training_ , and the resulting model, in
which the weights have already been initialized during pre-training, can be
_fine-tuned_ using the (potentially small amount of) task-specific data in a
supervised fashion. Self-supervised learning tasks are NLP tasks for which the
label associated with a training instance can be derived automatically from
the text itself. Consider, for instance, one of the most well-known self-
supervised learning tasks, Masked Language Modeling (MLM) (Devlin et al.
2019). Given a sequence of word tokens in which a certain percentage of tokens
is _masked_ randomly, the goal of MLM is to predict the masked tokens. As can
be imagined, a model for MLM can therefore be trained on instances where each
one is composed of a partially masked sequence of word tokens and the
associated “class” value is the masked tokens themselves. Because no human
annotation is needed, a model can be pre-trained on a very large amount of
labeled data that can be automatically generated. Various studies have shown
that pre-training allows a model to learn universal language representations
that encode both linguistic and commonsense knowledge, and a PLM, after being
fine-tuned, can offer substantially improved performance on a wide variety of
NLP tasks.

Existing PLMs differ primarily in terms of (1) what is being pre-trained (is
it the encoder, the decoder, or both of them?); (2) the self-supervised
learning tasks used; and (3) the network architecture. While early work has
focused on pre-training the encoder (e.g., BERT (Devlin et al. 2019), RoBERTa
(Liu et al. 2019), ELECTRA (Clark et al. 2020) or the decoder (e.g., GPT-2
(Svyatkovskiy et al. 2020)), recent work has focused on jointly pre-training
the encoder and the decoder (e.g., T5 (Raffel et al. 2020), BART (Lewis et al.
2020)). The most successful PLMs are based on the Transformer (Vaswani et al.
2017), a fully-connected self-attention model. Throughout this paper we will
simply use the term PLMs to refer to Transformer-based PLMs.

## Capturing Commonsense Knowledge

How well do PLMs capture commonsense knowledge ? Researchers have employed
_probing_ to answer this question. To probe a PLM for commonsense knowledge,
most of the probing methods use a hand-crafted _template_ to convert a
relational fact from a knowledge base (KB), which is typically represented in
the form of a triple <$s,r,o$> where $s$ is the Subject, $r$ is the Relation,
and $o$ is the Object, into a natural language sentence. One template needs to
be defined for each relation. As illustrated in Figure 1, each triple having
the KB relation “place of birth” would be translated to a sentence of the form
“Subject was born in Object”. Note that a template keeps the entities intact
while approximately the Relation to a set of hand-coded verbs/relations that
can generalize on numerous entities (e.g., Atlocation may be translated to “is
in”). One of the entities (i.e., the Subject or the Object) in the sentence
will then be masked. The resulting _clozed_ sentence can then provide an
automated and flexible way to probe PLMs for stored knowledge. Specifically,
if a PLM can fill in the blank in the given clozed sentence correctly (i.e.,
the answer is the same as the entity that appears in the triple originally
used to generate the sentence), then the PLM is considered to possess the
knowledge being probed. Note that we are not asking the PLM to derive _new_
knowledge: only inference is performed by the PLM to check for stored
knowledge.

Several key observations are being revealed via probing experiments. First,
PLMs are becoming a promising alternative to KBs. BERT shows signs of
capturing relational knowledge in a zero-shot setting reasonably well compared
to supervised alternatives (Petroni et al. 2019). It can recall factual
knowledge for some relations such as one-to-one, but it struggles to perform
well on other relations such as N-to$\mathbf{M}$ and N-to-one.

Figure 1: A common approach to probe PLMs for stored knowledge (Petroni et al. 2019). Prompt |  Model Predictions
---|---
_A has fur, is big, has claws, has teeth, is an animal, eats, is brown, and lives in woods._ |  bear, wolf, cat, …
Table 1: Masked token predictions about stereotypical assumptions get refined
as more properties are appended. (Nathaniel Weir 2020).

Second, PLMs do not generalize well on unseen entities. While BERT can predict
the top 100 triples mined from Wikipedia fairly accurately, which suggests
that BERT can generalize to specific data sources (Davison, Feldman, and Rush
2019), PLMs do not generalize well on entities not encountered during pre-
training due to their heavy reliance on memorization in the pre-training
process (Logan et al. 2019).

Third, PLMs can perform comparisons and categorization of entities.
Specifically, they can compare physical objects along a particular attribute
such as weight or size (e.g., a chair is _smaller than_ a room) (Goel, Feng,
and Boyd-Graber 2019). When it comes to categorization, they work reasonably
well on knowledge types that are ontological in nature, such as “mango isA
fruit” (Hwang et al. 2021).

Finally, analyzing the top-$k$ predictions made by PLMs on the association
between an entity and its attributes, we see that PLMs can learn stereotypical
associations reasonably well from large text corpora. As more properties are
appended to provide contextual knowledge (see the example in Table 1), the
performance of RoBERTa-L increases, with predictions going from being sensible
to more acceptable as per human interpretation. Specifically, PLMs do better
on functional (e.g., “eat fish”) and encyclopedic (e.g., “are found in
forests”) knowledge than visual-perceptual variants (e.g., “bears have fur”).
Although this result is encouraging, when asked for widely acceptable
properties about an entity, the ranked predictions provided by PLMs do not
correlate strongly with those of humans (Nathaniel Weir 2020).

## Reasoning with Commonsense Knowledge

In this section, we examine how well PLMs perform commonsense reasoning by
considering five types of commonsense reasoning tasks.

### Linguistic Reasoning

Linguistic reasoning is concerned with understanding text for which the
correct interpretation requires commonsense knowledge. A representative
benchmark for linguistic reasoning is Winogrande (Sakaguchi et al. 2020),
which consists of Winograd schema-inspired problems that require linguistic,
social or physical reasoning (Levesque, Davis, and Morgenstern 2012). As an
example, given the sentence “The plant took up too much room in the urn,
because the  was large” and two answer candidates “plant” and “urn”, the goal
is to determine which candidate should be used to fill in the blank.

Several observations can be made based on the performance of PLMs on this and
other linguistic reasoning tasks.

First, BERT shows poor linguistic sensitivity and becomes fragile on negated
and misprimed sentences (Kassner and Schütze 2020; Ettinger 2020). For
example, BERT fails to distinguish between the two sentences “Birds cannot
$[\mathrm{MASK}]$ ”) and “Birds can [MASK]” and tends to get distracted if
they are prepended with “misprimes” such as “Talk? Birds can [MASK]”. The fact
that its predictions do not change with such major changes indicates that BERT
does not attend to the prominent cues in the desired manner.

Second, PLMs perform poorly on numerical knowledge out-of-the-box (Lin et al.
2020a; Wallace et al. 2019; Chen et al. 2019; Bhagavatula et al. 2020). For
instance, given the sentence “Birds have [MASK] legs”, BERT predicts “four” to
be the answer, suggesting that pre-training does not facilitate the
acquisition of numerical knowledge.

Third, as the sentences in a given reasoning task require more turns of
logical reasoning (i.e., the task becomes increasingly complex), BERT’s
performance deteriorates (Zhou et al. 2020b; Richardson and Sabharwal 2020;
Huang et al. 2019). These are typically sentences with complex semantics such
as riddles, where PLMs are required to understand figurative language (Lin et
al. 2021b).

Several attempts have been made to improve the _robustness_ of PLMs for
linguistic reasoning tasks.

Semantic similarity. Niu et al. (2021) show that semantic similarity matching
can be used to make PLMs robust against irrelevant factors such as word
frequencies. Specifically, we can first use PLMs to generate plausible answers
so that we can compute the similarity between each generated answer and each
of the provided answer candidates, and then we can select the answer candidate
that has the highest similarity score as the correct answer.
Attention maps. Klein and Nabi (2019) show that attention maps obtained from
BERT can be used for coreference resolution in long sentences, suggesting
their potential usefulness for pronoun disambiguation.
Numerical reasoning. To improve numerical reasoning, Geva, Gupta, and Berant
(2020) pre-train BERT on two numerical tasks, one involving predicting what
comes after a sentence such as “3+4+5” and the other involving answering
numerical questions (e.g., given a historical passage about Spain, answer the
question of “How many Japanese families were in Spain?”), so that BERT is
endowed with the ability to understand computations expressed in pseudo-
natural language (text+digits).

Figure 2: Simulating what might happen next in order to enable LMs to encode
language “form” and “meaning” (Zellers et al. 2021).

### Reasoning about Physical World

Physical commonsense reasoning involves understanding concepts based on the
physical properties of objects, including the _affordances_ of objects (i.e.,
the actions applicable to them) and how they can be manipulated. A
representative benchmark for physical commonsense reasoning is PIQA (Bisk et
al. 2020). Given a sentence such as “When boiling butter, when it’s ready, you
can  ” , the goal is to fill in the blank with one of two answer options, such
as “Pour it onto a plate” and “Pour it onto a jar”.

Perception and Interaction are among the key components behind how humans
learn to reason about the physical world. However, static input
representations which current neural models are fed are inadequate since they
cannot compensate for the information humans derive from being in a dynamic
physical world. So, a key question posed by PIQA is whether PLMs can reason
over physical commonsense questions without interacting with the physical
world.

Several observations can be made about the performance of PLMs on physical
commonsense reasoning questions. First, PLMs predominantly learn property
associations that are explicitly mentioned in text, achieving higher
accuracies on entities that have simple affordances (e.g., “spoon”) than on
entities that have a a long tail of affordances (e.g., “water”). Second, PLMs
struggle to understand fundamental relations (e.g., “before/after”,
“top/bottom”, and find it hard to reason when common objects are used in
unconventional ways (e.g., a glue stick is used as a paper weight). Finally,
although neural representations are dexterous at capturing the affordances
(“boats can be driven”) and properties (“boats require fuel”) of objects, PLMs
struggle to understand the connection between affordances and properties
(Forbes, Holtzman, and Choi 2019; Zhao, Papalexakis, and Ma 2020).

In light of the weakness associated with the lack of interaction with the
physical world, (Zellers et al. 2021) explore the benefits of providing PLMs
access to world dynamics. World dynamics include information that one would
obtain after interacting with objects. As an example, consider Figure 2. If an
action heatUp is applied to a pan, the model will learn that the temperature
of an egg has risen to become hot and is now in a cooked state. Predicting
object states after an action has been taken drastically improves a PLM’s
ability to make correct inferences about object states.

### Abductive Reasoning

Abductive reasoning involves finding the most likely explanation for a set of
incomplete observations. There are at least two representative benchmarks for
abductive reasoning, CosmosQA (Huang et al. 2019) and Hellaswag (Zellers et
al. 2019b). CosmosQA is a commonsense comprehension task where, given a
context, the goal is to choose the answer to the question based on the context
from four answer candidates. This benchmark contains questions that require
abductive reasoning, such as “what might I continue to do after the situation
described in the context?” Hellaswag is a benchmark in which the goal is to
choose the best plausible ending of a given context out of four options.

Several observations can be made. First, (Huang et al. 2019) attribute the
errors made by their model on Cosmosqa to two reasons. First, PLMs struggle on
examples where the context becomes intricate enough to require cross-sentence
interpretation and reasoning. In such examples, PLMs are required to
understand the important parts of the passage and jointly attend to the
identified parts. In addition, PLMs do not understand what situations are
inconsistent with human commonsense. For example, they may choose “leaving a
baby alone at home is not safe” over “she would try to find a babysitter” when
asked the question “what would happen if she does not find a daycare”.
Interestingly,when one of the answer options is “None of the above”, PLMs
often struggle to choose this option since the words in this option do not
provide enough signal for why this option might be correct.

Second, PLMs struggle with selecting the most plausible ending given a context
for Hellaswag. Since a given context can have multiple correct endings,
determining which one would be the most plausible requires prior reasoning of
what humans relate to the most with their commonsense knowledge. Zellers et
al. (2018, 2019b) show that when more surface cues are eliminated, PLMs are
less likely to be able to predict the most plausible ending even though it
might be trivial for humans to do so.

Figure 3: Learning to reason about dynamic context from a static image
(Zellers et al. 2019a).

### Social Reasoning

Social reasoning involves modeling the mental states of others and their
likely actions to the extent that reasoning can be performed over their
behaviors and emotions. A representative benchmark for social reasoning is
Socialiqa (Sap et al. 2019b), which evaluates commonsense reasoning based on
social situations and interactions. Consider the following example taken from
socialiqa, in which the correct answer is boldfaced:

“Context”: “Tracy had accidentally pressed upon Austin in the small elevator
and it was awkward.”
“Question” : “Why did Tracy do this?”
“Choice A”: “get very close to Austin”;
“Choice B”: “squeeze into the elevator”;
“Choice C”: “get flirty with Austin”

Several observations can be made. First, BERT finds examples of effects (“what
will happen to X?”) and motivation (“why did X do that to Y?”) easier than
those that involve understanding descriptions (“how would you describe X?”)
(Sap et al. 2019b). Second, it performs better on examples where the answer
exhibits cues about emotions than those involving spatial commonsense
(Bhagavatula et al. 2020).

### Multimodal Reasoning

Textual representations are restricted to what can be expressed through
natural language and therefore are unable to represent the multi-modal
information that humans could have access to or infer from by being in a
dynamic world, such as a constant stream of images and a sequence of
interactions in the physical world. Vision naturally becomes a next step
towards enabling learning through joint interaction (Baldwin 1995). However,
merely using raw visual images along with their descriptions is by no means
sufficient to provide grounded understanding (Marasović et al. 2020). For
example, inferring the intentions of the entities in images can only be well
dealt with if we have some prior information (either behavioral or temporal)
to rely on to make justifiable inferences. This has led the community to look
into approaches that can help provide a tighter integration of linguistic and
visual modalities.

There are two well-known benchmarks for multimodal reasoning. Visual
commonsense reasoning (Vcr) (Zellers et al. 2019a) seeks to answer cognition-
level questions from images. Concretely, given an image with a list of regions
and a question, the goal is to choose the answer to the question out of a set
of possible candidates and provide a rationale that can explain why the chosen
answer is correct. An example can be found in Figure 3. Visual commonsense
graphs (Park et al. 2020) checks how well PLMs reason about the dynamic
context from a static image and an event. Specifically, given an image and a
textual description of an event at present, the task is to generate the rest
of the visual commonsense graph that is connected to the event. For example,
given an image of a man who is drowning in the river and a textual description
of the event, the goal is to generate a commonsense graph with nodes such as
“the man wanted to save himself from drowning”, “the man is waiting for help”,
“the man senses his own death”, and “the man needs to swim towards the river
bank”. Empirical results reveal that for both benchmarks, models that exploit
both visual and textual information outperform those that only use textual
information. This suggests that visual features help make higher quality
commonsense inferences.

### Temporal Reasoning

Time is an inherent aspect of events. Broadly, temporal reasoning involves two
subtasks. _Temporal attribute prediction_ involves understanding an event
mentioned in text or dialogue through reasoning with its temporal dimensions
such as the duration of the event, when the event typically happens, how long
the event is going to be stationary, and how often it happens. _Temporal
relation identification_ involves understanding how an event is temporally
related to other events mentioned in the same text or dialogue (e.g., did an
event take place _before_ or _after_ another event?). Temporal reasoning is
challenging because the timestamp associated with an event and the
aforementioned temporal dimensions may not be mentioned explicitly and
therefore need to be inferred.

Two commonly-used benchmarks have been developed for temporal reasoning. Mc-
taco (Zhou et al. 2020a) is a question-answering benchmark involving temporal
commonsense comprehension. Here is an example:

“Context”: The massive ice sheet, called a glacier, caused the features on the
land you see today
“Question”: When did the glacier start to impact the land’s features ?
“options”: a) centuries ago; b) hours ago; c) 10 years ago; d) tens of
millions of years ago

Timedial (Qin et al. 2021) involves temporal reasoning in dialogues. Here is
an example:
A: May we see the wine list please.
B: Sure. Our special wine today is a 1989 Chardonnay.
A: I’d like a bottle please.
B: I’ll need to see your ID please.
A: Here you go.
B: Sorry about the inconvenience, you look so young. I had to make sure you
are over.
a) 21 years old; b) 30 years old; c) 4 years old; d) 18 years old

Ideally, one can train _time-aware_ PLMs to address these temporal reasoning
tasks. An obstacle to the development of such PLMs concerns the lack of large-
scale KBs that incorporate the notion of time into the facts that they encode
over entities and events. For instance, the Location relation (i.e., where a
person lives) and the Employment relation (i.e., the company a person is
affiliated with) are dependent on time, but existing KBs typically fail to
encode the time period for which a given relation holds true. Such time-aware
KBs should also encode temporal commonsense knowledge such as “if a student
attends a university, s/he will likely graduate and work after a few years”.

Given the lack of such KBs, time-aware PLMs can only be trained on the
annotated training data provided by Mctaco and Timedial. For instance, (Zhou
et al. 2020a) propose Taco-lm, a Bert-based PLM that is trained to be
temporally aware by contextually estimating duration and time via (1)
extracting the important events and their temporal information, and then (2)
asking the model to predict the masked tokens that talk about some temporal
aspect. However, Taco-lm only provides marginal improvements over Bert w.r.t.
duration, frequency, and when the event typically takes place. More recently,
Qin et al. (2021) have shown that fine-tuned PLMs struggle to perform well on
Timedial primarily because they largely fail to understand the context of the
given dialogue and instead simply rely on shallow cues about the temporal
patterns in the context.

## Generating Commonsense Knowledge

Commonsense knowledge generation is a critical component in building
commonsense knowledge resources. Broadly, we can divide commonsense knowledge
generation tasks into two categories, as described below.

### Knowledge Base Completion

A KB is a collection of relational facts, each of which is represented as a
triple <$s,r,o$>, where $s$ is the subject, $r$ is the relation, and $o$ is
the object. KB completion is the task of automatically inferring missing facts
by reasoning about the information already present in the KB.

To date, the most successful knowledge generation approach with PLMs is
arguably Commonsense Transformer (COMET) (Bosselut et al. 2019). COMET can be
used to generate $o$ given $s$ and $r$, after being pre-trained on a knowledge
base such as ConceptNet (Singh et al. 2002; Speer, Chin, and Havasi 2016),
which represents (mostly taxonomic) commonsense knowledge as a graph of
concepts (words or phrases) connected by relations (edge types), or ATOMIC
(Sap et al. 2019a), which is a large-scale KG consisting of textual
descriptions of inferential knowledge (_if-then_ relations).

### Constrained Commonsense Text Generation

Next we examine studies on how PLMs can be used to generate commonsense text
subject to a set of constraints.

Tasks There are three benchmarks commonly used to evaluate commonsense
generation approaches.

Commongen (Lin et al. 2020b): Given a concept set (e.g., $\\{$dog, frisbee,
catch, throw$\\}$), the goal is to generate a coherent sentence describing an
everyday event using all the provided concepts.

Commonsense explanations (COS-E) (Rajani et al. 2019): is a dataset targeted
at a task known as explanation generation. Given that a model selects an
answer (from a set of candidates) to a given question, the goal is to generate
an explanation of why the selected answer is correct. The resulting
explanation may help us understand the reasoning that a model relies on to
arrive at the selected answer.

$\alpha$ NLG (Bhagavatula et al. 2020): Given two observations/events $O_{1}$
and $O_{2}$ that happen in two different timesteps, the goal is to generate a
valid hypothesis $h$ of what happened between the observations/events.

Challenges These benchmarks reveal that PLMs, when used as commonsense
knowledge generators, suffer from several shortcomings.

  * •

Poor coherency: The generated sentences do not necessarily adhere to the human
notion of commonsense. For instance, given the concept set {dog, catch, throw,
frisbee}, GPT2 generates the sentence “A dog throws a frisbee at a football
player”. Although this sentence is grammatically correct, it suffers from poor
coherency.

  * •

Insufficient concept coverage: PLMs continue to produce sentences that fail to
include all concepts from the provided concept set. In the previous example,
the concept “catch” was not used to generate the output.

  * •

Limited reasoning capability: It is not clear what kind of reasoning is used
by PLMs to arrive at an answer. Though a solution to explanation generation
could shed light on this question, some studies show that existing approaches
generate either trivial or noisy explanations, providing little or no evidence
of how a PLM arrives at the selected answer (Ji et al. 2020a). Other studies
show that the reasoning used by PLMs is often not fully correct (McCoy,
Pavlick, and Linzen 2019; Shwartz and Choi 2020). Overall, the reasoning
capability of PLMs is far from satisfactory.

Figure 4: Using structural relational knowledge for Multi hop reasoning (Ji et
al. 2020b). Blue nodes correspond to concepts in story, orange nodes
correspond to those in story ending and green nodes are intermediate concepts
that connect their blue and orange counterparts.

Improving Coherency and Concept Coverage Several approaches have been proposed
to address two of the challenges, poor coherency and insufficient concept
coverage.
Using prototypes. (Guu et al. 2018) propose a sentence generation mechanism
that involves selecting a sentence from the training data (known as a
_prototype_) and editing it into a form that satisfies a given set of
constraints. Their hypothesis is that it is easier to edit a sentence that is
grammatically correct and semantically coherent than to generating one from
scratch. If provided with a concept set {trailer, shirt, side, sit, road} from
commongen, a PLM may generate “A man sits on the side of a trailer and a
shirt”, whereas a prototype such as “Two guys in red shirts are sitting on
chairs, by the side of the road, behind that open trailer” maybe edited by the
PLM to form “a man in a white shirt and black pants sits on the side of a
trailer on the road”, which has better coverage and coherency.
Using knowledge graphs (KGs). KGs can play a crucial role in enabling PLMs to
improve the semantic correctness of text (and thus coherency) of text as they
can provide PLMs with information that may not be captured reliably by PLMs,
such as entity representations and their dependency relations (i.e., how
concepts are related), which may not be captured reliably by PLMs. For
instance, Li et al. (2020) extract concept-specific relations from a KG and
inject them into a PLM to make the generated text more coherent.
Reasoning over multi-hop relational paths in KGs. The sparse connections
between the nodes in KGs may make it hard for PLMs to learn rich relations
from them. These rich relations, however, may be needed by PLMs to generate
commonsense sentences with rich semantic structures in order to boost
coherency. A solution to this problem is to perform _multi-hop reasoning_ ,
which involves reasoning over multiple edges/relations in a KG. When
performing multi-hop reasoning, models are required to attend to different
parts of a given context to answer a question. Figure 4 shows an example of a
task known as _story ending_ generation, where the goal is to generate the end
of a story given the story context. In this example, external commonsense
knowledge in the form of relational paths can guide the generation of the two
nodes lava and substance by providing background knowledge such as (volcano,
madeOf, lava). Capturing what lava and substance that appear in the story
ending refer to in the story context is a non-trivial task for PLMs,
especially when the story is long. To address this drawback, (Ji et al. 2020b)
perform reasoning over multi-hop relational paths as a way to extract
structural and semantic knowledge from a KG.
Using iterative refinements. When provided with information about past and
future events (as in $\alpha\mathrm{NLG}$ ), humans can easily reason about
these events by using contextual and prior knowledge. This kind of non-
monotonic reasoning is crucial to improving generation coherency. However,
nonmonotonic reasoning is difficult for neural models since the generation
process happens predominantly while conditioning on the left context (Welleck
et al. 2019). To address this problem, (Qin et al. 2020) propose a decoding
approach that involves sampling from the combined output vector
representations computed from both forward and backward propagation. In other
words, iterative refinements are made on the generated text through
alternating between forward and backward passes, yielding generated text with
improved coherency.

## Concluding Remarks

Despite recent progresses on using PLMs to address commonsense knowledge
reasoning and generation tasks, many of these tasks are far from being solved.
We conclude our discussion with key challenges in this area of research.

Dataset | Model | Human | Dataset | Model | Human
---|---|---|---|---|---
hellaswag | 93.85 | 95.6 | winogrande | 86.64 | 94.0
comosqa | 91.79 | 94.0 | socialiqa | 83.15 | 88.1
piqa | 90.13 | 94.9 | vcr | 63.15. | 85.0
Table 2: Results of state-of-the-art models and human baselines on widely-used
commonsense reasoning benchmarks.

Improving benchmarks. While state-of-the-art models have achieved near-human
performance on many of the benchmarks mentioned in this paper (see Table 2),
the reasoning tasks underlying these benchmarks are still far from being
solved. Consequently, it is not clear what the performance gains on a
particular benchmark mean. Bender and Koller (2020) point out that acing a
benchmark has led us to over-estimate the capability of PLMs, which in turn
has given rise to misleading definitions of “understanding”. It is therefore
important to re-think what is being learned by PLMs and how benchmarks can be
made more representative such that performance gains on them translate to
meaningful progress towards the bigger goal of “understanding”.
Reducing biases. Biases in benchmarks such as predictable question structures,
annotation artifacts, and lexical overlap provide easy shortcuts for PLMs to
arrive at correct answers without involving reasoning. To mitigate biases,
researchers have used _adversarial filtering_ wherein easily solvable options
are replaced iteratively by new ones until the discriminator misclassifies it
(Zellers et al. 2018; McCoy, Pavlick, and Linzen 2019; Bras et al. 2020). To
robustify data, several workarounds have been proposed that revolve around
reducing lexical overlap, creating complex reasoning questions that require
additional context, and employing adversarial approaches with newer models
(Gardner et al. 2019). Bias reduction in benchmarks remains an active research
area.
Exploring new components of commonsense knowledge. These are numerous
components of commonsense knowledge that are partially understood and not
covered by the present research. One primary reason for this is that we do not
have a comprehensive understanding of how humans learn. A concrete example can
be derived from Kahneman’s (2011) cognitive system of intuition. There is no
clear way of representing a human’s mental and emotional states that can be
readily used by our algorithms. Modeling multiple mental states with natural
language is a highly non-trivial process (Sap et al. 2020).
Addressing the reporting bias. Much commonsense knowledge is assumed rather
than mentioned explicitly in text (Grice 1975; Durme and Benjamin 2009; Gordon
and Van Durme 2013). This results in what is known as the “reporting bias”,
which, when combined with the scarcity of training data for many NLP tasks,
makes it hard for PLMs to receive appropriate signal about a particular
concept (Zhao, Papalexakis, and Ma 2020) . This could lead to over-
generalization of associations and amplification of biases (Shwartz and Choi
2020). How to address the reporting bias remains an open question.
Improving existing KGs. While many approaches rely on KGs to obtain rich
contextual knowledge, existing KGs have at least two key weaknesses that can
potentially limit their usefulness for commonsense reasoning. The first is
_sparsity_ : many concepts and relations are missing in existing KGs (Li et
al. 2016). This sparsity problem in turn limits the amount of knowledge that
can be extracted from KGs for commonsense reasoning (Zhao, Papalexakis, and Ma
2020). The second is _non-contextualization_ : finding the nodes that are most
relevant to a query is difficult, particularly by propagation-based
algorithms, because many of them are non-contextual in nature (Fadnis et al.
2019). To address the non-contextualization problem, (Malaviya et al. 2020)
have attempted to use the structural and semantic connections of the nodes in
a KG to obtain contextual information, which addresses the non-
contextualization problem. The resulting contextual information can then be
explicitly encoded in a KG by creating additional nodes, which alleviates the
sparsity problem. How to densify KGs and contextualize their nodes is an
ongoing research topic (Wang et al. 2020).
Harnessing commonsense knowledge from different modalities. There are many
instances wherein the visual modality is required along with text to make
sense of a particular situation (Park et al. 2020). While humans learn
commonsense knowledge predominantly through perception and interaction with
the physical world, neural models are primarily trained on text data.
Harnessing commonsense knowledge from different modalities can potentially
take these models to the next level of performance.
Towards multilinguality. An important but unexplored direction is multi-
lingual commonsense reasoning and generation. Studies have shown that the
performance of cross-lingual PLMs is poor when evaluated on non-English
commonsense reasoning benchmarks (Lin et al. 2021a). These models perform
tremendously poorly when evaluated on a test set that was translated to
English, leading to staggering transfer reasoning capabilities to other
languages and restricting the research scope to only certain languages (Ponti
et al. 2020).

## References

  * Bender and Koller (2020) Bender, E. M.; and Koller, A. 2020.  Climbing towards NLU: On Meaning, Form, and Understanding in the Age of Data.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , 5185–5198. Online: Association for Computational Linguistics.
  * Bhagavatula et al. (2020) Bhagavatula, C.; Bras, R. L.; Malaviya, C.; Sakaguchi, K.; Holtzman, A.; Rashkin, H.; Downey, D.; tau Yih, W.; and Choi, Y. 2020.  Abductive Commonsense Reasoning.  In _International Conference on Learning Representations_.
  * Bisk et al. (2020) Bisk, Y.; Zellers, R.; Bras, R. L.; Gao, J.; and Choi, Y. 2020.  PIQA: Reasoning about Physical Commonsense in Natural Language.  In _Thirty-Fourth AAAI Conference on Artificial Intelligence_.
  * Bosselut et al. (2019) Bosselut, A.; Rashkin, H.; Sap, M.; Malaviya, C.; Çelikyilmaz, A.; and Choi, Y. 2019.  COMET: Commonsense Transformers for Automatic Knowledge Graph Construction.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (ACL)_.
  * Bras et al. (2020) Bras, R. L.; Swayamdipta, S.; Bhagavatula, C.; Zellers, R.; Peters, M. E.; Sabharwal, A.; and Choi, Y. 2020.  Adversarial Filters of Dataset Biases.  In _Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event_ , volume 119 of _Proceedings of Machine Learning Research_ , 1078–1088. PMLR.
  * Chen et al. (2019) Chen, M.; D’Arcy, M.; Liu, A.; Fernandez, J.; and Downey, D. 2019.  CODAH: An Adversarially-Authored Question Answering Dataset for Common Sense.  In _Proceedings of the 3rd Workshop on Evaluating Vector Space Representations for NLP_ , 63–69. Minneapolis, USA: Association for Computational Linguistics.
  * Clark et al. (2020) Clark, K.; Luong, M.-T.; Le, Q. V.; and Manning, C. D. 2020.  ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators.  In _International Conference on Learning Representations_.
  * Davison, Feldman, and Rush (2019) Davison, J.; Feldman, J.; and Rush, A. 2019.  Commonsense Knowledge Mining from Pretrained Models.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , 1173–1178. Hong Kong, China: Association for Computational Linguistics.
  * Devlin et al. (2019) Devlin, J.; Chang, M.-W.; Lee, K.; and Toutanova, K. 2019.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , 4171–4186. Minneapolis, Minnesota: Association for Computational Linguistics.
  * Durme and Benjamin (2009) Durme, V.; and Benjamin, D. 2009.  Extracting implicit knowledge from text.
  * Ettinger (2020) Ettinger, A. 2020.  What BERT Is Not: Lessons from a New Suite of Psycholinguistic Diagnostics for Language Models.  _Transactions of the Association for Computational Linguistics_ , 8: 34–48.
  * Fadnis et al. (2019) Fadnis, K. P.; Talamadupula, K.; Kapanipathi, P.; Ishfaq, H.; Roukos, S.; and Fokoue, A. 2019.  Heuristics for Interpretable Knowledge Graph Contextualization.  _CoRR_ , abs/1911.02085.
  * Forbes, Holtzman, and Choi (2019) Forbes, M.; Holtzman, A.; and Choi, Y. 2019.  Do Neural Language Representations Learn Physical Commonsense?  In _CogSci_.
  * Gardner et al. (2019) Gardner, M.; Berant, J.; Hajishirzi, H.; Talmor, A.; and Min, S. 2019.  On Making Reading Comprehension More Comprehensive.  In _Proceedings of the 2nd Workshop on Machine Reading for Question Answering_ , 105–112. Hong Kong, China: Association for Computational Linguistics.
  * Geva, Gupta, and Berant (2020) Geva, M.; Gupta, A.; and Berant, J. 2020.  Injecting Numerical Reasoning Skills into Language Models.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , 946–958. Online: Association for Computational Linguistics.
  * Goel, Feng, and Boyd-Graber (2019) Goel, P.; Feng, S.; and Boyd-Graber, J. 2019.  How Pre-trained Word Representations Capture Commonsense Physical Comparisons.  In _Proceedings of the First Workshop on Commonsense Inference in Natural Language Processing_ , 130–135. Hong Kong, China: Association for Computational Linguistics.
  * Gordon and Van Durme (2013) Gordon, J.; and Van Durme, B. 2013.  Reporting Bias and Knowledge Acquisition.  In _Proceedings of the 2013 Workshop on Automated Knowledge Base Construction_ , AKBC ’13, 25–30. New York, NY, USA: Association for Computing Machinery.  ISBN 9781450324113.
  * Grice (1975) Grice, H. P. 1975.  Logic and Conversation.  In Ezcurdia, M.; and Stainton, R. J., eds., _The Semantics-Pragmatics Boundary in Philosophy_ , 47. Broadview Press.
  * Guu et al. (2018) Guu, K.; Hashimoto, T. B.; Oren, Y.; and Liang, P. 2018.  Generating Sentences by Editing Prototypes.  _Transactions of the Association for Computational Linguistics_ , 6: 437–450.
  * Han et al. (2021) Han, X.; Zhang, Z.; Ding, N.; Gu, Y.; Liu, X.; Huo, Y.; Qiu, J.; Zhang, L.; Han, W.; Huang, M.; Jin, Q.; Lan, Y.; Liu, Y.; Liu, Z.; Lu, Z.; Qiu, X.; Song, R.; Tang, J.; Wen, J.-R.; Yuan, J.; Zhao, W. X.; and Zhu, J. 2021.  Pre-Trained Models: Past, Present and Future.  _AI Open_.
  * Huang et al. (2019) Huang, L.; Le Bras, R.; Bhagavatula, C.; and Choi, Y. 2019.  Cosmos QA: Machine Reading Comprehension with Contextual Commonsense Reasoning.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , 2391–2401. Hong Kong, China: Association for Computational Linguistics.
  * Hwang et al. (2021) Hwang, J. D.; Bhagavatula, C.; Bras, R. L.; Da, J.; Sakaguchi, K.; Bosselut, A.; and Choi, Y. 2021.  COMET-ATOMIC 2020: On Symbolic and Neural Commonsense Knowledge Graphs.  In _AAAI_.
  * Ji et al. (2020a) Ji, H.; Ke, P.; Huang, S.; Wei, F.; and Huang, M. 2020a.  Generating Commonsense Explanation by Extracting Bridge Concepts from Reasoning Paths.  In _Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing_ , 248–257. Suzhou, China: Association for Computational Linguistics.
  * Ji et al. (2020b) Ji, H.; Ke, P.; Huang, S.; Wei, F.; Zhu, X.; and Huang, M. 2020b.  Language Generation with Multi-Hop Reasoning on Commonsense Knowledge Graph.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , 725–736. Online: Association for Computational Linguistics.
  * Kalyan, Rajasekharan, and Sangeetha (2021) Kalyan, K. S.; Rajasekharan, A.; and Sangeetha, S. 2021.  AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processing.  _CoRR_ , abs/2108.05542.
  * Kassner and Schütze (2020) Kassner, N.; and Schütze, H. 2020.  Negated and Misprimed Probes for Pretrained Language Models: Birds Can Talk, But Cannot Fly.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , 7811–7818. Online: Association for Computational Linguistics.
  * Klein and Nabi (2019) Klein, T.; and Nabi, M. 2019.  Attention Is (not) All You Need for Commonsense Reasoning.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , 4831–4836. Florence, Italy: Association for Computational Linguistics.
  * Lenat (1995) Lenat, D. B. 1995.  CYC: A Large-Scale Investment in Knowledge Infrastructure.  _Commun. ACM_ , 38(11): 33–38.
  * Levesque, Davis, and Morgenstern (2012) Levesque, H. J.; Davis, E.; and Morgenstern, L. 2012.  The Winograd Schema Challenge.  In _Proceedings of the Thirteenth International Conference on Principles of Knowledge Representation and Reasoning_ , KR’12, 552–561. AAAI Press.  ISBN 9781577355601.
  * Lewis et al. (2020) Lewis, M.; Liu, Y.; Goyal, N.; Ghazvininejad, M.; Mohamed, A.; Levy, O.; Stoyanov, V.; and Zettlemoyer, L. 2020.  BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , 7871–7880. Online: Association for Computational Linguistics.
  * Li et al. (2016) Li, X.; Taheri, A.; Tu, L.; and Gimpel, K. 2016.  Commonsense Knowledge Base Completion.  In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , 1445–1455. Berlin, Germany: Association for Computational Linguistics.
  * Lin et al. (2020a) Lin, B. Y.; Lee, S.; Khanna, R.; and Ren, X. 2020a.  Birds have four legs?! NumerSense: Probing Numerical Commonsense Knowledge of Pre-Trained Language Models.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , 6862–6868. Online: Association for Computational Linguistics.
  * Lin et al. (2021a) Lin, B. Y.; Lee, S.; Qiao, X.; and Ren, X. 2021a.  Common Sense Beyond English: Evaluating and Improving Multilingual Language Models for Commonsense Reasoning.  _CoRR_ , abs/2106.06937.
  * Lin et al. (2021b) Lin, B. Y.; Wu, Z.; Yang, Y.; Lee, D.; and Ren, X. 2021b.  RiddleSense: Answering Riddle Questions as Commonsense Reasoning.  _CoRR_ , abs/2101.00376.
  * Lin et al. (2020b) Lin, B. Y.; Zhou, W.; Shen, M.; Zhou, P.; Bhagavatula, C.; Choi, Y.; and Ren, X. 2020b.  CommonGen: A Constrained Text Generation Challenge for Generative Commonsense Reasoning.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , 1823–1840. Online: Association for Computational Linguistics.
  * Liu et al. (2019) Liu, Y.; Ott, M.; Goyal, N.; Du, J.; Joshi, M.; Chen, D.; Levy, O.; Lewis, M.; Zettlemoyer, L.; and Stoyanov, V. 2019.  RoBERTa: A Robustly Optimized BERT Pretraining Approach.  _CoRR_ , abs/1907.11692.
  * Logan et al. (2019) Logan, R.; Liu, N. F.; Peters, M. E.; Gardner, M.; and Singh, S. 2019.  Barack’s Wife Hillary: Using Knowledge Graphs for Fact-Aware Language Modeling.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , 5962–5971. Florence, Italy: Association for Computational Linguistics.
  * Malaviya et al. (2020) Malaviya, C.; Bhagavatula, C.; Bosselut, A.; and Choi, Y. 2020.  Commonsense Knowledge Base Completion with Structural and Semantic Context.  _Proceedings of the 34th AAAI Conference on Artificial Intelligence_.
  * Marasović et al. (2020) Marasović, A.; Bhagavatula, C.; Park, J. s.; Le Bras, R.; Smith, N. A.; and Choi, Y. 2020.  Natural Language Rationales with Full-Stack Visual Reasoning: From Pixels to Semantic Frames to Commonsense Graphs.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , 2810–2829. Online: Association for Computational Linguistics.
  * McCoy, Pavlick, and Linzen (2019) McCoy, T.; Pavlick, E.; and Linzen, T. 2019.  Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , 3428–3448. Florence, Italy: Association for Computational Linguistics.
  * Nathaniel Weir (2020) Nathaniel Weir, B. V. D., Adam Poliak. 2020.  Probing Neural Language Models for Human Tacit Assumptions.  In _42nd Annual Virtual Meeting of the Cognitive Science Society (CogSci)_.
  * Niu et al. (2021) Niu, Y.; Huang, F.; Liang, J.; Chen, W.; Zhu, X.; and Huang, M. 2021.  A Semantic-based Method for Unsupervised Commonsense Question Answering.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , 3037–3049. Online: Association for Computational Linguistics.
  * Park et al. (2020) Park, J. S.; Bhagavatula, C.; Mottaghi, R.; Farhadi, A.; and Choi, Y. 2020.  VisualCOMET: Reasoning about the Dynamic Context of a Still Image.  In _In Proceedings of the European Conference on Computer Vision (ECCV)_.
  * Petroni et al. (2019) Petroni, F.; Rocktäschel, T.; Riedel, S.; Lewis, P.; Bakhtin, A.; Wu, Y.; and Miller, A. 2019.  Language Models as Knowledge Bases?  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , 2463–2473. Hong Kong, China: Association for Computational Linguistics.
  * Ponti et al. (2020) Ponti, E. M.; Glavaš, G.; Majewska, O.; Liu, Q.; Vulić, I.; and Korhonen, A. 2020.  XCOPA: A Multilingual Dataset for Causal Commonsense Reasoning.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , 2362–2376. Online: Association for Computational Linguistics.
  * Qin et al. (2021) Qin, L.; Gupta, A.; Upadhyay, S.; He, L.; Choi, Y.; and Faruqui, M. 2021.  TIMEDIAL: Temporal Commonsense Reasoning in Dialog.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , 7066–7076. Online: Association for Computational Linguistics.
  * Qin et al. (2020) Qin, L.; Shwartz, V.; West, P.; Bhagavatula, C.; Hwang, J. D.; Le Bras, R.; Bosselut, A.; and Choi, Y. 2020.  Back to the Future: Unsupervised Backprop-based Decoding for Counterfactual and Abductive Commonsense Reasoning.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , 794–805. Online: Association for Computational Linguistics.
  * Qiu et al. (2020) Qiu, X.; Sun, T.; Xu, Y.; Shao, Y.; Dai, N.; and Huang, X. 2020.  Pre-trained Models for Natural Language Processing: A Survey.  _CoRR_ , abs/2003.08271.
  * Raffel et al. (2020) Raffel, C.; Shazeer, N.; Roberts, A.; Lee, K.; Narang, S.; Matena, M.; Zhou, Y.; Li, W.; and Liu, P. J. 2020.  Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer.  _Journal of Machine Learning Research_ , 21(140): 1–67.
  * Rajani et al. (2019) Rajani, N. F.; McCann, B.; Xiong, C.; and Socher, R. 2019.  Explain Yourself! Leveraging Language Models for Commonsense Reasoning.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , 4932–4942. Florence, Italy: Association for Computational Linguistics.
  * Richardson and Sabharwal (2020) Richardson, K.; and Sabharwal, A. 2020.  What Does My QA Model Know? Devising Controlled Probes Using Expert Knowledge.  _Transactions of the Association for Computational Linguistics_ , 8: 572–588.
  * Rogers, Kovaleva, and Rumshisky (2020) Rogers, A.; Kovaleva, O.; and Rumshisky, A. 2020.  A Primer in BERTology: What We Know About How BERT Works.  _Transactions of the Association for Computational Linguistics_ , 8: 842–866.
  * Sakaguchi et al. (2020) Sakaguchi, K.; Bras, R. L.; Bhagavatula, C.; and Choi, Y. 2020.  WINOGRANDE: An Adversarial Winograd Schema Challenge at Scale.  In _AAAI_.
  * Sap et al. (2019a) Sap, M.; Bras, R. L.; Allaway, E.; Bhagavatula, C.; Lourie, N.; Rashkin, H.; Roof, B.; Smith, N. A.; and Choi, Y. 2019a.  ATOMIC: An Atlas of Machine Commonsense for If-Then Reasoning.  In _The Thirty-Third AAAI Conference on Artificial Intelligence, AAAI 2019, The Thirty-First Innovative Applications of Artificial Intelligence Conference, IAAI 2019, The Ninth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2019, Honolulu, Hawaii, USA, January 27 - February 1, 2019_ , 3027–3035. AAAI Press.
  * Sap et al. (2019b) Sap, M.; Rashkin, H.; Chen, D.; Bras, R. L.; and Choi, Y. 2019b.  Social IQA: Commonsense Reasoning about Social Interactions.  In _EMNLP 2019_.
  * Sap et al. (2020) Sap, M.; Shwartz, V.; Bosselut, A.; Choi, Y.; and Roth, D. 2020.  Commonsense Reasoning for Natural Language Processing.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: Tutorial Abstracts_ , 27–33. Online: Association for Computational Linguistics.
  * Shwartz and Choi (2020) Shwartz, V.; and Choi, Y. 2020.  Do Neural Language Models Overcome Reporting Bias?  In _Proceedings of the 28th International Conference on Computational Linguistics_ , 6863–6870. Barcelona, Spain (Online): International Committee on Computational Linguistics.
  * Singh et al. (2002) Singh, P.; Lin, T.; Mueller, E. T.; Lim, G.; Perkins, T.; and Zhu, W. L. 2002.  Open Mind Common Sense: Knowledge Acquisition from the General Public.  In _OTM_.
  * Speer, Chin, and Havasi (2016) Speer, R.; Chin, J.; and Havasi, C. 2016.  ConceptNet 5.5: An Open Multilingual Graph of General Knowledge.  _CoRR_ , abs/1612.03975.
  * Svyatkovskiy et al. (2020) Svyatkovskiy, A.; Deng, S. K.; Fu, S.; and Sundaresan, N. 2020.  _IntelliCode Compose: Code Generation Using Transformer_ , 1433–1443.  New York, NY, USA: Association for Computing Machinery.  ISBN 9781450370431.
  * Vaswani et al. (2017) Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A. N.; Kaiser, L. u.; and Polosukhin, I. 2017.  Attention is All you Need.  In Guyon, I.; Luxburg, U. V.; Bengio, S.; Wallach, H.; Fergus, R.; Vishwanathan, S.; and Garnett, R., eds., _Advances in Neural Information Processing Systems_ , volume 30. Curran Associates, Inc.
  * Wallace et al. (2019) Wallace, E.; Wang, Y.; Li, S.; Singh, S.; and Gardner, M. 2019.  Do NLP Models Know Numbers? Probing Numeracy in Embeddings.  _CoRR_ , abs/1909.07940.
  * Wang et al. (2020) Wang, P.; Peng, N.; Ilievski, F.; Szekely, P.; and Ren, X. 2020.  Connecting the Dots: A Knowledgeable Path Generator for Commonsense Question Answering.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , 4129–4140. Online: Association for Computational Linguistics.
  * Welleck et al. (2019) Welleck, S.; Brantley, K.; Daumé, H.; and Cho, K. 2019.  Non-Monotonic Sequential Text Generation.  In _ICML_.
  * Zang et al. (2013) Zang, L.; Cao, C.; Cao, Y.; Wu, Y.; and Cao, C. 2013.  A Survey of Commonsense Knowledge Acquisition.  _Journal of Computer Science and Technology_ , 28: 689–719.
  * Zellers et al. (2019a) Zellers, R.; Bisk, Y.; Farhadi, A.; and Choi, Y. 2019a.  From Recognition to Cognition: Visual Commonsense Reasoning.  In _The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * Zellers et al. (2018) Zellers, R.; Bisk, Y.; Schwartz, R.; and Choi, Y. 2018.  SWAG: A Large-Scale Adversarial Dataset for Grounded Commonsense Inference.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , 93–104. Brussels, Belgium: Association for Computational Linguistics.
  * Zellers et al. (2019b) Zellers, R.; Holtzman, A.; Bisk, Y.; Farhadi, A.; and Choi, Y. 2019b.  HellaSwag: Can a Machine Really Finish Your Sentence?  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , 4791–4800. Florence, Italy: Association for Computational Linguistics.
  * Zellers et al. (2021) Zellers, R.; Holtzman, A.; Peters, M.; Mottaghi, R.; Kembhavi, A.; Farhadi, A.; and Choi, Y. 2021.  PIGLeT: Language Grounding Through Neuro-Symbolic Interaction in a 3D World.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics_.
  * Zhao, Papalexakis, and Ma (2020) Zhao, Z.; Papalexakis, E.; and Ma, X. 2020.  Learning Physical Common Sense as Knowledge Graph Completion via BERT Data Augmentation and Constrained Tucker Factorization.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , 3293–3298. Online: Association for Computational Linguistics.
  * Zhou et al. (2020a) Zhou, B.; Ning, Q.; Khashabi, D.; and Roth, D. 2020a.  Temporal Common Sense Acquisition with Minimal Supervision.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , 7579–7589. Online: Association for Computational Linguistics.
  * Zhou et al. (2020b) Zhou, X.; Zhang, Y.; Cui, L.; and Huang, D. 2020b.  Evaluating Commonsense in Pre-Trained Language Models.  In _The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020_ , 9733–9740. AAAI Press.