Beam Energy Scan a Case for the Chiral Magnetic Effect in Au-Au Collisions.
R.S. Longacrea
aBrookhaven National Laboratory, Upton, NY 11973, USA

###### Abstract

The Chiral Magnetic Effect (CME) is predicted for Au-Au collisions at RHIC.
However many backgrounds can give signals that make the measurement hard to
interpret. The STAR experiment has made measurements at different collisions
energy ranging from $\sqrt{s_{NN}}$=7.7 GeV to 62.4 GeV. In the analysis that
is presented we show that the CME turns on with energy and is not present in
central collisions where the induced magnetic is small.

##  1 Introduction

Topological configurations should occur in the hot Quantum Chromodynamic (QCD)
vacuum of the Quark-Gluon Plasma (QGP) which can be created in heavy ion
collisions. These topological configurations form domains of local strong
parity violation (P-odd domains) in the hot QCD matter through the so-called
sphaleron transitions. The domains might be detected using the Chiral Magnetic
Effect (CME)[1] where the strong external magnetic field(electrodynamic) at
the early stage of a collision(non-central), through the sphaleron transitions
which induces a charge separation along the direction of the magnetic field
perpendicular to the reaction plane. Such an out of plane charge separation,
however, varies its orientation from event to event, either parallel or anti-
parallel to the magnetic field (sphaleron or antisphaleron). Also the magnetic
field can be up or down with respect to the reaction plane depending if the
ions pass in a clockwise or anti-clockwise manner. Any P-odd observable will
vanish and only the variance of such observable may be detected.

The STAR collaboration[2] has published a measurement of charge particle
azimuthal correlations consistent with CME expectations. In Ref.[3] and used
by STAR the CME can be indirectly approached through a two-particle azimuthal
correlation given by

$\gamma=\langle cos(\phi_{1}+\phi_{2}-2\Psi_{RP})\rangle=\langle
cos(\phi_{1}-\Psi_{RP})cos(\phi_{2}-\Psi_{RP})\rangle-\langle
sin(\phi_{1}-\Psi_{RP})sin(\phi_{2}-\Psi_{RP})\rangle,$ (1)

where $\Psi_{RP}$, $\phi_{1}$, $\phi_{2}$ denote the azimuthal angles of the
reaction plane, produced particle 1, and produced particle 2. This two
particle azimuthal correlation measures the difference between the in plane
and out of plane projected azimuthal correlation. If we would rotate all
events such that $\Psi_{RP}$ = 0.0, then $\gamma$ would become

$\gamma=\langle cos(\phi_{1}+\phi_{2})\rangle=\langle
cos(\phi_{1})cos(\phi_{2})\rangle-\langle sin(\phi_{1})sin(\phi_{2})\rangle.$
(2)

The CME predicts that $\gamma$ $>$ 0 for opposite sign-pairs and $\gamma$ $<$
0 for same sign-pairs. There are other two particle azimuthal correlation
effects that can depend on the reaction plane driven by elliptic flow even
though the underlying correlation may be independent of the reaction plane.
These backgrounds are summarized in Ref.[4]. It was pointed in Ref.[4] that
the $\phi$ difference correlation ($\delta$) which is independent of the
reaction plane gives a constraint on the CME and backgrounds.

$\delta=\langle cos(\phi_{1}-\phi_{2})\rangle=\langle
cos(\phi_{1})cos(\phi_{2})\rangle+\langle sin(\phi_{1})sin(\phi_{2})\rangle.$
(3)

In Ref.[4] Transverse Momentum Conservation (TMC) is derived and demonstrated
that if there is no other correlation in the data except elliptic flow TMC
will give a negative $\langle cos(\phi_{1}+\phi_{2}-2\Psi_{RP})\rangle$
($\gamma$) which is $v_{2}$ smaller than $\langle
cos(\phi_{1}-\phi_{2})\rangle$ ($\delta$). $\delta$ is a negative number given
by TMC and scales as 1/N (N is the number of particles). Also in Ref.[4] Local
Charge Conservation (LCC) is another important background and the details of
LCC is found in Ref.[5]. The authors of Ref.[5] point out that for same sign
pairs LCC should give a small negative sign, while for opposite sign pairs LCC
should give a large positive correlation. Using the same coupling effect to
the reaction plane as TMC, one should expect $\langle
cos(\phi_{1}+\phi_{2}-2\Psi_{RP})_{+-}\rangle$ ($\gamma$) equal $v_{2}$ times
$\langle cos(\phi_{1}-\phi_{2})_{+-}\rangle$ ($\delta$) for the LCC.

Ref.[1] has pointed out that P-odd domains on the surface of the fireball omit
same charge sign particles in the direction of the magnetic field. The
particles that escape the surface would be of the same sign while the charge
particles moving in the opposite direction would be of opposite sign. These
particles would run into the fireball and be thermalized and loss their
direction (quenched). This effect will be taken into account in the latter
part of this report.

The paper is organized in the following manner:

Sec. 1 is the introduction to correlations. Sec. 2 presents the STAR[6]
correlation data which will be used in the analysis. Sec. 3 separates the
correlations up into reaction plane dependent and independent parts with a
further separation into charge dependent and charge independent amplitudes.
Sec. 4 introduce ratios of charge dependent to total amplitudes. Sec. 5 brings
the idea of quenching into the reaction plane dependent analysis. Sec. 6
presents the summary and discussion.

##  2 $\gamma$ and $\delta$ from the Beam Energy Scan at RHIC

In this analysis we use STAR[6] data coming from charged particles produced in
Au-Au collisions at RHIC. 8M $\sqrt{s_{NN}}$= 62.4 GeV (2005), 100M at 39.0
GeV (2010), 46M at 27.0 GeV (2011), 20M at 19.6 GeV (2011), 10M at 11.5 GeV
(2010) and 4M at 7.7 GeV (2010) were used. $\gamma$ ($\langle
cos(\phi_{1}+\phi_{2}-2\Psi_{RP})\rangle$) and $\delta$ ($\langle
cos(\phi_{1}-\phi_{2})\rangle$) were extracted for like and unlike sign
charged pairs as a function of centrality. Figure 1 - 4 show scatter plots of
the correlation data. We have plotted centrality vs log($\sqrt{s_{NN}}$). In
the plots the label E is equal to $\sqrt{s_{NN}}$ and the most central
collisions are at 10% while most peripheral are at 70%. The vertical scale is
measured in units of $10^{-4}$. Figure 5 show the average $v_{2}$ plotted in
the same type of scatter plot. This $v_{2}$ is needed to extract the reaction
plane dependent and independent correlations which we define in Sec. 3.

Figure 1:  (Color) Scatter plot of the correlation(units of $10^{-4}$) of same
sign charge pairs $\langle cos(\phi_{1}+\phi_{2}-2\Psi_{RP})\rangle$
($\gamma$) calculated from Au-Au collisions with acceptance cuts of 0.15 $<$
$p_{t}$ $<$ 2 GeV/c and $|\eta|$ $<$ 1.0. The two axes are centrality vs beam
energy. The beam energy is plotted as log($\sqrt{s_{NN}}$) with data from
STAR[6] experiment at RHIC(see text). log($\sqrt{s_{NN}}$ = 64.4 GeV) = 4.1
and log($\sqrt{s_{NN}}$ = 7.7 GeV) = 2.0. Centrality ranges from most central
collisions at 10%, while most peripheral at 70%.

Figure 2:  (Color) Scatter plot of the correlation(units of $10^{-4}$) of
opposite sign charge pairs $\langle cos(\phi_{1}+\phi_{2}-2\Psi_{RP})\rangle$
($\gamma$) calculated from Au-Au collisions with acceptance cuts of 0.15 $<$
$p_{t}$ $<$ 2 GeV/c and $|\eta|$ $<$ 1.0. The two axes are centrality vs beam
energy. The beam energy is plotted as log($\sqrt{s_{NN}}$) with data from
STAR[6] experiment at RHIC(see text). log($\sqrt{s_{NN}}$ = 64.4 GeV) = 4.1
and log($\sqrt{s_{NN}}$ = 7.7 GeV) = 2.0. Centrality ranges from most central
collisions at 10%, while most peripheral at 70%.

Figure 3:  (Color) Scatter plot of the correlation(units of $10^{-4}$) of same
sign charge pairs $\langle cos(\phi_{1}-\phi_{2})\rangle$ ($\delta$)
calculated from Au-Au collisions with acceptance cuts of 0.15 $<$ $p_{t}$ $<$
2 GeV/c and $|\eta|$ $<$ 1.0. The two axes are centrality vs beam energy. The
beam energy is plotted as log($\sqrt{s_{NN}}$) with data from STAR[6]
experiment at RHIC(see text). log($\sqrt{s_{NN}}$ = 64.4 GeV) = 4.1 and
log($\sqrt{s_{NN}}$ = 7.7 GeV) = 2.0. Centrality ranges from most central
collisions at 10%, while most peripheral at 70%.

Figure 4:  (Color) Scatter plot of the correlation(units of $10^{-4}$) of
opposite sign charge pairs $\langle cos(\phi_{1}-\phi_{2})\rangle$ ($\delta$)
calculated from Au-Au collisions with acceptance cuts of 0.15 $<$ $p_{t}$ $<$
2 GeV/c and $|\eta|$ $<$ 1.0. The two axes are centrality vs beam energy. The
beam energy is plotted as log($\sqrt{s_{NN}}$) with data from STAR[6]
experiment at RHIC(see text). log($\sqrt{s_{NN}}$ = 64.4 GeV) = 4.1 and
log($\sqrt{s_{NN}}$ = 7.7 GeV) = 2.0. Centrality ranges from most central
collisions at 10%, while most peripheral at 70%.

Figure 5:  (Color) Scatter plot of the average $v_{2}$ calculated from Au-Au
collisions with acceptance cuts of 0.15 $<$ $p_{t}$ $<$ 2 GeV/c and $|\eta|$
$<$ 1.0. The two axes are centrality vs beam energy. The beam energy is
plotted as log($\sqrt{s_{NN}}$) with data from STAR[6] experiment at RHIC(see
text). log($\sqrt{s_{NN}}$ = 64.4 GeV) = 4.1 and log($\sqrt{s_{NN}}$ = 7.7
GeV) = 2.0. Centrality ranges from most central collisions at 10%, while most
peripheral at 70%.

##  3 Reaction Plane Dependent and Reaction Plane Independent Correlations

Using the $\gamma$ and $\delta$ measurements we can define reaction plane
dependent(H) and reaction plane independent(F) correlations. The
definitions[6] from Ref.[7] are written as

$\gamma=v_{2}F-H,$ (4)

and

$\delta=F+H.$ (5)

We can solve for H by using equations 4 and 5 obtaining

$H=(v_{2}\delta-\gamma)/(1+v_{2}).$ (6)

Thus F is given by

$F=\delta-H.$ (7)

###  3.1 Charge Dependent and Charge Independent Amplitudes

The CME is a charge dependent amplitude. The CME amplitude has an opposite
sign depending on whether we consider same sign charge pairs or opposite sign
charge pairs. Also the CME is a reaction plane dependent amplitude acting on
pairs which are moving along the B field perpendicular to the reaction plane.
For this analysis we will assume that the CME is the only such effect at work
when we isolate the reaction plane dependent(H) and charge dependent
amplitude.

$\rm HCME=\it(H_{ss}-H_{os})\rm/2,$ (8)

where $H_{ss}$ is the same sign charge pairs reaction plane dependent
correlation and $H_{os}$ is the opposite sign charge pairs reaction plane
dependent correlation(see Figure 6). At low log($\sqrt{s_{NN}}$ = 7.7 GeV) =
2.0 HCME is very small, while at log($\sqrt{s_{NN}}$ = 64.4 GeV) = 4.1 HCME
becomes large $\sim$.0003. The highest value of HCME $\sim$.0005 is an
isolated point at top energies(see rotated plot Figure 7). At all energies the
HCME is small for central collisions(note B field is small for central
collisions).

We can define three other amplitudes HCI(reaction plane dependent amplitude
with no charge dependence), FCD(reaction plane independent amplitude with a
charge dependence) and FCI(reaction plane independent amplitude with no charge
dependence).

$\rm HCI=\it(H_{ss}+H_{os})\rm/2,$ (9) $\rm FCD=\it(F_{ss}-F_{os})\rm/2,$ (10)

and

$\rm FCI=\it(F_{ss}+F_{os})\rm/2.$ (11)

Figure 6:  The HCME is assumed to dominate this charge sign dependent reaction
plane dependent amplitude(units of $10^{-4}$). The amplitude is calculated
from Au-Au collisions with acceptance cuts of 0.15 $<$ $p_{t}$ $<$ 2 GeV/c and
$|\eta|$ $<$ 1.0. The two axes are centrality vs beam energy. The beam energy
is plotted as log($\sqrt{s_{NN}}$) with data from STAR[6] experiment at
RHIC(see text). log($\sqrt{s_{NN}}$) = 64.4 GeV) = 4.1 and log($\sqrt{s_{NN}}$
= 7.7 GeV) = 2.0. Centrality ranges from most central collisions at 10%, while
most peripheral at 70%. HCME becomes large $\sim$.0003. The highest value of
HCME $\sim$.0005 is an isolated at top energies(see rotated plot Figure 7). At
all energies the HCME is small for central collisions(note B field is small
for central collisions).

Figure 7:  The HCME has been rotated so one can see that HCME $\sim$.0005 is
an isolated value at top energies while sitting on a plateau of $\sim$.0003 in
value. The amplitude(units of $10^{-4}$) is calculated from Au-Au collisions
with acceptance cuts of 0.15 $<$ $p_{t}$ $<$ 2 GeV/c and $|\eta|$ $<$ 1.0. The
two axes are centrality vs beam energy. The beam energy is plotted as
log($\sqrt{s_{NN}}$) with data from STAR[6] experiment at RHIC(see text).
log($\sqrt{s_{NN}}$ = 64.4 GeV) = 4.1 and log($\sqrt{s_{NN}}$ = 7.7 GeV) =
2.0. Centrality ranges from most central collisions at 10%, while most
peripheral at 70%

.

The HCI is dominate at low beam energy and most peripheral. HCI is a charge
sign independent reaction plane dependent amplitude(units of $10^{-4}$) as
defined above. At lower beam energies and most peripheral the shower of charge
particles from the Au-Au collision defines the reaction plane. This amplitude
is negative which is driven by the back to back nature of the shower of
particles due to momentum conservation. At the lowest beam energy the
amplitude scales as 1/multiplicity(see Figure 8).

The FCD amplitude is a very complex two particle correlation where the same
charge sign pair correlation has the opposite charge sign pair correlation
subtracted from it. At the highest beam energies opposite sign particles are
correlated with each other moving together and creating a positive sign(see
Figure 4 and Ref.[8]). This positive correlation is stronger in the most
peripheral collisions. The same charge sign correlation is negative arising
from back to back correlation between like sign particles. The difference thus
is an overall negative correlation which is strongest for the most peripheral
collisions. When we plot the FCD in our usual scatter plot(see Figure 9) this
high beam energy behavior is hidden by the more complicated action at lower
beam energies. In Figure 10 we have rotated the plot so we can see this
negative value at the most peripheral which decreases in absolute value as
1/multiplicity with centrality.

The FCI is dominate at low beam energy and most peripheral. FCI is a charge
sign independent reaction plane independent amplitude(units of $10^{-4}$) as
defined above. At lower beam energies and most peripheral the shower of charge
particles from the Au-Au collision have a back to back nature due to momentum
conservation. The same charge sign pairs follow this basic behavior. Opposite
charge sign pairs go against this behavior and cancel out this display of
momentum conservation except for the most peripheral and lowest beam energy
where momentum conservation is the strongest effect giving the largest
negative amplitude of $\sim$-.0035 in this analysis(see Figure 11).

Figure 8: The HCI is dominate at low beam energy and most peripheral. HCI is a
charge sign independent reaction plane dependent amplitude(units of
$10^{-4}$). The amplitude is calculated from Au-Au collisions with acceptance
cuts of 0.15 $<$ $p_{t}$ $<$ 2 GeV/c and $|\eta|$ $<$ 1.0. The two axes are
centrality vs beam energy. The beam energy is plotted as log($\sqrt{s_{NN}}$)
with data from STAR[6] experiment at RHIC(see text). log($\sqrt{s_{NN}}$ =
64.4 GeV) = 4.1 and log($\sqrt{s_{NN}}$ = 7.7 GeV) = 2.0. Centrality ranges
from most central collisions at 10%, while most peripheral at 70%. HCI has a
large negative value $\sim$-.0025. This is a back to back pair correlation of
momentum conservation. At the lowest beam energy the scaling is one of
1/multiplicity.

Figure 9:  The FCD amplitude is a very complex two particle pair correlation
where the same charge sign pair correlation has the opposite charge sign pair
correlation subtracted from it. When we plot the FCD in our usual scatter plot
the high beam energy behavior is hidden by the more complicated action at
lower beam energies. In Figure 10 we have rotated the plot to so one can see
the hidden behavior. FCD is a charge sign dependent reaction plane independent
amplitude(units of $10^{-4}$). The amplitude is calculated from Au-Au
collisions with acceptance cuts of 0.15 $<$ $p_{t}$ $<$ 2 GeV/c and $|\eta|$
$<$ 1.0. The two axes are centrality vs beam energy. The beam energy is
plotted as log($\sqrt{s_{NN}}$) with data from STAR[6] experiment at RHIC(see
text). log($\sqrt{s_{NN}}$ = 64.4 GeV) = 4.1 and log($\sqrt{s_{NN}}$ = 7.7
GeV) = 2.0. Centrality ranges from most central collisions at 10%, while most
peripheral at 70%. FCD has a large negative value $\sim$-.0018 for lowest beam
energy and centrality of 55%.

Figure 10:  As part of the FCD amplitude at the highest beam energies the
opposite sign particles pairs are correlated with each other moving together
and creating a positive sign(see Figure 4 and Ref.[8]). This positive
correlation is stronger in the most peripheral collisions but has a negative
contribution to FCD. The same charge sign pair correlation is negative arising
from back to back correlation between like sign particles and also makes the
FCD more negative for peripheral collisions. Thus this overall negative
correlation is strongest for the most peripheral collisions($\sim$-.0018).
When we plot the FCD in our usual scatter plot(see Figure 9) this high beam
energy behavior is hidden by the more complicated action at lower beam
energies. In this figure we have rotated the plot so we can see this negative
value at the most peripheral which decreases in absolute value as
1/multiplicity with centrality. The FCD is a charge sign dependent reaction
plane independent amplitude(units of $10^{-4}$). The amplitude is calculated
from Au-Au collisions with acceptance cuts of 0.15 $<$ $p_{t}$ $<$ 2 GeV/c and
$|\eta|$ $<$ 1.0. The two axes are centrality vs beam energy. The beam energy
is plotted as log($\sqrt{s_{NN}}$) with data from STAR[6] experiment at
RHIC(see text). log($\sqrt{s_{NN}}$ = 64.4 GeV) = 4.1 and log($\sqrt{s_{NN}}$
= 7.7 GeV) = 2.0. Centrality ranges from most central collisions at 10%, while
most peripheral at 70%.

Figure 11:  The FCI is dominate at low beam energy and most peripheral. FCI is
a charge sign independent reaction plane independent amplitude(units of
$10^{-4}$). At lower beam energies and most peripheral the shower of charge
particles from the Au-Au collision have a back to back nature due to momentum
conservation. The same charge sign pairs follow this basic behavior. Opposite
charge sign pairs go against this behavior and cancel out this display of
momentum conservation except for the most peripheral and lowest beam energy
where momentum conservation is the strongest effect giving $\sim$-.0035 the
largest negative amplitude in this analysis. The amplitude is calculated from
Au-Au collisions with acceptance cuts of 0.15 $<$ $p_{t}$ $<$ 2 GeV/c and
$|\eta|$ $<$ 1.0. The two axes are centrality vs beam energy. The beam energy
is plotted as log($\sqrt{s_{NN}}$) with data from STAR[6] experiment at
RHIC(see text). log($\sqrt{s_{NN}}$ = 64.4 GeV) = 4.1 and log($\sqrt{s_{NN}}$
= 7.7 GeV) = 2.0. Centrality ranges from most central collisions at 10%, while
most peripheral at 70%.

Figure 12:  The ratio(RF) of the charge dependent to the square root of the
sum of the squares of the charge dependent(FCD) and the charge independent
amplitudes(FCI). This ratio RF for reaction plane independent amplitudes for
the higher beam energies the difference between the same charge sign pair
correlation and the opposite charge sign pair correlation becomes vary large
leading to dominance of charge dependent amplitudes. The ratio is calculated
from Au-Au collisions with acceptance cuts of 0.15 $<$ $p_{t}$ $<$ 2 GeV/c and
$|\eta|$ $<$ 1.0. The two axes are centrality vs beam energy. The beam energy
is plotted as log($\sqrt{s_{NN}}$) with data from STAR[6] experiment at
RHIC(see text). log($\sqrt{s_{NN}}$ = 64.4 GeV) = 4.1 and log($\sqrt{s_{NN}}$
= 7.7 GeV) = 2.0. Centrality ranges from most central collisions at 10%, while
most peripheral at 70%.

##  4 Ratio Charge Dependent Amplitudes to Total

For the higher beam energies the difference between the same charge sign pair
correlation and the opposite charge sign pair correlation becomes vary large
leading to dominance of charge dependent amplitudes. We can express this
dominance by a ratio of the charge dependent to the square root of the sum of
the squares of the charge dependent and the charge independent amplitudes.
This ratio for the F or reaction plane independent amplitudes is given by

$RF=FCD/\sqrt{FCD^{2}+FCI^{2}}.$ (12)

RF is shown in Figure 12 where for high beam energy this ratio is 1.0 dropping
to 0.3 at the lowest energy and most peripheral.

The Chiral Magnetic Effect (HCME) is the dominate reaction plane and charge
dependent amplitude for peripheral collision where there will be a magnetic
field and high enough energy to have deconfined quarks. This ratio(RH) of the
charge dependent to the square root of the sum of the squares of the charge
dependent and the charge independent amplitudes is

$RH=HCME/\sqrt{HCME^{2}+HCI^{2}}.$ (13)

RH is shown in Figure 13 where the magnetic field is large and the energy is
high enough for deconfined quarks the ratio is 1.0 dropping to zero at central
events and low energies.

##  5 Quenching of the CME

Ref.[1] has pointed out that P-odd domains on the surface of the fireball omit
same charge sign particles in the direction of the magnetic field. The
particles that escape the surface would be of the same sign while the charge
particles moving in the opposite direction would be of opposite sign. These
particles would run into the fireball and be thermalized and loss their
direction (quenched). This implies that $H_{ss}$ would be unaffected by
quenching, thus

$H_{ss}=HCME+HCI.$ (14)

If we would consider a quenching factor q such that when q equals 1 we have
maximum quenching, $H_{os}$ becomes

$H_{os}=-(1-q)HCME+HCI.$ (15)

Even though the CME is mainly in the same charge sign pairs the definition of
the charge sign dependent reaction plane dependent amplitude remains the same.
When there is quenching the relationship changes and the HCME becomes

$(H_{ss}-H_{os})/2=\left(2-q\over 2\right)HCME.$ (16)

This causes in the maximum quenching case the HCME to be twice the value of
the charge sign dependent reaction plane dependent amplitude(see Figure 14).
the charge sign independent reaction plane dependent amplitude now picks up a
component of the CME,

$(H_{ss}+H_{os})/2=\left(q\over 2\right)HCME+HCI.$ (17)

However this has a very small effect on this amplitude(see Figure 15). Finally
The ratio(RH) of the charge dependent to the square root of the sum of the
squares of the charge dependent and the charge independent amplitudes is shown
in Figure 16. We see that this ratio appears to have a nice gaussian shape
which is consistent with there being quenching present in the reacting
systems.

Figure 13:  The ratio(RH) of the charge and reaction plane dependent to the
square root of the sum of the squares of the charge dependent(HCME) and the
charge independent amplitudes(HCI). This ratio RH for reaction plane dependent
amplitudes for the higher beam energies(deconfined quarks) and peripheral
collision (magnetic field) becomes vary large leading to dominance of charge
dependent amplitudes. The ratio is calculated from Au-Au collisions with
acceptance cuts of 0.15 $<$ $p_{t}$ $<$ 2 GeV/c and $|\eta|$ $<$ 1.0. The two
axes are centrality vs beam energy. The beam energy is plotted as
log($\sqrt{s_{NN}}$) with data from STAR[6] experiment at RHIC(see text).
log($\sqrt{s_{NN}}$ = 64.4 GeV) = 4.1 and log($\sqrt{s_{NN}}$ = 7.7 GeV) =
2.0. Centrality ranges from most central collisions at 10%, while most
peripheral at 70%.

##  6 Summary and Discussion

We use the STAR[6] correlation data in an analysis that can separate the
correlations up into reaction plane dependent and independent parts with a
further separation into charge dependent and charge independent amplitudes.
This gives us a clean separation of the The Chiral Magnetic Effect into an
amplitude HCME. This amplitude is isolated to top energies and peripheral
collisions where the magnetic is field large and quarks are not confined. We
also show that the idea of quenching is supported by the data best expressed
by the ratio RH. This ratio(RH) of the charge dependent to the square root of
the sum of the squares of the charge dependent and the charge independent
amplitudes which we show in Figure 16 appears to have a nice gaussian shape.
This is consistent with there being quenching present in the reacting systems.

Figure 14:  The HCME is assumed to dominate this charge sign dependent
reaction plane dependent amplitude(units of $10^{-4}$). Here we plot the HCME
which would exist if there was maximum quenching. The amplitude is calculated
from Au-Au collisions with acceptance cuts of 0.15 $<$ $p_{t}$ $<$ 2 GeV/c and
$|\eta|$ $<$ 1.0. The two axes are centrality vs beam energy. The beam energy
is plotted as log($\sqrt{s_{NN}}$) with data from STAR[6] experiment at
RHIC(see text). log($\sqrt{s_{NN}}$) = 64.4 GeV) = 4.1 and log($\sqrt{s_{NN}}$
= 7.7 GeV) = 2.0. Centrality ranges from most central collisions at 10%, while
most peripheral at 70%. HCME becomes large $\sim$.0006. The highest value of
HCME $\sim$.0010 is an isolated at top energies. At all energies central
collisions the HCME is small(note B field is small at central collisions).

Figure 15: The HCI is dominate at low beam energy and most peripheral. Since
this is the region where the CME is small quenching has little effect on this
amplitude. HCI is a charge sign independent reaction plane dependent
amplitude(units of $10^{-4}$). The ratio is calculated from Au-Au collisions
with acceptance cuts of 0.15 $<$ $p_{t}$ $<$ 2 GeV/c and $|\eta|$ $<$ 1.0. The
two axes are centrality vs beam energy. The beam energy is plotted as
log($\sqrt{s_{NN}}$) with data from STAR[6] experiment at RHIC(see text).
log($\sqrt{s_{NN}}$ = 64.4 GeV) = 4.1 and log($\sqrt{s_{NN}}$ = 7.7 GeV) =
2.0. Centrality ranges from most central collisions at 10%, while most
peripheral at 70%. HCI has a large negative value $\sim$-.0025. This is a back
to back pair correlation of momentum conservation. At the lowest beam energy
the scaling is one of 1/multiplicity.

Figure 16:  The ratio(RH) of the charge and reaction plane dependent to the
square root of the sum of the squares of the charge dependent(HCME) and the
charge independent amplitudes(HCI). This ratio RH for reaction plane dependent
amplitudes for the higher beam energies(deconfined quarks) and peripheral
collision (magnetic field) becomes vary large leading to dominance of charge
dependent amplitudes. The amplitude is calculated from Au-Au collisions with
acceptance cuts of 0.15 $<$ $p_{t}$ $<$ 2 GeV/c and $|\eta|$ $<$ 1.0. The two
axes are centrality vs beam energy. The beam energy is plotted as
log($\sqrt{s_{NN}}$) with data from STAR[6] experiment at RHIC(see text).
log($\sqrt{s_{NN}}$ = 64.4 GeV) = 4.1 and log($\sqrt{s_{NN}}$ = 7.7 GeV) =
2.0. Centrality ranges from most central collisions at 10%, while most
peripheral at 70%.

##  7 Acknowledgments

This research was supported by the U.S. Department of Energy under Contract
No. DE-AC02-98CH10886.

## References

  * [1] D.E. Kharzeev, L.D. McLerran and H.J. Warringa, Nucl. Phys. A 803 (2008) 227.
  * [2] STAR Collaboration, B.I. Abelev et al., Phys. Rev. Lett. 103 (2009) 251601, Phys. Rev. C 81 (2010) 054908.
  * [3] S.A. Voloshin, Phys. Rev. C 70 (2004) 057901.
  * [4] A. Bzdak, V. Koch and J. Liao, Phys. Rev. C 83 (2011) 014905.
  * [5] S. Schlichting and S. Pratt, arXiv:1005.5341[nucl-th].
  * [6] STAR Collaboration, L. Adamczyk et al., Phys. Rev. Lett. 113 (2014) 052302.
  * [7] A. Bzdak, V. Koch and J. Liao, Lect. Notes Phys. 871 (2013) 503\.
  * [8] J. Adams et al., Phys. Lett B 634 (2006) 347, STAR Collaboration, M. Daugherity et al., arXiv:0611.032[nucl-ex], STAR Collaboration, B.I. Abelev et al., arXiv:0806.0513[nucl-ex].


# CoColm: COmplex COmmonsense Enhanced
Language Model

Changlong Yu, $\\*$ Hongming Zhang∗, Yangqiu Song, Wilfred Ng
Department of Computer Science and Engineering
The Hong Kong University of Science and Technology
{cyuaq, hzhangal, yqsong<EMAIL_ADDRESS>
Equal Contribution.

###### Abstract

Large-scale pre-trained language models have demonstrated strong knowledge
representation ability. However, recent studies suggest that even though these
giant models contains rich simple commonsense knowledge (e.g., bird can fly
and fish can swim.), they often struggle with the complex commonsense
knowledge that involves multiple eventualities (verb-centric phrases, e.g.,
identifying the relationship between “Jim yells at Bob” and “Bob is upset”).
To address this problem, in this paper, we propose to help pre-trained
language models better incorporate complex commonsense knowledge. Different
from existing fine-tuning approaches, we do not focus on a specific task and
propose a general language model named CoColm. Through the careful training
over a large-scale eventuality knowledge graphs ASER, we successfully teach
pre-trained language models (i.e., BERT and RoBERTa) rich complex commonsense
knowledge among eventualities. Experiments on multiple downstream commonsense
tasks that requires the correct understanding of eventualities demonstrate the
effectiveness of CoColm.

##  1 Introduction

Recently, large-scale pre-trained language representation models (LMs) [1, 2]
have demonstrated the strong ability to discover useful linguistic properties
of syntax and remember an impressive amounts of knowledge with self-supervised
training over a large unlabeled corpus [3]. On top of that, with the help of
the fine-tuning step, LMs can learn how to use the memorized knowledge for
different tasks, and thus achieve outstanding performance on many downstream
natural language processing (NLP) tasks.

Query |  Answer
---|---
Birds can [MASK]. |  fly
Cars are used for [MASK]. |  transport
Jim yells at Bob, [MASK] Jim is upset. |  but
Jim yells at Bob, [MASK] Bob is upset. |  but
Table 1: Exploring knowledge contained in pre-trained language models
following LAMA [3]. Exploiting queries and prediction returned by BERT-large
are presented. Semantically plausible and implausible prediction are indicated
with blue and red colors.

As discussed in [4], while language models has already captured rich
knowledge, they often only perform well when the semantic unit is a single
token while poorly when the semantic unit is more complex (e.g., a multi-token
named entity which is a noun phrase or an eventuality which is a verb-centric
phrase). For example, as shown in Table 1, if we follow LAMA [3] to analyze
the knowledge contained in BERT-large [1] with a token prediction task, we can
find out that BERT can understand that bird can fly, and car is used for the
transportation, but it fails to understand the relation between “Jim yells at
Bob” and relevant eventualities. An important reason behind this is that
current language models heavily rely on the token-level masked language model
(MLM) as the loss function, which can effectively represent and memorize the
token-level co-occurrence information but struggle at perceiving multi-token
concepts. For example, to correctly understand the relation between “Jim yells
at Bob” and “Bob is upset”, the model needs perceive these two eventualities
as independent semantic units while current LMs often failed to do so. Even
though BERT introduced an extra next sentence prediction task to capture more
complex semantics, its effect is not very significant because different from
tokens, entities, or eventualities, the semantics of a sentence is often the
combination of multiple semantic units rather than a single one [5].

To address this problem and equip LMs with complex and accurate human
knowledge, [4] proposed to regularize the language model with key-value based
entity representations from external knowledge graphs. While such approach has
been proved effective merging accurate structured knowledge into the language
models, it still has two limitations: (1) It is restricted to a fixed set of
named entities, and cannot be easily generated to more complex semantic
components (i.e., eventualities) because there are enormous eventualities; (2)
The key-value structure can only encode one-hop knowledge and thus the
language models still struggle to understand complex knowledge that involves
high-order inference. As a result, it can only be used to model entity-based
factual knowledge rather than more complex commonsense knowledge.

In this paper, to effectively inject complex commonsense knowledge about
eventualities into pre-trained language representation models, we propose a
knowledge injection framework CoColm. Through carefully controlled walk over a
large-scale eventuality knowledge graph, ASER [6], we manage to collect rich
knowledge about the discourse relations among eventualities (e.g., “being
hungry” can cause “eat food” and “being hungry” often happens at the same time
as “being tired”). After that, we design a special masking strategy to help
language representation models to view each eventuality as a whole. By doing
so, we successfully inject rich eventuality knowledge into pre-trained
language representation models. As ASER is also automatically extracted from
unlabeled corpus, the proposed pipeline is cheap and scalable. Experiments on
multiple eventuality-relevant natural language understanding tasks show that
the proposed solution can help improve the performance of all four language
models (i.e., BERT-base, BERT-large, RoBERTa-base, and RoBERTa-large).
Extensive analysis are conducted to show the effect and contribution of all
components in CoColm. The main contributions of this paper are as follows:

Figure 1: The overall framework of CoColm. On top of base pre-trained language
models, complex commonsense knowledge from the eventuality sequences is
injected by fine-tuning masked language models and auxiliary discourse tasks.

  * •

We propose CoColm, a new contextualized language model enhanced by complex
commonsense knowledge from high-order discourse relations. CoColm is trained
to predict the whole eventuality among the sequences using a large-scale
eventuality KG.

  * •

We introduce two auxiliary discourse tasks to help incorporate discourse-
related knowledge into pre-trained language models, which complement the
special eventuality masking strategy.

  * •

CoColm achieves stronger performance than the baseline LMs on multiple
datasets that require the understanding of complex commonsense knowledge about
eventualities.

Our code and pre-trained models are publicly available at
https://github.com/HKUST-KnowComp/CoCoLM.

##  2 Methods

The overall framework of CoColm is presented in Figure 1. Given a pre-trained
language model, we inject complex commonsense knowledge about eventualities
into the pre-trained model. Specifically, we first generate eventuality
sequences based on carefully controlled walks over existing eventuality
knowledge graphs, and then use the sequences as the context to help LMs handle
eventualities. Besides the original masked language model loss, we also
introduce the eventuality-based masking loss function and several auxiliary
tasks to assist the training. As this new language model training is not task
specific, the resulting language model can be easily applied to any downstream
tasks via another task-specific fine-tuning. Details about all steps are as
follows.

###  2.1 Eventuality Sequence Generation

As aforementioned, we leverage ASER, which uses eventualities as nodes and the
discourse relations as edges, as the eventuality knowledge resource. ASER
contains rich eventuality knowledge such as “being hungry” and “being tired”
often happen together and people often “make a call” before they go.
Interestingly, beyond the single edges, higher-order connections over ASER can
also reflect insightful eventuality knowledge. For example, “sleep” and “go”
are not likely to happen at the same time because “sleep” can be caused by
“being tired” and there exist a contrast connection between “being tired” and
“go”. To include the higher-order knowledge into the model, we propose to take
the whole graph into consideration rather than just the edges. At the same
time, considering the large-scale of ASER, it is infeasible to input the whole
graph into a graph model. Motivated by DeepWalk [7], we randomly sample paths
to simulate the overall graph structure.

Type |  Sequences
---|---
Temporal |  The police said. Succession The father of boy be charged murder yesterday. Co-occurrence Eleven people were arrested yesterday in raid.
Casual |  They speak. Condition They have a interest. Reason they come there.
Others |  The police said. Conjunction He was taken to house. Contrast Officer Schwarz held him.
Table 2: Sampled eventuality sequences with different relations from ASER.
Discourse relations are highlighted with pink.

Given the initial knowledge graph $\mathcal{G}=(\mathcal{E},\mathcal{R})$,
where $\mathcal{E}$ is the eventuality set and $\mathcal{R}$ is the relation
set, we conduct the weighted random walk based on the edge weights over
$\mathcal{G}$ to sample eventuality paths. We denote each path as
$(E_{0},r_{0},E_{1},r_{1},...,r_{l-1},E_{l})$, where $E$ means an eventuality,
$r$ a discourse edge connecting two eventualities, and $l$ the numbers of
eventualities along the sequence. To convert the sampled sentence into a token
list, we keep all words in each event as a sentence and use representative
connectives for each discourse relation to connect them. As ASER is
automatically extracted from raw documents, it may contains noise. To minimize
the influence of the noise and improve the quality of sampled paths, we
required the selected paths to fulfill the following requirements:

  1. 1.

To filter out rare eventualities, the frequency of starting eventualities has
to be larger than five.

  2. 2.

Other than the relations that have the transitive property (e.g., Precedence,
Result), each selected path should not contain successive edges with repeated
relations.

  3. 3.

To make sampled sequences more informative, we manually improve the sampling
probability of selecting sub-sequence patterns like “$E_{i}$ Condition $E_{j}$
Reason $E_{k}$”. Since it has been proven that if-then rules [8] and if-then-
because rules [9] are crucial for reasoning.

Implementation details and careful analysis are presented in Section 4 and
several examples are shown in the Table 2.

Figure 2: Illustration of CoColm-(Complex commonsense pre-training stage).
Given an eventuality sequence, it is either masked by the whole eventuality
masking (in blue) or discourse connective masking strategy (in pink). Besides
the regular masked language model, the discourse relation labels are jointly
predicted for masked connective tokens (on $\mathbf{x}_{4}$, $\mathbf{x}_{8}$
and $\mathbf{x}_{12}$). Co-occurrence prediction (on $\mathbf{x}_{1}$) is
conducted for both masking strategies.

###  2.2 Eventuality-Level Mask

Masking strategy plays a crucial role for the training of language
representation models. Besides the random token-level masking strategy, many
other masking strategies have been explored by previous literature such as
the-whole-word masking [1, 10], named entity masking [11] or text span masking
[12]. Similarly, to effectively help the model view each eventuality as an
independent semantic unit, we propose the following two masking strategies:
(1) Whole Eventuality Masking: Similar to the whole word masking or entity
masking strategies, the whole eventuality masking aims to reduce the prior
biases of eventuality tokens. For example, given an eventuality sequence “I
feel sleepy because I drink a cup of [MASK].”, BERT would easily predict
“coffee” or “tea” because of the prior knowledge of “cup of” inside the
eventuality. Instead of that, masking the whole “I drink a cup of coffee”
would encourage the prediction to treat each eventuality as an independent
semantic unit and focus on the relations between them. For each sampled
sequence, we randomly mask at most one eventuality to fulfill the masking
budget, which is typically 25% of the sequence token length. (2) Discourse
Connective Masking: Besides masking the eventualities, to effectively encode
the discourse information, we also tried masking the discourse connectives.

Examples of the two masking strategies are shown in Figure 2. It is worth
mentioning that for each sequence, we only randomly select one type of masking
strategy to guarantee that enough information is kept in the left tokens for
the prediction. The formal masking strategy is defined as follows. Given a
tokenized sampled sequence $X=(x_{1},x_{2},...,x_{n})$, after randomly passing
several tokens, we pass it to a transformer encoder [13] and denote the
resulting vector representations as
$\mathbf{x_{1}},\mathbf{x_{2}},...\mathbf{x_{n}}$. The training loss
$\mathcal{L}_{mlm}$ can thus be defined as:

$\mathcal{L}_{mlm}=-\frac{1}{|\mathbf{M}|}\sum_{i\in\mathbf{M}}{\rm
log}P(x_{i}|\mathbf{x_{i}}),$ (1)

where M means the set of masked tokens following the aforementioned masking
strategies.

###  2.3 Auxiliary Tasks

A limitation of the MLM loss is that the prediction is over the entire
vocabulary, and as a result the model could not effectively learn the
connection between eventualities and connective words. To remedy this and
force the model to learn the discourse knowledge, we propose to add an
additional classification layer after the last layer of transformer encoder
and it feeds the output vector $\mathbf{x_{i}}$ of connective token $x_{i}$
into a softmax layer over the set of discourse relation labels as follows.

$P(l_{i}|\mathbf{x_{i}})={\rm softmax}(\mathbf{x_{i}}\mathbf{W}+\mathbf{b}),$
(2) $\mathcal{L}_{rel}=-\sum_{i\in\mathbf{M}_{R}}\log
P(l_{i}=\widetilde{l}_{i}|\mathbf{x}_{i}),$ (3)

where $\mathbf{M}_{R}$ is the index set of masked discourse connective tokens
(e.g., because, and, so) in Figure 2, $l_{i}$ is the predicted discourse
relation label, and $\widetilde{l}_{i}$ the label provided by ASER.
$\mathbf{W}$ and $\mathbf{b}$ are trainable parameters.

Besides the aforementioned discourse relations, ASER also provides the Co-
Occurrence relations between eventualities, which means that two eventualities
appear in the same sentence, but there is no explicit discourse relations
between them. Even though compared with discourse relations, the Co-Occurrence
relations are less informative, we still think it reflects rich knowledge
about eventualities. Motivated by this, we propose another auxiliary task to
help the model to learn such knowledge. Specially, given an eventuality
sequence $S=(E_{0},r_{0},E_{1},r_{1},...,r_{l-1},E_{l})$ and an eventuality
$E_{c}$, we format the input111The starting and separation tokens are based on
the base model. For example, if we use BERT as the base model, we add “[CLS]”
and “[SEP]”, and if we use RoBERTa, we add “¡s¿” and “¡/s¿”. All notations in
the rest of this paragraph are based on BERT. as “[CLS] $S$ [SEP] $E_{c}$
[SEP]”. We set 50% of the time $E_{c}$ to be the positive co-occurred
eventuality with one of eventualities in the sequence while 50% of the time
$E_{c}$ is randomly sampled negative in ASER. Similar to the next sentence
prediction in the original BERT, on top of the vector representation of token
[CLS], i.e., $\mathbf{x}_{cls}$, we add another classification layer to
predict whether the Co-Occurrence relations appears or not. The training
objective $\mathcal{L}_{occur}$ for binary classification is similar to
$\mathcal{L}_{rel}$:

$\mathcal{L}_{occur}=-\log P(l_{i}=\widetilde{l}_{i},|\mathbf{x}_{cls}),$ (4)

where $\widetilde{l}_{i}$ is the true co-occurrence label (positive or
negative) for the sequence.

Merging all three losses together, we can then define the overall loss
function $\mathcal{L}$ as:

$\mathcal{L}=\mathcal{L}_{mlm}+\mathcal{L}_{rel}+\mathcal{L}_{occur}.$ (5)

Figure 3: The distribution of lengths along with relation edges for generated
eventuality sequences.

##  3 Implementation Details

In this work, we use the released ASER-core
version222https://github.com/HKUST-KnowComp/ASER extracted from multi-domain
multi-resource corpora, which contains over 27.6 million eventualities and 8.8
million relations. We follow the heuristic rules in the Sec. 2.1 to sample
eventuality sequences for pre-training. Overall we generated 4,901,511
eventuality sequences, ranging from one to five hops and the one-hop sequence
means the direct (first-order) edge in the ASER. We also discard some edges
with uninformative relation types such as Co-Occurrence except those used for
auxiliary tasks and down-sample eventuality nodes with extremely high
frequency such as I see. The sequence distribution over different lengths is
shown in Figure 3.

We select BERT-base, BERT-large [1], RoBERTa-base, and RoBERTa-large [2] as
the base language model. All models are implemented based with the Huggingface
library [14]. For the continual pre-training phrase, we use the Adam optimizer
for 10 epochs with batch size 128, learning rate 1e-5 and weight decay 0.01.
Considering the relative longer span of masked eventualities, we enlarge the
masking proportion from 15% to 25%.

Dataset |  Type |  # Train |  # Dev |  # Test |  Example
---|---|---|---|---|---
ROCStories |  Narrative |  1,771 |  100 |  1,871 |  Context: The Mills next door had a new car. The car was stolen during the weekend. They came to my house and asked me if I knew anything. I told them I didn’t, but for some reason they suspected me. Positive Ending: They called the police to come to my house. Negative Ending: They liked me a lot after that.
MATRES |  Temporal |  230* |  25 |  20 |  Fidel Castro {$e_{1}$: invited} John Paul to {$e_{2}$: come} for a reason. Label: BEFORE
COPA |  Causal |  400 |  100 |  500 |  Premise: I knocked on my neighbor’s door. Positive Hypothesis: My neighbor invited me in. Negative Hypothesis: My neighbor left his house.
Table 3: The statistics and examples for all commonsense evaluation datasets.
The tasks are binary or multiple classification problems. Note the dataset of
MATRES is split at the article level as in the previous work [15].

##  4 Experiments

In this section, we conduct experiments on three widely used downstream tasks
that require the correct understanding of complex commonsense knowledge about
events:

  1. 1.

ROCStories [16] is widely used for story comprehension tasks such as Story
Cloze Test. It contains 98,162 five-sentence coherent stories as the unlabeled
training dataset, 1,872 four-sentence story contexts along with two candidate
ending sentences in the development and test datasets. We follow the dataset
split for the story ending prediction task in [17].

  2. 2.

MATRES [18] is a pairwise event temporal ordering prediction dataset, where
each event pair in one document is annotated with a temporal relation (Before,
After, Equal, Vague). It contains 13,577 event pairs extracted from 256
English documents.

  3. 3.

COPA [19] is a binary-choice commonsense causal reasoning task, which requires
models to predict which the candidate hypothesis is the plausible effect/cause
of the given premise. We follow the training/dev/test split in SuperGLUE [20].

Statistics and examples of the three selected datasets are presented in Table
3. We implemented the experiments with Huggingface [14] and run experiments on
eight Nvidia V100 32GB GPUs. For all experiments, we set the learning rate to
be 1e-5, and maximize the sequence length and batch size such that they can
fit into GPU memory.

For the ROCStories task, as mentioned in [21], there is strong bias about the
human-created negative endings such that the model can distinguish the
positive and negative endings without seeing the first four events. Even
though [21] tried to filter the annotations, the bias still cannot be fully
relieved. As a result, to clearly show the effect of adding complex knowledge
about events into the LM, besides the most widely used supervised setting, we
also report the performance of a debiased setting, where the model randomly
selects events from other stories as the negative ending during the training.
The debiased setting is indicated with “D”. Following previous works, we
report accuracy for the ROCStories, MATRES and COPA tasks. For MATRES, we also
report the metric of F1 by considering the task as general relation extraction
and treating the label of vague as no relation [22]. All models are trained
until converge on the training set and the best model on the dev set is
selected to be evaluated333The only exception is the supervised setting of
ROCStories. As only the development and testing set are provided with the
human-created negative endings, following previous works, we use development
as the training set and directly test the converged model on the testing set..

##  5 Experimental Results

The experimental results are presented in Table 4, from which we can see that
CoColm consistently outperform all the baselines on all three different
commonsense tasks, especially on the de-biased setting of ROCStories. Besides
that we can make the following observations:

  1. 1.

For the ROCStories dataset, Compared with the original supervised setting, the
de-biased setting is more challenging for all models, which helps verify our
assumptions that previous models benefit from the bias.

  2. 2.

The improvement of our model is more significant on ROCStories and COPA rather
than MATRES, which is mainly because events in MATRES are associated with the
context and many temporal relations have to be inferred with the local context
rather than memorized by the language model. As a comparison, Both the
ROCStories and COPA do not have any extra context, and thus require the pre-
trained LM to know the essential knowledge to solve those problems.

In the rest of this section, we conduct extensive experiments and case studies
to demonstrate the contribution of different components. For the experiment
efficiency, in all analysis experiments, we use BERT-large as the base
language model and ROCStories as the evaluation dataset.

###  5.1 Ablation Study

From the results in Table 5, we can see that all components contribute to the
final success of our model, especially the Co-Occurrence relation. This result
proves our previous assumption that even though compared other discourse
relations (e.g., Before and Cause), the Co-occurrence relations has relatively
weaker semantic, it still can help models to better understand events due to
its large scale.

| ROCStories | MATRES | COPA
---|---|---|---
Model | Accuracy | Accuracy (D) | Accuracy | F1 | Accuracy
BERT-base | 52.9 | 52.9 | 71.5 | 77.2 | 69.8
CoColm (BERT-base) | 84.2 | 65.2 | 72.8 | 77.8 | 73.8
BERT-large | 88.9 | 69.1 | 73.5 | 78.9 | 70.6
CoColm (BERT-large) | 91.9 | 71.2 | 73.9 | 79.2 | 75.8
RoBERTa-base | 93.3 | 73.2 | 74.0 | 79.2 | 85.4
CoColm (RoBERTa-base) | 94.1 | 75.2 | 74.2 | 79.8 | 86.2
RoBERTa-large | 97.4 | 88.1 | 75.2 | 81.0 | 91.6
CoColm (RoBERTa-large) | 97.7 | 89.2 | 75.3 | 81.6 | 91.8
Table 4: All three commonsense task evaluation results. The best performance is highlighted in bold. Note our proposed CoColm with RoBERTa-large achieves the best scores over all the tasks. Method | Accuracy | $\Delta$ | Accuracy (D) |  $\Delta$ (D)
---|---|---|---|---
CoColm (BERT-large) | 91.9 | - | 71.2 | -
\- Co-Occurrence Loss | 91.3 | -0.6 | 70.4 | -0.8
\- Eventuality Mask | 91.1 | -0.8 | 70.2 | -1.0
\- Relation Loss | 90.5 | -1.4 | 69.6 | -1.6
\- Co-Occurrence Relation | 90.3 | -1.6 | 69.3 | -1.9
Table 5: Ablation study on ROCStories test set by removing different model
components.

###  5.2 Effect of Different Event Knowledge Resources

Besides ASER [6], another important event knowledge resource is ATOMIC [8],
which is a crowdsourced commonsense knowledge graph that contains nine human-
defined relations between daily events. As ATOMIC is a bipolar graph and thus
we could not find a multi-hop path, we did not select it as the event
knowledge resource in our final model. In this experiment, we report the
performance of using ATOMIC as the event knowledge resource. As ATOMIC does
not contain the multi-hop knowledge, we also add ASER (single-hop), which only
uses the single-hop knowledge in ASER, as another baseline approach.

The overall results are shown in Table 6, from which we can see that there is
still a notable gap between ATOMIC and our final model. At the same time, we
can also see that ATOMIC can outperform the single-hop version of ASER. Based
on these two observations, we can make the following conclusions: (1) Compared
with ASER, ATOMIC is cleaner because it is created with human annotations; (2)
Multi-hop knowledge is crucial for LMs to understand events. We leave how to
combine ATOMIC and ASER to get more high-quality multi-hop event knowledge as
our future work.

Method | Accuracy | $\Delta$ | Accuracy (D) |  $\Delta$ (D)
---|---|---|---|---
ASER (Multi-hop) | 91.9 | - | 71.2 | -
ATOMIC (Single-hop) | 88.2 | -3.7 | 68.2 | \- 3.0
ASER (Single-hop) | 85.4 | -6.5 | 67.5 | -3.7
Table 6: Effect of different event knowledge resources.

###  5.3 Case Study

We present the case study in Table 7 and use a probing approach to further
investigate the reason for our success. Similar to [3], we put a “[MASK]”
token between two events and try to ask the model to predict the connective.
Take the case from COPA dataset as an example, connectives predicted by CoColm
clearly show the effect relation between two events. However predictions from
the baseline models reveal weaker (temporal, conjunction) or wrong (contrast)
relations. Similar observations could be drawn from another two datasets.
These observations show that compared with the original LM, CoColm manages to
memorize richer event knowledge.

Dataset |  Example |  BERT [MASK] |  CoColm [MASK]
---|---|---|---
ROCStories |  Context: Ed made beef jerky for a living. He ran the business out of his garage. One day he woke up and noticed his garage jarred open. He looked inside and noticed everything in disarray Positive Ending: Ed was delighted to see this. Negative Ending: Ed was shocked called the police for an investigation. |  Context + [MASK] + Ending: P: when, then, while N: and, but, so |  Context + [MASK] + Ending: P: so, hence, therefore N: or, and, though
MATRES |  The last surviving member of the team which first conquered Everest in 1953 has {$e_{1}$: died} in a Derbyshire nursing home. George Lowe, 89, {$e_{2}$: died} in Ripley on Wednesday after a long-term illness, with his wife Mary by his side. |  S1, +[MASK] + S2: and, sir, Dr |  S1, +[MASK] + S2: then, afterwards, till
COPA |  Premise: The girl found a bug in her cereal. Positive Hypothesis: She lost her appetite. Negative Hypothesis: She poured milk in the bowl. |  Pre+ [MASK] + Hypo: P: then, but, and N: then, next, so |  Pre + [MASK] + Hypo: P: so, therefore, thus. N: but, instead, and.
Table 7: Case study from three evaluation datasets. Connectives in blue are
predicted by the BERT-large model and ones in pink are predicted by
CoColm(BERT-large). “P” and “N” are used to represent the positive and
negative candidates, respectively.

##  6 Related Work

Understanding Events. It is important to represent and learn the commonsense
knowledge for deeply understanding the causality and correlation between
events. Recently various kinds of tasks requiring multiple dimensional event
knowledge are proposed such as story ending prediction [16], event temporal
ordering prediction [23], and event casual reasoning [19]. Prior studies have
incorporated external commonsense knowledge from ConceptNet [24] and ATOMIC
[8] for solving event representation [25], story generation tasks [26], event
KG completion [27]. However, their event-level knowledge is sparse and
incomplete due to the human-annotated acquisition, which thus limits the model
capacity, especially when injecting into pre-trained LMs. [6] builds a large-
scale eventuality knowledge graph, ASER, by specifying eventuality relations
mined from discourse connectives. It explicitly provides structural high-order
discourse information between events spanning from temporal, casual to co-
occurred relations, which has been proven to be transferable to human-defined
commonsense [28]. In this work, we aim at making full use of multi-dimensional
high-order event knowledge in the ASER to help pre-trained LMs understand
events.

Injecting Knowledge into LMs. Though [3] shows that pre-trained LMs store
factual relational knowledge without fine-tuning, still, LMs can not handle
knowledge-intensive tasks such as open-domain question answering or
commonsense reasoning. Previous works explore different ways to inject various
knowledge into pre-trained LMs for downstream task performance improvement.
They mainly differ from the knowledge resources, masking strategies, and
training objectives. From the perspective of knowledge resources, entity-
centric knowledge graphs are infused into LMs in the form of linked entities
[29, 30, 31] or triplets [32, 33, 34]. Besides that, linguistic knowledge
(e.g.,synonym/hypernym relations [35], word-supersense knowledge [36],
dependency parsing [34], and constituent parsing [37]) also plays a critical
role to improve the pre-trained LMs. Last but not least, domain-specific
knowledge is also customized to improve relevant tasks such as mined sentiment
word [38], event temporal patterns [39], and numerical reasoning data [40]. In
this work, we aim at injecting complex commonsense into pre-trained LMs with
two significant difference against previous works: (1) we use the event rather
than tokens as the semantic unit, and propose to use an event-based masking
strategy as well as two auxiliary tasks to help LMs understand events; (2) We
first leverage the random walk process on a large-scale knowledge graph to
include multi-hop knowledge.

##  7 Conclusion

In this work, we aim at helping pre-trained language models understand complex
commonsense about events. Specifically, we first conduct the random walk over
a large-scale eventuality-based knowledge graph to collect multi-hop event
knowledge and then inject the knowledge into the pre-trained LMs with an
event-based mask strategy as well as two auxiliary tasks. Experiments on three
downstream tasks as well as extensive analysis demonstrate the effectiveness
of the proposed model. As our approach is a general solution, we believe that
it can also be helpful for other tasks that require complex commonsense about
events. Both the code and pre-trained model CoColm are released.

## References

  * [1] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In Proceedings of NAACL, pages 4171–4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics.
  * [2] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov.  Roberta: A robustly optimized bert pretraining approach.  arXiv preprint arXiv:1907.11692, 2019.
  * [3] Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick S. H. Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander H. Miller.  Language models as knowledge bases?  In Proceedings of EMNLP-IJCNLP 2019, pages 2463–2473, 2019.
  * [4] Pat Verga, Haitian Sun, Livio Baldini Soares, and William W. Cohen.  Facts as experts: Adaptable and interpretable neural memory over symbolic knowledge.  CoRR, abs/2007.00849, 2020.
  * [5] Ray Jackendoff.  Semantic structures, volume 18.  MIT press, 1992.
  * [6] Hongming Zhang, Xin Liu, Haojie Pan, Yangqiu Song, and Cane Wing-Ki Leung.  Aser: A large-scale eventuality knowledge graph.  In Proceedings of The Web Conference 2020, pages 201–211, 2020\.
  * [7] Bryan Perozzi, Rami Al-Rfou, and Steven Skiena.  Deepwalk: online learning of social representations.  In The 20th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD ’14, New York, NY, USA - August 24 - 27, 2014, pages 701–710, 2014.
  * [8] Maarten Sap, Ronan Le Bras, Emily Allaway, Chandra Bhagavatula, Nicholas Lourie, Hannah Rashkin, Brendan Roof, Noah A Smith, and Yejin Choi.  Atomic: An atlas of machine commonsense for if-then reasoning.  In Proceedings of the AAAI, volume 33, pages 3027–3035, 2019.
  * [9] Forough Arabshahi, Jennifer Lee, Mikayla Gawarecki, Kathryn Mazaitis, Amos Azaria, and Tom Mitchell.  Conversational neuro-symbolic commonsense reasoning.  arXiv preprint arXiv:2006.10022, 2020.
  * [10] Yiming Cui, Wanxiang Che, Ting Liu, Bing Qin, Ziqing Yang, Shijin Wang, and Guoping Hu.  Pre-training with whole word masking for chinese bert.  arXiv preprint arXiv:1906.08101, 2019.
  * [11] Yu Sun, Shuohuan Wang, Yukun Li, Shikun Feng, Xuyi Chen, Han Zhang, Xin Tian, Danxiang Zhu, Hao Tian, and Hua Wu.  Ernie: Enhanced representation through knowledge integration.  arXiv preprint arXiv:1904.09223, 2019.
  * [12] Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S Weld, Luke Zettlemoyer, and Omer Levy.  Spanbert: Improving pre-training by representing and predicting spans.  TACL, 8:64–77, 2020.
  * [13] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In Advances in neural information processing systems, pages 5998–6008, 2017.
  * [14] Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush.  Transformers: State-of-the-art natural language processing.  In Proceedings of EMNLP, pages 38–45, Online, 2020. Association for Computational Linguistics.
  * [15] Miguel Ballesteros, Rishita Anubhai, Shuai Wang, Nima Pourdamghani, Yogarshi Vyas, Jie Ma, Parminder Bhatia, Kathleen McKeown, and Yaser Al-Onaizan.  Severing the edge between before and after: Neural architectures for temporal ordering of events.  In Proceedings of the EMNLP 2020, 2020.
  * [16] Nasrin Mostafazadeh, Nathanael Chambers, Xiaodong He, Devi Parikh, Dhruv Batra, Lucy Vanderwende, Pushmeet Kohli, and James Allen.  A corpus and cloze evaluation for deeper understanding of commonsense stories.  In Proceedings of the NAACL, pages 839–849, San Diego, California, June 2016.
  * [17] Zhongyang Li, Xiao Ding, and Ting Liu.  Story ending prediction by transferable bert.  In Proceedings of IJCAI, pages 1800–1806. AAAI Press, 2019.
  * [18] Qiang Ning, Hao Wu, and Dan Roth.  A multi-axis annotation scheme for event temporal relations.  In Proceedings of the ACL, pages 1318–1328, Melbourne, Australia, July 2018.
  * [19] Andrew Gordon, Zornitsa Kozareva, and Melissa Roemmele.  SemEval-2012 task 7: Choice of plausible alternatives: An evaluation of commonsense causal reasoning.  In Proceedings of SemEval 2012, pages 394–398, Montréal, Canada, 7-8 June 2012.
  * [20] Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman.  Superglue: A stickier benchmark for general-purpose language understanding systems.  In Advances in Neural Information Processing Systems, pages 3266–3280, 2019.
  * [21] Rishi Sharma, James Allen, Omid Bakhshandeh, and Nasrin Mostafazadeh.  Tackling the story ending biases in the story cloze test.  In Proceedings of ACL 2018, pages 752–757, 2018.
  * [22] Qiang Ning, Sanjay Subramanian, and Dan Roth.  An improved neural baseline for temporal relation extraction.  In Proceedings of the EMNLP-IJCNLP, pages 6203–6209, Hong Kong, China, November 2019. Association for Computational Linguistics.
  * [23] Qiang Ning, Zhili Feng, Hao Wu, and Dan Roth.  Joint reasoning for temporal and causal relations.  In Proceedings of ACL, pages 2278–2288, Melbourne, Australia, July 2018.
  * [24] Robyn Speer, Joshua Chin, and Catherine Havasi.  Conceptnet 5.5: An open multilingual graph of general knowledge.  pages 4444–4451, 2017.
  * [25] Xiao Ding, Kuo Liao, Ting Liu, Zhongyang Li, and Junwen Duan.  Event representation learning enhanced with external commonsense knowledge.  In Proceedings of EMNLP-IJCNLP, pages 4896–4905, 2019.
  * [26] Jian Guan, Fei Huang, Zhihao Zhao, Xiaoyan Zhu, and Minlie Huang.  A knowledge-enhanced pretraining model for commonsense story generation.  TACL, 8:93–108, 2020.
  * [27] Antoine Bosselut, Hannah Rashkin, Maarten Sap, Chaitanya Malaviya, Asli Celikyilmaz, and Yejin Choi.  Comet: Commonsense transformers for automatic knowledge graph construction.  In Proceedings of ACL, pages 4762–4779, 2019.
  * [28] Hongming Zhang, Daniel Khashabi, Yangqiu Song, and Dan Roth.  Transomcs: From linguistic graphs to commonsense knowledge.  In Proceedings of IJCAI, 2020.
  * [29] Zhengyan Zhang, Xu Han, Zhiyuan Liu, Xin Jiang, Maosong Sun, and Qun Liu.  Ernie: Enhanced language representation with informative entities.  In Proceedings of ACL, pages 1441–1451, 2019.
  * [30] Matthew E Peters, Mark Neumann, Robert Logan, Roy Schwartz, Vidur Joshi, Sameer Singh, and Noah A Smith.  Knowledge enhanced contextual word representations.  In Proceedings of the EMNLP-IJCNLP, pages 43–54, 2019.
  * [31] Wenhan Xiong, Jingfei Du, William Yang Wang, and Veselin Stoyanov.  Pretrained encyclopedia: Weakly supervised knowledge-pretrained language model.  In International Conference on Learning Representations, 2020.
  * [32] Liang Yao, Chengsheng Mao, and Yuan Luo.  Kg-bert: Bert for knowledge graph completion.  arXiv preprint arXiv:1909.03193, 2019.
  * [33] Weijie Liu, Peng Zhou, Zhe Zhao, Zhiruo Wang, Qi Ju, Haotang Deng, and Ping Wang.  K-BERT: enabling language representation with knowledge graph.  In AAAI February 7-12, 2020, pages 2901–2908, 2020.
  * [34] Ruize Wang, Duyu Tang, Nan Duan, Zhongyu Wei, Xuanjing Huang, Cuihong Cao, Daxin Jiang, Ming Zhou, et al.  K-adapter: Infusing knowledge into pre-trained models with adapters.  arXiv preprint arXiv:2002.01808, 2020.
  * [35] Anne Lauscher, Ivan Vulić, Edoardo Maria Ponti, Anna Korhonen, and Goran Glavaš.  Specializing unsupervised pretraining models for word-level semantic similarity.  In Proceedings of COLING, pages 1371–1383, Barcelona, Spain (Online), December 2020.
  * [36] Yoav Levine, Barak Lenz, Or Dagan, Ori Ram, Dan Padnos, Or Sharir, Shai Shalev-Shwartz, Amnon Shashua, and Yoav Shoham.  SenseBERT: Driving some sense into BERT.  In Proceedings of ACL, pages 4656–4667, Online, July 2020. Association for Computational Linguistics.
  * [37] Junru Zhou, Zhuosheng Zhang, and Hai Zhao.  Limit-bert: Linguistic informed multi-task bert.  arXiv preprint arXiv:1910.14296, 2019.
  * [38] Hao Tian, Can Gao, Xinyan Xiao, Hao Liu, Bolei He, Hua Wu, Haifeng Wang, and Feng Wu.  SKEP: Sentiment knowledge enhanced pre-training for sentiment analysis.  In Proceedings of ACL, pages 4067–4076, Online, July 2020. Association for Computational Linguistics.
  * [39] Ben Zhou, Qiang Ning, Daniel Khashabi, and D. Roth.  Temporal common sense acquisition with minimal supervision.  In Proceedings of ACL, 2020.
  * [40] Mor Geva, Ankit Gupta, and Jonathan Berant.  Injecting numerical reasoning skills into language models.  In Proceedings of ACL, 2020.