# Predictions of Additional Baryons and Mesons

Paul H. Frampton

Dipartimento di Matematica e Fisica ”Ennio De Giorgi”,
Università del Salento, Via Arnesano, 73100 Lecce, Italy.

<EMAIL_ADDRESS>

###### Abstract

We discuss the predictions of the bilepton model which is an extension of the
standard model in which the group $SU(2)\times U(1)$ is changed to
$SU(3)\times U(1)$ and the fermion families are treated non-sequentially with
the third assigned differently from the first two. Cancellation of triangle
anomalies and asymptotic freedom require three families. The predicted new
physics includes bileptons and three heavy quarks ${\cal D}$. ${\cal S}$ and
${\cal T}$. QCD will bind the heavy quarks to light quarks and to each other
to form baryons and mesons which, unlike bileptons, are beyond the reach of
the LHC but accessible in a hypothetical 100 TeV proton-proton collider.

To appear in the Proceedings of the 25th Bled Workshop ”What Comes Beyond the
Standard Model”. July 3-10, 2022.

Keywords: Triangle anomaly cancellation; three families; TeV quarks;
additional baryons; additional mesons.

##  1 Introduction

In this talk we shall discuss what now seems likely to be the first new
particle beyond the standard model and which is now being actively searched
for at the LHC. The bilepton model, a better name than the 331-Model, was
invented as an example of what then was expected to be a new class of models
which require the existence three families. That invention was in 1992 [1] but
it required a couple more years to realise that the expected new class of
models has only one member.

We remain optimistic that LHC can find a discovery signal for the bilepton
gauge boson in the remainder of 2022. What we can say generally is that to
invent a model which is beyond the standard model, one generally aims to both
(i)address and solve a question unanswered within the standard model, and to
(ii) provide explicit predictions which are testable. The bilepton model
beautifully fulfils both of these criteria.

Although not a sub-theory, the model originated from studying an interesting
$SU(15)$ model[2] in which the 224 gauge bosons couple to all possible pairs
of the 15 states

$(u,d)_{\alpha};(\bar{u})_{\alpha};(\bar{d})_{\alpha};(\nu_{e}.e);(\bar{e}).$
(1)

Every gauge boson therefore has a well-defined B and L so there can be no
proton decay by tree-level gauge boson exchange.

In the $SU(15)$ model there is one unaesthetic feature that anomalies are
cancelled by adding mirror fermions as in $15+\bar{15}$. But persisting
further, we considered the subgroups in $SU(15)\rightarrow SU(12)_{q}\times
SU(3)_{l}$, especially the $SU(3)_{l}$ which contains an antitriplet
$(e^{+},\nu_{e},e^{-})$ where the $|L|=|Q|=2$ bilepton can first be seen,
coupling electron to positron.

The question then was: can a chiral model contain bileptons? After hundreds of
trials and errors we found only one solution of the anomaly cancellation
equations. This required non-sequential families where the third is assigned
differently from the first two and explains why there must be three families.
This is the bilepton model. It provides an answer to Rabi’s famous question
when the muon was discovered in 1936: ”Who ordered that?”

The non-sequentiality of families offers one explanation for the failure of
the $SU(5)$ model studied first in 1974 [3, 4] then in hundreds of other
papers. $SU(5)$ assumed sequentiality of families of the form $3(10+\bar{5})$.

In 1977 Weinberg[5] and in 1984 Glashow [6] both considered upgrading the
electroweak $SU(2)$ of the standard model to $SU(3)$ but overlooked the
assignments which explain three families.

##  2 Bilepton model

The gauge group is:

$SU(3)_{C}\times SU(2)_{L}\times U(1)_{X}$ (2)

The simplest choice for the electric charge is

$Q=\frac{1}{2}\lambda_{L}^{3}+\left(\frac{\sqrt{3}}{2}\right)\lambda_{L}^{8}+X\left(\frac{\sqrt{3}}{\sqrt{2}}\right)\lambda^{9}$
(3)

where

$Tr(\lambda_{L}^{a}\lambda_{L}^{b})=2\delta^{ab}$ (4)

and

$\lambda^{9}\equiv\left(\frac{\sqrt{2}}{\sqrt{3}}\right)diag(1,1,1)$ (5)

Thus a triplet has charges $(X+1,X,X-1)$.

Leptons are treated democratically in each
of the three families. They are colour singlets in antitriplets of $SU(3)_{L}$
:

$(e^{+},\nu_{e},e^{-})_{L}$

$(\mu^{+},\nu_{\mu},\mu^{-})_{L}$

$(\tau^{+},\nu_{\tau},\tau^{-})_{L}$

All have $X=0$.

Quarks in the first family are assigned to a left-handed triplet plus three
singlets of $SU(3)_{L}$.

$(u^{\alpha},d^{\alpha},D^{\alpha})_{L}~{}~{}~{}(\bar{u}_{\alpha})_{L},(\bar{d}_{\alpha})_{L},(\bar{D}_{\alpha})_{L}$

Similarly for the second family

$(c^{\alpha},s^{\alpha},S^{\alpha})_{L}~{}~{}~{}(\bar{c}_{\alpha})_{L},(\bar{s}_{\alpha})_{L},(\bar{S}_{\alpha})_{L}$

The X values are for the triplets are $X=-1/3$ and for the singlets
$X=-2/3,+1/3,+4/3$ respectively. The electric charge of the new quarks $D,S$
is $-4/3$.

The quarks of the third family are treated differently. They are assigned to a
left-handed antitriplet and three singlets under $SU(3)_{L}$

$(b^{\alpha},t^{\alpha},T^{\alpha})_{L}~{}~{}~{}(\bar{b}_{\alpha})_{L},(\bar{t}_{\alpha})_{L},(\bar{T}_{\alpha})_{L}$

The antitriplet has $X=+2/3$ and the singlets carry $X=+1/3,-2/3,-5/3$
respectively. The new quark $T$ has $Q=5/3$.

Some of the relevant LHC phenomenology is discussed in [7]. A refined mass
estimate [8] for the bilepton is is $M(Y^{\pm\pm})=(1.29\pm 0.06)$ TeV where
faute de mieux it was assumed that the symmetry breaking of $SU(3)_{L}$ is
closely similar to that of $SU(2)_{L}$. It will be pleasing if the physical
mass is consistent with this.

##  3 New Quarks

Because the quarks are in triplets and anti-triplets of $SU(3)_{L}$, rather
than only in doublets of $SU(2)_{L}$ as in the standard model, there is
necessarily an additional quark in each family. In the first and second
families they are the ${\cal D}$ and ${\cal S}$ respectively, both with charge
$Q=-4/3$ and lepton number $L=+2$. In the third family is the ${\cal T}$ with
charge $Q=+5/3$ and lepton number $L=-2$. All the three TeV scale quarks are
colour triplets with spin-$\frac{1}{2}$ and baryon number $B=\frac{1}{3}$.
Their masses are yet to be measured but may be expected to be below the
ceiling of $4.1TeV$ which is the upper limit for symmetry breaking of
$SU(3)_{L}$ and probably above $1TeV$. By analogy with the known quarks, one
might expect $M({\cal T})>M({\cal S})>M({\cal D})$, although without
experimental data this is conjecture.

The heavy quarks and antiquarks will be bound to light quarks and antiquarks,
and to each other, to form an interesting spectroscopy of mesons and baryons.
Let us first display, in Tables 1, 2 the TeV mesons, then in Tables 3,4,5 the
TeV baryons. The charge conjugate states are equally expected, and will
reverse the signs of $Q$ and $L$.

##  4 Additional Baryons and Mesons

.

Table 1: TeV mesons ${\cal Q}\bar{q}$ ${\cal Q}$ | $\bar{q}$ | Q | L
---|---|---|---
${\cal D/S}$ |  $\bar{u}$ etc. | -2 | +2
${\cal D/S}$ |  $\bar{d}$ etc. | -1 | +2
${\cal T}$ |  $\bar{u}$ etc. | +1 | -2
${\cal T}$ |  $\bar{d}$ etc. | +2 | -2

Table 2: TeV mesons ${\cal Q}\bar{{\cal Q}}$ ${\cal Q}$ | $\bar{{\cal Q}}$ | Q | L
---|---|---|---
${\cal D/S}$ | ${\cal\bar{D}/\bar{S}}$ | 0 | 0
${\cal D/S}$ | $\bar{{\cal T}}$ | -3 | +4
${\cal T}$ | $\bar{T}$ | 0 | 0

Table 3: TeV baryons ${\cal Q}qq$ ${\cal Q}$ | qq | Q | L
---|---|---|---
${\cal D/S}$ | dd etc. | -2 | +2
${\cal D/S}$ | ud etc. | -1 | +2
${\cal D/S}$ | uu etc. | 0 | +2
${\cal T}$ | dd etc. | +1 | -2
${\cal T}$ | ud etc. | +2 | -2
${\cal T}$ | uu etc, | +3 | -2

Table 4: TeV baryons ${\cal Q}{\cal Q}q$ ${\cal Q}{\cal Q}$ | $q$ | Q | L
---|---|---|---
${\cal(D/S)(D/S)}$ | d etc. | -3 | +4
${\cal(D/S)(D/S)}$ | u etc. | -2 | +4
${\cal(D/S)T}$ | d etc. | 0 | 0
${\cal(D/S)T}$ | u etc. | +1 | 0
${\cal TT}$ | d etc. | +3 | -4
${\cal TT}$ | u etc. | +4 | -4
Table 5: TeV baryons ${\cal Q}{\cal Q}{\cal Q}$ ${\cal Q}{\cal Q}{\cal Q}$ | Q | L
---|---|---
${\cal(D/S)(D/S)(D/S)}$ | -4 | +6
${\cal(D/S)(D/S)T}$ | -1 | +2
${\cal(D/S)TT}$ | +2 | -2
${\cal TTT}$ | +5 | -6

Although the ${\cal Q}$ masses are unknown, it may be reasonable first to make
a preliminary discussion of these states by assuming that

$M({\cal T})>M({\cal S})+2M_{t}>M({\cal D})+4M_{t}$ (6)

where $M_{t}$ is the top quark mass so that the lightest of the TeV baryons
and mesons are those containing just one ${\cal D}$ quark or one
${\cal\bar{D}}$ antiquark. The next lightest are the TeV baryons and mesons
containing just one ${\cal S}$ quark or one $\bar{{\cal S}}$ antiquark.

We begin by discussing the decay modes of the ${\cal D}\bar{q}$ mesons in
Table 1, focusing on final states from the first family. The decays of ${\cal
D}$ include, taking care of $L$ conservation,

$\displaystyle{\cal D}$ $\displaystyle\rightarrow$ $\displaystyle d+Y^{-}$
$\displaystyle\rightarrow$ $\displaystyle d+(e^{-}+\nu_{e})$
$\displaystyle\rightarrow$ $\displaystyle d+(\mu^{-}+\nu_{\mu})$
$\displaystyle\rightarrow$ $\displaystyle d+(\tau^{-}+\nu_{\tau})$

which implies that decays of the $({\cal D}\bar{u})$ meson include

$\displaystyle({\cal D}\bar{u})$ $\displaystyle\rightarrow$
$\displaystyle\pi^{-}+(e^{-}+\nu_{e})$ $\displaystyle\rightarrow$
$\displaystyle\pi^{-}+(\mu^{-}+\nu_{\mu})$ $\displaystyle\rightarrow$
$\displaystyle\pi^{-}+(\tau^{-}+\nu_{\tau})$

and variants thereof where $\pi^{-}$ is replaced by any other non-strange
negatively charged meson. The $d$ in Eq.(LABEL:calDdecay) can be replaced by
$s$ or $b$ which subsequently decay.

An alternative to Eq.(LABEL:calDdecay) is

$\displaystyle{\cal D}$ $\displaystyle\rightarrow$ $\displaystyle u+Y^{--}$
$\displaystyle\rightarrow$ $\displaystyle u+(e^{-}+e^{-})$
$\displaystyle\rightarrow$ $\displaystyle u+(\mu^{-}+\mu^{-})$
$\displaystyle\rightarrow$ $\displaystyle u+(\tau^{-}+\tau^{-})$

which implies additional decay modes of the $({\cal D}\bar{u})$ meson which
include

$\displaystyle({\cal D}\bar{u})$ $\displaystyle\rightarrow$
$\displaystyle\pi^{0}+(e^{-}+e^{-})$ $\displaystyle\rightarrow$
$\displaystyle\pi^{0}+(\mu^{-}+\mu^{-})$ $\displaystyle\rightarrow$
$\displaystyle\pi^{0}+(\tau^{-}+\tau^{-})$

and variants obtained by flavour replacements. Eqs.(LABEL:calDudecay) and
(LABEL:calDudecay2), and their generalisations to other flavours, suffice to
illustrate the richness of $({\cal D}\bar{u})$ decays.

Turning to the meson ${\cal D}\bar{d}$, we can use Eq.(LABEL:calDdecay) to
identify amongst its possible decays

$\displaystyle({\cal D}\bar{d})$ $\displaystyle\rightarrow$
$\displaystyle\pi^{0}+(e^{-}+\nu_{e})$ $\displaystyle\rightarrow$
$\displaystyle\pi^{0}+(\mu^{-}+\nu_{\mu})$ $\displaystyle\rightarrow$
$\displaystyle\pi^{0}+(\tau^{-}+\nu_{\tau})$

and variants thereof where $\pi^{0}$ is replaced by any other non-strange
neutral meson. When $u$ in Eq.(LABEL:calDdecay) is replaced by $c$ or $t$
which subsequently decay, we arrive at many other decay channels additional to
Eq.(LABEL:calDddecay).

Employing instead the ${\cal D}$ decays in Eq.(LABEL:calDdecay2) implies
additional decay modes of $({\cal D}\bar{d})$ meson that include

$\displaystyle({\cal D}\bar{d})$ $\displaystyle\rightarrow$
$\displaystyle\pi^{+}+(e^{-}+e^{-})$ $\displaystyle\rightarrow$
$\displaystyle\pi^{+}+(\mu^{-}+\mu^{-})$ $\displaystyle\rightarrow$
$\displaystyle\pi^{+}+(\tau^{-}+\tau^{-})$

and variants obtained by flavour replacement. Eqs.(LABEL:calDddecay) and
(LABEL:calDddecay2), merely illustrate a few of the simplest $({\cal
D}\bar{d})$ decays. There are many more.

Next we consider the lightest TeV baryons in Table 3 with ${\cal Q}={\cal D}$.
Using the ${\cal D}$ decays from Eq.(LABEL:calDdecay) we find for $({\cal
D}uu)$ decay

$\displaystyle({\cal D}uu)$ $\displaystyle\rightarrow$ $\displaystyle
p+(l_{i}^{-}+\nu_{i}).$

together with flavour rearrangements. Here, as in subsequent equations,
$i=e,\mu,\tau$.

Alternatively, the ${\cal D}$ decays from Eq.(LABEL:calDdecay2) lead to

$\displaystyle({\cal D}uu)$ $\displaystyle\rightarrow$ $\displaystyle
N^{*++}+Y^{--}.$ $\displaystyle\rightarrow$ $\displaystyle
p+\pi^{+}+(l_{i}^{-}+l_{i}^{-})..$

Looking at the TeV baryon $({\cal D}ud)$ the respective sets of decays
corresponding to Eq.(LABEL:calDdecay) are

$\displaystyle({\cal D}ud)$ $\displaystyle\rightarrow$ $\displaystyle
n+(l_{i}^{-}+\nu_{i})$

where only the simplest light baryon is exhibited.

Corresponding to ${\cal D}$ decays in Eq.(LABEL:calDdecay2) there are also

$\displaystyle({\cal D}ud)$ $\displaystyle\rightarrow$ $\displaystyle
p+(l_{i}^{-}+l_{i}^{-})$

in the simplest cases.

Finally, of the $({\cal D}qq)$ TeV baryons, we write out the decays for
$({\cal D}dd)$, first for the ${\cal D}$ decays in Eq.(LABEL:calDdecay)

$\displaystyle({\cal D}dd)$ $\displaystyle\rightarrow$ $\displaystyle
N^{*-}+Y^{-}$ $\displaystyle\rightarrow$ $\displaystyle
n+\pi^{-}+(l_{i}^{-}+\nu_{i}).$

within flavour variations.

With the Eq.(LABEL:calDdecay2) decays of ${\cal D}$ there are also decays

$\displaystyle({\cal D}dd)$ $\displaystyle\rightarrow$ $\displaystyle
n+(l_{i}^{-}+l_{i}^{-})$ (18)

again with more possibilities by choosing alternative flavours.

We now replace the TeV quark ${\cal D}$ by the next heavier TeV quark ${\cal
S}$ and repeat our study of decays whereupon we shall encounter the first
example of decay not only to the known quarks but also to a TeV quark.

The TeV quark ${\cal S}$ has possible decay channels

$\displaystyle{\cal S}$ $\displaystyle\rightarrow$ $\displaystyle d+Y^{-}$
$\displaystyle\rightarrow$ $\displaystyle d+(e^{-}+\nu_{e})$
$\displaystyle\rightarrow$ $\displaystyle d+(\mu^{-}+\nu_{\mu})$
$\displaystyle\rightarrow$ $\displaystyle d+(\tau^{-}+\nu_{\tau})$
$\displaystyle\rightarrow$ $\displaystyle{\cal D}+Z^{\prime}$
$\displaystyle\rightarrow$ $\displaystyle d+(e^{-}+\nu_{e})+(e^{+}+e^{-})$
$\displaystyle\rightarrow$ $\displaystyle d+(e^{-}+\nu_{e})+(\mu^{+}+\mu^{-})$
$\displaystyle\rightarrow$ $\displaystyle
d+(e^{-}+\nu_{e})+(\tau^{+}+\tau^{-})$ $\displaystyle\rightarrow$
$\displaystyle d+(\mu^{-}+\nu_{\mu})+(e^{+}+e^{-})$ $\displaystyle\rightarrow$
$\displaystyle d+(\mu^{-}+\nu_{\mu})+(\mu^{+}+\mu^{-})$
$\displaystyle\rightarrow$ $\displaystyle
d+(\mu^{-}+\nu_{\mu})+(\tau^{+}+\tau^{-})$ $\displaystyle\rightarrow$
$\displaystyle d+(\tau^{-}+\nu_{\tau})+(e^{+}+e^{-})$
$\displaystyle\rightarrow$ $\displaystyle
d+(\tau^{-}+\nu_{\tau})+(\mu^{+}+\mu^{-})$ $\displaystyle\rightarrow$
$\displaystyle d+(\tau^{-}+\nu_{\tau})+(\tau^{+}+\tau^{-})$

where we note the opening up of channels due to ${\cal S}\rightarrow{\cal D}$
decay.

With Eq.(LABEL:calSdecay) in mind, the decays of the TeV meson $({\cal
S}\bar{u})$ include

$\displaystyle({\cal S}\bar{u})$ $\displaystyle\rightarrow$
$\displaystyle\pi^{-}+(l_{i}^{-}+\nu_{i})$ $\displaystyle\rightarrow$
$\displaystyle\pi^{-}+(l_{i}^{-}+\nu_{i})+(l_{j}^{+}+l_{j}^{-})$

where the second line involves a ${\cal D}$ intermediary.

An alternative to Eq.(LABEL:calSdecay) is

$\displaystyle{\cal S}$ $\displaystyle\rightarrow$ $\displaystyle u+Y^{--}$
$\displaystyle\rightarrow$ $\displaystyle u+(e^{-}+e^{-})$
$\displaystyle\rightarrow$ $\displaystyle u+(\mu^{-}+\mu^{-})$
$\displaystyle\rightarrow$ $\displaystyle u+(\tau^{-}+\tau^{-})$

which implies additional decay modes of $({\cal S}\bar{u})$

$\displaystyle({\cal S}\bar{u})$ $\displaystyle\rightarrow$
$\displaystyle\pi^{0}+(l_{i}^{-}+l_{i}^{-})$

and variants which replace $\pi^{0}$ by another neutral non-strange meson.
Eqs.(LABEL:calSudecay) and (LABEL:calSudecay2), illustrate sufficiently
$({\cal S}\bar{u})$ decays.

Turning to the meson (${\cal S}\bar{d}$), we can use Eq.(LABEL:calSdecay) to
identify its possible decays

$\displaystyle({\cal S}\bar{d})$ $\displaystyle\rightarrow$
$\displaystyle\pi^{0}+(l_{i}^{-}+\nu_{i})$

When $u$ in Eq.(LABEL:calSdecay) is replaced by $c$ or $t$ which subsequently
decay, we arrive at many other decay channels additional to
Eq.(LABEL:calSddecay).

Employing instead the ${\cal S}$ decays in Eq.(LABEL:calSdecay2) implies
additional decay modes of $({\cal S}\bar{d})$ that include

$\displaystyle({\cal S}\bar{d})$ $\displaystyle\rightarrow$
$\displaystyle\pi^{+}+(l_{i}^{-}+l_{i}^{-})$

and variants obtained by flavour replacement. Eqs.(LABEL:calSddecay) and
(LABEL:calSddecay2), illustrate only a few of the simplest $({\cal S}\bar{d})$
decays. There are many more.

Next we consider the lightest TeV baryons in Table 3 with one ${\cal Q}={\cal
S}$. Using the ${\cal S}$ decays from Eq.(LABEL:calSdecay) we find for (${\cal
S}uu$) decay

$\displaystyle({\cal S}uu)$ $\displaystyle\rightarrow$ $\displaystyle
p+(l_{i}^{-}+\nu_{i}).$

together with flavour rearrangements.

Alternatively, the ${\cal S}$ decays from Eq.(LABEL:calSdecay2) lead to

$\displaystyle({\cal S}uu)$ $\displaystyle\rightarrow$ $\displaystyle
N^{*++}+(l_{i}^{-}+l_{i}^{-})..$ $\displaystyle\rightarrow$ $\displaystyle
p+\pi^{+}+(l_{i}^{-}+l_{i}^{-}).$

Looking at the TeV baryon $({\cal S}ud)$ the respective sets of decays
corresponding to Eq.(LABEL:calSdecay) are

$\displaystyle({\cal S}ud)$ $\displaystyle\rightarrow$ $\displaystyle
n+(l_{i}^{-}+\nu_{i})$

where only the simplest version is exhibited.

Corresponding to the ${\cal S}$ decays in Eq.(LABEL:calSdecay2) there are the
decays

$\displaystyle({\cal S}ud)$ $\displaystyle\rightarrow$ $\displaystyle
p+(l_{i}^{-}+l_{i}^{-})$

For baryon $({\cal S}dd)$, firstly from the ${\cal S}$ decays in
Eq.(LABEL:calSdecay) we have

$\displaystyle({\cal S}dd)$ $\displaystyle\rightarrow$ $\displaystyle
N^{*-}+Y^{-}$ $\displaystyle\rightarrow$ $\displaystyle
n+\pi^{-}+(l_{i}^{-}+\nu_{i}).$

within flavour variations.

Secondly, from the Eq.(LABEL:calSdecay2) decays of ${\cal S}$ there are baryon
decays of the type

$\displaystyle({\cal S}dd)$ $\displaystyle\rightarrow$ $\displaystyle
n+(l_{i}^{-}+l_{i}^{-})$

with more possibilities by choosing alternative flavours.

##  5 Discussion

We could continue further to study decays of all the baryons and mesons in our
Tables. However, it seems premature to do so, until we know from experimental
data the masses and mixings of ${\cal D,S,T}$. We remark only that the type of
lepton cascade which we have exhibited in Eq.(LABEL:calSdecay) becomes ever
more prevalent as the lepton number of the decaying hadron increases.

We may expect, by analogy with the top quark mass being close to the weak
scale that the mass of the ${\cal T}$ quark, although probably below $4.1$ TeV
for the symmetry-breaking reason discussed ut supra, might be not much below.
For example it might exceed $3$ TeV whereupon the mass of a (${\cal TTT}$)
baryon could exceed $9$ TeV. Since this baryon has high lepton number, it must
be pair produced and such production is far beyond the reach of the $14$ TeV
LHC. Its study would require a $100$ TeV collider of the type presently under
preliminary discussion. As a foretaste of the physics accessible to such a
hypothetical collider, the simplest decay of the $({\cal TTT})$ baryon we can
find is

$p+4(e^{+})+2(\bar{\nu}_{e}).\\\ $

which would be very exciting to confirm.

At the time of writing, the particles exhibited in our Tables are conjectural.
After the bilepton is discovered the existence of all the additional baryons
and mesons in our five Tables would become a sharp predictions.

The bilepton resonance in $\mu^{\pm}\mu^{\pm}$ has been the subject of
searches by the ATLAS and CMS Collaborations at the LHC, starting in March
2021. In March 2022, ATLAS published an inconclusive result [9] about the
existence of the resonance, putting only a lower mass limit $M_{Y}>1.08$ TeV.
CMS has better momentum resolution and, what is the same thing, charge
identification than ATLAS and should be able to investigate the bilepton
resonance proper. The high sensitivity of CMS is a result of serendipity
because it was designed in 1993 not for the bilepton but to search for heavy
Z-primes[11]. A second serendipity was an accidental 2015 meeting in London
between us and Sir Tejinder Virdee who helped design the CMS detector.

Our strong belief in the existence of the bilepton lies partly in the close
relationship between the 1961 paper[10] which solved the parity puzzle and our
1992 paper[1] which solved the family puzzle. We regard these two papers which
span three decades as well-matched bookends,

According to our calculations [7], the Run 2 data with 139/fb collected by
2018 are sufficient for a CMS discovery of the bilepton. If not, future LHC
runs up to their target integrated luminosity of 4/ab can provide 28 times as
many events and bilepton discovery would be merely postponed. We do hope,
however, that a great discovery will be made by the LHC within six months from
today (July 25, 2022).

Note added:
We answer here one interesting question received after our talk: Why are these
heavy states not as unstable as the top quark which lives for less than a
trillion trillionth of a second? The answer is that they decay via bilepton
exchange. This fact renders their lifetimes a trillion times longer than the
top quark lifetime.

## Acknowlegement

We thank the organisers Norma Borstnik, Maxim Khlopov and Holger Nielsen for
their invitation to present this talk.

## References

  * [1] P.H. Frampton,
Phys. Rev. Lett. 69, 2889 (1992).

  * [2] P.H. Frampton and B.-H. Lee,
Phys. Rev Lett. 64, 619 (1990).

  * [3] H. Georgi and S.L. Glashow,
Phys. Rev. Lett. 32, 438 (1974).

  * [4] H. Georgi, H.R. Quinn and S. Weinberg,
Phys. Rev. Lett. 33, 451 (1974).

  * [5] B.W.Lee and S.Weinberg,
Phys.Rev. Lett. 38, 1237 (1977).

  * [6] S.L. Glashow,
Fifth Workshop on Grand Unification.
World Scientific Publishing Company (1984).

  * [7] G. Corcella, C, Coriano, A, Costantini and P.H. Frampton,
Phys. Lett. B773, 544 (2017); ibid. 785, 73 (2018);
ibid. B826, 136904 (2022).

  * [8] C. Coriano and P.H. Frampton,
Mod. Phys. Lett. A36, 2150118 (2021).

  * [9] ATLAS Collaboration,
ATLAS-CONF-2022-010 (March 2022).

  * [10] S.L. Glashow,
Nucl. Phys. 22, 579 (1961).

  * [11] T. Virdee,
Several private communications beginning in 2015.


# BOOSTR: A Dataset for Accelerator Control Systems

Diana Kafkes  Jason St. John  Fermi National Accelerator Laboratory, Batavia,
IL 60510, USA

###### Abstract

BOOSTR (Booster Operation Optimization Sequential Time-Series for
Reinforcement) was created to provide cycle-by-cycle time series of readings
and settings from instruments and controllable devices of the Booster,
Fermilab’s Rapid-Cycling Synchrotron (RCS) operating at 15 Hz. BOOSTR provides
a time series from 55 device readings and settings which pertain most directly
to the high-precision regulation of the Booster’s Gradient Magnet Power Supply
(GMPS). To our knowledge, this is the first known dataset of accelerator
device parameters made publicly available. We are releasing it in the hopes
that it can be used to demonstrate aspects of artificial intelligence for
advanced control systems, such as reinforcement learning and autonomous
anomaly detection.

## Background & Summary

Tuning and controlling particle accelerators is challenging and time
consuming. Even marginal improvements to accelerator operation can translate
very efficiently into improved scientific yield for an experimental particle
physics program. The data released here was collected in the hopes of
achieving improvement in precision for the Fermilab Booster Gradient Magnet
Power Supply (GMPS) regulatory system, which is detailed below.

The Fermilab Booster receives a 400 MeV proton beam from the Linear
Accelerator and accelerates it to 8 GeV through synchronously raising
accelerator cavity radiofrequency and instigating a controlled magnetic field
to steer the beam with combined-function bending and focusing electromagnets,
known as gradient magnets. These magnets are powered by the GMPS, which
operates on a 15 Hz cycle between a minimum current (at injection) and a
maximum current (at beam extraction). The GMPS is realized as four power
supplies, evenly distributed around the Booster, and each powers one fourth of
the gradient magnets. The role of the GMPS regulator is to calculate and apply
small compensating offsets in the GMPS driving signal in order to improve the
agreement of the resulting observed minimum and maximum currents with their
set points. Without regulation, the fitted minimum of the magnetic field may
vary from the set point by as much as a few percent.

At beam injection, a perturbation of only a percent is enough to significantly
decrease beam transfer efficiency and thereby reduce the beam flux ultimately
available to the high-energy particle physics experiments run at the lab.
Disturbances to the magnet current can occur due to ambient temperature
changes, other nearby high-power pulsed radio-frequency systems, and
electrical ground movement induced by power supplies of other particle
accelerators at the complex. The current GMPS regulation involves a PID
(Proportional-Integral-Derivative) control scheme (see Figure 1 for
schematic). The regulator calculates estimates for the minimum and maximum
currents of the offset-sinusoidal magnetic field from the previous 15 Hz
cycle. These values are then used to adjust the power supply program and
decrease systemic error in the next cycle’s current, such that it more closely
matches the set point. Presently, the PID system achieves regulation errors
corresponding to roughly 0.1% of the set value.

Although some 200,000 entries populate the device database of Fermilab’s
accelerator control system [1], the 55 device value time series presented here
in BOOSTR [2] were collected in accordance with suggestion by Fermilab
accelerator subject matter experts (SMEs). These values exhibit correlations
with GMPS power supply perturbations. The full data were collected during two
separate periods: from June 3, 2019 to July 11, 2019 — when the accelerator
was shut down for regular maintenance — and from December 3, 2019 to April 13,
2020 — when the accelerator was shut down in response to the Covid-19
Pandemic.

Data from a single day of BOOSTR was previously described in a Datasheet [3].
A proof-of-concept paper [4] (submitted to Physical Review Accelerators and
Beams) used this subset of BOOSTR and demonstrated the viability of training a
reinforcement learning agent to control GMPS regulation better than the
existing PID system. Relative to the original Datasheet [3], this manuscript
is expanded with more SME input, describes more than 100 times more data, and
includes documentation of validation not presented in the original Datasheet.

## Methods

#### Collection Process

A data collection node was created and set to request data at 15 Hz from the
Data Pool Manager of the Accelerator Control Network (ACNET) [1]. The created
scheme involved front-end nodes, each managing their respective devices,
replying with timestamped values at the stated rate barring differences of
clock speed, input-output (I/O) lag time variations due to network traffic
fluctuations, and higher-priority interruptions from competing processes on
the front-end node. These inconsistencies were later addressed through a time-
alignment process discussed in the Data Processing Section. The collection
node stored the data in a circular buffer approximately 10 days deep.

A Python script managed by a nightly cron job polled the data collection node
for the most recent midnight-to-midnight 24 hours of timestamped data for each
of the 55 time series identified by SMEs. A second cron-managed script did the
same for relevant accelerator control events issued in the same period. These
event data correspond to important cycles achieved through controlling the
devices at the accelerator. Event data were requested by a separate data
collection node.

Each day’s data harvest was originally stored in HDF5 (Hierarchical Data
Format Version 5) files. Any data instances missing from the parquet files
released here were not included in the original data buffers from which this
dataset was drawn.

### Data Processing

Each instance was created through a concatenation of each device’s timestamp
data table within every HDF5 file and then saved in parquet format. A similar
procedure was undertaken for one of the accelerator control event signals
polled, Event0C, as its broadcast is synchronized with the GMPS magnetic field
minimum. Event0C was collected to correct a problem in the observed sampling
frequency: there was an issue of the sampling of each device being nominally
at 15 Hz, but in reality synchrony was demonstrably imperfect, and the time
intervals between successive timestamps display varying lags.

Since Event0c serves as the baseline or heartbeat of the Booster at
approximately 15 Hz and is synchronized with the smoothly varying electrical
current GMPS regulates, we used Event0c to time-align our data. The alignment
approximates the data available to the GMPS regulator operating in real time.
We used the GMPS-synchronized Event0C’s timestamp as the moment to begin
forward inference, taking the value for each device time series which had the
most recent corresponding timestamp. In practice, this required timestamp-
sorted series for each device and finding the most recent device value,
relative to Event0c timestamp, in a lookback window equal to the maximum
interval between device timestamps (necessarily excluding the five month gap
between our two data collection periods). This time-alignment step was run
over the whole dataset in multiple parallel processes using Apache Spark.

Notably, the data recorded for Event0c was missing the period from July 1 to
11, 2019. Therefore aligning on this variable discarded some of the data
collected during our first period of collection.

## Data Records

The data release is stored on Zenodo [2]. Each instance is a zip compressed
parquet of one of the 55 aligned time series with columns corresponding to the
aligned time stamp, original time stamp, difference between time stamps, and
the reading/setting value [2]. The original timestamp and time difference is
included to demonstrate the mechanics of our alignment process and enable a
check for reproducibility. All timestamps are in Greenwich Mean Time (UTC).

Our data release contains device data from each of the four gradient magnet
power supplies, the GMPS PID regulator, and the Main Injector, where the beam
is directed after acceleration via the Booster. Minimum and maximum current
information readings and settings, the feedback and transductor gain settings,
and the feed-forward start trigger are collected as part of the current PID
regulation scheme. The “inhibit” value controls whether the GMPS regulator
accepts settings changes for parameters other than the minimum and maximum
current, such as the gain settings (any positive value will prevent changes).
Additionally, $\dot{\vec{B}}$, the rate of change of the magnetic field, is
recorded as a proxy for the magnetic field we are interested in regulating.
Timing information derived from $\dot{\vec{B}}$ = 0 synchronizes the current
PID regulator system.

We acknowledge that the ACNET parameter names are by no means standardized
across different particle accelerators and that they will appear especially
abstruse for those well-versed in control systems who are new to working with
accelerators. In Table 1, we detail explanations of each of the parameters
read from devices (devices whose first letter is followed by :) and indicate
whether the device setting was included in the dataset (devices whose first
letter is followed by _ and appear in Setting column), since describing these
corresponding pairs would be redundant. In Figures 2 and 3, we visualize
metadata trends for each “nonconstant" parameter in each data collection
period (see Table 2 for a list of values we considered to be virtually
unchanging within the two periods) and also provide the mean and standard
deviation of device readings across the two collection periods in Table 1.
Furthermore, Table 1 includes dates missing in the data recorded for each
reading. As a reminder the data were collected during two separate periods:
from June 3, 2019 to June 30, 2019 (July is missing due to time-alignment with
Event0C) and from December 3, 2019 to April 13, 2020. Finally, in Figure 4 we
demonstrate the centrality of each recorded parameter with a heatmap of
histogram values.

Additionally, we provide the PID regulator status values B|GMPSSC (ACNET
status parameters include |), whose 16 bits contain various motherboard
states. Here we are concerned with bit 3, which indicates whether or not the
GMPS regulator was on (1), and bit 7, which indicates whether the Booster is
in its normal alternating current (AC) mode (1) or “coasting beam” direct
current (DC) mode at constant beam energy (0). Unlike the rest of the devices,
this status value is presently recorded at only 1 Hz because it was not
included in our initial data node request and was relegated to an archived
data node at lesser frequency. While the same time-aligning described above
was applied to align B|GMPSSC, due to the slow sampling rate, we caution the
user to refer closely to the original timestamp such that they might make
decisions about whether to use data when GMPS was off and to inform them of
potential problems when interpolating in a region immediately before or after
a status change. See Figure 5 for more details on B|GMPSSC values.

## Technical Validation

In order to verify the quality of this dataset, we pored over the electronic
logbook (Elog) [5] that Fermilab Booster technicians and operators use to
record changes to device settings as well as observations while in operation.
We used these Elog entries to authenticate our data’s viability across
timescales. First, we used the Elog to corroborate expert acknowledgement of
the major spikes observed in Figures 2 and 3. These outlier changes, typically
seen in the value’s mean and standard deviation, represented major changes
made on that specific day, including when the Booster was switched from
alternating current (AC) to direct current (DC) mode (see Figure 5) as well as
when the GMPS regulator was turned off altogether. These reconciliations are
presented in Table 3.

Furthermore, we pinpoint changes in the AC vs. DC settings according to the
Elog [5] for June 24, 2019 and March 11, 2020 in Figure 6. Here applying a
bitmask reveals that a B|GMPSSC value of 159 indicates AC mode/GMPS on, while
31 indicates DC mode/GMPS on, and 407 indicates AC mode/GMPS off. In this
figure, the plotted timestamps were offset to Central Time (UTC-5) in order to
align with times given in the Elog, which were not recorded in UTC. On June
24, the trace of B|GMPSSC clearly shows GMPS regulation briefly switching off
before commencing DC studies from 8:00 AM - 6:00 PM with a value of 31, then
being turned back to 159. On March 11, B|GMPSSC is at 159 before 6:00 AM, off
at 407 from 6:00 - 9:50 AM, and then is set to AC mode from 9:50 AM - 12:45
PM, to DC mode from 12:45 - 3:49 PM, and back to AC mode for the rest of the
day, as per the Elog [5]. The close correspondence of these changes in our
data to the recorded actions and observations of Booster personnel boost our
confidence in the quality and relevance of the collected dataset.

Additionally, we plot settings changes on March 10 and 11 documented in the
Elog [5] in Figure 7. The blip in B:ACMNPG from 6.5 to 13.5 is visible as is
the slight decrease in B_VIMIN around 4:00 PM CST, which were mentioned in
Table 3.

## Usage Notes

BOOSTR could be used to train various control networks for accelerator
regulation, to construct “digital twins” of the Fermilab Booster regulator’s
control environment, or to develop anomaly detection/categorization
capabilities. Please note: there are no legal or ethical ramifications of
using this data as it was collected from a machine, and not collected from or
representative of people.

In the future, the dataset could feasibly be expanded to include more of the
200,000 available ACNET system parameters and therefore be used to control,
mimic, or monitor further aspects of the particle accelerator complex. One
could argue that this initial dataset might become the foundation upon which
substantial fine-tuning of particle accelerators could depend.

## Code availability

The preprocessing code is available here. When using BOOSTR data, the authors
recommend ordering by time immediately, as the parquet files do not store the
data entries sequentially [2].

## Acknowledgements

This dataset was created as part of the “Accelerator Control with Artificial
Intelligence” Project conducted under the auspices of the Fermilab Laboratory
Directed Research and Development Program (Project ID FNAL-LDRD-2019-027). The
manuscript has been authored by Fermi Research Alliance, LLC under Contract
No. DE-AC02-07CH11359 with the U.S. Department of Energy, Office of Science,
Office of High Energy Physics and is registered at Fermilab as Technical
Report Number FERMILAB-PUB-21-005-SCD.

We are extremely grateful for Brian Schupbach and Kent Triplett for lending
their Booster technical expertise, without which we could not have validated
our dataset. Additionally, we would like to acknowledge Burt Holzman for
guidance on getting set up in the cloud, and the help of the Databricks
federal support staff, in particular Sheila Stewart. Furthermore, we would
like to recognize Jovan Mitrevski and Aleksandra Ciprijanovic for useful
discussions and a careful reading of this manuscript.

## Author contributions statement

J.S. created the data collection script, and set up and maintained the cron
jobs to record the data in HDF5 files. D.K. migrated the data from on-premise
storage to the cloud, wrote the preprocessing and time-alignment code, and
validated the data. Both authors reviewed this manuscript.

## Competing interests

The authors declare no competing interests.

## References

  * [1] Cahill, K. & et. al.  The fermilab accelerator control system.  _ICFA Beam Dyn. Newslett._ 47, 106–124 (2008).
  * [2] Kafkes, D. & St. John, J.  BOOSTR: A Dataset for Accelerator Control Systems (Full Release), 10.5281/zenodo.4382663 (2021).
  * [3] Kafkes, D. & St. John, J.  BOOSTR: A Dataset for Accelerator Control Systems (Partial Release), 10.5281/zenodo.4088982 (2020).
  * [4] John, J. S. _et al._ Real-time artificial intelligence for accelerator control: A study at the fermilab booster (2020).  2011.07371.
  * [5] Hazelwood, K. J.  Fermilab electronic logbook.  www-bd.fnal.gov/Elog.

## Figures & Tables

Figure 1: Overview of current GMPS control system [4]. Presently, a human
operator specifies a target program for B:VIMIN and B:VIMAX, the GMPS
compensated minimum and maximum currents respectively, via the Fermilab
Accelerator Control Network that is transmitted to the GMPS control board.
Table 1: Description of BOOSTR dataset parameters. Here “GMPS" denotes the
Gradient Magnet Power Supplies (1-4), “MI" means main injector, “MDAT" refers
to Fermilab’s Machine Data communications protocol. Device parameters that
begin with B relate to the accelerator Booster, whereas device parameters that
begin with I relate to the Main Injector. Parameter mean and standard
deviation have been truncated to two decimal points.

Parameter | Details (Units) | Setting | Mean (Std) | Missing Dates
---|---|---|---|---
B:ACMNIG | Min AC integral feedback gain | B_ACMNIG | 0.75 (0) | 2019: 6/14, 12/12
B:ACMNPG | Min AC proportional feedback gain | B_ACMNPG | 9.1968 (0.75) | 2019: 6/14, 12/12
B:ACMXIG | Max AC integral feedback gain | B_ACMXIG | 0.30 (0) | 2019: 6/14, 12/12
B:ACMXPG | Max AC proportional feedback gain | B_ACMXPG | 3.00 (0) | 2019: 6/14, 12/12
B:DCIG | DC integral feedback gain | B_DCIG | 0 (0) | 2019: 6/14, 12/12
B:DCPG | DC proportional feedback gain | B_DCPG | 0.10 (1.35) | 2019: 6/14, 12/12
B:GMPS1V | GMPS1 output voltage (V) |  | 81.82 (89.56) | 2019: 6/14, 12/12, 12/30-31
|  |  |  | and 2020: 1/01-4/12
B:GMPS2V | GMPS2 output voltage (V) |  | 85.29 (96.00) | 2019: 6/14, 12/12, 12/30-31
|  |  |  | and 2020: 1/01-4/12
B:GMPS3V | GMPS3 output voltage (V) |  | 63.67 (61.19) | 2019: 6/14, 12/12, 12/30-31
|  |  |  | and 2020: 1/01-4/12
B:GMPS4V | GMPS4 output voltage (V) |  | 61.08 (65.19) | 2019: 6/14, 12/12
B:GMPSBT |  $\partial B/\partial t$ offset trigger (s) | B_GMPSBT | 0 (0) | 2019: 6/14, 12/12
B|GMPSSC | Binary status control of GMPS |  | N/A (N/A) | 2019: 6/07, 6/12, 6/14-15
|  |  |  | and 2020: 1/18, 3/08, 3/15
B:GMPSFF | Feedforward start trigger (s) | B_GMPSFF | 1.32 (1.52) | 2019: 6/14, 12/12
B:IMAXXG | Max transductor gain (A/V) | B_IMAXXG | -117.12 (1.80) | 2019: 6/14, 12/12
B:IMAXXO | Max transductor offset (A) | B_IMAXXO | 10.00 (0) | 2019: 6/14, 12/12
B:IMINXG | Min transductor gain (A/V) | B_IMINXG | -11.73 (0.23) | 2019: 6/14, 12/12
B:IMINXO | Min transductor offset (A) | B_IMINXO | 0 (0) | 2019: 6/14, 12/12
B:IMINER | Discrepancy from setting at min (0.1 A) |  | 1.93 (3.86) | 2019: 6/14, 12/12
B:IMINST |  $\partial B/\partial t$ sample off | B_IMINST | 0 (0) | 2019: 6/14, 12/12
B:IPHSTC | Phase regulator time constant | B_IPHSTC | 20.00 (0.01) | 2019: 6/14, 12/12
B:LINFRQ | 60 Hz line frequency offset (mHz) |  | -0.44 (16.31) | 2019: 6/03-7/11, 12/12
|  |  |  | 12/30-31 and 2020: 1/01-06
B:NGMPS | Number of GMPS suppliers |  | 4.00 (0) | 2019: 6/14, 12/12
B:PS1VGM | GMPS1 V- to ground (V) |  | -2.30 (23.56) | 2019: 6/14, 12/12, 12/30-31
|  |  |  | and 2020: 1/01-4/12
B:PS2VGM | GMPS2 V- to ground (V) |  | -21.29 (27.52) | 2019: 6/14, 12/12, 12/30-31
|  |  |  | and 2020: 1/01-4/12
B:PS3VGM | GMPS3 V- to ground (V) |  | -15.13 (14.11) | 2019: 6/14, 12/12, 12/30-31
|  |  |  | and 2020: 1/01-4/12
B:PS4VGM | GMPS4 V- to ground (V) |  | -26.27 (17.22) | 2019: 6/14, 12/12, 12/30-31
|  |  |  | and 2020: 1/01-4/12
B:PS1VGP | GMPS1 V+ to ground (V) |  | 52.00 (34.82) | 2019: 6/14, 12/12, 12/30-31
|  |  |  | and 2020: 1/01-4/12
B:PS2VGP | GMPS2 V+ to ground (V) |  | 26.53 (30.53) | 2019: 6/14, 12/12, 12/30-31
|  |  |  | and 2020: 1/01-4/12
B:PS3VGP | GMPS3 V+ to ground (V) |  | 20.17 (13.74) | 2019: 6/14, 12/12, 12/30-31
|  |  |  | and 2020: 1/01-4/12
B:PS4VGP | GMPS4 V+ to ground (V) |  | 9.14 (12.75) | 2019: 6/14, 12/12, 12/30-31
|  |  |  | and 2020: 1/01-4/12
B:VIMAX | Compensated max GMPS current (A) | B_VIMAX | 772.43 (385.98) | 2019: 6/14, 12/12
B:VIMIN | Compensated min GMPS current (A) | B_VIMIN | 83.20 (40.68) | 2019: 6/14,
B:VINHBT | Inhibit value | B_VINHBT | 1.00 (0.04) | 2019: 6/14, 12/12
B:VIPHAS | GMPS ramp phase wrt line voltage (rad) | B_VIPHAS | 1.80 (0.14) | 2019: 6/14, 12/12
I:IB | MI lower bend current (A) |  | 2275.71 (2571.03) | 2019: 6/03-19, 12/12
I:MDAT40 | MDAT measured MI current (A) |  | 2400.57 (2576.37) | 2019: 6/03-7/11, 12/12
I:MXIB | Main dipole bend current (A) |  | 2279.68 (2564.53) | 2019: 6/03-11, 6/13-19, 12/12

Table 2: Summary of nearly constant variables in both periods of data collection. Here “nearly constant" denotes variables having a standard deviation less than $10^{-5}$ across both periods. Device | Setting | Constant Value
---|---|---
B:DCIG | B_DCIG | 0
B:GMPSBT | B_GMPSBT | 0.0003
B:IMINST | B_IMINST | 0
B:IMINXO | B:IMINXO | 0
Table 3: Summary of Booster-related electronic log (Elog) [5] entries corresponding to spikes in Figures 2 and 3. Original Central Time times are given with values in parenthesis designating UTC. Here “RF" denotes radiofrequency. Date | Elog Entry Summary
---|---
6/8/2019 | GMPS in AC mode until 8:00 AM (13:00) on 6/08, then switched off until 6/17
6/10/2019 | GMPS was locked/tagged out for outage, West Booster gallery RF off from
8:30-10:00 (13:30-3:00) for work
6/17/2019 | GMPS turned back on and put in AC mode
6/22/2019 | High energy physics beam turned off at 8:00 PM (1:00 +1) (GMPS remained in AC mode)
6/24/2019 | DC studies from 8:00 AM - 6:00 PM (13:00 - 23:00), back to AC mode
6/26/2019 | Alternated between AC and DC mode, GMPS off for 30 min around 5:30 PM (22:30)
6/27/2019 | GMPS in AC mode all day, but removing certain study events caused bias to creep up,
eventually tripping the RF
6/28/2019 | Alternated between AC and DC mode
12/8/2019 |  B_VIMIN adjusted, GMPS in AC mode
12/12/2019 | AC mode, operators reset virtual machine environment locally
12/28 - 12/30/2019 | No beam from injector, GMPS in AC mode
12/31/2019 | Booster injection back, GMPS in AC mode
1/1/2020 | GMPS off for 15 min around 9:30 AM (14:30), in AC mode for rest of day
2/4/2020 | RF sparking in gallery due to reverting of RF capture settings, GMPS in AC mode
2/5/2020 | GMPS off from 6:00 AM - 3:30 PM (11:00 - 20:30), then in AC mode for rest of day
2/6/2020 | Lowered beam intensity to users, but GMPS was in AC mode all day
3/5/2020 | Beam tails were large, so turned B_VIMIN down
3/10/2020 | GMPS in AC mode all day, B:ACMNPG changed from 6.5 to 13.5, B_VIMIN decreased
from 103.440 to 103.420
3/11/2020 | GMPS in AC mode from 12:00 AM - 6:00 AM (5:00 - 11:00), off from 6:00 - 10:00 AM
(11:00 - 15:00), then alternated between AC and DC mode, B_VIMIN adjusted from
| 103.418 to 103.386
3/13/2020 | GMPS off from 9:30 AM - 11:00 AM (14:30-16:00), back on and put in AC mode
3/20/2020 | Booster turned off on account of Covid-19 pandemic at 12:00 PM (17:00)

Figure 2: Metadata variable trends for Period 1: June 3, 2019 to June 30,
2019. The graphs show the mean for each variable on the given date and shades
in the standard deviation of that variable on that date.

Figure 3: Metadata variable trends for Period 2: December 2, 2019 to April 13,
2020. The graphs show the mean for each variable on the given date and shades
in the standard deviation of that variable on that date. Figure 4: Heatmap of
histogram distributions for each reading and setting variable with equal
sampling. This is only meant to characterize the centrality of each recorded
value. See Fig. 2 and 3 for actual metadata value ranges.

Figure 5: Daily values of B|GMPSSC (should be interpreted as taking the mode
for each day) whose bits encode relevant Booster statuses. Figure 6: Values of
status B|GMPSSC corresponding to Table 3 entries for June 24, 2019 and March
11, 2020 (timestamps were put in Central Time to align with Elog). Recall: a
value of 159 indicates AC study/GMPS on, 31 indicates DC study/GMPS on, and
407 indicates AC study/GMPS off. These traces display this value at a much
greater granularity than Figure 5.

Figure 7: Switches corresponding to Table 3 entries for March 10, 2020 and
March 11, 2020: B:ACMNPG changed from 6.5 to 13.5 and B_VIMIN decreased from
103.418 to 103.386 (timestamps were put in Central Time to align with Elog).
The sudden large increase in B_VIMIN from 12:45 - 3:49 PM CST to a value off
the plotted region corresponds to the DC mode observed in Figure 6.


# DEEM: Dynamic Experienced Expert Modeling for Stance Detection

Xiaolong Wang*,1, Yile Wang*,2, Sijie Cheng1,2, Peng Li🖂,2, Yang Liu🖂,1,2
1Department of Computer Science and Technology, Tsinghua University, Beijing,
China
2Institute for AI Industry Research (AIR), Tsinghua University, Beijing, China
<EMAIL_ADDRESS><EMAIL_ADDRESS>
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Recent work has made a preliminary attempt to use large language models (LLMs)
to solve the stance detection task, showing promising results. However,
considering that stance detection usually requires detailed background
knowledge, the vanilla reasoning method may neglect the domain knowledge to
make a professional and accurate analysis. Thus, there is still room for
improvement of LLMs reasoning, especially in leveraging the generation
capability of LLMs to simulate specific experts (i.e., multi-agents) to detect
the stance. In this paper, different from existing multi-agent works that
require detailed descriptions and use fixed experts, we propose a Dynamic
Experienced Expert Modeling (DEEM) method which can leverage the generated
experienced experts and let LLMs reason in a semi-parametric way, making the
experts more generalizable and reliable. Experimental results demonstrate that
DEEM consistently achieves the best results on three standard benchmarks,
outperforms methods with self-consistency reasoning, and reduces the bias of
LLMs.

11footnotetext: Equal contribution.11footnotetext: Corresponding authors.

##  1 Introduction

Stance detection (Hasan and Ng, 2014; Küçük and Can, 2020) is a natural
language processing (NLP) task that automatically identifies the stance
towards a specific target in a given text. For example, the stance of
“Secretary SecPompeo is as corrupt as every other member of the Trump” is
against Donald Trump. Such a task has been shown to play an important role in
gaining insights into public opinion (Darwish et al., 2017; Lai et al., 2020),
understanding political polarization (Darwish et al., 2018), and tracking
ideological trends from social media (Conforti et al., 2020).

Recently, large language models (LLMs; Brown et al., 2020; Chowdhery et al.,
2022; Ouyang et al., 2022; Wang et al., 2023a) are developing rapidly and can
be applied to various tasks. For example, Zhang et al. (2022) empirically
confirms that ChatGPT can achieve impressive performance to detect stance in a
zero-shot setting. Zhang et al. (2023a) further improve the results by using
chain-of-thought reasoning strategies (Wei et al., 2022; Cheng et al., 2023).
These works have opened up new directions in stance detection.

Despite the success of applying LLMs, conventional reasoning techniques with
LLMs could cause hallucinations and factual errors (Guerreiro et al., 2023; Ji
et al., 2023), particularly in stance detection. Texts in stance detection
usually originate from social media AlDayel and Magdy (2021), which are
typically short and intricate, necessitating additional domain expertise He et
al. (2022). For example, to detect the stance towards Biden in “Are you
actually trying, as president of the U.S., to start a war??!!
#VoteBlueToSaveAmerica2020 #Biden”, we need to know which camp Biden belongs
to and the meaning of “#VoteBlueToSaveAmerica2020”.

Figure 1: Illustration of our DEEM method. Top: Vanilla multi-agent reasoning
for stance detection through generation. Bottom: Our method first generates
and filters experienced experts by leveraging training data, then retrieves
the related ones during reasoning.

Inspired by the wisdom of crowds in sociological theory (Minsky, 1988; Piaget,
2013), we intuitively propose designing multiple capable experts to
collaborate in order to come up with a comprehensive stance prediction.
Previous studies (Du et al., 2023; Wang et al., 2023e) have attempted to solve
reasoning tasks with multi-agent debate and multi-persona self-collaboration.
However, their designed agents are generally pre-defined or automatically
generated by LLMs, which either require strong prior knowledge or need to be
further improved for stance detection tasks. Obviously, pre-defined agents are
fixed, thus it is difficult to adapt to different contexts in social media.
Moreover, fully generated agents by LLMs may not be suitable due to the
intricate contextualized information, especially in specific domains.

In this work, we propose DEEM, a Dynamic Experienced Expert Modeling method to
solve stance detection tasks, as shown in Figure 1. In particular, to better
gather the potential expertise for stance detection, we first leverage labeled
samples from the existing training data to generate diverse experts. Then, we
design two heuristic rules, namely occurrence numbers and response accuracy,
to filter the experienced experts and construct an expert repository. Finally,
instead of using a fully generative approach, we adopt a dynamic retrieval
method to identify relevant experienced experts for new sentences,
facilitating discussions for the final prediction.

We evaluate DEEM across both single-target and multi-target stance detection
tasks on three widely used datasets, including P-Stance (Li et al., 2021),
SemEval-2016 (Mohammad et al., 2016), and MTSD (Sobhani et al., 2017).
Experimental results demonstrate that DEEM with dynamic experienced experts
can gain substantial improvement across all datasets. Furthermore, it also
outperforms reasoning with self-consistency that requires multiple responses
and shows potential for reducing the bias of LLMs.

##  2 Related Work

Stance Detection. Early works on stance detection mainly take it as a
classification task, leveraging the small language models (Devlin et al.,
2019; Nguyen et al., 2020) and learning features from either in-domain or
cross-domain training datasets (Augenstein et al., 2016; Zhang et al., 2019;
Allaway et al., 2021; Liu et al., 2021; Liang et al., 2022b). With the
emergence of LLMs, Zhang et al. (2022, 2023a) first try using ChatGPT to solve
the task directly by zero-shot or few-shot reasoning with chain-of-thought,
which only requires simple prompts to obtain the political stance from the
generated responses. In comparison to their methods, we take inspiration from
the multi-agent (Wang et al., 2023b; Xi et al., 2023) and introduce a novel
dynamic expert mechanism, enabling LLMs to generate responses from multiple
perspectives, providing more comprehensive responses and improve the
prediction accuracy.

LLMs Reasoning with Multi-Agent. Multi-agent strategies have proven to be
effective in LLMs reasoning Talebirad and Nadiri (2023); Li et al. (2023). By
using prompts or instructions that specify the desired role or persona, the
model can generate responses based on its understanding of that role and can
apply to more complicated scenarios such as social interaction (Park et al.,
2023), court simulation (Hamilton, 2023), code development (Qian et al.,
2023), and engaging communication games (Xu et al., 2023b). The above works
often require detailed role specialization at the beginning of each task (Xu
et al., 2023a). In contrast, we propose to discover the experienced personas
automatically with the least human effort by using the existing labeled
samples, then call for dynamic expertise by retrieving the collected
“sentence-expert” pairs. The entire process minimizes the prior knowledge
required from stance detection.

##  3 Methods

Figure 2: An example of a few-shot prompt with expert modeling for stance
detection. The underlined parts are the sentence, target, and label,
respectively. The green texts indicate the manually written experts. Figure 3:
The overall pipeline of our DEEM method. We first use the prompt and training
sentences to generate diverse experts ($\S$ 3.1). Then we filter experienced
experts according to their occurrence numbers and performance ($\S$ 3.2).
Finally, we build the sentence-expert pairs and retrieve the experienced
experts for each new sentence ($\S$ 3.3).

The overall pipeline of the proposed DEEM is shown in Figure 3. DEEM first
generates diverse experts by leveraging the training data. Then it filters the
experienced experts that are generalizable and reliable. Finally, it retrieves
the experts according to new queried sentences and makes responses.

###  3.1 Generating Diverse Experts

Conventionally, training datasets are used to fine-tune the parameters of
small models. Recently, they can be chosen as prompts for better few-shot in-
context learning for LLMs Liu et al. (2022); Zhang et al. (2023b); Shum et al.
(2023). In this paper, we leverage the existing training datasets in a novel
way to help generate potentially useful experts for solving stance detection,
without using much prior knowledge or detailed role descriptions.

Firstly, given the training dataset with sentence-target-label triplets
$\mathcal{D}=\\{s_{j},t_{j},l_{j}\\}_{j=1}^{|\mathcal{D}|}$, we randomly
select some of them as held-out ones to construct a prompt as shown in Figure
2:

${\rm prompt}=s_{p}\oplus t_{p}\oplus\mathcal{E}_{p}\oplus l_{p},\ $ (1)

where $\oplus$ denotes textual concatenation, $s_{p}$, $t_{p}$, $l_{p}$ are
the sentence, target, and label of the selected instance in prompt,
$\mathcal{E}_{p}=\\{e^{1}_{p},...,e^{k}_{p}\\}$ are the manually written
experts corresponding to the selected instance, such as “Social Media Expert”.

Then, we use the LLM $\mathcal{M}$ to generate experts and predicted labels
for all other sentences via few-shot in-context learning (Brown et al., 2020):

$\mathcal{E}_{j},\hat{l}_{j}={\mathcal{M}}({\rm prompt}\oplus s_{j}\oplus
t_{j}),M$ (2)

where $\mathcal{E}_{j}=\\{e^{1}_{j}$, …, $e^{k}_{j}\\}$ denotes the generated
experts, and $\hat{l}_{j}$ indicates the predicted label to the sentence
$s_{j}$ and target $t_{j}$.

It is worth noting that more than 1,400 distinct experts can be generated,
showing LLMs’ strong capability of in-context learning. We further show the
detailed expert distributions in Section 5.1.

###  3.2 Filtering Experienced Experts

The generated expert candidates in $\mathcal{E}$ are directly generated by
LLMs, which are diverse enough but many do not always match the sentences. To
make them more generalizable and reliable to new sentences, we designed two
heuristic rules to filter the experienced experts among all candidates.

First, despite a large number of generated experts, many of them only appear a
few times. These low-frequency experts are usually from unrelated domains,
making it difficult to generalize to new sentences for stance detection. Thus,
we require experts who are experienced in stance detection tasks. To fulfill
this requirement, the first heuristic rule is the total occurrence numbers
${\rm Count}(\cdot)$ of each expert $e_{j}^{m}\in\mathcal{E}_{j}$ in the
training dataset as below:

${\rm
Count}(e_{j}^{m})=\sum_{e_{i}\in\mathcal{E}}\mathbbm{1}({e_{j}^{m}=e_{i})},$
(3)

where $e_{i}$ is the $i$-th element in the collection of generated experts
$\mathcal{E}$, and $\mathbbm{1}(e_{j}^{m}=e_{i})$ is the indicator function
that equals 1 when $e_{j}^{m}$ equals to $e_{i}$ and 0 otherwise.

Second, it is well known that LLMs can occasionally generate hallucinations
Guerreiro et al. (2023); Ji et al. (2023), thus they could make the responses
unreliable and lead to incorrect final predictions. Therefore, experienced
experts need to be accurate in analyzing the stance towards the target, thus
the second heuristic rule is the total prediction accuracy ${\rm Acc}(\cdot)$
of each expert $e_{j}^{m}\in\mathcal{E}_{j}$:

$\displaystyle{\rm
Acc}(e_{j}^{m})=\frac{\sum_{e_{j}^{m}=e_{i}}\mathbbm{1}(\hat{l}_{i}=l_{i})}{\sum_{e_{i}\in\mathcal{E}}\mathbbm{1}({e_{j}^{m}=e_{i})}},$
(4)

where $\hat{l}_{i}$ is the predicted label corresponding to the $i$-th element
$e_{i}$ in Eq. 2, and $l_{i}$ is the ground-truth label for sentence $s_{i}$.

Finally, we discard the expert $e_{j}^{m}\in\mathcal{E}_{j}$ who have low
prediction accuracy as the threshold (e.g., ${\rm Acc}(e^{m}_{j})<50\%$). Then
we select the rest of the top-$k$ experts (e.g., $k=10\sim 30$) according to
their occurrence numbers ${\rm Count}(e_{j}^{m})$. We take these selected
experts $\mathcal{E}^{\prime}_{j}$ in each sentence $s_{j}$ as the final
experienced experts. Moreover, we regard all of them as the expert pool
$\mathcal{E}^{\prime}=\mathcal{E}^{\prime}_{1}\cup\cdots\cup\mathcal{E}^{\prime}_{|\mathcal{D}|}$
to solve the stance detection for a new sentence. We discuss the settings of
${\rm Acc}(e^{m}_{j})$ and top-$k$ in Section 5.2.

###  3.3 Retrieving Related Experts

Verified on the training set, all the experts in the expert pool are
experienced with both high occurrence and accuracy, these pieces of
information can be utilized during the testing phase for retrieval and placed
in the prompt as inputs to the model Guo et al. (2023); Wang et al. (2023d).
Overall, to ascertain the stance of a new sentence, it is more effective to
choose experienced experts from similar sentences, rather than having LLMs
generate potentially unrelated or inexperienced experts directly.

Specifically, we construct a repository to better match the new sentence and
these experienced experts. We assign the sentence $s_{j}$ in the training
dataset as the key. Meanwhile, its corresponding filtered experienced experts
$\mathcal{E}^{\prime}_{j}$ as the value. The resulting sentence-expert pair
$\langle s_{j},\mathcal{E}^{\prime}_{j}\rangle$ indicates that the filtered
experienced experts $\mathcal{E}^{\prime}_{j}$ can accurately and expertly
detect the stance of the sentence $s_{j}$, thus they have potential to be
applied to the similar sentence to $s_{j}$.

In the inference phase, given a new sentence $s$ and the constructed sentence-
expert repository, we can retrieve the top-$h$ related experienced experts
according to the textual similarity scores:

$\displaystyle\text{Sim}(s,s_{j})=\frac{\exp({\rm Enc}(s)\cdot{\rm
Enc}(s_{j}))}{\sum_{i=1}^{\left|\mathcal{D}\right|}\exp({\rm Enc}(s)\cdot{\rm
Enc}(s_{i}))},$ (5)

where ${\rm Enc}(\cdot)$ is a sentence encoder, such as SimCSE Gao et al.
(2021).

Method | Including | Multi- | Verified | Reasoning
---|---|---|---|---
Explanations | Roles | Experts | Type
Few-Shot | ✗ | ✗ | - | Gen
CoT | ✓ | ✗ | - | Gen
Auto-CoT | ✓ | ✗ | - | Re+Gen
ExpertPrompt | ✓ | ✗ | ✗ | Gen
SPP | ✓ | ✓ | ✗ | Gen
DEEM (ours) | ✓ | ✓ | ✓ | Re+Gen

Table 1: Comparison to typical reasoning methods including Few-Shot, chain-of-
thought (CoT), Auto-CoT, ExpertPrompt, and solo performance prompting (SPP).
Re: Retrieval. Gen: Generation.

Finally, we obtain the top-$h$ experienced experts and directly append them to
the prompt as shown in Figure 3. Then we use the whole prompt as the input for
LLMs to generate the upcoming experts’ discussion and the final predicted
answer.

###  3.4 Comparison with Other Methods

We compare our method with typical reasoning approaches in Table 1. To involve
explanations or specific roles during the reasoning process, CoT Wei et al.
(2022), ExpertPrompt Xu et al. (2023a) and SPP Wang et al. (2023e) let LLMs
fully generate the explanations or discussions between experts by using the
prompt. As for our method, we propose to explore experienced experts from
training samples and introduce a retrieval mechanism during the reasoning
process.

According to the retrieval mechanism (Borgeaud et al., 2022), Auto-CoT Zhang
et al. (2023b) finds similar samples as demonstrations in the prompt. In
contrast, we retrieve according to constructed “sentence-expert” pairs during
reasoning, which makes the involved experts more related to the current
sentence in a semi-parametric manner.

##  4 Experiments

###  4.1 Experimental Setups

Datasets. To evaluate the effectiveness of our method, we comprehensively use
three standard stance detection datasets, including both single-target and
multi-target tasks: (1) P-stance (Li et al., 2021) is a political stance
detection dataset, with each tweet annotated for its stance towards one of
three politicians; (2) SemEval-2016 (Mohammad et al., 2016) introduces a
shared task on stance detection from tweets, including six targets with one
target exclusively for testing; (3) MSTD (Sobhani et al., 2017) is a dataset
for multi-target stance detection, primarily focusing on four presidential
candidates in the 2016 US election using specific hashtags. Among these
datasets, we mainly focus on politicians, such as “Donald Trump”. The detailed
statistics are shown in Table 2.

Datasets | Target | Train | Test
---|---|---|---
P-Stance | Trump | 6,362 | 796
Biden | 5,806 | 745
Sanders | 5,056 | 635
SemEval-2016 | Clinton | 1,898 | 984
Trump | 2,194 | 707
MTSD | Trump-Clinton | 1,240 | 355
Trump-Cruz |  922 | 263
Clinton-Sanders |  957 | 272

Table 2: Statistics of P-Stance, SemEval-2016, and MTSD datasets in our
experiments.

Type | Method | P-Stance | SemEval-2016 | MTSD | Avg.
---|---|---|---|---|---
DT | JB | BS | HC | DT | DT-HC | DT-TC | HC-BS
FT | BiCond (Augenstein et al., 2016)♠ | 73.0 | 69.4 | 64.6 |  32.7† |  30.5† | - | - | - | -
BERT (Devlin et al., 2019)♠ | 81.6 | 81.7 | 78.4 |  49.6† |  40.1† | - | - | - | -
BERTweet (Nguyen et al., 2020) | 82.4 | 81.0 | 78.1 |  50.9† |  42.2† | 69.2 | 70.7 | 69.0 | 67.9
JointCL (Liang et al., 2022b)♡ | - | - | - |  54.8† |  50.5† | - | - | - | -
(text-davinci-003)
ZS | Zero-Shot (Brown et al., 2020) | 73.8 | 83.3 | 77.5 | 71.8 | 68.3 | 61.6 | 64.7 | 61.4 | 70.3
DQA (Zhang et al., 2022) | 73.0 | 80.8 | 76.1 | 72.7 | 69.9 | 58.9 | 66.4 | 63.3 | 70.1
1-11 FS | Few-Shot (Brown et al., 2020) | 79.9 | 85.2 | 78.6 | 79.4 | 73.5 | 68.6 | 65.9 | 70.7 | 75.2
($d=2$) | Manual-CoT (Wei et al., 2022) | 79.3 | 84.9 | 78.4 | 77.2 | 72.5 | 75.0 | 75.6 | 68.8 | 76.5
StSQA (Zhang et al., 2023a) | 75.2 | 85.2 | 78.9 | 78.3 | 72.3 | 72.6 | 75.9 | 72.0 | 76.3
Auto-CoT (Zhang et al., 2023b) | 82.9 | 84.7 | 78.4 | 80.7 | 73.8 | 67.9 | 67.4 | 75.3 | 76.4
ExpertPrompt (Xu et al., 2023a) | 82.8 | 85.5 | 78.7 | 85.2 | 73.0 | 74.1 | 76.8 | 71.5 | 78.5
SPP (Wang et al., 2023e) | 83.4 | 85.5 | 79.6 | 85.5 | 73.3 | 73.0 | 78.0 | 76.8 | 79.4
DEEM (ours) | 83.7 | 86.0 | 80.4 | 85.7 | 74.8 | 76.5 | 80.1 | 81.3 | 81.1
$\Delta$ (compare w/ second-best results) | +0.3 | +0.5 | +0.8 | +0.2 | +1.0 | +1.5 | +2.1 | +4.5 | +1.7
(gpt-3.5-turbo-0301)
ZS | Zero-Shot (Brown et al., 2020) | 83.3 | 82.5 | 79.4 | 79.3 | 71.4 | 73.5 | 67.0 | 73.6 | 76.3
DQA (Zhang et al., 2022)♠ | 83.2 | 82.0 | 79.4 | 78.0 | 71.3 | 66.2 | 63.2 | 69.3 | 74.1
1-11 FS | Few-Shot (Brown et al., 2020) | 83.6 | 83.1 | 80.8 | 79.3 | 71.6 | 76.6 | 78.2 | 72.8 | 78.3
($d=2$) | Manual-CoT (Wei et al., 2022) | 85.4 | 83.8 | 80.9 | 79.5 | 71.2 | 77.0 | 77.5 | 76.7 | 79.0
StSQA (Zhang et al., 2023a)♠ | 85.7 | 82.8 | 80.8 | 78.9 | 71.6 | 77.5 | 78.2 | 81.2 | 79.6
Auto-CoT (Zhang et al., 2023b) | 84.1 | 82.8 | 80.6 | 84.6 | 73.5 | 77.0 | 76.9 | 76.7 | 79.5
ExpertPrompt (Xu et al., 2023a) | 84.7 | 84.7 | 81.2 | 83.8 | 77.4 | 80.6 | 77.0 | 79.0 | 81.1
SPP (Wang et al., 2023e) | 85.1 | 84.6 | 81.5 | 85.3 | 79.5 | 79.5 | 79.8 | 79.8 | 81.9
DEEM (ours) | 86.4 | 86.1 | 82.1 | 85.9 | 80.5 | 81.7 | 80.7 | 83.5 | 83.4
$\Delta$ (compare w/ second-best results) | +0.7 | +1.4 | +0.6 | +0.6 | +1.0 | +1.1 | +0.9 | +2.3 | +2.5

Table 3: Main results of baselines and our proposed DEEM. FT: Fine-tuning, ZS:
Zero-shot, FS: Few-shot. DT: Donald Trump, JB: Joe Biden, BS: Bernie Sanders,
HC: Hillary Clinton, TC: Ted Cruz. ${\dagger}$: cross-target setting.
$\spadesuit$: reported by Zhang et al. (2023a), $\heartsuit$: reported by
Liang et al. (2022b). The other results are achieved via our implementation.
The best results are in bold and the second-best results are underlined.

Baselines. Besides the methods that require supervised fine-tuning, we also
compared our method with recent methods by using LLMs without modifying model
parameters:

  * •

DQA (Zhang et al., 2022) uses the template “What is the attitude of the
sentence : [Tweet] to the target: [Target]. ‘favor’ or ‘against’.” and extract
the answers by question answering.

  * •

CoT (Wei et al., 2022) manually provides the explanations in demonstrations
and enhances the chain-of-thought reasoning ability of LLMs.

  * •

StSQA (Zhang et al., 2023a) proposes automatic “thought-inducing” and add them
to the demonstrations for step-by-step question answering.

  * •

Auto-CoT (Zhang et al., 2023b) automatically selects demonstrations from
training data according to semantic diversity.

  * •

ExpertPrompt (Xu et al., 2023a) introduces the identity of experts and
customizes information descriptions for LLMs before giving responses.

  * •

SPP (Wang et al., 2023e) proposes solo performance prompting by engaging in
multi-turn collaboration with multi-persona during reasoning.

Metric. Following Liang et al. (2022a) and Zhang et al. (2023a), we use the
F1avg, i.e., the average of F1-score on the label ‘favor’ and ‘against’, as
the metric for evaluation and comparison.

Implementation Details. We employ InstructGPT (text-davinci-003) and ChatGPT
(gpt-3.5-turbo-0301) as in Zhang et al. (2022, 2023a) through OpenAI
API111https://platform.openai.com/, we also tried the newer version model of
gpt-3.5-turbo-0613, but the results did not show significant improvement.. We
set the number of demonstrations $d$ in both the first and third stages as 2.
The number of experts (i.e., the number $k$) in the first stage is 3, and the
number of top-$h$ in the third stage is also 3. The temperature is set to 0 to
ensure the reproducibility of the LLMs’ responses.

###  4.2 Main Results

The main comparison results on three standard datasets are reported in Table
3. For fine-tuning models, BERT and BERTweet achieve comparable results on
P-Stance, while BERTweet achieves relatively good performance on both
SemEval-2016 and MTSD datasets. Moreover, JointCL obtains the highest
performance on SemEval-2016. These phenomena show that small models can
capture domain knowledge by fine-tuning useful data to further enhance the
results.

Target | Frequency (Proportion) | Accuracy (Proportion)
---|---|---
>1% (2.98%) | 0.05-1% (24.32%) | <0.05% (72.70%) | >80% (58.06%) | 50-80% (10.67%) | <50% (31.27%)
Sanders | Political | LeaderShip | Future_Prediction | Immigration | Media | Banking
Ethics | History | Progmatism | Economic | Political | Comedy
Economic | Technology | Transparency | History | Polling | Alcohol
Trump | Immigration | Corruption | Energy | Religious | Political | Slang
Political | Social_Policy | Ethanol | Election | Social_Media | Nationality
Military_Science | Taxation | Deception | Ethics | Economic | Endorsement
Clinton | Gender | Technology | Fash_Food | Ethics | Unity | Geography
Healthcare | Labor | Alcohol_Policy | Legal | Gender | Diversity
Political | Fashion | Leadership | National_Security | Political | Anarchist
Cruz | Media | Unity | Values | National_Security | Political | Music
Political | Technology | Nationality | History | Language | TeaParty
Legal | Polling | Chess | Religious_Studies | Election | Voting

Table 4: Distributions and examples of diverse generated experts with
different frequencies (Left) and prediction accuracy (Right) to different
targets.

As for LLMs, zero-shot methods using InstructGPT can not surpass fine-tuning
models in both the P-Stance and MTSD datasets, showing that LLMs with a larger
number of parameters do not achieve better results without using specific
strategies. Traditional few-shot learning and their reasoning methods obtain
significant improvements, especially under InstructGPT, indicating
demonstrations and chain-of-thoughts strategies are effective for solving
complex tasks. Moreover, the strong performances of ExpertPrompt and SPP prove
that using experts is quite useful for stance detection. However, their
improvement is not always stable. For example, SPP performs well on P-stance
tasks but struggles on SemEval-2016 by using Instruct.

Our method DEEM consistently yields superior results across all three
datasets, regardless of whether the InstructGPT or ChatGPT model is utilized.
One of the reasons for this superior performance is that DEEM effectively
leverages the knowledge within LLMs and adapts it to specific tasks using
expert domain knowledge. In some cases, the advantage of DEEM is particularly
large (e.g., in the multi-target setting), possibly because it can better
capture the underlying structure and relationships within the data. Moreover,
compared with expert-based methods, the performance of DEEM is much more
stable, indicating that introducing the retrieving module can recall more
suitable experts. Overall, DEEM can generalize well and perform better than
all other methods, benefiting from dynamic experienced experts and our
retrieval mechanism.

##  5 Analyses and Discussion

In this section, we conduct a series of analyses to probe the reason behind
the effectiveness of our proposed method DEEM. Specifically, we mainly conduct
these experiments on the only multi-target stance detection task, i.e., the
MTSD dataset.

###  5.1 Expert Distributions

We first investigate expert types according to their frequency and accuracy in
the generating stage (Section 3.1). The examples and the distributions of the
generated experts are shown in Table 4.

Frequency. As for the frequency, we can see that “political experts” appear
many times for all targets, showing that LLMs can uncover the shared
characteristics of political character and use this to solve the stance
detection task. We also find that different targets can exhibit some
distinctive types of experts. For example, the “Gender Expert” appears when
the target is “Hillary Clinton”. Overall, it shows an unbalanced distribution
where 72.70% experts have a low frequency, i.e., less than 0.05%. Intuitively,
the majority of these low-frequency experts (e.g., “Ethanol Expert”, “Chess
Expert”) do indeed have limited generalizability for stance detection tasks.

Accuracy. According to the accuracy, some experts do not show good performance
for stance detection. For example, over 30% experts achieve an accuracy of
less than 50%. As for the “political experts”, the accuracy is mediocre
(ranging from 50% to 80%), showing that high frequency does not always lead to
high accuracy. Thus we need to combine both the frequency and accuracy to
filter experienced ones for better solving stance detection tasks.

###  5.2 Filtering Strategies

We then investigate the accuracy and frequency threshold for filtering
experienced experts (Section 3.2), the averaged results for three multi-target
pairs are shown in Figure 4.

Figure 4: Impact of filtering strategies according to accuracy (Top) and
frequency (Bottom).

(a) Number of Demonstrations |  (b) Number of Experts |  (c) Number of Discussion Turns
---|---|---

Figure 5: Impact of (a) demonstrations, (b) retrieved experts, and (c)
discussion turns during reasoning.

Accuracy Threshold. The impact of the accuracy threshold is shown at the top
of Figure 4. The results are relatively bad when the threshold is low (e.g.,
around 80% when accuracy is lower than 30%). This shows that the experts with
low accuracy may not be generalizable enough. We find that the 50% threshold
leads to the best results (around 82%), and the performance does not improve
when the threshold continues to increase. The reason can be that an
intermediate threshold can maintain both the generalization and diversity of
potential useful experts for new sentences, thus showing the best results for
test sentences.

Frequency Threshold. We set different frequency thresholds, i.e., different
top-$k$ selected experts in the expert pool. The results are shown at the
bottom of Figure 4. We can see that 10$\sim$30 experts achieve better results.
The performance largely reduces when the number of $k$ increases, showing the
negative influence of involving the possible unrelated experts with useless
information.

###  5.3 Dynamic Experts vs. Fixed Experts

Figure 6: Comparing dynamic experts with fixed experts according to frequency
and accuracy.

To prove the effectiveness of our dynamic experts mechanism, we compare fixed
experts that directly rely on frequency and accuracy for selecting experts.
Specifically, we attempt to use the top-3 and bottom-3 experts according to
frequency and accuracy. The results are shown in Figure 6.

As we can see, the performance by fixed experts is worse than DEEM with
dynamic experts, even if we set the top-3 ones with the highest frequency or
accuracy. The performance by using the experts with the least frequency and
accuracy further decreases the performance. These findings show that
dynamically retrieving the suitably experienced experts according to specific
sentences is useful.

###  5.4 Impact of Demonstrations, Experts, and Discussion Turns

For retrieving the experts and prompting LLMs for predictions (Section 3.3),
we investigate the impact of the numbers of demonstrations, retrieved experts,
and discussion turns. The overall results are shown in Figure 5.

Demonstrations. The results depicted in Figure 5(a) indicate sub-optimal
performance during zero- or one-shot reasoning, i.e., when the number of
demonstrates is 0 or 1. However, the performance progressively improves and
eventually plateaus with the presence of 2 or more demonstrates. This trend
suggests that LLMs can deliver commendable performance when provided with
exemplars labeled both ’favor’ and ’against’.

Retrieved Experts. In Figure 5(b), we find that 2 to 5 experts achieve the
best performance, and the performance dramatically drops when the experts are
10 or more. This shows that engaging more experts does not consistently lead
to better performance, which can be due to introducing noise from unrelated
experts, as discussed in Section. 5.2.

Discussion Turns. From Figure 5(c), it is apparent that a single turn can
already yield satisfactory performance. Additional turns (e.g., 3 or 4) bring
improvement. This could be attributed to two primary factors: 1) The
difficulty in generating high-quality expert discussions increases with
multiple turns, potentially diluting the effectiveness of demonstrations. 2)
Complex multi-step reasoning may not be a requisite for stance detection
within a sentence.

###  5.5 Effect of Expert Modeling

To investigate the effect of “experts”, we compare self-consistency reasoning
and two variants of our proposed DEEM. The results are shown in Table 5.

Method | DT-HC | DT-TC | HC-BS
---|---|---|---
Few-Shot | 76.6 | 78.2 | 72.8
Few-Shot + SC (N=3) | 76.3 | 80.1 | 76.7
DEEM w/ “Person A/B/C” | 77.2 | 79.0 | 77.0
DEEM w/ “Expert A/B/C” | 78.6 | 78.7 | 76.7
DEEM (Ours) | 81.7 | 80.7 | 83.5

Table 5: Results of few-shot reasoning, self-consistency (SC) reasoning, and
variants of our DEEM method. By leveraging multiple experts, our DEEM method
can outperforms self-consistency-based methods in a single response.

Comparison with Self-Consistency Reasoning. Unlike methods without using
experts, our method models three experts and generates the prediction
according to their analysis, thus it integrates contextualized information
from multiple reasoning paths, which is similar to the self-consistency
reasoning method Wang et al. (2023c). The difference is that self-consistency
reasoning requires multiple API calls and multiple responses to get the
prediction through voting. From the results we can see that using self-
consistency reasoning can improve the few-shot results, showing that the
single-reply approach limits the performance. Our method with multi-expert can
obtain comparable or better predictions through a single response.

Comparison with Substitute Roles. In our method, we specify the profession of
the experts, i.e., social media experts, political experts, etc. To make a
comparison, we remove the profession and use the substitute roles “expert
A/B/C” or “person A/B/C” to involve the multi-role discussion. From the
results we can see that our method with specified experts achieves the best
results, showing that expert modeling is useful for stance detection, which
can offer more reliable results.

###  5.6 Multi-Experts as Bias Reduction

Method | Trump | Biden | Sanders | Avg.
---|---|---|---|---
BERTweet | 47.6 | 53.6 | 53.6 | 51.6
DQA | 43.9 | 45.7 | 45.2 | 44.9
StSQA | 54.5 | 55.1 | 53.6 | 54.4
DEEM (Ours) | 60.9 | 64.8 | 67.6 | 64.4

Table 6: Comparison between methods for the samples in P-Stance dataset with
the label “neutral”. The results show that our DEEM method show less bias
towards these neutral expressions.

Given the training data distribution, LLMs have shown bias towards specific
targets. As Zhang et al. (2023a) demonstrates, ChatGPT tends to show bias
towards certain topics with no evident stance, either through direct prompting
or chain-of-thought reasoning.

We explore whether the incorporation of multiple experts can mitigate this
stance bias. We focus on sentences labeled as ‘neutral’ and compare predicted
stances using different methods. The results are shown in Table 6. For neutral
stance samples, the outcomes from DQA and StSQA sometimes fall below or match
those from the fine-tuned BERTweet model. This suggests that large models do
carry some stance bias, corroborating Zhang et al. (2023a)’s findings.
However, our multi-expert method consistently yields the best results for
‘neutral’ cases, demonstrating the potential of multiple experts in reducing
stance bias, even in the few-shot settings without further parameter updates.

Figure 7: Example of solo performance prompting Wang et al. (2023e) with
generated experts (Middle) and our method with filtered experienced experts
through retrieval (Bottom).

###  5.7 Case Study

One example is given in Figure 7 to show the comparison between the fully
generated expert and the retrieved experienced expert. We can see that LLMs
misunderstand the text “DOCTOR” and generate the unmatched “Medical Expert”,
giving incorrect responses with hallucinations. This reflects an important
issue with current LLMs in role-playing, which is the inability to generate
appropriate characters to solve specific tasks when the roles are not
provided, especially in non-general domain problems.

In contrast, our DEEM method can determine that this is not a problem related
to the medical domain based on the experience of collecting diverse experts,
retrieving the experienced “Political Expert” according to this specific
question and show much more reliable responses in the final.

##  6 Conclusion

We propose DEEM, a dynamic experienced expert modeling method for stance
detection. Different from existing multi-agent reasoning methods, DEEM first
generates possible diverse experts without leveraging domain knowledge and
detailed expert descriptions, then filters the experienced experts to fulfill
the generalizability and reliability, and finally it involves a retrieval
mechanism during reasoning. Experimental results on both InstructGPT and
ChatGPT model show that DEEM achieves a consistent improvement over all
baselines across three benchmark datasets, outperforming methods with self-
consistency reasoning, reducing the bias of LLMs, and mitigate potential
hallucinations brought from generated inappropriate experts.

## Limitations

There are several limitations that should be considered for our proposed DEEM
method:

Limited Generalizability to Other Domains: The present study confines its
scope to three established benchmark datasets, raising questions about the
extent to which our DEEM method generalizes to diverse domains. A more
comprehensive investigation into the method’s applicability across various
content categories and targets is warranted.

Dependency on Large Language Models: The efficacy of DEEM heavily relies on
the availability and accessibility of large language models. This dependence
may pose challenges in resource-constrained environments, thus limiting the
practical deployment of the proposed approach.

## References

  * AlDayel and Magdy (2021) Abeer AlDayel and Walid Magdy. 2021.  Stance detection on social media: State of the art and trends.  _Information Processing & Management_, 58(4):102597.
  * Allaway et al. (2021) Emily Allaway, Malavika Srikanth, and Kathleen McKeown. 2021.  Adversarial learning for zero-shot stance detection on social media.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 4756–4767, Online. Association for Computational Linguistics.
  * Augenstein et al. (2016) Isabelle Augenstein, Tim Rocktäschel, Andreas Vlachos, and Kalina Bontcheva. 2016.  Stance detection with bidirectional conditional encoding.  In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_ , pages 876–885, Austin, Texas. Association for Computational Linguistics.
  * Borgeaud et al. (2022) Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, George Bm Van Den Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, et al. 2022.  Improving language models by retrieving from trillions of tokens.  In _International conference on machine learning_ , pages 2206–2240. PMLR.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020.  Language models are few-shot learners.  In _Advances in Neural Information Processing Systems_ , volume 33, pages 1877–1901. Curran Associates, Inc.
  * Cheng et al. (2023) Sijie Cheng, Zhiyong Wu, Jiangjie Chen, Zhixing Li, Yang Liu, and Lingpeng Kong. 2023.  Unsupervised explanation generation via correct instantiations.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 37, pages 12700–12708.
  * Chowdhery et al. (2022) Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. 2022.  Palm: Scaling language modeling with pathways.  _arXiv preprint arXiv:2204.02311_.
  * Conforti et al. (2020) Costanza Conforti, Jakob Berndt, Mohammad Taher Pilehvar, Chryssi Giannitsarou, Flavio Toxvaerd, and Nigel Collier. 2020.  Will-they-won’t-they: A very large dataset for stance detection on Twitter.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 1715–1724, Online. Association for Computational Linguistics.
  * Darwish et al. (2018) Kareem Darwish, Walid Magdy, Afshin Rahimi, Timothy Baldwin, and Norah Abokhodair. 2018.  Predicting online islamophobic behavior after #parisattacks.  _The Journal of Web Science_ , 4(3):34–52.
  * Darwish et al. (2017) Kareem Darwish, Walid Magdy, and Tahar Zanouda. 2017.  Trump vs. hillary: What went viral during the 2016 us presidential election.  In _Social Informatics_ , pages 143–161, Cham. Springer International Publishing.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Du et al. (2023) Yilun Du, Shuang Li, Antonio Torralba, Joshua B Tenenbaum, and Igor Mordatch. 2023.  Improving factuality and reasoning in language models through multiagent debate.  _arXiv preprint arXiv:2305.14325_.
  * Gao et al. (2021) Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021.  SimCSE: Simple contrastive learning of sentence embeddings.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 6894–6910, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Guerreiro et al. (2023) Nuno M. Guerreiro, Elena Voita, and André Martins. 2023.  Looking for a needle in a haystack: A comprehensive study of hallucinations in neural machine translation.  In _Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics_ , pages 1059–1075, Dubrovnik, Croatia. Association for Computational Linguistics.
  * Guo et al. (2023) Zhicheng Guo, Sijie Cheng, Yile Wang, Peng Li, and Yang Liu. 2023.  Prompt-guided retrieval augmentation for non-knowledge-intensive tasks.  _arXiv preprint arXiv:2305.17653_.
  * Hamilton (2023) Sil Hamilton. 2023.  Blind judgement: Agent-based supreme court modelling with gpt.  _arXiv preprint arXiv:2301.05327_.
  * Hasan and Ng (2014) Kazi Saidul Hasan and Vincent Ng. 2014.  Why are you taking this stance? identifying and classifying reasons in ideological debates.  In _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 751–762, Doha, Qatar. Association for Computational Linguistics.
  * He et al. (2022) Zihao He, Negar Mokhberian, and Kristina Lerman. 2022.  Infusing knowledge from Wikipedia to enhance stance detection.  In _Proceedings of the 12th Workshop on Computational Approaches to Subjectivity, Sentiment & Social Media Analysis_, pages 71–77, Dublin, Ireland. Association for Computational Linguistics.
  * Ji et al. (2023) Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. 2023.  Survey of hallucination in natural language generation.  _ACM Computing Surveys_ , 55(12):1–38.
  * Küçük and Can (2020) Dilek Küçük and Fazli Can. 2020.  Stance detection: A survey.  _ACM Computing Surveys (CSUR)_ , 53(1):1–37.
  * Lai et al. (2020) Mirko Lai, Alessandra Teresa Cignarella, Delia Irazú Hernández Farías, Cristina Bosco, Viviana Patti, and Paolo Rosso. 2020.  Multilingual stance detection in social media political debates.  _Computer Speech & Language_, 63:101075.
  * Li et al. (2023) Guohao Li, Hasan Abed Al Kader Hammoud, Hani Itani, Dmitrii Khizbullin, and Bernard Ghanem. 2023.  Camel: Communicative agents for "mind" exploration of large scale language model society.
  * Li et al. (2021) Yingjie Li, Tiberiu Sosea, Aditya Sawant, Ajith Jayaraman Nair, Diana Inkpen, and Cornelia Caragea. 2021.  P-stance: A large dataset for stance detection in political domain.  In _Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021_ , pages 2355–2365, Online. Association for Computational Linguistics.
  * Liang et al. (2022a) Bin Liang, Zixiao Chen, Lin Gui, Yulan He, Min Yang, and Ruifeng Xu. 2022a.  Zero-shot stance detection via contrastive learning.  In _Proceedings of the ACM Web Conference 2022_ , WWW ’22, page 2738–2747, New York, NY, USA. Association for Computing Machinery.
  * Liang et al. (2022b) Bin Liang, Qinglin Zhu, Xiang Li, Min Yang, Lin Gui, Yulan He, and Ruifeng Xu. 2022b.  JointCL: A joint contrastive learning framework for zero-shot stance detection.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 81–91, Dublin, Ireland. Association for Computational Linguistics.
  * Liu et al. (2022) Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. 2022.  What makes good in-context examples for GPT-3? In _Proceedings of Deep Learning Inside Out (DeeLIO 2022): The 3rd Workshop on Knowledge Extraction and Integration for Deep Learning Architectures_ , pages 100–114, Dublin, Ireland and Online. Association for Computational Linguistics.
  * Liu et al. (2021) Rui Liu, Zheng Lin, Yutong Tan, and Weiping Wang. 2021.  Enhancing zero-shot and few-shot stance detection with commonsense knowledge graph.  In _Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021_ , pages 3152–3157, Online. Association for Computational Linguistics.
  * Minsky (1988) Marvin Minsky. 1988.  _Society of mind_.  Simon and Schuster.
  * Mohammad et al. (2016) Saif Mohammad, Svetlana Kiritchenko, Parinaz Sobhani, Xiaodan Zhu, and Colin Cherry. 2016.  SemEval-2016 task 6: Detecting stance in tweets.  In _Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016)_ , pages 31–41, San Diego, California. Association for Computational Linguistics.
  * Nguyen et al. (2020) Dat Quoc Nguyen, Thanh Vu, and Anh Tuan Nguyen. 2020.  BERTweet: A pre-trained language model for English tweets.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pages 9–14, Online. Association for Computational Linguistics.
  * Ouyang et al. (2022) Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022.  Training language models to follow instructions with human feedback.  _Advances in Neural Information Processing Systems_ , 35:27730–27744.
  * Park et al. (2023) Joon Sung Park, Joseph C O’Brien, Carrie J Cai, Meredith Ringel Morris, Percy Liang, and Michael S Bernstein. 2023.  Generative agents: Interactive simulacra of human behavior.  _arXiv preprint arXiv:2304.03442_.
  * Piaget (2013) Jean Piaget. 2013.  _The construction of reality in the child_ , volume 82.  Routledge.
  * Qian et al. (2023) Chen Qian, Xin Cong, Cheng Yang, Weize Chen, Yusheng Su, Juyuan Xu, Zhiyuan Liu, and Maosong Sun. 2023.  Communicative agents for software development.  _arXiv preprint arXiv:2307.07924_.
  * Shum et al. (2023) KaShun Shum, Shizhe Diao, and Tong Zhang. 2023.  Automatic prompt augmentation and selection with chain-of-thought from labeled data.  _arXiv preprint arXiv:2302.12822_.
  * Sobhani et al. (2017) Parinaz Sobhani, Diana Inkpen, and Xiaodan Zhu. 2017.  A dataset for multi-target stance detection.  In _Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers_ , pages 551–557, Valencia, Spain. Association for Computational Linguistics.
  * Talebirad and Nadiri (2023) Yashar Talebirad and Amirhossein Nadiri. 2023.  Multi-agent collaboration: Harnessing the power of intelligent llm agents.  _arXiv preprint arXiv:2306.03314_.
  * Wang et al. (2023a) Guan Wang, Sijie Cheng, Xianyuan Zhan, Xiangang Li, Sen Song, and Yang Liu. 2023a.  Openchat: Advancing open-source language models with mixed-quality data.
  * Wang et al. (2023b) Lei Wang, Chen Ma, Xueyang Feng, Zeyu Zhang, Hao Yang, Jingsen Zhang, Zhiyuan Chen, Jiakai Tang, Xu Chen, Yankai Lin, et al. 2023b.  A survey on large language model based autonomous agents.  _arXiv preprint arXiv:2308.11432_.
  * Wang et al. (2023c) Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V. Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2023c.  Self-consistency improves chain of thought reasoning in language models.  In _ICLR 2023_.
  * Wang et al. (2023d) Yile Wang, Peng Li, Maosong Sun, and Yang Liu. 2023d.  Self-knowledge guided retrieval augmentation for large language models.  In _Findings of the Association for Computational Linguistics: EMNLP 2023_ , pages 10303–10315, Singapore. Association for Computational Linguistics.
  * Wang et al. (2023e) Zhenhailong Wang, Shaoguang Mao, Wenshan Wu, Tao Ge, Furu Wei, and Heng Ji. 2023e.  Unleashing cognitive synergy in large language models: A task-solving agent through multi-persona self-collaboration.  _arXiv preprint arXiv:2307.05300_.
  * Wei et al. (2022) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed Chi, Quoc V Le, and Denny Zhou. 2022.  Chain-of-thought prompting elicits reasoning in large language models.  In _Advances in Neural Information Processing Systems_ , volume 35, pages 24824–24837. Curran Associates, Inc.
  * Xi et al. (2023) Zhiheng Xi, Wenxiang Chen, Xin Guo, Wei He, Yiwen Ding, Boyang Hong, Ming Zhang, Junzhe Wang, Senjie Jin, Enyu Zhou, et al. 2023.  The rise and potential of large language model based agents: A survey.  _arXiv preprint arXiv:2309.07864_.
  * Xu et al. (2023a) Benfeng Xu, An Yang, Junyang Lin, Quan Wang, Chang Zhou, Yongdong Zhang, and Zhendong Mao. 2023a.  ExpertPrompting: Instructing large language models to be distinguished experts.  _arXiv preprint arXiv:2305.14688_.
  * Xu et al. (2023b) Yuzhuang Xu, Shuo Wang, Peng Li, Fuwen Luo, Xiaolong Wang, Weidong Liu, and Yang Liu. 2023b.  Exploring large language models for communication games: An empirical study on werewolf.  _arXiv preprint arXiv:2309.04658_.
  * Zhang et al. (2022) Bowen Zhang, Daijun Ding, and Liwen Jing. 2022.  How would stance detection techniques evolve after the launch of chatgpt?  _arXiv preprint arXiv:2212.14548_.
  * Zhang et al. (2023a) Bowen Zhang, Xianghua Fu, Daijun Ding, Hu Huang, Yangyang Li, and Liwen Jing. 2023a.  Investigating chain-of-thought with chatgpt for stance detection on social media.  _arXiv preprint arXiv:2304.03087_.
  * Zhang et al. (2019) Chen Zhang, Qiuchi Li, and Dawei Song. 2019.  Aspect-based sentiment classification with aspect-specific graph convolutional networks.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 4568–4578, Hong Kong, China. Association for Computational Linguistics.
  * Zhang et al. (2023b) Zhuosheng Zhang, Aston Zhang, Mu Li, and Alex Smola. 2023b.  Automatic chain of thought prompting in large language models.  In _The Eleventh International Conference on Learning Representations (ICLR 2023)_.