# A cornucopia of antineutrons and hyperons from super
$J\kern-1.49994pt/\kern-1.49994pt\psi\kern 1.49994pt$ factory
for next-generation nuclear and particle physics high-precision experiments

Chang-Zheng Yuan<EMAIL_ADDRESS>Institute of High Energy Physics, Chinese
Academy of Sciences, Beijing 100049, China  University of Chinese Academy of
Sciences, Beijing 100049, China  Marek Karliner<EMAIL_ADDRESS>School
of Physics and Astronomy, Tel Aviv University, Tel Aviv 69978, Israel

###### Abstract

In order to study the interactions and structure of various types of matter
one typically needs to carry out scattering experiments utilizing many
different particles as projectiles. Whereas beams of $e^{\pm}$, $\mu^{\pm}$,
$\pi^{\pm}$, $K^{\pm}$, proton, antiproton, and of various heavy ions have
been produced and have enabled many scientific breakthroughs, beams of
antineutrons, hyperons ($\Lambda$, $\Sigma$ and $\Xi$) and their antiparticles
are typically not easy to obtain. Here we point out and investigate a new
high-quality source of these particles: a super
$J\kern-1.49994pt/\kern-1.49994pt\psi\kern 1.49994pt$ factory with capability
of accumulating trillions of $J\kern-1.49994pt/\kern-1.49994pt\psi\kern
1.49994pt$ decays each year. In the relevant
$J\kern-1.49994pt/\kern-1.49994pt\psi\kern 1.49994pt$ decays the desired
particle is produced together with other final state particles that can be
tagged. This allows accurate determination of the flux and momentum of the
projectile, enabling unprecedented precision-study of the corresponding
interactions with a broad range of targets. These novel high-statistics
sources of baryons and antibaryons with precisely known kinematics open fresh
opportunities for applications in particle and nuclear physics, including
antinucleon-nucleon interaction, nonvalence $s\bar{s}$ component of the
nucleon, (anti)hyperon-nucleon interaction, OZI violation, (multi-strange)
hypernuclei, exotic light hadron spectroscopy and many others, as well as
calibration of Monte Carlo simulation for hadronic and medical physics.

antineutron, hyperons, hypernuclei, exotic hadrons,
$J\kern-1.49994pt/\kern-1.49994pt\psi\kern 1.49994pt$, $e^{+}e^{-}$
annihilation

Scattering experiments using many different kinds of beams are the mainstay of
experiments investigating fundamental interactions and structure of matter at
subatomic level. Beams of long-lived charged particles and of photons are easy
to produce and so many experiments using charged projectiles have been carried
out during more than 100 years since the trailblazing experiment shooting
$\alpha$ particles into gold foil enabled Rutherford to infer the existence of
the atomic nucleus rutherford . Since then, $e^{\pm}$, $\mu^{\pm}$,
$\pi^{\pm}$, $K^{\pm}$, proton, antiproton, photon and various heavy ion beams
have been produced and have served as enablers of many scientific
breakthroughs. Beams of some neutral particles, like neutrons and neutrinos
are relatively easy to produce, but difficult to control, i.e. have large
momentum spread. Beams of other neutral particles, such as antineutron and
$K^{0}/\bar{K}^{0}$, and of long-lived hyperons ($\Lambda$, $\Sigma^{+,-}$,
$\Xi^{0,-}$) and their antiparticles ($\bar{\Lambda}$, $\bar{\Sigma}^{-,+}$,
$\bar{\Xi}^{0,+}$) have great physics potential, but typically are much more
difficult to produce and control.

Although potentially extremely useful for investigating nonperturbative QCD
and nuclear structure, experimental studies using antineutron beams have been
very limited till now, due to the severe difficulties in accumulating
sufficient number of antineutrons with known flux and momentum Bressani:2003pv
. The best antineutron sources so far have been obtained at BNL E-767 E767
(with momentum between 100 and 500 MeV/$c$) and CERN OBELIX experiments OBELIX
(with momentum between 50 and 400 MeV/$c$), enabling quite a wide range of
physics topics to be studied, from nuclear physics to hadron spectroscopy,
albeit with limited statistics. Scattering of antineutrons on nuclei made it
possible to investigate nucleon-antinucleon annihilation inside matter,
without complications due to Coulomb interaction Bressani:2003pv .

In the above experiments antineutrons were produced in proton-antiproton
annihilation via charge exchange (CEX) $\bar{p}p\to\bar{n}n$. The
disadvantages of this method are obvious: production rate is low and
antineutron momentum and direction are hard to control. Selection of
antineutrons with momentum in a specific direction results in discarding a
large fraction of antineutrons. Further higher rate experiments have been
proposed, utilizing high-intensity antiproton beams, but within this approach
one does not expect a significant improvement in the antineutron accumulation
rate.

Ground state $SU(3)$ octet hyperons ($\Lambda$, $\Sigma^{+,-}$, $\Xi^{0,-}$)
and their antiparticles have relatively long lifetime (typical $c\tau$ of a
few cm) and are essential for investigating several important physics
questions, including hyperon-nucleon interaction and and the possible role of
hyperons in neutron stars Haidenbauer:2013oca ; Vidana:2018bdi ; Tolos:2020aln
. The relevant experimental studies started in 1960s and have lasted for more
than half a century Engelmann:1966 ; SechiZorn:1969hk ; Alexander:1969cx ;
Eisele:1971mk ; Kadyk:1971tc ; Hauptman:1977hr ; Ahn:1997wa ; Aoki:1998sv ;
Kondo:2000hn ; Ahn:2005jz , using $\pi^{\pm}$ or $K^{\pm}$ beams shot into
bubble chambers or Scintillating Fiber (SciFi) targets. The statistics of
these experiments are low, with typically a few tens to a few hundreds
observed events.

In this article we propose a new source of antineutrons and (anti)hyperons,
based on a completely different technique. Due to its large production cross
section in $e^{+}e^{-}$ and large decay branching fractions to final states
with antineutron or (anti)hyperon, $J\kern-1.49994pt/\kern-1.49994pt\psi\kern
1.49994pt$ particles produced in an $e^{+}e^{-}$ annihilation experiment can
serve as a new source of antineutrons and (anti)hyperons. By tagging the
particles recoiling against an antineutron or (anti)hyperon in
$J\kern-1.49994pt/\kern-1.49994pt\psi\kern 1.49994pt$ decays, the momentum and
direction of these source particles can be determined precisely, and the
interaction with the target material placed outside of a very small radius
$e^{+}e^{-}$ beam pipe allows substantial high-precision measurements related
to particle and nuclear physics.

We show below that antineutrons produced in
$\,J\kern-1.49994pt/\kern-1.49994pt\psi\kern 1.49994pt\to p\pi^{-}\bar{n}\,$
and (anti)hyperons produced in similar
$J\kern-1.49994pt/\kern-1.49994pt\psi\kern 1.49994pt$ decay modes at a tau-
charm factory like BESIII bes3  at BEPCII bepc2 , or a super tau-charm factory
like STCF STCF  or SCTF SCTF  can serve as a perfect source of these particles
for many physics studies, thanks to the huge $e^{+}e^{-}\to
J\kern-1.49994pt/\kern-1.49994pt\psi\kern 1.49994pt$ production rate and the
modern, multipurpose high-performance detectors. We use the existing BESIII
experiment as a case study, with its accumulated 10 billions
$J\kern-1.49994pt/\kern-1.49994pt\psi\kern 1.49994pt$-s dataset, to provide a
proof of concept of using antineutrons from
$J\kern-1.49994pt/\kern-1.49994pt\psi\kern 1.49994pt$ decays. We extend our
discussion to $SU(3)$ octet hyperon and antihyperon sources and to future
higher-luminosity experiments under consideration. We point out that by
including in the detector design an option for inserting a variety of specific
materials as targets for these particles, a wide range of novel high-precision
physics measurements can be carried out. These include antinucleon-nucleon
interaction Klempt:2005pp , OZI violation and nonvalence $s\bar{s}$ component
of the nucleon Ellis:1988jwa ; Ellis:1994ww ; Ellis:1999er , (anti)hyperon-
nucleon interaction Haidenbauer:2013oca ; Tolos:2020aln , (multi-strange)
hypernuclei Gal:2016boi ; Botta:2012xi ; Pochodzalla:2016ncu , light hadron
spectroscopy Rosner:2006jz ; Klempt:2007cp , including exotics and many others
Bressani:2003pv , as well as cross section of antineutrons with material for
calibration of Monte Carlo simulation codes for particle physics and medical
applications, such as fluka FLUKA  and geant4 GEANT .

The $J\kern-1.49994pt/\kern-1.49994pt\psi\kern 1.49994pt$ was discovered in
1974 ting ; richter  and has been studied in many generations of detectors.
Due to its large partial width to $e^{+}e^{-}$, the production cross section
of $J\kern-1.49994pt/\kern-1.49994pt\psi\kern 1.49994pt$ in $e^{+}e^{-}$
annihilation is about 90 $\mu$b. However, the actual production rate suffers
from initial state radiation and the energy spread of the $e^{+}e^{-}$ beams
wym . As a matter of fact, the experimental production cross section of
$J\kern-1.49994pt/\kern-1.49994pt\psi\kern 1.49994pt$ at BEPCII is 3500 nb
with an energy spread of 0.9 MeV for the center-of-mass (CM) energy bes3_mm .
The BEPCII bepc2  is a symmetric $e^{+}e^{-}$ collider operating in CM
energies from 2 to 5 GeV, with design luminosity of $10^{33}~{}{\rm
cm}^{-2}s^{-1}$ at 3.77 GeV, the peak of the $\psi(3770)$ resonance. The
luminosity at 3.097 GeV, the $J\kern-1.49994pt/\kern-1.49994pt\psi\kern
1.49994pt$ resonance, is about $0.47\times 10^{33}~{}{\rm cm}^{-2}s^{-1}$. The
energies of the $e^{+}$ and $e^{-}$ beams are known to 0.1 MeV and they
collide with a crossing angle of 22 mrad in the horizontal plane, so the
$J\kern-1.49994pt/\kern-1.49994pt\psi\kern 1.49994pt$ particle is basically
static with a very small boost which is known precisely. This is equivalent to
a production rate of 1600 Hz for $J\kern-1.49994pt/\kern-1.49994pt\psi\kern
1.49994pt$ events.

Amongst more than 300 decay modes reported,
$J\kern-1.49994pt/\kern-1.49994pt\psi\kern 1.49994pt\to p\pi^{-}\bar{n}\,$ is
the best source of antineutrons: the branching fraction, $(2.12\pm 0.09)\times
10^{-3}$, is large pdg  and there are only two charged tracks originating from
the interaction point. They can be selected and identified with high
efficiency in a detector like BESIII.

The BESIII detector is described in detail in Ref. bes3 . The cylindrical core
of the detector covers 93% of the full solid angle and consists of a helium-
based multilayer drift chamber (MDC) outside of a beryllium beam pipe, a
plastic scintillator time-of-flight system (TOF), and a CsI(Tl)
electromagnetic calorimeter (EMC), which are all enclosed in a superconducting
solenoidal magnet providing a 1.0 T magnetic field. The solenoid is supported
by an octagonal flux-return yoke with resistive plate counter muon
identification modules interleaved with steel. The charged-particle momentum
resolution at $1~{}{\rm GeV}/c$ is $0.5\%$, and the $dE/dx$ resolution is
$6\%$ for electrons from Bhabha scattering. The EMC measures photon energies
with a resolution of $2.5\%$ ($5\%$) at $1$ GeV in the barrel (end cap)
region. The time resolution in the TOF barrel region is 68 ps, while that in
the end cap region is 110 ps. The end cap TOF system was upgraded in 2015
using multi-gap resistive plate chamber technology, providing a time
resolution of 60 ps etof . In four runs in 2009, 2012, 2018 and 2019, BESIII
has accumulated 10 billion high-quality
$J\kern-1.49994pt/\kern-1.49994pt\psi\kern 1.49994pt$ events. These have
yielded a large number of significant published results, based on a part of or
the full data sample bes3 .

With known four-momenta of the initial
$J\kern-1.49994pt/\kern-1.49994pt\psi\kern 1.49994pt$ and those of the
selected proton and $\pi^{-}$, the antineutron can be selected by requiring
the recoiling mass of $p\pi^{-}$ to agree with that of an antineutron. The
momentum and direction of the antineutron can be determined as well, with an
uncertainty of a few MeV and a few milliradians, respectively. This estimate
is based on published results on $J\kern-1.49994pt/\kern-1.49994pt\psi\kern
1.49994pt\to p\pi^{-}\bar{n}$ from BES pnbarpi_bes , whose momentum resolution
is not as good as that of BESIII, and $\psi(2S)\to\gamma\chi_{cJ}\to\gamma
p\pi^{-}\bar{n}$ from BESIII chicj_pnbarpi_bes3 , where one more photon is
involved in the final state. With ${\cal
B}(J\kern-1.49994pt/\kern-1.49994pt\psi\kern 1.49994pt\to
p\pi^{-}\bar{n})\approx 2\times 10^{-3}$, the $\,{\sim}40$% tagging efficiency
indicates that 8 million antineutrons can be tagged in the 10 billion
$J\kern-1.49994pt/\kern-1.49994pt\psi\kern 1.49994pt$ event sample.

The maximum momentum of the antineutron is 1174 MeV/$c$, corresponding to the
case when the proton and $\pi^{-}$ fly in the same direction, opposite to the
antineutron; the minimum momentum of the antineutron is zero, when the proton
and $\pi^{-}$ fly back-to-back with the same absolute value of the momentum.
The momentum spectrum of the antineutron, shown in the right panel of Fig. 1,
is determined by the $J\kern-1.49994pt/\kern-1.49994pt\psi\kern 1.49994pt\to
p\pi^{-}\bar{n}$ decay dynamics. It can be obtained by converting the
$p\pi^{-}$ invariant mass distribution pnbarpi_bes  (shown in the left panel
of Fig. 1) into antineutron momentum distribution.

Figure 1: Invariant mass distribution of $p\pi^{-}$ from
$J\kern-1.49994pt/\kern-1.49994pt\psi\kern 1.49994pt\to p\pi^{-}\bar{n}$
selected from BES experiment (left), and the corresponding antineutron
momentum distribution (fraction of events in 10 MeV/$c$) (right). The plots
have been remade with data in Ref. pnbarpi_bes .

Comparing this sample with that achieved by the OBELIX experiment
Astrua:2002zg , we find that the BESIII sample is already a quarter of that in
OBELIX and the momentum range is wider. The momentum of the antineutron at
BESIII is known with an uncertainty of about 6–7 MeV/$c$ in the full momentum
range whereas that at OBELIX is 3-5% of 50 to 400 MeV/$c$. In particular,
antineutrons with momentum higher than 500 MeV/$c$ (about 80% of the tagged
antineutrons) are unique in the world. These events can be used to study many
open problems involving the physics of antineutrons, as discussed in the
review Bressani:2003pv .

Ground state $SU(3)$ octet hyperons ($\Lambda$, $\Sigma^{+,-}$, $\Xi^{0,-}$)
and their antiparticles can be produced copiously in
$J\kern-1.49994pt/\kern-1.49994pt\psi\kern 1.49994pt$ decays and they can be
tagged similarly to antineutrons. Since their $c\tau$ is on the order of a few
cm, comparable to inner radius of the beam pipe (see below), only some make it
to the beam pipe and/or the inner tube of the MDC, but the surviving fraction
is significant. This makes the study of hyperon and antihyperon interactions
with nuclei feasible. Table 1 lists the main
$J\kern-1.49994pt/\kern-1.49994pt\psi\kern 1.49994pt$ decay modes relevant for
hyperon/antihyperon production, together with their $c\tau$, to be compared
with the geometry of BESIII beam pipe and of MDC inner tube. A rough estimate
of the available tagged particles in the 10 billion
$J\kern-1.49994pt/\kern-1.49994pt\psi\kern 1.49994pt$ event sample can be made
according to the tag efficiency (${\sim}20\%\div 40\%$, depending on
multiplicity of final state particles), the $\beta\gamma$ of the produced
hyperons and their lifetimes. The numbers of particles reaching the beam pipe
(as target material at BESIII) are listed in Table 1.

Table 1: Hyperon and antihyperon production in the 10 billion $J\kern-1.49994pt/\kern-1.49994pt\psi\kern 1.49994pt$ or 3 billion $\psi(2S)$ event data sample at BESIII. The yield of hyperons is the same as that of antihyperons, since particles and antiparticles are produced with the same rates in $J\kern-1.49994pt/\kern-1.49994pt\psi\kern 1.49994pt$ (or $\psi(2S)$) decays via strong or electromagnetic interactions. $p_{\rm max}$ is the maximum momentum of the antihyperon, $n^{Y}_{\rm BP}$ is the number of tagged antihyperons reaching the beam pipe; “—” means not available. Antihyperon |  $c\tau$ (cm) | decay mode | Branching Fraction ($\times 10^{-3}$) |  $p_{\rm max}$ (MeV/$c$) |  $n^{Y}_{\rm BP}$ ($\times 10^{5}$)
---|---|---|---|---|---
$\bar{\Lambda}$ | 7.89 | $J\kern-1.49994pt/\kern-1.49994pt\psi\kern 1.49994pt\to\Lambda\bar{\Lambda}$ | 1.89 | 1074 | 26
|  | $J\kern-1.49994pt/\kern-1.49994pt\psi\kern 1.49994pt\to pK^{-}\bar{\Lambda}$ | 0.87 | 876 | 9
$\bar{\Sigma}^{-}$ | 2.40 | $J\kern-1.49994pt/\kern-1.49994pt\psi\kern 1.49994pt\to\Sigma^{+}\bar{\Sigma}^{-}$ | 1.50 | 992 | 4
|  | $J\kern-1.49994pt/\kern-1.49994pt\psi\kern 1.49994pt\to\Lambda\pi^{+}\bar{\Sigma}^{-}$ | 0.83 | 950 | 1
$\bar{\Sigma}^{+}$ | 4.43 | $J\kern-1.49994pt/\kern-1.49994pt\psi\kern 1.49994pt\to\Lambda\pi^{-}\bar{\Sigma}^{+}$ | — | 945 | —
$\bar{\Xi}^{0}$ | 8.71 | $J\kern-1.49994pt/\kern-1.49994pt\psi\kern 1.49994pt\to\Xi^{0}\bar{\Xi}^{0}$ | 1.17 | 818 | 7
|  | $J\kern-1.49994pt/\kern-1.49994pt\psi\kern 1.49994pt\to\Xi^{-}\pi^{+}\bar{\Xi}^{0}$ | — | 685 | —
$\bar{\Xi}^{+}$ | 4.91 | $J\kern-1.49994pt/\kern-1.49994pt\psi\kern 1.49994pt\to\Xi^{-}\bar{\Xi}^{+}$ | 0.97 | 807 | 3
|  | $J\kern-1.49994pt/\kern-1.49994pt\psi\kern 1.49994pt\to\Xi^{0}\pi^{-}\bar{\Xi}^{+}$ | — | 686 | —
$\bar{\Omega}^{+}$ | 2.46 | $\psi(2S)\to\Omega^{-}\bar{\Omega}^{+}$ | 0.05 | 774 | 0.05
|  | $\psi(2S)\to K^{-}\Xi^{0}\bar{\Omega}^{+}$ | — | 606 | —

The $\Omega^{-}$ and its antiparticle $\bar{\Omega}^{+}$ can not be produced
in $J\kern-1.49994pt/\kern-1.49994pt\psi\kern 1.49994pt$ decays, due to their
high mass, but they are accessible through $\psi(2S)$ decays. As ${\cal
B}[\psi(2S)\to\Omega^{-}\bar{\Omega}^{+}]=5.2\times 10^{-5}$, they can be
studied with the BESIII $\psi(2S)$ data sample bes3 . There could also be
other $\psi(2S)$ decay modes with $\Omega^{-}$ or $\bar{\Omega}^{+}$ in the
final state, such as $\psi(2S)\to K^{-}\Xi^{0}\bar{\Omega}^{+}+c.c.$ These,
too, are listed in Table 1, together with the number of $\Omega$s reaching the
beam pipe in the 3 billion $\psi(2S)$ event data sample bes3 . Of course the
$\psi(2S)$ data sample will be able to supply more decay modes for all these
hyperons and antihyperons, including the decays from the secondary charmonium
states [$\chi_{cJ}$ ($J=0$, $1$, $2$),
$J\kern-1.49994pt/\kern-1.49994pt\psi\kern 1.49994pt$] produced in $\psi(2S)$
radiative and hadronic transitions note . These new decay modes will broaden
the momentum range of the source particles.

The BESIII $J\kern-1.49994pt/\kern-1.49994pt\psi\kern 1.49994pt$ data sample
has been collected already, so we do not have an opportunity to gather data
with a custom-made target. However, the detector material close to the
interaction point in the inner detector serves as an effective target,
allowing us to carry out a substantial study with the collected antineutrons.

The BESIII bes3  beam pipe is 1000 mm long with an inner diameter of 63 mm and
an outer diameter of 114 mm. The central part of the beam pipe is 296 mm long
with two layers of beryllium. The inner wall that maintains the ultra high
vacuum in the beam pipe is 0.8 mm thick and the outer beryllium wall is 0.6 mm
thick. A 0.8 mm channel between the two walls is used for circulating the
cooling fluid (high-purity mineral oil). Outside of the beam pipe is the MDC.
The MDC consists of an outer chamber and an inner chamber, which are joined
together at the end plates, sharing a common gas volume. The inner tube of the
MDC is made of 1.2 mm thick carbon fiber with a radius of 59.2 mm that also
provides some mechanical strength. The inner chamber of the MDC, together with
the inner tube can be replaced in case it is damaged by radiation. This design
even allows us put some target material into the current BESIII detector if we
want to do antineutron physics in the near future, as will be discussed below.

The materials in the beam pipe and the inner tube of the MDC can be treated as
targets of beryllium and carbon. Since all $\bar{n}$ interactions with these
materials occur in the central part of the detector, all the final state
particles can be detected by the full detector, just like those produced in
the primary $J\kern-1.49994pt/\kern-1.49994pt\psi\kern 1.49994pt$ decays, in
terms of tracking, particle identification, interaction vertex determination,
photon reconstruction and so on. A schematic diagram of $e^{+}e^{-}\to
J\kern-1.49994pt/\kern-1.49994pt\psi\kern 1.49994pt\to p\pi^{-}\bar{n}$,
followed by $\bar{n}$ interaction with a proton in the beam pipe material is
shown in Fig. 2. The charged pions and the photons from neutral pion decays
are shown for a possible process $\bar{n}p\to\pi^{+}\pi^{+}\pi^{-}\pi^{0}$,
$\pi^{0}\to\gamma\gamma$.

Figure 2: Schematic diagram of $e^{+}e^{-}\to
J\kern-1.49994pt/\kern-1.49994pt\psi\kern 1.49994pt\to p\pi^{-}\bar{n}$,
followed by $\bar{n}$ interaction with a proton in the beam pipe material,
$\bar{n}p\to\pi^{+}\pi^{+}\pi^{-}\pi^{0}$, $\pi^{0}\to\gamma\gamma$.

The good momentum resolution of the BESIII detector makes it possible to
pinpoint the interaction vertex with a resolution of about 1 mm vertex , so as
to determine whether an antineutron interacted with beryllium or carbon, but
the resolution is not sufficient to separate interaction with inner or outer
wall of the beam pipe, nor with the mineral oil between the two layers of
beryllium. The background from direct
$J\kern-1.49994pt/\kern-1.49994pt\psi\kern 1.49994pt$ decays is low, since all
the final state particles are reconstructed and the total energy of the event
is larger than the background $e^{+}e^{-}$ CM energy. The background from
particle interaction with target material, but with final states different
from the one under study can be suppressed thanks to the good particle
identification and good momentum/energy resolution of the detector.

One can use the materials’ budget to estimate the fraction of antineutrons
interacting with specific components of the inner detector. Assuming an
average of 100 mb cross section per $\bar{n}p$ or $\bar{n}n$ collision, we
expect 1–2% of the tagged antineutrons to interact with beryllium and another
1–2% with carbon fiber targets. These are equivalent to about 100,000 events
each for $\bar{n}+{\rm Be}$ and $\bar{n}+{\rm C}$ interactions, respectively.

Accelerators at tau-charm energy region with luminosity 100 times higher than
current one are being proposed, focusing on a variety of physics associated
with strong and electroweak interactions. Both STCF STCF  and SCTF SCTF
operate in a way similar to BEPCII. The detectors will be even better than
BESIII in term of tracking, particle identification, and photon detection.
Experiments at these new facilities will be able to accumulate one trillion
($10^{12}$) $J\kern-1.49994pt/\kern-1.49994pt\psi\kern 1.49994pt$ events in
one year’s running time, i.e. 100 times more than the total dataset
accumulated by BESIII.

If one were to just simply rescale the numbers in previous sections
accordingly, one would immediately see that such a machine provides a superb
opportunity for physics with baryon and antibaryon sources discussed above. In
fact, the case is even more striking, since several additional features in the
accelerator and the detector design will result in significant improvements in
the quality of both the sources and the targets.

In the current BEPCII accelerator the energy spread at the
$J\kern-1.49994pt/\kern-1.49994pt\psi\kern 1.49994pt$ peak is 0.9 MeV. Since
the natural width of $J\kern-1.49994pt/\kern-1.49994pt\psi\kern 1.49994pt$ is
only about 90 keV, the peak cross section for
$J\kern-1.49994pt/\kern-1.49994pt\psi\kern 1.49994pt$ production will be
significantly increased once the beam energy spread is further reduced. It is
expected that use of the monochromator scheme will significantly reduce the
energy spread and increase the peak cross section, by a factor between a few
times and orders of magnitude mono1 ; mono2 ; mono3 .

The range of physics topics investigated with these baryons and antibaryons
and be expanded through use of custom-made targets can be designed and placed
inside the detector, just outside of the beam pipe. The radius of the beam
pipe can be made smaller than that at BEPCII, allowing the targets be located
closer to the interaction point, enlarging acceptance of the target material
and reducing the fraction of hyperon decays. In principle one can design a
barrel with several different types of materials as targets, allowing
simultaneous study of different interactions. This will be extremely
efficient, since all studies share the same environment and systematic effects
can be investigated once for all. Of course, removable targets will be helpful
for even more studies. This does not involve technical difficulties, as the
current BESIII bes3  design has a replaceable inner drift chamber, and the
inner tube of the chamber has been used as a target in the data taking.

In order to produce baryon and antibaryon particles with higher momenta, one
can use asymmetric $e^{+}e^{-}$ beams which will produce
$J\kern-1.49994pt/\kern-1.49994pt\psi\kern 1.49994pt$ in motion.
Alternatively, one can utilize similar decay modes of $\psi(2S)$, or both. One
may also broaden the momentum spectra of baryons produced in
$J\kern-1.49994pt/\kern-1.49994pt\psi\kern 1.49994pt$ decays, by utilizing
$J\kern-1.49994pt/\kern-1.49994pt\psi\kern 1.49994pt$ boosted through large
crossing angle collision mono3 .

Detector design can be optimized to enhance the detection of the final states
specific to the reactions discussed in the present work. The current detector
design is optimized for measurements of light particles such as $e$, $\mu$,
$\pi$, $K$, $p$, and photon. A special-purpose subdetector can be devised for
identification of particles, like deuteron, triton, and even heavier nuclei.

In summary, we demonstrate that $J\kern-1.49994pt/\kern-1.49994pt\psi\kern
1.49994pt$ produced in high-luminosity $e^{+}e^{-}$ annihilation can provide
large numbers of baryon and antibaryon particles as sources for novel nuclear
and particle physics studies. These sources include all the long-lived baryons
and antibaryons, especially antineutron, hyperons $\Lambda$, $\Sigma^{+,-}$,
$\Xi^{0,-}$, and their antiparticles, as well as smaller, but still large
fluxes of $\Omega^{-}$ and $\bar{\Omega}^{+}$. By placing specific custom-made
targets in a BESIII-like detector, one can perform a rich variety of
experiments in nuclear and particle physics. Besides those discussed in the
paper or listed in Table 1, other decay modes with antineutron or hyperons can
also be used as sources of these particles. Although the tagging efficiency
may be low in multi-particle final states, more low momentum source particles
can be produced in these modes.

With the existing 10 billion $J\kern-1.49994pt/\kern-1.49994pt\psi\kern
1.49994pt$ event data sample accumulated at BESIII, many studies of the
interactions of these long-lived baryons with the material in the beam pipe
and the inner tube of the MDC detector can already be performed. With minor
modifications of the accelerator and detector designed for STCF STCF  and SCTF
SCTF , the resulting $J\kern-1.49994pt/\kern-1.49994pt\psi\kern 1.49994pt$
sample can serve as a superb source of all kinds of long-lived baryons and
antibaryons, opening a new era for novel high-precision nuclear and particle
physics studies.

Traditional setups need to produce many different kinds of beams for different
dedicated experiments, and need to share accelerator time among them. This
requires large resources in terms of manpower and funding, impeding such
experiments. In contrast, the approach proposed here will allow experiments
with different beams at the same time, requiring no additional infrastructure
and minimal further investments.

This work is supported in part by National Key Research and Development
Program of China under Contract No. 2020YFA0406300, National Natural Science
Foundation of China (NSFC) under contract Nos. 11961141012, 11835012, and
11521505; and the CAS Center for Excellence in Particle Physics (CCEPP), as
well as NSFC-ISF grant No. 3423/19. C.Z.Y. thanks Sicheng Yuan for stimulating
conversations about antineutron-material interactions, and M.K. thanks Avraham
Gal, Simon Eydelman and Jon Rosner for helpful discussions.

## References

  * (1) E. Rutherford, Phil. Mag. Ser. 6 21, 669-688 (1911).
  * (2) For a review, see T. Bressani and A. Filippi, Phys. Rept. 383, 213-297 (2003).
  * (3) T. Armstrong et al. [BROOKHAVEN-HOUSTON-PENNSYLVANIA STATE-RICE], Phys. Rev. D 36, 659-673 (1987).
  * (4) M. Agnello, et al., Nucl. Instrum. Meth. A 399, 11-26 (1997).
  * (5) J. Haidenbauer, S. Petschauer, N. Kaiser, U. G. Meissner, A. Nogga and W. Weise, Nucl. Phys. A 915, 24-58 (2013).
  * (6) I. Vidaña, Proc. Roy. Soc. Lond. A 474, 0145 (2018).
  * (7) L. Tolos and L. Fabbietti, Prog. Part. Nucl. Phys. 112, 103770 (2020).
  * (8) R. Engelmann, H. Filthuth, V. Hepp, E. Kluge, Phys. Lett. 21, 587-589 (1966).
  * (9) B. Sechi-Zorn, B. Kehoe, J. Twitty and R. A. Burnstein, Phys. Rev. 175, 1735-1740 (1968).
  * (10) G. Alexander, U. Karshon, A. Shapira, G. Yekutieli, R. Engelmann, H. Filthuth and W. Lughofer, Phys. Rev. 173, 1452-1460 (1968).
  * (11) F. Eisele, H. Filthuth, W. Foehlisch, V. Hepp and G. Zech, Phys. Lett. B 37, 204-206 (1971).
  * (12) J. A. Kadyk, G. Alexander, J. H. Chan, P. Gaposchkin and G. H. Trilling, Nucl. Phys. B 27, 13-22 (1971).
  * (13) J. M. Hauptman, J. A. Kadyk and G. H. Trilling, Nucl. Phys. B 125, 29-51 (1977).
  * (14) J. K. Ahn et al. [KEK-PS E-251], Nucl. Phys. A 648, 263-279 (1999).
  * (15) S. Aoki, S. Asai, S. Y. Bahk, S. H. Chung, H. Funahashi, C. H. Hahn, M. Hanabata, T. Hara, K. Hoshino and M. Ieiri, et al. Nucl. Phys. A 644, 365-385 (1998).
  * (16) Y. Kondo et al. [KEK-PS-E289], Nucl. Phys. A 676, 371-387 (2000).
  * (17) J. K. Ahn, S. Aoki, K. S. Chung, M. S. Chung, H. En’yo, T. Fukuda, H. Funahashi, Y. Goto, A. Higashi and M. Ieiri, et al. Phys. Lett. B 633, 214-218 (2006).
  * (18) M. Ablikim et al. [BESIII], Nucl. Instrum. Methods Phys. Res. Sect. A 614, 345 (2010); D. M. Asner et al., Int. J. Mod. Phys. A 24, S1 (2009); M. Ablikim et al. [BESIII], Chin. Phys. C 44, no.4, 040001 (2020).
  * (19) Q. Qin, L. Ma, J. Wang and C. Zhang, Conf. Proc. C 100523, WEXMH01 (2010) IPAC-2010-WEXMH01.
  * (20) Z. G. Zhao, talk at the “International Workshop on Physics at Future High Intensity Collider 2-7 GeV in China”, January 13-16, 2015, University of Chinese Academy of Sciences (UCAS), Hefei, China.
  * (21) E. Levichev, talk at “The 9th Intnl. Workshop on Charm Physics”, May 21–25, 2018, Novosibirsk, Russia, https://indico.inp.nsk.su/event/10/session/1/contribution/65/material/slides/0.pdf
  * (22) E. Klempt, C. Batty and J. M. Richard, Phys. Rept. 413, 197-317 (2005).
  * (23) J. R. Ellis, E. Gabathuler and M. Karliner, Phys. Lett. B 217, 173-180 (1989).
  * (24) J. R. Ellis, M. Karliner, D. E. Kharzeev and M. G. Sapozhnikov, Phys. Lett. B 353, 319-328 (1995).
  * (25) J. R. Ellis, M. Karliner, D. E. Kharzeev and M. G. Sapozhnikov, Nucl. Phys. A 673, 256-278 (2000).
  * (26) A. Gal, E. V. Hungerford and D. J. Millener, Rev. Mod. Phys. 88, no.3, 035004 (2016).
  * (27) E. Botta, T. Bressani and G. Garbarino, Eur. Phys. J. A 48, 41 (2012).
  * (28) J. Pochodzalla et al. [$\overline{\hbox{P}}$ANDA Collaboration], JPS Conf. Proc. 17, 091002 (2017) [arXiv:1609.01916 [nucl-ex]].
  * (29) J. L. Rosner, J. Phys. G 34, S127-S148 (2007).
  * (30) E. Klempt and A. Zaitsev, Phys. Rept. 454, 1-202 (2007).
  * (31) G. Battistoni et al., Annals Nucl. Energy 82, 10-18 (2015); T. T. Böhlen et al., Nucl. Data Sheets 120, 211-214 (2014).
  * (32) S. Agostinelli et al. [GEANT4], Nucl. Instrum. Meth. A 506, 250-303 (2003); J. Allison et al., Nucl. Instrum. Meth. A 835, 186-225 (2016).
  * (33) J. J. Aubert et al. [E598 Collaboration], Phys. Rev. Lett. 33, 1404 (1974).
  * (34) J. E. Augustin et al. [SLAC-SP-017 Collaboration], Phys. Rev. Lett. 33, 1406 (1974).
  * (35) P. Wang, C. Z. Yuan, X. H. Mo and D. H. Zhang, Phys. Lett. B 593, 89-94 (2004).
  * (36) M. Ablikim et al. [BESIII], Phys. Lett. B 791, 375-384 (2019).
  * (37) P. A. Zyla et al. (Particle Data Group), Prog. Theor. Exp. Phys. 2020, 083C01 (2020).
  * (38) X. Li et al., Radiat. Detect. Technol. Methods 1, 13 (2017); Y. X. Guo et al., Radiat. Detect. Technol. Methods 1, 15 (2017); P. Cao et al., Nucl. Instrum. Meth. A 953, 163053 (2020).
  * (39) M. Ablikim et al. [BES], Phys. Rev. Lett. 97, 062001 (2006).
  * (40) M. Ablikim et al. [BESIII], Phys. Rev. D 86, 052011 (2012).
  * (41) M. Astrua, E. Botta, T. Bressani, D. Calvo, C. Casalegno, A. Feliciello, A. Filippi, S. Marcello, M. Agnello and F. Iazzi, Nucl. Phys. A 697, 209-224 (2002).
  * (42) M. Xu, K. L. He, Z. P. Zhang, Y. F. Wang, J. M. Bian, G. F. Cao, X. X. Cao, S. J. Chen, Z. Y. Deng and C. D. Fu, et al. Chin. Phys. C 33, 428-435 (2009).
  * (43) It should be noted that particles other than those which are the focus of this paper are also produced copiously in $J\kern-1.49994pt/\kern-1.49994pt\psi\kern 1.49994pt$ and $\psi(2S)$ decays. With similar methods, $\pi^{\pm}$, $K^{\pm}$, $K^{0}/\bar{K}^{0}$, $p/\bar{p}$, neutron, photon, and other particles can be also be tagged (for $K_{L}$ cf. Amaryan:2020xhw ). Although the corresponding samples may not be large compared with those from dedicated experiments, they are produced for free, their momentum is known with high precision and they can be used for physics studies with the same targets and detector. In a super $J\kern-1.49994pt/\kern-1.49994pt\psi\kern 1.49994pt$ factory the corresponding yields will be higher by two orders of magnitude or more, so in certain cases the samples may become competitive with existing data from dedicated experiments.
  * (44) M. Amaryan et al. [KLF], [arXiv:2008.08215 [nucl-ex]].
  * (45) A. A. Avdienko, G. A. Kornyukhin, I. Y. Protopopov, A. N. Skrinsky, A. B. Temnykh, G. M. Tumaikin and A. A. Zholents, Conf. Proc. C 830811, 186-189 (1983).
  * (46) K. Wille and A. W. Chao, SLAC/AP-032.
  * (47) V. I. Telnov, [arXiv:2008.13668 [physics.acc-ph]].


# Benchmark Self-Evolving:
A Multi-Agent Framework for Dynamic LLM Evaluation

Siyuan Wang1111Equal contribution., Zhuohan Long2111Equal contribution.,
Zhihao Fan3, Zhongyu Wei1, Xuanjing Huang1
1Fudan University, 2Tongji University, 3Alibaba Inc.
<EMAIL_ADDRESS><EMAIL_ADDRESS>

###### Abstract

This paper presents a benchmark self-evolving framework to dynamically
evaluate rapidly advancing Large Language Models (LLMs), aiming for a more
accurate assessment of their capabilities and limitations. We utilize a multi-
agent system to manipulate the context or question of original instances,
reframing new evolving instances with high confidence that dynamically extend
existing benchmarks. Towards a more scalable, robust and fine-grained
evaluation, we implement six reframing operations to construct evolving
instances testing LLMs against diverse queries, data noise and probing their
problem-solving sub-abilities. With this framework, we extend benchmark
datasets of four tasks. Experimental results show a general performance
decline in most LLMs against their original results. This decline under our
scalable and robust evaluations, alongside our fine-grained evaluation, more
accurately reflect models’ capabilities. Besides, our framework widens
performance discrepancies both between different models and within the same
model across various tasks, facilitating more informed model selection for
specific tasks 111Code and data are available at
https://github.com/NanshineLoong/Self-Evolving-Benchmark.

Benchmark Self-Evolving:
A Multi-Agent Framework for Dynamic LLM Evaluation

Siyuan Wang1111Equal contribution., Zhuohan Long2111Equal contribution.,
Zhihao Fan3, Zhongyu Wei1, Xuanjing Huang1 1Fudan University, 2Tongji
University, 3Alibaba Inc<EMAIL_ADDRESS><EMAIL_ADDRESS>

##  1 Introduction

Recent advancements in Large Language Models (LLMs) Touvron et al. (2023);
Chiang et al. (2023); OpenAI (2023); Jiang et al. (2023) have demonstrated
remarkable performance across various tasks, ranging from text generation to
complex problem-solving. The evaluation of LLMs thus has emerged as a crucial
area of research Chang et al. (2023); Espejel et al. (2023). It can provide a
comprehensive understanding of the capabilities and limitations in these
models, and guide the selection of the most applicable LLM for specific
applications. Besides, a systematic assessment of LLMs would inspire further
potential improvement.

A multitude of benchmark datasets Hendrycks et al. (2020); Liang et al.
(2022); bench authors (2023) have been proposed to evaluate LLMs. However,
with the rapid development and emerging abilities of ever-evolving LLMs, these
static datasets are increasingly inadequate for a thorough assessment.
Besides, the extensive use of data for improving LLMs leads to data
contamination issues Zhou et al. (2023); Shi et al. (2023), where in-domain
training or even public test data may be inadvertently included during LLM
training, resulting in skewed evaluations.

Figure 1: The evolution of LLMs necessitates benchmark self-evolving.

These challenges necessitate continual updates of static benchmark datasets,
enabling a more dynamic and accurate evaluation of LLMs. Since annotating new
benchmarks from scratch is costly Kiela et al. (2021), Wei et al. (2023)
evaluate LLMs using perplexity on re-sampled data. However, this over-reliance
on perplexity may not fully reflect LLMs’ performance beyond predictive
accuracy. Zhu et al. (2023) dynamically synthetize test samples based on
directed acyclic graphs, but this method struggles in generalizing to tasks
that cannot be graph-represented. In this work, we propose to flexibly update
existing benchmark datasets instead of constructing entirely new ones.

We introduce a benchmark self-evolving framework, which reframes existing
benchmark instances into new variants for dynamic evaluation, by modifying
their contexts or questions, and corresponding answers. This framework propels
existing benchmarks towards self-evolution in three directions, providing a
systematical dynamic evaluation of LLMs. First, to examine LLMs’ ability to
generalize across diverse and increasingly challenging queries, we introduce
scalable evaluation by creating alternative or more complex questions based on
original contexts. Second, to counteract LLMs’ tendency to exploit shortcut
biases Gallegos et al. (2023); Yang et al. (2023) and their sensitivity to
data noise Dong et al. (2023); Pezeshkpour and Hruschka (2023), our framework
implements robust evaluation. This involves incorporating various
perturbations to the contexts of original instances, including paraphrasing,
adding noise, and reversing polarity. Finally, to mitigate the impact that
outdated data and bias susceptibility could skew capability assessments, we
design fine-grained evaluation to probe LLMs’ sub-abilities for solving
different problems.

We design a multi-agent system to dynamically generate evolving instances from
existing benchmarks while ensuring their accuracy. It comprises four key
components: an instance pre-filter, an instance creator, an instance verifier
and a candidate option formulator. The workflow begins with the pre-filter to
select manageable instances from the original evaluation set. The instance
creator crafts new instances by editing their contexts or questions with
answers, which the verifier checks for correctness. To further enhance
reliability, the candidate option formulator subsequently creates an incorrect
answer option for each new context-question pair, which the verifier need to
identify as inconsistent with the new context-question. These rigorously
generated and double-verified instances will be used for dynamic evaluation.
All these components are powered by GPT-4 to leverage its generative and
verification strengths.

Based on our framework, we dynamically extend benchmark datasets of four
different tasks, including mathematical reasoning (GSM8K), logical reasoning
(CLUTRR), commonsense reasoning (StrategyQA) and reading comprehension
(BoolQ), and re-evaluate various closed-source and open-source LLMs. Results
show that our scalable and robust evaluation are more challenging compared to
original benchmarks, leading to a general performance decline for all models.
It helps reveal the limited generalizability and robustness of models to
diverse and complex queries. This along with sub-ability probing offers a more
accurate reflection of LLMs’ true capabilities. Besides, our framework expands
the performance gap between various models and also the differences of a
single model across various tasks, which benefits the selection of the most
suitable LLM for specific applications.

##  2 Benchmark Self-Evolving Framework

Original Instance
---
Context: Janet’s ducks lay 16 eggs per day. She eats three for breakfast every
morning and bakes muffins for her friends every day with four. She sells the
remainder at the farmers’ market daily for $2 per fresh duck egg.
Original Question: How much in dollars does she make every day at the farmers’
market?
Original Answer: Janet sells 16 - 3 - 4 = <<16-3-4=9>>9 duck eggs a day. She
makes 9 * 2 = $<<9*2=18>>18 every day at the farmer’s market. #### 18

Directions |  Operation Types |  Changed Items |  Examples
---|---|---|---
Scalable |  Question Alternating |  context, answer |  Alternative Question: If Janet decides to use 2 of her daily eggs to make a special omelette for dinner each day, how much will she earn at the farmers’ market in a week?
Alternative Answer: $98
Question Complicating |  context, answer |  Complex Question: How many days will it take for Janet to save $100 from her earnings at the farmers’ market?
Complex Answer: 6 days
Robust |  Context Paraphrasing |  context |  Paraphrased Context: Janet’s daily egg production from her ducks is 16. Each morning, she consumes three eggs for breakfast and uses four more to bake muffins for her friends. The remaining eggs are then sold at the farmers’ market for $2 each.
Context Noising |  context |  Noised Context: Janet’s ducks lay 16 eggs per day and _her cows product 4L milk_ per day. She eats three eggs and 1L milk for breakfast every morning and bakes muffins for her friends every day with four eggs. She _keeps the remainder milk for herself_ and only sells the remainder eggs at the farmers’ market daily for $2 per fresh duck egg.
Polarity Reversing |  context, answer |  Reversed Context: Janet’s ducks lay _20 eggs_ per day. She eats _five_ for breakfast every morning and bakes muffins for her friends every day with four. She sells the remainder at the farmers’ market daily for _$2.5_ per fresh duck egg.
Reversed Answer: 27.5
Fine-grained |  Sub-ability Question Generation |  context, answer |  New Question: What are the detailed reasoning steps required to calculate how much in dollars Janet makes every day at the farmers’ market?
New Answer: The solution involves 2 reasoning steps. [Step 1] calculates the
number of eggs can be sold. [Step 2] calculate the money she earns.

Table 1: The reframing operations and examples for generating evolving
instances.

The overall illustration of our framework is presented in Figure 1. We first
introduce different directions that we modify the contexts or questions of
original instances along with their answers for newly evolving instances (see
Section 2.1). We employ a multi-agent system to facilitate collaboration on
evolving instance generation and double-verification. (see Section 2.2)

Figure 2: The workflow of our Multi-Agent Evolving Instance Generator system.

###  2.1 Evolving Instance Taxonomy

An instance can be formulated as a triplet consisting of a context ($C$), a
question ($Q$) and an answer ($A$). For tasks involving only a question and an
answer, the context is designated as null. Given an original evaluation
instance $(C_{o},Q_{o},A_{o})$, we either perturb the context $C_{o}$ or alter
the questions $Q_{o}$, simultaneously forming the corresponding answer. We
thereby reframe an evolving instance as $(C_{e},Q_{o},A_{e})$ or
$(C_{o},Q_{e},A_{e})$, for scalable, robust and fine-grained evaluation.

Scalable Evaluation For scalable evaluation of evolving LLMs, we create
various questions with their corresponding new answers based on the original
instance to examine whether LLMs can generalize to diverse and increasingly
challenging queries. Our approach includes the creation of alternative
questions (Question Alternating) that examine different facets of the original
context, as well as more complex questions requiring additional reasoning
steps (Question Complicating). To maintain the accuracy of evolving instance,
we conduct question generation without changing original contexts.

Robust Evaluation For more robust evaluation of LLMs, we introduce various
perturbations to the contexts of original instances to generate evolving
instances. Specifically, we apply three perturbation strategies on the
contexts as follows. (1) Context Paraphrasing: paraphrasing the original
context to obtain diverse formulations; (2) Context Noising: adding noise by
introducing irrelevant or adversarial sentences into the original context; (3)
Polarity Reversing: reversing the polarity or altering key details of the
original context. The first two perturbations require maintaining the original
answer labels while the third approach necessitates a corresponding answer
change, offering a more rigorous test of the model’s adaptability.

Fine-grained Evaluation We design fine-grained evaluation by generating sub-
ability questions to probe LLMs’ problem-solving capabilities. We focus on
three explainability-related sub-abilities: (1) task planning capability that
inquires about the details of planned reasoning steps, (2) implicit knowledge
identification capability for recognizing underlying facts or rules, and (3)
relevant context retrieval capability for extracting pertinent information
from the given context to support its responses.

The detailed operations to reframe evolving instances and corresponding
examples are in Table 1.

###  2.2 Multi-Agent Evolving Instance Generator

To generate evolving instances and ensure their correctness, we design a
multi-agent instance creator system, incorporating four key agents including
an instance pre-filter, an instance creator, an instance verifier and a
candidate option formulator. All agents are built upon GPT-4 to fulfill their
roles. The system’s workflow is presented in Figure 2.

Instance Pre-Filter The instance pre-filter is designed to go through the
original dataset to identify manageable instances that fall within GPT-4’s
capability to accurately answer. As the whole system is powered by GPT-4, this
instance pre-filter can establish a correct foundation for subsequent
operations and enhance the overall system’s reliability. It takes the context
and question of the original instance as inputs, prompting GPT-4 to predict
the answer and compare its prediction with the reference answer. A two-shot
chain of thought Wei et al. (2022) prompting setting is utilized to select
manageable cases as $(C_{o},Q_{o},A_{o})$.

Instance Creator The instance creator agent is pivotal in generating different
types of newly evolving instances $(C_{e},Q_{e},A_{e})$. Given an original
instance including a context, a question, an answer, and its task description
like “mathematical reasoning task”, the instance creator is prompted to either
modify the contexts ($Q_{e}=Q_{o}$) or form new questions ($C_{e}=C_{o}$). We
design different prompts for six pre-defined reframing operations as Table 1.
For operations without altering the answer ($A_{e}=A_{o}$), the instance
creator is specifically prompted to maintain the original answer during the
operation. For operations leading to new answers, the instance creator think
step-by-step to infer its new answer after the reformulation of the context or
question. This process adopts a one-shot prompting strategy for better
understanding operation requirements. An example prompt for generating complex
question is provided below. Prompts for other reframing operations are listed
in Appendix A.2.

Prompt on Question Complicating You are an expert Question Creator. You will
receive an instance of {task description}, including a context, an original
question and its answer. Your task is to generate a more complex question and
its corresponding answer based on the given context, with the goal of
incorporating additional reasoning steps beyond what is required by the
original question and answer. Please do not change the context but just edit
the question and the answer. Please first generate the question. Then think
step-by-step in one line to give an brief analysis of the question, Finally,
directly present a short answer omitting the intermediate steps, in a single
line. Context: {context $C_{o}$} Original Question: {question $Q_{o}$}
Original Answer: {answer $A_{o}$} Alternative Question:

Instance Verifier The primary function of the instance verifier agent is to
validate the correctness of the newly evolving instance $(C_{e},Q_{e},A_{e})$,
i.e., whether its answer can correctly support the corresponding context and
question. Since these evolving instances are auto-generated through a GPT-4
based agent, the inclusion of this verifier is essential to control our data
quality. The instance verifier directly takes the context, question and answer
of the new instance as inputs, and adopts a two-shot CoT prompting strategy.
It utilizes both a correct and an incorrect demonstrations to avoid potential
biases. The specific prompt used for instance verifier can be found in
Appendix A.2.

Candidate Option Formulator The candidate option formulator aims to generate
an incorrect answer option $O_{w}$ for each new context-question pair
$(C_{e},Q_{e})$. It has two primary purposes: (1) mitigating the impact of
GPT-4’s shortcut biases on data reliability. It enables the instance verifier
to conduct a double-check which simultaneously confirms the validity of the
previous-generated instance $(C_{e},Q_{e},A_{e})$ and ascertains that the
candidate option $O_{w}$ is unable to answer the context-question pair
$(C_{e},Q_{e})$. (2) It additionally provides a standardized binary-choice
assessment method for more accurate evaluation metrics. For new instances with
fine-grained questions where their free-form answers are not easy to evaluate,
we adopt this binary-choice evaluation. Specifically, it takes the context-
question pair and the correct answer as inputs, and adopts a one-shot
prompting strategy to output a wrong candidate option. The specific prompt is
provided in Appendix A.2.

Dataset | Manageable | Scalable | Robust | Fine-Grained |  Total
---|---|---|---|---|---
Alternating | Complicating | Paraphrasing | Noising | Reversing | Planning | Knowledge | Retrieval
GSM8K | 96/100 | 65 | 55 | 90 | 90 | 61 | 71 | / | / | 432
CLUTRR | 96/100 | 88 | 78 | 76 | 80 | 72 | 69 | 81 | 64 | 608
StrategyQA | 83/100 | / | 57 | / | / | / | 78 | 65 | / | 200
BoolQ | 90/100 | 88 | 68 | 90 | 86 | 50 | / | / | 67 | 382

Table 2: Statistics of our evolving instances from four original datasets.

System Workflow The systematic procedure of our system is delineated in the
following steps. First, the instance pre-filter selects manageable instances
$(C_{o},Q_{o},A_{o})$ from the original evaluation set. From these, the
instance creator creates new instances $(C_{e},Q_{e},A_{e})$ across various
evolving types, and the candidate option formulator subsequently generates an
incorrect candidate option $O_{w}$ for each new context-question pair. Then
the instance verifier need to validate the correctness of both the new
generated instance $(C_{e},Q_{e},A_{e})$ and its corresponding incorrect
alternative $(C_{e},Q_{e},O_{w})$. Only those instances that pass this double-
verification process, i.e., the generated instance is examined as correct
while the alternative is incorrect, will be utilized for dynamic evaluation.
For a detailed exposition of the underlying algorithms, refer to Appendix A.1

##  3 Experiments

###  3.1 Setup

Tasks and Datasets Using our benchmark self-evolving framework, we dynamically
extend benchmark datasets of four different tasks, including mathematical
reasoning, logical reasoning, commonsense reasoning and reading comprehension.
Specifically, we utilize GSM8K Cobbe et al. (2021), CLUTRR Sinha et al.
(2019), StrategyQA Geva et al. (2021), and BoolQ Clark et al. (2019)
respectively for these four tasks, all involving multi-step complex reasoning.
Detailed dataset descriptions are provided in Appendix B.1.

We randomly select 100 instances from publicly available dev/test sets of each
dataset222The instances from CLUTRR are selected within the clauses of length
$k\leq 3$., and feed each instance into our multi-agent system to generate new
evaluation instances of various reframing types. For GSM8K, CLUTRR and BoolQ,
we generate new instances across all six types as in Table 1. For StrategyQA
without context, we only generate instances with complex and fine-grained
questions. Specifically for fine-grained sub-abilities, we focus on the task
planning ability for GSM8K and BoolQ, both task planning and implicit
knowledge identification for StrategyQA, and all three sub-abilities for
CLUTRR. Statistics of generated evolving datasets are summarized as Table 2.

Examined LLMs We evaluate both closed-source models, ChatGPT and ChatGLM Zeng
et al. (2023), and open-source models, LLama Touvron et al. (2023) and Mistral
Jiang et al. (2023), using our evolving evaluation datasets. We compare their
performance against on original datasets to demonstrate the effectiveness of
our framework. For closed-source models, we use gpt-3.5-turbo-1106 and
chatglm-turbo versions, while for open-source models, we employ
LLama2-70B-Chat and Mistral-7B-Instruct-v0.2. We also evaluate GPT-4 despite
its involvement in generating evolving instances, to test whether they can
also provide more scalable and robust evaluation for GPT-4.

Task Formats We adopt two task formats tailored to different evaluation
directions. For scalable and robust evaluations, we adhere to the original
datasets’ task types, and employ a two-shot CoT prompting strategy. For fine-
grained evaluation, we create binary-choice questions featuring two options, A
and B, among which one is randomly selected as the correct answer and the
other, developed by our Candidate Option Formulator, is incorrect. We
implement a zero-shot prompting approach for fine-grained evaluation.

###  3.2 Overall Comparison

We first provide an overall assessment of LLMs with scalable and robust
evaluations, leaving the fine-grained evaluation in Section 3.4. We conduct
scalable evaluation using instances with alternative and complex questions,
while performing robust evaluation using instances with paraphrased, noised
and reversed contexts, and compare their performance against on corresponding
original instances. For fair comparison, the average performance of original
instances involved in each evaluation type is reported as their original
performance. Table 3 presents the main comparisons, with arrows indicate
shifts from original to evolving evaluation results. We have following
findings.

Models | Scalable Evaluation | Robust Evaluation | Overall
---|---|---|---
(Evolving$\leftarrow$Original) | (Evolving$\leftarrow$Original) | (Evolving$\leftarrow$Original)
GSM8K
GPT-4 | 85.00 $\leftarrow$ 100.0 (-15.00) | 97.10 $\leftarrow$ 100.0 (- 2.90) | 93.07 $\leftarrow$ 100.0 (\- 6.93)
ChatGPT | 60.83 $\leftarrow$ 93.33 (-32.50) | 79.25 $\leftarrow$ 91.29 (-12.04) | 73.13 $\leftarrow$ 91.97 (-18.84)
ChatGLM | 42.50 $\leftarrow$ 66.67 (-24.17) | 62.66 $\leftarrow$ 67.22 (- 4.56) | 55.96 $\leftarrow$ 67.04 (-11.08)
LLama | 40.83 $\leftarrow$ 60.00 (-19.17) | 60.58 $\leftarrow$ 58.51 (+ 2.07) | 54.02 $\leftarrow$ 59.00 (\- 4.98)
Mistral | 27.50 $\leftarrow$ 41.67 (-14.17) | 35.27 $\leftarrow$ 39.42 (- 4.15) | 32.69 $\leftarrow$ 40.17 (\- 7.48)
CLUTRR
GPT-4 | 77.11 $\leftarrow$ 100.0 (-22.89) | 93.42 $\leftarrow$ 100.0 (- 6.58) | 86.55 $\leftarrow$ 100.0 (-13.45)
ChatGPT | 65.66 $\leftarrow$ 83.13 (-17.47) | 78.51 $\leftarrow$ 82.02 (- 3.51) | 73.10 $\leftarrow$ 82.49 ( -9.39)
ChatGLM | 55.42 $\leftarrow$ 73.49 (-18.07) | 67.11 $\leftarrow$ 74.56 (- 7.45) | 62.18 $\leftarrow$ 74.11 (-11.93)
LLama | 47.59 $\leftarrow$ 36.14 (+11.45) | 36.40 $\leftarrow$ 33.77 (+ 2.63) | 41.12 $\leftarrow$ 34.77 (+ 5.35)
Mistral | 45.78 $\leftarrow$ 55.42 (- 9.64) | 50.00 $\leftarrow$ 53.95 (- 3.95) | 48.22 $\leftarrow$ 54.57 (\- 6.35)
StrategyQA
GPT-4 | 98.25 $\leftarrow$ 100.0 (- 1.75) | / | 98.25 $\leftarrow$ 100.0 (\- 1.75)
ChatGPT | 64.91 $\leftarrow$ 91.23 (-26.32) | / | 64.91 $\leftarrow$ 91.23 (-26.32)
ChatGLM | 66.67 $\leftarrow$ 73.68 (- 7.01) | / | 66.67 $\leftarrow$ 73.68 (\- 7.01)
LLama | 78.95 $\leftarrow$ 75.44 (+ 3.51) | / | 78.95 $\leftarrow$ 75.44 (+ 3.51)
Mistral | 77.19 $\leftarrow$ 73.68 (+ 3.51) | / | 77.19 $\leftarrow$ 73.68 (+ 3.51)
BoolQ
GPT-4 | 99.36 $\leftarrow$ 100.0 (- 0.64) | 97.35 $\leftarrow$ 100.0 (- 2.65) | 98.17$\leftarrow$ 100.0 (\- 1.83)
ChatGPT | 92.31 $\leftarrow$ 91.03 (+ 1.28) | 91.15 $\leftarrow$ 90.27 (+ 0.88) | 91.62 $\leftarrow$ 90.58 (+ 1.04)
ChatGLM | 86.54 $\leftarrow$ 89.10 (- 2.56) | 90.71 $\leftarrow$ 88.05 (+ 2.66) | 89.01 $\leftarrow$ 88.48 (+ 0.53)
LLama | 84.62 $\leftarrow$ 92.31 (- 7.69) | 91.60 $\leftarrow$ 91.60 (- 0.00) | 88.74 $\leftarrow$ 91.88 (\- 3.14)
Mistral | 76.92 $\leftarrow$ 80.13 (- 3.21) | 83.19 $\leftarrow$ 79.20 (+ 3.99) | 80.63 $\leftarrow$ 79.58 (+ 1.05)

Table 3: Comparison of evolving and original evaluation. Left-side values of
the arrow are evolving results, while the right-side denotes original
performance on respective instances. Values in parentheses are performance
changes.

  1. (1)

Overall, most models exhibit reduced performance in our scalable and robust
evaluation compared to their original results across the first three tasks.
This offers a more accurate measure of LLMs’ capabilities, highlighting that
original results potential overestimate their proficiency. The decline of all
models except LLama in robust evaluation on GSM8K and CLUTRR indicating their
limited robustness.

  2. (2)

GPT-4, though achieving 100% accuracy on manageable original instances, still
experiences performance drops on our evolving instances. This is because these
evolving instances are generated by GPT-4 given the original ones with correct
answers, aiding model’s reasoning. Yet, directly evaluating GPT-4 with
evolving instances would uncover their limitations.

  3. (3)

Our scalable and robust evaluation effectively expand the performance gap
between models. Initially, GPT-4 and ChatGPT exhibit less than a 10% accuracy
difference on GSM8K, while this gap increased to 20% under our evolved
evaluation. On the BoolQ dataset where all models consistently perform well,
our scalable evaluation further highlights their disparities.

  4. (4)

Our framework also widens performance discrepancies of the same model across
various tasks. For example, while ChatGPT consistently achieves 80$\sim$90%
accuracy on four tasks, its proficiency notably diverges following scalable
evaluation, only maintaining stable performance on BoolQ. Similarly, GPT-4
shows performance disparities after our evaluation, maintaining effectiveness
on most datasets while showing a decrease on CLUTRR.

###  3.3 Analysis of Varied Reframing Operations

To further assess the impact of various reframing operations on model
evaluation, we gather results of each operation across all datasets and
compare them with corresponding original results. Our analysis as detailed in
Figure 3 shows that among five reframing operations, question complicating
cause the most disruption to models, followed by polarity reversing and
question alternating. In contrast, context paraphrasing and context noising
have a limited impact on model performance. These findings suggest that our
framework primarily enhances the original benchmarks by highlighting the
limitations of these LLMs regarding question generalizability and their
susceptibility to adversarial attacks.

Figure 3: Comparison of evolving results using various reframing operations
versus original results. Darker bars show accuracy for each operation across
all datasets, with lighter bars ahead representing original accuracy.

We further offer a perplexity-based analysis in Appendix B.4 demonstrating
that our generated instances exhibit greater complexity and diversity than the
original instances for dynamic evaluation, along with an error analysis in
Appendix B.5.

###  3.4 Further Analysis on Sub-Ability

The fine-grained evaluation in our framework paves a way to dissect the sub-
abilities of models. We aggregate the results of each sub-ability across all
datasets and compare the models’ rankings against their original ranking, as
shown in Figure 4a.

Substantial Discrepancy Between Original and Fine-grained Evaluations. Our
findings reveal that ChatGLM, initially lagging behind ChatGPT in original
evaluation, surprisingly outperforms ChatGPT in all sub-ability evaluations.
Upon scrutinizing ChatGPT’s results, we observe a significant selection bias
towards option ’A’. We hypothesize that such bias in certain LLMs impairs
their decision-making, leading to poorer performance.

Presence of Selection Bias in Certain LLMs. Following Zheng et al. (2023a), We
estimate the prior prediction distribution of different LLMs on options ID ’A’
and ’B’. The result in Figure 4b shows that ChatGPT, LLama, and Mistral
exhibit a significant preference towards ’A’, in contrast to the neutral
stance of GPT-4 and ChatGLM. For a fair model evaluation, we utilize a bias
calibrating method to obtain debiased results as shown in Figure 4c, with
methodologies for bias calculation and mitigation detailed in Appendix B.3.

a Biased results.

b Selection Bias of Various LLMs.

c Debiased results.

Figure 4: Results of fine-grained sub-ability evaluation. Figure 5: Comparison
of LLama-2-7B-Chat models under different contamination conditions. “Vanilla”,
“In-domain Cont.” and “Direct Cont.” denotes the original model, the in-domain
contaminated and direct contaminated models.

Improvement Potential for Planning Ability As Figure 4c, GPT-4 consistently
performs best across all three sub-abilities while Mistral showing the lowest
performance. Among three sub-abilities, planning emerges as the weakest skill
for all LLMs, highlighting a key area for further enhancements.

###  3.5 Quality of Evolving Instances

Human Verification To demonstrate the reliability of our dynamic evaluation,
we sample a subset of our generated instances and conduct a human annotation
333Manually verified by the authors. to assess their quality. Specifically, we
randomly select five instances that are incorrectly answered by ChatGPT for
each reframing operation across all datasets, with a total of 115 instances.
Following human verification, 110 out of 115 instances (95.7%) are deemed
accurate, reinforcing the credibility of our evolving instances.

Instance Filter Rate Our system incorporates a pre-filter and a double-
verification process to enhance the reliability of generated instances. The
pre-filter discards nearly 9% of the original instances that exceed the
capabilities of GPT-4. Following this, the double-verification stage filters
out approximately 24% which initially processed correctly by GPT-4 (Detailed
statistics are in Appendix B.2). This underscores the importance of our
double-verification strategy on instance quality.

###  3.6 Impact on Data Contamination

To further analyze our framework’s ability to mitigate data contamination, we
design controlled experiments to simulate data contamination. We construct two
instruction-tuning datasets: one simulating in-domain contamination by
including portions of our evaluation benchmark’s training set, and the other
simulating direct contamination by incorporating both training and evaluation
sets. We respectively use these two datasets to fine-tune LLama-2-7B-Chat
(details in Appendix B.6). We assess the original model and two fine-tuned
models using original evaluation instances and our generated evolving
instances, with results shown in Figure 5.

  * •

Compared to the original model, both in-domain and direct contaminated models
show notable improvement under original evaluation, revealing how data
contamination can skew evaluations. In contrast, under our dynamic evaluation,
the performance gap between contaminated models and the original model
decreases, especially in scalable and fine-grained assessments. This indicates
our framework’s resistance against data contamination.

  * •

In fine-grained evaluations, the in-domain contaminated model surpasses the
original model, indicating that in-domain training can enhance task-related
abilities. Yet, the direct contaminated model performs worse than the
original, suggesting that memorizing original answers may hinder problem-
solving in new contexts. This highlights the value of fine-grained evaluations
in mitigating data contamination effects.

##  4 Related Work

#### LLMs Evaluation

Recent advanced large language models OpenAI (2023); Jiang et al. (2023);
Touvron et al. (2023), have shown remarkable performance across various tasks
and sparked significant interest in their evaluation Chang et al. (2023); Liu
et al. (2023a). The evaluation for LLMs encompass automatic evaluation, human
evaluation and LLM-based evaluation that respectively utilizes automatic
metrics Liang et al. (2022), human preference Zheng et al. (2023b) and LLM
feedback Liu et al. (2023b); Zhang et al. (2023). Automatic evaluation offers
a most cost-effective approach for extensive and comprehensive assessments,
necessitating diverse benchmark datasets, including task-specific benchmarks
Yu et al. (2023); Holmes et al. (2023) and general benchmarks Hendrycks et al.
(2020); bench authors (2023). However, with the rapid evolution of LLMs and
potential data contamination issue, these static benchmarks tend to be
inadequate for a thorough assessment. To address this, our benchmark self-
evolving framework dynamically updating instances via a multi-agent system,
providing a dynamic solution for LLM evaluation.

Data Contamination The expansion of training datasets for LLMs introduces a
significant challenge of data contamination. This results in an in-domain
distribution overlap with existing datasets, or even publicly available
development and test sets, risking biased evaluation Sainz et al. (2023). Such
contamination undermines the fairness and accuracy of benchmarks Zhou et al.
(2023), casting doubt on whether high performance reflects true generalization
or merely data memorization Biderman et al. (2023). Shi et al. (2023) and
Golchin and Surdeanu (2023) propose detecting contaminated data from benchmark
for LLMs and use untainted data for evaluation. Besides, Wei et al. (2023)
utilize perplexity as an evaluation metric on newly sampled data without
additional annotations, yet this may not fully reflect models’ capabilities.
Our benchmark self-evolving framework can mitigate evaluation bias caused by
data contamination.

##  5 Conclusion

In this study, we introduce a benchmark self-evolving framework that employs a
multi-agent system to enhance existing benchmarks for a more scalable, robust
and fine-grained evaluation on LLMs. Results indicate a general decline in LLM
performance, alongside significant discrepancies in performance across various
models and tasks. This highlights the effectiveness of our framework in
providing a more accurate and comprehensive evaluation of LLMs, as well as
distinguishing the capabilities of various models. Our study aids in selecting
the most suitable LLMs for specific applications, and guide their further
development.

## Limitations

#### Limitation on benchmark coverage

Due to computational limit, our dynamic evaluation study only explores four
datasets across various textual tasks and select 100 instances from each
dataset to construct nearly 1600 evolving instances. Our framework can
flexibly generalize to other tasks and even different modalities for a broader
analysis.

#### Limitation on examine LLMs

We evaluate three closed-source (GPT-4, ChatGPT, ChatGLM) and two open-sourec
LLMs (LLama, Mistral) using our crafted evolving instances to illustrate our
scalable, robust and fine-grained evaluation. We acknowledge the limitation in
the scope of LLMs, and will later provide further experiments on more LLMs.

#### Limitation on instance accuracy

Despite incorporating a pre-filtering and double-verification procedure, our
system, which is entirely powered by GPT-4, may inevitably generate a small
number of instances with inaccuracies, as evidenced by human verification.
This might result in less accurate assessments of LLMs.

## Risks

#### Introduction of Factual Errors

For benchmark datasets containing factual information, such as BoolQ, our
framework may generate counterfactual information to alter the key details of
the original context during the polarity reversing operation. Such
inaccuracies, if inadvertently used as learning material by the models, could
negatively impact their performance and reliability.

#### Environmental Impact

A significant risk associated with our methodology is the potential increase
in environmental impact due to the extensive use of OpenAI’s APIs for large
language models. This is particularly concerning for benchmarks of substantial
size, as the energy consumption and carbon footprint associated with
generating evolving instances could be considerable.

## Ethics Statement

All data utilized in our benchmark self-evolving framework are sourced from
publicly available datasets, including GSM8K, CLUTRR, StrategyQA and BoolQ.
Our generated evolving instances for dynamic evaluation are also publicly
released for usage and have been subjected to a thorough review by the
authors. This setting guarantees transparency and reproducibility in our
experiments, allowing other researchers to evaluate and expand upon our work.
Our benchmark-evolving framework is strictly limited to be used for instance
generation that follow the ethical guidelines of the community. The authors
emphatically denounce the use of our framework for generating inaccurate or
harmful instances.

## References

  * bench authors (2023) BIG bench authors. 2023.  Beyond the imitation game: Quantifying and extrapolating the capabilities of language models.  _Transactions on Machine Learning Research_.
  * Biderman et al. (2023) Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O’Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. 2023.  Pythia: A suite for analyzing large language models across training and scaling.  In _International Conference on Machine Learning_ , pages 2397–2430. PMLR.
  * Chang et al. (2023) Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Kaijie Zhu, Hao Chen, Linyi Yang, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, et al. 2023.  A survey on evaluation of large language models.  _arXiv preprint arXiv:2307.03109_.
  * Chiang et al. (2023) Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. 2023.  Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality.
  * Clark et al. (2019) Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019.  Boolq: Exploring the surprising difficulty of natural yes/no questions.  _NAACL_.
  * Cobbe et al. (2021) Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. 2021.  Training verifiers to solve math word problems.  _arXiv preprint arXiv:2110.14168_.
  * Dong et al. (2023) Guanting Dong, Jinxu Zhao, Tingfeng Hui, Daichi Guo, Wenlong Wan, Boqi Feng, Yueyan Qiu, Zhuoma Gongque, Keqing He, Zechen Wang, and Weiran Xu. 2023.  Revisit input perturbation problems for llms: A unified robustness evaluation framework for noisy slot filling task.
  * Espejel et al. (2023) Jessica López Espejel, El Hassane Ettifouri, Mahaman Sanoussi Yahaya Alassan, El Mehdi Chouham, and Walid Dahhane. 2023.  Gpt-3.5, gpt-4, or bard? evaluating llms reasoning ability in zero-shot setting and performance boosting through prompts.  _Natural Language Processing Journal_ , 5:100032.
  * Gallegos et al. (2023) Isabel O Gallegos, Ryan A Rossi, Joe Barrow, Md Mehrab Tanjim, Sungchul Kim, Franck Dernoncourt, Tong Yu, Ruiyi Zhang, and Nesreen K Ahmed. 2023.  Bias and fairness in large language models: A survey.  _arXiv preprint arXiv:2309.00770_.
  * Geva et al. (2021) Mor Geva, Daniel Khashabi, Elad Segal, Tushar Khot, Dan Roth, and Jonathan Berant. 2021.  Did aristotle use a laptop? a question answering benchmark with implicit reasoning strategies.  _Transactions of the Association for Computational Linguistics_ , 9:346–361.
  * Golchin and Surdeanu (2023) Shahriar Golchin and Mihai Surdeanu. 2023.  Time travel in llms: Tracing data contamination in large language models.  _arXiv preprint arXiv:2308.08493_.
  * Hendrycks et al. (2020) Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2020.  Measuring massive multitask language understanding.  _arXiv preprint arXiv:2009.03300_.
  * Holmes et al. (2023) Jason Holmes, Zhengliang Liu, Lian Zhang, Yuzhen Ding, Terence T Sio, Lisa A McGee, Jonathan B Ashman, Xiang Li, Tianming Liu, Jiajian Shen, et al. 2023.  Evaluating large language models on a highly-specialized topic, radiation oncology physics.  _arXiv preprint arXiv:2304.01938_.
  * Jiang et al. (2023) Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. 2023.  Mistral 7b.  _arXiv preprint arXiv:2310.06825_.
  * Kiela et al. (2021) Douwe Kiela, Max Bartolo, Yixin Nie, Divyansh Kaushik, Atticus Geiger, Zhengxuan Wu, Bertie Vidgen, Grusha Prasad, Amanpreet Singh, Pratik Ringshia, et al. 2021.  Dynabench: Rethinking benchmarking in nlp.  _arXiv preprint arXiv:2104.14337_.
  * Liang et al. (2022) Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, et al. 2022.  Holistic evaluation of language models.  _arXiv preprint arXiv:2211.09110_.
  * Liu et al. (2023a) Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, et al. 2023a.  Agentbench: Evaluating llms as agents.  _arXiv preprint arXiv:2308.03688_.
  * Liu et al. (2023b) Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023b.  Gpteval: Nlg evaluation using gpt-4 with better human alignment.  _arXiv preprint arXiv:2303.16634_.
  * OpenAI (2023) OpenAI. 2023.  Gpt-4 technical report.
  * Peng et al. (2023) Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao. 2023.  Instruction tuning with gpt-4.  _arXiv preprint arXiv:2304.03277_.
  * Pezeshkpour and Hruschka (2023) Pouya Pezeshkpour and Estevam Hruschka. 2023.  Large language models sensitivity to the order of options in multiple-choice questions.
  * Sainz et al. (2023) Oscar Sainz, Jon Ander Campos, Iker García-Ferrero, Julen Etxaniz, Oier Lopez de Lacalle, and Eneko Agirre. 2023.  Nlp evaluation in trouble: On the need to measure llm data contamination for each benchmark.  _arXiv preprint arXiv:2310.18018_.
  * Shi et al. (2023) Weijia Shi, Anirudh Ajith, Mengzhou Xia, Yangsibo Huang, Daogao Liu, Terra Blevins, Danqi Chen, and Luke Zettlemoyer. 2023.  Detecting pretraining data from large language models.  _arXiv preprint arXiv:2310.16789_.
  * Sinha et al. (2019) Koustuv Sinha, Shagun Sodhani, Jin Dong, Joelle Pineau, and William L Hamilton. 2019.  Clutrr: A diagnostic benchmark for inductive reasoning from text.  _arXiv preprint arXiv:1908.06177_.
  * Touvron et al. (2023) Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023.  Llama: Open and efficient foundation language models.  _arXiv preprint arXiv:2302.13971_.
  * Wei et al. (2022) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022.  Chain-of-thought prompting elicits reasoning in large language models.  _Advances in Neural Information Processing Systems_ , 35:24824–24837.
  * Wei et al. (2023) Tianwen Wei, Liang Zhao, Lichang Zhang, Bo Zhu, Lijie Wang, Haihua Yang, Biye Li, Cheng Cheng, Weiwei Lü, Rui Hu, Chenxia Li, Liu Yang, Xilin Luo, Xuejie Wu, Lunan Liu, Wenjun Cheng, Peng Cheng, Jianhao Zhang, Xiaoyu Zhang, Lei Lin, Xiaokun Wang, Yutuan Ma, Chuanhai Dong, Yanqi Sun, Yifu Chen, Yongyi Peng, Xiaojuan Liang, Shuicheng Yan, Han Fang, and Yahui Zhou. 2023.  Skywork: A more open bilingual foundation model.
  * Yang et al. (2023) Jingfeng Yang, Hongye Jin, Ruixiang Tang, Xiaotian Han, Qizhang Feng, Haoming Jiang, Bing Yin, and Xia Hu. 2023.  Harnessing the power of llms in practice: A survey on chatgpt and beyond.  _arXiv preprint arXiv:2304.13712_.
  * Yu et al. (2023) Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James T Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. 2023.  Metamath: Bootstrap your own mathematical questions for large language models.  _arXiv preprint arXiv:2309.12284_.
  * Zeng et al. (2023) Aohan Zeng, Xiao Liu, Zhengxiao Du, Zihan Wang, Hanyu Lai, Ming Ding, Zhuoyi Yang, Yifan Xu, Wendi Zheng, Xiao Xia, Weng Lam Tam, Zixuan Ma, Yufei Xue, Jidong Zhai, Wenguang Chen, Zhiyuan Liu, Peng Zhang, Yuxiao Dong, and Jie Tang. 2023.  GLM-130b: An open bilingual pre-trained model.  In _The Eleventh International Conference on Learning Representations (ICLR)_.
  * Zhang et al. (2023) Xinghua Zhang, Bowen Yu, Haiyang Yu, Yangyu Lv, Tingwen Liu, Fei Huang, Hongbo Xu, and Yongbin Li. 2023.  Wider and deeper llm networks are fairer llm evaluators.  _arXiv preprint arXiv:2308.01862_.
  * Zheng et al. (2023a) Chujie Zheng, Hao Zhou, Fandong Meng, Jie Zhou, and Minlie Huang. 2023a.  Large language models are not robust multiple choice selectors.  _arXiv e-prints_ , pages arXiv–2309.
  * Zheng et al. (2023b) Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric. P Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023b.  Judging llm-as-a-judge with mt-bench and chatbot arena.
  * Zhou et al. (2023) Kun Zhou, Yutao Zhu, Zhipeng Chen, Wentong Chen, Wayne Xin Zhao, Xu Chen, Yankai Lin, Ji-Rong Wen, and Jiawei Han. 2023.  Don’t make your llm an evaluation benchmark cheater.  _arXiv preprint arXiv:2311.01964_.
  * Zhu et al. (2023) Kaijie Zhu, Jiaao Chen, Jindong Wang, Neil Zhenqiang Gong, Diyi Yang, and Xing Xie. 2023.  Dyval: Graph-informed dynamic evaluation of large language models.

##  Appendix A Details of Framework

###  A.1 Algorithm Design

The pseudo-code for the algorithm of our multi-agent evolving instance setting
system is presented in Algorithm 1.

Algorithm 1 Multi-Agent Evolving Instance Setter

0: An original evaluation instance $(C_{o},Q_{o},A_{o})$, its task description
$s$.

0: An evolving instance $(C_{e},Q_{e},A_{e})$.

1: $(C_{o},Q_{o},A_{o})\leftarrow$ Instance Pre-filter($(C_{o},Q_{o},A_{o})$,
$s$)

2: $(C_{e},Q_{e},A_{e})\leftarrow$ Instance Generator($(C_{o},Q_{o},A_{o})$,
$s$)

3: $O_{w}\leftarrow$ Candidate Option Formulator($(C_{e},Q_{e},A_{e})$, $s$)

4: if Instance Verifier$(C_{e},Q_{e},A_{e})$ and not Instance
Verifier$(C_{e},Q_{e},O_{w})$ then

5: return $(C_{e},Q_{e},A_{e})$

6: else

7: return  _NULL_

8: end if

###  A.2 Prompts of Multi-Agent Evolving Instance Setter

Table 4, 5, 6, 7, 8, 9 and 10 present prompts for Instance Generator of
different reframing operations. Table 11 presents the prompt for Instance
Verifier. Table 12 presents the prompt for Candidate Option Formulator.

Table 4: Prompt on Question Alternating

Prompt for Instance Generator on Question Alternating You are an expert
Question Creator. You will receive an instance of {task description},
including a context, a question and its answer. You are tasked with creating
an alternative question to explore a different aspect of the original problem.
Please do not change the context but just edit the question and the answer.
Please first generate the question. Then think step-by-step in one line to
give an brief analysis of the question, Finally, directly present a short
answer omitting the intermediate steps, in a single line. Context: {context
$C_{o}$} Original Question: {question $Q_{o}$} Original Answer: {answer
$A_{o}$} Alternative Question:

Table 5: Prompt on Context Paraphrasing

Prompt for Instance Generator on Context Paraphrasing You are an expert
Question Creator.You will receive an instance of {task description}, including
a context, a question and its answer. Your task is to rephrase the given
context in a short and easy-readable manner without summarizing or explaining.
Confirm that the rephrased context do not change the answer to the original
question. Simply output the rephrased context and do not output the original
question. Context: {context $C_{o}$} Original Question: {question $Q_{o}$}
Original Answer: {answer $A_{o}$} Alternative Context:

Table 6: Prompt on Context Noising

Prompt for Instance Generator on Context Noising You are an expert Question
Creator. You will receive an instance of {task description}, including a
context, a question and its answer. You are tasked with creating a new context
by inserting irrelevant facts within the critical sentences of the original
context. Make sure these facts shouldn’t change the correct answer to the
question. Simply output the rephrased context and do not output the original
question. Context: {context $C_{o}$} Original Question: {question $Q_{o}$}
Original Answer: {answer $A_{o}$} Alternative Context:

Table 7: Prompt on Polarity Reversing

Prompt for Instance Generator on Polarity Reversing You are an expert Question
Creator. You will receive an instance of {task description}, including a
context, a question and its answer. Your task is to generate a new context by
altering key details in the original context. Ensure that the rest of the
original context remains unchanged. The altered details should change the
answer to the question. Please first output the rephrased context. Then give
an one-line step-by-step analysis of the original question based on the new
context. Finally, generate the corresponding direct answer in a newline.
Context: {context $C_{o}$} Original Question: {question $Q_{o}$} Original
Answer: {answer $A_{o}$} Alternative Context:

Table 8: Prompt on Planning

Prompt for Instance Generator on Planning You are an expert Task Planner. You
will receive an instance of {task description}, including a context, a
question and its answer. Your task is to generate a new question and its
corresponding answer, aiming to ask about the plan to solve the original
question given the context. Your new question can either inquire about all
reasoning steps required or ask for the specific details about a certain
(e.g., first, second, or last) step. Please first generate the question. Then
think step-by-step in one line to give an brief analysis of the question,
Finally, directly present a short answer omitting the intermediate steps, in a
single line. Context: {context $C_{o}$} Original Question: {question $Q_{o}$}
Original Answer: {answer $A_{o}$} Alternative Question:

Table 9: Prompt on Retrieval

Prompt for Instance Generator on Retrieval You are an expert Relevant Context
Retriever. You will receive an instance of {task description}, including a
context, a question and its answer. Your task is to generate a new question
and its corresponding answer, aiming to identify the relevant information from
the given context necessary to solve the original question with the original
answer. Your answer must be exclusively from the given context, to contain all
required information to solve the original question and cover the original
answer. Please first generate the question. Then think step-by-step in one
line to give an brief analysis of the question, Finally, directly present a
short answer omitting the intermediate steps, in a single line. Context:
{context $C_{o}$} Original Question: {question $Q_{o}$} Original Answer:
{answer $A_{o}$} Alternative Question:

Table 10: Prompt on Knowledge

Prompt for Instance Generator on Knowledge You are an expert Relevant Context
Retriever. You will receive an instance of {task description}, including a
context, a question and its answer. Your task is to generate a new question
and its corresponding answer, aiming to ask about the implicit knowledge
(e.g., facts, rules, commonsense, …) required to solve the original question.
Your new answer should directly list all required implicit knowledge for the
question. Please first generate the question. Then think step-by-step in one
line to give an brief analysis of the question, Finally, directly present a
short answer omitting the intermediate steps, in a single line. Context:
{context $C_{o}$} Original Question: {question $Q_{o}$} Original Answer:
{answer $A_{o}$} Alternative Question:

Table 11: Prompt for Instance Verifier

Prompt for Instance Verifier You are an expert Question-Answer Validator. You
will receive an instance of {task description}, including a context, a
question and its answer. Your task is to validate whether the answer is
correct to solve the question given the context. Please think step-by-step in
one line to analyze whether the answer is correct for the question and the
context. Then give your final judgement with Yes or No in a newline. Context:
{context $C$} Question: {question $Q$} Answer: {answer $A$} Judgement:

Table 12: Prompt for Candidate Option Formulator

Prompt for Candidate Option Formulator You are an expert Candidate Option
Generator. You will receive an instance of {task description}, including a
context, a question and its answer. Your task is to modify the provided answer
to generate a candidate option that wrongly answer the question given the
context. Context: {context $C$} Question: {question $Q$} Answer: {answer $A$}
Option:

##  Appendix B Experimental Analysis

###  B.1 Dataset Descriptions

  * •

GSM8K: a collection of grade school math problems in a free-form QA format,
featuring diverse arithmetic and algebraic problems.

  * •

CLUTRR: a synthesized free-form question answering dataset designed for
evaluating logical reasoning over kinship relationships.

  * •

StrategyQA: consists of crowdsourced yes/no questions that require implicit
reasoning steps and commonsense strategies. The instances in StrategyQA
consist solely of questions and answers, with their contexts being null.

  * •

BoolQ: a reading comprehension dataset sourced from Google’s Natural
Questions, offers yes/no questions based on real Google searches paired with
answers from Wikipedia articles.

###  B.2 Instance Filtering Statistics

Table 13 shows the percentage of instances filtered by double-verification.

Table 13: Percentage (%) of instances filtered by double-verification.

Dataset | Scalable | Robust | Fine-grained |  Average
---|---|---|---|---
Alternating | Complicating | Paraphrasing | Noising | Reversing | Planning | Knowledge | Retrieval
GSM8K | 32.29% | 42.71% | 6.25% | 6.25% | 36.46% | 26.04% | / | / | 25.00%
CLUTRR | 8.33% | 18.75% | 20.83% | 16.67% | 25.00% | 28.13% | 15.63% | 33.33% | 20.83%
StrategyQA | / | 31.33% | / | / | / | 6.02% | 21.69% | / | 19.68%
BoolQ | 2.22% | 24.44% | 0.00% | 4.44% | 44.44% | / | / | 25.56% | 29.26%
Average | 14.28% | 29.17% | 9.03% | 9.12% | 35.30% | 20.06% | 18.66% | 29.45% | 23.69%

###  B.3 Selection Bias Analysis

In our discussion, selection bias denotes the tendency that a model inherently
assigns a higher probability to specific ID tokens, such as A or B, in multi-
choice questions. The impact of selection bias varies across different models
and tasks, influencing the models’ performance by reducing their robustness in
handling multi-choice problems Zheng et al. (2023a). The permutation-based
debiasing method, which averages the model’s prediction distributions across
various option permutations, theoretically eliminates selection bias. However,
due to limited access to the prediction distributions of closed-source models,
we employ a sampling approximation approach to estimate and mitigate selection
bias.

To formalize our discussion, we use $C$ to denote the concatenation of context
and question. Suppose an multi-choice problem consists of $n$ options,
$id_{i}$ denotes the $i$th option ID in the default order, and $o_{i}$ denotes
the $i$th option content in the default order, $i\in\\{1,2,...,n\\}$. The
option IDs remain in their default order in all multi-choice questions, while
the option contents can be rearranged through different permutations to match
with different option IDs. We introduce $I$ as a permutation of the set
$\\{1,2,...,n\\}$, and $\mathcal{I}$ represents the set of all possible
permutations $I$. We use $x^{I}$ to denote the concatenation of the default-
ordered option IDs with the option contents permuted according to $I$.

We assume that given $C$ and $x^{I}$, the observed probability of the model
selecting $id_{i}$ is $P_{\text{biased}}\left(id_{i}\mid C,x^{I}\right)$,
which can be decomposed as:

$\displaystyle Z_{x^{I}}^{-1}P_{\text{prior}}\left(id_{i}\mid
C\right)P_{\text{debiased}}\left(o_{f_{I}(i)}\mid C,x^{I}\right)$ (1)

where $Z_{x^{I}}^{-1}$ is the normalization factor,
$P_{\text{prior}}\left(id_{i}\mid C\right)$ represents the prior probability
of selecting $id_{i}$, $P_{\text{debiased}}\left(o_{f_{I}(i)}\mid
C,x^{I}\right)$ represents the debiased prediction probability of
$o_{f_{I}(i)}$, and $f_{I}(i)$ denotes the $i$th element in permutation $I$.

We assume $P_{\text{debiased}}$ is invariant to the ordering of options. By
applying logarithms to both $P_{\text{biased}}$ and its decomposed expression,
and then summing across all $I$ in $\mathcal{I}$ on both sides of the
equation, we can simplify and compute $P_{\text{prior }}\left(id_{i}\mid
C\right)$ as:

$\displaystyle\operatorname{softmax}\left(\frac{1}{|\mathcal{I}|}\sum_{I\in\mathcal{I}}\log
P_{\text{biased }}\left(id_{i}\mid C,x^{I}\right)\right)$ (2)

Based on the estimated $P_{\text{prior }}\left(id_{i}\mid C\right)$ and
Equation 1, we can compute the normalized $P_{\text{debiased
}}\left(o_{f_{I}(i)}\mid C,x\right)$.

For our analysis, the $P_{\text{prior }}\left(id_{i}\mid C\right)$ and
$P_{\text{debiased }}\left(o_{f_{I}(i)}\mid C,x\right)$ are computed
independently for different datasets. Based on the correct option’s ID, we
categorize all multi-choice questions in a dataset into different permutation
sets to support $x^{I}$. We calculate the frequency of the model selecting
$id_{i}$ at different permutation sets to estimate $P_{\text{biased
}}\left(id_{i}\mid C,x^{I}\right)$.

###  B.4 Dataset Perplexity Analysis

Perplexity is a metric that quantifies the complexity and the predictability
of a dataset. By analyzing the perplexity of dataset, we can gain insights
into the relative difficulty models may encounter during testing, as well as
the diversity of information within the dataset. In this analysis, we
calculate the perplexity of newly evolving datasets derived by reframing the
original GSM8K dataset and compared them with their original counterparts. The
comparison results are presented in Figure 6.

Our findings indicate that the datasets created through different reframing
operations exhibit an increase in perplexity compared to the original
instances. This indicates that the reframed instances are more complex and
less predictable. These results, aligning with the experimental observations
discussed in Section 3.3, suggest that our framework has the ability to
generate instances with enhanced linguistic structure and diversity compared
to the original instances.

Figure 6: Perplexity comparison between original and reframed datasets.

###  B.5 Error Analysis

Tables 14, 15and 16 present cases from GSM8K where ChatGPT successfully
responds to the original instance but fails to answer the evolved version. The
case in Table 14 introduces an additional reasoning step to the original
instance, which causes ChatGPT to misinterpret the context and perform an
incorrect calculation. In Table 15, the evolved instance alters two data
points from the original case, leading ChatGPT to neglect the critical
information "4 times a week," resulting in an inaccurate response. The example
in Table 16 demonstrates that, although ChatGPT correctly calculates the total
number of hours the candle will burn in the original instance, it fails to
address a direct question about the first reasoning step in the new instance.

By utilizing instances from our evolving framework, we can expose model’s
limitations more effectively, including its vulnerability to perturbations,
diminished generalization capabilities on more complex problems, and
inadequacies in addressing questions targeting specific sub-abilities.

Table 14: Instance Before and After Question Complicating Original Instance |  New Instance
---|---
Context: A robe takes 2 bolts of blue fiber and half that much white fiber. |  Context: A robe takes 2 bolts of blue fiber and half that much white fiber.
Question: How many bolts in total does it take? |  Question: If a merchant has 20 bolts of blue fiber and 10 bolts of white fiber, how many robes can he make using all the fibers without any leftovers?
Answer: 3 |  Answer: 10
Response: It takes 2 + 1 = 3 bolts in total. |  Response: The merchant can make 20/2 = 10 robes using the blue fiber. He can make 10/2 = 5 robes using the white fiber. So, he can make a total of 5 robes using all the fibers without any leftovers.
Judgement: ✓ |  Judgement: $\times$
Table 15: Instance Before and After Polarity Reversing Original Instance |  New Instance
---|---
Context: Jim spends 2 hours watching TV and then decides to go to bed and reads for half as long. He does this 3 times a week. |  Context: Jim spends 3 hours watching TV and then decides to go to bed and reads for a third as long. He does this 4 times a week.
Question: How many hours does he spend on TV and reading in 4 weeks? |  Question: How many hours does he spend on TV and reading in 4 weeks?
Answer: 36 |  Answer: 64
Response: Jim spends 2 + (2/2) = 3 hours on TV and reading each time. So, in 4 weeks, he spends 3 * 3 * 4 = 36 hours on TV and reading. The answer is 36. |  Response: Jim spends 3 + (3/3) = 4 hours on TV and reading each time. In 4 weeks, he spends 4 * 4 = 16 hours on TV and reading. The answer is 16.
Judgement: ✓ |  Judgement: $\times$
Table 16: Instance Before and After Planning Operation Original Instance |  New Instance
---|---
Context: A candle melts by 2 centimeters every hour that it burns. |  Context: A candle melts by 2 centimeters every hour that it burns.
Question: How many centimeters shorter will a candle be after burning from 1:00 PM to 5:00 PM? |  Question: What is the first reasoning step to calculate how many centimeters shorter a candle will be after burning from 1:00 PM to 5:00 PM?
|  A. The first reasoning step is to measure the initial height of the candle.
|  B. The first reasoning step is to determine the total number of hours the
candle will burn.
Answer: 8 |  Answer: B
Response: The candle burns for 5-1=4 hours. So, it will be 2*4=8 centimeters shorter. The answer is 8. |  Response: A
Judgement: ✓ |  Judgement: $\times$

###  B.6 Data Contamination Experiment Details

We construct an instruction tuning dataset comprising 4,000 general instances
from alpaca-gpt4-data Peng et al. (2023) and additional 4,000 instances, with
1,000 each from the training sets of GSM8K, CLUTRR, StrategyQA and BoolQ. This
dataset is used to fine-tune a model to simulate in-domain contamination.
Furthermore, we incorperate 400 instances from the original benchmark into the
instruction tuning dataset to fine-tune another model to simulate the direct
contamination.