# Circumgalactic Ly$\alpha$ Nebulae in Overdense Quasar Pair Regions Observed
with the Palomar Cosmic Web Imager

Jessica S. Li11affiliation: Department of Astronomy and Steward Observatory,
University of Arizona, Tucson, AZ, USA , Carlos J. Vargas11affiliation:
Department of Astronomy and Steward Observatory, University of Arizona,
Tucson, AZ, USA , Donal O’Sullivan22affiliation: Cahill Center for
Astrophysics, California Institute of Technology, 1216 East California
Boulevard, Mail code 278-17, Pasadena, CA 91125, USA , Erika
Hamden11affiliation: Department of Astronomy and Steward Observatory,
University of Arizona, Tucson, AZ, USA , Zheng Cai33affiliation: Department of
Astronomy, Tsinghua University, Beijing 100084, China;
<EMAIL_ADDRESS>44affiliation: UCO/Lick Observatory, University of
California, 1156 High Street, Santa Cruz, CA 95064, USA , Mateusz
Matuszewski22affiliation: Cahill Center for Astrophysics, California Institute
of Technology, 1216 East California Boulevard, Mail code 278-17, Pasadena, CA
91125, USA , Christopher Martin22affiliation: Cahill Center for Astrophysics,
California Institute of Technology, 1216 East California Boulevard, Mail code
278-17, Pasadena, CA 91125, USA , Miriam Keppler11affiliation: Department of
Astronomy and Steward Observatory, University of Arizona, Tucson, AZ, USA ,
Haeun Chung 11affiliation: Department of Astronomy and Steward Observatory,
University of Arizona, Tucson, AZ, USA , Nicole Melso 11affiliation:
Department of Astronomy and Steward Observatory, University of Arizona,
Tucson, AZ, USA , Shiwu Zhang 33affiliation: Department of Astronomy, Tsinghua
University, Beijing 100084, China<EMAIL_ADDRESS>
(Accepted in ApJ May 2023)

###### Abstract

The recent discovery of enormous Ly$\alpha$ nebulae (ELANe), characterized by
physical extents $>200$ kpc and Ly$\alpha$ luminosities $>10^{44}$ erg s-1,
provide a unique opportunity to study the intergalactic and circumgalactic
medium (IGM/CGM) in distant galaxies. Many existing ELANe detections are
associated with local overdensities of active galactic nuclei (AGN). We have
initiated a search for ELANe around regions containing pairs of quasi-stellar
objects (QSOs) using the Palomar Cosmic Web Imager (PCWI). The first study of
this search, Cai et al. (2018), presented results of ELAN0101+0201 which was
associated with a QSO pair at $z=2.45$. In this study, all targets residing in
QSO pair environments analyzed have Ly$\alpha$ detections, but only one of the
four targets meets the classification criteria of an ELANe associated with a
QSO pair region (z$\sim 2.87$). The other three sample detections of
Ly$\alpha$ nebulae do not meet the size and luminosity criteria to be
classified as ELANe. We find kinematic evidence that the ELANe J1613, is
possibly powered mostly by AGN outflows. The analysis of circularly-averaged
surface brightness profiles of emission from the Ly$\alpha$ regions show that
the Ly$\alpha$ emission around $z\sim 2$ QSO pairs is consistent with emission
around individual QSOs at $z\sim 2$ (Cai et al., 2019), which is fainter than
that around $z\sim 3$ QSOs (Arrigoni Battaia et al., 2019b; Cai et al., 2019;
Fossati et al., 2021). A larger sample of Ly$\alpha$ at z$\sim$2 will be
needed to determine if there is evidence of redshift evolution when compared
to nebular emissions at z$\sim$3 from other studies.

##  1 Introduction

The intergalactic medium (IGM) and circumgalactic medium (CGM) are of crucial
importance in regulating galaxy formation and evolution. They act as
reservoirs for fueling star formation in galaxies, and are subject to feedback
processes responsible for the ejection of matter from galaxies. Their
morphology and kinematics provide stringent constraints to models of galaxy
formation (Stewart et al., 2011; Martin et al., 2015a). In particular, giant
Ly$\alpha$ nebulae (also known as “Ly$\alpha$” blobs, LABs) are characterized
by a high luminosity of Ly$\alpha$ line emission (L${}_{\rm{Ly\alpha}}\gtrsim$
$10^{43}$ erg s-1), and a spatially large Ly$\alpha$ emitting region up to
hundreds of kpc (Dey et al., 2005). For example, Matsuda et al. (2004) reports
the detection of 35 LABs at z = 3.1, two of which are extremely Ly$\alpha$
bright and have large physical extents (Steidel et al., 2000). In another
study, Yang et al. (2009) discovers four new LABs, where two of the four are
powered by QSOs at z = 2.3. A broadband survey for Ly$\alpha$ nebulae by
Prescott et al. (2013) confirmed the presence of four large Ly$\alpha$
nebulae, with Ly$\alpha$ luminosities ranging from $\sim$ 1043 to 1044 erg s-1
at z $\sim$ 2-3. A subset of LABs, referred to as enormous Ly$\alpha$ nebulae
(ELANe; Cai et al. 2017a; Arrigoni Battaia et al. 2018), are defined as having
extents $>200$ kpc and L${}_{\rm{Ly\alpha}}>10^{44}$ erg s-1.

The Ly$\alpha$ line is the primary coolant of cold $T\sim 10^{4}$ K low
metallicity gas and can be used to map the CGM/IGM via emission. Several
mechanisms have been identified that should lead to Ly$\alpha$ emission from
the CGM: cooling, in-falling gravitationally heated gas (Haiman et al., 2000),
cooling following superwind-driven shocks (Taniguchi & Shioya, 2000), and
Ly$\alpha$ fluorescence induced by exposure to UV radiation. Each of these
mechanisms is likely to be dominant at different scales, with multiple
processes responsible for powering a single Ly$\alpha$ nebula.

Ly$\alpha$ fluorescence caused by the much stronger UV radiation from a nearby
source can also boost emission from a Ly$\alpha$ nebula into the detectable
regime (Rees, 1988; Haiman & Rees, 2001; Kollmeier et al., 2010). The surface
brightness of an optically thin Ly$\alpha$ nebulae is roughly proportional to
$M_{\rm{gas}}^{2}$. Strongly clustered, ultra-luminous QSO groups provide us
with both conditions: (1) a strong meta-galactic ionising flux from an over-
density of AGN; (2) a significant gas overdensity in this system. These
strongly clustered groups are the best sites to initially search for giant
Ly$\alpha$ nebulae, where Ly$\alpha$ emission can reach IGM-scale distances of
$\gtrsim 500$ kpc.

###  1.1 Confirmed LABs and ELANe

Several systematic efforts have been made to search for LABs and ELANe at
$z\approx 2$. Using the VLT/X-shooter spectrograph, Zafar et al. (2011)
measures a LAB with a velocity gradient that may indicate gas inflow or
outflow that is associated with a QSO pair. Through deep narrow-band imaging
surveys around ultra-luminous type-I QSOs, Cantalupo et al. (2014) and Hennawi
et al. (2015) discovered two different giant Ly$\alpha$ nebulae with spatial
extent $\gtrsim 400$ kpc. The Slug nebula (Cantalupo et al., 2014) is the
first LAB that extends to IGM scales of 500 kpc. It is associated with a QSO
pair consisting of one bright and one faint QSO. The Jackpot nebula (Hennawi
et al., 2015) resides in a massive overdensity of 4 AGN. These two nebulae
have similar sizes and surfaces brightnesses (SBs) although QSO luminosities
differ by an order of magnitude. This suggests that the QSOs sufficiently
photoionized the surrounding gas resulting in optically thin ELANe. Besides
AGN photo-ionization, additional mechanisms, e.g., cooling flows, may play a
role in powering Ly$\alpha$ emission in massive halos (Arrigoni Battaia et
al., 2019a). Using the Palomar Cosmic Web Imager (PCWI), Martin et al. (2015a)
reveals large velocity shifts of a giant LAB, the Slug Nebula. Assuming these
shifts are due to kinematics, these PCWI observations yielded the first 3-D
picture of how massive galaxies acquire gas from the IGM. The results suggest
that the extended Ly$\alpha$ emission is consistent with a cooling flow of
accretion (Stewart et al., 2011) – the inflow deposits gas and angular
momentum into the circumgalactic medium (CGM), transports cool gas to the
galaxy, and maintains large star formation rates at high-redshift. However,
observations of Ly$\alpha$ emission alone are unable to isolate whether the
velocity shifts are due to radiative transfer effects or kinematics. More
observations using non-resonant lines (e.g. H$\alpha$, H$\beta$, etc.)
alongside Ly$\alpha$ emissions suggest that there are variations of physical
distance between emission regions and the associated quasar (Ouchi et al.,
2020; Arrigoni Battaia et al., 2019a). These variations imply that the
emission structure is oriented along our line of sight with its length
extending out to Mpc scales. The Cantalupo et al. (2019) Multi Unit
Spectroscopic Explorer (MUSE) non-resonant line observations of the Slug
Nebula are different from those of PCWI, since MUSE data shows abrupt velocity
gradients in the nebula’s neighboring regions. This implies that there may be
more complex kinematics than the rotational motion proposed by Martin et al.
(2015a). Cai et al. (2017a) detected a giant LAB, MAMMOTH-1, extending to
nearly $450$ kpc despite a significantly shallower observation than either the
Jackpot or Slug nebulae. MAMMOTH-1, observed with narrow-band imaging, is
$1.5\times$ the size of the Slug (z $\simeq$ 2.28) and Jackpot (z $\simeq$
2.04) nebulae when measured from SB contours of $4.8\times 10^{-18}$ ergs s-1
cm-2 arcsec-2. MAMMOTH-1 resides in a unique field where there are several
strong Ly$\alpha$ absorbers at $z=2.32\pm 0.02$ which are associated with 5
background QSOs, projected within 20 $h^{-1}$ Mpc. The MAMMOTH survey has also
detected a second LAB (Cai et al., 2017b), indicating that these large LABs
are well associated with clustered QSOs.

In addition to this evidence for a LAB connection with clustered QSOs,
Borisova et al. (2016a) have targeted ultra-luminous QSOs with $g<19$ at $z=3$
and find 100% incidence of extended Ly$\alpha$ emission. Cai et al. (2019)
found extended Ly$\alpha$ emission associated with 14 out of their 16 QSO
sample at $z=2.1-2.3$. Among those 14 detected Ly$\alpha$ nebulae, 4 had
physical extents and luminosities consistent with ELANe. Results from the
Fluorescent Lyman-Alpha Structures in High-z Environments (FLASHES) Survey
(O’Sullivan et al., 2020a) indicates significant detections of Ly$\alpha$
emission around galaxies with 2.4 $<$ z $<$ 2.75, including several QSOs
covered by the Keck Baryonic Structure Survey (Rudie et al., 2012). With the
Multi Unit Spectroscopic Explorer (MUSE), Arrigoni Battaia et al. (2019a)
observed a total sample of 61 extended Ly$\alpha$ emission nebulae that have
an average maximum projected distance of 80 kpc, each associated with quasars
at $z=3.03-3.46$. At higher redshifts, Bielby et al. (2020) detected an
extended Ly$\alpha$ nebula around a z$\sim$5.26 quasar. Additionally, in the
Reionization Epoch QUasar InvEstigation with MUSE (REQUIEM) Survey, 12 of the
31 z$=5.7-6.6$ quasars were associated with Ly$\alpha$ nebulae (Farina et al.,
2019). Drake et al. (2019) confirmed the presence of extended Ly$\alpha$ halos
from 4 out of 5 observations at z$\sim 6$. In light of these detections of
extended gas around both clustered and single QSOs at a range of redshifts, we
attempt to more systematically understand the extent and kinematics of gas
around the most densely clustered QSOs at a slightly lower redshift, focusing
on z $\simeq$ 2.4.

Here, we describe our results from initial observations of our 4 survey
targets using the Palomar Cosmic Web Imager (PCWI) at the 200 inch Hale
Telescope.

##  2 Target Selection

We selected our targets from the SDSS-IV QSO database of 200,000 QSO spectra
(Pâris et al., 2017), providing a set of targets between $z=2.2-2.8$ that are
accessible with PCWI. From this sample we select 12 QSO groups for a multi-
year observing program. These fields were selected because they contain the
strongest clustered QSO groups in 1’ area (to roughly match the FoV of PCWI).
The extreme over-dense nature of these fields is further suggested by strong
IGM Ly$\alpha$ absorption in the spectrum of background QSOs on a larger scale
of $\sim 10\ h^{-1}$ Mpc (Cai et al., 2016). Of these 12 target QSO groups,
only 4 were observed with reliable data to date (see Section 3).

Our selection criteria involved finding QSO pairs with separations less than
1’, redshifts within $\Delta$z=0.02 of each other, at least one QSO having g
$<$ $18$ mag, and observable with CWI. In the first stage of our CWI program,
we search for extended emission that bridge the QSO pairs and are in the
periphery of the QSOs. Higher spectral resolution follow up observations with
KCWI would provide higher signal to noise observations that illuminate details
such as gas kinematics and metal emission lines. ELAN0101+0201 at z=2.4, the
closest separation pair, is the first of our survey to be published by Cai et
al. (2018). Details of the QSO pair sample are included in Table 1.

##  3 Observations

The Palomar Cosmic Web Imager is a slicer integral field unit (IFU), designed
for observing low surface brightness, diffuse objects, in particular emission
from the CGM (Matuszewski et al., 2010). CWI uses a $40\arcsec$ by $60\arcsec$
image slicer to divide the field into 24 slices with area $40\arcsec$ by
$2\arcsec.5$, resulting in a sampling of $\sim 0.55^{\prime\prime}$/pix. CWI
uses a novel observing mode, nod-and-shuffle, which allows for near perfect
sky subtraction. CWI is mounted on the Hale 5 m telescope Cassegrain focus, at
Palomar Observatory. CWI, with a full suite of gratings, covers a bandpass
from $380$ to $770$ nm, with the ability to observe a wavelength range of $45$
nm at a time. Our observations used the Richardson grating (R $=2000$) and
blue filter. A more thorough description of the instrument, observing
strategies, and data pipeline can be found in Martin et al. (2014a).

Name | RA | DEC | RA | DEC | $z^{a}$ | $z^{a}$ |  g-band Maga |  g-band Maga |  i-band Maga |  i-band Maga | separ. |  Central $\lambda$
---|---|---|---|---|---|---|---|---|---|---|---|---
|  QSO1 |  QSO1 |  QSO2 |  QSO2 |  QSO1 |  QSO2 |  QSO1 |  QSO2 |  QSO1 |  QSO2 | arcsec | Å
ELAN0101b | 01:01:16.5 | +02:01:57.4 | 01:01:16.9 | +02:01:49.8 | 2.443 | 2.459 |  18.18$\pm$0.01 |  21.68$\pm$0.06 |  18.19$\pm$0.01 |  21.62$\pm$0.11 |  9$\arcsec$ | 3920
J1613 (T2) | 16:13:02.0 | +08:08:14.3 | 16:13:01.7 | +08:08:06.1 | 2.3864 | 2.3818 |  18.91$\pm$0.01 |  19.55$\pm$0.01 |  18.80$\pm$0.01 |  19.47$\pm$0.02 |  9.45$\arcsec$ | 4120
J1120 (T3) | 11:20:53.2 | +46:33:35.4 | 11:20:54.2 | +46:33:27.2 | 2.5122 | 2.5108 |  18.62$\pm$0.01 |  21.87$\pm$0.06 |  18.60$\pm$0.01 |  21.67$\pm$0.12 |  13.17$\arcsec$ | 4270
J1334 (T4) | 13:34:24.7 | +45:28:55.4 | 13:34:23.5 | +45:29:02.3 | 2.2620 | 2.2519 |  17.97$\pm$0.01 |  19.32$\pm$0.01 |  17.78$\pm$0.01 |  19.31$\pm$0.02 |  14.38$\arcsec$ | 3960
J1342 (T5) | 13:42:10.8 | +60:35:22.3 | 13:42:06.1 | +60:15:06.1 | 2.3975 | 2.3850 |  17.93$\pm$0.01 |  21.33$\pm$0.04 |  21.97$\pm$0.20 |  21.45$\pm$0.09 |  47.9$\arcsec$ | 4120
Table 1:  Table of QSO pair characteristics and observational parameters for
our PCWI sample. The seeing for observations varied between 1-2”.
aValues from SDSS DR17.
bResults from Cai et al. (2018).

###  3.1 Data Reduction

We used the standard CWI pipeline to perform the data reduction (Martin et
al., 2014c). The CWI data pipeline consists of a multi-stage process which
yields a 3-D data cube from each image. We use the pipeline to build a master
bias and flat field image (ideally from twilight flats) for each night of
observations. Cosmic rays are then removed, and a geometric solution is
derived to identify and extract slices, and to verify wavelength coverage
across the field of view. For nod and shuffle observations, as in this case,
the sky image is subtracted from the object image before further processing. A
sky cube is also generated for verification and diagnostics. Slice to slice,
or relative response, corrections are made based on the master flat and final
standard star calibration is performed to convert the data cube pixel
intensities to physical flux units. For our chosen observing mode, one data
cube is generated, containing $\sim$20 minutes of sky-subtracted data on the
target object. For multiple images of the same target, data cubes are
registered by hand to ensure targets are spatially aligned, removing flexure
or pointing errors as much as possible.

Further data analysis outside of the CWI pipeline was done using
CWITools111https://github.com/dbosul/cwitools, a software library developed by
author Donal O’Sullivan. For each of our targets, we stacked all cubes,
aligning along the bright central QSO. PSF subtraction is performed using the
same methodology as O’Sullivan & Chen (2020) to subtract bright QSOs and
isolate extended nebular emission. First, a broadband image is created by
summing over all the wavelength layers of the cropped cube. DAOStarFinder from
the package Photutils (Bradley et al., 2020) is used to identify QSOs in the
white light image above a signal-to-noise ratio (SNR) threshold of $\sim
5\sigma$ from the broadband image. For each QSO, a 3D PSF model is built and
subtracted from the data cube. In order to create a 3D PSF model, a 2D PSF
model is made by integrating over wavelength windows of 150Å, centered on each
wavelength channel. Then, the 2D PSF model is scaled to match the PSF in each
wavelength pixel (0.55Å wide). The scaling factor is determined by minimizing
the residual sum of squares within a radius of $\sim 1".5$, and the scaled PSF
is subtracted from each wavelength channel out to a radius of $\sim 5"$, which
is typically about twice the seeing. We found a SNR threshold of $\sim
5\sigma$ and a radius of $\sim 5"$ sufficiently removed the bright region of
PSFs. Pixels used for scaling purposes are masked and excluded from future
analysis. Ly$\alpha$ wavelengths were not masked in building the empirical PSF
model. The analysis between masked and unmasked PSF models showed that the
5$"$ radius circular masks centered on PSF subtracted QSOs in the moment maps
mask the entire area of the PSF subtraction and any differences that would
arise between the two models.

Residual background is present due to small amounts of diffuse scattering from
the camera, the grating, and light leakage under the nod-and-shuffle mask. The
background has no structure comparable to extended emission lines, but removal
creates a cleaner smoothed image. We therefore subtract the remaining
background structure from these cubes using a first order polynomial fit.

The data cubes are then smoothed using an adaptive kernel at a SNR threshold
of $\sim$3.5 to better outline faint diffuse emission. They are smoothed using
boxcar wavelength kernels and gaussian spatial kernels, first spectrally, then
spatially. First, data cubes are spectrally smoothed by 2Å kernels. Then,
spatial smoothing starts at a FWHM of 3 pixels followed by larger smoothing
kernels in 2 pixel increments up to 20 pixels. When a spaxel (spatial pixel
that can move in a third dimension, the wavelength axis) exceeds the noise
threshold, it is selected for the final cube as a detection. Before the next
spatial smoothing cycle, flux within the smoothed spaxel is subtracted from
the unsmoothed cube. Subsequently, this unsmoothed difference cube is
spectrally smoothed by a 4Å kernel, afterwards, spatial smoothing is repeated.
This algorithm repeats a last time using an 8Å kernel for spectral smoothing.
This process also creates a SNR cube for each object that is used to define
the 2$\sigma$ contours in Figures 1, 2, 3, and 4 (Martin et al., 2014b, c,
2015b, 2016, 2019; O’Sullivan et al., 2020b).

After smoothing, potential Ly$\alpha$ emission in the cube was found by
segmenting objects in the data cube and creating an object cube. During
segmentation, voxels (0.594” x 0.594” x 0.55Å) above a SNR of 4 were
identified as detections, but only groups of 500 voxels or more were selected
to create the object cube. Finally, pseudo-narrow band (PNB) images are
created by summing up all wavelength channels in the final unsmoothed data
cube with the chosen objects from the object segmentation cube, and converting
the image from flux (erg s-1 cm-2 Å-1) to surface brightness (erg s-1 cm-2
arcsec-2) units. First (velocity) and second (dispersion) moment maps are
created from the PNB image. One-dimensional spectra are made by summing along
the two spatial axes for the specified object. O’Sullivan & Chen (2020)
details the process of creating these data products within CWITools.

##  4 Results

Extended Ly$\alpha$ emission was detected in all four targets, but only one
target has characteristics consistent with the definition of an ELAN
($>$200kpc and $>$1044erg s-1). The 1$\sigma_{\textrm{SB}}$ sensitivity of
each target (Table 2) was calculated over a rest-frame velocity range within
$\pm$500 km s-1 of the Ly$\alpha$ line based on the mean redshift of each QSO
pair. An image was made for each target by integrating over wavelength
channels of this velocity range. At least fourteen 1arcsec2 circular apertures
were used on each $\pm$500 km s-1 integrated image to calculate the
1$\sigma_{\textrm{SB}}$ sensitivity. We characterize each nebula’s physical
extent by measuring the largest angular separation on the 2$\sigma$ central
large surface brightness contour within the produced pseudo-narrowband
Ly$\alpha$ images. Each target’s total Ly$\alpha$ luminosity was measured by
summing background-subtracted emission in the pseudo-narrowband Ly$\alpha$
images. Spectral resolution limits velocity dispersion measurements to a floor
of $\sim$200 km/s. Nebulae calculated star formation rates were derived from
the H$\alpha$ star formation rate (Kennicutt, 1998). In Table 2, it is assumed
that star formation is due to case-B recombination, where the relation for
H$\alpha$ and Ly$\alpha$ luminosity of the source is $L_{Ly_{\alpha}}$ =
8.7$L_{H_{\alpha}}$ (e.g., Dijkstra & Westra, 2010).

###  4.1 J1613 (T2)

We present the PCWI PSF-subtracted Ly$\alpha$ cube products for target J1613
in Figure 1. The 6600s total exposure time on target J1613 yields an empirical
1$\sigma_{SB}$ of 8.69$\times$10-18 erg s-1 cm-2 arcsec-2 within a wavelength
range of 4107.72 to 4122.57Å. The PNB image contains wavelengths 4110.47 to
4140.72Å, and the 2$\sigma$ contour encompasses an area of 576 arcsec2.
Ly$\alpha$ emission in the PNB has an angular extent of $37".4$, corresponding
to a projected physical size of 305 kpc at the mean redshifts of QSO 1 and 2.
The total Ly$\alpha$ luminosity associated with target J1613 is
1.80$\times$1044 erg s-1. The extended size and Ly$\alpha$ luminosity both
sufficiently exceed the criteria for ELAN classification. The data depth,
extent, and Ly$\alpha$ luminosity for all sample targets are displayed in
Table 2.

Like the quasar pair seen in Arrigoni Battaia et al. (2019b) where there is an
intergalactic bridge connecting two QSOs, the Ly$\alpha$ emission morphology
resembles structure extending between the two QSOs. The velocity gradient
across where the J1613 QSO pair resides may indicate interaction. The velocity
dispersion is limited by the spectral resolution of the instrument $\lesssim$
200 km s-1, however dispersions greater than that appear between the QSOs and
around both QSOs. Shock heating and photoionization are potential powering
mechanisms for the ELANe in the CGM of this system (see Section 5.2).

Figure 1: Target J1613 PCWI Ly$\alpha$ emission results. The upper left panel
shows the pseudo-narrowband (PNB) Ly$\alpha$ image created from the wavelength
range shown in the shaded red region in the bottom panel with the full nebula
1-D spectrum. Black (QSO 1) and gray (QSO 2) circles in the upper panels
represent the positions and PSF sizes of the QSOs that were PSF subtracted
out. White contours in the PNB image represent the calculated 2$\sigma$
Ly$\alpha$ emission surface brightness level. The upper middle and upper right
panels show the velocity map (moment 1), and velocity dispersion map (moment
2) respectively. The moment maps use the average Ly$\alpha$ line emission of
the QSO pair at z$\sim 2.385$ as the rest-frame velocity of the system. They
are calculated based on data points within the central large 2$\sigma$ contour
in the PNB image. The smaller 2$\sigma$ contours are excluded since they
appear to be regions of noise. The PNB, velocity, and velocity dispersion maps
are smoothed relative to adjacent pixels using gouraud shading. The lower
panel shows the full range of 1-D nebular spectrum plotted with the associated
QSO pair. A red vertical solid (dotted) line is drawn in the 1-D spectrum at
the wavelength of Ly$\alpha$ emission at the redshift of QSO 1 (QSO 2). The
1-D QSO 1 and QSO 2 spectra fluxes are scaled by $10^{-4}$ to fit on the same
axis range as the nebular spectrum.

###  4.2 J1120 (T3)

The data products for target J1120 are shown in Figure 2. The 3600s total
exposure time yields a 1$\sigma_{SB}$ of 8.92$\times$10-18 erg s-1 cm-2
arcsec-2 within a wavelength range of 4261.33 to 4275.63Å. The PNB image
contains wavelengths 4278.38 to 4290.48 Å, and the 2$\sigma$ contour
encompasses an area of 62 arcsec2. The Ly$\alpha$ emission has an angular
extent of $20".2$, corresponding to a projected physical size of 163 kpc at
the average redshifts of QSO 1 and 2. The total Ly$\alpha$ luminosity
associated with target J1120 is 0.11$\times$1044 erg s-1, which does not meet
the criteria for ELAN classification. Future Lya observations of J1120 that
probe to deeper surface brightness levels may uncover additional Lya emission.
The kinematic distribution shows a bulk of the gas flows near the systemic
velocity.

There is a bulk of blueshifted features (negative velocity in the velocity
map) on the northernmost edge of the Ly$\alpha$ emission region. It is
possible that this feature is a shock front propagating outward from a cluster
galaxy within the nebula. The low average velocity dispersion is potentially
caused by the relative lack of mechanical energy input from feedback.

Figure 2: Target J1120 PCWI Ly$\alpha$ emission results. The upper left panel
shows the PNB Ly$\alpha$ image created the wavelength range shown in the
shaded red region in the bottom right panel of the full nebular 1-D spectrum.
The upper right and lower left panels show the velocity and velocity
dispersion maps respectively. These moment maps use the average redshifted
Ly$\alpha$ emission line of the QSO pair at z$\sim 2.51$ as the rest-frame
velocity of the system. The 1-D QSO1 and QSO2 spectra fluxes are scaled by
$10^{-2.5}$ and $10^{-1.5}$ respectively to fit on the same axis range as the
nebular spectrum. The panel display characteristics are mirrored from Figure
1.

###  4.3 J1334 (T4)

The data products for target J1334 are shown in Figure 3. The 6000s total
exposure time yields a 1$\sigma_{SB}$ of 8.58$\times$10-18 erg s-1 cm-2
arcsec-2 within a wavelength range of 3952.31 to 3965.5 Å. The PNB image
contains wavelengths 3967.71 to 3988.06 Å, and the 2$\sigma$ contour
encompasses an area of 311 arcsec2. The Ly$\alpha$ emission has an angular
extent of $34".5$ corresponding to a projected physical size of 284 kpc at the
average redshifts of QSO 1 and 2. The total Ly$\alpha$ luminosity associated
with target J1334 is $0.40\times 10^{44}$ erg s-1, which does not meet the
criteria for ELANe classification. However, the structure of J1334 is
remarkably clumpy, so the source may host numerous Ly$\alpha$ emitting
galaxies (LAEs). Future KCWI observations with higher spatial and spectral
resolution can make identifying individual sources possible.

Figure 3: Target J1334 PCWI Ly$\alpha$ emission results. Velocity and velocity
dispersion maps use the average Ly$\alpha$ emission line of the QSO pair at
z$\sim 2.257$ as the rest-frame velocity of the system. The 1-D QSO 1 and QSO
2 spectra fluxes are scaled by $10^{-2.5}$ and $10^{-2}$ respectively to fit
on the same axis range as the nebular spectrum. The panel layout and display
characteristics are mirrored from Figure 2.

###  4.4 J1342 (T5)

The data products for target J1342 are shown in Figure 4. The 7200s total
exposure time yields a 1$\sigma_{SB}$ of 8.39$\times$10-18 erg s-1 cm-2
arcsec-2 within a wavelength range of 4115.74 to 4129.49 Å. The PNB image
contains wavelengths 4105.84 to 4158.64 Å, and the 2$\sigma$ contour
encompasses an area of 121 arcsec2. The Ly$\alpha$ emission has an angular
extent of $27".5$ at the average redshifts of QSO 1 and 2. The total
Ly$\alpha$ luminosity associated with target J1342 is $0.37\times 10^{44}$ erg
s-1. The Ly$\alpha$ luminosity does not meet the criteria for ELANe
classification. Features surrounding the main central 2$\sigma$ contour were
masked in the calculation of velocity and velocity dispersion maps to exclude
artifacts that resemble noise. The north to south velocity gradient suggests
the presence of a bipolar flow from QSO 1.

Figure 4: Target J1342 PCWI Ly$\alpha$ emission results. The moment maps use
the average Ly$\alpha$ emission line of the QSO pair at z$\sim 2.39$ as the
rest-frame velocity. The 1-D QSO 1 and QSO 2 spectra fluxes are scaled by
$10^{-3.5}$ and $10^{-2}$ respectively to fit on the same axis range as the
nebular spectrum. The red arrows are in the direction of the location of QSO 2
outside of the field of view. The panel layout and display characteristics are
mirrored from Figure 2. Figure 5: Ly$\alpha$ surface brightness (SB) profiles
as a function of radius from the brighter QSO of the QSO pair. These SB
profiles were corrected for cosmological dimming. Profiles were created from
averaging the SB within annuli defined at increasing distances from the
brighter QSO. The black lines are the SB profiles from the current PCWI study,
each target is associated with a QSO pair at z$\sim$2.2-2.5 (Table 2). The red
line is the median, and the red shaded region represents the SB within 25 to
75 percentiles of this study. Purple data points represent the median, and
their error bars show the 25 to 75 percentile of the SB profile from the Cai
et al. (2019) KCWI sample of 16 QSOs at z=2.1-2.3. Green data points represent
radial profiles around two individual QSOs, a bright and an intermediate
Ly$\alpha$ emission detection from the FLASHES survey (O’Sullivan et al.,
2020b). Detection drops to around zero for this study at $\gtrsim$ 50 kpc.

Target | t exp | 1$\sigma_{SB}$ ($\times 10^{-18}$ | Angular Extent | LLyα |  aSFRLyα | ELANe?
---|---|---|---|---|---|---
Name | (sec) | erg s-1 cm-2 arcsec-2) | (kpc) | ($\times 10^{44}$ erg s-1) | (M⊙ yr-1) |
ELAN0101b | 14400 | 4.50 | $232$ | 4.5 | 409 | Yes
J1613 (T2) | 6600 | 8.69 | $305$ | 1.80 | 163 | Yes
J1120 (T3) | 3600 | 8.92 | $163$ | 0.11 | 10 | No
J1334 (T4) | 6000 | 8.58 | $284$ | 0.40 | 36 | No
J1342 (T5) | 7200 | 8.39 | $224$ | 0.19 | 17 | No

Table 2:  Data depth and Ly$\alpha$ nebulae characteristics. Calculations for
1$\sigma_{SB}$ are based off of 5$\sigma$ contour measurements. Angular
extents are maximum extents

on the largest central 2$\sigma$ surface brightness contour of each target’s
FoV. Distance calculations are based on a $\Lambda$CDM cosmology with
$\Omega_{m}=0.3$, $\Omega_{\Lambda}=0.7$, and $h=0.7$ (Wright, 2006).
Ly$\alpha$ luminosity and SFR are calculated for emission regions within
2$\sigma$ detection.
aCalibration from Kennicutt (1998) and Dijkstra & Westra (2010).

##  5 Discussion

Ly$\alpha$ emission was found around all QSO pair targets, but only one
reaches the criteria of an ELANe, this target (J1613) has a Ly$\alpha$
luminosity $>$ 1044 erg s-1 and a projected physical size $>300$ kpc. These
enormous and extremely bright nebulae are exceedingly rare near $z\sim 2$.
Below, we explore relationships between these nebulae and their host QSOs, as
well as their powering mechanisms.

###  5.1 Integrated Sample Properties with QSO Characteristics

The Ly$\alpha$ nebulae sizes and integrated Ly$\alpha$ luminosities are shown
in Table 2. We explore correlations of these results with properties of both
QSO 1 and QSO 2 for each target. In our data, there is no observed correlation
between LLyα and QSO magnitude in any of the SDSS bands, or between Ly$\alpha$
nebula size and QSO magnitude. Lastly, there is no redshift evolution over the
range of system redshifts in our sample with neither LLyα nor Ly$\alpha$
nebula size.

###  5.2 ELANe Powering Mechanisms

We follow the approach of Cai et al. (2018) to explore the powering mechanisms
of the nebulae in our new sample. Extended Ly$\alpha$ emission can be powered
by shock heating from AGN outflows, photoionization, gravitational cooling,
and resonant scattering from the QSO broadline region (Cantalupo, 2017).
First, AGN outflows are expected to behave like fast shocks which ionize the
material around quasars, resulting in Ly$\alpha$ emission where $F_{Ly\alpha}$
$\propto$ $n_{H}$$v_{\rm{shock}}^{3}$ (e.g., Allen et al. (2008)). Second,
Ly$\alpha$ emission could be produced by recombinations following
photoionization of hydrogen by the hard spectrum of quasars. In this scenario,
as we discuss below, there are two limiting cases for optically thin and
optically thick gas due to ionizing radiation (e.g., Hennawi & Prochaska
(2013)). Ly$\alpha$ fluorescence on the edges of optically thick clouds arises
where hydrogen is photoionized by the UV radiation of quasars. Within these
clouds, resonant scattering of Ly$\alpha$ photons experience absorption and
re-emission by ground state hydrogen atoms until they finally escape the cloud
(Cantalupo et al., 2005; Prochaska et al., 2013). Lastly, Ly$\alpha$ emission
could be due to gravitational cooling as collisionally excited neutral
hydrogen at $\gtrsim$ 104-5 K flows into galaxies (Geach et al., 2016; Steidel
et al., 2011; Dijkstra & Loeb, 2009).

Since kinematics show evidence of potential AGN outflows and that nebulae may
be photoionized by one or both QSOs in each QSO pair, the subsequent analysis
will only focus on the former two scenarios. AGN outflows eject gas
symmetrically outward, shock heating surrounding gas. These motions could
manifest as a bipolar kinematic pattern the nebular emission velocity maps,
depending upon the orientation of the outflow feature. Two of our sample
targets, J1613 (T2) and J1342 (T5), have kinematic patterns that may indicate
the presence the bipolar flows. Alexander et al. (2010) and Harrison et al.
(2014) suggest that AGN outflows can eject matter at velocities upward of
$v_{\rm{max}}\gtrsim 1500$ km s-1. Though the velocity maps of ELANe J1613
(T2) and J1342 (T5) do not show velocities greater than $1500$ km s-1, the
coherent velocity structures reach larger extrema than our two other sample
targets. Furthermore, an outflow oriented slightly away from our line of sight
may lead to an underestimate of the absolute velocity of the gas due to
projection. However, improved higher resolution data is needed to definitively
determine whether or not these targets have bipolar flows.

To explore the photoionization scenario, we assume that a large majority of
ionizing photons come from QSO 1, and hence the nebulae are photoionized by
QSO 1, except for J1613 where the QSOs are similar in brightness. Hennawi &
Prochaska (2013) find the surface brightness of an optically thin cloud due to
recombination follows:

$\displaystyle{\rm{SB}^{\rm{thin}}_{\rm{Ly}\alpha}=8.8\times
10^{-20}\left(\frac{1+z}{3.253}\right)^{-4}\times\left(\frac{f^{\rm{thin}}_{\rm{C}}}{0.5}\right)}\times$
$\displaystyle\left(\frac{N_{\rm{H}}\times{n_{\rm{H}}}}{10^{20.5}\rm{cm}^{-5}}\right)\rm{erg}\rm{s}^{-1}\rm{cm}^{-2}\rm{arcsec}^{-2}$

For this calculation, we assume values of covering factor, hydrogen column
density, and hydrogen volume density. Covering factor, is
$f_{\rm{C}}^{\rm{thin}}=0.5$, as suggested by Arrigoni Battaia et al. (2015a,
b); Hennawi et al. (2015) for a relatively smooth Ly$\alpha$ morphology.
Hydrogen column density is approximately $N_{\rm{H}}\approx 1\times
10^{20.5\pm 1.0}$ cm-2 based on photoionization models by Lau et al. (2016)
used to estimate hydrogen ionization fraction, where they showed that hydrogen
column density did not vary significantly up to a distance of 200 kpc from the
QSO. Hydrogen number density is taken to be n${}_{\textrm{H}}$ $\approx 1$
cm-3 for cold T$\sim$104 K gas in the CGM (Tumlinson et al., 2013), where the
SB${}_{\textrm{Ly}\alpha}\sim$ 10-19 to 10-17 erg s-1cm-2arcsec-2 at $\sim$ 50
kpc of QSO 1 for all targets. The expected surface brightness of an optically
thin cloud due to recombination is then $\sim$8$\times$10-20 erg s-1 cm-2
arcsec-2. This value is below the observed surface brightness of our sample
within 200 kpc of QSO 1, which suggests that the CGM density of our extended
Ly$\alpha$ nebulae sample is $\sim$2-3 orders of magnitude higher than nebulae
found in more isolated QSOs at $z\sim 2$ (Arrigoni Battaia et al., 2016). The
same is also found with nebulae from the Cai et al. (2019) KCWI study and
FLASHES Survey which have SB$\sim 10^{-17}$ erg s-1 cm-2 arcsec-2. For the
optically thick case (Hennawi & Prochaska, 2013; Cai et al., 2018):

$\displaystyle\rm{SB}^{\rm{thick}}_{\rm{Ly}\alpha}=5.3\times
10^{-17}\left(\frac{1+z}{3.45}\right)^{-4}\times\left(\frac{f^{\rm{thick}}_{\rm{C}}}{0.5}\right)\times$
$\displaystyle\left(\frac{\rm{R}}{160\rm{kpc}}\right)^{-2}\times\left(\frac{L_{\nu_{\rm{LL}}}}{10^{30.9}\rm{erg}\rm{s}^{-1}\rm{Hz}^{-1}}\right)\rm{erg}\rm{s}^{-1}\rm{cm}^{-2}\rm{arcsec}^{-2}$

For this estimation, we assume $f^{\rm{thick}}_{\rm{C}}=0.5$. Cai et al.
(2018) estimates Log
$\frac{\rm{L}_{\nu_{\rm{LL}}}}{\rm{erg}\rm{s}^{-1}}=31.1$, by scaling the
composite spectrum of QSO 1 to match the i-band magnitude of QSO 1 in
ELAN0101, where $\rm{L}_{\nu_{\rm{LL}}}$ is the specific luminosity at the
Lyman edge (Arrigoni Battaia et al., 2015b; Lusso et al., 2015). The predicted
surface brightness from the optically thick case at $\sim 50$ kpc from QSO 1
is SB${}^{\rm{thick}}_{\rm{Ly}\alpha}$ $\sim$ 10-15 erg s-1cm-2arcsec-2 which
is $\sim$ 2 orders of magnitude higher than our SB${}_{\textrm{Ly}\alpha}$.
This suggests that the covering factor of the optically thick gas may be lower
than what is assumed for this estimation, or photoionization of optically
thick gas is not the mechanism that significantly contributes to the observed
Ly$\alpha$ emission.

###  5.3 Surface Brightness Profiles of Ly$\alpha$ Emission

Redshift-corrected circularly-averaged surface brightness profiles, are shown
in Figure 5. A characteristic surface brightness of $\sim 10^{-17}$ erg s-1
cm-2 arcsec-2 is detected in the QSO pair samples at z $\approx$ 2.3. The
chosen lower radial limit for all targets is 18.1 kpc from the brighter QSO of
each QSO pair. Surface brightness is calculated in increments of 10log(r)+0.1
kpc, where $r$ is radial distance from the brighter QSO in kpc. These radial
increments are the midlines of annuli with $\pm$10log(r)+0.05 kpc thickness.
Each data point is the average surface brightness within a spatial annulus
calculated from the targets’ pseudo-narrowband images. The red line shows the
median of our PCWI sample and the red region spans between the 25th to 75th SB
percentile of the sample at incremental distances from QSO1. Radial profiles
are plotted from $\sim$15 to 75 kpc, following the Cai et al. (2019) study
that a power law profile of Ly$\alpha$ SB centered at QSO 1 is only valid at
this range for QSOs at z$\approx$2.3. QSO 2 is beyond 75 kpc from QSO 1 for
all targets. These nebulae are likely in fairly disturbed environments and may
be drawing their energy from multiple external sources. Hence, their surface
brightness profiles may be more affected by their environments than Ly$\alpha$
nebulae identified through other means. Despite this, these profiles provide a
valid comparison to other samples of LABs and ELANe. We compare the results of
Figure 5 to surface brightness profiles of other Ly$\alpha$ nebulae at various
redshifts and environments. Arrigoni Battaia et al. (2016) found 7 out of a
sample of 15 QSOs at z$\sim$2 have Ly$\alpha$ emission within $\sim$50 kpc of
the QSO. At z$\sim$2.65, an analysis of 92 UV-continuum selected galaxies
showed a presence of Ly$\alpha$ emission in 55$\%$ of the sample, where the
Ly$\alpha$ nebulae had an extent of at least $\sim$80 kpc (Steidel et al.,
2011). Ly$\alpha$ nebulae in both studies had an average
SB${}_{\textrm{Ly}\alpha}\sim$10-17erg s-1 cm-2 arcsec-2. Higher redshift
studies at z$\sim 3-6$ by (Wisotzki et al., 2016) detected Ly$\alpha$ emission
in 21 of 26 galaxies, and compared with Ly$\alpha$ halos at z$\sim$0 concluded
that higher redshift galaxies have Ly$\alpha$ regions that are 5$\times$ more
extended with SBs of 10-18 to 10-17 erg s-1 cm-2 arcsec-2. At z$\sim 3.17$, a
MUSE sample of 61 extended Ly$\alpha$ nebulae around QSOs showed that most of
the Ly$\alpha$ nebulosities’ SB ($\gtrsim$10-18erg s-1 cm-2 arcsec-2) reside
within 50kpc of the host QSO, regardless of environmental variants such as
radio-loudness and the presence of active QSO companions (Arrigoni Battaia et
al., 2019a). A smaller MUSE study of 17 QSOs at z$\sim 3.5$ found that every
QSO is surrounded by giant Ly$\alpha$ nebulae of projected sizes ranging from
$\gtrsim$100 to 320 kpc. Therefore, Borisova et al. (2016b) suggests that
Ly$\alpha$ emission regions are more readily found around bright QSOs at
z$\sim$3-4, in contrast to the low detection rate of Ly$\alpha$ nebulae around
QSOs at z$\sim$2\. The bulk SB of Ly$\alpha$ nebulosities around QSOs seem to
mostly reside within $<$50 kpc of its host QSO at all redshifts for all these
studies.

We include the median of the sample from Cai et al. (2019) as purple data
points with error bars spanning between 25th to 75th percentile of the sample
in Figure 5. Figure 4 of Cai et al. (2019) shows surface brightness profiles
of the composite of these studies, and we include a redshift-corrected y-axis
in Figure 5 for direct comparison to that study. Shown in green are examples
of a bright (z=2.49) and an intermediate (z=2.79) detection of quasar fields
from the FLASHES (Fluorescent Lyman-Alpha Structures in High-z Environments)
Survey observed with PCWI. The FLASHES Survey consists of 48 quasar fields at
an average z$\sim$2.6 and the redshift dimming corrected characteristic SB for
the sample is $\sim$6$\times$ 10-18 erg s-1 cm-2 arcsec-2 (O’Sullivan et al.,
2020b). These comparisons show relative agreement with the $z\sim 2$ results
of Cai et al. (2019), since the surface brightness values of our sample are
generally slightly lower overall than those at $z\sim 3$. As discussed in that
study, this redshift evolution can be explained through two possibilities:
either nebular emission at $z\sim 2$ has less circular morphology or covering
fraction than that at $z\sim 3$, or the nebular surface brightness at $z\sim
2$ is intrinsically fainter. This would imply a lower local density or gas
mass at $z\sim 2$ than at $z\sim 3$. However, a larger sample size of
Ly$\alpha$ nebulae at z $\sim$ 2 would be needed to confirm whether or not
there is strong evidence of redshift evolution between z $\sim$ 2-3.

###  5.4 Detection Frequency in QSO Pair Systems and Future Observations

The discovery of ELANe around QSO pair systems represents an opportunity to
efficiently study the extreme CGM. To date, nearly all QSO pair systems
observed with PCWI/KCWI show evidence of extended Ly$\alpha$ emission (Cai et
al. 2018 and this study). Future surveys of the CGM could use this success as
a marker for target selection that would lead to greater detection confidence.
QSO pair systems trace overdense regions that are likely progenitors to high
mass clusters at $z=0$ (Onoue et al., 2018). Only one ELANe has been
discovered in a sample of 61 QSOs at $z\sim 3$ (Arrigoni Battaia et al.,
2019a), representing a $\sim 1\%$ probability of detection in a given QSO
system. More ELANe can be discovered by specifically targeting multiple QSO
systems at $z\sim 3$ and beyond, like the system found by (Arrigoni Battaia et
al., 2018), where Ly$\alpha$ emission was associated with 2 Ly$\alpha$
emitters and 2 QSOs, and another system where Ly$\alpha$ emitting
intergalactic structures form bridges between a QSO pair (Arrigoni Battaia et
al., 2019b). The need for higher resolution follow-up observations of these
targets is great. The higher spatial resolution of KCWI would grant the
ability to study individual substructures in the ELANe, such as LAEs and
bright nodes in detail. The larger spectral resolution would allow for robust
Ly$\alpha$ line profile fitting within individual substructures. Such an
analysis could reveal asymmetries and broad/narrow line components. Line
asymmetries are more accurate indicators of ISM outflows or line of sight
absorption characteristics. The characteristics of broad/narrow line
components could also reveal the existence of AGN in nebular substructures
(Trainor et al., 2016).

##  6 Conclusion

The discovery of ELANe around QSO pair systems at cosmic noon represents a
unique opportunity to study the CGM without the need for entirely blind
surveys. We detect extended Ly$\alpha$ emission in the PCWI data from all four
sample targets between $2.26<z<2.51$. One of the four targets had intense
Ly$\alpha$ emission consistent with the definition of an ELAN. We classify
this emission region as ELAN, based on the threshold definition in Ly$\alpha$
extent and luminosity. To date, many QSO pair systems observed with PCWI/KCWI
have shown extended and intense Ly$\alpha$ emission. We find that ELANe J1613
is possibly powered by AGN outflows, as inferred by the kinematic distribution
of the nebular emission. Also, the circularly-averaged surface brightness of
our targets are slightly lower than that from other studies of nebular
emission at $z\sim 3$. This may imply that there is a lower local density or
gas mass at $z\sim 2$ than at $z\sim 3$. KCWI would grant the ability to study
individual substructures in the ELANe, such as LAEs and bright nodes in
detail.

##  7 Acknowledgements

Resources supporting this work were provided by NSF AAG Grant 1716907 and the
California Institute of Technology. ZC acknowledges National Key R&D Program
of China (grant no. 2018YFA0404503), the National Science Foundation of China
(grant no. 12073014). The science research grants from the China Manned Space
Project with No. CMS-CSST2021-A05.

Palomar

## References

  * Alexander et al. (2010) Alexander, D. M., Swinbank, A. M., Smail, I., McDermid, R., & Nesvadba, N. P. H. 2010, MNRAS, 402, 2211
  * Allen et al. (2008) Allen, M. G., Groves, B. A., Dopita, M. A., Sutherland, R. S., & Kewley, L. J. 2008, ApJS, 178, 20
  * Arrigoni Battaia et al. (2016) Arrigoni Battaia, F., Hennawi, J. F., Cantalupo, S., & Prochaska, J. X. 2016, ApJ, 829, 3
  * Arrigoni Battaia et al. (2015a) Arrigoni Battaia, F., Hennawi, J. F., Prochaska, J. X., & Cantalupo, S. 2015a, ApJ, 809, 163
  * Arrigoni Battaia et al. (2019a) Arrigoni Battaia, F., Hennawi, J. F., Prochaska, J. X., et al. 2019a, MNRAS, 482, 3162
  * Arrigoni Battaia et al. (2018) Arrigoni Battaia, F., Prochaska, J. X., Hennawi, J. F., et al. 2018, MNRAS, 473, 3907
  * Arrigoni Battaia et al. (2015b) Arrigoni Battaia, F., Yang, Y., Hennawi, J. F., et al. 2015b, ApJ, 804, 26
  * Arrigoni Battaia et al. (2019b) Arrigoni Battaia, F., Obreja, A., Prochaska, J. X., et al. 2019b, A&A, 631, A18
  * Bielby et al. (2020) Bielby, R. M., Fumagalli, M., Fossati, M., et al. 2020, MNRAS, 493, 5336
  * Borisova et al. (2016a) Borisova, E., Cantalupo, S., Lilly, S. J., et al. 2016a, ArXiv:1605.01422
  * Borisova et al. (2016b) —. 2016b, ApJ, 831, 39
  * Bradley et al. (2020) Bradley, L., Sipőcz, B., Robitaille, T., et al. 2020, astropy/photutils: 1.0.0, doi:10.5281/zenodo.4044744
  * Cai et al. (2016) Cai, Z., Fan, X., Peirani, S., et al. 2016, ApJ, 833, 135
  * Cai et al. (2017a) Cai, Z., Fan, X., Yang, Y., et al. 2017a, ApJ, 837, 71
  * Cai et al. (2017b) Cai, Z., Fan, X., Bian, F., et al. 2017b, ApJ, 839, 131
  * Cai et al. (2018) Cai, Z., Hamden, E., Matuszewski, M., et al. 2018, ApJ, 861, L3
  * Cai et al. (2019) Cai, Z., Cantalupo, S., Prochaska, J. X., et al. 2019, ApJS, 245, 23
  * Cantalupo (2017) Cantalupo, S. 2017, Astrophysics and Space Science Library, Vol. 430, Gas Accretion and Giant Ly$\alpha$ Nebulae, ed. A. Fox & R. Davé, 195
  * Cantalupo et al. (2014) Cantalupo, S., Arrigoni-Battaia, F., Prochaska, J. X., Hennawi, J. F., & Madau, P. 2014, Nature, 506, 63
  * Cantalupo et al. (2005) Cantalupo, S., Porciani, C., Lilly, S. J., & Miniati, F. 2005, ApJ, 628, 61
  * Cantalupo et al. (2019) Cantalupo, S., Pezzulli, G., Lilly, S. J., et al. 2019, MNRAS, 483, 5188
  * Dey et al. (2005) Dey, A., Bian, C., Soifer, B. T., et al. 2005, ApJ, 629, 654
  * Dijkstra & Loeb (2009) Dijkstra, M., & Loeb, A. 2009, Monthly Notices of the Royal Astronomical Society, 400, 1109
  * Dijkstra & Westra (2010) Dijkstra, M., & Westra, E. 2010, MNRAS, 401, 2343
  * Drake et al. (2019) Drake, A. B., Farina, E. P., Neeleman, M., et al. 2019, ApJ, 881, 131
  * Farina et al. (2019) Farina, E. P., Arrigoni-Battaia, F., Costa, T., et al. 2019, ApJ, 887, 196
  * Fossati et al. (2021) Fossati, M., Fumagalli, M., Lofthouse, E. K., et al. 2021, Monthly Notices of the Royal Astronomical Society, 503, 3044
  * Geach et al. (2016) Geach, J. E., Narayanan, D., Matsuda, Y., et al. 2016, ApJ, 832, 37
  * Haiman & Rees (2001) Haiman, Z., & Rees, M. J. 2001, ApJ, 556, 87
  * Haiman et al. (2000) Haiman, Z., Spaans, M., & Quataert, E. 2000, ApJ, 537, L5
  * Harrison et al. (2014) Harrison, C. M., Alexander, D. M., Mullaney, J. R., & Swinbank, A. M. 2014, MNRAS, 441, 3306
  * Hennawi & Prochaska (2013) Hennawi, J. F., & Prochaska, J. X. 2013, ApJ, 766, 58
  * Hennawi et al. (2015) Hennawi, J. F., Prochaska, J. X., Cantalupo, S., & Arrigoni-Battaia, F. 2015, Science, 348, 779
  * Kennicutt (1998) Kennicutt, Robert C., J. 1998, ApJ, 498, 541
  * Kollmeier et al. (2010) Kollmeier, J. A., Zheng, Z., Davé, R., et al. 2010, ApJ, 708, 1048
  * Lau et al. (2016) Lau, M. W., Prochaska, J. X., & Hennawi, J. F. 2016, ApJS, 226, 25
  * Lusso et al. (2015) Lusso, E., Worseck, G., Hennawi, J. F., et al. 2015, MNRAS, 449, 4204
  * Martin et al. (2014a) Martin, D. C., Chang, D., Matuszewski, M., et al. 2014a, ApJ, 786, 106
  * Martin et al. (2014b) —. 2014b, ApJ, 786, 106
  * Martin et al. (2014c) —. 2014c, ApJ, 786, 107
  * Martin et al. (2015a) Martin, D. C., Matuszewski, M., Morrissey, P., et al. 2015a, Nature, 524, 192
  * Martin et al. (2015b) —. 2015b, Nature, 524, 192
  * Martin et al. (2016) —. 2016, ApJ, 824, L5
  * Martin et al. (2019) Martin, D. C., O’Sullivan, D., Matuszewski, M., et al. 2019, Nature Astronomy, 3, 822
  * Matsuda et al. (2004) Matsuda, Y., Yamada, T., Hayashino, T., et al. 2004, The Astronomical Journal, 128, 569
  * Matuszewski et al. (2010) Matuszewski, M., Chang, D., Crabill, R. M., et al. 2010, in Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series, Vol. 7735, Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series
  * Onoue et al. (2018) Onoue, M., Kashikawa, N., Uchiyama, H., et al. 2018, PASJ, 70, S31
  * O’Sullivan & Chen (2020) O’Sullivan, D., & Chen, Y. 2020, arXiv e-prints, arXiv:2011.05444
  * O’Sullivan et al. (2020a) O’Sullivan, D. B., Martin, C., Matuszewski, M., Hoadley, K., & Hamden, E. 2020a, ApJ, 894, doi:10.3847/1538-4357/ab838c
  * O’Sullivan et al. (2020b) O’Sullivan, D. B., Martin, C., Matuszewski, M., et al. 2020b, ApJ, 894, 3
  * Ouchi et al. (2020) Ouchi, M., Ono, Y., & Shibuya, T. 2020, ARA&A, 58, 617
  * Pâris et al. (2017) Pâris, I., Petitjean, P., Ross, N. P., et al. 2017, A&A, 597, A79
  * Prescott et al. (2013) Prescott, M. K. M., Dey, A., & Jannuzi, B. T. 2013, ApJ, 762, 38
  * Prochaska et al. (2013) Prochaska, J. X., Hennawi, J. F., Lee, K.-G., et al. 2013, The Astrophysical Journal, 776, 136
  * Rees (1988) Rees, M. J. 1988, MNRAS, 231, 91p
  * Rudie et al. (2012) Rudie, G. C., Steidel, C. C., Trainor, R. F., et al. 2012, ApJ, 750, 67
  * Steidel et al. (2000) Steidel, C. C., Adelberger, K. L., Shapley, A. E., et al. 2000, ApJ, 532, 170
  * Steidel et al. (2011) Steidel, C. C., Bogosavljević, M., Shapley, A. E., et al. 2011, ApJ, 736, 160
  * Stewart et al. (2011) Stewart, K. R., Kaufmann, T., Bullock, J. S., et al. 2011, ApJ, 735, L1
  * Taniguchi & Shioya (2000) Taniguchi, Y., & Shioya, Y. 2000, ApJ, 532, L13
  * Trainor et al. (2016) Trainor, R. F., Strom, A. L., Steidel, C. C., & Rudie, G. C. 2016, ApJ, 832, 171
  * Tumlinson et al. (2013) Tumlinson, J., Thom, C., Werk, J. K., et al. 2013, ApJ, 777, 59
  * Wisotzki et al. (2016) Wisotzki, L., Bacon, R., Blaizot, J., et al. 2016, A&A, 587, A98
  * Wright (2006) Wright, E. L. 2006, PASP, 118, 1711
  * Yang et al. (2009) Yang, Y., Zabludoff, A., Tremonti, C., Eisenstein, D., & Davé, R. 2009, ApJ, 693, 1579
  * Zafar et al. (2011) Zafar, T., Møller, P., Ledoux, C., et al. 2011, A&A, 532, A51


# Cooperation on the Fly: Exploring Language Agents for Ad Hoc Teamwork in the
Avalon Game

Zijing Shi1, Meng Fang2, Shunfeng Zheng1, Shilong Deng2, Ling Chen1, Yali Du3
1 AAII, University of Technology Sydney, Australia
{zijing.shi<EMAIL_ADDRESS><EMAIL_ADDRESS>
2 University of Liverpool, United Kingdom
{Meng.Fang<EMAIL_ADDRESS>
3 King’s College London, United Kingdom
<EMAIL_ADDRESS>
###### Abstract

Multi-agent collaboration with Large Language Models (LLMs) demonstrates
proficiency in basic tasks, yet its efficiency in more complex scenarios
remains unexplored. In gaming environments, these agents often face situations
without established coordination protocols, requiring them to make intelligent
inferences about teammates from limited data. This problem motivates the area
of ad hoc teamwork, in which an agent may potentially cooperate with a variety
of teammates to achieve a shared goal. Our study focuses on the ad hoc
teamwork problem where the agent operates in an environment driven by natural
language. Our findings reveal the potential of LLM agents in team
collaboration, highlighting issues related to hallucinations in communication.
To address this issue, we develop CodeAct, a general agent that equips LLM
with enhanced memory and code-driven reasoning, enabling the repurposing of
partial information for rapid adaptation to new teammates.

Cooperation on the Fly: Exploring Language Agents for Ad Hoc Teamwork in the
Avalon Game

Zijing Shi1, Meng Fang2, Shunfeng Zheng1, Shilong Deng2, Ling Chen1, Yali Du3
1 AAII, University of Technology Sydney, Australia {zijing.shi,
<EMAIL_ADDRESS><EMAIL_ADDRESS>2 University of
Liverpool, United Kingdom {Meng.Fang<EMAIL_ADDRESS>3 King’s
College London, United Kingdom<EMAIL_ADDRESS>

##  1 Introduction

Large Language Models (LLMs) have exhibited impressive abilities in reasoning
and generalization, showing their potential in building autonomous agents and
driving many advancements Kojima et al. (2022); Xi et al. (2023). Recent
research involving multiple LLM agents working together has shown success in
task execution Wang et al. (2023c); Wu et al. (2023); Dong et al. (2023).
However, these studies often employ explicit coordination protocols for fully
cooperative tasks, assuming full information of the teammate and the task.
While some research has explored simulating LLM agents in more open-ended
settings, these agents remain limited by pre-established strategies and
behaviors Park et al. (2023).

In dynamic environments, maintaining consistent coordination protocols among
agents is a challenge. Such contexts demand an agent capable of swift
adaptation and on-the-fly cooperation. Without pre-established team
strategies, an agent, often referred to as the learner, must observe its
teammates’ behaviors and adapt effectively to achieve their shared goals. This
challenge is known as the Ad Hoc Teamwork (AHT) problem Stone et al. (2010);
Macke et al. (2021).

A significant challenge for the learner is to infer the role of their
teammates. Traditional methods rely on pre-programmed rules or data-driven
algorithms trained on historical data Mirsky et al. (2022). In contrast, LLM
agents can directly communicate with their teammates in natural language.
However, the communication dynamics within the environment may exhibit
conflicting and competitive elements. Moreover, due to the inherent traits of
LLMs, teammates may provide inaccurate or misleading information.

In this study, we explore the AHT abilities of LLM agents, utilizing the
multi-agent game The Resistance: Avalon to simulate AHT environments. We
introduce the AvalonPlay benchmark, which challenges agents with hidden roles
to participate in strategic gameplay across multiple rounds. In each round, a
learner is tasked with selecting teammates without prior strategies and full
information. Our research integrates LLM agents into AvalonPlay to evaluate
their performance in AHT. Rather than developing coordination protocols for
the entire team, we prioritize designing agents adept at cooperating in
environments without such explicit protocols.

Figure 1: A flowchart of the AvalonPlay benchmark showing team sides and roles
on the left and a detailed round pipeline on the right. Each round includes
four stages: leader assignment, team selection, discussion and voting, quest
execution.

A communication channel is designed within the AvalonPlay benchmark to enhance
collaboration. We observe that LLM agents may sometimes produce hallucinations
and forget early information. To address this, we introduce CodeAct, a general
agent leveraging the advanced reasoning capabilities of LLMs, focusing on
reasoning about various potential teammate types. We begin with a memory
retrieval system that collects information and utilize the knowledge gained
from these interactions. Subsequently, we incorporate code-driven reasoning
with action to effectively refer the roles of new teammates. With the teammate
role information, our agent is well-equipped to effectively cooperate with its
teammates.

In summary, our contributions are as follows:

  * •

We explore the AHT capabilities of LLM agents, utilizing the AvalonPlay
benchmark with its multi-round collaborative tasks as a testing ground for
these abilities.

  * •

We develop CodeAct, an LLM agent utilizing a code-driven prompting approach,
aimed at boosting the agent’s efficiency in AHT tasks.

  * •

We conduct comprehensive experiments within the AvalonPlay benchmark,
demonstrating the effectiveness of our proposed agent.

##  2 The AvalonPlay Benchmark

We introduce the AvalonPlay benchmark to explore the performance of LLM agents
on AHT. This benchmark is a language-based, multi-agent platform comprised of
multi-round tasks. During the gameplay, agents have limited knowledge about
each other’s roles. In each round, a team leader, also regarded as the
learner, is selected to deduce the roles and intentions of other agents. The
leader must adapt their team’s strategies dynamically as the game evolves.
This section details the key elements of the proposed benchmark.

###  2.1 Teammate Role

We utilize a setup involving seven players indexed by $j$, each controlled by
an LLM agent. In each game, agents are randomly assigned one of six distinct
roles, dividing them into two factions: the just side (including Merlin,
Percival, and two Loyal Servants of Arthur) and the evil side (including
Morgana, Minion, and Assassin).

Each role possesses unique abilities, as shown on the left of Figure 1. On the
just side, Merlin is aware of the factions but not the specific roles, while
Percival can recognize Merlin and Morgana but is unable to distinguish between
them. The Loyal Servants of Arthur lack any special knowledge. On the evil
side, in contrast, all roles can recognize their teammates. This setup puts
the just side at a disadvantage due to their limited information. Our study
focuses on scenarios where the just side takes the lead role, aiming to
enhance the leader’s ability to deduce the various roles of agents.

Figure 2: An overview of the proposed CodeAct agent as the leader during team
selection. We begins by establishing a memory retrieval system that distills
information from past interactions, enabling the agent to access relevant
information. Then, we integrate code-driven reasoning with action to determine
teammate roles effectively. Finally, we employ an interpreter to execute the
generated code, equipping the agent with self-debug capabilities.

###  2.2 Pipeline

In AvalonPlay, a single game is composed of five quest rounds indexed by $i$.
Each round, defined as a full quest cycle, is divided into the following four
phases, as shown in the right of Figure 1.

  * •

Leader Assignment: Each round begins with the selection of a leader who
nominates teammates for the quest. The first round’s leader is randomly
chosen, and in subsequent rounds, leader rotates sequentially to the next
player. This approach ensures diverse leader roles while incorporating
randomness.

  * •

Team Selection: In this phase, a leader nominates $n_{i}$ teammates, with the
team size $n_{i}$ varying per round. Adhering to the game rule, the team sizes
are: $n_{1}=2,n_{2}=3,n_{3}=3,n_{4}=4,n_{5}=4$. We align the leader’s response
into corresponding actions. For instance, the response “I choose myself and
player 2 for the quest team” is mapped to the specific selection of teammates.

  * •

Discussion and Voting: We design a communication protocol that enables the
agent to acquire information while accounting for the potential of
misinformation. Post team selection, agents participate in discussions to
assess the proposed team. Following the discussion, each agent secretly votes
$\mathbf{1}_{j}$ indicates agent $j$’s approval of the current nominated team.
To enhance agents’ decision-making, they are required to generate a JSON
containing their reasoning and vote. A team is approved if the total votes
reaches a majority (i.e., $\sum_{j=1}^{7}\mathbf{1}_{j}\geq 4$). Otherwise,
the team selection is repeated with a new leader. A team is automatically
formed if approval is not achieved after five consecutive attempts.

  * •

Quest Execution: For an approved team, teammates from the just side must vote
to approve, while teammates from the evil side must vote against. Let $Q_{i}$
denote the total votes of evil members. The quest is sabotaged if $Q_{i}\geq
1$ for rounds $i\in\\{1,2,3\\}$, or $Q_{i}\geq 2$ for rounds $i\in\\{4,5\\}$.
In other cases, it is completed successfully.

We adhere to game rule by including an assassination phase in the Avalonplay
benchmark, where the Assassin identify Merlin after all rounds. However, our
study aims to assess the teamwork of LLM agents, not the accuracy of
assassinations, though this feature is included for prospective future
research.

###  2.3 Observation Understanding

The AvalonPlay benchmark includes a moderator responsible for overseeing the
game’s progress. This involves providing action descriptions to agents and
recording events that occur during gameplay. The moderator’s role is non-
interactive with the LLMs but adheres to predefined procedures. The
information generated by the moderator is a vital part of the agents’
observation.

We develop structured prompts to assist LLM agents in comprehending both the
game’s rules and its current state. In general, an LLM agent’s observation is
divided into four key variables to navigate the game:

  * •

Game Rule: We begin with a basic understanding of the game (e.g., “You are
playing the Avalon game with other players...”).

  * •

Role Assignment: For each player, we randomly assign their roles and create a
unique role assignment prompt (e.g., Merlin is informed: “You are
Merlin...Your objective is to ensure the success of the quests as much as
possible”).

  * •

Role Identification: Roles with special abilities get extra information (e.g.,
Merlin receives a moderator’s note: “Players 1, 2, and 6 belong to the evil
side”).

  * •

Game History: This includes the game trajectory from previous rounds, as well
as earlier time steps of the current round.

  * •

Action Description: This includes a description of the next possible actions
for the agent, provided by the moderator (e.g., Player 2 receives a
moderator’s note: “Now it’s your turn, player 2. Please discuss whether to
agree with the nominated team”).

The game rule, role assignment, and role identification collectively
constitute the global prompt, consistently employed at each time step. Game
history and action description serve as real-time prompt, effectively
capturing the game’s present state. For additional details, please refer to
Appendix A.

##  3 Methodology

###  3.1 Overview

Reasoning about various potential teammate types is a key issue in AHT
research. To improve the agent’s reasoning abilities, we introduce the CodeAct
agent, which utilizes the code-based reasoning capabilities of LLMs. Our
CodeAct agent does not focus on creating prompts or demonstrations for
specific tasks. Instead, it emphasizes a general strategy that transforms
complex semantic tasks into flexible code structures.

Our approach unfolds in three stages, as shown in Figure 2. Firstly, we
establish a memory retrieval system to extract information from previous
interactions. Then, we integrate the code-driven reasoning with action to
reason teammate roles. Lastly, we employ an interpreter to execute the code
and equip the agent with the self-debug capability.

###  3.2 Memory Retrieval

During game playing, we implement global memory for each LLM agent to prevent
the over-extension of token inputs. Particularly during the team selection
phase, the team leader is equipped with a customized memory module,
strategically designed to aid in deducing the roles of its teammates.

#### Global Memory

The global memory for each agent comprises identified key information along
with messages from the most recent $k$ steps. This key information includes:
1) Notes from the game moderator; 2) Contributions from agents during
discussions, which reveal details of personal identities or deduce the
identities of others.

#### Leader-Specific Memory

LLM agents often base their reasoning on the latest information and frequently
experience hallucinations, as detailed in Section 4.5. In response, for each
game round’s team selection stage, we systematically retrieve factual data
gathered from earlier interactions to aid the current leader. This includes:
1) Private Information, derived from unique skills tied to an agent’s role,
like Percival’s insight: “Among players 1 and 4, one is good and the other
evil, but you can’t discern who is who.” 2) Public Information, pertaining to
historical game outcomes and events, accessible to all agents, such as: “In
the initial round, players 1 and 2 were selected for the quest, which ended in
failure.”

⬇

You are engaged in a game with 7 players, each potentially categorized as
’good’ or ’bad’. There are 4 good players and 3 bad players among them.

You have the fllowing information.

{PRIVATE_INFORMATION}

{PUBLIC_INFORMATION}

Please generate a code program that selects {self._current_team_number}
players most likely to be good person. Please print the final 4 players.’’’})

Figure 3: An example of the CodeAct agent’s prompt.

⬇

Action: Python_REPL

Action Output:

players = [None] * 7 # Initialize the players list with None

# Apply Round 2 information

players[1] = True # Player 2 is good

players[3] = True # Player 4 is good

players[6] = True # Player 7 is good

# Apply Round 3 information

# Since Player 2 and Player 4 are good, Player 1 must be bad

players[0] = False # Player 1 is bad

# Apply Round 4 information

# Since Player 2 and Player 7 are good, and we know Player 1 is bad, Player 3
must be good

players[2] = True # Player 3 is good

# This leaves Player 5 as the only possible bad player from this group

players[4] = False # Player 5 is bad

# We already have 2 bad players (Player 1 and Player 5), so Player 6 must be
good

players[5] = True # Player 6 is good

# Print the final 4 players most likely to be good

good_players = [index + 1 for index, is_good in enumerate(players) if is_good]

print(good_players)

Observation: [2, 3, 4, 6, 7]

Final Answer: The teammates are Players 2, 3, 4, and 6.

Figure 4: An example of the CodeAct agent’s output.

###  3.3 Code-driven Reasoning with Action

In the team selection phase, we utilize a customized memory system, allowing
the current leader to deduce the roles of its teammates. Drawing inspiration
from ReAct Yao et al. (2022), our agent is designed for interactive generation
of both thoughts and actions through a program in a zero-shot manner. We
utilize a program that processes useful information by reasoning over the
agent’s current private and public data to support future actions. As
illustrated in Figure 3, we further enhance this process by enabling the
leader to generate reasoning substeps in a code-like format. In this study, we
focus on Python, but our approach is applicable to any programming language.
Figure 4 provide an excerpt generated by our CodeAct, with more details
available in Appendix B.

Our code-driven approach, on the one hand, leverages advantages from semantic
reasoning that involve intermediate steps. On the other hand, it leverages
code-based reasoning, enabling reliable inference through the formal yet
expressive code structure and the powerful computational tools it offers.

(a) Game Win Rate by Quest

(b) Total Quest Win Rate

(c) Team Selection Accuracy

Figure 5: The results of different models acting as the good and evil sides,
playing games against each other. Under each setting, 30 games were conducted,
totaling 150 quests.

(a) GPT-3.5 as Good Side

(b) GPT-4 as Good Side

(c) LLaMA2 as Good Side

Figure 6: The turn-by-turn quest win rate results when different models assume
the role of the good side. Each model, as the good side, played 90 games,
resulting in 450 quests evenly distributed across 5 turns.

###  3.4 Code Execution with Self-Debug

We equip the leader with the reflection mechanism Shinn et al. (2023); Chen et
al. (2023b), allowing them to refine their programs. The code-based action,
once formulated, is then processed by a code interpreter. In this study, we
equip the agent with a Read-Eval-Print-Loop Python interpreter (i.e.,
Python_REPL). If the code executes successfully, the agent’s decision is
informed by the resulting program output. Conversely, if execution falters,
the agent will revise the code, taking into account the issue encountered and
previous outputs to enhance its generation.

##  4 Experiments

Our experiments aim to explore the following questions: 1) The AHT
capabilities of agents employing different LLMs as backend models; 2) The
effect of a natural language-based communication protocol on the AHT
capabilities of LLM agents; 3) The impact of diverse prompting strategies on
the AHT capabilities of LLM agents. We compare the proposed CodeAct agent with
other semantic-based reasoning approaches.

###  4.1 Baselines

To tackle question 1, we conduct a preliminary evaluation of three advanced
LLMs within the multi-agent AvalonPlay benchmark. These models include
GPT-3.5, GPT-4, and LLaMA 2. Specifically, we utilize the versions
gpt-3.5-turbo-0613, gpt-4-1106-preview, and Llama2-chat-13B respectively.
These LLM agents, equipped with different backend LLMs, are deployed on both
the good and evil sides to compete against one another.

To address question 2, we compare scenarios in AvalonPlay both with and
without communication protocols. In scenarios lacking communication protocols,
all players proceed to a secret vote immediately following the team leader’s
selection of the team.

To tackle question 3, we focus on the team selection by the leader, compare
semantic reasoning approaches (i.e., Chain of Thought Wei et al. (2022) and
ReAct Yao et al. (2022)) with our proposed CodeAct agent.

###  4.2 Metrics

We design the following three metrics to measure an LLM agent’s performance in
the AvalonPlay benchmark.

  * •

Game Win Rate by Quest (Game Win), which refers to the percentage of games
where the good side wins by completing three out of five quests.

  * •

Total Quest Win Rate (Quest Win), which refers to the percentage of quests
where the good side wins.

  * •

Team Selection Accuracy (Team Acc), which refers to the percentage of times
the just side, when in the role of leader, selects the team for a successful
quest.

###  4.3 Implementation Details

For all LLM agent, we set the sampling temperature as 0.7 for text generation
and 0 for code generation. We set the window size $k$ to be 15 for the global
memory retrieval. During gameplay, all roles are randomly assigned to each
player to avoid biasing the data towards certain players based on the turn
order.

###  4.4 Main Results

#### Question 1: Model Performance.

Figure 5 displays the performance of agents with three different backend LLMs,
playing on different sides. Figure 6 shows the quest success rates of these
LLM agents as the good side across different rounds. Our findings indicate
that 1) More advanced LLMs achieve higher win rates both in games and in
overall tasks when playing as the good side, with GPT-4 agents exhibiting the
best performance. 2) Notably, GPT-4 agents demonstrate higher quest success
rates in later rounds compared to earlier ones, indicating their ability to
reason from observations. In contrast, GPT-3.5 and LLaMA agents do not show
significant progress across rounds. This suggests that the GPT-4 agent is more
adept at learning and adapting, leading us to utilize the GPT-4 model in
subsequent experiments.

The 5th Round Team Selection and Discussion
---
Moderator: |  Player 3, as the team leader, please select 4 team members to undertake the quest. You may also choose yourself.
Player 3: |  I choose player 2, player 3, player 6, and player 7 for the quest. (Team selection by the current leader)
Moderator: |  The selected team by the leader include [’player 2’, ’player 3’, ’player 6’, ’player 7’]. Now everyone discuss if you agree the team.
Player 1: |  This team includes players who have been part of successful quests before. It’s a logical choice to maintain a strong team dynamic. I agree with the team selection.
Player 2: |  Player 3, player 6, and player 7 were on the last successful quest with me, and I believe this consistency will help us ensure another victory. I’m in favor of this team. (In the previous round, players 2, 3, 6, and 7 executed the quest together and succeeded. Player 2 effectively used this information for reasoning)
Player 3: |  I’ve chosen players who have either been part of successful quests or have not shown any suspicious behavior so far. It’s important to maintain a strong team dynamic, and I believe this selection will lead us to victory.
Player 4: |  The team seems solid with previous successful quest members. Including player 6 could either solidify their trustworthiness or provide us with more information. I agree to proceed.
Player 5: |  The inclusion of player 6, who hasn’t been on a quest recently, along with proven successful quest members, provides a chance to validate their loyalty. We should be wary, but I am inclined to give this team a chance. (Player 5, however, produced hallucination, with agent’s statement being inconsistent with the facts shared by everyone)
Player 6: |  The team includes players who have consistently been part of successful quests, and as a trusted member myself, I feel confident we can achieve success once more. I agree with this selection and am prepared to contribute to the quest’s success.
Player 7: |  The team selected by player 3 includes myself and other players who have been part of successful quests, which instills confidence in this group’s potential for success. Player 6’s inclusion is a variable, but it is essential to evaluate all players. Based on the track record of success we’ve had, I agree with this team selection and am optimistic about our prospects.

Table 1: An example of the GPT-4 agent’s participation in discussions during
gameplay. In the fifth round, the agent’s response is based on previous
observations and demonstrates some reasoning ability. However, the agent
utilizes only limited information and is prone to experiencing hallucinations.

#### Question 2: Communication Channel.

Table 2 compares the GPT-4 agent’s performance in self-playing, both with and
without communication. The introduction of communication does not
significantly enhance the game performance of LLM agents. This is possibly due
to the fact that, besides the communication protocol, LLM agents also have
access to observe and reason based on the outcomes of quests in previous
rounds. These findings indicate that communication does not always lead to
improved collaborative efficiency in LLM agents. We further explore this
phenomenon in Section 4.5.

Table 2: Results from the GPT-4 agent self-playing in the AvalonPlay
benchmark, both with and without communication. Under each setting, 30 games
were conducted, totaling 150 quests.

Agent/ Metric | Game Win | Quest Win | Team Acc
---|---|---|---
Base | 0.500 | 0.513 | 0.640
Base W/O Communication | 0.466 | 0.480 | 0.693

#### Question 3: Code-driven Reasoning.

We campare the proposed CodeAct with two semantic reasoning methods: Chain of
Thought (CoT) and ReAct. Table 3 shows that our proposed CodeAgent achieves
the best performance across three metrics, with a team selection accuracy of
0.830, demonstrating the superiority of code-based reasoning.

Table 3: Results of the GPT-4 agent self-playing in the AvalonPlay benchmark
using different prompting strageties. Under each setting, 30 games were
conducted, totaling 150 quests.

Agent/ Metric | Game Win | Quest Win | Team Acc
---|---|---|---
Base with CoT | 0.533 | 0.547 | 0.707
Base with ReAct | 0.500 | 0.526 | 0.634
CodeAct | 0.633 | 0.593 | 0.830

###  4.5 Quantitative Analysis

In our preliminary analysis of the experimental logs targeting question 1, we
observe that employing LLM agents in multi-turn tasks often results in memory
forgetting. In cooperative tasks involving communication, these agents tend to
exhibit cascading hallucinations and logical inconsistencies. We present Table
1 for a more detailed illustration.

#### Forgetting Early Information.

Players typically focus on information from the last round, failing to
integrate key information from the early stages of the game into their later
decisions and strategies.

#### Hallucination Generation.

In the game, statements that contradict a player’s actual role are not
classified as hallucinations, as deception is a common strategy. Similarly,
fabricating scenarios to falsely accuse others is also not considered
hallucinations, as this mirrors the strategies of human players. We define
inconsistencies within a response and counterfactual content in an iteration
as hallucinations. When GPT-3.5 and LLaMa 2 models serve as agents, they tend
to produce responses inconsistent with the prompt. A notable example is their
role as team leaders, where they often select a number of teammates that
diverges from the required amount. Additionally, the final votes they generate
may not correspond with the reasoning process outlined in the previous step.
This problem is less pronounced with the GPT-4 agent during gameplay. However,
all models, including GPT-4, are prone to generating responses that contradict
established facts, such as modifying or denying events from previous game
rounds. Table 1 provides an example of GPT-4 agent’s participation in
discussions. Player 5’s statements are inconsistent with the facts. As player
2 points out, playe 6 had already successfully participated in the quest in
the previous round.

This suggests that the communication protocol does not necessarily offer
optimal solutions. The most important aspect is the effective verification of
the factuality and correctness of communications. This distinguishes it from
prior AHT research with agents trained from historical data Mirsky et al.
(2020). In our study, we address this by utilizing memory retrieval to extract
factual information. Additionally, we enhance the agent’s reasoning abilities
for different types of teammates by integrating code-based reasoning with
actions.

##  5 Related Work

#### LLMs and Agents.

The advent of generative agents for interactive tasks marks a significant
paradigm shift. Unlike traditional methods that relied on Deep Reinforcement
Learning (DRL) Shi et al. (2022, 2023); Xu et al. (2020, 2021, 2022), which
confined agents to constrained settings, the emergence of LLMs has notably
broadened the scope of possibilities OpenAI (2023). Efforts are increasingly
focused on leveraging LLMs for complex tasks like planning and reasoning Wang
et al. (2023c); Wu et al. (2023); Dong et al. (2023). Innovative prompting
methods, including ReACT Yao et al. (2022), Chain of Thought (CoT) Wei et al.
(2022), and Tree of Thought (ToT) Yao et al. (2023), significantly enhance the
planning and reasoning capabilities of LLMs. Some research integrates complex
human cognitive functions into LLM agents, such as memory Wang et al. (2023b),
reflection Shinn et al. (2023), and tool usage Huang et al. (2023).

#### Multi-agent Interaction.

Recent research has explored that utilizing the interactions among multiple
LLM agents to achieve several goals Chen et al. (2023a); Wu et al. (2023).
These studies focus on two main areas. First, it involves assigning distinct
roles to agents to enhance their task performance Wang et al. (2023a). Notable
applications include using multiple agents in debates to counter self-
reflection biases Liang et al. (2023); Chan et al. (2023), and diversifying
agent roles for complex projects like software development Qian et al. (2023);
Hong et al. (2023). Second, researchers develop multiple LLM agents in sandbox
environments to mimic real human behavior Park et al. (2023); Zhou et al.
(2023). Some of these investigate the capabilities that emerge in multi-agent
settings, such as deception and leadership Xu et al. (2023).

#### Ad Hoc Teamwork.

Ad Hoc teamwork aims to generalise the agent’s cooperation capacity with novel
agents Stone et al. (2010); Du et al. (2023) The progress in AHT has shifted
from relying on preset action-selection rules to a data-driven strategy. This
strategy employs methods like probabilistic models Rahman et al. (2023), deep
learning Chen et al. (2020), or reinforcement learning Rahman et al. (2021);
Lou et al. (2023); Yan et al. (2023) to predict the behaviors of various
agents in different states, aiming to optimize and refine the learner’s
behavior. Recent research has also explored the communication in AHT,
considering scenarios where teammates follow a shared communication protocol
Macke et al. (2021); Mirsky et al. (2020). However, these studies are often
confined to a limited multi-agent setting. Our work stands apart from previous
studies by focusing on dynamic environments based on natural language
communication. We employ agents powered by LLMs to execute AHT, and explore
ways to improve their performance in these settings.

##  6 Conclusion and Future Work

In this study, we investigate the AHT capabilities of LLM agents using the
AvalonPlay benchmark. LLMs endow agents with powerful reasoning abilities, but
challenges such as early memory forgetting and hallucination generation also
arise. We address these challenges by proposing the CodeAct agent. We first
employ a memory retrieval system to collect factual information from prior
interactions. Subsequently, we integrate a code-driven reasoning with action
to discern new teammate roles, enhancing our agent’s ability to cooperate more
effectively. Lastly, we employ an interpreter to execute the code and equip
the agent with the self-debug capability.

Despite our initial research, there are some unresolved issues in the AHT
research with language agents that require further investigation. For intance,
our AvalonPlay benchmark involves moderator-guided communication for agents,
making it crucial to investigate the agent’s ability to autonomously decide
when to communicate. Furthermore, significant work lies in reducing the
effects of hallucinations and enhancing their practical application in real-
world scenarios. In this study, we employ a memory retrieval system for
factual information extraction, and another promising research direction
involves developing systems for evaluating and verifying facts to address
these challenges.

##  7 Limitations

Regarding limitations, our study does not incorporate experience pools from
human players. Future research will focus on developing robust strategies for
leveraging this experience and enhancing comparison with human performance.

## References

  * Chan et al. (2023) Chi-Min Chan, Weize Chen, Yusheng Su, Jianxuan Yu, Wei Xue, Shanghang Zhang, Jie Fu, and Zhiyuan Liu. 2023.  Chateval: Towards better llm-based evaluators through multi-agent debate.  _arXiv preprint arXiv:2308.07201_.
  * Chen et al. (2020) Shuo Chen, Ewa Andrejczuk, Zhiguang Cao, and Jie Zhang. 2020.  Aateam: Achieving the ad hoc teamwork by employing the attention mechanism.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , pages 7095–7102.
  * Chen et al. (2023a) Weize Chen, Yusheng Su, Jingwei Zuo, Cheng Yang, Chenfei Yuan, Chen Qian, Chi-Min Chan, Yujia Qin, Yaxi Lu, Ruobing Xie, et al. 2023a.  Agentverse: Facilitating multi-agent collaboration and exploring emergent behaviors in agents.  _arXiv preprint arXiv:2308.10848_.
  * Chen et al. (2023b) Xinyun Chen, Maxwell Lin, Nathanael Schärli, and Denny Zhou. 2023b.  Teaching large language models to self-debug.  _arXiv preprint arXiv:2304.05128_.
  * Dong et al. (2023) Yihong Dong, Xue Jiang, Zhi Jin, and Ge Li. 2023.  Self-collaboration code generation via chatgpt.  _arXiv preprint arXiv:2304.07590_.
  * Du et al. (2023) Yali Du, Joel Z Leibo, Usman Islam, Richard Willis, and Peter Sunehag. 2023.  A review of cooperation in multi-agent learning.  _arXiv preprint arXiv:2312.05162_.
  * Hong et al. (2023) Sirui Hong, Xiawu Zheng, Jonathan Chen, Yuheng Cheng, Jinlin Wang, Ceyao Zhang, Zili Wang, Steven Ka Shing Yau, Zijuan Lin, Liyang Zhou, et al. 2023.  Metagpt: Meta programming for multi-agent collaborative framework.  _arXiv preprint arXiv:2308.00352_.
  * Huang et al. (2023) Yue Huang, Jiawen Shi, Yuan Li, Chenrui Fan, Siyuan Wu, Qihui Zhang, Yixin Liu, Pan Zhou, Yao Wan, Neil Zhenqiang Gong, et al. 2023.  Metatool benchmark for large language models: Deciding whether to use tools and which to use.  _arXiv preprint arXiv:2310.03128_.
  * Kojima et al. (2022) Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022.  Large language models are zero-shot reasoners.  _Advances in neural information processing systems_ , 35:22199–22213.
  * Liang et al. (2023) Tian Liang, Zhiwei He, Wenxiang Jiao, Xing Wang, Yan Wang, Rui Wang, Yujiu Yang, Zhaopeng Tu, and Shuming Shi. 2023.  Encouraging divergent thinking in large language models through multi-agent debate.  _arXiv preprint arXiv:2305.19118_.
  * Lou et al. (2023) Xingzhou Lou, Jiaxian Guo, Junge Zhang, Jun Wang, Kaiqi Huang, and Yali Du. 2023.  Pecan: Leveraging policy ensemble for context-aware zero-shot human-ai coordination.  In _Proceedings of the 2023 International Conference on Autonomous Agents and Multiagent Systems(AAMAS)_ , pages 679–688.
  * Macke et al. (2021) William Macke, Reuth Mirsky, and Peter Stone. 2021.  Expected value of communication for planning in ad hoc teamwork.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , pages 11290–11298.
  * Mirsky et al. (2022) Reuth Mirsky, Ignacio Carlucho, Arrasy Rahman, Elliot Fosong, William Macke, Mohan Sridharan, Peter Stone, and Stefano V Albrecht. 2022.  A survey of ad hoc teamwork: Definitions, methods, and open problems.  In _European Conference on Multiagent Systems_.
  * Mirsky et al. (2020) Reuth Mirsky, William Macke, Andy Wang, Harel Yedidsion, and Peter Stone. 2020.  A penny for your thoughts: The value of communication in ad hoc teamwork.  _Good Systems-Published Research_.
  * OpenAI (2023) R OpenAI. 2023.  Gpt-4 technical report.
  * Park et al. (2023) Joon Sung Park, Joseph C O’Brien, Carrie J Cai, Meredith Ringel Morris, Percy Liang, and Michael S Bernstein. 2023.  Generative agents: Interactive simulacra of human behavior.  _arXiv preprint arXiv:2304.03442_.
  * Qian et al. (2023) Chen Qian, Xin Cong, Cheng Yang, Weize Chen, Yusheng Su, Juyuan Xu, Zhiyuan Liu, and Maosong Sun. 2023.  Communicative agents for software development.  _arXiv preprint arXiv:2307.07924_.
  * Rahman et al. (2023) Arrasy Rahman, Elliot Fosong, Ignacio Carlucho, and Stefano V Albrecht. 2023.  Generating teammates for training robust ad hoc teamwork agents via best-response diversity.  _Transactions on Machine Learning Research_.
  * Rahman et al. (2021) Muhammad A Rahman, Niklas Hopner, Filippos Christianos, and Stefano V Albrecht. 2021.  Towards open ad hoc teamwork using graph-based policy learning.  In _International Conference on Machine Learning_ , pages 8776–8786. PMLR.
  * Shi et al. (2022) Zijing Shi, Meng Fang, Yunqiu Xu, Ling Chen, and Yali Du. 2022.  Stay moral and explore: Learn to behave morally in text-based games.  In _The Eleventh International Conference on Learning Representations_.
  * Shi et al. (2023) Zijing Shi, Yunqiu Xu, Meng Fang, and Ling Chen. 2023.  Self-imitation learning for action generation in text-based games.  In _Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics_ , pages 703–726.
  * Shinn et al. (2023) Noah Shinn, Beck Labash, and Ashwin Gopinath. 2023.  Reflexion: an autonomous agent with dynamic memory and self-reflection.  _arXiv preprint arXiv:2303.11366_.
  * Stone et al. (2010) Peter Stone, Gal Kaminka, Sarit Kraus, and Jeffrey Rosenschein. 2010.  Ad hoc autonomous agent teams: Collaboration without pre-coordination.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , pages 1504–1509.
  * Wang et al. (2023a) Lei Wang, Chen Ma, Xueyang Feng, Zeyu Zhang, Hao Yang, Jingsen Zhang, Zhiyuan Chen, Jiakai Tang, Xu Chen, Yankai Lin, et al. 2023a.  A survey on large language model based autonomous agents.  _arXiv preprint arXiv:2308.11432_.
  * Wang et al. (2023b) Weizhi Wang, Li Dong, Hao Cheng, Xiaodong Liu, Xifeng Yan, Jianfeng Gao, and Furu Wei. 2023b.  Augmenting language models with long-term memory.  _arXiv preprint arXiv:2306.07174_.
  * Wang et al. (2023c) Zhenhailong Wang, Shaoguang Mao, Wenshan Wu, Tao Ge, Furu Wei, and Heng Ji. 2023c.  Unleashing cognitive synergy in large language models: A task-solving agent through multi-persona self-collaboration.  _arXiv preprint arXiv:2307.05300_.
  * Wei et al. (2022) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022.  Chain-of-thought prompting elicits reasoning in large language models.  _Advances in Neural Information Processing Systems_ , 35:24824–24837.
  * Wu et al. (2023) Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Shaokun Zhang, Erkang Zhu, Beibin Li, Li Jiang, Xiaoyun Zhang, and Chi Wang. 2023.  Autogen: Enabling next-gen llm applications via multi-agent conversation framework.  _arXiv preprint arXiv:2308.08155_.
  * Xi et al. (2023) Zhiheng Xi, Wenxiang Chen, Xin Guo, Wei He, Yiwen Ding, Boyang Hong, Ming Zhang, Junzhe Wang, Senjie Jin, Enyu Zhou, et al. 2023.  The rise and potential of large language model based agents: A survey.  _arXiv preprint arXiv:2309.07864_.
  * Xu et al. (2021) Yunqiu Xu, Meng Fang, Ling Chen, Yali Du, and Chengqi Zhang. 2021.  Generalization in text-based games via hierarchical reinforcement learning.  In _Findings of the Association for Computational Linguistics: EMNLP 2021_.
  * Xu et al. (2022) Yunqiu Xu, Meng Fang, Ling Chen, Yali Du, Joey Zhou, and Chengqi Zhang. 2022.  Perceiving the world: Question-guided reinforcement learning for text-based games.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_.
  * Xu et al. (2020) Yunqiu Xu, Meng Fang, Ling Chen, Yali Du, Joey Tianyi Zhou, and Chengqi Zhang. 2020.  Deep reinforcement learning with stacked hierarchical attention for text-based games.  _Advances in Neural Information Processing Systems_ , 33:16495–16507.
  * Xu et al. (2023) Yuzhuang Xu, Shuo Wang, Peng Li, Fuwen Luo, Xiaolong Wang, Weidong Liu, and Yang Liu. 2023.  Exploring large language models for communication games: An empirical study on werewolf.  _arXiv preprint arXiv:2309.04658_.
  * Yan et al. (2023) Xue Yan, Jiaxian Guo, Xingzhou Lou, Jun Wang, Haifeng Zhang, and Yali Du. 2023.  An efficient end-to-end training approach for zero-shot human-ai coordination.  In _Thirty-seventh Conference on Neural Information Processing Systems_.
  * Yao et al. (2023) Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L Griffiths, Yuan Cao, and Karthik Narasimhan. 2023.  Tree of thoughts: Deliberate problem solving with large language models.  _arXiv preprint arXiv:2305.10601_.
  * Yao et al. (2022) Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2022.  React: Synergizing reasoning and acting in language models.  _arXiv preprint arXiv:2210.03629_.
  * Zhou et al. (2023) Pei Zhou, Aman Madaan, Srividya Pranavi Potharaju, Aditya Gupta, Kevin R McKee, Ari Holtzman, Jay Pujara, Xiang Ren, Swaroop Mishra, Aida Nematzadeh, et al. 2023.  How far are large language models from agents with theory-of-mind?  _arXiv preprint arXiv:2310.03051_.

##  Appendix A Global Prompt for Agents

Here, we show the global prompt used for the LLM agent in the AvalonPlay
benchmark. We provide the game rule and role assignment as the global prompt.
Figure 7 displays the specific game rule for all agents, while Table 4 shows
the role assignments for different agents.

⬇

You are playing the Avalon game with some other players. The moderator is also
host, he organised this game and you need to follow his instructions
correctly. Do not talk with the moderator. What the modeler said is
unquestionable and absolutely true. There are six roles in the game. Merlin,
Percival, and Loyal Servants of Arthur form the just side, with members only
knowing their own role and voting to approve in secret ballots to make quests
successful. Morgana, Assassin, and Minion belong to the evil side. They are
privy to their teammates’ identities and vote against in secret ballots to
make the quests fail.

Tips: If you think a player is evil side, try not to let them be selected into
the quest team. If this player is the leader, you can disagree with the
establishment of this team. Reasoning based on facts you have observed and you
cannot perceive information (such as acoustic info) other than text. Do not
pretend you are other players or the moderator.

Figure 7: Game rule prompt for all LLM agents in the AvalonPlay benchmark.

|  Role |  Global Prompt
---|---|---
|  Merlin |  {Game Rule} + You are Merlin, aware of the evil presences, yet you must remain discreet in sharing this knowledge. Should the assassin identify you after the just side completes three quests, the evil side shall emerge victorious. Your objective is to ensure the success of the quests.
|  Percival |  {Game Rule} + As Percival, you recognize Merlin, but be wary as Morgana might mislead you by appearing as Merlin. Your allegiance is with the just side. Your objective is to ensure the success of the quests.
Good |  Servant |  {Game Rule} + You are a Servant of Arthur, with no special abilities, belonging to the just side. Your objective is to ensure the success of the quests.
|  Morgana |  {Game Rule} + You are Morgana, appearing as Merlin to Percival, weaving a web of confusion. Your objective is to ensure the quest fails.
|  Assassin |  {Game Rule} + As the Assassin, your critical role comes into play once the just side successfully executes three quests; identifying Merlin can secure a win for the evil side. Your objective is to ensure the quest fails.
Evil |  Minion |  {Game Rule} + You are a Minion of Mordred, without special abilities, on the evil side. Your objective is to ensure the quest fails.

Table 4: Role assignment prompt for LLM agents in various roles within the
AvalonPlay benchmark.

##  Appendix B An Example of Code-driven Reasoning Generation

In this section, we provide a complete example of code-driven reasoning with
action generation produced by the GPT-4 model, as shown in Figure 8.

⬇

Thought Output:

Let’s start by creating a list of 7 elements to represent the players, with
None as the initial value since we don’t know their alignment yet. We will
then apply the constraints from each round to update this list.

- From Round 2, we know that Players 2, 4, and 7 are good.

- From Round 1, we know that either Player 1 or Player 3 is bad, but not both.

- From Round 3, we know that there is at least one bad person among Player 1, Player 2, and Player 4. Since Players 2 and 4 are good, Player 1 must be bad.

- From Round 4, we know that there is at most one bad person among Player 2, Player 3, Player 5, and Player 7. Since Player 2 and Player 7 are good, and we know Player 1 is bad, Player 3 must be good, leaving Player 5 as the only possible bad player from this group.

After applying these constraints, we should be able to identify the 4 players
most likely to be good.

Let’s write the code to apply these constraints and find the 4 good players.

Action: Python_REPL

Action Output:

players = [None] * 7 # Initialize the players list with None

# Apply Round 2 information

players[1] = True # Player 2 is good

players[3] = True # Player 4 is good

players[6] = True # Player 7 is good

# Apply Round 1 information

# We cannot determine which one is bad yet, so we leave Player 1 and Player 3
as None

# Apply Round 3 information

# Since Player 2 and Player 4 are good, Player 1 must be bad

players[0] = False # Player 1 is bad

# Apply Round 4 information

# Since Player 2 and Player 7 are good, and we know Player 1 is bad, Player 3
must be good

players[2] = True # Player 3 is good

# This leaves Player 5 as the only possible bad player from this group

players[4] = False # Player 5 is bad

# Now we can determine the remaining players’ goodness based on the total
number of bad players

# We already have 2 bad players (Player 1 and Player 5), so Player 6 must be
good

players[5] = True # Player 6 is good

# Print the final 4 players most likely to be good

good_players = [index + 1 for index, is_good in enumerate(players) if is_good]

print(good_players)

Observation: [2, 3, 4, 6, 7]

Thought:I now know the final answer

Final Answer: The 4 players most likely to be good are Players 2, 3, 4, and 6.

Figure 8: An example of the code-driven reasoning with action generation.