# Discovery of a Rare Eclipsing Be/X-ray Binary System, Swift J010902.6-723710
= SXP 182

Thomas M. Gaudin Pennsylvania State University  Jamie A. Kennea Pennsylvania
State University  M.J. Coe Physics and Astronomy, The University of
Southampton, SO17 1BJ, UK  I. M. Monageng South African Astronomical
Observatory, PO Box 9, Observatory, Cape Town 7935, South Africa  Department
of Astronomy, University of Cape Town, Private Bag X3, Rondebosch 7701, South
Africa  Andrzej Udalski Astronomical Observatory, University of Warsaw, Al.
Ujazdowskie 4, 00-478 Warszawa, Poland  L. J. Townsend Southern African Large
Telescope, PO Box 9, Observatory, Cape Town 7935, South Africa  South African
Astronomical Observatory, PO Box 9, Observatory, Cape Town 7935, South Africa
David A.H. Buckley  South African Astronomical Observatory, PO Box 9,
Observatory, Cape Town 7935, South Africa  Southern African Large Telescope,
PO Box 9, Observatory, Cape Town 7935, South Africa  Department of Astronomy,
University of Cape Town, Private Bag X3, Rondebosch 7701, South Africa
Department of Astronomy, University of the Free State, PO Box 339,
Bloemfontein, Cape Town, 9300, South Africa  Phil A. Evans  Leicester
University, UK

###### Abstract

We report on the discovery of Swift J010902.6-723710, a rare eclipsing
Be/X-ray Binary system by the Swift SMC Survey (S-CUBED). Swift
J010902.6-723710 was discovered via weekly S-CUBED monitoring observations
when it was observed to enter a state of X-ray outburst on 10 October 2023.
X-ray emission was found to be modulated by a 182s period. Optical
spectroscopy is used to confirm the presence of a highly-inclined
circumstellar disk surrounding a B0-0.5Ve optical companion. Historical UV and
IR photometry are then used to identify strong eclipse-like features re-
occurring in both light curves with a 60.623 day period, which is adopted as
the orbital period of the system. Eclipsing behavior is found to be the result
of a large accretion disk surrounding the neutron star. Eclipses are produced
when the disk passes in front of the OBe companion, blocking light from both
the stellar surface and circumstellar disk. This is only the third Be/X-ray
Binary to have confirmed eclipses. We note that this rare behavior provides an
important opportunity to constrain the physical parameters of a Be/X-ray
Binary with greater accuracy than is possible in non-eclipsing systems.

††facilities: Swift (XRT and UVOT), OGLE, SALT

##  1 Introduction

Be/X-ray Binaries (BeXRBs) are a type of interacting High Mass X-ray Binary
(HMXB) that contain a main sequence OBe star and a compact object, typically a
neutron star (NS). These binaries are characterized by moderately eccentric
elliptical orbits ($e\sim 0.3$), orbital periods on the order of $\sim$10-1000
days, and the presence of strong emission lines such as the Balmer series
H$\alpha$ line in the optical spectrum of the OBe star (for a comprehensive
review of BeXRBs, see Reig (2011)). The optical emission lines are interpreted
to be a strong indication of a geometrically thin circumstellar disk
(Rivinius, 2019) that surrounds the donor star and is variable in size.
Interactions between the NS and disk are capable of producing intermittent
X-ray outbursts, which are perhaps the most prominent feature of BeXRB
systems.

There are two types of outburst that can be produced by NS interactions with
the circumstellar disk (Stella et al., 1986; Okazaki & Negueruela, 2001; Reig,
2011). Type I outbursts are periodic in nature and associated with NS-disk
interactions that occur at the periastron passage in each orbit(Stella et al.,
1986; Okazaki & Negueruela, 2001). Type II outbursts are much more luminous
than Type I outbursts (Okazaki & Negueruela, 2001; Reig, 2011) and can even
reach super-Eddington luminosities, such as during the SMC X-3 outburst
described by Townsend et al. (2017) and the SMC X-2 outburst that occurred in
2015 (Jaisawal et al., 2023; Roy et al., 2022). These luminous outbursts are
independent of the orbital phase of the NS at the time of outburst and can
last for multiple orbits (Reig, 2011). Due to both the long duration and phase
independence of these events, Type II outbursts are thought to be related to
the growth and shape of the Be star’s disk (Martin et al., 2014; Monageng et
al., 2017) causing it to interact with the NS for a longer time.

Since X-ray outbursts in BeXRBs are dependent on the disk of their donor star,
these systems are prone to experiencing long quiescent states during which
they are hard to detect and identify (Coe et al., 2023b). Swift
J010902.6-723710 is an example of a system that escaped identification due to
a long period of quiescence. This newly-discovered system resides in the Small
Magellanic Cloud (SMC), a satellite galaxy of the Milky Way that is well-
documented to have a large population of HMXBs (Antoniou et al., 2010) due to
an unusually-active period of star formation that occurred in the SMC’s recent
past (Harris & Zaritsky, 2004; Rezaeikh et al., 2014). New SMC BeXRBs are
still occasionally discovered (Maitra et al., 2023), indicating that a hidden
population of previously-undetected systems still exists within the dwarf
galaxy.

In an effort identify X-ray outbursts and detect new sources, the Swift SMC
Survey (S-CUBED) (Kennea et al., 2018) has operated since 2016 to provide
regular X-ray and UV monitoring of the SMC. This survey utilizes the Neil
Gehrels Swift Observatory (Gehrels et al., 2004) to tile the galaxy on a
weekly cadence with 142 overlapping tiles and $\sim$1 minute spent on each
tile. Data recorded by S-CUBED are automatically analyzed to flag new X-ray
outburst events that are detected by Swift’s X-ray Telescope (XRT; Burrows et
al. 2005). S-CUBED also observes all tiles using the Ultraviolet/Optical
Telescope (UVOT; Roming et al. 2005) in the uvw1 band centered at 2600Å. The
archive of X-ray and UV photometric data taken by S-CUBED has proved to be an
invaluable resource for discovering previously-undetected BeXRBs (Monageng et
al., 2020; Kennea et al., 2020; Coe et al., 2021; Kennea et al., 2021;
Monageng et al., 2022). It was through an X-ray selected search of the S-CUBED
archive that Swift J010902.6-723710 was first identified as a candidate source
in the summer of 2023, several months before it entered outburst during
October 2023.

In this paper, we report on the recent X-ray outburst that was experienced by
Swift J010902.6-723710 which confirms its status as a newly-identified BeXRB
system (Coe et al., 2023a). In the following sections, we present the results
of observations made both before and during outburst using Swift, the Optical
Gravitational Lensing Experiment (OGLE), and the Southern African Large
Telescope (SALT). Additionally, all results are combined and analyzed in an
effort to obtain information about the orbital and physical properties of both
the NS and its mass donor OBe companion.

##  2 Observations

###  2.1 Swift - XRT

Figure 1: XRT light curve for Swift J010902.6-723710 from August 2023 to
present, combining data from S-CUBED observations and follow-up TOOs. Note
source visibility to Swift is indicated by the grey background, white vertical
strips indicate periods when the target could not be observed. Vertical dotted
lines mark predicted times for the periastron passages.

Swift J010902.6-723710 has remained in a quiescent state for most of the
duration of S-CUBED monitoring. XRT first detected this source during weekly
S-CUBED monitoring in March of 2020. Two more detections occurred in July and
September of 2021. This mini-flare of X-ray photons went un-reported due to
its sparse detection frequency. The X-ray luminosity of the flare peaked at an
XRT count rate of $0.071_{-0.035}^{+0.051}$ counts s-1 before returning to a
quiescent state. The peak XRT count rate implies an approximate 0.3-10 keV
band luminosity of $L_{X}=4.76\times 10^{36}$ erg s-1 if the standard distance
of 62.44 kpc (Graczyk et al., 2020) to the SMC is assumed.

The source was identified as a possible candidate BeXRB by examining the
optical/IR SEDs of all unidentified X-ray sources in the S-CUBED survey
(Gaudin et al. in prep.). This identification motivated a deep 10ks Swift
observation in August of 2023 to try and constrain the hardness ratio of the
source’s X-ray spectrum. However, the source was only weakly detected with a
mean luminosity of $L_{X}=9.07\times 10^{35}$ erg s-1.

The results of all XRT observations during the recent outburst are shown in
Figure 1. Weekly S-CUBED monitoring detected X-ray emission from the source as
it entered outburst on October 10th and again at peak XRT count rate on
October 31st. The peak XRT count rate of Swift J010902.6-723710 was
$0.22_{-0.058}^{+0.071}$ counts s-1, implying a peak 0.3-10 keV band
luminosity of $L_{X}=1.35\times 10^{37}$ erg s-1 at a distance of 62.44 kpc.
S-CUBED detection of outbursting behavior triggered deeper follow-up Swift
observations of 5 ks per day on November 2nd and 4th in an effort to both
monitor the outburst and constrain the spin period of the NS. Deep
observations show the XRT count rate start to decline exponentially as
expected, reaching a mean XRT count rate value of 0.11 counts s-1 during the
early November observations. Additional observations taken on November 22nd
and December 2nd confirm the continuing trend of exponential decline in X-ray
brightness as the outburst fades. However, weekly S-CUBED detections
persisted, with additional detections occurring on December 12th and December
26th. These two detections had 0.3-10 keV luminosities of $L_{X}=3.55\times
10^{36}$ erg s-1 and $L_{X}=3.98\times 10^{36}$ erg s-1, respectively, which
do not fit the trend of steadily declining flux values after the peak
luminosity was reached.

Figure 2: $Z_{2}^{2}$ Periodogram for four observations with exposures of
$\simeq$5 ks of Swift XRT data taken in PC mode. The periodogram shows a peak
at 182 seconds in all four observations made after outburst was detected by
S-CUBED weekly monitoring. There is evidence for pulsar spin-up over time,
however the errors on period measurement are large meaning this result is not
statistically significant.

All 5 ks follow-up Swift observations were searched for the presence of
periodic pulsations that are indicative of the spin period of the X-ray
pulsar. The results of this search, carried out using a $Z_{2}^{2}$
Periodogram (Buccheri et al., 1983), are shown in Figure 2. In all 5 ks
observations, the periodogram shows peaks at around $182$s (see Figure 2 for
individual period measurements). Pulsar periods show small evidence for spin-
up during the outburst, although given large errors on the periodicities due
to the low numbers of counts and low (2.5s) PC mode frame time, the spin-up is
not strongly detected. Folding the light curve of each individual XRT
observation of 182s reveals an asymmetric double-peaked periodic signature,
which is the expected shape for an X-ray pulsar light curve. Based on the
phase-folded light curve and the strength of the periodogram peaks, we argue
that the true spin period of the pulsar is 182s, allowing Swift
J010902.6-723710 to be designated as SXP 182.

Spectral fitting is used to derive values for the spectral index of this
source and column density along the line of sight. This is done by fitting the
time-averaged 0.3-10 keV spectrum to an absorbed power-law model using the
methods presented in Evans et al. (2014). Using these methods, best-fit
parameters and a 68% confidence interval can be derived for both properties.
SXP 182 is shown to have a hard X-ray spectrum, with a derived photon index
for the source of $\Gamma=0.52_{-0.15}^{+0.16}$, which is consistent with the
photon index of $\Gamma\sim 0-1$ expected of a BeXRB system (Kennea et al.,
2018). Additionally, power law fitting indicates that the column density along
the line of sight towards this source is $N_{H}=5.1_{-1.9}^{+2.3}\times
10^{21}$ cm-2 which is higher than the average value of the column density of
$N_{H}=5.34\times 10^{20}\text{cm}^{-2}$ towards the SMC (Willingale et al.,
2013; Kennea et al., 2018).

Figure 3: OGLE IV, Swift UVOT, and Swift XRT light curves for Swift
J010902.6-723710.

###  2.2 Swift - UVOT

Swift J010902.6-723710 has been observed 221 times by UVOT in the uvw1-band
since 2016 as part of S-CUBED monitoring, which represents approximately
weekly coverage. UVOT light curves are generated for S-CUBED sources using the
FTOOLS software package (Blackburn et al., 1999). Photometric data are
extracted from a circle with 5 arcsecond radius around the XRT source position
for the object using the uvotsource method that is part of FTOOLS. Using this
method, the light curve presented in Figure 3 was generated for the entire
duration of S-CUBED.

UVOT data identifies Swift J010902.6-723710 as a persistent emitter in the
uvw1 band as it is well-detected in every observation with a 14.4 mean
magnitude. The most prominent feature of this light curve is the presence of
strong UV variability on timescales of 500-1000 days. The source reached its
minimum average magnitude of 14.7 in October 2021 after years of gradual
dimming from its initial brightness of 14.1 at the start of S-CUBED
monitoring. At this point, the source began a period of rapid brightening
lasting for over 500 days, reaching a maximum magnitude of 13.7 on June 27,
2023. After the peak magnitude was reached, the source has entered a steep
dimming phase that corresponds with the start of the XRT outburst. This
dimming phase was observed to be concluded on MJD 60283, when the source was
observed to be at its pre-outburst uvw1 magnitude.

The long period of UV brightening that Swift J010902.6-723710 underwent was
crucial in identifying the system as a candidate BeXRB before the outburst
occurred. This type of UV increase leading to outburst has been observed in
other sources such as Swift J004516.6–734703 (Kennea et al., 2020) which
demonstrated very little UV variability before experiencing a similar
brightening in the lead up to a Type I outburst. In the outburst of Swift
J004516.6–734703, the UV brightening was interpreted to be the result of a
circumstellar disk forming around the companion Be star in the newly-
identified binary. For Swift J010902.6-723710, we can interpret the long-term
variability to be an indication of growth and decay in the size of the disk,
where the disk expanded over the last 500+ days to a large-enough radius for
NS-disk interactions to occur.

###  2.3 OGLE - the Optical Gravitational Lensing Experiment

The OGLE project (Udalski et al., 2015) provides long term I-band photometry
with a cadence of 1-3 days for sources in the Magellanic Clouds. From the
X-ray position the optical/IR counterpart to Swift J010902.6-723710 is
identified as 2MASS J01090226-7237101. It was observed continuously for nearly
3 decades by OGLE, though the observations were interrupted for approximately
3 years during COVID-19.

The counterpart to the X-ray source is identified in the OGLE catalogue as:

OGLE II (I-band): smc_sc11.107571
OGLE III (I-band): smc110.3.22311
OGLE IV (I band): smc726.26.15515
OGLE IV (V band): smc726.26.v.22358

The I band data from just the OGLE IV project are shown in the top panel of
Figure 3 for comparison with the contemporaneous Swift UVOT and XRT data.

Figure 4: Top panel - the whole OGLE data set covering 26 years. Lower panel - the above data detrended with a simple polynomial in preparation for timing analysis. Figure 5: Generalised Lomb-Scargle analysis of the detrended OGLE data set. The peak is at a period of $60.623\pm 0.001$ days.The horizontal dashed line represents the False Alarm Probabilty of 1%. Figure 6: Detrended OGLE data divided into 5 consecutive epochs (see Table 1) and each then folded using the ephemeris given in Equation 1. For the purpose of being able to see the separate profiles, those numbered 2-4 have been arbitrarily shifted in the y-axis such as to avoid overlap. Table 1: OGLE data date ranges used in Figure 6 Data block | Dates | Average I band
---|---|---
number | JD-2450000 | magnitude
1 | 600-2000 | 16.26
2 | 2000-3100 | 16.22
3 | 3100-5000 | 16.35
4 | 5200-6400 | 16.37
5 | 6400-9000 | 16.34

For the purpose of period analysis the whole of the 26.4 years of OGLE data
were used and first detrended using a simple polynomial - see Figure 4. This
data set was then analysed with a generalised Lomb-Scargle routine and the
resulting power spectrum is shown in Figure 5. The peak in the power spectrum
is at a period of $60.623\pm 0.001$ days. This peak is driven by the sharp
eclipse-like features that can be clearly seen by eye in the top panels of
Figures 3 and 4. It is assumed that this represents the orbital period of the
system.

If all the detrended OGLE data are folded at the proposed binary period, with
a phase 0.0 set to the date of the first OGLE measurement (JD 2450627.9) then
the sharp eclipse-like dips are clearly seen in Figure 6. The average FWHM of
the dips is 0.1 in phase. It is also clear that the ingress into the eclipse
is much less sharp than the egress. To explore the changes in the shape of the
eclipse over time the OGLE data were divided up into 5 separate time segments
representing the times when visible changes were occurring in the profile. The
resulting separate profiles are shown in Figure 6. The 5 epochs chosen are
listed in Table 1. Using the phase of the dip in the I-band shown in Figure 6
as a reference point, the ephemeris for the time of the optical eclipses is
given by Equation 1:

$T_{ecl}=2450645.1+N(60.62\pm{0.01})~{}\textrm{~{}JD}$ (1)

In addition to the regular I-band measurements the OGLE IV project also
records V band magnitudes every few days. This provides the opportunity to
investigate the overall colour changes seen in the system as function of
brightness - a colour-magnitude diagram (CMD). Since the V band measurements
are less frequent than the I band, the determination of (V-I) can only occur
when the I and V measurements are close enough together in time. In this
instance the proximity of the 2 measurements is set to be less than 3 days.
This means that occasionally one V band measurement may be partnered with more
than one I band measurements. The result is shown in Figure 7 and discussed
below.

###  2.4 SALT - the Southern African Large Telescope

Figure 7: $(V-I)-I$ (left) and $(U-I)-I$ (right) colour-magnitude diagrams of Swift J010902.6-723710 Table 2: The H$\alpha$ equivalent width (EW) and peak separation ($\Delta V$) measured from SALT observations. Date | MJD | EW (Å) | Grating |  $\Delta V$ (km/s)
---|---|---|---|---
03-11-2023 | 60251.88 | -7.4$\pm$0.3 | PG0900 | $-$
09-11-2023 | 60257.87 | -7.3$\pm$0.5 | PG2300 | 373.6$\pm$0.4
02-12-2023 | 60280.85 | -7.4$\pm$0.4 | PG2300 | 383.4$\pm$0.9

Swift J010902.6-723710 was observed with the Southern African Large Telescope
(SALT; Buckley et al. 2006) with the Robert Stobie Spectrograph (Burgh et al.,
2003; Kobulnicky et al., 2003) using different settings. On 03-11-2023
(MJD60251.88) the PG0900 grating was used (grating angle of 15.125∘) with an
exposure time of 1200 sec covering a wavelength range of $4200-7250$ Å. The
PG2300 grating was used on 09-11-2023 (MJD60257.87) and 02-12-2023
(MJD60280.85) (grating angle of 48.875∘) with an exposure time of 1200 sec
covering a wavelength range $6100-6900$ Å. An additional observation was taken
on 09-11-2023 (MJD60257.87) with the PG2300 grating (grating angle of 30.5∘)
with an exposure time of 1500 sec covering a wavelength range $3840-4915$ Å.
The primary reductions, which include overscan corrections, bias subtraction,
gain and amplifier cross-talk corrections, were performed with the SALT
science pipeline (Crawford et al., 2012). The remainder of the data reduction
steps, which comprise wavelength calibration, background subtraction, and
extraction of the one-dimensional spectrum, were done in iraf. All spectra
were corrected for the heliocenter and redshift of the SMC of 145.6 km/s
(McConnachie, 2012).

Figure 8 shows the H$\alpha$ emission line profiles. The observations obtained
on MJD60257.87 and MJD60280.85 with the PG2300 grating exhibit asymmetric
double-peak profiles as a result of the Keplerian distribution of matter in
the disc when the disc is viewed at non-zero inclination angles. The
observation performed on MJD60251.88 with the PG0900 grating shows an
asymmetric single-peak profile since the resolution of the grating is
insufficient to resolve the two peaks. The H$\alpha$ equivalent width
measurements are recorded in Table 2.

##  3 Discussion

###  3.1 Corbet Diagram

Combining the spin period with the orbital period allows this new system to be
placed on the Corbet diagram. The original Corbet diagram (Corbet, 1984)
showed a correlation between the spin and orbital periods of neutron stars in
BeXRBs. However, instead of a correlation between spin and orbital periods, a
modern version (shown in Figure 9) shows that these properties are well-
constrained to a specific region of the Corbet diagram. BeXRB systems are
expected to fall above the diagonal of the diagram, which represents a log-
linear relationship between the orbital period in days and the spin period in
seconds of the NS. Figure 9 shows the location of SXP 182 when placed on the
Corbet diagram with all BeXRBs in the SMC that have known spin and orbital
periods (Haberl & Pietsch, 2005; Coe & Kirk, 2015; Haberl & Sturm, 2016;
McBride et al., 2017; Carpano et al., 2017; Kennea et al., 2018; Lazzarini et
al., 2019; Kennea et al., 2020; Carpano et al., 2022; Maitra et al., 2023).
The location of SXP 182 is consistent with the trend shown by other SMC
BeXRBs. This serves as a check that the periods derived in Sections 2.1 and
2.3 are consistent with the observational properties of other BeXRBs. For
Swift J010209.6-723710, the derived orbital period of 60.623 days and pulsar
spin period of 182 seconds place the system near the center of the
distribution of orbit and spin periods for SMC BeXRBs, providing a strong
piece of evidence validating the source as a newly-discovered BeXRB.

###  3.2 Inclination angle of the Be disc

The peak separation of the double-peak H$\alpha$ emission lines (Figure 8) can
be used to estimate the size of the H$\alpha$ emitting region (Huang, 1972):

$R=\frac{GM_{\ast}\sin^{2}i}{(0.5\Delta V)^{2}}$ (2)

Using $M_{\ast}=17.8$ M⊙ (Cox, 2000) based on the spectral type of the massive
companion, this results in a disc size radius range of $92-97\sin^{2}i$ R⊙. We
can estimate the inclination angle of the Be disc, assuming that the disc and
orbital planes are aligned and that during the period of X-ray activity the NS
was accreting matter from the outermost parts of the Be disc at periastron
passage. The semi-major axis of the orbit can be estimated by assuming a
companion mass of $M_{\ast}=17.8$ M⊙ and a neutron star mass of $M_{NS}=1.4$
M⊙. Similarly, the periastron passage can be estimated using a conservative
value for the eccentricity of $e\sim 0.5$ based on the relationship between
eccentricity and orbital period presented in Townsend et al. (2011). This
results in a range of disc inclination angles of $72-90^{\circ}$. The
suggestion of a high disc inclination angle is corroborated by the H$\alpha$
emission line displaying a double peak morphology with a deep central
depression from the high-resolution observations.

Figure 8: The evolution of the H$\alpha$ emission line from SALT observations.
The spectra are corrected for the heliocenter and redshift of the SMC.

Figure 7 shows the $(V-I)-I$ and $(U-I)-I$ colour-magnitude diagrams. The
general trend of the colour-magnitude plots is a redder-when-brighter pattern,
which is indicative of inclination angles below $90^{\circ}$(Harmanec, 1983;
Rajoelimanana et al., 2011; Reig & Fabregat, 2015). This trend is more
noticeable in the $(V-I)-I$ plot where the range in colour is broader since
the simultaneous $V$ and $I$ band observations were taken during a period of
substantial optical variability.

Figure 9: Corbet diagram for all BeXRBs in the SMC with known orbital periods
and spin periods (Haberl & Pietsch, 2005; Coe & Kirk, 2015; McBride et al.,
2017; Carpano et al., 2017; Kennea et al., 2018; Lazzarini et al., 2019;
Kennea et al., 2020; Carpano et al., 2022; Maitra et al., 2023). SXP 182 is
plotted alongside these data and is shown to demonstrate the same correlation
between X-ray pulsar spin period and NS orbital period that is seen in the
rest of the SMC BeXRB population.

###  3.3 Spectral Classification of the massive counterpart

Figure 10 shows the SALT spectrum of Swift J010902.6-723710 covering the blue
wavelength range. The spectrum shows clear signatures of an early-type star
with several Balmer and helium lines present. The H$\beta$ line is in weak
emission with an asymmetric profile, along with the other absorption lines
exhibiting infilling due to the presence of the circumstellar disc. According
to the criteria presented in Evans et al. (2004), the strong presence of the
He I lines at 4026, 4143, 4388 and 4471 Å as well as the weak presence of the
He II lines at 4541 and 4686 Å constrains the spectral type to B0-0.5. The
faintest V-band observation from OGLE measurements during our period of
monitoring is $\sim$16.5 magnitudes. Using this and the distance modulus of
the SMC of 18.95 (Graczyk et al., 2013), this results in a luminosity class of
V (Straizys & Kuriliene, 1981; Pecaut & Mamajek, 2013). In summary, the
spectral class of the massive companion in Swift J010902.6-723710 is B0-0.5 V.

Figure 10: The SALT spectrum of Swift J010902.6-723710 covering the blue
region with different line species labeled at their expected rest wavelengths.
The spectrum is corrected for the heliocenter and redshift of the SMC.

###  3.4 Eclipsing Behavior

The orbital period of Swift J010902.6-723710 derived from long-term OGLE
monitoring can be used to re-examine the UVOT light curve in search of
periodic behavior. This was done by de-trending the UVOT data using a 5th-
order polynomial and folding the de-trended data at the 60.623 day orbital
period of the system. Figure 11 shows the comparison between the binned and
time-averaged OGLE and UVOT light curves starting at MJD 57563, which is the
date of the first S-CUBED observation. Despite S-CUBED’s weekly cadence
providing infrequent sampling of any singular orbit, the eclipsing behavior is
clearly visible in the de-trended and folded UVOT light curve. When compared
to the OGLE data, it becomes evident that the shape of the eclipse profile is
wavelength-dependent. The UVOT data shows increasing emission that peaks just
before each eclipse begins, which is similar to the behavior exhibited by the
system during the 4th OGLE data block shown in Figure 6. Over the lifetime of
S-CUBED, the UVOT eclipse profile is also shown to be broader than the OGLE
eclipse profile with similar depth at the time of maximum eclipse. At the
maximum depth of the eclipse, the uvw1-band magnitude drops by an average of
0.22 magnitudes and I-band magnitude decreases by an average of 0.25
magnitudes. Converting these magnitude decreases to flux measurements for both
telescopes, the implied relative decrease in flux is at both wavelengths
$\frac{\Delta F}{F}\sim 0.2$.

Figure 11: A comparison of the OGLE IV and UVOT light curves folded at the
proposed binary period of 60.623 days using Equation 1. The OGLE data are
offset vertically to avoid overlap.

Equation 2 from Maggi et al. (2013) can be used to estimate the size of the
accretion disk based on the relative decrease in flux that is observed during
eclipse:

$\frac{\Delta F}{F}=\left(\frac{R_{X}}{R_{C}}\right)^{2}$ (3)

where $R_{X}$ is the radius of the eclipsing object and $R_{C}$ is the radius
of the optical companion. The value for $R_{X}$ can be estimated due to the
constraints placed on $R_{C}$ via the results of Section 3.3. The typical
radius for a spectral class B0V star is 7.4 $R_{\odot}$ (Allen, 1976). This
value can be used to constrain the upper limit for the radius of the eclipsing
object at $R_{X}=R_{C}\sqrt{\frac{\Delta F}{F}}=3.3\,R_{\odot}$. A radius of
this size rules out the NS, a large planet, or a Sun-like tertiary star as the
cause of the eclipsing behavior. Therefore, it can be concluded that the
eclipsing object is likely an extended accretion disk that surrounds the NS.
Maggi et al. (2013) derives the theoretical upper limit for the size of an
accretion disk in a BeXRB system to be $r_{c}\sim 11.5\,R_{\odot}$, assuming
Bondi-Hoyle accretion and a 10 $M_{\odot}$ companion star. The 3.3 $R_{\odot}$
accretion disk size derived for Swift J010902.6-723710 is well within this
upper limit value, providing further evidence in favor of an eclipsing NS
accretion disk.

Very few BeXRBs have been observed to demonstrate eclipsing behavior. Swift
J010902.6-723710 is only the third known eclipsing BeXRB system and the second
to be found in the SMC. LXP 168.8 (Maggi et al., 2013) was found to have a
eclipsing accretion disk with a 24.3 day orbital period. SXP 5.05 (Coe et al.,
2015) was observed to contain a Be star that eclipses its NS companion every
17 days. This newly-discovered eclipsing system thus provides a unique
opportunity to further constrain the physical parameters of the system such as
the sizes of both disks and the masses of both stars. More observations are
needed to further characterize the BeXRB using subsequent eclipses.

###  3.5 Outburst Type

Date | MJD | Phase |  $L_{X}$ (erg s-1)
---|---|---|---
10-10-2023 | 60227 | 0.06 | $4.14_{-2.61}^{+4.27}\times 10^{36}$
10-31-2023 | 60248 | 0.40 | $1.35_{-0.391}^{+0.475}\times 10^{37}$
11-02-2023 | 60250.0 | 0.47 | $5.66_{-0.275}^{+0.275}\times 10^{36}$
11-02-2023 | 60250.9 | 0.48 | $7.06_{-0.527}^{+0.527}\times 10^{36}$
11-04-2023 | 60252.1 | 0.50 | $5.30_{-0.508}^{+0.508}\times 10^{36}$
11-05-2023 | 60253.0 | 0.52 | $4.96_{-0.253}^{+0.253}\times 10^{36}$
11-05-2023 | 60253.6 | 0.53 | $4.03_{-0.385}^{+0.385}\times 10^{36}$
11-06-2023 | 60254 | 0.54 | $4.95_{-0.518}^{+0.518}\times 10^{36}$
11-21-2023 | 60269.9 | 0.80 | $3.20_{-2.04}^{+3.33}\times 10^{36}$
11-22-2023 | 60270.6 | 0.81 | $2.25_{-0.190}^{+0.190}\times 10^{36}$
12-02-2023 | 60280 | 0.96 | $1.83_{-0.177}^{+0.177}\times 10^{36}$
12-08-2023 | 60286 | 0.06 | $1.43_{-0.192}^{+0.192}\times 10^{36}$
12-12-2023 | 60290 | 0.13 | $3.55_{-2.28}^{+3.78}\times 10^{36}$
12-14-2023 | 60292 | 0.16 | $1.17_{-0.397}^{+0.397}\times 10^{36}$
12-26-2023 | 60304 | 0.36 | $3.98_{-2.26}^{+3.69}\times 10^{36}$
Table 3: S-CUBED and Swift TOO XRT detection dates, orbital phases from
Equation 1, and luminosities during outburst.  Figure 12: Truncated OGLE IV,
Swift UVOT, and Swift XRT light curves for Swift J010902.6-723710 showing
emission from MJD 60100 to MJD 60300. The times of optical eclipse, calculated
using Equation 1, are shown as vertical grey lines. Blue arrows show the
calculated XRT flux upper limit during S-CUBED observations where Swift
J010902.6-723710 is not detected.

The type of outburst observed in Swift J010902.6-723710 has remained uncertain
as long-term monitoring of the source continues to produce XRT detections well
past the peak of the initial outburst event. In Type I outbursts, X-ray
emission is often limited both by the duration of an orbit and the current
orbital phase. Emission is expected to be detected at a narrow range of values
near the periastron passage of the orbit and is expected to last for no longer
than a full orbital period (Stella et al., 1986). Additionally, Type I
outbursts typically demonstrate a moderate increase in luminosity, peaking at
$L_{X}\sim 10^{36}-10^{37}$ ergs s-1 (Okazaki & Negueruela, 2001; Reig, 2011).
Table 3 shows the date of XRT detections of Swift J010902.6-723710 and the
orbital phase at which they occurred with respect to Equation 1. The light
curve of XRT detections during outburst is plotted with respect to the time of
optical eclipses in Figure 12. Both the figure and table show the unusually
long duration of this outburst. The first S-CUBED detection of the outburst on
October 10th occurred at Phase 0.06, which is approximately the periastron
passage of orbit. However, other subsequent detections continue to occur well
past the periastron passage of the orbit, demonstrating phase independence of
the emission. The last two S-CUBED detections, occurring on December 12th and
December 26th, correspond to Phase 0.13 and 0.36 of a new orbit. These
subsequent detections are not consistent with the typical behavior
demonstrated by a Type I outburst and have more in common with the emission
signature of a Type II outburst (Okazaki & Negueruela, 2001; Townsend et al.,
2017; Tamang et al., 2022). If this is indeed a Type II outburst, then the
peak luminosity is much smaller than is typical for these outbursts. The peak
X-ray luminosity of $L_{X}=1.35\times 10^{37}$ erg s-1 is the only detection
to occur above $10^{37}$ erg s-1. Based on these features, one of two
situations has occurred. One possibility is that the system has produced an
abnormally long-duration Type I outburst with a typical peak luminosity
(Okazaki & Negueruela, 2001; Reig, 2011). Alternatively, the system has
produced a Type II outburst that fails to reach the characteristic large peak
luminosity that is expected during these events (Okazaki & Negueruela, 2001;
Reig, 2011).

However, it is important to remember that this traditional classification
scheme is very quantised - either it is called Type I or Type II. Since
originally proposed in 1986 (Stella et al., 1986) there have been many more
diverse examples of X-ray outbursts from BeXRB systems observed. In reality
there must be a scope for a whole spectrum of outburst types since the
outburst duration and the X-ray luminosity seen depend almost entirely on the
interaction between the circumstellar disc and the neutron star. That in turn
depends upon the characteristics of the circumstellar disc (density, size,
inclination, mass outflow rate etc.) and the characteristics of the neutron
star orbit (eccentricity, inclination to the circumstellar disk plane, phase
of periastron etc.). So, given all those free parameters one should expect a
whole range of observational characteristics, and not be driven to simply call
them Type I or Type II. SXP 182 is an excellent example of how the original
classification scheme can be overly simplistic.

##  4 Conclusion

This paper reports the detection of a previously-unknown BeXRB via weekly
observations of the S-CUBED survey. This new system, Swift J010902.6-723710,
was identified via a transient X-ray outburst and followed up via multi-
wavelength observations. Deep follow-up X-ray observations identify a proposed
spin period of 182s for the NS in this binary system. Historical light curve
analysis of both UV and IR emission reveal the presence of strong eclipse-like
features that re-appear every 60.623 days in the light curve, which is adopted
as the proposed orbital period of the system. Optical spectroscopy reveals a
strongly double-peaked H$\alpha$ emission line, indicating a highly-inclined
system with an inclination of $i=72-90^{\circ}$. Spectroscopic observations
are also used to constrain the spectral class of the optical companion as a
B0-0.5 star of spectral class Ve.

The proposed spin and orbital periods place Swift J010902.6-723710 place in
the center of the expected distribution for similar BeXRBs on the Corbet
Diagram. Eclipsing behavior is found to be caused by a $3.3R_{\odot}$
accretion disk that surrounds the NS, making this the third eclipsing BeXRB to
be detected so far. Finally, the type of outburst observed is found to be
uncertain, with characteristics of both Type I and Type II outbursts being
found in the X-ray emission of the source. More observations of this system
are needed, particularly during subsequent eclipses, in order to further
constrain the physical parameters of this rare eclipsing system and better
understand this long-lasting, moderately luminous X-ray outburst.

##  5 acknowledgments

This work made use of data supplied by the UK Swift Science Data Centre at the
University of Leicester. JAK and TMG acknowledge the support of NASA contract
NAS5-00136. We acknowledge the use of public data from the Swift data archive.

The SALT observations reported in this paper were obtained under the SALT
Large Science Programme on transients (2021-2-LSP-001; PI: DAHB), which is
also supported by Poland under grant no. MNiSW DIR/WK/2016/07.

PAE acknowledges UKSA support. DAHB acknowledges research support from the
National Research Foundation. LJT is supported by the SALT Foundation and the
NRF.

## References

  * Allen (1976) Allen, C. W. 1976, Astrophysical Quantities
  * Antoniou et al. (2010) Antoniou, V., Zezas, A., Hatzidimitriou, D., & Kalogera, V. 2010, ApJ, 716, L140, doi: 10.1088/2041-8205/716/2/L140
  * Blackburn et al. (1999) Blackburn, J. K., Shaw, R. A., Payne, H. E., Hayes, J. J. E., & Heasarc. 1999, FTOOLS: A general package of software to manipulate FITS files, Astrophysics Source Code Library, record ascl:9912.002.  http://ascl.net/9912.002
  * Buccheri et al. (1983) Buccheri, R., Bennett, K., Bignami, G. F., et al. 1983, A&A, 128, 245
  * Buckley et al. (2006) Buckley, D. A. H., Swart, G. P., & Meiring, J. G. 2006, in Proc. SPIE, Vol. 6267, Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series, 62670Z, doi: 10.1117/12.673750
  * Burgh et al. (2003) Burgh, E. B., Nordsieck, K. H., Kobulnicky, H. A., et al. 2003, in Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series, Vol. 4841, Proc. SPIE, ed. M. Iye & A. F. M. Moorwood, 1463–1471, doi: 10.1117/12.460312
  * Burrows et al. (2005) Burrows, D. N., Hill, J. E., Nousek, J. A., et al. 2005, Space Sci. Rev., 120, 165, doi: 10.1007/s11214-005-5097-2
  * Carpano et al. (2022) Carpano, S., Haberl, F., Maitra, C., et al. 2022, A&A, 661, A20, doi: 10.1051/0004-6361/202141082
  * Carpano et al. (2017) Carpano, S., Haberl, F., & Sturm, R. 2017, A&A, 602, A81, doi: 10.1051/0004-6361/201629299
  * Coe et al. (2015) Coe, M. J., Bartlett, E. S., Bird, A. J., et al. 2015, MNRAS, 447, 2387, doi: 10.1093/mnras/stu2568
  * Coe et al. (2021) Coe, M. J., Kennea, J. A., Evans, P. A., et al. 2021, MNRAS, 504, 1398, doi: 10.1093/mnras/stab972
  * Coe et al. (2023a) Coe, M. J., Kennea, J. A., Gaudin, T. M., et al. 2023a, The Astronomer’s Telegram, 16321, 1
  * Coe et al. (2023b) Coe, M. J., Kennea, J. A., Monageng, I. M., et al. 2023b, MNRAS, 524, 3263, doi: 10.1093/mnras/stad1987
  * Coe & Kirk (2015) Coe, M. J., & Kirk, J. 2015, MNRAS, 452, 969, doi: 10.1093/mnras/stv1283
  * Corbet (1984) Corbet, R. H. D. 1984, A&A, 141, 91
  * Cox (2000) Cox, A. N. 2000, Allen’s astrophysical quantities
  * Crawford et al. (2012) Crawford, S. M., Still, M., Schellart, P., et al. 2012, PySALT: SALT science pipeline, Astrophysics Source Code Library.  http://ascl.net/1207.010
  * Evans et al. (2004) Evans, C. J., Howarth, I. D., Irwin, M. J., Burnley, A. W., & Harries, T. J. 2004, MNRAS, 353, 601, doi: 10.1111/j.1365-2966.2004.08096.x
  * Evans et al. (2014) Evans, P. A., Osborne, J. P., Beardmore, A. P., et al. 2014, ApJS, 210, 8, doi: 10.1088/0067-0049/210/1/8
  * Gehrels et al. (2004) Gehrels, N., Chincarini, G., Giommi, P., et al. 2004, ApJ, 611, 1005, doi: 10.1086/422091
  * Graczyk et al. (2013) Graczyk, D., Pietrzyński, G., Pilecki, B., et al. 2013, in IAU Symposium, Vol. 289, Advancing the Physics of Cosmic Distances, ed. R. de Grijs, 222–225, doi: 10.1017/S1743921312021436
  * Graczyk et al. (2020) Graczyk, D., Pietrzyński, G., Thompson, I. B., et al. 2020, ApJ, 904, 13, doi: 10.3847/1538-4357/abbb2b
  * Haberl & Pietsch (2005) Haberl, F., & Pietsch, W. 2005, A&A, 438, 211, doi: 10.1051/0004-6361:20042470
  * Haberl & Sturm (2016) Haberl, F., & Sturm, R. 2016, A&A, 586, A81, doi: 10.1051/0004-6361/201527326
  * Harmanec (1983) Harmanec, P. 1983, Hvar Observatory Bulletin, 7, 55
  * Harris & Zaritsky (2004) Harris, J., & Zaritsky, D. 2004, AJ, 127, 1531, doi: 10.1086/381953
  * Huang (1972) Huang, S.-S. 1972, ApJ, 171, 549, doi: 10.1086/151309
  * Jaisawal et al. (2023) Jaisawal, G. K., Vasilopoulos, G., Naik, S., et al. 2023, MNRAS, 521, 3951, doi: 10.1093/mnras/stad781
  * Kennea et al. (2020) Kennea, J. A., Coe, M. J., Evans, P. A., et al. 2020, MNRAS, 499, L41, doi: 10.1093/mnrasl/slaa154
  * Kennea et al. (2021) —. 2021, MNRAS, 508, 781, doi: 10.1093/mnras/stab2632
  * Kennea et al. (2018) Kennea, J. A., Coe, M. J., Evans, P. A., Waters, J., & Jasko, R. E. 2018, ApJ, 868, 47, doi: 10.3847/1538-4357/aae839
  * Kobulnicky et al. (2003) Kobulnicky, H. A., Nordsieck, K. H., Burgh, E. B., et al. 2003, in Proc. SPIE, Vol. 4841, Instrument Design and Performance for Optical/Infrared Ground-based Telescopes, ed. M. Iye & A. F. M. Moorwood, 1634–1644, doi: 10.1117/12.460315
  * Lazzarini et al. (2019) Lazzarini, M., Williams, B. F., Hornschemeier, A. E., et al. 2019, ApJ, 884, 2, doi: 10.3847/1538-4357/ab3f32
  * Maggi et al. (2013) Maggi, P., Haberl, F., Sturm, R., et al. 2013, A&A, 554, A1, doi: 10.1051/0004-6361/201321238
  * Maitra et al. (2023) Maitra, C., Haberl, F., Kaltenbrunner, D., et al. 2023, The Astronomer’s Telegram, 15886, 1
  * Martin et al. (2014) Martin, R. G., Nixon, C., Armitage, P. J., Lubow, S. H., & Price, D. J. 2014, ApJ, 790, L34, doi: 10.1088/2041-8205/790/2/L34
  * McBride et al. (2017) McBride, V. A., González-Galán, A., Bird, A. J., et al. 2017, MNRAS, 467, 1526, doi: 10.1093/mnras/stx181
  * McConnachie (2012) McConnachie, A. W. 2012, AJ, 144, 4, doi: 10.1088/0004-6256/144/1/4
  * Monageng et al. (2017) Monageng, I. M., McBride, V. A., Coe, M. J., Steele, I. A., & Reig, P. 2017, MNRAS, 464, 572, doi: 10.1093/mnras/stw2354
  * Monageng et al. (2020) Monageng, I. M., Coe, M. J., Buckley, D. A. H., et al. 2020, MNRAS, 496, 3615, doi: 10.1093/mnras/staa1739
  * Monageng et al. (2022) Monageng, I. M., Coe, M. J., Townsend, L. J., et al. 2022, MNRAS, 511, 6075, doi: 10.1093/mnras/stac106
  * Okazaki & Negueruela (2001) Okazaki, A. T., & Negueruela, I. 2001, A&A, 377, 161, doi: 10.1051/0004-6361:20011083
  * Pecaut & Mamajek (2013) Pecaut, M. J., & Mamajek, E. E. 2013, ApJS, 208, 9, doi: 10.1088/0067-0049/208/1/9
  * Rajoelimanana et al. (2011) Rajoelimanana, A. F., Charles, P. A., & Udalski, A. 2011, MNRAS, 413, 1600, doi: 10.1111/j.1365-2966.2011.18243.x
  * Reig (2011) Reig, P. 2011, Ap&SS, 332, 1, doi: 10.1007/s10509-010-0575-8
  * Reig & Fabregat (2015) Reig, P., & Fabregat, J. 2015, A&A, 574, A33, doi: 10.1051/0004-6361/201425008
  * Rezaeikh et al. (2014) Rezaeikh, S., Javadi, A., Khosroshahi, H., & van Loon, J. T. 2014, MNRAS, 445, 2214, doi: 10.1093/mnras/stu1807
  * Rivinius (2019) Rivinius, T. 2019, IAU Symposium, 346, 105, doi: 10.1017/S1743921318008207
  * Roming et al. (2005) Roming, P. W. A., Kennedy, T. E., Mason, K. O., et al. 2005, Space Sci. Rev., 120, 95, doi: 10.1007/s11214-005-5095-4
  * Roy et al. (2022) Roy, A., Cappallo, R., Laycock, S. G. T., et al. 2022, ApJ, 936, 90, doi: 10.3847/1538-4357/ac82b6
  * Stella et al. (1986) Stella, L., White, N. E., & Rosner, R. 1986, ApJ, 308, 669, doi: 10.1086/164538
  * Straizys & Kuriliene (1981) Straizys, V., & Kuriliene, G. 1981, Ap&SS, 80, 353, doi: 10.1007/BF00652936
  * Tamang et al. (2022) Tamang, R., Ghising, M., Tobrej, M., Rai, B., & Paul, B. C. 2022, MNRAS, 515, 5407, doi: 10.1093/mnras/stac2135
  * Townsend et al. (2011) Townsend, L. J., Coe, M. J., Corbet, R. H. D., & Hill, A. B. 2011, MNRAS, 416, 1556, doi: 10.1111/j.1365-2966.2011.19153.x
  * Townsend et al. (2017) Townsend, L. J., Kennea, J. A., Coe, M. J., et al. 2017, MNRAS, 471, 3878, doi: 10.1093/mnras/stx1865
  * Udalski et al. (2015) Udalski, A., Szymański, M. K., & Szymański, G. 2015, Acta Astron., 65, 1.  https://arxiv.org/abs/1504.05966
  * Willingale et al. (2013) Willingale, R., Starling, R. L. C., Beardmore, A. P., Tanvir, N. R., & O’Brien, P. T. 2013, MNRAS, 431, 394, doi: 10.1093/mnras/stt175


#  EnDex: Evaluation of Dialogue Engagingness at Scale

Guangxuan Xu1 Ruibo Liu2
Fabrice Harel-Canada1 Nischal Reddy Chandra1 Nanyun Peng1
1University of California, Los Angeles 2Dartmouth College
{gxu21, violetpeng} @cs.ucla.edu

###### Abstract

We propose EnDex, the first human-reaction based model to evaluate dialogue
engagingness. EnDex is trained on 80k Reddit-based Engagement Dataset (RED)
curated using a novel distant-supervision framework. Engagingness is a key
measure that captures high-level quality of AI dialogue systems and closely
reflects actual user experience. However, data shortage, plus the abstract and
extensive definition of engagingness makes it challenging to develop an
automatic metric. Our work departs from mainstream approaches that use
synthetic negative examples to train binary classifiers, and instead, proposes
a solution using distant-supervision from human-reaction feedback. To support
the soundness of our EnDex metric, we offer a theoretical foundation for
engagement, an extensive ablation study, and empirical evidence of high
correlation on five engagingness related datasets.111Off-the-shelf EnDex model
and the RED dataset is available at https://github.com/gxxu-ml/EnDex.

##  1 Introduction

Many modern generative language models are trained to maximize a likelihood
objective, but this paradigm tends to assign high probability to generic
responses (Li et al., 2016), such as “I don’t know.”. Prior research has
established that people prefer to converse with interesting, creative, and
informative agents (See et al., 2019), all concepts broadly related to the
notion of _engagingness_. Furthermore, engagingness is recognized as a key
evaluation metric for the quality of dialogue systems (Zhang et al., 2018;
Ghazarian et al., 2020). For example, FAIR’s ParlAI (Miller et al., 2017)
incorporated Engagingness as the default testing metric in the Blenderbot
system (Roller et al., 2021); dialogue data challenges, like ConvAI2 (Dinan et
al., 2019), Amazon Alexa Prize222https://www.amazon.science/alexa-prize, and
ensemble metrics like FED (Mehri and Eskenazi, 2020), all measure engagingness
to benchmark dialogue quality.

Figure 1: Example of an online post with scores for emotional engagement (EE),
attentional engagement (AE), and behavioral engagement (BE) in blue to
represent the 3 dimensions of _human engagement_ ; reply engagement (RE) in
red; and the aggregated EnDex score in green. We apply z-score to EnDex Score
and pick a hyper-parameter threshold to cluster posts into positive and
negative samples.

However, the current evaluation of engagingness still primarily relies on
expensive human annotation rather than off-the-shelf automatic tools, due to
several theoretical and technical challenges: firstly, unlike more well-
characterized properties such as fluency, the definition of engagingness is
significantly more abstract and multi-dimensional (See et al., 2019),
requiring well-tuned quality metrics for each sub-dimension to aggregate a
final score. Secondly, what qualifies as engaging is open-ended and many
different answers may embody the concept (Ghazarian et al., 2020). Therefore,
reference-based metrics requiring unique ground truth, such as BLEURT (Sellam
et al., 2020) and BERTScore (Zhang et al., 2020), cannot apply. Thirdly,
there’s an acute shortage of large-scale, high-quality data annotated for
engagingness.

Ghazarian et al. (2020) jump-started efforts to automatically measure dialogue
engagement, where they fine-tuned a BERT-based model Devlin et al. (2019) on
the ConvAI2 and DialyDialog datasets (Li et al., 2017) to predict an
engagingness score. However, finetuning on small size supervised data could
easily lead to overfitting and generalization problems. Another high
performing metric on engagingness USL-H (Phy et al., 2020) assumes a positive
set and generates synthetic negative samples to train model. However, credible
positive samples are not always available, and synthetic negative samples may
not be challenging enough to further advance classifier performance.

In light of the above challenges, we propose EnDex, a novel metric trained
with distantly supervised data to predict turn-level dialogue engagingness
(Figure 1). EnDex requires neither human annotations nor direct
disentanglement of engagingness. Instead, we leverage observed _user
reactions_ to posts as distant signals to model engagingness, which marks a
departure from mainstream approach to train on synthetic negative samples (Lan
et al., 2020; Ghazarian et al., 2022; Tao et al., 2018; Sato et al., 2020).
EnDex trains on real conversations sourced from Reddit, that are automatically
annotated as positive and negative examples with our framework. The novoel
dataset is named RED (Reddit Engagement Dataset) with over 80k labelled
samples. EnDex framework derives its theoretical underpinning from relevant
HCI works, and has shown superior performance on five benchmark datasets.

##  2 EnDex Metric

Engagingness is not only a linguistic concept useful for dialogue systems, but
also manifests itself in multi-modalities and is extensively leveraged to
benchmark gaming and online learning experiences (Silpasuwanchai et al., 2016;
Chen et al., 2005; Mcmahan, 2003; Schoenau-Fog, 2011). Our work is inspired by
HCI study of Human Engagement Ma (2018), which decomposes engagingness into
three major dimensions including attentional engagement (e.g., clicks and
scrolls), behavior engagement (e.g., facial expressions), and emotional
engagement (e.g., heart rate).

EnDex metric follows the same intuition: we can infer engagingness of a text
by analyzing human reactions to it, for which there is abundant data in social
media. EnDex metric learns from our distant-supervised RED dataset, which
measures dialogue engagement along four dimensions as shown in Figure 1;
three-dimensions correspond to the original Human Engagement definition, and
one distinct Reply Engagement dimension for the dialogue specific task.

###  2.1 Reddit Engagement Dataset (RED)

We curate the Reddit Engagement Dataset (RED), a distant-supervision set, with
80k single-turn conversations. We source RED from Reddit, sampling from 43
popular subreddits, and processed a total of 5 million posts, filtering out
data that was either non-conversational, toxic, or posts not possible to
ascertain popularity; the resulting data distribution of RED is shown in Table
1. The following sections will explain the procedure to automatically annotate
EnDex scores and cluster samples into positive and negative sets.

We also curated a RED testset with 150 human annotated samples obtained from a
different split from RED. The inter-annotator agreement is 0.34 Fleiss-Kappa,
indicating fair agreement, which reflects the challenge of determining
engagingness.

| Engaging | Non-engaging
---|---|---
# of samples | 40,162 | 40,162
Emotional | .605 $\pm$ .273 | .152 $\pm$ .120
Attentional | .759 $\pm$ .127 | .203 $\pm$ .100
Behavioral | .659 $\pm$ .274 | .318 $\pm$ .285
Reply | .718 $\pm$ .154 | .354 $\pm$ .980
EnDex | .709 $\pm$ .048 | .259 $\pm$ .033

Table 1: RED dataset has two classes, engaging and non-engaging, clustered by
applying z-score on EnDex score. This table shows the mean and standard
deviation of sub-dimension scores for both classes; the last row displays the
distribution of the overall EnDex score.

###  2.2 Distantly-Supervised Engagingness Scores

We use distant-supervision to provide samples in RED an EnDex Score, which is
the aggregate of 4 engaging dimensions. Section 2.2 discusses the intuition
for each engagingness dimension; section 2.3 explains how to adjust raw score
by thread popularity; section 2.4 lays out the formula to normalize and
aggregate sub-dimensions into the overall engagingness score; section 2.5
explains sampling with z-score to convert the task into binary classification.

  * •

Emotional Engagement (EE): Emotional connection is a key sign of human
engagement (Savin-Baden et al., 2014); and we model EE using a multi-class
emotional classifier (Demszky et al., 2020) on post replies. If post receives
positive and emotional replies, it’s engaging; negative or neutral replies
indicates non-engaging.

  * •

Attentional Engagement (AE): More user time spent indicates higher engagement
(Attfield et al., 2011). We model AE of a post by examining whether it has
editted replies, and the information specificity in the replies.

  * •

Behavioral Engagement (BE): Human behavioral features closely correlate with
their engagement state (Attfield et al., 2011), and we model BE by examining
Reddit post scores, adjusted by popularity.

  * •

Reply Engagement (RE): Following definition from (Ghazarian et al., 2020), if
a certain post is very likely to be continued by following threads, it is
considered engaging; reply_counts are also popularity adjusted.

###  2.3 Adjustment for Popularity

Raw score for Behavior Engagement(upvotes) and Reply Engagement(reply counts)
are heavily influenced by the popularity of the thread in which the post
appears. A non-engaging post may receive high user interaction because it
simply receives a lot of exposure; on the flip side, a very engaging most may
receive zero user interaction simply because it is rarely seen. To mitigate
the imbalanced exposure problem, we calculate a popularity value for each
thread, and adjust posts scores by the popularity value of the thread it
resides.

Popularity Value(PV) The PV of a post is given by the amount of exposure its
parent post attracts. Let the target post be $\theta$ and its parent $\sigma$,
$R_{eply}$ obtains the reply counts of a post, and $U_{pvote}$ obtains the
upvotes of a post. The PV is defined in equation (1), where coefficient 2 is
adopted to give equal weight for reply and upvotes; popularity value adjusted
RE score is given by PVRE in equation (6), where $M_{pv}$ and $M_{re}$ are the
median of popularity value and reply counts in the entire dataset. Only
popularity adjusted scores are used for calculating EnDex score.

$\textrm{PV}(\theta)=2*R_{eply}(\sigma)+U_{pvote}(\sigma)$ (1)

$\textrm{PVRE}(re)=re+\frac{M_{pv}}{M_{\textrm{re}}}*\frac{re}{\textrm{PV(re)}}*re$
(2)

###  2.4 Monotone Submodular Normalization

The final EnDex score is essentially a weighted sum of the 4 respective sub-
dimension scores; an importance nuance is the usage of submodular
normalization (shown in Eq. 8) for 3 dimensions to bring raw scores to the
scale of 0-1. We observe that unit increase in raw score lead to diminishing
positive effect on engagingness. For example, a sentence with 100 replies
should be more engaging than one with 1 replies, but not 99 times more; thus,
we normalize engagingness score with a monotone submodular function
$f(x)=\frac{x}{x+\alpha}$.

$N(x)=\Bigg{(}\displaystyle\sum_{n=1}^{3}w_{i}*\frac{x_{i}}{x_{i}+\alpha_{i}}\Bigg{)}+w_{\textrm{EE}}*x_{\textrm{EE}},$
(3)

$N$ is the normalized score for sample $x$, $x_{i}$ is $x$’s raw score on
dimension $i$, where $i\in\\{RE,BE,AE\\}$; $\alpha_{i}$ is the median of
$i$-th dimension; $w_{i}$ is the weight for $i$ dimension; $w_{\textrm{EE}}$
is the weight for EE dimension. The weight can be tuned for your own usage of
RED; 333For EnDex, $\alpha$ for three dimensions RE, BE, AE are 1, 2, 18,
respectively; we also applied weights of 3, 3, 2, 1 for RE, AE, EE, and BE.

###  2.5 Clustering with z-score

Essentially, engagingness prediction is a classification task, and we want to
prepare dataset for binary classification. We use $z$-score on the EnDex Score
to easily sample and cluster the data according to standard deviation from
mean. A confidence threshold $\kappa$(ours is 1) needs to be picked, which
means that we regard samples that fall between $\kappa$ standard deviation
from mean as uncertain, and are thus discarded. And we cluster positive and
negative samples using the following equation (9).

$Polarity(x)=\begin{cases}1&\text{if }z\\_score(x)>\kappa\\\ 0&\text{if
}z\\_score(x)<-\kappa\end{cases}$ (4)

The EnDex metric is then trained as a binary classification task by finetuning
a RoBERTa-large model (Liu et al., 2019) on turn-level RED data.

##  3 Experiments

Method | Better | PredEng-600 | Fed-eng | Red-Test | Grade
---|---|---|---|---|---
P | S | P | S | P | S | P | S | P | S
Random (ref.) | 0.025 | 0.025 | -0.012 | -0.013 | 0.080 | 0.081 | -0.053 | -0.053 | 0.053 | 0.045
Question | 0.167 | 0.167 | 0.073 | 0.074 | 0.320 | 0.320 | 0.194 | 0.194 | 0.009 | 0.008
Specificity | 0.357 | 0.357 | 0.076 | 0.102 | 0.254 | 0.254 | 0.122 | 0.122 | -0.090 | -0.090
USL-H | 0.356 | 0.343 | 0.688 | 0.699 | 0.267 | 0.277 | 0.121 | 0.125 | 0.234 | 0.243
Pred_En | 0.234 | 0.310 | -0.137 | -0.134 | 0.250 | 0.340 | 0.044 | 0.178 | -0.090 | -0.060
Pred_En  (FT+DD) | 0.338 | 0.368 | 0.390 | 0.450 | 0.253 | 0.195 | 0.237 | 0.258 | 0.194 | 0.173
Ours: EnDex | 0.414* | 0.397* | 0.397 | 0.348 | 0.235* | 0.225* | 0.381** | 0.375** | 0.266 | 0.248
Ours: EnDex+NS |  0.478* |  0.455* | 0.597** | 0.577** | 0.229* | 0.214* |  0.389** |  0.378** | 0.308 |  0.282*
Ours: EnDex-Best | 0.521 | 0.511 | 0.620 | 0.629 | 0.286 | 0.253 | 0.414 | 0.405 | 0.406 | 0.352
Table 2: The correlation between engagement scores and ground truth human
judgment. Best scores are emboldened and second-best are underlined. We train
EnDex and EnDex+NS 10 times and report the mean with * and ** indicating a
stdev < 0.05 and < 0.03, respectively. EnDex-Best is the best score observed
over the 10 runs. Compared to existing metrics, the EnDex-framework achieves
SOTA correlation with human judgement on engagingness, leading by far on our
newly proposed Red-Test dataset with more complex and longer texts than
chitchats.

###  3.1 Experiment Set-up

We test the performance of the EnDex metric on 5 golden evaluation sets that
have turn-level labels. Among them, Better (Ghazarian et al., 2019),
PredEeng-600 (Ghazarian et al., 2020) are annotated specifically for
engagingness with high annotator agreement. Better samples are taken from
human conversation, while half of PredEng-600 are chatbot generations. FED
(Mehri and Eskenazi, 2020) annotates dialogue for 9 different dimensions, and
we use their engagingness scores as target labels. GRADE (Huang et al., 2020)
contains quality annotations for dialogue coherence, and we include it to test
whether our model can also have good zero-shot performance on related tasks.
Lastly, our own RED-Test is sourced from Reddit, contains discussions on
various topics. A table of evaluation set statistics is provided in the
Appendix3.

###  3.2 Ablation Study on Engaging Dimensions

To test the robustness of the 4 engagingness dimensions 2.2 of EnDex, we
conducted ablation study to train model using only signals from each of the 4
dimensions. We hypothesize that dimensions with high positive contribution
towards final results should have very successful clustering of engaging and
non-engaging samples by itself; so, if we train model on data clustered by
such dimension, we can still get good performance models.

We train five different models on different subsets of RED. All datasets
included the same 40k negative (i.e. non-engaging) samples drawn according to
our overall engagement score. However, the other 40k positive (i.e. engaging)
samples were selected according to a particular dimension score (e.g. EE, AE,
BE, and RE), except for EnDex, which is our aggregate score model. Figure 2
shows that all four dimensions correlate with engagingness to some degree, but
RE, AE, and EE are especially effective. We also observe a synergistic effect
of training on a composite score rather than any one dimension individually.
The experiment highlights and corroborates the multi-dimensionality of
engagingness previously reported in the literature (See et al., 2019).
Overall, having an aggregate score is crucial for successful distant-
supervised annotation of negative and positive examples.

Figure 2: Ablation study of our four engagement dimensions. The EnDex model
was trained on our aggregate engagingness score, while RE, EE, AE, and BE
indicate models trains only on scores reflecting that particular dimension.

###  3.3 Comparison with Related Works

We compare our EnDex metric, and heuristics-augmented EnDex+NS metric with
five baselines. Three baselines are rule-based, including Random, information
Specificity (See et al., 2019) that counts number of non-stopword tokens, and
Inquisitiveness (Ghandeharioun et al., 2019) that examines question asking
ability. We included them because in some dataset, rule-based system could
work surprisingly well (Yeh et al., 2021).

We selected USL-H (Phy et al., 2020) as a baseline because it is the top
performing metric on the PredEng-600 and Fed engagingness evaluation sets Yeh
et al. (2021). USL-H is designed to measure high-level dialog quality,
including understandability, sensibleness, and likability; it trains 3 BERT-
based (Devlin et al., 2019) classifiers for each component, and uses a
composite score named USL-H for overall assessment. Pred_En (Ghazarian et al.,
2020) uses BERT embedding plus MLP layer and train on ConvAI dataset (Dinan et
al., 2019) to make engagement score predictions. PRED_EN (FT+DD) further
finetunes the original PRED_EN metric on the DailyDialogue dataset, to get
better results.

Our model has two versions: EnDex is solely trained on human-reaction based
data. +NS means non-engaging samples set is mixed with some rule-based
negative samples, created by random insertion, random deletion, copying, and
generic replies;

The experiments in Table 2 demonstrate that our model achieves strong
performance on 4 engagingness related datasets, and good correlation with one
coherence dataset(Grade). EnDex surpasses Pred_En and USL-H by a large margin
on two real human conversations, Better and Red-Test. USL-H still leads in
PredEng-600, and EnDex+NS’s best model is a close second. Yeh et al. (2021)
shows achieving high score on FED-eng is challenging, with no one surpassing
0.3 spearman in 12 tested metrics. A strong rule-based question detection
algorithm surprisingly claims the highest result, and EnDex a close second.

We find that training solely with human reaction distant supervision signals
suffices for building competitive models on par or even surpassing mainstream
metrics, and it shows better generalization capability in new domains, which
seems to echo recent success on modeling human preferences via upvotes in
Reddits (Gao et al., 2020).

##  4 Conclusion

This paper proposes the first human reaction based model, EnDex, to evaluate
dialogue engagingness, and curates an 80k Reddit Engagement Dataset (Red)
using a novel distant-supervision framework. The success of EnDex demonstrates
the validity of training automatic metrics with human reaction signals,
offering a strong complement to a synthetic negative sampling approach. We
also release an off-the-shelf EnDex model, and a large scale dataset to
facilitate future research.

## Limitation

One limitation is that we only curated data for turn-level dialogue. Multi-
turn dialogues could also be useful, but it was computationally infeasible to
interactively query Reddit for entire threads of conversation. Future work can
explore this direction to produce dialogue-level and system-level engagingness
metrics.

We also haven’t fully explore our model’s performance on non-dialogue domains,
such as on story or creative generations. The training data distribution from
the Reddit corpus is diverse enough that it could potentially achieve good
performance in non-dialogue settings. A valuable direction of future work is
to adapt our method for more general engagingness, or another evaluation
metric for open-domain generation.

## Ethics

A caveat of using framing our approach around human attention is that not all
texts attracting high attention are good and ethical. Since being engaging
often carries a positive connotation, we made a deliberate design decision to
mitigate forms of _negative engagement_ in our metric. For example, we assign
lower scores to samples flagged by Reddit as controversial, and our behavioral
engagement dimension subtracts downvotes from upvotes to punish negative,
biased (Liu et al., 2021), and aggressive comments. Moreover, we implemented
our emotional engagement algorithm to reward posts with positive emotional
replies and punish posts that prompt negative emotions. Future may try to
account for the darker aspects of engagingness into our framework and improve
the EnDex metric to differentiate between positive and negative engagement.

Human annotations for RED-Test were obtained via Amazon Mechanical Turks. We
filtered out toxic samples to reduce the likelihood of offensive content and
paid $0.30 USD per instance for an expected hourly wage of $20 USD.

## Acknowledgement

We want to give special thanks to Sarik Ghazarian for great advice and help
with the baseline model; we also appreciate the brainstorming session with
Hongyuan Yang and the suggestion to use z-score for sampling from Zeyi Wang.

## References

  * Attfield et al. (2011) Simon Attfield, Gabriella Kazai, and Benjamin Piwowarski. 2011.  Towards a science of user engagement ( position paper ).
  * Chen et al. (2005) Mark Chen, Beth E. Kolko, Elisabeth Cuddihy, and Eliana Medina. 2005.  Modeling and measuring engagement in computer games.  In _DiGRA Conference_.
  * Demszky et al. (2020) Dorottya Demszky, Dana Movshovitz-Attias, Jeongwoo Ko, Alan Cowen, Gaurav Nemade, and Sujith Ravi. 2020.  GoEmotions: A dataset of fine-grained emotions.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 4040–4054, Online. Association for Computational Linguistics.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _ArXiv_ , abs/1810.04805.
  * Dinan et al. (2019) Emily Dinan, Varvara Logacheva, Valentin Malykh, Alexander H. Miller, Kurt Shuster, Jack Urbanek, Douwe Kiela, Arthur D. Szlam, Iulian Serban, Ryan Lowe, Shrimai Prabhumoye, Alan W. Black, Alexander I. Rudnicky, Jason Williams, Joelle Pineau, Mikhail S. Burtsev, and Jason Weston. 2019.  The second conversational intelligence challenge (convai2).  _ArXiv preprint_ , abs/1902.00098.
  * Gao et al. (2020) Xiang Gao, Yizhe Zhang, Michel Galley, Chris Brockett, and Bill Dolan. 2020.  Dialogue response ranking training with large-scale human feedback data.  _ArXiv_ , abs/2009.06978.
  * Ghandeharioun et al. (2019) Asma Ghandeharioun, Judy Hanwen Shen, Natasha Jaques, Craig Ferguson, Noah Jones, Àgata Lapedriza, and Rosalind W. Picard. 2019.  Approximating interactive human evaluation with self-play for open-domain dialog systems.  In _Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada_ , pages 13658–13669.
  * Ghazarian et al. (2019) Sarik Ghazarian, Johnny Wei, Aram Galstyan, and Nanyun Peng. 2019.  Better automatic evaluation of open-domain dialogue systems with contextualized embeddings.  In _Proceedings of the Workshop on Methods for Optimizing and Evaluating Neural Language Generation_ , pages 82–89, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Ghazarian et al. (2020) Sarik Ghazarian, Ralph M. Weischedel, Aram Galstyan, and Nanyun Peng. 2020.  Predictive engagement: An efficient metric for automatic evaluation of open-domain dialogue systems.  In _The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020_ , pages 7789–7796. AAAI Press.
  * Ghazarian et al. (2022) Sarik Ghazarian, Nuan Wen, A. G. Galstyan, and Nanyun Peng. 2022.  Deam: Dialogue coherence evaluation using amr-based semantic manipulations.  _ArXiv preprint_ , abs/2203.09711.
  * Hanu and Unitary team (2020) Laura Hanu and Unitary team. 2020.  Detoxify.  Github. https://github.com/unitaryai/detoxify.
  * Huang et al. (2020) Lishan Huang, Zheng Ye, Jinghui Qin, Liang Lin, and Xiaodan Liang. 2020.  GRADE: Automatic graph-enhanced coherence metric for evaluating open-domain dialogue systems.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 9230–9240, Online. Association for Computational Linguistics.
  * Lan et al. (2020) Tian Lan, Xian-Ling Mao, Wei Wei, Xiaoyan Gao, and Heyan Huang. 2020.  Pone: A novel automatic evaluation metric for open-domain generative dialogue systems.  _ArXiv preprint_ , abs/2004.02399.
  * Li et al. (2016) Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. 2016.  A diversity-promoting objective function for neural conversation models.  In _Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 110–119, San Diego, California. Association for Computational Linguistics.
  * Li et al. (2017) Yanran Li, Hui Su, Xiaoyu Shen, Wenjie Li, Ziqiang Cao, and Shuzi Niu. 2017.  DailyDialog: A manually labelled multi-turn dialogue dataset.  In _Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 986–995, Taipei, Taiwan. Asian Federation of Natural Language Processing.
  * Liu et al. (2021) Ruibo Liu, Chenyan Jia, Jason Wei, Guangxuan Xu, Lili Wang, and Soroush Vosoughi. 2021.  Mitigating political bias in language models through reinforced calibration.  _Proceedings of the AAAI Conference on Artificial Intelligence_ , 35(17):14857–14866.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Roberta: A robustly optimized bert pretraining approach.  _ArXiv preprint_ , abs/1907.11692.
  * Ma (2018) Xiaojuan Ma. 2018.  Towards human-engaged AI.  In _Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence, IJCAI 2018, July 13-19, 2018, Stockholm, Sweden_ , pages 5682–5686. ijcai.org.
  * Mcmahan (2003) Alison Mcmahan. 2003.  Immersion, engagement, and presence: A method for analyzing 3-d video games.  _The Video Game Theory Reader_ , pages 67–86.
  * Mehri and Eskenazi (2020) Shikib Mehri and Maxine Eskenazi. 2020.  Unsupervised evaluation of interactive dialog with DialoGPT.  In _Proceedings of the 21th Annual Meeting of the Special Interest Group on Discourse and Dialogue_ , pages 225–235, 1st virtual meeting. Association for Computational Linguistics.
  * Miller et al. (2017) Alexander Miller, Will Feng, Dhruv Batra, Antoine Bordes, Adam Fisch, Jiasen Lu, Devi Parikh, and Jason Weston. 2017.  ParlAI: A dialog research software platform.  In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pages 79–84, Copenhagen, Denmark. Association for Computational Linguistics.
  * Phy et al. (2020) Vitou Phy, Yang Zhao, and Akiko Aizawa. 2020.  Deconstruct to reconstruct a configurable evaluation metric for open-domain dialogue systems.  In _Proceedings of the 28th International Conference on Computational Linguistics_ , pages 4164–4178, Barcelona, Spain (Online). International Committee on Computational Linguistics.
  * Roller et al. (2021) Stephen Roller, Emily Dinan, Naman Goyal, Da Ju, Mary Williamson, Yinhan Liu, Jing Xu, Myle Ott, Eric Michael Smith, Y-Lan Boureau, and Jason Weston. 2021.  Recipes for building an open-domain chatbot.  In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_ , pages 300–325, Online. Association for Computational Linguistics.
  * Sato et al. (2020) Shiki Sato, Reina Akama, Hiroki Ouchi, Jun Suzuki, and Kentaro Inui. 2020.  Evaluating dialogue generation systems via response selection.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 593–599, Online. Association for Computational Linguistics.
  * Savin-Baden et al. (2014) Maggi Savin-Baden, Gemma Tombs, Roy Bhakta, and David J. H. Burden. 2014.  Students’ experiences of emotional connection with pedagogical agents.
  * Schoenau-Fog (2011) Henrik Schoenau-Fog. 2011.  The player engagement process - an exploration of continuation desire in digital games.  In _DiGRA Conference_.
  * See et al. (2019) Abigail See, Stephen Roller, Douwe Kiela, and Jason Weston. 2019.  What makes a good conversation? how controllable attributes affect human judgments.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 1702–1723, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Sellam et al. (2020) Thibault Sellam, Dipanjan Das, and Ankur Parikh. 2020.  BLEURT: Learning robust metrics for text generation.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 7881–7892, Online. Association for Computational Linguistics.
  * Silpasuwanchai et al. (2016) Chaklam Silpasuwanchai, Xiaojuan Ma, Hiroaki Shigemasu, and Xiangshi Ren. 2016.  Developing a comprehensive engagement framework of gamification for reflective learning.  pages 459–472.
  * Tao et al. (2018) Chongyang Tao, Lili Mou, Dongyan Zhao, and Rui Yan. 2018.  RUBER: an unsupervised method for automatic evaluation of open-domain dialog systems.  In _Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, (AAAI-18), the 30th innovative Applications of Artificial Intelligence (IAAI-18), and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence (EAAI-18), New Orleans, Louisiana, USA, February 2-7, 2018_ , pages 722–729. AAAI Press.
  * Yeh et al. (2021) Yi-Ting Yeh, Maxine Eskénazi, and Shikib Mehri. 2021.  A comprehensive assessment of dialog evaluation metrics.  _ArXiv preprint_ , abs/2106.03706.
  * Zhang et al. (2018) Saizheng Zhang, Emily Dinan, Jack Urbanek, Arthur Szlam, Douwe Kiela, and Jason Weston. 2018.  Personalizing dialogue agents: I have a dog, do you have pets too? In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 2204–2213, Melbourne, Australia. Association for Computational Linguistics.
  * Zhang et al. (2020) Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020\.  Bertscore: Evaluating text generation with BERT.  In _8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020_. OpenReview.net.

##  Appendix A Appendix

###  A.1 RED Processing Steps

Our Reddit data is downloaded from Pushshift.io
444https://files.pushshift.io/reddit/comments/, and we processed approximately
5M data to curate an 80k sample RED dataset. We deleted posts that do not have
an immediate parent thread, because we need pair turn-level data. Our
preprocessing removes non-conversational data, such as posts including &
gt(reply to symbol). We also removed explicitly toxic data filtered by
Detoxify (Hanu and Unitary team, 2020).

We also applied a key data processing trick to reduce noisy signals – the
exposure variable. It helps measure the amount exposure each post receives to
help normalize its upvote/reply score. We reward posts that are in low-
exposure, unpopular threads, while penalizing posts in high-exposure, popular
threads, because high upvotes and replies in popular threads may be more due
to exposure than true engagingness.

After computing the normalized score given in Equation 8, we also apply
another z-score to normalize the final EnDex score according to standard
deviation, so that we can easily sample our data from it. A score with higher
standard deviation will imply a higher probability that the sample is engaging
according to our metric. We apply a cut-off to sample high probability
engaging and non-engaging samples, and arrive at the RED dataset.

###  A.2 Model training details

Our RED-Test set contains 300 human labeled data. The train validation split
during training is 0.8 and 0.2.

We used 4 Nvidia A6000 GPUs for training, and 1 Nvidia A6000 GPU for
inference. The average runtime for training one model is 2 minutes per epoch,
and inference time is in seconds, negligible for the testset. The estimated
energy cost per model is, assuming per second gpu energy cost of 245W:
245W*4*60 = 58800 per model.

We trained our model for 2 epochs, and only save the best checkpoints, with
learning rate of 5e-5 with no extensive hyperparameter search.

We used specificity and question examination inspired from (See et al., 2019);
USL-H (Phy et al., 2020) and PredEn is taken from a GitHub
repo555https://github.com/exe1023/DialEvalMetrics and modified to use a local
bert-base-uncased since the original ‘bert-as-service‘ code no longer
functions.

The Formula for calculating each dimension is given in the following:

  * •

Reply Engagement; The raw Reply Engagement score(re) is just the reply counts
of a post. Popularity value adjusted RE score is given by PVRE in equation
(6), where $M_{pv}$ and $M_{re}$ are the median of popularity value and reply
counts in the entire dataset. Please refer to equation (1) for calculation of
the popularity value.

$\textrm{PVRE}(re)=re+\frac{M_{pv}}{M_{\textrm{re}}}*\frac{re}{\textrm{PV(re)}}*re$
(5)

  * •

Behavioral Engagement; The raw BE score of a certain post(be) is obtained by
subtracting downvotes to upvotes and set to 0 if given controversy flag.
Popularity value adjusted BE score is given by PVBE in equation
(LABEL:eq:pvbe), where $M_{pv}$ and $M_{be}$ are the median of popularity
value and raw BE score in the entire dataset. Please refer to equation (1) for
calculation of the popularity value.

$\textrm{PVBE}(be)=be+\frac{M_{pv}}{M_{\textrm{be}}}*\frac{be}{\textrm{PV(be)}}*be$
(6)

  * •

Attentional Engagement; It is calculated using maximum information
specificity, or the the maximum number of non-stopword tokens in a post’s
replies, and whether its children posts are editted; t is the maximum reply
specificity, and e stands for number of edited replies.

$\textrm{AE}(x)=t+10*e$ (7)

  * •

Emotional Engagement; The EE score is the aggregate probability for all
positive emotion categories, produced by the go-emotion classifier (Demszky et
al., 2020).

###  A.3 Submodular Normalization and z-score Clustering

After we obtained the sub-dimension scores, we want to aggregate them into a
single normalized EnDex Score, and lastly cluster them into positive and
negative sets to train a binary classifier. The formulas are list in the
following:

$\textsc{EnDex}(x)=\Bigg{(}\displaystyle\sum_{n=1}^{3}w_{i}*\frac{x_{i}}{x_{i}+\alpha_{i}}\Bigg{)}+w_{\textrm{EE}}*x_{\textrm{EE}},$
(8)

$N$ is the normalized score for sample $x$, $x_{i}$ is $x$’s raw score on
dimension $i$, where $i\in\\{RE,BE,AE\\}$; $\alpha_{i}$ is the median of
$i$-th dimension; $w_{i}$ is the weight for $i$ dimension; $w_{\textrm{EE}}$
is the weight for EE dimension. The weight can be tuned for your own usage of
RED;

A confidence threshold $\kappa$(ours is 1) needs to be picked, which means
that we regard samples that fall between $\kappa$ standard deviation from mean
as uncertain, and are thus discarded. And we cluster positive and negative
samples using the following equation (9).

$Polarity(x)=\begin{cases}1&\text{if }z\\_score(x)>\kappa\\\ 0&\text{if
}z\\_score(x)<-\kappa\end{cases}$ (9)

Figure 3: The screenshot of the task description of our Amazon MTurk
questionnaire. We have prepared instructions, demonstrations, and proper
warning of offensive content. Figure 4: The screenshot of the labeling area of
our Amazon MTurk questionnaire. Each pair will be labelled by three
annotators.

###  A.4 Annotation data and test data

We performed annotation on Amazon Mechanical Turk, and selected annotators
based in the United State; in implemented restrictions to annotator with 98%
pass rate. We give four examples and clear instruction for the task carried
out. A screenshot of our annotation interface is provided below.

Table 3 gives summary of the evaluation datasets we used.

Dataset | # of Samples | Context Length | Response Length | Source | Agreement Rate
---|---|---|---|---|---
BETTER | 297 | 6 | 8 | Human | N/A
PREDENG-600 | 600 | 12 | 14 | Human+Bot | 0.51
FED-ENG | 261 | 26 | 12 | Human+Bot | N/A
RED-TEST | 150 | 16 | 17 | Human | 0.34
GRADE | 150 | 12 | 14 | Human | N/A

Table 3: Dataset Statistics for the 5 golden evaluation sets, with number of
samples, context-length, response length, and if applicable, inter-annotator
agreement rate.