# CDF-S XT1: The off-axis afterglow of a neutron star merger at $z=2.23$

Nikhil Sarin11affiliation<EMAIL_ADDRESS>, Gregory Ashton, Paul D.
Lasky, Kendall Ackley, Yik-Lun Mong, Duncan K. Galloway  School of Physics and
Astronomy, Monash University, Vic 3800, Australia
OzGrav: The ARC Centre of Excellence for Gravitational-wave Discovery,
Clayton, Victoria 3800, Australia
Royal Holloway University of London, Egham, Surrey, TW20 0EX, U.K.

###### Abstract

CDF-S XT1 is a fast-rising non-thermal X-ray transient detected by Chandra in
the Deep-Field South Survey. Although various hypotheses have been suggested,
the origin of this transient remains unclear. Here, we show that the
observations of CDF-S XT1 are well explained as the X-ray afterglow produced
by a relativistic structured jet viewed off-axis. We measure properties of the
jet, showing that they are similar to those of GRB170817A, albeit at
cosmological distances. We measure the observers viewing angle to be
$\theta_{\textrm{obs}}=10^{\circ}\pm 3^{\circ}$ and the core of the ultra-
relativistic jet to be $\theta_{\textrm{core}}=4.4^{\circ}\pm 0.9^{\circ}$,
where the uncertainties are the $68\%$ credible interval. The inferred
properties and host galaxy combined with Hubble, radio, and optical non
detections favour the hypothesis that CDF-S XT1 is the off-axis afterglow of a
binary neutron star merger. We find that other previously suggested hypotheses
are unable to explain all properties of CDF-S XT1. At a redshift of $z=2.23$,
this is potentially the most distant observed neutron star merger to date and
the first orphan afterglow of a short gamma-ray burst. We discuss the
implications of a binary neutron star merger at such a high redshift for the
star-formation rate in the early Universe, the nucleosynthesis of heavy
elements, and the prospect of identifying other off-axis afterglows.

###### Subject headings:

gamma-ray burst: general, X-rays: burst

citnum

##  1\. Introduction

Gamma-ray bursts are highly energetic explosions caused by either the collapse
of a massive star or the merger of a neutron star binary. While the mechanism
behind the prompt gamma-ray emission is still uncertain (e.g., Pe’er, 2015),
the weaker broadband afterglow emission observed extensively in X-rays,
optical and radio has been studied in significant detail. This broadband
afterglow is widely believed to be synchrotron radiation produced from the
interaction of a relativistic jet with the surrounding interstellar medium
(e.g., Sari  _et al._ , 1999).

It has long been accepted that gamma-ray bursts are collimated into an ultra-
relativistic jet (e.g., Kumar and Zhang, 2015), a point spectacularly
confirmed by very-long baseline interferometry of GRB170817A (e.g., Mooley
_et al._ , 2018), the gamma-ray burst that accompanied the first
gravitational-wave observation of a binary neutron star merger (Abbott  _et
al._ , 2017a, b). In general, as the jet ploughs into the surrounding
interstellar medium, the initial high velocity of the material implies the
emission is relativistically beamed and can only be seen for an observer close
to the emission axis. As that material slows, the beaming cone broadens, and
off-axis observers begin to see the burst afterglow. This relativistic beaming
necessarily implies the existence of orphan afterglows, where the observer
misses the early prompt emission but sees the afterglow as the jet broadens.

Orphan afterglows are notoriously difficult to find. Despite numerous searches
in multiple diverse data sets, there are only three putative long gamma-ray
burst orphan afterglow candidates, one observed in radio (Marcote  _et al._ ,
2019) and two in optical (Ho  _et al._ , 2020). In 2014, the Chandra Deep-
Field South Survey detected a fast-rising transient known as CDF-S XT1 (Bauer
_et al._ , 2017). CDF-S XT1 is associated with a host galaxy in the CANDELS
survey (Grogin  _et al._ , 2011) with a photometric redshift of
$z=2.23^{+0.98}_{-1.84}$ ($2\sigma$). With a non-thermal spectrum, a photon
index $\gamma\approx 1.43$ (Bauer  _et al._ , 2017) (i.e., a declining
spectrum), and non-detections in optical and radio, the transient properties
are unlike many other high-energy transients (Bauer  _et al._ , 2017). Since
its discovery, various hypotheses have been suggested to explain the
observations and properties of CDF-S XT1. Hypotheses such as a supernova-shock
breakout (Bauer  _et al._ , 2017; Alp and Larsson, 2020), a tidal disruption
event of a white dwarf with an intermediate-mass black hole (Bauer  _et al._ ,
2017; Peng  _et al._ , 2019), and the trapped emission from a millisecond
magnetar (Sun  _et al._ , 2019) have all been explored previously to varying
success.

In this Letter, we show that CDF-S XT1 observations can instead be well
interpreted as the X-ray afterglow produced by a relativistic jet viewed off-
axis. In Sec. 2 we describe our new method for fitting the raw photon count
data with detailed structured jet models. In Sec. 3 we discuss the results and
present evidence for the observations being explained as the X-ray afterglow
of a gamma-ray burst viewed off axis. In Sec. 4 we utilise the multi-
wavelength non detections and the spectrum to build evidence for CDF-S XT1
being the X-ray afterglow of a short gamma-ray burst produced by the merger of
a neutron star binary. In Sec. 5 we discuss the weaknesses of the alternative
interpretations. We conclude and discuss the implications of our results in
Sec. 6.

##  2\. Method

CDF-S XT1 was observed on October 1, 2014. The first photon arrived
$1.68\times 10^{4}\,\mathrm{s}$ into the observation period (Bauer  _et al._ ,
2017). During the subsequent $\sim 100\,\mathrm{s}$, the photon count rate
sharply increased, before decreasing over the next $\sim 10^{5}\,\mathrm{s}$.
We develop a new method for analysing the data of CDF-S XT1 to fit the data
with various models and infer parameters of the burst including the time of
the prompt emission, density of the interstellar medium, the opening angle and
energetics of the relativistic jet, and the observer viewing angle. We use
individual photon arrival times from the source region of CDF-S XT1 as defined
in Bauer  _et al._ (2017) with a $0.26"$ spatial extraction region. We use the
Chandra Interactive Analysis of Observations (ciao) (Fruscione  _et al._ ,
2006) software package to extract $115$ source photons in the
$0.3-7\,\mathrm{keV}$ energy range. Given the location of the transient and
the point-spread function, this observation is unlikely to suffer from pileup
(two incident photons count as one, or get rejected) (Bauer  _et al._ , 2017).
There are no other photons in this region across the entire observation.

The first photon has been the source of debate, with Sun  _et al._ (2019)
considering that photon to be consistent with the background. We analyse the
entire $\approx 5\times 10^{4}\,\mathrm{s}$ observation epoch to determine
whether this photon is consistent with the background. We calculate the
background rate from an off-source region to be $3\times
10^{-5}~{}\mathrm{cts~{}s^{-1}}$. Given this rate, the probability that one
background photon arrives within $\sim 50$s of all the other photons is
$1\times 10^{-4}$, suggesting that it is unlikely this photon is consistent
with the background. Conversely, given our fitting allows us to estimate the
source rate, the probability that this photon is from the source is $\gtrsim
74$%.

We use the raw photon time of arrivals from Chandra that are binned with a
constant bin size of $\Delta T=3.2\,\mathrm{s}$ across the entire observation
period. We perform Bayesian inference using a Poisson likelihood on this
binned data. Our likelihood is,

$\mathcal{L}(\vec{d}|\vec{\theta})=\prod_{i}\frac{r_{i}\Delta Te^{-r\Delta
T}}{k_{i}!},$ (1)

where $r_{i}$ is the rate of photons in the $i^{\rm th}$ bin, $k_{i}$ is the
number of photons in that bin, and $\Delta T$ is the bin size. We calculate
the rate in the $i^{\rm th}$ bin using,

$r_{i}=\frac{\sigma_{i}\int_{0.3\,\mathrm{keV}}^{7.0\,\mathrm{keV}}F(t_{i},\vec{\theta},\nu)d\nu}{E_{i}}+\Lambda_{\rm{background}},$
(2)

where $\sigma_{i}$ is the effective area of the detector, $E_{i}$ is the
averaged energy of the photons in the given area, $F(t_{i},\vec{\theta},\nu)$
is the flux density of our model at time $t_{i}$ at a given frequency $\nu$
with associated vector of parameters $\vec{\theta}$, and
$\Lambda_{\rm{background}}$ is the background rate. We note that this
expression is an approximation and calculation of the true rate requires
modelling the detector response and point-spread function. However, the
uncertainties caused by this approximation are small compared to model
uncertainties. We calculate the flux density using the smooth power-law
structured jet model in afterglowpy (Ryan  _et al._ , 2020) with inverse
Compton emission and jet spreading. We note that we also perform inference
without these effects and our conclusions do not change. The smooth power-law
structured jet model defines the energy distribution of the jet as,

$E(\theta_{\mathrm{obs}})=E_{0}\left(1+\frac{\theta_{\mathrm{obs}}^{2}}{\beta\theta_{\mathrm{core}}^{2}}\right)^{-\beta/2}.$
(3)

Here $\beta$, is the exponent dictating the slope of the power-law jet
structure. The Lorentz factor of the jet follows
$E_{\theta_{\textrm{obs}}}^{1/2}$, with an initial Lorentz factor
$\Gamma_{0}$. There is an additional parameter, $\theta_{\rm{wing}}$ which is
a truncation angle outside of which the energy is initially zero. This
structured jet interacts with the surrounding interstellar medium accelerating
a fraction of electrons, $\xi_{n}$, with a fraction of the total thermal
energy, $\epsilon_{e}$ and fraction of the thermal energy in the magnetic
field, $\epsilon_{b}$. The synchrotron radiation produced by these electrons
is responsible for the emission we observe. We set the flux before the onset
of the burst $T_{\rm{start}}$ to zero. We sample over the redshift by putting
a uniform prior on $z$ between $0.39-3.21$ corresponding to the 95% credible
interval from the host galaxy photometric redshift. We use Planck-15 cosmology
to convert the redshift into a luminosity distance to the source (Planck
Collaboration  _et al._ , 2016).

To reduce the computational cost, we use a larger binning size of $\Delta
T=64\,\mathrm{s}$ compared to the raw data. We verify there is no systematic
bias introduced by this larger bin size by repeating the calculation with
different bin sizes with afterglowpy. We infer consistent parameters across
$\Delta T=64,128,256\,\mathrm{s}$. We note that bin sizes larger than
$256\mathrm{s}$ destroy the structure of the data and change the results. To
ensure there is no bias introduced by not using the raw data, we also train a
neural network algorithm to compute the flux density, calibrating it to the
output of afterglowpy. This trained model can be evaluated approximately three
orders of magnitude faster than afterglowpy making the analysis tractable with
the raw data, but it introduces a systematic uncertainty due to the nature of
machine learning. We infer consistent posteriors (at $1-\sigma$) with the
neural network model on the raw data binned at $\Delta T=3.2\,\mathrm{s}$ and
the larger bin sizes with afterglowpy. We have also verified that our machine
learning model is accurate by performing tests on untrained data, simulating
signals generated with afterglowpy and recovering them with consistent
parameters with the trained neural network model using the infrastructure
described here. We will present details of the neural network model in future
work.

##  3\. An off-axis gamma-ray burst

Several properties of CDF-S XT1 are consistent with a gamma-ray burst
afterglow, including the event rate (Bauer  _et al._ , 2017; Sun  _et al._ ,
2019). Firstly, the energetics and high redshift ($z=2.23^{+0.98}_{-1.84}$)
demands a highly energetic transient, with upwards of $10^{50}\,\mathrm{erg}$
of energy. The non-thermal spectrum immediately rules out any thermal
transient such as a supernova, while the fast rise and slow decay rule out a
persistent non-thermal source. These properties, in particular the rise and
slow decay are indicative of an off-axis gamma-ray burst afterglows (Granot
_et al._ , 2002). Moreover, the declining spectrum and rising lightcurve is
likely due to misaligned evolution i.e.,
$\theta_{\rm{obs}}>\theta_{\rm{core}}$ (e.g., Ryan  _et al._ , 2020).

To investigate the above interpretation in detail, we first consider the X-ray
data. We apply our method described in Sec. 2 and fit the X-ray data of CDF-S
XT1. We infer the parameters of the system using Bilby (Ashton  _et al._ ,
2019) and the pyMultinest sampler (Feroz  _et al._ , 2009). In Tab. 1 we list
the full set of parameters $\vec{\theta}$, their descriptions, their
associated priors used in the analysis, and their posteriors with $68\%$
posterior credible interval.

Table 1Parameters associated with the smooth power-law structured jet model along with a brief description, the prior used in our analysis and the posterior. We note that the posterior values quoted here are from analysis with the neural network model run on data binned at $\Delta T=3.2\mathrm{s}$. Parameter [unit] | Description | Prior | Posterior
---|---|---|---
$T_{\textrm{start}}$ [s] | burst start time into the observation | $\textrm{Uniform}[10,16800]$ | $16784^{+9}_{-13}$
$z$ | redshift | $\textrm{Uniform}[0.39,3.21]$ | $1.7^{+0.8}_{-0.7}$
$\Gamma_{0}$ | initial Lorentz factor | $\textrm{Uniform}[1,1000]$ | $420^{+300}_{-240}$
$\Lambda_{\rm{background}}$ [counts/s] | background rate | $\log\textrm{Uniform}[10^{-6},10^{-4}]$ | $10^{-4}\pm 10^{-5}$
$\theta_{\textrm{obs}}$ [∘] | observers viewing angle | $\textrm{Cosine}[0,0.7]$ | $10^{\circ}\pm 3^{\circ}$
$\log_{10}(E_{\textrm{iso}}/\rm{erg})$ | isotropic-equivalent energy | $\textrm{Uniform}[10^{44},10^{54}]$ | $52.2\pm 0.5$
$\theta_{\textrm{core}}$ [∘] | half-width of jet core | $\textrm{Uniform}[0.01,0.1]$ | $4.4^{\circ}\pm 0.9^{\circ}$
$\theta_{\textrm{wing}}$ [∘] | wing truncation angle of the jet | $\textrm{Uniform}[0.01,0.7]$ | $10^{\circ}\pm 3^{\circ}$
$\beta$ | power for power-law structure | $\textrm{Uniform}[0.5,10]$ | $4.7\pm 1.7$
$\log_{10}(n_{\textrm{ism}}/\mathrm{cm^{-3}})$ | number density of ISM | $\textrm{Uniform}[-5,2]$ | $1.8^{+0.6}_{-0.8}$
$p$ | electron distribution power-law index | $\textrm{Uniform}[2,3]$ | $2.2\pm 0.1$
$\log_{10}\epsilon_{e}$ | thermal energy fraction in electrons | $\textrm{Uniform}[-5,0]$ | $-0.7\pm 0.3$
$\log_{10}\epsilon_{b}$ | thermal energy fraction in magnetic field | $\textrm{Uniform}[-5,0]$ | $-0.8\pm 0.5$
$\xi_{N}$ | fraction of accelerated electrons | $\textrm{Uniform}[0,1]$ | $0.8\pm 0.2$

In Fig. 1, we plot the raw X-ray photon counts detected by Chandra in the
$0.3-7.0\,\mathrm{keV}$ band. For visualisation purposes, we bin the photon
arrival times with a bin size of 128 s. Times in Fig. 1 are referenced to our
estimated burst start time. In red, we show model-predicted counts from 100
random draws of the posterior distribution. In the same Figure, we also show a
schematic view of the physics creating the various features of the light
curve. In particular, the jet slowing and subsequent broadening of the beaming
cone give rise to the sharp increase in photon count seen approximately 100 s
after the burst.

Figure 1.— CDF-S XT1 data and model light curves. X-ray counts as a function
of time binned at $\Delta T=128$s intervals in blue, and posterior predictions
for the off-axis short gamma-ray burst model in red. The schematic illustrates
the physical process responsible for the different observational epochs. In
epoch A, the observer sees no emission as the ultra-relativistic jet (blue
cone) and mildly relativistic wings (grey cone) are beamed away from Earth’s
line of sight. As the structured jet slows, its beaming cone broadens such
that the mildly relativistic wings are now in Earth’s line of sight. At this
point, the observer on Earth starts to see X-ray photons from the source
(epoch B). As the jet slows down further it continues to broaden at which
point the jet core becomes visible to the observer on Earth and the photon
rate peaks (epoch C). The subsequent gradual decay is a product of further
slowing down of the jet and subsequent broadening of the jet.

We measure the observer angle to be $\theta_{\rm obs}=10^{\circ}\pm 3^{\circ}$
and the core of the jet to be $\theta_{\rm core}=4.4^{\circ}\pm 0.9^{\circ}$.
This implies that CDF-S XT1 was observed from outside the jet-core naturally
explaining the lack of prompt gamma-ray emission due to relativistic beaming.

We derive the start time of the burst to be
$T_{\rm{start}}=16784^{+9}_{-13}\,\mathrm{s}$ into the observation which
implies that the afterglow peaks approximately $150\,\mathrm{s}$ after the
predicted time of the prompt emission. Within our off-axis interpretation,
this peak timescale $t_{\rm{peak}}$ is dictated by the timescale for the edge
of the ultra-relativistic core to start becoming visible to the off-axis
observer i.e., for $\Gamma(\theta_{\rm{obs}}-\theta_{\rm{wing}})\sim 1$. A
back-of-the-envelope estimate for this peak timescale is (e.g., Nakar and
Piran, 2021)

$t_{p}\approx
130\,\mathrm{d}\left(\frac{E}{10^{51}\,\mathrm{erg}}\frac{10^{-3}\,\mathrm{cm^{-3}}}{n_{\rm{ism}}}\right)^{1/3}\left(\frac{\theta_{\rm{obs}}-\theta_{\rm{wing}}}{15^{\circ}}\right)^{2}.$

From our estimated parameters, this gives $t_{p}\sim 700\,\mathrm{s}$, which
is comparable to the observations.

We note that this peak timescale is different to the peak timescale seen for
GRB170817A, which came $160\,\mathrm{d}$ following the prompt. That peak time
was the jet break time. For CDF-S XT1 this can be estimated by (e.g., Granot
_et al._ , 2018)

$t_{\rm{break}}=70\,\mathrm{d}E_{51}^{1/3}n_{0}^{-1/3}\theta_{\rm obs}^{2}.$

From our estimated parameters this is $t_{\rm{break}}\sim 4\,\mathrm{days}$,
which is significantly shorter than the $t_{\rm{break}}$ of GRB170817A.
However, this discrepancy in timescales is not inconsistent and can be
ascribed to the smaller viewing angle $\theta_{\rm{obs}}$ and the higher
interstellar medium density $n_{\rm ism}$ for CDF-S XT1.

Our inferred estimates for the isotropic energy is also consistent with a
back-of-the-envelope estimate. The reported fluence for CDF-S XT1 is
$4.2^{+3.5}_{-0.2}\times 10^{-9}\mathrm{erg~{}cm^{-2}}$ (Bauer  _et al._ ,
2017). At a redshift of $z=2.23$, this implies a jet with energy $\sim
10^{50}\,\mathrm{erg}$, which is comparable to the estimated jet energy from
our analysis. Comparing the energetics CDF-S XT1 to X-ray afterglows seen in
other gamma-ray bursts, the total energy of CDF-S XT1 would be lower by two
orders of magnitude compared to the dimmest X-ray afterglows we see unless
CDF-S XT1 is at $z\gtrsim 2$. Although, since CDF-S XT1 is observed off-axis,
some difference in brightness is to be expected.

Although Figure 1 shows the fit on a $128\,\mathrm{s}$ binned timescale, we
infer similar parameters across all bin sizes i.e., our conclusion is robust
to the choice of binning. Similarly, even accounting for the redshift
uncertainty, our results and conclusion are quantitatively similar: CDF-S XT1
is consistent with being an orphan afterglow of an off-axis gamma-ray burst.
We note that the posterior values mentioned above and those listed in Tab. 1
are marginalised over the redshift uncertainty.

To probe the nature of CDF-S XT1, the source region was searched in several
other electromagnetic bands. In optical, the Very Large Telescope (VLT)
observed the location of the transient $80$ minutes after the arrival of the
first photon, and again $18$ days later, setting stringent $r$-band upper
limits (Treister  _et al._ , 2014; Bauer  _et al._ , 2017). The Hubble Space
Telescope (HST) searched the source region $111$ days after the first X-ray
observations, setting the most sensitive limits for the source region in the
$F110W$ filter (Bauer  _et al._ , 2017). In radio, the Australian Telescope
Compact Array (ATCA) observed the source region seven days after the first
X-ray photons, setting limits at different radio frequencies (Burlon  _et al._
, 2014). In gamma rays, neither Swift nor Fermi had coverage in the direction
of CDF-S XT1 for the few hours surrounding the event (Bauer  _et al._ , 2017).
None of these observations found a counterpart to CDF-S XT1. To correctly
identify the nature of CDF-S XT1, we must be able to explain these non-
detections. Gamma-ray burst afterglow models make reliable predictions about
emission across all of these bands. Therefore these non-detections can be used
to verify our proposed off-axis gamma-ray burst hypothesis.

We evaluate the predicted flux density light curves in optical at 640 nm and
1179 nm, and radio at 5 GHz, corresponding to the VLT $r$-band, HST
$F110W$-band, and ATCA observations, respectively. We show these lightcurves,
as well as the upper limits in optical and radio in Fig. 2 at a fixed redshift
$z=2.23$ corresponding to the median value of the host galaxy photometric
redshift for easier comparison. The left and middle panels show the optical
light curves for $100$ different samples randomly drawn from our posterior
distribution as predicted in the VLT $r$-band and the HST $F110W$-band, while
the right panel shows the corresponding flux density in the radio band. The
black arrows indicate the upper limits set by VLT, HST and ATCA in the left,
middle and right panels, respectively.

We estimate the total extinction from its association with the neutral
hydrogen column density, which we infer from the spectra. We infer a total
extinction of $A_{V}=5.96^{+6.98}_{-3.22}$ at a redshift $z=2.23$, largely due
to the strong redshift dependence of the inferred hydrogen column density
(Bauer  _et al._ , 2017). In Fig. 2, These observed upper limits are above the
corresponding optical and radio light curve predictions, consistent with the
non-detections in these electromagnetic bands. We note that in Fig. 2 we have
used the median value of $A_{V}=5.96$ as the total extinction in the $r$-band.
The $F110W$-band and radio do not suffer such high extinction. Given our
uncertainty on the extinction, we estimate that there is a less than $10^{-4}$
% probability that the optical afterglow in $r$-band would have been
detectable.

Figure 2.— Optical and radio upper limits are above predictions from our
model. Flux density in optical ($\lambda=640$nm; left panel, and
$\lambda=1179$nm; middle panel) and radio ($5$GHz; right panel), with arrows
representing VLT, HST and ATCA upper limits, respectively (Bauer  _et al._ ,
2017). Each red curve is a predicted light curve, calculated by drawing
parameters randomly from our posterior probability distribution. The arrows
correspond to upper limits derived from VLT, HST and ATCA observations
respectively (Treister  _et al._ , 2014; Burlon  _et al._ , 2014). The blue
points are data from the kilonova associated with GW170817 but scaled assuming
a redshift of $z=2.23$. Likewise, the green points are the data for supernova
SN1998bw but scaled to a redshift of $z=2.23$. Our posterior predictive models
are consistent with the interpretation of this event as a compact object
coalescence, but not necessarily with that of a supernova.

In gamma rays, neither Swift nor Fermi had coverage in the direction of CDF-S
XT1 for the few hours surrounding the event (Bauer  _et al._ , 2017). Our
inferred parameters from fitting the X-ray data and the large redshift implied
from the identification of the host galaxy imply no prompt emission would have
been detected from this system even with gamma-ray coverage. Furthermore, the
inferred parameters can explain the non-detections in other electromagnetic
bands. This analysis above suggests CDF-S XT1 is potentially the first orphan
afterglow detected in X rays and one of the most distant orphan afterglow
candidates ever.

##  4\. A compact object merger?

Having established that CDF-S XT1 is potentially an orphan X-ray afterglow of
a gamma-ray burst we now turn to answer whether CDF-S XT1 is an orphan
afterglow of a long or short gamma-ray burst? In other words, was this the
orphan afterglow due to the collapse of a massive star, or the merger of a
neutron star binary? Here, we investigate the various properties of CDF-S XT1
and the multi-wavelength observations to answer this question.

A short gamma-ray burst would likely be accompanied by an optical/infrared
kilonova, such as AT2017gfo for GW170817 (e.g., Abbott  _et al._ , 2017b) and
several other short gamma-ray bursts. Similarly, a long gamma-ray burst should
be accompanied by a core-collapse supernova, such as the case for GRB980425
accompanied by SN1998bw (e.g., Clocchiatti  _et al._ , 2011) and GRB111209A
accompanied by SN2011kl (e.g., Greiner  _et al._ , 2015). In the left and
middle panels of Fig. 2 we show the prediction for a GW170817-like kilonova
(blue points) and a SN1998bw-like supernova (green points) if either had
accompanied CDF-S XT1 at $z=2.23$. We pick SN1998bw as it is one of the
dimmest supernovae to accompany a long gamma-ray burst, and largely due to its
proximity, the best observed. In particular, we take data from AT2017gfo and
scale it to a distance of $z=2.23$. The VLT upper limits being above the blue
points indicates that the kilonova associated with GW170817 would not have
been seen by VLT had it been at a redshift of $z=2.23$. Moreover, that the red
predictive curves are consistent with the GW170817-like kilonova indicate too
that, if CDF-S XT1 was a short gamma-ray burst with associated kilonova, one
would not have expected to see it with VLT observations.

The green dots on the left and middle panels of Fig. 2 take the $r$ and
$i$-band observations from SN1998bw, respectively, and scale them to a
distance of $z=2.23$. We note that SN1998bw was not observed by HST and the
$i$-band observations are likely a lower estimate of what SN1998bw would have
looked like realistically with HST. SN1998bw was a relatively dim supernova
compared to other supernovae that accompany long gamma-ray bursts (e.g.,
Clocchiatti  _et al._ , 2011). That the green dots are close to the HST upper
limit implies an SN1998bw-like supernova would have been marginally
detectable. Had a brighter supernova been associated with CDF-S XT1, it would
have been potentially observed by the HST. This deep constraint from HST rules
out most potential supernovae up to a redshift $z\sim 2$ (Richardson  _et al._
, 2014). A similar point could be made for the VLT observations if this system
does not suffer from severe extinction. If a supernova accompanied CDF-S XT1,
it would have had to be relatively dim to not be observed by HST. We note that
there have been observations of long gamma-ray bursts without an accompanying
supernova, however, these are likely due to observational selection effects
(Lyutikov, 2013). This non-detection of a supernova adds weight to the
hypothesis that the progenitor of CDF-S XT1 was the merger of a neutron star
binary rather than the collapse of a massive star.

The star-formation rate ($1.15\pm 0.04~{}M_{\odot}$yr -1) of the putative host
galaxy CANDELS $28573$ and the off-set ($0.13"$) are also consistent with
other short gamma-ray bursts (Bauer  _et al._ , 2017; Sun  _et al._ , 2019).
We note that the uncertainty in the location of the transient is smaller than
the size of a typical galaxy, such that the effect of the incompleteness of
galaxy catalogues does not significantly affect this association. Furthermore,
given the implied high redshift, a requirement based on energetics, the effect
of the supernova kick also does not add significant uncertainty to affect this
association. We note that our inferred interstellar medium density suggests
that the gamma-ray burst was not significantly offset from the host. The mass
of the host galaxy is on the smaller end for short gamma-ray bursts, while the
star formation rate is comparable. However, the limited number of known short
gamma-ray bursts beyond redshift $z\approx 2$ implies this is not a
statistically robust statement (Sun  _et al._ , 2019).

Our inferred interstellar-medium density
$n_{\textrm{ism}}=1.8^{+0.6}_{-0.8}\,\mathrm{cm^{-3}}$ is more akin to long
gamma-ray bursts than short (Fong  _et al._ , 2015). However, approximately 5
to 20% of short gamma-ray bursts (Fong  _et al._ , 2015) have $n_{\rm
ism}\gtrsim 1\,\mathrm{cm^{-3}}$. Moreover, systematic studies of
interstellar-medium densities to date typically fix the energy fraction in the
magnetic field and electrons, which systematically underestimates $n_{\rm
ism}$. We do not fix these parameters but instead marginalise over this
uncertainty.

Finally, we consider the implied spectrum of CDF-S XT1. Gamma-ray bursts show
an empirical correlation between the isotropic gamma-ray energy and the rest-
frame peak energy in gamma-rays. For long gamma-ray bursts this is known as
the Amati correlation (Amati, 2006), which differs from that measured for
short gamma-ray bursts with observed redshifts. Therefore, identifying the
consistency with the Amati relation offers another way to probe whether the
gamma-ray burst is long or short (e.g., Amati, 2006; Minaev and Pozanenko,
2020).

We use our inferred isotropic energy from the afterglow and the observed
distribution of prompt emission gamma-ray efficiencies (Fong  _et al._ , 2015)
to determine the isotropic energy in gamma rays emitted from CDF-S XT1,
finding $\log_{10}(E_{\gamma,\rm{iso}}/\mathrm{erg})=50\pm 0.6$. We use the
relation between the X-ray photon index and rest-frame peak energy (Virgili
_et al._ , 2012) to infer the rest-frame peak energy,
$E_{\rm{peak,z}}=830^{+1200}_{-500}~{}\mathrm{keV}$. We emphasise that this
analysis is agnostic to whether a gamma-ray burst is from the collapse of a
massive star or the merger of a neutron star binary. In Fig. 3, we show that
the inferred parameters for CDF-S XT1 are inconsistent with observed long
gamma-ray bursts and the Amati relation. However, these parameters are
consistent with several observed short gamma-ray bursts. This adds significant
weight to the hypothesis that CDF-S XT1 is the orphan afterglow of a short
gamma-ray burst produced in the merger of a neutron star binary.

Figure 3.— The observed isotropic gamma-ray energy and peak rest-frame energy
of CDF-S XT1 (blue) and a sample of observed long (black) and short (green)
gamma-ray bursts (Minaev and Pozanenko, 2020). The empirical Amati relation
and associated uncertainty is shown in red. The uncertainty on CDF-S XT1 is
obtained by propagating uncertainties in the spectral slope, the inferred
kinetic energy and the distribution of gamma-ray radiative efficiencies (e.g.,
Fong  _et al._ , 2015). CDF-S XT1 is inconsistent with the Amati relation,
while being consistent with observed short gamma-ray bursts, adding weight to
our argument that CDF-S XT1 is the X-ray afterglow of a binary neutron star
merger. Note that the Amati relation is derived from a significantly larger
sample of observed gamma-ray bursts than shown here (Minaev and Pozanenko,
2020)

##  5\. Alternative interpretations

Three alternate explanations were offered for this transient in the discovery
paper (Bauer  _et al._ , 2017) and explored briefly elsewhere (Sun  _et al._ ,
2019; Peng  _et al._ , 2019; Alp and Larsson, 2020). Here we critically assess
each of these scenarios, providing a summary of the different scenarios and
their failure to explain observations in Tab. 2.

Table 2Class of systems and emission mechanisms and whether they can explain the different observational constraints on CDF-S XT1. System | Emission mechanism | Spectrum | Luminosity | Optical constraints | Host-galaxy properties
---|---|---|---|---|---
Tidal disruption | Tidal disruption event | ? | x | ✓ | ✓
| Jetted tidal disruption event | ? | x | ? | ✓
Collapse of massive star | Supernova-shock breakout | x | x | x | ✓
| Long gamma-ray burst orphan afterglow | x | ✓ | ? | ✓
Compact-object coalescence | Short gamma-ray burst orphan afterglow | ✓ | ✓ | ✓ | ✓
| Kilonova with energy injection | ? | ✓ | ? | ✓

It has been argued that CDF-S XT1 could be a supernova-shock breakout from a
core-collapse supernova (Bauer  _et al._ , 2017; Alp and Larsson, 2020) in
which the shock wave from the supernova blasts out of the progenitor star’s
surface. While the timescale and steep-rise/gradual decay of the light curve
is consistent with such phenomena, the spectrum is not. Supernova shock-
breakouts are thermal transients and the radiation they produce is typically
soft X rays or ultra-violet depending on the temperature of the system (e.g.,
Tominaga  _et al._ , 2011). For typical shock breakouts, the dominant emission
is expected at $0.01-1\,\mathrm{keV}$ i.e., a thermal spectrum, inconsistent
with the non-thermal spectrum of CDF-S XT1. Furthermore, the peak luminosity
from typical supernova-shock breakouts (Tominaga  _et al._ , 2011) of
$10^{44}-10^{45}\,\mathrm{erg}$ is inconsistent with the inferred peak
luminosity of CDF-S XT1 at $z\gtrsim 0.4$. To explain the observations of
CDF-S XT1 at $z=2.23$, a supernova-shock breakout would have to be more
energetic than the most energetic shock-breakout observed by two orders of
magnitude. Importantly, a supernova-shock breakout will also be accompanied by
a supernova. However, the upper limits set by HST observations confidently
rule out a supernova at redshift $z\lesssim 2$ (Richardson  _et al._ , 2014).
The combination of these constraints suggests that the supernova-shock
breakout scenario is a poor explanation of the observations: the supernova-
shock breakout would have to be exceptionally bright but accompanied by a
relativity dim supernova and explain the hard spectrum. While we cannot
definitively rule out a supernova-shock breakout, the event would be like
nothing else we have seen before.

It has been suggested that CDF-S XT1 could be caused by a tidal disruption
event (Bauer  _et al._ , 2017; Peng  _et al._ , 2019). The fast rise and hard
X-ray flux of CDF-S XT1 imply that the only viable part of the parameter space
for a tidal disruption event that can explain the observations is a compact
white dwarf around an intermediate-mass black hole of around
$~{}10^{3}-10^{4}M_{\odot}$ (Bauer  _et al._ , 2017; Peng  _et al._ , 2019).
However, such a scenario is problematic because the Eddington limit for such a
system is $2-3$ orders of magnitude below what is required given the
associated redshift (Bauer  _et al._ , 2017). This discrepancy in energy could
be explained by a strongly beamed, jetted tidal disruption event. One would
expect such a scenario to also be observed in radio and optical as is the case
for other jetted tidal disruptions (Cenko  _et al._ , 2012) which also have
longer decay timescales. However, we do not observe radio and optical
counterparts and a long decay timescale is inconsistent with the decay of
CDF-S XT1. The only proposed way to generate sufficient magnetic fields to
create a jetted tidal disruption event is by anchoring the magnetic field to a
black hole (Tchekhovskoy  _et al._ , 2014) or through a highly magnetised star
from a recent merger (e.g., Mandel and Levin, 2015). In both cases, the
timescale for the rise and decay of the light curve would be significantly
longer than that observed in CDF-S XT1 and inconsistent with the observations.

Finally, a more exotic scenario was developed to explain both CDF-S XT1 and
CDF-S XT2 in a unified framework as being the emission from a millisecond
magnetar combined with the kilonova from the ejecta of a neutron star merger
(Sun  _et al._ , 2019). In this scenario, CDF-S XT1 was a neutron star merger
that produced a remnant neutron star that acted as a central engine injecting
energy into the surrounding ejecta. To explain the lack of detection in
optical, such a system would need to be unusually dim given the additional
energy injection, which is possible for comparatively lower ejecta masses than
for the kilonova associated with GW170817. However, the ejecta cannot be too
small either, as such a system would then not be optically thick and not
explain the rise seen in the light curve of CDF-S XT1. It is important to note
that the typical timescale for such a system to become optically thin is $\sim
9\,\mathrm{hr}$ (Metzger and Piro, 2014), which would imply such a model is
inconsistent with CDF-S XT1. However, there is substantial latitude in the
timescale given the uncertain opacity of the ejecta. Furthermore, the spectrum
from such a model would be a combination of the non-thermal spectrum from the
spin-down of the nascent neutron star and quasi-thermal spectrum from the
ejecta, which is at odds with the non-thermal spectrum inferred for CDF-S XT1.
Moreover, this model requires the formation of an infinitely stable neutron
star We note that to explain the observations of CDF-S XT1 with such a model,
Sun  _et al._ (2019) considered the first photon of the observation as being
consistent with the background—in Sec. 2 we argue that the first photon was
from CDF-S XT1, implying their fit to CDF-S XT1 is inconsistent with the data.

Even with an afterglow interpretation, there are other ways to interpret the
data. Within our interpretation, to explain the relatively fast rise and
subsequent decay, the data requires the jet to be truncated near the observer.
The angle at which the jet truncates is set by $\theta_{\rm{wing}}$, which we
keep as a free parameter. This implies that the jet energy drops sharply at an
angle close to the observer. Such a sharp drop is implicitly built into other
jet models such as the top-hat (e.g., Ryan  _et al._ , 2020). We analyse the
data by not allowing such a sharp drop in energy by fixing
$\theta_{\rm{wing}}=f\times\theta_{\rm{core}}$ where $f$ is some factor such
that the jet energy is $\approx 0$, i.e., we do not truncate the jet. Such a
model is perhaps more plausible than one with a sharp drop, although without
detailed hydrodynamical simulations or extensive observations of off-axis
gamma-ray bursts one cannot know for certain what the true jet structure is.
We note that past hydrodynamical simulations do produce sharp drops in jet
energy (e.g., Aloy  _et al._ , 2005).

If instead, we demand a jet structure that does not truncate, we would need to
explain the rise time with the pre-deceleration behaviour of an on-axis
relativistic jet. Fixing $f=8$, and fitting to the data, we find that the data
can be explained by having a weakly relativistic jet with Lorentz factor
$\Gamma_{0}=46^{+30}_{-20}$ and the observer being on-axis. In such a
scenario, the rise and decay can be explained by the deceleration of a low
Lorentz factor ($\lesssim 100$) jet. Such a relativistic jet may not produce
prompt gamma-rays due to the pair production threshold for gamma-ray
production, also known as the compactness problem. However, the threshold for
prompt gamma-ray emission is not well understood. Such gamma-ray bursts are
also known as failed gamma-ray bursts or dirty fireballs (e.g., Huang  _et
al._ , 2002; Rhoads, 2003). We note, however, that this solution is not
preferred by the data, with a Bayesian odds, assuming both hypotheses are
equally likely, of $\mathcal{O}=3.3$ in favour of truncating the jet. That is,
the solution suggesting the jet truncates near the observer is $\sim 3$ times
more favourable than not.

##  6\. Implications and Conclusions

Considering the limitations of the other three models posed in the literature
(see Sec. 5 and Tab. 2 for a summary) and the success of the orphan afterglow
interpretation presented here in explaining the various constraints, we
believe CDF-S XT1 is the orphan afterglow of an off-axis gamma-ray burst. This
makes CDF-S XT1 potentially one of the first orphan afterglows ever detected,
considering the uncertain nature of other candidates (e.g., Cenko  _et al._ ,
2012; Marcote  _et al._ , 2019; Ho  _et al._ , 2020; Sarin  _et al._ , 2021).
The method developed here is ideally suited for detecting orphan afterglows
from other X-ray surveys.

Our inferred parameters, the host galaxy properties, optical upper limits, and
inconsistency with the empirical Amati relation imply that CDF-S XT1 is more
likely to be the afterglow of a short gamma-ray burst afterglow than a long,
implying the most likely progenitor model is that of a binary neutron star
coalescence similar to the first multimessenger gravitational-wave detection
GW170817 (Abbott  _et al._ , 2017c, b). This has far-reaching implications.
Most notably, at a potential redshift of $z=2.23$, this is likely one of the
most distant short gamma-ray burst ever observed (GRB090426, with a redshift
of $z=2.61$, was initially classified as a short gamma-ray burst (Antonelli
_et al._ , 2009), but is now likely considered a long gamma-ray burst
(Levesque  _et al._ , 2010)).

Observing neutron star mergers so early in the Universe has implications for
our understanding of stellar evolution. The best-fit redshift for CDF-S XT1
places it before the peak of star formation, implying the binary that merged
to produce the gamma-ray burst must have formed and merged on a relatively
short timescale. Population synthesis studies estimate that the rate for short
gamma-ray bursts peaks in the redshift range $z=0.6$–1 (Wiggins  _et al._ ,
2018), similar to observations of short gamma-ray bursts with known redshifts
(Fong  _et al._ , 2015). The existence of CDF-S XT1 and its potential
progenitor being the merger of a neutron star binary suggests that the rate of
the merger of these systems at high redshifts is not negligible, which has
important implications for understanding binary stellar evolution and for
detecting gravitational waves from mergers of these objects. Furthermore, it
may imply that there are significant short delay-time binary neutron star
mergers, which may mean that a relatively high fraction of binary mergers
occurs in dense stellar environments such as globular clusters, or that
common-envelope evolution is more efficient at reducing the orbital separation
than previously realised. This also has implications for the offset of short
gamma-ray bursts from their host galaxy, particularly at higher redshifts.

A neutron star merger progenitor hypothesis for CDF-S XT1 also has
implications for heavy element nucleosynthesis. The multimessenger
observations of GW170817 confirmed that binary neutron star mergers are the
production sites of heavy $r$-process elements such as gold, which are
difficult to produce in ordinary supernovae. However, owing to the relative
lack of expected mergers at high redshifts, they are not believed to be the
only source (Siegel  _et al._ , 2019). If instead there are more neutron star
mergers at high redshifts than otherwise expected, this could imply that these
systems play a more critical role in the chemical evolution of the Universe.

##  7\. Acknowledgements

We are grateful to Carl Knox for the creation of the graphics in Fig. 1, and
Franz Bauer for help with collecting and processing the data. We also thank
Geoffrey Ryan for their help in understanding the afterglowpy model. We are
grateful to Ilya Mandel, Ryan Shannon, and Matthew Bailes for their valuable
comments and discussions. Computations were performed on the OzStar
supercomputer. N.S is supported by an Australian Government Research Training
Program (RTP) Scholarship. P.D.L. is supported through Australian Research
Council Future Fellowship FT160100112 and CE170100004. P.D.L and G.A are
supported by ARC Discovery Project DP180103155.

## References

  * Pe’er (2015) A. Pe’er, Advances in Astronomy 2015, 907321 (2015).
  * Sari  _et al._ (1999) R. Sari, T. Piran, and J. P. Halpern, ApJ 519, L17 (1999).
  * Kumar and Zhang (2015) P. Kumar and B. Zhang, Phys. Rep. 561, 1 (2015).
  * Mooley  _et al._ (2018) K. P. Mooley, A. T. Deller, O. Gottlieb, E. Nakar, G. Hallinan, S. Bourke, D. A. Frail, A. Horesh, A. Corsi, and K. Hotokezaka, Nature 561, 355 (2018).
  * Abbott  _et al._ (2017a) B. P. Abbott, R. Abbott, T. D. Abbott, F. Acernese, K. Ackley, _et al._ , Astrophys. J. 848, L13 (2017a).
  * Abbott  _et al._ (2017b) B. P. Abbott, R. Abbott, T. D. Abbott, _et al._ , ApJ 848, L12 (2017b).
  * Marcote  _et al._ (2019) B. Marcote, K. Nimmo, O. S. Salafia, Z. Paragi, J. W. T. Hessels, E. Petroff, and R. Karuppusamy, ApJ 876, L14 (2019).
  * Ho  _et al._ (2020) A. Y. Q. Ho, D. A. Perley, P. Beniamini, S. B. Cenko, S. R. Kulkarni, I. Andreoni, L. P. Singer, K. De, M. M. Kasliwal, C. Fremling, E. C. Bellm, R. Dekany, A. Delacroix, D. A. Duev, D. A. Goldstein, V. Z. Golkhou, A. Goobar, M. J. Graham, D. Hale, T. Kupfer, R. R. Laher, F. J. Masci, A. A. Miller, J. D. Neill, R. Riddle, B. Rusholme, D. L. Shupe, R. Smith, J. Sollerman, and J. van Roestel, ApJ 905, 98 (2020).
  * Bauer  _et al._ (2017) F. E. Bauer, E. Treister, K. Schawinski, S. Schulze, B. Luo, D. M. Alexander, W. N. Brand t, A. Comastri, F. Forster, R. Gilli, D. A. Kann, K. Maeda, K. Nomoto, M. Paolillo, P. Ranalli, D. P. Schneider, O. Shemmer, M. Tanaka, A. Tolstov, N. Tominaga, P. Tozzi, C. Vignali, J. Wang, Y. Xue, and G. Yang, MNRAS 467, 4841 (2017).
  * Grogin  _et al._ (2011) N. A. Grogin _et al._ , ApJS 197, 35 (2011), arXiv:1105.3753 [astro-ph.CO] .
  * Alp and Larsson (2020) D. Alp and J. Larsson, ApJ 896, 39 (2020).
  * Peng  _et al._ (2019) Z.-K. Peng, Y.-S. Yang, R.-F. Shen, L.-J. Wang, J.-H. Zou, and B.-B. Zhang, ApJ 884, L34 (2019).
  * Sun  _et al._ (2019) H. Sun, Y. Li, B.-B. Zhang, B. Zhang, F. E. Bauer, Y. Xue, and W. Yuan, ApJ 886, 129 (2019).
  * Fruscione  _et al._ (2006) A. Fruscione _et al._ , in  _Observatory Operations: Strategies, Processes, and Systems_, Society of Photo-Optical Instrumentation Engineers (SPIE), Vol. 6270 (2006) p. 62701V.
  * Ryan  _et al._ (2020) G. Ryan, H. van Eerten, L. Piro, and E. Troja, ApJ 896, 166 (2020).
  * Planck Collaboration  _et al._ (2016) Planck Collaboration _et al._ , A&A 594, A13 (2016), arXiv:1502.01589 [astro-ph.CO] .
  * Granot  _et al._ (2002) J. Granot, A. Panaitescu, P. Kumar, and S. E. Woosley, ApJ 570, L61 (2002).
  * Ashton  _et al._ (2019) G. Ashton, M. Hübner, P. D. Lasky, C. Talbot, K. Ackley, S. Biscoveanu, Q. Chu, A. Divakarla, P. J. Easter, B. Goncharov, F. Hernandez Vivanco, J. Harms, M. E. Lower, G. D. Meadors, D. Melchor, E. Payne, M. D. Pitkin, J. Powell, N. Sarin, R. J. E. Smith, and E. Thrane, ApJS 241, 27 (2019).
  * Feroz  _et al._ (2009) F. Feroz, M. P. Hobson, and M. Bridges, MNRAS 398, 1601 (2009).
  * Nakar and Piran (2021) E. Nakar and T. Piran, ApJ 909, 114 (2021).
  * Granot  _et al._ (2018) J. Granot, R. Gill, D. Guetta, and F. De Colle, MNRAS 481, 1597 (2018), arXiv:1710.06421 [astro-ph.HE] .
  * Treister  _et al._ (2014) E. Treister, F. Bauer, and K. Schawinski, The Astronomer’s Telegram 6603, 1 (2014).
  * Burlon  _et al._ (2014) D. Burlon, K. Bannister, P. Hancock, M. Bell, T. Murphy, M. Huynh, and B. Gaensler, The Astronomer’s Telegram 6583, 1 (2014).
  * Clocchiatti  _et al._ (2011) A. Clocchiatti, N. B. Suntzeff, R. Covarrubias, and P. Candia, ApJ 141, 163 (2011).
  * Greiner  _et al._ (2015) J. Greiner, P. Mazzali, D. Kann, T. Krühler, E. Pian, S. Prentice, F. Olivares E., A. Rossi, S. Klose, S. Taubenberger, F. Knust, P. Afonso, C. Ashall, J. Bolmer, C. Delvaux, R. Diehl, J. Elliott, R. Filgas, J. Fynbo, J. Graham, A. Guelbenzu, S. Kobayashi, G. Leloudas, S. Savaglio, P. Schady, S. Schmidl, T. Schweyer, V. Sudilovsky, M. Tanga, A. Updike, H. van Eerten, and K. Varela, Nature 523, 189 (2015).
  * Richardson  _et al._ (2014) D. Richardson, I. Jenkins, Robert L., J. Wright, and L. Maddox, Astronomical Journal 147, 118 (2014).
  * Lyutikov (2013) M. Lyutikov, ApJ 768, 63 (2013).
  * Fong  _et al._ (2015) W. Fong, E. Berger, R. Margutti, and B. A. Zauderer, ApJ 815, 102 (2015).
  * Amati (2006) L. Amati, MNRAS 372, 233 (2006).
  * Minaev and Pozanenko (2020) P. Y. Minaev and A. S. Pozanenko, MNRAS 492, 1919 (2020).
  * Virgili  _et al._ (2012) F. J. Virgili, Y. Qin, B. Zhang, and E. Liang, MNRAS 424, 2821 (2012).
  * Tominaga  _et al._ (2011) N. Tominaga, T. Morokuma, S. I. Blinnikov, P. Baklanov, E. I. Sorokina, and K. Nomoto, ApJS 193, 20 (2011), arXiv:1102.2360 [astro-ph.HE] .
  * Cenko  _et al._ (2012) S. B. Cenko, H. A. Krimm, A. Horesh, A. Rau, D. A. Frail, J. A. Kennea, A. J. Levan, S. T. Holland, N. R. Butler, R. M. Quimby, J. S. Bloom, A. V. Filippenko, A. Gal-Yam, J. Greiner, S. R. Kulkarni, E. O. Ofek, F. Olivares E., P. Schady, J. M. Silverman, N. R. Tanvir, and D. Xu, ApJ 753, 77 (2012), arXiv:1107.5307 [astro-ph.HE] .
  * Tchekhovskoy  _et al._ (2014) A. Tchekhovskoy, B. D. Metzger, D. Giannios, and L. Z. Kelley, MNRAS 437, 2744 (2014), arXiv:1301.1982 [astro-ph.HE] .
  * Mandel and Levin (2015) I. Mandel and Y. Levin, ApJ 805, L4 (2015).
  * Metzger and Piro (2014) B. D. Metzger and A. L. Piro, MNRAS 439, 3916 (2014).
  * Aloy  _et al._ (2005) M. A. Aloy, H. T. Janka, and E. Müller, A&A 436, 273 (2005).
  * Huang  _et al._ (2002) Y. F. Huang, Z. G. Dai, and T. Lu, MNRAS 332, 735 (2002).
  * Rhoads (2003) J. E. Rhoads, ApJ 591, 1097 (2003).
  * Sarin  _et al._ (2021) N. Sarin _et al._ , “A new class of low-efficiency long gamma-ray bursts,” (2021), in prep.
  * Abbott  _et al._ (2017c) B. P. Abbott, R. Abbott, T. D. Abbott, F. Acernese, K. Ackley, _et al._ , Phys. Rev. Lett. 119, 161101 (2017c).
  * Antonelli  _et al._ (2009) L. A. Antonelli, P. D’Avanzo, R. Perna, L. Amati, S. Covino, S. Cutini, V. D’Elia, S. Gallozzi, A. Grazian, E. Palazzi, S. Piranomonte, A. Rossi, S. Spiro, L. Stella, V. Testa, G. Chincarini, A. di Paola, F. Fiore, D. Fugazza, E. Giallongo, E. Maiorano, N. Masetti, F. Pedichini, R. Salvaterra, G. Tagliaferri, and S. Vergani, A&A 507, L45 (2009).
  * Levesque  _et al._ (2010) E. M. Levesque, J. S. Bloom, N. R. Butler, D. A. Perley, S. B. Cenko, J. X. Prochaska, L. J. Kewley, A. Bunker, H.-W. Chen, R. Chornock, A. V. Filippenko, K. Glazebrook, S. Lopez, J. Masiero, M. Modjaz, A. Morgan, and D. Poznanski, MNRAS 401, 963 (2010).
  * Wiggins  _et al._ (2018) B. K. Wiggins, C. L. Fryer, J. M. Smidt, D. Hartmann, N. Lloyd-Ronning, and C. Belcynski, ApJ 865, 27 (2018).
  * Siegel  _et al._ (2019) D. M. Siegel, J. Barnes, and B. D. Metzger, Nature 569, 241 (2019).


# Network Generalization Prediction
for Safety Critical Tasks in Novel Operating Domains

Molly O’Brien1
<EMAIL_ADDRESS>Mike Medoff 2
<EMAIL_ADDRESS>Julia Bukowski3
<EMAIL_ADDRESS>Greg Hager1
<EMAIL_ADDRESS>1 Department of Computer Science, Johns Hopkins University,
Baltimore MD 21218
2exida LLC, Sellersville PA 18960
3Department of Electrical and Computer Engineering, Villanova University,
Villanova, PA 19085

###### Abstract

It is well known that Neural Network (network) performance often degrades when
a network is used in novel operating domains that differ from its training and
testing domains. This is a major limitation, as networks are being integrated
into safety critical, cyber-physical systems that must work in unconstrained
environments, e.g., perception for autonomous vehicles. Training networks that
generalize to novel operating domains and that extract robust features is an
active area of research, but previous work fails to predict what the network
performance will be in novel operating domains. We propose the task Network
Generalization Prediction: predicting the expected network performance in
novel operating domains. We describe the network performance in terms of an
interpretable Context Subspace, and we propose a methodology for selecting the
features of the Context Subspace that provide the most information about the
network performance. We identify the Context Subspace for a pretrained Faster
RCNN network performing pedestrian detection on the Berkeley Deep Drive (BDD)
Dataset, and demonstrate Network Generalization Prediction accuracy within
$5\%$ or less of observed performance. We also demonstrate that the Context
Subspace from the BDD Dataset is informative for completely unseen datasets,
JAAD and Cityscapes, where predictions have a bias of $10\%$ or less.

##  1 Introduction

Deep Neural Networks (networks) are being integrated into commercial, safety
critical, autonomous systems that operate in unconstrained environments, e.g.,
perception for autonomous vehicles. When a network is deployed in an
unconstrained environment, the operating domain, i.e., the distribution of
context features that describe the network’s environment, can change
significantly from the testing domain, i.e., the distribution of context
features that describe the test data. Safety critical systems are regulated by
international functional safety standards, e.g., ISO 26262 for the automotive
industry, IEC 61508 for electronics and software. Functional safety standards
leverage various techniques to verify the safety of software, including
requirement specification, i.e., linking required system behavior to specific
code modules, white box testing, i.e., testing specific inputs that cover all
branches or behavior in the code, and code inspection and review to identify
human error. These techniques are challenging or impossible to apply directly
to networks, e.g., labeled data is used to implicitly specify the correct
behavior in supervised learning, networks are black box systems, and network
weights cannot be manually inspected to identify failure cases.

New techniques are needed to bridge the gap between the high performance of
deep networks and the verification required for safety critical systems. In
particular, the ability to predict how a network’s performance will change in
a novel operating domain can enable verifying the required level of
performance before a network is deployed, we denote this task Network
Generalization Prediction. We propose a methodology for Network Generalization
Prediction for networks trained via supervised learning. Our contributions are
as follows:

  1. 1.

We introduce the concept of a Context Subspace, a low-dimensional space,
encoding the context features most informative about the network performance.

  2. 2.

We propose a greedy feature selection algorithm for identifying the Context
Subspace by 1) ranking the context features by the information they provide
about the network loss, and 2) selecting the subspace dimensionality that
leads to accurate Network Generalization Prediction.

  3. 3.

We leverage a Context Subspace for accurate Network Generalization Prediction
for pedestrian detection in diverse operating domains, with a prediction error
from $0.5\%$ to $2\%$ for not safety critical pedestrians (pedestrians not in
the road), and a prediction error from $2\%$ to $5\%$ for safety critical
pedestrians (pedestrians in the road).

  4. 4.

We demonstrate that the Context Subspace identified for the Berkeley Deep
Drive Dataset can be used to predict pedestrian recall in completely unseen
datasets, the JAAD and Cityscapes Datasets, with a prediction bias of $10\%$
or less.

##  2 Background

###  2.1 Network Dependability

Avizienis et al. defined software dependability as “the ability to deliver
service that can justifiably be trusted,” where dependability encompasses
availability, reliability, safety, integrity, and maintainability[1]. To
describe the dependability of a learned model, O’Brien et al. defined ML
Dependability as “the probability that a model will succeed when operated
under specified conditions”[14]. Cheng et al. proposed that Robustness,
Interpretability, Completeness, and Correctness contribute to a network’s
Dependability [4]. Ponn et al. trained a random forest to predict whether a
network would detect a pedestrian, based on pedestrian attributes; they denote
this task Detection Performance Modeling[15]. Where Detection Performance
Modeling predicts whether one specific object will be detected, Network
Generalization Prediction predicts the expected network performance for a
given operating domain, described by a distribution of context features.

###  2.2 Network Generalization

It has been shown that underspecification causes network performance to
degrade when deployed in operating conditions different from the training and
testing conditions[6]. The WILDS benchmark was released to provide datasets
with “in-the-wild” distribution shifts between the training and test data
[11]. Subbaswamy et al. propose to evaluate a model’s robustness to
distribution shifts with one fixed evaluation set [18]. Common techniques to
improve network generalization include extracting features robust to changing
conditions[19], [10], zero or few-shot learning [24], [23], and identifying
when an input is outside the network’s training distribution [13], [7].

###  2.3 Feature Selection

Feature selection algorithms aim to select a subset of the available features,
typically to use the features as input to train a model for a given task.
Feature selection algorithms can be classified as filter methods, i.e.,
features are scored according to their association with the task label,
wrapper methods, i.e., features are selected to minimize task error, and
embedded methods, i.e., features are selected in the model training process
[3]. The Mutual Information [12] is often used in filter methods to measure
the information between a given feature and the desired label [22]. As
exhaustive feature selection search is typically intractable, greedy feature
selection algorithms are often used [9], [21], [8]. Note, greedy feature
selection is related to matching pursuit in the sparse approximation
literature [20] and has applications in compressed sensing [2].

Figure 1: Overview of Network Generalization Prediction.

##  3 Methods

###  3.1 Problem Formulation

It is well known that in supervised learning, a network, $f$, is trained to
produce a label, $y_{i}$, from data, $x_{i}$, and a loss function,
$l(f(x_{i}),y_{i})$, is used to drive training. In Network Generalization
Prediction, we are not training $f$. Instead, we aim to predict the
performance of a fixed network $f$, trained via supervised learning, when
deployed in an operating domain, $\mathcal{O}$, that differs from the testing
domain, $\mathcal{T}$, see Figure 1. The performance of $f$ is measured using
test data, $X=\\{x_{i}\\}_{i=1}^{N}$, and test labels,
$Y=\\{y_{i}\\}_{i=1}^{N}$, via a loss function
$L=\\{l(f(x_{i}),y_{i})\\}_{i=1}^{N}$, where the elements of $L$ are assumed
to be discrete and bounded, e.g., an object detection flag, whether a safety
criteria was satisfied, or a discretized classification error.

$\mathcal{T}$ is described via $J$ context features,
$\textbf{C}=\\{\textbf{c}_{i}\\}_{i=1}^{N}$, where $\textbf{c}_{i}$ indicates
a $J$ dimensional context vector associated with $x_{i}$. Context features,
e.g., image brightness, weather, or robot speed, can be categorical or
numerical; numerical features are assumed to be discrete or discretized. It is
possible for multiple test samples to map to the same context, i.e.,
$\textbf{c}_{i}=\textbf{c}_{j},i\neq j$. $p_{\mathcal{T}}(\textbf{c})$ denotes
the probability of encountering c in $\mathcal{T}$. $\mathcal{O}$ is described
by the probability of encountering c in $\mathcal{O}$,
$p_{\mathcal{O}}(\textbf{c})$. In many practical applications, the likelihood
of encountering a context may be known without annotated data, e.g., there is
a $25\%$ chance of snow in Boston, etc. Note, labeled test data from
$\mathcal{O}$ is not required. We assume that while the distribution of
contexts shifts between the testing and operating domains, i.e.,
$p_{\mathcal{T}}(\textbf{c})\neq p_{\mathcal{O}}(\textbf{c})$, the expected
network performance in context c is stable for both the testing and operating
domains. Table 1 describes the Notation used in the Methods Section.

As is typical, we approximate the posterior expected loss in $\mathcal{T}$,
$\mathcal{L}^{\mathcal{T}}$, using the empirical loss:

$\mathcal{L}^{\mathcal{T}}=E[l(f(X),Y)]=\frac{1}{N}\sum_{i=1}^{N}l(f(x_{i}),y_{i})$
(1)

We define $g(\textbf{c})=E[l(f(X),Y|\textbf{c})]$. Let
$\mathbb{I}(\textbf{a},\textbf{b})$ be an indicator function that is equal to
$1$ if $\textbf{a}=\textbf{b}$ and $0$ otherwise. $g(\textbf{c})$ can be
computed as:

$g(\textbf{c})=\frac{\sum_{i=1}^{N}{\mathbb{I}(\textbf{c}_{i},\textbf{c})*l(f(x_{i}),y_{i})}}{\sum_{i=1}^{N}{\mathbb{I}(\textbf{c}_{i},\textbf{c})}}$
(2)

$\mathcal{L}^{\mathcal{T}}$ can equivalently be computed as:

$\mathcal{L}^{\mathcal{T}}=\sum_{\textbf{c}\in\textbf{C}}p_{\mathcal{T}}(\textbf{c})g(\textbf{c})$
(3)

Likewise, we can now express the Network Generalization Prediction,
$\hat{\mathcal{L}}^{\mathcal{O}}$, as:

$\hat{\mathcal{L}}^{\mathcal{O}}=\sum_{\textbf{c}\in\textbf{C}}p_{\mathcal{O}}(\textbf{c})g(\textbf{c})$
(4)

This formulation holds theoretically for any number of context features $J$.
However, as $J$ grows linearly, computing Eqn. 4 requires exponentially more
test samples to cover every possible $\textbf{c}\in\textbf{C}$. Thus, we
introduce the Context Subspace, $\textbf{C}^{S_{K}}$, a low-dimensional space,
encoding the context features most informative about the network performance.

| Notation
---|---
$X=\\{x_{i}\\}_{i=1}^{N}$ | The Test Data
$Y=\\{y_{i}\\}_{i=1}^{N}$ | The Test Labels
$f$ | The trained network
$L=\\{l(f(x_{i}),y_{i})\\}_{i=1}^{N}$ | The Test Set Loss
C | The context features
$\textbf{c}\in\textbf{C}$ | A context vector
$g(\textbf{c})$ | The expected loss of $f$ in c
$\textbf{C}^{S_{K}}$ | The Context Subspace
$\mathcal{T}$ | The Testing Domain
$\mathcal{O}$ | The Operating Domain
$p_{\mathcal{T}}(\textbf{c})$, $p_{\mathcal{O}}(\textbf{c})$ | The probability of c in $\mathcal{T}$, $\mathcal{O}$
$\mathcal{L}^{\mathcal{T}}$ | The observed loss in $\mathcal{T}$
$\hat{\mathcal{L}}^{\mathcal{O}}$ | The predicted loss in $\mathcal{O}$
Table 1: Notation.

###  3.2 Defining a Context Subspace

We are interested in selecting the $K$ context features that provide the most
information about the network loss, to include these features in
$\textbf{C}^{S_{K}}$. Let $S_{K}=\\{s_{k}\\}_{k=1}^{K}$ be the indices of
context features of interest and
$\textbf{C}^{S_{K}}=\\{C^{s_{k}}\\}_{k=1}^{K}$, where
$C^{s_{k}}=\\{c_{i}^{s_{k}}\\}_{i=1}^{N}$ are the annotated attributes for
each example in the test set for context feature ${s_{k}}$. To select the
context features to include in $\textbf{C}^{S_{K}}$, we 1) rank the context
features by how much information they provide about the network loss, 2)
select the $\textbf{C}^{S_{K}}$ dimensionality $K$ to enable accurate Network
Generalization Prediction.

####  3.2.1 Ranking Context Features

Recall, the Mutual Information is often used to rank features in filter
feature selection algorithms and is computed as $I(L,C^{j})$ for loss $L$ and
context feature $C^{j}$:

$I(L,C^{j})=\sum_{\ell\in L}\sum_{c\in
C^{j}}p(\ell,c)log(\frac{p(\ell,c)}{p(\ell)p(c)})$ (5)

where $p(\ell,c)$ indicates the joint probability of $\ell$ and $c$, and
$p(\ell)$ and $p(c)$ indicate the marginal probabilities for $\ell$ and $c$,
respectively. The Interaction Information is a generalization of the Mutual
Information to $K$ features. The Interaction Information between $L$ and the
context features $C^{s_{1}},...,C^{s_{K}}$ is defined as:

$I(L,C^{s_{1}},...,C^{s_{K}})=I(L,C^{s_{1}},...,C^{s_{K-1}})\\\
-I(L,C^{s_{1}},...,C^{s_{K-1}}|C^{s_{K}})$ (6)

For two features, this becomes:

$I(L,C^{s_{1}},C^{s_{2}})=I(L,C^{s_{2}})-I(L,C^{s_{2}}|C^{s_{1}})$ (7)

Where $I(L,C^{s_{2}}|C^{s_{1}})$ can be computed as:

$I(L,C^{s_{2}}|C^{s_{1}})=\sum_{\ell\in L}\sum_{c_{2}\in C_{2}}\sum_{c_{1}\in
C_{1}}p(\ell,c_{2},c_{1})\\\ \times
log\left(\frac{p(\ell,c_{2},c_{1})}{p(\ell,c_{1})p(c_{2},c_{1})}\right)$ (8)

The computational complexity of $I(L,C^{s_{1}},...,C^{s_{K}})$ grows
combinatorially with $K$. We are interested in ranking the context features by
the Interaction Information, but computing the exact Interaction Information
becomes intractable as $K$ grows. To make computation tractable, we propose
$\Delta I(L,C^{s_{1}},...,C^{s_{K}})$ to approximate how much more information
including context feature $C^{s_{K}}$ in the Context Subspace provides about
$L$.

$\Delta
I(L,C^{s_{1}},...,C^{s_{K}})=I(L,C^{s_{K}})-\sum_{k=1}^{K-1}I(C^{s_{k}},C^{s_{K}})$
(9)

Intuitively, $\Delta I(L,C^{s_{1}},...,C^{s_{K}})$ subtracts the redundant
information in $C^{s_{K}}$, $\sum_{k=1}^{K-1}I(C^{s_{k}},C^{s_{K}})$, from the
information it provides about the loss, $I(L,C^{s_{K}})$. Note that the
computational complexity of computing $\Delta I(L,C^{s_{1}},...,C^{s_{K}})$
grows linearly with $K$. Like the Interaction Information, $\Delta
I(L,C^{s_{1}},...,C^{s_{K}})$ can be positive or negative. In Appendix A, we
show that for independent features in the Context Subspace, $\Delta
I(L,C^{s_{1}},C^{s_{2}})$ approaches $I(L,C^{s_{1}},C^{s_{2}})$ as $C^{s_{2}}$
approaches perfect information on $L$. We propose a greedy algorithm to
iteratively select the $K$ most informative features from the context, see
Algorithm 1.

Algorithm 1 Greedy $\Delta I$ Context Selection

1:$S_{K}=\\{\\}$

2:for $k=1:K$ do

3: $s_{k}^{*}\leftarrow argmax_{j}[I(C^{j},L)-\sum_{s_{k}\in
S_{K}}I(C^{j},C^{s_{k}})]$

4:$\quad\quad\forall j\in J\setminus S_{K}$

5: $S_{K}=S_{K}\cup s_{k}^{*}$

6:end for

7:$g(\textbf{c}^{S_{K}})=E[l(f(X),Y|\textbf{c}^{S_{K}})]$

####  3.2.2 Selecting the Context Subspace Dimensionality

Selecting the number of features, $K$, to include in $\textbf{C}^{S_{K}}$ is
not trivial: including more features can lead to a more descriptive
$\textbf{C}^{S_{K}}$ but can also lead to many untested contexts in
$\textbf{C}^{S_{K}}$. To select $K$, we compute the expected prediction error
for a given subspace dimensionality, $\epsilon_{K}$. Using the $K$ most
informative context features,
$g(\textbf{c}^{S_{K}})=E[l(f(X),Y|\textbf{c}^{S_{K}})]$ can be computed
according to Eqn. 2. where $\textbf{c}^{S_{K}}$ is a $K$ dimensional feature
vector in $\textbf{C}^{S_{K}}$. We iteratively compute the prediction error
within the test set, $\epsilon_{K}$, to estimate the expected prediction error
$\tilde{\epsilon}_{K}$, see Algorithm 2. First, we randomly partition the test
set into a $fit$ set and a $val$ set: $X^{fit}$, $Y^{fit}$, $\textbf{C}^{fit}$
with $N^{fit}$ samples and $X^{val}$, $Y^{val}$, $\textbf{C}^{val}$ with
$N^{val}$ samples respectively. We estimate $g^{fit}(\textbf{c}^{S_{K}})$
using the $fit$ set. We compute the observed loss from the $val$ set,
$\mathcal{L}^{val}$. Let $p^{val}(\textbf{c}^{S_{K}})$ indicate the
probability of encountering context $\textbf{c}^{S_{K}}$ in
$\textbf{C}^{val}$. The prediction error, $\epsilon_{K}$, is the difference
between the observed validation loss, $\mathcal{L}^{val}$, and the predicted
validation loss using $g^{fit}(\textbf{c}^{S_{K}})$. This procedure can be
iterated multiple times, and the subsequent $\epsilon_{K}$’s averaged, to
estimate the expected prediction error, $\tilde{\epsilon}_{K}$, for different
random $fit$ and $val$ partitions of the test set. We select the $K$ that
minimizes $\tilde{\epsilon}_{K}$.

Algorithm 2 Context Subspace Dimensionality Selection

1:$\tilde{\epsilon}_{K}=\\{\\}$

2:for $K=1:J$ do

3: $\epsilon_{K}s=[]$

4: for $iteration$ do

5: split test set into $fit$ and $val$ set

6: $g^{fit}(\textbf{c}^{S_{K}})=E[l(f(X^{fit}),Y^{fit}|\textbf{c}^{S_{K}})]$

7:
$\mathcal{L}^{val}=\frac{1}{N^{val}}\sum_{i=1}^{N^{val}}l(f(x^{val}_{i}),y^{val}_{i})$

8:
$\epsilon_{K}=|\mathcal{L}^{val}-\sum_{\textbf{c}^{S_{K}}\in\textbf{C}^{S_{K}}}p^{val}(\textbf{c}^{S_{K}})g_{k}^{fit}(\textbf{c}^{S_{K}})|$

9: $\epsilon_{K}s.append(\epsilon_{K})$

10: end for

11: $\tilde{\epsilon}_{K}=mean(\epsilon_{K}s)$

12:end for

13:$K\leftarrow argmin_{K}\quad\tilde{\epsilon}_{K}$

$\tilde{\epsilon}_{K}$ measures the expected prediction error within
$\mathcal{T}$. When the context is informative about the loss, we expect
$\tilde{\epsilon}_{K}$ to decrease as $K$ increases until an optimal $K^{*}$
is reached, then $\tilde{\epsilon}_{K}$ will begin to rise as $K$ increases
and there are many untested contexts. If $\tilde{\epsilon}_{K}$ is flat or
increasing as $K$ increases, it indicates that the context features available
are not informative about the loss.

After we have ranked the context features and selected the number of features
to include in the subspace, we can form $\textbf{C}^{S_{K}}$. The $K$ most
informative context features form the axes of the subspace. Recall, we assumed
the context features are categorical or numerical and discrete, this yields a
finite set of context partitions, $\textbf{c}^{S_{K}}\in\textbf{C}^{S_{K}}$.

###  3.3 Using the Context Subspace

We use $\textbf{C}^{S_{K}}$ to describe the expected network loss in different
contexts, $g(\textbf{c}^{S_{K}})$, and to describe the probability of
encountering a context in the operating domain,
$p_{\mathcal{O}}(\textbf{c}^{S_{K}})$. We can compute $g(\textbf{c}^{S_{K}})$
using Eqn. 2, note we use the entire test set to compute
$g(\textbf{c}^{S_{K}})$ once we have selected the subspace dimensionality $K$.
Recall, we do not assume to have labeled test data in $\mathcal{O}$, but we do
assume to know $p_{\mathcal{O}}(\textbf{c}^{S_{K}})$. Individual context
feature probabilities can be multiplied to obtain a joint probability
distribution if the context feature probabilities are assumed to be
independent.

###  3.4 Network Generalization Prediction

We can now perform Network Generalization Prediction, where
$\hat{\mathcal{L}}^{\mathcal{O}}$ is the predicted loss in $\mathcal{O}$:

$\hat{\mathcal{L}}^{\mathcal{O}}=\sum_{\textbf{c}^{S_{K}}\in\textbf{C}^{S_{K}}}p_{\mathcal{O}}(\textbf{c}^{S_{K}})g(\textbf{c}^{S_{K}})$
(10)

Recall, we selected a small number of informative context features so that it
would be practical to describe the unique contexts
$\textbf{c}^{S_{K}}\in\textbf{C}^{S_{K}}$, but there may be untested contexts
in $\textbf{C}^{S_{K}}$. For conservative predictions, we assume the maximum
loss in untested contexts. The maximum loss may correspond to a binary failure
or a large expected error. Leveraging $\mathbf{C}^{S_{K}}$ renders Network
Generalization Prediction practical for interestingly complex applications,
like perception for autonomous vehicles.

##  4 Experimental Results

###  4.1 Pedestrian Detection Generalization

Perception for autonomous vehicles is an active area of research, and systems
that use deep networks to detect and avoid obstacles, like pedestrians, while
driving are commercially available. Some of these commercial systems can be
used in any driving conditions, at the user’s discretion, and the operating
domains can vary significantly in terms of the lighting conditions, e.g.,
daytime compared to night, road conditions, e.g., clear weather compared to
rainy or snowy weather, and obstacle density, e.g., a residential street
compared to a restricted access highway. It would be impractical for
autonomous vehicle developers to test a perception system in every possible
operating domain, but it is also imperative to know whether it is safe to use
a perception system in a given operating domain. We perform experiments
analogous to an autonomous vehicle developer: we test a fixed network in one
testing domain, $\mathcal{T}$, and predict the network’s performance in novel
operating domains, where the distribution of context features vary
significantly from $\mathcal{T}$. Our goal is to accurately predict the
observed network performance when the network is used in a novel operating
domain, $\mathcal{O}$.

We test a pretrained Faster RCNN [17] object detector for pedestrian
detection, where the objects detected as $person$ are used as pedestrian
detections. In our analysis, we consider pedestrians whose ground truth
bounding box area is $\geq 300$ pixels. We evaluate the network performance at
the pedestrian level. Pedestrians correctly detected with an $IoU>0.5$ and a
confidence score $>0.5$ are assigned a loss of $0$; pedestrians that are not
detected are assigned a loss of $1$111We are predicting the network’s recall.
We do not assign a loss for false positive detections; this same methodology
can be used to predict network precision if that is of interest. We focus on
recall because failing to predict a pedestrian who is truly present in the
scene is a higher safety risk than trying to avoid a pedestrian who is not
present. . Pedestrians in images with multiple people are considered
independently; images with no pedestrians present are not assigned any loss.

Figure 2: Defining the Context Subspace. 1) Rank Context Features: The $\Delta
I(L,C)$ between different context features and the loss in the BDD Test Set
for the first three rounds of Algorithm 1. Note that in iteration one, $\Delta
I(L,C)=I(L,C)$ so the features’ scores are non-negative. 2) Select K: We
estimate the expected prediction error for different Context Subspace
dimensionalities, $K$, and choose the dimensionality with the lowest expected
prediction error: in this case, $K=3$. We form the Context Subspace with the
three most informative context features: brightness, safety critical flag, and
the scene type. Right: heatmaps of the probability of encountering a context
in the testing domain, $p_{\mathcal{T}}(\textbf{c}^{S_{K}})$, and the expected
network loss in different contexts, $g(\textbf{c}^{S_{K}})$. X-axis:
brightness (dark to bright from left to right). Y-axis: (top to bottom) scene
type ‘H’ highway, ‘R’ residential, ‘C’ city street. Separate heatmaps shown
for NSC and SC pedestrians.

The Berkeley Deep Drive (BDD) Dataset [25] was recorded across the continental
US and includes data from varying times of day (daytime, dawn/dusk, or night),
weather conditions (clear, partly cloudy, overcast, rainy, foggy, or snowy),
and scene types (city street, residential, or highway). BDD images are of size
$720\times 1280$. We use $10,000$ images from the BDD Dataset for testing,
denoted the BDD Test Set. We use the remaining $70,000$ images in the BDD
Dataset, denoted the BDD Operating Set, to define novel operating domains. The
BDD Test Set and BDD Operating Set correspond to the BDD “Validation” and
“Train” folds, respectively.

###  4.2 Defining the Context Subspace

We evaluate the network performance at the pedestrian level; therefore,
context features are assigned to individual pedestrians. We do not know a
priori which pedestrian attributes are informative about the network loss, so
we include all available context features. The BDD dataset includes metadata
on the image time of day, weather, and scene type. We include the metadata as
context features. We also include the image brightness and the pedestrian
bounding box brightness. We define the road(s) to be the safety critical (SC)
region(s) in the images. Pedestrians in the road are labeled SC, pedestrians
outside the road, e.g., on the sidewalk, are labeled not safety critical
(NSC). The road is defined using the drivable area annotations. Whether a
pedestrian is SC, denoted the safety critical flag, is included as a context
feature. To capture information about the obstacle density in the scene, we
include the total number of pedestrians, the number of SC pedestrians, and the
number of NSC pedestrians in the image as context features.

####  4.2.1 Ranking Context Features

We use Algorithm 1 to rank the context features by how much information they
provide about the network loss. When computing the mutual information for a
numerical feature with more than 10 unique values, we uniformly partition the
feature into 10 discrete bins. Categorical features are labeled discretely
with their assigned labels. See Figure 2 for the $\Delta I$ computed for the
first three iterations of Algorithm 1. The 6 most informative features were
found to be: 1) image brightness, 2) safety critical flag, 3) scene , 4)
number SC pedestrians, 5) time of day, and 6) bounding box brightness.

####  4.2.2 Selecting the Context Subspace Dimensionality

To select the number of features to include in the Context Subspace, we
compute $\tilde{\epsilon}_{K}$ for values of $K$ from $1$ to $6$. For each
dimensionality, $K$, we compute $\epsilon_{K}$ 50 times by randomly
partitioning the test data into $50\%$ for fitting $g(\textbf{c}^{S_{K}})$ and
$50\%$ for validation. We select the $K$ with the minimum expected prediction
error $\tilde{\epsilon}_{K}$ over the $50$ iterations. $K=3$ was found to be
optimal, with an average prediction error of $0.63\%$, see Figure 2 center. We
subsequently define the Context Subspace with three dimensions: 1) image
brightness, 2) safety critical flag, and 3) scene.

The image brightness is a continuous feature; we uniformly partition the image
brightness into 10 bins. The safety critical flag and the scene type are
discrete and categorical features with 2 and 3 possible values, respectively.
This results in a Context Subspace, $\textbf{C}^{S_{K}}$, with $60$ discrete
contexts, $\textbf{c}^{S_{K}}$.

Figure 3: Top: BDD Novel Operating Domains. (a) Sample images of NSC
pedestrians in the operating domain, NSC pedestrians outlined in cyan.
Drivable area shown in transparent color. (b) NSC pedestrian
$p_{\mathcal{O}}(\textbf{c}^{S_{K}})$. (c) Sample images of SC pedestrians in
the operating domain, SC pedestrians outlined in red. Drivable area shown in
transparent color. (d) SC pedestrian $p_{\mathcal{O}}(\textbf{c}^{S_{K}})$.
Bottom: Pedestrian Generalization Prediction Results. NSC pedestrian recall
and SC pedestrian recall are shown separately. X-Axis: Ground Truth (GT)
recall in the operating domain $\mathcal{O}$. Y-Axis: predicted recall.
Perfect predictions would fall on the diagonal line. Predictions in the shaded
region are conservative, i.e., the predicted recall is less than the GT
recall.

###  4.3 Using the Context Subspace

We use $\textbf{C}^{S_{K}}$ to estimate the expected network loss in a
context, $g(\textbf{c}^{S_{K}})$, and to describe the probability of
encountering a context in $\mathcal{O}$,
$p_{\mathcal{O}}(\textbf{c}^{S_{K}})$, see Figure 2 right. For all tested
contexts, $g(\textbf{c}^{S_{K}})$ is computed according to Eqn. 2. All
untested contexts are assigned an expected loss of $1$, i.e., a $100\%$ chance
of failing to detect a pedestrian. The BDD Operating Set is used to define
four novel operating domains: 1) daytime, small groups; 2) daytime, large
groups; 3) night, small groups; and 4) night, large groups. The time of day
annotated in the images was used to assign “daytime” or “night”. The SC and
NSC pedestrians are considered independently. Pedestrians in images with fewer
than $5$ (N)SC pedestrians are categorized as small groups; pedestrians in
images with $5$ or more (N)SC pedestrians are categorized as large groups,
i.e., in an image with 2 SC pedestrians and 15 NSC pedestrians, the SC
pedestrians would be labeled ‘small group’ and the NSC pedestrians would be
labeled ‘large group’. We compute $p_{\mathcal{O}}(\textbf{c}^{S_{K}})$ for
each $\mathcal{O}$ by counting the number of pedestrians that fall into each
$\textbf{c}^{S_{K}}\in\textbf{C}^{S_{K}}$ and dividing by the total number of
pedestrians.

###  4.4 Pedestrian Detection Generalization Prediction

We predict the network loss in the novel operating domains defined in 4.3
using Eqn. 10. The heatmaps of $p_{\mathcal{O}}(\textbf{c}^{S_{K}})$ in Figure
3 illustrate that the novel operating domains are significantly different from
each other and the testing domain, see $p_{\mathcal{T}}(\textbf{c}^{S_{K}})$
in Figure 2. Our network loss is equivalent to the fraction of pedestrians
that are not detected by the network; we convert the predictions into the
predicted network recall by subtracting the fraction of pedestrians that are
not detected from 1, see Figure 3. We then pass the BDD Operating Set through
the network; the observed recall is computed as the fraction of pedestrians
that were correctly detected. Figure 3 illustrates that our predictions are
accurate with Network Generalization Prediction accuracy between $0.5\%$ and
$2.5\%$ for NSC pedestrian recall and $2\%$ and $5\%$ for SC pedestrian
recall. All the SC predictions underpredict the observed recall; this
demonstrates that our predictions are conservative. Note, the only prediction
with significant error is for night, large group SC pedestrians. Only one
image in the BDD Operating Set falls into this category, so the observed
performance is based on minimal data.

Figure 4: Left: Unseen Dataset Novel Operating Domains. (a) Sample images from
the unseen datasets. NSC pedestrians outlined in blue. SC pedestrians outlined
in red. (b) NSC pedestrian $p_{\mathcal{O}}(\textbf{c}^{S_{K}})$. (c) SC
pedestrian $p_{\mathcal{O}}(\textbf{c}^{S_{K}})$. Right: Generalization
Prediction for Unseen Datasets. NSC pedestrian recall and SC pedestrian recall
are shown separately. X-Axis: Ground Truth (GT) recall in the dataset. Y-Axis:
predicted recall. Perfect predictions would fall on the diagonal line.
Predictions in the shaded region are conservative, i.e., the predicted recall
is less than the GT recall.

###  4.5 Generalization Prediction for Unseen Datasets

As a preliminary study, we investigate whether the Context Subspace,
$\textbf{C}^{S_{K}}$, defined using the BDD Test Set and the network loss,
$g(\textbf{c}^{S_{K}})$, estimated from the BDD Test Set provide information
about completely unseen datasets. Unseen datasets include shifts in the
context feature distributions, as well as changes in camera parameters and
physical setup that are not captured by the test set. As such, we expect
predictions for unseen dataset to contain bias, i.e., the prediction error for
an unseen dataset will have a consistent non-zero offset. We are interested in
determining the magnitude of this prediction bias to evaluate the usefulness
of Network Generalization Prediction across datasets. We perform Network
Generalization Prediction for the JAAD Dataset [16], and the Cityscapes
Dataset with the gtFine labels [5], see Figure 4 for sample images. For both
datasets, the (N)SC pedestrian image brightness distribution is computed from
the images.

The JAAD Dataset was recorded in North America and Europe; it includes
primarily daytime images from residential and city streets in varying weather
conditions. JAAD images are of size $1080\times 1920$. For the JAAD Dataset,
we sampled images every three seconds from the videos to limit temporal
correspondence between frames; this resulted in 1,031 images. Pedestrians in
the road were manually annotated as SC, all others were labeled NSC. Scene
annotations are not available for the JAAD dataset. To estimate the
probability distribution of scenes, the scene type was annotated for a subset
of 100 images, we assume the distribution holds for the entire dataset. The
marginal (N)SC image brightness distributions and scene type distribution are
multiplied to obtain the joint probability distributions for the JAAD Dataset.

The Cityscapes Dataset contains $3,475$ images recorded in 50 cities across
Germany in the daytime during fair weather conditions. Cityscapes images are
of size $1024\times 2048$. We defined the pedestrian bounding boxes using the
outermost edges of the labeled person instance segmentations, and we used the
semantic segmentation of the road to define the SC region in the image. For
Cityscapes, the scene type is known to be “city street”.

We make Network Generalization Predictions for the JAAD and Cityscapes
Datasets using $g(\textbf{c}^{S_{K}})$, estimated using the BDD Test Set. The
prediction bias is consistently around $10\%$, with a minimum prediction error
of $5\%$ for SC pedestrian recall in the JAAD Dataset. We underpredict
pedestrian recall for the JAAD Dataset and we overpredict pedestrian recall
for the Cityscapes Dataset.

##  5 Discussion

We make accurate Network Generalization Predictions for the BDD Operating Set,
where the observed recall varies from $47\%$ to $87\%$. This demonstrates that
a fixed test set can be used to predict a network’s performance in diverse,
novel operating domains. The observed recall for SC pedestrians is about
$20\%$ higher than for NSC pedestrians. This makes intuitive sense, as SC
pedestrians tend to be central in the image and closer to the vehicle. This is
encouraging, because the performance of perception systems for autonomous
vehicles will ultimately be determined by how well they detect SC pedestrians
and obstacles. However, in the BDD Test Set there are many more examples of
NSC pedestrians, $11,169$, than SC pedestrians, $484$. This leads to more
untested contexts for the SC pedestrians, which in turn leads to the slight
underprediction of SC recall.

For unseen datasets, we find a Network Generalization Prediction bias of
$10\%$; we believe these results are promising and that the results indicate
the Context Subspace identified for one dataset, e.g., one camera setup and
one physical setup, can be informative for unseen datasets. Investigating how
network performance changes between datasets and identifying what physical
changes lead to performance differences is a direction for future work.

Network Generalization Prediction can be used to link network behavior in
novel operating domains to required levels of performance. The Context
Subspace can be leveraged for quasi-white box testing by testing the network
across variations in context features that are known to impact network
behavior. The Context Subspace also makes the network behavior interpretable
by elucidating where failure is more likely. In addition to making the Network
Generalization Prediction tractable, we believe the Context Subspace can be
used during network training to extract features that are robust to changes in
the Context Subspace. The Context Subspace can also be used for online error
monitoring, e.g., an autonomous vehicle could notify the driver if it detects
the surrounding scene is a context with subpar expected performance. We
believe the Context Subspace is a tool that can make network performance more
interpretable during training, testing, and deployment.

##  6 Conclusions

We propose the task Network Generalization Prediction and leverage a Context
Subspace to render Network Generalization Prediction tractable with scarce
test samples. We identify the Context Subspace automatically and demonstrate
accurate Network Generalization Prediction for Faster RCNN used for pedestrian
detection in diverse operating domains. We show that the Context Subspace
identified for the BDD Dataset is informative for completely unseen datasets.
We believe that accurate Network Generalization Prediction, with an
interpretable Context Subspace, is a step towards bridging the gap between the
high performance of deep networks and the verification required for safety
critical systems.

## References

  * [1] Algirdas Avizienis, J-C Laprie, Brian Randell, and Carl Landwehr.  Basic concepts and taxonomy of dependable and secure computing.  IEEE transactions on dependable and secure computing, 1(1):11–33, 2004.
  * [2] Gábor Braun, Sebastian Pokutta, and Yao Xie.  Info-greedy sequential adaptive compressed sensing.  IEEE Journal of selected topics in signal processing, 9(4):601–611, 2015.
  * [3] Jie Cai, Jiawei Luo, Shulin Wang, and Sheng Yang.  Feature selection in machine learning: A new perspective.  Neurocomputing, 300:70–79, 2018.
  * [4] Chih-Hong Cheng, Chung-Hao Huang, Harald Ruess, Hirotoshi Yasuoka, et al.  Towards dependability metrics for neural networks.  In 2018 16th ACM/IEEE International Conference on Formal Methods and Models for System Design (MEMOCODE), pages 1–4. IEEE, 2018.
  * [5] Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele.  The cityscapes dataset for semantic urban scene understanding.  In Proc. of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
  * [6] Alexander D’Amour, Katherine Heller, Dan Moldovan, Ben Adlam, Babak Alipanahi, Alex Beutel, Christina Chen, Jonathan Deaton, Jacob Eisenstein, Matthew D Hoffman, et al.  Underspecification presents challenges for credibility in modern machine learning.  arXiv preprint arXiv:2011.03395, 2020.
  * [7] Yen-Chang Hsu, Yilin Shen, Hongxia Jin, and Zsolt Kira.  Generalized odin: Detecting out-of-distribution image without learning from out-of-distribution data.  In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
  * [8] Jianhao Jiao, Yilong Zhu, Haoyang Ye, Huaiyang Huang, Peng Yun, Linxin Jiang, Lujia Wang, and Ming Liu.  Greedy-based feature selection for efficient lidar slam.  arXiv preprint arXiv:2103.13090, 2021.
  * [9] Rajiv Khanna, Ethan Elenberg, Alex Dimakis, Sahand Negahban, and Joydeep Ghosh.  Scalable greedy feature selection via weak submodularity.  In Artificial Intelligence and Statistics, pages 1560–1568. PMLR, 2017.
  * [10] Byungju Kim, Hyunwoo Kim, Kyungsu Kim, Sungjin Kim, and Junmo Kim.  Learning not to learn: Training deep neural networks with biased data.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9012–9020, 2019.
  * [11] Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Balsubramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Sara Beery, et al.  Wilds: A benchmark of in-the-wild distribution shifts.  arXiv preprint arXiv:2012.07421, 2020.
  * [12] Alexander Kraskov, Harald Stögbauer, and Peter Grassberger.  Estimating mutual information.  Physical review E, 69(6):066138, 2004.
  * [13] Shiyu Liang, Yixuan Li, and Rayadurgam Srikant.  Enhancing the reliability of out-of-distribution image detection in neural networks.  arXiv preprint arXiv:1706.02690, 2017.
  * [14] Molly O’Brien, William Goble, Greg Hager, and Julia Bukowski.  Dependable neural networks for safety critical tasks.  In International Workshop on Engineering Dependable and Secure Machine Learning Systems, pages 126–140. Springer, 2020.
  * [15] Thomas Ponn, Thomas Kröger, and Frank Diermeyer.  Identification and explanation of challenging conditions for camera-based object detection of automated vehicles.  Sensors (Basel, Switzerland), 20(13), 2020.
  * [16] Amir Rasouli, Iuliia Kotseruba, and John K Tsotsos.  Are they going to cross? a benchmark dataset and baseline for pedestrian crosswalk behavior.  In Proceedings of the IEEE International Conference on Computer Vision Workshops, pages 206–213, 2017.
  * [17] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun.  Faster r-cnn: Towards real-time object detection with region proposal networks.  arXiv preprint arXiv:1506.01497, 2015.
  * [18] Adarsh Subbaswamy, Roy Adams, and Suchi Saria.  Evaluating model robustness to dataset shift.  arXiv preprint arXiv:2010.15100, 2020.
  * [19] Saeid Asgari Taghanaki, Mohammad Havaei, Alex Lamb, Aditya Sanghi, Ara Danielyan, and Tonya Custis.  Jigsaw-vae: Towards balancing features in variational autoencoders.  arXiv preprint arXiv:2005.05496, 2020.
  * [20] Joel A Tropp and Anna C Gilbert.  Signal recovery from random measurements via orthogonal matching pursuit.  IEEE Transactions on information theory, 53(12):4655–4666, 2007\.
  * [21] Ioannis Tsamardinos, Giorgos Borboudakis, Pavlos Katsogridakis, Polyvios Pratikakis, and Vassilis Christophides.  A greedy feature selection algorithm for big data of high dimensionality.  Machine learning, 108(2):149–202, 2019.
  * [22] Jorge R Vergara and Pablo A Estévez.  A review of feature selection methods based on mutual information.  Neural computing and applications, 24(1):175–186, 2014.
  * [23] Wei Wang, Vincent W Zheng, Han Yu, and Chunyan Miao.  A survey of zero-shot learning: Settings, methods, and applications.  ACM Transactions on Intelligent Systems and Technology (TIST), 10(2):13, 2019.
  * [24] Yongqin Xian, Christoph H Lampert, Bernt Schiele, and Zeynep Akata.  Zero-shot learning—a comprehensive evaluation of the good, the bad and the ugly.  IEEE transactions on pattern analysis and machine intelligence, 41(9):2251–2265, 2018.
  * [25] Fisher Yu, Wenqi Xian, Yingying Chen, Fangchen Liu, Mike Liao, Vashisht Madhavan, and Trevor Darrell.  Bdd100k: A diverse driving video database with scalable annotation tooling.  arXiv preprint arXiv:1805.04687, 2018.

##  Appendix A Comparing $\Delta I$ and $I$

We propose $\Delta I(L,C^{s_{1}},...,C^{s_{K}})$ to approximate the
Interaction Information between $K$ context features and the network loss $L$,
$I(L,C^{s_{1}},...,C^{s_{K}})$. The computational complexity of computing
$\Delta I(L,C^{s_{1}},...,C^{s_{K}})$ grows linearly with $K$, as compared to
the computational complexity of computing $I(L,C^{s_{1}},...,C^{s_{K}})$ which
grows combinatorially with $K$. We investigate the difference between
$I(L,C^{s_{1}},...,C^{s_{K}})$ and $I(L,C^{s_{1}},...,C^{s_{K}})$. To simplify
the notation, we denote $C^{s_{1}}$ as $C^{1}$ and $C^{s_{2}}$ as $C^{2}$. It
is trivial to compute the Mutual Information between the context features and
$L$ and select $C^{1}$ to be the feature most informative about the loss. We
assume $C^{1}$ has been selected and we compare $I(L,C^{1},C^{2})$ and $\Delta
I(L,C^{1},C^{2})$.

$I(L,C^{1},C^{2})=I(L,C^{2})-I(L,C^{2}|C^{1})$ (11) $\Delta
I(L,C^{1},C^{2})=I(L,C^{2})-I(C^{1},C^{2})$ (12)

The difference between $I(L,C^{1},C^{2})$ and $\Delta I(L,C^{1},C^{2})$ is:

$I(L,C^{1},C^{2})-\Delta I(L,C^{1},C^{2})=I(C^{1},C^{2})-I(L,C^{2}|C^{1})$
(13)

As we would like the context features in $\textbf{C}^{S_{K}}$ to be roughly
independent, let us assume that $C^{1}$ is not informative of $C^{2}$, i.e.,
$I(C^{1},C^{2})=0$.

$I(L,C^{1},C^{2})-\Delta I(L,C^{1},C^{2})=-I(L,C^{2}|C^{1})$ (14)

The reader is reminded that the conditional mutual information is computed as:

$I(L,C^{2}|C^{1})=\sum_{\ell\in L}\sum_{c_{1}\in C_{1}}\sum_{c_{2}\in
C_{2}}p(\ell,c_{1},c_{2})\\\ \times
log\left(\frac{p(c_{1})p(\ell,c_{1},c_{2})}{p(\ell,c_{1})p(c_{1},c_{2})}\right)$
(15)

For simplicity, let us consider the point wise conditional mutual information
at $\ell$, $c_{1}$, and $c_{2}$:

$log\left(\frac{p(c_{1})p(\ell,c_{1},c_{2})}{p(\ell,c_{1})p(c_{1},c_{2})}\right)$
(16)

Recall, it was assumed that $C^{1}$ and $C^{2}$ are independent, thus
$p(c_{1},c_{2})=p(c_{1})p(c_{2})$. The joint probability $p(\ell,c_{1},c_{2})$
can also be factored as $\frac{1}{Z}\psi(\ell,c_{1})\psi(\ell,c_{2})$.

$=log\left(\frac{p(c_{1})\psi(\ell,c_{1})\psi(\ell,c_{2})}{Zp(\ell,c_{1})p(c_{1})p(c_{2})}\right)$
(17)
$=log\left(\frac{\psi(\ell,c_{1})\psi(\ell,c_{2})}{Zp(\ell,c_{1})p(c_{2})}\right)$
(18)

Note $\psi(\ell,c_{1})\propto p(\ell,c_{1})$ and $\psi(\ell,c_{2})\propto
p(\ell,c_{2})$. Thus, the difference between the proposed $\Delta I$ and the
Interaction Information is proportional to

$\propto log\left(p(\ell|c_{2})\right)$ (19)

If we consider only combinations of $\ell$ and $c_{2}$ that exist in the test
set, $p(\ell|c_{2})>0$. As the new context feature becomes more informative,
$p(\ell|c_{2})\rightarrow 1$ and the difference
$log\left(p(\ell|c_{2})\right)\rightarrow 0$. This demonstrates that, if the
context features are informative about the loss, $\Delta I$ is a good
approximation of the Interaction Information.