# Exploring the sky localization and early warning capabilities of third
generation gravitational wave detectors in three-detector network
configurations.

YUFENG LI  Key Laboratory for Computational Astrophysics, National
Astronomical Observatories, Chinese Academy of Sciences, Beijing 100101, China
School of Astronomy and Space Science, University of Chinese Academy of
Sciences, Beijing 100049, China  SUPA, School of Physics and Astronomy,
University of Glasgow, Glasgow G12 8QQ, United Kingdom  Ik Siong Heng  SUPA,
School of Physics and Astronomy, University of Glasgow, Glasgow G12 8QQ,
United Kingdom Ik Siong Heng<EMAIL_ADDRESS>Man Leong Chan  Department
of Applied Physics, Fukuoka University, Nanakuma 8-19-1, Fukuoka 814-0180,
Japan  Chris Messenger  SUPA, School of Physics and Astronomy, University of
Glasgow, Glasgow G12 8QQ, United Kingdom  XILONG FAN  School of Physics and
Technology, Wuhan University, Wuhan 430072, China

###### Abstract

This work characterises the sky localization and early warning performance of
different networks of third generation gravitational wave detectors,
consisting of different combinations of detectors with either the Einstein
Telescope or Cosmic Explorer configuration in sites in North America, Europe
and Australia. Using a Fisher matrix method which includes the effect of earth
rotation, we estimate the sky localization uncertainty for
$1.4\text{M}\odot$-$1.4\text{M}\odot$ binary neutron star mergers at distances
$40\text{Mpc}$, $200\text{Mpc}$, $400\text{Mpc}$, $800\text{Mpc}$,
$1600\text{Mpc}$, to characterize each network’s performance for binary
neutron star observations at a given distance. We also characterize each
network’s sky localization capability for an assumed astrophysical population
up to redshift $\leq 2$. Furthermore, we also study the capabilities for the
different networks to localize a binary neutron star merger prior to merger
(early warning) and characterise the network performance for sky localization
uncertainty between 1 and 30 square degrees. We find that, for example, for
binary neutron star mergers at $200\text{Mpc}$ and a network consisting of the
Einstein Telescope, Cosmic Explorer and an extra Einstein Telescope-like
detector in Australia (2ET1CE), the upper limit of the size of the 90%
credible region for the best localized 90% signals is $0.25\text{deg}^{2}$.
For the simulated astrophysical distribution, this upper limit is
$91.79\text{deg}^{2}$. If the Einstein Telescope-like detector in Australia is
replaced with a Cosmic Explorer-like detector (1ET2CE), for signals at
$200\text{Mpc}$, the size of the 90% credible region for the best localized
90% signals is $0.18\text{deg}^{2}$, while the corresponding value for the
best localized 90% sources following the astrophysical distribution is
$56.77\text{deg}^{2}$. We note that the 1ET2CE network can detect 7.2$\%$ more
of the simulated astrophysical population than the 2ET1CE network. In terms of
early warning performance (e.g. 200 Mpc), we find that a network of 2ET1CE and
1ET2CE networks can both provide early warnings of the order of 1 hour prior
to merger with sky localization uncertainties of 30 square degrees or less. In
some cases, the 2ET1CE network is capable of estimating the sky location with
an uncertainty of 5 square degrees or less on timescales of about 1 hour prior
to merger. Our study concludes that the 1ET2CE network is a good compromise
between binary neutron stars detection rate, sky localization and early
warning capabilities.

gravitational wave sources (677); Gravitational wave detectors (676);

##  1 Introduction

The first directly detected gravitational wave (GW) event GW150914 by Advanced
LIGO in September, 2015 opened a new window on exploring the Universe (Abbott
et al., 2016). During the first two observation runs (O1 and O2) of Advanced
LIGO, and Advanced Virgo (joined in O2) (Acernese et al., 2015), a total of
ten binary black holes (BBHs) mergers and one binary neutron star (BNS) merger
were identified (Abbott et al., 2019). The follow-up observations of the
electromagnetic (EM) counterparts of the BNS event have initiated the GW and
EM multi-messenger astronomy era (Abbott et al., 2017). In 2019, the Advanced
LIGO and Advanced Virgo started the third observation run (O3) with improved
sensitivities. O3 has finished and 39 GW candidates from the first half of O3
have been released in 2020 as the second Gravitational-Wave Transient Catalog
(GWTC-2) (Abbott et al., 2020). More recently, a deeper list of candidate
events over the same period of GWTC-2 was reported recently in The LIGO
Scientific Collaboration et al. (2021) as GWTC-2.1, which revealed 44 high-
significance candidates. It is expected that a large number of GW events will
come into our view with the improvement in sensitivity of upcoming the fourth
observation run and more distant third generation detector era.

In the frequency band relevant to ground based GW detectors, the most common
sources of GW events are expected to be compact binary coalescences.
In addition to the exploration of the source properties for a single CBC
system, GW detections could also shed light on the merger rate of CBC system
as a function of distance from us, thus could test theoretical models for CBC
merger rate and further give clues to studies of formation of single CBC
system. For example, the BNS merger rate is constrained to be
$1540^{+3200}_{-1220}\mathrm{Gpc}^{-3}\mathrm{yr}^{-1}$ by the detection of
GW170817 (Abbott et al., 2017); and the latest BNS merger rate is found to be
320${}^{+490}_{-240}\mathrm{Gpc}^{-3}\mathrm{yr}^{-1}$ by the second LIGO-
Virgo Gravitational-Wave Transient Catalog (Abbott et al., 2021). Beyond that,
a fundamental method to estimate merger rate distribution, delay time
distribution (DTD), attracts some attempts to evaluate its validity by current
second-generation GW detectors and future third generation GW detectors
(Safarzadeh & Berger, 2019a; Safarzadeh et al., 2019b, c). As explored in a
series researches not limited to Hachisu et al. (2008); Piran (1992);
Safarzadeh & Berger (2019a); Safarzadeh et al. (2019b, c); Zheng & Ramirez-
Ruiz (2007), the merger rate distribution is a convolution of the cosmic star
formation rate and the distribution of delay time (DTD) between the system
form and merger. Specifically, the DTD is usually parametrized as a power law
distribution with a slope $\Gamma$ above minimum merger timescale
$t_{\text{{min}}}$. In later analysis, we generated GW sources following DTD
distribution to simulate a relative realistic distribution in the Universe.
Systems of CBCs which include at least one neutron star, such as the mergers
of binary neutron star (BNS) or neutron star black hole (NSBH), are likely to
generate observable EM emissions. These emissions include short-duration
gamma-ray bursts, powered by accretion onto the central compact object
(Metzger & Berger, 2012), and an isotropic thermal emission powered by the
radioactive decay of heavy elements in the merger ejecta, known as kilonova
(Metzger et al., 2010), as well as optical and radio afterglows (Hajela et
al., 2019). Successful EM follow-up observations of a GW event can identify
the host galaxy, and provide information on the progenitor local environment
and the hydrodynamics of the merger (Abbott et al., 2017; Branchesi, 2016;
Nuttall & Sutton, 2010). Coupled with the estimates of a model-independent
luminosity distance derived from GWs, it is even possible to infer the values
of the Hubble constant (Abbott et al., 2021; Hotokezaka et al., 2019; Margutti
et al., 2018). In terms of EM follow-up observations, the most needed
information from GWs is the estimate of the sky location of the source, the
size of the error region, and the time at merger. To increase the likelihood
of successful EM follow-up observations, therefore, improvement in
localization of GWs is desired. On the other hand, if the in-band duration of
a GW signal is long enough that its presence can be identified and information
on its sky location obtained well before the merger, astronomers will be
allowed more time to prepare, further increasing the likelihood of successful
EM follow-up observations. This is known as early warning (Cannon et al.,
2012). For BNSs, the in-band duration of a signal mainly depends on the low
cut-off frequency of GW detectors and the masses of the systems. For example,
the in-band duration of the insprial signal of a $1.4M_{\odot}-1.4M_{\odot}$
BNS meger in a GW detector with a low cut-off frequency of $10$ Hz is
$\mathcal{O}(10^{3})$s.
Third generation GW detectors such as the Einstein Telescope (ET) (Punturo et
al., 2014) and Cosmic Explorer (CE) (Abbott et al., 2017) are expected to be
built in the 2030s. With sentivities better than that of the second generation
detectors, observing GWs from binary black hole (BBH) and BNS sources located
at distances far greater than the horizon of second generation detectors will
be possible, thus could contribute to explorations of compact object evolution
so as to further study the evolution of the Universe. Localization of CBC
sources detectable by the second generation detectors and CBC sources at
greater distances could also benefit from improved sensitivity of the third
generation detectors due to higher SNR. And the improved sensitivity of third
generation GW detectors at low frequency band, for example, 1 Hz-10 Hz, allows
for a longer in-band duration (e.g. low cut-off frequency 1 Hz, in-band
duration will be 5 days) for CBC signals as the time to merger is inversely
related to the instantaneous GW frequency which we will introduce later, thus
jointly lead to a promising EM follow-up observations as stated in previous
researches such as Akcay (2018); Akcay et al. (2018); Chan et al. (2018);
Mills et al. (2018); Zhao & Wen (2018). It should be noted that we considered
the earth’s rotation due to the long duration of simulated signals.
In this paper, using a Fisher matrix approach, we estimate the localization
uncertainty and early warning prospect for networks of third generation ground
based GW detectors for BNSs. We simulate BNS mergers at fixed distances as
well as a population following the DTD of a power law distribution with a
slope $\Gamma=-1.5$ and a minimum merger time $t_{\mathrm{min}}=1$ Gyr.

Previous studies have shown the benefits of building a detector in Australia
because of the long baseline that such a detector will form with other
detectors (Barriga et al., 2010; Howell et al., 2018; Nissanke et al., 2011).
This paper aims to characterise the performance of a network of 3 third
generation detectors which consists of different permutations of detectors in
CE and ET configurations. Thus, in addition to the proposed third generation
GW detectors, the ET and CE, we assume an ET-like detector or a CE-like
detector in Australia.
This paper is arranged as follows: in Section 2, the basic information of the
third generation detectors are introduced. In Section 3, we present the
information of sources in simulations and explain the Fisher matrix method
used to estimate the localisation errors. And the results of the simulations
are discussed in Section 4. Finally, in Section 5, a discussion and a
conclusion are provided.

##  2 Third generation detectors

Third generation GW detectors aim to improve the sensitivity across the entire
frequency band. (Abbott et al., 2017; Punturo et al., 2010). At present, the
two proposed third generation detectors are the ET and CE. ET is composed of
three interferometers with 10 km long arms. The interferometers will each have
an opening angle equal to $60^{\circ}$ and each of them will be rotated
relative to the others by $120^{\circ}$, forming an equilateral triangular
structure. Proposed locations for ET include Sardinia, Italy and the border
region between the Netherlands, Belgium and Germany. The target sensitivity of
the ET is shown in Figure 1. For frequency larger than 10Hz, compared to the
Advanced LIGO and Advanced Virgo, the sensitivity of ET will be improved by a
factor of $\sim 10$, and even better for frequencies below $10$Hz. A detailed
description of ET can be found in Punturo et al. (2014, 2010).

Unlike the ET, CE will have the typical L-shaped configuration employed by
second generation detectors, with an arm length of 40 km. For frequencies
$\geq 10$Hz, as shown in Figure 1, the sensitivity is improved by a factor of
$\sim 30$ compared to Advanced LIGO and Advanced Virgo (Abbott et al., 2017;
Dwyer et al., 2015). The planned location for CE is North America. More
extensive technology details about CE can be found in Reitze et al. (2019a,
b).

Figure 1: The sensitivity curve of the second generation detectors LIGO, VIGO
and the third generation detectors ET and CE.

With the improvement of sensitivity, ET is expected to be able to detect
mergers of BNS at redshifts of about $2$, and it will be even higher for CE
(Zhao, 2018). On the one hand, the improved sensitivity of GW detectors at
frequencies $\geq 10$Hz will directly lead to a higher signal-to-noise-ratio
(SNR) so as to improve the localization of GW events. On the other hand, the
improved sensitivity for the frequency band of $1$Hz - $10$Hz will greatly
extend the in-band duration of GW signals, so as to improve the localization
as shown in Chan et al. (2018); Zhao (2018).
Assuming the detector is at the center or a spherical coordinate system, the
antenna pattern of each interferometer of a ET-like detector can be expressed
as follows (Regimbau et al., 2012):

$\centering\begin{array}[]{r}F_{+}^{1}(\theta,\phi,\psi)=-\frac{\sqrt{3}}{4}\left[\left(1+\cos^{2}\theta\right)\sin
2\phi\cos 2\psi\right.\\\ +2\cos\theta\cos 2\phi\sin 2\psi],\\\
F_{\times}^{1}(\theta,\phi,\psi)=\frac{\sqrt{3}}{4}\left[\left(1+\cos^{2}\theta\right)\sin
2\phi\sin 2\psi\right.\\\ -2\cos\theta\cos 2\phi\cos 2\psi],\\\
F_{+,\times}^{2}(\theta,\phi,\psi)=F_{+,\times}^{1}\left(\theta,\phi+\frac{2\pi}{3},\psi\right),\\\
F_{+,\times}^{3}(\theta,\phi,\psi)=F_{+,\times}^{1}\left(\theta,\phi-\frac{2\pi}{3},\psi\right)\end{array}\@add@centering$
(1)

where $F_{+}^{n}$ is the plus polarisation response of the $n$-th
interferometer and $F_{\times}^{n}$ is the cross polarisation response. The
antenna patterns are a function of the azimuthal angle, $\theta$, the polar
angle, $\phi$ and the polarization angle, $\psi$, of the GW source. The
antenna pattern of CE can be expressed as (Schutz, 2011):

$\begin{array}[]{r}F_{+}(\theta,\phi,\psi)=\frac{1}{2}\left(1+\cos^{2}\theta\right)\cos
2\phi\cos 2\psi\\\ -\cos\theta\sin 2\phi\sin 2\psi,\\\
F_{\times}(\theta,\phi,\psi)=-\frac{1}{2}\left(1+\cos^{2}\theta\right)\cos
2\phi\sin 2\psi\\\ -\cos\theta\sin 2\phi\cos 2\psi.\end{array}$ (2)

where, as before, $F_{+}$ is the plus polarisation response for CE and
$F_{\times}$ is the cross polarisation response. In addition, better
sensitivity of detector such as ET in low frequency band enables longer in-
band signal duration, generally, 1 Hz corresponds to about 5 days as explained
in section 3. Therefore, the azimuthal angle and polar angle will change over
time as the earth keeps rotating, which means the antenna pattern will also be
time-dependent. Additionally, due to the long duration of signal, the signal
will experience Doppler effect because the the detector moves relative to the
signal, in terms of Doppler effect, Chan et al. (2018) suggests the Doppler
effect is not important to the localization uncertainty estimation, while for
completeness, we also included the Doppler effect in analysis.
For our study, we assume that the ET is located at the Virgo site in Italy
(longitude, latitude) = ($10.4^{\circ}\text{E},~{}43.7^{\circ}\text{N}$), and
CE at the LIGO Hanford site in the United States (longitude
$-119.41^{\circ}\text{E}$, latitude $46.45^{\circ}\text{N}$). Though it is
likely that the final locations of CE and the ET will not be at these assumed
locations, the impact of any change in location within the United States and
Europe will be small since our study is looking at the results averaged across
a population of simulated signals. Additionally, we consider a third detector
located in Australia (longitude $115.87^{\circ}\text{E}$, latitude
$-31.95^{\circ}\text{N}$) for our study. Specifically, we assume two different
scenarios. In the first scenario, the detector located in Australia is a
detector with sensitivity and configuration identical to those of the ET,
denoted by ET-A, while in the other scenario, the detector will be similar to
CE, referred to as CE-A. To further explore the localisation capabilities of
different combinations of third generation GW detectors, two detectors,
denoted by ET-L and CE-V, are introduced. Similarly, ET-L indicates a ET-like
detector at the location of LIGO Hanford, and CE-V refers to a CE-like
detector at the Virgo site.

We simulate 4 networks of GW detectors. The simulated networks can be found in
Table 1.

| Australia | LIGO Hanford | Italy
---|---|---|---
| ( $115.87^{\circ}\text{E}$,$-31.95^{\circ}\text{N}$) | ($-119.41^{\circ}\text{E}$, $46.45^{\circ}\text{N}$) | ($10.4^{\circ}\text{E}$, $43.7^{\circ}\text{N}$)
2ET1CE | ET-A | CE | ET
3ET | ET-A | ET-L | ET
1ET2CE | CE-A | CE | ET
3CE | CE-A | CE | CE-V
Table 1: The simulated GW detector networks in this paper. The first row is
the locations of the GW detectors in longitude and latitude. The left column
shows the names of the GW detector networks, which directly indicates the
network configuration. ET-A and ET-L represent an ET-like detector in
Australia and the LIGO Hanford site respectively. Similarly, CE-A and CE-V
indicate a CE-like detector located in Australia and Italy.

We use the numbers and the types of the detectors in a network to refer to the
network. For example, 2ET1CE refers to a network consisting of $2$ ET-like
detectors and $1$ CE-like detector, and 3ET to a network consisting of $3$ ET-
like detectors. We estimate the localization uncertainty and the prospect for
early warning for these networks.

##  3 Methodology

###  3.1 Binary neutron star simulations

Generally, the evolution of BNSs is considered to go through three stages:
inspiral, merger and ringdown. The GW signal during the adiabatic inspiral
phase can be well described by post-Newtonian (Abadie et al., 2012). Our major
concern is the localization performance achieved before the merger of BNSs to
evaluate early warning capabilities, thus our demand for an accurate
description of the merger and ringdown is not high. In this context, we use
the TaylorT3 waveform (specifically, Eq (3.10b) in Buonanno et al. (2009))
approximant to simulate signals in this work. In addition to the signal
templates, the duration of simulated signals also need to be taken into
consideration. The duration of simulated signal, in other words, the time
remaining to merger, can be simply estimated according to a leading order
post-Newtonian approximation as follows (Maggiore, 2007):

$\tau_{\mathrm{c}}=\frac{5}{256}\frac{c^{5}}{G^{\frac{5}{3}}}\frac{\left(\pi
f_{\mathrm{s}}\right)^{-\frac{8}{3}}}{\mathcal{M}^{\frac{5}{3}}},$ (3)

where $\tau_{\mathrm{c}}$ is the time to merger, $c$ the speed of light, $G$
the gravitational constant, $\mathcal{M}$ the chirp mass
($\mathcal{M}=\frac{\left(m_{1}m_{2}\right)^{3/5}}{\left(m_{1}+m_{2}\right)^{1/5}}$,
where $m1,m2$ are component masses), and $f_{\mathrm{s}}$ the low cut-off
frequency. It should be noted that component masses, chirp masses, and total
masses ($M=m_{1}+m_{2}$) in our simulations are all redshifted to observed
mass $M_{\mathrm{obs}}$ by:

$M_{\mathrm{obs}}=M_{\mathrm{local}}(1+z).$ (4)

Now assuming a low cut-off frequency of 1 Hz, and a BNS of
$1.4M_{\odot}$-$1.4M_{\odot}$, the in-band duration of the GW signal can be
estimated to be about 130 hours. Naturally, during detection for signals with
duration of 130 hours, the earth keeps rotating, in this context, time-
dependent antenna pattern needs to be applied for a reliable estimation for
localization.
To estimate the localisation for different populations of BNS mergers, two
scenarios are considered, BNSs at fixed distances, and a population of BNSs
with a redshift distribution following the power law DTD described in
Safarzadeh et al. (2019b). For BNSs at fixed distances, we select
$40\text{Mpc}$, $200\text{Mpc}$, $400\text{Mpc}$, $800\text{Mpc}$ and
$1600\text{Mpc}$ as examples. Specifically, 100 BNSs are simulated for each
distance. $500$ BNSs sources following the DTD of redshift up to $2$ are
simulated.

The mass of the simulated BNSs are all $1.4M_{\odot}$-$1.4M_{\odot}$ in local
frame. The sky locations (the right ascension $\alpha$ and declination
$\delta$) of the simulated BNS mergers for all cases are randomly sampled to
simulate uniform distribution in the whole sky. The cosine of inclination
angle $cos~{}\iota$ are randomised in range between -1 and 1, polarization
angle $\psi$ are also randomised in range between 0 and $2\pi$.
For the BNS mergers following the DTD used in this work, the BNS merger rate
as a function of redshift $z$ can be expressed as,

$R(z)=\int_{z_{b}=10}^{z_{b}=z}\lambda\frac{dP_{m}}{dt}\left(t-t_{b}-t_{\min}\right)SFR\left(z_{b}\right)\frac{dt}{dz}\left(z_{b}\right)dz_{b}.$
(5)

Here, $R(z)$ represents the BNS intrinsic merger rate density
($\text{Mpc}^{-3}\text{year}^{-1}$), and $SFR$ represents cosmic star
formation rate density ($\text{M}_{\odot}\text{yr}^{-1}\text{Mpc}^{-3}$).
$\lambda$ is the BNS mass efficiency which is assumed to be a constant
$10^{-5}\mathrm{M}_{\odot}^{-1}$, $t_{b}$ is the time at the redshift $z_{b}$
and $dP_{m}/dt$ is the DTD, which is assumed to follow a power law
distribution with a minimum delay time, and $\Gamma$ as the power law index,
given by

$dP_{m}/dt\propto t^{\Gamma},$ (6)

$t_{\text{min}}$ in Eq (5) is the minimum delay time which corresponds to the
time since the star first joins the main sequence on the Hertzsprung-Russell
diagram. In this paper, we adopt $t_{\text{{min}}}=1$Gyr and $\Gamma=-1.5$.
The derivative of time, $dt/dz$, with respect to redshift is given by

$dt/dz=-\left[(1+z)E(z)H_{0}\right]^{-1},$ (7)

where

$E(z)=\sqrt{\Omega_{m,0}(1+z)^{3}+\Omega_{k,0}(1+z)^{2}+\Omega_{\Lambda}(z)},$
(8)

where $\Omega_{M}$ is the matter density, $\Omega_{\Lambda}$ vacuum density ,
$\Omega_{K}$ curvature, and $H_{0}$ the Hubble constant. We adopt the values
from the Planck 2015 results (Planck Collaboration et al., 2016). In
particular, $\Omega_{M}=0.308$, $\Omega_{\Lambda}=0.692$, $\Omega_{K}=0$, and
$H_{0}=67.8~{}\mathrm{km~{}s^{-1}~{}Mpc^{-1}}$ respectively. For the cosmic
star formation rate density $SFR\left(z\right)$, we adopted the following
formula from (Safarzadeh et al., 2019b; Madau & Dickinson, 2014):

$SFR(z)=0.015\frac{(1+z)^{2.7}}{1+[(1+z)/2.9]^{5.6}}\mathrm{M}_{\odot}\mathrm{yr}^{-1}\mathrm{Mpc}^{-3},$
(9)

where $z$ represents the redshift. In Eq. 5 we chose the lower limit of
integration to be redshift of 10 based on Safarzadeh et al. (2019b), because
the choice of maximum redshift has little impact on the calculation. In the
case of $t_{\text{{min}}}$ = 1 Gyr, $\Gamma=-1.5$, we calculated its
corresponding BNS merger rate density for a redshift range from 0 to 2 via Eq.
5. And by assuming the integral of BNS merger rate density over redshift range
from 0 to 2 is 1 (that is, all sources are distributed within redshift range
of 0 - 2), the redshift distribution (namely, the probability density function
of redshift) of sources could be obtained as shown in Figure 2. We simply
chose 2 as maximum redshift cut-off.

Figure 2: The redshift distribution of simulated sources. The x-axis
represents the redshift. The y-axis represents the normalized distribution
which makes the area under the curve is 1. The orange line is the redshift
distribution calculated using Eq. 5 with
$t_{\text{{min}}}=1\text{Gyr},\Gamma=-1.5$, and the blue line is the
distribution of simulated sources.

###  3.2 Fisher matrix analysis

The Fisher matrix translates errors on observed quantities measured directly
into constraints on parameters of interest in the underlying model (Bassett et
al., 2011), which has tested as an applicable tool in a series of researches
(Fairhurst, 2009; Wen & Chen, 2010; Fairhurst, 2011; Chan et al., 2018; Zhao &
Wen, 2018; Mills et al., 2018) to estimate localization errors for GW
detectors with not high computation cost compared to other methods. As
elaborated in Vallisneri (2008), the inverse Fisher matrix represents the
covariance matrix of parameter errors for the true signal, which could also be
interpreted as the frequentist error covariance for the maximum-likelihood
parameter estimator assuming Gaussian noise in the case of high SNR limit
(that means, the waveforms could be considered as linear functions of source
parameters). And in the high SNR limit, the maximum-likelihood naturally
achieves the Cramér-Rao bound. Therefore that’s why we set a SNR threshold for
further localization uncertainty analysis. Only those signals detected with
SNR larger than or equal to SNR threshold are seen as detectable signals. In
this paper, we chose a network SNR of 8 as detectable threshold. In addition,
for waveforms of low SNR or waveforms with poor priors estimation for its
parameters, the Fisher matrix could not be applied to parameter estimation of
GW data credibly. Also, it’s worth noting that the detector network consisting
of three detectors significantly improves the sky localization accuracy
compared to two-detector networks and thus enhances the reliability to
estimate localization error via the Fisher matrix method (Berry et al., 2015).
We consider a vector, $\boldsymbol{\theta}$, of $9$ parameters in our Fisher
matrix analysis as in Chan et al. (2018): $\alpha$; $\delta$; the arrival time
of the signal at the center of the earth, $t_{0}$; the log of distance,
$log_{10}d$; polarization angle, $\psi$; the log of the total mass,
$log_{10}M$; the cosine of the inclination angle, $cos~{}\iota$; the symmetric
mass ratio, $\eta=m_{1}\times m_{2}/M^{2}$; the initial phase of the wave when
it arrives at the center of the earth, $\phi_{0}$. The elements of
$\boldsymbol{\theta}$ in the Fisher matrix then could be obtained by,

$\rm FIM_{ij}=\sum_{d=1}^{N}2\int_{0}^{\infty}\frac{\frac{\partial
h_{d}^{*}}{\partial\theta_{i}}\frac{\partial
h_{d}}{\partial\theta_{j}}+\frac{\partial
h_{d}^{*}}{\partial\theta_{j}}\frac{\partial
h_{d}}{\partial\theta_{i}}}{P_{d}}df,$ (10)

where $h_{\text{d}}$ represents the incoming gravitational wave strain in the
frequency domain at the $\text{d}^{th}$ detector. $\frac{\partial
h_{\text{d}}}{\partial\theta_{i}}$ denotes the partial derivative of
$h_{\text{d}}$ with respect to the $i^{th}$ unknown parameter $\theta_{i}$.
$P_{\text{d}}$ refers to the power spectral density for $\text{d}^{th}$
interferometer. The optimal SNR, $\rho_{\text{d}}$, for the $\text{d}^{th}$
detector can be expressed as,

$\rho_{\text{d}}=\sqrt{4\int_{0}^{\infty}\frac{\left|h_{\text{{d}}}(\boldsymbol{\theta},f)\right|^{2}}{P_{\text{d}}(f)}df}.$
(11)

The combined SNR for a signal from a network of detectors can be obtained by

$\rho_{\mathrm{network}}=\sqrt{\sum_{\text{d}=1}^{N}\rho_{\text{d}}^{2}}.$
(12)

In simulation, we divide the strain data in the time domain into S segments
with length of 100 seconds (the last segment shorter than 100 seconds), and
use Eq. 10 to calculate the Fisher matrix for each segment. The Fisher matrix
for the entire signal is then obtained by summing the Fisher matrices over the
segments,

$\text{FIM}_{ij}^{\text{t}}=\sum_{k=1}^{S}\text{FIM}_{ij}^{k},$ (13)

where t represents variable for the entire signal, and $k$ indicates the
$k^{\text{th}}$ segment. The total accumulated SNR of single detector for the
entire signal $\rho_{d}^{t}$ is obtained by

$\rho_{d}^{\text{t}}=\sqrt{\sum_{k=1}^{S}\left(\rho_{d}^{k}\right)^{2}},$ (14)

where $\rho_{d}^{k}$ represents SNR of $k^{\text{th}}$ segment for
$d^{\text{th}}$ detector. The covariance matrix can then be derived by
computing the inverse of the Fisher matrix,

${\text{cov}_{ij}^{\text{t}}={(\text{FIM}_{ij}^{\text{t}})}^{-1}.}$ (15)

The localization uncertainty $\Delta\Omega$ is given by,

$\Delta\Omega=\pi\sqrt{\lambda_{\alpha}\lambda_{\delta}}\cos\delta,$ (16)

where $\lambda_{\alpha}$ and $\lambda_{\delta}$ are the eigenvalues of
$\mathrm{cov}^{t}_{ij}$ corresponding to $\alpha$ and $\delta$. Finally, the
localization uncertainty at confidence level $p$ is obtained by,

$\Delta\Omega_{p}=-2\log(1-p)\Delta\Omega.$ (17)

##  4 Results

We use a network SNR of $8$ to determine whether an event is detectable. To
avoid ambiguity, the localization errors shown in the remaining of this paper
are all at $90\%$ confidence.

###  4.1 Sources at fixed distances

####  4.1.1 Localization uncertainty

Almost all BNS sources simulated at the chosen fixed distances were observed
with network SNR greater than 8. In Table 2, we list the percentage of the
detectable sources which can be localised to within $30\mathrm{deg^{2}}$,
$10\text{deg}^{2}$, $5\text{deg}^{2}$, and $1\mathrm{deg^{2}}$ with each
detector network.

All simulated BNS signals at 40 Mpc are detectable to all simulated detector
networks and can be localised to an area smaller than $1\mathrm{deg^{2}}$ and
are, thus, not shown in Table 2. We also note that all sources at 200 Mpc are
localized to within $1\text{deg}^{2}$ by all networks. As expected, the
localization uncertainty gets increases with distance increases. Less than 10%
sources at 1600 Mpc could be detected within $1\text{deg}^{2}$ by the 3ET
network and 2ET1CE networks, while 1ET2CE and 3CE can localize respectively
16% and 40% sources.

|  | 3ET | 2ET1CE | 1ET2CE | 3CE
---|---|---|---|---|---
200Mpc | $30\mathrm{deg^{2}}$ | 100% | 100% | 100% | 100%
$10\mathrm{deg^{2}}$ | 100% | 100% | 100% | 100%
$5\mathrm{deg^{2}}$ | 100% | 100% | 100% | 100%
$1\mathrm{deg^{2}}$ | 100% | 100% | 100% | 100%
400Mpc | $30\mathrm{deg^{2}}$ | 100% | 100% | 100% | 100%
$10\mathrm{deg^{2}}$ | 100% | 100% | 100% | 100%
$5\mathrm{deg^{2}}$ | 100% | 100% | 100% | 100%
$1\mathrm{deg^{2}}$ | 91% | 89% | 95% | 96%
800Mpc | $30deg^{2}$ | 100% | 100% | 100% | 100%
$10\mathrm{deg^{2}}$ | 100% | 99% | 98% | 98%
$5\mathrm{deg^{2}}$ | 95% | 93% | 96% | 96%
$1\mathrm{deg^{2}}$ | 50% | 53% | 64% | 77%
1600Mpc | $30\mathrm{deg^{2}}$ | 99% | 99% | 98% | 98%
$10\mathrm{deg^{2}}$ | 80% | 79% | 89% | 91%
$5\mathrm{deg^{2}}$ | 59% | 58% | 70% | 82%
$1\mathrm{deg^{2}}$ | 7% | 9% | 16% | 40%
Table 2: A table showing the fraction of detectable sources which can be localised to within $\mathrm{30deg^{2},10deg^{2},5deg^{2}}$, and $1\text{deg}^{2}$ with the GW networks. |  |  3ET ($\text{deg}^{2}$) |  2ET1CE ($\text{deg}^{2}$) |  1ET2CE ($\text{deg}^{2}$) |  3CE ($\text{deg}^{2}$)
---|---|---|---|---|---
200Mpc |  90% |  $0.23$ |  $0.25$ |  $0.18$ |  $0.14$
50% |  $0.06$ |  $0.06$ |  $0.04$ |  $0.02$
10% |  $0.02$ |  $0.02$ |  $0.01$ |  $0.01$
400Mpc |  90% |  $0.94$ |  $1.02$ |  $0.72$ |  $0.56$
50% |  $0.23$ |  $0.22$ |  $0.15$ |  $0.09$
10% |  $0.07$ |  $0.06$ |  $0.06$ |  $0.02$
800Mpc |  90% |  $3.75$ |  $4.07$ |  $2.89$ |  $2.25$
50% |  $0.93$ |  $0.91$ |  $0.60$ |  $0.35$
10% |  $0.26$ |  $0.26$ |  $0.23$ |  $0.09$
1600Mpc |  90% |  $14.58$ |  $14.65$ |  $10.07$ |  $9.01$
50% |  $3.69$ |  $3.57$ |  $2.30$ |  $1.39$
10% |  $1.06$ |  $1.03$ |  $0.90$ |  $0.38$
Table 3: The 90% credible regions of localization uncertainty of every
detector network for binary neutron star mergers respectively at 200 Mpc, 400
Mpc, 800 Mpc, 1600 Mpc with SNR$>$8\. $90\%$, $50\%$ and $10\%$ respectively
represents the best localized 100%, 90% and 50% of the detectable sources.

Figure 3: The cumulative distributions of the size of the $90\%$ credible
regions for the BNS mergers at 200 Mpc and at 1600 Mpc. The x-axes show the
size of the 90% credible region and the upper limit of the x-axes corresponds
to the size of the whole sky.

(a) $1\text{deg}^{2}$

(b) $5\text{deg}^{2}$

(c) $10\text{deg}^{2}$

(d) $30\text{deg}^{2}$

Figure 4: Histograms showing the distributions of time to merger for the BNS
mergers at $200$ Mpc with 4 detector networks. The x-axis is the time to
merger when the signal meets the early warning criteria. The y-axis is the
fraction of detectable events that achieve these early warning criteria. The
percentage of sources that could be detected within required localization
uncertainty could be found in Table 2.

The cumulative distribution of the size of the 90% credible regions for the
BNS mergers at fixed distances are presented in Figure 4. Results for 200 Mpc
and 1600 Mpc are shown as an example for simplicity here and results of other
fixed distances are presented in Appendix. We note that the relative behaviour
of the 4 networks remain largely unchanged for sources at 200 Mpc and 1600
Mpc. Taking 200 Mpc as an example, there’s a trend of gradual change from 3CE
to 1ET2CE to 2ET1CE network which gradually accumulate more higher values of
localization uncertainty. Additionally, the cumulative distribution of 3ET
network is very similar to that of 2ET1CE network.
In Table 3, we present the upper limits of the size of the $90\%$ credible
regions for the $90\%$, $50\%$ and $10\%$ best localized sources.
Particularly, $90\%$ of the best localized sources at 200 Mpc can be
constrained to within an area of $\sim 0.3\text{deg}^{2}$ with all 4 networks.
Even for sources at a distance equal to $1600$Mpc, the size of the $90\%$
credible region is still smaller than $15\text{deg}^{2}$, indicating a
promising prospect for EM follow-up observations. Furthermore, 50% of the best
localized sources respectively at 200 Mpc, 400 Mpc, 800 Mpc, 1600 Mpc, could
be limited to within $0.07\text{deg}^{2}$, $0.30\text{deg}^{2}$,
$1.00\text{deg}^{2}$, $4.00\text{deg}^{2}$ by all GW networks. We also found
that the localization uncertainty achieved by 2ET1CE for 90% best localized
sources at any individual distance is $\sim 1.4$ times of that of 1ET2CE
network. And it becomes $\sim 1.5$ times for 50% best localized sources. For
10% best localized sources, the ratio ranges from 1 to 1.18. Overall, as shown
in this table, 2ET1CE network has localization uncertainty no less than 1ET2CE
network for simulated sources at fixed distances.

####  4.1.2 Early warning capabilities

As mentioned in Section 3, the in-band duration of the GW signals from BNS
mergers have been significantly extended for third generation detectors. It is
therefore possible for a signal to accumulate enough SNR to be deemed
significant before the merger. This in turn makes it possible that a trigger
may be released prior to merger, increasing the likelihood of a successful EM
follow-up observation.

To evaluate the early warning prospect of the simulated networks of GW
detectors, we require two conditions to be satisfied before a trigger can be
released. The first condition is that the signal has to accumulate a network
SNR of no less than $8$ at the time the trigger is released, and the other
condition is that the source of the signal has to be localised to an error no
greater than a given certain area with $90\%$ confidence. The second condition
is required because a signal may accumulate enough SNR to the point where the
event is considered significant prior to merger, but the localisation error is
still too large for any meaningful EM follow-up observations to be carried
out. These two conditions will be referred to as early warning criteria in the
remaining of this paper. Furthermore, in terms of the second condition of
early warning, we chose localization area of respectively
$30~{}\text{deg}^{2}$, $10~{}\text{deg}^{2}$, $5~{}\text{deg}^{2}$ and
$1~{}\text{deg}^{2}$ as requirements to evaluate corresponding time to merger
distribution under these conditions.
Figure 4 presents the histogram distribution of time to merger of 4 detector
network for BNSs at 200Mpc, where each subplot denotes the time to merger
distribution given maximum allowable localization uncertainty of respectively
$\mathrm{30~{}deg^{2}}$, $10~{}\mathrm{deg^{2}}$, $5~{}\mathrm{deg^{2}}$ and
$1~{}\mathrm{deg^{2}}$. It should be noted that the sum of percentage for each
network may not be 100% because there are some sources could be detected
within the required localization uncertainty only at the time of merger. Early
warning of BNSs at 200Mpc will be discussed in detail here, the related
figures of BNSs at other fixed distances could be found in Appendix.
Given the maximum localization uncertainty of $30\mathrm{deg^{2}}$, 100% BNS
signals at 200 Mpc could be detected by all tested GW detector networks at
least 100 seconds before merger. The time to merger of BNSs detected by 1ET2CE
network ranges from 5 minutes to 6 hours, and is concentrated in range of 10
minutes to 1 hour. While for 2ET1CE network, it ranges from 5 minutes to 10
hours, and is mainly distributed between 10 minutes and 6 hours. The range of
time to merger indicates a significant possibility of third generation GW
detector networks to give early warning alert to EM follow-up observations.
When the maximum allowable localization uncertainty decreases, to
$10\mathrm{deg^{2}}$, $5\mathrm{deg^{2}}$ and $1\mathrm{deg^{2}}$, detectable
sources which meet the early warning criteria become less and less, part of
BNS signals could only meet the early warning criteria at the time of merger,
and the distribution of time to merger move towards to a direction of smaller
value.

###  4.2 Astrophysical population

####  4.2.1 Signal-to-noise-ratio distributions

For the BNS mergers following the DTD, the SNR distribution is presented in
Figure 5. In total, there are 63.0%, 89.8%, 97%, 99.4% BNS mergers detectable
to 3ET, 2ET1CE, 1ET2CE, 3CE respectively as listed in Table 4. For networks
consisting of more CE-like detectors such as 1ET2CE and 3CE, the distribution
of SNRs peak at higher value, which is largely due to the better sensitivity
of CE at higher frequencies.

Figure 5: The probability density distributions of the SNR for the BNS which
follow the DTD. The horizontal axis represents the SNR in log scale. The
network combined SNR threshold of 8 is denoted by a red dashed vertical line.
The area under the histogram is normalized to be 1.

####  4.2.2 Localization uncertainty

For all simulated BNS sources following the DTD, all detector networks can
localize BNS signals within $\mathcal{O}\left(10^{2}\right)\text{deg}^{2}$.
The cumulative distribution of size of the $90\%$ credible regions are shown
in Figure 7. It should be noted that only localization results of those
detectable sources as described in Sec 4.2.1 are presented in Figure 7. As we
can see, the curve of 2ET1CE, 1ET2CE, and 3CE have a trend of gradual change,
toward a direction of occupying more proportion at smaller values of
localization uncertainty. Behavior of 3ET is deviated from prediction of this
gradual change, which originates from detectable source selection effect. The
upper limits of the localisation errors for the $90\%$, $50\%$ and $10\%$ best
localised sources are shown in Table 4. It could be found from Table 4 that
for best localized 90% detectable sources, 2ET1CE network could localize
sources within $91.79\text{deg}^{2}$, and maximum localization uncertainty of
1ET2CE network shrinks by about 0.38 times compared to that of 2ET1CE network.
In addition, 1ET2CE network could localize best localized half of detectable
sources within $12.54\text{deg}^{2}$, which is about 0.28 times smaller than
$17.57\text{deg}^{2}$ of 2ET1CE network. For the best localized 10% of
detectable sources, the differences are smaller, but the trend remains the
same, 1ET2CE could localize sources within $1.17\text{deg}^{2}$ which is still
less than $1.37\text{deg}^{2}$ of 2ET1CE network.

|  3ET ($\text{deg}^{2}$) |  2ET1CE ($\text{deg}^{2}$) |  1ET2CE ($\text{deg}^{2}$) |  3CE ($\text{deg}^{2}$)
---|---|---|---|---
|  63.0% |  89.8% |  97% |  99.4%
90% |  $31.59$ |  $91.79$ |  $56.77$ |  $44.68$
50% |  $10.25$ |  $17.57$ |  $12.54$ |  7.02
10% |  $0.97$ |  $1.37$ |  $1.17$ |  0.61
Table 4: A table showing the upper limits for the size of the 90% credible
regions for the best localized $90\%$, $50\%$ and $10\%$ of the detectable
sources following DTD. The second row denotes the percentage of the detectable
sources for the detector networks.

Figure 6: The cumulative distribution of the size of the $90\%$ credible
region for the BNS mergers following the DTD. The x-axis show the size of 90%
credible region in deg2.

(a) 3ET

(b) 2ET1CE

(c) 1ET2CE

(d) 3CE

Figure 7: Two-dimensional histograms showing the distributions of the time to
merger for BNS mergers following DTD. The horizontal axis of every subplot is
the size of the $90\%$ credible region, and the vertical axis represents the
time to merger. The color represents the number of sources which achieve the
early warning criteria with the localisation requirement being $\leq
30\mathrm{deg^{2}}$ .

####  4.2.3 Early warning capabilities

The results are presented in Figure 7 as a two-dimensional histogram showing
the distribution of time to merger versus the size of $90\%$ credible region
for BNS mergers following DTD. The chosen value of $30\text{deg}^{2}$ is
motivated by the fact that all simulated events at $1600$Mpc can be localised
to within such an area with all simulated networks. For comparison, we also
employ $100\text{deg}^{2}$ as the value for the second condition. The result
is shown in Appendix.

From Figure 7, 3ET network and 2ET1CE network both could detect several BNS
signals more than 10 hours before merger with localization uncertainty smaller
than $\mathrm{30deg^{2}}$. In terms of the detectable source quantity, more CE
included network could detect more sources which achieve the early warning
criteria. But in terms of the time to merger distribution, it is obvious that
early warning of more ET included network are more widely distributed and
peaked at higher value in vertical direction. For example, for 1ET2CE network,
which replaces one CE with ET in 3CE network, the upper right corner of the
figure fills in the blanks, which means joining of ET improves the early
warning performance in some extent, and could give an early warning alert as
early as 10 hours. Besides, compared to 1ET2CE network, 2ET1CE network could
still detect some signals as early as 10 hours while with smaller localization
uncertainty, which indicates a greater opportunity to perform successful EM
follow-up observations.

##  5 Conclusion and Discussion

GW detections are rapidly evolving as a routine tool in the multi-messenger
era. Maximizing GW’s scientific output by associating with its EM counterpart
is our major concern in multi-messenger astronomy. Localization and early
warning performance of GW detector largely affects the efficiency of EM
follow-up operation. In this paper, we calculated and compared the
localization uncertainty and early warning performance of four third
generation ground based GW detector networks for BNS sources at fixed distance
and BNS sources following DTD. The duration of simulated signals is about 5
days long, therefore we included the earth rotation effect. And considering
that during detection, the detector has motion relative to source, thus we
also included this Doppler effect. Besides ET to be built in Europe, and CE to
be built in North America, we also simulated four detector networks involving
combinations of the aforementioned detectors and the following: ET in
Australia (ET-A), CE in Australia (CE-A), ET in North America and CE in Italy.
The comparison of localization capabilities of ET-A included network and CE-A
included network is our major concern.

In terms of SNR, for a BNS population following the DTD approach, networks
with more CE detectors tends to have higher SNRs due to CE having better
sensitivity at medium to high frequency band, lead to a smaller source
localization uncertainty. Using Fisher matrix analysis, 90% of the best
localized sources can be localized within an area of respectively
$31.59\text{deg}^{2}$, $91.79\text{deg}^{2}$, $56.77\text{deg}^{2}$ and
$44.68\text{deg}^{2}$ for 3ET, 2ET1CE, 1ET2CE, 3CE network. The 3ET network
seems to give an excellent source localization estimate but only because
poorly localized sources are rejected due to their SNR not being above the
required threshold of 8. For BNS mergers at fixed distances, the comparison of
localization performance between tested networks remain basically similar to
that of BNS mergers following DTD.

In terms of time to merger distribution, for BNSs at 200 Mpc, all tested
networks achieved the early warning criteria of $30\text{deg}^{2}$ at least
100 seconds before merger, and the time to merger distribution is widely
distributed in range of 100 seconds to 10 hours, which indicates a bright
future for successful EM follow-up observations in third generation GW
detector era. And for both BNS mergers at fixed distances and following DTD,
more ET included network is more likely to detect BNS signals earlier because
the better sensitivity of ET in low frequency band, leading to BNS signals
being in-band for longer.

Overall, compared with the 2ET1CE network, the 1ET2CE network detects more BNS
signals with achieving a better localization uncertainty. However, the 2ET1CE
network tends to giving alert earlier than the 1ET2CE network. All things
considered, we think building a CE-like detector in Australia to create the
1ET2CE network is a reasonable compromise, as this would lead to an excellent
localization performance without losing too much early warning performance
compared to 2ET1CE network.

We are grateful to Prof. Mohammadtaher Safarzadeh and Prof. Seyed Alireza
Mortazavi for helpful discussion about delay time distribution of binary
neutron stars. We would like to thank Jan Harms and Christopher Berry for
reviewing the paper during the internal LIGO review. We are also grateful for
computational resources provided by Cardiff University, and funded by an STFC
grant supporting UK Involvement in the Operation of Advanced LIGO. Y. L
acknowledges scholarship provided by the University of Chinese Academy of
Sciences (UCAS). Y. L is supported by the National Program on Key Research and
Development Project through grant No. 2016YFA0400804, and by the National
Natural Science Foundation of China with grant No. Y913041V01, and by the
Strategic Priority Research Program of the Chinese Academy of Sciences through
grant No. XDB23040100.

## References

  * Abadie et al. (2012) Abadie, J., Abbott, B. P., Abbott, R., et al. 2012, Phys. Rev. D, 86, 069903
  * Abbott et al. (2016) B. P. Abbott et al. 2016, Phys. Rev. Lett., 116, 061102
  * Abbott et al. (2017) Abbott, B. P., Abbott, R., Abbott, T. D., et al. 2017, ApJ, 850, L40. doi:10.3847/2041-8213/aa93fc
  * Abbott et al. (2017) Abbott, B. P., Abbott, R., Abbott, T. D., et al. 2017, Classical and Quantum Gravity, 34, 044001. doi:10.1088/1361-6382/aa51f4
  * Abbott et al. (2017) Abbott, B. P., Abbott, R., Abbott, T. D., et al. 2017, Phys. Rev. Lett., 119, 161101
  * Abbott et al. (2019) Abbott, B. P., Abbott, R., Abbott, T. D., et al. 2019, Physical Review X, 9, 031040
  * Abbott et al. (2020) Abbott, R., Abbott, T. D., Abraham, S., et al. 2020, arXiv:2010.14527
  * Abbott et al. (2021) Abbott, B. P., Abbott, R., Abbott, T. D., et al. 2021, ApJ, 909, 218. doi:10.3847/1538-4357/abdcb7
  * Abbott et al. (2021) Abbott, R., Abbott, T. D., Abraham, S., et al. 2021, ApJ, 913, L7. doi:10.3847/2041-8213/abe949
  * Acernese et al. (2015) Acernese, F., Agathos, M., Agatsuma, K., et al. 2015, Classical and Quantum Gravity, 32, 024001. doi:10.1088/0264-9381/32/2/024001
  * Akcay (2018) Akcay, S. 2018, Annalen der Physik, 531, 1. doi:10.1002/andp.201800365
  * Akcay et al. (2018) Akcay, S., Fraser, M., & Martin-Carrillo, A. 2018, arXiv:1812.07307
  * Akutsu et al. (2018) Akutsu, T., Ando, M., Arai, K., et al. 2018, arXiv e-prints, arXiv:1811.08079
  * Berry et al. (2015) Berry, C. P. L., Mandel, I., Middleton, H., et al. 2015, ApJ, 804, 114. doi:10.1088/0004-637X/804/2/114
  * Essick et al. (2017) Essick, R., Vitale, S., & Evans, M. 2017, Phys. Rev. D, 96, 084004. doi:10.1103/PhysRevD.96.084004
  * ET science team (2011) Amaro-Seoane P et al. 2009, Einstein Telescope Conceptual design study. ET internal note: ET-0106C-10
  * Barriga et al. (2010) Barriga, P., Blair, D. G., Coward, D., et al. 2010, Classical and Quantum Gravity, 27, 084005. doi:10.1088/0264-9381/27/8/084005
  * Bassett et al. (2011) Bassett, B. A., Fantaye, Y., Hlozek, R., et al. 2011, International Journal of Modern Physics D, 20, 2559
  * Branchesi (2016) Branchesi, M. 2016, Journal of Physics Conference Series, 718, 022004. doi:10.1088/1742-6596/718/2/022004
  * Buonanno et al. (2009) Buonanno, A., Iyer, B. R., Ochsner, E., et al. 2009, Phys. Rev. D, 80, 084043
  * Cannon et al. (2012) Cannon, K., Cariou, R., Chapman, A., et al. 2012, ApJ, 748, 136. doi:10.1088/0004-637X/748/2/136
  * Chan et al. (2018) Chan, M. L., Messenger, C., Heng, I. S., et al. 2018, Phys. Rev. D, 97, 123014
  * Dwyer et al. (2015) Dwyer, S., Sigg, D., Ballmer, S. W., et al. 2015, Phys. Rev. D, 91, 082001. doi:10.1103/PhysRevD.91.082001
  * Fairhurst (2009) Fairhurst, S. 2009, New Journal of Physics, 11, 123006. doi:10.1088/1367-2630/11/12/123006
  * Fairhurst (2011) Fairhurst, S. 2011, Classical and Quantum Gravity, 28, 105021. doi:10.1088/0264-9381/28/10/105021
  * Hachisu et al. (2008) Hachisu, I., Kato, M., & Nomoto, K. 2008, ApJ, 683, L127. doi:10.1086/591646
  * Hajela et al. (2019) Hajela, A., Margutti, R., Alexander, K. D., et al. 2019, ApJ, 886, L17. doi:10.3847/2041-8213/ab5226
  * Hotokezaka et al. (2019) Hotokezaka, K., Nakar, E., Gottlieb, O., et al. 2019, Nature Astronomy, 3, 940. doi:10.1038/s41550-019-0820-1
  * Howell et al. (2018) Howell, E. J., Chan, M. L., Chu, Q., et al. 2018, MNRAS, 474, 4385. doi:10.1093/mnras/stx3077
  * Madau & Dickinson (2014) Madau, P. & Dickinson, M. 2014, ARA&A, 52, 415
  * Maggiore (2007) M. Maggiore, Gravitational Waves. Vol. 1: Theory and Experiments, (Oxford University Press, Oxford, England, 2007)
  * Margutti et al. (2018) Margutti, R., Cowperthwaite, P., Doctor, Z., et al. 2018, arXiv e-prints, arXiv:1812.04051
  * Metzger et al. (2010) Metzger, B. D., Martínez-Pinedo, G., Darbha, S., et al. 2010, MNRAS, 406, 2650. doi:10.1111/j.1365-2966.2010.16864.x
  * Metzger & Berger (2012) Metzger, B. D., & Berger, E. 2012, ApJ, 746, 48
  * Mills et al. (2018) Mills, C., Tiwari, V., & Fairhurst, S. 2018, Phys. Rev. D, 97, 104064. doi:10.1103/PhysRevD.97.104064
  * Nissanke et al. (2011) Nissanke, S., Sievers, J., Dalal, N., et al. 2011, ApJ, 739, 99. doi:10.1088/0004-637X/739/2/99
  * Nuttall & Sutton (2010) Nuttall, L. K. & Sutton, P. J. 2010, Phys. Rev. D, 82, 102002. doi:10.1103/PhysRevD.82.102002
  * Piran (1992) Piran, T. 1992, ApJ, 389, L45. doi:10.1086/186345
  * Planck Collaboration et al. (2016) Planck Collaboration, Ade, P. A. R., Aghanim, N., et al. 2016, A&A, 594, A13
  * Punturo et al. (2010) Punturo, M., Abernathy, M., Acernese, F., et al. 2010, Classical and Quantum Gravity, 27, 194002
  * Punturo et al. (2014) Punturo, M., Lück, H., & Beker, M. 2014, Advanced Interferometers and the Search for Gravitational Waves, 333
  * Regimbau et al. (2012) Regimbau, T., Dent, T., Del Pozzo, W., et al. 2012, Phys. Rev. D, 86, 122001
  * Reitze et al. (2019a) Reitze, D., LIGO Laboratory: California Institute of Technology, LIGO Laboratory: Massachusetts Institute of Technology, et al. 2019, BAAS, 51, 141
  * Reitze et al. (2019b) Reitze, D., Adhikari, R. X., Ballmer, S., et al. 2019, BAAS
  * Safarzadeh & Berger (2019a) Safarzadeh, M. & Berger, E. 2019, ApJ, 878, L12. doi:10.3847/2041-8213/ab24df
  * Safarzadeh et al. (2019b) Safarzadeh, M., Berger, E., Ng, K. K. Y., et al. 2019, ApJ, 878, L13
  * Safarzadeh et al. (2019c) Safarzadeh, M., Berger, E., Leja, J., et al. 2019, ApJ, 878, L14. doi:10.3847/2041-8213/ab24e3
  * Sathyaprakash et al. (2011) Sathyaprakash, B., Abernathy, M., Acernese, F., et al. 2011, arXiv e-prints, arXiv:1108.1423
  * Schutz (2011) Schutz, B. F. 2011, Classical and Quantum Gravity, 28, 125023. doi:10.1088/0264-9381/28/12/125023
  * The LIGO Scientific Collaboration et al. (2021) The LIGO Scientific Collaboration, the Virgo Collaboration, Abbott, R., et al. 2021, arXiv:2108.01045
  * Vallisneri (2008) Vallisneri, M. 2008, Phys. Rev. D, 77, 042001. doi:10.1103/PhysRevD.77.042001
  * Vitale & Evans (2017) Vitale, S. & Evans, M. 2017, Phys. Rev. D, 95, 064052. doi:10.1103/PhysRevD.95.064052
  * Wen & Chen (2010) Wen, L. & Chen, Y. 2010, Phys. Rev. D, 81, 082001. doi:10.1103/PhysRevD.81.082001
  * Zhao (2018) Zhao, W. 2018, Scientia Sinica Physica, Mechanica & Astronomica, 48, 079805
  * Zhao & Wen (2018) Zhao, W. & Wen, L. 2018, Phys. Rev. D, 97, 064031. doi:10.1103/PhysRevD.97.064031
  * Zheng & Ramirez-Ruiz (2007) Zheng, Z. & Ramirez-Ruiz, E. 2007, ApJ, 665, 1220. doi:10.1086/519544

##  Appendix A Appendix

The cumulative distribution of localization uncertainty for BNS mergers at 40
Mpc, 400 Mpc, 800 Mpc are shown in Figure 8. In addition, Figure 9, Figure 10,
Figure 11, Figure 12 are the histogram distribution of time to merger
respectively for BNS mergers at 40 Mpc, 400 Mpc, 800 Mpc, 1600 Mpc. Finally,
Figure 13 show the two-dimensional histograms distributions of the time to
merger for BNS mergers following DTD with maximum allowable region of 100
deg2.

Figure 8: The cumulative distributions of the size of the $90\%$ credible
regions for the BNS mergers at 40 Mpc, 400 Mpc and at 800 Mpc. The x-axes show
the size of the 90% credible region and the upper limit of the x-axes
corresponds to the size of the whole sky.

(a) $1\text{deg}^{2}$

(b) $5\text{deg}^{2}$

(c) $10\text{deg}^{2}$

(d) $30\text{deg}^{2}$

Figure 9: The histogram distribution of time to merger by 4 detector networks
for BNS at 40 Mpc with required localization uncertainty denoted as caption of
each subplot.

(a) $1\text{deg}^{2}$

(b) $5\text{deg}^{2}$

(c) $10\text{deg}^{2}$

(d) $30\text{deg}^{2}$

Figure 10: The histogram distribution of time to merger by 4 detector networks
for BNS at 400 Mpc with required localization uncertainty denoted as caption
of each subplot.

(a) $5\text{deg}^{2}$

(b) $10\text{deg}^{2}$

(c) $30\text{deg}^{2}$

Figure 11: The histogram distribution of time to merger by 4 detector networks
for BNS at 800 Mpc with required localization uncertainty denoted as caption
of each subplot.

(a) $10\text{deg}^{2}$

(b) $30\text{deg}^{2}$

Figure 12: The histogram distribution of time to merger by 4 detector networks
for BNS at 1600 Mpc with required localization uncertainty denoted as caption
of each subplot.

(a) 3ET

(b) 2ET1CE

(c) 1ET2CE

(d) 3CE

Figure 13: Two-dimensional histograms showing the distributions of the time to
merger for BNS mergers following DTD. The horizontal axis of every subplot is
the size of the $90\%$ credible region, and the vertical axis represents the
time to merger. The color represents the number of sources which achieve the
early warning criteria with the localisation requirement being $\leq
100\mathrm{deg^{2}}$ .

  *[GW]: gravitational wave
  *[EM]: electromagnetic
  *[CBC]: compact binary coalescence
  *[DTD]: delay time distribution
  *[CBCs]: compact binary coalescence
  *[BNS]: binary neutron star
  *[NSBH]: neutron star black hole
  *[GWs]: gravitational wave
  *[BNSs]: binary neutron star
  *[ET]: Einstein Telescope
  *[CE]: Cosmic Explorer
  *[BBH]: binary black hole
  *[SNR]: signal-to-noise-ratio
  *[SNRs]: signal-to-noise-ratio


# Learning Deep Neural Networks by Iterative Linearisation

Adrian Goldwaser, Hong Ge  Adrian Goldwaser
<EMAIL_ADDRESS>Hong Ge
<EMAIL_ADDRESS>
(University of Cambridge)

###### Abstract

The excellent real-world performance of deep neural networks has received
increasing attention. Despite the capacity to overfit significantly, such
large models work better than smaller ones. This phenomenon is often referred
to as the scaling law by practitioners. It is of fundamental interest to study
why the scaling law exists and how it avoids/controls overfitting. One
approach has been looking at infinite width limits of neural networks (e.g.,
Neural Tangent Kernels, Gaussian Processes); however, in practise, these do
not fully explain finite networks as their infinite counterparts do not learn
features. Furthermore, the empirical kernel for finite networks (i.e., the
inner product of feature vectors), changes significantly during training in
contrast to infinite width networks. In this work we derive an iterative
linearised training method. We justify iterative lineralisation as an
interpolation between finite analogs of the infinite width regime, which do
not learn features, and standard gradient descent training which does. We show
some preliminary results where iterative linearised training works well,
noting in particular how much feature learning is required to achieve
comparable performance. We also provide novel insights into the training
behaviour of neural networks.

##  1 Introduction

Deep neural networks perform well on a wide variety of tasks despite their
overparameterisation and capacity to memorise random labels [Zhang et al.,
2017], often with improved generalisation behaviour as the number of
parameters increases [Nakkiran et al., 2020]. This goes contrary to classical
beliefs around learning theory and overfitting, meaning there is likely some
implicit regularisation inducing an inductive bias which encourages the
networks to converge to well-generalising solutions. One approach to
investigate this has been to examine infinite width limits of neural networks
using the _Neural Tangent Kernel_ (NTK) [Jacot et al., 2018, Lee et al.,
2019], interestingly these often do worse than standard neural networks,
though with extra tricks they can perform equivalently well or better under
certain scenarios [Lee et al., 2020]. Similarly, despite their use for
analysis due to having closed-form expressions for many terms, they don’t
predict finite network behaviour very closely in many regards. For example,
due to not learning features, they cannot be used for transfer learning and
the empirical NTK (outer product of Jacobians) changes significantly
throughout training whereas NTK theory states in the infinite limit that this
is constant. This raises important questions about in what ways they are
different, most of which can be summarised as how the use of feature learning
impacts the networks that can be learnt.

To work towards answering this question, we look at an interpolation between
standard training and a finite analog of infinite training where we fix the
empirical NTK by performing weight space linearisation. One interpretation of
this is that we are varying the amount of feature learning allowed. We find
that essentially any amount of feature learning is enough to eventually
converge to a similar performing network, assuming learning rates are small
enough.

###  1.1 Related Work

Li et al. [2019] create an enhanced NTK for CIFAR10 with significantly better
empirical performance than the standard one, however it still performs less
well than the best neural networks. Yang and Hu [2021] use a different limit
to allow feature learning, however neither of these give much insight as to
why the standard parameterisation doesn’t work well.

Lee et al. [2020] run an empirical study comparing finite and infinite
networks under many scenarios and Fort et al. [2020] look at how far SGD
training is from fixed-NTK training, and at what point they tend to converge.
Lewkowycz et al. [2020] investigate at what points in training the kernel
regime applies as a good model of finite network behaviour. Both find better
agreement later in training.

Chizat et al. [2019] consider a different way to make finite networks closer
to their infinite width analogs by scaling in a particular way, finding that
as they get closer to their infinite width analogs, they perform less well
empirically.

##  2 Problem Formulation

Consider a neural network $f_{\theta}(x)$ parameterised by weights $\theta$
and a mean squared error loss function111We use MSE for simplicity and
compatability with NTK results here. While this is needed for some NTK
results, it does not effect the algorithms we propose where any differentiable
loss function can be used — see Appendix A
$\mathcal{L}(\hat{Y})=\frac{1}{2}||\hat{Y}-Y||^{2}$, where we minimise
$\mathcal{L}\left(f_{\theta}(X)\right)$ for data $X$ and labels $Y$. We can
write the change in the function over time under gradient flow with learning
rate $\eta$ as:

$\displaystyle\dot{\theta}_{t}$ $\displaystyle=-\eta\nabla
f_{\theta_{t}}(X)^{\top}(f_{\theta_{t}}(X)-Y)$ (1)
$\displaystyle\dot{f}_{\theta_{t}}(X)$
$\displaystyle=-\eta\hat{\Theta}_{t}(X,X)(f_{\theta_{t}}(X)-Y)$
$\displaystyle\text{where }\>\left[\hat{\Theta}_{t}\right]_{ij}$
$\displaystyle=\left<\frac{\partial
f_{\theta_{t}}(X_{i})}{\partial\theta},\frac{\partial
f_{\theta_{t}}(X_{j})}{\partial\theta}\right>$ (2)

It has been shown [Jacot et al., 2018, Lee et al., 2019, Arora et al., 2019]
that in the infinite width limit the _empirical neural tangent kernel_ ,
$\hat{\Theta}_{t}$, converges to a deterministic NTK, $\Theta$. This is a
matrix dependent only on architecture and data and does not change during
training. From this perspective, training the infinite width model under
gradient flow (or gradient descent with a small step size) is equivalent to
training the weight-space linearisation of the neural network [Lee et al.,
2019]. This raises a number of interesting observations about why this doesn’t
work well in finite networks and what is different in them. This is likely due
to lack of enough random features whereas running gradient descent on the full
network allows features to be learnt, reducing the reliance on having enough
initial random features.

##  3 Iterative Linearisation

NTK theory says that if the width is large enough then training the weight-
space linearisation is equivalent to training the full network [Lee et al.,
2019]. However in practise training the fully linearised network performs very
poorly for practically sized networks Lee et al. [2020]. In this section we
propose _iterative linearisation_ in order to interpolate between training of
the standard network and the linearised network.

Consider standard (full batch) gradient descent on a neural network.

$\theta_{t+1}=\theta_{t}-\eta\phi_{t}(f_{\theta_{t}}(X)-Y)\quad\quad\quad\text{where
}\>\phi_{t}=\nabla_{\theta}f_{\theta_{t}}(X)^{\top}$

Here we can think of this as two separate variables we update each step, the
weights $\theta_{t}$ and the features $\phi_{t}$. However there is no
requirement that we always update both, giving rise to the following
generalised algorithm:

$\displaystyle\theta_{t+1}$
$\displaystyle=\theta_{t}-\eta\phi_{s}^{\text{lin}}\left(f^{\text{lin}}_{s,t}(X)-Y\right)$
(3) $\displaystyle\phi_{s}^{\text{lin}}$
$\displaystyle=\nabla_{\theta}f_{\theta_{s}}(X)^{\top}$ (4)

where $s=K*\lfloor\frac{t}{K}\rfloor$. In addition, we write the linearised
version of a neural network $f_{\theta}$ using its first order Taylor
expansion at the weights $\theta_{s}$ as

${f}^{\text{lin}}_{s,t}(x)=f_{\theta_{s}}(x)+\nabla_{\theta}f_{\theta_{s}}(x)^{\top}(\theta_{t}-\theta_{s})$

Using this framework, when $K=1$ this is simply gradient descent and when
$K=\infty$ it is fully linearised training. Other values of $K$ interpolate
between these two extremes. See Algorithm 1 for more details. Note that we can
also generalise this to not be periodic in terms of when we update $\phi$ so
we call this _fixed period_ iterative linearisation.

Input: learning rate $\eta$, update periodicity $K$, pre-initialised
parameters $\theta_{0}$

for  _t = 1..epochs_ do

$\theta_{t}\xleftarrow{}\theta_{t-1}-\eta\nabla\mathcal{L}(f^{\text{lin}}(X))$;

if  _t mod K = 0_ then

$f^{\text{lin}}(X)\xleftarrow{}f_{\theta_{t}}(X)+\nabla
f_{\theta_{t}}(X)^{\top}(\theta_{t}-\theta_{0})$;

end if

end for

Algorithm 1 Iterative Linearisation (fixed period)

###  3.1 Interpreting Iterative Linearisation

It is insightful to look a bit more closely at what is happening in Algorithm
1. In the case of standard training of a linear model, (e.g.
$f^{\text{lin}}(\cdot)$), the ‘features’ are fixed and the learning decides
how to use those features. This is what is happening in Equation (3) and with
$K=\infty$, we don’t learn any new features beyond the random ones we got
through the initialisation of the network. Interestingly we can say the same
for infinite width networks, this idea that infinite width networks don’t
learn features is not new (see Yang and Hu [2021] for work trying to avoid
this pitfall), but the finite analogy gives us a new perspective on what is
happening in Algorithm 1. We then call Equation (4) _feature learning_ ,
noting that the feature learning cannot be happening in Equation (3). From
this interpretation, the Jacobian $\phi_{t}$ are the features we are using at
time $t$ and $K$ tells us how frequently to update features, putting a limit
on the frequency of feature learning updates.

##  4 Results

Figure 1: Iterative linearisation results on MNIST and CIFAR10 with learning
rates of 1e-4 and 1e-5 Figure 2: Iterative linearisation on CIFAR10 with large
$K$

To examine the effect of increasing $K$, or equivalently reducing the feature
learning frequency, we run a number of experiments on MNIST and CIFAR10 with a
slightly larger variant of LeNet [LeCun et al., 1998] with 50 channels in each
convolutional layer and softmax output. The purpose of these experiments is
not to achieve amazing performance on the datasets (it only gets to $\sim
50\%$ for CIFAR10) but to examine how training changes as the learning rate
$\eta$ and update frequency $K$ changes. For these experiments we note that
unlike the general NTK theory, no process of our derivation relies on the use
of MSE so we instead use cross-entropy loss as is standard for image
classification. We additionally include the softmax in the loss function so it
is not linearised by the algorithm, this both means that we continue to have
an output for which taking the cross-entropy loss is meaningful as well as
avoiding numerical issues caused by having the linearisation happening in the
middle of the logsumexp trick.

Figure 1 shows the results of these experiments. For MNIST, all $K\leq 100$
follow almost the exact same learning trajectory, converging very quickly and
not saturating the amount learnt from each set of features before updating.
For $K=1000$, it can be seen that the training levels off each time before
$\phi$ is updated. This shows that the we can get close to 100% accuracy on
MNIST by only updating the features from their initialisation twice (all
except $K=\infty$ are within a 0.4% range at the end of training). From this
we can conclude that the initialisation of a neural network with this
architecture creates features that are not too far from what is needed to
solve MNIST. Compare this with CIFAR10 (learning rate 1e-5) where it still
only takes a few feature vector updates to reach the performance of $K=1$
however this is a much lower accuracy. It is still unclear from these
experiments how the few updates necessary interacts with different
architectures on the same dataset.

Comparing CIFAR10 with learning rates of $1e-4$ and $1e-5$ also gives
interesting conclusions. For $\eta=1e-4$, training diverges as soon as $K$
goes above 5, and is unstable even for $K=5$, whereas for $\eta=1e-5$ training
stays stable for all $K$ up to $K=20000$ (see Figure 2). Note that this is not
simply because it’s moving less far as this is a 10x reduction in the step
size but a 4000x increase in $K$. We include SGD training for comparison,
noting that plotting per-step rather than per-epoch gives similar performance
to GD training and so we should expect most runs to reach 60% accuracy if run
for twice as long.

##  5 Conclusion

This paper has proposed _iterative linearisation_ , a new training algorithm
that interpolates between gradient descent on the standard and linearised
neural network as a parallel infinite width vs finite networks. We show that,
at least in the case of a LeNet-like architecture with small learning rates,
any amount of features learning is enough to converge to a similar performing
model. This provides an important step towards understanding feature learning
and the distinction between how infinite and finite width networks learn.
Better understanding how networks change with large amounts of parameters has
important connections to empirical phenomena such as explaining deep double
descent [Nakkiran et al., 2020].

###  5.1 Future Work

It is important to do more rigourous empirical investigations to confirm these
results, in particular to scale up to larger models in order to disentangle
the impact of iterative linearisation training from the fact that this
architecture will never do particularly well on CIFAR10. This is also
important to better understand under what architectures/learning
rates/frequencies iterative linearisation training is stable.

Another direction is to better understand the types of solutions that
iterative linearisation finds for various values of $K$. This will shed light
onto how the inductive bias is changing, in particular understanding if all
$K<\infty$ find similar solutions and the infinite width limit is a step
change similar to the test performance in the experiments here, or if this is
a gradual change towards the solutions which don’t learn features.

Finally, we only consider _fixed period_ iterative linearisation here where we
update the feature vector $\phi$ at regular intervals. However Fort et al.
[2020] showed that the empirical NTK changes faster earlier in training so it
makes sense for $K$ to be more adaptive if this was to be used directly for
training.

## References

  * Zhang et al. [2017] Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals.  Understanding deep learning requires rethinking generalization.  In _5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings_. OpenReview.net, 2017.  URL https://openreview.net/forum?id=Sy8gdB9xx.
  * Nakkiran et al. [2020] Preetum Nakkiran, Gal Kaplun, Yamini Bansal, Tristan Yang, Boaz Barak, and Ilya Sutskever.  Deep double descent: Where bigger models and more data hurt.  In _8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020_. OpenReview.net, 2020.  URL https://openreview.net/forum?id=B1g5sA4twr.
  * Jacot et al. [2018] Arthur Jacot, Clément Hongler, and Franck Gabriel.  Neural tangent kernel: Convergence and generalization in neural networks.  In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Cesa-Bianchi, and Roman Garnett, editors, _Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montréal, Canada_ , pages 8580–8589, 2018.  URL https://proceedings.neurips.cc/paper/2018/hash/5a4be1fa34e62bb8a6ec6b91d2462f5a-Abstract.html.
  * Lee et al. [2019] Jaehoon Lee, Lechao Xiao, Samuel S. Schoenholz, Yasaman Bahri, Roman Novak, Jascha Sohl-Dickstein, and Jeffrey Pennington.  Wide neural networks of any depth evolve as linear models under gradient descent.  In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d’Alché-Buc, Emily B. Fox, and Roman Garnett, editors, _Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada_ , pages 8570–8581, 2019.  URL https://proceedings.neurips.cc/paper/2019/hash/0d1a9651497a38d8b1c3871c84528bd4-Abstract.html.
  * Lee et al. [2020] Jaehoon Lee, Samuel S. Schoenholz, Jeffrey Pennington, Ben Adlam, Lechao Xiao, Roman Novak, and Jascha Sohl-Dickstein.  Finite versus infinite neural networks: an empirical study.  In Hugo Larochelle, Marc’Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin, editors, _Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual_ , 2020.  URL https://proceedings.neurips.cc/paper/2020/hash/ad086f59924fffe0773f8d0ca22ea712-Abstract.html.
  * Li et al. [2019] Zhiyuan Li, Ruosong Wang, Dingli Yu, Simon S. Du, Wei Hu, Ruslan Salakhutdinov, and Sanjeev Arora.  Enhanced convolutional neural tangent kernels.  _CoRR_ , abs/1911.00809, 2019.  URL http://arxiv.org/abs/1911.00809.
  * Yang and Hu [2021] Greg Yang and Edward J. Hu.  Tensor programs IV: feature learning in infinite-width neural networks.  In Marina Meila and Tong Zhang, editors, _Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event_ , volume 139 of _Proceedings of Machine Learning Research_ , pages 11727–11737. PMLR, 2021.  URL http://proceedings.mlr.press/v139/yang21c.html.
  * Fort et al. [2020] Stanislav Fort, Gintare Karolina Dziugaite, Mansheej Paul, Sepideh Kharaghani, Daniel M. Roy, and Surya Ganguli.  Deep learning versus kernel learning: an empirical study of loss landscape geometry and the time evolution of the neural tangent kernel.  In Hugo Larochelle, Marc’Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin, editors, _Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual_ , 2020.  URL https://proceedings.neurips.cc/paper/2020/hash/405075699f065e43581f27d67bb68478-Abstract.html.
  * Lewkowycz et al. [2020] Aitor Lewkowycz, Yasaman Bahri, Ethan Dyer, Jascha Sohl-Dickstein, and Guy Gur-Ari.  The large learning rate phase of deep learning: the catapult mechanism.  _CoRR_ , abs/2003.02218, 2020.  URL https://arxiv.org/abs/2003.02218.
  * Chizat et al. [2019] Lénaïc Chizat, Edouard Oyallon, and Francis R. Bach.  On lazy training in differentiable programming.  In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d’Alché-Buc, Emily B. Fox, and Roman Garnett, editors, _Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada_ , pages 2933–2943, 2019.  URL https://proceedings.neurips.cc/paper/2019/hash/ae614c557843b1df326cb29c57225459-Abstract.html.
  * Arora et al. [2019] Sanjeev Arora, Simon S. Du, Wei Hu, Zhiyuan Li, Ruslan Salakhutdinov, and Ruosong Wang.  On exact computation with an infinitely wide neural net.  In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d’Alché-Buc, Emily B. Fox, and Roman Garnett, editors, _Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada_ , pages 8139–8148, 2019.  URL https://proceedings.neurips.cc/paper/2019/hash/dbc4d84bfcfe2284ba11beffb853a8c4-Abstract.html.
  * LeCun et al. [1998] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner.  Gradient-based learning applied to document recognition.  _Proc. IEEE_ , 86(11):2278–2324, 1998.  doi: 10.1109/5.726791.  URL https://doi.org/10.1109/5.726791.

##  Appendix A Iterative linearisation with a general loss function

In Section 3 we show how to get to iterative linearisation from standard
gradient under mean squared error loss. The use of mean squared error is more
instructive due to its similarities with NTK results, however it is not
strictly necessary. For completeness we include here the same idea but for a
general loss function $\mathcal{L}(\cdot)$.

Standard gradient descent on a function $f_{\theta}(\cdot)$ parameterised by
$\theta$, with step size $\eta$ and data $X$ can be written as

$\theta_{t+1}=\theta_{t}-\eta\nabla_{\theta}\mathcal{L}(f_{\theta_{t}}(X))$

We can apply the chain rule, resulting in

$\theta_{t+1}=\theta_{t}-\eta\nabla_{\theta}f_{\theta_{t}}(X)\mathcal{L}^{\prime}(f_{\theta_{t}}(X))$

Where $\mathcal{L}^{\prime}(\cdot)$ is the derivative of $\mathcal{L}(\cdot)$
(in the case of mean squared error, this is the residual:
$\mathcal{L}^{\prime}(\hat{Y})=\hat{Y}-Y$). Now again using
$\phi_{t}=\nabla_{\theta}f_{\theta_{t}}(X)$, we can write this as

$\theta_{t+1}=\theta_{t}-\eta\phi_{t}\mathcal{L}^{\prime}(f_{\theta_{t}}(X))$

With a similar argument to Section 3, we note that we don’t need to update the
features $\phi_{t}$ every step, resulting in the following formulation.

$\displaystyle\theta_{t+1}$
$\displaystyle=\theta_{t}-\eta\phi_{s}^{\text{lin}}\mathcal{L}^{\prime}\left(f^{\text{lin}}_{s,t}(X)-Y\right)$
(5) $\displaystyle\phi_{s}^{\text{lin}}$
$\displaystyle=\nabla_{\theta}f_{\theta_{s}}(X)$ (6)

where $s=K*\lfloor\frac{t}{K}\rfloor$

This now lets us use softmax followed by cross-entropy in the loss
$\mathcal{L}(\cdot)$ while maintaining the same interpretation, as we do for
the MNIST and CIFAR10 results.