#  gwforge: A user-friendly package to generate gravitational-wave mock data

Koustav Chandra Institute for Gravitation & the Cosmos and Department of
Physics, The Pennsylvania State University, University Park PA 16802, USA

###### Abstract

The next-generation gravitational-wave detectors, with their improved
sensitivity and wider frequency bandwidth, will be capable of observing almost
every compact binary coalescence signal from epochs before the first stars
began to form, increasing the number of detectable binaries to hundreds of
thousands annually. This will enable us to observe compact objects through
cosmic time, probe extreme matter phenomena, do precision cosmology, study
gravity in strong field dynamical regimes and potentially allow observation of
fundamental physics beyond the standard model. However, the richer data sets
produced by these detectors will pose new computational, physical and
astrophysical challenges, necessitating the development of novel algorithms
and data analysis strategies. To aid in these efforts, this paper introduces
gwforge, a user-friendly, lightweight Python package, to generate mock data
for next-generation detectors. We demonstrate the package’s capabilities
through data simulation examples and highlight a few potential applications:
performance loss due to foreground noise, bright-siren cosmology and impact of
waveform systematics on binary parameter estimation.

##  1 Introduction

The next-generation (XG) ground-based gravitational-wave (GW) detectors, such
as the Cosmic Explorer (CE) and the Einstein Telescope (ET), promise to
significantly improve both the detection horizon and the frequency bandwidth
of observable compact binary coalescence (CBC) signals (Punturo et al., 2010;
Reitze et al., 2019; Maggiore et al., 2020; Evans et al., 2021; Borhanian &
Sathyaprakash, 2022; Branchesi et al., 2023; Evans et al., 2023; Gupta et al.,
2023). This expanded detection horizon will allow us to observe compact binary
coalescence (CBC) signals from greater distances, thereby increasing the
number of detectable binaries by several orders of magnitude. Additionally,
the increased bandwidth of these detectors could allow for the observation of
a CBC signal over durations spanning minutes to even hours (Regimbau et al.,
2012; Evans et al., 2021; Gupta et al., 2023).

Given this, the XG GW detectors will have far-reaching scientific potential.
However, the “richer” GW detector data will present computational, physical
and astrophysical challenges. Existing GW analysis tools, waveform models and
detector calibration techniques will be insufficient in XG era (Evans et al.,
2021; Pizzati et al., 2022; Hu & Veitch, 2023). Also, the sheer volume of
observed signals will necessitate the development of novel algorithms for
detecting the signals and extracting the science out of them. It is,
therefore, imperative to quantify the limitations of current tools when
employed in XG data analysis.

To support this effort and ensure that XG detectors will achieve their
scientific potential, it is crucial to generate playground datasets to allow
the GW community to develop and tune their algorithms. Such an effort will
drive the development of new data analysis strategies, such as faster search
and parameter estimation (PE) techniques, improved waveform modelling, dealing
with non-stationary noise and methods to identify subtle physical effects,
such as finite size effects in neutron star (NS).

This paper introduces gwforge, a user-friendly Python package to simulate the
mock data for the proposed XG detectors based on our understanding of our
universe. One primary reason for developing gwforge is the anticipated
abundance of signals in XG data, necessitating systematic data generation that
accounts for realistic population models and the complexities of overlapping
signals. This code is built on existing GW packages, such as PyCBC, Bilby,
gwpy and gwpopulation.

The subsequent sections of this paper are structured to highlight the user-
friendly nature of the code. Sec 2 provides an overview of the code, including
installation instructions in Sec. 2.1. Sec. 3 provides the necessary
theoretical background and an example to generate a day’s worth of coloured
Gaussian XG network data with CBC signals. Sec. 4 is devoted to a few studies
one can do using the prepared dataset in the previous section, and we conclude
in Sec. 5.

##  2 Code Overview

###  2.1 Installation

The source code is available on the git repository (Chandra, 2024a), which
includes instructions for contributing to code development. Comprehensive
documentation regarding installation, functionality, and user syntax is
provided on the project’s website (Chandra, 2024b). Example setup scripts
discussed in this work are available in the git repository.

###  2.2 Packages

At the top level, gwforge comprises three main packages: population, ifo and
inject. The population package provides functions essential to generate
individual compact binary properties $\boldsymbol{\theta}$ based on hyper-
parameters $\boldsymbol{\Lambda}$ that describe the population
characteristics. These characteristics are defined by phenomenological
population models or detailed physical simulations, such as population
synthesis or N-body dynamical simulations generated using population synthesis
codes such as Compas (Stevenson et al., 2017; Vigna-Gómez et al., 2018).
gwforge integrates all phenomenological population models from gwpopulation
and includes additional distribution functions such as a DoubleGaussian and
LogNormal. Users can also utilise outputs from population synthesis
simulations to generate their own populations and define cosmological
parameters, such as the Hubble constant $H_{0}$, the CMB temperature
$T_{\mathrm{CMB}}$, etc., to specify their Universe.

The ifo package provides core functionality for generating either zero-noise
or coloured Gaussian noise given a noise power spectral density (PSD). It
implements current and future ground-based GW detectors, detailing their
locations, orientations, and various noise PSDs of existing and future
instruments. Users can also implement custom GW detectors akin to Bilby
(Ashton et al., 2019). Additionally, this package allows users to introduce
data gaps, enabling the simulation of detector downtimes. This is important,
as it creates a more realistic scenario, reflecting the potential for missing
signals or having signals that are only partially present. By accounting for
detector downtimes and data gaps, one can better understand the impact of
these factors on gravitational wave detection and analysis, thereby improving
the robustness of signal detection and parameter estimation methodologies.

Finally, the inject module provides essential functionalities for injecting or
adding simulated GW signals into the data. It supports all standard waveform
approximants available via the lalsimulation package (LIGO Scientific
Collaboration et al., 2018) and/or those made available via PyCBC’s waveform
plugin.

Future updates will expand this module to include the capability to add non-
Gaussian transients, Supernova (SNe) signals, and other GW transients, thereby
making the network output more closely “mock” the real data.

###  2.3 Executables

To facilitate usability, we developed three key executables using the
aforementioned packages: gwforge_population, gwforge_noise, and
gwforge_inject. Each executable operates based on a configuration file
tailored to its specific function. Additionally, the gwforge_workflow
executable integrates all three tools and utilises the high-throughput
computing framework HTCondor. This integration significantly accelerates the
data generation process and simplifies user interaction by managing all
ancillary tasks automatically. The following section provides an explicit
example of using these executables.

##  3 Simulating a day’s worth of mock data

###  3.1 Background

The observed GW strain $s(t\mid\boldsymbol{\theta},I)$ at a detector $I$ can
be expressed as:

$s(t\mid\boldsymbol{\theta},I)=\frac{1}{D_{L}}\mathbb{R}\left[F^{I}h\right]$
(1)

where $\boldsymbol{\theta}$ is a vector representing the source’s properties,
$D_{L}$ is the luminosity distance to the source, $\mathbb{R}$ denotes the
real part, and

$F=F_{+}^{I}+iF_{\times}^{I}$

is the antenna response function that depends on the sky location
$(\alpha,\delta)$ and the polarization angle $(\psi)$. The term $h$ represents
the complex binary waveform on a sphere. This waveform can be decomposed as a
sum of spin-weighted spherical harmonic modes $h_{\ell m}$, so that the
waveform along any direction $(\iota,\varphi)$ in the binary’s source frame is
given by:

$h=\sum_{\ell\geq
2}\sum_{m=-\ell}^{\ell}{}^{-2}{Y}_{\ell,m}(\iota,\varphi)h_{\ell
m}(t\mid\boldsymbol{\Xi})$ (2)

Here, ${}^{-2}{Y}_{\ell,m}(\iota,\varphi)$ are the spin $-2$ weighted
spherical harmonics, and $(\iota,\varphi)$ represent the binary’s inclination
and azimuth, respectively. The vector $\boldsymbol{\Xi}$ denotes the binary’s
intrinsic parameters whose dimensionality depends on the binary type. For
example, in quasi-spherical binary black holes, $\boldsymbol{\Xi}$ includes
the two component masses $m_{i}$ and the six spin degrees of freedom
$\boldsymbol{\chi}_{i}$. In contrast, for quasi-circular or non-precessing
binary neutron stars, $\boldsymbol{\Xi}$ represents the component masses
$m_{i}$, the tidal parameters $\Lambda_{i}$ and the aligned spin components
$\boldsymbol{\chi}_{i}\cdot\hat{L}$, with $\boldsymbol{L}$ being the binary’s
orbital angular momentum.

Since the source’s total merger rate and redshift distribution determine the
number of signals in a given simulated dataset, gwforge begins by first
estimating the average time interval $\langle\Delta t\rangle$ between two
adjacent signals of the same type given the source population and the
cosmological parameters. It assumes that all CBC systems are formed through
isolated formation channels involving a common envelope phase. This allows it
to derive the redshift distribution of CBCs in the source frame as follows
(Zhu et al., 2021; Wu & Nitz, 2023):

$\displaystyle\mathcal{R}(z)$
$\displaystyle\propto\int_{\tau_{\text{min}}}^{\infty}\mathcal{R}_{\ast}[z_{f}(z,\tau)]P(\tau)d\tau$
(3)
$\displaystyle\propto\int_{z}^{\infty}\mathcal{R}_{\ast}(z_{f})P[\tau(z,z_{f})]\frac{dt(z_{f})}{dz_{f}}\,dz_{f}~{}.$

Here, $\mathcal{R}_{\ast}$ is the cosmological star formation rate density,
$P(\tau)$ is the probability density function of the delay time, $z_{f}$ is
the redshift at which the stellar binary forms, $z$ is the redshift at which
the compact binary merges, $\tau=t(z)-t(z_{f})$ is the delay time and $t(z)$
is the lookback time at redshift $z$. The proportionality constant is defined
such that $\mathcal{R}(z=0)=\mathcal{R}_{0}$ where $\mathcal{R}_{0}$ is the
local merger rate density of the CBC source.

The lookback time,

$t(z)=\frac{1}{H_{0}}\int_{z}^{\infty}\frac{dz}{(1+z)\sqrt{\Omega_{\Lambda}+\Omega_{m}(1+z)^{3}}},$
(4)

depends on the Hubble constant, $H_{0}$, the dark matter, $\Omega_{\Lambda}$
and matter density, $\Omega_{m}$. All these cosmological parameters, along
with the star formation rate, are adjustable by the user, allowing them to
simulate their own Universe. However, gwforge currently only implements
$P(\tau)\propto 1/\tau$.

Given this, the merger rate per redshift bin in the detector frame is then
given by (Regimbau et al., 2012):

$\frac{dR}{dz}=\frac{\mathcal{R}}{1+z}\frac{dV(z)}{dz},$ (5)

where the comoving volume element is given as:

$\frac{dV(z)}{dz}=\frac{c}{H_{0}}\frac{4\pi
D_{L}^{2}}{(1+z)^{2}\sqrt{\Omega_{\Lambda}+\Omega_{m}(1+z)^{3}}}~{}.$ (6)

Here, $c$ is the light speed, and $D_{L}$ is the luminosity distance to the
source defined as:

$D_{L}=\frac{c(1+z)}{H_{0}}\int_{0}^{z}\frac{dz^{\prime}}{\sqrt{\Omega_{\Lambda}+\Omega_{m}(1+z^{\prime})^{3}}}$
(7)

The $1/(1+z)$ factor in Eq. (5) accounts for time dilation.

gwforge obtains $\langle\Delta t\rangle$ as follows:

$\langle\Delta
t\rangle=1/\Big{(}\int^{z_{\mathrm{max}}}_{0}\frac{dR}{dz}dz\Big{)}~{}.$ (8)

where $z_{\mathrm{max}}$ is the maximum redshift used in the simulation. Based
on $\langle\Delta t\rangle$ and the total analysis time, $T_{a}$ of the
network, gwforge determines the expected number of signals,
$N=T_{a}/\langle\Delta t\rangle$, and correspondingly draws the binary
parameters from the provided population models.

Below, we discuss an example to build a mock catalogue based on our current
understanding of BBH binary population distribution.

###  3.2 Simulating the mock GW catalogue

To create a mock BBH catalogue using gwforge, one must first define a
configuration file like the one provided below:

⬇

1[Redshift]

2redshift-model = MadauDickinson

3redshift-parameters = {‘gamma’: 2.7, ‘kappa’: 5.6, ‘z_peak’: 1.9}

4local-merger-rate-density = 22

5maximum-redshift = 30

6; custom cosmology

7cosmology = custom

8H0 = 67.7

9Om0 = 0.31

10Ode0 = 0.69

11Tcmb0 = 2.735

12; analysis start time

13gps-start-time = 1893024018

14

15[Mass]

16mass-model = PowerLaw+Peak

17mass-parameters = {‘alpha’:3.37, ‘beta’: 0.76, ‘delta_m’:5.23, ‘mmin’:4.89,
‘mmax’:88.81, ‘lam’:0.04, ‘mpp’: 33.60, ‘sigpp’:4.59}

18

19[Spin]

20spin-model = default

21spin-parameters = {‘mu_chi’:0.26, ‘sigma_squared_chi’:0.02, ‘sigma_t’:0.87,
‘xi_spin’:0.76}

22

23[Extrinsic]’

Within the [Redshift] section, users must specify the redshift evolution
model, local merger rate density in $\mathrm{/Gpc^{3}/yr}$, analysis start
time (in GPS time), and the maximum redshift. Currently, gwforge supports two
different redshift evolution models: the power-law redshift evolution model
and the other that convolves the Madau-Dickinson star-formation rate with an
inverse time-delay model to determine the CBC’s redshift evolution model
(Fishbach et al., 2018). Users can define custom cosmological and redshift
parameters; otherwise, default values from Aghanim et al. (2020) and Madau-
Dickinson parameters are used (Madau & Dickinson, 2014).

Similarly, the [Mass] and [Spin] sections provide options for various
phenomenological mass and spin distribution models, as detailed in Chandra
(2024b). Besides predefined models, gwforge can utilize outputs from
population-synthesis codes, creating interpolated distribution functions from
variable and associated probability arrays, enabling sampling from simulated
distributions.

Similarly, the [Mass] and [Spin] sections offer a selection of
phenomenological mass and spin distribution models, as documented in Chandra
(2024b). The [Extrinsic] section deals with the other extrinsic parameters.
Unless otherwise specified, gwforge employs the following default distribution
choices for these parameters (See Table 2) for definitions of the symbols):

variable | unit | distribution | minimum | maximum
---|---|---|---|---
$\alpha$ | rad. | uniform | 0 | $2\pi$
$\delta$ | rad. | $\cos$ | $-\pi/2$ | $\pi/2$
$\iota$ | rad. | sin | 0 | $\pi$
$\psi$ | rad. | uniform | 0 | $\pi$
$\phi$ | rad. | uniform | 0 | $2\pi$
Table 1: The default distribution functions for the sky location parameters,
$(\alpha,\delta)$, inclination, $\iota$, polarisation angle $\psi$ and azimuth
$\phi$.

Finally, if the binary contains a NS or a pair of NSs, gwforge can set the
tidal parameters $\Lambda_{i}$ based on an Equation of State (EoS). This
necessitates a pre-shipped ASCII table, examples of which can be found at
Chandra (2024a). Additional configuration file examples are also provided at
the same location.

Once the configuration file is defined, the next step is to execute:

⬇

1 gwforge_population \--config-file bbh.ini \--output-file bbh.h5

to create the mock BBH catalogue and store as in an hdf5 file.

###  3.3 Mock noise generation

After generating the mock GW catalogue, the next step is to define the
detector network and simulate their noise. This can be achieved by creating a
configuration file, for example, xg.ini, as shown below:

⬇

1[IFOS]

2detectors = [‘CE20’, ‘CE40’, ‘ET’]

3sampling-frequency = 8192

4noise = gaussian’

and executing:

⬇

1 gwforge_noise \--config-file xg.ini \--output-directory output/data \--gps-
start-time 1895616018 \--gps-end-time 1895656978

This initialises a three-detector network consisting of two L-shaped CE
observatories at two fiducial locations: CE40 off the coast of Washington
State and CE20 off the coast of Texas, along with an underground triangular ET
located in Sardinia, Italy (Evans et al., 2023; Gupta et al., 2023). The
simulated noise thus produced will be Gaussian coloured by the detector noise
PSD, with data sampled at 8192 Hz over approximately one day. The data will be
stored as HDF5 files in the output/data directory, named in the format IFO-
GPS-TIME.h5, with channel names IFO:INJ. To simulate zero-noise realisations,
the user has to set the flag noise = zero.

gwforge also includes the two LIGO detectors in the US (Aasi et al., 2015),
Virgo in PISA (Acernese et al., 2015), LIGO-Aundha in India (Iyer et al.,
2011) and the KAGRA detector in Japan (Aso et al., 2013). It is also possible
to define new detectors characterised by their geometry, location and
frequency response. This feature is particularly useful for developing the
science case for new proposals and optimising the design and placement of new
detectors, such as placing the ET observatory in the Netherlands or building a
pair of ET observatories at two different locations (Branchesi et al., 2023).

###  3.4 Mock data generation

Figure 1: A representative section of data generated using gwforge. The grey
shows the Gaussian noise while the blue represents the sum of all GW signals,
namely from BNS, BHNS and BBH.

The final step involves injecting the simulated GW signals into the network
data. Similar to the previous steps, this requires a configuration file, for
example, inject-bbh.ini, as shown below:

⬇

1[IFOS]

2detectors = [‘CE20’, ‘CE40’, ‘ET’]

3channel-dict = {‘CE20’:‘CE20:INJ’, ‘CE40’:‘CE40:INJ’, ‘ET’:‘ET:INJ’}

4sampling-frequency = 8192

5minimum-frequency = 6

6

7[Injections]

8injection-file = bbh.h5

9injection-type = bbh

10waveform-approximant = IMRPhenomXO4a’

As with the noise generation step, the [IFOS] section requires specification
of the detector network and sampling frequency. Additionally, this section
must include the minimum frequency cut-off for signal generation and the
channel names for the frame files. If not specified, gwforge assumes that the
reference frequency for measuring the spin angles is the same as the minimum
frequency used for signal generation.

The [Injections] section should contain the following keys: injection-file,
which points to the file containing the mock BBH catalogue (e.g., bbh.h5);
injection-type, which specifies the type of injection being made; and
waveform-approximant, which indicates the waveform model to use for signal
generation, with IMRPhenomXO4a being an example (Thompson et al., 2024).
Currently, gwforge supports all waveform approximants available via the
lalsimulation package (LIGO Scientific Collaboration et al., 2018). To use
waveforms beyond those available in lalsimulation, a new plugin package must
be created to advertise the new waveform model to PyCBC for waveform
generation (Usman et al., 2016).

To inject the signal waveforms into the generated data over the specified GPS
time interval, the user needs to execute the following command:

⬇

1gwforge_inject \--config-file forged-data/inject-bbh.ini \--data-directory
/output/data \--gps-start-time 1895616018 \--gps-end-time 1895656978

This command injects the signal waveforms into the generated noise over the
specified GPS time interval, creating the mock data for further study.

###  3.5 Using gwforge_workflow

Since the data generation process can be computationally expensive and time-
consuming, users are encouraged to take advantage of gwforge_workflow, which
leverages the high-throughput computing framework HTCondor. This significantly
accelerates the data generation process by abstracting all other housekeeping
logic away from the user. Once the mock catalogues are generated and the
different configuration files are created, the following commands can be
executed:

⬇

1# Activate Conda environment

2conda activate gwforge-test || { echo "Failed to activate Conda environment."
>&2; exit 1; }

3

4# Set output directory

5output_directory=forged-data

6

7# Workflow submission script

8gwforge_workflow \

9 \--gps-start-time 1895616018 \

10 \--gps-end-time 1895702418 \

11 \--output-directory "${output_directory}/output" \

12 \--noise-configuration-file "${output_directory}/xg.ini" \

13 \--bbh-configuration-file "${output_directory}/bbh-inject.ini" \

14 \--bns-configuration-file "${output_directory}/bns.ini" \

15 \--nsbh-configuration-file "${output_directory}/nsbh.ini" \

16 \--workflow-name work \

17 \--submit-now

This will create an HTCondor meta-scheduler to manage dependencies between
jobs and systematically parallelize the tasks, accelerating the mock data
generation process. Fig 1 shows 1024s of simulated data when assuming Gaussian
noise, including signals from Population-A (See Sec. 4.1 for details). As can
be seen, the data will never be “signal-free”.

##  4 Example usage of the mock data

###  4.1 Foreground Noise

Figure 2: Comparison of Welch ASD estimates for different realizations of 512
seconds of CE40 mock data. The black dotted line represents the ideal CE40
design sensitivity curve. The red curve indicates the estimate when the noise
includes signals from population A, while the blue curve represents the
estimate when the noise includes signals from population B. Both the red and
blue curves include a 90% confidence band. The comparison demonstrates that
depending on the number of signals, the ASD estimate can deviate from the
expected noise ASD at 15Hz. Figure 3: Comparison of horizon distance reach
when using the median Welch ASD estimates. The black dotted line represents
the ideal CE40 design sensitivity curve. The red curve indicates the estimate
when the noise includes signals from population A, while the blue curve
represents the estimate when the noise includes signals from population B. The
comparison shows that depending on the number of signals, the horizon distance
of a XG detector can be affected, especially for BBHs with a high detector
frame (redshifted) total mass.

The increased bandwidth of XG detectors will allow low-mass CBC signals to
remain within the detector’s sensitive bandwidth for hours. Consequently,
multiple signals will simultaneously appear in the data stream, the number of
which will depend on the sources’ astrophysical merger rate density.
Characterised by their rapid frequency evolution, these signals will
predominantly overlap at low frequencies and “separate” as they approach
merger. Therefore, current modelled detection techniques can identify and
measure their merger time to $\mathcal{O}(10ms)$ accuracy (Relton et al.,
2022). As a result, unlike in LISA, where overlapping signals lead to source
confusion, these signals will be resolvable and only contribute to foreground
noise.

However, as the number of overlapping signals per time-frequency bin is small,
their combined contribution won’t obey the central limit theorem. Therefore,
the data cannot be assumed to follow a wide-sense stationary Gaussian process
— an assumption critical to the currently used noise PSD estimation technique
such as the Welch averaging (Welch, 1967; Allen et al., 2012) and the
BayesLine method (Littenberg & Cornish, 2015; Cornish & Littenberg, 2015;
Cornish et al., 2021). As a result, using these techniques without
modification can result in biased PSD estimates. Such a deviation can
introduce systematic biases in parameter inference and reduce CBC
detectability.

To demonstrate this, we simulate multiple realisations of 512s mock CE40 data
using the gwforge package and estimate the noise PSD using the Welch method.
Mathematically, this method assumes the data to be wide-sense stationary and
divides the discrete time domain data $d[j]$ into $K$ overlapping segments
$d_{k}[j]$ of length $L$, each windowed by $w[j]$ which for our purpose, is
the Hann truncation function. The discrete Fourier transform
$\tilde{d}_{k}[i]$ of each windowed segment is computed using an FFT routine,
and the periodogram $P_{k}[i]$ is obtained as

$P_{k}[i]=\frac{1}{U}|\tilde{d}_{k}[i]|^{2},$

where $U=\sum_{n=0}^{L-1}w[j]^{2}$. We, following Allen et al. (2012), use
mean-median averaging of the periodograms over all segments to obtain the
final PSD estimate. This approach significantly reduces the variance of the
spectral estimate due to glitches, thereby providing a more accurate
representation of the data’s power distribution across frequencies.

We assume two different source populations — population-A quantified by BNS
and BHNS merger rate density of
$\mathcal{R}_{\mathrm{BNS}}^{\mathrm{A}}=320/\mathrm{Gpc}^{3}/\mathrm{yr},~{}\mathcal{R}_{\mathrm{BHNS}}^{\mathrm{A}}=90/\mathrm{Gpc}^{3}/\mathrm{yr}$
respectively and population-B with merger-rate density of
$\mathcal{R}_{\mathrm{BNS}}^{\mathrm{B}}=1700/\mathrm{Gpc}^{3}/\mathrm{yr},~{}\mathcal{R}_{\mathrm{BHNS}}^{\mathrm{B}}=200/\mathrm{Gpc}^{3}/\mathrm{yr}$
(LVK, 2023, 2024). While overlapping BBH signals will lead to larger
foreground noise, their lower local merger rate and shorter duration in the
detector band suggest that they will rarely contribute to the foreground
noise. We therefore use
$\mathcal{R}_{\mathrm{BBH}}=30/\mathrm{Gpc}^{3}/\mathrm{yr}$ in both the
cases.

Fig. 2 shows the simulated and the estimated CE40 ASD, with the latter
obtained using the Welch method. It is estimated by dividing multiple-
realisation of 512s of data into 16s subsegments and 50% overlap, akin to that
used in a PyCBC search. Consistent with previous works (Wu & Nitz, 2023; Evans
et al., 2021), we find that the deviation is mainly concentrated between
6-30Hz, reaching $\gtrsim 10\%$ at 15Hz for the upper merger rate. This can
reduce the detectability of BBHs with a high detector frame (redshifted) total
mass, as shown in Fig. 3. In particular, depending on the binary’s detector
frame mass, the horizon distance 111The horizon distance, $D_{H}$, for our
case, is the farthest luminosity distance to which we can confidently detect
an optimally oriented, located, non-spinning symmetric mass BBH assuming an
optimal signal-to-noise ratio (SNR) threshold of 8. loss to systems with
$M_{T}(1+z)\gtrsim 1000M_{\odot}$ can be $\gtrsim 15\%$. Similar to the Welch
estimate, the median BayesLine output, which is widely used in GW PE, will
also show a deviation, unless a joint spectral and signal estimate is
performed with precision (see Appendix A for a discussion).

###  4.2 Bright-siren cosmology

Table 2: True value for the parameters used for the simulated IMBH binary. The values are inspired by the maximum likelihood values of GW190521 as inferred by Islam et al. (2023). Parameter | Symbol | Value
---|---|---
Mass ratio | $q$ | 0.95
Detector-frame total mass | $M_{T}(1+z)$ | $273.83M_{\odot}$
Primary spin magnitude | $\chi_{1}$ | 0.83
Secondary spin magnitude | $\chi_{2}$ | 0.96
Primary tilt a | $\theta_{1}$ | 1.59
Secondary tilt | $\theta_{2}$ | 1.96
Azimuthal inter-spin angle | $\phi_{12}$ | 0.23
Azimuthal precession cone angle | $\phi_{JL}$ | 0.36
Effective aligned spin | $\chi_{\text{eff }}$ | -0.19
Effective precessing spin | $\chi_{p}$ | 0.83
Coalescence phase | $\phi$ | 1.06
Polarization angle | $\psi$ | 1.28
Coalescence GPS time | $t_{c}$ | 1242442967.43s
Right ascension | $\alpha$ | 4.39
Declination | $\delta$ | 0.85
Redshift | $z$ | 0.76
Inclination angle | $\theta_{JN}$ | 2.09
Hubble constant | $H_{0}$ | $67.7\mathrm{km}/\mathrm{s}/\mathrm{Mpc}$
Matter density of Universe | $\Omega_{m}$ | $0.31$

GW signals from CBC sources can be used as probes for cosmic expansion
(Schutz, 1986). The luminosity distance, $D_{L}$ to a CBC can be directly
estimated using the GW waveform $s(t\mid\boldsymbol{\theta})\propto 1/D_{L}$
without needing external distance calibrators (Cutler & Flanagan, 1994). If
the CBC’s cosmological redshift, $z$, can be estimated by other means, then
one can infer the local expansion rate, $H_{0}$ and other cosmological
parameters (Holz & Hughes, 2005; Nissanke et al., 2010), such as the
fractional energy density of dark energy, $\Omega_{\Lambda}$ and dark matter,
$\Omega_{m}$. For example, observing the electromagnetic (EM) counterparts
from GW170817 allowed us to identify the source’s host galaxy, NGC 4993,
enabling the first direct measurement of $H_{0}$ using GWs (Abbott et al.,
2017).

Another promising EM-bright CBC source is BBHs mergers occurring in gas-rich
environments such as active galactic nuclei (AGN) disks (McKernan et al.,
2014; Bartos et al., 2017; McKernan et al., 2019; Graham et al., 2020). BBHs
in such environments can efficiently merge due to gas torques and dynamical
encounters, potentially producing detectable EM radiation during or after the
merger. Such detection is feasible even against bright AGN disks, provided the
disks are thin and relatively low in luminosity or if the merger remnant is
ejected from the optically thick midplane and undergoes super-Eddington
accretion.

XG detectors promise to play a pivotal role in inferring cosmological
parameters (Chen et al., 2024; Corsi et al., 2024). To demonstrate how the
mock data can be used to perform bright-siren cosmology, we analyse 8s of XG
detector data segment containing a GW190521-like binary that coincides with an
EM counterpart with
$\\{\alpha_{\mathrm{em}},\delta_{\mathrm{em}},\mathrm{z}_{\mathrm{em}}\\}$
same as the simulated signal. The data includes Gaussian noise and
Population-A binaries, as introduced in the previous subsection. The binary
and cosmological parameters used for the analysis are tabulated in Table 2. We
use the Bayesian PE library Bilby (Ashton et al., 2019), waveform model
NRSur7dq4 (Varma et al., 2019) and off-the-shelf dynamic nested sampler
dynesty (Speagle, 2020). We use the same waveform model for injection and
recovery to prevent waveform systematics. Consistent with the mock data
generation set-up, we use a minimum-frequency cutoff of $f_{\mathrm{min}}=6$Hz
and perform a flare-unrestricted PE with standard priors (Abbott et al., 2020)
and PSD calculated using the Welch method.

Figure 4: Posterior probability density for the sky-location of the
GW190521-like signal in XG network data. The blue square shows the sky
location of the simulated binary, and the black dot shows the maximum
likelihood sky location

Fig. 4 shows the posterior probability distributions for the sky-location of
simulated binary. As can be seen, the sky location of the flare and the GW
signal are consistent. In fact, the odds of a common source hypothesis $C$,
against a random coincidence, $\mathcal{O}_{C/R}>2\times 10^{7}$, if we assume
the prior odds, $\pi_{C/R}$, to be unity.

Figure 5: The posterior pdf of $H_{0}$ for the event under the assumption of
flat $\Lambda$CDM cosmology and physical matter constraints from Aghanim et
al. (2020). The red curve assumes no selection effect, while the green and
blue curve assumes that the underlying BBH population follows a Uniform in
Comoving volume and star formation rate as modelled by Madau & Dickinson
(2014) respectively.

Having confirmed such an association, we use the obtained posterior samples to
infer $H_{0}$ (Abbott et al., 2017; Chen et al., 2022) given the GW and EM
datasets, $d_{\mathrm{gw}}$ and $d_{\mathrm{em}}$:

$\displaystyle p(H_{0}$ $\displaystyle\mid
d_{\mathrm{em}},d_{\mathrm{gw}})\propto\,\pi(H_{0})$ (9)
$\displaystyle\times\mathcal{L}\left(d_{\mathrm{gw}}\mid\alpha_{\mathrm{em}},\delta_{\mathrm{em}},D_{L}(z_{\mathrm{em}}\mid
H_{0})\right)$
$\displaystyle\times\frac{\pi_{\mathrm{pop}}(z_{\mathrm{em}}\mid
H_{0})}{\beta(H_{0})}~{}.$

Here $\pi(H_{0})$ is the prior on $H_{0}$, $\mathcal{L}$ is the marginalised
GW likelihood function evaluated at the distance and sky location of the
electromagnetic source, $\pi_{\mathrm{pop}}(z_{\mathrm{em}}\mid H_{0})$ is the
assumed redshift distribution of the binary population and $\beta(H_{0})$ is
the fraction of binary population that will be simultaneously detected in GW
and EM window.

Figure 6: The joint posterior pdf of $H_{0}$ and $\Omega_{m}$ for the event
using uniform priors for all the parameters in a flat $\Lambda$CDM $(w=-1)$
cosmology. The contour encloses 90% CI. The blue square shows the cosmological
parameters used for the simulations, while the black dot shows the maximum
likelihood sky location.

Fig 5 shows our results, assuming a flat $\Lambda$CDM cosmology and matter
constraints from Aghanim et al. (2020). We assume
$\pi(H_{0})=\mathcal{U}(35,140)\mathrm{km/s/Mpc}$ and consider two redshift
distribution models: one that assumes that BBH merger rate is constant up to
redshift $z=10$ and the other that follows the Madau-Dickinson star-formation
rate used to simulate the binary population. For both cases, we first estimate
$\beta(H_{0})$ following Farr & Gair (2018) and then infer $H_{0}$ to be
$67.7^{+0.36}_{-0.36}\mathrm{km/s/Mpc}$ at 68% CI, thereby being consistent
with the value used in simulating the binary population. It shows that if the
uncertainty in luminosity distance and the counterpart redshift measurement is
small, XG EM-bright sources can precisely measure $H_{0}$, irrespective of the
underlying population distribution assumption.

In Fig 6, we present the measurements of $\Omega_{m}$ and $H_{0}$, when using
flat priors in the range $H_{0}=[35,140]\mathrm{km/s/Mpc}$, $\Omega_{m}=[0,1]$
and assuming $\Lambda$CDM cosmology. For this, we obtain the joint posterior
on $(H_{0},\Omega_{m})$ by modifying Eq. (9) to the following:

$\displaystyle p(H_{0},\Omega_{m}$ $\displaystyle\mid
d_{\mathrm{em}},d_{\mathrm{gw}})\propto\,\pi(H_{0},\Omega_{m})$ (10)
$\displaystyle\times\mathcal{L}\left(d_{\mathrm{gw}}\mid\alpha_{\mathrm{em}},\delta_{\mathrm{em}},D_{L}(z_{\mathrm{em}}\mid
H_{0},\Omega_{m})\right)$
$\displaystyle\times\frac{\pi_{\mathrm{pop}}(z_{\mathrm{em}}\mid
H_{0},\Omega_{m})}{\beta(H_{0},\Omega_{m})}~{}.$

Consistent with Fig. 5, the maximum likelihood point is close to the true
value. However, as expected, the posterior for $H_{0}$ is broader.

###  4.3 PE biases due to waveform systematics

Figure 7: Comparison corner plot for the posteriors obtained for the
redshifted total mass, mass ratio, effective aligned and precessing parameters
and the redshift for the GW190521-like signal when using three different
waveform approximants. The cyan lines correspond to the true values.

GWs from CBCs carry characteristic information about the astrophysical
properties of compact objects, such as their masses and spins. A significant
challenge in inferring these parameters is the systematic error introduced by
the waveform approximants. If two waveform models, $\mathcal{W}_{1}$ and
$\mathcal{W}_{2}$ do not agree for a particular choice of
$\boldsymbol{\theta}$, i.e. $\delta
h=h_{1}(t\mid\boldsymbol{\theta},\mathcal{W}_{1})-h_{2}(t\mid\boldsymbol{\theta},\mathcal{W}_{2})\not\rightarrow
0$, then it can significantly skew PE outcomes. This is especially problematic
if the difference exceeds the detector noise level.

More rigorously, if the two waveform models are not faithful up to a
particular level determined by the signal’s SNR, then the disagreement between
the waveform models will lead to differences in PE outputs. This issue is
particularly critical in XG era, where we anticipate observing over $10^{5}$
compact binary signals per year, few of which can have a SNR greater than 100.
Since the statistical uncertainty of parameter estimates decreases with an
increase in SNR, the measurability of signal parameters can be significantly
affected if the current waveform models are used, particularly when analysing
high SNR events (Pürrer & Haster, 2020; Dhani et al., 2024; Kapil et al.,
2024). Consequently, it is crucial to understand how waveform systematics can
lead to incongruous results.

While a detailed study is deferred to future work, we provide an illustrative
example of using the simulated mock data to identify biases in PE due to
waveform systematics. We note that these studies must be performed in zero-
noise so that no systematics are introduced due to noise. Here, we give an
example where the noise is Gaussian and includes population-A signals.

To this end, we re-analyse the mock data segment containing the GW190521-like
NRSur7dq4 signal with parameters tabulated in Table 2 with three different
state-of-the-art waveform models, namely NRSur7dq4, IMRPhenomXO4a and
SEOBNRv5PHM (Ramos-Buades et al., 2023). For each of these analyses, we use
Bilby, employ standard priors and the standard likelihood function for GW
transients (Finn, 1992) and evaluate it from a fixed minimum frequency of
$f_{\mathrm{min}}=6$Hz identical to the reference frequency choice.

Fig. 7 shows a corner plot comparing a few binary parameters. As can be seen,
depending on the waveform approximant, the recovered parameters can be
significantly different from the injected value. This is because these
waveform models differ in their construction; therefore, measurements relying
on them can have systematic uncertainties. This has not been an issue for
almost all second-generation GW candidates, as differences caused by different
approximant choices have been lower than those induced by noise fluctuations.
But for loud XG events, this can introduce significant biases as the
statistical uncertainty of parameter estimates decreases with an increase in
SNR (Kapil et al., 2024; Dhani et al., 2024). Standard approaches to account
for modelling errors, such as combining posterior for analysis performed using
different approximants to yield a joint posterior that is effectively
marginalised over the uncertainty inherent in the waveform models, won’t alone
be able to account for the errors and biases that occur during the recovery of
source parameters due to waveform modelling choices.

###  4.4 Other possible challenges

The data generated using gwforge can be used to address several other
challenges, some of which are summarised below:

  1. 1.

Testing faster and more robust search and PE algorithms: The XG detectors are
expected to observe hundreds of thousands of binary signals annually.
Developing optimal signal identification methods and rapid parameter
estimation algorithms tailored to compact binary analysis will be essential
for extracting science from the signal-rich dataset. In this context, the
traditional requirement of achieving a $\gtrsim 5\sigma$ confidence level may
no longer be necessary. Instead, search design requirements can be adapted to
enable faster detection and PE algorithms can be developed and/or tuned for
faster and more precise estimation of signal properties.

Moreover, non-stationary noise due to glitches and foreground noise will
necessitate a joint analysis of the signal of interest and the remaining data.
In fact, current search techniques can be replaced by coherent Bayesian search
algorithms that will allow for the identification of potential candidates,
give rapid, fully coherent parameter estimates of the signal source, and
assign astrophysical probabilities of the candidate events (Isi et al., 2018).
The simulated datasets will allow the GW community to develop, test, and
refine algorithms to meet these scientific objectives.

  2. 2.

Measurement of NS EoS: GWs from binaries consisting of a pair of NS, or a NS
and a Black hole (BH) can be used to measure the tidal deformability of NSs
and constrain the nature of matter at supernuclear densities (Hinderer et al.,
2010; Damour et al., 2012; Del Pozzo et al., 2013). Given that these effects
appear in the fifth post-Newtonian order, this will be best measured for
signals with SNR of order thousands, akin to those expected to be observed in
XG era. The gwforge dataset will contain such signals depending on the initial
conditions and can be used to verify how precisely and accurately one can
measure the NS EoS. Further, it can be used to quantify any observed bias due
to the foreground noise. Also, depending on the waveform model used, the post-
inspiral portion of the signals may also be present in the simulated dataset
which can be used to study how well we can constrain the presence of phase
transitions at several times the nuclear densities (Radice et al., 2017; Most
et al., 2019, 2020; Prakash et al., 2024).

  3. 3.

Testing rapid pre-merger sky localisation algorithms: Low latency detection
and pre-merger localisation of potential BNS signals, especially those within
the reach of electromagnetic observatories, will be an important scientific
goal for XG CBC algorithms as they can allow for a multiband understanding of
BNSs. The gwforge dataset can be used to mimic online analysis. It will allow
the community to combine multi-messenger observation of future mergers to more
precisely determine the NS EoS and learn about the engine powering gamma-ray
bursts, kilonovae and the formation of heavy elements.

  4. 4.

Estimate the strength of the stochastic background due to compact binary
population: The aggregate of all sources, particularly binary neutron stars
will contribute to foreground noise which could interfere with the study or
the establishment of upper limits on the cosmological GW background.
Therefore, it is crucial to either remove or account for this foreground to
measure the properties of a primordial background accurately. gwforge datasets
will facilitate extensive testing of XG algorithms on large and realistic
samples.

  5. 5.

Investigating Deviations from general relativity (GR) and False Violations:
gwforge allows injecting non-GR signals through a PyCBC plugin, enabling users
to introduce and analyze these signals. This feature facilitates studies on
the efficacy of conventional tests in detecting deviations from GR.
Additionally, users can inject GR signals to investigate scenarios where tests
of GR may fail. Such failures could arise due to waveform systematics, sub-
optimal detector noise modelling, and various astrophysical aspects. By
examining both non-GR and GR signals, gwforge dataset will allow us to perform
mock tests of GR in XG era.

  6. 6.

Determining Merger Rates and Reconstructing the Underlying Distribution of
Source Populations: Given the anticipated high number of detections, utilizing
current hierarchical Bayesian inference techniques to determine the population
properties of a CBC type may become impractical. However, the sheer volume of
signals allows for an alternative approach: multiplying the posteriors of
individual events of a particular source type to infer population properties.
The gwforge dataset will enable the validation of this hypothesis and
facilitate the testing of faster and more accurate Bayesian inference
techniques.

##  5 Conclusion

In the upcoming decades, XG detectors are expected to replace the current GW
detection facilities. With longer baselines and new infrastructures, these
detectors can make transformative discoveries across astronomy, physics and
cosmology. However, the “richer” GW detector data will pose new computational,
physical and astrophysical challenges, which may not be straightforward to
mitigate.

For example, the sheer volume of signals will require developing novel signal
identification and extraction algorithms. Non-stationarity due to foreground
noise will pose additional challenges, especially in noise modelling,
complicating subsequent analysis. Furthermore, detecting loud signals will
necessitate the development of improved waveform models to achieve
unprecedented precision in parameter estimations and avoid false indications
of deviations from GR.

To simulate these challenges, we introduce gwforge, a user-friendly and
lightweight Python package designed to generate mock data for XG detectors.
Through various data simulation examples, we demonstrate the capabilities of
gwforge. These examples include examining foreground noise from overlapping
signals with intersecting time-frequency tracks, demonstrating the potential
of bright sirens for $H_{0}$ measurement, and discussing parameter estimation
biases due to waveform systematics. We also highlight a few potential
challenges that can be addressed using the simulated dataset. These
demonstrations underscore the potential applications of gwforge in supporting
the development of new analysis methods and preparing the scientific community
for the next era of gravitational wave astronomy.

## Acknowledgements

The author thanks B.S. Sathyaprakash, Ian Harry, Rossella Gamba and Ish Gupta
for their insightful comments and valuable suggestions. He extends special
thanks to Sophie Hourihane for her assistance with BayesLine. Heartfelt
appreciation is also given to Lorine Talhaoui, Tobias Forge, Papa Nihil, and
N. Ghouls for their continuous inspiration throughout the development of this
work.

The author acknowledges the generous support of the National Science
Foundation (NSF) through grant PHY-2207638. He is also grateful for the
computational resources provided by the Gwave (PSU) cluster, which were
essential for the numerical work conducted in this research.

The gwforge package extensively utilises the following gw packages: Bilby
(Ashton et al., 2019), PyCBC (Usman et al., 2016), gwpy (Macleod et al.,
2021), and Lalsuite (LIGO Scientific Collaboration et al., 2018). Furthermore,
various Python packages such as numpy (Harris et al., 2020), matplotlib
(Hunter, 2007), corner (Foreman-Mackey, 2016), and seaborn (Waskom, 2021) are
used in this work.

##  Appendix A BayesLine PSD estimate

Figure 8: Comparison of BayesLine median ASD estimates of 512 seconds of CE40
mock data. The black dotted line represents the ideal CE40 design sensitivity
curve. The red curve indicates the estimate when the noise includes signals
from population A, while the blue curve represents the estimate when the noise
includes signals from population B. The comparison demonstrates that depending
on the number of signals, the ASD estimate can deviate significantly at lower
frequencies from the expected noise.

The Welch method for estimating noise power spectral density (PSD) generates
an “off-source” noise PSD by median-averaging the spectral estimates of
neighbouring data segments. However, it is not widely used in GW PE due to its
sensitivity to non-Gaussian transients or “glitches”. Moreover, even for wide-
sense stationary Gaussian noise, the data whitened by the Welch method doesn’t
produce a $\mathcal{N}(0,1)$ distribution, leading to biased PE estimates
(Talbot & Thrane, 2020; Chatziioannou et al., 2019).

Hence, current LIGO-Virgo-Kagra Collaboration (LVK) analyses use the “on-
source” Bayesian spectral estimates, such as those provided by the BayesLine
algorithm (Littenberg & Cornish, 2015; Cornish & Littenberg, 2015; Cornish et
al., 2021). This method approximates the noise PSD by generating a
parameterised fit involving a combination of splines and Lorentzians. This has
yielded unbiased and statistically consistent results, with whitened residuals
that follow a $\mathcal{N}(0,1)$ distribution.

Furthermore, this methodology, originally developed for joint spectral and
signal inference, will be indispensable in the upcoming XG era, where no
signal-free off-source data will be available. However, current LVK analysis
uses sequential modelling, which may lead to notable discrepancies from joint
analyses as detector sensitivities improve. This discrepancy becomes more
pronounced, particularly for ground-based XG detectors, as the sum of
foreground noise won’t follow a Gaussian distribution.

To illustrate that, we re-compute the noise PSD of a 512s segment of data as
discussed in Sec. 4.1 using the BayesLine algorithm. Fig. 8 shows our result.
Like the Welch estimate, the foreground noise significantly affects noise
characterisation at lower frequencies, with a more pronounced impact observed
for population-B data, which assumes upper merger rates for BNS and BHNS
systems. This can affect PE results, potentially broadening constraints on
parameters like chirp mass ($\mathcal{M}_{c}$) and/or symmetric mass ratio
($\eta$) for longer signals while not significantly affecting the
measurability of parameters appearing at higher post-Newtonian (PN) orders
(see Johnson et al. (2024) for a possible way to mitigate this). However, this
may substantially impact binary parameter measurements for short-lived
signals, such as IMBH binaries. A detailed investigation is deferred to future
work.

## References

  * LVK (2023) 2023, Phys. Rev. X, 13, 011048, doi: 10.1103/PhysRevX.13.011048
  * LVK (2024) 2024\.  https://arxiv.org/abs/2404.04248
  * Aasi et al. (2015) Aasi, J., et al. 2015, Class. Quant. Grav., 32, 074001, doi: 10.1088/0264-9381/32/7/074001
  * Abbott et al. (2017) Abbott, B. P., et al. 2017, Nature, 551, 85, doi: 10.1038/nature24471
  * Abbott et al. (2020) Abbott, R., et al. 2020, Phys. Rev. Lett., 125, 101102, doi: 10.1103/PhysRevLett.125.101102
  * Acernese et al. (2015) Acernese, F., et al. 2015, Class. Quant. Grav., 32, 024001, doi: 10.1088/0264-9381/32/2/024001
  * Aghanim et al. (2020) Aghanim, N., et al. 2020, Astron. Astrophys., 641, A6, doi: 10.1051/0004-6361/201833910
  * Allen et al. (2012) Allen, B., Anderson, W. G., Brady, P. R., Brown, D. A., & Creighton, J. D. E. 2012, Phys. Rev. D, 85, 122006, doi: 10.1103/PhysRevD.85.122006
  * Ashton et al. (2019) Ashton, G., et al. 2019, Astrophys. J. Suppl., 241, 27, doi: 10.3847/1538-4365/ab06fc
  * Aso et al. (2013) Aso, Y., Michimura, Y., Somiya, K., et al. 2013, Phys. Rev. D, 88, 043007, doi: 10.1103/PhysRevD.88.043007
  * Bartos et al. (2017) Bartos, I., Kocsis, B., Haiman, Z., & Márka, S. 2017, Astrophys. J., 835, 165, doi: 10.3847/1538-4357/835/2/165
  * Borhanian & Sathyaprakash (2022) Borhanian, S., & Sathyaprakash, B. S. 2022.  https://arxiv.org/abs/2202.11048
  * Branchesi et al. (2023) Branchesi, M., et al. 2023, JCAP, 07, 068, doi: 10.1088/1475-7516/2023/07/068
  * Chandra (2024a) Chandra, K. 2024a, GWForge: A lightweight code to generate mock gravitational wave detector data, https://github.com/koustavchandra/gwforge
  * Chandra (2024b) —. 2024b, GWForge Documentation, https://koustavchandra.github.io/gwforge/index.html
  * Chatziioannou et al. (2019) Chatziioannou, K., Haster, C.-J., Littenberg, T. B., et al. 2019, Phys. Rev. D, 100, 104004, doi: 10.1103/PhysRevD.100.104004
  * Chen et al. (2024) Chen, H.-Y., Ezquiaga, J. M., & Gupta, I. 2024, Class. Quant. Grav., 41, 125004, doi: 10.1088/1361-6382/ad424f
  * Chen et al. (2022) Chen, H.-Y., Haster, C.-J., Vitale, S., Farr, W. M., & Isi, M. 2022, Mon. Not. Roy. Astron. Soc., 513, 2152, doi: 10.1093/mnras/stac989
  * Cornish & Littenberg (2015) Cornish, N. J., & Littenberg, T. B. 2015, Class. Quant. Grav., 32, 135012, doi: 10.1088/0264-9381/32/13/135012
  * Cornish et al. (2021) Cornish, N. J., Littenberg, T. B., Bécsy, B., et al. 2021, Phys. Rev. D, 103, 044006, doi: 10.1103/PhysRevD.103.044006
  * Corsi et al. (2024) Corsi, A., et al. 2024.  https://arxiv.org/abs/2402.13445
  * Cutler & Flanagan (1994) Cutler, C., & Flanagan, E. E. 1994, Phys. Rev. D, 49, 2658, doi: 10.1103/PhysRevD.49.2658
  * Damour et al. (2012) Damour, T., Nagar, A., & Villain, L. 2012, Phys. Rev. D, 85, 123007, doi: 10.1103/PhysRevD.85.123007
  * Del Pozzo et al. (2013) Del Pozzo, W., Li, T. G. F., Agathos, M., Van Den Broeck, C., & Vitale, S. 2013, Phys. Rev. Lett., 111, 071101, doi: 10.1103/PhysRevLett.111.071101
  * Dhani et al. (2024) Dhani, A., Völkel, S., Buonanno, A., et al. 2024.  https://arxiv.org/abs/2404.05811
  * Evans et al. (2021) Evans, M., et al. 2021.  https://arxiv.org/abs/2109.09882
  * Evans et al. (2023) —. 2023.  https://arxiv.org/abs/2306.13745
  * Farr & Gair (2018) Farr, W. M., & Gair, J. R. 2018, H0StatisticalLikelihood, https://github.com/farr/H0StatisticalLikelihood
  * Finn (1992) Finn, L. S. 1992, Phys. Rev. D, 46, 5236, doi: 10.1103/PhysRevD.46.5236
  * Fishbach et al. (2018) Fishbach, M., Holz, D. E., & Farr, W. M. 2018, Astrophys. J. Lett., 863, L41, doi: 10.3847/2041-8213/aad800
  * Foreman-Mackey (2016) Foreman-Mackey, D. 2016, The Journal of Open Source Software, 1, 24, doi: 10.21105/joss.00024
  * Graham et al. (2020) Graham, M. J., et al. 2020, Phys. Rev. Lett., 124, 251102, doi: 10.1103/PhysRevLett.124.251102
  * Gupta et al. (2023) Gupta, I., et al. 2023.  https://arxiv.org/abs/2307.10421
  * Harris et al. (2020) Harris, C. R., Millman, K. J., van der Walt, S. J., et al. 2020, Nature, 585, 357, doi: 10.1038/s41586-020-2649-2
  * Hinderer et al. (2010) Hinderer, T., Lackey, B. D., Lang, R. N., & Read, J. S. 2010, Phys. Rev. D, 81, 123016, doi: 10.1103/PhysRevD.81.123016
  * Holz & Hughes (2005) Holz, D. E., & Hughes, S. A. 2005, Astrophys. J., 629, 15, doi: 10.1086/431341
  * Hu & Veitch (2023) Hu, Q., & Veitch, J. 2023, Astrophys. J., 945, 103, doi: 10.3847/1538-4357/acbc18
  * Hunter (2007) Hunter, J. D. 2007, Computing In Science & Engineering, 9, 90
  * Isi et al. (2018) Isi, M., Smith, R., Vitale, S., et al. 2018, Phys. Rev. D, 98, 042007, doi: 10.1103/PhysRevD.98.042007
  * Islam et al. (2023) Islam, T., Vajpeyi, A., Shaik, F. H., et al. 2023.  https://arxiv.org/abs/2309.14473
  * Iyer et al. (2011) Iyer, B. R., Souradeep, T., Unnikrishnan, C. S., et al. 2011, LIGO-India, Proposal of the Consortium for Indian Initiative in Gravitational-Wave Observations (IndIGO)
  * Johnson et al. (2024) Johnson, A. D., Chatziioannou, K., & Farr, W. M. 2024, Phys. Rev. D, 109, 084015, doi: 10.1103/PhysRevD.109.084015
  * Kapil et al. (2024) Kapil, V., Reali, L., Cotesta, R., & Berti, E. 2024, Phys. Rev. D, 109, 104043, doi: 10.1103/PhysRevD.109.104043
  * LIGO Scientific Collaboration et al. (2018) LIGO Scientific Collaboration, Virgo Collaboration, & KAGRA Collaboration. 2018, LVK Algorithm Library - LALSuite, Free software (GPL), doi: 10.7935/GT1W-FZ16
  * Littenberg & Cornish (2015) Littenberg, T. B., & Cornish, N. J. 2015, Phys. Rev. D, 91, 084034, doi: 10.1103/PhysRevD.91.084034
  * Macleod et al. (2021) Macleod, D. M., Areeda, J. S., Coughlin, S. B., Massinger, T. J., & Urban, A. L. 2021, SoftwareX, 13, 100657, doi: 10.1016/j.softx.2021.100657
  * Madau & Dickinson (2014) Madau, P., & Dickinson, M. 2014, Ann. Rev. Astron. Astrophys., 52, 415, doi: 10.1146/annurev-astro-081811-125615
  * Maggiore et al. (2020) Maggiore, M., et al. 2020, JCAP, 03, 050, doi: 10.1088/1475-7516/2020/03/050
  * McKernan et al. (2014) McKernan, B., Ford, K. E. S., Kocsis, B., Lyra, W., & Winter, L. M. 2014, Mon. Not. Roy. Astron. Soc., 441, 900, doi: 10.1093/mnras/stu553
  * McKernan et al. (2019) McKernan, B., Ford, K. E. S., Bartos, I., et al. 2019, Astrophys. J. Lett., 884, L50, doi: 10.3847/2041-8213/ab4886
  * Most et al. (2020) Most, E. R., Jens Papenfort, L., Dexheimer, V., et al. 2020, Eur. Phys. J. A, 56, 59, doi: 10.1140/epja/s10050-020-00073-4
  * Most et al. (2019) Most, E. R., Papenfort, L. J., Dexheimer, V., et al. 2019, Phys. Rev. Lett., 122, 061101, doi: 10.1103/PhysRevLett.122.061101
  * Nissanke et al. (2010) Nissanke, S., Holz, D. E., Hughes, S. A., Dalal, N., & Sievers, J. L. 2010, Astrophys. J., 725, 496, doi: 10.1088/0004-637X/725/1/496
  * Pizzati et al. (2022) Pizzati, E., Sachdev, S., Gupta, A., & Sathyaprakash, B. 2022, Phys. Rev. D, 105, 104016, doi: 10.1103/PhysRevD.105.104016
  * Prakash et al. (2024) Prakash, A., Gupta, I., Breschi, M., et al. 2024, Phys. Rev. D, 109, 103008, doi: 10.1103/PhysRevD.109.103008
  * Punturo et al. (2010) Punturo, M., et al. 2010, Class. Quant. Grav., 27, 194002, doi: 10.1088/0264-9381/27/19/194002
  * Pürrer & Haster (2020) Pürrer, M., & Haster, C.-J. 2020, Phys. Rev. Res., 2, 023151, doi: 10.1103/PhysRevResearch.2.023151
  * Radice et al. (2017) Radice, D., Bernuzzi, S., Del Pozzo, W., Roberts, L. F., & Ott, C. D. 2017, Astrophys. J. Lett., 842, L10, doi: 10.3847/2041-8213/aa775f
  * Ramos-Buades et al. (2023) Ramos-Buades, A., Buonanno, A., Estellés, H., et al. 2023, Phys. Rev. D, 108, 124037, doi: 10.1103/PhysRevD.108.124037
  * Regimbau et al. (2012) Regimbau, T., et al. 2012, Phys. Rev. D, 86, 122001, doi: 10.1103/PhysRevD.86.122001
  * Reitze et al. (2019) Reitze, D., et al. 2019, Bull. Am. Astron. Soc., 51, 035.  https://arxiv.org/abs/1907.04833
  * Relton et al. (2022) Relton, P., Virtuoso, A., Bini, S., et al. 2022, Phys. Rev. D, 106, 104045, doi: 10.1103/PhysRevD.106.104045
  * Schutz (1986) Schutz, B. F. 1986, Nature, 323, 310, doi: 10.1038/323310a0
  * Speagle (2020) Speagle, J. S. 2020, Mon. Not. Roy. Astron. Soc., 493, 3132, doi: 10.1093/mnras/staa278
  * Stevenson et al. (2017) Stevenson, S., Vigna-Gómez, A., Mandel, I., et al. 2017, Nature Commun., 8, 14906, doi: 10.1038/ncomms14906
  * Talbot & Thrane (2020) Talbot, C., & Thrane, E. 2020, Phys. Rev. Res., 2, 043298, doi: 10.1103/PhysRevResearch.2.043298
  * Thompson et al. (2024) Thompson, J. E., Hamilton, E., London, L., et al. 2024, Phys. Rev. D, 109, 063012, doi: 10.1103/PhysRevD.109.063012
  * Usman et al. (2016) Usman, S. A., et al. 2016, Class. Quant. Grav., 33, 215004, doi: 10.1088/0264-9381/33/21/215004
  * Varma et al. (2019) Varma, V., Field, S. E., Scheel, M. A., et al. 2019, Phys. Rev. Research., 1, 033015, doi: 10.1103/PhysRevResearch.1.033015
  * Vigna-Gómez et al. (2018) Vigna-Gómez, A., et al. 2018, Mon. Not. Roy. Astron. Soc., 481, 4009, doi: 10.1093/mnras/sty2463
  * Waskom (2021) Waskom, M. L. 2021, Journal of Open Source Software, 6, 3021, doi: 10.21105/joss.03021
  * Welch (1967) Welch, P. 1967, IEEE Trans. Audio Electroacoust., 15, 70, doi: 10.1109/TAU.1967.1161901
  * Wu & Nitz (2023) Wu, S., & Nitz, A. H. 2023, Phys. Rev. D, 107, 063022, doi: 10.1103/PhysRevD.107.063022
  * Zhu et al. (2021) Zhu, J.-P., et al. 2021, Astrophys. J., 917, 24, doi: 10.3847/1538-4357/abfe5e

  *[XG]: next-generation
  *[GW]: gravitational-wave
  *[CE]: Cosmic Explorer
  *[ET]: Einstein Telescope
  *[CBC]: compact binary coalescence
  *[PE]: parameter estimation
  *[NS]: neutron star
  *[PSD]: power spectral density
  *[SNe]: Supernova
  *[CBCs]: compact binary coalescence
  *[BBH]: binary black hole
  *[NSs]: neutron star
  *[EoS]: Equation of State
  *[BNS]: binary neutron star
  *[BHNS]: black-hole neutron star binary
  *[ASD]: amplitude spectral density
  *[BBHs]: binary black hole
  *[SNR]: signal-to-noise ratio
  *[IMBH]: intermediate-mass black hole
  *[EM]: electromagnetic
  *[GWs]: gravitational-wave
  *[AGN]: active galactic nuclei
  *[BH]: Black hole
  *[BNSs]: binary neutron star
  *[GR]: general relativity
  *[LVK]: LIGO-Virgo-Kagra Collaboration
  *[PN]: post-Newtonian


# Spatio-Temporal Human Action Recognition Model with Flexible-interval
Sampling and Normalization

Yuke Yang
<EMAIL_ADDRESS>missing

###### Abstract

Human action recognition[Zhou_2018_CVPR] is a well-known computer vision and
pattern recognition task of identifying which action a man is actually doing.
Extracting the keypoint information of a single human with both spatial and
temporal features of action sequences plays an essential role to accomplish
the task[2018_CVPR, cao2018openpose, simon2017hand, cao2017realtime,
wei2016cpm, fang2017rmpe, xiu2018poseflow]. In this paper, we propose a human
action system for Red-Green-Blue(RGB) input video with our own designed
module. Based on the efficient Gated Recurrent Unit(GRU)[pmlr-v63-gao30,
chung2014empirical, 8053243] for spatio-temporal feature extraction, we add
another sampling module and normalization module to improve the performance of
the model in order to recognize the human actions. Furthermore, we build a
novel dataset with a similar background and discriminative actions for both
human keypoint prediction and behavior recognition. To get a better result, we
retrain the pose model with our new dataset to get better performance.
Experimental results demonstrate the effectiveness of the proposed model on
our own human behavior recognition dataset and some public datasets.

###### Index Terms:

human action, gated recurrent unit, convoluted neural network

##  I Introduction

Human action recognition has been a fundamental and challenging task in
computer vision over these years. Many related areas such as intelligent video
surveillance[Wang:2013:IMV:2397196.2397216], human-computer
interaction[Karray_human-computerinteraction:], video summary and
understanding[Miki_Haseyama2013], and smart device action
recognition[Zhou_2018_CVPR] are prevalent due to the development of human
action recognition. Most of the current methods focus on three types of input
– RGB, optical and skeleton videos[Song:2017:ESA:3298023.3298186, 8237377,
Liu_2017_CVPR, 8026287]. Among these three types, RGB is most closed to the
practical usage of real-time human recognition techniques because it is the
direct source of the data. For RGB videos, it has been proved that extracting
discriminative spatial-temporal features[10.1007/978-3-642-53932-9_52] would
effectively model the spatial and temporal evolutions of different actions.

Even though the current human action recognition task has been popular, public
dataset still has two problems. The first one is about the background. For
example, UCF-101[abs-1212-0402] has lots of action types that differ from each
other a lot, but the backgrounds of these images also vary a lot. So for any
algorithm with high precision on UCF-101, it may classify video based on the
background instead of recognizing the behaviors
precisely.[10.1145/2939672.2939778] Meanwhile, for some datasets containing
similar backgrounds, their action classification is too closed even for human-
beings. For example, for the HON4D Action dataset[Oreifej_2013_CVPR], lifting
the cup and drinking water are too resembled to distinguish even for humans so
we can expect it is difficult for the model to hold a high accuracy there. The
second one is not as serious as the first one, but it still needs to be
noticed here – the localization problem. Some models own good performance on
certain datasets such as the UCLA-Northwestern Action Dataset[Wang_2014_CVPR]
because most of their actions are taken place in the same areas of videos. The
models can be sensitive to the actions in these areas but not actions on the
other areas. As a result, a dataset containing similar backgrounds and
discriminative actions is in demand.

Three kinds of methods are prevalent through the whole human recognition on
RGB task: two-stream method[NIPS2014_5353, Feichtenhofer_2016_CVPR], three-
dimension convolution method[AAAI1817205, 801497123211, Tran_2015_ICCV,
2017arXiv171108200D] and spatio-temporal types of
method[10.1007/978-3-319-46487-9_50, Zhang_2017_ICCV]. However, the first and
second methods suffer from severe shortcomings. Two-stream methods extensively
developed these years, and some of them achieved a relatively high precision
on public datasets such as UCF-101. But according to some
research[10.1145/2939672.2939778], most of the two-stream methods depend
largely on the background of the RGB videos so this type of method did not
recognize the action but background instead. Some modern
work[10.1007/978-3-030-12939-2_20] have already shown that the precision of
the two-stream method has fallen down on the datasets with a similar
background, let alone the two-stream method costs a lot of computing
resources. Three-dimension convolution methods such as
T3D[2017arXiv171108200D] and C3D[AAAI1817205] can extract some spatio-temporal
feature but they are limited. They do not concentrate on adequate length of
the video so they do not have such high precision on some datasets such as
HMDB51[Kuehne11]. Consequently, for the RGB videos, the spatio-temporal method
would be most useful.

However, not all aspects are perfect for the current spatio-temporal
methods[10.1007/978-3-319-46487-9_50, Zhang_2017_ICCV]. Previous work has made
some achievements in human action recognition but their methods own two
drawbacks. The first one is the results would be misleading if the original
video contains sufficient time interference
factor[10.1007/978-3-030-12939-2_20, abs-1212-0402]. For example, a 10-second
falling video may contain 4-second standing time, 3-second falling time and
3-second lying time so that the algorithm would classify the video as standing
or lying, which would be unexpected because it is more important to focus on
the rapid change in the video. The second one is for some of these spatio-
temporal methods with sequential sampling[8297027], each sample may be
misleading because it may not cover the whole action. Some actions such as
archery in UCF-101 only cover 4 frames, but the whole video is much longer. 20
sequential frames of sampling in these methods must be insufficient. According
to these two difficulties, a sampling module can be effective.

Faced with these challenges, we build up our own dataset and propose a human
action recognition system. Specifically for our own dataset, we count certain
common behaviors with almost the same background in similar places and light
conditions with only one person inside each video. For our model, it
recognizes the human behaviors after extracting the bounding box from yolo-
version three[Redmon2018YOLOv3AI] and keypoint information extracting from
pose model[2018_CVPR, cao2018openpose, simon2017hand, cao2017realtime,
wei2016cpm, fang2017rmpe, xiu2018poseflow]. This paper integrates the pose
model and the latter spatio-temporal network. To solve the challenges above,
this paper proposes a solution to deal with the keypoint information with
flexible-interval sampling and normalization module inside the model in both
training and verifying process. The flexible-interval sampling and
normalization both work for increasing our model generalization – sampling
works on the time and normalization works on the space. Our work replaces the
conventional frequently-used LSTM[Hochreiter, Sak2014LongSM] with
GRU[pmlr-v63-gao30, chung2014empirical, 8053243], because the latter one
demonstrates better performance on this type of work and can have fewer
parameters. This method not only assists the model to be more precise but
generate more training set.

To summarize, we have made four main contributions:

  * •

we build our own dataset with a similar background and discriminative actions.

  * •

we add a unique sampling module to improve the accuracy of the model.

  * •

we add a unique normalization method to increase generalization on various
kinds of videos.

  * •

we retrain the pose model with our own labeled data to improve the accuracy of
the pose model.

The remainder of this paper is structured as follows. Section II presents how
we collect dataset and Section III shows the composition of our model. Section
IV describes how we conduct experiments. Results and discussions are shown in
Section V. Conclusion and future research directions are given in Section VI.

##  II Dataset

In this section, we first introduce the details about collecting our action
dataset named Spatio-Temporal Human Dataset(STH-Dataset) consisting of
multiple actions including standing, sitting, walking, waving, kicking,
falling down and other unrecognized behaviors. The eventual category may
contain the videos with an incomplete body which leads to unidentifiable
action. Figure 1 shows examples of our dataset.

Figure 1: Example of Dataset. STH-Dataset sampled from our collected human
action behavior dataset. The original source is videos and we display five
sequential frames to represent the whole behavior.

###  II-A Collecting Method

We collect our data[doi:10.1177/1729881417709079] in three kinds of scenes:
the light office with adequate decorations, the light office with few
decorations and the dark office with even no decoration. Generally, these
backgrounds are semblable but they all have nuances between themselves so that
the perfect performance on our dataset can prove that the model really focuses
on actions. Specifically, each scene consists of 9 cameras: top-front left
corner, top-front right corner, top-back left corner, top-back right corner,
bottom-front left corner, bottom-front right corner, bottom-back left corner,
bottom-back right corner and one more camera with the same level as the normal
human-beings’ height(1.35 meters when sitting on the chair). These cameras
take videos of the same action from different angles.

The cameras are named DS-2CD3T45-I3 camera produced by Haikang Inc. . One
mobile phone is connected to only one camera to record each video and store
them on the SD card. Finally, we clear up the dataset and store them according
to the category.

###  II-B Composition

According to table 1, our dataset consists of 7 kinds of actions including
standing, sitting, moving, waving, falling down and other unrecognized
behaviors with different persons. The total number of our dataset has already
reached 9604. The detailed information can be seen in table 1. All of the
videos are marked with 18 keypoint information and the type of action.

TABLE I: The Number of Each Action in STH-Dataset Action Type |  Number
---|---
Wave |  879
Walk |  321
Stand |  1808
Fall |  3851
Kick |  1234
Sit |  1136
Others |  375
Total |  9604
a The table contains a specific number of each category of action.

##  III Our Proposed Model

In this section, we describe our proposed Spatio-Temporal Human Action Model
in detail. We first introduce the background information of the Pose
model[2018_CVPR, cao2018openpose, simon2017hand, cao2017realtime, wei2016cpm,
fang2017rmpe, xiu2018poseflow] and the Gated Recurrent Unit
technique[pmlr-v63-gao30, chung2014empirical, 8053243]. Both of them are in
the leading position of the current method. Second, we notice how our model is
designed based on these two kinds of techniques with our own sampling module.

###  III-A Background information

#### Pose

Modern techniques for multiple human keypoint estimation have been various and
effective. We utilize these mature models to be trained on the combination of
our dataset and the public dataset – coco dataset
[10.1007/978-3-319-10602-1_48]. Specifically, we use existed model called
AlphaPose[fang2017rmpe, xiu2018poseflow] to train a pose model with the input
of RGB image and output the keypoint information of human beings.
Openpose[2018_CVPR, cao2018openpose, simon2017hand, cao2017realtime,
wei2016cpm] is another similar model but here we choose the AlphaPose because
of its excellent performance.

Here is a brief explanation of how the AlphaPose model works. The AlphaPose
model is a Regional Multi-person Pose Estimation(RMPE)[Fang_2017_ICCV] work,
which means it can predict keypoints in one image of multiple persons
utilizing the proposal method[NIPS2015_5638]. Generating the human bounding
boxes by the human detector, these images with different bounding boxes are
fed into the Symmetric Spatial Transformer Network(SSTN)[NIPS2015_5854] and
Single-Person Pose Estimator(SPPE) module, and they would be transferred into
the pose proposals automatically. SSTN is useful to extract a high-quality
single person region from an inaccurate bounding box and SPPE is useful to
extract keypoint information from a single-human image. The pose proposal
selection module would finish the final selection. With Non-Maximum-
Suppression(NMS)[Cai_2019_CVPR] method, the final output keypoints would be
filtered out.

This work won’t introduce the details of this model. In this section, we
utilize the AlphaPose model to predict human keypoints. Besides the original
coco[10.1007/978-3-319-10602-1_48] dataset to train the pose model, we add one
another public dataset downloading from the AI challenge contest
website(https://challenger.ai/) and our own dataset to supplement the human
keypoint dataset. The keypoints criterion is identical to the coco human
keypoints – 18 keypoints for each human. The details would be in the
experiment section.

#### Gated Recurrent Unit

Better than long short-term memory (LSTM)[Hochreiter, Sak2014LongSM], gated
recurrent units(GRU)[pmlr-v63-gao30, chung2014empirical, 8053243] technique is
extensively used in recurrent neural networks these years. It lacks an output
gate compared to the LSTM, consequently, most of the GRU has fewer parameters
than LSTM. It has been proved that for some large datasets, GRU works as well
as the LSTM. However, for some small datasets, GRU even performs more
effectively.

For this model, the GRU model utilization details are as
follows[69e088c8129341ac89810907fe6b1bfe] modulating the flow of information
inside the unit without having separate memory cells.

Figure 2: Gated Recurrent Unit Network Illustration. The r gate sets how much
the gate would forget about its input information and the z gate would control
which input it would depend on.

Figure 3: The Whole Model Illustration. The upper area of the figure
demonstrates training details of the model and the lower area of the figure
demonstrates the verifying detail of the model which would infer the action
name. The original input is the video of falling action and the pose model is
improved by the single image with its keypoint information. The skeleton image
is to illustrate how the keypoint of the image would be marked. In the figure,
the interval is set to 5 so 5 sequences are passed to the GRU in the training
process.

This paper will not describe the details of the GRU as demonstrating the
detailed function here. We use almost the same structure of GRU as it was
presented in the official implementation. The GRU consists of three kinds of
gates to learn spatio-temporal information. The reset gate started to update
the next iteration and normalize the data. Our activation function
$\varphi(\cdot)$ here would take effect. Finally, the unique update gate for
all the units would update the stored information. After adequate iterations,
we would calculate our expecting result.

###  III-B Our design

This subsection would introduce the structure of our model. Broadly, taking an
input of a sequence of videos, our model would predict the action type. The
model consists of two portions: the first one is the pose model and the second
one is the GRU model with our unique sampling method and normalization module
between these two portions. The pose model would extract the keypoints to
convert the two-dimension video into a one-dimension sequence. After the
handle of the sampling module, the several times larger augmented sequence
would be generated in the sampling module. Finally, the action would be
predicted according to the sequences. The details of the model are elaborated
in the following paragraph.

The training process is shown in figure 3. For the training process, the video
would first be transferred into human keypoint information. The groundtruth
keypoints of the single human image would supervise the pose model. Obviously,
not all the keypoints are available here because some of the keypoints may not
exist in the image, and we would note that $n_{k}$ as the number of the
keypoints that we have detected, and $(x_{i},y_{i})$ as each pair of the
keypoint with $i$ from 0-17. To be noticed, the keypoints that are not
available here would be assigned as $(0,0)$. Then the normalization and
sampling would take effect here.

The normalization method works on each pair of the body keypoints
$(x_{i},y_{i})$. The normalization module concentrates on two tasks – one is
globalization and the other is standardization. The globalization process
begins with selecting an initial point to replace the original point $(0,0)$,
which can eliminate the space interference factor. The new original point is
selected by computing the average of the valid keypoints by the formula:

$\displaystyle\overline{x}=\frac{\sum_{i=0}^{17}{x_{i}}}{n_{k}}$ (1)
$\overline{y}=\frac{\sum_{i=0}^{17}{y_{i}}}{n_{k}}$ (2)

The $(\overline{x},\overline{y})$ would be the new original point. Then the
standardization process starts. Assuming the width of the image is w and the
height of the image is h. Consequently, each point would be updated by:

$x_{i}=\frac{(x_{i}-\overline{x})}{w}$ (3)
$y_{i}=\frac{(y_{i}-\overline{y})}{h}$ (4)

The generating new pair $(x_{i},y_{i})$ would range from (-1,1). The
standardization puts all the points in the same scope and would make our model
more robust.

Then the sampling module would convert the video keypoint vectors into
multiple keypoint sequences according to the number of intervals, each of
which contains a concrete number of frames’ keypoints. For example, 2
intervals mean that the sequences would be generated by sampling the keypoint
vectors per 2 vectors, and the original videos would generate 2 sequences. 10
keypoint vectors generated from a video with interval as 2 would be divided
into two parts: the 1st, 3rd, 5th, 7th, 9th sequence and the 2nd, 4th, 6th,
8th, 10th sequence. Furthermore, these sequences would then be recognized as
the same action as they are generated from the same video. Meanwhile, all
these sequences can be recognized as separate which means we make our training
dataset much larger. The rest GRU would supervise itself based on its own
marked action.

The verifying process is shown in figure 3. The verifying process contains a
queue-based model that is similar to the function of the sampling module. It
shares the identical initial stage – first, we need the keypoint of the video
sequential frames and do the normalization process. Then each frame keypoints
would be pushed into a queue with a concrete capacity, which is manually set.
Once the queue reached its own capacity, the queue would pop out sequences as
the interval it sets. For example, a full queue with 50 capacity and 10
intervals would pop out a sequence with 1st, 11th, 21st, 31st and 41st vector
as the output. Then according to the action type and these sequences, the GRU
can finish its verifying process.

For real-time action recognition, the action result would be immediately
output to the screen or other connected devices. However, things can be
different for a video. For a video, the model would store all the actions of
the sequences and output the action name that is most probably the theme of
the video. It can be possible because most of the time, we can abandon the
static actions and focus on the longest-lasting dynamic behavior, which means
rapid change.

##  IV Experiments

###  IV-A Data Prepossessing and Division

All the frames in the video would firstly enter the pretrained yolo-version-
three model to predict the bounding box of a single person. As all of our
training datasets only contain video with a single person, we would not
consider the tracking information of each person. Then the bounding box would
be put into the pose model to finish the rest training process.

###  IV-B Model Training

This section describes the details of model training according to the method
above. For pose model, besides the existed data in the previous dataset, we
add other unused public datasets to the pose training process. For each pair
of data, we supplement the image with 18 keypoints: two eyes, one nose, one
forehead, one neck, two shoulders, one chest, two elbows, two hands, two
crotches, two thighs, and two feet. All the keypoint information is marked
with JSON File with labelme(https://github.com/wkentaro/labelme) tool at first
and transferred into a one-dimension vector.

For our GRU[pmlr-v63-gao30, chung2014empirical, 8053243] model, the model
takes in a one-dimension vector and output an action name. The whole process
is recognized as a classification task. For our model, we use three GRU layers
for the model. The activation function is tanh function. The classification
loss function is the cross-entropy function. Four dropout layers are
implemented to avoid overfitting.

##  V Results and Analysis

###  V-A Performance on our own dataset

We compare three kinds of models on our own dataset. The first one is the
baseline – the model with the conventional dense sampling method – the
sampling method is dense here without any interval. The second one is the
model with flexible-interval sampling but without normalization. It means that
we only deal with the data by dividing the width and height without the new
original point selection. And the last one is the model with both flexible-
interval sampling and normalization module. It is the model that we describe
above.

For our designed model, the accuracy is elaborated in figure 4. Figure 4
describes how the loss function and accuracy develops in the whole training
process. The whole model becomes steady after 50 epochs.

Figure 4: a)Training Result on Our STH-Dataset. The three whole loss vibrates
a lot before 50 epochs as the graph shows. The accuracy of the model on our
own dataset comes close to 1. Our methods improve the accuracy of the model,
especially on the last two categories. b)We compare the confusion matrix of
three models on our own dataset. The x-axis denotes the predicted label of the
model which is the result of the model. The y-axis denotes the truth label of
the model which is the groundtruth. Our method has an accuracy at 92% with
norm and 90% without norm, and the baseline has an accuracy at 80%. It is
clear to see that the last two methods own better performance.

Figure 4 compares these three models on the accuracy of the multiple action
types, which obviously shows the high precision with flexible-sampling. It can
be demonstrated that our flexible-sampling method is pretty effective.
However, the promotion by the normalization is not as effective as the
previous one. In the last two confusion matrix, the numbers centralize in the
diagonal so most of the videos are predicted right. For each action type, the
result gets improved.

###  V-B Performance on public dataset

We conduct more experiments to demonstrate our model can still be high-
efficient in other similar public action datasets. Specifically, we use the
UCLA-Northwestern Action Dataset[Wang_2014_CVPR] and SYSU
Dataset[hu2017jointly]. SYSU Dataset provides multiple sequential photos, and
we recover videos from these photos. Both of these two datasets hold similar
actions, so we select them as our subjects. The details of these datasets are
elaborated in the following paragraph.

#### UCLA-Northwestern Action Dataset

This dataset contains RGB, depth and human skeleton data captured
simultaneously by three Kinect cameras. For our task, we only utilize RGB
videos. These datasets include 8 action categories. Each action is performed
by 10 actors. All of these actions are taken place in the same location field.
This dataset contains data taken from a variety of viewpoints.

#### SYSU Dataset

The action dataset contains two categories of input: RGB photos and depth
photos. The dataset focuses on human-object interaction, which will be made
available to the human action benchmarking and analysis. The dataset contains
12 actions and each action is conducted by 40 persons.

For each dataset, we use our pose model[2018_CVPR, cao2018openpose,
simon2017hand, cao2017realtime, wei2016cpm, fang2017rmpe, xiu2018poseflow] to
infer the keypoints of each video to convert them into one-dimension vectors.
We first retrain our model based on these different datasets and then test the
accuracy of the model. And the accuracy result on each dataset is demonstrated
in figure 5.

Figure 5: Figure 5 demonstrates the comparison of accuracy between the
baseline and our model. Although it seems a little fluctuant at first, the
final accuracy becomes stable. The accuracy in the table demonstrates that on
both two datasets our improvement is pretty effective.

As figure 5 demonstrates, our model can significantly promote the accuracy of
the action recognition task on these two datasets. The promotion of the effect
on the UCLA-Northwestern Action Dataset is not obvious because each video
contains too few frames. Consequently, we can only choose 2 or 3 as our
interval for most of the videos with less than 10 frames. However, the effect
is still obvious. For SYSU dataset, it demonstrates the power of the
normalization. The human taking action in the SYSU dataset is not always fixed
in one place, he or she may take actions in different regions of the videos.
Normalization unifies them so that the spatio-temporal network can concentrate
on the action itself despite which region the human stays. Some of the other
methods utilize the depth figures or skeleton images in the dataset. On the
RGB video task, our method owns much better performance than the others. The
importance of the RGB video has already been placed in the introduction.

##  VI Conclusions and Future Work

In this work, we proposed a novel spatio-temporal human action with flexible-
interval sampling and normalization by taking advantage of pose model and GRU.
To improve the accuracy of the current model and make the current model more
robust, we propose a novel model with sampling and normalization. To take
sufficient advantage of all the training dataset, our own-made sampling module
helps to create more training data and help the model understand more what the
real behavior is. Finally, we combine the existed pose model, sampling module
and GRU to finally generate with a cross-entropy loss function. Experimental
result demonstrates that our model can classify the action accurately despite
the similar background.