# Measuring the Variance of the Macquart Relation in z-DM Modeling

Jay Baptista Department of Astronomy and Astrophysics, Yale University, New
Haven, CT 06520, USA  Kavli Institute for Particle Astrophysics & Cosmology,
Stanford University, P.O. Box 2450, Stanford, CA 94305, USA  J. Xavier
Prochaska Department of Astronomy and Astrophysics, University of California,
Santa Cruz, CA 95064, USA  Kavli Institute for the Physics and Mathematics of
the Universe (Kavli IPMU), 5-1-5 Kashiwanoha, Kashiwa, 277-8583, Japan
Division of Science, National Astronomical Observatory of Japan, 2-21-1 Osawa,
Mitaka, Tokyo 181-8588, Japan  Alexandra G. Mannings  Department of Astronomy
and Astrophysics, University of California, Santa Cruz, CA 95064, USA  C.W.
James International Centre for Radio Astronomy Research, Curtin University,
Bentley, WA 6102, Australia  R. M. Shannon Centre for Astrophysics and
Supercomputing, Swinburne University of Technology, Hawthorn, VIC 3122,
Australia  Stuart D. Ryder  Department of Physics & Astronomy, Macquarie
University, NSW 2109, Australia  A. T. Deller  Centre for Astrophysics and
Supercomputing, Swinburne University of Technology, Hawthorn, VIC 3122,
Australia  Danica R. Scott  International Centre for Radio Astronomy Research,
Curtin University, Bentley, WA 6102, Australia  Marcin Glowacki  International
Centre for Radio Astronomy Research, Curtin University, Bentley, WA 6102,
Australia  Nicolas Tejos  Instituto de Física, Pontificia Universidad Católica
de Valparaíso, Casilla 4059, Valparaíso, Chile

###### Abstract

The Macquart relation describes the correlation between the dispersion measure
(DM) of fast radio bursts (FRBs) and the redshift $z$ of their host galaxies.
The scatter of the Macquart relation is sensitive to the distribution of
baryons in the intergalactic medium (IGM) including those ejected from
galactic halos through feedback processes. The width of the distribution in
DMs from the cosmic web (${\rm DM}_{\rm cosmic}$) is parameterized by a
fluctuation parameter $F$, which is related to the cosmic DM variance by
$\sigma_{\rm DM}=Fz^{-0.5}$. In this work, we present a new measurement of $F$
using 78 FRBs of which 21 have been localized to host galaxies. Our analysis
simultaneously fits for the Hubble constant $H_{0}$ and the DM distribution
due to the FRB host galaxy. We find that the fluctuation parameter is
degenerate with these parameters, most notably $H_{0}$, and use a uniform
prior on $H_{0}$ to measure $\log_{10}F>-0.89$ at the $3\sigma$ confidence
interval and a new constraint on the Hubble constant
$H_{0}=85.3_{-8.1}^{+9.4}\,{\rm km\,s^{-1}\,Mpc^{-1}}$. Using a synthetic
sample of 100 localized FRBs, the constraint on the fluctuation parameter is
improved by a factor of $\sim 2$. Comparing our $F$ measurement to simulated
predictions from cosmological simulation (IllustrisTNG), we find agreement
between $0.4<z<2$. However, at $z<0.4$, the simulations underpredict $F$ which
we attribute to the rapidly changing extragalactic DM excess distribution at
low redshift.

Radio transient sources (2008), Radio bursts (1339), Cosmological parameters
(339), Intergalactic medium (813), Hubble constant (758)

##  1 Introduction

In galaxy formation models, AGN and stellar feedback have provided mechanisms
for regulating star formation and evacuating gas out of low-mass halo (Cen &
Ostriker, 2006; Davé et al., 2011). In simulations without baryonic outflows,
galaxies simply produce too many stars and have higher than observed star
formation rates (Davé et al., 2011). These outflow processes are also critical
in understanding how the IGM becomes enriched and how the galaxies and the IGM
co-evolve.

Not only is understanding the nature of feedback crucial in reproducing
realistic galaxy properties in cosmological-baryonic simulations but also in
understanding the channels where these “missing” baryons may have left halos
and are prevented from re-accretion. Gas accretion onto galaxies from cold gas
filaments is exceptionally efficient. Baryonic feedback is a preventative
process that not only enriches the IGM but also removes baryons and restricts
accretion from the IGM (Kereš et al., 2005). For example, in the Simba suite
of cosmological hydrodynamic simulations, feedback from AGN jets can cause 80%
of baryons in halos to be evacuated by $z=0$ (Davé et al., 2019; Appleby et
al., 2021; Sorini et al., 2022).

A comparison of simulation suites shows that different feedback prescriptions
can eject baryons at various distances beyond the halo boundary—feeding gas
into the reservoir of diffuse baryons (e.g. Ayromlou et al., 2023). Thus, to
constrain the strength of AGN and stellar feedback processes, one must be able
to constrain the distribution of baryons in the IGM to discriminate between
these feedback models.

Determining the distribution of these ejected baryons is difficult. Emission
and absorption lines from baryons in the IGM are extremely difficult to detect
due to their high temperatures and low densities (Fukugita et al., 1998; Cen &
Ostriker, 2006; Shull et al., 2012; McQuinn, 2014). However, the advent of
fast radio bursts (FRBs) presents a new opportunity to probe the intergalactic
distribution of baryons and provide a novel approach to measuring feedback
strength (McQuinn, 2014; Muñoz & Loeb, 2018).

FRBs are sensitive to the line-of-sight free electron density where the
integrated free electron density yields the dispersion measure (DM) of the
signal (Lorimer et al., 2007). A redshift can be estimated if the spatial
localization of the FRB overlaps with a galaxy with a known redshift (assuming
the FRB progenitor indeed lies within that galaxy; Aggarwal et al., 2021).
This FRB redshift-extragalactic dispersion measure (z-${\rm DM}_{\rm EG}$)
correlation known as the Macquart relation (Macquart et al., 2020) is
sensitive to cosmological properties of the universe (e.g. James et al.,
2022a). By using a sophisticated FRB observational model that can account for
observational biases, intergalactic gas distribution, burst width, and DM, it
is possible to use FRB surveys to infer the distribution of baryons in the
universe (McQuinn, 2014; James et al., 2022a; Lee et al., 2022).

Halos with weaker feedback retain their baryons more effectively leading to
halos with higher ${\rm DM}_{\rm EG}$ contributions and voids with lower ${\rm
DM}_{\rm EG}$ contributions (McQuinn, 2014). An FRB can travel either through
extremely low DM voids or pass through extremely high DM halos suddenly,
leading to enhanced scatter in the $z$-DMEG distribution. On the other hand,
halos with stronger feedback will cause the $z$-DMEG distribution to show less
scatter. Feedback processes are able to more effectively relocate halo baryons
into the IGM, causing inter-halo voids to have higher DM. This leads to a more
homogeneous universe. The goal of this work is to forecast and measure a
precise galactic feedback prescription as determined by a sample of FRBs.

This paper is organized as follows: Section 2 outlines the modeling of the
$z-{\rm DM}$ distribution and how the fluctuation parameter $F$ influences
observations of the FRB distribution; Section 3.1 presents our measurements on
the fluctuation parameter $F$ and the fits on other parameters used in the
$z$-DMEG model; Section 3.2 details our forecast on $F$ by sampling 100
synthetic FRBs and finding the probability distribution functions of our model
parameters based on the synthetic survey; Section 4 discusses our results in
the context of constraining cosmic feedback strength, parameter degeneracies,
and compares our measurements to simulations.

##  2 Methods

###  2.1 Basic Formalism

This work makes use of the FRB code zdm developed by James et al. (2022a) to
model observables of FRB populations. The model assumes that the DM
measurement of an FRB can be decomposed as ${\rm DM}_{\rm FRB}={\rm DM}_{\rm
ISM}+{\rm DM}_{\rm halo}+{\rm DM}_{\rm EG}$, where ${\rm DM}_{\rm ISM}$ and
${\rm DM}_{\rm halo}$ are the DM contributions due to the Milky Way’s
interstellar medium (modeled using NE2001; Cordes & Lazio, 2002a) and diffuse
ionized gas in the Galaxy’s halo (Prochaska & Zheng, 2019; Cook et al., 2023;
Ravi et al., 2023). The ${\rm DM}_{\rm EG}$ term is the extragalactic DM
contribution and is decomposed into ${\rm DM}_{\rm EG}={\rm DM}_{\rm
cosmic}+{\rm DM}_{\rm host}$ where ${\rm DM}_{\rm cosmic}={\rm DM}_{\rm
IGM}+{\rm DM}_{\rm halo,EG}$ is the contribution due to baryons in the IGM and
intersecting halos in the line-of-sight, and ${\rm DM}_{\rm host}$ is the
contribution due to the host galaxy of the FRB signal. The contribution from
the host (${\rm DM}_{\rm host}$) is modeled as a log-normal distribution with
a width of $\exp(\mu)$ and a logarithmic scatter of ${\sigma}_{\rm host}$
where $\mu$ and ${\sigma}_{\rm host}$ are free parameters of the model.

The width of the $z$-DMEG distribution at a fixed redshift is characterized in
part by the probability distribution of measuring the ${\rm DM}_{\rm cosmic}$
of an FRB above or below $\langle{\rm DM}_{\rm cosmic}\rangle$. This
distribution is described by $p_{\rm cosmic}(\Delta)$, where $\Delta\equiv{\rm
DM}_{\rm cosmic}/\langle{\rm DM}_{\rm cosmic}\rangle$:

$p_{\rm
cosmic}(\Delta)=A\Delta^{-\beta}\exp(-\frac{(\Delta^{-\alpha}-C_{0})^{2}}{2\alpha^{2}\sigma_{\rm
DM}^{2}}),$ (1)

where $\alpha\simeq 3$ and $\beta\simeq 3$ are the inner and outer slopes of
the gas profile density of intervening halos (based on numerical simulations
from Macquart et al., 2020), $C_{0}$ shifts the distribution such that
$\langle\Delta\rangle=1$, and $\sigma_{\rm DM}$ represents the spread of the
distribution (Macquart et al., 2020). This non-Gaussian probability
distribution function is motivated by theoretical treatments of the IGM and
galaxy halos (Macquart et al., 2020). For example, in the limit where
$\sigma_{\rm DM}$ is small, the distribution becomes Gaussian to capture the
Gaussianity of large-scale structures. This is physically motivated as halo
gas is more diffuse in this limit and thus contributions to the variance due
to halo gas are insignificant. In the limit where $\sigma_{\rm DM}$ is large,
the halo gas contribution becomes significant (Macquart et al., 2020).

Owing to the approximately Poisson nature of intersecting halos, one expects
$\sigma_{\rm DM}\propto z^{-1/2}$ (Macquart et al., 2020) and one is motivated
to introduce a fluctuation parameter $F$

$\sigma_{\rm DM}(\Delta)=Fz^{-0.5}\;\;\;,$ (2)

As the fluctuation parameter increases, i.e. $F\sim 1$, the spread of ${\rm
DM}_{\rm cosmic}$ increases. Figure 1 shows $p({\rm DM}|z)$ for two extreme
values of $F$ and the resultant, substantial changes to the width of the ${\rm
DM}_{\rm EG}$ distribution at any given redshift.

Figure 1: Upper panel: The $p({\rm DM_{\rm EG}}|z)$ distribution which admits
a high fluctuation parameter (low galactic feedback efficiency). Lower panel:
The $p({\rm DM_{\rm EG}}|z)$ distribution which admits a low fluctuation
parameter (high galactic feedback efficiency). The white dashed-line indicates
the 95 percentile contour. Note that the distribution primarily falls below
the mean due to the rare population of high DM FRBs that result from
intersections with the host galaxy and/or very massive galaxy halos along the
LOS.

The variance in ${\rm DM}_{\rm EG}$, however, is influenced by both
${\sigma}_{\rm host}$ and $F$. However, at high redshift, the contribution to
the variance of ${\rm DM}_{\rm EG}$ due to ${\sigma}_{\rm host}$ may decrease
relative to the contributions by ${\rm DM}_{\rm cosmic}$ and, inherently, $F$.
In James et al. (2022b), their work assumes that uncertainties attributed to
the fixed value of $F$ can be aggregated into uncertainties in ${\sigma}_{\rm
host}$; however, at high redshift, the assumption breaks down as the
uncertainty in $F$ becomes larger than the true constraint in ${\sigma}_{\rm
host}$.

Although Macquart et al. (2020) restrict their fitting of the fluctuation
parameter to $F\in[0.09,0.32]$ based on semi-analytic models, we sample a wide
range of $F\in[0,1]$. We opt for a logarithmic sampling of the fluctuation
parameter to efficiently sample this domain: $\log_{10}F\in[-2,0]$.

The additional parameters used in the model include $H_{0}$ (acceleration of
the Universe’s expansion), the ${\rm DM}_{\rm EG}$ contribution due to the FRB
host galaxy, and other parameters that govern the FRB luminosity function and
redshift distribution. The model assumes that the ${\rm DM}_{\rm EG}$
contribution from the FRB host galaxy can be modeled as a log-normal
distribution with a mean of $\mu_{\rm host}$ (or ${\rm DM}_{\rm host}$) and a
spread of $\sigma_{\rm host}$.

In terms of the luminosity function, the maximum burst energy is given as
$E_{\rm max}$, and the integral slope of the FRB luminosity function is
controlled by $\gamma$. The volumetric burst rate ($\Phi$) is controlled by
the parameter $n_{\rm sfr}$ assuming a star-formation rate: $\Phi\propto{\rm
SFR(z)}^{n_{\rm sfr}}$. Additionally, $\alpha$ is the spectral index that sets
a frequency-dependent FRB rate as $\Phi(z,\nu)=\Phi(z)\nu^{\alpha}$ (James et
al., 2022b).

###  2.2 Measuring $F$ Using FRB Survey Data

To measure the fluctuation parameter, we perform a simultaneous fit of the
parameters in the zdm model implemented by James et al. (2022b). We obtain the
probability distributions of each parameter by a brute-force grid search based
on the ranges specified in Table 2, and calculating the likelihoods for each
permutation of parameter values.

We fit these parameters using both the FRB sample used in James et al. (2022b)
and newly detected or analyzed FRBs (see Table 1) which were collected from
the Parkes and ASKAP telescopes. Of this sample of 78 measured FRBs, 57 FRBs
do not have measured redshifts. Constraining the redshift of an FRB greatly
increases the statistical power as a single FRB with a redshift can have the
same constraining power as roughly 20 FRBs without redshifts (James et al.,
2022b). Thus our inclusion of seven new CRAFT/ICS FRBs detected over three
frequency ranges (four with host redshifts), and our identification of the
host galaxy of FRB20211203C at $z=0.344$, provides a significant increase in
statistical precision.

There is a slight bias in this sample, as we include FRB20220610A, which has
an energy exceeding the previously estimated turnover $E_{\rm max}$ by a
factor of 3.5–10, depending on the assumed spectral behavior (Ryder et al.,
2022). FRB20210912A has a lower DM of 1234.5 ${\rm pc\,cm^{-3}}$, but it does
not have an identified redshift, perhaps due to the distance to its host
galaxy (Marnoch et al., in prep). Therefore, the inclusion of some data is
redshift-dependent. Given that our sample is statistically limited, we assume
the resulting bias to be small compared to the gain in precision.

In contrast to the James et al. (2022b) analysis, we hold model parameters
that are not degenerate with $F$ to their fiducial values. These parameters
were determined to be non-degenerate with $F$ running the model using a low-
resolution grid search on synthetic data to determine if $F$ correlated with
any of the other model parameters. From this preliminary analysis, we fix the
following parameters that were found to be non-degenerate with $F$: $\alpha$,
$\gamma$, $E_{\rm max}$, and $n_{\rm sfr}$. On the other hand, we expect the
fluctuation parameter to be degenerate with the other model parameters. In
particular, we expect the Hubble constant $H_{0}$–—the cosmological parameter
that quantifies the expansion of the universe–—to be degenerate with the
fluctuation parameter $F$.

We examine this degeneracy further in Figure 2 which shows the 95 percentile
contours for $p({\rm DM}|z)$ for two models with very different $F$ and
$H_{0}$ values. One notes that the lower contours (${\rm DM}_{\rm EG}\lesssim
720,\,z\lesssim 1.3$) of both realizations look nearly identical. Although the
contours differ above the mean, the bulk of the constraining power on $F$ is
in the lower contour or “DM cliff”. Therefore, we anticipate $F$ and $H_{0}$
to be highly correlated.

The distribution at the low DM end of $p({\rm DM_{\rm EG}}|z)$ exhibits a
sharp cut-off and provides strong constraints on $H_{0}$ since there is a
minimum imparted ${\rm DM}_{\rm cosmic}$ from voids and is not impeded by the
${\rm DM}_{\rm EG}$ contributions from large-scale structures like filaments
or halos. And while the contours do have modest differences at high $z$, high
${\rm DM}_{\rm EG}$, these can be difficult to distinguish from host galaxy
contributions to ${\rm DM}_{\rm EG}$.

Figure 2: $95^{\rm th}$ percentile contours of two $p({\rm DM_{\rm EG}}|z)$ distributions with different prescriptions on the Hubble constant $H_{0}$ and the fluctuation parameter $F$. Note that the lower contours (“the DM cliff”) of both models are nearly identical to each other. Since the DM cliff places a stronger constraint on $H_{0}$ and $F$ than the upper contour, we expect a high degree of degeneracy between $H_{0}$ and $F$. The units of ${\rm DM}_{\rm EG}$ are in pc cm-3 and the units of $H_{0}$ are $\rm km\,s^{-1}\,Mpc^{-1}$. Table 1: New FRB detections detected in 2022 used in addition to the FRB surveys used in James et al. (2022b). The FRB name, SNR-maximizing DM, ${\rm DM}_{\rm ISM}$ estimated using the NE2001 model of Cordes & Lazio (2002b), central frequency of observation $\nu$, measured signal-to-noise ratio SNR, redshift $z$, and original reference. Where redshifts are not given, this is because (a): no voltage data were dumped, preventing radio localization; (b) optical follow-up observations are not yet complete; (c) Substantial Galactic extinction has challenged follow-up optical observations; (d) the host galaxy appears too distant to accurately measure a redshift.  Name | DM | ${\rm DM}_{\rm ISM}$ | $\nu$ | SNR | $z$ | Ref.
---|---|---|---|---|---|---
| (${\rm pc\,cm^{-3}}$) | (${\rm pc\,cm^{-3}}$) | (MHz) |  |  |
CRAFT/ICS $900\,{\rm MHz}$
20211203C | 636.2 | 63.4 | 920.5 | 14.2 | 0.344 | Shannon et al. (in prep.)
20220501C | 449.5 | 30.6 | 863.5 | 16.1 | 0.381
20220725A | 290.4 | 30.7 | 920.5 | 12.7 | 0.1926
CRAFT/ICS $1.3\,{\rm GHz}$
20220531A | 727.0 | 70.0 | 1271.5 | 9.7 | – | Shannon et al. (in prep.)
20220610A | 1458.1 | 31.0 | 29.8 | 1.016 | Ryder et al. (2022)
20220918A | 656.8 | 40.7 | 26.4 | – | Shannon et al. (in prep.)
CRAFT/ICS $1.6\,{\rm GHz}$
20220105A | 583.0 | 22.0 | 1632.5 | 9.8 | 0.2785 | Shannon et al. (in prep.)
20221106A | 344.0 | 34.8 | 1631.5 | 35.1 | –

###  2.3 Forecasting the fluctuation parameter $F$ using Synthetic FRBs

Table 2: z-DM grid parameters

Parameter | Unit | Fiducial | Min | Max | N
---|---|---|---|---|---
$H_{0}$ | km s-1 Mpc-1 | 67.4 | 60.0 | 80.0 | 21
$\log F$ | - | 0.32 | -1.7 | 0 | 30
${\mu}_{\rm host}$ | $\log{\rm pc\,cm^{-3}}$ | 2.16 | 1.7 | 2.5 | 10
${\sigma}_{\rm host}$ | $\log{\rm pc\,cm^{-3}}$ | 0.51 | 0.2 | 0.9 | 10
$\log_{10}E_{\rm max}$ | $\log{\rm erg}$ | 41.84 | – | – | –
$n_{\rm sfr}$ | - | 1.77 | – | – | –
$\alpha$ | - | 1.54 | – | – | –
$\gamma$ | - | -1.16 | – | – | –

Note. — This table indicates the parameters of the high-resolution grid run.
Non-degenerate parameters are held to the fiducial values. $N$ is the number
of cells between the minimum and maximum parameter values.

Future radio surveys are expected to widely increase the number of sub-
arcsecond localized FRBs. Thus, the constraining power on $F$ will greatly
increase. To explore this scenario we generate a forecast on the fluctuation
parameter by replicating our analysis using a synthetic FRB survey. A sample
of 100 localized synthetic FRBs was drawn assuming the distribution of FRBs
followed the fiducial $z-{\rm DM}$ distribution (Table 2). With this synthetic
survey, we calculate the associated 4D likelihood matrix and make a forecast
on the fluctuation parameter by adopting different priors on $H_{0}$.

##  3 Results

###  3.1 Parameter Likelihoods from FRB Surveys

In Figure 3, we present the 1D PDFs of each parameter determined from the 78
FRBs collected from the ASKAP and Parkes Radio Telescopes. In comparison to
James et al. (2022b), there is a significant loss of constraining power on
$H_{0}$ by including $F$ as a free parameter. We measure a Hubble constant to
be $85.3_{-8.1}^{+9.4}$ which is 1.5 times more uncertain than the $H_{0}$
measurement in James et al. (2022b). We attribute the uncertainty to the
degeneracy between $H_{0}$ and $F$ as indicated by the strong anti-correlation
in Figures 2 and 5.

To understand how our survey data at different frequencies contribute to the
constraining power on $H_{0}$ and $F$, we replicate the survey contribution
determination from James et al. (2022b) which provides the 1D parameter
likelihood across different FRB surveys with the Murriyang (Parkes) and
Australian Square Kilometre Array (ASKAP). Figure 4 shows the 1D PDFs of
$H_{0}$ and $\log_{10}F$ across the different surveys used in this analysis.
We observe that the CRAFT 1.3 GHz and 900 MHz surveys tend to have stronger
constraining power as they contain more FRBs with measured distances (10 and 7
redshifts respectively) than the rest of the surveys.

Figure 3: The calculated 1D likelihood functions using 78 FRBs (21 FRBs with
redshifts). $F$ is measured to be $\log_{10}F=-0.75_{-0.25}^{+0.33}$ with no
priors on $H_{0}$. There is a loss of constraining power on $H_{0}$ compared
to the measurement by James et al. (2022b) when allowing the $F$ parameter to
vary.

Figure 4: Upper panel: 1D likelihood functions of $H_{0}$ based on different
FRB surveys used in this work. Compared to the survey contribution constraints
(see Figure 7 in James et al., 2022b), the constraining power of each survey
is diminished. Lower panel: Same as upper panel for likelihood functions of
$\log_{10}F$. The CRAFT/ICS 900 MHz and 1.3 GHz surveys provide the most
constraining power on $F$ and $H_{0}$ as they contain more redshifts than the
other surveys.

In Figure 5, we present the 2D likelihoods of each parameter against $F$. We
observe correlations between $F$ and the FRB host galaxy parameters
(${\mu}_{\rm host}$, ${\sigma}_{\rm host}$), which we expect to be degenerate
given that they both influence the variance of $\langle{\rm DM}_{\rm
cosmic}\rangle$. As expected from Figure 2, the degeneracy in the lower bound
(or cliff) of the $z$-DMEG distribution results in the strong anti-correlation
of $H_{0}$ and $F$, resulting in a loss of constraining power on $H_{0}$ when
allowing $F$ to vary.

Our initial simultaneous fit does not implement any priors on the model
parameters. As motivated by the $H_{0}$-$F$ degeneracy in Figure 2, we
determine the 1D likelihoods of $\log_{10}F$ by limiting our grid to different
values of $H_{0}$. We consider a uniform prior on $H_{0}$ between 67.4 and
73.04 ${\rm km\,s^{-1}\,Mpc^{-1}}$—the lower bound is motivated by the $H_{0}$
constraint from Planck Collaboration et al. (2020), and the upper bound is
motivated by cosmological constraints using type-1a supernovae (SNe) from
Riess et al. (2022).

In Figure 6, we present the 1D likelihood of the fluctuation parameter
assuming different priors on $H_{0}$. Assuming a uniform prior between the CMB
and SNe-derived values of $H_{0}$, we measure the fluctuation parameter to be
$\log_{10}F=-0.48^{+0.26}_{-0.18}$ within $1\sigma$
($\log_{10}F=\log_{10}F>-0.89$ with 99.7% confidence). We present all
measurements of the $F$ parameter with different priors on $H_{0}$ in Table 3.

Figure 5: The 2D likelihood functions for each parameter compared against
$\log_{10}F$ derived from 78 FRBs (21 FRBs with redshifts). There is a strong
anti-correlation between $F$ and $H_{0}$. Additionally, we observe strong
correlations between $F$ and the host galaxy ${\rm DM}_{\rm EG}$ contribution
(${\mu}_{\rm host},{\sigma}_{\rm host}$). Figure 6: 1D survey likelihoods of
$\log_{10}F$ assuming different priors on $H_{0}$. The dotted gray line is the
original 1D likelihood without any priors on $H_{0}$. The blue line is the
likelihood which adopts a uniform prior on $H_{0}\in[67.4,73.04]$. The dashed
gray line is the likelihood adopting $H_{0}=67.0\,{\rm kms^{-1}Mpc^{-1}}$. The
dash-dotted gray line is the likelihood adopting $H_{0}=73.0\,{\rm
kms^{-1}Mpc^{-1}}$. Adopting a prior on $H_{0}$ or fixing the values of
$H_{0}$ greatly improves the constraint on $F$. Table 3: Measurements of the
$F$ Parameter

Survey | No Prior | Uniform $H_{0}$ Prior | CMB $H_{0}$ | SNe $H_{0}$
---|---|---|---|---
Observed | $-0.75_{-0.25}^{+0.33}$ | $-0.48^{+0.26}_{-0.18}$ | $-0.35_{-0.15}^{+0.23}$ | $-0.52_{-0.17}^{+0.26}$
Synthetic | $-0.58_{-0.15}^{+0.15}$ | $-0.60_{-0.10}^{+0.09}$ | $-0.52_{-0.06}^{+0.07}$ | $-0.67_{-0.06}^{+0.06}$

Note. — This table lists the measurements of the $F$ parameter from the
observational FRB survey (76 localized FRBs with 16 redshifts) and the
synthetic CRACO survey (100 localized FRBs; all with redshifts). The
measurements are presented without a prior, a uniform prior between the CMB
($H_{0}=67.0\,{\rm km\,s^{-1}\,Mpc^{-1}}$) and SNe ($H_{0}=73.0\,{\rm
km\,s^{-1}\,Mpc^{-1}}$) with their respective Gaussian errors on each side,
and fixing $H_{0}$ to the CMB or SNe estimates.

###  3.2 Parameter Likelihoods from Synthetic Surveys

Figure 7: The 1D likelihood functions for each parameter using 100 synthetic
FRBs. The constraint on $F$ is enhanced as the uncertainty due to sample size
is reduced. Similarly, the constraint on $H_{0}$ is improved compared to the
observational fit with only 21 redshifts.

We use a synthetic sample of 100 localized CRACO FRBs to investigate the
improvement in constraining power on both $F$ and $H_{0}$. In Figure 7, we
present the PDFs of each parameter in the grid. We observe that the constraint
on $H_{0}$ has significantly improved by a factor of 1.7 and is more Gaussian
than the previous run with $69.2_{-4.9}^{+5.5}$. Assuming a survey of 100
localized FRBs, the best measurement we can make on $H_{0}$ if we adopt a
Gaussian prior on $\log_{10}F$ (assuming $1\sigma$ corresponds to a 20% error
in the measurement) is $67.6_{-3.4}^{+3.5}\,{\rm km\,s^{-1}\,Mpc^{-1}}$ (see
Table 4).

In Figure 8 we show posterior estimates for $\log_{10}F$ using different
priors (see Table 3). Using the uniform prior, we obtain a forecast on the
fluctuation parameter of $\log_{10}F=-0.60_{-0.18}^{+0.19}$ within $2\sigma$.
We note that when compared to Figure 3, there is a definitive upper limit on
the fluctuation parameter rather than only a lower limit. Incorporating the
uniform prior enhances the constraint on $F$ by a factor of $\sim 1.5$, and
fixing the value of $H_{0}$ can increase the constraint by a factor of $2.5$.

Figure 8: Same as Figure 6 but based on fits to the synthetic FRB sample.
Assuming a uniform prior between CMB and SNe values of $H_{0}$ equipped with
their associated Gaussian errors on both sides, we find
$\log_{10}F=-0.60_{-0.10}^{+0.09}$. Fixing the value of $H_{0}$ also greatly
enhances the constraint on $F$ by a factor of $>1.5$. Table 4: Measurements of
$H_{0}$

Survey | No Prior | Gaussian Prior
---|---|---
Observed | $85.3_{-8.1}^{+9.4}$ | -
Synthetic | $69.2_{-4.9}^{+5.5}$ | $67.6_{-3.4}^{+3.5}$

Note. — This table lists the measurements of $H_{0}$ from the observational
FRB survey (76 localized FRBs with 16 redshifts) and the synthetic CRACO
survey (100 localized FRBs; all with redshifts). The measurements are
presented without a prior on $F$ and a Gaussian prior on $F$ centered at
$\log_{10}F\simeq-0.49$ with $\sigma\simeq 0.1$ (20% error on $F$).

##  4 Discussion

###  4.1 Measurement of the fluctuation parameter

Our principle result from the population analysis of 78 FRBs (21 with
redshifts) is a lower limit on $F$ which is $\log_{10}F=-0.48^{+0.26}_{-0.18}$
($\log_{10}F>-0.89$ at 99.7% confidence). This measurement is motivated by
James et al. (2022b), where they noted that for future localization of FRBs
beyond $z\gtrsim 1$, $F$ may need to be fitted explicitly. We note that this
observation is only made when adopting a prior between the CMB and SNe values
of $H_{0}$.

###  4.2 Fluctuation Parameter Degeneracies

Our findings indicate a strong degeneracy between the Hubble constant $H_{0}$
and the fluctuation parameter $F$ when simultaneously fitting both within the
z-DM modeling framework adopted by James et al. (2022a) which uses $F=0.32$
which falls within the accepted range of our measurement.

Aside from the degeneracy between $F$ and $H_{0}$, we would like to call
attention to the possible degeneracy between $F$ and $\sigma_{8}$–the RMS
amplitude of the matter density field when smoothed with an $8h^{-1}$ Mpc
filter. In the case of w feedback ($F\rightarrow 1$), more mass would be
concentrated within cosmic filaments, increasing the variance of a fixed-mass
filter (i.e., $\sigma_{8}$). We expect these two parameters to be inversely
coupled. A preliminary analysis varying $\sigma_{8}$ in the CAMELS
IllustrisTNG cosmological simulations does show a positive correlation between
$F$ and $\sigma_{8}$ (Medlock et al. in prep.).

###  4.3 Forecasting enhanced constraints on $F$

Using a sample of 100 synthetic FRBs (see Figure 8), we are able to constrain
both upper and lower limits on the fluctuation parameter out to $3\sigma$.
Since we are only able to effectively constrain a lower limit on $F$, we
compare the lower-sided half-maximum widths. We find the left-sided half-
maximum width of the synthetic distribution is half the width of the current
measured distribution. We expect this constraint to only improve with more
localizations, which will be easily facilitated with next-generation all-sky
radio observatories.

Additionally, it is of interest to see how this method compares to other ways
of measuring the baryon distribution in the IGM. For example, an alternative
method to constrain AGN and stellar feedback focuses on small-scale deviations
in the matter power spectrum (van Daalen et al., 2020). As baryonic feedback
significantly influences the mass distribution at smaller scales (higher $k$),
probes of the gas density at those scales (thermal Sunyaev-Zel’dovich effect)
can measure the intergalactic baryon distribution (Pandey et al., 2023).

###  4.4 Comparing with Fluctuation Parameter in IllustrisTNG

In a work by Zhang et al. (2021) to highlight the utility of FRBs in probing
the IGM, they generated thousands of FRB sightlines in IllustrisTNG and fitted
the observed extragalactic DM excess $p_{\rm cosmic}(\Delta)$. They provide
the fitted parameters as well as the dispersion in the $z$-DMEG distribution
$\sigma_{\rm DM}$. We convert these values into the fluctuation parameter $F$
by assuming $\sigma_{\rm DM}=Fz^{-0.5}$.

In Figure 9, we present these derived $\log_{10}F$ values as a function of
redshift compared to our measured values. Between $0.4<z<2$, our measurements
are in fine agreement. However, we observe that the fluctuation parameter in
Illustris appears to be higher at $z\lesssim 0.4$ and lower when $z>2$.

From the redshift-dependent ${\rm DM}_{\rm IGM}$ distributions derived from
IllustrisTNG (figure 2 from Zhang et al. (2021)), distributions between
$0.1<z<0.4$ are wider and the modes of each distribution are spread further
apart. This may explain why the IllustrisTNG fluctuation parameter is higher
than our measurement as the ${\rm DM}_{\rm IGM}$ distribution functions have
larger variance at those redshifts.

To make a proper comparison between our work and Zhang et al. (2021), it may
be necessary to introduce a free parameter for the redshift evolution of
$\sigma_{\rm DM}$ instead of simply fixing the redshift exponent to $-1/2$
(Equation 2).

Figure 9: Fluctuation parameters derived from this work, the fiducial value
from James et al. (2022b), and the IllustrisTNG values from Zhang et al.
(2021). Our measurement on $F$ agrees with the simulated $F$ parameter between
$0.4<z<2$.

##  5 Conclusions

In this work, we have implemented variance in ${\rm DM}_{\rm cosmic}$ as a
free parameter in a forward model of the $z$-DMEG distribution of FRBs. With
this adapted model and a survey of 78 ASKAP and Parkes FRBs, we constrain a
value for the fluctuation parameter, explore degeneracies within the model,
and generate a forecast of the constraint on the fluctuation parameter with a
synthetic survey of 100 localized FRBs. The conclusions we draw from this
analysis are:

  * •

Incorporating survey data of 78 (21 with redshifts) FRBs yields a firm lower
limit on $F$. We place the lower limit on $F$ as measured by the survey sample
to be $\log_{10}F>-0.89$ at 99.7% confidence. The 900 MHz and 1.3 GHz surveys
dominate this constraint due to their higher number of localizations to host
galaxies and their associated redshifts.

  * •

Forward modeling the FRB data from Parkes and ASKAP, the fluctuation parameter
is degenerate with the Hubble constant $H_{0}$.

  * •

We forecast that 100 localized FRBs are sufficient to constrain both an upper
and lower limit on the fluctuation parameter. With the greater count of
localizations, the half-maximum width of the distribution decreases by
$\approx 50\%$.

  * •

Extrapolation of the fluctuation parameter from IllustrisTNG shows agreement
between $0.4<z<2.0$. Zhang et al. (2021) measure a higher fluctuation
parameter at low redshift ($z<0.4$) and a lower fluctuation parameter beyond
$z>2$. The former result is likely to be an effect of the rapidly evolving
${\rm DM}_{\rm IGM}$ distribution at low redshift.

Next-generation radio observatories will significantly improve the constraint
on the fluctuation parameter. For example, the Deep Synoptic Array 2000
(DSA-2000) is expected to localize on the order of 10,000 FRBs each year —
enough FRBs to sufficiently characterize the baryonic contents of the IGM
(Hallinan et al., 2019; Ravi et al., 2019).

Additionally, the FRB Line-of-sight Ionization Measurement From Lightcone
AAOmega Mapping (FLIMFLAM) survey is an upcoming spectroscopic survey that
seeks to map the intervening cosmic structures and diffuse cosmic baryons in
front of localized FRBs (Lee et al., 2022). These FRB foreground data taken in
the Southern hemisphere will be used in conjunction with ASKAP FRB
measurements to improve the constraints on the intergalactic baryon
distribution (Lee et al., 2022).

These expansions in FRB surveys with localizations are expected to greatly
improve the constraints on the fluctuation parameter. With these improved
constraints on $F$, one may leverage this novel observable for investigating
feedback and cosmological prescriptions in simulations.

Combining the $F$ parameter with other observables like the thermal SZ effect
that trace the intergalactic baryon distribution, there is ample opportunity
to better inform subgrid feedback models (Muñoz & Loeb, 2018; Pandey et al.,
2023).

## Acknowledgements

J.B. acknowledges support from the University of California Santa Cruz under
the Lamat REU program, funded by NSF grant AST-1852393, and the Yale Science
Technology and Research Scholars Fellowship funded by the Yale College Dean’s
Office. Authors A.G.M. and J.X.P., as members of the Fast and Fortunate for
FRB Follow-up team, acknowledge support from NSF grants AST-1911140,
AST-1910471 and AST-2206490. The authors acknowledge the use of the Nautilus
cloud computing system which is supported by the following US National Science
Foundation (NSF) awards: CNS-1456638, CNS-1730158, CNS-2100237, CNS-2120019,
ACI-1540112, ACI-1541349, OAC-1826967, OAC-2112167.

CWJ and MG acknowledge support by the Australian Government through the
Australian Research Council’s Discovery Projects funding scheme (project
DP210102103).

RMS and ATD acknowledge support through Australian Research Council Future
Fellowship FT190100155 and Discovery Project DP220102305.

The Australian SKA Pathfinder is part of the Australia Telescope National
Facility (https://ror.org/05qajvd42) which is managed by CSIRO. Operation of
ASKAP is funded by the Australian Government with support from the National
Collaborative Research Infrastructure Strategy. ASKAP uses the resources of
the Pawsey Supercomputing Centre. Establishment of ASKAP, the Murchison Radio-
astronomy Observatory and the Pawsey Supercomputing Centre are initiatives of
the Australian Government, with support from the Government of Western
Australia and the Science and Industry Endowment Fund. We acknowledge the
Wajarri Yamatji people as the traditional owners of the Observatory site.

This research is based on observations collected at the European Southern
Observatory under ESO programmes 0102.A-0450(A), 0103.A-0101(A),
0103.A-0101(B), 105.204W.001, 105.204W.002, 105.204W.003, 105.204W.004,
108.21ZF.001, 108.21ZF.002, 108.21ZF.005, 108.21ZF.006, and 108.21ZF.009.

## References

  * Aggarwal et al. (2021) Aggarwal, K., Budavári, T., Deller, A. T., et al. 2021, ApJ, 911, 95, doi: 10.3847/1538-4357/abe8d2
  * Appleby et al. (2021) Appleby, S., Davé, R., Sorini, D., Storey-Fisher, K., & Smith, B. 2021, MNRAS, 507, 2383, doi: 10.1093/mnras/stab2310
  * Ayromlou et al. (2023) Ayromlou, M., Kauffmann, G., Anand, A., & White, S. D. M. 2023, MNRAS, 519, 1913, doi: 10.1093/mnras/stac3637
  * Cen & Ostriker (2006) Cen, R., & Ostriker, J. P. 2006, ApJ, 650, 560, doi: 10.1086/506505
  * Cook et al. (2023) Cook, A. M., Bhardwaj, M., Gaensler, B. M., et al. 2023, ApJ, 946, 58, doi: 10.3847/1538-4357/acbbd0
  * Cordes & Lazio (2002a) Cordes, J. M., & Lazio, T. J. W. 2002a, arXiv e-prints, astro.  https://arxiv.org/abs/astro-ph/0207156
  * Cordes & Lazio (2002b) —. 2002b, ArXiv Astrophysics e-prints
  * Davé et al. (2019) Davé, R., Anglés-Alcázar, D., Narayanan, D., et al. 2019, MNRAS, 486, 2827, doi: 10.1093/mnras/stz937
  * Davé et al. (2011) Davé, R., Oppenheimer, B. D., & Finlator, K. 2011, MNRAS, 415, 11, doi: 10.1111/j.1365-2966.2011.18680.x
  * Fukugita et al. (1998) Fukugita, M., Hogan, C. J., & Peebles, P. J. E. 1998, ApJ, 503, 518, doi: 10.1086/306025
  * Hallinan et al. (2019) Hallinan, G., Ravi, V., Weinreb, S., et al. 2019, in Bulletin of the American Astronomical Society, Vol. 51, 255, doi: 10.48550/arXiv.1907.07648
  * James et al. (2022a) James, C. W., Prochaska, J. X., Macquart, J. P., et al. 2022a, MNRAS, 509, 4775, doi: 10.1093/mnras/stab3051
  * James et al. (2022b) James, C. W., Ghosh, E. M., Prochaska, J. X., et al. 2022b, MNRAS, 516, 4862, doi: 10.1093/mnras/stac2524
  * Kereš et al. (2005) Kereš, D., Katz, N., Weinberg, D. H., & Davé, R. 2005, MNRAS, 363, 2, doi: 10.1111/j.1365-2966.2005.09451.x
  * Lee et al. (2022) Lee, K.-G., Ata, M., Khrykin, I. S., et al. 2022, ApJ, 928, 9, doi: 10.3847/1538-4357/ac4f62
  * Lorimer et al. (2007) Lorimer, D. R., Bailes, M., McLaughlin, M. A., Narkevic, D. J., & Crawford, F. 2007, Science, 318, 777, doi: 10.1126/science.1147532
  * Macquart et al. (2020) Macquart, J. P., Prochaska, J. X., McQuinn, M., et al. 2020, Nature, 581, 391, doi: 10.1038/s41586-020-2300-2
  * McQuinn (2014) McQuinn, M. 2014, ApJ, 780, L33, doi: 10.1088/2041-8205/780/2/L33
  * Muñoz & Loeb (2018) Muñoz, J. B., & Loeb, A. 2018, Phys. Rev. D, 98, 103518, doi: 10.1103/PhysRevD.98.103518
  * Pandey et al. (2023) Pandey, S., Lehman, K., Baxter, E. J., et al. 2023, arXiv e-prints, arXiv:2301.02186, doi: 10.48550/arXiv.2301.02186
  * Planck Collaboration et al. (2020) Planck Collaboration, Aghanim, N., Akrami, Y., et al. 2020, A&A, 641, A6, doi: 10.1051/0004-6361/201833910
  * Prochaska & Zheng (2019) Prochaska, J. X., & Zheng, Y. 2019, MNRAS, 485, 648, doi: 10.1093/mnras/stz261
  * Ravi et al. (2019) Ravi, V., Battaglia, N., Burke-Spolaor, S., et al. 2019, BAAS, 51, 420, doi: 10.48550/arXiv.1903.06535
  * Ravi et al. (2023) Ravi, V., Catha, M., Chen, G., et al. 2023, arXiv e-prints, arXiv:2301.01000, doi: 10.48550/arXiv.2301.01000
  * Riess et al. (2022) Riess, A. G., Yuan, W., Macri, L. M., et al. 2022, ApJ, 934, L7, doi: 10.3847/2041-8213/ac5c5b
  * Ryder et al. (2022) Ryder, S. D., Bannister, K. W., Bhandari, S., et al. 2022, arXiv e-prints, arXiv:2210.04680, doi: 10.48550/arXiv.2210.04680
  * Shull et al. (2012) Shull, J. M., Smith, B. D., & Danforth, C. W. 2012, ApJ, 759, 23, doi: 10.1088/0004-637X/759/1/23
  * Sorini et al. (2022) Sorini, D., Davé, R., Cui, W., & Appleby, S. 2022, MNRAS, 516, 883, doi: 10.1093/mnras/stac2214
  * van Daalen et al. (2020) van Daalen, M. P., McCarthy, I. G., & Schaye, J. 2020, MNRAS, 491, 2424, doi: 10.1093/mnras/stz3199
  * Zhang et al. (2021) Zhang, Z. J., Yan, K., Li, C. M., Zhang, G. Q., & Wang, F. Y. 2021, ApJ, 906, 49, doi: 10.3847/1538-4357/abceb9


11institutetext: Leiden University, Niels Bohrweg 1, 2333 CA Leiden, The
Netherlands 22institutetext: TNO, Oude Waalsdorperweg 63, 2597 AK The Hague,
The Netherlands
22email<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
# Spot What Matters: Learning Context
Using Graph Convolutional Networks
for Weakly-Supervised Action Detection

Michail Tsiaousis 1Corresponding author1Corresponding authorThis work was
carried out during an internship at TNOThis work was carried out during an
internship at TNO 0000-0002-9355-5026 Gertjan Burghouts 22 0000-0001-6265-7276
Fieke Hillerström 22 0000-0003-1301-3073 Peter van der Putten 11
0000-0002-6507-6896

###### Abstract

The dominant paradigm in spatiotemporal action detection is to classify
actions using spatiotemporal features learned by 2D or 3D Convolutional
Networks. We argue that several actions are characterized by their context,
such as relevant objects and actors present in the video. To this end, we
introduce an architecture based on self-attention and Graph Convolutional
Networks in order to model contextual cues, such as actor-actor and actor-
object interactions, to improve human action detection in video. We are
interested in achieving this in a weakly-supervised setting, i.e. using as
less annotations as possible in terms of action bounding boxes. Our model aids
explainability by visualizing the learned context as an attention map, even
for actions and objects unseen during training. We evaluate how well our model
highlights the relevant context by introducing a quantitative metric based on
recall of objects retrieved by attention maps. Our model relies on a 3D
convolutional RGB stream, and does not require expensive optical flow
computation. We evaluate our models on the DALY dataset, which consists of
human-object interaction actions. Experimental results show that our
contextualized approach outperforms a baseline action detection approach by
more than 2 points in Video-mAP. Code is available at
https://github.com/micts/acgcn.

###### Keywords:

Weakly-Supervised action detection graph convolutional networks relational
reasoning actor-context relations

##  1 Introduction

††Paper presented at the International Workshop on Deep Learning for Human-
Centric Activity Understanding (DL-HAU2020), January 11, 2021

Human action recognition is an important part of video understanding, with
potential applications in robotics, autonomous driving, surveillance, video
retrieval and healthcare. Given a video, spatiotemporal action detection aims
to localize all human actions in space and time, and classify the actions
being performed. The dominant paradigm in action detection is to extend CNN-
based object detectors [13, 18] to learn appearance and motion representations
in order to jointly localize and classify actions in video. The desired output
are action tubes [9]: sequences of action bounding boxes connected in time
throughout the video.

In contrast to object detection, action detection requires learning of both
appearance and motion features. Although spatiotemporal features are essential
for action recognition and detection, they might prove insufficient for
actions that share similar characteristics in terms of appearance and motion.
For example, spatiotemporal features might not be sufficient to differentiate
the action ”Taking Photos” in Fig. 1 from a similar one, such as ”Phoning”,
since both share similar characteristics in space and time (i.e. similar
posture, motion around the head). As humans, we make use of context to put
actions and objects in perspective, which can be an important cue to improve
action recognition. Such contextual cues can refer to actor-object and actor-
actor interactions. For instance, a person holding a camera is more likely to
perform the action ”Taking Photos” than ”Phoning”, and vice versa. CNNs are
able to capture such abstract or distant visual interactions only implicitly
by stacking several convolutional layers, which increases the overall
complexity and number of parameters. Hence, an approach to explicitly model
contextual cues would be beneficial.

We introduce an approach to explicitly learn contextual cues, such as actor-
actor and actor-object interactions, to aid action classification for the task
of action detection. Our model, inspired by recent work on graph neural
networks [12, 27, 32], learns context by performing relational reasoning on a
graph structure using Graph Convolutional Networks (GCN) [12]. A high-level
overview is illustrated in Fig. 1. Given a detected actor in a short video
clip, we construct a graph with an actor node encoding actor features, and
context nodes encoding context features, such as objects and other actors in
the scene. The graph’s adjacency matrix consists of relation values encoding
the importance of context nodes to the actor node, and is learned during
training via gradient descent. Graph convolutions accumulate the learned
context to the actor to obtain contextualized/updated actor features for
action classification. Our model aids explainability by visualizing the
learned adjacency matrix as an attention map that highlight the relevant
context for recognizing the action.

We are interested in an approach to learn these contextual cues using as few
annotated data as possible. Recent works [7, 23, 25, 33] that model contextual
cues for action detection rely on full supervision in terms of actor bounding
box annotations. However, extensive video annotation is time consuming and
expensive [15]. In this work, we are interested in learning context for the
task of weakly-supervised action detection, i.e. action detection when only a
handful of annotated frames are available throughout the action instance.
Following the setting of sparse spatial supervision [31], we train our
contextual model by using up to five actor bounding box annotations throughout
the action instance.

We evaluate our models on the challenging Daily Action Localization in YouTube
(DALY) dataset [31], which consists of 10 action classes of human-object
interactions (e.g. Drinking, Phoning, Brushing Teeth), and is annotated based
on sparse spatial supervision. Therefore, DALY is a suitable test bed to model
context for the task of weakly-supervised action detection.

Our contributions are as follows: 1) We introduce an architecture employing
Graph Convolutional Networks [12] in order to model contextual cues to improve
classification of human actions in videos; 2) Our model aids explainability by
visualizing the graph’s adjacency matrix in the form of attention maps that
highlight the learned context, even in a zero-shot setting, i.e. for actions
and objects unseen during training; 3) We achieve 1) and 2) in a weakly-
supervised setting, i.e. when annotated data are sparse throughout the action
instance; 4) We introduce an intuitive metric based on recall of retrieved
objects in attention maps, in order to quantitatively evaluate how well the
model highlights the important context. As attention maps are often used for
qualitative inspection only, this metric may be of general use beyond our use
case.

In Section 3, we present the baseline model and our approach on learning
context by performing reasoning on a graph structure using GCN [12].
Additionally, we discuss training of these models using sparse spatial
supervision [31]. We conduct experiments and report results in Section 4. In
Section 5, we perform a qualitative and quantitative analysis of attention
maps.

Figure 1: Given spatiotemporal features extracted from an input clip, we
construct a graph with an actor (grey) node and context (numbered) nodes in
order to model relations, such as actor-actor and actor-object interactions.
Graph convolutions accumulate the learned context to the actor to obtain
updated actor features for classification.

##  2 Related Work

###  2.1 Action Recognition and Detection

Action recognition aims to classify the action taking place in a video. Early
approaches relied on two-stream 2D CNNs [21] operating on RGB and optical flow
inputs, or on detectors to classify bounding boxes of components in keyframes
leaving action classification at the video level to traditional machine
learning techniques [3]. Recent works focus on (two-stream) 3D CNNs [4, 17,
24], which perform spatiotemporal (3D) convolutions. While action recognition
considers the classification task, action detection carries out both
classification and detection of actions. Although action detection is usually
addressed using full supervision [7, 9, 23, 25, 30, 33], we are interested in
weak supervision, which allows us to reduce annotation cost by training models
using very few action bounding box annotations per action instance.

Sivan and Xiang [22] approach weakly-supervised action detection using
Multiple Instance Learning (MIL). Their approach requires binary labels at the
video level, indicating the presence of an action. Mettes et al. [15] propose
action annotation using points, instead of boxes. Chéron et al. [5] present a
unified framework for action detection by incorporating varying levels of
supervision in the form of labels at the video level, a few bounding boxes,
etc. Weinzaepfel et al. [31] introduce DALY and the setting of sparse spatial
supervision, in which, up to five bounding boxes are available per action
instance. Chesneau et al. [6] produce full-body actor tubes inferred from
detected body parts, even when the actor is occluded or part of the actor is
not included in the frame.

###  2.2 Visual Relational Reasoning

There has been recent research on augmenting deep learning models with the
ability to perform visual relational reasoning.

Santoro et al. [20] propose the relation network, which models relations
between pairs of feature map pixels for visual question answering. This idea
has been extended for action detection [23, 25] to model actor-context
relations. Similar to [23, 25], we treat every 1$\times$1$\times$1 location of
the feature map as context. In these works, learned actor-context relations
are used either directly [23] or to highlight features of the feature map [25]
for action classification. In contrast, we encode actor-context relations as
edges in a graph, and graph convolutions output updated actor features for
action classification.

With regard to visual attention, non-local neural networks [28] compute the
output of a feature map pixel as a weighted sum of all input pixels. For
action detection, Girdhar et al. [7] extend the Transformer architecture [26]
for action detection. Although a Transformer can be represented as a graph
neural network and vice versa, we argue that a graph representation is simpler
and more intuitive compared to the Transformer representation of Queries, Keys
and Values. Whilst [7] use two transformations of a feature map to represent
context as Keys and Values, this representation does not have a direct
interpretation in a graph. In contrast, we use a single transformation to
obtain context features, which correspond to context nodes in the graph.
Furthermore, our model does not require residual connections [11] nor Layer
Normalization [2], two essential components of the Transformer.

In this work, we apply Graph Convolutional Networks (GCN) [12], which provide
a structured and intuitive way to model relations between nodes in a graph.
Recently, GCN have been used for visual relational reasoning for the tasks of
action recognition [29] and group activity recognition [32]. Zhang et al. [33]
employ GCN [12] for action detection, where nodes represent detected actors
and objects. As we do not require an external object detector, our approach is
suitable to reason with respect to arbitrary context and objects which cannot
be detected, e.g. because the object detector has not been trained to do so.

Whilst aforementioned approaches [7, 23, 25, 33] rely on full actor
supervision during training, our work focuses on learning contextual cues
using weak supervision in terms of actor bounding box annotations. Although
similar attention maps are also presented in [7, 23, 25], our work is the
first to generate them in a zero-shot setting, and introduce a metric to
quantitatively evaluate them.

##  3 Learning Context with GCN

We propose an approach to learn contextual cues, such as actor-actor and
actor-object interactions, by performing relational reasoning on a graph
structure using Graph Convolutional Networks (GCN) [12]. We expect such
contextual cues to improve action classification, as they can be
discriminative for the actions performed, and provide more insight into what
the model has learnt, which benefits interpretability. Moreover, we are
interested in learning context using weak supervision in terms of actor
bounding box annotations. An overview of our proposed GCN model is illustrated
in the second branch of Fig. 2. The input is a short video clip with at least
one actor performing an action. A 3D convolutional network extracts
spatiotemporal features for the input clip, up to a convolutional layer. We
treat every $1\times 1\times 1$ spatiotemporal location of the output feature
map as context, while actor features are extracted by RoI Pooling [8] on the
detected actor’s bounding box and subsequent 3D convolutional layers. We
construct a graph consisting of context nodes and an actor node, with
connections drawn from every context node to the actor node. Relation values
between nodes are encoded in the adjacency matrix, and learned using a dot-
product self-attention mechanism [26, 27]. Graph convolutions accumulate the
learned context to the actor in order to obtain updated/contextualized actor
features for action classification. We compare the GCN model to a baseline
model that uses no context, and classifies the action using the feature
representation corresponding to the actor bounding box.

In this section, we first present the 3D convolutional backbone network to
learn spatiotemporal features using weak supervision. Next, we present our
approach on learning contextual cues for action detection using GCN and we
provide implementation details.

Figure 2: The lowest part shows the graph representation of the GCN model for
a single graph, while its implementation using matrix operations is shown in
the middle part. The top part illustrates the construction of multiple graphs
(multi-head attention) in order to learn different types of actor-context
relations.

###  3.1 Feature Extraction with Weak Supervision

####  3.1.1 Backbone Network

Spatiotemporal features for the whole input clip are extracted using a 3D
convolutional backbone network. There are several 3D architectures in
literature [4, 17, 24]. We opt for I3D [4], which is widely used and has
demonstrated very positive results in action recognition. The input is a
sequence of frames of size $C\times T\times H\times W$, where $C$ denotes the
number of channels, $T$ is the number of input frames, and $H$ and $W$
represent the height and width of the input sequence. Features are extracted
up to Mixed_4f layer, which has an output feature map of size
$D^{\prime}\times T^{\prime}\times H^{\prime}\times W^{\prime}$, where
$D^{\prime}$ denotes the number of feature channels, $T^{\prime}=\frac{T}{8}$,
$H^{\prime}=\frac{H}{16}$, and $W^{\prime}=\frac{W}{16}$.

####  3.1.2 Actor Feature Extraction with Weak Supervision

We are interested in learning spatiotemporal features using only a handful of
annotated frames per action instance. For an annotated frame, also called
keyframe, annotation is in the form of an action bounding box and
corresponding class label. Due to the limited number of available annotated
frames, training a Region Proposal Network (RPN) [18] to produce actor box
proposals would be sub-optimal. To this end, we train our models using sparse
spatial supervision as introduced in [31]. In detail, a Faster R-CNN [18]
detects all actors in each frame, and detections are tracked throughout the
action instance using a tracking-by-detection approach [30], which produces
class-agnostic action tubes. In practice, we use tubes provided by [31]. Tubes
are labeled based on spatiotemporal Intersection over Union (IoU) with sparse
annotations, i.e. ground truth tubes comprised of up to 5 bounding boxes
throughout the action instance. Tubes with spatiotemporal IoU greater than 0.5
are assigned to the action class of the ground truth tube with the highest
IoU. If no such ground truth tube exists, the action tube is labeled as
background. The backbone is augmented with a RoI pooling layer [8] to extract
features for each actor for action classification. Boxes of each tube are
appropriately scaled and mapped to the output feature map of Mixed_4f layer,
with a temporal stride of four frames. For each action tube, RoI pooling
extracts actor features of size $D^{\prime}\times T^{\prime}\times 7\times 7$.
Actor features are then passed through I3D tail consisted of 3D convolutional
layers Mixed_5b and Mixed_5c. Finally, a spatiotemporal (3D) average pooling
layer reduces the size to $D^{\prime\prime}\times 1\times 1\times 1$.

###  3.2 Graph Convolutional Networks

####  3.2.1 Learning Relations

Our graph consists of two types of nodes: context nodes and actor nodes.
Context node features, $f_{j}^{\prime}\in\mathbb{R}^{D^{\prime}\times 1\times
1\times 1}$, $j=1,2,\dots,M$, $M=T^{\prime}H^{\prime}W^{\prime}$, correspond
to every $1\times 1\times 1$ spatiotemporal location of the output feature map
of Mixed_4f layer. Actor node features,
$a_{i}^{\prime}\in\mathbb{R}^{D^{\prime\prime}\times 1}$, $i=1,2,\dots,N$,
where $N$ is the number of detected actors in the input clip, are extracted as
described in Section 3.1. Relations between actor features and context
features, shown in orange arrows in Fig. 2, are learned using a dot-product
self-attention operation [26, 27], after projecting the features in a lower
dimensional space using a linear transformation. Formally,

$\displaystyle e_{ij}=\theta(a_{i}^{\prime})^{T}\cdot\phi(f_{j}^{\prime})$ (1)

where

$\displaystyle
a_{i}=\theta(a_{i}^{\prime})=\mathbf{W}_{\theta}a_{i}^{\prime}+\mathbf{b}_{\theta}$
(2) $\displaystyle
f_{j}=\phi(f_{j}^{\prime})=\mathbf{W}_{\phi}f_{j}^{\prime}+\mathbf{b}_{\phi}$
(3)

Eq. 2–3 are transformations for actor features and context features,
respectively, with $\mathbf{W}_{\theta}\in\mathbb{R}^{D^{\prime\prime}\times
D},\mathbf{W}_{\phi}\in\mathbb{R}^{D^{\prime}\times D}$;
$\mathbf{b}_{\theta},\mathbf{b}_{\phi}\in\mathbb{R}^{D\times 1}$;
$D<D^{\prime},D^{\prime\prime}$. In matrix form,
$\mathbf{A}\in\mathbb{R}^{N\times D}$ for transformed actor features and
$\mathbf{F}\in\mathbb{R}^{M\times D}$ for transformed context features. The
graph is represented by an adjacency matrix, $\mathbf{G}\in\mathbb{R}^{N\times
M}$, where $g_{ij}\in\mathbf{G}$ denotes the relation or attention value,
indicating the importance of context feature, $f_{j}$, to actor feature,
$a_{i}$. Consequently, $\mathbf{G}$ is a directed graph connecting every
context node to every actor node. Relation or attention values, $g_{ij}$, are
obtained by applying softmax normalization on $e_{ij}$ (output of dot-product)
across context features

$\displaystyle g_{ij}=\frac{\exp(e_{ij})}{\sum_{k}\exp(e_{ik})}$ (4)

####  3.2.2 Graph Convolutions

Having defined the graph and a mechanism for learning actor-context relations,
we perform reasoning on the graph in order to obtain updated actor features.
This is achieved by accumulating information from context nodes to the actor
node using graph convolutions. Updated actor features,
$\mathbf{Z}\in\mathbb{R}^{N\times D}$, are obtained by

$\displaystyle\mathbf{Z}=\sigma\Big{(}\Big{(}\mathbf{G}\mathbf{F}+\mathbf{A}\Big{)}\mathbf{W}\Big{)}$
(5)

The operation is shown in blue arrows in Fig. 2. The weighted average of
$\mathbf{F}$ with the relation values $\mathbf{G}$ produces weighted context
features. Adding actor features $\mathbf{A}$ to the resulting representation
imposes identity links for all actor nodes in the graph. The output is passed
through a learnable linear transformation $\mathbf{W}\in\mathbb{R}^{D\times
D}$ and a non-linear activation function $\sigma(\cdot)$ implemented as ReLU
[10].

In order to capture multiple types of relations between the actor and the
context, we perform multi-head attention [26] by constructing multiple graphs
at a given layer and merging their outputs using concatenation or summation.
Weight matrices $\mathbf{W}_{\theta},\mathbf{W}_{\phi}$, $\mathbf{W}$ are
independent across graphs. Finally, in order to encode updated actor features
on a higher level, we stack multiple GCN layers by providing the output of
multiple graphs as input to the next GCN layer.

####  3.2.3 Location Embedding

Location information, such as the position of an actor with respect to other
actors and objects, is important for modeling contextual cues. However, such
information, encoded indirectly by regular convolutions, is lost when applying
convolutions on a graph structure.

We incorporate location information in both context features and actor
features. For context features, we concatenate coordinates $(x,y)$ along the
channel dimension before applying $\mathbf{W}_{\phi}$, indicating the location
of the feature on the output feature map. For actor features, we concatenate
coordinates $(cx,cy,w,h)$ before applying $\mathbf{W}_{\theta}$, corresponding
to the average center, width and height of the actor tube across the input
clip. Coordinates are normalized in $[-1,1]$.

###  3.3 Implementation Details

We implement our models in PyTorch [16]. I3D is pre-trained on ImageNet [19]
and then on the Kinetics [4] action recognition dataset, while the external
detector is pre-trained on the MPII Human Pose dataset [1]. The input is a
clip of 32 RGB frames with spatial resolution of $224\times 224$. The output
feature map of Mixed_4f layer has $D^{\prime}=832$ channels, while actor
features have $D^{\prime\prime}=1024$. Transformations $\mathbf{W}_{\theta}$,
$\mathbf{W}$ are implemented as fully connected layers and $\mathbf{W}_{\phi}$
as a 3D convolutional layer with kernel size $1\times 1\times 1$. We set
$D=256$. We apply 3-dimensional dropout to context features before
$\mathbf{W}_{\phi}$. Additionally, 1-dimensional dropout is applied to actor
features before $\mathbf{W}_{\theta}$ in the first GCN layer, before
$\mathbf{W}$ in all GCN layers and prior to the final classification layer (in
both GCN and baseline model). Dropout probability is 0.5 in all cases. All
fully connected layers are initialized using a Normal distribution according
to [10]. We set the gain parameter to 1 for $\mathbf{W}_{\theta}$ and to
$\sqrt{2}$ for the rest of the fully connected layers. $\mathbf{W}_{\phi}$ is
initialized using a Uniform distribution according to [10] in the range
$(-b+0.01,b-0.01)$ for the first GCN layer, and in the range $(-b,b)$ for
subsequent layers, using a gain of $\frac{1}{\sqrt{3}}$. Biases of all layers
are initialized to zero.

Models are optimized using SGD and cosine learning rate annealing, with
learning rate $2.5\cdot 10^{-4}$ over 150 epochs, and $4.7\cdot 10^{-5}$ over
450 epochs, for the baseline and GCN model, respectively. We use a batch size
of 3 clips, where each clip is randomly sampled from a video in the training
set. Tubes of each clip are scored using the softmax scores produced by the
model. During inference, we sample 10 32-frame clips from each video, and
tubes are scored by averaging the softmax scores across the clips. The same
clips are sampled in order to facilitate fair comparison between different
models. Training time is approximately one day for GCN and less than half a
day for the baseline on a GTX 1080 Ti GPU.

##  4 Experiments

In this section, we first describe the DALY dataset and the evaluation metric
used throughout the experiments. Next, we conduct experiments to evaluate the
performance of the GCN model, and we compare it with the baseline and the
state of the art on DALY. Finally, we evaluate the GCN model using minimal
spatial supervision i.e. one bounding box per action instance.

###  4.1 Dataset and Evaluation Metric

We develop and evaluate our models on the Daily Action Localization in Youtube
(DALY) [31] dataset. It consists of 510 videos of 10 human actions, such as
”Drinking”, ”Phoning” and ”Brushing Teeth”. In this paper, we do not perform
temporal localization, and we assume that the temporal boundaries of each
action instance within a video are known. An action instance has an average
duration of 8 seconds and may contain more than one person performing an
action. Each of the 10 classes contains an interaction between a person and an
object that define the action taking place. There are 31 training videos and
20 test videos per class. We fine-tune our models by holding out a subset of
the training set as a validation set, consisted of 10 videos from each class.
We evaluate models using Video-mAP at 0.5 IoU threshold<EMAIL_ADDRESS>[9].

###  4.2 Evaluation of Architecture Choices

In this section, we experimentally evaluate the GCN model with respect to
several architecture choices. Specifically, we experiment with up to two GCN
layers and up to three graphs per layer. Additionally, we compare
concatenation and summation as merging functions to combine the output of
multiple graphs. Finally, we measure the impact of including the location
embedding and the I3D tail (convolutional layers Mixed_5b and Mixed_5c) to
extract actor features.

Results with respect to different number of layers and graphs are shown in
Table 1, along with the number of parameters for every configuration (I3D
parameters are not included). Note that for two GCN layers, the first layer
always employs concatenation as a merging function. Building multiple graphs
is beneficial for model performance, for both functions. It is interesting
that mAP increases for a 2-layer GCN model with concatenation, but not with
summation. Concatenation outperforms summation in nearly all configurations.
For the rest of the experiments, we choose a 2-layer, 2-graph GCN model with
concatenation, which provides a good trade-off between performance and number
of parameters.

In order to measure the impact on model performance obtained by the location
embedding and I3D tail, we remove them from the architecture and examine the
difference in model performance. By removing the location embedding, the model
has no information of the actor’s location relatively to other actors and
objects, and relations are calculated based solely on visual features. This
results in a decrease of 1.1 points in mAP (50.7), which indicates that
modeling spatial actor-context relations improves performance. By removing the
I3D tail, actor features are extracted from the output feature map of Mixed_4f
layer. This results in a significant decrease of more than four points in mAP
(47.42), highlighting the importance of using the I3D tail to encode actor
features.

Table 1: Validation mAP with respect to different number of layers, number of graphs per layer and merging functions to combine the output of multiple graphs. The number of model parameters are provided for every configuration. # Layers | # Graphs | Merging Function | # Parameters | Val. mAP
---|---|---|---|---
1 | 1 | - | 543K | 49.39
1 | 2 | Sum | 1.084M | 51.39
Concat | 1.086M | 51.3
1 | 3 | Sum | 1.624M | 50.7
Concat | 1.630M | 50.98
2 | 1 | - | 887K | 47.28
2 | 2 | Sum | 1.903M | 50.1
Concat | 1.906M | 51.82
2 | 3 | Sum | 3.050M | 49.98
Concat | 3.055M | 52.09

###  4.3 Comparison with Baseline and State of the Art

We compare the GCN model with the baseline model and the state-of-the-art [6,
31] on the DALY test set in Table 2.

The baseline model classifies actor features obtained from I3D (see Section
3.1) using a linear layer that outputs classification scores for $C$ action
classes and a background class. Results are shown in Table 2. Across five
repetitions, the GCN model outperforms the baseline model by 2.24 (3.7%)
points in mean mAP, and by 2.94 (4.9%) points in maximum mAP. The left-hand
side of Fig. 3 illustrates per-class average precision for the baseline and
GCN model. GCN performs comparably or better than the baseline model in all
classes except ”TakingPhotosOrVideos”. On the right-hand side of Fig. 3, we
visualize t-sne [14] actor feature embeddings, colored by the respective
action, for the GCN (top) and baseline model (bottom). The GCN model produces
tighter and more distinct clusters compared to the baseline model.

Comparing the GCN model with the state-of-the-art [6, 31], we obtain slightly
improved performance in comparison to [31], while Chesneau et al. [6] achieve
better performance by 1.69 points in mean mAP and 0.78 points in maximum mAP.
We argue that this is due to the following reasons. Firstly, our models are
trained using fewer videos, since we hold out a part of the training set as a
validation set for fine-tuning. Secondly, [6, 31] train their models on the
region proposals produced by the detector (see Section 3.1), while we train
our models on the data provided by [31], which are only the final detections
of the detector. Consequently, we train our models using fewer videos and
boxes compared to [6, 31]. It is worth noting that, in contrast to [6, 31],
our model does not employ expensive optical flow computation.

Table 2: Comparison of GCN model with the baseline model and state-of-the-art
on the test set. We report model architecture and input modalities (RGB,
Optical Flow).

Model | Architecture | Input | Test mAP
---|---|---|---
Weinzaepfel et al. [31] | Fast R-CNN (VGG-16) | RGB, OF | 61.12
Chesneau et al. [6] | Fast R-CNN (VGG-16) | RGB, OF | 63.51
Baseline (Ours) | I3D | RGB | 59.58 ($\pm$ 0.22) (59.79)
GCN (Ours) | I3D | RGB | 61.82 ($\pm$ 0.51) (62.73)

.

Figure 3: Per-class Video-AP on the test set across five repetitions of GCN
and baseline model, and t-SNE actor feature embeddings of GCN (top) and
baseline model (bottom).

###  4.4 Reducing Annotation to One Bounding Box

We examine model performance when minimal spatial supervision is used, i.e.
one bounding box per action instance. We label tubes based on spatial IoU with
the ground truth box of a randomly selected keyframe for each action instance.
A GCN model is then trained using the newly labeled tubes. Using only a single
keyframe to label tubes, we obtain a small decrease in mAP, from 61.82 to
61.07. On the other hand, when one keyframe is used during evaluation too,
performance decreases by 3.15 points in mAP. The reason for such a decrease is
that mAP is not adequately estimated using only one evaluation keyframe.

##  5 Analysis of Attention

Our GCN model aids explainability by visualizing the adjacency matrix in the
form of attention maps that highlight the learned context, even in a zero-shot
setting, i.e. for actions and objects unseen during training. Although similar
attention maps are presented in previous works [7, 23, 25] (albeit not in a
zero-shot setting), in this paper, we go one step further to quantitatively
evaluate the ability of the attention to highlight the relevant context. To
this end, we propose a metric based on recall of objects retrieved by
attention maps. In this Section, we present qualitative and quantitative
results of attention maps.

###  5.1 Evaluation of Attention Maps

The adjacency matrix contains the relation or attention values, indicating the
importance of every context node (spatiotemporal location of feature map) to
the actor node. By visualizing the adjacency matrix, we obtain an attention
map that highlights, for a given actor, the important context regions the
model pays most attention to. The map is interpolated to the original input
size and overlaid on the input clip. Our model is even able to generalize its
attention in a zero-shot setting i.e. for actions that the model has not been
trained to recognize. To achieve this, we train a GCN model by excluding two
action classes, and then visualize the attention maps for the excluded
classes.

####  5.1.1 Qualitative Evaluation

Fig. 4 illustrates examples of attention maps, where each one is the
combination of four adjacency matrices (2 GCN layers; 2 graphs per layer) by
summing their values along the spatial dimensions. Each example contains four
attention maps, representing time progression along the input clip. The last
row of Fig. 4 contains zero-shot attention maps for classes ”Ironing” and
”TakingPhotosOrVideos”. The attention maps show that our GCN model highlights
relevant context, such as objects, hands and faces, and is also able to track
objects along time. Finally, our model is able to highlight relevant objects
(e.g. Iron, Camera) for actions unseen during training (last row of Fig. 4).

Figure 4: Per-class object recall curves, along with visualizations of
attention maps (last row illustrates zero-shot cases) for actions “Drinking”,
“CleaningFloor”, “CleaningWindows”, “BrushingTeeth”, “FoldingTextile”,
“Ironing”, “TakingPhotosOrVideos”.

####  5.1.2 Quantitative Evaluation

We evaluate how well the attention maps highlight relevant objects by
introducing a metric based on recall of objects retrieved by the attention.
DALY provides object bounding box annotations on annotated frames. Given the
attention map produced for a detected actor, we sum the attention values
inside the object’s bounding box. An instance is a true positive if the sum of
values is larger than a threshold, and a false negative, otherwise.

A per-class quantitative evaluation of attention maps is shown in Fig. 4, with
recall on the $y$-axis and the attention threshold on the $x$-axis. Dashed
curves correspond to zero-shot cases. Our metric suggests that objects are
retrieved by the attention with relatively high recall, even for large
attention thresholds, which shows the effectiveness of our model to highlight
the relevant context.

##  6 Conclusion

We propose an approach using Graph Convolutional Networks [12] to model
contextual cues, such as actor-actor and actor-object interactions, to improve
action detection in video. On the challenging DALY dataset [31], our model
outperforms a baseline, which uses no context, by more than 2 points in Video-
mAP, performing on par or better in all action classes but one. The learned
adjacency matrix, visualized as an attention map, aids explainability by
highlighting the learned context, such as objects relevant for recognizing the
action, even in a zero-shot setting, i.e. for actions unseen during training.
We quantitatively evaluate the attention maps using our proposed metric based
on recall of objects retrieved by the attention. Results show the
effectiveness of our model to highlight the relevant objects with high recall.
All the above are achieved in a weakly-supervised setting using only up to
five or even one actor box annotation per action instance. Future work
includes end-to-end model training using weak supervision and modeling
relations between consecutive clips and videos of the same action.

#### Acknowledgements

We would like to thank Philippe Weinzaepfel for providing us with the
predicted action tubes of their tracking-by-detection model.

## References

  * [1] Andriluka, M., Pishchulin, L., Gehler, P., Schiele, B.: 2d human pose estimation: New benchmark and state of the art analysis. In: 2014 IEEE Conference on Computer Vision and Pattern Recognition. pp. 3686–3693 (2014)
  * [2] Ba, J., Kiros, J.R., Hinton, G.E.: Layer normalization. ArXiv abs/1607.06450 (2016)
  * [3] van Boven, B., van der Putten, P., Åström, A., Khalafi, H., Plaat, A.: Real-time excavation detection at construction sites using deep learning. In: Duivesteijn, W., Siebes, A., Ukkonen, A. (eds.) Advances in Intelligent Data Analysis XVII. pp. 340–352. Springer International Publishing, Cham (2018)
  * [4] Carreira, J., Zisserman, A.: Quo vadis, action recognition? a new model and the kinetics dataset. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition. pp. 4724–4733 (2017)
  * [5] Chéron, G., Alayrac, J.B., Laptev, I., Schmid, C.: A flexible model for training action localization with varying levels of supervision. In: Advances in Neural Information Processing Systems 31, pp. 942–953. Curran Associates, Inc. (2018)
  * [6] Chesneau, N., Rogez, G., Alahari, K., Schmid, C.: Detecting parts for action localization. ArXiv abs/1707.06005 (2017)
  * [7] Girdhar, R., João Carreira, J., Doersch, C., Zisserman, A.: Video action transformer network. In: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 244–253 (2019)
  * [8] Girshick, R.: Fast r-cnn. In: 2015 IEEE International Conference on Computer Vision. pp. 1440–1448 (2015)
  * [9] Gkioxari, G., Malik, J.: Finding action tubes. 2015 IEEE Conference on Computer Vision and Pattern Recognition pp. 759–768 (2015)
  * [10] He, K., Zhang, X., Ren, S., Sun, J.: Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In: 2015 IEEE International Conference on Computer Vision. pp. 1026–1034 (2015)
  * [11] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition. pp. 770–778 (2016)
  * [12] Kipf, T.N., Welling, M.: Semi-supervised classification with graph convolutional networks. In: 5th International Conference on Learning Representations. OpenReview.net (2017)
  * [13] Liu, W., et al.: Ssd: Single shot multibox detector. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) Computer Vision – ECCV 2016. pp. 21–37. Springer International Publishing, Cham (2016)
  * [14] van der Maaten, L., Hinton, G.: Visualizing data using t-SNE. Journal of Machine Learning Research 9, 2579–2605 (2008)
  * [15] Mettes, P., Snoek, C.G.: Pointly-supervised action localization. Int. J. Comput. Vision 127(3), 263–281 (2019)
  * [16] Paszke, A., et al.: Pytorch: An imperative style, high-performance deep learning library. In: Advances in Neural Information Processing Systems 32, pp. 8024–8035. Curran Associates, Inc. (2019)
  * [17] Qiu, Z., Yao, T., Mei, T.: Learning spatio-temporal representation with pseudo-3d residual networks. In: 2017 IEEE International Conference on Computer Vision (ICCV). pp. 5534–5542 (2017)
  * [18] Ren, S., He, K., Girshick, R., Sun, J.: Faster r-cnn: Towards real-time object detection with region proposal networks. IEEE Transactions on Pattern Analysis and Machine Intelligence 39(6), 1137–1149 (2017)
  * [19] Russakovsky, O., et al.: Imagenet large scale visual recognition challenge. Int. J. Comput. Vision 115(3), 211–252 (2015)
  * [20] Santoro, A., et al.: A simple neural network module for relational reasoning. In: Advances in Neural Information Processing Systems 30, pp. 4967–4976. Curran Associates, Inc. (2017)
  * [21] Simonyan, K., Zisserman, A.: Two-stream convolutional networks for action recognition in videos. In: Advances in Neural Information Processing Systems 27, pp. 568–576. Curran Associates, Inc. (2014)
  * [22] Siva, P., Xiang, T.: Weakly supervised action detection. In: Proceedings of the British Machine Vision Conference. BMVA Press (2011)
  * [23] Sun, C., Shrivastava, A., Vondrick, C., Murphy, K., Sukthankar, R., Schmid, C.: Actor-centric relation network. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) Computer Vision – ECCV 2018. pp. 335–351. Springer International Publishing, Cham (2018)
  * [24] Tran, D., Wang, H., Torresani, L., Ray, J., LeCun, Y., Paluri, M.: A closer look at spatiotemporal convolutions for action recognition. In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 6450–6459 (2018)
  * [25] Ulutan, O., Rallapalli, S., Srivatsa, M., Manjunath, B.S.: Actor conditioned attention maps for video action detection. 2020 IEEE Winter Conference on Applications of Computer Vision (WACV) pp. 516–525 (2020)
  * [26] Vaswani, A., et al.: Attention is all you need. In: Advances in Neural Information Processing Systems 30, pp. 5998–6008. Curran Associates, Inc. (2017)
  * [27] Velickovic, P., Cucurull, G., Casanova, A., Romero, A., Liò, P., Bengio, Y.: Graph attention networks. ArXiv abs/1710.10903 (2018)
  * [28] Wang, X., Girshick, R., Gupta, A., He, K.: Non-local neural networks. In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 7794–7803 (2018)
  * [29] Wang, X., Gupta, A.: Videos as space-time region graphs. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) Computer Vision – ECCV 2018. pp. 413–431. Springer International Publishing, Cham (2018)
  * [30] Weinzaepfel, P., Harchaoui, Z., Schmid, C.: Learning to track for spatio-temporal action localization. In: 2015 IEEE International Conference on Computer Vision. pp. 3164–3172 (2015)
  * [31] Weinzaepfel, P., Martin, X., Schmid, C.: Towards weakly-supervised action localization. ArXiv abs/1605.05197 (2016)
  * [32] Wu, J., Wang, L., Wang, L., Guo, J., Wu, G.: Learning actor relation graphs for group activity recognition. In: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 9956–9966 (2019)
  * [33] Zhang, Y., Tokmakov, P., Hebert, M., Schmid, C.: A structured model for action detection. In: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 9967–9976 (2019)


# A phenomenological note on the missing $\rho_{2}$ meson

Shahriyar Jafarzade

###### Abstract

The $\rho_{2}$ meson is the missing isovector member of the meson nonet with
the quantum numbers $J^{PC}=2^{--}$. It belongs to the class of $\rho$-mesons
such as the vector meson $\rho(770)$, the excited vector $\rho(1700)$ and the
tensor $\rho_{3}(1690)$. Yet, despite the rich experimental and theoretical
studies for other $\rho$-meson states, no resonance that could be assigned to
the $\rho_{2}$ meson has been measured. In this note, we present the results
for the mass and dominant decay channels of the $\rho_{2}$ meson within the
extended Linear Sigma Model.

##  1 Introduction

PDG contains various mesons denoted with the letter $\rho$ [1] . These are the
isovector resonances with quantum number of isospin ($I=1$), of parity
$(P=-1)$, and of charge conjugation ($C=-1$). For instance, the vector mesons
$\rho(770)$ with quantum number $J^{PC}=1^{--}$ [2], the excited vector mesons
$\rho(1450)$, $\rho(1700)$ [5, 4, 3], and the tensor meson $\rho_{3}(1690)$
with quantum number $J^{PC}=3^{--}$ [6]. Despite the prediction of the
$\rho_{2}$ in the Relativistic Quark model [7], it is still missing
experimentally. We only have the following data which were observed from
different experimental groups and listed as “further states” in PDG [1]:
$\rho_{2}(1940)$ and $\rho_{2}(2225)$ with the total decay widths
$\Gamma^{\text{tot}}_{\rho_{2}(1940)}\simeq 155\pm 40$ MeV and
$\Gamma^{\text{tot}}_{\rho_{2}(2225)}\simeq 335^{+100}_{-50}$ MeV accordingly.
Axial tensor mesons are studied in recent LQCD simulations [8], where the
authors consider the mass of $\rho_{2}$ is about $1.7$ GeV as the
$\rho_{3}(1690)$. We present the results about the missing $\rho_{2}$ [9]
within a chiral effective model which is so-called the extended Linear Sigma
Model (eLSM) [2].

##  2 Effective Model and Results

The physical resonances such as the pseudoscalar mesons
{$\pi,K,\eta,\eta^{\prime}(958)$}, the vector mesons {$\rho(770)$,
$\overline{K}^{\star}(892)$, $\omega(782)$, $\phi(1020)$}, and the tensor
mesons {$a_{2}(1320)$, $\overline{K}^{\star}_{2}(1430)$, $f_{2}(1270)$,
$f_{2}^{\prime}(1525)$} together with their chiral partners construct chiral
nonets. Table 1 describes the transformation of the chiral fields under
different symmetries for (pseudo) scalars (P) S, (axial-) vector
($A_{1}^{\mu}$) $V^{\mu}$ and (axial-) tensor ($A_{2}^{\mu\nu}$) $T^{\mu\nu}$
nonets.

Nonet | Parity $(P)$ | Charge conjugation $(C)$ | $U_{R}(3)\times U_{L}(3)$
---|---|---|---
$\Phi(t,\vec{x}):=S(t,\vec{x})+iP(t,\vec{x})$ | $\Phi^{\dagger}(t,-\vec{x})$ | $\Phi^{t}(t,\vec{x})$ | $U_{L}\Phi U^{\dagger}_{R}$
$R^{\mu}(t,\vec{x}):=V^{\mu}(t,\vec{x})-A_{1}^{\mu}(t,\vec{x})$ | $L_{\mu}(t,-\vec{x})$ | $-(L^{\mu}(t,\vec{x}))^{t}$ | $U_{R}R^{\mu}U_{R}^{\dagger}$
$L^{\mu}(t,\vec{x}):=V^{\mu}(t,\vec{x})+A_{1}^{\mu}(t,\vec{x})$ | $R_{\mu}(t,-\vec{x})$ | $-(R^{\mu}(t,\vec{x}))^{t}$ | $U_{L}L^{\mu}U_{L}^{\dagger}$
$\mathbf{R}^{\mu\nu}(t,\vec{x}):=T^{\mu\nu}(t,\vec{x})-A_{2}^{\mu\nu}(t,\vec{x})$ | $\mathbf{L}_{\mu\nu}(t,-\vec{x})$ | $(\mathbf{L}^{\mu\nu}(t,\vec{x}))^{t}$ | $U_{R}\mathbf{R}^{\mu\nu}U^{\dagger}_{R}$
$\mathbf{L}^{\mu\nu}(t,\vec{x}):=T^{\mu\nu}(t,\vec{x})+A_{2}^{\mu\nu}(t,\vec{x})$ | $\mathbf{R}_{\mu\nu}(t,-\vec{x})$ | $(\mathbf{R}^{\mu\nu}(t,\vec{x}))^{t}$ | $U_{L}\mathbf{L}^{\mu\nu}U^{\dagger}_{L}$
Table 1: Transformations of the chiral multiplets under P, C, and
$U_{R}(3)\times U_{L}(3)$.

The chiral invariant Lagrangian that generates the masses of the spin-2 mesons
reads

$\displaystyle\mathcal{L}_{\text{mass}}=\text{Tr}\Big{[}\Big{(}\frac{m_{\text{ten}}^{2}}{2}+\Delta^{\text{ten}}\Big{)}\Big{(}\mathbf{L}_{\mu\nu}^{2}+\mathbf{R}_{\mu\nu}^{2}\Big{)}+\mathbf{R}^{\mu\nu}\mathbf{R}_{\mu\nu}\Big{]}+2h_{3}^{\text{ten}}\text{Tr}\Big{[}\Phi\mathbf{R}^{\mu\nu}\Phi^{\dagger}\mathbf{L}_{\mu\nu}\Big{]}\text{,}$
(1)

where
$\Delta^{\text{ten}}=\text{diag}\\{0\,,0\,,\delta_{S}^{\text{ten}}=m_{K_{2}}^{2}-m_{\textbf{a}_{2}}^{2}\\}$.

The following three equations are coming from the extended version of the
above lagrangian and relate the masses of spin-2 chiral partners:

$\displaystyle
m_{\rho_{2}}^{2}=m_{a_{2}}^{2}-h_{3}^{\text{ten}}\phi_{N}^{2},\quad
m_{K_{2A}}^{2}=m_{K_{2}}^{2}-\sqrt{2}h_{3}^{\text{ten}}\phi_{N}\phi_{S},\quad
m_{\omega_{2,S}}^{2}$
$\displaystyle=m_{f_{2,S}}^{2}-2h_{3}^{\text{ten}}\phi_{S}^{2}\text{ ,}$ (2)

which leads to $m_{\rho_{2}}=1663$ MeV where we have assumed
$m_{K_{2A}}=m_{K_{2}(1820)}$. The same assumption in the last term of Eq (1)
implies $\Gamma(\rho_{2}\rightarrow a_{2}(1320))\pi\approx 88$ MeV which is
200 MeV in [10]. Our prediction for the mass of $\rho_{2}$ from the
spontaneous breaking of the chiral symmetry is near to the prediction in [7].

The simplest Lagrangian which describes tree level decays has the following
form

$\displaystyle\mathcal{L}=\frac{g_{2}^{\text{ten}}}{2}\Big{(}\text{Tr}\Big{[}\mathbf{L}_{\mu\nu}\\{L^{\mu},L^{\nu}\\}\Big{]}+\text{Tr}\Big{[}\mathbf{R}_{\mu\nu}\\{R^{\mu},R^{\nu}\\}\Big{]}\Big{)}\text{
.}$ (3)

We firstly present the results for $a_{2}(1320)$ with the quantum number
$J^{PC}=2^{++}$ based on the Lagrangian (3).

Decay process (in model) | eLSM [9] | PDG [1]
---|---|---
$\,\;a_{2}(1320)\longrightarrow\bar{K}\,K$ | $4.06\pm 0.14$ | $7.0^{+2.0}_{-1.5}$
$\,\;a_{2}(1320)\longrightarrow\pi\,\eta$ | $25.37\pm 0.87$ | $18.5\pm 3.0$
$\,\;a_{2}(1320)\longrightarrow\pi\,\eta^{\prime}(958)$ | $1.01\pm 0.03$ | $0.58\pm 0.10$
Table 2: Decay rates of the $a_{2}(1320)$ into the pseudoscalar mesons in MeV.

Secondly, we present the results in Table 3 for the missing $\rho_{2}$. Note
that, we have used the PDG data in Table 2 to obtain the coupling
$g_{2}^{\text{ten}}$ for presenting the results in the second row of the Table
3. We expect the dominant $\rho_{2}$ decay widths in the interval between the
second and the third rows of the following table which implies it is being
broad despite some uncertainties.

Decay process (in model) | eLSM | eLSM (LQCD) | LQCD [8]
---|---|---|---
$\,\;\rho_{2}(?)\longrightarrow\rho(770)\,\eta$ | $87$ | $30$ | $-$
$\,\;\rho_{2}(?)\longrightarrow\bar{K}^{\ast}(892)\,K+\textbf{c}.\mathrm{c}.$ | $77$ | $27$ | $36$
$\,\;\rho_{2}(?)\longrightarrow\omega(782)\,\pi$ | $376$ | $122$ | $125$
$\,\;\rho_{2}(?)\longrightarrow\phi(1020)\,\pi$ | $0.8$ | $0.3$ | $-$
Table 3: Decay rates of $\rho_{2}$ into the vector and the pseudoscalar mesons
in MeV.

We finally present the result for the well-established tensor meson
$\rho_{3}(1690)$ in Table 4. The decay channel of
$\Gamma(\rho_{3}(1690)\rightarrow\omega(782)\pi)$ which is measured
experimentally too, is 5-6 times smaller than
$\Gamma(\rho_{2}\rightarrow\omega(782)\pi)$ within LQCD simulations in spite
of having the same mass.

Decay process (in model) | PDG [1] | eLSM [6] | LQCD [8]
---|---|---|---
$\,\;\rho_{3}(1690)\longrightarrow\rho(770)\,\eta$ | $-$ | $3.8\pm 0.8$ | $-$
$\,\;\rho_{3}(1690)\longrightarrow\bar{K}^{\ast}(892)\,K+\textbf{c}.\mathrm{c}.$ | $-$ | $3.4\pm 0.7$ | $2$
$\,\;\rho_{3}(1690)\longrightarrow\omega(782)\,\pi$ | $25.8\pm 9.8$ | $35.8\pm 7.4$ | $22$
$\,\;\rho_{3}(1690)\longrightarrow\phi(1020)\,\pi$ | $-$ | $0.036\pm 0.007$ | $-$
Table 4: Decay rates of $\rho_{3}(1690)$ into the vector and the pseudoscalar
mesons in MeV.

##  3 Conclusion

We have studied $\rho_{2}$ axial-tensor meson, chiral partner of the tensor
meson $a_{2}(1320)$ in the framework of a chiral model for low-energy QCD. We
predict its mass to be around $1.663$ GeV similar to the Relativistic Quark
model prediction. Because of the chiral symmetry, the parameter determined in
the tensor sector allows to make predictions for unknown ground-state axial-
tensor resonance. The effective model fitting to the LQCD results is also
presented.

## Acknowledgement

I am thankful to Adrian Königstein for collaboration in [6] , Milena
Piotrowska, Arthur Verijeken in [9] and Francesco Giacosa for his reading the
manuscript as well as collaboration in [6, 9]. I acknowledge financial support
through the project “Development Accelerator of the Jan Kochanowski University
of Kielce”, co-financed by the European Union under the European Social Fund,
with no. POWR.03.05. 00-00-Z212 / 18 and support through the NCN OPUS no.
2018/29/B/ST2/02576.

## References

  * [1] R. L. Workman et al. [Particle Data Group], PTEP 2022 (2022), 083C01 doi:10.1093/ptep/ptac097
  * [2] D. Parganlija, P. Kovacs, G. Wolf, F. Giacosa and D. H. Rischke, Phys. Rev. D 87 (2013) no.1, 014011 doi:10.1103/PhysRevD.87.014011 [arXiv:1208.0585 [hep-ph]].
  * [3] M. Piotrowska and F. Giacosa, PoS Hadron2017 (2018), 237 doi:10.22323/1.310.0237 [arXiv:1712.05617 [hep-ph]].
  * [4] M. Piotrowska and F. Giacosa, EPJ Web Conf. 182 (2018), 02097 doi:10.1051/epjconf/201818202097 [arXiv:1712.01087 [hep-ph]].
  * [5] M. Piotrowska, C. Reisinger and F. Giacosa, Phys. Rev. D 96 (2017) no.5, 054033 doi:10.1103/PhysRevD.96.054033 [arXiv:1708.02593 [hep-ph]].
  * [6] S. Jafarzade, A. Koenigstein and F. Giacosa, Phys. Rev. D 103 (2021) no.9, 096027 doi:10.1103/PhysRevD.103.096027 [arXiv:2101.03195 [hep-ph]].
  * [7] S. Godfrey and N. Isgur, Phys. Rev. D 32 (1985), 189-231 doi:10.1103/PhysRevD.32.189
  * [8] C. T. Johnson et al. [Hadron Spectrum], Phys. Rev. D 103 (2021) no.7, 074502 doi:10.1103/PhysRevD.103.074502 [arXiv:2012.00518 [hep-lat]].
  * [9] S. Jafarzade, A. Vereijken, M. Piotrowska and F. Giacosa, Phys. Rev. D 106 (2022) no.3, 036008 doi:10.1103/PhysRevD.106.036008 [arXiv:2203.16585 [hep-ph]].
  * [10] D. Guo, C. Q. Pang, Z. W. Liu and X. Liu, Phys. Rev. D 99 (2019) no.5, 056001 doi:10.1103/PhysRevD.99.056001 [arXiv:1901.03518 [hep-ph]].