# Single-event likelihood of star cluster properties with LIGO-Virgo-Kagra
binary black hole observations

Ken K. Y. Ng<EMAIL_ADDRESS>Konstantinos Kritos  Andrea Antonelli  Roberto
Cotesta  Emanuele Berti  William H. Miller III Department of Physics and
Astronomy, Johns Hopkins University, Baltimore, Maryland 21218, USA

###### Abstract

The population of binary black hole mergers observed in gravitational waves,
together with astrophysical simulations, can help us to understand the
properties of the progenitors and the binary formation mechanisms in different
astrophysical scenarios. Here we focus on dynamical formation in star
clusters. We use the third gravitational-wave transient catalog (GWTC-3) and
Rapster, a rapid code to simulate cluster dynamics, to show that it is
possible to construct the single-event likelihood of star cluster properties
from individual observations. We find that the measured primary mass in a
binary black hole merger correlates with the measured star cluster mass,
because the mass spectrum of the primary component increases with the mass of
the cluster. This trend may be caused by two physical mechanisms: (i) the more
efficient production of hierarchical mergers with primary mass above $\sim
40~{}M_{\odot}$ for cluster masses of $\gtrsim 10^{6}~{}M_{\odot}$, and (ii)
the suppression of more massive first-generation binaries, which happens
because ejected binaries do not merge within the lookback time for cluster
masses of $\lesssim 10^{5}~{}M_{\odot}$. The formalism presented here can be
generalized to infer the population properties of binary progenitors in more
realistic scenarios involving multiple formation channels.

##  I Introduction

The recent catalogs of gravitational-wave (GW) transients released by the
LIGO-Virgo-Kagra Collaboration [1, 2, 3, 4, 5, 6] motivated efforts to
investigate the properties of the binary black hole (BBH) population and their
possible formation channels. There are various ways to address this problem.

One approach is to construct phenomenological models that reproduce the main
distinctive features of astrophysical formation channels [7, 8, 9]. This is a
sensible approach because it requires minimal astrophysical modeling.

For instance, certain broad features of the population – such as the BH spin
alignment with the orbital angular momentum [10, 11, 12, 13, 14, 15, 16, 17,
18] or the measurement of binary component masses populating the pair
instability supernova (PISN) mass gap, above $\sim 40M_{\odot}$ [19, 20, 21,
22, 23] – may provide evidence for multiple formation channels. This is
because isolated binary evolution is expected to produce mostly binaries with
aligned spins and masses below the mass gap [24, 25, 26, 27, 28, 29, 30, 31,
32, 33, 34, 35, 36, 37, 38], while the spins of BHs produced through dynamical
formation in star clusters should be isotropically oriented, and hierarchical
mergers can populate the PISN mass gap [39, 40, 41, 42, 43, 44, 45, 46, 47,
48, 44, 49, 50, 51, 52, 53]. Other features that can be captured by
phenomenological models include the time (or redshift) evolution of the merger
rate density [54, 55, 56, 57, 58, 59], or the presence of peaks and tails in
the redshift evolution of merger rate densities due to putative Population III
or primordial BBH components, which could be observable with next-generation
GW detectors [58, 60].

One drawback of phenomenological models is that they are affected by modeling
systematics: for example, the class of parametrized functions used to
reproduce the data may be too restrictive, leading to an erroneous mapping
between the parametrized models and detailed astrophysical simulations.

A second approach is to infer the empirical distribution using data-driven
models, leaving the interpretation of the resulting distribution to the
postprocessing stage [61, 62, 63, 64]. Even within this approach, finding a
suitable statistical metric connecting data with astrophysical simulations
could be problematic.

A third approach (and one that we follow in this paper) is to build a direct
mapping between the measured parameters of a BBH merger event and the
observables predicted by astrophysical simulations [65, 66, 67, 68, 69, 70,
71, 72]. While the inference is still limited by our incomplete knowledge of
astrophysical formation scenarios, this approach allows for in-depth studies
of the astrophysical mechanisms that correspond to certain features seen in
the populations. There have been many attempts to infer some of the key
astrophysical parameters affecting the isolated binary evolution scenario, as
well as the relative contribution (or branching ratios) of multiple formation
channels: see e.g. [73, 74, 75, 76, 77, 78] for an incomplete list.

In this paper we avoid the complications related to multiple formation
channels, and we focus on the dynamical formation scenario in dense star
clusters. Each cluster in the star cluster population has different
properties, and therefore it produces a different BBH subpopulation. Here we
develop a two-level hierarchical Bayesian framework that can ultimately infer
the properties of the star cluster population from BBH merger observations
(Sec. II). We focus on the first step in this framework, which consists in
constructing the single-event likelihood of star cluster properties: i.e., we
aim to identify clusters with parameters which are more likely to generate a
particular BBH observed in the third GW transient catalog (GWTC-3) [3, 79]. To
this end, we use a code for rapid simulations of cluster dynamics, Rapster
[44], to build a statistical mapping between the BBH parameters and the star
cluster parameters. By analyzing these simulations we observe a positive
correlation between the measured BBH primary mass and the inferred cluster
mass. As we discuss in Sec. III, this correlation may be explained by the
cluster mass scaling of the efficiency in the production of hierarchical
mergers and by the inspiral timescale of the ejected binaries. In Sec. IV we
discuss some technical aspects and future prospects to interpret the observed
BBH population using astrophysical simulations. In Appendix A we give details
on the cluster simulations and on the kernel density estimation (KDE) we use
to approximate the joint distribution from the simulated mergers.

##  II Two-level hierarchical Bayesian framework

As the Universe evolves, numerous star clusters form with redshift-dependent
rates and with different physical properties (such as total mass, radius, and
metallicity) [80]. Each cluster evolves dynamically and produces an ensemble
of BBHs whose statistical distribution depends on the properties of the host
cluster. Therefore, the distribution of BBH properties observed by LVK in the
cluster scenario should be modeled by considering the population of BBHs
originating from a population of star clusters. The “inverse problem” consists
of inferring the properties of the star cluster population that can host BBH
mergers observable in GW detectors.

One may attempt to perform the full hierarchical analysis by simulating BBHs
drawing from different realizations of cluster populations. However, it is
more beneficial to consider a two-level hierarchical model that can break down
the inference, as follows.

In the first level of hierarchy, we map the single-event likelihood of BBH
parameters to the likelihood of parameters of individual clusters that are
likely to produce them, using the BBH properties predicted by star cluster
simulations. In the second level of hierarchy, we combine these single-event
cluster likelihoods and infer the distribution of the cluster properties.

To see how the single-event cluster likelihood enters the hierarchical
framework, we derive it using a top-down approach. The full hierarchical
likelihood based on a Poisson process of data generation is given by [81, 82]

$\displaystyle p(\Lambda^{\rm cl}|\pmb{d})$ $\displaystyle\propto e^{-N^{\rm
B}_{\rm det}(\Lambda^{\rm cl})}\prod_{i=1}^{N}\int p(d_{i}|\theta^{\rm
B}_{i})\frac{dN^{\rm B}}{d\theta^{\rm B}}(\theta^{\rm B}_{i}|\Lambda^{\rm
cl})d\theta^{\rm B}_{i},$ (1)

where $\pmb{d}=\left\\{d_{i}\right\\}_{i=1}^{N}$ is the data set of $N$ BBH
observations, $p(d_{i}|\theta^{\rm B}_{i})$ is the individual likelihood of
the $i$-th BBH characterized by parameters $\theta^{\rm B}_{i}$ such as
component masses and spins, $dN^{\rm B}/d\theta^{\rm B}$ is the differential
number of BBHs expected for a given cluster population characterized by
hyperparameters $\Lambda^{\rm cl}$, and $N^{\rm B}_{\rm det}$ is the number of
detectable BBHs:

$\displaystyle N^{\rm B}_{\rm det}(\Lambda^{\rm cl})=\int\frac{dN^{\rm
B}}{d\theta^{\rm B}}(\theta^{\rm B}|\Lambda^{\rm cl})\epsilon_{\rm
det}(\theta^{\rm B})d\theta^{\rm B},$ (2)

where $0\leq\epsilon_{\rm det}(\theta^{\rm B})\leq 1$ is the detection
efficiency for a BBH merger with binary parameters $\theta^{\rm B}$.

The differential rate can be written as

$\displaystyle\indent\frac{dN^{\rm B}}{d\theta^{\rm B}}(\theta^{\rm
B}|\Lambda^{\rm cl})=\int\frac{d^{2}N^{\rm B}}{d\theta^{\rm B}d\lambda^{\rm
cl}}(\theta^{\rm B},\lambda^{\rm cl}|\Lambda^{\rm cl})d\lambda^{\rm cl}$
$\displaystyle=\int p(\theta^{\rm B}|\lambda^{\rm cl})\eta(\lambda^{\rm
cl})\frac{dN^{\rm cl}}{d\lambda^{\rm cl}}(\lambda^{\rm cl}|\Lambda^{\rm
cl})d\lambda^{\rm cl},$ (3)

where $p(\theta^{\rm B}|\lambda^{\rm cl})$ is the distribution of $\theta^{\rm
B}$ originating from a single cluster characterized by some $\lambda^{\rm
cl}$, $\eta(\lambda^{\rm cl})$ is the number of BBHs produced by the cluster,
and $N^{\rm cl}$ is the total number of clusters. For example, $\lambda^{\rm
cl}$ could be the mass of a single cluster, and $\Lambda^{\rm cl}$ the power
law index of the cluster mass function. Both $p(\theta^{\rm B}|\lambda^{\rm
cl})$ and $\eta(\lambda^{\rm cl})$ are predicted by the simulation, while
$p(d_{i}|\theta^{\rm B}_{i})$ is obtained from GW observations.

The integral in Eq. (1) is equivalent to the expected number of BBHs averaged
over the individual BBH likelihoods. It can be rewritten as

$\displaystyle\langle N^{\rm B}\rangle_{i}(\Lambda^{\rm cl})$ $\displaystyle=$
$\displaystyle\iint p(d_{i}|\theta^{\rm B}_{i})p(\theta^{\rm
B}_{i}|\lambda^{\rm cl}_{i})\eta(\lambda^{\rm cl}_{i})\frac{dN^{\rm
cl}}{d\lambda^{\rm cl}}(\lambda^{\rm cl}_{i}|\Lambda^{\rm cl})d\lambda^{\rm
cl}_{i}d\theta^{\rm B}_{i}$ $\displaystyle=$ $\displaystyle\int
p(d_{i}|\lambda^{\rm cl}_{i})\eta(\lambda^{\rm cl}_{i})\frac{dN^{\rm
cl}}{d\lambda^{\rm cl}}(\lambda^{\rm cl}_{i}|\Lambda^{\rm cl})d\lambda^{\rm
cl}_{i},$ (4)

where the individual cluster likelihood of the $i$-th observation,
$p(d_{i}|\lambda^{\rm cl}_{i})$, is the marginalization of $p(\theta^{\rm
B}_{i}|\lambda^{\rm cl}_{i})$ over the individual BBH likelihoods:

$\displaystyle p(d_{i}|\lambda^{\rm cl}_{i})\equiv\int p(d_{i}|\theta^{\rm
B}_{i})p(\theta^{\rm B}_{i}|\lambda^{\rm cl}_{i})d\theta^{\rm B}_{i}.$ (5)

This result may also be obtained by applying Bayes’ theorem and marginalizing
over $\theta^{\rm B}_{i}$ on the joint distribution $p(d_{i},\theta^{\rm
B}_{i}|\lambda^{\rm cl}_{i})$ “directly” in the bottom-up approach.

This procedure is practically advantageous as we only need to approximate
$p(\theta^{\rm B}|\lambda^{\rm cl})$ and $\eta(\lambda^{\rm cl})$ once with
the finite samples produced by the simulations. On the contrary, the emulation
of the entire BBH population $dN^{\rm B}/d\theta^{\rm B}$ originating from all
possible cluster populations is limited to the choices of the prior functions
used in the training set, and thus hinders the use of the more flexible
nonparametric models for the cluster population that we are interested in. In
the following, we will study solely the single-event cluster likelihood in Eq.
(5) for selected events in GWTC-3.

##  III Individual likelihood of cluster properties

The best-measured BBH parameters in current GW observations, and therefore the
parameters that are most informative in the inference of formation channels,
are the (source-frame) masses of the primary, $m_{1}$, and secondary, $m_{2}$;
the effective spin projected along the orbital angular momentum, $\chi_{\rm
eff}$; and the redshift, $z$.

The Rapster code has a total of 19 input parameters. While it is hopeless to
constrain all of these parameters, as a proof of principle we explore two of
the most important intrinsic properties of individual clusters: the total mass
of the cluster, $M_{\rm cl}$, and the half-mass radius at the time of cluster
formation, $r_{h}$. In other words, we set $\theta^{\rm
B}=(m_{1},m_{2},\chi_{\rm eff},z)$ and $\lambda^{\rm cl}=(M_{\rm cl},r_{h})$
in the formalism of Sec. II. We reweigh the LVK posterior samples and obtain
the likelihood samples of $\theta^{\rm B}$.

We limit the cluster parameter space to the ranges $M_{\rm
cl}\in[10^{4},~{}10^{7}]~{}M_{\odot}$ and $r_{h}\in[0.5,~{}3]~{}\rm pc$,
respectively, based on current observations of young star clusters [80]. The
simulation samples for constructing the KDE are generated by the following
settings. The initial cluster masses $M_{\rm cl}$ are drawn from a power-law
distribution with a spectral index $-2$ in the range
$[10^{3.7},\,10^{7.3}]M_{\odot}$. To avoid hard cutoffs in the range
$[10^{4},~{}10^{7}]M_{\odot}$ where we construct our KDE, we taper the
distribution using a Tukey window function with shape parameter 0.18. The
initial half-mass radius $r_{h}$ is drawn from a linear distribution in the
range $[0.3,\,3]$ pc. This choice is to balance the number of mergers per
cluster in the simulation set, which scales with the inverse of the cluster
radius. In the inference, we obtain the likelihood of $M_{\rm cl}$ and $r_{h}$
by reweighing the chosen priors. We note that the above initial conditions are
reweighed out eventually and that they do not affect the evaluation of the
likelihood, as shown in Eq. (5).

The other cluster parameters and the initial cluster conditions are fixed as
follows. We use SEVN to compute the initial mass function of BHs so that the
PISN cut-off is at $\sim 40M_{\odot}$, with the exact value depending on the
metallicity [83]. The dimensionless natal spin of first-generation BHs is
sampled from a uniform distribution in the range $[0,~{}0.2]$, as in Ref.
[84]. The masses and spins of BBH merger remnants are computed using the
precession code [85]. The initial central stellar density is calculated as
$3M_{\rm cl}/(4\pi(r_{h}/1.3)^{3})$, assuming a Plummer profile [86]. This
choice of mass-radius relation is motivated by observations of star clusters
in the local Universe [80]. A detailed cross-validation of the cluster
population resulting from this assumption with other surveys would be
interesting, but is beyond the scope of this paper.

Moreover, we assume a young massive cluster population with the redshift of
cluster formation and mean metallicity sampled from the Madau-Fragos
distribution [87]. We also apply a log-normal spread with a variance of 0.3 in
the metallicity distribution at each redshift. The rest of the input
parameters are set to their default Rapster values, as listed in Table I of
Ref. [44]. With one node (48 processors) at the Maryland Advanced Research
Computing Center at Johns Hopkins, we can simulate $\sim 10^{6}$ star clusters
within 2 days.

To approximate the conditional probability distribution $p(\theta^{\rm
B}|\lambda^{\rm cl})$, we employ Gaussian KDE on a set of $\sim 7\times
10^{5}$ simulated BBH mergers generated by the synthesis code Rapster. Given
$p(\theta^{\rm B}|\lambda^{\rm cl})$, we can evaluate Eq. (5) for each BBH
observation in GWTC-3 released by the LVK Collaboration. Since the integral is
generally intractable, we sample the likelihood in Eq. (5) by Monte Carlo
methods. Technical details on the KDE and on the integration are given in
Appendix A.

Figure 1: The joint likelihoods of $(M_{\rm cl},r_{h})$ for GW190521 (blue)
and GW151226 (orange). The marginalized $M_{\rm cl}$ likelihood for GW190521
(GW151226) favors values above (below) $M_{\rm cl}\sim 10^{6}~{}M_{\odot}$,
while the $r_{h}$ likelihoods are mostly flat.  Figure 2: The 90% credible
intervals of the likelihood of $m_{1}$ (horizontal bars) and $M_{\rm cl}$
(vertical bars) for all BBH events in GWTC-3, with markers indicating the
median values of $m_{1}$ and $M_{\rm cl}$. Six events (GW190521, GW191109,
GW190412, GW150914, GW151226, and GW190930) are highlighted in color to better
illustrate the correlation between $m_{1}$ and $M_{\rm cl}$.

In Fig. 1 we show the joint likelihoods of $(M_{\rm cl},r_{h})$ for two
events: GW190521 [88], and GW151226 [89]. These events were chosen because
they have very different primary masses: GW190521 has $m_{1}\sim
100M_{\odot}$, suggestive of a hierarchical merger origin [90], while GW151226
has $m_{1}\sim 14M_{\odot}$, a more typical value for events in the GWTC-3
catalog [9].

We find that the likelihood for $r_{h}$ is almost uninformative even for
GW190521. This is because, in the Rapster simulations [44], the compactness of
the cluster mostly affects the number of BBHs produced in the cluster, i.e.
$\eta$, rather than the shape of the BBH mass distribution. As $\eta$ is not
involved in the single-event cluster likelihood, we do not extract any new
information about $r_{h}$ from the single-event analysis. However, we note
that the current version of Rapster does not include stellar mergers, which
would allow for the possibility to form initial BHs within the PISN mass gap.
This mechanism may skew the distribution of $m_{1}$ to higher values for very
compact clusters [52].

The key feature of Fig. 1 is that the likelihood of $M_{\rm cl}$ in the two
systems is very different: GW190521 favors $M_{\rm cl}\gtrsim
10^{6}M_{\odot}$, and GW151226 favors $M_{\rm cl}\lesssim 10^{6}M_{\odot}$.
This may hint at a positive correlation between the primary BH mass $m_{1}$
and the probable cluster mass $M_{\rm cl}$ that produces the corresponding BBH
merger event.

To test this hypothesis, we have analyzed all BBH events in GWTC-3. The
results of this analysis are shown in Fig. 2, where we show the inferred value
of $M_{\rm cl}$ as a function of $m_{1}$ for the GWTC-3 catalog. Most events
are in grey, but a selected subset (listed in the legend) is highlighted in
color. The highlighted subset is chosen to cover three different ranges of
$m_{1}$: values in the PISN mass gap (GW190521 and GW191109, with
$m_{1}\gtrsim 40~{}M_{\odot}$), events with $40\gtrsim m_{1}\gtrsim
20M_{\odot}$ (GW150914 and GW190412), and low-primary mass events with
$m_{1}\lesssim 20M_{\odot}$ (GW151226 and GW190930). As anticipated, we
observe that the inferred values of $M_{\rm cl}$ (within the 90% credible
interval) tend to increase as a function of the measured values of $m_{1}$. We
have also checked that the correlation of $M_{\rm cl}$ with other binary
parameters (such as $\chi_{\rm eff}$ and $q$) is not as significant as the
correlation with $m_{1}$.

To understand this correlation, in Fig. 3 we compare the primary mass
distributions $p(m_{1})$ generated by clusters having masses $M_{\rm cl}$ in
different ranges (highlighted by histograms in different colors). These mass
distribution histograms have two major features.

First of all, the relative fraction of BBHs above the PISN mass gap is larger
when $M_{\rm cl}\geq 10^{5}M_{\odot}$ (i.e., for the orange and green
histograms). Hierarchical mergers within the mass gap occur more frequently in
more massive clusters, because these clusters have larger escape velocities
and thus they are more likely to retain the merger remnants despite their
gravitational recoils. This is compatible with the correlation between $M_{\rm
cl}$ and primary BHs having $m_{1}\gtrsim 40M_{\odot}$ observed in Fig. 2.

Secondly, the mass distribution of first-generation mergers below the PISN
mass gap (those with $m_{1}\lesssim 40M_{\odot}$) is skewed toward lower
values for $M_{\rm cl}\leq 10^{5}M_{\odot}$ (blue histogram): for these light
clusters, the peak in $m_{1}$ decreases from $\sim 35M_{\odot}$ to $\sim
15M_{\odot}$. This trend may be qualitatively explained by a combination of
the ejection mechanism discussed above, and the finite merging time window. In
a star cluster, first-generation mergers are typically formed by a combination
of mass segregation and exchange interactions. The majority of first-
generation mergers are nearly equal-mass systems, whose critical semimajor
axis for getting ejected out of the cluster after a binary-single interaction
scales with $a_{\rm ej}\propto m_{1}/M_{\rm cl}$: see e.g. Eq. (8) in Ref.
[44], or Eq. (8) in Ref. [91]. Therefore, in less massive clusters the more
massive BBHs are ejected at an earlier stage of their inspiral evolution.
Since the GW inspiral timescale $\tau\propto a_{\rm ej}^{4}/m_{1}^{3}$ [92],
the typical inspiral time for ejected mergers has the scaling $\tau\propto
m_{1}/M_{\rm cl}^{4}$. As the binaries can only merge within the (finite)
cosmic time since their formation, the critical $m_{1}$ below which BBHs can
merge scales with $m_{1}\propto M_{\rm cl}^{4}$. This leads to the observed
shift in the primary mass distribution as $M_{\rm cl}$ decreases. Note that
this is only a qualitative explanation, and the quantitative correlation
between $M_{\rm cl}$ and $m_{1}$ for first-generation BHs is very likely
model-dependent. The ejection efficiency and the resulting merging timescale
are sensitive to nonlinear effects in cluster dynamics, to the formation
redshift and to the cluster metallicity. All of these effects may modify the
shape of the primary mass distribution at different merger redshifts.

Figure 3: Histograms of the primary mass distributions generated by clusters
with masses in the range $M_{\rm cl}\in[10^{4},10^{5}]~{}M_{\odot}$ (blue),
$[10^{5},10^{6}]~{}M_{\odot}$ (orange), and $[10^{6},10^{7}]~{}M_{\odot}$
(green), using a prior $p(M_{\rm cl})\propto M_{\rm cl}^{-2}$. The peak
location of the $m_{1}$ distribution, which represents the most probable
first-generation mergers, shifts from $m_{1}\sim 15M_{\odot}$ to $m_{1}\sim
35M_{\odot}$ as $M_{\rm cl}$ increases from $10^{4}~{}M_{\odot}$ to
$10^{6}~{}M_{\odot}$.

##  IV Discussion

In this paper we propose a two-level hierarchical framework to analyze the BBH
population observed in GWs under the assumption that the merger events are
produced dynamically in star clusters. The two-level hierarchy is based on the
idea that each cluster in the cluster population has different physical
properties, and therefore produces a different BBH population. The implication
is that we can not only characterize the BBH population produced by an assumed
cluster population, but (vice versa) we can also infer the population
properties of the clusters, given a physical model of cluster dynamics.

To carry out this hierarchical inference, we first need to perform single-
event inference – that is, we need to identify the cluster properties that are
most likely to produce any observed BBH merger event. Estimating the single-
event likelihood of cluster properties requires a knowledge of how the
distribution of BBH parameters (such as binary masses, spins, and redshift)
depends on the individual cluster properties, such as the cluster mass and
radius. In this paper, we carried out a proof-of-principle demonstration that
this hierarchical inference is possible using, for illustrative purposes,
astrophysical models built with the Rapster code. With Rapster we can simulate
the formation of BBHs in star clusters, and then approximate the joint
distribution of BBH parameters and cluster parameters from the simulated BBH
samples by KDE methods.

We find that the inferred cluster mass is correlated with the measured BBH
primary mass, as shown in Fig. 2. The correlation is a result of the variation
of the primary mass distribution as a function of cluster mass observed in
Fig. 3: more massive clusters enhance the production of hierarchical mergers
above the PISN mass gap, and less massive clusters eject more massive first-
generation binaries at semimajor axes that are too large to efficiently
produce mergers. As emphasized in the main text, this is a mostly qualitative
explanation: the extent of the shift of the primary mass distribution in
first-generation BBHs is sensitive to the details of cluster dynamics and to
the initial conditions of cluster formation (including redshift and
metallicity).

For cluster radii $r_{h}\in[0.5,3]$ pc, the radius only affects the overall
number of BBHs produced in each cluster (because stellar mergers are not been
included in the model), and it has a negligible effect on the primary mass
distribution. In a more realistic scenario, the radius should play an
important role in the inference of cluster population properties, because the
number of BBHs produced per cluster $\eta$ entering the next hierarchy depends
on $r_{h}$, and affects the expected BBH merger rate. Our results are again
model-dependent, and therefore it would be useful to validate this trend by
comparing against other existing codes, especially for low-mass clusters with
$M_{\rm cl}\lesssim 10^{5}M_{\odot}$, whose evolution is more sensitive to the
details of every dynamical process [39, 44, 93, 47, 43]. Another source of
uncertainty is the initial binary fraction, which we assumed to be 10% in the
simulations. While differences in the initial binary star population have an
impact on the exchange channel in Rapster, the effect of this population on
the black hole mass-cluster mass correlation is probably more dependent on the
physics of binary star evolution. The study of this effect would require
complementing Rapster with input from a binary star evolution code to simulate
common envelope and stable mass transfer for the original binary stars. This
is an interesting problem, but it is beyond the scope of our work.

While the KDE method we employed suffices to capture the broad features
discussed above, it may not be robust enough to proceed to the next hierarchy
and infer the properties of the cluster population. Performing the full
hierarchical analysis may require more advanced techniques, such as deep
generative modeling, to better approximate the multidimensional probability
density functions involved in Eq. (5). For example, one may learn
$p(\theta^{\rm B}|\lambda^{\rm cl})$ and $\eta(\lambda^{\rm cl})$ separately
by simulating the BBH populations given a set of cluster properties
$\\{\lambda^{\rm cl}\\}$, or work with the joint distribution $p(\theta^{\rm
B},\lambda^{\rm cl}|\tilde{\Lambda}^{\rm cl})$ at a chosen cluster population
characterized directly by $\tilde{\Lambda}^{\rm cl}$, and obtain
$p(\theta^{\rm B}|\lambda^{\rm cl})\eta(\lambda^{\rm cl})\propto p(\theta^{\rm
B},\lambda^{\rm cl}|\tilde{\Lambda}^{\rm cl})/p(\lambda^{\rm
cl}|\tilde{\Lambda}^{\rm cl})$ by reweighing the chosen prior of the cluster
population.

Finally, we note that the two-level hierarchy may be generalized to include
contributions from multiple formation channels. This may be achieved by using
the relevant parametrization to set up Eq. (II) for each channel and build a
mixture model in Eq. (1). For example, one may obtain the likelihood of
progenitor redshift and metallicity based on binary evolution simulations for
galactic field binaries (either by backpropagation or though other numerical
techniques, as in Refs. [67, 68]); combine with the similar likelihood of
cluster binaries; and trace the evolution of star formation rate, cluster
formation rate, and stellar metallicity all at once. The full hierarchical
inference will be presented in future work.

###### Acknowledgements.

We thank Davide Gerosa, Will Farr, Chase Kimball, Sharan Banagiri, and Vicky
Kalogera for fruitful discussions. KKYN is supported by Miller Fellowship at
Johns Hopkins University and Croucher Fellowship by the Croucher Foundation.
K.K., A.A., R.C. and E.B. are supported by NSF Grants No. AST-2006538,
PHY-2207502, PHY-090003 and PHY-20043, and NASA Grants No. 20-LPS20- 0011 and
21-ATP21-0010. This research project was conducted using computational
resources at the Maryland Advanced Research Computing Center (MARCC). This
research has made use of data or software obtained from the Gravitational Wave
Open Science Center (gwosc.org), a service of LIGO Laboratory, the LIGO
Scientific Collaboration, the Virgo Collaboration, and KAGRA. LIGO Laboratory
and Advanced LIGO are funded by the United States National Science Foundation
(NSF) as well as the Science and Technology Facilities Council (STFC) of the
United Kingdom, the Max-Planck-Society (MPS), and the State of
Niedersachsen/Germany for support of the construction of Advanced LIGO and
construction and operation of the GEO600 detector. Additional support for
Advanced LIGO was provided by the Australian Research Council. Virgo is
funded, through the European Gravitational Observatory (EGO), by the French
Centre National de Recherche Scientifique (CNRS), the Italian Istituto
Nazionale di Fisica Nucleare (INFN) and the Dutch Nikhef, with contributions
by institutions from Belgium, Germany, Greece, Hungary, Ireland, Japan,
Monaco, Poland, Portugal, Spain. KAGRA is supported by Ministry of Education,
Culture, Sports, Science and Technology (MEXT), Japan Society for the
Promotion of Science (JSPS) in Japan; National Research Foundation (NRF) and
Ministry of Science and ICT (MSIT) in Korea; Academia Sinica (AS) and National
Science and Technology Council (NSTC) in Taiwan.

##  Appendix A Construction of KDE and importance sampling

Rapster enables rapid simulations of BBHs generated from a population of
clusters. Therefore, it is more convenient to (i) choose a cluster population
(parametrized by $\tilde{\Lambda}_{\rm cl}$) that produces enough simulated
BBHs in the range of the BBH parameter space $\theta^{\rm B}$ (our simulated
samples), (ii) perform kernel density estimation (KDE) to approximate the
joint distribution $\tilde{p}(\theta^{\rm B},\lambda^{\rm
cl}|\tilde{\Lambda}^{\rm cl})$ from the simulated mergers, and (iii) obtain
$p(\theta^{\rm B}|\lambda^{\rm cl})=\tilde{p}(\theta^{\rm B},\lambda^{\rm
cl}|\tilde{\Lambda}^{\rm cl})/\tilde{p}(\lambda^{\rm cl}|\tilde{\Lambda}^{\rm
cl})$ from Bayes’ theorem.

Here, $\tilde{p}(\lambda^{\rm cl}|\tilde{\Lambda}^{\rm cl})$ contains a factor
of the differential merger rate, $\eta(\lambda^{\rm cl})$, on top of the
chosen prior of the cluster population put into the simulations, since each
input cluster may produce a different number of mergers. To ensure that
$p(\theta^{\rm B}|\lambda^{\rm cl})$ is properly normalized to unity for each
$\lambda^{\rm cl}$, we require a second KDE for $\tilde{p}(\lambda^{\rm
cl}|\tilde{\Lambda}^{\rm cl})\propto\eta(\lambda^{\rm cl})p(\lambda^{\rm
cl}|\tilde{\Lambda}^{\rm cl})$, which can also be constructed from the
simulated samples, because the count of $\lambda^{\rm cl}$ is proportional to
the differential merger rate. We employ gaussian_kde from scipy, written in
the jax infrastructure, to speed up the KDE. We use $\sim 7\times 10^{5}$
simulation points, with a bandwidth of $\sim 0.25$ set by Silverman’s rule.

One may attempt to first approximate the integral of Eq. (5) by an importance
sum over the $\theta^{\rm B}$ samples of individual BBH likelihoods, and then
draw the $\lambda^{\rm cl}$ sample from the approximated $\lambda^{\rm cl}$
likelihood by Monte Carlo methods. Since the parameter space of $\lambda^{\rm
cl}$ has a relatively low dimension, we simplify the sampling procedures
further by importance sampling of the joint distribution $p(d_{i},\theta^{\rm
B}_{i}|\lambda^{\rm cl}_{i})\equiv p(d_{i}|\theta^{\rm B}_{i})p(\theta^{\rm
B}_{i}|\lambda^{\rm cl}_{i})$. In practice, we append an additional set of
$\lambda^{\rm cl}_{i}$ samples, $\\{\lambda^{\rm cl}_{i,j}\\}_{j=1}^{K}$,
drawn from a uniform distribution $U(\lambda^{\rm cl}_{i})\propto 1$ to the
set of BBH likelihood samples, $\\{\theta^{\rm B}_{i,j}\\}_{j=1}^{K}$. The
aggregated set $\\{\theta^{\rm B}_{i,j},\lambda^{\rm cl}_{i,j}\\}_{j=1}^{K}$
follows the joint distribution $p(d_{i}|\theta^{\rm B}_{i})U(\lambda^{\rm
cl}_{i})$. The desired set of $\lambda^{\rm cl}_{i}$ samples that are
representative of the marginalized likelihood $p(d_{i}|\lambda^{\rm cl}_{i})$
is equivalent to the set of $\\{\lambda^{\rm cl}_{i,j}\\}_{j=1}^{K}$ weighed
by $\\{w^{\rm cl}_{i,j}\propto p(\theta^{\rm B}_{i,j}|\lambda^{\rm
cl}_{i,j})\\}_{j=1}^{K}$.

## References

  * Abbott  _et al._ [2019a] B. Abbott _et al._ (LIGO Scientific, Virgo), Phys. Rev. X 9, 031040 (2019a), arXiv:1811.12907 [astro-ph.HE] .
  * Abbott  _et al._ [2021a] R. Abbott _et al._ (LIGO Scientific, Virgo), Phys. Rev. X 11, 021053 (2021a), arXiv:2010.14527 [gr-qc] .
  * Abbott  _et al._ [2021b] R. Abbott _et al._ (LIGO Scientific, VIRGO, KAGRA), GWTC-3: Compact Binary Coalescences Observed by LIGO and Virgo During the Second Part of the Third Observing Run (2021b), arXiv:2111.03606 [gr-qc] .
  * Aasi  _et al._ [2015] J. Aasi _et al._ (LIGO Scientific), Class. Quant. Grav. 32, 074001 (2015), arXiv:1411.4547 [gr-qc] .
  * Acernese  _et al._ [2015] F. Acernese _et al._ (VIRGO), Class. Quant. Grav. 32, 024001 (2015), arXiv:1408.3978 [gr-qc] .
  * Aso  _et al._ [2013] Y. Aso, Y. Michimura, K. Somiya, M. Ando, O. Miyakawa, T. Sekiguchi, D. Tatsumi, and H. Yamamoto (KAGRA), Phys. Rev. D 88, 043007 (2013), arXiv:1306.6747 [gr-qc] .
  * Abbott  _et al._ [2019b] B. Abbott _et al._ (LIGO Scientific, Virgo), Astrophys. J. Lett. 882, L24 (2019b), arXiv:1811.12940 [astro-ph.HE] .
  * Abbott  _et al._ [2021c] R. Abbott _et al._ (LIGO Scientific, Virgo), Astrophys. J. Lett. 913, L7 (2021c), arXiv:2010.14533 [astro-ph.HE] .
  * Abbott  _et al._ [2021d] R. Abbott _et al._ (LIGO Scientific, VIRGO, KAGRA), The population of merging compact binaries inferred using gravitational waves through GWTC-3 (2021d), arXiv:2111.03634 [astro-ph.HE] .
  * Gerosa  _et al._ [2013] D. Gerosa, M. Kesden, E. Berti, R. O’Shaughnessy, and U. Sperhake, Phys. Rev. D 87, 104028 (2013), arXiv:1302.4442 [gr-qc] .
  * Vitale  _et al._ [2017] S. Vitale, R. Lynch, R. Sturani, and P. Graff, Class. Quant. Grav. 34, 03LT01 (2017), arXiv:1503.04307 [gr-qc] .
  * Farr  _et al._ [2018] B. Farr, D. E. Holz, and W. M. Farr, Astrophys. J. Lett. 854, L9 (2018), arXiv:1709.07896 [astro-ph.HE] .
  * Farr  _et al._ [2017] W. M. Farr, S. Stevenson, M. Coleman Miller, I. Mandel, B. Farr, and A. Vecchio, Nature 548, 426 (2017), arXiv:1706.01385 [astro-ph.HE] .
  * Ng  _et al._ [2018] K. K. Y. Ng, S. Vitale, A. Zimmerman, K. Chatziioannou, D. Gerosa, and C.-J. Haster, Phys. Rev. D98, 083007 (2018), arXiv:1805.03046 [gr-qc] .
  * Talbot and Thrane [2017] C. Talbot and E. Thrane, Phys. Rev. D 96, 023012 (2017), arXiv:1704.08370 [astro-ph.HE] .
  * Wysocki  _et al._ [2018] D. Wysocki, D. Gerosa, R. O’Shaughnessy, K. Belczynski, W. Gladysz, E. Berti, M. Kesden, and D. E. Holz, Phys. Rev. D97, 043014 (2018), arXiv:1709.01943 [astro-ph.HE] .
  * Fishbach  _et al._ [2022] M. Fishbach, C. Kimball, and V. Kalogera, Astrophys. J. Lett. 935, L26 (2022), arXiv:2207.02924 [astro-ph.HE] .
  * Fishbach and Fragione [2023] M. Fishbach and G. Fragione, Globular cluster formation histories, masses and radii inferred from gravitational waves (2023), arXiv:2303.02263 [astro-ph.GA] .
  * Gerosa and Berti [2017] D. Gerosa and E. Berti, Phys. Rev. D 95, 124046 (2017), arXiv:1703.06223 [gr-qc] .
  * Fishbach  _et al._ [2017] M. Fishbach, D. E. Holz, and B. Farr, Astrophys. J. Lett. 840, L24 (2017), arXiv:1703.06869 [astro-ph.HE] .
  * Talbot and Thrane [2018] C. Talbot and E. Thrane, Astrophys. J. 856, 173 (2018), arXiv:1801.02699 [astro-ph.HE] .
  * Farah  _et al._ [2023] A. M. Farah, B. Edelman, M. Zevin, M. Fishbach, J. M. Ezquiaga, B. Farr, and D. E. Holz, Things that might go bump in the night: Assessing structure in the binary black hole mass spectrum (2023), arXiv:2301.00834 [astro-ph.HE] .
  * Kimball  _et al._ [2020] C. Kimball, C. Talbot, C. P. L.Berry, M. Carney, M. Zevin, E. Thrane, and V. Kalogera, Astrophys. J. 900, 177 (2020), arXiv:2005.00023 [astro-ph.HE] .
  * Belczynski  _et al._ [2008] K. Belczynski, V. Kalogera, F. A. Rasio, R. E. Taam, A. Zezas, T. Bulik, T. J. Maccarone, and N. Ivanova, Astrophys. J. Suppl. 174, 223 (2008), arXiv:astro-ph/0511811 .
  * Belczynski  _et al._ [2016a] K. Belczynski, D. E. Holz, T. Bulik, and R. O’Shaughnessy, Nature 534, 512 (2016a), arXiv:1602.04531 [astro-ph.HE] .
  * Belczynski  _et al._ [2020] K. Belczynski _et al._ , Astron. Astrophys. 636, A104 (2020), arXiv:1706.07053 [astro-ph.HE] .
  * Dominik  _et al._ [2012] M. Dominik, K. Belczynski, C. Fryer, D. Holz, E. Berti, T. Bulik, I. Mandel, and R. O’Shaughnessy, Astrophys. J. 759, 52 (2012), arXiv:1202.4901 [astro-ph.HE] .
  * Dominik  _et al._ [2013] M. Dominik, K. Belczynski, C. Fryer, D. E. Holz, E. Berti, T. Bulik, I. Mandel, and R. O’Shaughnessy, Astrophys. J. 779, 72 (2013), arXiv:1308.1546 [astro-ph.HE] .
  * Dominik  _et al._ [2015] M. Dominik, E. Berti, R. O’Shaughnessy, I. Mandel, K. Belczynski, C. Fryer, D. E. Holz, T. Bulik, and F. Pannarale, Astrophys. J. 806, 263 (2015), arXiv:1405.7016 [astro-ph.HE] .
  * Bavera  _et al._ [2020] S. S. Bavera, T. Fragos, Y. Qin, E. Zapartas, C. J. Neijssel, I. Mandel, A. Batta, S. M. Gaebel, C. Kimball, and S. Stevenson, Astron. Astrophys. 635, A97 (2020), arXiv:1906.12257 [astro-ph.HE] .
  * Breivik  _et al._ [2020] K. Breivik _et al._ , Astrophys. J. 898, 71 (2020), arXiv:1911.00903 [astro-ph.HE] .
  * Stevenson  _et al._ [2017] S. Stevenson, A. Vigna-Gómez, I. Mandel, J. W. Barrett, C. J. Neijssel, D. Perkins, and S. E. de Mink, Nature Commun. 8, 14906 (2017), arXiv:1704.01352 [astro-ph.HE] .
  * Broekgaarden  _et al._ [2019] F. S. Broekgaarden, S. Justham, S. E. de Mink, J. Gair, I. Mandel, S. Stevenson, J. W. Barrett, A. Vigna-Gómez, and C. J. Neijssel, Mon. Not. Roy. Astron. Soc. 490, 5228 (2019), arXiv:1905.00910 [astro-ph.HE] .
  * Baibhav  _et al._ [2019] V. Baibhav, E. Berti, D. Gerosa, M. Mapelli, N. Giacobbo, Y. Bouffanais, and U. N. Di Carlo, Phys. Rev. D100, 064060 (2019), arXiv:1906.04197 [gr-qc] .
  * de Mink and Belczynski [2015] S. de Mink and K. Belczynski, Astrophys. J. 814, 58 (2015), arXiv:1506.03573 [astro-ph.HE] .
  * Belczynski  _et al._ [2016b] K. Belczynski _et al._ , Astron. Astrophys. 594, A97 (2016b), arXiv:1607.03116 [astro-ph.HE] .
  * Woosley [2017] S. E. Woosley, Astrophys. J. 836, 244 (2017), arXiv:1608.08939 [astro-ph.HE] .
  * Riley  _et al._ [2022] J. Riley _et al._ (COMPAS Team, Team COMPAS), Astrophys. J. Supp. 258, 34 (2022), arXiv:2109.10352 [astro-ph.IM] .
  * Askar  _et al._ [2017] A. Askar, M. Szkudlarek, D. Gondek-Rosińska, M. Giersz, and T. Bulik, Mon. Not. Roy. Astron. Soc. 464, L36 (2017), arXiv:1608.02520 [astro-ph.HE] .
  * Rodriguez  _et al._ [2022] C. L. Rodriguez _et al._ , Astrophys. J. Supp. 258, 22 (2022), arXiv:2106.02643 [astro-ph.GA] .
  * Mapelli  _et al._ [2022] M. Mapelli, Y. Bouffanais, F. Santoliquido, M. A. Sedda, and M. C. Artale, Mon. Not. Roy. Astron. Soc. 511, 5797 (2022), arXiv:2109.06222 [astro-ph.HE] .
  * Antonini and Gieles [2020a] F. Antonini and M. Gieles, Mon. Not. Roy. Astron. Soc. 492, 2936 (2020a), arXiv:1906.11855 [astro-ph.HE] .
  * Sedda  _et al._ [2021] M. A. Sedda, M. Mapelli, M. Benacquista, and M. Spera, Population synthesis of black hole mergers with B-POP: the impact of dynamics, natal spins, and intermediate-mass black holes on the population of gravitational wave sources (2021), arXiv:2109.12119 [astro-ph.GA] .
  * Kritos  _et al._ [2022] K. Kritos, V. Strokov, V. Baibhav, and E. Berti, Rapster: a fast code for dynamical formation of black-hole binaries in dense star clusters (2022), arXiv:2210.10055 [astro-ph.HE] .
  * Rodriguez  _et al._ [2019] C. L. Rodriguez, M. Zevin, P. Amaro-Seoane, S. Chatterjee, K. Kremer, F. A. Rasio, and C. S. Ye, Phys. Rev. D100, 043027 (2019), arXiv:1906.10260 [astro-ph.HE] .
  * Rodriguez  _et al._ [2015] C. L. Rodriguez, M. Morscher, B. Pattabiraman, S. Chatterjee, C.-J. Haster, and F. A. Rasio, Phys. Rev. Lett. 115, 051101 (2015), [Erratum: Phys.Rev.Lett. 116, 029901 (2016)], arXiv:1505.00792 [astro-ph.HE] .
  * Rodriguez  _et al._ [2016] C. L. Rodriguez, S. Chatterjee, and F. A. Rasio, Phys. Rev. D 93, 084029 (2016), arXiv:1602.02444 [astro-ph.HE] .
  * Rodriguez  _et al._ [2018] C. L. Rodriguez, P. Amaro-Seoane, S. Chatterjee, and F. A. Rasio, Phys. Rev. Lett. 120, 151101 (2018), arXiv:1712.04937 [astro-ph.HE] .
  * Doctor  _et al._ [2019] Z. Doctor, D. Wysocki, R. O’Shaughnessy, D. E. Holz, and B. Farr, ArXiv e-print 10.3847/1538-4357/ab7fac (2019), arXiv:1911.04424 [astro-ph.HE] .
  * Di Carlo  _et al._ [2019] U. N. Di Carlo, N. Giacobbo, M. Mapelli, M. Pasquato, M. Spera, L. Wang, and F. Haardt, Mon. Not. Roy. Astron. Soc. 487, 2947 (2019), arXiv:1901.00863 [astro-ph.HE] .
  * Antonini and Gieles [2020b] F. Antonini and M. Gieles, Phys. Rev. D 102, 123016 (2020b), arXiv:2009.01861 [astro-ph.HE] .
  * Kremer  _et al._ [2020] K. Kremer, M. Spera, D. Becker, S. Chatterjee, U. N. Di Carlo, G. Fragione, C. L. Rodriguez, C. S. Ye, and F. A. Rasio, Astrophys. J. 903, 45 (2020), arXiv:2006.10771 [astro-ph.HE] .
  * Gerosa and Fishbach [2021] D. Gerosa and M. Fishbach, Nature Astron. 5, 749 (2021), arXiv:2105.03439 [astro-ph.HE] .
  * Fishbach  _et al._ [2018] M. Fishbach, D. E. Holz, and W. M. Farr, Astrophys. J. Lett. 863, L41 (2018), arXiv:1805.10270 [astro-ph.HE] .
  * Fishbach and Kalogera [2021] M. Fishbach and V. Kalogera, Astrophys. J. Lett. 914, L30 (2021), arXiv:2105.06491 [astro-ph.HE] .
  * Mapelli  _et al._ [2019] M. Mapelli, N. Giacobbo, F. Santoliquido, and M. C. Artale, Mon. Not. Roy. Astron. Soc. 487, 2 (2019), arXiv:1902.01419 [astro-ph.HE] .
  * Rodriguez and Loeb [2018] C. L. Rodriguez and A. Loeb, Astrophys. J. Lett. 866, L5 (2018), arXiv:1809.01152 [astro-ph.HE] .
  * Ng  _et al._ [2021] K. K. Y. Ng, S. Vitale, W. M. Farr, and C. L. Rodriguez, Astrophys. J. Lett. 913, L5 (2021), arXiv:2012.09876 [astro-ph.CO] .
  * Vitale  _et al._ [2019] S. Vitale, W. M. Farr, K. Ng, and C. L. Rodriguez, Astrophys. J. Lett. 886, L1 (2019), arXiv:1808.00901 [astro-ph.HE] .
  * Ng  _et al._ [2022] K. K. Y. Ng, G. Franciolini, E. Berti, P. Pani, A. Riotto, and S. Vitale, Astrophys. J. Lett. 933, L41 (2022), arXiv:2204.11864 [astro-ph.CO] .
  * Mandel  _et al._ [2017] I. Mandel, W. M. Farr, A. Colonna, S. Stevenson, P. Tiňo, and J. Veitch, Mon. Not. Roy. Astron. Soc. 465, 3254 (2017), arXiv:1608.08223 [astro-ph.HE] .
  * Rinaldi and Del Pozzo [2021] S. Rinaldi and W. Del Pozzo, Mon. Not. Roy. Astron. Soc. 509, 5454 (2021), arXiv:2109.05960 [astro-ph.IM] .
  * Edelman  _et al._ [2023] B. Edelman, B. Farr, and Z. Doctor, Astrophys. J. 946, 16 (2023), arXiv:2210.12834 [astro-ph.HE] .
  * Wong and Cranmer [2022] K. W. K. Wong and M. Cranmer, in  _39th International Conference on Machine Learning Conference_ (2022) arXiv:2207.12409 [astro-ph.IM] .
  * Zevin  _et al._ [2017] M. Zevin, C. Pankow, C. L. Rodriguez, L. Sampson, E. Chase, V. Kalogera, and F. A. Rasio, Astrophys. J. 846, 82 (2017), arXiv:1704.07379 [astro-ph.HE] .
  * Wong  _et al._ [2020a] K. W. Wong, G. Contardo, and S. Ho, Phys. Rev. D 101, 123005 (2020a), arXiv:2002.09491 [astro-ph.IM] .
  * Wong  _et al._ [2022] K. W. K. Wong, K. Breivik, W. M. Farr, and R. Luger, Backward Population Synthesis: Mapping the Evolutionary History of Gravitational-Wave Progenitors (2022), arXiv:2206.04062 [astro-ph.HE] .
  * Andrews  _et al._ [2018] J. J. Andrews, A. Zezas, and T. Fragos, Astrophys. J. Suppl. 237, 1 (2018), arXiv:1710.11030 [astro-ph.HE] .
  * Taylor and Gerosa [2018] S. R. Taylor and D. Gerosa, Phys. Rev. D98, 083017 (2018), arXiv:1806.08365 [astro-ph.HE] .
  * Mould  _et al._ [2022] M. Mould, D. Gerosa, and S. R. Taylor, Phys. Rev. D 106, 103013 (2022), arXiv:2203.03651 [astro-ph.HE] .
  * Mould  _et al._ [2023] M. Mould, D. Gerosa, M. Dall’Amico, and M. Mapelli, One to many: comparing single gravitational-wave events to astrophysical populations (2023), arXiv:2305.18539 [astro-ph.HE] .
  * Antonelli  _et al._ [2023] A. Antonelli, K. Kritos, K. K. Y. Ng, R. Cotesta, and E. Berti, Classifying the generation and formation channels of individual LIGO-Virgo-KAGRA observations from dynamically formed binaries (2023), arXiv:2306.11088 [gr-qc] .
  * Stevenson  _et al._ [2015] S. Stevenson, F. Ohme, and S. Fairhurst, Astrophys. J. 810, 58 (2015), arXiv:1504.07802 [astro-ph.HE] .
  * Bouffanais  _et al._ [2019] Y. Bouffanais, M. Mapelli, D. Gerosa, U. N. Di Carlo, N. Giacobbo, E. Berti, and V. Baibhav, Astrophys. J. 886, 10.3847/1538-4357/ab4a79 (2019), arXiv:1905.11054 [astro-ph.HE] .
  * Wong  _et al._ [2020b] K. W. Wong, K. Breivik, K. Kremer, and T. Callister, arXiv e-print (2020b), arXiv:2011.03564 [astro-ph.HE] .
  * Wong  _et al._ [2021] K. W. K. Wong, G. Franciolini, V. De Luca, V. Baibhav, E. Berti, P. Pani, and A. Riotto, Phys. Rev. D 103, 023026 (2021), arXiv:2011.01865 [gr-qc] .
  * Zevin  _et al._ [2020] M. Zevin, S. S. Bavera, C. P. Berry, V. Kalogera, T. Fragos, P. Marchant, C. L. Rodriguez, F. Antonini, D. E. Holz, and C. Pankow, arXiv e-print (2020), arXiv:2011.10057 [astro-ph.HE] .
  * Andrews  _et al._ [2021] J. J. Andrews, J. Cronin, V. Kalogera, C. P. L. Berry, and A. Zezas, Astrophys. J. Lett. 914, L32 (2021), arXiv:2011.13918 [astro-ph.HE] .
  * Abbott  _et al._ [2023] R. Abbott _et al._ (LIGO Scientific, VIRGO, KAGRA), Open data from the third observing run of LIGO, Virgo, KAGRA and GEO (2023), arXiv:2302.03676 [gr-qc] .
  * Krumholz  _et al._ [2019] M. R. Krumholz, C. F. McKee, and J. Bland-Hawthorn, Annual Reviews of Astronomy and Astrophysics 57, 227 (2019), arXiv:1812.01615 [astro-ph.GA] .
  * Mandel  _et al._ [2019] I. Mandel, W. M. Farr, and J. R. Gair, Mon. Not. Roy. Astron. Soc. 486, 1086 (2019), arXiv:1809.02063 [physics.data-an] .
  * Vitale  _et al._ [2020] S. Vitale, D. Gerosa, W. M. Farr, and S. R. Taylor, Inferring the properties of a population of compact binaries in presence of selection effects, in  _Handbook of Gravitational Wave Astronomy_, edited by C. Bambi, S. Katsanevas, and K. D. Kokkotas (Springer Singapore, Singapore, 2020) pp. 1–60.
  * Spera and Mapelli [2017] M. Spera and M. Mapelli, Mon. Not. Roy. Astron. Soc. 470, 4739 (2017), arXiv:1706.06109 [astro-ph.SR] .
  * Baibhav  _et al._ [2021] V. Baibhav, E. Berti, D. Gerosa, M. Mould, and K. W. K. Wong, Phys. Rev. D 104, 084002 (2021), arXiv:2105.12140 [gr-qc] .
  * Gerosa and Kesden [2016] D. Gerosa and M. Kesden, Phys. Rev. D 93, 124066 (2016), arXiv:1605.01067 [astro-ph.HE] .
  * Plummer [1911] H. C. Plummer, Mon. Not. Roy. Astron. Soc. 71, 460 (1911).
  * Madau and Fragos [2017] P. Madau and T. Fragos, Astrophys. J. 840, 39 (2017), arXiv:1606.07887 [astro-ph.GA] .
  * Abbott  _et al._ [2020a] R. Abbott _et al._ (LIGO Scientific, Virgo), Phys. Rev. Lett. 125, 101102 (2020a), arXiv:2009.01075 [gr-qc] .
  * Abbott  _et al._ [2016] B. P. Abbott _et al._ (LIGO Scientific, Virgo), Phys. Rev. Lett. 116, 241103 (2016), arXiv:1606.04855 [gr-qc] .
  * Abbott  _et al._ [2020b] R. Abbott _et al._ (LIGO Scientific, Virgo), Astrophys. J. Lett. 900, L13 (2020b), arXiv:2009.01190 [astro-ph.HE] .
  * Antonini and Rasio [2016] F. Antonini and F. A. Rasio, Astrophys. J. 831, 187 (2016), arXiv:1606.04889 [astro-ph.HE] .
  * Peters [1964] P. C. Peters, Phys. Rev. 136, B1224 (1964).
  * Joshi  _et al._ [2000] K. Joshi, F. Rasio, and S. F. Portegies Zwart, Astrophys. J. 540, 969 (2000), arXiv:astro-ph/9909115 .


11institutetext: University of British Columbia, Vancouver, BC, Canada
22institutetext: Adobe Research, San Francisco, CA, USA
22email: {lzxing<EMAIL_ADDRESS>
22email: {qtran, caba, dernonco, syoon, zhawang<EMAIL_ADDRESS>
# Multi-Modal Video Topic Segmentation with Dual-Contrastive Domain Adaptation

Linzi Xing Work done while the first author was an intern at Adobe Research.11
Quan Tran 22 Fabian Caba 22 Franck Dernoncourt 22 Seunghyun Yoon 22 Zhaowen
Wang 22 Trung Bui 22 Giuseppe Carenini 11

###### Abstract

Video topic segmentation unveils the coarse-grained semantic structure
underlying videos and is essential for other video understanding tasks. Given
the recent surge in multi-modal, relying solely on a single modality is
arguably insufficient. On the other hand, prior solutions for similar tasks
like video scene/shot segmentation cater to short videos with clear visual
shifts but falter for long videos with subtle changes, such as livestreams. In
this paper, we introduce a multi-modal video topic segmenter that utilizes
both video transcripts and frames, bolstered by a cross-modal attention
mechanism. Furthermore, we propose a dual-contrastive learning framework
adhering to the unsupervised domain adaptation paradigm, enhancing our model’s
adaptability to longer, more semantically complex videos. Experiments on short
and long video corpora demonstrate that our proposed solution, significantly
surpasses baseline methods in terms of both accuracy and transferability, in
both intra- and cross-domain settings.

###### Keywords:

Topic Segmentation Video Understanding NLP

##  1 Introduction

Video Topic Segmentation aims to break stretches of videos into smaller
segments consisting of video frames or clips consistently addressing a common
topic. As an example given in Figure 1, video topic segmentation does the job
of segmenting a creative livestream video (e.g., from livestream platform
Behance111https://www.behance.net/live) into a sequence of topical-coherent
pieces (Seg1 – Seg8), placing a boundary where a topic transition happens.
This task can enhances both human-to-human and human-to-system interactions in
modern social contexts, improving real-time engagement on live streaming
platform [9]. In particular, the relatively coarse-grained temporal structure
of the input video produced by video topic segmentation is shown to not only
simplify video comprehension and helps viewers find content of interest
easily. More importantly, it can substantially benefit other key video
understanding tasks such as video summarization [28], and query-driven video
localization [37].

Figure 1: A Behance exemplar about making a choking victim poster. The left
side of the figure illustrates the video’s timeline after topic segmentation.
The right side shows the transcript with words indicating a segment’s topic
highlighed.

Early computational models for video segmentation primarily targeted shot or
scene detection by merely leveraging surface visual features like
spatiotemporal aspects or frame colors [32, 31, 30, 6]. These approaches
typically measure the temporal similarity along a video’s timeline to predict
shot/scene boundaries. Despite the difference in definition from shot/scene
segmentation (discussed in §2), the task of video topic segmentation focuses
more on topic-related semantics in the video, which is not necessarily aligned
with visual changes. As shown in Figure 1, the visual background remains
similar in Seg3 – Seg7 for a considerable time, even though the streaming
topic changes drastically. Moreover, prior video segmentation methods mostly
focused on short videos with clear visual changes and simple patterns [13, 18,
16]. These distinctive features of short videos could be emphasized in model
design or learned in supervised setups, making such models less adaptable to
longer, more nuanced video content, like documentaries or instructional
livestreams.

To address the aforementioned issues, in this paper, we first propose a simple
yet effective multi-modal model for video topic segmentation, which can take
both the aligned video transcript and visual frames as input. This ability
considerably enhances the model’s performance, as textual and visual features
can work together to more comprehensively reflect the input video’s topic-
related semantics [17, 2]. Similar to the formulation used for text topic
segmentation [21, 39, 24], we treat video topic segmentation as a sequence
labeling task and introduce a neural model equipped with a cross-modal
attention mechanism going beyond simple fusion to effectively integrate
textual and visual signals in a complementary manner. We initially conduct
intra-domain experiments by training and testing our proposed segmenter on a
newly collected YouTube corpus equipped with high-quality human labeling.
Empirical results show that our multi-modal approach outperforms a set of
baseline video segmenters by a substantial margin.

To further adapt our proposed model trained on YouTube videos to longer videos
with complex visuals and semantics, we propose an unsupervised domain
adaptation strategy empowered by the sliding window inference and dual-
contrastive learning scheme. Further experiments on two out-of-domain long
video corpora demonstrate that the model’s generality can be significantly
improved through applying our dual-contrastive adaptation approach.

##  2 Related Work

#### Topic Segmentation

seeks to uncover the semantic structure of a document (either monologue [39]
or dialogue [38]) by dividing it, typically a sequence of sentences, into
topical-coherent segments. Recently, a number of supervised neural solutions
have been introduced owing to the availability of large-scale labeled corpora
sampled from Wikipedia [21], with section marks as gold segment boundaries.
Most of these neural segmentation approaches follow the same strategy to
simply interpret text segmentation as a sequence labeling problem and further
tackle it using a variety of hierarchical neural sequence labelers [21, 23,
39, 40].

Inspired by above-mentioned neural text segmenters, our paper similarly frame
video topic segmentation as a sequence labeling task due to the availability
of textual input (video transcript) and utilize a hierarchical neural sequence
labeling framework as the basic architecture of our proposal. Then we extend
such framework into the multi-modal setting, with the injection of visual
signals from video frames by adding a cross-modal attention network on top of
it. More details will be presented in §3.

Shot and Scene Segmentation are closely related to video topic segmentation
but more narrowly focused. A shot is a sequence of frames from a continuous
camera capture. Hence, most shot segmentation techniques mainly use the visual
modality to group video frames into shots. Conversely, a scene is more
semantically intricate than a shot, representing a series of related shots
that depict events defined by elements like actions, places, and characters.
These elements are mostly found in narrative videos such as movies. Thus, past
scene segmentation techniques are primarily developed for movies exclusively,
aiming to group consecutive shots into scenes based on their visual
consistency, spatiotemporal features, or shot color similarity. Similar but
more difficult than shot or scene segmentation, our work focuses on video
topic segmentation, which can be deemed as an extension of text topic
segmentation with "topic" defined as a relatively self-contained collection of
semantically close information. Notably, the topic’s definition relies heavily
on the video’s context and domain, and isn’t strictly bound to the concepts of
shots or scenes as previously defined. From a machine learning perspective,
the more dynamic nature of shot/scene segmentation favours the visual signal,
while in our task, the video topic segmentation relies more heavily on the
semantic signals carried by natural language [17, 2]. Therefore, our video
topic segmenter is designed to extend from a topic segmentation framework for
text, by integrating visual frames as the auxiliary signal.

Contrastive Learning algorithms aim to learn data representations by enlarging
the distance between dissimilar samples and meanwhile minimizing the distance
between similar samples with contrastive loss functions [27]. These techniques
have been observed promising in domain adaptation for both uni-modal and
cross-modal settings [19, 20, 5]. Recent works on using contrastive learning
for video domain adaptation mainly focus on transferring models from source to
target domain, where both domains consist of short videos [20]. In contrast,
our work attempts to adapt the model trained on short videos to the low-
resource domain containing long videos with subtler visual changes, by
utilizing the semantic overlaps within a single modality or between two
modalities to guide the contrastive learning process.

##  3 Neural Video Topic Segmentation

###  3.1 Problem Definition

Inspired by recent neural-based supervised approaches for text topic
segmentation for text, we similarly frame video topic segmentation as a
sequence labeling task, with sentences in the video transcript as the units
for labeling. More precisely, given an input video with (1) the transcript as
a sequence of timestamped sentences, and (2) a sequence of timestamped video
frames, our model will predict a binary label for each transcript sentence to
indicate whether or not the sentence indicates a topic segment boundary.
Formally,

Given: A video $v$ with its transcript $T_{v}$ as a sequence of sentences
$\\{s_{1},s_{2},...,s_{n}\\}$ along with start time offsets
$\\{b_{1},b_{2},...,b_{n}\\}$ / end time offsets
$\\{e_{1},e_{2},...,e_{n}\\}$, and video frames $X_{v}=\\{x_{1},x_{2},...$
$,x_{m}\\}$, as a single frame $x_{i}$ has timestamp $t_{i}$.

Predict: A sequence of labels $\\{l_{1},l_{2},...,l_{n-1}\\}$ for the sequence
of transcript sentences, where $l$ is a binary label, $1$ means the
corresponding sentence overlaps a video topic segment boundary, $0$ otherwise.
We do not predict the label for the last sentence $s_{n}$, as it is by
definition equal to 1, i.e., the end of the last segment.

Figure 2: Left: The overall architecture of our proposed multi-modal video
topic segmentation model, with four sub-modules (in §3.2) coded in different
colors. Right: The detailed illustration of cross-modal attention.

###  3.2 Model Architecture

Figure 2 illustrates the detailed framework of our proposed video topic
segmenter, which is similar in architecture to TextSeg [21]. It comprises two
hierarchically linked encoding layers: one as text encoder for contextualized
encoding within a sentence (orange in Figure 2) and the other for
contextualized encoding between sequence units (green in Figure 2). To allow
both textual and visual modalities contribute complementarily to the model’s
prediction, we add a frame encoder module (blue in Figure 2) and a cross-modal
attention mechanism (red in Figure 2). This design choice extends the
framework to the multi-modal setting, accepting now both the video transcript
and frames as input and making prediction of topic segment boundaries based on
them.

The text encoder module $E_{t}$ yields low-level features for sentences in the
video transcript. Different from the proposal in [21] using the BiLSTM +
attention as the backbone for text encoder, here we adopt the pre-trained
vanilla BERT [7] in accordance to its achieved superiority on text
segmentation observed in [39] and [26]. Parallel to the text encoder, a frame
encoder $E_{f}$ is introduced to extract features for visual signals with the
standard pretrained ResNet-18 [14]. Formally, given a transcript sentence
$s_{i}$ with its time interval as $[b_{i},e_{i}]$ and a set of video frames
$X_{i}=\\{x^{i}_{1},...,x^{i}_{m}\\}$ associated with this sentence, where
each frame (e.g., the $k^{th}$ frame) in the set has timestamp
$t^{i}_{k}\in[b_{i},e_{i}]$, we can obtain the sentence representation
$tr_{i}=E_{t}(s_{i})$ and its corresponding set of frame representations
FR${}_{i}=\\{fr^{i}_{1},...,fr^{i}_{m}\\}$, where
$fr^{i}_{k}=E_{f}(x^{i}_{k})$.

Next, we propose to use a cross-modal attention mechanism to produce a text-
aware visual representation for each sentence, rather than obtaining the
visual representation by naively operating mean-pooling over the set of frame
representations covered by the sentence interval. This design is motivated by
the observation that transcript sentences and video frames are in an one-to-
many relation, and frames sharing more semantics with the text should be given
more attention weights for visual representation generation. Irrelevant frames
sharing no or little semantics with the text may negatively affect the quality
of the fused multi-modal representation passed to the subsequent module [36].
In practice, the cross-modal attention module adopts the standard scaled dot-
product attention function proposed in [35]. With the transcript sentence
representation $tr_{i}$ and its corresponding frame representation set
FR${}_{i}=\\{fr^{i}_{1},...,fr^{i}_{m}\\}$ as query and key (value), we
compute the text-aware visual representation ${vr}_{i}$ as:

${vr}_{i}=A_{i}V_{i},$ (1)
$A_{i}=softmax(\frac{q_{i}K_{i}^{T}}{\sqrt{d_{k}}})$ (2)

where $q_{i}\in\mathbb{R}^{1\times d_{k}}$, $K_{i}\in\mathbb{R}^{m\times
d_{k}}$, and $V_{i}\in\mathbb{R}^{m\times d_{k}}$ denote the query vector, key
and value matrices generated by passing the sentence representation and frame
representations through three parallel feedfoward layers, namely Q-FFN, K-FFN
and V-FFN respectively. More formally, we have $q_{i}=$ Q-FFN$(tr_{i})$,
$K_{i}=$ K-FFN$($FR${}_{i})$, $V_{i}=$ V-FFN$($FR${}_{i})$, where K-FFN and
V-FNN share the same parameters and thus produce identical key and value
matrices following [25, 22].

Then all the obtained text-aware visual representations
$\\{vr_{1},...,vr_{n}\\}$ are concatenated with their corresponding sentence
representations $\\{tr_{1},...,tr_{n}\\}$ and fed into a BiLSTM layer which
performs contextualization and returns hidden states. Next, a multilayer
perceptron (MLP) followed by Softmax serves as a topic boundary predictor to
make binary predictions regarding the input hidden states according to a
threshold $\tau$ tuned on the validation set. More specifically, if a
transcript sentence’s output probability exceeds $\tau$, it’s marked as $1$,
indicating a segment boundary. The entire model is fine-tuned using cross-
entropy loss.

We train and test this model on a newly collected YouTube corpus (in §5.1) and
empirically verify its in-domain effectiveness (reported in Table 2). We
leverage it as the source model in the next section to help deliver the domain
adaptation strategy coupled with our proposed multi-modal framework.

##  4 Long Video Adaptation

The rise of (live-)streaming platforms has increased the demand for topical
segmentation of videos on these platforms. Unlike videos (e.g., on YouTube)
with careful pre-editing and segment labeling provided by their creators,
videos on (live-)streaming platforms (e.g., BBC documentaries and creative
livestreams on Behance) are usually extremely long, with sparse visual
changes, and more importantly, time-consuming to obtain segment annotations.
Thus, it is impractical to learn a fully-supervised model on such videos, and
the segmentation model described in §3.2 (source model) trained on short
YouTube videos (source domain) might underperform when applied to these
extensively long videos (target domain). To adapt the source model for the
above-described target domain, we propose to equip it with two strategies,
namely Sliding Window Inference (§4.1) and Dual-Contrastive Adaptation (§4.2).

###  4.1 Simple Sliding Window Inference

Due to the length discrepancy between videos from the source and target
domains, directly applying the source model to the target input taking the
full transcript sentence sequence as input is observed to cause extremely
sparse boundary predictions [12]. Therefore, we propose to calibrate the input
length by first breaking the long target input into snippets by a sliding
window with the size consistent with the source input length (i.e., the
average length of the YouTube videos). After applying the source model on
every snippet and aggregating outputs of all snippets, we eventually make
binary prediction for each sentence if its aggregated probability exceeds the
threshold $\tau$ pre-tuned on the source domain. Formally, given a long video
input from the target domain with length $=n$ and a fixed window size $=k$
($n\gg k$), we can create $n-k+1$ snippets
$\\{\mathbb{S}_{1},...,\mathbb{S}_{n-k+1}\\}$ where each snippet consists of
$k$ consecutive sentences by sliding the window over the input with the stride
of 1. As a result, each transcript sentence $s_{m}$ can be covered by up to
$k$ snippets. Once we apply the source model to all snippets, we can obtain
multiple probability predictions for $s_{m}$. We then aggregate these
probability predictions associated with $s_{m}$ by taking average:

$\bar{p}_{m}=\frac{1}{k}\sum_{i=1}^{k}p_{m}^{\mathbb{S}_{i}}$ (3)

Finally, we predict that $s_{m}$ falls on a segment boundary if
$\bar{p}_{m}>\tau$.

###  4.2 Dual-Contrastive Adaptation

To transfer the pre-trained source model to the target domain while still
preserving its performance on the source domain, we follow a more
sophisticated unsupervised domain adaptation paradigm, which leverages both
labeled source data and unlabeled target data. Specifically, we fix the frame
and text encoder ($E_{t}$ and $E_{f}$) while updating the rest of the model in
two steps, where the first step updates the model on unlabeled target data
using two contrastive learning objectives namely intra-modal contrastive loss
and cross-modal contrastive loss, while the second step updates the model on
source data with the supervised training scheme described in §3.2.

As the overview of the first step shown in Figure 3, given a collection of the
paired sentence representations and their corresponding frame representation
sets $\\{tr_{i},$FR${}_{i}\\}_{i=1}^{b}$ in a training batch (size=$b$)
produced by the text and frame encoder from the target domain, we need two
distinct projection heads to map these sentence/frame representations into a
shared space. Here we use Q-FFN and K-FNN (which shares parameters with V-FNN)
in the cross-modal attention to serve as projection heads for textual and
visual modality respectively. Thus we have:

$q_{i}=\textit{Q-FFN(}tr_{i}\textit{)},K_{i}=\textit{K-FFN(}FR_{i}\textit{)}$
(4)

where $q_{i}$ and $K_{i}=\\{k^{i}_{1},...,k^{i}_{m}\\}$ denote the projected
sentence embedding and the set of projected frame embeddings covered by the
sentence. As frames covered by the same sentence are more likely to share
similar semantics, we pull the frames attached to the same sentence closer and
push the ones from different sentences far apart, by minimizing the intra-
modal contrastive loss for visual modality defined as:

$l^{intra}=-\sum_{i=1}^{b}log\frac{exp(\tilde{k}^{i}\cdot\tilde{k}^{i})/\tau}{\sum_{j=1}^{b}exp(\tilde{k}^{i}\cdot\tilde{k}^{j})/\tau}$
(5)

where $\tilde{k}^{i}$$\in_{R}$ FRi and $\tau$ is the hyper-parameter of
temperature. To learn the semantic relation between modalities in the target
domain, we first obtain the visual representation matched with a sentence by
averaging all frames covered by the sentence, and then bring semantically
close sentence-visual pairs together and push away non-related pairs by
minimizing the cross-modal contrastive loss:

$l^{cross}=-\sum_{i=1}^{b}log\frac{exp(q_{i}\cdot\textit{MP}(K_{i}))/\tau}{\sum_{j=1}^{b}exp(q_{i}\cdot\textit{MP}(K_{j}))/\tau}$
(6)

where MP denotes Mean-Pooling. The total dual-contrastive loss is formed as:

$l=l^{intra}+l^{cross}$ (7)

The above dual-contrastive learning phase is followed by the second step. In
this step, the model is further trained on the labeled source data again to
leverage the target-domain signals learned by the parameters in cross-modal
attention for boundary prediction. This step ensures the model adapted to the
target domain still preserves some level of effectiveness when applied to the
source domain.

Figure 3: An illustration of updating cross-modal attention module with dual-
contrastive adaptation.

Dataset | YouTube | BBC | Behance
---|---|---|---
# of Vids. | 5,422 | 11 | 575
# of Tokens/Vid. | 1,411 | 2,868 | 11,554
# of Sents./Vid. | 108 | 216 | 1,287
# of Segs./Vid. | 6.7 | 29.1 | 5.2
# of Tokens/Seg. | 209 | 99 | 2,229
# of Sents./Seg. | 16 | 8 | 248
# of Frames/Sec. | 10 | 4 | 4
Avg. Length | 0:09:30 | 0:45:02 | 2:07:51

Table 1: Statistics of corpora used for training and evaluation in §5.

##  5 Experimental Setup

To evaluate the effectiveness and generality of our proposed video topic
segmentation model (in §3) and long video adaptation strategy (in §4), we
conduct experiments on two different settings, namely Intra-Domain Segment
Inference and Cross-Domain Segment Inference.

###  5.1 Intra-Domain Dataset – YouTube

For intra-domain segment inference, we train and test models with the data
from the same domain (corpus). Due to the lack of large-scale human-annotated
dataset in the field of video segmentation, we collect a novel corpus
consisting of 5,422 user-generated videos randomly sampled from YouTube.
During the video collecting process, we applied filtering criteria to
eliminate unsuitable samples to ensure the quality of the dataset to
construct. These criteria include constraints on video length ($>$ 100s), word
count ($>$ 0.5 word/second on average), chapter durations (all chapters with
length $>$ 10s), and sentence length (all sentences with length $<$ 60 tokens)
inspired by [4]. Each video in this corpus is associated with a series of
user-defined chapter timestamps indicating the beginning of each topic chapter
contained in this video. We thereby use these available chapter beginning
timestamps as ground-truth topic segment boundaries since a topic chapter is
by definition a main thematic division within a video. The video transcripts
are generated by YouTube ASR with token-level offsets and we further exploit a
top performing punctuation restoration model [1] to boost the quality of
transcript sentence segmentation. Compared with other existing video
understanding corpora such as BBC [2] (for scene segmentation, size = 11) and
TVSum [33] (for video summarization, size = 50), our constructed YouTube
corpus is (1) larger in size; (2) covering more diverse topics; (3) with
reliable segmentation which has already been specified by video creators. We
split this corpus into train/dev/test portions with size: 5148/134/140.

###  5.2 Datasets for Cross-Domain Inference

To evaluate our proposal’s robustness in cases where a domain-shift is
present, we conduct experiments for cross-domain segment inference, in which
our proposed supervised segmenter is initially trained on the YouTube video
corpus, and tested on two corpora with videos significantly longer than
YouTube videos:

BBC Planet Earth [2] consists of 11 episodes from the BBC educational TV
series Planet Earth. Topic segment boundaries of the dataset have been
manually annotated by human experts and transcript sentences are obtained
using Whisper222https://openai.com/research/whisper [29]. As the statistics in
Table 1 indicates, this dataset has longer videos with more segments than
YouTube videos, while each segment covers much fewer sentences and tokens. We
split this dataset into 5 and 6 for validation and testing.

Behance Livestream consists of 575 videos sampled from the creative livestream
platform Behance. Livestreams, in general, are very challenging to segment
into coherent sections, since they contain mixes of multi-user dialogues, with
visual features that change little over the video. They are also very long,
with hours of content in each video. Behance livestreams, in particular,
contain another layer of challenge, since they are highly specialized in
creative tasks such as animation and image edit with unknown entities and
intricate visual operations. We believe finding structures in such noisy
environment would be not only a good benchmark for future topic segmentation
research, but also of great practical value for consumers, who would find it
hard to navigate contents in a back-and-forth conversation. Thus, we collect
such a corpus and annotate its videos with topic segments given the help from
domain experts in animation and image edits. For each video, we pay $23.16 for
human annotators to watch and create segments based on the video content. The
total cost to annotate 575 videos is $13,317. We split this dataset into
validation and test sets with 57 (10%), 518 (90%).

###  5.3 Baselines

We consider the following representative baselines as comparisons:

\- Random: Given a video with its transcript consisting of $k$ sentences, we
first randomly sample the segment number $b\in\\{0,...,k-1\\}$ for this video,
and then determine if a sentence overlaps a segment boundary with probability
$\frac{b}{k}$.

\- BayesSeg [8] originally proposed for text segmentation by predicting
segment boundaries through modeling the lexical cohesion in a Bayesian
context.

\- GraphSeg [11] generates a semantic relatedness graph with sentences as
nodes. Segments are then predicted as the maximal cliques in graph.

\- Cross-BERT [26] is a supervised text topic segmenter representing candidate
segment boundaries using their left and right contexts encoded with BERT. The
model is trained on YouTube and applied to longer BBC and Behance videos.

\- TransNet [34]: This model adopts stacked Dilated DCNN blocks as its basic
framework and targets video segmentation on shot level. Here we use its
publicly available version trained on synthesized video corpora off-the-shelf.

\- LGSS [30] is a multi-modal movie scene segmenter pre-trained on the
limited-scale MovieScenes corpus. We use its public version with visual frames
as input.

\- X-Tiling is our extension of TextTiling [15]. While TextTiling only allows
textual embeddings as input, X-Tiling accepts both textual and visual
embeddings and their concatenations as input to compute semantic coherence and
then make segment boundary predictions for videos. For fair comparison, the
input textual and visual embeddings are produced by pre-trained BERT and mean-
pooling over ResNet-18 respectively.

All hyper-parameters required by the above baselines are tuned on the
validation portion of the datasets included in this paper.

###  5.4 Evaluation Metrics

We apply three standard metrics in previous literature to evaluate the
performances of our proposal and baselines. They are:

- $\boldsymbol{F_{1}}$, with higher scores denoting better performance. It measures the exact match between ground truth and model’s prediction.

\- $\boldsymbol{P_{r}}$ error score [10], which fixes the inadequacies of
$P_{k}$ [3] and WindowDiff [pevzner-hearst-2002-critique], previously
considered as two standard evaluation metrics for text segmentation.
Concretely, $P_{r}$ is the mean of missing and false alarm probabilities,
calculated based on the overlap between ground-truth segments and model’s
predictions within a certain size sliding window. Since it is a penalty
metric, lower score indicates better performance.

\- mIoU score [41], shortened from the mean Intersection-over-Union. It is
calculated by taking average over maximal IoUs of all ground-truth segments to
predicted segments. Higher score indicates better performance.

###  5.5 Implementation Details

In our multi-modal video topic segmenter, we employ the [CLS] token
representation from bert-base-uncased (dimension $d=768$) for sentence
representation and ResNet-18’s avg-pooling layer output (dimension $d=512$)
for frame representation. The cross-modal attention’s feedforward layers have
an output dimension of 768. Our BiLSTM has 2 layers with a hidden size of 256.
For YouTube training, we use the Adam optimizer with a learning rate of
$1e^{-3}$ and a batch size of 16. In the long video adaptation with dual-
contrastive learning, the SGD optimizer is applied with a mini-batch size of
256, learning rate of $3e^{-2}$, and softmax temperature of $1e^{-1}$.
Training spans 10 epochs for both supervised learning and domain adaptation,
with results averaged over 3 runs. The segmentation threshold, $\tau$, is
tuned on validation sets for both intra-domain and cross-domain evaluations.

Method | Modalities | Pr $\downarrow$ | F1 $\uparrow$ | mIoU $\uparrow$
---|---|---|---|---
Random | – | 45.84 | 48.98 | 37.76
X-Tiling | Text | 39.94 | 51.56 | 50.97
BayesSeg | Text | 40.70 | 50.24 | 49.69
GraphSeg | Text | 38.12 | 51.41 | 51.73
Cross-BERT∗ | Text | 32.89 | 60.48 | 60.00
X-Tiling | Visual | 37.53 | 52.08 | 53.24
TransNet | Visual | 40.14 | 51.01 | 50.66
LGSS | Visual | 39.77 | 50.87 | 51.13
X-Tiling | Text + Visual | 38.78 | 52.30 | 51.45
NeuralSeg∗ (Ours) | Text | 31.91 | 63.23 | 61.36
Visual | 50.18 | 47.59 | 16.15
Text + Visual | 30.61 | 65.29 | 63.11

Table 2:  Results on YouTube for intra-domain evaluation. Bold results
indicate the best performance across all comparisons. Underlined results
indicate the best performance within their own sub-section. * indicates a
fully supervised setting.

##  6 Results and Discussion

Intra-Domain Segment Inference: Table 2 reports the performance of the chosen
baselines and our proposal (NeuralSeg in the table) on the YouTube testing
set, while NeuralSeg is trained on the YouTube training set with different
input modality settings. Notably, our model significantly outperforms the best
baseline, Cross-BERT, even when only using video transcripts (text) as input.
But on the other hand, if we train the model by using only video frames
(visual) as input, the model’s performance is even considerably worse than the
random baseline, possibly because that too diverse visual and topic presence
in the corpus makes it difficult for the model to learn a meaningful visual
input-to-prediction projection. Yet, by combining both modalities with cross-
modal attention fusion, the model’s performance can be further enhanced
compared with only textual modality. These results confirm that the visual
information itself may not be sufficient to capture a video’s topic-related
semantics under a supervised setting, but fusing it together with the textual
information can provide a more clear picture of the video’s underlying topics.

Method | Modalities | BBC | Behance
---|---|---|---
|  |  Pr $\downarrow$ |  F1 $\uparrow$ |  mIoU $\uparrow$ |  Pr $\downarrow$ |  F1 $\uparrow$ |  mIoU $\uparrow$
Random | – | 48.29 | 47.36 | 30.66 | 46.27 | 50.00 | 36.24
X-Tiling [15] | Text | 41.59 | 50.72 | 33.27 | 45.93 | 49.95 | 43.16
BayesSeg [8] | Text | 42.01 | 51.01 | 33.79 | 49.01 | 48.16 | 37.55
GraphSeg [11] | Text | 45.11 | 49.28 | 30.23 | 46.76 | 50.12 | 40.38
Cross-BERT [26] | Text | 44.63 | 49.70 | 31.51 | 46.15 | 49.88 | 41.02
X-Tiling [15] | Visual | 44.48 | 49.89 | 33.26 | 44.56 | 50.73 | 41.81
TransNet [34] | Visual | 42.54 | 49.82 | 31.12 | 46.34 | 49.88 | 40.54
LGSS [30] | Visual | 42.88 | 50.02 | 32.66 | 45.67 | 50.12 | 39.90
X-Tiling [15] | Text + Visual | 43.22 | 50.44 | 32.68 | 45.87 | 50.01 | 41.72
Ours (Plain) | Text + Visual | 43.14 | 54.13 | 31.23 | 45.71 | 51.25 | 34.33
Ours (Window) | Text + Visual | 40.66 | 54.92 | 37.50 | 43.03 | 51.83 | 47.66
1-8 Ours (Window + CL-Cross) | Text + Visual |  36.45† | 54.91 |  50.92† | 42.72 | 51.68 |  48.65†
Ours (Window + CL-Intra) | Text + Visual |  37.14† | 55.10 |  49.22† | 42.56 | 51.84 | 48.39
Ours (Window + CL-Dual) | Text + Visual |  36.01† | 55.75 |  51.56† |  42.25† | 51.91 |  49.35†

Table 3:  Results on BBC and Behance corpora for cross-domain evaluation.
Bolded and underlined results indicate the best performance across all
comparisons and within their own section. $\dagger$ indicates results applied
contrastive domain adaptation which are significantly different ($p<0.05$)
from Ours (Window).

Cross-Domain Segment Inference: Table 3 compares the performance of the
baselines and our proposed segmenter pre-trained on YouTube w/ or w/o applying
our long video adaptation strategy to two challenging long video corpora. To
better investigate the effectiveness of the two contrastive objectives
associated with long video adaptation, Table 3 also shows the results by
adding each objective (CL-Cross/CL-Intra) individually. The primary takeaway
is that our long video adaptation strategy (Window + CL-Dual) significantly
and consistently improves the performance achieved by the segmenter initially
trained on short YouTube videos (Plain). In more detail, we can observe that
utilizing the sliding window inference strategy (Plain $\rightarrow$ Window)
already yields noticeable performance gains on both corpora. Furthermore,
adding contrastive learning objectives can further boost accuracy on two long
video corpora. The greatest enhancement is seen when the model is tuned with
dual-contrastive losses.

##  7 Conclusion and Future Work

We present a multi-modal video topic segmentation model accepting both video
transcripts and frames as input. Further, we propose a novel unsupervised
domain adaptation strategy empowered by a dual-contrastive learning framework
to generalize our model pre-trained on short videos to longer videos with more
complex content and subtler visual changes. Experiments on two settings
(intra-domain and cross-domain segment inference) show that (1) our system
achieves the SOTA performance on a newly collected YouTube corpus consisting
of large-scale but short videos; (2) When we apply our long video adaption
strategy, the model for short videos achieves better performance when
transferred to two domains comprising long (live-)stream videos.

## References

  * [1] Alam, T., Khan, A., Alam, F.: Punctuation restoration using transformer models for high-and low-resource languages. In: Proceedings of the Sixth Workshop on Noisy User-generated Text (W-NUT 2020). pp. 132–142 (2020)
  * [2] Baraldi, L., Grana, C., Cucchiara, R.: A deep siamese network for scene detection in broadcast videos. In: Proceedings of ACM MM 2015. pp. 1199–1202 (2015)
  * [3] Beeferman, D., Berger, A., Lafferty, J.: Statistical models for text segmentation. Machine Learning 34(1), 177–210 (1999)
  * [4] Cao, X., Chen, Z., Le, C., Meng, L.: Multi-modal video chapter generation. ArXiv abs/2209.12694 (2022)
  * [5] Chen, D., Wang, D., Darrell, T., Ebrahimi, S.: Contrastive test-time adaptation. In: Proceedings of CVPR 2022. pp. 295–305 (2022)
  * [6] Chen, S., Nie, X., Fan, D.D., Zhang, D., Bhat, V., Hamid, R.: Shot contrastive self-supervised learning for scene boundary detection. In: Proceedings of CVPR 2021. pp. 9791–9800 (2021)
  * [7] Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: BERT: Pre-training of deep bidirectional transformers for language understanding. In: Proceedings of NAACL 2019. pp. 4171–4186. Association for Computational Linguistics (2019)
  * [8] Eisenstein, J., Barzilay, R.: Bayesian unsupervised topic segmentation. In: Proceedings of EMNLP 2008. pp. 334–343 (2008)
  * [9] Fraser, C., Kim, J., Shin, H., Brandt, J., Dontcheva, M.: Temporal segmentation of creative live streams. In: Proceedings of CHI 2020. pp. 1–12 (2020)
  * [10] Georgescul, M., Clark, A., Armstrong, S.: An analysis of quantitative aspects in the evaluation of thematic segmentation algorithms. In: Proceedings of SIGdial 2006. pp. 144–151 (2006)
  * [11] Glavaš, G., Nanni, F., Ponzetto, S.P.: Unsupervised text segmentation using semantic relatedness graphs. In: Proceedings of the Fifth Joint Conference on Lexical and Computational Semantics. pp. 125–130 (2016)
  * [12] Glavas, G., Somasundaran, S.: Two-level transformer and auxiliary coherence modeling for improved text segmentation. In: Proceeding of AAAI-20. pp. 2306–2315 (2020)
  * [13] Gygli, M., Grabner, H., Riemenschneider, H., Van Gool, L.: Creating summaries from user videos. In: Proceedings of ECCV 2014. pp. 505–520 (2014)
  * [14] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of CVPR 2016. pp. 770–778 (2016)
  * [15] Hearst, M.A.: Text tiling: Segmenting text into multi-paragraph subtopic passages. Computational Linguistics 23(1), 33–64 (1997)
  * [16] Jadon, S., Jasim, M.: Unsupervised video summarization framework using keyframe extraction and video skimming. In: Proceedings of ICCCA 2020. pp. 140–145 (2020)
  * [17] James, N., Todorov, K., Hudelot, C.: Combining visual and textual modalities for multimedia ontology matching. In: Semantic Multimedia. pp. 95–110 (2011)
  * [18] Jayaraman, D., Grauman, K.: Slow and steady feature analysis: Higher order temporal coherence in video. In: Proceeding of CVPR 2016. pp. 3852–3861 (2016)
  * [19] Kang, G., Jiang, L., Yang, Y., Hauptmann, A.G.: Contrastive adaptation network for unsupervised domain adaptation. In: Proceedings of CVPR 2019. pp. 4893–4902 (2019)
  * [20] Kim, D., Tsai, Y.H., Zhuang, B., Yu, X., Sclaroff, S., Saenko, K., Chandraker, M.: Learning cross-modal contrastive features for video domain adaptation. In: Proceedings of ICCV 2021. pp. 13598–13607 (2021)
  * [21] Koshorek, O., Cohen, A., Mor, N., Rotman, M., Berant, J.: Text segmentation as a supervised learning task. In: Proceedings of NAACL 2018. pp. 469–473 (2018)
  * [22] Kumar, A., Mittal, T., Manocha, D.: MCQA: multimodal co-attention based network for question answering. CoRR abs/2004.12238 (2020)
  * [23] Li, J., Sun, A., Joty, S.: Segbot: A generic neural text segmentation model with pointer network. In: Proceedings of IJCAI-18. pp. 4166–4172 (2018)
  * [24] Lo, K., Jin, Y., Tan, W., Liu, M., Du, L., Buntine, W.: Transformer over pre-trained transformer for neural text segmentation with enhanced topic coherence. In: EMNLP 2021 (Findings). pp. 3334–3340 (2021)
  * [25] Lu, J., Yang, J., Batra, D., Parikh, D.: Hierarchical question-image co-attention for visual question answering. In: Proceedings of NeurIPS 2016. pp. 289–297 (2016)
  * [26] Lukasik, M., Dadachev, B., Papineni, K., Simões, G.: Text segmentation by cross segment attention. In: Proceedings of EMNLP 2020. pp. 4707–4716 (2020)
  * [27] van den Oord, A., Li, Y., Vinyals, O.: Representation learning with contrastive predictive coding. ArXiv abs/1807.03748 (2018)
  * [28] Qiu, J., Zhu, J., Xu, M., Dernoncourt, F., Bui, T., Wang, Z., Li, B., Zhao, D., Jin, H.: Semantics-consistent cross-domain summarization via optimal transport alignment. ArXiv abs/2210.04722 (2022)
  * [29] Radford, A., Kim, J.W., Xu, T., Brockman, G., McLeavey, C., Sutskever, I.: Robust speech recognition via large-scale weak supervision. ArXiv abs/2212.04356 (2022)
  * [30] Rao, A., Xu, L., Xiong, Y., Xu, G., Huang, Q., Zhou, B., Lin, D.: A local-to-global approach to multi-modal movie scene segmentation. In: Proceedings of CVPR 2020. pp. 10146–10155 (2020)
  * [31] Rasheed, Z., Shah, M.: Scene detection in hollywood movies and tv shows. In: Proceedings of CVPR 2003. pp. II–343 (2003)
  * [32] Rui, Y., Huang, T.S., Mehrotra, S.: Exploring video structure beyond the shots. In: Proceedings of the IEEE International Conference on Multimedia Computing and Systems. pp. 237–240 (1998)
  * [33] Song, Y., Vallmitjana, J., Stent, A., Jaimes, A.: Tvsum: Summarizing web videos using titles. In: Proceeding of CVPR 2015. pp. 5179–5187 (2015)
  * [34] Souček, T., Lokoč, J.: Transnet v2: An effective deep network architecture for fast shot transition detection. ArXiv abs/2008.04838 (2020)
  * [35] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L.u., Polosukhin, I.: Attention is all you need. In: Advances in Neural Information Processing Systems. vol. 30 (2017)
  * [36] Wang, Z., Zhong, Y., Miao, Y., Ma, L., Specia, L.: Contrastive video-language learning with fine-grained frame sampling. In: Proceedings of AACL-IJCNLP 2022\. pp. 694–705 (2022)
  * [37] Xiao, S., Chen, L., Zhang, S., Ji, W., Shao, J., Ye, L., Xiao, J.: Boundary proposal network for two-stage natural language video localization. In: Proceedings of AAAI-21 (2021)
  * [38] Xing, L., Carenini, G.: Improving unsupervised dialogue topic segmentation with utterance-pair coherence scoring. In: Proceedings of SIGdial 2021. pp. 167–177 (2021)
  * [39] Xing, L., Hackinen, B., Carenini, G., Trebbi, F.: Improving context modeling in neural topic segmentation. In: Proceedings of AACL-IJCNLP 2020. pp. 626–636 (2020)
  * [40] Xing, L., Huber, P., Carenini, G.: Improving topic segmentation by injecting discourse dependencies. In: Proceedings of the 3rd Workshop on Computational Approaches to Discourse. pp. 7–18 (2022)
  * [41] Zhu, W., Pang, B., Thapliyal, A.V., Wang, W.Y., Soricut, R.: End-to-end dense video captioning as sequence generation. In: Proceedings of COLING 2022. pp. 5651–5665 (2022)