11institutetext: ASTRON, Netherlands Institute for Radio Astronomy, Oude
Hoogeveensedijk 4, 7991 PD, Dwingeloo, The Netherlands

# Fourier-domain dedispersion

C. G. Bassa  J. W. Romein  B. Veenboer  S. van der Vlugt  S. J. Wijnholds

(Received ; Accepted )

We present and implement the concept of the Fourier-domain dedispersion (FDD)
algorithm, a brute-force incoherent dedispersion algorithm. This algorithm
corrects the frequency-dependent dispersion delays in the arrival time of
radio emission from sources such as radio pulsars and fast radio bursts. Where
traditional time-domain dedispersion algorithms correct time delays using time
shifts, the FDD algorithm performs these shifts by applying phase rotations to
the Fourier-transformed time-series data. Incoherent dedispersion to many
trial dispersion measures (DMs) is compute, memory-bandwidth and I/O intensive
and dedispersion algorithms have been implemented on Graphics Processing Units
(GPUs) to achieve high computational performance. However, time-domain
dedispersion algorithms have low arithmetic intensity and are therefore often
memory-bandwidth limited. The FDD algorithm avoids this limitation and is
compute limited, providing a path to exploit the potential of current and
upcoming generations of GPUs. We implement the FDD algorithm as an extension
of the dedisp time-domain dedispersion software. We compare the performance
and energy-to-completion of the FDD implementation using an NVIDIA Titan RTX
GPU against the standard as well as an optimized version of dedisp. The
optimized implementation already provides a factor of 1.5 to 2 speedup at only
66% of the energy utilization compared to the original algorithm. We find that
the FDD algorithm outperforms the optimized time-domain dedispersion algorithm
by another 20% in performance and 5% in energy-to-completion when a large
number of DMs ($\gtrsim 512$) are required. The FDD algorithm provides
additional performance improvements for FFT-based periodicity surveys of radio
pulsars, as the Fourier transform back to the time domain can be omitted. We
expect that this computational performance gain will further improve in the
future since the Fourier-domain dedispersion algorithm better matches the
trends in technological advancements of GPU development.

###### Key Words.:

methods: data analysis – pulsars: general

##  1 Introduction

Radio waves propagating through the ionized intergalactic, interstellar and/or
interplanetary medium undergo dispersion, which introduces a frequency-
dependent time delay between emission at the source and reception at the
telescope (e.g. Lorimer & Kramer, 2012). Hence, pulsed radio signals from
astrophysical sources like radio pulsars and fast radio bursts (see Kaspi &
Kramer, 2016; Petroff et al., 2019, for reviews) have a distinctive dispersion
measure (DM), directly related to the electron column density between the
source and the observer. If not properly corrected for, the dispersion delays
lead to smearing and hence loss of signal-to-noise of the received radio
pulses.

Correcting for the dispersion is called dedispersion, and can be performed
using two techniques, depending on the state of the radio observation. For
Nyquist-sampled time-series data of orthogonal polarizations (voltage data
hereafter), the phase information can be used to coherently dedisperse the
signals through a convolution with the inverse of the response function of
dispersion (Hankins, 1971; Hankins & Rickett, 1975). This technique completely
removes the effects of dispersion. With coherent dedispersion, dedispersion is
performed before squaring the orthogonal polarizations signals to form the
Stokes parameters. For signals that have already been squared to form the
Stokes parameters, the incoherent dedispersion method is used. The time-series
data of finite channels are shifted in time to correct for the dispersion
delay between channels, while dispersive smearing within channels remains
uncorrected. Hence, the size of frequency channels and time samples is
typically optimized to minimize dispersive smearing within channels.

In surveys for new radio pulsars or fast radio bursts, the dispersion measure
of a new source is a priori unknown and the input data need to be dedispersed
to many, usually thousands, of trial dispersion measures. The majority of
current surveys perform incoherent dedispersion on Stokes I time-series,
typically with sample times between 50 to 100 $\upmu$s, and channel sizes of
0.1 to 0.5 MHz (Keith et al. 2010; Stovall et al. 2014; Lazarus et al. 2015;
Bhattacharyya et al. 2016, see also Lyon et al. 2016), as this requires lower
data rates and less storage compared to voltage data. Only at the lowest
observing frequencies ($\nu\lesssim 300$ MHz), voltage data are recorded to
perform a combination of coherent and incoherent dedispersion (Bassa et al.,
2017a, b; Pleunis et al., 2017).

The computational cost to incoherently dedisperse is significant, as the input
data typically consist of $N_{\nu}\sim 10^{3}$ frequency channels for
$N_{t}\sim 10^{6}$ time samples, and needs to be dedispersed to
$N_{\mathrm{DM}}\sim 10^{4}$ dispersion measures. Hence, algorithms have been
developed to either reduce the computational cost by reducing the complexity,
by using e.g. a tree algorithm (Taylor, 1974) or a piece-wise linear
approximation of the dispersion delay (Ransom, 2001), a transformation
algorithm (Zackay & Ofek, 2017) and/or by using hardware acceleration of the
direct, brute force, dedispersion approach. A review of these algorithms, and
an implementation of the direct dedispersion algorithm on graphics processing
units (GPUs), is presented by Barsdell et al. (2012).

Sclocco et al. (2016) found that the direct, brute force, dedispersion
algorithm on many-core accelerators is inherently memory-bandwidth bound, due
to the low arithmetic intensity of the required calculations. Hence, the
direct dedispersion algorithm does not optimally benefit from advances in the
computing performance of many-core accelerators. In this paper we investigate
the feasibility of the direct dedispersion approach through, what we call,
Fourier-domain dedispersion, where the time delays due to dispersion are
performed as phase rotations in the Fourier transform of the time-series data
of each channel111To avoid confusion with observing frequencies, we use the
term Fourier-domain instead of frequency-domain..

We present the description of the Fourier-domain dedispersion algorithm in § 2
and compare it to the time-domain variant. § 3 provides a GPU implementation
of the Fourier-domain dedispersion algorithm along with an optimization of the
GPU accelerated time-domain direct dedispersion algorithm by Barsdell et al.
(2012), the results for the different implementations are compared in § 4. We
discuss and conclude our feasibility study in § 5.

##  2 Algorithm description

###  2.1 Time-domain dedispersion

Dispersion in the ionized intergalactic, interstellar and interplanetary
medium leads to delays in the arrival time of radio signals, The magnitude of
this delay depends on the frequency of the radio signal observed. For a given
dispersion measure $\mathrm{DM}=\int n_{e}\mathrm{d}l$, the path integral over
the electron density $n_{e}$ along the line of sight, signals arrive with a
time delay

$\Delta t\left(\nu,\mathrm{DM}\right)=\mathrm{DM}\leavevmode\nobreak\
k_{\mathrm{DM}}\left(\nu^{-2}-\nu_{0}^{-2}\right),$ (1)

where $\nu$ is the observing frequency and $\nu_{0}$ is the reference
frequency. The proportionality constant is $k_{\mathrm{DM}}=(2.41\times
10^{-4})^{-1}$ MHz2 cm3 pc-1 s (Manchester & Taylor, 1972; Kulkarni, 2020).

The standard incoherent dedispersion method takes an input dynamic spectrum
$I\left(t,\nu\right)$ with Stokes-I values as a function of observing time $t$
and observing frequency $\nu$ and shifts the time series at the individual
observing frequencies to the nearest time sample given the delays described in
Eqn. 1 based on an assumed DM. This results in a new dynamic spectrum in which
all time series are aligned for the assumed DM. The shifted time series are
then summed over observing frequency to give the intensity as function of time
observed at the assumed DM

$I\left(t,\mathrm{DM}\right)=\sum_{\nu}^{N_{\nu}}I\left(t-\Delta
t\left(\nu,\mathrm{DM}\right),\nu\right).$ (2)

When the DM of the source is not known, this procedure needs to be repeated
for many trial DMs.

The brute-force time-domain dedispersion algorithm requires significant memory
bandwidth while requiring only minimal computations once the data are properly
aligned. This led us to explore the benefits of analyzing the Fourier
transform of the time series, where the time shifts can be replaced by phase
rotations through the Fourier shift theorem (e.g. Bracewell, 1986), resulting
in dense matrix multiplications, which can be computed efficiently. We
describe this Fourier-domain dedispersion (FDD) method in the next section.

Figure 1: Comparison between time-domain dedispersion (top row) and FDD
(bottom row) with (a) and (c) representing the input dynamic spectrum, (b)
showing the dynamic spectrum after applying the appropriate delays for each
frequency, (d) showing the Fourier transform of the time series (amplitude and
phase) along with the phasors applied in
$\mathcal{T}_{\nu\rightarrow\mathrm{DM}}$, (e) showing the result after
applying the phasors and (f) showing the dynamic spectrum after the inverse
Fourier transform $\mathcal{F}_{t\rightarrow f_{\mathrm{s}}}^{-1}$. The hashed
regions in (b) and (f) represent data that should be discarded.

###  2.2 Fourier-domain dedispersion

In FDD, the time series for each observing frequency is Fourier transformed to
the Fourier domain of time, frequency. To distinguish these frequencies from
the observing frequencies of the data $\nu$, we will refer to the Fourier
transform of the time series as the spin frequencies $f_{\mathrm{s}}$ that
would be associated with a periodic signal from a radio pulsar. The Fourier
transform results in a new dynamic spectrum
$I\left(f_{\mathrm{s}},\nu\right)$, in which the data are represented as a
function of spin frequency $f_{\mathrm{s}}$ and observing frequency $\nu$. We
describe this as

$I\left(f_{\mathrm{s}},\nu\right)=\int
I\left(t,\nu\right)e^{-2\pi\mathrm{i}f_{\mathrm{s}}t}\mathrm{d}t=\mathcal{F}_{t\rightarrow
f_{\mathrm{s}}}\left\\{I\left(t,\nu\right)\right\\},$ (3)

where the Fourier transform operator $\mathcal{F}$ has subscript $t\rightarrow
f_{\mathrm{s}}$ to emphasize that this is a mapping from time domain to spin-
frequency domain instead of the regular 2-D Fourier transform. This notation
also allows us to use the Fourier transform for other mappings later on
without causing confusion.

In (spin) frequency space, the time shifts from Eqn. 1 can be represented by
complex phase rotations, which we will refer to as phasors, of the form

$W\left(f_{\mathrm{s}},\nu,\mathrm{DM}\right)=e^{-2\pi\mathrm{i}f_{\mathrm{s}}\Delta
t(\nu,\mathrm{DM})}.$ (4)

The dedispersed time-series can now be recovered through the inverse Fourier
transform $\mathcal{F}_{t\rightarrow f_{\mathrm{s}}}^{-1}$ and a summation
over observing frequency

$I\left(t,\mathrm{DM}\right)=\sum_{\nu}^{N_{\nu}}\mathcal{F}_{t\rightarrow
f_{\mathrm{s}}}^{-1}\left\\{W\left(f_{\mathrm{s}},\nu,\mathrm{DM}\right)I\left(f_{\mathrm{s}},\nu\right)\right\\}$
(5)

Due to the linearity of the Fourier transform, the summation and inverse
Fourier transform can be interchanged, yielding

$I\left(t,\mathrm{DM}\right)=\mathcal{F}_{t\rightarrow
f_{\mathrm{s}}}^{-1}\left\\{\sum_{\nu}^{N_{\nu}}W\left(f_{\mathrm{s}},\nu,\mathrm{DM}\right)I\left(f_{\mathrm{s}},\nu\right)\right\\},$
(6)

which reduces the number of inverse Fourier transforms from
$N_{\mathrm{DM}}\times N_{\mathrm{\nu}}$ to $N_{\mathrm{DM}}$.

Furthermore, it is instructive to define

$I\left(f_{\mathrm{s}},\mathrm{DM}\right)=\sum_{\nu}^{N_{\nu}}W\left(f_{\mathrm{s}},\nu,\mathrm{DM}\right)I\left(f_{\mathrm{s}},\nu\right)=\mathcal{T}_{\nu\rightarrow\mathrm{DM}}\left\\{I\left(f_{\mathrm{s}},\nu\right)\right\\},$
(7)

since phase rotations provide a mapping from observing frequencies to
dispersion measures. As the phase rotations depend on spin frequency, the data
for each spin frequency need to be processed independently. For a given spin
frequency, this mapping resembles a direct Fourier transform. However, it
cannot be implemented as a fast Fourier transform due to the non-linear
dependence of $\Delta t$ on observing frequency. Despite these two issues that
complicate the implementation, we note that Eqn. 7 can be viewed as a
transformation that maps the space of observing frequencies to the space of DM
values and denote this transformation by
$\mathcal{T}_{\nu\rightarrow\mathrm{DM}}$.

Figure 1 shows a cartoon of the traditional time-domain dedispersion algorithm
and the Fourier-domain algorithm. The panels in the top row show the
traditional algorithm where time-series at individual observing frequencies
are shifted in time for a given DM. The steps presented in Eqns. 3, 6 and 7
are illustrated in the bottom row of Fig. 1. After Fourier transforming the
input time series for each frequency, shown in panel 1(c), a spectrum as
function of spin frequency is obtained, shown in panel 1(d), showing the
amplitude and phase of the complex values. The phase of the Fourier
transformed time series encodes the arrival time of the pulse. The phase slope
of the $W\left(f_{\mathrm{s}},\nu,\mathrm{DM}\right)$ phasor is shown by the
green curves in panel 1(d), and depends on the dispersion delay as given in
Eqn. 4. The transformation $\mathcal{T}_{\nu\rightarrow\mathrm{DM}}$ corrects
the measured phases with the appropriate phase slope. In our example, this
results in the phases shown in panel 1(e). As this is a pure phase rotation,
the amplitudes are unaltered. After applying the inverse Fourier transform to
each of the spectra, we obtain a shifted time series at each frequency. When
the correct phasors are applied, this results in the peaks being aligned as
shown in 1(f) and we obtain the same results as with time-domain dedispersion
as shown in 1(b) for comparison.

The use of the Fourier shift theorem and the cyclic nature of the Fourier
transform contaminates the end of each time series with samples from the
start. Hence, given a maximum $\mathrm{DM}$ ($\mathrm{DM}_{\mathrm{max}}$) and
the extrema of the observing band ($\nu_{\mathrm{min}}$ and
$\nu_{\mathrm{max}}$) at least $\Delta
t_{\mathrm{max}}=\mathrm{DM}_{\mathrm{max}}\leavevmode\nobreak\
k_{\mathrm{DM}}\left(\nu_{\mathrm{min}}^{-2}-\nu_{\mathrm{max}}^{-2}\right)$
at the end of each time series should be discarded when dedispersing against
the highest observing frequency channel, see Fig. 1(f). In the time-domain
dedispersion algorithm the same amount of data should be discarded, see Fig.
1(b).

As dispersive smearing increases with increasing $\mathrm{DM}$, time-domain
dedispersion algorithms down-sample the input time-series when the smearing
exceeds one or more native time samples. In the Fourier-domain algorithm, this
down-sampling can be efficiently implemented by reducing the length of the
inverse Fourier transform $\mathcal{F}_{t\rightarrow f_{\mathrm{s}}}^{-1}$.

###  2.3 Algorithm optimization

The transformation $\mathcal{T}_{\nu\rightarrow\mathrm{DM}}$ poses two
implementation challenges: the phasors described by (4) depend on the spin
frequency and the non-linear dependence of $\Delta t$ on observing frequency
precludes the use of the fast Fourier transform. In this section, we present
three approximations to reduce the computational burden incurred by these two
challenges.

Calculation of the phasors for each trial DM, observing frequency and spin
frequency consumes a significant amount of computing resources. Also, an
actual implementation of (7) leads to a matrix-vector product of a matrix of
phasors for constant $f_{\mathrm{s}}$ extracted from the phasor tensor
$W(f_{\mathrm{s}},\nu,\mathrm{DM})$ and a vector with data values for constant
$f_{\mathrm{s}}$ extracted from the data matrix
$I(f_{\mathrm{s}},\mathrm{DM})$. As GPUs can perform matrix multiplications
efficiently, it is expected to be beneficial to process data for multiple spin
frequencies simultaneously. Therefore, it may be worthwhile to accept a spin-
frequency-dependent set of DM values and re-sample those afterwards to the
desired set of trial DMs. To this end, consider the following spin-frequency-
dependent DM

$\mathrm{DM}_{f_{\mathrm{s}}}=\mathrm{DM}_{f_{s,0}}\frac{f_{s,0}}{f_{\mathrm{s}}}.$
(8)

Substitution in (4) gives

$\displaystyle W\left(f_{\mathrm{s}},\nu,\mathrm{DM}_{f_{\mathrm{s}}}\right)$
$\displaystyle=$
$\displaystyle\exp\left\\{-2\pi\mathrm{i}f_{\mathrm{s}}\mathrm{DM}_{f_{\mathrm{s}}}k_{\mathrm{DM}}\left(\frac{1}{\nu^{2}}-\frac{1}{\nu_{0}^{2}}\right)\right\\}$
(9) $\displaystyle=$
$\displaystyle\exp\left\\{-2\pi\mathrm{i}f_{\mathrm{s}}\mathrm{DM}_{f_{s,0}}\frac{f_{s,0}}{f_{\mathrm{s}}}k_{\mathrm{DM}}\left(\frac{1}{\nu^{2}}-\frac{1}{\nu_{0}^{2}}\right)\right\\}$
$\displaystyle=$
$\displaystyle\exp\left\\{-2\pi\mathrm{i}f_{s,0}\mathrm{DM}_{f_{s,0}}k_{\mathrm{DM}}\left(\frac{1}{\nu^{2}}-\frac{1}{\nu_{0}^{2}}\right)\right\\}$
$\displaystyle=$ $\displaystyle\exp\left\\{-2\pi\mathrm{i}f_{s,0}\Delta
t(\nu,\mathrm{DM}_{f_{s,0}})\right\\}.$

Note that these phasors are independent of spin frequency, implying that these
phasors can be applied over a range of spin frequencies reasonably close to
the reference spin frequency $f_{s,0}$. The extent of this range is limited by
the tolerable amount of contraction or expansion of the set of DM values for
spin frequencies other than $f_{s,0}$ as given by (8).

Although the structure of the phasors describing the mapping to DM space,
$\mathcal{T}_{\nu\rightarrow\mathrm{DM}}$, appears to resemble the Fourier
transform very closely, the exact transformation from observing frequencies to
DM values cannot be represented by a discrete Fourier transform (DFT) matrix,
let alone a fast Fourier transform (FFT) due to the non-linear dependence of
the phases of the phasors on observing frequencies. This can be remedied by
using a first order Taylor approximation for the frequency dependence, i.e.,

$\left(\frac{1}{\nu^{2}}-\frac{1}{\nu_{0}^{2}}\right)\Bigg{|}_{\nu_{0}}\approx\frac{2}{\nu_{0}^{2}}-\frac{2\nu}{\nu_{0}^{3}}.$
(10)

Substitution of this approximation in (4) gives

$W\left(f_{\mathrm{s}},\nu,\mathrm{DM}\right)=\exp\left\\{-2\pi\mathrm{i}f_{\mathrm{s}}\mathrm{DM}k_{\mathrm{DM}}\cdot
2\left(\frac{\nu_{0}-\nu}{\nu_{0}^{3}}\right)\right\\}.$ (11)

This approximation can be exploited to implement
$\mathcal{T}_{\nu\rightarrow\mathrm{DM}}$ by a FFT per spin frequency or even
multiple spin frequencies when combined with the optimization from the
previous subsection. The observing frequency range over which this gives
acceptable results depends on DM and spin frequency.

To be able to treat the $\mathcal{T}_{\nu\rightarrow\mathrm{DM}}$
transformation as a DFT, and be able to use the advantages of an FFT for
precision and performance over a DFT, the observing frequencies $\nu$ would
have to be remapped to a linear scale following $\nu^{-2}$. Such a remapping
has been used by Manchester et al. (2001). This mapping could be obtained
through either interpolation or inserting empty channels in the linearized
scale at $\nu^{-2}$ values that do not map to integer observing channels. It
is likely that such a remapping would increase GPU memory usage, as well as
introduce some smearing in the dedispersed time series. The analysis of this
optimization and assessment of its performance and accuracy over the direct
computation of the transform goes beyond the scope of this paper.

##  3 Implementation

We have implemented Fourier-domain dedispersion as an extension of the dedisp
library222{https://github.com/ajameson/dedisp} by Barsdell et al. (2012). This
library implements the brute-force, direct incoherent dedispersion algorithm
on NVIDIA GPUs and is widely used (e.g. Jameson & Barsdell 2019; Sclocco et
al. 2015). To make a fair comparison, we have updated the dedisp library by
Barsdell et al. (2012) to make use of newer hardware and software features. In
the following, we will refer to the original implementation as dedisp and to
the updated implementation as tdd (time-domain dedispersion).

We refer to our implementation of Fourier-domain dedispersion as fdd. For this
work we have implemented the algorithm as described under § 2.2, additional
optimizations as proposed in § 2.3, as well as the use of new GPU tensor-core
technology are left as future work.

Sclocco et al. (2016) found that brute-force dedispersion in the time domain
has a low arithmetic intensity and is inherently memory-bandwidth bound. They
show that auto-tuning of kernel parameters significantly improves performance
on various accelerators. For this work we explore the feasibility of
dedispersion in the Fourier domain and compare performance for a
representative parameter space according to the use cases outlined in the
introduction. The performance of time-domain dedispersion is predominantly
determined by the amount of data reuse and memory bandwidth.

The work of Barsdell et al. (2012) already shows that major performance
improvements are achieved when moving the implementation from a central
processing unit (CPU) to a GPU. Performance improvements are mainly attributed
to the higher memory bandwidth of the GPU. We compare dedisp, tdd and fdd on
the NVIDIA Titan RTX GPU. The full measurement set-up is listed in Table 1. We
refer, in NVIDIA CUDA terminology, to the CPU as host and to the GPU as
device. The main application runs on the host and launches computational
kernels on the device. Input and output data are copied between the host and
device over a PCIe interface.

Dedispersion is usually part of a pipeline running on a GPU where
initialization of and input/output to the GPU kernels are part of the
pipeline. The input to dedispersion is a matrix of size $N_{\nu}\times N_{t}$
of packed 8-bit data, with $N_{\nu}$ frequency channels and $N_{t}$ time
samples. For time-domain dedispersion these data have to be transposed,
unpacked and dedispersed. The resulting output is a matrix of
$N_{\mathrm{DM}}\times N_{t}$ 32-bit floating point data, with $N_{DM}$ trial-
DMs.

For time-domain dedispersion the input data $I\left(t,\nu\right)$ can be
segmented in the time dimension, taking a segment of $M_{t}(<N_{t})$ of time
samples, for all frequency channels. dedisp implements the above described
pipeline with batch processing where a segment of input data is copied from
host to device, the GPU applies the kernel pipelines, parallelizing over DMs,
and the resulting output data $I\left(t,\mathrm{DM}\right)$ for the segment is
copied back to the host before copying a new segment of data to the device.

We extended the dedisp framework with an optional tdd implementation that
contains several improvements over dedisp. First, the transfers of input and
output data were overlapped with computations on the GPU, moving the copying
of input and output out of the critical path. Transfer speeds were increased
by a factor of 2 to 3 by changing the memory transfers from paged to pinned
memory. Furthermore, the unpacking and transpose operations were combined into
one kernel, thus requiring only a single pass over the data. Finally, the use
of texture memory is optionally disabled, as this optimization for old GPUs
reduces performance on recent GPUs. These improvements are currently available
as a fork333https://github.com/svlugt/dedisp of the dedisp library by Barsdell
et al. (2012) and are being integrated with the original library.

For the implementation of fdd, the input data cannot straightforwardly be
segmented in time. It follows from Eqn. 3 that splitting in the time dimension
affects the outcome of the FFT. Instead we split input data along the
frequency-channel dimension (shown as channel job on line 1 of Algorithm
listing 1). Such an input data segment is copied to the GPU, where the data
are transposed and unpacked, like in the preprocessing step in tdd. Next, the
data are zero padded such that the number of samples becomes a power of two.
The last step of preprocessing is the Fourier transformation (Eqn. 3) of the
input samples. Dedispersion is performed as phase rotations (Eqn. 7) in the
Fourier domain, followed by a summation over frequency channels. The result is
transformed back to the time domain (Eqn. 6) before it is scaled and
transferred back from device to host. Both forward and backward Fourier
transformations are implemented with the NVIDIA cuFFT library. Continuing on
Algorithm listing 1, we show that within batches of channels the work is split
into batches of DMs (DM job starting at line 1) to efficiently parallelize the
workload on the GPU. The backward FFT is implemented as a batched series of 1D
FFTs for each DM. To allow for efficient batching of the backwards FFTs, the
output of the dedispersion kernel is buffered on the GPU for all batches of
channels before it is transformed back to the time domain (starting at line
1). Due to GPU memory size constraints, we cannot process all the DMs at once
for a large number of DMs. We solve this by processing the DMs in batches with
an additional $DM\\_outer$ dimension (line 1). This approach has the advantage
that the output data are only copied once, after all input for the current
batch of DMs is processed and postprocessing has taken place. The input data,
on the other hand, has to be copied and preprocessed multiple times for
different batches of DMs. We minimize the cost of these repeated operations by
overlapping them with the copy of output data. Batching of channels and DMs
also allows for data reuse in the other GPU kernels.

44 generate_spin_frequency_table();

45 allocate_gpu_memory();

46 initialize_gpu_fft();

47 intialize_jobs();

48 copy_delay_table();

49 for  _$idm\\_outer=0\dots dm\\_jobs.size()$_ :

50 for  _$ichannel=0\dots channel\\_jobs.size()$_ :

// input

51 if  _$ichannel==0$_ then

52 copy_to_gpu($input_{ichannel}$);

53

54 end if

// preprocessing

55 transpose_and_unpack($input_{ichannel}$);

56 zero_pad($input_{ichannel}$);

57 apply_r2c_FFT($input_{ichannel}$);

// dedispersion

58 for  _$idm\\_inner=0\dots
ndm\\_bu\mkern-2.0muf\mkern-3.0muf\mkern-2.0muers$_ :

59 apply_dedispersion($input_{ichannel}$, $output_{idm\\_inner}$);

60

61

// input

62 if  _$ichannel+1 <channel\\_jobs.size()$_ then

63 copy_to_gpu($input_{ichannel+1}$);

64

65 end if

66

67

68 for  _$idm\\_inner=0\dots
ndm\\_bu\mkern-2.0muf\mkern-3.0muf\mkern-2.0muers$_ :

// postprocessing

69 apply_c2r_FFT($output_{idm\\_inner}$);

70 apply_scaling($output_{idm\\_inner}$);

// output

71 copy_from_gpu($output_{idm\\_inner}$);

72

73

74

75

Algorithm 1 This listing shows batch processing for the fdd implementation.
This process starts with a number of initialization steps on the host
(highlighted in blue). Next, DMs are processed in batches ($idm_{outer}$). For
every batch of DMs, frequency channels are also processed in batches. This
enables overlapping of data transfers (highlighted in red) with computation on
the GPU (highlighted in green). After dedispersion has finished, the output is
again processed in batches such that postprocessing is overlapped with data
transfers.

##  4 Results

###  4.1 Performance comparison

To allow benchmarking of performance we have generated datasets containing
simulated astrophysical signals. The datasets cover a range of trial-DMs that
number from $N_{\mathrm{DM}}=128$ to 4096\. Each dataset consists of 5 minutes
of data with a sampling resolution of 64 $\upmu$s, 1024 channels, 400 MHz
bandwidth and a maximum frequency of 1581 MHz, similar to the observational
setup used by the ongoing SUPERB survey (Keane et al., 2018) which uses the
dedisp library. Trial-DMs start at 0 pc cm-3 and increment by 2 pc cm-3 up to
the maximum specified DM. These datasets are sufficiently large to neglect any
inaccuracies in CPU or GPU timers. Furthermore, for tdd and fdd the size of
the dataset requires splitting it into smaller batches where input and output
for one batch is overlapped with the computation of another batch.
Measurements were performed on a GPU node of the DAS-5 cluster (Bal et al.,
2016) and the hardware and software setup of the GPU node are listed in Table
1.

Table 1: The setup used for evaluation of the different implementations

CPU | Intel Xeon E5-2660 v3 (one socket, 10 cores/socket)
---|---
GPU | NVIDIA TITAN RTX 24 GiB GDDR6 (PCIe Gen3 16 lanes)
RAM | 128 GiB DDR4 2133 MHz
OS | CentOS 7 kernel 3.10.0-693 (64 bit)
Compiler | GNU GCC 8.3.0
GPU driver | 450.36.06
CUDA | CUDA version 11.0.1 kernels compiled for CC 7.5

Figure 2: Runtime distribution for dedisp, tdd and fdd for 1024 trial-DMs.
Postprocessing for fdd is optional and depends on the processing pipeline.
Figure 3: Comparison of runtime for dedisp, tdd and fdd for a range of trial-
DMs that number from 128 to 4096. The runtime for fdd includes postprocessing
to transform the output data to the time-domain.

Dedispersion is usually a step in a processing pipeline, but for this work we
have isolated the dedispersion implementation. We define the runtime of an
implementation as the recurring time of the implementation which is taken as
the summation of individual timings of components in the critical path. The
critical path for the runtime differs per implementation.

In Fig. 2 we show the distribution of runtimes for dedisp, tdd and fdd. We
find that in dedisp all components (input, preprocessing, dedispersion and
output) are executed serially. The time required for input and output of data
is left out of scope in the work by Barsdell et al. (2012) and Sclocco et al.
(2016). We however argue that the time spend on I/O _cannot_ be ignored for
dedisp, as the GPU is idle while the memory transfers between host and device
take place.

Our implementation of tdd, on the other hand, overlaps computation on the GPU
with I/O which is reflected by a lower over-all runtime of a factor 1.5 to 2,
depending on the amount of trial-DMs. Furthermore, the preprocessing time
(transpose and unpack) has been reduced while the timing for the dedispersion
kernel itself is almost the same as in dedisp.

In fdd, the input and output are overlapped as well. The dedispersion time has
been reduced significantly, but at the cost of longer preprocessing time
(transpose, unpack and FFT) and additional postprocessing time (FFT and
scaling). We note that most of the preprocessing time is spent performing the
FFT. We observe a similar distribution of runtime for other numbers of trial-
DMs. Figure 3 shows a comparison of runtime for a range of trial-DMs from 128
to 4096. For a low number of trial-DMs tdd is faster than fdd. Looking at the
runtime distribution for 128 to 512 DMs we observe that the preprocessing time
is almost equal from 128 until 512 DMs and then increases linearly with the
amount of DMs. From 512 DMs onward the runtime for fdd is up to 20% faster
than tdd. The performance for a low number of trial-DMs might be improved with
tuning of batch size parameters. However performance is mostly of interest for
larger numbers of trial-DMs. Furthermore, when profiling the GPU
implementations we observe that, despite part of the runtime of the fdd
implementation being limited by the available GPU memory444This is a
limitation of the memory size, not to be confused with a memory bandwidth
limitation. A larger memory size would allow for larger and consequently more
efficient batch sizes., the dedispersion algorithm itself is compute bound,
except for the FFT.

###  4.2 Energy to completion

Figure 4: Energy utilization per implementation for 1024 trial-DMs, measured
as energy to completion. Percentages shown are the percentage of either the
CPU or GPU energy utilization compared to the total per implementation.

In many applications the total energy consumption of the system is also an
important factor. A reduction of runtime of the implementation might lead to a
reduction in energy as well, but does not necessarily scale linearly as
compute utilization of the device and I/O operations also contribute to the
total energy utilization. We have measured the energy to completion for our
three implementations with the same setup used for the runtime measurements.
Energy usage on the CPU was measured with LIKWID (Gruber et al., 2021) and on
the GPU with NVIDIA NVML. Figure 4 shows the obtained energy measurements per
implementation. We find that, as one might expect, the energy to completion
was reduced significantly for both CPU and GPU between dedisp and tdd. The
implementation of tdd leads to the same result as dedisp at only 66% of the
energy utilization. Also the fdd implementation leads to a reduction in energy
to completion by another 5%. We observe that the GPU energy utilization was
reduced compared to tdd, at a slightly increased CPU energy utilization.

##  5 Discussion and conclusions

We have presented the concept of Fourier-domain dedispersion and shown that
this direct incoherent dedispersion algorithm is a viable alternative to the
traditional time-domain dedispersion. We have implemented the Fourier-domain
dedispersion algorithm (fdd) in the dedisp library by Barsdell et al. (2012).
An improved version of the dedisp time-domain dedispersion algorithm (tdd) was
also added to allow for a fair comparison.

We find that for a small number of DMs, the time-domain algorithm (tdd) is
faster than Fourier-domain dedispersion (fdd). However, for typical use cases
with a large number (hundreds) of DMs, fdd is the fastest dedispersion
implementation as it outperforms tdd by about 20%. We also show that the
energy to completion is lower for tdd and fdd than for dedisp. The energy to
completion for 1024 trial-DMs with fdd is only 61% of the energy utilization
of dedisp on the system used. Furthermore, where dedisp is largely I/O bound
(Sclocco et al., 2016), we were able to alleviate some of the I/O bottlenecks
in tdd: we optimized the memory transfers, and we re-implemented the kernels
that pre-process input data. While some parts of fdd (such as the FFTs) are
also I/O bound, the most dominant fdd dedispersion kernel is compute bound. We
therefore expect fdd to scale better with advances in GPU technology, where
compute performance typically increases much faster than memory bandwidth.

For pulsar surveys using FFT-based periodicity searches, the fdd algorithm
provides even higher performance compared to the dedisp and tdd algorithms.
For the time-domain dedispersion algorithms, the FFT-based periodicity
searches would perform an FFT of the dedispersed time-series to produce power
spectra to search of periodic signals. By using the fdd algorithm this FFT of
the periodicity search, as well as the fdd postprocessing step in which the
data is Fourier-transformed back to the time-domain can be omitted. Instead,
the amplitudes and phases of the dedispersed spectra from fdd can be directly
used for the subsequent processing steps of periodicity searches, like
Fourier-domain acceleration searches (Ransom, 2001). In any situation where
the number of DMs exceeds the number of (observing) frequency channels,
Fourier-domain dedispersion will require fewer FFTs for dedispersion compared
to time-domain dedispersion.

Besides the algorithmic optimizations proposed in 2.3, we consider some
optimizations that may further improve performance. First, the throughput of
the current implementation of Fourier-domain dedispersion depends on
parameters such as the data size (e.g. number of channels, number of DMs), as
well as the amount of available GPU memory and the PCIe bandwidth. Autotuning
(Sclocco et al., 2016) may be useful to find the best-performing set of tuning
parameters, such as the optimal batch dimensions.

Second, the current implementation performs all operations in 32-bit floating
point. The use of 16-bit half precision or even 8-bit integers would not only
double/quadruple the amount of samples that can be processed in one batch, it
would also allow the use of _tensor cores_ to perform the dedispersion (tensor
cores are special-function units in modern GPUs, that perform mixed-precision
matrix multiplications much faster than regular GPU cores). Unfortunately,
neither 8-bit FFTs, nor tensor-core dedispersion is simple to implement. In
another signal-processing application (a correlator), tensor cores are 5 to 10
times more (energy) efficient compared to regular GPU cores (Romein, 2021).

Furthermore, the performance of the Fourier-domain algorithm can be improved
by reusing phase rotations (Eqn. 4) when multiple observations with the same
observational setup are available. This is the case for telescope where
signals from multiple beams are obtained simultaneously, or for survey
observations with many similar observations. We note that this approach
reduces the number of DMs or time samples that can be processed in one batch,
as the amount of available GPU memory is limited, and requires careful tuning.

###### Acknowledgements.

We thank Scott Ransom for very constructive comments on the manuscript and are
indebted to Andrew Jameson for help incorporating fdd into the dedisp library.
This project received funding from the Netherlands Organization for Scientific
Research (NWO) through the Fourier-Domain Dedispersion (OCENW.XS.083) and the
DAS-5 (621.018.201) grants.

## References

  * Bal et al. (2016) Bal, H. et al. 2016, IEEE Computer, 49, 54
  * Barsdell et al. (2012) Barsdell, B. R., Bailes, M., Barnes, D. G., & Fluke, C. J. 2012, MNRAS, 422, 379–392
  * Bassa et al. (2017a) Bassa, C. G., Pleunis, Z., & Hessels, J. W. T. 2017a, Astronomy and Computing, 18, 40
  * Bassa et al. (2017b) Bassa, C. G., Pleunis, Z., Hessels, J. W. T., et al. 2017b, ApJ, 846, L20
  * Bhattacharyya et al. (2016) Bhattacharyya, B., Cooper, S., Malenta, M., et al. 2016, ApJ, 817
  * Bracewell (1986) Bracewell, R. N. 1986, The Fourier Transform and its applications
  * Gruber et al. (2021) Gruber, T., Eitzinger, J., Hager, G., & Wellein, G. 2021, RRZE-HPC/likwid: likwid-4.2.1
  * Hankins (1971) Hankins, T. H. 1971, ApJ, 169, 487
  * Hankins & Rickett (1975) Hankins, T. H. & Rickett, B. J. 1975, Methods in Computational Physics, 14, 55
  * Jameson & Barsdell (2019) Jameson, A. & Barsdell, B. R. 2019, Heimdall Transient Detection Pipeline
  * Kaspi & Kramer (2016) Kaspi, V. M. & Kramer, M. 2016, arXiv e-prints, arXiv:1602.07738
  * Keane et al. (2018) Keane, E. F., Barr, E. D., Jameson, A., et al. 2018, MNRAS, 473, 116
  * Keith et al. (2010) Keith, M. J., Jameson, A., van Straten, W., et al. 2010, MNRAS, 409, 619
  * Kulkarni (2020) Kulkarni, S. R. 2020, arXiv e-prints, arXiv:2007.02886
  * Lazarus et al. (2015) Lazarus, P., Brazier, A., Hessels, J. W. T., et al. 2015, ApJ, 812, 81
  * Lorimer & Kramer (2012) Lorimer, D. R. & Kramer, M. 2012, Handbook of Pulsar Astronomy (Cambridge University Press)
  * Lyon et al. (2016) Lyon, R. J., Stappers, B. W., Cooper, S., Brooke, J. M., & Knowles, J. D. 2016, MNRAS, 459, 1104
  * Manchester et al. (2001) Manchester, R. N., Lyne, A. G., Camilo, F., et al. 2001, MNRAS, 328, 17
  * Manchester & Taylor (1972) Manchester, R. N. & Taylor, J. H. 1972, Astrophys. Lett., 10, 67
  * Petroff et al. (2019) Petroff, E., Hessels, J. W. T., & Lorimer, D. R. 2019, A&A Rev., 27, 4
  * Pleunis et al. (2017) Pleunis, Z., Bassa, C. G., Hessels, J. W. T., et al. 2017, ApJ, 846, L19
  * Ransom (2001) Ransom, S. M. 2001, PhD thesis, Harvard University
  * Romein (2021) Romein, J. W. 2021, A&A, to appear
  * Sclocco et al. (2016) Sclocco, A., van Leeuwen, J., Bal, H., & van Nieuwpoort, R. 2016, Astronomy and Computing, 14
  * Sclocco et al. (2015) Sclocco, A., van Leeuwen, J., Bal, H. E., & van Nieuwpoort, R. V. 2015, in 2015 IEEE Global Conference on Signal and Information Processing (GlobalSIP), 468–472
  * Stovall et al. (2014) Stovall, K., Lynch, R. S., Ransom, S. M., et al. 2014, ApJ, 791, 67
  * Taylor (1974) Taylor, J. H. 1974, A&AS, 15, 367
  * Zackay & Ofek (2017) Zackay, B. & Ofek, E. O. 2017, ApJ, 835, 11


# Generic Event Boundary Detection in Video with Pyramid Features

Van Thong Huynh, Hyung-Jeong Yang, Guee-Sang Lee, Soo-Hyung Kim
Department of AI Convergence, Chonnam National University
<EMAIL_ADDRESS>Corresponding author

###### Abstract

Generic event boundary detection (GEBD) aims to split video into chunks at a
broad and diverse set of actions as humans naturally perceive event
boundaries. In this study, we present an approach that considers the
correlation between neighbor frames with pyramid feature maps in both spatial
and temporal dimensions to construct a framework for localizing generic events
in video. The features at multiple spatial dimensions of a pre-trained
ResNet-50 are exploited with different views in the temporal dimension to form
a temporal pyramid feature map. Based on that, the similarity between neighbor
frames is calculated and projected to build a temporal pyramid similarity
feature vector. A decoder with 1D convolution operations is used to decode
these similarities to a new representation that incorporates their temporal
relationship for later boundary score estimation. Extensive experiments
conducted on the GEBD benchmark dataset show the effectiveness of our system
and its variations, in which we outperformed the state-of-the-art approaches.
Additional experiments on TAPOS dataset, which contains long-form videos with
Olympic sport actions, demonstrated the effectiveness of our study compared to
others.

##  1 Introduction

Temporal action localization attempts to spot the range of action instances in
timestamps from the video stream with well-known including THUMOS [14],
ActivityNet [11], HACS [31], FineAction [21]. However, with the growth of
video contents, the number of classes is expanding, and the predefined target
classes could not cover completely. That can be resolved with semi-supervised
or unsupervised learning, but it still focuses on specific actions. To
increase generalization, Shou  _et al_. [23] constructed a new benchmark,
Generic Event Boundary Detection (GEBD), which increases the ability to detect
a board and diverse set of boundaries between taxonomy-free events, as humans
naturally perceive. Following cognitive studies [1], GEBD [23] considered four
high-level causes that involve changes in subject, action, shot, environment,
and object of interaction to form event boundaries. These changes can occur
with varying speed, duration and even do not involve any scene changes.

The area of GEBD is attracting considerable interest because of its
application in the field of video understanding. The findings of GEBD can be
applied to uniform sampling a video into a fixed number of frames based on
generic boundaries to achieve high classification accuracy [23]. It can also
provide a cue to select a set of frames for video summarization. Furthermore,
knowledge of GEBD is needed for understanding in downstream tasks [28]. For
example, boundary captioning means to generate description for the status
change at the boundary, or boundary grounding that required to locate the
boundary based on given description, or boundary caption-video retrieval aims
to retrieve the video containing boundaries with provided description from
video corpus.

Due to its taxonomy-free property, GEBD contains a variety of spatial and
temporal characteristics that lead to difficulties in localizing these events
as human perception. Still in an early stage, only a few studies have been
conducted to solve the GEBD problem. In previous studies [23, 25], the authors
considered a set of consecutive frames as input, and its label corresponds to
the label of the center frame. Although it reduced the computational burden
for a single forward pass, but needed to repeat that process many times for
the whole video and also lacked the relationships between frames in the longer
view. Besides that, Li  _et al_. [17] worked on the entire video but in the
compressed domain, which caused deficiencies and reduced their performance.

In this study, we consider the entire video as an input that takes advantage
of the complete semantic relationship between frames to localize event
boundaries, and divide the video into meaningful units. First, based on the
observation that the output from different layers of CNN-based architecture
contains its characteristic to distinguish with the others, we collected
features for each frame on multiple scales of spatial dimension, with a CNN
backbone on its way to downsampling the frame. Second, motivated by the fact
that two adjacent frames are almost similar in most of the cases in a video,
we compute the similarity between adjacent frames by exploiting the
correlation between frames in short and longer views together with different
dilation rates in 1D temporal convolutions. In addition, our work also
exploits the pyramid temporal relationship of the similarities to decode them
into boundary scores. In summary, the main contributions of this study are as
follows:

  * •

We exploited the characteristic at different spatial dimensions to distinguish
frames at multiple levels.

  * •

Taking advantage of dilation rate in 1D convolution operation, we propose to
use temporal pyramid similarity (TPS) module that builds upon this operation
with dilation rates to construct the similarity between frames in multiple
views to solve the GEBD problem.

  * •

To enrich the information in higher views, a residual connection is used to
integrate spatial and temporal information, and a 1D depthwise convolution is
inserted at the beginning of these views to incorporate local information to
them without increasing too much computational cost.

  * •

Thanks to stacking multiple 1D convolution layers with different dilation
rates to decode the similarity representation, our framework obtained
improvements in performance compared to the state-of-the-art approach.

##  2 Related Works

#### Temporal modeling for spatial features

Although 3D convolution is designed as a native architecture to process a
sequence of images [13, 26, 2], but it takes a lot of computing resources.
Leveraging 2D CNN to extract spatial features and then modeling temporal
relationship with other architectures (e.g., 1D CNN, LSTM), is one approach to
tackle that problem. The authors in [30] and [32] partitioned feature map to
various levels and applied pooling operation on partitions, then concatenate
results to achieve temporal pyramid features. Wang  _et al_. [27] was inspired
by DeepLab [4, 5], they deployed multiple parallel temporal convolutions with
different atrous rates to capture multiscale contextual features. MS-TCN++
[18] stacked multiple blocks of dual dilated temporal convolutions to capture
temporal dependencies in a sequence.

#### Temporal activity detection in video

Early methods [24, 7, 3, 19] formulate the task to proposal generation and
action classification. The generation of action proposals aims to find
candidate segments, including a pair of starting and ending boundaries, in a
long video that may contain actions. Shou  _et al_. [24] deployed 3D
convolution operations to classify segments generated from sliding windows of
varied length into two nodes (action or background). The authors in [7, 3, 19]
used 1D convolution operations to capture context information between features
sampling on multiple scales. TURN [9] divided a long video into video units
and calculated CNN features for each unit. TURN employed an anchor unit to
construct clip pyramid features and compute confidence scores. MGG [20]
deployed multiple perspectives of granularity, depending on visual
characteristics with embedded position information. Yang  _et al_. [29]
employed background constraints that take advantage of the rich information
about the action and background to suppress low quality proposals.

#### Generic event boundary detection

Shou  _et al_. [23] introduced the first benchmark dataset to detect generic
event boundaries, based on Kinetics-400 videos [33]. They extended the
boundary matching (BM) mechanism [19], which considered each pair of starting
and ending boundaries as a proposal and computed their confidence scores, to
determine these boundaries. The authors in [15] exploited the pairwise self-
similarity between video frames to construct a temporal self-similarity matrix
(TSM) for the entire video to use as an intermediate stage and integrated with
contrastive learning to produce boundary indices. Li  _et al_. [17] encoded
GOPs (group of pictures separated from video with modern codecs) based on
their I-frames and P-frames to form unified representations and explore
temporal dependence with the temporal contrastive module. DDM-Net [25]
leveraged multi-level features of space and scale for calculating the
difference at multiple scales and construct multi-level dense difference maps.
These maps are then aggregated and fused with progressive attention.

##  3 Method

To estimate the boundary score for every frame in a video with $T$ frames, the
proposed systems consider the similarity between each frame and their
neighbors with the encoded features in a multi-view manner. These similarities
are then decoded with 1D CNNs and post-processed with a Gaussian filter for
smoothing and reducing noise on the estimated scores. An illustration of our
system is shown in Fig. 1.

Figure 1: An overview of the proposed system.

###  3.1 Spatial Pyramid Features

Over the last decade, many deep learning architectures have been developed to
tackle the image classification task, which can be used as a base for the
other downstream tasks (e.g., activity recognition, video classification),
with continuous improvement over time. Some of them are considered standard
architectures when constructing the baseline model to solve the other
problems. Following prior works [23, 15, 25, 17], our network is built upon
ResNet-50 architecture [10], pre-trained on ImageNet, to construct spatial
features for each frame in video. Given a frame resized to $224\times 224$,
our model leverages the output from the last convolution blocks in which the
input is downsampled in a pyramid scheme by 4, 8, 16 and 32 times as they
contain the most detail and vital information for each size of spatial
dimensions. The average pooling is applied on the spatial dimensions of these
outputs to obtain the 1D feature vector for each of them, as illustrated in
Feature extractor part of Fig. 1. Based on that, this work takes advantage of
4 feature vectors of size $T\times d_{i}$ for a video with $T$ frames, where
$d_{i}=2^{i+7}$ with $i\in\\{1,2,3,4\\}$ as produced by ResNet-50.

###  3.2 Temporal Pyramid Similarity

Based on the observation that two adjacent frames are almost similar in most
cases when recording video, we perform 1D convolution operations with
different dilation rates separately to achieve multiple views changing from
short- to long-term correlation. Given a sequence $S$ and a filter $K$ of size
$2m+1$ with $m\in\mathbb{N}$, convolution $G$ of these inputs with dilation
rate $r$ is formulated as follows:

$G_{i}^{r}=\sum_{j=-m}^{m}K_{j}\cdot S_{i+r\cdot j}$ (1)

where $i+r\cdot j$ accounts for the dependence on the past and future of an
output at time $i$. This work deploys $n=4$ blocks with kernel size of $3$ and
dilation rates $r_{i}=2^{i-1}$ with $i\in\\{1,2,...,n\\}$, implies the system
can observe up to $8$ frames in the temporal dimension to form the
representations for obtaining the similarities between frames and their
neighbors. To exploit the richer relationship between short-term and long-term
correlation, for blocks with $r_{i}>1$, a 1D depthwise convolution [6] is
added at the beginning of the block to form the adjacent relationship (local
correlation), as presented in Tab. 1 with $ks$ is kernel size and $r$ is
dilation rate.

Table 1: Details of each Conv1D block $r_{i}$ in TPS module. Layer | $ks$ | $r$ | Included | Output size
---|---|---|---|---
input | $-$ | $-$ | ✓ | $T\times d_{k}$
depthwise conv1d | 3 | 1 | $r_{i}>1$ | $T\times d_{k}$
conv1d | $3$ | $r_{i}$ | ✓ | $T\times d_{k}$
layer norm | $-$ | $-$ | ✓ | $T\times d_{k}$
activation (gelu) | $-$ | $-$ | ✓ | $T\times d_{k}$

In addition, the output from blocks is also concatenated and projected to the
lower dimension as the same as input, in which the most important properties
are compressed to produce a comprehensive representation. In total, the system
uses $n+1$ feature vectors with each size of $T\times d_{k}$,
$k\in\\{1,2,3,4\\}$ corresponding to 4 stages in the feature extractor step,
to calculate the similarity between neighbor frames with Euclidean distance,
as done in [25], for each stage, as shown in Fig. 2. To incorporate the
difference in spatial dimension, we add a residual connection to connect each
stage output with its input before measuring the similarity between frames.
Formally, let $F_{k}^{i}$ is the output of the feature vector $I_{k}$ at stage
$k$ with dilation rate $r_{i}$, we obtain $n$ similarity vectors by the
following equation

$\displaystyle R_{k}^{i}$ $\displaystyle=\frac{F_{k}^{i}+I_{k}}{\lVert
F_{k}^{i}+I_{k}\rVert},$ (2) $\displaystyle\mathcal{D}(q)$
$\displaystyle=\lVert R_{k,t}^{i}-R_{k,t+q}^{i}\rVert^{2},$ (3) $\displaystyle
D_{k}^{i}$
$\displaystyle=[\mathcal{D}(-l),\dots,\mathcal{D}(-1),\mathcal{D}(1),\dots,\mathcal{D}(l)],$
(4)

with $\mathcal{D}(q)$ denotes the Euclidean distance between frame at time $t$
and $t+q$. In this work, we consider neighbors in 1 second for each side, in
other words, $l$ equals the frames per second of video. And $R_{k}^{n+1}$, the
comprehensive representation of $F_{k}^{i}$, is calculated as

$\displaystyle P$
$\displaystyle=\mathcal{F}_{p}([F_{k}^{1},F_{k}^{2},\dots,F_{k}^{n}]),$ (5)
$\displaystyle R_{k}^{n+1}$ $\displaystyle=\frac{P}{\lVert P\rVert},$ (6)

where $\mathcal{F}_{p}$ is a 1D convolution with kernel size of $1$, and $P$
has the same dimension with $F_{k}^{i}$. These similarities, $R_{k}$, are
concatenated and projected into higher dimension,

$\mathcal{T}_{k}=\mathcal{F}_{t}([R_{k}^{1},R_{k}^{2},\dots,R_{k}^{n},R_{k}^{n+1}]),$
(7)

with $\mathcal{F}_{t}$ is 1D convolution with $d_{out}$ filters of kernel size
of $1$. Consequently, this module formed a feature vector, $\mathcal{T}_{k}$,
of size $T\times d_{out}$ for each stage output. At this point, we concatenate
the feature vectors of 4 stages and project to a new dimension to obtain a
final feature vector, $\mathcal{T}$, as

$\mathcal{T}=\mathcal{F}([\mathcal{T}_{1},\mathcal{T}_{2},\mathcal{T}_{3},\mathcal{T}_{4}]),$
(8)

in which, the projection $\mathcal{F}$ is a 1D convolution block with
$d_{out}$ filters of kernel size $3$ with aim to reduce the dimension of the
concatenation vector and incorporate these information together.

Figure 2: Temporal pyramid similarity module.

###  3.3 Similarity Decoder

To decode the similarity information for further prediction, $n_{s}$ 1D
convolution blocks with dilation rates $r_{i}$, $i\in\\{1,2,...,n_{s}\\}$, are
stacked together to form a similarity decoder (SD) block. Due to dilation rate
of $1$ is applied to the previous projection block, this module employs
dilation rates greater than $1$ and does not include 1D depthwise
convolutions. As 1D convolutions are stacked instead of applying separately as
in the previous module, this decoder can observe more frames to compute the
output at each timestamp. For simplicity, the same dilation rates as in the
temporal pyramid similarity module are used. In detail, $n_{s}=n-1$ for this
part and $r_{i}=2^{i}$. Formally, the output $\mathcal{S}$ of SD block is
defined as

$\mathcal{S}=\left(\mathcal{F}^{r_{n_{s}}}\circ\mathcal{F}^{r_{n_{s}-1}}\circ\cdots\circ\mathcal{F}^{r_{1}}\right)(\mathcal{T}),$
(9)

with $\mathcal{T}$ from Eq. 8, and $\mathcal{F}^{r_{i}}$ denotes the 1D
convolution block with dilation $r_{i}$.

###  3.4 Prediction Head

Our network uses two layers of 1D convolutions, followed by a sigmoid
function, to estimate the boundary score for each frame. These scores tend to
be noisy and difficult to use only a fixed-threshold (e.g., 0.5) to get exact
boundaries. To address these issues, we use a Gaussian kernel with standard
deviation of $1$ and window size equal to the number of frames in 1 second, as
shown in Fig. 3. Then each score is compared to neighbors in $0.5$ seconds and
a minimum threshold of $0.1$, which works similar to max pooling, to consider
as a maximum (boundary) or not.

##  4 Experimental Results

Table 2: Kinetics-GEBD validation score with ResNet50 as the backbone at
Rel.Dis.

Method |<EMAIL_ADDRESS>|<EMAIL_ADDRESS>|<EMAIL_ADDRESS>|<EMAIL_ADDRESS>|<EMAIL_ADDRESS>|<EMAIL_ADDRESS>|<EMAIL_ADDRESS>|<EMAIL_ADDRESS>|<EMAIL_ADDRESS>|<EMAIL_ADDRESS>| Avg
---|---|---|---|---|---|---|---|---|---|---|---
BMN [19] | 0.186 | 0.204 | 0.213 | 0.220 | 0.226 | 0.230 | 0.233 | 0.237 | 0.239 | 0.241 | 0.223
BMN-StartEnd [19] | 0.491 | 0.589 | 0.627 | 0.648 | 0.660 | 0.668 | 0.674 | 0.678 | 0.681 | 0.683 | 0.640
TCN [16] | 0.588 | 0.657 | 0.679 | 0.691 | 0.698 | 0.703 | 0.706 | 0.708 | 0.710 | 0.712 | 0.685
PC [23] | 0.625 | 0.758 | 0.804 | 0.829 | 0.844 | 0.853 | 0.859 | 0.864 | 0.867 | 0.870 | 0.817
SBoCo [15] | 0.732 | 0.827 | 0.853 | 0.877 | 0.882 | 0.891 | 0.894 | 0.899 | 0.899 | 0.907 | 0.866
DDM-Net [25] | 0.764 | 0.843 | 0.866 | 0.880 | 0.887 | 0.892 | 0.895 | 0.898 | 0.900 | 0.902 | 0.873
CVRL [17] | 0.743 | 0.830 | 0.857 | 0.872 | 0.880 | 0.886 | 0.890 | 0.893 | 0.896 | 0.898 | 0.865
Ours | 0.770 | 0.842 | 0.863 | 0.874 | 0.880 | 0.884 | 0.887 | 0.890 | 0.891 | 0.893 | 0.867

Table 3: TAPOS validation score compared to previous works.

Method |<EMAIL_ADDRESS>|<EMAIL_ADDRESS>|<EMAIL_ADDRESS>|<EMAIL_ADDRESS>|<EMAIL_ADDRESS>|<EMAIL_ADDRESS>|<EMAIL_ADDRESS>|<EMAIL_ADDRESS>|<EMAIL_ADDRESS>|<EMAIL_ADDRESS>| Avg
---|---|---|---|---|---|---|---|---|---|---|---
ISBA [8] | 0.106 | 0.170 | 0.227 | 0.265 | 0.298 | 0.326 | 0.348 | 0.369 | 0.382 | 0.396 | 0.302
TCN [16] | 0.237 | 0.312 | 0.331 | 0.339 | 0.342 | 0.344 | 0.347 | 0.348 | 0.348 | 0.348 | 0.330
CTM [12] | 0.244 | 0.312 | 0.336 | 0.351 | 0.361 | 0.369 | 0.374 | 0.381 | 0.383 | 0.385 | 0.350
TransParser [22] | 0.289 | 0.381 | 0.435 | 0.475 | 0.500 | 0.514 | 0.527 | 0.534 | 0.540 | 0.545 | 0.474
PC [23] | 0.522 | 0.595 | 0.628 | 0.646 | 0.659 | 0.665 | 0.671 | 0.676 | 0.679 | 0.683 | 0.642
DDM-Net [25] | 0.604 | 0.681 | 0.715 | 0.735 | 0.747 | 0.753 | 0.757 | 0.760 | 0.763 | 0.767 | 0.728
Ours | 0.616 | 0.701 | 0.740 | 0.760 | 0.772 | 0.780 | 0.786 | 0.790 | 0.794 | 0.796 | 0.754

###  4.1 Dataset and Evaluation

#### Kinetics-GEBD

Our approach is evaluated primarily on Kinetics-GEBD, a benchmark dataset for
locating the boundaries of generic events in video, created by Shou  _et al_.
[23]. It consists of around $60\,000$ 10-second videos, with the training,
validation, and testing partition ratio approximately 1:1:1. The training and
testing videos are randomly selected from Kinetics-400 [33] Train split, while
the validation set used all videos in Kinetics-400 Val split. Each video is
annotated by 5 annotators with average 4.77 boundaries. Due to the annotation
of test set not available publicly, our network is trained on training set and
evaluated on validation set.

#### TAPOS

In addition, we conduct experiment on TAPOS dataset [22] containing Olympics
sport videos across 21 actions. Follow [23], we perform boundaries
localization between sub-actions in each action instance, as annotated in
[22]. The average duration of instances is 9.4 seconds, and maximum around 5
minutes. TAPOS contains $13\,094$ action instances for training and $1790$
instances for validation. Due to the variety of duration over instances, we
split instances into 10-second clips with overlap of 5 seconds to perform
training and evaluation. During evaluation, the boundary scores from clips are
merged with summation to form a unique score for each timestamp in the whole
action instance to get the final boundaries. Similar to Kinetics-GEBD, our
network is trained on the training set and evaluated on the validation set.

#### Evaluation

Following [23], F1 score with Relative Distance (Rel.Dis.) measurement is used
to evaluate the system performance. Rel.Dis. determines whether a detection is
correct or incorrect by calculating the error of detection, $E_{d}$, as

$E_{d}=\frac{\lvert\mathrm{Detected~{}point}-\mathrm{Ground~{}truth~{}point}\rvert}{\mathrm{Length~{}of~{}video}},$
(10)

and comparing with a threshold, $\tau$. If $E_{d}\leq\tau$, the detection is
correct, otherwise incorrect. The reported results are with thresholds $\tau$
from $0.05$ to $0.5$ with a gap of $0.05$.

###  4.2 Implementation

Because of the variety of frame rate of each video, we sample frames at 25 fps
(frame per second), as done in [2]. For this task, video at 25 fps may contain
redundant information. We then sample video to 5 fps that led to 50 frames per
video or $T=50$, which is suitable to run the inference on a usual computing
resource. The network is built with TensorFlow 2.9 and trained end-to-end with
ResNet-50 backbone and Adam optimizer at batch size of $8$ in 10 epochs. We
linearly increased the learning rate from 0 to $4\text{\times}{10}^{-4}$ in 2
epochs, and then used a cosine decay schedule to reduce the learning rate to
$4\text{\times}{10}^{-6}$ in 8 epochs. The model from the last epoch is used
to generate the prediction for evaluation. All experiments are conducted on a
single NVIDIA RTX 3090 GPU equipped machine.

###  4.3 Results

Table 2 shows the comparison of our approach with the previous method on the
Kinetics-GEBD validation set with the same spatial backbone (ResNet-50).
Compared to PC [23], the GEBD benchmark baseline, our system achieves a huge
improvement of almost $14.5$ percent, demonstrating the effectiveness of the
proposed system. DDM-Net [25] took consecutive frames with a gap between them
that created a longer view in their system. Consequently, they also surpass
the baseline, but still has a small gap, $0.6$ percent, with our method at
strict threshold $\tau=0.05$ as multiple longer views are exploited in our
method. We also achieved a remarkable improvement compared to CVRL [17] and
SBoCo [15] with around $3$ percent at $\tau=0.05$.

The results of the comparison between our method and previous methods on TAPOS
are summarized in Tab. 3. Since our approach is able to learn different
context in multiple view, we surpass PC [23] and DDM-Net [25] as TAPOS
contains instances with duration up to 5 minutes. We obtain an increasing of
around $8.2\%$ and $1.2\%$ on<EMAIL_ADDRESS>compared to PC and DDM-Net,
respectively. These results prove the effectiveness of our method on short
videos, around 10 seconds, and long videos with up to 5 minutes.

###  4.4 Ablation study

To evaluate the effectiveness of our approach, several ablations on the
network are conducted with the same training mechanism.

#### Effects of stages in the feature extractor

We experiment our system with different number of feature maps to analyze how
it effects the overall performance. As high-level feature maps (high stages)
contain more specific information compared to low-level feature maps (low
stages), we add/remove feature maps from high to low in an orderly way as in
Tab. 4. The system has increased steadily in terms of<EMAIL_ADDRESS>with a
difference of up to $1.22$ percent between the single stage and the
combination of 4 stages. This can be explained because each feature map level
contains its own knowledge that makes the different neighboring frames, which
makes their combination useful for the whole system.

Table 4: Effects of the combination of stages in feature extractor with results on Kinetics-GEBD. Stage |<EMAIL_ADDRESS>|<EMAIL_ADDRESS>|<EMAIL_ADDRESS>
---|---|---|---
Stage 4 | 0.6965 | 0.8330 | 0.7587
Stage 3, 4 | 0.6967 | 0.8462 | 0.7642
Stage 2, 3, 4 | 0.7026 | 0.8460 | 0.7677
Stage 1, 2, 3, 4 | 0.7076 | 0.8450 | 0.7702
Stage 1, 2, 3 | 0.7015 | 0.8491 | 0.7683
Stage 1, 2 | 0.6956 | 0.8514 | 0.7657
Stage 1 | 0.6924 | 0.8373 | 0.7580

#### Effects of dilation rates

Table 5 shows the results when adding dilation rates of $2^{i-1}$ to compute
the similarity in TPS module. In general, F1 scores increase consistently
around $0.1$ percent for each new dilation rate and reach a total gain of
$0.35\%$ based on the improvement $0.5\%$ of precision. Although these
increases are not remarkable, they still show a potential further extension of
how to incorporate them in an efficient manner and achieve significant
improvements.

Table 5: Effects of dilation rates in the TPS module with results on Kinetics-
GEBD.

Modification |<EMAIL_ADDRESS>|<EMAIL_ADDRESS>|<EMAIL_ADDRESS>
---|---|---|---
$n=1,r_{i}=2^{i-1}$ | 0.7013 | 0.8455 | 0.7667
$n=2,r_{i}=2^{i-1}$ | 0.7045 | 0.8438 | 0.7679
$n=3,r_{i}=2^{i-1}$ | 0.7037 | 0.8472 | 0.7688
$n=4,r_{i}=2^{i-1}$ | 0.7076 | 0.8450 | 0.7702

#### Effects of local correlation

In this study, we incorporate residual connection (Res.Con.) and 1D depthwise
convolution to enrich long-term correlation with local contextual information
in the TPS module. Table 6 shows their effects on overall system performance.
According to that, Res.Con. achieves an increase of $1.2\%$, while the other
does not have many effects. Consequently, it encourages refinement of features
with local context in a whole view with Res.Con. to improve the generalization
instead of on individual channels as to what depthwise convolution is doing.

Table 6: Effects of local correlation in TPS module with results on Kinetics-
GEBD, in which Res.Con. indicates residual connection.

Depthwise conv1d |  Res.Con. |<EMAIL_ADDRESS>|<EMAIL_ADDRESS>|<EMAIL_ADDRESS>
---|---|---|---|---
|  | 0.6943 | 0.8361 | 0.7586
|  ✓ | 0.7063 | 0.8470 | 0.7703
✓ |  | 0.6933 | 0.8373 | 0.7585
✓ |  ✓ | 0.7076 | 0.8450 | 0.7702

#### Effects of similarity decoder block

We conduct experiments with a fixed dilation rate in SD block along with the
removal of SD block to compare their results, as shown in Tab. 7. Generally,
SD block helps improve performance at least $0.8\%$ with no dilation and at
most $1.2\%$ with pyramid dilation rates. The dilation rates can improve the
performance, but large dilation can also cause the drop. In case of the
pyramid dilation rates, it creates balances by increasing the receptive field
step by step, which consistently boosts the performance.

Table 7: Effects of similarity decoder block with results on Kinetics-GEBD. In
these experiments, $n_{s}$ is set to $3$.

Modification |<EMAIL_ADDRESS>|<EMAIL_ADDRESS>|<EMAIL_ADDRESS>
---|---|---|---
Excluded SD block | 0.6998 | 0.8280 | 0.7585
SD block $r_{i}=1$ | 0.7059 | 0.8395 | 0.7669
SD block $r_{i}=2$ | 0.7036 | 0.8462 | 0.7684
SD block $r_{i}=4$ | 0.7059 | 0.8443 | 0.7689
SD block $r_{i}=8$ | 0.7060 | 0.8333 | 0.7644
SD block $r_{i}=2^{i}$ | 0.7076 | 0.8450 | 0.7702

Table 8: Effects of Gaussian smoothing in prediction head with results on
Kinetics-GEBD.

Gaussian smoothing |<EMAIL_ADDRESS>|<EMAIL_ADDRESS>|<EMAIL_ADDRESS>
---|---|---|---
Training | Inference
|  | 0.6717 | 0.8631 | 0.7555
| ✓ | 0.7426 | 0.8038 | 0.7720
✓ |  | 0.6206 | 0.9103 | 0.7381
✓ | ✓ | 0.7076 | 0.8450 | 0.7702

Figure 3: A visualization of detected boundaries with or without GF smoothing.

#### Effects of Gaussian smoothing in prediction head

In this work, the Gaussian filter (GF) is attached and runs on the GPU
together with the other parts of the network during training and inference. If
GF includes in training, the learning process will optimize the smoothing
scores instead of the raw ones. To see its impacts on the overall performance,
we do include and exclude it during training and inference, respectively. As
we can see in Tab. 8, the recall is very high compared to precision when GF is
excluded for inference, which makes more noise in scores with many local
maximums and produces many boundaries. On the other hand, the network focus on
the smoothing score and cause a huge loss, nearly $3.21\%$ that twice compared
to the other ($1.65\%$). Figure 3 shows an example result in which GF reduced
4 false positive detection with Rel.Dis. threshold of $0.05$.

##  5 Conclusion

In this work, we proposed an approach based on temporal pyramid similarity to
detect the boundary of generic events in video. Our work exploited the
correlation between frames in different temporal and spatial scales in a
pyramid scheme to compute the distance between adjacent frames, parallelly.
Together, we applied this operation on multiple spatial scales to combine the
unique characteristics on each scale to improve the system performance. Thanks
to 1D operation for temporal modeling, we explored the temporal dimension to
decode the similarity information and achieve a considerable improvement for
the whole system. Our method outperforms the previous state-of-the-art methods
on the Kinetics-GEBD and TAPOS benchmarks at a strict relative distance. More
research is needed to expand our method so that it can work with untrimmed
videos, as we only evaluate our approach for trimmed video with action.

## References

  * [1] Roger G Barker and Herbert F Wright.  Midwest and its children: The psychological ecology of an american town.  1955\.
  * [2] Joao Carreira and Andrew Zisserman.  Quo vadis, action recognition? a new model and the kinetics dataset.  In CVPR, July 2017.
  * [3] Yu-Wei Chao, Sudheendra Vijayanarasimhan, Bryan Seybold, David A Ross, Jia Deng, and Rahul Sukthankar.  Rethinking the faster r-cnn architecture for temporal action localization.  In CVPR, pages 1130–1139, 2018.
  * [4] Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan L Yuille.  Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs.  IEEE TPAMI, 40(4):834–848, 2017.
  * [5] Liang-Chieh Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam.  Encoder-decoder with atrous separable convolution for semantic image segmentation.  In ECCV, pages 801–818, 2018.
  * [6] François Chollet.  Xception: Deep learning with depthwise separable convolutions.  In CVPR, pages 1251–1258, 2017.
  * [7] Xiyang Dai, Bharat Singh, Guyue Zhang, Larry S Davis, and Yan Qiu Chen.  Temporal context network for activity localization in videos.  In ICCV, pages 5793–5802, 2017.
  * [8] Li Ding and Chenliang Xu.  Weakly-supervised action segmentation with iterative soft boundary assignment.  In CVPR, pages 6508–6516, 2018.
  * [9] Jiyang Gao, Zhenheng Yang, Kan Chen, Chen Sun, and Ram Nevatia.  Turn tap: Temporal unit regression network for temporal action proposals.  In ICCV, pages 3628–3636, 2017.
  * [10] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In CVPR, pages 770–778, 2016.
  * [11] Fabian Caba Heilbron, Victor Escorcia, Bernard Ghanem, and Juan Carlos Niebles.  ActivityNet: A large-scale video benchmark for human activity understanding.  In CVPR, jun 2015.
  * [12] De-An Huang, Li Fei-Fei, and Juan Carlos Niebles.  Connectionist temporal modeling for weakly supervised action labeling.  In ECCV, pages 137–153. Springer, 2016.
  * [13] Shuiwang Ji, Wei Xu, Ming Yang, and Kai Yu.  3d convolutional neural networks for human action recognition.  IEEE TPAMI, 35(1):221–231, 2012.
  * [14] Y.-G. Jiang, J. Liu, A. Roshan Zamir, G. Toderici, I. Laptev, M. Shah, and R. Sukthankar.  THUMOS challenge: Action recognition with a large number of classes.  http://crcv.ucf.edu/THUMOS14/, 2014.
  * [15] Hyolim Kang, Jinwoo Kim, Taehyun Kim, and Seon Joo Kim.  Uboco: Unsupervised boundary contrastive learning for generic event boundary detection.  In CVPR, pages 20073–20082, June 2022.
  * [16] Colin Lea, Austin Reiter, René Vidal, and Gregory D Hager.  Segmental spatiotemporal cnns for fine-grained action segmentation.  In ECCV, pages 36–52. Springer, 2016.
  * [17] Congcong Li, Xinyao Wang, Longyin Wen, Dexiang Hong, Tiejian Luo, and Libo Zhang.  End-to-end compressed video representation learning for generic event boundary detection.  In CVPR, pages 13967–13976, June 2022.
  * [18] Shi-Jie Li, Yazan AbuFarha, Yun Liu, Ming-Ming Cheng, and Juergen Gall.  Ms-tcn++: Multi-stage temporal convolutional network for action segmentation.  IEEE Transactions on Pattern Analysis and Machine Intelligence, pages 1–1, 2020.
  * [19] Tianwei Lin, Xiao Liu, Xin Li, Errui Ding, and Shilei Wen.  BMN: Boundary-matching network for temporal action proposal generation.  In ICCV, oct 2019.
  * [20] Yuan Liu, Lin Ma, Yifeng Zhang, Wei Liu, and Shih-Fu Chang.  Multi-granularity generator for temporal action proposal.  In CVPR, pages 3604–3613, 2019.
  * [21] Yi Liu, Limin Wang, Xiao Ma, Yali Wang, and Yu Qiao.  Fineaction: A fine-grained video dataset for temporal action localization.  May 2021.
  * [22] Dian Shao, Yue Zhao, Bo Dai, and Dahua Lin.  Intra-and inter-action understanding via temporal action parsing.  In CVPR, pages 730–739, 2020.
  * [23] M. Z. Shou, S. W. Lei, W. Wang, D. Ghadiyaram, and M. Feiszli.  Generic event boundary detection: A benchmark for event segmentation.  In ICCV, oct 2021.
  * [24] Zheng Shou, Dongang Wang, and Shih-Fu Chang.  Temporal action localization in untrimmed videos via multi-stage cnns.  In CVPR, pages 1049–1058, 2016.
  * [25] Jiaqi Tang, Zhaoyang Liu, Chen Qian, Wayne Wu, and Limin Wang.  Progressive attention on multi-level dense difference maps for generic event boundary detection.  In CVPR, pages 3355–3364, June 2022.
  * [26] Du Tran, Lubomir Bourdev, Rob Fergus, Lorenzo Torresani, and Manohar Paluri.  Learning spatiotemporal features with 3d convolutional networks.  In ICCV, pages 4489–4497, 2015.
  * [27] Jiahao Wang, Zhengyin Du, Annan Li, and Yunhong Wang.  Atrous temporal convolutional network for video action segmentation.  In ICIP, sep 2019.
  * [28] Yuxuan Wang, Difei Gao, Licheng Yu, Stan Weixian Lei, Matt Feiszli, and Mike Zheng Shou.  Geb+: A benchmark for generic event boundary captioning, grounding and text-based retrieval.  In ECCV, 2022.
  * [29] Haosen Yang, Wenhao Wu, Lining Wang, Sheng Jin, Boyang Xia, Hongxun Yao, and Hujie Huang.  Temporal action proposal generation with background constraint.  In AAAI, volume 36, pages 3054–3062, 2022.
  * [30] Zhesong Yu, Xiaoshuo Xu, Xiaoou Chen, and Deshun Yang.  Temporal pyramid pooling convolutional neural network for cover song identification.  In IJCAI, aug 2019.
  * [31] Hang Zhao, Antonio Torralba, Lorenzo Torresani, and Zhicheng Yan.  HACS: Human action clips and segments dataset for recognition and temporal localization.  In ICCV, oct 2019.
  * [32] Zhenxing Zheng, Gaoyun An, Dapeng Wu, and Qiuqi Ruan.  Spatial-temporal pyramid based convolutional neural network for action recognition.  Neurocomputing, 358:446–455, sep 2019.
  * [33] Andrew Zisserman, Joao Carreira, Karen Simonyan, Will Kay, Brian Zhang, Chloe Hillier, Sudheendra Vijayanarasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, and Mustafa Suleyman.  The kinetics human action video dataset.  2017\.