# Revisiting Foreground and Background Separation in Weakly-supervised
Temporal Action Localization: A Clustering-based Approach

Qinying Liu1 Zilei Wang 1 Shenghai Rong 1 Junjie Li 1 Yixin Zhang1,2
1 University of Science and Technology of China
2 Institute of Artificial Intelligence, Hefei Comprehensive National Science
Center
{lydyc, rongsh, hnljj<EMAIL_ADDRESS>{zhyx12<EMAIL_ADDRESS>Corresponding author

###### Abstract

Weakly-supervised temporal action localization aims to localize action
instances in videos with only video-level action labels. Existing methods
mainly embrace a localization-by-classification pipeline that optimizes the
snippet-level prediction with a video classification loss. However, this
formulation suffers from the discrepancy between classification and detection,
resulting in inaccurate separation of foreground and background (F&B)
snippets. To alleviate this problem, we propose to explore the underlying
structure among the snippets by resorting to unsupervised snippet clustering,
rather than heavily relying on the video classification loss. Specifically, we
propose a novel clustering-based F&B separation algorithm. It comprises two
core components: a snippet clustering component that groups the snippets into
multiple latent clusters and a cluster classification component that further
classifies the cluster as foreground or background. As there are no ground-
truth labels to train these two components, we introduce a unified self-
labeling mechanism based on optimal transport to produce high-quality pseudo-
labels that match several plausible prior distributions. This ensures that the
cluster assignments of the snippets can be accurately associated with their
F&B labels, thereby boosting the F&B separation. We evaluate our method on
three benchmarks: THUMOS14, ActivityNet v1.2 and v1.3. Our method achieves
promising performance on all three benchmarks while being significantly more
lightweight than previous methods. Code is available at
https://github.com/Qinying-Liu/CASE

##  1 Introduction

Temporal action localization (TAL) [53, 34, 37] is a task to localize the
temporal boundaries of action instances and recognize their categories in
videos. In recent years, numerous works put effort into the fully supervised
manner and gain great achievements. Albeit successful, these methods require
extensive manual frame-level annotations, which are expensive and time-
consuming. Without the requirement of frame-level annotations, weakly-
supervised TAL (WTAL) has received increasing attention, as it allows us to
detect the action instances with only video-level action labels.

Figure 1: Conceptual illustration of our clustering-based F&B separation
algorithm. In snippet clustering, we partition the snippets (or frames) into
multiple clusters with explicit characteristics. In cluster classification, we
classify the clusters as foreground or background. The above results are
attained according to the predictions of our method.

There has been a wide spectrum of WTAL methods developed in the literature
[59, 70, 46, 36]. With only video-level labels, mainstream methods employ a
localization-by-classification pipeline, which formulates WTAL as a video
action classification problem to learn a temporal class activation sequence
(T-CAS). For this pipeline, foreground (i.e., action) and background
separation remains an open question since video-level labels do not provide
any cue for background class. There are two types of existing approaches to
solve it. The first type [59, 70] is based on the multiple instance learning
(MIL), which uses the T-CAS to select the most confident snippets for each
action class. The second type [46, 33] introduces an attention mechanism to
learn class-agnostic foreground weights that indicate the probabilities of the
snippets belonging to foreground. Despite recent progress, these methods
typically rely on the video classification loss to guide the learning of the
T-CAS or the attention weights. There is an inherent downside: the loss is
easily minimized by the salient snippets [42] and fails to explore the
distribution of the whole snippets, resulting in erroneous T-CAS or attention
weights. This issue is rooted in the supervision gap between the
classification and detection tasks. Recent studies [48, 40] are devoted to
producing snippet-level pseudo-labels to bridge the gap. However, the pseudo-
labels are still derived from the unreliable T-CAS or attention weights.

Deep clustering [8], which automatically partitions the samples into different
groups, has been proven to be capable of revealing the intrinsic distribution
of the samples in many label-scarce tasks [1, 5, 12, 30, 35]. A natural issue
arises: is it possible to adopt the clustering to capture the distribution of
snippets? Since clustering can be conducted in a self-supervised manner, it is
immune to the video classification loss. This suggests a great potential of
clustering for F&B separation in WTAL. A brute-force solution would be to
group the snippets into two clusters, one for foreground and one for
background. Whereas, we empirically find that it underperforms in practice
(cf. Sec. 5.3). We argue that the reason is that snippets, regardless of
foreground or background, can differ dramatically in appearance (cf. Fig. 1
(a)). As a result, it may be difficult for a self-supervised model to group
them accurately. Fortunately, in real-world videos, there are common
characteristics (e.g., ”interview”, ”running”) shared by a group of snippets
(cf. Fig. 1 (b)). Compared to learning two clusters for F&B in the complex
video content, it may be easier to explore the snippet clusters with clear and
distinctive characteristics. This necessitates a clustering algorithm with
multiple clusters. Furthermore, it can be observed that the characteristics of
clusters are sometimes indicative cues for F&B separation. For example, we can
confidently classify the ”running” cluster to foreground and the ”interview”
cluster to background according to the cluster-level characteristics.
Consequently, it is promising to further leverage the cluster-level
representations to assist F&B separation.

In light of the above discussion, we propose a novel Clustering-Assisted F&B
SEparation (CASE) network. We begin by constructing a standard WTAL baseline
that provides a primary estimation of F&B snippets. We then introduce a
clustering-based F&B separation algorithm (cf. Fig. 1) to refine the F&B
separation. This algorithm is comprised of two main components: snippet
clustering for dividing the snippets into multiple clusters, and cluster
classification for classifying the clusters as foreground or background.
Considering that no ground-truth labels are available to train the components,
we propose a unified self-labeling mechanism to generate high-quality pseudo-
labels for them. Specifically, we formulate the label assignment in both
components as a unified optimal transport problem, which allows us to flexibly
impose several customized constraints on the distribution of pseudo-labels.
After training these two components, we can transform the cluster assignments
of the snippets to their F&B assignments, which can be used to refine the F&B
separation of the baseline.

It is demonstrated that our method yields favorable performance while being
much more lightweight compared to prior approaches. In summary, our
contributions are three-fold. 1) We propose a clustering-based F&B separation
algorithm for WTAL, which casts the problem of F&B separation as a combination
of snippet clustering and cluster classification. 2) We propose a unified
self-labeling mechanism based on optimal transport to guide snippet clustering
and cluster classification. 3) We conduct extensive experiments that
demonstrate the effectiveness and efficiency of our method compared to
existing approaches.

##  2 Related Work

Deep clustering. Current deep clustering approaches [66, 65, 11] could be
roughly divided into two categories. The first one iteratively computes the
clustering assignment from the up-to-date model and supervises the network
training processes by the estimated information [62, 63, 8, 4, 7, 61].
DeepCluster [62] is a typical method that iteratively groups the features and
uses the subsequent assignments to update the deep network. The second one
simultaneously learns feature representation and clustering assignment [15,
25, 18], which has gained popularity in recent years. Asano et al. [1] propose
to enforce a balanced label assignment constraint to avoid degenerate
solution. Caron et al. [5] use the algorithm in [1] to introduce a swapped
mechanism that employs two random transformations of the same images to guide
each other. In this work, we extend [1] from image classification to WTAL, and
incorporate it with task-specific designs, _e.g_., imposing multiple sensible
constraints on the distribution of pseudo-labels. It is worth noting that in
the above methods, the number of clusters is typically set to the number of
ground-truth (GT) classes so that clusters and GT classes can be mapped one-
by-one during testing [57, 4]. There are attempts that utilize an extra over-
clustering technique to learn a larger number of clusters than GT, which is
believed to be conducive to representation learning [25, 12]. However, these
methods commonly treat the technique as an auxiliary tool independent of their
main task. In contrast, we are committed to building an explicit
correspondence between learned clusters and F&B classes, thus unleashing the
full potential of clustering in the WTAL task.

Weakly-supervised temporal action localization. Existing WTAL approaches can
be categorized into four broad groups. The first group aims to improve feature
discrimination ability. Various techniques, _e.g_., deep metric learning [42,
44] and contrastive learning [70, 29], have been explored. The second group
seeks to discover complete action regions. [42, 54, 72] hide some snippets to
press the models in exploring more action regions, while [33, 24] use a multi-
branch framework to discover complementary snippets. The third group is
concerned with learning attention weights. [69, 45] design losses to
regularize the values of the attention weights. [48, 40] generate pseudo-
labels for them. However, the pseudo-labels are derived from the primary
predictions of snippets, which are still optimized using the video
classification loss. The last group is the most closely related to ours, which
introduces auxiliary classes in addition to the action classes. [51]
introduces a video-level context class. [39, 58, 33] mine the action units or
sub-actions shared across action categories. Class-specific sub-action is
explored in [21, 20]. Recently, [31] learns a set of visual concepts for fine-
grained action localization. Our method is superior to these methods in three
noticeable aspects. 1) These methods rely on video-level supervision to
discover the auxiliary classes. Conversely, we develop the clusters in a self-
supervised manner that is orthogonal to video-level supervision. 2) These
methods devise multiple loss terms to regularize the auxiliary classes. In
contrast, we introduce regularization into optimal transport, which can be
resolved in a principled way. 3) Our method significantly outperforms these
methods.

##  3 Preliminaries and Baseline Setup

In each training iteration, we first sample a mini-batch of $B$ videos. For
each video, we have access only to its video-level label
$\bm{Y}\in\mathbb{R}^{G}$, where $G$ is the number of ground-truth action
classes. By convention, we first sample a sequence of $T$ snippets from each
video, and then extract snippet features with a pre-trained feature extractor
for both RGB and optical-flow streams. For simplicity, only one stream is
presented hereafter. As a result, we obtain a sequence of snippet features
$\bm{F}\in\mathbb{R}^{T\times D}$. Here, $D$ is the channel dimension.

For the baseline, following convention [27], we use a two-branch framework
consisting of a video classification branch and an attention branch, as shown
in Fig. 2(a). In the former branch, we first feed the input features $\bm{F}$
to an embedding encoder followed by an action classifier to get the temporal
class activation sequence (T-CAS) $\bm{P^{V}}\in\mathbb{R}^{T\times G}$. In
the latter branch, $\bm{F}$ is first passed through another embedding encoder
to obtain the snippet embeddings, and the embeddings are then sent to an
attention layer to extract one-dimension Attention weights
$\bm{P^{A}}\in\mathbb{R}^{T}$, which represent the foreground probabilities of
the snippets.

We apply the popular multiple instance learning (MIL) to train the video
classification branch. Briefly (see Supplementary for details), we first
calibrate T-CAS with the attention weights to restrain background snippets.
Then we select the top-$k$ snippets for each class based on the activations to
construct video-level scores $\bm{\bar{P}^{V}}\in\mathbb{R}^{G}$. Finally, we
optimize a video classification loss with the known video labels $\bm{Y}$:

$\displaystyle\mathcal{L}_{V}=\mathcal{L}_{CE}(\bm{\bar{P}^{V}},\bm{Y}).$ (1)

To train the attention branch, we adopt the pseudo-label-based scheme proposed
by [41] due to its conciseness and effectiveness. Specifically, we define
foreground pseudo-labels $\bm{Q^{A}}\in\mathbb{R}^{T}$ as follows: snippets
appearing in the top-$k$ activations for the ground-truth video-level classes
are positive, and the other snippets are negative. To improve the robustness
of the model against label noise, we use the generalized binary cross-entropy
loss [71, 41]:

$\displaystyle\mathcal{L}_{A}=$
$\displaystyle\frac{1}{N_{\text{pos}}}\sum_{t=1}^{T}\bm{Q^{A}}_{t}\frac{1-{(\bm{P^{A}}_{t})}^{\gamma}}{\gamma}+$
(2)
$\displaystyle\frac{1}{N_{\text{neg}}}\sum_{t=1}^{T}(1-\bm{Q^{A}}_{t})\frac{1-{(1-\bm{P^{A}}_{t})}^{\gamma}}{\gamma},$

where $\gamma\in(0,1)$ controls the noise tolerance, and $N_{\text{pos}}$ and
$N_{\text{neg}}$ represent the number of positive and negative snippets.

Figure 2: Framework of our CASE. (a) depicts the baseline, which includes a
video classification branch and an attention branch. (b) illustrates our
proposed clustering-based F&B separation algorithm, which comprises a snippet
clustering component (SCC) and a cluster classification component (CCC). Both
are trained using a unified self-labeling mechanism based on optimal transport
(OT). (c) shows the clustering-assisted testing technique that utilizes the
results of SCC and CCC to assist F&B separation during inference.

##  4 Our Method

In this section, we present our clustering-based F&B algorithm, as depicted in
Fig. 2(b), which is built upon the above baseline. We begin by providing an
overview of our algorithm, which consists of two main components: snippet
clustering component (SCC) and cluster classification component (CCC). Next,
we introduce a unified self-labeling mechanism that we employ to provide
pseudo-labels for both SCC and CCC. Lastly, we explain how SCC and CCC are
used in the training and testing procedures.

###  4.1 Overview

#### Snippet Clustering Component.

SCC is proposed to group snippets into latent clusters. To enable joint
learning of the attention layer and snippet clustering, we append SCC over the
embeddings in the attention branch, as shown in Fig. 2(b). For notation
simplicity, we henceforth term the total number of snippets as $N=BT$ for a
batch of $B$ videos, and call the snippet embeddings
$\bm{E}\in\mathbb{R}^{N\times D}$. We feed $\bm{E}$ into a clustering head
composed of a linear classifier with $K$ classes (clusters), producing
Snippet-level cluster assignment probabilities dubbed
$\bm{P^{S}}\in\mathbb{R}^{N\times K}$. Inspired by self-supervised learning
[47, 50], we set $K$ as a predefined parameter, which we find to be robust in
practice. To train the clustering head, we first generate (soft) pseudo-labels
$\bm{Q^{S}}\in\mathbb{R}^{N\times K}$ for $\bm{P^{S}}$, which will be
described in Sec. 4.2. Then, we minimize the following loss:

$\displaystyle\mathcal{L}_{S}=\frac{1}{N}\sum_{n=1}^{N}\mathcal{L}_{CE}(\bm{Q^{S}}_{n},\bm{P^{S}}_{n}).$
(3)

#### Cluster Classification Component.

CCC enforces each cluster to be classified into foreground or background by
mapping the cluster prototypes to the F&B prototypes, as shown in Fig. 2(b).
Specifically, based on the pseudo cluster assignments of snippets
$\bm{Q^{S}}\in\mathbb{R}^{N\times K}$ obtained from SCC, we can compute the
$k$-th cluster prototype over the snippet embeddings
$\bm{E}\in\mathbb{R}^{N\times D}$:

$\displaystyle\bm{\bar{E}^{S}}_{k}=\frac{\sum_{n=1}^{N}\bm{Q^{S}}_{n,k}\cdot\bm{E}_{n}}{\sum_{n=1}^{N}\bm{Q^{S}}_{n,K}}.$
(4)

where $\bm{\bar{E}^{S}}_{k}\in\mathbb{R}^{D}$. In a similar vein, using the
foreground pseudo-labels $\bm{Q^{A}}\in\mathbb{R}^{N}$ and background labels
$1-\bm{Q^{A}}$, we can calculate the F&B prototypes
$\bm{\bar{E}^{A}}\in\mathbb{R}^{2\times D}$. $\bm{\bar{E}^{A}}_{1}$ and
$\bm{\bar{E}^{A}}_{2}$ correspond to foreground and background, respectively.
Then, we compute Cluster-level classification probabilities dubbed
$\bm{P^{C}}\in\mathbb{R}^{K\times 2}$ by measuring the similarities between
cluster prototypes and F&B prototypes:

$\displaystyle\bm{P^{C}}_{k,i}=\mathop{Softmax}\limits_{i}\big{(}\rho\cdot\operatorname{cos}(\bm{\bar{E}^{S}}_{k},\bm{\bar{E}^{A}}_{i})\big{)},$
(5)

where $\operatorname{cos}(\cdot)$ indicates the cosine similarity function and
$\rho$ is the temperature. $\bm{P^{C}}_{k,i}$ represents the probability that
$k$-th cluster belongs to foreground ($i=1$) or background ($i=2$). To
optimize the component, we generate (soft) labels $\bm{Q^{C}}$ for
$\bm{P^{C}}$, which will be described in Sec. 4.2. Accordingly, we will get a
loss term:

$\displaystyle\mathcal{L}_{C}=\frac{1}{K}\sum_{k=1}^{K}\mathcal{L}_{CE}(\bm{Q^{C}}_{k},\bm{P^{C}}_{k}).$
(6)

Notably, although $\bm{Q^{C}}$ is computed from each mini-batch of data, we
observe that it will quickly converge to a stable status near the one-hot form
during training. This suggests that a global and clear correspondence between
clusters and F&B is established.

###  4.2 Self-Labeling via Unified Optimal-Transport

This section explains the self-labeling mechanism that generates the labels
$\bm{Q^{S}}$ for $\bm{P^{S}}$ in SCC and labels $\bm{Q^{C}}$ for $\bm{P^{C}}$
in CCC. First, we describe a basic labeling formulation shared in both SCC and
CCC. This formulation converts the label assignment to an optimal transport
problem while imposing constraints on the distribution of the labels. Then, we
discuss the unique adaptations required for SCC and CCC individually. For SCC,
we introduce a prior distribution for $\bm{Q^{S}}$ to avoid the uncertain
label assignment issue observed in SCC. As for CCC, we leverage snippet-level
F&B labels to estimate the prior marginal distribution of $\bm{Q^{C}}$. These
adaptations can be seamlessly integrated into the optimal-transport
formulation, resulting in a unified solution that is easy to implement, as
demonstrated in Alg. 1.

#### Basic formulation.

Regarding the generation of pseudo-labels $\bm{Q}$, a straightforward solution
is to search for a reasonable $\bm{Q}$ that is close to the current model
predictions $\bm{P}$, _e.g_., by applying $\operatorname{\arg\max}$ to
$\bm{P}$. However, in our unsupervised setting, this way may lead to trivial
solutions, _e.g_., all samples are assigned to only a class (cf. Sec. 5.3).
Instead, when searching for $\bm{Q}$, we propose to impose a constraint on the
proportion of elements assigned to each class. Formally, we formulate this as
an optimization problem:

$\displaystyle\mathop{\min}_{\bm{Q}\in\Omega}\mathrm{E}(\bm{P},\bm{Q}),$ (7)

where
$\small{\mathrm{E}(\bm{P},\bm{Q})=-\sum_{n}^{N}\sum_{k}^{K}\bm{Q}_{n,k}\log\bm{P}_{n,k}}$
measures the distance between $\bm{Q}$ and $\bm{P}$, $N$ is the number of
samples and $K$ is the number of classes. The constraint $\Omega$ is defined
as:

$\displaystyle\Omega=\\{\bm{Q}\in\mathbb{R}_{+}^{N\times
K}|\bm{Q}\mathbf{1}^{K}=\bm{\alpha},{\bm{Q}}^{\top}\mathbf{1}^{N}=\bm{\beta}\\},$
(8)

where $\bm{\alpha}$ and $\bm{\beta}$ are the marginal distributions of
$\bm{Q}$ onto its rows and columns, respectively. We set
$\bm{\alpha}=\mathbf{1}^{N}$ to ensure that $\bm{Q}$ is a probability matrix.
$\bm{\beta}\in\mathbb{R}^{K}$ represents the proportion of elements belonging
to each of the $K$ classes. When there is no prior knowledge, equipartition
[1, 5] can serve as a general inductive bias and be utilized to set
$\bm{\beta}$:

$\displaystyle\bm{\beta}=\frac{N}{K}\mathbf{1}^{K}.$ (9)

This ensures that, on average, each class is assigned the same number of
samples, thereby averting trivial solutions.

It is noteworthy that Eq. 7 is an optimal transport problem, which is
computationally expensive to solve. Following [10], an entropy term is
introduced to it:

$\displaystyle\mathop{\min}_{\bm{Q}\in\Omega}\
\mathrm{E}(\bm{P},\bm{Q})-\frac{1}{\epsilon}\operatorname{H}(\bm{Q}),$ (10)

where $\epsilon>0$ and $\operatorname{H}(\bm{Q})$ is the entropy of $\bm{Q}$.
The advantage of the term is that Eq. 10 can be efficiently solved by
Sinkhorn-Knopp algorithm [10].

#### Prior distribution for SCC.

In Eq. 10, an entropy term is subtracted to make it tractable with affordable
complexity. Nevertheless, maximizing the entropy can also lead to uncertain
label assignments, where the samples are assigned to different classes with
equal probability. In practice, the issue is pronounced in SCC but not in CCC.
This may be because the former involves much more instances and classes,
rendering the algorithm harder to converge (cf. Fig. 4).

To remedy the defect in SCC, we repurpose an early sequence-matching method
[56] to introduce a prior distribution for the pseudo-labels $\bm{Q^{S}}$ in
SCC, denoted as $\bm{\hat{Q}^{S}}\in\mathbb{R}^{N\times K}$, which represents
the probability of assigning $N$ snippets to $K$ clusters. A sensible prior
distribution should encourage foreground snippets to have relatively high
probabilities of belonging to foreground clusters, and the same is true for
background. To implement this, we first sort the snippets according to their
foreground probabilities $\bm{P^{A}}$ in ascending order, and denote the
resulting ranks of the $N$ snippets as ${rank}\in\mathbb{R}^{N}$. We then
construct the prior distribution $\bm{\hat{Q}^{S}}$, such that the snippets
with high ranks (i.e, ${rank}$) are more likely to be assigned to the clusters
with high foreground probabilities (i.e, $\bm{Q^{C}}_{:,1}$) and vice versa.
Formally, $\bm{\hat{Q}^{S}}$ is defined as a Gaussian distribution:

$\displaystyle\bm{\hat{Q}^{S}}_{n,k}=\frac{1}{\sigma\sqrt{2\pi}}\exp\big{(}-\frac{{|\frac{{rank}_{n}}{N}-\bm{Q^{C}}_{k,1}|}^{2}}{2\sigma^{2}}\big{)},$
(11)

where ${rank}_{n}$ is the order of $n$-th snippet, and $\bm{Q^{C}}_{k,1}$ is
the foreground probability of $k$-th cluster. Finally, we replace Eq. 10 with
the following objective function:

$\displaystyle\mathop{\min}_{\bm{Q^{S}}\in\Omega^{S}}\
\mathrm{E}(\bm{P^{S}},\bm{Q^{S}})+\frac{1}{\epsilon}\operatorname{KL}(\bm{Q^{S}},\bm{\hat{Q}^{S}}),$
(12)

where $\operatorname{KL}(\cdot)$ is the Kullback-Leibler divergence. By
minimizing the KL term, we encourage the labels $\bm{Q^{S}}$ to be close to
the prior distribution $\bm{\hat{Q}^{S}}$, which helps to avoid uncertain
label assignments caused by the original entropy term. Importantly, Eq. 12 can
still be efficiently addressed by Sinkhorn-Knopp algorithm. For detailed
derivation, we refer to Supplementary.

#### Prior marginal distribution for CCC.

Although equipartition (_i.e_., Eq. 9) is a common prior in traditional
clustering, it is not suitable for enforcing equipartition on the marginal
distribution of the cluster-level F&B labels $\bm{Q^{C}}$, namely
$\bm{\beta^{C}}\in\mathbb{R}^{2}$. This is due to the fact that
$\bm{\beta^{C}}$ represents the proportions of clusters assigned to F&B, which
are not always balanced. However, since SCC enforces equipartition on the
snippet level, each cluster contains a similar number of snippets.
Consequently, the proportions of F&B clusters are expected to be close to the
proportions of F&B snippets. To this end, instead of using Eq. 9, we estimate
$\bm{\beta^{C}}$ empirically based on the distribution of the snippet-level
foreground labels $\bm{Q^{A}}$:

$\displaystyle\bm{\beta^{C}}=[\frac{1}{N}\sum_{n=1}^{N}\bm{Q^{A}}_{n},\
\frac{1}{N}\sum_{n=1}^{N}(1-\bm{Q^{A}}_{n})].$ (13)

Algorithm 1 Pseudo-code of labeling procedures of SCC and CCC in Pytorch-like
style.

⬇

1# L_S: logit output of clustering head (NxK)

2# Q_hat_S: prior distribution in SCC (NxK)

3# L_C: logit output of cluster classification (Kx2)

4# Beta_C: prior marginal distribution in CCC (2)

5

6# generating labels Q_S for SCC

7Q_S = SK(L_S, Q_hat=Q_hat_S) # (NxK)

8# generating labels Q_C for CCC

9Q_C = SK(L_C, Beta=Beta_C) # (Kx2)

10

11# resolve the optimal-transport problem by iterative Sinkhorn-Knopp (SK)
algorithm

12def SK(L, Q_hat=None, Beta=None, n_iter=3):

13 N, K = L.size()

14 # uniform distribution is the default when prior distribution is not given

15 Q_hat = 1/K if Q_hat is None else Q_hat

16 Beta = 1/K if Beta is None else Beta

17 Q = exp(L / eps)

18 Q = (Q * Q_hat).T

19 Q /= Q.sum()

20 for _ in range(n_iter):

21 Q /= Q.sum(dim=1)

22 Q = Q * Beta

23 Q /= Q.sum(dim=0)

24 Q /= N

25 return Q.t() * N

###  4.3 Training and Testing

#### Joint training.

We train all components together in an end-to-end fashion. The overall
objective is written as:

$\displaystyle\mathcal{L}=(\mathcal{L}_{V}+\mathcal{L}_{A})+\lambda_{S}\mathcal{L}_{S}+\lambda_{C}\mathcal{L}_{C},$
(14)

where $\lambda_{S}$ and $\lambda_{C}$ represent the loss weights. As the
baseline and our proposed algorithm share the same embedding encoder in the
attention branch, the joint training also facilitates the training of the
baseline model.

#### Clustering-assisted testing.

In the inference period, using the cluster-level foreground probabilities
$\bm{Q^{C}}_{:,1}$, we can transform the snippet-level cluster assignments
$\bm{P^{S}}$ to snippet-level foreground probabilities dubbed $\bm{P^{T}}$
based on law of total probability: $\bm{P^{T}}=\bm{P^{S}}\bm{Q^{C}}_{:,1}$, as
depicted in Fig. 2 (c). Considering that $\bm{Q^{C}}$ is stable during
training, we simply use the $\bm{Q^{C}}$ from the last training iteration for
inference. Moreover, as verified in Table 7, the transformed foreground
probability $\bm{P^{T}}$ is complementary to the foreground probability
$\bm{P^{A}}$ from the attention layer. Hence, we fuse $\bm{P^{A}}$ and
$\bm{P^{T}}$ by convex combination: $\bm{P^{M}}=0.5\bm{P^{A}}+0.5\bm{P^{T}}$.
The combined probability $\bm{P^{M}}$ is then used to help localize action
instances during inference.

##  5 Experiments

###  5.1 Datasets and Evaluation Metric

THUMOS14 [26] contains videos with $20$ classes. We use the $200$ videos in
validation set for training and the $213$ videos in testing set for
evaluation. ActivityNet [3] has two release versions, i.e., ActivityNet v1.3
and ActivityNet v1.2. ActivityNet v1.3 [3] covers 200 action categories with
10, 024 and 4, 926 videos in the training and validation sets, respectively.
ActivityNet v1.2 is a subset of ActivityNet v1.3, and covers 100 action
categories with 4, 819 and 2, 383 videos in the training and validation sets,
respectively We follow the standard evaluation protocol by reporting mean
Average Precision (mAP) under various temporal intersection over union (tIoU)
thresholds. We refer to Supplementary for more experimental details about
architecture, setup, baseline, etc.

Method | MACs | Params | mAP @ IoU |  AVG 0.1:0.5 | AVG
---|---|---|---|---|---
0.1 | 0.3 | 0.5 | 0.7
W-TALC [49] | - | - | 55.2 | 40.1 | 22.8 | 7.6 | 39.8 | -
BaS-Net [27] | 38.60 | 26.26 | 58.2 | 44.6 | 27.0 | 10.4 | 43.6 | 35.3
TSCN [69] | - | - | 63.4 | 47.8 | 28.7 | 10.2 | 47.0 | 37.8
ACM-Net [51] | 9.48 | 12.63 | 68.9 | 55.0 | 34.6 | 10.8 | 53.2 | 42.6
CoLA [70] | 9.47 | 12.62 | 66.2 | 51.5 | 32.2 | 13.1 | 50.3 | 40.9
UGCT [64] | - | - | 69. | 55.5 | 35.9 | 11.4 | 54.0 | 43.6
CO2-Net [17] | 20.88 | 34.13 | 70.1 | 54.5 | 38.3 | 13.4 | 54.4 | 44.6
FTCL [13] | 9.45 | 14.73 | 69.6 | 55.2 | 35.6 | 12.2 | 53.8 | 43.6
RSKP [23] | 6.90 | 4.20 | 71.3 | 55.8 | 38.2 | 12.5 | 55.6 | 45.1
ASM-Loc [16] | 13.77 | 46.21 | 71.2 | 57.1 | 36.6 | 13.4 | 55.4 | 45.1
DELU [9] | 20.88 | 34.13 | 71.5 | 56.5 | 40.5 | 15.5 | 56.5 | 46.4
CASE | 1.60 | 2.13 | 72.3 | 59.2 | 37.7 | 13.7 | 57.1 | 46.2

Table 1: Comparison on THUMOS14 in terms of mAP (%), MACs (G), and Params (M).
AVG(0.1:0.5) and AVG represent the average mAP under IoU thresholds of 0.1:0.5
and 0.1:0.7. MACs are computed from a video with 750 snippets.

| Our baseline | CO2-Net [17] | AICL [32] | Zhou et al. [73]
---|---|---|---|---
Base | 42.1 | 44.6 | 46.4 | 48.3
Base + CASE | 46.2 | 46.6 | 48.0 | 49.2

Table 2: Compatibility with different baselines.

###  5.2 Comparison with SOTA Methods

In Table 1, we compare our CASE with SOTA WTAL methods on THUMOS14. Besides
mAP, we also report the model complexity in terms of multi-accumulative
operations (MACs), and the number of trainable parameters (Params). We can
observe that CASE achieves comparable performance to the recent SOTA method
DELU [9] and evidently outperforms other approaches. Notably, CASE is
significantly more lightweight than the competitors, with MACs and Params less
than 1/10 of those of DELU. This is because our method does not require a
heavy cross-modal consensus module [9, 17] or multi-step proposal refinement
[16]. We can increase the complexity of CASE by combining it with previous
methods. The results are presented in Table 2. We can observe that our CASE
consistently and evidently boosts the performances of all the baselines,
highlighting its strong compatibility with these baselines.

Additionally, we compare our CASE with previous methods on ActivityNet
v1.2&v1.3 in Table 3. As can be seen, CASE achieves impressive improvements
over the previous methods on both datasets.

Method | mAP @ IoU
---|---
0.5 | 0.75 | 0.95 | AVG
BaS-Net [27] | 38.5 | 24.2 | 5.6 | 24.3
RPN [19] | 37.6 | 23.9 | 5.4 | 23.3
EM-MIL [40] | 37.4 | - | - | 20.3
TSCN [69] | 37.6 | 23.7 | 5.7 | 23.6
WUM [28] | 41.2 | 25.6 | 6.0 | 25.9
CoLA [70] | 42.7 | 25.7 | 5.8 | 26.1
ASL [41] | 40.2 | - | - | 25.8
D2-Net [43] | 42.3 | 25.5 | 5.8 | 26.0
ACGNet [67] | 41.8 | 26.0 | 5.9 | 26.1
DELU [9] | 44.2 | 26.7 | 5.4 | 26.9
CASE | 43.8 | 27.2 | 6.7 | 27.9

(a) ActivityNet v1.2

Method | mAP @ IoU
---|---
0.5 | 0.75 | 0.95 | AVG
BaS-Net [27] | 34.5 | 22.5 | 4.9 | 22.2
TSCN [69] | 35.3 | 21.4 | 5.3 | 21.7
MSA [20] | 36.5 | 22.8 | 6.0 | 22.9
ACM-Net [51] | 40.1 | 24.2 | 6.2 | 24.6
UGCT [64] | 39.1 | 22.4 | 5.8 | 23.8
AUMN [39] | 38.3 | 23.5 | 5.2 | 23.5
FAC-Net [22] | 37.6 | 24.2 | 6.0 | 24.0
FTCL [13] | 40.0 | 24.3 | 6.4 | 24.8
RSKP [23] | 40.6 | 24.6 | 5.9 | 25.0
ASM-Loc [16] | 41.0 | 24.9 | 6.2 | 25.1
CASE | 43.2 | 26.2 | 6.7 | 26.8

(b) ActivityNet v1.3

Table 3: Results on ActivityNet v1.2&v1.3. AVG indicates the average mAP at
IoU thresholds 0.5:0.05:0.95.

###  5.3 Ablation Study

Contribution of core components. Our method contains two core components:
snippet clustering component (SCC) and cluster classification component (CCC),
which are successively stacked on the baseline model and jointly trained with
it. Table 10 quantifies the contributions of each component. When compared to
the baseline (line #1), both components contribute to the performance.
Specifically, when the clustering-assisted testing technique is not applied,
SCC and CCC yield gains of 1.1% (line #2) and 0.7% (line #3), respectively,
indicating that the joint training with SCC and CCC indeed improves the
performance of the baseline. The reason is that through deliberated clustering
of snippets, the embedding space is shaped as well-structured, as illustrated
in Fig. 7, thereby facilitating the learning of the baseline. After training
SCC and CCC, we leverage the clustering-assisted testing to further make use
of SCC and CCC during testing, which brings about a substantial promotion of
2.3% (line #4). In a nutshell, SCC and CCC are beneficial in both the training
and testing stages.

# | Method | mAP @ IoU (%)
---|---|---
0.3 | 0.5 | 0.7 | AVG
1 | Baseline | 53.8 | 31.9 | 11.9 | 42.1
2 | \+ SCC | 55.3 | 33.7 | 12.4 | 43.2+1.1
3 | \+ SCC + CCC | 56.1 | 34.9 | 12.8 | 43.9+0.7
4 | \+ SCC + CCC (T) | 59.2 | 37.7 | 13.7 | 46.2+2.3

Table 4: Ablation study of core components. ”(T)” indicates that clustering-
assisted testing is applied.

Necessity of converting label assignment to optimal transport (OT). We propose
an OT-based labeling strategy that generates labels by solving an OT problem:
the objective Eq. 7 subject to a constraint Eq. 8 on the proportion of each
class. To assess the necessity of our design, we use the labeling strategy in
SCC as an example and compare it with alternative labeling strategies. The
results are summarized in Table 5. The terms ”Hard” and ”Soft” indicate
whether the labels are one-hot or soft, respectively. The strategy without OT,
denoted as ”w/o OT”, refers to solving Eq. 7 without Eq. 8, which is no longer
an OT problem and can be solved immediately. In particular, in the case of
”w/o OT+Soft”, the labels would be nearly identical to the model predictions,
resulting in almost no learning. As a consequence, it shows poor performance.
On the other hand, ”w/o OT+Hard” involves applying $\operatorname{\arg\max}$
to the model predictions, but it also performs poorly. To understand why, we
calculate the entropy of the proportions of clusters, namely
$\operatorname{H}(\bm{\bar{P}^{S}})$, where
$\bm{\bar{P}^{S}}=\frac{1}{N}\sum_{n}^{N}\bm{P^{S}}_{n}$. In Table 5, we
observe that ”w/o OT+Hard” results in a low
$\operatorname{H}(\bm{\bar{P}^{S}})$ close to zero, indicating a trivial
solution where most snippets are assigned to only a few clusters. In contrast,
our proposed ”w/ OT” greatly alleviates this issue and achieves much better
performance. Notably, ”w/ OT+Hard” performs worse than ”w/ OT+Soft”, which we
attribute to the aggressive nature of obtaining hard labels. Overall, the
results confirm the efficacy of our design.

Metric |  w/o OT |  w/ OT
---|---|---
Soft | Hard | Soft | Hard
mAP | 41.2 | 40.8 | 46.2 | 44.7
$\operatorname{H}(\bm{\bar{P}^{S}})$ | 2.77 | 0.01 | 2.76 | 2.75

Table 5: Ablation study of OT-based labeling.

Analysis of the number of clusters $K$. We cluster snippets into multiple
clusters ($K>2$), even though only F&B separation is required. To verify the
correctness of our design, we compare the performances under different numbers
of $K$ in Fig. 4. As can be seen, a small $K$ results in inferior performance,
with $K=2$ even causing a performance decline relative to baseline. The reason
may be that the clustering results deviate too much from the true distribution
of F&B snippets. Hence, clustering into multiple clusters is necessary.
Besides, the performance becomes stable ($\pm$0.2%) and robust to $K$ as long
as enough clusters are set ($K>16$), making it easy to tune an appropriate $K$
in practice. These findings are in accordance with self-supervised learning
[50, 47, 60].

Figure 3: Ablation study of the number of clusters $K$.  Figure 4: The
evolution of $H(\bm{Q^{C}})$ and $H(\bm{Q^{S}})$.

# |  Method |  mAP
---|---|---
1 |  our CASE |  46.2
2 |  using Eq. 10 in SCC |  45.4
3 |  using Eq. 9 in CCC |  44.0

Table 6: Analysis of unique techniques in SCC and CCC.

Analysis of snippet clustering component (SCC).  In SCC, we enforce pseudo-
labels $\bm{Q^{S}}$ to match the prior distribution $\bm{\hat{Q}^{S}}$ by
transitioning the objective of OT problem from the original objective Eq. 10
to the current objective Eq. 12. To evaluate the impact of this modification,
in Table 6, we compare the results of our method (line #1) against those
obtained using the original Eq. 10 (line # 2). The results show that our
modified objective outperforms the original one, indicating that matching the
prior distribution is beneficial. To further examine this phenomenon, in Fig.
4, we plot the evolution of the entropy of $\bm{Q^{S}}$,
$\operatorname{H}(\bm{Q^{S}})$, using current Eq. 12 or original Eq. 10. A
large $\operatorname{H}(\bm{Q^{S}})$ is typically undesired as it indicates
that the assignment is uncertain. Additionally, we report the entropy of
$\bm{Q^{C}}$ in CCC, $\operatorname{H}(\bm{Q^{C}})$, as an indication, though
a direct comparison is not entirely fair. The results reveal that
$\operatorname{H}(\bm{Q^{C}})$ converges rapidly to a small value, indicating
that optimization can easily yield a solution close to a one-hot form in CCC.
Conversely, $\operatorname{H}(\bm{Q^{S}})$ converges slowly and remains high,
suggesting uncertain assignments in SCC. However, this issue is mitigated when
using our proposed Eq. 12.

Analysis of cluster classification component (CCC).  In CCC, we empirically
estimate the prior marginal distribution of pseudo-labels $\bm{Q^{C}}$ based
on Eq. 13, as opposed to using equipartition Eq. 9. To evaluate the impact of
this design choice, we present the results of using equipartition Eq. 9 in
line #3 of Table 6. The clear superiority of our method over the use of
equipartition highlights the importance of using a data-driven approach to
estimate the prior distribution of pseudo-labels in CCC.

Method |  $\bm{P^{A}}$ |  $\bm{P^{T}}$ |  $\bm{P^{M}}$
---|---|---|---
mAP |  43.9 |  44.1 |  46.2

Table 7: Analysis of the clustering-assisted testing.

Analysis of clustering-assisted testing. We propose to fuse the foreground
probabilities $\bm{P^{A}}$ from the baseline model and the foreground
probabilities $\bm{P^{T}}$ transformed from the cluster assignments, resulting
in the fused one $\bm{P^{M}}$. In Table 7, we compare the performances of
$\bm{P^{A}}$, $\bm{P^{T}}$, and $\bm{P^{M}}$. It can be seen that the fused
one $\bm{P^{M}}$ outperforms both $\bm{P^{A}}$ and $\bm{P^{T}}$, which well
demonstrates that $\bm{P^{A}}$ and $\bm{P^{T}}$ are complementary to each
other. This is further confirmed by the qualitative results exemplified in
Fig. 5. Concretely, it delivers the following inspiring observations: 1)
Compared to $\bm{P^{A}}$, $\bm{P^{T}}$ tends to activate more action-related
regions (e.g., ②, ④) and clearer action boundaries (e.g., ③, ⑤). This is in
line with our motivation for using $\bm{P^{T}}$, which is designed to be more
independent of the video classification loss, capturing a more comprehensive
distribution of snippets rather than being biased by discriminative regions.
2) $\bm{P^{T}}$ is not always superior to $\bm{P^{A}}$. When optimizing
$\bm{P^{T}}$, CASE regards the snippets as independent samples and fails to
make use of the video-level labels, resulting in irrelevant action activation
(e.g., ①). These observations further support the complementary nature of
$\bm{P^{A}}$ and $\bm{P^{T}}$.

Figure 5: Qualitative results of two videos on THUMOS14. We show $\bm{P^{T}}$,
$\bm{P^{A}}$ and $\bm{P^{M}}$, and ground-truth (GT). Figure 6: Qualitative
results of snippet clustering and cluster classification. We show three
clusters belonging to foreground at the top and there clusters belonging to
background at the bottom. Figure 7: TSNE visualization of $10,000$ snippet
embeddings and $16$ cluster prototypes. ”-RGB” and ”-Flow” indicate RGB stream
and optical-flow stream, respectively. The yellow, red, black, and blue marks
represent foreground snippets, background snippets, foreground cluster
prototypes, and background cluster prototypes, respectively.

Visualization results. To gain more insight, we illustrate qualitative results
of snippet clustering and cluster classification in Fig. 6. It can be seen
that: 1) Our method effectively identifies the snippet groups with common
characteristics, such as ”swinging arms” for the 1st row, ”standing” for the
5th row. 2) Our method is accurate in classifying clusters into F&B classes.
For instance, some foreground frames in 1st&3rd rows and background frames in
5th row are visually similar, yet our method can still separate them into
different clusters with correct F&B labels. Overall, these results vividly
substantiate the efficacy of our method in effectively grouping and
classifying snippets.

Furthermore, we visualize the snippet embeddings and cluster prototypes in
Fig. 7 and make the following observations: 1) Each cluster prototype is
surrounded by a couple of snippets, and the cluster prototypes are
distinguishable between foreground and background. 2) The embedding space
learned by our method exhibits a clearer boundary between foreground and
background classes compared to the baseline. These results clearly demonstrate
that 1) the learned clusters are highly representative of the snippets; 2) the
embedding space in our method is well-shaped and captures a comprehensive
distribution of the snippets.

##  6 Conclusion and Limitation

In this work, we propose a WTAL framework named CASE, which leverages snippet
clustering to improve F&B separation. Specifically, CASE comprises a snippet
clustering component that partitions snippets into multiple clusters, followed
by a cluster classification component that identifies the F&B clusters. To
optimize these components, we employ a unified self-labeling strategy based on
optimal transport. Our extensive analysis demonstrates the effectiveness and
efficiency of CASE. One limitation of our method is the requirement for a WTAL
baseline to provide semantic-level reference of F&B classes to classify the
clusters into F&B. A more self-contained clustering-based framework is our
future work.

##  7 Acknowledgements

This work is supported by the National Natural Science Foundation of China
under Grant 62176246 and Grant 61836008. This work is also supported by Anhui
Province Key Research and Development Plan (202304a05020045), Anhui Provincial
Natural Science Foundation 2208085UD17 and the Fundamental Research Funds for
the Central Universities (WK3490000006).

## References

  * [1] YM Asano, C Rupprecht, and A Vedaldi.  Self-labelling via simultaneous clustering and representation learning.  In ICLR, 2019.
  * [2] Alberto Borobia and Rafael Cantó.  Matrix scaling: A geometric proof of sinkhorn’s theorem.  Linear algebra and its applications, 268:1–8, 1998.
  * [3] Fabian Caba Heilbron, Victor Escorcia, Bernard Ghanem, and Juan Carlos Niebles.  Activitynet: A large-scale video benchmark for human activity understanding.  In CVPR, 2015.
  * [4] Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze.  Deep clustering for unsupervised learning of visual features.  In ECCV, 2018.
  * [5] Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin.  Unsupervised learning of visual features by contrasting cluster assignments.  NeurIPS, 2020.
  * [6] Joao Carreira and Andrew Zisserman.  Quo vadis, action recognition? a new model and the kinetics dataset.  In CVPR, 2017.
  * [7] Jianlong Chang, Yiwen Guo, Lingfeng Wang, Gaofeng Meng, Shiming Xiang, and Chunhong Pan.  Deep discriminative clustering analysis.  arXiv preprint arXiv:1905.01681, 2019.
  * [8] Jianlong Chang, Lingfeng Wang, Gaofeng Meng, Shiming Xiang, and Chunhong Pan.  Deep adaptive image clustering.  In ICCV, 2017.
  * [9] Mengyuan Chen, Junyu Gao, Shicai Yang, and Changsheng Xu.  Dual-evidential learning for weakly-supervised temporal action localization.  In ECCV, 2022.
  * [10] Marco Cuturi.  Sinkhorn distances: Lightspeed computation of optimal transport.  In NeurIPS, 2013.
  * [11] Zhiyuan Dang, Cheng Deng, Xu Yang, and Heng Huang.  Multi-scale fusion subspace clustering using similarity constraint.  In CVPR, 2020.
  * [12] Enrico Fini, Enver Sangineto, Stéphane Lathuilière, Zhun Zhong, Moin Nabi, and Elisa Ricci.  A unified objective for novel class discovery.  In ICCV, 2021.
  * [13] Junyu Gao, Mengyuan Chen, and Changsheng Xu.  Fine-grained temporal contrastive learning for weakly-supervised temporal action localization.  In CVPR, 2022.
  * [14] Spyros Gidaris and Nikos Komodakis.  Dynamic few-shot visual learning without forgetting.  In CVPR, 2018.
  * [15] Philip Haeusser, Johannes Plapp, Vladimir Golkov, Elie Aljalbout, and Daniel Cremers.  Associative deep clustering: Training a classification network with no labels.  In GCPR, 2018.
  * [16] Bo He, Xitong Yang, Le Kang, Zhiyu Cheng, Xin Zhou, and Abhinav Shrivastava.  Asm-loc: Action-aware segment modeling for weakly-supervised temporal action localization.  In CVPR, 2022.
  * [17] Fa-Ting Hong, Jia-Chang Feng, Dan Xu, Ying Shan, and Wei-Shi Zheng.  Cross-modal consensus network for weakly supervised temporal action localization.  In ACMMM, 2021.
  * [18] Jiabo Huang, Shaogang Gong, and Xiatian Zhu.  Deep semantic clustering by partition confidence maximisation.  In CVPR, 2020.
  * [19] Linjiang Huang, Yan Huang, Wanli Ouyang, and Liang Wang.  Relational prototypical network for weakly supervised temporal action localization.  In AAAI, 2020.
  * [20] Linjiang Huang, Yan Huang, Wanli Ouyang, and Liang Wang.  Modeling sub-actions for weakly supervised temporal action localization.  Transactions on Image Processing, 2021.
  * [21] Linjiang Huang, Yan Huang, Wanli Ouyang, and Liang Wang.  Two-branch relational prototypical network for weakly supervised temporal action localization.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021.
  * [22] Linjiang Huang, Liang Wang, and Hongsheng Li.  Foreground-action consistency network for weakly supervised temporal action localization.  In ICCV, 2021.
  * [23] Linjiang Huang, Liang Wang, and Hongsheng Li.  Weakly supervised temporal action localization via representative snippet knowledge propagation.  In CVPR, 2022.
  * [24] Ashraful Islam, Chengjiang Long, and Richard Radke.  A hybrid attention mechanism for weakly-supervised temporal action localization.  In AAAI, 2021.
  * [25] Xu Ji, João F Henriques, and Andrea Vedaldi.  Invariant information clustering for unsupervised image classification and segmentation.  In ICCV, 2019.
  * [26] Y.-G. Jiang, J. Liu, A. Roshan Zamir, G. Toderici, I. Laptev, M. Shah, and R. Sukthankar.  THUMOS challenge: Action recognition with a large number of classes, 2014.
  * [27] Pilhyeon Lee, Youngjung Uh, and Hyeran Byun.  Background suppression network for weakly-supervised temporal action localization.  In AAAI, 2020.
  * [28] Pilhyeon Lee, Jinglu Wang, Yan Lu, and Hyeran Byun.  Weakly-supervised temporal action localization by uncertainty modeling.  In AAAI, 2021.
  * [29] Jingjing Li, Tianyu Yang, Wei Ji, Jue Wang, and Li Cheng.  Exploring denoised cross-video contrast for weakly-supervised temporal action localization.  In CVPR, 2022.
  * [30] Ruihuang Li, Shuai Li, Chenhang He, Yabin Zhang, Xu Jia, and Lei Zhang.  Class-balanced pixel-level self-labeling for domain adaptive semantic segmentation.  In CVPR, 2022.
  * [31] Zhi Li, Lu He, and Huijuan Xu.  Weakly-supervised temporal action detection for fine-grained videos with hierarchical atomic actions.  In ECCV, 2022.
  * [32] Zhilin Li, Zilei Wang, and Qinying Liu.  Actionness inconsistency-guided contrastive learning for weakly-supervised temporal action localization.  In AAAI, 2023.
  * [33] Daochang Liu, Tingting Jiang, and Yizhou Wang.  Completeness modeling and context separation for weakly supervised temporal action localization.  In CVPR, 2019.
  * [34] Qinying Liu and Zilei Wang.  Progressive boundary refinement network for temporal action detection.  In AAAI, 2020.
  * [35] Qinying Liu and Zilei Wang.  Collaborating domain-shared and target-specific feature clustering for cross-domain 3d action recognition.  In ECCV, 2022.
  * [36] Qinying Liu, Zilei Wang, Ruoxi Chen, and Zhilin Li.  Unleashing the potential of adjacent snippets for weakly-supervised temporal action localization.  In ICME, 2023.
  * [37] Qinying Liu, Zilei Wang, and Shenghai Rong.  Improve temporal action proposals using hierarchical context.  Pattern Recognition, 140:109560, 2023.
  * [38] Ziyi Liu, Le Wang, Wei Tang, Junsong Yuan, Nanning Zheng, and Gang Hua.  Weakly supervised temporal action localization through learning explicit subspaces for action and context.  In AAAI, 2021.
  * [39] Wang Luo, Tianzhu Zhang, Wenfei Yang, Jingen Liu, Tao Mei, Feng Wu, and Yongdong Zhang.  Action unit memory network for weakly supervised temporal action localization.  In CVPR, 2021.
  * [40] Zhekun Luo, Devin Guillory, Baifeng Shi, Wei Ke, Fang Wan, Trevor Darrell, and Huijuan Xu.  Weakly-supervised action localization with expectation-maximization multi-instance learning.  In ECCV, 2020.
  * [41] Junwei Ma, Satya Krishna Gorti, Maksims Volkovs, and Guangwei Yu.  Weakly supervised action selection learning in video.  In CVPR, 2021.
  * [42] Kyle Min and Jason J Corso.  Adversarial background-aware loss for weakly-supervised temporal activity localization.  In ECCV, 2020.
  * [43] Sanath Narayan, Hisham Cholakkal, Munawar Hayat, Fahad Shahbaz Khan, Ming-Hsuan Yang, and Ling Shao.  D2-net: Weakly-supervised action localization via discriminative embeddings and denoised activations.  In ICCV, 2021.
  * [44] Sanath Narayan, Hisham Cholakkal, Fahad Shabaz Khan, and Ling Shao.  3c-net: Category count and center loss for weakly-supervised action localization.  In ICCV, 2019.
  * [45] Phuc Nguyen, Ting Liu, Gautam Prasad, and Bohyung Han.  Weakly supervised action localization by sparse temporal pooling network.  In CVPR, 2018.
  * [46] Phuc Xuan Nguyen, Deva Ramanan, and Charless C Fowlkes.  Weakly-supervised action localization with background modeling.  In ICCV, 2019.
  * [47] Bo Pang, Yifan Zhang, Yaoyi Li, Jia Cai, and Cewu Lu.  Unsupervised visual representation learning by synchronous momentum grouping.  In ECCV, 2022.
  * [48] Alejandro Pardo, Humam Alwassel, Fabian Caba, Ali Thabet, and Bernard Ghanem.  Refineloc: Iterative refinement for weakly-supervised action localization.  In WACV, 2021.
  * [49] Sujoy Paul, Sourya Roy, and Amit K Roy-Chowdhury.  W-talc: Weakly-supervised temporal activity localization and classification.  In ECCV, 2018.
  * [50] Qi Qian, Yuanhong Xu, Juhua Hu, Hao Li, and Rong Jin.  Unsupervised visual representation learning by online constrained k-means.  2022\.
  * [51] Sanqing Qu, Guang Chen, Zhijun Li, Lijun Zhang, Fan Lu, and Alois Knoll.  Acm-net: Action context modeling network for weakly-supervised temporal action localization.  Transactions on Image Processing, 2021.
  * [52] Zheng Shou, Hang Gao, Lei Zhang, Kazuyuki Miyazawa, and Shih-Fu Chang.  Autoloc: Weakly-supervised temporal action localization in untrimmed videos.  In ECCV, 2018.
  * [53] Zheng Shou, Dongang Wang, and Shih-Fu Chang.  Temporal action localization in untrimmed videos via multi-stage cnns.  In CVPR, 2016.
  * [54] Krishna Kumar Singh and Yong Jae Lee.  Hide-and-seek: Forcing a network to be meticulous for weakly-supervised object and action localization.  In ICCV, 2017.
  * [55] Richard Sinkhorn.  Diagonal equivalence to matrices with prescribed row and column sums.  The American Mathematical Monthly, 74(4):402–405, 1967.
  * [56] Bing Su and Gang Hua.  Order-preserving wasserstein distance for sequence matching.  In CVPR, 2017.
  * [57] Wouter Van Gansbeke, Simon Vandenhende, Stamatios Georgoulis, Marc Proesmans, and Luc Van Gool.  Scan: Learning to classify images without labels.  In ECCV, 2020.
  * [58] Binglu Wang, Xun Zhang, and Yongqiang Zhao.  Exploring sub-action granularity for weakly supervised temporal action localization.  IEEE Transactions on Circuits and Systems for Video Technology, 2021.
  * [59] Limin Wang, Yuanjun Xiong, Dahua Lin, and Luc Van Gool.  Untrimmednets for weakly supervised action recognition and detection.  In CVPR, 2017.
  * [60] Xudong Wang, Ziwei Liu, and Stella X Yu.  Unsupervised feature learning by cross-level instance-group discrimination.  In CVPR, 2021.
  * [61] Jianlong Wu, Keyu Long, Fei Wang, Chen Qian, Cheng Li, Zhouchen Lin, and Hongbin Zha.  Deep comprehensive correlation mining for image clustering.  In ICCV, 2019.
  * [62] Junyuan Xie, Ross Girshick, and Ali Farhadi.  Unsupervised deep embedding for clustering analysis.  In ICML, 2016.
  * [63] Jianwei Yang, Devi Parikh, and Dhruv Batra.  Joint unsupervised learning of deep representations and image clusters.  In CVPR, 2016.
  * [64] Wenfei Yang, Tianzhu Zhang, Xiaoyuan Yu, Tian Qi, Yongdong Zhang, and Feng Wu.  Uncertainty guided collaborative training for weakly supervised temporal action detection.  In CVPR, 2021.
  * [65] Xu Yang, Cheng Deng, Kun Wei, Junchi Yan, and Wei Liu.  Adversarial learning for robust deep clustering.  In NeurIPS, 2020.
  * [66] Xu Yang, Cheng Deng, Feng Zheng, Junchi Yan, and Wei Liu.  Deep spectral clustering using dual autoencoder network.  In CVPR, 2019.
  * [67] Zichen Yang, Jie Qin, and Di Huang.  Acgnet: Action complement graph network for weakly-supervised temporal action localization.  In AAAI, 2022.
  * [68] C Zach, T Pock, and H Bischof.  A duality based approach for realtime tv-l 1 optical flow.  Pattern Recognition, 2007.
  * [69] Yuanhao Zhai, Le Wang, Wei Tang, Qilin Zhang, Junsong Yuan, and Gang Hua.  Two-stream consensus network for weakly-supervised temporal action localization.  In ECCV, 2020.
  * [70] Can Zhang, Meng Cao, Dongming Yang, Jie Chen, and Yuexian Zou.  Cola: Weakly-supervised temporal action localization with snippet contrastive learning.  In CVPR, 2021.
  * [71] Zhilu Zhang and Mert Sabuncu.  Generalized cross entropy loss for training deep neural networks with noisy labels.  In NeurIPS, 2018.
  * [72] Jia-Xing Zhong, Nannan Li, Weijie Kong, Tao Zhang, Thomas H. Li, and Ge Li.  Step-by-step erasion, one-by-one collection: A weakly supervised temporal action detector.  In ACMMM, 2018.
  * [73] Jingqiu Zhou, Linjiang Huang, Liang Wang, Si Liu, and Hongsheng Li.  Improving weakly supervised temporal action localization by bridging train-test gap in pseudo labels.  In CVPR, 2023.

###### Contents

  1. 1 Introduction
  2. 2 Related Work
  3. 3 Preliminaries and Baseline Setup
  4. 4 Our Method
    1. 4.1 Overview
    2. 4.2 Self-Labeling via Unified Optimal-Transport
    3. 4.3 Training and Testing
  5. 5 Experiments
    1. 5.1 Datasets and Evaluation Metric
    2. 5.2 Comparison with SOTA Methods
    3. 5.3 Ablation Study
  6. 6 Conclusion and Limitation
  7. 7 Acknowledgements
  8. A Notation
  9. B Architecture
  10. C Additional Ablation Experiments
    1. C.1 Ablation on multiple datasets
    2. C.2 Analysis of baseline model
    3. C.3 Analysis of ranking indices ${rank}$
  11. D Additional Visualizations
    1. D.1 Comparison to baseline
    2. D.2 Failure cases
  12. E Additional Discussion on Related Work
  13. F Implementation Details
    1. F.1 Baseline model
    2. F.2 Co-labeling
    3. F.3 Training details
    4. F.4 Testing details
  14. G Theoretical Derivation

##  Appendix A Notation

In Table 8, we summarize the key notations used in our paper for easy
reference.

##  Appendix B Architecture

Table 9 presents the details of the architecture of our proposed CASE. As can
be seen, the architecture is simple and lightweight, consisting of only a few
1D convolutional layers with a kernel size of 1 and linear layers, leading to
the high efficiency of our method.

| Notation | Shape | Description
---|---|---|---
Baseline | $\bm{P^{A}}$ | $N$ | foreground probability
$\bm{Q^{A}}$ | $N$ |  pseudo-labels of $\bm{P^{A}}$
$\bm{Z}$ | $N\times D$ | snippet embedding
SCC | $\bm{P^{S}}$ | $N\times K$ | cluster assignment probability
$\bm{Q^{S}}$ | $N\times K$ |  pseudo-labels of $\bm{P^{S}}$
$\bm{\hat{Q}^{S}}$ | $N\times K$ |  prior distribution of $\bm{Q^{S}}$
CCC | $\bm{P^{C}}$ | $K\times 2$ | cluster classification probability
$\bm{Q^{C}}$ | $K\times 2$ |  pseudo-labels of $\bm{P^{C}}$
$\bm{\beta^{C}}$ | $2$ |  prior marginal distribution of $\bm{Q^{C}}$
Testing | $\bm{P^{T}}$ | $N\times K$ | transformed foreground probability
$\bm{P^{M}}$ | $N\times K$ | fused foreground probability

Table 8: Key notations in this paper.

component | layer | kernel | stride | dim | act | output size
---|---|---|---|---|---|---
| Embedding Encoder
Baseline | $\operatorname{Conv1d}$ | 1 | 1 | 512 | $\operatorname{Relu}$ |  512$\times T$
Action Classifier
$\operatorname{Conv1d}$ | 1 | 1 | $G$ | $\operatorname{Softmax}$ | $G\times T$
Embedding Encoder
$\operatorname{Conv1d}$ | 1 | 1 | 512 | $\operatorname{Relu}$ |  512$\times T$
Attention layer
$\operatorname{Conv1d}$ | 1 | 1 | 1 | $\operatorname{Sigmoid}$ | $1\times T$
Our Algorithm | Clustering Head
$\operatorname{Linear}$ | 1 | 1 | $K$ | $\operatorname{Softmax}$ | $K\times T$

Table 9: The detailed architecture of CASE, where the RGB stream and optical
flow stream share the same structure.

##  Appendix C Additional Ablation Experiments

###  C.1 Ablation on multiple datasets

To show the effectiveness of our method in various scenarios, we perform a
component-wise ablation study for the snippet clustering component (SCC) and
the cluster classification component (CCC) on THUMOS14, ActivityNet v1.2 and
v1.3. The corresponding results are provided in Table 10. We observe
consistent trends across all datasets, indicating the robustness and
effectiveness of our approach.

###  C.2 Analysis of baseline model

We carry out several ablation experiments to analyze the components of the
baseline. The results are illustrated in Table 11. It can be seen that the
attention branch largely increases the performance, demonstrating the
significance of class-agnostic F&B separation. Additionally, we find that the
use of the generalized binary cross-entropy loss yields better results than
the traditional binary cross-entropy loss, proving that enhancing the label
noise tolerance is advantageous.

###  C.3 Analysis of ranking indices ${rank}$

In SCC, we use the distance between the normalized ranking indices of the
snippets ${rank}/N$ and the cluster-level pseudo-labels $\bm{Q^{C}}$ to
compute a 2D gaussian distribution. In principle, ${rank}/N$ can be replaced
by $\bm{P^{A}}$. However, we experimentally find that the performance of using
$\bm{P^{A}}$ is inferior to that of using ${rank}/N$ (i.e., $45.1$ for
$\bm{P^{A}}$ vs. $46.2$ for ${rank}/N$ on average mAP). To explain it, we show
the statistics ( i.e., maximum, average and minimum) of $\bm{P^{A}}$ and
$\bm{Q^{C}}$ in Fig. 8. The statistics are computed over each batch (i.e.,
iteration). Notably, the maximum, average, and minimum of ${rank}/N$ are
always $\frac{1}{N}\simeq 0$, $0.5+0.5\frac{1}{N}\simeq 0.5$ and $1$,
respectively. As can be seen, compared with $\bm{P^{A}}$, ${rank}/N$ is more
comparable to $\bm{Q^{C}}$. For example, both the average of $\bm{Q^{C}}$ and
the average of ${rank}/N$ are around $0.5$ and are evidently larger than the
average of $\bm{P^{A}}$. This observation confirms the validity of our
approach.

| THUMOS14 | ActivityNet v1.2 | ActivityNet v1.3
---|---|---|---
Baseline | 42.1 | 25.6 | 24.7
\+ SCC | 43.2 | 26.5 | 25.4
\+ SCC + CCC | 43.9 | 27.0 | 25.7
\+ SCC + CCC (T) | 46.2 | 27.9 | 26.8

Table 10: Component-wise ablation study on THUMOS14, ActivityNet v1.2 and
v1.3. ”(T)” indicates that the clustering-assisted testing technique is
appiled.

VTB |  ATB |  GBCE |  mAP
---|---|---|---
✓ |  |  |  32.0
✓ |  ✓ |  |  41.7
✓ |  ✓ |  ✓ |  42.1

Table 11: Ablation study on the baseline. VTB, ATB, and GBCE indicate video
classification branch, attention branch, and generalized binary cross-entropy
loss, respectively. Notably, if GBCE is not used, we use the traditional
binary cross-entropy loss to train the ATB. Figure 8: The maximum, average,
and minimum values of $\bm{P^{A}}$ and $\bm{Q^{C}}$ of each iteration during
training.  Figure 9: Comparison between our CASE and the baseline. The solid
and dashed boxes represent the regions where CASE outperforms and
underperforms the baseline, respectively.

##  Appendix D Additional Visualizations

###  D.1 Comparison to baseline

In Fig. 9, four visualized examples are provided to illustrate the differences
between the F&B separation results of CASE and that of baseline. It can be
observed that: 1) CASE is advantageous to capture fine-grained patterns of
snippets that are helpful to distinguish different snippets (see the solid
boxes). For instance, in the region of ’4’, which is near the boundary of a
’diving’ action instance, the foreground snippets and the background snippets
are visually similar. However, CASE can accurately classify these snippets
into correct F&B classes, whereas the baseline cannot, showing that CASE can
capture the underlying fine-grained structure of the snippets. 2) CASE
performs worse than the baseline in some ’suspicious’ regions (see the dashed
boxes). To name a few, in the region of ’8’, an athlete raises her leg,
causing CASE to mistake the region for an action instance. This mistake may be
avoided by the baseline model because the video-level labels used to train the
baseline can offer instructive information for the potential action types
within the videos.

Figure 10: Samples of failure cases. The dashed boxes represent the regions
with wrong predictions.

###  D.2 Failure cases

We showcase some examples of failure cases of our method in Fig. 10. From the
figure, we conjecture the reason that accounts for the failure cases are: 1)
low quality of images, e.g., ’1’ and ’8’; 2) indistinguishable body motions,
e.g., ’3’ and ’7’; 3) small objects, e.g., ’2’ and ’4’; 4) incorrect
annotation, e.g., ’5’ and ’6’. These challenging cases represent future
directions for our work.

##  Appendix E Additional Discussion on Related Work

In our main paper, we extensively discussed the differences between our method
and previous deep clustering and WTAL methods in the Related Work section. In
this section, we would like to provide additional insights on other related
methods.

In the snippet clustering component (SCC), we draw inspiration from the early
sequence-matching method [56] to construct a prior distribution for the
pseudo-labels of cluster assignments of snippets. However, our method differs
significantly from [56] in both purpose and solution. Specifically, [56] aims
to measure the distance between two sequences by matching the frames of one
sequence with the frames of another sequence with similar temporal positions.
Consequently, it constructs a prior distribution for the mapping between the
frames in different sequences based on their temporal locations. In contrast,
we aim to disambiguate the assignments between snippets and clusters by
enforcing the snippets with high foreground/background probabilities to be
assigned to the clusters with high foreground/background probabilities. To
achieve this, we construct the prior distribution for the cluster assignments
of the snippets based on the distance between the foreground probabilities of
snippets and the foreground probabilities of clusters. Besides, to better suit
our approach, we rank the snippets according to their foreground
probabilities, resulting in ranking indices that are more comparable with the
foreground probabilities of the clusters. This approach allows us to better
match the snippets to the appropriate clusters, as demonstrated in Sec. C.3.

Furthermore, our method is somewhat related to context-based methods [51, 38].
Previous context-based methods [51, 38] typically regard the context as a
special type of background. That is, they divide foreground and background
snippets into three latent groups: action, context, and normal background.
This approach provides a more detailed description of the background
distribution. Our method extends this approach by dividing snippets into
multiple latent groups, which allows for a more comprehensive description of
both the foreground and background distributions. The visualized results
(e.g., 5th row of Fig.6 in the main paper) reveal that some of the learned
clusters are very close to the concept of context. From this view, our model
already has some contextual modeling capabilities.

##  Appendix F Implementation Details

###  F.1 Baseline model

Here we present more details about the multiple instance learning (MIL) used
in the baseline. Specifically, we first calibrate T-CAS
$\bm{P^{V}}\in\mathbb{R}^{T\times G}$ with the attention weights
$\bm{P^{A}}\in\mathbb{R}^{T}$ to highlight foreground snippets and suppress
background snippets, resulting in the calibrated T-CAS (dubbed
$\bm{\hat{P}^{V}}\in\mathbb{R}^{T\times G}$). It can be implemented in
multiple ways. Here following [41, 51], we fuse the scores by weighted
summation, $\bm{\hat{P}^{V}}=\omega\bm{P^{V}}+(1-\omega)\bm{P^{A}}$. $\omega$
is a predefined weight. Thereafter, we select $K$ snippets from each video for
each class based on $\bm{\hat{P}^{V}}$:

$\displaystyle\small\Gamma_{c}=\arg\max_{\begin{subarray}\Gamma\subset\\{1,..,T\\}\\\
\quad|\Gamma|=K\end{subarray}}\sum_{\tau\in\Gamma}{\bm{\hat{P}^{V}}_{\tau,c}},$
(15)

where $K$ is a hyper-parameter. Temporal pooling is applied to the selected
snippets in $\Gamma_{c}$ to build video-level class prediction
$\bm{\bar{P}}\in\mathbb{R}^{G}$:

$\displaystyle\small{\bm{\bar{P}}_{c}=\mathop{Softmax}_{c}(\frac{1}{K}\sum_{\tau\in\Gamma_{c}}{\bm{P^{V}}_{\tau,c}}}).$
(16)

Finally, $\bm{\bar{P}}$ is used to compute a video classification loss, as
shown in the main paper.

###  F.2 Co-labeling

In our framework, there are several procedures of pseudo-labeling that can be
summarized with a unified formulation as $\bm{Q}=\Psi(\bm{P})$. Here $\bm{P}$
is the prediction of the model, $\Psi$ is the function of generating pseudo-
labels, $\bm{Q}$ is the pseudo-labels. To improve the quality of the pseudo-
labels, following [69], we propose to apply the two-stream co-labeling (TSCL)
strategy, which is model-agnostic and naturally compatible with our method.
That is, we aggregate the predictions of RGB and optical-flow streams to
generate the modality-sharing pseudo-labels, i.e.,
$\bm{Q}=\Psi(0.5\bm{P^{\text{RGB}}}+0.5\bm{P^{\text{Flow}}})$. To be specific,
for $\bm{Q^{C}}$, we fuse the cluster assignments of RGB stream (dubbed
$\bm{P^{C,\text{RGB}}}$) and that of Flow stream (dubbed
$\bm{P^{C,\text{Flow}}}$) by:

$\displaystyle\small\bm{P^{C}}=0.5\bm{P^{C,\text{RGB}}}+0.5\bm{P^{C,\text{Flow}}}.$
(17)

Then the pseudo-labels $\bm{Q^{C}}$ is generated by:

$\displaystyle\small\mathop{\min}_{\bm{Q^{C}}\in\Omega^{C}}\
\langle\bm{Q^{C}},-\operatorname{log}\bm{P^{C}}\rangle.$ (18)

As for $\bm{Q^{R}}$, the prediction of cluster classifier of RGB stream
(dubbed $\bm{P^{R,\text{RGB}}}$) and that of Flow stream (dubbed
$\bm{P^{R,\text{Flow}}}$) are combined as follows

$\displaystyle\small\bm{P^{R}}=0.5\bm{P^{R,\text{RGB}}}+0.5\bm{P^{R,\text{Flow}}}.$
(19)

Then the pseudo-labels $\bm{Q^{R}}$ is generated by:

$\displaystyle\small\mathop{\min}_{\bm{Q^{R}}\in\Omega^{R}}\
\langle\bm{Q^{R}},-\operatorname{log}\bm{P^{R}}\rangle.$ (20)

Moreover, the top-$K$ selection used in Eq. 15 can be regarded as a procedure
of defining the F&B snippets. Hence, we utilize the TSCL to improve the
quality of the top-$K$ selection. Specifically, we fuse the calibrated T-CAS
of RGB stream (dubbed $\bm{\hat{P}^{V,\text{RGB}}}$) and that of optical-flow
stream (dubbed $\bm{\hat{P}^{V,\text{Flow}}}$) as follows:

$\displaystyle\small\bm{\hat{P}^{V}}=0.5\bm{\hat{P}^{V,\text{RGB}}}+0.5\bm{\hat{P}^{V,\text{Flow}}}.$
(21)

Then $\bm{\hat{P}^{V}}$ is used for top-$K$ selection. Notably, the results of
the top-$K$ selection also influences the definition of $\bm{Q^{A}}$. we use
Eq. 15 to determine the foreground and background snippets. , which can
influence the learning of both the video classification module and attention
module.

In Table 12, we present an evaluation of the effect of the two-stream co-
labeling strategy on both the baseline model and our clustering-based F&B
algorithm. It can be seen that the TSCL is important to the baseline, boosting
its performance from $38.3\%$ to $42.1\%$. However, the additional use of the
TSCL in our algorithm results in only a small improvement compared to not
using the TSCL in our algorithm (from $45.6\%$ to $46.2\%$). This suggests
that the main reason for the performance improvement of our algorithm over the
baseline model is our proposed clustering-based approach, rather than the two-
stream co-labeling strategy.

Method | mAP
---|---
Baseline w/o TSCL | 38.3
Baseline w/ TSCL | 42.1
Baseline w/ TSCL + Our algorithm w/o TSCL | 45.6
Baseline w/ TSCL + Our algorithm w/ TSCL | 46.2

Table 12: Ablation study of two-stream co-labeling (TSCL).

###  F.3 Training details

TVL1 [68] is applied to extract optical-flow stream from RGB stream in
advance. Each stream is divided into $16$-frame snippets. Following
convention, we employ the I3D [6] network pre-trained on Kinetics-$400$ [6] to
extract snippet-level features from each stream, where the channel dimension
$D$ is $1024$. The number of sampled snippets $T$ is set to $750$ for THUMOS14
and $50$ for ActivityNet v1.2 and v1.3. Both streams share the same structure
but have separate parameters. The embedding encoders are comprised of a
temporal convolution layer with $512$ channels and a $\operatorname{ReLU}$
layer. The action classifier consists of a $\operatorname{FC}$ layer and a
$\operatorname{Softmax}$ layer. The clustering head is composed of a linear
cosine classifier [14] with a temperature of $10$ and a
$\operatorname{Softmax}$ layer. The attention layer consists of a
$\operatorname{FC}$ layer and a $\operatorname{Sigmoid}$ layer. We set the
classes $K$ of the clustering head to $16$ for THUMOS14 and $64$ for
ActivityNet v1.2 and v1.3. Following previous methods [51, 41], the $k$ for
top-$k$ selection is set to $T//8$ in THUMOS14 and $T//2$ in ActivityNet v1.2
and v1.3, while the batch size $B$ is set to $16$, the $\gamma$ is set to
$0.7$ and the $\omega$ is set to $0.25$ for all datasets. Following [5], the
$\epsilon$ is set to $20$. The temperature $\rho$ is set to $10$. The standard
deviation $\sigma$ is set to $10$. The loss weights are set as
$\lambda_{S}=1,\lambda_{C}=0.3$ for all datasets. We utilize Adam optimizer
with a learning rate of $10^{-4}$ for all datasets. We run each experiment
three times and report their mean accuracy for reliability. The model
implemented by Pytorch is trained on a Nvidia 1080Ti GPU.

###  F.4 Testing details

During inference, the video-level scores and snippet-level scores (_i.e_.,
T-CAS) of both the RGB stream and optical-flow stream are fused by averaging.
Then, a threshold is applied to the video-level scores to determine the action
categories. For the selected action class, a threshold strategy is applied to
the T-CAS, as done in [51, 13], to obtain action proposals. Next, the outer-
inner-contrastive technique [52] is used to calculate the class-specific score
for each proposal. To increase the pool of proposals, multiple thresholds are
applied, and non-maximum suppression (NMS) is employed to remove duplicate
proposals.

##  Appendix G Theoretical Derivation

Here we provide the derivation of the solution to the following optimal-
transport problem in SCC:

$\displaystyle\mathop{\min}\
\langle\bm{Q^{S}},-\operatorname{log}\bm{P^{S}}\rangle+\frac{1}{\epsilon}\operatorname{KL}(\bm{Q^{S}},\bm{\hat{Q}^{S}})\quad\textit{s.t.},\bm{Q^{S}}\in\Omega^{S}$
(22) $\displaystyle\Omega^{S}=\\{\bm{Q^{S}}\in\mathbb{R}_{+}^{N\times
K}|\bm{Q^{S}}\mathbf{1}^{K}=\bm{\alpha^{S}},{\bm{Q^{S}}}^{\top}\mathbf{1}^{N}=\bm{\beta^{S}}\\}.$

For notation simplicity, we remove the superscript $S$. Then the problem is
rewritten as

$\displaystyle\mathop{\min}\
\langle\bm{Q},-\operatorname{log}\bm{P}\rangle+\frac{1}{\epsilon}\operatorname{KL}(\bm{Q}||\bm{\hat{Q}})\quad\textit{s.t.},\bm{Q}\in\Omega$
(23) $\displaystyle\Omega=\\{\bm{Q}\in\mathbb{R}_{+}^{N\times
K}|\bm{Q}\mathbf{1}^{K}=\bm{\alpha},{\bm{Q}}^{\top}\mathbf{1}^{N}=\bm{\beta}\\}.$

To address the problem, we first write the Lagrangian function of Eq. 23 as
follows:

$\displaystyle\small\mathcal{L}(\bm{Q},\bm{\mu},\bm{\nu})=$
$\displaystyle\langle\bm{Q},-\operatorname{log}\bm{P}\rangle+\frac{1}{\epsilon}\operatorname{KL}(\bm{Q}||\bm{\hat{Q}})$
(24)
$\displaystyle+\bm{\mu}^{\top}(\bm{Q}\mathbf{1}^{K}-\bm{\alpha})+\bm{\nu}^{\top}({\bm{Q}}^{\top}\mathbf{1}^{N}-\bm{\beta})$
$\displaystyle=$
$\displaystyle\sum_{n=1}^{N}\sum_{k=1}^{K}(-\bm{Q}_{n,k}\operatorname{log}\bm{P}_{n,k}+\frac{1}{\epsilon}\bm{Q}_{n,k}\operatorname{log}\frac{\bm{Q}_{n,k}}{\bm{\hat{Q}}_{n,k}}$
$\displaystyle+\bm{\mu}_{n}\bm{Q}_{n,k}+\bm{\nu}_{k}\bm{Q}_{n,k})-\bm{\mu}^{\top}\bm{\alpha}-\bm{\nu}^{\top}\bm{\beta}$

where $\bm{\mu}\in\mathbb{R}^{N}$ and $\bm{\nu}\in\mathbb{R}^{K}$ are the dual
variables so that $\bm{Q}\mathbf{1}^{K}=\bm{\alpha}$ and
${\bm{Q}}^{\top}\mathbf{1}^{N}=\bm{\beta}$. The derivative of
$\mathcal{L}(\bm{Q},\bm{\mu},\bm{\nu})$ w.r.t. $\bm{Q}_{n,k}$ is:

$\displaystyle\small\frac{\partial\mathcal{L}(\bm{Q},\bm{\mu},\bm{\nu})}{\partial\bm{Q}_{n,k}}=-\operatorname{log}\bm{P}_{n,k}+\frac{1}{\epsilon}\operatorname{log}\frac{\bm{Q}_{n,k}}{\bm{\hat{Q}}_{n,k}}+\frac{1}{\epsilon}+\bm{\mu}_{n}+\bm{\nu}_{k}.$
(25)

Note that the optimal $\bm{Q}$ exists and is unique, as both the objective and
the constraint in Eq. 23 are convex. Hence, to obtain the optimal $\bm{Q}$, we
set
$\frac{\partial\mathcal{L}(\bm{Q},\bm{\mu},\bm{\nu})}{\partial\bm{Q}_{n,k}}=0$,
and then get:

$\displaystyle\small\bm{Q}_{n,k}=e^{-\frac{1}{2}-\epsilon\bm{\mu}_{n}-\frac{1}{2}}(\bm{\hat{Q}}_{n,k}{\bm{P}^{\epsilon}_{n,k}})e^{-\frac{1}{2}-\epsilon\bm{\nu}_{k}}.$
(26)

Let us denote $\bm{S}=\bm{\hat{Q}}\cdot{\bm{P}^{\epsilon}}$. Obviously, all
elements of $\bm{S}$ are strictly positive. According to [55, 2, 56], there
exist diagonal matrices $\operatorname{diag}(\bm{u})$ and
$\operatorname{diag}(\bm{v})$ with strictly positive diagonal elements so that
$\operatorname{diag}(\bm{u})S\operatorname{diag}(\bm{v})$ belongs to $\Omega$.

In summary, the optimal $\bm{Q}$ has the form as:

$\displaystyle\small\bm{Q}=\operatorname{diag}(\bm{u})\bm{S}\operatorname{diag}(\bm{v})=\operatorname{diag}(\bm{u})(\bm{\hat{Q}}\cdot{\bm{P}^{\epsilon}})\operatorname{diag}(\bm{v}),$
(27)

where $\bm{u}\in\mathbb{R}^{N}$ and $\bm{v}\in\mathbb{R}^{K}$ are two
renormalization vectors that make the resulting matrix $\bm{Q}$ to be a
probability matrix. Throughout our work, we follow [5] to implement the
algorithm due to its conciseness. Formally, Eq. 27 is replaced as follows:

$\displaystyle\small\bm{Q}=\operatorname{diag}(\bm{u})(\bm{\hat{Q}}\cdot{\exp(\epsilon\bm{L}}))\operatorname{diag}(\bm{v}),$
(28)

where $\bm{L}$ indicates the logits before the $\operatorname{Softmax}$ layer,
namely $\bm{P}=\operatorname{Softmax}(\bm{L})$. Note that Eq. 27 and Eq. 28
are equivalent in principle. The main difference lies in the placement of the
factor $\epsilon$ that sharpens the labels. In Eq. 27, the factor is applied
before $\operatorname{Softmax}$, while in Eq. 28, it is applied after
$\operatorname{Softmax}$. Similarly, for the cluster classification component
(without $\bm{\hat{Q}}$), we can obtain the solution as follows:

$\displaystyle\small\bm{Q}=\operatorname{diag}(\bm{u}){\exp(\epsilon\bm{L}})\operatorname{diag}(\bm{v}).$
(29)

Both Eq. 28 and Eq. 29 can be efficiently computed using the iterative
Sinkhorn-Knopp algorithm [10]. We refer to [10] for more details. This
algorithm is highly efficient on GPU as it only involves a couple of matrix
multiplication, enabling online computation.