# Distill and Collect for Semi-Supervised Temporal Action Segmentation

Sovan Biswas  Anthony Rhodes  Ramesh Manuvinakurike  Giuseppe Raffa  Richard
Beckwith
Intel Labs
<EMAIL_ADDRESS>
###### Abstract

Recent temporal action segmentation approaches need frame annotations during
training to be effective. These annotations are very expensive and time-
consuming to obtain. This limits their performances when only limited
annotated data is available. In contrast, we can easily collect a large corpus
of in-domain unannotated videos by scavenging through the internet. Thus, this
paper proposes an approach for the temporal action segmentation task that can
simultaneously leverage knowledge from annotated and unannotated video
sequences. Our approach uses multi-stream distillation that repeatedly refines
and finally combines their frame predictions. Our model also predicts the
action order, which is later used as a temporal constraint while estimating
frames labels to counter the lack of supervision for unannotated videos. In
the end, our evaluation of the proposed approach on two different datasets
demonstrates its capability to achieve comparable performance to the full
supervision despite limited annotation.

##  1 Introduction

Segmenting and understanding a long video sequence is very important for many
applications, such as surveillance, intelligent advertisement, _etc_. It is of
high importance to automatic human assistance systems where video sequences
tend to have a temporally ordered list of actions. For example, an
instructional video of ‘making coffee’ tends to have a temporal order of
actions such as ‘add milk’, ‘take cup’, ‘stir’, _etc_. Recently, several
approaches have achieved significant success in segmenting activities in
videos [19, 21, 9, 38, 6]. Despite their success, these approaches are limited
by their need for time-consuming and costly-to-obtain frame-level annotation.
To overcome these need for frame annotations, many have started to explore
approaches to train with weaker supervision in the form of timestamp
supervision [25, 27, 26], transcripts [2, 31, 22, 35], or even sets [10, 30,
23]. In timestamp supervision, only a single frame of each action segment is
annotated. At the same time, the videos are annotated with an ordered list of
actions devoid of the starting and ending time in transcript-level
supervision. Finally, in set-level supervision, only the set of actions are
used in training. These set level annotations lack any order information or
indication of the repetition for each action within a video.

Figure 1: In a semi-supervised setting, we have a combination of a
considerable number of unannotated videos and a few annotated video sequences.
The aim is to leverage additional information from the ‘in-domain’ unannotated
video sequences to boost the performance. As annotating frame labels are
costly, semi-supervision can help in scaling the training data without
additional cost.

On the one hand, weak supervision, such as transcripts or sets, drastically
reduces the annotation cost, the overall performance is far from satisfaction
as these approaches often fail to correctly align actions in time. On the
other hand, an annotator needs to go through each video in detail to generate
timestamp supervision. This is a significant bottleneck towards scaling the
training data substantially without incurring a high cost. In this paper, we
propose a semi supervision approach for the action segmentation task that
addresses the limitations of the current level of supervisions. Our semi-
supervised data consist of a small amount of detailed frame annotated videos
and a large corpus of unannotated videos as shown in Fig 1. Though at a
limited quantity, the use of frame labels provides more substantial
supervision than the transcript or set annotation, as it helps to locate and
better understand the action variations within a segment.

Due to the lack of substantial supervision in unannotated videos, leveraging
the additional knowledge from these videos can be challenging. A simple
approach to solve this problem would be to use the confident frames in the
unannotated video based on a threshold as additional frame labels. This,
however, raises a question in the form of defining the confidence as
classification confidence grows over the training cycle. Further, any early-
stage misidentification might propagate the error in later stages - resulting
in poor overall performance. Moreover, any model can easily overfit the
training data due to few annotations, resulting in false confidence on
unannotated videos. Finally, longer video sequences tend to have temporal
order across various action segments, which the above approach fails to
consider.

In this work, we propose a different approach where we use predictions from
multiple streams of segmentation models to compute temporally ordered frame
labels on the unannotated data. Despite limited annotations, the proposed
multiple streams generate more accurate and robust predictions by repeated
refinement and accumulation. Further, our method also learns to predict an
ordered list of actions with a sequence-to-sequence model for the unannotated
videos. This predicted ordered list is later used as an alignment constraint
to maintain the temporal order while estimating frame labels based on the
frame predictions. Any error in the temporal order prediction can
significantly impact frame label computing. We, therefore, use beam search to
generate multiple ordered candidates to find the optimal matching. This
ensures better chances of estimating the correct frame labels for the
unannotated data.

Our contributions thus are two folds as follows:

  * •

We introduce an approach to train a temporal action segmentation model from
semi supervision. The approach computes frame labels for an unannotated video
aligned to the predicted temporal order of actions.

  * •

We propose multi-stream segmentation models that repeatedly refine and
accumulate predictions to reduce any noise that may arise from overfitting on
the limited training data.

##  2 Related Work

Fully Supervised Action Segmentation: The goal of action recognition [33, 3,
11] has been to classify short trimmed videos. This is quite different from
temporal action segmentation where one requires to capture long-range
dependencies in order to classify each frame of the input video. Many
approaches in past combined frame-wise classifiers with grammars [37, 28] or
with hidden Markov models (HMMs) [20, 16, 18], to capture this dependency.
With the recent advancement of neural networks, many recent approaches
utilized temporal convolutional networks to capture long-range dependencies
for the temporal action segmentation task [19, 21]. Despite accurate
predictions, these approaches still suffer from an over-segmentation problem.
In order to overcome this problem, recent state-of-the-art methods proposes a
multi-stage architecture with dilated temporal convolutions [9, 38, 24, 13,
14, 5]. Chen et al. [6] further enhances the performance by utilizing activity
level domain adaptation to compute consistent action labels and reduce over-
segmentation. Yet, these approaches rely on each from annotation resulting in
high cost towards real world deployment. On the contrary, in this paper, we
perform the temporal action segmentation task in a semi-supervised setup.

Weakly Supervised Action Segmentation: To circumvent the need for expensive
annotations of full-supervision, weakly supervised action segmentation has
been gaining significant interest recently. Early approaches, in this context,
used discriminative clustering based on distinctive action segments to detect
and locate temporal action segments using movie scripts [2, 8]. Despite the
apparent presence of temporal ordering in the scripts, these approaches failed
to harness them in a meaningful manner. However, Bojanowski et al. [2] were
the first to use an ordered list of actions as supervision, thus explicitly
modeling the action order to detect and segment actions in a video. Due to the
inherent ordering of the actions and the video frames, many researchers
recently started addressing the task as an alignment problem between video
frames and the transcripts using connectionist temporal classification [12],
dynamic time warping [4] or energy-based learning [22]. On the other hand,
approaches such as [17, 7, 29, 18] iteratively generate and refine pseudo
ground truth labels for the training. Few approaches, such as [31, 35, 34],
use the temporal ordering constraints over-frame predictions to generate the
target labels. Lastly, some approaches [10, 30, 23] tried tackling the action
segmentation problem with a much weaker set level supervision without the
temporal order. Eventhough these approaches have improved recently, their
qualitative performance is far from that of fully supervised approaches due to
errors in aligning these actions in time. Bridging the gap, recently [25]
proposed an approach with timestamp supervision that focused on estimating the
action transition in time. Even though we aim to reduce the annotation cost
similar to these approaches, we also want to leverage additional information
from unlabeled data. This will help in scaling the training data without any
additional cost.

##  3 Semi-Supervised Temporal Action Segmentation

The task of the temporal action segmentation task is to predict frame action
labels for a given input video. Formally, for a given video of $T$ frames
$X=[x_{1},\ldots,x_{T}]$, the objective is to obtain frame action labels
$Y=[y_{1},\ldots,y_{T}]$. Here $x_{t}\in\mathbb{R}^{D}$ and $y_{t}\in C$ are
the frame embedding and action label at time $t$ respectively. $C$ is the set
of action classes. The frame labels $Y$ can also be perceived as sequence of
$N$ action steps $S=[s_{1},\ldots,s_{N}]$. Here, $s_{n}\in C$ is the $n^{th}$
action step. This ordered sequence of steps $S=[s_{1},\ldots,s_{N}]$ is also
known as video transcript. Note the two representations $Y$ and $S$ for are
related.

For the current semi-supervised formulation, we consider that only a tiny
percentage of the training videos ($X_{s}$) have frame-level supervision
($Y_{s}$). We scale the training data with additional videos ($X_{u}$) with
neither frame annotations nor action transcripts. Note, both supervised and
unsupervised videos are assumed to be ‘in-domain’. The focus is to leverage
additional information from unannotated data and improve the overall
performance paving the way towards scaling training data with less cost. We
now present the proposed method in detail.

##  4 Proposed Method

Figure 2: The figure showcases the overview of our method. All bidirectional
arrows indicate both the forward and backward pass. During training, we feed a
sequence of pre-computed frame features into our proposed multi-stream MSTCN++
to predict frame-level action predictions. For the annotated video sequence,
we use these predictions to train our ‘Transcription Generation Module’ and
‘Multi-Stream MSTCN++’ using $\mathcal{L}_{s}^{g}$ and $\mathcal{L}_{s}^{f}$
respectively. Here, the Transcription Generation Module’ predicts the $M$
possible candidate transcripts for an unannotated video sequence. We, then,
compute the frame labels $\hat{Y}_{u}$ for these videos by finding an optimal
match in the Sequence Matching Module. Lastly, the optimally matched frame
labels $\hat{Y}_{u}$ is also used to further train both ‘Transcription
Generation Module’ and ‘Multi-Stream MSTCN++’ using $\mathcal{L}_{u}^{g}$ and
$\mathcal{L}_{u}^{f}$.

Figure 2 illustrates the overview of the proposed method. Here, the objective
of the method is to leverage additional information from unannotated training
videos. This leveraging can be done by computing frame labels for those
videos. Despite all these videos being ‘in-domain’, computing frame labels for
an unannotated video is not trivial. Moreover, wrong labels might confuse the
model and thus impacting the overall performance. We know that videos have a
natural order of actions _e.g_. while making an omelet, pour oil always
precede pour egg action. Thus, our approach focuses on predicting the action
order/transcripts for the videos. We argue that the action transcripts should
be aligned to their frame action predictions and thus can be used as
constraints while computing frame labels for the unannotated videos.
Furthermore, noisy or erroneous predictions can impact the label generation
despite the transcript constraints. Thus, we also propose multi-stream
distillation and collection to repeatedly refine the predictions in a single
forward pass in an end-to-end manner.

We first present our temporal action segmentation model based on multi-stream
distillation and collection in Section 4.1. It is then followed by the details
of sequence-to-sequence modeling used for the transcript generation model in
Section 4.2. In Section 4.3, we describe our sequence matching module to
compute the additional frame labels. Finally, we provide the details of all
the loss functions in Section 4.4.

###  4.1 Distill and Collect: Multi-Stream Predictions

As shown in Figure 2, we generate the additional frame labels and the action
transcripts based on the frame action predictions. Thus, any noisy predictions
can impact the overall performance. To reduce the noise, we propose a
refinement strategy based on multiple streams of segmentation. The objective
of the refinement strategy is to repeatedly distill the next segmentation
stream and finally collect all the frame predictions across multiple streams.

####  4.1.1 MSTCN++: Baseline Model

Our proposed approach uses multiple streams of MSTCN++[9, 24] segmentation
models. We first briefly describe the standard MSTCN++ model, which is the
current state-of-the-art architecture for action segmentation. The
architecture uses pre-computed frame features $X=[x_{1},\ldots,x_{T}]$ to
compute frame labels $\hat{Y}=[\hat{y}_{1},\hat{y}_{2},\ldots,\hat{y}_{t}]$.
It is performed by multiple stages of the temporal convolutions network. The
initial stage is also known as prediction or generation stage, consisting of a
dual dilated layer that combines two temporal convolutions with different
dilation factors. It is, then, followed by multiple layers of a single-stage
temporal convolution network as known as  SS-TCN. A combination of frame-wise
cross-entropy loss and smoothing loss [9, 24] is used to train the
segmentation model.

In the current semi-supervised setting, where $(X_{s},Y_{s})$ and
$(X_{u},\hat{Y}_{u})$ corresponds to annotated and un-annotated video
sequence, our frame-wise losses are defined as follows:

$\displaystyle\mathcal{L}_{s}^{f}=\frac{1}{T}\sum_{t}{-\log\hat{y}_{t,c}}+\beta*\frac{1}{TC}\sum\hat{\Delta}_{t,c}^{2}$
(1)
$\displaystyle\mathcal{L}_{u}^{f}=\alpha*\frac{1}{T}\sum_{t}{-\log\hat{y}_{t,\hat{c}}}+\beta*\frac{1}{TC}\sum\hat{\Delta}_{t,c}^{2}$
(2)

Here, $\mathcal{L}_{s}^{f}$ and $\mathcal{L}_{u}^{f}$ are the frame loss over
the annotated and unannotated videos, respectively. $\hat{y}_{t,c}$ and
$\hat{y}_{t,\hat{c}}$ are the predicted probability for true class $c$ and
estimated-label $\hat{c}$, respectively. Same as in [9, 24],
$\hat{\Delta}_{t,c}=\min(\tau,|\log{\hat{y}}_{t,c}-\log{\hat{y}}_{t-1,c}|)$
and $\tau=4$ and $\beta=0.15$. $\alpha$ is a weight factor to control the
impact of computed-labels as they might be erroneous.

####  4.1.2 Distill - Learn from previous stream:

Figure 3: Distill and Collect - Multiple streams. The Base MSTCN++ is only
trained with ground-truth frame-annotations ($Y_{s}$). Whereas in the later
streams, the predictions of the previous stream are distilled to the first
stage of the MSTCN. Note, each refinement stream is additionally trained with
both ground truth ($Y_{s}$) and estimated frame labels ($\hat{Y}_{u}$).

In 4.1.1, we have explained how a single vanilla stream MSTCN++ is trained
using ground truth frame labels and estimated labels. However, the estimated
frame labels for unannotated videos are computed based on the model
predictions, as explained later in Section 4.3. Hence, computing robust and
generalized frame predictions is imperative to reduce subsequent frame label
computation errors. Similar to [36], we can use multiple streams to computer
robust. However, it has some limitations: 1) It is very time-consuming as each
stream requires individual computation for label computation; 2) Estimated
labels for each stream may vary, which will impact the convergence; 3) It
fails to harness the internal structure of MSTCN++.

Instead, we propose distillation across multiple streams as shown in Figure 3.
Note, the distilled knowledge is only passed to the first stage of the
following MSTCN stream instead of the final stage of MSTCN. This enables the
following MSTCN to build upon the distilled knowledge. We define our
distillation loss ($\mathcal{L}^{d}$) as follows:

$\displaystyle\mathcal{L}^{d}_{l}=\frac{1}{TC}\sum_{t,c}\min(\tau,|\log{\hat{y}}^{l}_{t,c}-\log{\hat{y}}^{l-1}_{t,c}|)$
(3) $\displaystyle\mathcal{L}^{d}=\sum_{l=2}^{L}\mathcal{L}^{d}_{l}$ (4)

Here, $l$ and $L$ denotes the index of a stream and the total number of
streams, respectively. $\hat{y}^{l}_{t,c}$ is the predictions of $l^{th}$
stream at time $t$ for a class label $c$. $\tau=4$.

Note, the $1^{st}$ stream is only trained with ground truth labels from the
annotated videos. Although less generalized, the usage of clean labels at the
first stream ensures no-incorrect seed predictions are passed onto the
refinement streams. However, in the refinement streams, we additionally use
the estimated labels to increase the generalization.

####  4.1.3 Collect - Multiple Streams:

Even though using multi-stream distillation, as explained in 4.1.2, helps in
refining predictions, ambiguity in predictions still exists. This is
especially true at time instants where action transition takes place in the
video. We further reduce the noise in predictions by collecting the
predictions of all the streams. This collection step is defined as follows:

$\log{\hat{y}}_{t,c}=\frac{1}{L}\sum_{l}\log{\hat{y}}^{l}_{t,c}$ (5)

$l$ and $L$ denotes the index of a stream and the total number of streams,
respectively. $\hat{y}^{l}_{t,c}$ is the predictions of $l^{th}$ stream at
time $t$ for a class label $c$.

Note, our usage of multiple streams increases the size of the overall model in
comparison to a single MSTCN++. Post-training, this might be a bottleneck
during inference of a test sample where memory is limited. In such cases, the
predictions only from the final MSTCN++ stream can be used. Experiments at
Section 5.4.6 showcases the inference performance tradeoff when using the
final stream in comparison to the overall model.

###  4.2 Transcript Generation Module:

In this section, we present the details of our transcript generation module
that will be used in the computation of frame labels for the unannotated
videos in Section 4.3. The objective of the transcript generation module is to
translate the temporal frame probabilities of a video into its action
transcript. To this end, we use sequence-to-sequence model with a
bidirectional LSTM encoder to encode the input sequence. This is followed by
an LSTM with MLP attention as decoder [1]. It is well known that such
sequence-to-sequence model is very ineffective in handling long sequences.
This is a critical problem as the input video sequence can be significantly
longer for such models to handle. However, due to high similarity between
consecutive frames, most temporal information is similar and redundant. Thus,
we reduce the effective length of the input video sequence by dividing it into
$K$ non-overlapping temporal segments. We then temporally ‘maxpool’ the class
probabilities within these segments to collapse them into $K$ length temporal
sequence as shown in Figure 2. In this way, we can significantly reduce the
temporal length of the video sequence needed for the sequence-to-sequence
model to generate action transcripts effectively.

Ideally, we expect the decoder of our transcript generation module to generate
the exact action order for those videos. However, in reality, our transcripts
may contain a few errors. We use beam-search decoding to generate $M$ possible
candidate transcripts for estimating the frame labels, to counteract such
errors. Further, if we have additional knowledge about the unannotated video,
such as the knowledge of the overall activity, we can add the domain-
heuristics during beam-search to generate more relevant transcripts. _e.g_.
while preparing coffee, it is highly unlikely that the action crack egg is
performed. Thus, our decoding can ensure the action crack egg is not present
in the candidate transcripts.

We train our Transcript Generation Module with both annotated videos and
unannotated videos. The video transcript $S_{s}$ is easily obtained from the
frame labels $Y_{s}$ to train upon for annotated videos. However, the
unannotated videos lack any such information. Instead, we use the best
$\hat{S}_{u}$ among the $M$ candidates after our Sequence Matching Module
explained later in Section 4.3 to train our transcript generation module upon.
We define our transcript predictions loss over both annotated and unannotated
videos as follows:

$\displaystyle\mathcal{L}_{s}^{g}=\frac{1}{N+1}\sum_{n}{-\log\hat{s}_{n,c}}$
(6)
$\displaystyle\mathcal{L}_{u}^{g}=\alpha*\frac{1}{N+1}\sum_{n}{-\log\hat{s}_{n,\hat{c}}}$
(7)

Here, $\mathcal{L}_{s}^{g}$ and $\mathcal{L}_{u}^{g}$ are a cross-entropy loss
over ground truth and estimated transcripts respectively. $N$ is the length of
the transcript excluding the EOS token. Note, $\hat{s}_{n,c}$ and
$\hat{s}_{n,\hat{c}}$ are $n^{th}$ step transcript predictions for annotated
step $c$ and estimated step $\hat{c}$ respectively. $\alpha$ is a weight
factor to control the impact of estimated transcripts as they might be
erroneous.

###  4.3 Sequence Matching Module:

Given the possible set of transcripts $\mathcal{S}_{u}$ and the frame
probabilities $\hat{y}_{t}$, our aim is to compute the frame-labels
($\hat{Y}_{u}$) for the unannotated videos. This will enable us to utilize the
additional labels from the unannotated videos to train the model better. To
this end, we use Dynamic Time Warping [32] to find an optimal alignment
between a transcript and the frame predictions. We select the alignment
corresponding to the least alignment scores over $M$ candidate transcripts.
Note, $|\mathcal{S}_{u}|=M$. The overall sequence matching module is defined
as follows:

$\displaystyle
Y_{S}^{*}=\operatorname*{arg\,min}_{Y\in\mathcal{Y}}\langle{Y,C_{S}}\rangle$
(8)
$\displaystyle\hat{Y}_{u},\hat{S}_{u}=\operatorname*{arg\,min}_{Y_{S}^{*},S\in\mathcal{S}_{u}}\langle{Y_{S}^{*},C_{S}}\rangle$
(9) $\displaystyle C_{S}(t,n)=1-\hat{y}_{t,s_{n}}$ (10)

(8) is the equation for the DTW where $C_{S}$ is the cost matrix between frame
probabilities and a candidate transcript $S$. $Y_{S}^{*}$ is a optimal
alignment of the frame predictions with a candidate transcript $S$.
$\hat{Y}_{u}$ and $\hat{S}_{u}$ are computed frame labels and transcripts
based on DTW matching over $M$ candidate transcripts. $\hat{y}_{t,s_{n}}$ is
the action probability at time $t$ and action ($s$) at transcript step $n$.

###  4.4 Loss Functions:

As shown in (1) and (2), MSTCN++ combines cross-entropy loss and smoothing
loss over both ground-truth and estimated labels. The distillation loss in (4)
enables the model to refine the predictions across multiple streams
repeatedly. Finally, the transcript prediction loss from (6) and (7) helps in
predicting the action order that is used to generate the estimated labels on
the unannotated data. Thus, the overall loss is defined as:

$\displaystyle\mathcal{L}_{s}=\mathcal{L}_{s}^{f}+\mathcal{L}_{s}^{g}$ (11)
$\displaystyle\mathcal{L}_{u}=\mathcal{L}_{u}^{f}+\mathcal{L}_{u}^{g}$ (12)
$\displaystyle\mathcal{L}=\mathcal{L}_{s}+\mathcal{L}_{s}+\beta*\mathcal{L}_{d}$
(13)

Here, $\mathcal{L}_{s}$ and $\mathcal{L}_{u}$ are the overall losses for
annotated videos and unannoated videos respectively. Note, we set $\alpha=0.3$
in (1) and (7) and $\beta=0.15$ for our experiments.

##  5 Dataset and Experiments

###  5.1 Datasets and Metrics

Datasets: We evaluate our approach on two datasets: Breakfast [15], and
Hollywood Extended [2].

The Breakfast dataset consist of 1712 videos of breakfast preparing activities
with roughly 3.6M frames. Here, the frames are annotated with 48 action
classes. The dataset has four splits. We follow the leave-one-split-out
strategy for evaluation as explained in [15]. However, to mimic annotated and
un-annotated data for the training, we additionally considered leave-one-split
annotated, thus effectively reducing the amount of annotation roughly by
$1/3$. Note, at any moment, one split is used for testing, another one split
as annotated data, and the rest two are unlabeled data. Our final result is an
average of all possible split combinations.

The Hollywood extended dataset contains 937 video sequences taken from
Hollywood movies. The videos contain 16 different action classes. We divided
that data into training and testing sets of 843 and 94 videos, respectively.
This corresponds to seed 5 of the original setting [2]. We, further, divide
the training set into three equal splits of 281 videos. To mimic annotated and
unannotated data, we assumed only one training split is annotated, and the
rest two are unannotated, thus reducing the amount of available annotation by
$1/3$. The final result over the test set is the average of all possible
combinations of annotated and unannotated training splits.

Metrics: We use the standard metrics for temporal action segmentation such as
frame accuracy (MoF), segmental edit distance (Edit), and segmental F1 scores
at overlapping thresholds $10\%$, $25\%$ and $50\%$ on the Breakfast dataset.
For Hollywood extended, we use the standard non-background frame accuracy
(MoF-BG) and intersection over Detection (IoD) as explained in [16].
Additionally, we also report the segmental edit distance (Edit) and segmental
F1 scores at overlapping thresholds $10\%$, $25\%$, and $50\%$ for the
Hollywood dataset.

Baselines: Our baseline is a standard MSTCN++ [24] when trained only with the
annotated data, using the above experimental framework of only one split being
annotated. Whereas, MSTCN++ [24] trained on original training data is
considered as full-supervised

###  5.2 Implementation Details:

As explained used MSTCN++ [24] as our base model. We train our model for 12000
steps with Adam optimizer. To minimize the impact of initialization, we
compute the frame-wise cross-entropy loss and segment-wise cross-entropy loss
on the unannotated data only after the first 2000 steps. We set the learning
rate to 0.0005, and the batch size to 3. We use the I3D features per frame for
both Breakfast and Hollywood Extended as input to our model, respectively.

###  5.3 Comparison with the Baselines:

In this section, we compare the proposed approach for semi-supervised action
segmentation against a simple baseline (as explained before) that fails to
utilize the additional annotated data. The results on the two datasets are
shown in Table 1. Despite training on a few annotated data, our approach
significantly outperforms these baselines in all the evaluation metrics. It
achieves nearly $0.9$ times MoF and $0.87$ times MoF-BG on the full supervised
Breakfast and Hollywood extended dataset, respectively.

Table 1: Comparison of the proposed approach with the baseline on both Breakfast and HollywoodExtended Dataset Breakfast |  F1 = {$0.1$,$0.25$,$0.5$} | Edit | MoF
---|---|---|---
Baseline | $52.1$ | $46.4$ | $34.4$ | $50.4$ | $54.2$
Semi-Supervision | 61.5 | 55.2 | 41.7 | 59.9 | 60.1
Full-Supervision | $64.1$ | $58.6$ | $45.9$ | $65.6$ | $67.6$
Hollywood |  F1 = {$0.1$,$0.25$,$0.5$} | Edit | MoF-BG
Baseline | $29.8$ | $25.2$ | $16.1$ | $32.0$ | $35.4$
Semi-Supervision | 32.9 | 28.7 | 18.8 | 35.1 | 38.4
Full-Supervision | $34.5$ | $31.8$ | $21.4$ | $38.9$ | $44.1$

###  5.4 Ablation Studies:

####  5.4.1 Impact of Domain Knowledge:

Table 2: Performance impact of using correct transcripts to estimate labels on the additional videos. No-Heuristics, Domain-Heuristics and Mixed-Supervision corresponds to different level of transcript correctness. The table showcases the result over all the splits of the Breakfast dataset. |  F1 = {$0.1$,$0.25$,$0.5$} | Edit | MoF
---|---|---|---
No-Heuristics | $61.5$ | $55.2$ | $41.7$ | $59.9$ | $60.1$
Domain-Heuristics | $63.9$ | $57.6$ | $43.7$ | $61.8$ | $62.9$
Mixed-Supervision | $64.2$ | $58.1$ | $44.4$ | $62.9$ | $63.5$

The crux of the proposed algorithm lies in predicting the correct transcripts
for the unannotated videos. Any error in the transcripts impacts the sequence
matching module and the subsequent estimation of frame labels. Thus, we
compare our results (No-Heuristics) against the scenario where we have been
provided with ground-truth transcripts for the unannotated videos. We denote
it as mixed supervision. Note, we do not need our sequence generation module
in such a scenario. Additionally, we experimented with the case where our
action predictions are constrained by the domain-heuristics of the videos (as
explained in 4.2). Table 2 showcases the result on the Breakfast dataset with
increasing order of transcript correctness. Despite having no prior knowledge
about the video content, our approach can still achieve comparable results to
mixed supervision.

####  5.4.2 Impact of amount of Labelled Data:

The amount of labelled data impacts the performance of the temporal
segmentation. Our approach improves its performance with the increase in
labelled data as seen by results on Split 3 (test set) as shown in Table 3.
$100\%$ corresponds to the result with full supervision. We also display that
our approach achieves nearly $0.77\times$ frame wise accuracy with
$\frac{1}{5}$ annotation in comparison to full annotation.

Table 3: The experiment was performed on Split 3 of the Breakfast dataset. The total training data size remains the same here, including both with frame labels and no labels. $\%$ of labels |  F1 = {$0.1$,$0.25$,$0.5$} | Edit | MoF
---|---|---|---
$20\%$ | $59.3$ | $53.4$ | $39.9$ | $59.7$ | $53.6$
$35\%$ | ${61.5}$ | ${55.2}$ | ${42.1}$ | ${61.3}$ | ${60.7}$
$45\%$ | ${63.3}$ | ${57.9}$ | ${45.0}$ | ${60.1}$ | ${64.6}$
$100\%$ | ${65.6}$ | ${61.5}$ | ${50.0}$ | ${63.7}$ | ${69.7}$

####  5.4.3 Impact of Multiple-Stream ($L$):

The proposed approach accumulates the prediction of multiple streams to have a
consistent prediction. This is shown by the quantitative improvement in Table
4. In this experiment, we utilize the Breakfast dataset with domain
heuristics. The results improve from a frame accuracy of $57.9\%$ to $62.9\%$
when the number of streams increases from $2$ to $4$. We also experimented
with full-supervised models and usage of multiple streams. Our experiments
improve the frame accuracy from $67.6\%$ to $70.1\%$ when the number of
streams is increased from $1$ to $4$. Note, there is no change in each stream
of the MSTCN++ model during full-supervision apart from the use of proposed
distillation (4) and collection (5).

Table 4: Performance impact of using multiple streams on the split of Breakfast dataset. $L$ denotes the number of streams in the multi-stream architecture. Note, this experiment used domain-heuristics during the semi-supervised training. Semi-Supervision |  F1 = {$0.1$,$0.25$,$0.5$} | Edit | MoF
---|---|---|---
$L=2$ | $57.3$ | $51.2$ | $38.2$ | $56.0$ | $58.5$
$L=3$ | $62.7$ | $56.2$ | $42.7$ | $61.0$ | $61.5$
$L=4$ | 63.9 | 57.6 | 43.7 | 61.8 | 62.9
Full-Supervision |  F1 = {$0.1$,$0.25$,$0.5$} | Edit | MoF
$L=1$ | $64.1$ | $58.6$ | $45.9$ | $65.6$ | $67.6$
$L=4$ | 67.1 | 61.9 | 50.4 | 64.2 | 70.1

####  5.4.4 Impact of Distillation vs Collection:

Table 5: Performance impact of distillation and collection step. The experiments was performed only on the split 1 of the Breakfast dataset. Approach |  F1 = {$0.1$,$0.25$,$0.5$} | Edit | MoF
---|---|---|---
Baseline | $56.1$ | $50.5$ | $38.3$ | $54.6$ | $56.7$
Only Distil | $58.3$ | $52.4$ | $38.9$ | $59.7$ | $58.9$
Only Collect | $62.6$ | $56.4$ | $41.8$ | $61.1$ | $62.1$
Both | 63.4 | 57.3 | 43.5 | 61.6 | 63.0

Our proposed approach uses a combination of distillation and collection of
predictions to reduce noise and increase robustness. Though, in theory, both
contribute towards better prediction, the question remains with their overall
impact. In Table 5, we disentangle each of the processes individually. In the
experiment, Only Distil uses only distillation without the collection step
(5). In this case, the prediction of the final stream is used to compute the
labels and during inferencing, both. Only Collect uses multiple stream without
the distillation loss (4). Though distillation improves the result compared to
the baseline, the experiment showcases that the collection of predictions over
multiple streams contributes more towards the overall approach.

####  5.4.5 Impact of $M$ candidates:

Beam-search decoding enables our approach to generate multiple transcript
candidates. Table 6 showcases the impact of various candidates on the overall
performance. Using a value of $M=5$ increases the result slightly compared to
just greedy decoding ($M=1$) for a transcript. Note, the computation cost of
the sequence matching module increases with an increase in value of $M$. Thus,
we set $M=5$ for all our experiments, which enables us to rectify any greedy
decoding errors and retain tractable computation costs.

Table 6: Performance impact due to $M$ candidates. The experiment was performed on all the splits of the Breakfast dataset. |  F1 = {$0.1$,$0.25$,$0.5$} | Edit | MoF
---|---|---|---
$M=1$ | $61.2$ | $54.9$ | $40.8$ | $58.9$ | $59.9$
$M=5$ | 61.5 | 55.2 | 41.7 | 59.9 | 60.1

####  5.4.6 Performance tradeoff: Using Final Stream vs Overall Model:

Table 7: Performance tradeoff when using final stream vs overall model. Here, Baseline is the model trained only on the annotated data. The experiment was performed on all the splits of the Breakfast dataset. Model Size |  F1 = {$0.1$,$0.25$,$0.5$} | Edit | MoF
---|---|---|---
Baseline | $52.1$ | $46.4$ | $34.4$ | $50.4$ | $54.2$
Last Stream | $59.5$ | $53.1$ | $39.7$ | $58.4$ | $58.2$
Overall | 61.5 | 55.2 | 41.7 | 59.9 | 60.1

As stated earlier, we collect the predictions of all the streams to compute
the final prediction. This might be a bottleneck during inference with limited
memory. A simple tradeoff uses only the final stream instead of the whole
trained model and ignores the collection step. Table 7 showcases the
performance tradeoff in such cases. Despite the drop in performance when using
only the final stream, it is still far better than the Baseline model, where
additional training data was ignored.

##  6 Ablation-Studies: Impact of $\alpha$

In Eq. (2) and Eq. (7), we use a weight factor $\alpha$ to control the errors
in the estimated transcripts from impacting the overall model. If the
estimated transcripts were accurate and $\alpha=1$, this proposed semi-
supervised approach would be equivalent to full-supervision. Table 8 showcases
the impact of the $\alpha$ on the overall performance. Increasing the value of
$\alpha$ impacts the performance negatively as the model is forced to learn
the errors with higher confidence. On the other hand, setting $\alpha$ to a
lower value enables the approach to balance the impact of the transcripts
error towards the overall frame label prediction. Thus, we set our value to
$\alpha=0.3$ for all our experiments.

Table 8: Performance impact of $\alpha$ on the overall performance. The table showcases the result over all the splits of the Hollywoodextended dataset. $\alpha$ |  F1 = {$0.1$,$0.25$,$0.5$} | Edit | MoF-BG | IoD
---|---|---|---|---
Baseline | $29.8$ | $25.2$ | $16.1$ | ${32.0}$ | $35.4$ | -
$0.1$ | $32.2$ | $28.4$ | $17.9$ | 35.7 | $37.8$ | 52.1
$0.3$ | 32.9 | 28.7 | 18.8 | ${35.1}$ | 38.4 | $51.4$
$1.0$ | $28.5$ | $24.8$ | $15.1$ | $33.9$ | $35.1$ | $49.7$
Table 9: Comparison of the proposed approach with the state-of-the-art on the Breakfast dataset. $*$ indicates the authors have used more $\%$ of annotation in comparison. | Method |  F1 = {$0.1$,$0.25$,$0.5$} | Edit | MoF
---|---|---|---|---
Full |  MSTCN[9] | $52.6$ | $48.1$ | $37.9$ | $61.7$ | $66.3$
MSTCN++[24] | $64.1$ | $58.6$ | $45.9$ | $65.6$ | $67.6$
BCN[38] | $68.7$ | $65.5$ | $55.0$ | $66.2$ | $70.4$
ASRF[14] | $74.3$ | $68.9$ | $56.1$ | $72.4$ | $67.6$
Timestamps | MSTCN+Timestamps[25] | $70.5$ | $63.6$ | $47.4$ | $69.9$ | $64.1$
Semi |  HMM-RNN∗[18] | - | - | - | - | $61.3$
Ours | 63.9 | 57.6 | 43.7 | 61.8 | 62.9
Transcripts |  CDFL[22] | - | - | - | - | $50.2$
Mucon[35] | - | - | - | - | $47.1$
D3TW[4] | - | - | - | - | $45.7$
NN-Viterbi[31] | - | - | - | - | $43.0$
Sets |  SCT[10] | - | - | - | - | $30.4$
SCV[23] | - | - | - | - | $30.2$
Action-Sets[30] | - | - | - | - | $23.3$
Table 10: Comparison of the proposed approach with the state-of-the-art on the Hollywood dataset. | Method | MoF-BG | IoD
---|---|---|---
Transcripts |  CDFL[22] | $40.6$ | $52.9$
Mucon[35] | $41.6$ | $52.3$
D3TW[4] | $33.6$ | $50.9$
NN-Viterbi[31] | - | $48.7$
Semi |  HMM-RNN[18] | - | $13.7$
Ours | 38.4 | 51.4

###  6.1 Comparison to State-of-the-art

In this section, we compare our approach with semi-supervised annotation
against the recent state-of-the-art approaches. To the best of our knowledge,
semi supervision has not been studied extensively. Results for the Breakfast,
and Hollywood extended datasets are shown in Table 9 and Table 10,
respectively. Our approach is comparable to timestamp supervision and full-
supervision despite using roughly ${1/3}^{rd}$ of training data on Breakfast
dataset. This demonstrates the capability of our approach to scaling quickly
when additional ‘in-domain’ data is available. We also compare our approach
with the weakly supervised setup where the action order in the form of
transcripts is only provided. Due to the availability of detailed frame
annotations for a few videos, the model performs significantly better than any
transcripts based approach on Breakfast dataset. On Hollywood extended,
however, our approach is on par with few state-of-the-art transcript based
approaches.

##  7 Conclusion

In this paper, we proposed an approach that leverages additional knowledge
from ‘in-domain’ unannotated videos to train a temporal action segmentation
model when a limited annotation is available. Our approach is based on multi-
stream distillation that repeatedly refines and finally combines their frame
predictions. We further predicted the action order that is used as an order
constraint while computing the frame labels of unannotated videos. Our
experiments on two different action datasets showcase comparable performance
to full-supervised approaches despite limited annotation. Additionally, we
also demonstrate the impact of the proposed distillation and accumulation on
full supervision, pointing towards its potential across various levels of
supervision.

## References

  * [1] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio.  Neural machine translation by jointly learning to align and translate.  In ICLR, 2014.
  * [2] Piotr Bojanowski, Rémi Lajugie, Francis Bach, Ivan Laptev, Jean Ponce, Cordelia Schmid, and Josef Sivic.  Weakly supervised action labeling in videos under ordering constraints.  In ECCV, pages 628–643, 2014.
  * [3] Joao Carreira and Andrew Zisserman.  Quo vadis, action recognition? a new model and the kinetics dataset.  In CVPR, pages 6299–6308, 2017.
  * [4] Chien-Yi Chang, De-An Huang, Yanan Sui, Li Fei-Fei, and Juan Carlos Niebles.  D3tw: Discriminative differentiable dynamic time warping for weakly supervised action alignment and segmentation.  In CVPR, pages 3546–3555, 2019.
  * [5] Min-Hung Chen, Baopu Li, Yingze Bao, and Ghassan AlRegib.  Action segmentation with mixed temporal domain adaptation.  In WACV, 2020.
  * [6] Min-Hung Chen, Baopu Li, Yingze Bao, Ghassan AlRegib, and Zsolt Kira.  Action segmentation with joint self-supervised temporal domain adaptation.  In CVPR, 2020.
  * [7] Li Ding and Chenliang Xu.  Weakly-supervised action segmentation with iterative soft boundary assignment.  In CVPR, pages 6508–6516, 2018.
  * [8] Olivier Duchenne, Ivan Laptev, Josef Sivic, Francis Bach, and Jean Ponce.  Automatic annotation of human actions in video.  In ICCV, pages 1491–1498, 2009.
  * [9] Yazan Abu Farha and Juergen Gall.  MS-TCN: multi-stage temporal convolutional network for action segmentation.  In CVPR, pages 3575–3584, 2019.
  * [10] Mohsen Fayyaz and Jurgen Gall.  Sct: Set constrained temporal transformer for set supervised action segmentation.  In CVPR, pages 501–510, 2020.
  * [11] Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He.  Slowfast networks for video recognition.  In ICCV, pages 6202–6211, 2019.
  * [12] De-An Huang, Li Fei-Fei, and Juan Carlos Niebles.  Connectionist temporal modeling for weakly supervised action labeling.  In ECCV, pages 137–153, 2016.
  * [13] Yifei Huang, Yusuke Sugano, and Yoichi Sato.  Improving action segmentation via graph-based temporal reasoning.  In CVPR, pages 14024–14034, 2020.
  * [14] Yuchi Ishikawa, Seito Kasai, Yoshimitsu Aoki, and Hirokatsu Kataoka.  Alleviating over-segmentation errors by detecting action boundaries.  In WACV, pages 2322–2331, 2021.
  * [15] Hilde Kuehne, Ali Arslan, and Thomas Serre.  The language of actions: Recovering the syntax and semantics of goal-directed human activities.  In CVPR, pages 780–787, 2014.
  * [16] Hilde Kuehne, Juergen Gall, and Thomas Serre.  An end-to-end generative framework for video segmentation and recognition.  In WACV, pages 1–8, 2016.
  * [17] Hilde Kuehne, Alexander Richard, and Juergen Gall.  Weakly supervised learning of actions from transcripts.  CVIU, 163:78–89, 2017.
  * [18] Hilde Kuehne, Alexander Richard, and Juergen Gall.  A hybrid rnn-hmm approach for weakly supervised temporal action segmentation.  IEEE TPAMI, 42(4):765–779, 2018.
  * [19] Colin Lea, Michael D Flynn, Rene Vidal, Austin Reiter, and Gregory D Hager.  Temporal convolutional networks for action segmentation and detection.  In CVPR, pages 156–165, 2017.
  * [20] Colin Lea, Austin Reiter, René Vidal, and Gregory D Hager.  Segmental spatiotemporal cnns for fine-grained action segmentation.  In ECCV, pages 36–52, 2016.
  * [21] Peng Lei and Sinisa Todorovic.  Temporal deformable residual networks for action segmentation in videos.  In CVPR, pages 6742–6751, 2018.
  * [22] Jun Li, Peng Lei, and Sinisa Todorovic.  Weakly supervised energy-based learning for action segmentation.  In ICCV, pages 6243–6251, 2019.
  * [23] Jun Li and Sinisa Todorovic.  Set-constrained viterbi for set-supervised action segmentation.  In CVPR, pages 10820–10829, 2020.
  * [24] Shi-Jie Li, Yazan AbuFarha, Yun Liu, Ming-Ming Cheng, and Juergen Gall.  MS-TCN++: Multi-stage temporal convolutional network for action segmentation.  2020\.
  * [25] Zhe Li, Yazan Abu Farha, and Jurgen Gall.  Temporal action segmentation from timestamp supervision.  In CVPR, pages 8365–8374, 2021.
  * [26] Fan Ma, Linchao Zhu, Yi Yang, Shengxin Zha, Gourab Kundu, Matt Feiszli, and Zheng Shou.  Sf-net: Single-frame supervision for temporal action localization.  In ECCV, pages 420–437, 2020.
  * [27] Davide Moltisanti, Sanja Fidler, and Dima Damen.  Action recognition from single timestamp supervision in untrimmed videos.  In CVPR, pages 9915–9924, 2019.
  * [28] Hamed Pirsiavash and Deva Ramanan.  Parsing videos of actions with segmental grammars.  In CVPR, pages 612–619, 2014.
  * [29] Alexander Richard, Hilde Kuehne, and Juergen Gall.  Weakly supervised action learning with rnn based fine-to-coarse modeling.  In CVPR, pages 754–763, 2017.
  * [30] Alexander Richard, Hilde Kuehne, and Juergen Gall.  Action sets: Weakly supervised action segmentation without ordering constraints.  In CVPR, pages 5987–5996, 2018.
  * [31] Alexander Richard, Hilde Kuehne, Ahsan Iqbal, and Juergen Gall.  Neuralnetwork-viterbi: A framework for weakly supervised video learning.  In CVPR, pages 7386–7395, 2018.
  * [32] Hiroaki Sakoe and Seibi Chiba.  A dynamic programming approach to continuous speech recognition.  In ICASSP, page 65–69, 1971.
  * [33] Karen Simonyan and Andrew Zisserman.  Two-stream convolutional networks for action recognition in videos.  NeurIPS, page 568–576, 2014.
  * [34] Yaser Souri, Yazan Abu Farha, Fabien Despinoy, Gianpiero Francesca, and Juergen Gall.  Fifa: Fast inference approximation for action segmentation.  arXiv preprint arXiv:2108.03894, 2021.
  * [35] Yaser Souri, Mohsen Fayyaz, Luca Minciullo, Gianpiero Francesca, and Juergen Gall.  Fast weakly supervised action segmentation using mutual consistency.  TPAMI, 2021.
  * [36] Peng Tang, Xinggang Wang, Xiang Bai, and Wenyu Liu.  Multiple instance detection network with online instance classifier refinement.  In CVPR, pages 2843–2851, 2017.
  * [37] Nam N Vo and Aaron F Bobick.  From stochastic grammar to bayes network: Probabilistic parsing of complex activity.  In CVPR, pages 2641–2648, 2014.
  * [38] Zhenzhi Wang, Ziteng Gao, Limin Wang, Zhifeng Li, and Gangshan Wu.  Boundary-aware cascade networks for temporal action segmentation.  In ECCV, pages 34–51, 2020.