# Unsupervised Action Segmentation with Self-supervised
Feature Learning and Co-occurrence Parsing

Zhe Wang1 Hao Chen2 Xinyu Li2 Chunhui Liu2 Yuanjun Xiong2 Joseph Tighe2
Charless Fowlkes1

1Dept. of CS, UC Irvine 2Amazon Web Services This work was done during
internship at Amazon

###### Abstract

Temporal action segmentation is a task to classify each frame in the video
with an action label. However, it is quite expensive to annotate every frame
in a large corpus of videos to construct a comprehensive supervised training
dataset. Thus in this work we explore a self-supervised method that operates
on a corpus of unlabeled videos and predicts a likely set of temporal segments
across the videos. To do this we leverage self-supervised video classification
approaches to perform unsupervised feature extraction. On top of these
features we develop CAP, a novel co-occurrence action parsing algorithm that
can not only capture the correlation among sub-actions underlying the
structure of activities, but also estimate the temporal trajectory of the sub-
actions in an accurate and general way. We evaluate on both classic datasets
(Breakfast, 50Salads) and emerging fine-grained action datasets (FineGym) with
more complex activity structures and similar sub-actions. Results show that
our method achieves state-of-the-art performance on all three datasets with up
to 22% improvement, and can even outperform some weakly-supervised approaches,
demonstrating its effectiveness and generalizability.

##  1 Introduction

Temporal action segmentation aims to classify each frame in an untrimmed video
with an action label. The task is a key step in understanding the structure of
complex activities in videos. A recent study [44] also shows that the fine-
grained sub-action segmentation can help many of other tasks such as action
recognition and detection. Today’s high performing solutions for temporal
action segmentation require full frame-level supervision [16, 25, 52].
However, collecting per-frame annotations in untrimmed videos is extremely
expensive and often impractical. Some recent work aim to overcome this by
leveraging weak supervision to avoid the need of the full frame-level
annotations [17, 33, 41].

These solutions, even though they do not use full supervision, still need some
annotations that are costly to acquire, especially when expert knowledge on
fine-grained atoms of activities (i.e., sub-actions) is required. Recently,
temporal action segmentation in an unsupervised manner has received more
attention [1, 3, 42]. These solutions are clustering based and operate on
extracted frame features. However, unsupervised temporal action segmentation
is a very challenging problem. First, unlike in fully-supervised or weakly-
supervised settings where at least the number or the order of the sub-actions
are given, in unsupervised setting even the temporal order of sub-actions is
unknown and needs to be estimated. Second, to parse long video sequences into
sub-actions in a semantically meaningful way without any supervision requires
a solution which not only extracts good visual representations for each
individual frame, but also captures the temporal relations among frames and
sub-actions. Today there are still no effective solutions to precisely
estimate the temporal path of sub-actions, well capture the temporal
characteristics in frame features, and model temporal correlations among sub-
actions underlying the native structure of activities. As a result, the
performance of most today’s unsupervised action segmentation solutions [1, 3,
42] is still far from desired.

In this work, we address above issues by firstly introducing self-supervised
learning methods to extract frame-level feature representations that are
usually more discriminative. While self-supervised learning has been widely
exploited on other tasks such as image classification and action recognition
[7, 15, 36], the application of it on action segmentation is unexploited. We
explore a few of today’s most popular video based self-supervised learning
approaches and show superior performance over the current state-of-the-art
features in action segmentation. We further show that those self-supervised
approaches that learn temporal characteristics outperform other self-
supervised methods that focus on spatial transformation, which proves our
assumption that modeling temporal relations is a key for accurately segmenting
the frames into sub-actions.

We then propose a Co-occurrence Action Parsing (CAP) algorithm to cluster the
extracted features and decode the frames into sub-actions by considering both
the temporal order of occurrence and the co-occurrence relations of sub-
actions. These priors, calculated over the whole dataset, can statistically
depict the natural structure of activities and will be injected into each
video during inference. Moreover, CAP is able to handle complex structures of
activities and videos by taking multi-occurrence of sub-actions into account.
Results show that with CAP, our solution not only achieves state-of-the-art
(SOTA) for unsupervised action segmentation, and even outperforms some of the
weakly-supervised approaches, but also is more generalizable to different
scenarios.

Besides evaluating our model on traditional action segmentation benchmarks
such as Breakfast [27], Salad [45], we additionally test its performance on
more challenging and recently emerging dataset FineGym [43]. Actions in
traditional benchmarks are well structured - they are goal-directed (actors
have clear goals like “making salad” or “preparing scrambled egg”) and can be
decomposed into clear steps, with one step being as the prerequisite for
another one (e.g., you have to “get” an egg before “scramble” it). However,
actions in FineGym are with more complex structures - they are fast moving
gymnastics related actions, where several actors may perform the same action
multiple times in a single video. There are more similar fine-grained sub-
actions across different activities as well (for example: both ‘floor
exercise’ and ‘vault’ event involves a lot of basic sub-action like ‘running’
and ‘jumping’). Results show that our approach can achieve significant
performance improvements on this challenging dataset, demonstrating the
generalizability of our solution. To summarize, our main contributions are:

  * •

To the best of our knowledge, we are the first one to introduce the self-
supervised feature representation learning in solving the unsupervised action
segmentation problem. We study today’s most popular video self-supervised
learning approaches and show that self-supervised learning can bring notable
improvement for unsupervised action segmentation;

  * •

We propose a co-occurrence action parsing (CAP) algorithm, which takes into
account the correlations and temporal locations of sub-actions calculated
based on global statistics of all the videos, and is able to estimate the
temporal path of sub-actions in a more precise and general way, and therefore
outperforms today’s solutions;

  * •

Our solution not only achieves SOTA performance on unsupervised action
segmentation task (with an improvement up to 22%) but also outperforms some
weakly-supervised methods. We also study the action segmentation task on
emerging and more challenging dataset that contains more complex activities
structures and more similar fine-grained sub-actions. Results show that our
solution can even achieve more improvement (up to 126.5%) on it, demonstrating
the effectiveness and generalizability of our solution.

##  2 Related Work

Temporal action segmentation is a task to classify each frame in an untrimmed
video with a label of action. It is a key for understanding structures of
complex activities. While current techniques have achieved great success on
other action understanding tasks such as action recognition [8, 18, 19, 46,
55, 49, 57, 34, 50, 11] and action detection [35, 56, 54, 23], temporal action
segmentation is still far from being well-explored. Today’s high performing
solutions for temporal action segmentation require full frame-level
supervision [16, 25, 28, 52, 31, 30]. However, acquiring annotations on fine-
grained sub-actions for each frame is expensive. To reduce the amount of
labeled data needed, [10] proposes a domain adaptation approach based on self-
supervised auxiliary tasks and achieves good performance. More recent works
focus on developing weakly-supervised solutions: instead of having supervision
for every frame, only the list (with order information) [5, 32, 41, 39, 13,
24, 9] or the set (without order information) [17, 33, 40] of the sub-actions
are required during training. Other work uses script or instructions for weak
annotation [2].

Unsupervised temporal action segmentation starts to receive growing interests
these days, to further reduce the dependence on labeled data. [14] builds a
joint sequential subset selection method to produces a summary of the
procedure key-steps in instructional videos. [42] learns a function assigning
sub-activity scores to frame features and models the distribution over sub-
activity permutations by a Generalized Mallows Model. [1] proposes a self-
supervised predictive learning with recurrent neural networks (RNN) by
utilizing features of adjacent frames to find the action boundaries. [3]
proposes a combination of temporal embedding generated using a frame timestamp
prediction network, feature clustering and Viterbi decoding for consistent
frame-to-cluster assignment. However, these solutions do not take into account
the semantic correlations among sub-actions and the re-occurrence of sub-
actions in some complex activity structures.

Self-supervised learning of visual representation is a hot topic today, and
has shown success in some of action understanding tasks [6, 7, 15, 20]. Many
different video properties have been used as supervision to build good
representations, such as: cycle consistency between video frames [51];
distinguishing between a video frame sequence and a shuffled version of it
[36]; predicting the order of the shuffled clips [53]; solving space-time
cubic puzzles [26]; classifying the speed of the video [15, 4]; recognize the
rotation of the video [21]. Another common task is predicting the future,
either by predicting pixels of future frames [12], or an embedding of a future
video segment [22]. While today’s action segmentation solutions [16, 3, 52]
still mainly rely on hand-crafted features such as IDT [48], Fisher Vector
(FV) [37], or pre-trained features such as I3D on Kinetics dataset [8], we
believe that introducing self-supervised methods in feature representation
learning for unsupervised action segmentation can significantly boost the
performance, as it can not only bring strong discriminative power to the
features, but also bring a chance to improve the temporal characteristics of
the features.

Figure 1: Overview of our approach.

##  3 Technical Approach

Temporal action segmentation is a task to classify each frame in the video to
a sub-action. Specifically, given $M$ videos, each of them is represented by
$X_{m}$ which contains $N_{m}$ frames. The task is to predict the action label
$l_{mn}\in{1,...,K}$ for each video frame $x_{mn}$. In this paper, we study
the task in an unsupervised setting, i.e., the label $l_{mn}$ for each frame
is unknown during model training.

Fig. 1 is an overview of our approach. Briefly: given a set of videos, our
feature generation module extracts frame features with a self-supervised
representation learning method. Extracted frame features are then sent to the
co-occurrence action parsing (CAP) module, which clusters the frames and
decodes them into segments of unlabeled sub-actions. Finally, the semantic
action label is assigned to each cluster by using Hungarian algorithm [29]. We
introduce each module in details in the following sections.

###  3.1 Feature Generation with Self-supervision

Our co-occurrence action parsing (CAP) module relies on the feature
representation to have some semantic meaning so that the clustering step in
our CAP module can effectively separate different semantic concepts in the
video. In this work we explore a number of self-supervised feature leaning
methods that have been developed for image and video classification tasks. We
apply these methods for the task of video segmentation and study their
relative effectiveness for that task v.s. the classification task they were
designed for. Since one essential factor of the feature design for the
temporal action segmentation is to capture the temporal characteristics, we
want to further understand whether self-supervised learning methods built
based on temporal augmentations (e.g, adjusting the frame rate, shuffling the
frame or clip order, etc.) are also effective on temporal segmentation or not.

With these questions in mind, we explore three of today’s most popular video
self-supervised learning methods and compare them with the feature designs
used today in unsupervised action segmentation, including: IDT [48], FV [37],
pre-trained I3D features [8], and the continuous temporal features that
achieves SOTA performance today [3]. Specifically, we study the following
self-supervised methods: (1) SpeedNet [15], which augments the frame rate of
the video and predict it; (2) ShuffleLearn [36], which predict whether the
order of frames in the sequence is shuffled or not; (3) RotationNet [21],
which rotates the whole video sequence by several pre-defined degrees and
predict which degree the video is rotated. Note that RotationNet is a method
based on spatial augmentation, while Shuffle and Learn and SpeedNet are with
temporal augmentation that are expected to have stronger temporal reasoning
capability. Briefly we summarize the designs of these three methods in our
work:

SpeedNet [15]. Given a clip $C$ with fixed number of $N$ frames: $C$ =
($f_{1},f_{2},...,f_{N}$), we pre-define four frame rate settings
$[2,4,8,15]$. Each time when we generate a clip we randomly select one out of
these four settings. Note that we keep the number of frames in each clip as
the same, hence the time-range of the clip varies according to the frame rate
selected. We train the network to predict which frame-rate the clip is sampled
from (i.e., a 4-way classification).

ShuffleLearn [36]. Given a clip $C$ with fixed number of $N$ frames: $C$ =
($f_{1},f_{2},...,f_{N}$), we random shuffle $M$ frames ($0<M<N$) in a random
order. Each time when we generate the clip, we flip the coin to decide whether
to shuffle it or not. We train the network to predict whether a clip is
shuffled or not.

RotationNet [21]. Given a clip $C$ with fixed number of $N$ frames: $C$ =
($f_{1},f_{2},...,f_{N}$), we pre-define four rotation degree settings:
$[0,90,180,270]$. We also add some randomness when we rotate, i.e., in
$[-30,30]$ degree. Each time when we generate clip we randomly select one out
of these four settings and rotate the whole video sequence with the selected
degree. We train the network to predict which of the settings is the rotation
on (i.e., a 4-way classification).

Importance of feature. Finding an effective good feature for temporal action
segmentation task is a non-trival task as we need insight and efforts to
select good features and introduce them into action segmentation, and to
demonstrate its superiorty over traditional features (Similar to applying GCN,
LSTM, Transformers into specific vision tasks, which also requires some domain
knowledge.)

###  3.2 Co-occurrence Action Parsing

Our co-occurrence action parsing (CAP) algorithm consists of two steps:
clustering and decoding. Extracted frame-level features from Section 3.1 are
first clustered. Since we don’t have ground truth annotations on sub-actions,
here we aim at generating a number of clusters, with each of them representing
a unique but unlabeled sub-action. After clustering, we then generate three
outputs: (1) a video score matrix that represents the predicting score of each
frame to each cluster for the whole video; (2) a co-occurrence matrix that
captures the chance of any pair of clusters co-existing in videos; and (3) a
cluster temporal location histogram that captures the distribution of temporal
locations of each cluster in videos. In decoding, we first apply the co-
occurrence matrix onto the video score matrix to generate a refined video
score matrix. Then we estimate a temporal path with the cluster temporal
location histogram and decode the frames to final segments based on the
temporal path. The overall algorithm can be seen in Fig. 1. Below we introduce
the clustering and decoding in details.

####  3.2.1 Clustering

Given an activity, and assuming there are $K$ different sub-actions,we extract
all the frame-level features for all videos labeled with this activity, and
cluster them into $K$ clusters using k-means. To get the probability of the
frame $x_{mn}$ that belongs to the cluster $k$, we estimate a Gaussian
distribution for each cluster $k\in K$ :

$p(x_{mn}|k)=\mathcal{N}(x_{mn};\mu_{k},\Sigma_{k})$ (1)

We generate three outputs from the clustering step: (1) video score matrix;
(2) co-occurrence matrix and (3) cluster temporal location histogram.

Video score matrix. Given a video $X_{m}$ that contains $N_{m}$ frames, we
create a video score matrix $S_{m}\in\mathbb{R}^{K\times N_{m}}$ to hold the
probabilities calculated from Eq. (1), where $K$ is the number of clusters.
Each element $S_{m}[k,n]$, with $k\in\\{1,2,..K\\}$ and
$n\in\\{1,2,...N_{m}\\}$ represents the probability of the $n$-th frame
assigned to the cluster $k$.

Co-occurrence matrix. The purpose of creating the co-occurrence matrix is to
capture the correlations among sub-actions underlying the native structure of
activities from the global video sets. Given an activity, there are usually
strong correlations among its sub-actions across all videos. For example, in
the activity “Fried Egg”, “fry egg” always occurs with “take egg”and “crack
egg”. While previous solutions do not build models to capture such global
patterns, we leverage a co-occurrence matrix to capture the co-existing
relations among sub-actions. Note that, since we are dealing with an
unsupervised setting, we don’t have ground truth annotations on sub-actions,
hence in the real implementation, what we calculate is the co-occurrence
between two clusters $i$ and $j\in\\{1,2,..,K\\}$, instead of the real sub-
actions. The way to generate the co-occurrence matrix is simple: for each pair
of cluster $i$ and $j$, we iterate on all the videos and count the number of
times that cluster $i$ and $j$ appear together in each video as $C(i,j)$. We
also count the number of videos that contains the cluster $i$ as $C(i)$. To
avoid the noise introduced by the clustering algorithm, we set a threshold
$\tau_{1}$ in determining the appearance of a cluster, namely, while
calculating the $C(i)$ and $C(i,j)$, only when the ratio of frames of class
$i$ is larger than $\tau_{1}$, we count that class $i$ appears in the video.
We then calculate a conditional occurrence probability as
$P(j|i)=C(i,j)/C(i)$, which represents the probability of cluster $j$ appears
in a video given that cluster $i$ appears. We calculate $P(i|j)$ for all pairs
of clusters $(i,j)$ in this way and generate the $K\times K$ dimension co-
occurrence matrix. Details are shown in Algorithm 1. Since the co-occurrence
matrix is learned from all videos, it is capturing the global patterns of the
activity structures. The co-occurrence matrix is used for refining the video
score matrix, which is introduced in Section 3.2.2.

Cluster temporal location histogram. The purpose of generating the cluster
temporal location histogram is to estimate where each cluster generally
locates in temporal dimension. Given the frames associated to each cluster
from k-means, we compute the histogram of the temporal location for each
cluster based on the timestamp of each frame in the video. Specifically, for a
frame $x_{mn}$, i.e., the $n$-th frame in the video $X_{m}$, we calculate its
relevant timestamp as $t_{x_{mn}}=n/N_{m}$, where $N_{m}$ is the total number
of frames in video $X_{m}$. Then for each cluster $k$, we calculate the
histogram on $t_{x_{mn}}$ for all the frames $x_{mn}$ from the videos that are
clustered to $k$. The histogram indicates the potential temporal location of
the cluster $k$ in the videos, and will be used for temporal path decoding
introduced in Section 3.2.2. Note that in real cases a sub-action can appear
multiple times in different parts of videos, so each cluster might have more
than one significant bin in the cluster temporal location histogram.

####  3.2.2 Decoding

After clustering all the frames into $K$ clusters, the next step is to decode
them into segments of the sub-actions. There are two main steps, namely, (1)
refine the video score matrix with the co-occurrence matrix; (2) estimate the
temporal path of the clusters and decode frames to segments.

Refining the video score matrix. The first step in the decoding is to generate
the refined video score matrix $R_{m}\in\mathbb{R}^{K\times N_{m}}$ for each
video $X_{m}$. There are two main issues with directly using the original
video score matrix $S_{m}$ in decoding: (1) the video score matrix is
generated purely based on the frame feature representations, it does not
capture any global patterns of the activity structures across all videos, and
does not contain any correlation information among sub-actions; (2) the
clustering is conducted on $K$ clusters, which is the total number of
different sub-actions in the given activity. In many of cases, only a subset
of $K$ will appear in each individual video due to the diversity of the
activities being performed by different persons. Directly using video score
matrix may lead to the issue of over-segmenting video to the non-existing
classes in current video. To address these issues, we apply the co-occurrence
matrix to refine the score matrix $S_{m}$. Specifically, we construct a
salience pool to record the most “salient” clusters in a video. Given a video
$X_{m}$ and for each cluster $k$, we first calculate the number of frames
belonging to $k$ in the video, i.e., $c(k)$, and define the ratio of frames
for cluster $k$ as $r(k)=c(k)/N_{m}$, where $N_{m}$ is the total number of
frames in the video. We add the cluster $k_{0}$ with the largest $r(k)$ into
the salience pool first. Then for the rest of clusters $k$, we weigh the ratio
of frames as:

$\displaystyle r^{\prime}(k)=P(k_{0}|k)\cdot r(k),$ (2)
$\displaystyle\text{where }k\in 1,2,...,K,\text{and }k\neq k_{0},$

$P(k_{0}|k)$ is the conditional co-occurrence probability from the co-
occurrence matrix calculated before. We set a threshold $\tau_{2}$ and select
the cluster $k_{1}$ with the largest $r^{\prime}(k)$. We add $k_{1}$ into the
salience pool only if $r^{\prime}(k_{1})>\tau_{2}$. We keep iterating on this
procedure by conditioning on all clusters that has been added in the salience
pool, until none of the ratio of frames from the rest of the clusters is above
$\tau_{2}$.

Then we use the salience pool to calculate the refine video score matrix
$R_{m}$. Since the salience pool is generated based on co-occurrence matrix,
it brings the global information on the occurrence of clusters from all the
videos. Clusters that are not in the salience pool should have lower chances
to appear in the given video, hence we want to attenuate the scores of them.
Assuming there are $K_{s}\subset K$ sub-actions in the salience pool, we
introduce a decay factor $\eta$ to decay the probability scores of all the
cluster $k\notin K_{s}$ for each frame in the video score matrix $S_{m}$, to
generate the refined video score matrix $R_{m}$. Details are in Algorithm 1.
The probabilistic update is inspired by Kalman filter where the new
probability is updated based on the observation $P(k_{0}|k)$ and old
probability $r(k)$.

Data: Video score matrix $S_{m}$ for video $X_{m}$, with
$S_{m}[k,n]_{k=1,n=1}^{K,N_{m}}$, number of clusters: $K$, thresholds:
$\tau_{1}$,$\tau_{2}$, decay factor: $\eta$.

Result: Co-occurrence matrix: $P(i,j)_{i,j=1}^{K}$, clusters in salience pool
$\mathcal{C}=\\{k_{i}\\}_{i=1}^{K_{s}}$, where $K_{s}\ \subset K$, Refined
video score matrix $R_{m}$, with scores $R_{m}[k,n]$.

\- Compute the ratio of frames for each cluster in each video, if it is larger
than $\tau_{1}$, we treat the specific cluster occur in the video.

\- Iterate all the videos, count the times each cluster appears
$C(i)_{i=1}^{K}$, and the times different clusters co-occur together
$C(i,j)_{i,j=1}^{K}$. Normalize to make it as conditional probability
$P(j|i)=C(i,j)/C(i)$

\- Initialization: $\mathcal{C}\leftarrow k_{0}$. ($k_{0}$ is the cluster with
the largest ratio of frames $r(k_{0})$ in current video.)

while  _$len(\mathcal{C})\leq K$_ do

1\. For each remaining cluster $j$, update the ratio of frames with the
cluster $i$ just added into the salience pool: $r(j)=P(i|j)\cdot r(j),$

2\. Choose the class: $k^{*}\leftarrow\mathrm{arg}\max_{j}r(j)$.

3\. Update: $\mathcal{C}\leftarrow\mathcal{C}\cup\\{k^{*}\\}$, if
$r(k^{*})>\tau_{2}$, else break.

end while

for  _$n\leq N_{m}$_ do

Decay probabilities in video score matrix $S_{m}$, $R_{m}[k,n]=\eta
S_{m}[k,n]$, $\forall k\notin\mathcal{C}$.

end for

$-$ Return refined video score matrix: $R_{m}$.

Algorithm 1 Generating the Co-occurrence Matrix and the Refined Video Score
Matrix.

Estimating the temporal path and decoding. Having the refined video score
matrix, the final step is to decode the frame sequences into the sub-action
segments. Assume that we have a known temporal path of the clusters
$k_{1}\rightarrow...\rightarrow k_{T}$, then the decoding is to maximize the
probability of the sequence following this path based on our refined video
score matrix, i.e.,

$\hat{l}_{l}^{N_{m}}={argmax}_{l_{1},...,l_{N_{m}}}\prod^{N_{m}}_{n=1}p(x_{mn}|l_{n})p(l_{n}|l_{n-1}),$
(3)

where, $p(x_{mn}|l_{n}=k)$ is the probability of the frame $x_{mn}$ belonging
to the cluster $k$, obtained from the refined video score matrix, and
$p(l_{n}|l_{n-1})$ are the transition probabilities of moving from the label
$l_{n-1}$ at frame $n-1$ to the next label $l_{n}$ at frame $n$. Given a
temporal path, Eq. (3) can be solved with a Viterbi algorithm [38].

The temporal path estimation is the key for the decoding. While in weakly-
supervised action segmentation the temporal path (i.e, the order of sub-
actions) is usually assumed to be known, in unsupervised setting it can only
be estimated. Previous work [3] proposed an estimation of the temporal path
that is limited to be strictly single directional and each sub-action can
appear at most once in the path. Such a path may suit for dataset with simple
activity structures, e.g., Breakfast [27], however, begins to fail when the
structure of activities gets more complex. For instance, a single sub-action
may occur multiple times, and the order of the sub-actions can be bi-
directional (i.e., sub-action $i$ can go to sub-action $j$ and then go back to
$i$).

We design a multi-occur temporal path that is suitable for more general cases,
using the cluster temporal location histogram introduced in Section 3.2.1. We
first set a global threshold $\theta$, and for each cluster, we only select
the bins from temporal location histogram with counts (normalized to the total
counts across all the bins) larger than $\theta$. Then we merge the selected
bins from all the clusters and sort them based on the temporal location
associated with each bin. The order of bins with their corresponding cluster
labels forms our temporal path. Our multi-occur temporal path has the
following advantages: (1) it is able to capture the multi-time occurrence for
each cluster as well as the bi-direction transition among different clusters,
because more than one bins from each cluster can be selected; (2) by adjusting
the global threshold, it can control the length of the generated temporal
path.

##  4 Experiments

##### Datasets.

We evaluate our approach on two most widely used datasets in action
segmentation: Breakfast [27] and 50Salads [45], and an emerging fine-grained
action dataset FineGym [43]. Breakfast [27] is a large-scale dataset that
consists of ten different complex activities of performing common kitchen
activities with approximately eight sub-actions per activity class. The
duration of the videos varies significantly, e.g. ‘coffee’ has an average
duration of 30 seconds while ‘cooking pancake’ takes roughly 5 minutes.
50Salads [45] dataset contains 4.5 hours of different people performing a
single complex activity, making mixed salad. Compared to the other datasets,
the videos are much longer with an average video length of 10k frames. We also
observe that a single sub-action can occur multiple times along the video in
this dataset, which seldom happens in Breakfast. FineGym [43] is an emerging
dataset provided with fine-grained sports action annotations. Though it is
originally proposed for fine-grained action recognition, it contains rich
hierarchical information including annotations of start and end for the fine-
grained sub-actions, which is suitable for action segmentation evaluation.
These sports videos contain several athletes performing the same action and
their replays. The structure of the videos are more complex and it includes
many re-occurrences of a single sub-action, and includes more similar fine-
grained sub-actions (up to 35), which makes the dataset more challenging for
action segmentation than typical ones used today. We use Gym99 setting in
original paper. The statistics of it is listed in supplementary material. We
select 20 as bin size and $\theta$ is set as 0.15. They are depending on
native attributes of actions in the dataset.

Breakfast | MoF | F1 score
---|---|---
Unsupervised setting
GMM [42] | 0.346 | -
LSTM + AL [1] | 0.429* | -
CTE [3] | 0.418 | 0.264
Ours | 0.511 | 0.392
Weakly-supervised setting
Action Sets [40] | 0.284 | -
NNviterbi [41] | 0.430 | -
SCT [17] | 0.304 | -
SetViterbi [33] | 0.408 | -
EnergySeg [32] | 0.630 | -
Fully-supervised setting
HTK [28] | 0.259 | -
GTRM [25] | 0.650 | -
MS-TCN [16] | 0.663 | -
BCN [52] | 0.704 | -
Table 1: Results on Breakfast.

##### Evaluation Metrics.

Following [3] we use the Hungarian algorithm [29] to get a one-to-one matching
that maximizes the similarity between all the clusters and sub-action classes
for all the videos in evaluation. We report frame-wise accuracy as the mean
over frames (MoF), indicating the percentage of frames correctly labelled. We
also report F1 score the same as [3] to measure the quality of the temporal
segmentation.

###  4.1 Comparison with SOTA

We compare our approach with SOTA solutions on Breakfast in Table 1, 50Salads
in Table 2 and FineGym in Table 3. For all the datasets, We use SpeedNet [15]
trained on the Kinetics-400 [8] as the feature for our CAP algorithm. Details
on feature comparison can be referred to the Section 4.2. We can see our
method achieves SOTA performance on both Breakfast and 50Salads datasets under
unsupervised setting, with around 22% and 14% improvement on MoF
correspondingly. Our method on Breakfast even outperforms most of the weakly-
supervised solutions.

50Salads | MoF | F1 score
---|---|---
Unsupervised setting
LSTM + AL [1] | 0.606* | -
CTE [3] | 0.355 | -
Ours | 0.414 | 0.303
Weakly-supervised setting
NNviterbi [41] | 0.494 | -
EnergySeg [32] | 0.547 | -
Fully-supervised setting
HTK [28] | 0.247 |
GTRM [25] | 0.826 | -
MS-TCN [16] | 0.734 | -
BCN [52] | 0.844 | -
Table 2: Results on 50Salads. Note that ‘*’ uses a per-video instance cluster to ground-truth mapping, which uses additional video-level information, while we follow other work to use cluster to ground-truth mapping in a global manner in evaluation and do not use any additional information [47].  FineGym | MoF | F1 score
---|---|---
Baseline [3] | 0.294 | 0.167
Ours | 0.666 | 0.297
Table 3: Results on FineGym.

Since FineGym is new and we are the first to conduct action segmentation on
it, there is no existing related results yet. We generate the baseline by
running the SOTA apporach [3] on FineGym. We can see that on this emerging and
more challenging dataset, our solution achieves even more improvement (127% on
MoF and 78% on F1) comparing to the SOTA solution, demonstrating the
effectiveness of our designed solution in handling videos with more complex
structures. Overall the consistent improvement across all the benchmarks shows
that our approach is able to be generalized to different cases.

###  4.2 Comparison of Feature Learning

We study three different self-supervised feature learning methods: (1)
SpeedNet [15], (2) RotationNet [21], and (3) ShuffleLearn [36], and compare
the results on Breakfast dataset in Table 4. All the numbers in the table are
generated without using the co-occurrence matrix and the multi-occur temporal
path, so that we can purely evaluate the effectiveness of features without
bringing any additional factors.

We conduct the experiments on self-supervised training (1) only on Kinetics
dataset (i.e., (d) - (f)), (2) only on Breakfast dataset (i.e., (g) - (i)) and
(3) first on Kinectics dataset and then on Breakfast dataset (i.e., (j) -
(l)), to further explore that whether we need to train on target dataset
(i.e., Breakfast in our case) to get a good feature representation for the
task. From the table we can find that:

(1) Based on experiment (a) - (l), we conclude that SpeedNet trained on
Kinectics shown in (d) performs the best and gives better performance than
SOTA as in (c);

(2) Based on experiment (d) - (l), RotationNet consistently performs worse
than SpeedNet and ShuffleLearn. This may due to the fact that RotationNet is
using self-supervised signal from spatial dimension, which may lack of ability
to model the temporal characteristics between frames. SpeedNet and
ShuffleLearn, however, use temporal augmentation as the self-supervised
signal, so that they may have stronger capability in modeling the temporal
relations among frames and thus are more suitable for temporal action
segmentation. The results also indicate that it is important to capture
temporal relations among frames in feature learning for the segmentation task.
In addition, SpeedNet, as one of the most emerging and successful video self-
supervised learning approaches, consistently outperforms the ShuffleLearn,
showing the effectiveness of it;

(3) Based on experiment (b), (d), (e), (f), self-supervised features always
perform better than classical I3D feature pre-trained on Kinetics. We argue
that while the I3D feature trained for action recognition task optimizes
feature representation on video clip level, self-supervised learning
representation implicitly learns better frame-level representation and can
better capture the temporal structure inside the clip;

(4) Based on the experiments (d) - (l), we can see larger dataset like
Kinetics can help build better self-supervised representation, while smaller
ones contain less variety. It’s not needed to use target dataset to get a good
feature representation for the temporal action segmentation task.

| Feature Description | MoF | F1
---|---|---|---
Baseline
(a) | IDT [3] | 0.316 | -
(b) | K400 I3D [3] | 0.251 | -
(c) | CTE [3] | 0.418 | 0.264
Self-supervised on K400
(d) | K400 SpeedNet | 0.508 | 0.391
(e) | K400 RotationNet | 0.328 | 0.317
(f) | K400 shuffleLearn | 0.339 | 0.328
Self-supervised on Breakfast
(g) | Breakfast SpeedNet | 0.344 | 0.327
(h) | Breakfast RotationNet | 0.307 | 0.319
(i) | Breakfast shuffleLearn | 0.315 | 0.309
Self-supervised first on K400, then on Breakfast
(j) | K400, Breakfast, SpeedNet | 0.501 | 0.337
(k) | K400, Breakfast, RotationNet | 0.279 | 0.290
(l) | K400, Breakfast, shuffleLearn | 0.292 | 0.318
Table 4: Comparison of features for temporal action segmentation. ‘K400’ denotes features trained on Kinetics [8] dataset while ‘Breakfast’ represents features trained on Breakfast [27] dataset. ‘SpeedNet’, ‘RotationNet’ and ‘ShuffleLearn’ denote features trained using the self-supervised methods from [15], [21] and [36].  Dataset | Feat | C-Matrix | M-T-Path | MoF | F1
---|---|---|---|---|---
Breakfast |  |  |  | 0.418 | 0.264
✓ |  |  | 0.508 | 0.391
✓ | ✓ |  | 0.511 | 0.392
✓ | ✓ | ✓ | 0.511 | 0.392
50Salads |  |  |  | 0.355 | -
✓ |  |  | 0.372 | 0.281
✓ | ✓ |  | 0.378 | 0.290
✓ | ✓ | ✓ | 0.414 | 0.303
FineGym |  |  |  | 0.294 | 0.167
✓ |  |  | 0.425 | 0.246
✓ | ✓ |  | 0.442 | 0.248
✓ | ✓ | ✓ | 0.666 | 0.297
Table 5: Ablation study on all datasets. ‘Feat’ denotes whether to use the
SpeedNet feature trained on K400; ‘C-Matrix’ denotes whether to use the co-
occurrence matrix to refine the video score matrix; and ‘M-T-Path’ denotes
whether to use our designed multi-occur temporal path in decoding. If none of
them is selected then it’s a baseline setting from [3].

###  4.3 Ablation Study on CAP

Next, we conduct ablation studies on our CAP algorithm. Two main units are
investigated: (1) Co-occurrence Matrix; (2) Multi-occur Temporal Path.

Effectiveness of Co-occurrence Matrix. We evaluate the influence of using the
co-occurrence matrix to refine the video score matrix in Table 5 on all
datasets. From the table, our co-occurrence matrix brings performance
improvement over using original video score on all of the three datasets and
on both MoF and F1. The improvement on FineGym dataset is more notable than
the other two, indicating the importance of using the co-occurrence matrix
while handling more complex scenarios.

(a) P24 friedegg sequence from Breakfast dataset. The semantics order are
‘background’, ‘crack egg’,‘fry egg’, ‘take plate’, ‘put egg2plate’,
‘background’.

(b) P38 juice sequence from Breakfast dataset. The semantics order are
‘background’, ‘cut orange’, ‘squeeze orange’,‘take glass’,‘pour
juice’,‘background’.

Figure 2: Two qualitative examples (a) and (b). The first row is the frame-
wise ground-truth (x-axis stands for the time in the video while different
sub-action classes are colorized differently); the second row is the
prediction from [3]; the third row is our prediction; and the fourth row is
the uniform sampled 8 frames from the original video. Best viewed zoom in and
in color.

Effectiveness of Multi-occur Temporal Path. We then assess the impact of using
the proposed multi-occur temporal path in decoding. Results on all three
datasets are in table 5. We can see the designed multi-occur temporal path
decoding improves both MoF and F1 on 50Salads and FineGym, but does not on
Breakfast. We observe that the multi-occurrence of one sub-action in the same
video happens more frequently in 50Salads and FineGym, while in Breakfast,
most of the sub-actions only occur once in each video, therefore, the multi-
occur temporal path is not as helpful on Breakfast as on the other datasets.
It is worth to mention that our CAP algorithm actually generates an exact the
same temporal path (with each sub-action occurring only once) automatically as
turning off the function for the Breakfast, representing that the CAP
algorithm is self-applicable to different scenarios.

###  4.4 Qualitative Results

Fig. 2 visualizes two examples from Breakfast dataset. From both Fig. 2(a) and
Fig. 2(b) we can see that our CAP algorithm is able to effectively suppress
the over-segmentation issue in baseline by introducing the co-occurrence
relations among sub-actions in decoding.

##  5 Conclusion

In this work, we have shown that using self-supervised methods in feature
learning can effectively improve the performance of unsupervised temporal
action segmentation. We have also proposed a CAP algorithm that helps model
the co-occurrence among sub-actions and can better handle complex activity
structures in videos. Our approach has achieved SOTA performance on three
public benchmarks in unsupervised setting, and even outperform several
recently proposed weakly-supervised methods. The future direction can be
leveraging the deep clustering approaches in further improving the accuracy of
the task.

## Acknowledgement

We thank reviewers for the valuable suggestions. Zhe Wang personally thanks
Fanyi Xiao, Bing Shuai and Uta Büchler for helpful discussion

## References

  * [1] Sathyanarayanan N. Aakur and Sudeep Sarkar.  A perceptual prediction framework for self supervised event segmentation.  In CVPR, 2019.
  * [2] Jean-Baptiste Alayrac, Piotr Bojanowski, Nishant Agrawal, Josef Sivic, Ivan Laptev, and Simon Lacoste-Julien.  Unsupervised learning from narrated instruction videos.  In CVPR, 2016.
  * [3] AnnaKukleva, Hilde Kuehne, Fadime Sener, and Jurgen Gall.  Unsupervised learning of action classes with continuous temporal embedding.  In CVPR, 2019.
  * [4] Sagie Benaim, Ariel Ephrat, Oran Lang, Inbar Mosseri, William T. Freeman, Michael Rubinstein, Michal Irani, and Tali Dekel.  Speednet: Learning the speediness in videos.  In CVPR, 2020.
  * [5] Piotr Bojanowski, Rémi Lajugie, Francis Bach, Ivan Laptev, Jean Ponce, Cordelia Schmid, and Josef Sivic.  Weakly supervised action labeling in videos under ordering constraints.  In ECCV, 2014.
  * [6] Biagio Brattoli, Uta Büchler, Anna-Sophia Wahl, Martin E. Schwab, and Björn Ommer.  Lstm self-supervision for detailed behavior analysis.  In CVPR, 2017.
  * [7] Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze.  Deep clustering for unsupervised learning of visual features.  In ECCV, 2018.
  * [8] Joao Carreira and Andrew Zisserman.  Quo vadis, action recognition? a new model and the kinetics dataset.  In CVPR, 2017.
  * [9] Chien-Yi Chang, De-An Huang, Yanan Sui, Li Fei-Fei, and Juan Carlos Niebles.  D3tw: Discriminative differentiable dynamic time warping for weakly supervised action alignment and segmentation.  In CVPR, 2019.
  * [10] Min-Hung Chen, Baopu Li, Yingze Bao, Ghassan AlRegib, and Zsolt Kira.  Action segmentation with joint selfsupervised temporal domain adaptation.  In CVPR, 2020.
  * [11] George E. Dahl, Dong Yu, Li Deng, and Alex Acero.  Context-dependent pre-trained deep neural networks for large-vocabulary speech recognition.  In IEEE Transactions on Audio, Speech, and Language Processing, 2012\.
  * [12] Ali Diba, Vivek Sharma, Luc Van Gool, and Rainer Stiefelhagen.  Dynamonet: Dynamic action and motion network.  In Arxiv, 2019.
  * [13] Li Ding and Chenliang Xu.  Weakly-supervised action segmentation with iterative soft boundary assignment.  In CVPR, 2018.
  * [14] E. Elhamifar and Z. Naing.  Unsupervised procedure learning via joint dynamic summarization.  In ICCV, 2019.
  * [15] Dave Epstein, Boyuan Chen, and Carl Vondrick.  Oops! predicting unintentional action in video.  In CVPR, 2020.
  * [16] Yazan Abu Farha and Juergen Gall.  Ms-tcn: Multi-stage temporal convolutional network for action segmentation.  In CVPR, 2019.
  * [17] Mohsen Fayyaz and Juergen Gall.  Sct: Set constrained temporal transformer for set supervised action segmentation.  In CVPR, 2020.
  * [18] Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He.  Slowfast networks for video recognition.  In ICCV, 2019.
  * [19] Christoph Feichtenhofer, Axel Pinz, and Andrew Zisserman.  Convolutional two-stream network fusion for video action recognition.  In CVPR, 2016.
  * [20] Basura Fernando, Hakan Bilen, Efstratios Gavves, and Stephen Gould.  Self-supervised video representation learning with odd-one-out networks.  In CVPR, 2017.
  * [21] Spyros Gidaris, Praveer Singh, and Nikos Komodakis.  Unsupervised representation learning by predicting image rotations.  In ICLR, 2018.
  * [22] Tengda Han, Weidi Xie, and Andrew Zisserman.  Video representation learning by dense predictive coding.  In ICCV, 2019.
  * [23] Rui Hou, Chen Chen, and Mubarak Shah.  Tube convolutional neural network (t-cnn) for action detection in videos.  In ICCV, 2017.
  * [24] De-An Huang, Li Fei-Fei, and Juan Carlos Niebles.  Connectionist temporal modeling for weakly supervised action labeling.  In ECCV, 2016.
  * [25] Yifei Huang, Yusuke Sugano, and Yoichi Sato.  Improving action segmentation via graph based temporal reasoning.  In CVPR, 2020.
  * [26] Dahun Kim, Donghyeon Cho, and In So Kweon.  Self-supervised video representation learning with space-time cubic puzzles.  In AAAI, 2019.
  * [27] Hilde Kuehne, Ali Arslan, and Thomas Serre.  Recovering the syntax and semantics of goal-directed human activities.  In CVPR, 2014.
  * [28] Hilde Kuehne, Juergen Gall, and Thomas Serre.  An end-to-end generative framework for video segmentation and recognition.  In WACV, 2016.
  * [29] H. W. Kuhn.  The hungarian method for the assignment problem.  In Wiley Online Library, 1955.
  * [30] Colin Lea, Michael D Flynn, Rene Vidal, Austin Reiter, and Gregory D Hager.  Temporal convolutional networks for action segmentation and detection.  In CVPR, 2017.
  * [31] Colin Lea, Austin Reiter, René Vidal, and Gregory D Hager.  Segmental spatiotemporal cnns for fine-grained action segmentation.  In ECCV, 2016.
  * [32] Jun Li and Sinisa Todorovic.  Weakly supervised energy-based learning for action segmentation.  In ICCV, 2019.
  * [33] Jun Li and Sinisa Todorovic.  Set-constrained viterbi for set-supervised action segmentation.  In CVPR, 2020.
  * [34] Ji Lin, Chuang Gan, and Song Han.  Tsm: Temporal shift module for efficient video understanding.  In ICCV, 2019.
  * [35] Tianwei Lin, Xu Zhao, Haisheng Su, Chongjing Wang, and Ming Yang.  Bsn: Boundary sensitive network for temporal action proposal generation.  In ECCV, 2018.
  * [36] Ishan Misra, C. Lawrence Zitnick, and Martial Hebert.  Shuffle and learn: Unsupervised learning using temporal order verification.  In ECCV, 2016.
  * [37] Florent Perronnin, Jorge Sánchez, and Thomas Mensink.  Improving the fisher kernel for large-scale image classification.  In ECCV, 2010.
  * [38] T. Quach and M Farooq.  Maximum likelihood track formation with the viterbi algorithm.  In IEEE Conference on Decision and Control, 1994.
  * [39] Alexander Richard, Hilde Kuehne, and Juergen Gall.  Weakly supervised action learning with rnn based fine-to-coarse modeling.  In CVPR, 2017.
  * [40] Alexander Richard, Hilde Kuehne, and Juergen Gall.  Action sets: Weakly supervised action segmentation without ordering constraints.  In CVPR, 2018.
  * [41] Alexander Richard, Hilde Kuehne, Ahsan Iqbal, and Juergen Gall.  Neuralnetwork-viterbi: A framework for weakly supervised video learning.  In CVPR, 2018.
  * [42] Fadime Sener and Angela Yao.  Unsupervised learning and segmentation of complex activities from video.  In CVPR, 2018.
  * [43] Dian Shao, Yue Zhao, Bo Dai, and Dahua Lin.  Finegym: A hierarchical video dataset for fine-grained action understanding.  In CVPR, 2020.
  * [44] Dian Shao, Yue Zhao, Bo Dai, and Dahua Lin.  Intra- and inter-action understanding via temporal action parsing.  In CVPR, 2020.
  * [45] Sebastian Stein and Stephen J. McKenna.  Combining embedded accelerometers with computer vision for recognizing food preparation activities.  In UBICOMP, 2013.
  * [46] Du Tran, Lubomir Bourdev, Rob Fergus, Lorenzo Torresani, and Manohar Paluri.  Learning spatiotemporal features with 3d convolutional networks.  In ICCV, 2015.
  * [47] Rosaura G. VidalMata, Walter J. Scheirer, Anna Kukleva, David Cox, and Hilde Kuehne.  Joint visual-temporal embedding for unsupervised learning of actions in untrimmed sequences.  In Arxiv, 2020.
  * [48] Heng Wang and Cordelia Schmid.  Action recognition with improved trajectories.  In ICCV, 2013.
  * [49] Limin Wang, Yuanjun Xiong, Zhe Wang, Yu Qiao, Dahua Lin, Xiaoou Tang, and Luc Van Gool.  Temporal segment networks: Towards good practices for deep action recognition.  In ECCV, 2016.
  * [50] Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He.  Non-local neural networks.  In CVPR, 2018.
  * [51] Xiaolong Wang, Allan Jabri, and Alexei A Efros.  Learning correspondence from the cycle-consistency of time.  In CVPR, 2019.
  * [52] Zhenzhi Wang, Ziteng Gao, Limin Wang, Zhifeng Li, and Gangshan Wu.  Boundary-aware cascade networks for temporal action segmentation.  In ECCV, 2020.
  * [53] Dejing Xu, Jun Xiao, Zhou Zhao, Jian Shao, Di Xie, and Yueting Zhuang.  Self-supervised spatiotemporal learning via video clip order prediction.  In CVPR, 2019.
  * [54] Huijuan Xu, Abir Das, and Kate Saenko.  R-c3d: Region convolutional 3d network for temporal activity detection.  In ICCV, 2017.
  * [55] Ceyuan Yang, Yinghao Xu, Jianping Shi, Bo Dai, and Bolei Zhou.  Temporal pyramid network for action recognition.  In CVPR, 2020.
  * [56] Yue Zhao, Yuanjun Xiong, Limin Wang, Zhirong Wu, Xiaoou Tang, and Dahua Lin.  Temporal action detection with structured segment networks.  In ICCV, 2017.
  * [57] Bolei Zhou, Alex Andonian, Aude Oliva, and Antonio Torralba.  Temporal relational reasoning in videos.  In ECCV, 2018.


# Analyzing High-Resolution Clouds and Convection using Multi-Channel VAEs

Harshini Mangipudi
University of California Irvine
<EMAIL_ADDRESS>
Griffin Mooers
University of California Irvine
<EMAIL_ADDRESS>
Mike Pritchard
University of California Irvine
<EMAIL_ADDRESS>
Tom Beucler
University of Lausanne
<EMAIL_ADDRESS>
Stephan Mandt
University of California Irvine
<EMAIL_ADDRESS>

###### Abstract

Understanding the details of small-scale convection and storm formation is
crucial to accurately represent the larger-scale planetary dynamics.
Presently, atmospheric scientists run high-resolution, storm-resolving
simulations to capture these kilometer-scale weather details. However, because
they contain abundant information, these simulations can be overwhelming to
analyze using conventional approaches. This paper takes a data-driven approach
and jointly embeds spatial arrays of vertical wind velocities, temperatures,
and water vapor information as three "channels" of a VAE architecture. Our
"multi-channel VAE" results in more interpretable and robust latent structures
than earlier work analyzing vertical velocities in isolation. Analyzing and
clustering the VAE’s latent space identifies weather patterns and their
geographical manifestations in a fully unsupervised fashion. Our approach
shows that VAEs can play essential roles in analyzing high-dimensional
simulation data and extracting critical weather and climate characteristics.

##  1 Introduction

Long-term climate modeling is increasingly accurate, but any climate
simulation high resolution enough to explicitly resolve kilometer-scale
processes controlling clouds and storms not only requires high computational
expense but also produces massive amounts of data. This large data volume
overwhelms physically-informed methods traditionally used to better understand
storm organization and how small-scale clouds connect to the large-scale
circulation; even established dimensionality reduction techniques such as
Principle Component Analysis (PCA) [16] may fail to capture these nonlinear
relations. In contrast, unsupervised learning, more specifically deep
generative modeling [5, 3] can summarize explicitly-resolved storms without
omitting fine-scale, nonlinear information.

Variational Autoencoders (VAEs) in particular are well-suited for dynamics
discovery and understanding because of ability to learn meaningful latent
representations of the data [11, 7, 17, 13]. VAE applications on climate and
weather data have thus far been rare. Physical processes like a 2D laminar
flow have been successfully represented by a VAE [4]. Some analysis has been
conducted with VAEs on spatio-temporal earth dynamics and with state variables
like temperature and potential vorticity data to cluster phenomena like the
Polar Vortex [14, 15]. However, the training/test data applied to the VAEs in
all these cases was highly idealized and smaller in dimension than a storm-
resolving simulation output.

A single-channel VAE architecture (SVAE), modeling velocity fields in high-
resolution climate simulation has been implemented [12]; however,
relationships between other variables known to be important controls for
convection were ignored. To bypass the limit of using a single physical state
variable input in the context of deep generative modeling for climate
simulation analysis, we upgrade our network architecture to that of a three-
channel VAE - temperature and water vapor in addition to vertical velocity. We
hypothesize that this multi-variate approach can enable increasingly
satisfying physical clustering through a more thermodynamically-informed
latent space.

##  2 Methods

###  2.1 Data and preprocessing

To provide our VAE with a multi-variate, high resolution simulation, we use
output from the System for Atmospheric Modeling (SAM), a Storm-Resolving Model
(SRM). We extract 1.5e6 30x128 snapshots or "images" of vertical velocity
(m/s), temperature (K), and water vapor concentration (kg/kg) fields. The
dimensionality of our images comes from the simulation design of an atmosphere
with thirty vertical levels and 128 SRM columns embedded in each grid cell of
a planetary-scale General Circulation Model (GCM) to explicitly resolve deep
convection in thousands of embedded sub-domains [6]. The high-resolution
convection data comes from all tropical areas ranging from latitudes of 20
degrees south to 20 degrees North during boreal winter. This geographically
diverse output thus includes samples ranging from moist tropical rain forests
to arid deserts, spanning both marine and continental zones. Each of the three
variables is re-scaled separately from zero to one using standard
normalization techniques.

###  2.2 Model architecture and upgrades

We expand on a baseline VAE architecture designed in [12]. Our VAE is fully
convolutional and has four layers in the encoder and decoder. To address the
inherent rate-distortion trade-off native to VAEs [1] and align the latent
variables of the model more closely with the output, we implement a linear
annealing [2] in the Kullback–Leibler (KL) Divergence term on our VAE loss
function (Equation 1). We initially weight the KL Divergence as 0 in order to
prioritize learning in the encoder for a superior latent space but increase
$\beta$ to 1 over 1600 epochs.

$\mathrm{ELBO}\left(\textbf{x};\theta,\phi,\textbf{z}\right)=\mathbb{E}_{q_{\phi}(\textbf{z}|\textbf{x})}\left[\log
p_{\theta}(\textbf{x}|\textbf{z})\right]-\beta\times
D_{\mathrm{KL}}\left(q_{\phi}(\textbf{z}|\textbf{x})\,||\,p(\textbf{z})\right),$
(1)

In contrast to earlier work, we interpret temperature, water vapor, and raw
vertical velocity fields as three “channels” of an image, and adopt a VAE
architecture to jointly embed them. This also increases the compression from
the input vector to a latent space of 1024 from roughly 4x in [12] to 10x. We
also continue to use a SVAE trained on vertical velocity fields as a baseline
for comparisons.

##  3 Results

#### Latent space credibility and exploration

By expanding the input to three high-resolution variables, we are asking
substantially more of our encoder in terms of dimensionality reduction and
feature extraction. There is no guarantee this architecture can create the
same physically interpretable latent space as the SVAE in [12]. However, we
can gain better understanding of the latent space by "colorizing" it by the
magnitude of the corresponding vertical velocity field in the test dataset and
its land fraction. In the tropics there are regions of intense convection,
such as over the rain-forests and hot tropical waters of the Pacific warm
pool; however, there will also be dry regions like deserts where deep
convection will be suppressed. We want such known disparate convection types
to be separated in some capacity on the latent space for it to be useful for
physical analysis. When our latent space is colorized (Fig 1, a and b), we do
see a clear sorting by magnitude, and convection over land preferentially
appearing on different parts of the latent space than convection over the
ocean. The full animated visual can be found at this link. This suggests that
the added complexity is not overwhelming the VAE, and we now ask: What are the
advantages of the MVAE compared to the SVAE when it comes to anomaly detection
and regime classification?

#### Anomaly detection

We determine the similarity between the trained VAEs and their respective
latent spaces by first looking at density estimation properties. To test
whether the MVAE adds value compared to the SVAE, we calculate the Evidence
Lower Bound (ELBO) for both VAEs at each of the 1.5e6 samples in the test
dataset. The higher this ELBO score the more anomalous the VAE finds the
convection sample. We look at just the top five percent of ELBO anomalies for
both the SVAE and MVAE. We then plot the frequency at which these anomalies
occurred at each grid-cell in the tropics (Fig 2 b and c) and ask whether the
results agree with domain knowledge of where geographically rare convective
events tend to occur. Immediately we see different patterns lit up by the
different VAEs. The SVAE hones in on continental zones, especially over more
arid regions (Fig 2b). The MVAE highlights areas over both land and sea (Fig
2c), which is physically satisfying since rare storms are known to occur in
both regions. Specifically, the MVAE finds deep convection over the tropical
rain forests and the Inter-Tropical Convergence Zone to be most anomalous. But
more significantly than just disagreeing about the geographic location of the
most extreme anomalies, the two VAEs each find anomalies of very different
vertical structures. In the median vertical profile in each anomaly group, the
SVAE emphasizes strong convection near the surface of the earth, which is not
typically recognized as an interesting phenomenon in the atmospheric sciences,
whereas the MVAE focuses on convection deep in the upper Troposphere, a
reassuringly familiar extreme tropical storm structure (Fig 2a).

#### Data driven convection regimes

To analyze the latent space in an objective fashion, we bring in another
unsupervised learning algorithm: K-Means Clustering. We cluster separately on
both latent spaces to determine if the latent structures are different enough
for the MVAE to highlight different types of convective organization. Our
analysis turns up three distinct clusters in both the latent spaces (Figs 3).
However, there are significant differences in the vertical profiles and
geographic location of the regimes. The SVAE latent space successfully
separates the deepest convection from shallower forms that occur in the
subtropical ocean basins (Fig 3a and g, blue lines). But the third cluster in
the SVAE group, similar to the anomalies it finds (Fig 2b), reflects the same
puzzling mode of shallow convection over drier land areas (Fig 3d, blue line).
The MVAE clusters deep convection similarly to the SVAE but attractively does
not make a distinct regime for the mode of dry, continental shallow convection
(Fig 3 d,g green vs. blue). Instead, unlike the SVAE, the subtropical shallow
convective mode is satisfyingly separated into two distinct sub-regions – one
in the Central Pacific/Atlantic, and another in the eastern parts of these
ocean basins where especially distinct stratocumulus clouds are known to occur
[8, 9, 10]. Since these two forms of low cloud have different characteristic
horizontal eddy scales, we speculate that the inclusion of water vapor to the
input of the MVAE down-weights the importance of the dry, shallow mode (Fig
3d; green vs. blue) and allows a more satisfying physical clustering of
convective regimes.

##  4 Conclusion and broader impacts

We explore the ability of a Variational Autoencoder (VAE) to find a physically
interpretable representation of a multi-variate, high-resolution climate
simulation. We find that adding multiple variables as "channels" to our VAE
highlights drastically different convection anomaly types and geographic
locations compared to a VAE trained only on a single variable. The different
regimes identified by clustering both latent spaces suggest there is value to
a multi-variate approach using both dynamic and thermodynamic variables to
cluster convection regimes successfully in a fully unsupervised fashion. It is
possible that this multi-variate approach would also be valuable for global
storm-resolving climate simulations that produce even more data and augment
traditional physically-informed approaches to clustering the cloud regime-
dependence of climate change projections.

---
Figure 1:  The 3D projection of Principle Component Analysis of our multi-
channel VAE latent space. We colorize by land/sea fraction and intensity of
the vertical velocity field at each sample in the test dataset. The
organization of this latent structure appears physically interpretable.
---
Figure 2:  The vertical profiles (a, solid lines), 25th and 75th percentiles
(a, dashed lines) and geographic distributions (b,c,d) of convection anomalies
defined by our VAEs. We threshold the test data by the Evidence Lower Bound
(ELBO) anomalies and examine only the top 5 percent of ELBO scores. The multi-
channel VAE finds anomalies of different structure and geographic location
than the single-channel VAE.
---
Figure 3:  The results of K-Means clustering on the latent space of both VAEs
separately. We find that three is the largest number of clusters in which each
convective regime has both a distinct vertical profile (a,d,g) and distinct
geographic pattern of frequency of occurrence (b,c,e,f,h,i). However, the
three regimes found in each latent do not match suggesting unique organization
in each VAE.

##  5 Acknowledgements

The authors thank the MAPS program and NSF grant 1633631, OAC-1835863,
AGS-1734164, IIS-2047418, IIS-2003237 and IIS-2007719 for funding support and
co-funding by the Enabling Aerosol-cloud interactions at GLobal convection-
permitting scalES (EAGLES) project (74358), of the U.S. DOE Office of
Biological and Environmental Research, Earth System Model Development program
area. The link for training and analysis of the MVAE can be found at this
link. Computational resources were provided by the Extreme Science and
Engineering Discovery Environment supported by NSF Grant number ACI-1548562
(charge number TG-ATM190002).

## References

  * [1] A. A. Alemi, B. Poole, I. S. Fischer, J. V. Dillon, R. A. Saurous, and K. Murphy.  Fixing a broken elbo.  In ICML, 2018.
  * [2] S. R. Bowman, L. Vilnis, O. Vinyals, A. M. Dai, R. Józefowicz, and S. Bengio.  Generating sentences from a continuous space.  In CoNLL, 2016.
  * [3] D. Crommelin and W. Edeling.  Resampling with neural networks for stochastic parameterization in multiscale systems, 2020.
  * [4] S. Eismann, S. Bartzsch, and S. Ermon.  Shape optimization in laminar flow with a label-guided variational autoencoder, 2017.
  * [5] I. J. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio.  Generative adversarial networks, 2014.
  * [6] W. W. Grabowski and P. K. Smolarkiewicz.  Crcp: a cloud resolving convection parameterization for modeling the tropical convecting atmosphere.  Physica D: Nonlinear Phenomena, 133:171–178, 1999.
  * [7] H. Huang, Z. Li, R. He, Z. Sun, and T. Tan.  Introvae: Introspective variational autoencoders for photographic image synthesis, 2018.
  * [8] R. H. Johnson, T. M. Rickenbach, S. A. Rutledge, P. E. Ciesielski, and W. H. Schubert.  Trimodal characteristics of tropical convection.  Journal of Climate, 12(8):2397 – 2418, 1999.
  * [9] B. Khouider and A. J. Majda.  Multicloud convective parametrizations with crude vertical structure.  Theoretical and Computational Fluid Dynamics, 20(5):351–375, 2006\.
  * [10] B. Mapes, S. Tulich, J. Lin, and P. Zuidema.  The mesoscale convection life cycle: Building block or prototype for large-scale tropical waves?  Dynamics of Atmospheres and Oceans, 42(1):3–29, 2006.  Atmospheric Convection and Wave Interactions (title for cover & spine) Atmospheric Convection and Wave Interactions: Convective Life Cycles and Scale Interactions in Tropical Waves (title for Special Issue title page).
  * [11] L. Mescheder, S. Nowozin, and A. Geiger.  Adversarial variational bayes: Unifying variational autoencoders and generative adversarial networks, 2018.
  * [12] G. Mooers, J. Tuyls, S. Mandt, M. Pritchard, and T. Beucler.  Generative modeling for atmospheric convection, 2020.
  * [13] A. A. Pol, V. Berger, G. Cerminara, C. Germain, and M. Pierini.  Anomaly detection with conditional variational autoencoders, 2020.
  * [14] X.-A. Tibau, C. Requena-Mesa, C. Reimers, J. Denzler, V. Eyring, M. Reichstein, and J. Runge.  Supernovae: Vae based kernel pca for analysis of spatio-temporal earth data.  In Proceedings of the 10th International Conference on Climate Informatics, 2018.
  * [15] N. Tilinina, M. Krinitskiy, Y. Zyulyaeva, and S. Gulev.  Clustering of the Polar Vortex states using deep convolutional neural networks.  In EGU General Assembly Conference Abstracts, EGU General Assembly Conference Abstracts, page 17539, Apr. 2019.
  * [16] D. S. Wilks.  Statistical methods in the atmospheric sciences.  Elsevier, 2006.
  * [17] Y. Wu, Y. Burda, R. Salakhutdinov, and R. Grosse.  On the quantitative analysis of decoder-based generative models, 2017\.