# Multimodal Prototype-Enhanced Network for Few-Shot Action Recognition

Xinzhe Ni1, Yong Liu1, Hao Wen1, Yatai Ji1, Jing Xiao2, Yujiu Yang1
Corresponding author

###### Abstract

Current methods for few-shot action recognition mainly fall into the metric
learning framework following ProtoNet, which demonstrates the importance of
prototypes. Although they achieve relatively good performance, the effect of
multimodal information is ignored, e.g. label texts. In this work, we propose
a novel MultimOdal PRototype-ENhanced Network (MORN), which uses the semantic
information of label texts as multimodal information to enhance prototypes. A
CLIP visual encoder and a frozen CLIP text encoder are introduced to obtain
features with good multimodal initialization. Then in the visual flow, visual
prototypes are computed by a Temporal-Relational CrossTransformer (TRX) module
for example. In the text flow, a semantic-enhanced (SE) module and an
inflating operation are used to obtain text prototypes. The final multimodal
prototypes are then computed by a multimodal prototype-enhanced (MPE) module.
Besides, we define a PRototype SImilarity DiffErence (PRIDE) to evaluate the
quality of prototypes, which is used to verify our improvement on the
prototype level and effectiveness of MORN. We conduct extensive experiments on
four popular datasets, and MORN achieves state-of-the-art results on HMDB51,
UCF101, Kinetics and SSv2. When plugging PRIDE into the training stage, the
performance can be further improved.

## Introduction

(a) Prototype-based strategy.

(b) Multimodal-based strategy.

(c) Our multimodal prototype-enhanced strategy.

Figure 1: Existing metric-learning strategy (a) and (b) and our multimodal
prototype-enhanced strategy (c) for few-shot action recognition. Results on
TRX (Perrett et al. 2021) with CLIP visual encoder, multimodal feature-
enhanced strategy and multimodal prototype-enhanced strategy are shown.

Few-shot action recognition is a challenging problem in computer vision
because of the scarcity of labeled samples and the complicated temporal
evolution of videos. Most of the methods (Bishay, Zoumpourlis, and Patras
2019; Kumar Dwivedi et al. 2019; Zhu and Yang 2018; Zhang et al. 2020b; Fu et
al. 2020; Cao et al. 2020; Zhang et al. 2020a; Zhang, Zhou, and He 2021;
Perrett et al. 2021; Thatipelli et al. 2022; Wang et al. 2022; Huang, Yang,
and Sato 2022; Liu et al. 2021; Zhu et al. 2021; Wu et al. 2022) for few-shot
action recognition mainly fall into the metric-learning framework, whose key
idea is to learn a proper metric space tuned to a particular task. Among
various metric-learning works, ProtoNet (Snell, Swersky, and Zemel 2017) has
been followed by many works and has far-reaching influence. It proposes
Prototype as a representation of each category in a few-shot learning
scenario. For the scarcity of samples, it is likely that the overfitting
problem will arise with a complicated matching strategy. With prototypes,
however, the model reflects a simpler inductive bias and works better in a
limited-data regime, _i.e._ few-shot learning scenario. Since we notice the
importance of prototypes, it is natural to ask: _how to obtain representative
prototypes_?

Intuitively, it is useful to directly introduce a certain prototype-enhanced
module as shown in Figure 1a. Methods with prototype-based strategy (Kumar
Dwivedi et al. 2019; Perrett et al. 2021; Zhu et al. 2021) expand the concept
of prototype, which is not a simple average of video features. However, the
information provided by a single modality is limited especially in the few-
shot scenario, and they ignore the effect of multimodal information.

As we know, video is a multimodal carrier rather than a series of RGB images.
Multimodal information generally yields better results since different
modalities provide complementary information, and is proved by (Huang et al.
2021). Therefore, another strategy is to use multimodal information to obtain
a better distance metric as shown in Figure 1b. We select label texts as
another modality inspired by ActionCLIP (Wang, Xing, and Liu 2021). There is
abundant semantic information in label texts to assist the classification. For
example, if one is given several short videos of “lifting up one end of
something” without being told what action it is, he may not find the same
pattern in a short time. Instead, it is much easier to distinguish them from
other actions when knowing the exact label. Based on this fact, we
preliminarily combine the semantic information of label texts on the feature
level. Compared to unimodal prototype-enhanced strategy in Figure 1a, the
model with multimodal feature-enhanced strategy achieves further improvement,
which verifies the effectiveness of multimodal information. However, methods
with multimodal-based strategy (Zhang et al. 2020b; Fu et al. 2020) ignore the
importance of prototypes and only combine the two modalities on the feature
level. As a result, there is still room for improvement with multimodal
prototype-enhanced strategy.

Considering both prototype and multimodal information, we propose a novel
MultimOdal PRototype-ENhanced Network (MORN) based on our multimodal
prototype-enhanced strategy as shown in Figure 1c. It is simple but well-
performed, and can be applied to methods producing prototypes. For example on
TRX (Perrett et al. 2021), we firstly obtain visual and text features by a
CLIP (Radford et al. 2021) visual encoder and a frozen CLIP text encoder.
Then, visual prototypes are computed by TRX baseline, and text prototypes are
computed by a semantic-enhanced (SE) module and an inflating operation.
Finally, we combine visual prototypes and text prototypes to obtain
representative prototypes in a multimodal prototype-enhanced (MPE) module. The
final multimodal prototypes are used during the distance computation with
query videos. Compared to multimodal feature-enhanced strategy in Figure 1b,
MORN achieves better or comparable improvement, which verifies the importance
of prototypes.

To verify that our MORN achieves improvement on prototypes rather than other
aspects, there is a strong need to evaluate the quality of prototypes with our
MORN. Inspired by (Zhang et al. 2021), we define PRototype SImilarity
DiffErence (PRIDE). Firstly, we compute the real prototype of each category of
all samples in the meta-test stage, and the similarity of current prototypes
in an episode with all real prototypes. Then, the difference of similarity
between its own category and the average of other categories is used to
evaluate the quality of prototypes. With experiments on PRIDE, we are
convinced that our MORN achieves improvement on the prototype level. Since
PRIDE indicates the quality of prototypes, it can also be used to assist the
training process. Therefore, we plug PRIDE into the training stage of MORN and
achieve further performance gains.

Our contributions can be summarized as follows:

  * •

We use the semantic information of label texts to enhance prototypes and
propose a simple but well-performed Multimodal Prototype-Enhanced Network
(MORN).

  * •

We define Prototype Similarity Difference (PRIDE) to verify improvement on the
prototype level and effectiveness of MORN. When plugging PRIDE into the
training stage, MORN can achieve further performance gains.

  * •

We conduct extensive experiments on four popular action recognition datasets
and the application of PRIDE. Methods with MORN achieve state-of-the-art
results on HMDB51, UCF101, Kinetics and SSv2.

## Related Work

### Few-Shot Image Classification

Few-shot image classification methods can be widely divided into three
categories: augmentation-based, optimization-based and metric-based.

Augmentation-based methods. The objective of these methods is to use
augmentation techniques or extra data to increase samples for training, and
improve the diversity of data. Some prior attempts are intuitive, including
(Perez and Wang 2017; Ratner et al. 2017). Besides, some works focus on the
semantic feature level, including (Chen et al. 2018, 2019). Rather than
applying augmentation techniques, (Pahde et al. 2019, 2021) introduce a GAN
(Goodfellow et al. 2020) architecture to generate extra images based on the
text description to compensate for the lack of data.

Optimization-based methods. The objective of these methods is to train a model
under the meta-learning framework so that it can adapt to novel tasks with
only a few optimization steps. (Andrychowicz et al. 2016; Gui et al. 2018; Lee
et al. 2019; Liu, Schiele, and Sun 2020; Ravi and Larochelle 2017) use the
meta-learner as an optimizer. MAML (Finn, Abbeel, and Levine 2017) and its
variants (Antoniou, Edwards, and Storkey 2018; Jamal and Qi 2019; Sun et al.
2019) aim to learn a robust model initialization.

Metric-based methods. The objective of these methods is to learn feature
embeddings under a certain distance metric with a better generalization
ability. Samples of novel categories can be accurately classified via a
nearest neighbor classifier with different distance metrics such as cosine
similarity (Vinyals et al. 2016; Ye et al. 2020), Euclidean distance (Snell,
Swersky, and Zemel 2017; Yoon, Seo, and Moon 2019), non-linear metric with
CNNs (Sung et al. 2018; Hou et al. 2019; Li et al. 2019) or graph neural
networks (Kim et al. 2019; Garcia and Bruna 2018; Yang et al. 2020). Our work
falls into the metric-learning framework and aims to solve a more complicated
few-shot video action recognition problem.

### Few-Shot Action Recognition

Most of the methods for few-shot action recognition mainly fall into the
metric-learning framework. More specifically, we divide them into three
categories: feature-based, multimodal-based and prototype-based.

Feature-based methods. The objective of these methods is to obtain reliable
video features or apply an effective alignment strategy. CMN (Zhu and Yang
2018), TARN (Bishay, Zoumpourlis, and Patras 2019), OTAM (Cao et al. 2020) and
ARN (Zhang et al. 2020a) make preliminary attempts to obtain reliable video
features for classification. ITANet (Zhang, Zhou, and He 2021) introduces a
video representation based on a self-attention mechanism and an implicit
temporal alignment. MASTAF (Liu et al. 2021) uses self-attention and cross-
attention mechanisms to obtain reliable spatial-temporal features. MT-FAN (Wu
et al. 2022) designs a motion modulator and a segment attention mechanism, and
conducts a temporal fragment alignment. STRM (Thatipelli et al. 2022) proposes
a local patch-level and a global frame-level enrichment module based on TRX.
Similarly, HyRSM (Wang et al. 2022) proposes a hybrid relation module to
enrich video features.

Prototype-based methods. Prototype is a representation of each category, and
is proposed by ProtoNet (Snell, Swersky, and Zemel 2017). The objective of
these methods is to compute representative prototypes by some prototype-
enhanced strategies with a single modality. ProtoGAN (Kumar Dwivedi et al.
2019) generates extra samples by using a conditional GAN with category
prototypes. PAL (Zhu et al. 2021) matches a prototype with all the query
samples instead of matching a query sample with the space of prototypes. TRX
(Perrett et al. 2021) applies CrossTransformer (Doersch, Gupta, and Zisserman
2020) to a few-shot action recognition scenario to obtain query-specific
category prototypes.

Multimodal-based methods. The objective of these methods is to use multimodal
information to assist classification. AMeFu-Net (Fu et al. 2020) introduces
depth information as another modality to assist classification. CMMN (Zhang et
al. 2020b) also uses the semantic information of label texts as ours and is
similar to our multimodal setting. However, it fails to apply a feature
extractor with a good multimodal initialization and ignores the importance of
prototypes. CPMT (Huang, Yang, and Sato 2022) uses object features as
multimodal information and proposes a compound prototype-matching strategy.
Thus, CPMT can be regarded as a multimodal-based method with a prototype-based
design. Our work also uses multimodal information and further computes
representative prototypes. Furthermore, we notice the remarkable effect of the
semantic information of label texts and the prototype-enhanced strategy.

Figure 2:  Overview of our proposed MORN on TRX on a 2-way 1-shot problem with
1 video for each category in the query set. In the visual flow, a CLIP visual
encoder is first introduced on videos with $L$ frames to obtain video
features. Then, support video features are passed to the Temporal-Relational
CrossTransformer (TRX) module to compute visual prototypes. In the text flow,
a frozen CLIP text encoder is first introduced on the prompted label texts.
Then, the semantic features of label texts are passed to a semantic-enhanced
(SE) module and are inflated as text prototypes. The visual prototypes and the
text prototypes are combined through a multimodal prototype-enhanced (MPE)
module.

## Method

### Problem formulation

There are two sets in the $N$-way $K$-shot few-shot scenario: a support set
$S^{N}_{K}=\\{s^{1}_{1},\cdots,s^{1}_{K},\cdots,s^{N}_{1},\cdots,s^{N}_{K}\\}$
and a query set
$Q^{N}_{M}=\\{q^{1}_{1},\cdots,q^{1}_{M},\cdots,q^{N}_{1},\cdots,q^{N}_{M}\\}$,
where $N$ denotes the number of different action categories, $K$ denotes the
number of videos each category in the support set and $M$ denotes the number
of videos each category in the query set. The objective of few-shot action
recognition is to learn a model with great generalization ability using only a
few video samples. Specifically, the model classifies a completely novel query
action to the right category by matching it with the most similar video in the
support set. Meanwhile, the whole dataset is divided into a base dataset
$C_{base}=\\{(x_{i},y_{i})\\}^{|C_{base}|}_{i=1}$ and a novel dataset
$C_{novel}=\\{(x_{i},y_{i})\\}^{|C_{novel}|}_{i=1}$, where $y_{i}$ is the
action category of a video sample $x_{i}$. Note that the categories of
$C_{base}$ and $C_{novel}$ are label-wise non-overlapping, _i.e.
$C_{base}\,\cap\,C_{novel}=\emptyset$_. To make full use of the few video
samples, we follow the episode training manner (Vinyals et al. 2016). Each
episode contains $K$ videos in the support set and $M$ videos in the query set
of $N$ categories.

### MORN

Considering both prototype and multimodal information, we propose MORN. The
overall architecture is shown in Figure 2, including a visual flow, a text
flow and a multimodal prototype-enhanced (MPE) module.

Visual flow. For each input video, we uniformly sample $L$ frames as in (Wang
et al. 2016). The video $s^{n}_{k}$ in the support set and $q^{n}_{m}$ in the
query set can be further denoted as:

$\displaystyle s^{n}_{k}$ $\displaystyle=\\{s^{n}_{k1},\cdots,s^{n}_{kL}\\},$
(1) $\displaystyle q^{n}_{m}$
$\displaystyle=\\{q^{n}_{m1},\cdots,q^{n}_{mL}\\},$

where $s^{n}_{k}$, $q^{n}_{m}\in\mathbb{R}^{L\times H\times W\times 3}$. $H$
and $W$ denote the height and width of an image. Then, we apply a pre-trained
CLIP visual encoder for better multimodal initialization. The visual features
of each frame $s^{n}_{ki}\in\mathbb{R}^{H\times W\times 3}$ in the support set
and $q^{n}_{mi}\in\mathbb{R}^{H\times W\times 3}$ in the query set are defined
as:

$\displaystyle F^{V}_{sk}$
$\displaystyle=\\{f_{v}(s^{n}_{k1}),\cdots,f_{v}(s^{n}_{kL})\\},$ (2)
$\displaystyle F^{V}_{qm}$
$\displaystyle=\\{f_{v}(q^{n}_{m1}),\cdots,f_{v}(q^{n}_{mL})\\},$

where $F^{V}_{sk}$, $F^{V}_{qm}\in\mathbb{R}^{L\times d}$. The visual features
of videos in the support set and the query set are denoted as:

$\displaystyle F^{V}_{s}$
$\displaystyle=\\{F^{V}_{s1},\cdots,F^{V}_{s(NK)}\\},$ (3) $\displaystyle
F^{V}_{q}$ $\displaystyle=\\{F^{V}_{q1},\cdots,F^{V}_{q(NM)}\\},$

where $F^{V}_{s}\in\mathbb{R}^{NK\times L\times d}$ and
$F^{V}_{q}\in\mathbb{R}^{NM\times L\times d}$. Then, we sample tuples of video
frames $F^{Tuple}_{q}=\\{F^{Tuple}_{q1},\cdots,F^{Tuple}_{q(NM)}\\}$ and
compute visual prototypes of each episode through TRX:

$P^{V}=TRX(F^{V}_{q},F^{V}_{s},F^{V}_{s}),$ (4)

where $F^{Tuple}_{q},P^{V}\in\mathbb{R}^{NM\times C^{\omega}_{L}\times
d_{p}}$, and $C^{\omega}_{L}$ is the combinatorial number.

Text flow. In the training stage for each sample $(x_{i},y_{i})\in C_{base}$,
we firstly make $n_{temp}$ discrete templates
$P_{temp}=\\{p^{temp}_{1},\cdots,p^{temp}_{n_{temp}}\\}$ of $y_{i}$ as text
prompts. Then, we apply a CLIP text tokenizer to obtain tokenized text
sequences $T_{i}$:

$\displaystyle T_{i}$ $\displaystyle=Tokenizer(p^{temp}_{j}||y_{i}),$ (5)
$\displaystyle j$ $\displaystyle=1,\cdots,n_{temp},\,i=1,\cdots,N,$

where $||$ denotes a concatenation operation. In the practical meta-training
and meta-test stage, the template is randomly selected once per episode. Then,
the features of prompted label texts are obtained by a frozen CLIP text
encoder:

$F^{T}_{i}=f_{t}(T_{i}),\,\\{i\,|\,x_{i}\in S^{N}_{K}\,\cup\,Q^{N}_{M}\\},$
(6)

where $F^{T}_{i}\in\mathbb{R}^{1\times d}$. To obtain text features with more
reliable semantic information, we further apply a semantic-enhanced (SE)
module $g(\cdot)$:

$P^{T}_{i}=g(F^{T}_{i}),$ (7)

where $P^{T}_{i}\in\mathbb{R}^{1\times d_{p}}$ and a multi-head attention
mechanism with 4 heads is used as $g(\cdot)$. So we obtain
$\\{P^{T}_{1},\cdots,P^{T}_{NM}\\}$ in each episode. Since different frames of
videos in the same category have the same label, we simply inflate $P^{T}_{i}$
of the same category to keep the same dimension with visual prototypes. Then,
we obtain text prototypes $P^{T}$:

$P^{T}=\\{P^{T}_{1},\cdots,P^{T}_{1},\cdots,P^{T}_{N},\cdots,P^{T}_{N}\\},$
(8)

where $P^{T}\in\mathbb{R}^{NM\times C^{\omega}_{L}\times d_{p}}$.

Multimodal prototype-enhanced module. To use multimodal information to enhance
prototypes, we propose a simple but well-performed multimodal prototype-
enhanced (MPE) module. The choice of the MPE module is flexible containing
weighted average, multi-head attention and so on. Here, we apply weighted
average as our default setting:

$P^{M}=(1-\lambda)P^{V}+\lambda P^{T},$ (9)

where $P^{M}\in\mathbb{R}^{NM\times C^{\omega}_{L}\times d_{p}}$ and $\lambda$
is the multimodal enhanced hyper-parameter. The multimodal prototypes are used
as the final prototypes. Then, distances are computed between multimodal
prototypes and $F^{Tuple}_{q}$, and are passed as logits to a cross-entropy
loss.

### PRIDE

Denote the number of categories in the meta-test stage as $N_{novel}$. We
first compute the real prototype in a 1-way scenario by averaging all
prototypes of the same category $i$ in the novel dataset, which is based on
(Zhang et al. 2021):

$P^{real}_{i}=\frac{1}{|C^{i}_{novel}|}\sum_{(x,y)\in C^{i}_{novel}}P_{i}(x),$
(10)

where $P_{i}(x)$ is the query-specific category prototype of video $x$ and
$C^{i}_{novel}$ is the i-th category in $C_{novel}$. The whole set of real
prototypes is denoted as:

$P^{real}=\\{P^{real}_{1},\cdots,P^{real}_{N_{novel}}\\}.$ (11)

Then, the cosine similarity can be computed between a given $P_{i}(x)$ and
$P^{real}_{j}$:

$Sim_{j}=sim(P_{i}(x),P^{real}_{j}),\,j=1,\cdots,N_{novel},$ (12)

where $sim(\cdot,\cdot)$ is the cosine similarity operation. The similarity of
other categories is denoted as:

$Sim_{other}=\frac{1}{N_{novel}-1}\sum_{1\leq j\neq i\leq N_{novel}}Sim_{j}.$
(13)

We can now denote PRIDE as:

$PRIDE_{i}(x)=Sim_{i}-Sim_{other},\,(x,y)\in C^{i}_{novel}.$ (14)

According to the computation process, PRIDE can accurately reflect the
category separability of prototypes. Thus, PRIDE can be used to verify the
improvement on the prototype level and the effectiveness of our MORN. A higher
PRIDE value means more representative prototypes with a better discriminating
ability.

Method | Strategy | Backbone | HMDB51 | UCF101 | Kinetics | SSv2 | Average
---|---|---|---|---|---|---|---
ARN (Zhang et al. 2020a) | Feature-based | C3D | 60.6 | 83.1 | 82.4 | - | 75.4
OTAM (Cao et al. 2020) | ResNet-50 | 68.0 | 88.9 | 85.8 | 52.3 | 73.8
HyRSM (Wang et al. 2022) | 76.0 | 94.7 | 86.1 | 69.0 | 81.5
MT-FAN (Wu et al. 2022) | 74.6 | 95.1 | 87.4 | 60.4 | 79.4
STRM (Thatipelli et al. 2022) | 77.3 | 96.9 | 86.7 | 68.1 | 82.3
PAL (Zhu et al. 2021) | Prototype-based | ResNet-50 | 75.8 | 95.2 | 87.1 | 62.6 | 80.2
TRX (Perrett et al. 2021) | 75.6 | 96.1 | 85.9 | 64.6 | 80.6
CMMN (Zhang et al. 2020b) | Multimodal-based | C3D + MLP | 70.5 | 87.7 | - | - | 79.1
AMeFu-Net (Fu et al. 2020) | ResNet-50 + ResNet-50 | 75.5 | 95.5 | 86.8 | - | 85.9
CPMT (Huang, Yang, and Sato 2022) | ResNet-50 + Mask-RCNN | 85.1 | 92.3 | 87.9 | 73.5 | 84.7
Ours: STRMM (Weighted average) |  Prototype-based & Multimodal-based |  CLIP ResNet-50 + CLIP text encoder | 84.4 | 94.6 | 90.5 | 73.6 | 85.8
Ours: TRXM (Weighted average) | 86.3 | 96.9 | 91.6 | 71.1 | 86.5
Ours: STRMM (Multi-head Attention) | 86.2 | 96.4 | 93.8 | 71.9 | 87.1
Ours: TRXM (Multi-head Attention) | 87.1 | 97.7 | 94.6 | 71.7 | 87.8
Table 1: State-of-the-art comparison on four few-shot action recognition
datasets in terms of classification accuracy. “Weighted average” and “Multi-
head Attention” denotes different choices of the MPE module. *M denotes
methods with our MORN. The bold font and underline indicate the best and
second-best results respectively.

## Experiments

Datasets. We evaluate our method on four popular datasets: HMDB51 (Kuehne et
al. 2011), UCF101 (Soomro, Zamir, and Shah 2012), Kinetics (Carreira and
Zisserman 2017) and Something-Something V2 (SSv2) (Goyal et al. 2017). HMDB51
contains 51 action categories, each containing at least 101 clips for a total
of 6,766 video clips. For HMDB51, we adopt the same protocol as (Zhang et al.
2020a) with 31/10/10 categories for train/val/test. UCF101 contains 101 action
categories, each containing at least 100 clips for a total of 13,320 video
clips. For UCF101, we also adopt the same protocol as (Zhang et al. 2020a)
with 70/10/21 categories for train/val/test. Kinetics contains 400 action
categories with 400 or more clips for each category. For Kinetics, we adopt
the same protocol as (Zhu and Yang 2018) with 64/12/24 categories for
train/val/test. SSv2 contains 220,847 videos of fine-grained actions with only
subtle differences between categories, and is regarded as a more challenging
task. For SSv2, we adopt the same protocol as (Cao et al. 2020) with 64/12/24
categories for train/val/test.

Baseline. Our work adopts STRM (Thatipelli et al. 2022) and TRX (Perrett et
al. 2021) as baseline. STRM is a feature-based method based on TRX, and TRX
applies CrossTransformer (Doersch, Gupta, and Zisserman 2020) to the few-shot
action recognition scenario. CrossTransformer combines the information of
support images and query images through an attention operation. TRX further
samples ordered sub-sequences of video frames called tuples, and thus can
capture higher-order temporal relationships.

### Comparison with State-of-the-arts

As shown in Table 1, we apply MORN on STRM (Thatipelli et al. 2022) and TRX
(Perrett et al. 2021), and comprehensively compare four datasets for the
standard 5-way 5-shot action recognition task with state-of-the-art methods.
CPMT (Huang, Yang, and Sato 2022) uses both object features and a compound
prototype-matching strategy, thus achieves the best results before us on
HMDB51, Kinetics and SSv2. Compared to CPMT, our MORN achieves the best
results of 87.1%, 97.7%, 94.6% and 73.6% on all four datasets. Also, there is
significant improvement over STRM and TRX baseline with our MORN. When using
weighted average as MPE module, STRMM achieves performance gains of 7.1% on
HMDB51, 3.8% on Kinetics and 5.5% on SSv2 over STRM, and TRXM also achieves
similar performance gains. The improvement demonstrates the effectiveness of
our multimodal prototype-enhanced strategy.

To further improve the performance of MORN, we use a multi-head attention
mechanism with 8 heads as MPE module. With a better MPE, the performance on
HMDB51, UCF101 and Kinetics can be further improved, while the performance of
STRMM on SSv2 declines on the contrary. It is probably because some label
texts in SSv2 are so similar that it is hard to capture the useful semantic
information, _e.g._ _pouring something into something_ against _pouring
something onto something_. Plus, we notice the importance of multimodal
initialization and introduce CLIP encoders. A more detailed ablation study of
the MPE module and encoders will be illustrated in Section Ablation Study. In
summary, MORN achieves the best results of 87.1% and 87.8% with multi-head
attention on average. For simplicity and fewer parameters, we apply TRXM with
weighted average as our default setting of MORN.

(a) PRIDE gains by MORN.

(b) Accuracy gains by MORN.

Figure 3: Performance gains and correlation analysis of PRIDE and accuracy on HMDB51. MORN achieves PRIDE gains in (a) and accuracy gains in (b) in each category.  Method | HMDB51 | UCF101 | Kinetics | SSv2
---|---|---|---|---
STRMC | 10.0 | 17.5 | 16.5 | 8.6
Ours: STRMM | 17.1 | 23.5 | 26.5 | 48.2
TRXC | 7.4 | 15.3 | 16.4 | 10.4
Ours: TRXM | 17.4 | 24.4 | 26.9 | 47.3
Table 2: Average PRIDE values across prototypes computed in $C_{novel}$ on
HMDB51, UCF101, Kinetics and SSv2. *C denotes methods with CLIP visual
encoder.

### Multimodal Prototype-Enhanced Analysis

In this subsection, we explore the performance of our multimodal prototype-
enhanced strategy. As mentioned earlier, we use PRIDE to evaluate the quality
of prototypes. Firstly, we compute PRIDE and verify the effectiveness of our
MORN. Furthermore, we plug PRIDE into the training stage of MORN and achieve
further performance gains.

Firstly, we randomly sample 10000 test episodes to compute accuracy and PRIDE.
As shown in Figure 3a and 3b, MORN achieves significant gains on both PRIDE
and accuracy on each category of HMDB51. Then, we conduct experiments of
average PRIDE on all four datasets to verify the effectiveness of MORN. For a
fair comparison, we replace the original ResNet-50 in TRX and STRM with a CLIP
ResNet-50. As shown in Table 2, there is significant improvement of STRMM and
TRXM on PRIDE over STRM and TRX baseline. Specifically, STRMM achieves
performance gains of 7.1% on HMDB51, 6.0% on UCF101, 10.0% on Kinetics and
39.6% on SSv2 over STRM, and TRXM also achieves similar PRIDE gains. Note that
the improvement on SSv2 is particularly significant because the label texts in
SSv2 are more complicated with more semantic information. According to the
computation process, PRIDE can accurately reflect the category separability of
prototypes. With MORN, we obtain higher PRIDE values over STRM and TRX. It
turns out that MORN can improve the quality of prototypes, which verifies the
effectiveness of MORN.

Figure 4: Overview of our proposed MORN with PRIDE loss on a 2-way 1-shot problem with 2 videos for each category in the query set. CE loss and PRIDE loss are combined through a learnable weight. Method | HMDB51 | UCF101 | Kinetics | SSv2
---|---|---|---|---
STRMM | 84.4 | 94.6 | 90.5 | 73.6
STRMMP | 85.5 | 95.6 | 91.2 | 73.9
TRXM | 86.3 | 96.9 | 91.6 | 71.1
TRXMP | 86.4 | 96.9 | 91.9 | 72.6
Table 3: *MP denotes methods with our MORN and PRIDE loss. Plugging PRIDE into
the training stage of MORN achieves further performance gains, which verifies
the rationality of PRIDE and the effectiveness of our MORN.

To further explore the function of PRIDE, we plug it into the training stage
of MORN as a loss as shown in Figure 4. PRIDE loss is implemented through
InfoNCE (Oord, Li, and Vinyals 2018) with prototypes:

$\displaystyle{\mathcal{L}}_{PRIDE}=-\log\frac{\exp(P_{i}(x)\\!\cdot\\!P^{real}_{i}/\tau)}{\sum_{j=1}^{N_{novel}}\exp(P_{i}(x)\\!\cdot\\!P^{real}_{j}/\tau)}.$
(15)

We aim at increasing the similarity of prototypes in the same category and
decreasing that of prototypes in different categories. CE loss and PRIDE loss
are combined through a learnable weight. Since PRIDE indicates the quality of
prototypes, it is supposed to assist the training process. In Table 3, STRMMP
and TRXMP achieve extra performance gains over STRMM and TRXM. The results
verify both the rationality of PRIDE and the effectiveness of our MORN.

### Ablation Study

Method | Backbone | HMDB51 | UCF101 | Kinetics | SSv2
---|---|---|---|---|---
STRM | ResNet-50 | 77.3 | 96.9 | 86.7 | 68.1
TRX | 75.6 | 96.1 | 85.9 | 64.6
STRMC | CLIP ResNet-50 | 79.3 | 94.7 | 86.0 | 62.9
TRXC | 77.9 | 95.9 | 86.7 | 64.4
STRMRB | ResNet-50 + BERT | 73.4 | 94.8 | 83.2 | 59.3
TRXRB | 72.0 | 95.2 | 83.4 | 58.8
STRMM |  CLIP ResNet-50 + CLIP text encoder | 84.4 | 94.6 | 90.5 | 73.6
TRXM | 86.3 | 96.9 | 91.6 | 71.1
Table 4: Ablation study of various encoders on HMDB51, UCF101, Kinetics and
SSv2. *RB denotes methods with ResNet-50 and BERT-large. MORN achieves
significant gains with a good multimodal initialization.

Multimodal encoders. We employ label texts as an extra modality, and an
intuitive idea is to use BERT (Devlin et al. 2019) as the text encoder.
However, the performance of STRMRB and TRXRB declines a lot as shown in Table
4. We refer it to an inadequate multimodal initialization. To verify this, we
introduce CLIP encoders with a good multimodal initialization and our
prototype-enhanced strategy. STRMM and TRXM achieve best results and
significantly outperform STRMRB and TRXRB. Furthermore, to verify the
importance of the text flow, we only replace the original ResNet-50 with a
CLIP ResNet-50 and obtain STRMC and TRXC. The performance varies on four
datasets, and is worse than that of STRMM and TRXM, demonstrating the
importance of multimodal information. In addition, we show the advantage of
prototype-enhanced strategy over feature-enhanced strategy in Figure 1.
Combining the two aspects, the necessity and effectiveness of our MORN are
verified.

Figure 5: Ablation study of varying the SE module and multimodal enhanced hyper-parameter ($\lambda$) on HMDB51. MPE module | HMDB51 | UCF101 | Kinetics | SSv2
---|---|---|---|---
Concat + MLP | 62.0 | 79.9 | 76.6 | 58.3
MLP + Concat | 72.4 | 86.3 | 80.5 | 60.6
Weighted average | 86.3 | 96.9 | 91.6 | 71.1
Head = 1 | 81.0 | 95.5 | 91.8 | 72.6
Head = 4 | 85.1 | 96.9 | 94.1 | 70.3
Head = 8 | 87.1 | 97.7 | 94.6 | 71.7
Table 5: Ablation study of the MPE module on HMDB51, UCF101, Kinetics and
SSv2. “Concat” indicates a concatenation operation and “Head” indicates the
head number in multi-head attention mechanism. The bold font and underline
indicate the best and second-best results respectively.

Design choices. As shown in Figure 5, we compare various choices of the SE
module and multimodal enhanced hyper-parameter ($\lambda$) on HMDB51. When
using a two-layer MLP with a 1024-dimensional hidden layer, we obtain a low
accuracy. It is mainly because MLP has a simple architecture and thus fails to
fully explore the semantic information of label texts. Therefore, we focus on
the multi-head attention mechanism (Vaswani et al. 2017). As the head number
increases, the fitting ability of the SE module improves as well. However, the
accuracy fluctuates with different head numbers and $\lambda$ choices. We need
to balance the trade-off between head numbers and $\lambda$ choices. According
to our results, we use a multi-head attention mechanism with 4 heads as the SE
module and set $\lambda$ as 0.5 in our further experiments.

For the MPE module, applying a multi-head attention mechanism can further
improve the performance of MORN as shown in Table 5. It turns out that the
multimodal information can further enhance the prototypes with a better
multimodal combination strategy. MORN achieves the best results with 8 heads
on HMDB51, UCF101 and Kinetics, while with 1 head on SSv2. Although weighted
average is relatively simple with no parameters for training, our MORN with
weighted average can also achieve the second-best results of 86.3% on HMDB51
and 96.9% on UCF101, which has already outperformed those of prior methods.
For simplicity and fewer parameters for training, we use weighted average as
our default MPE module.

| SE module
---
(exist or not)
| CLIP text encoder
---
(freeze or not)
HMDB51 | UCF101 | Kinetics | SSv2
✘ | ✘ | 84.5 | 96.1 | 88.8 | 72.5
✘ | ✔ | 83.9 | 96.4 | 90.6 | 72.8
✔ | ✘ | 76.7 | 93.2 | 87.2 | 62.1
✔ | ✔ | 86.3 | 96.9 | 91.6 | 71.1
Table 6: Ablation study of the SE module and CLIP text encoder on HMDB51,
UCF101, Kinetics and SSv2.

SE module and CLIP text encoder. In the text flow, we first introduce a frozen
CLIP text encoder. Then, a semantic-enhanced (SE) module is used to enhance
the semantic information. Firstly on SSv2, MORN with a frozen CLIP text
encoder is enough to achieve the best result of 72.8% as shown in Table 6.
When coming to HMDB51, UCF101 and Kinetics, an SE module further achieves
performance gains, and MORN with the SE module and a frozen CLIP text encoder
achieves the best results. On the one hand, the function of the SE module is
similar to that of the enrichment module in (Thatipelli et al. 2022) to obtain
reliable semantic information. On the other hand, a frozen language encoder is
proven to be effective in several works (Tsimpoukelli et al. 2021; Rao et al.
2022) and our results further prove it. When we remove the SE module, the
results are lower than MORN whether we freeze the CLIP text encoder or not.
Besides, using the SE module and fine-tuning the CLIP text encoder leads to
the lowest results on all three datasets. It is mainly because the overfitting
problem becomes more serious as parameters increase. When coming to SSv2, MORN
with a frozen CLIP text encoder but without the SE module achieves the best
result. In general, MORN achieves significant performance gains with our
design.

## Conclusion

We propose a novel Multimodal Prototype-Enhanced Network (MORN) for few-shot
action recognition. Our MORN uses the semantic information of label texts as
multimodal information to compute more representative prototypes, and achieves
state-of-the-art results. We also define Prototype Similarity Difference
(PRIDE) to evaluate the quality of prototypes. By conducting experiments on
PRIDE, we verify our improvement on prototypes. Besides, plugging PRIDE into
the training stage of MORN can achieve further performance gains, verifying
the rationality of PRIDE and the importance of prototypes. In summary, we’d
like to shed light on a view: _multimodal prototypes with high quality are
extremely important in few-shot action recognition_.

## References

  * Andrychowicz et al. (2016) Andrychowicz, M.; Denil, M.; Gomez, S.; Hoffman, M. W.; Pfau, D.; Schaul, T.; Shillingford, B.; and De Freitas, N. 2016.  Learning to learn by gradient descent by gradient descent.  In _NIPS_ , volume 29.
  * Antoniou, Edwards, and Storkey (2018) Antoniou, A.; Edwards, H.; and Storkey, A. 2018.  How to train your MAML.  In _ICLR_.
  * Bishay, Zoumpourlis, and Patras (2019) Bishay, M.; Zoumpourlis, G.; and Patras, I. 2019.  Tarn: Temporal attentive relation network for few-shot and zero-shot action recognition.  In _BMVC_ , 154.
  * Cao et al. (2020) Cao, K.; Ji, J.; Cao, Z.; Chang, C.-Y.; and Niebles, J. C. 2020.  Few-shot video classification via temporal alignment.  In _CVPR_ , 10618–10627.
  * Carreira and Zisserman (2017) Carreira, J.; and Zisserman, A. 2017.  Quo vadis, action recognition? a new model and the kinetics dataset.  In _CVPR_ , 6299–6308.
  * Chen et al. (2018) Chen, Z.; Fu, Y.; Zhang, Y.; Jiang, Y.-G.; Xue, X.; and Sigal, L. 2018.  Semantic feature augmentation in few-shot learning.  _arXiv preprint arXiv:1804.05298_ , 86(89): 2.
  * Chen et al. (2019) Chen, Z.; Fu, Y.; Zhang, Y.; Jiang, Y.-G.; Xue, X.; and Sigal, L. 2019.  Multi-level semantic feature augmentation for one-shot learning.  _IEEE Transactions on Image Processing_ , 28(9): 4594–4605.
  * Devlin et al. (2019) Devlin, J.; Chang, M.-W.; Lee, K.; and Toutanova, K. 2019.  Bert: Pre-training of deep bidirectional transformers for language understanding.  In _NAACL-HLT_ , volume 1, 4171–4186.
  * Doersch, Gupta, and Zisserman (2020) Doersch, C.; Gupta, A.; and Zisserman, A. 2020.  Crosstransformers: spatially-aware few-shot transfer.  In _NIPS_ , volume 33, 21981–21993.
  * Finn, Abbeel, and Levine (2017) Finn, C.; Abbeel, P.; and Levine, S. 2017.  Model-agnostic meta-learning for fast adaptation of deep networks.  In _ICML_ , 1126–1135.
  * Fu et al. (2020) Fu, Y.; Zhang, L.; Wang, J.; Fu, Y.; and Jiang, Y.-G. 2020.  Depth guided adaptive meta-fusion network for few-shot video recognition.  In _ACMMM_ , 1142–1151.
  * Garcia and Bruna (2018) Garcia, V.; and Bruna, J. 2018.  Few-shot learning with graph neural networks.  In _ICLR_.
  * Goodfellow et al. (2020) Goodfellow, I.; Pouget-Abadie, J.; Mirza, M.; Xu, B.; Warde-Farley, D.; Ozair, S.; Courville, A.; and Bengio, Y. 2020.  Generative adversarial networks.  _Communications of the ACM_ , 63(11): 139–144.
  * Goyal et al. (2017) Goyal, R.; Ebrahimi Kahou, S.; Michalski, V.; Materzynska, J.; Westphal, S.; Kim, H.; Haenel, V.; Fruend, I.; Yianilos, P.; Mueller-Freitag, M.; et al. 2017.  The” something something” video database for learning and evaluating visual common sense.  In _ICCV_ , 5842–5850.
  * Gui et al. (2018) Gui, L.; Wang, Y.; Ramanan, D.; and Moura, J. M. 2018.  Few-shot human motion prediction via meta-learning.  In _ECCV_ , 432–450.
  * Hou et al. (2019) Hou, R.; Chang, H.; Ma, B.; Shan, S.; and Chen, X. 2019.  Cross attention network for few-shot classification.  In _NIPS_ , volume 32.
  * Huang et al. (2021) Huang, Y.; Du, C.; Xue, Z.; Chen, X.; Zhao, H.; and Huang, L. 2021.  What makes multi-modal learning better than single (provably).  In _NIPS_ , volume 34, 10944–10956.
  * Huang, Yang, and Sato (2022) Huang, Y.; Yang, L.; and Sato, Y. 2022.  Compound Prototype Matching for Few-shot Action Recognition.  In _ECCV_.
  * Jamal and Qi (2019) Jamal, M. A.; and Qi, G.-J. 2019.  Task agnostic meta-learning for few-shot learning.  In _CVPR_ , 11719–11727.
  * Kim et al. (2019) Kim, J.; Kim, T.; Kim, S.; and Yoo, C. D. 2019.  Edge-labeling graph neural network for few-shot learning.  In _CVPR_ , 11–20.
  * Kuehne et al. (2011) Kuehne, H.; Jhuang, H.; Garrote, E.; Poggio, T.; and Serre, T. 2011.  HMDB: a large video database for human motion recognition.  In _ICCV_ , 2556–2563.
  * Kumar Dwivedi et al. (2019) Kumar Dwivedi, S.; Gupta, V.; Mitra, R.; Ahmed, S.; and Jain, A. 2019.  Protogan: Towards few shot learning for action recognition.  In _CVPRW_.
  * Lee et al. (2019) Lee, K.; Maji, S.; Ravichandran, A.; and Soatto, S. 2019.  Meta-learning with differentiable convex optimization.  In _CVPR_ , 10657–10665.
  * Li et al. (2019) Li, H.; Eigen, D.; Dodge, S.; Zeiler, M.; and Wang, X. 2019.  Finding task-relevant features for few-shot learning by category traversal.  In _CVPR_ , 1–10.
  * Liu et al. (2021) Liu, R.; Zhang, H.; Pirsiavash, H.; and Liu, X. 2021.  MASTAF: A Spatio-Temporal Attention Fusion Network for Few-shot Video Classification.  _arXiv preprint arXiv:2112.04585_.
  * Liu, Schiele, and Sun (2020) Liu, Y.; Schiele, B.; and Sun, Q. 2020.  An ensemble of epoch-wise empirical bayes for few-shot learning.  In _ECCV_ , 404–421.
  * Oord, Li, and Vinyals (2018) Oord, A. v. d.; Li, Y.; and Vinyals, O. 2018.  Representation learning with contrastive predictive coding.  _arXiv preprint arXiv:1807.03748_.
  * Pahde et al. (2019) Pahde, F.; Ostapenko, O.; Hnichen, P. J.; Klein, T.; and Nabi, M. 2019.  Self-paced adversarial training for multimodal few-shot learning.  In _WACV_ , 218–226.
  * Pahde et al. (2021) Pahde, F.; Puscas, M.; Klein, T.; and Nabi, M. 2021.  Multimodal prototypical networks for few-shot learning.  In _WACV_ , 2644–2653.
  * Perez and Wang (2017) Perez, L.; and Wang, J. 2017.  The effectiveness of data augmentation in image classification using deep learning.  _arXiv preprint arXiv:1712.04621_.
  * Perrett et al. (2021) Perrett, T.; Masullo, A.; Burghardt, T.; Mirmehdi, M.; and Damen, D. 2021.  Temporal-relational crosstransformers for few-shot action recognition.  In _CVPR_ , 475–484.
  * Radford et al. (2021) Radford, A.; Kim, J. W.; Hallacy, C.; Ramesh, A.; Goh, G.; Agarwal, S.; Sastry, G.; Askell, A.; Mishkin, P.; Clark, J.; et al. 2021.  Learning transferable visual models from natural language supervision.  In _ICML_ , 8748–8763.
  * Rao et al. (2022) Rao, Y.; Zhao, W.; Chen, G.; Tang, Y.; Zhu, Z.; Huang, G.; Zhou, J.; and Lu, J. 2022.  Denseclip: Language-guided dense prediction with context-aware prompting.  In _CVPR_ , 18082–18091.
  * Ratner et al. (2017) Ratner, A. J.; Ehrenberg, H.; Hussain, Z.; Dunnmon, J.; and Ré, C. 2017.  Learning to compose domain-specific transformations for data augmentation.  In _NIPS_ , volume 30.
  * Ravi and Larochelle (2017) Ravi, S.; and Larochelle, H. 2017.  Optimization as a model for few-shot learning.  In _ICLR_.
  * Snell, Swersky, and Zemel (2017) Snell, J.; Swersky, K.; and Zemel, R. 2017.  Prototypical networks for few-shot learning.  In _NIPS_ , volume 30.
  * Soomro, Zamir, and Shah (2012) Soomro, K.; Zamir, A. R.; and Shah, M. 2012.  UCF101: A dataset of 101 human actions classes from videos in the wild.  _arXiv preprint arXiv:1212.0402_.
  * Sun et al. (2019) Sun, Q.; Liu, Y.; Chua, T.-S.; and Schiele, B. 2019.  Meta-transfer learning for few-shot learning.  In _CVPR_ , 403–412.
  * Sung et al. (2018) Sung, F.; Yang, Y.; Zhang, L.; Xiang, T.; Torr, P. H.; and Hospedales, T. M. 2018.  Learning to compare: Relation network for few-shot learning.  In _CVPR_ , 1199–1208.
  * Thatipelli et al. (2022) Thatipelli, A.; Narayan, S.; Khan, S.; Anwer, R. M.; Khan, F. S.; and Ghanem, B. 2022.  Spatio-temporal relation modeling for few-shot action recognition.  In _CVPR_ , 19958–19967.
  * Tsimpoukelli et al. (2021) Tsimpoukelli, M.; Menick, J. L.; Cabi, S.; Eslami, S.; Vinyals, O.; and Hill, F. 2021.  Multimodal few-shot learning with frozen language models.  In _NIPS_ , volume 34, 200–212.
  * Vaswani et al. (2017) Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A. N.; Kaiser, Ł.; and Polosukhin, I. 2017.  Attention is all you need.  In _NIPS_ , volume 30.
  * Vinyals et al. (2016) Vinyals, O.; Blundell, C.; Lillicrap, T.; Wierstra, D.; et al. 2016.  Matching networks for one shot learning.  In _NIPS_ , volume 29.
  * Wang et al. (2016) Wang, L.; Xiong, Y.; Wang, Z.; Qiao, Y.; Lin, D.; Tang, X.; and Gool, L. V. 2016.  Temporal segment networks: Towards good practices for deep action recognition.  In _ECCV_ , 20–36.
  * Wang, Xing, and Liu (2021) Wang, M.; Xing, J.; and Liu, Y. 2021.  Actionclip: A new paradigm for video action recognition.  _arXiv preprint arXiv:2109.08472_.
  * Wang et al. (2022) Wang, X.; Zhang, S.; Qing, Z.; Tang, M.; Zuo, Z.; Gao, C.; Jin, R.; and Sang, N. 2022.  Hybrid Relation Guided Set Matching for Few-shot Action Recognition.  In _CVPR_ , 19948–19957.
  * Wu et al. (2022) Wu, J.; Zhang, T.; Zhang, Z.; Wu, F.; and Zhang, Y. 2022.  Motion-Modulated Temporal Fragment Alignment Network for Few-Shot Action Recognition.  In _CVPR_ , 9151–9160.
  * Yang et al. (2020) Yang, L.; Li, L.; Zhang, Z.; Zhou, X.; Zhou, E.; and Liu, Y. 2020.  Dpgn: Distribution propagation graph network for few-shot learning.  In _CVPR_ , 13390–13399.
  * Ye et al. (2020) Ye, H.-J.; Hu, H.; Zhan, D.-C.; and Sha, F. 2020.  Few-shot learning via embedding adaptation with set-to-set functions.  In _CVPR_ , 8808–8817.
  * Yoon, Seo, and Moon (2019) Yoon, S. W.; Seo, J.; and Moon, J. 2019.  Tapnet: Neural network augmented with task-adaptive projection for few-shot learning.  In _ICML_ , 7115–7123.
  * Zhang et al. (2021) Zhang, B.; Li, X.; Ye, Y.; Huang, Z.; and Zhang, L. 2021.  Prototype completion with primitive knowledge for few-shot learning.  In _CVPR_ , 3754–3762.
  * Zhang et al. (2020a) Zhang, H.; Zhang, L.; Qi, X.; Li, H.; Torr, P. H.; and Koniusz, P. 2020a.  Few-shot action recognition with permutation-invariant attention.  In _ECCV_ , 525–542.
  * Zhang et al. (2020b) Zhang, L.; Chang, X.; Liu, J.; Luo, M.; Prakash, M.; and Hauptmann, A. G. 2020b.  Few-shot activity recognition with cross-modal memory network.  _PR_ , 108: 107348.
  * Zhang, Zhou, and He (2021) Zhang, S.; Zhou, J.; and He, X. 2021.  Learning implicit temporal alignment for few-shot video classification.  In _IJCAI_.
  * Zhu and Yang (2018) Zhu, L.; and Yang, Y. 2018.  Compound memory networks for few-shot video classification.  In _ECCV_ , 751–766.
  * Zhu et al. (2021) Zhu, X.; Toisoul, A.; Perez-Rua, J.-M.; Zhang, L.; Martinez, B.; and Xiang, T. 2021.  Few-shot action recognition with prototype-centered attentive learning.  In _BMVC_.


# Forecasting landslides using community detection on geophysical satellite
data

V. Desai  F. Fazelpour  Physics Department, North Carolina State University,
Raleigh, NC, USA  A. Handwerger  Joint Institute for Regional Earth System
Science and Engineering, University of California Los Angeles, Los Angeles,
CA, USA and Jet Propulsion Laboratory, California Institute of Technology,
Pasadena, CA, USA  Karen E. Daniels  Physics Department, North Carolina State
University, Raleigh, NC, USA

###### Abstract

As a result of extreme weather conditions, such as heavy precipitation,
natural hillslopes can fail dramatically; these slope failures can occur on a
dry day due to time lags between rainfall and pore-water pressure change at
depth, or even after days to years of slow-motion. While the pre-failure
deformation is sometimes apparent in retrospect, it remains challenging to
predict the sudden transition from gradual deformation (creep) to runaway
failure. We use a network science method – multilayer modularity optimization
– to investigate the spatiotemporal patterns of deformation in a region near
the 2017 Mud Creek, California landslide. We transform satellite radar data
from the study site into a spatially-embedded network in which the nodes are
patches of ground and the edges connect the nearest neighbors, with a series
of layers representing consecutive transits of the satellite. Each edge is
weighted by the product of the local slope (susceptibility to failure)
measured from a digital elevation model and ground surface deformation
(current rheological state) from interferometric synthetic aperture radar
(InSAR). We use multilayer modularity optimization to identify strongly-
connected clusters of nodes (communities) and are able to identify both the
location of Mud Creek and nearby creeping landslides which have not yet
failed. We develop a metric, community persistence, to quantify patterns of
ground deformation leading up to failure, and find that this metric increases
from a baseline value in the weeks leading up to Mud Creek’s failure. These
methods promise as a technique for highlighting regions at risk of
catastrophic failure.

landslides, networks, communities

##  I Introduction

Slow-moving landslides creep downhill at rates from millimeters to several
meters a year [1], a process that sometimes proceeds hundred of years [2, 3].
Yet, a slow-moving landslide can reach runaway acceleration, and in many cases
this is likely to be triggered by a period of drought followed by heavy
precipitation, as happened in the 2017 Mud Creek landslide located on the Big
Sur Coast, CA. The processes which influence these transitions are myriad, and
additionally include topographic slope, hydrological and mechanical
properties, and natural or man-made disturbances. Advancement in remote
sensing techniques allows for the detection of ground deformation at
spatiotemporal resolutions that were not achievable a decade ago [4, 5].
Previous work in landslide forecasting has focused on either (1) estimating
the location and shape of a landslide using spatial analysis of slope
stability [6] or (2) the time of failure using temporal analysis of ground
deformation [7]. There has been recent progress on forecasting both location
and time [8, 5]. Linking kinematic data and the underlying micro-mechanics of
granular failure together has shown to be effective in forecasting the
location and time of granular failure [5, 9, 10].

Interferometric processing of synthetic aperture radar (InSAR) makes studies
of landslide and other ground deformation possible at high spatial and
temporal resolution. Ground deformation over large surface areas of the Earth
is measured with centimeter to millimeter accuracy using InSAR [11] by
exploiting the phase difference between two SAR snapshots acquired over the
same area at two different times. This quantifies the amount of ground
deformation during that time interval, projected along the radar line-of-sight
(LOS) direction [12, 13]. InSAR techniques have been applied in the study of
landslides and are starting to be incorporated into early warning systems
(EWS)[14, 15, 16, 16, 13, 6].

EWS have incorporated these advancements in radar monitoring data to monitor
areas of known risk and identify regions heavily affected by disaster [17],
but there are still significant drawbacks and limitations to the forecasting
of landslides and other hazards [9]. One such drawback is the use of
thresholds that are defined to predict the onset of failure in landslides
[18], and recent studies have shown that these thresholds are site-specific
and subjective [9]. A common practice is to only consider the region that is
accelerating rapidly [18], but important information about the interactions
occurring between the region and the surrounding area then go undetected [5,
19, 20].

The transition of a creeping landslide to a catastrophic failure is complex
and the system’s behavior emerges from dynamics that are occurring beneath the
surface [21]. The landslide system is modeled as a collection of patches that
are interacting with each other in such a way that the emerging behaviour is
not immediately predictable [22], which is why it is crucial to not only look
at the immediate region, but also the surrounding area. Thus, one of the
biggest challenges to EWS in regards to landslides is harnessing the full
potential of these data sets using the analysis and interpretation of the
underlying micro-mechanics in the time before granular failure [5].

Recent studies have shown that granular failure does not occur spontaneously,
but instead there is a transitional period between stable deformation and
catastrophic failure, which is indicated by a distinct dynamical pattern,
influenced by kinematics, that is formed [8, 9, 23]. Consider a region that
consists of multiple closely connected patches, or clusters. As the region
approaches failure, the spatiotemporal dynamics have been characterized into
two pieces: (1) similarity within a cluster increases as they begin to move
together and (2) similarity between clusters decreases [8]. The transition to
imminent failure is guided by the spatiotemporal persistence of community $C$,
followed by a sharp increase in the motion, or velocity, of $C$ in respect to
the rest of the system, or area [8].

Our goal is to use kinematic data retrieved from remote sensing techniques to
find patches of ground, defined as communities, that are likely to fail and
their impending time of failure. To develop a holistic approach that
integrates the dynamics of ground deformation, along with the physics of slope
stability, we use network science techniques of micro-scale granular failure
(grains on the order of 1 mm to 1 cm) [24, 25, 26] and apply it on a
macroscale system (patches of area on the order of 7x7 to 20x20 m) [8]. A
network framework gives us the ability to infer the structure and underlying
dynamics of a creeping landslide using information such as displacement that
can be retrieved using remote sensing techniques [22].

###  I.1 Study Site: Mud Creek vicinity

Within the California Coast Ranges, landslides are abundant due to
mechanically weak rocks, active uplift, and high seasonal precipitation [27,
28, 29, 30]. More than 650 active slow-moving landslides have been identified
and mapped within the region, but few of them have failed catastrophically
[30, 29]. On a dry day (May 20, 2017) following a long period of heavy
rainfall that ended on April 17, 2017, the Mud Creek landslide, which had been
slowly moving for decades, failed catastrophically and destroyed part of
California State Highway 1 (CA-1) [27, 31].

Figure 1: The root mean square (RMS) of the line of sight (LOS) velocity
(black circles) for the entire study area leading up to the day of failure
($T=0$); data from Handwerger  _et al._ [27]. Blue bars are the total weekly
precipitation, from PRISM [32]. The wet seasons correspond to roughly December
to May each year.

The Central Coast Ranges, which contain Mud Creek, receive about $\sim
1~{}\mathrm{m}$ of rain per year, of which 80% falls between October and May
[27, 33]. The precipitation in this region varies significantly from year to
year due to changes in the frequency and strength of atmospheric rivers;
climate models predict that California will experience a dramatic increase in
frequency of shifts from extreme dry seasons to extreme wet seasons over the
next couple of decades [33]. Since landslide dynamics can be driven by
seasonal changes, as seen in Fig. 1, the predicted increase in changes from
dry to wet seasons is a cause for concern [27, 33, 34].

Changes in the shear strength of a hillslope are dependent on the water-
saturation level of the material [35, 36, 37]. If the material is not fully
saturated, then capillary effects increase shear strength and help stabilize a
hillslope [36, 38, 35]. On the other hand, if the material is almost or fully
saturated, then an increase in the pore-water pressure leads to a reduction in
the shear strength of the material [36, 37], and may possibly lead to
instability within the hillslope [38, 37, 35, 21]. Note that the pore-water
pressure varies strongly with depth, as rain permeates the hillslope over
time. In addition, the parameters that influence slope stability originate
from critical-state soil mechanics with a Mohr-Coulomb failure criterion,
which assumes that failure is controlled by maximum shear strength [39, 40].
Saturated material, corresponding to high pore-water pressure, plus steep
slopes increases the susceptibility of catastrophic failure in creeping
landslides [21].

In this paper, we examine the area in and around Mud Creek
($3~{}\mathrm{km}\times 3~{}\mathrm{km}$), shown in Fig. 1, to explore the
dynamics within Mud Creek and compare it to the rest of the area, which
consists of 2 other creeping landslides that did not fail. We are interested
in analyzing, and with the eventual aim of forecasting, the transition of a
slow-moving landslide to sudden catastrophic failure.

###  I.2 Comparing Mud Creek to Other Regions

Since there is only one creeping landslide that catastrophically failed within
the study area, we create a set of controls to test for sensitivity and
specificity. For this, we additionally selected three regions within the study
area to analyze alongside Mud Creek (see Fig. 2). Of these, Region 2 is a
creeping landslide that was not identified in Handwerger  _et al._ [27], but
instead was recently observed with airborne InSAR data (see Fig. 2); and
Regions 1 and 3 are selected because they are steep areas that are not
creeping. For each of these regions, the LOS velocity (from Sentinel-1 InSAR
data, described in more detail in §II.2), plotted in Fig. 2, shows that there
is not a significant difference between them and Mud Creek for the majority of
the time period. In Fig. 2(b), the velocities for the four regions are mapped
at two different time steps ($-176$ and $-32$ days) to show that the regions
are moving similarly to each other at $-176$ days and later at $-32$ days Mud
Creek is moving significantly faster than the other three regions.

Figure 2: (a) The root mean square (RMS) of the line of sight (LOS) velocity,
for each of the 4 regions. The wet seasons shown in Fig. 1 are shaded in blue.
Inset: bare earth hillshade map from lidar showing the four regions within the
study area. (b) Maps of the LOS velocity for each of the 4 regions at two
times: $-32$ and $-176$ days before failure. Data is from [27].

###  I.3 Networks

It is difficult to quantify the structural changes happening beneath the
surface, and thereby forecasting when and where a landslide is likely to
occur. Thus, it is desirable to turn to reduced descriptions of the system.
One such technique, network science, has been successful at quantifying and
understanding the spatiotemporal dynamics of complex systems without the
inclusion of a detailed model of the internal interactions. Previous studies,
such as [41, 42], have shown that a network clustering technique known as
community detection (reviewed in Fortunato [43]) can be used to extract hidden
relations among nodes in static single-layer networks. Recent studies, such as
the analysis of the evolution of the force networks within granular materials
by Papadopoulos  _et al._ [44], have extended that technique to be used in
multilayer networks to analyze spatially-embedded networks that are more
similar to earth materials.

Network science provides a way to characterize this complex system for
consistent patterns of causality via community-detection techniques. For each
snapshot of ground deformation, there is a correlating static network layer
that is connected together into a multilayer network, where a simplistic
version of a multilayer network is shown in Fig. 3. We analyze the transition
of a slow-moving landslide, as represented in a multilayer network, to sudden
catastrophic failure using community persistence, a measure we have developed
to quantify the consistency of community structures. Using this measure, we
are able to narrow down the period in which Mud Creek could be said to be at
risk of failure, compared to what would be possible using InSAR alone.

##  II Data Assimilation

We use two types of data to create our multilayer network: digital elevation
models (DEM) for spatial information and slope and displacement snapshots from
InSAR to estimate the current rheological state. This section details the
process by which we define our multilayer network from the DEM and InSAR data;
the edges within each time step (layer $\ell$) are weighted by both the local
slope (steeper slopes are more susceptible to failure) and the local creep
velocity (faster speeds indicate a current soil state with more fluid
rheology). Local slope is held fixed over the duration of the analysis.

Figure 3: Schematic of the construction and analysis of our multilayer
network. We sample the digital elevation model (DEM) and a series of velocity
maps on a disordered mesh to create a multilayer network weighted by both.
Using this network, we selected the spatial coupling parameter $\gamma$ and
the temporal coupling parameter $\omega$ to detect communities of interest.

###  II.1 Digital Elevation Model

We measure the elevation at all positions in the study site using a Digital
Elevation Model (DEM) at 1 m resolution, downloaded from OpenTopography (May
17 2021), chosen because it penetrates vegetation and measures the true ground
surface. This DEM is from the USGS, 3D Elevation Program: ARRA-CA
CentralCoast-Z3 2010 [45]. We define the elevation as a field $h({\vec{r}})$
with position $\vec{r}=(x,y)$ given in UTM coordinates. From this, we
calculate the slope as $s=\frac{|\Delta h|}{|\Delta\vec{r}|}$, which has a
flat distribution for $0^{\circ}\leq s\leq 18^{\circ}$, after which steeper
slopes are rarer, up to a maximum of $32^{\circ}$ (such as within Mud Creek).
The mean slope for the other three regions are: $25^{\circ}$, $24^{\circ}$,
$25^{\circ}$.

###  II.2 InSAR

We utilize an existing dataset from Handwerger  _et al._ [27], comprising
InSAR snapshots from the Copernicus Sentinel-1 A/B satellites (see Handwerger
_et al._ [27] for InSAR processing details). The resulting series of snapshots
provides a total of 63 layers with $10~{}\mathrm{m}\times 12~{}\mathrm{m}$
resolution estimating ground deformation in a $3~{}\mathrm{km}\times
3~{}\mathrm{km}$ region in the vicinity of Mud Creek area from February 2015
to May 2017. Each InSAR snapshot provides LOS displacements, where the
direction of motion is either towards or away from the satellite. Using the
InSAR data, we have a layer $\ell$ for each snapshot within the multilayer
network. These layers span 6, 12, or 24 days due to either missed orbits
(repeat cycle is 6 days) or low coherence.

For each layer $\ell$ (corresponding to an InSAR snapshot), we defined
displacement as a field $U(x,y,\ell)$. Cumulative displacement is the
displacement of the surface from the first reference frame ($\ell=0$), for
which $U(x,y,0)=0$ everywhere. Subsequent layers contain the cumulative
displacement that has happened during layers $0$ to $\ell$. To quantify
deformation, we calculate the incremental displacement $u$ between pairs of
adjacent snapshots as $u(x,y,\ell)=U(x,y,\ell)-U(x,y,\ell-1)$. Since the
acquisition interval for the satellites varied from 6, 12, and 24 days, we
converted incremental displacement into velocity,
$v(x,y,\ell)=\frac{u(x,y,\ell)-u(x,y,\ell-1)}{\Delta{t}}$, where $\Delta{t}$
is time between any 2 consecutive snapshots. From the original dataset, we
additionally apply a $3\times 3$ cell matrix of a Gaussian smoothing kernel to
reduce the noise within each layer.

###  II.3 Defining the multilayer network

Our analysis takes place on a multilayer network, where each layer $\ell$
represents a time in space (corresponding to an InSAR snapshot). Each layer is
represented as a spatially embedded network consisting of nodes and edges.
Each node represents a patch of area, and each edge is the connection between
patches of areas that are neighbors. This network consists of the same nodes
and edge connections for all layers within the system, which we define as a
mesh.

To create the spatially embedded network, we sampled the DEM at $n\sim 5000$
nodes for the study area shown in Fig. 2, selected to be large enough to be
computationally efficient but small enough to provide spatial resolution
within ridges and valleys. We considered uniform, random, and uniform random
grids to create a mesh that would not introduce artefacts into the results or
lead to mesh-specific results. Using Poisson disc sampling [46], a type of
uniform random grid, we randomly selected node locations spaced an average of
20 m apart, without any preferred orientation, as shown in Fig. 3. Pairs of
adjacent nodes are connected by edges using Delaunay triangulation (selects
the closest nodes to each node), defining our binary adjacency matrix
$B_{ij}$. We created an ensemble of 10 such meshes that are uniformly sampled
across the area while also allowing us to test for sensitivity of the results
to the mesh that was used.

Information about the current state of rheology and the susceptibility of
failure was incorporated into the network using data from the DEM and InSAR to
weight the edges. For each edge ${ij}$, we calculated the slope
$s_{ij}=\frac{|h_{i}-h_{j}|}{|\vec{r}_{i}-\vec{r}_{j}|}$, where $h_{i}$ is the
average elevation (given by the DEM) of the patch of area represented by node
$i$ and $\vec{r}_{i}$ is the central coordinates of the patch of area, or node
$i$. For an InSAR snapshot (layer $\ell$), each node $i$ has some velocity
$v_{i\ell}$ that represents the backward-difference velocity, selected since
it is convenient for use in real-time monitoring. We then calculated velocity
for each edge ${ij}$ as the absolute average velocity defined as
$|{\bar{v}}_{ij\ell}|=|\frac{v_{i\ell}+v_{j\ell}}{2}|$. The edges are weighted
as $s_{ij}|{\bar{v}}_{ij\ell}|$ for each layer $\ell$, where $s_{ij}$ is
constant for all layers, but $|{\bar{v}}_{ij\ell}|$ changes for each layer
$\ell$. We define our adjacency matrix as

$A_{ij\ell}=s_{ij}|{\bar{v}}_{ij\ell}|B_{ij}$ (1)

for a single layer $\ell$ as a snapshot in time, with node indices ${i,j}$
specifying spatial location.

##  III Network Analysis

A community within a network is a collection of nodes that are more strongly
related to each other than to other nodes (see Fig. 3) [47, 43, 48]. We seek
to identify whether communities are present within the Mud Creek dataset, and
evaluate whether there are particularly unstable regions or periods within the
system. For example, clusters of nodes that are connected by edges, and moving
at relatively higher speeds and/or on steeper slopes, could be identified as a
community. The community structure of a multilayer network shows the evolution
of the communities over time, where communities are formed from partitioning
the graph based on a defining characteristic [47].

###  III.1 Community detection

In this study, we analyze our multilayer network (Eq. 1) using a method based
on optimizing the modularity $Q$; the method partitions nodes into communities
(each community is thereby given an arbitrary ID by the algorithm to identify
it) where nodes are more strongly connected within the community than to the
rest of the network [49, 50]. We optimize the modularity of the community
structures using the NetWiki GenLouvain implementation for multilayer networks
[51, 48]. The multilayer modularity is calculated as

$Q=\frac{1}{2\mu}\sum_{ij\ell m}[(A_{ij\ell}-\gamma P_{ij\ell})\delta_{\ell
m}+\omega_{j\ell m}\delta_{ij}]\delta(c_{i\ell},c_{jm}),$ (2)

where $\mu$ is the sum of all edge weights in the network, $c_{i\ell}$ is the
community of node $i$, $c_{j\ell}$ is the community of node $j$, $\gamma$ is
the structural resolution parameter, $\omega_{j\ell m}$ is the interlayer
coupling, and $P_{ij\ell}$ is the expected edge weight between nodes $i$ and
$j$ under a specified null model in layer $\ell$ [48]. Note that using a
global null model disregards the temporal fluctuations and does not improve
the results of the community detection.

To optimize the modularity, the nodes are partitioned into communities in such
a way that the total connection strength — the cumulative weight of connected
edges — within a community is much greater than what would be expected from a
null model. For any network, the choice of null model provides the expected
weight of a randomly-selected pair of nodes, and affects the interpretation of
the network [52]. Because our system is a spatially-embedded network (nodes
only interact with their direct neighbors), the work of Bassett et al. [53,
44] guides our choice of the geographical null model rather than the more
common Newman-Girvan null model [49, 50] for which all edges are equally
connected to each other. The geographical null model is defined as

$P_{ij\ell}=\rho_{\ell}B_{ij},$ (3)

where $\rho_{\ell}=\langle s_{ij}|v_{ij\ell}|\rangle_{ij}$ is the mean edge
weight of network layer $\ell$ (averaged over all ${ij}$ edges) and $B_{ij}$
is the binary adjacency matrix, which is constant through all layers and
encodes the geophysical proximity.

There are two resolution parameters used to calculate modularity $Q$, both of
which are selected to control the total number of communities detected, as
seen in Fig. 3. The structural resolution parameter $\gamma$ sets the weight
of the geographic null model, and allows the user to adjust the size
(conversely, the total number) of communities. Higher values of $\gamma$ leads
to smaller communities and vice versa. The corresponding temporal resolution
parameter is contained within the interlayer coupling $\omega_{j\ell m}$,
which is the weight of the edge coupling node $j$ to itself in layers
$(\ell,m)$. Here, we take $\omega_{j\ell m}=\omega$ (constant), for
simplicity. We want to consider $\omega$ values that take into account the
strain-loading history. Higher values of $\omega$ more strongly favor the
continuity of communities across the series of layers, while for $\omega=0$,
each layer is considered independently (no communities span the layers), and
thus no consideration of the strain-loading history.

###  III.2 Selecting the coupling parameters ($\gamma$, $\omega$)

We wish to select resolution parameters that highlight communities within the
network which (1) have stronger weights than average and (2) continue that
behavior over some time interval. To satisfy the former, we start from the
standard choice of $\gamma=1$. This corresponds to favoring the clustering of
nodes which have stronger-than-average weights (larger than $\langle
P_{\ell}\rangle$). We will revisit this choice in §IV.2. In contrast, the
interlayer coupling $\omega$ is normally defined to be in the range of 0 and 1
[44]. Thus to guide our choice of $\omega$, we explore two measures:
flexibility and stationarity [44, 54]. We want to find a value of $\omega$
which is large enough that important communities persist over time, but not so
large that we fail to detect changes in the temporal state of the study area.

We quantify the local properties of the nodes using a measure known as
flexibility [54]. It is the measure of the number of layers, $g_{i}$, that
each node $i$ changes community allegiance over the total possible layers
($L-1$). The flexibility of the multilayer network as a whole for all nodes
$N$ is defined as:

$\Xi=\frac{1}{N}\sum_{i}\frac{g_{i}}{L-1}.$ (4)

To capture the global properties of the communities found within the network,
we measure the consistency of nodes in communities over time using a measure
known as stationarity [55, 41]. As given in [44], the stationarity of a
multilayer network is $\zeta$ for all nodes $N$:

$\zeta=\frac{1}{C_{\ell}}\sum_{c}\frac{\sum_{\ell_{i}}^{\ell_{f}-1}J(c_{\ell},c_{\ell+1})}{\ell_{f}-\ell_{i}},$
(5)

where $C_{\ell}$ is the number of communities in layer $\ell$ and
$J(c_{\ell},c_{\ell+1})$ is the joint fraction (the ratio of the number of
nodes in community $c$ in layers $\ell$ and $\ell+1$ to the number of nodes
only in layers $\ell$ or $\ell+1$).

Fig. 4 shows the values of both the flexibility (Eq. 4) and the stationarity
(Eq. 5), repeated for each of the 10 meshes. We observe that as $\omega$
increases, $\Xi$ decreases and approaches zero; $\zeta$ increases overall and
is least sensitive to the mesh used at $\omega=0.01,0.5,1.5$; and the
differences between the meshes are minimal in (a) compared to (b).

Using these plots, we consider a few things when picking $\omega$. Low
flexibility and/or high stationarity would result in communities that are
insensitive to real temporal fluctuations which inform the forecasting. On the
other end of the spectrum, high flexibility and/or low stationarity would mean
that the strain loading is not being taken into account. Also, at high
stationarity values, we observe mesh-dependence among the ensemble of 10
meshes used, which means that the method is sensitive to the type of mesh that
is generated. Therefore, in §IV.2 we will select $\omega=1.5$ for two reasons:
(1) it is not sensitive to the mesh used and (2) it is not too strong to be
insensitive to changes to the state of the study area nor too weak to account
for temporal history.

Figure 4: (a) Network flexibility $\Xi$ (Eq. 4) and (b) network stationarity
$\zeta$ (Eq. 5) as a function of the choice of a particular temporal coupling
parameter $\omega$, for fixed spatial coupling $\gamma=1$. The effect of
$\gamma$ is further explored in §IV.2. Each data point is for one of the 10
meshes, distinguished by color.

##  IV Results

We aim to determine whether we have identified the (known) geometry and
location of Mud Creek with the results of the community detection algorithm
(explored below). Once this is achieved, we examine the details of the degree
of community persistence over time, and use our observations to construct a
measure which is able to forecast increasing risk of failure.

###  IV.1 Quantifying community structures

In order to identify areas of concern, we run community detection using the
modularity function defined in Eq. 2 with parameters $\gamma=1$ and
$\omega=1.5$. For each InSAR snapshot, there is a resulting community
structure: 62 in total, 6 of which are shown in Fig. 5. We only present the
results of one mesh in Fig. 5, but this is representative of the results for
all 10 meshes, indicating that the algorithm is not sensitive to the
particular mesh used. Mud Creek is identified as a community early on, clearly
showing up in Fig. 5(c) in orange and remains persistent for the next $\sim$
400 days prior to its ultimate failure.

Figure 5: Results of community detection using $\gamma$ = 1, $\omega=1.5$,
shown for six representative layers prior to failure at $T=0$ days. Nodes that
are members of the same community have the same (arbitrary) color,
corresponding to a single community ID number. The black outline is Mud Creek.

To identify the underlying pattern within these identified communities, we
visualize the evolution of the community structures in Fig. 6(a), where each
horizontal line represents a single community, whose ID is assigned in order
of its identification by the algorithm, and the color of the line represents
the approximate location of that community (see inset).

There are two characteristics of a community that we considered: size and
strength. The strength of a community is the mean of the weights of the edges,
and the size of a community is the number of nodes. We find that a joint
measure of community size and strength, normalized by global values, is useful
for highlighting the significance $\Psi_{c,\ell}$ of particular communities
detected, calculated as

$\Psi_{c,\ell}=\frac{n_{c,\ell}}{N}\frac{S_{c,\ell}}{S_{\ell}},$ (6)

where $n_{c,\ell}$ is the number of nodes in community $c$ at layer $\ell$,
$N$ is the total number of nodes, and $S_{c,\ell}$ and $S_{\ell}$ is the
strength of community $c$ and layer $\ell$, respectively. Large values of
$\Psi_{c,\ell}$ correspond to a community $c$ being large, fast-moving, and/or
steeply sloped.

Figure 6: Strip-chart showing the evolution of the community structures
detected for parameters $\gamma=1$ and $\omega=1.5$, separately highlighting
location, duration, and significance $\Psi_{c}$. Each detected community found
has an ID number that is arbitrary (but corresponds to the order of
detection), and is plotted as a line with three properties: color specifies
its location (see inset map), the extent along the horizontal axis specifies
its duration, and the thickness of of the line measures the size and strength
of that community at layer $\ell$, according to Eq. 6.

Within Fig. 6, we see a summary of changes to the overall community structure.
From this, we see 3 persistent communities that are picked up: Mud Creek,
Region 1, and Region 2 . We observe that Mud Creek is consistently picked up
as a community from approximately $-400$ days to $-8$ days. Region 1
fluctuates in and out as a significant community and has not been identified
as a creeping landslide. Since Region 2 has been identified as a creeping
landslide, when InSAR alone could not, but has not yet catastrophically
failed, its detection (beyond just Mud Creek) supports community detection as
a method of forecasting where failure is most likely to happen. We note that
Region 3 is correctly not identified as a community because, while it shares
similar topographic characteristics to Mud Creek, it is not creeping. Since
these observations are not sufficient enough to detect when Mud Creek will
fail, we wish to develop a metric that quantitatively distinguishes a time of
failure.

###  IV.2 Forecasting failure via community persistence

We observed that Mud Creek is identified as the same community for the last
$\sim$ 400 days in Fig. 6. Mud Creek is one of the few communities that
clearly persisted through time. Using these observations, we propose a new
measure: community persistence $\Pi_{\ell}$ inspired by Rieck  _et al._ [56],
González-Avella  _et al._ [57]. This measures the stability of the nodal
composition for each community in relation to the community’s size and is
defined as

$\Pi_{\ell}=\frac{1}{N}\sum_{c}\frac{c_{\ell-1}\cap c_{\ell}}{n_{c,\ell}},$
(7)

where $N$ is the total number of nodes in the network, $n_{c,\ell}$ is the
number of nodes in community $c$ at layer $\ell$, and $c_{\ell-1}\cap
c_{\ell}$ is the number of nodes present in community $c$ in both layers
$\ell$ and $\ell-1$. An increase in community persistence indicates a patch of
nodes that are consistently being identified as a community $c$, and that
community $c$’s strength is stronger than the rest of the layer, such that the
other communities start to break down into singleton nodes, as determined by
Eq. 2.

To further explore our choice of $\omega=1.5$ and $\gamma=1$ from §III.2, we
show $\Pi_{\ell}$ for varying values of $\omega$ from 0.5 to 2.5 at $\gamma=1$
and varying values of $\gamma$ from 0.5 to 1.5 at $\omega=1.5$ in Fig. 7(a-b).
The signal flattens out for two reasons: (1) for $\omega<1.0$ because the
community structure is not strongly correlated through time and (2) for
$\gamma>1.1$ because only a few communities are being picked up consistently
with a similar subset of nodes from beginning to end. For high values of
$\omega$ and small values of $\gamma$, there is a signal that is being picked
up, but it is not as strong as the signal that is picked up by our chosen
parameter values: $\gamma=1,\omega=1.5$. There are metrics and tests that we
use to decide what $\gamma$ and $\omega$ values to pick. One is using
flexibility (Eq. 4) and stationarity (Eq. 5) (shown in Fig. 4), and the other
is comparing the results to see what information is being lost or gained by
changing the parameter values. Using both Fig. 4 & Fig. 7, we pick $\gamma=1$
and $\omega=1.5$ as the values that best capture the underlying physical
dynamics without overpowering temporal fluctuations. To ensure that our
analysis is not sensitive to the ensemble of 10 meshes that we use, we run
community detection on 10 randomized meshes, and the mean for those 10 meshes
is shown in Fig. 7(a-b).

With these parameters now defined, we see a sharp increase in community
persistence (shown in Fig. 7(c)) about 56 days before failure for all 10
meshes. This is indicative of a group of nodes, each one linked to spatial
coordinates, that is continually being established as a community between each
layer, in this case the nodes contained within the Mud Creek geometry. As this
community becomes stronger (correlated with increasing velocity) and more
persistent, we observe that the other communities start to break down into
smaller communities, including the community representing the creeping
landslide found within Region 2. This happens because the mean weight to
partition nodes into communities (see Eq. 2) is increasing as we get closer to
failure (reference Fig. 2).

Figure 7: Community persistence $\Pi_{\ell}$ (Eq. 7) as a function of time
before failure, averaged over 10 meshes, (a) for different choices of spatial
coupling parameter $\gamma$ at fixed $\omega=1.5$ and (b) for different
choices of temporal coupling parameter $\omega$ at fixed $\gamma=1.0$) (c) The
mean community persistence $\Pi_{\ell}$ over 10 meshes using
$\gamma=1,\omega=1.5$. Standard error, measured across the ensemble of 10
meshes, is smaller than the symbol size. The dashed lines represent the 6
snapshots of the community structures that are shown in Fig. 5. The horizontal
solid line shows the mean of $\Pi_{\ell}$ for all $\ell$. The red shaded area
highlights where $Pi_{\ell}$ starts to significantly rise above the mean.

##  V Discussion/Conclusion

We have developed an algorithm that takes a geospatial surface and represents
it as a mesh of nodes. A few things we considered when creating the mesh are
sensitivity to the mesh we use, artificial results, and good coverage of the
area, all of which are addressed by using Poisson disc sampling. To test the
spatial resolution of the DEM that is used, we compare a DEM with resolution
of 12 m and the DEM we used for network analysis, as described in §II.1, of a
higher resolution of 1 m, and note that there are no significant differences
between the two, as we downsample the 1 m DEM to 20 m.

We are able to model the dynamics and susceptibility of the system to identify
Mud Creek both spatially (through community detection) and temporally (through
the persistence of those communities). We do this by incorporating information
about both the current state of rheology using velocity data derived from
InSAR and the geometry of the area using slope. We found that using only
velocity or slope was insufficient to distinguish creeping landslides from the
rest of the area: slope-weighted networks identified watershed basins and
velocity-weighted networks identified large communities without separating
them into geophysically-relevant areas. Through the inclusion of velocity and
slope, we are able to pick up clusters of areas that are strongly-connected to
each other. As shown in Fig. 6, we observe three significant areas: one is a
creeping landslide, one is Mud Creek, and one is an area of interest for
future monitoring. Combined with high-quality InSAR data, this method has
allowed us to forecast the location and geometry of the impending failure from
400 days out, without having to visually inspect a map (see Handwerger  _et
al._ [27]).

We quantify these results in Fig. 7; as we approach time of failure, there is
a sharp increase in community persistence $\Pi_{\ell}$. This indicates that
time of failure is close, but is not predictive enough to pinpoint a range of
days. We also note in Fig. 7 that the method is not sensitive to the ensemble
of 10 meshes that are being used.

This raises several promising avenues for improvement in utility. From the
standpoint of continually monitoring, it would be beneficial to instead use
pixel tracking displacement or automatically processed InSAR data, instead of
InSAR data that has been manually corrected, such as the work of Handwerger
_et al._ [27]. Previous examination of the InSAR time series by Handwerger
_et al._ [27] reveal that Mud Creek exhibited strong seasonal displacement
driven by precipitation. The displacement increased over the study period due
to increasing rainfall during the study period. Although the landslide
behavior changed from year to year, and the landslide was moving notably
faster in 2017 than previous years, it was concluded by Handwerger  _et al._
[27] that there was no displacement pattern that is indicative of a slope
approaching runaway instability. However, we note here that this is likely a
result of bias in the InSAR data itself. InSAR cannot detect large
displacements ($>$ half the radar wavelength), which means that if the slope
is moving too fast between acquisitions there can be phase bias and an
underestimate of true displacement. Warrick  _et al._ [31] showed that the
landslide moved over 10 m between March and May 2017, which is too large of
displacement for InSAR to measure. From this, we know that the InSAR-measured
displacement over the final months prior to collapse is much too slow.
Therefore, the quantities used are a lower bound, and improved data would
provide for an even stronger signal than is reported here. Significant outlay
of human resources would be required to avoid underestimating ground
displacements and being dominated by noise, and it remains to be understood
what level of spatial, temporal, and velocity resolution is required to
achieve the type of results we observed here.

The inclusion of time-delayed rainfall layers, or even approximations to soil
moisture (proxy for pore-water pressure) would likely improve the fidelity of
the multilayer network in describing the evolving state of the study site.
Furthermore, the topography of Earth’s surface is constantly changing due to
ground deformation, so using a static DEM may introduce challenges and loss of
crucial information over a long time period.

While community detection improves the signals of forecasting failure in the
Mud Creek area and aids in the identification of creeping landslides, an
important open question is how does this method fare on other areas, as well
as how to improve the temporal signal of failure.

##  VI Acknowledgement

Part of this research was carried out at the Jet Propulsion Laboratory,
California Institute of Technology, under a contract with the National
Aeronautics and Space Administration (80NM0018D0004).

## References

  * Lacroix  _et al._ [2020] P. Lacroix, A. L. Handwerger, and G. Bièvre, Nature Reviews Earth & Environment 1, 404 (2020).
  * Hu  _et al._ [2020] X. Hu, R. Bürgmann, W. H. Schulz, and E. J. Fielding, Nature Communications 11, 2792 (2020).
  * Rutter and Green [2011] E. Rutter and S. Green, Journal of the Geological Society 168, 359 (2011).
  * Aryal  _et al._ [2012] A. Aryal, B. A. Brooks, M. E. Reid, G. W. Bawden, and G. R. Pawlak, Journal of Geophysical Research: Earth Surface 117, 10.1029/2011JF002161 (2012).
  * Tordesillas  _et al._ [2021] A. Tordesillas, S. Kahagalage, L. Campbell, P. Bellett, E. Intrieri, and R. Batterham, Scientific Reports 11, 9729 (2021).
  * Carlà  _et al._ [2019] T. Carlà, E. Intrieri, F. Raspini, F. Bardi, P. Farina, A. Ferretti, D. Colombo, F. Novali, and N. Casagli, Scientific Reports 9, 1 (2019).
  * Intrieri  _et al._ [2019] E. Intrieri, T. Carlà, and G. Gigli, Earth-Science Reviews 193, 333 (2019).
  * Singh and Tordesillas [2020] K. Singh and A. Tordesillas, Entropy 22, 67 (2020).
  * Tordesillas  _et al._ [2018] A. Tordesillas, Z. Zhou, and R. Batterham, Mechanics Research Communications 92, 137 (2018).
  * Cascini  _et al._ [2022] L. Cascini, M. R. Scoppettuolo, and E. Babilio, Landslides 10.1007/s10346-022-01934-3 (2022).
  * Gabriel  _et al._ [1989] A. K. Gabriel, R. M. Goldstein, and H. A. Zebker, Journal of Geophysical Research 94, 9183 (1989).
  * Franceschetti and Lanari [2018] G. Franceschetti and R. Lanari,  _Synthetic Aperture Radar Processing_, 1st ed. (CRC Press, 2018).
  * Di Traglia  _et al._ [2021] F. Di Traglia, C. De Luca, M. Manzo, T. Nolesini, N. Casagli, R. Lanari, and F. Casu, Remote Sensing of Environment 260, 112441 (2021).
  * Teshebaeva  _et al._ [2019] K. Teshebaeva, H. P. Echtler, B. Bookhagen, and M. R. Strecker, Earth Surface Processes and Landforms , esp.4648 (2019).
  * Teshebaeva  _et al._ [2015] K. Teshebaeva, S. Roessner, H. Echtler, M. Motagh, H.-U. Wetzel, and B. Molodbekov, Remote Sensing 7, 8973 (2015).
  * Carlà  _et al._ [2018] T. Carlà, P. Farina, E. Intrieri, H. Ketizmen, and N. Casagli, Engineering Geology 235, 39 (2018).
  * Bernardi  _et al._ [2021] M. S. Bernardi, P. C. Africa, C. de Falco, L. Formaggia, A. Menafoglio, and S. Vantini, Mathematical Geosciences 53, 1781 (2021).
  * Dick  _et al._ [2015] G. J. Dick, E. Eberhardt, A. G. Cabrejo-Liévano, D. Stead, and N. D. Rose, Canadian Geotechnical Journal 52, 515 (2015).
  * Zhou  _et al._ [2022] S. Zhou, A. Tordesillas, E. Intrieri, F. Di Traglia, G. Qian, and F. Catani, Journal of Geophysical Research: Solid Earth 127, 10.1029/2021JB022957 (2022).
  * Das and Tordesillas [2019] S. Das and A. Tordesillas, Remote Sensing 11, 2777 (2019).
  * Agliardi  _et al._ [2020] F. Agliardi, M. M. Scuderi, N. Fusi, and C. Collettini, Nature Communications 11, 1352 (2020).
  * Torres  _et al._ [2021] L. Torres, A. S. Blevins, D. Bassett, and T. Eliassi-Rad, SIAM Review 63, 435 (2021).
  * Le Bouil  _et al._ [2014] A. Le Bouil, A. Amon, S. McNamara, and J. Crassous, Physical Review Letters 112, 246001 (2014).
  * Fazelpour [2022] F. Fazelpour,  _Failure and Flow in Sheared Granular Materials_ , PhD Thesis, North Carolina State University (2022).
  * Mei  _et al._ [2022] J. Mei, G. Ma, Q. Wang, T. Wu, and W. Zhou, International Journal of Solids and Structures 251, 111763 (2022).
  * Tordesillas [2007] A. Tordesillas, Philosophical Magazine 87, 4987 (2007).
  * Handwerger  _et al._ [2019] A. L. Handwerger, M.-H. Huang, E. J. Fielding, A. M. Booth, and R. Bürgmann, Scientific Reports 9, 1569 (2019).
  * Roering  _et al._ [2015] J. J. Roering, B. H. Mackey, A. L. Handwerger, A. M. Booth, D. A. Schmidt, G. L. Bennett, and C. Cerovski-Darriau, Geomorphology 236, 109 (2015).
  * Handwerger  _et al._ [2015] A. L. Handwerger, J. J. Roering, D. A. Schmidt, and A. W. Rempel, Geomorphology 246, 321 (2015).
  * Scheingross  _et al._ [2013] J. S. Scheingross, B. M. Minchew, B. H. Mackey, M. Simons, M. P. Lamb, and S. Hensley, Geological Society of America Bulletin 125, 473 (2013).
  * Warrick  _et al._ [2019] J. A. Warrick, A. C. Ritchie, K. M. Schmidt, M. E. Reid, and J. Logan, Landslides 16, 1201 (2019).
  * noa [2015] PRISM Climate Group, Oregon State U (2015), accessed on 2022-01-13.
  * Swain  _et al._ [2018] D. L. Swain, B. Langenbrunner, J. D. Neelin, and A. Hall, Nature Climate Change 8, 427 (2018).
  * Allen and Luptowitz [2017] R. J. Allen and R. Luptowitz, Nature Communications 8, 16055 (2017).
  * Iverson  _et al._ [2000] R. M. Iverson, M. E. Reid, N. R. Iverson, R. G. LaHusen, M. Logan, J. E. Mann, and D. L. Brien, Science 290, 513 (2000).
  * Mitarai and Nori [2006] N. Mitarai and F. Nori, Advances in Physics 55, 1 (2006).
  * Schulz  _et al._ [2009] W. H. Schulz, J. P. McKenna, J. D. Kibler, and G. Biavati, Landslides 6, 181 (2009).
  * Lu and Godt [2013] N. Lu and J. Godt,  _Hillslope Hydrology and Stability_ (Cambridge University Press, Cambridge, 2013).
  * Girard  _et al._ [2010] L. Girard, D. Amitrano, and J. Weiss, Journal of Statistical Mechanics: Theory and Experiment 2010, P01013 (2010).
  * Jaeger  _et al._ [2007] J. C. Jaeger, N. G. W. Cook, and R. W. Zimmerman,  _Fundamentals of rock mechanics_ , 4th ed. (Blackwell Pub, Malden, MA, 2007).
  * Palla  _et al._ [2007] G. Palla, A.-L. Barabási, and T. Vicsek, Nature 446, 664 (2007).
  * Whitaker  _et al._ [2019] K. A. Whitaker, Z. Varga, L. C. Hsiao, M. J. Solomon, J. W. Swan, and E. M. Furst, Nature Communications 10, 2237 (2019).
  * Fortunato [2010] S. Fortunato, Physics Reports 486, 75 (2010), arXiv: 0906.0612.
  * Papadopoulos  _et al._ [2016] L. Papadopoulos, J. G. Puckett, K. E. Daniels, and D. S. Bassett, Physical Review E 94, 032908 (2016).
  * noa [2010] USGS, 2014, 3D Elevation Program:ARRA-CA CentralCoast-Z3 (2010), accessed on 2021-05-17.
  * Cook [1986] R. L. Cook, ACM Transactions on Graphics 5, 51 (1986).
  * Porter  _et al._ [2009] M. A. Porter, J.-P. Onnela, and P. J. Mucha, Notices of the American Mathematical Society 56, 1082 (2009).
  * Mucha  _et al._ [2010] P. J. Mucha, T. Richardson, K. Macon, M. A. Porter, and J.-P. Onnela, Science 328, 876 (2010).
  * Newman and Girvan [2004] M. E. J. Newman and M. Girvan, Physical Review E 69, 026113 (2004).
  * Newman [2006] M. E. J. Newman, Proceedings of the National Academy of Sciences 103, 8577 (2006).
  * Jeub  _et al._ [1 19] L. G. S. Jeub, M. Bazzi, I. S. Jutla, and P. J. Mucha, A generalized louvain method for community detection implemented in matlab (2011-19), downloaded on 2020-05-15.
  * Sarzynska  _et al._ [2016] M. Sarzynska, E. A. Leicht, G. Chowell, and M. A. Porter, Journal of Complex Networks 4, 363 (2016).
  * Bassett  _et al._ [2015] D. S. Bassett, E. T. Owens, M. A. Porter, M. L. Manning, and K. E. Daniels, Soft Matter 11, 2731 (2015).
  * Bassett  _et al._ [2011] D. S. Bassett, N. F. Wymbs, M. A. Porter, P. J. Mucha, J. M. Carlson, and S. T. Grafton, Proceedings of the National Academy of Sciences 108, 7641 (2011).
  * Bassett  _et al._ [2013] D. S. Bassett, M. A. Porter, N. F. Wymbs, S. T. Grafton, J. M. Carlson, and P. J. Mucha, Chaos: An Interdisciplinary Journal of Nonlinear Science 23, 013142 (2013).
  * Rieck  _et al._ [2018] B. Rieck, U. Fugacci, J. Lukasczyk, and H. Leitte, IEEE Transactions on Visualization and Computer Graphics 24, 822 (2018).
  * González-Avella  _et al._ [2014] J. C. González-Avella, M. G. Cosenza, J. L. Herrera, and K. Tucci, EPL (Europhysics Letters) 107, 28002 (2014).