# LLM Reading Tea Leaves: Automatically Evaluating Topic Models with Large
Language Models

Xiaohao Yang⋄ He Zhao† Dinh Phung⋄ Wray Buntine‡ Lan Du⋄

⋄Monash University, Melbourne, VIC, Australia
†CSIRO’s Data61, Sydney, NSW, Australia
‡VinUniversity, Hanoi, Vietnam
<EMAIL_ADDRESS><EMAIL_ADDRESS>
<EMAIL_ADDRESS>
###### Abstract

Topic modeling has been a widely used tool for unsupervised text analysis.
However, comprehensive evaluations of a topic model remain challenging.
Existing evaluation methods are either less comparable across different models
(e.g., perplexity) or focus on only one specific aspect of a model (e.g.,
topic quality or document representation quality) at a time, which is
insufficient to reflect the overall model performance. In this paper, we
propose WALM (Words Agreement with Language Model), a new evaluation method
for topic modeling that comprehensively considers the semantic quality of
document representations and topics in a joint manner, leveraging the power of
large language models (LLMs). With extensive experiments involving different
types of topic models, WALM is shown to align with human judgment and can
serve as a complementary evaluation method to the existing ones, bringing a
new perspective to topic modeling. Our software package will be available at
https://github.com/Xiaohao-Yang/Topic_Model_Evaluation, which can be
integrated with many widely used topic models.

##  1 Introduction

Topic modeling Blei et al. (2003), a popular unsupervised text analysis
technique, has been applied to various domains, including information
retrieval (Yi and Allan, 2009), marketing analysis (Reisenbichler and
Reutterer, 2019), social media analysis (Laureate et al., 2023),
bioinformatics (Liu et al., 2016), and more. A topic model typically learns a
set of global topics to interpret a text corpus and the topic proportion of a
document as its semantic representation.

Although topic models have been time-tested for two decades, as an
unsupervised technique, comprehensive evaluations of a topic model remain
challenging (Zhao et al., 2021). Originally, topic models are implemented as
probabilistic graphical models such as Latent Dirichlet Allocation (LDA) (Blei
et al., 2003) and many of its Bayesian extensions. For these models, it has
been common practice to measure the log-likelihood or perplexity of a model on
held-out test documents. While log-likelihood or perplexity provides a
straightforward quantitative comparison between models, several issues still
persist: As topic models are not primarily designed for predicting words in
documents but rather for semantically learning interpretable topics and
document representations, these metrics do not measure these qualities.
Furthermore, estimating the predictive probability is often intractable for
Bayesian models, and different papers may employ different sampling or
approximation techniques (Wallach et al., 2009; Buntine, 2009). For recently
proposed neural topic models (NTMs) (Zhao et al., 2021), the computation of
log-likelihood is even more inconsistent.

In addition to log-likelihood or perplexity, document representation quality
and topic quality are evaluated separately. For document representation
quality, downstream task performance is typically used as a metric, such as
document classification (Yang et al., 2023), clustering (Zhao et al., 2021),
and retrieval (Larochelle and Lauly, 2012). For topic quality, the ultimate
evaluation method is human evaluation, which is time-consuming and expensive.
Thus, various automatic metrics have been proposed, such as topic coherence
(Lau et al., 2014), which measures how semantically coherent the
representative words in a topic are, and topic diversity (Dieng et al., 2020),
which measures how diverse discovered topics are. To comprehensively evaluate
the performance of a topic model, one needs to report multiple metrics on both
document representation and topic qualities. However, these metrics can be
contradictory; e.g., a topic model with good topic quality may not preserve
good quality on document representation, and vice versa. This discrepancy
complicates the model selection process for topic models in practice.

In this paper, we aim to develop a new evaluation approach for topic modeling
that comprehensively considers the semantic quality of document
representations and topics in a joint manner, leveraging the power of large
language models (LLMs). Our key idea is as follows: After being trained, a
topic model can infer a document’s distribution over topics and each topic is
a distribution over vocabulary words. With these two distributions, a model
can generate a set of “topical” words given a document, such as by looking at
its representative topics and the representative words of each topic. The
generation of the topical words takes both the topic distribution of a
document and the word distributions of the topics into account, which captures
the semantic summary of the document and is expected to align with the
keywords identified by humans. Thus, the quality of the generated topical
words of documents informatively and comprehensively reflects the quality of a
topic model. As human evaluation is typically costly, we propose querying LLMs
with appropriate prompts as a proxy to generate keywords for the document and
then compare them with the topical words generated by a topic model. Finally,
to quantify the agreement between the words from the topic model and the LLM,
a series of WALM (Words Agreement with Language Model) metrics are proposed.
WALM has the following appealing properties:

  * •

It is a joint metric that evaluates both document representation and topic
qualities.

  * •

It measures how well a topic model captures a document’s semantics, which is
the focus of topic modeling.

  * •

It can be compared across different kinds of topic models.

To examine WALM series metrics, we conduct extensive experiments using various
popular topic models on different datasets, comparing them with other widely
used topic model evaluation metrics. Moreover, human evaluation is also
conducted to demonstrate the alignment of WALM with human judgment.

##  2 Related Work

As an unsupervised technique for uncovering hidden themes in text, evaluating
topic models remains challenging. Early evaluations of a topic model rely on
the log-likelihood or perplexity of held-out documents (Blei et al., 2003),
which measures how well the model predicts the words of documents. As the
computation of predictive probability is often intractable for conventional
Bayesian topic models, various sampling or approximation techniques have been
proposed Wallach et al. (2009); Buntine (2009). Apart from the inconsistent
estimation, held-out likelihood is regarded as not correlated with the
interpretability of topics from a human perspective (Chang et al., 2009),
prompting the direct evaluation of topics and document representation quality.

As for the evaluation of topics, Chang et al. (2009) design the word and topic
intrusion tasks for human annotators, where high-quality topics or document
representations are those where annotators can easily identify the intruders.
Newman et al. (2010) and Mimno et al. (2011) evaluate topic coherence by
direct ratings from human experts. Although human judgment is commonly
regarded as the gold standard, it is expensive and impractical for large-scale
evaluation. Automated evaluation of topic coherence is more practical, such as
Normalized Pointwise Mutual Information (NPMI) (Lau et al., 2014), which
relies on the co-occurrence of the topic’s top words in the reference corpus
to measure topic coherence, with the underlying assumption that a large
reference corpus such as Wikipedia can capture prevalent language patterns.
Although they automate the evaluation of topics and strongly correlate with
human judgment (Newman et al., 2010), counting word co-occurrence in a large
reference corpus is still relatively expensive. Recent works propose
leveraging word embeddings (Nikolenko, 2016) or contextualized embeddings
(Hoover et al., 2021) for efficiently evaluating topic coherence,
incorporating semantics from pre-trained embeddings. Due to common posterior
collapse issues (Lucas et al., 2019) in the growing field of neural topic
models (Zhao et al., 2021), recent works also consider topic diversity (Dieng
et al., 2020) during evaluation, which measures how distinct the top words of
each topic are.

As for the evaluation of document representation, early works focus on how
well the topic proportion of a document represents the document content,
assessed through a topic intrusion task by human annotators (Chang et al.,
2009), which is further extended as automated metrics (Bhatia et al., 2017,
2018). Recent topic models often use the topic proportions as document
representations, the quality of which is commonly investigated through
downstream tasks, including their use as features for document classification
(Nguyen and Luu, 2021), clustering (Zhao et al., 2020), and retrieval
(Larochelle and Lauly, 2012). Recently, the generalization ability of topic
models is investigated by evaluating their quality of document representations
across different unseen corpora (Yang et al., 2023).

In the era of large language models (LLMs) (Thoppilan et al., 2022; Chowdhery
et al., 2022; OpenAI, 2022; Touvron et al., 2023a, b), recent research has
begun leveraging LLMs to evaluate topic models, such as using ChatGPT (OpenAI,
2022) as a proxy for human annotators for word intrusion and topic rating
tasks for evaluating topic coherence (Rahimi et al., 2023; Stammbach et al.,
2023). The focus of these works is still on topic quality only.

In this work, we propose new evaluation metrics for topic models, differing
from previous works in the following ways: (1) Unlike evaluations that focus
on only sub-components of a topic model (i.e., topics or document
representations), our evaluation metrics offer a joint approach to topic model
evaluation, considering both topics and document representations together. (2)
Compared with log-likelihood or perplexity, which also evaluate based on
documents, our evaluation metrics consider semantics from documents and align
with the focus of topic modeling. (3) Different from recent LLM-based
evaluations that use LLMs for topic quality evaluation, ours is a more
comprehensive method that considers both topic quality and document
representation quality and our use of LLMs is quite different from previous
works.

Figure 1: Performance rankings of topic quality (NPMI) and document
representation quality (ACC) during model selection. The best model can be
determined using either NPMI or ACC as the selection criterion. However, it
can be observed that the rankings for topic quality and document
representation quality are inconsistent under the same selection criteria.
Experiments are conducted five times, with the number of topics set to 50.

##  3 Background

Given a document collection $\mathcal{D}:=\\{\bm{d}_{1},...,\bm{d}_{M}\\}$
with $V$ vocabulary words, a topic model is typically trained on their Bag-of-
Words (BOWs), e.g., $\bm{x}\in\mathbb{N}^{V}$. The topic model can infer a
distribution over $K$ topics for each document by running its inference
process:

$\bm{z}:=f_{\bm{\theta}}(\bm{x}),$ (1)

where $\bm{\theta}$ denotes the model parameters of the inference process;
$\bm{z}\in\Delta^{K}$ ($\Delta$ denotes the probability simplex) indicates the
proportion of each topic present in the document and is commonly used as its
semantic representation. Besides, the topic model also discovers $K$ global
topics for the corpus (i.e., $\bm{T}:=\\{\bm{t}_{1},...,\bm{t}_{K}\\}$), where
each topic $\bm{t}\in\Delta^{V}$ is a distribution over $V$ vocabularies.
Ideally, each topic captures a semantic concept that can be interpreted by its
top-weighted words. To train a topic model, one often needs to generate or
reconstruct the word distribution of the document from $\bm{z}$ by running its
generative process:

$\bm{w}:=f_{\bm{\phi}}(\bm{z},\bm{T}),$ (2)

where $\bm{\phi}$ are the model parameters of the generative process;
$\bm{w}\in\Delta^{V}$ is the per-document word distribution from which
$\bm{x}$ is sampled. Let $\bm{Z}:=\\{\bm{z}_{1},...,\bm{z}_{N}\\}$ be the
semantic representations of $N$ test documents and $\bm{T}$ be the $K$ learned
topics, current evaluation of a topic model is commonly conducted based on
either $\bm{Z}$ or $\bm{T}$ separately.

##  4 Method

###  4.1 Motivation

Both topics and document representations are important components of a topic
model. To comprehensively evaluate a topic model, it is common practice to
report the performance of both parts. This can be done by measuring topic
quality using metrics such as NPMI and assessing document representation
quality through downstream classification accuracy (ACC) (see section 5.1 for
details of metrics calculation). However, a model that prioritizes topic
quality (e.g., NPMI) may not perform well in terms of document representations
(e.g., ACC), and vice versa, which creates difficulty during model selection,
as illustrated in Figure 1. This inconsistency in the performance of the two
components is also indicated by Bhatia et al. (2017). Therefore, evaluating a
topic model based on sub-components only is insufficient to reveal the entire
model’s performance. Recent topic models usually focus on improving topic
quality, such as clustering-based topic models in Sia et al. (2020);
Grootendorst (2022), but they do not evaluate their performance for document
representation. In this work, we aim to introduce a novel evaluation method
for topic modeling that jointly assesses the semantic quality of both topics
and document representations, with the help of large language models.

###  4.2 Key Idea

We propose to conduct the evaluation in a joint manner that considers both
document representations and topics, rather than evaluating them separately as
in previous works. To do so, we obtain the document-word distribution $\bm{w}$
for a given document from the topic model by running both its inference and
generative process:

$\bm{w}:=f_{\bm{\phi}}(f_{\bm{\theta}}(\bm{x})).$ (3)

The inference process infers the document representation $\bm{z}$ for a given
document $\bm{x}$, as in Eq. 1; the generative process generates or
reconstructs the word distribution $\bm{w}$ based on the document
representation $\bm{z}$ and topics $\bm{T}$, as in Eq. 2. Therefore, the
evaluation based on $\bm{w}$ involves both document representations and
topics.

Next, we take the top-weighted words $\mathbf{w}$ from the word distribution
$\bm{w}$ generated by the topic model as the “topical” words of the document.
Those topical words can be regarded as a semantic summary of the document from
the target topic model’s perspective. To generate high-quality topical words
for a document, a topic model should learn good global topics as well as good
document representations. Now, the evaluation of the topic model can be
converted to the evaluation of the quality of the topical words. Suppose the
true representative words $\mathbf{k}$ of document $\bm{x}$ are given, then we
can formulate our evaluation task as:

$S(\mathbf{w},\mathbf{k}),$ (4)

where $S(\cdot,\cdot)$ is a score function (Sec. 4.4) to quantify the
agreement between $\mathbf{w}$ and $\mathbf{k}$ (Sec. 4.3).

Figure 2: An example prompt and output of using LLAMA2 for document keywords
summarization.

###  4.3 Document summarization by LLM

Following our evaluation task in Eq. 4, the ideal representative words
$\mathbf{k}$ are from human summary of the document. However, this is
expensive and impractical for large-scale evaluation. With the recent
advancements in LLMs, which have demonstrated performance akin to human
capabilities in various NLP tasks, including text summarization Wang et al.
(2023); Tang et al. (2023); Zhang et al. (2024), we propose to leverage LLMs
by prompting to extract key words of documents:

$\mathbf{k}:=\text{LLM}(\text{Prompt($\bm{d}$)}).$ (5)

Specifically, we query keywords $\mathbf{k}$ for a given document $\bm{d}$
from an LLM by proposing the prompt shown in Figure 2. The prompt consists of
a task instruction, an in-context example (Liu et al., 2021), and the queried
document.

###  4.4 Choices of the Score Function

For the score function $S(\cdot,\cdot)$ in Eq. 4, we propose different ways to
calculate it: (1) Overlap-based, which computes the number of overlapping
words between $\mathbf{w}$ and $\mathbf{k}$, and (2) Embedding-based, which
calculates the overall semantic similarity between the two word sets using
pre-trained word embeddings.

#### Word Overlap

A straightforward choice of the score function is directly counting the
overlaps between $\mathbf{w}$ and $\mathbf{k}$. Considering the potential
variant in forms of the same word, we convert each word to its root form
before counting, formulated as:

$S_{\text{overlap}}:=C(f_{\text{root}}(\mathbf{w})\cap
f_{\text{root}}(\mathbf{k}))\times f_{\text{n}}(\mathbf{w},\mathbf{k}),$ (6)

where $C(\cdot)$ and $f_{\text{root}}(\cdot)$ are the counting and rooting
(e.g., stemming or lemmatization) operation, respectively;
$f_{\text{n}}(\mathbf{w},\mathbf{k}):=1/(N+M)$ returns the normalising factor
based on two input word sets, where $N$ and $M$ are the number of words in
$\mathbf{w}$ and $\mathbf{k}$, respectively.

#### Synset Overlap

Considering the case that different words may describe the same or similar
concept (e.g., both “puppy” and “dog”), we leverage the synset of WordNet
(Miller, 1995) when counting the number of overlaps: if there is an overlap
between two words’ synsets, then it is considered as an overlap, formulated
as:

$\displaystyle
S_{\text{synset}}:=\sum_{i=1}^{N}\sum_{j=1}^{M}\mathds{1}(C(f_{\text{synset}}(\mathbf{w}_{i})\cap$
$\displaystyle f_{\text{synset}}(\mathbf{k}_{j}))>0)\times
f_{\text{n}}(\mathbf{w},\mathbf{k}),$ (7)

where $\mathds{1}(\cdot)$ denotes the indicator function;
$f_{\text{synset}}(\cdot)$ is a function that returns the synset for a given
word.

#### Word Optimal Assignment

We consider another choice of $S$, which measures the overall semantic
similarity between two sets of words with pre-trained word embeddings (Mikolov
et al., 2013; Pennington et al., 2014). Since the alignments between words
from $\mathbf{w}$ and $\mathbf{k}$ are unknown, directly measuring similarity
between word embeddings is not feasible. To automatically find the alignment
for each word of $\mathbf{w}$ to each word of $\mathbf{k}$, we formulate it as
the following optimal assignment (OA) problem and solve it using the Hungarian
algorithm Kuhn (1955): Given a word set $\mathbf{w}$ that has $N$ words:
$\mathbf{w}:=\\{\mathbf{w}_{1},\mathbf{w}_{2},...,\mathbf{w}_{N}\\}$ and their
embedding vectors
$\bm{E}^{\mathbf{w}}:=\\{\bm{e}^{\mathbf{w}_{1}},\bm{e}^{\mathbf{w}_{2}},...,\bm{e}^{\mathbf{w}_{N}}\\}$;
and another word set $\mathbf{k}$ that has $M$ words:
$\mathbf{k}:=\\{\mathbf{k}_{1},\mathbf{k}_{2},...,\mathbf{k}_{M}\\}$ with
related embedding vectors
$\bm{E}^{\mathbf{k}}:=\\{\bm{e}^{\mathbf{k}_{1}},\bm{e}^{\mathbf{k}_{2}},...,\bm{e}^{\mathbf{k}_{M}}\\}$.
Define a cost matrix $\bm{C}\in\mathbb{R}_{\geq 0}^{N\times M}$ whose entry
$\bm{C}_{i,j}:=\text{CosD}(\bm{e}^{\bm{\mathbf{w}}_{i}},\bm{e}^{\bm{\mathbf{k}}_{j}})$,
where $\text{CosD}(\cdot,\cdot)$ denotes the cosine distance function; and a
binary matrix $\bm{A}\in\\{0,1\\}^{N\times M}$ whose entry $\bm{A}_{i,j}=1$ if
word $\bm{\mathbf{w}}_{i}$ is assigned to word $\bm{\mathbf{k}}_{j}$, and 0
otherwise. The goal is to solve the following optimal assignment problem:

$\displaystyle\min_{\bm{A}}\sum_{i=1}^{N}$
$\displaystyle\sum_{j=1}^{M}\bm{C}_{i,j}\times\bm{A}_{i,j},$ subject to:
$\displaystyle\sum_{j=1}^{M}\bm{A}_{i,j}=1,$
$\displaystyle\sum_{i=1}^{N}\bm{A}_{i,j}=1.$ (8)

By finding the optimal binary matrix $\bm{A}^{*}$, we obtain the distance
between $\mathbf{w}$ and $\mathbf{k}$ by:

$S_{\text{oa}}:=D_{\text{oa}}(\bm{\mathbf{w}},\bm{\mathbf{k}}):=\sum_{i=1}^{N}\sum_{j=1}^{M}\bm{C}_{ij}\times\bm{A}_{ij}^{*}.$
(9)

#### Word Optimal Transport

Considering that both $\mathbf{w}$ and $\mathbf{k}$ are top words of
probability distributions, where each word essentially retains a portion of
probability mass. Our previous calculations ignore the probability mass of
words and treat each word in the set as equal. Now we include the probability
mass and formulate the similarity calculation between $\mathbf{w}$ and
$\mathbf{k}$ as an optimal transport (OT) problem: Given two discrete
distributions $\mu(\mathbf{w},\bm{w})$ and $\mu(\mathbf{k},\bm{k})$, where
$\mathbf{w}:=\\{\mathbf{w}_{1},\mathbf{w}_{2},...,\mathbf{w}_{N}\\}$ and
$\mathbf{k}:=\\{\mathbf{k}_{1},\mathbf{k}_{2},...,\mathbf{k}_{M}\\}$ are the
supports of those two distributions; $\bm{w}\in\Delta^{N}$ and
$\bm{k}\in\Delta^{M}$ are their related probability vectors222We assume
$\mu(\mathbf{k},\bm{k})$ is a uniform distribution over the keywords
$\mathbf{k}$ from the LLM. Thus, $\bm{k}$ is a uniform probability vector.;
following the same construction of cost matrix $\bm{C}$ in the previous OA
problem, the OT problem between $\mu(\mathbf{w},\bm{w})$ and
$\mu(\mathbf{k},\bm{k})$ is defined as:

$\displaystyle\min_{\bm{P}}\sum_{i=1}^{N}$
$\displaystyle\sum_{j=1}^{M}\bm{C}_{i,j}\times\bm{P}_{i,j},$ subject to:
$\displaystyle\sum_{j=1}^{M}\bm{P}_{i,j}=\bm{w}_{i},$
$\displaystyle\sum_{i=1}^{N}\bm{P}_{i,j}=\bm{k}_{j},$ (10)

where $\bm{P}\in\mathbb{R}_{\geq 0}^{N\times M}$ is the transport plan, whose
entry $\bm{P}_{i,j}$ indicates the amount of probability mass moving from
$\bm{w}_{i}$ to $\bm{k}_{j}$. Similarly, by finding the optimal transport plan
$\bm{P}^{*}$ using solvers such as those in Flamary et al. (2021), the OT
distance between $\mu(\mathbf{w},\bm{w})$ and $\mu(\mathbf{k},\bm{k})$ is
obtained by:

$\displaystyle S_{\text{ot}}$
$\displaystyle:=D_{\text{ot}}(\mu(\mathbf{w},\bm{w}),\mu(\mathbf{k},\bm{k}))$
$\displaystyle:=\sum_{i=1}^{N}\sum_{j=1}^{M}\bm{C}_{ij}\times\bm{P}_{ij}^{*}.$
(11)

Compared with our OA and OT formulations for WALM, they are similar in that
they both construct the cost matrix $\bm{C}$ using cosine distance between
pre-trained word embeddings. However, they differ in the following ways: (1)
OA treats words in the set as equal, while OT considers probability mass of
each word. (2) OA can be viewed as a “hard” assignment problem between two
word sets because the entries of $\bm{A}$ are binary. In contrast, OT can be
regarded as a “soft” assignment because of the spread of probability mass in
$\bm{P}$.

Figure 3: Topic models’ performance in terms of WALM on 20News (top row) and
DBpedia (bottom row).

##  5 Experiments

###  5.1 Experimental Setup

#### Datasets

Two widely used datasets, 20Newsgroup (Lang, 1995) (20News), which contains
long documents, and DBpedia (Auer et al., 2007), which includes short
documents, are used for our experiments. We will release our pre-processed
dataset along with the software package in Github.

#### Evaluated Models

We conduct experiments on 7 popular topic models from traditional
probabilistic to recent neural topic models. (1) Latent Dirichlet Allocation
(LDA) (Blei et al., 2003), the most popular probabilistic topic model that
assumes a document is generated by a mixture of topics. (2) LDA with Products
of Experts (PLDA) (Srivastava and Sutton, 2017), an early NTM that applies the
product of experts instead of the mixture of multinomials in LDA. (3) Neural
Variational Document Model (NVDM) (Miao et al., 2017), a pioneer NTM that uses
a Gaussian as the prior distribution of topic proportions of documents. (4)
Embedded Topic Model (ETM) (Dieng et al., 2020), an NTM that involves word and
topic embeddings in the generative process. (5) Neural Topic Model with
Covariates, Supervision, and Sparsity (SCHOLAR) (Card et al., 2017), an NTM
that applies a logistic normal prior on topic proportions and leverages extra
information from metadata. (6) Neural Sinkhorn Topic Model (NSTM) (Zhao et
al., 2020), a recent NTM based on an optimal transport framework. (7)
Contrastive Learning Neural Topic Model (CLNTM) (Nguyen and Luu, 2021), a
recent NTM that uses contrastive learning to regularize document
representations. We keep all these models’ default settings as suggested in
their implementations. All experiments are conducted 5 times with different
random seeds; mean and std values are reported.

#### Settings of WALM

We evaluate topic models using WALM series metrics we propose in Sec. 4.4,
including $S_{\text{overlap}}$, $S_{\text{synset}}$, $S_{\text{oa}}$, and
$S_{\text{ot}}$. As for the settings of WALM, we use the word embeddings from
GloVe (Pennington et al., 2014) pre-trained on
Wikipedia333https://nlp.stanford.edu/projects/glove/. We set the number of top
words for a document generated by the topic model to 10; The LLM we use is
LLAMA2-13b-chat (Touvron et al., 2023b), which has 13 billion parameters and
is fine-tuned for chat completion; We use a 4-bit
quantization444https://huggingface.co/TheBloke/Llama-2-13B-chat-GGML version
of this model for memory efficiency and leverage the
llama.cpp555https://github.com/abetlen/llama-cpp-python library for easy
inference. Notably, our evaluation metrics can be used with any LLMs. We use
LLAMA2 in our study because it is open-sourced and cost-efficient.

#### Settings of Existing Metrics

We also evaluate topic models with existing commonly used metrics to compare
with ours. (1) Topic Coherence and Diversity: we apply NPMI to evaluate topic
coherence using Wikipedia as the reference corpus, done by the Palmetto
package666https://github.com/dice-group/Palmetto (Röder et al., 2015).
Following standard protocol, we consider the top 10 words of each topic and
obtain the average NPMI score of topics by selecting the top 50% coherent
topics. As for topic diversity (TD), we compute the percentage of unique words
in the top 25 words of all topics, as defined in Dieng et al. (2020). (2)
Document Representation Quality: we conduct document classification and
clustering to evaluate the representation capability of topic models. As for
classification, we train a Random Forest classifier based on the training
documents’ representation and test the accuracy (ACC) in the testing
documents, as in previous works such as Nguyen and Luu (2021). As for
clustering, we conduct K-Means clustering based on test documents’
representation and report the Purity (KM-Purity) and Normalized Mutual
Information (KM-NMI), as in previous works such as Zhao et al. (2020). (3)
Perplexity: we use document completion perplexity (Wallach et al., 2009) to
evaluate the predictive ability of topic models. We split each test document
into two equal-length folds randomly. Then we compute the document completion
perplexity (DC-PPL) on the second fold of documents based on the topic
proportion inferred from the first fold, as in previous works such as Dieng et
al. (2020).

Figure 4: Learning curves of topic models in terms of WALM on test sets of
20News (top row) and DBpedia (bottom row), where the number of topics is set
to 50.

Document | Model | Topical Words
---|---|---
9cmIt’s my understanding that, when you format a magneto-optical disc, (1) the formatting software installs a driver on the disc, (2) if you insert the disc in a different drive, then this driver is loaded into the computer’s memory and then controls the drive, and (3) if this driver is incompatible with the drive, then the disc can not be mounted and/or properly read/written. Is that correct? | LDA_B | drive disk card controller hard mb file scsi bios power
LDA_C | drive disk scsi hard card controller mb floppy ide sale
NVDM_B | driver, drive, problem, card, time, file, thanks, need, email, work
NVDM_C | drive, driver, hard, scsi, window, cd, mb, floppy, disc, work
| LLAMA2 | driver format magneto-optical disc operating system communication incompatibility
| Human | driver disc drive computer hardware software memory formatting format incompatible
9cmWrong World. Wrong World is a 1985 Australian film directed by Ian Pringle. It was filmed in Nhill and Melbourne in Victoria Australia. | LDA_B | film american released directed football album summer played team hospital
LDA_C | film played directed baseball league australian major drama football award
NVDM_B | specie, album, school, known, located, north, film, directed, american, released
NVDM_C | film, album, released, second, south, new, directed, american, australian, known
| LLAMA2 | wrong world film australian directed ian pringle victoria melbourne
| Human | film movie directed directer australian melbourne victoria

Table 1: Documents’ topical words from topic models at the beginning phase
(e.g., NVDM_B, LDA_B) and convergence phase (e.g., NVDM_C, LDA_C) according to
WALM, where the number of topics is set to 50.

###  5.2 Results and Analysis

#### Topic Model Evaluation with WALM

We assess topic models’ performance based on our evaluation metrics on both
20News and DBpedia. We have the following observations based on our results
illustrated in Figure 3: (1) The WALM values of most models on DBpedia show
better performance than 20News, which indicates that it is easier for topic
models to generate informative topical words for short documents than long
documents. (2) The performance indicated by overlap-based metrics (e.g.,
$S_{\text{overlap}}$ and $S_{\text{synset}}$) and embedding-based metrics
(e.g., $S_{\text{oa}}$ and $S_{\text{ot}}$) is slightly different. The reason
is that embedding-based metrics consider the semantic distance among words,
which can be more flexible than the exact match in overlap-based metrics. (3)
It can be observed that there is little improvement from recent NTMs over LDA
and NVDM in terms of our joint metrics. The potential reason is that most
contemporary NTMs primarily focus on enhancing topic coherence while
neglecting the generation of documents, thus showing weak performance in
generating topical words of documents as indicated by WALM. (4) WALM shows
consistent performance ranking across different settings of $K$ (e.g.,
$S_{\text{overlap}}$ and $S_{\text{synset}}$) for long documents, which
potentially alleviates the effort of extensive model selection in terms of $K$
during model comparison.

#### Learning Curves of WALM

We illustrate the learning curves of topic models in terms of WALM in Figure
4, which clearly shows how our metrics change along with the learning of a
topic model. It demonstrates that our metrics are informative for model
selection.

#### Qualitative Analysis on Topical Words for Documents

We qualitatively investigate the topical words of documents by topic models at
different stages in Table 1, where we randomly sample one document for 20News
and DBpedia, respectively. We have the following observations based on our
results: (1) The topical words at the beginning phase contain less
semantically related words about the documents than those at convergence,
which aligns with the learning status (as in Figure 4) indicated by WALM. (2)
The topical words of NVDM include more words that reveal the documents’ main
messages than LDA, which aligns with the ranking (as in Figure 3) suggested by
WALM. (3) The words from LLAMA2 are comparable with those from human
annotators in the longer document. For the short document, LLAMA2 focuses on
detailed information such as the movie title (e.g., “wrong” and “world”),
which human annotators ignore. Further analysis between using LLAMA2 and human
judgment to obtain the “true” topical words is described in Sec. 5.4.

Figure 5: Pearson’s correlation coefficient among evaluation metrics.

#### Correlation to Other Metrics

We compute Pearson’s correlation coefficients among existing and WALM series
metrics, similar to the correlation analysis in previous works such as Doogan
and Buntine (2021) and Rahimi et al. (2023). Pearson’s correlation
coefficients among the metrics are plotted in a heatmap in Figure 5. Based on
the results, we observe that: (1) WALM variants are highly correlated with
each other since they originate from the same mechanism. (2) Compared with DC-
PPL, which also evaluates the entire model based on documents, WALM show weak
correlations, suggesting a new family of evaluation metrics. (3) Compared with
other types of evaluations, WALM has moderate correlations with document
representation metrics (e.g., KM-Purity, KM-NMI, and ACC), and weak
correlations with topic quality metrics (e.g., NPMI and TD). This indicates
that our joint evaluation metrics consider both components. Those observations
demonstrate the comprehensive nature of WALM, which can be used as a
complementary evaluation method to the existing ones.

Figure 6: Learning curves of NVDM in terms of embedding-based metrics and
their contextualized variants on 20News (top row) and DBpedia (bottom row).

###  5.3 Contextualized Embeddings for WALM

#### Obtaining Contextualized Embeddings

Recall that in Eq. 9 and Eq. 4.4, the cost matrix $\bm{C}$ is constructed
using cosine distances between word embeddings. Here, we change our
construction of $\bm{C}$ from using static word embeddings from GloVe
(Pennington et al., 2014) to the contextualized word embeddings from LLAMA2,
considering that the same word may have different semantic meanings in
different contexts. We obtain the contextualized embeddings of a word given a
document differently in two cases: (1) When the target word appears in the
context document, we take the average embeddings of each occurrence as the
contextualized embedding. (2) When there is no occurrence of the target word
in the given document, we add an auxiliary sentence to the document in the
following format:

> “<Given Document>. This document is talking about <Target Word>.”

Then, we obtain the contextualized embedding of the target word given the
document with the auxiliary sentence. By replacing the global word embeddings
with contextualized word embeddings, we have new variants of our embedding-
based WALM (i.e., $S_{\text{oa}}$ and $S_{\text{ot}}$), i.e.,
$\bf{S}_{\text{oa\\_c}}$ and $\bf{S}_{\text{ot\\_c}}$.

#### Observations

Since the cost of obtaining contextualized embeddings is high for LLMs, we
compute $S_{\text{oa\\_c}}$ and $S_{\text{ot\\_c}}$ in a case study, where we
test NVDM on 100 documents randomly sampled from the test sets of 20News and
DBpedia, respectively. We plot the learning curves on test documents in Figure
6. We observe that using word embeddings or contextualized embeddings in our
embedding-based scores exhibits similar trends but with different values on
both datasets. Further analysis of both approaches is conducted in Sec. 5.4.

###  5.4 Comparisons with Human Evaluation

WALM computes the difference between documents’ topical words generated by
topic models and an LLM, treating the words from the LLM as the ground truth.
Here, we investigate the gap between using LLM and human judgment as the true
topical words in WALM. To quantify this gap, we propose the following
calculation:

$G:=\frac{|S(\mathbf{w},\mathbf{k}^{\text{LLM}})-S(\mathbf{w},\mathbf{k}^{\text{Human}})|}{S(\mathbf{w},\mathbf{k}^{\text{Human}})},$
(12)

where $S$ is the score function we propose in Sec. 4.4 (and the contextualized
variants in Sec. 5.3); $\mathbf{w}$, $\mathbf{k}^{\text{LLM}}$, and
$\mathbf{k}^{\text{Human}}$ are the topical words from the topic model, LLM
and human annotators, respectively. We empirically observe that the topical
words generated by topic models always differ from those identified by humans,
thus the denominator in Eq. 12 will not be zero.

#### Case Study Setup

As human annotation is expensive for large-scale investigation, we randomly
sample 200 test documents from 20News and DBpedia as a case study. We ask
human annotators to provide keywords that capture the main points of each
document. Then, given a trained topic model, we compute the gap between using
the words from the LLM and human judgment in our metrics using Eq. 12.

Figure 7: Evaluation gap between using LLAMA2 and human judgment as the “true”
topical words.

#### Observations

The results are illustrated in Figure 7, where the evaluated model is NVDM
with $K=50$ trained on 20News and DBpedia, respectively. We have the following
observations based on the results: (1) Comparing the datasets, the gap between
using human judgment and the LLM in 20News is lower than in DBpedia in most
cases. This indicates that for long documents such as those in 20News, the
topical words generated by the LLM are closer to human judgment than in short
documents in DBpedia. (2) Comparing the metrics, $S_{\text{oa}}$ exhibits the
lowest gap among WALM metrics, with a gap value of 0.03 and 0.15 on 20News and
DBpedia, respectively. This shows the effectiveness of using the LLM as a
proxy for human judgment when applied in $S_{\text{oa}}$. (3) Comparing the
embeddings, using contextualized embeddings from the LLM can further narrow
the evaluation gap for $S_{\text{oa}}$ and $S_{\text{ot}}$ on short documents.

##  6 Conclusion

In this work, we propose WALM (Words Agreement with Language Model), a new
evaluation method for topic models, which takes both topic and document
representation quality into account jointly. WALM measures the agreement
between the topical words generated by topic models and those from the LLM for
given documents. We propose different calculations to quantify their
agreement, including overlap-based and embedding-based metrics. Through
extensive experiments, WALM series metrics show their effectiveness in
reflecting topic models’ capability in terms of semantic summary of documents.
We show that WALM metrics align with human judgment and can serve as an
informative complementary method for topic model evaluation. In the future, we
believe that WALM metrics can be further enhanced through improved document
summarization by LLMs and word set similarity calculations, and topic models
can be further optimized based on WALM series targets.

## References

  * Auer et al. (2007) Sören Auer, Christian Bizer, Georgi Kobilarov, Jens Lehmann, Richard Cyganiak, and Zachary Ives. 2007.  Dbpedia: A nucleus for a web of open data.  In _international semantic web conference_ , pages 722–735. Springer.
  * Bhatia et al. (2017) Shraey Bhatia, Jey Han Lau, and Timothy Baldwin. 2017.  An automatic approach for document-level topic model evaluation.  _arXiv preprint arXiv:1706.05140_.
  * Bhatia et al. (2018) Shraey Bhatia, Jey Han Lau, and Timothy Baldwin. 2018.  Topic intrusion for automatic topic model evaluation.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 844–849.
  * Blei et al. (2003) David M Blei, Andrew Y Ng, and Michael I Jordan. 2003.  Latent dirichlet allocation.  _Journal of machine Learning research_ , 3(Jan):993–1022.
  * Buntine (2009) Wray Buntine. 2009.  Estimating likelihoods for topic models.  In _Asian conference on machine learning_ , pages 51–64. Springer.
  * Card et al. (2017) Dallas Card, Chenhao Tan, and Noah A Smith. 2017.  Neural models for documents with metadata.  _arXiv preprint arXiv:1705.09296_.
  * Chang et al. (2009) Jonathan Chang, Sean Gerrish, Chong Wang, Jordan Boyd-Graber, and David Blei. 2009.  Reading tea leaves: How humans interpret topic models.  _Advances in neural information processing systems_ , 22.
  * Chowdhery et al. (2022) Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. 2022.  Palm: Scaling language modeling with pathways.  _arXiv preprint arXiv:2204.02311_.
  * Dieng et al. (2020) Adji B Dieng, Francisco JR Ruiz, and David M Blei. 2020.  Topic modeling in embedding spaces.  _Transactions of the Association for Computational Linguistics_ , 8:439–453.
  * Doogan and Buntine (2021) Caitlin Doogan and Wray Buntine. 2021.  Topic model or topic twaddle? re-evaluating semantic interpretability measures.  In _Proceedings of the 2021 conference of the North American chapter of the association for computational linguistics: human language technologies_ , pages 3824–3848.
  * Flamary et al. (2021) Rémi Flamary, Nicolas Courty, Alexandre Gramfort, Mokhtar Z Alaya, Aurélie Boisbunon, Stanislas Chambon, Laetitia Chapel, Adrien Corenflos, Kilian Fatras, Nemo Fournier, et al. 2021.  Pot: Python optimal transport.  _Journal of Machine Learning Research_ , 22(78):1–8.
  * Grootendorst (2022) Maarten Grootendorst. 2022.  Bertopic: Neural topic modeling with a class-based tf-idf procedure.  _arXiv preprint arXiv:2203.05794_.
  * Hoover et al. (2021) Jacob Louis Hoover, Alessandro Sordoni, Wenyu Du, and Timothy J O’Donnell. 2021.  Linguistic dependencies and statistical dependence.  _arXiv preprint arXiv:2104.08685_.
  * Kuhn (1955) Harold W Kuhn. 1955.  The hungarian method for the assignment problem.  _Naval research logistics quarterly_ , 2(1-2):83–97.
  * Lang (1995) Ken Lang. 1995.  Newsweeder: Learning to filter netnews.  In _Machine learning proceedings 1995_ , pages 331–339. Elsevier.
  * Larochelle and Lauly (2012) Hugo Larochelle and Stanislas Lauly. 2012.  A neural autoregressive topic model.  _Advances in Neural Information Processing Systems_ , 25.
  * Lau et al. (2014) Jey Han Lau, David Newman, and Timothy Baldwin. 2014.  Machine reading tea leaves: Automatically evaluating topic coherence and topic model quality.  In _Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics_ , pages 530–539.
  * Laureate et al. (2023) Caitlin Doogan Poet Laureate, Wray Buntine, and Henry Linger. 2023.  A systematic review of the use of topic models for short text social media analysis.  _Artificial Intelligence Review_ , pages 1–33.
  * Liu et al. (2021) Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. 2021.  What makes good in-context examples for gpt-$3$?  _arXiv preprint arXiv:2101.06804_.
  * Liu et al. (2016) Lin Liu, Lin Tang, Wen Dong, Shaowen Yao, and Wei Zhou. 2016.  An overview of topic modeling and its current applications in bioinformatics.  _SpringerPlus_ , 5(1):1–22.
  * Lucas et al. (2019) James Lucas, George Tucker, Roger B Grosse, and Mohammad Norouzi. 2019.  Don’t blame the elbo! a linear vae perspective on posterior collapse.  _Advances in Neural Information Processing Systems_ , 32.
  * Miao et al. (2017) Yishu Miao, Edward Grefenstette, and Phil Blunsom. 2017.  Discovering discrete latent topics with neural variational inference.  In _International conference on machine learning_ , pages 2410–2419. PMLR.
  * Mikolov et al. (2013) Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013.  Efficient estimation of word representations in vector space.  _arXiv preprint arXiv:1301.3781_.
  * Miller (1995) George A Miller. 1995.  Wordnet: a lexical database for english.  _Communications of the ACM_ , 38(11):39–41.
  * Mimno et al. (2011) David Mimno, Hanna Wallach, Edmund Talley, Miriam Leenders, and Andrew McCallum. 2011.  Optimizing semantic coherence in topic models.  In _Proceedings of the 2011 conference on empirical methods in natural language processing_ , pages 262–272.
  * Newman et al. (2010) David Newman, Jey Han Lau, Karl Grieser, and Timothy Baldwin. 2010.  Automatic evaluation of topic coherence.  In _Human language technologies: The 2010 annual conference of the North American chapter of the association for computational linguistics_ , pages 100–108.
  * Nguyen and Luu (2021) Thong Nguyen and Anh Tuan Luu. 2021.  Contrastive learning for neural topic model.  _Advances in neural information processing systems_ , 34:11974–11986.
  * Nikolenko (2016) Sergey I Nikolenko. 2016.  Topic quality metrics based on distributed word representations.  In _Proceedings of the 39th International ACM SIGIR conference on Research and Development in Information Retrieval_ , pages 1029–1032.
  * OpenAI (2022) OpenAI. 2022.  Chatgpt: Optimizing language models for dialogue.  _http://web.archive.org/web/ 20230109000707/https://openai.com/ blog/chatgpt/_.
  * Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher D Manning. 2014.  Glove: Global vectors for word representation.  In _Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP)_ , pages 1532–1543.
  * Rahimi et al. (2023) Hamed Rahimi, Jacob Louis Hoover, David Mimno, Hubert Naacke, Camelia Constantin, and Bernd Amann. 2023.  Contextualized topic coherence metrics.  _arXiv preprint arXiv:2305.14587_.
  * Reisenbichler and Reutterer (2019) Martin Reisenbichler and Thomas Reutterer. 2019.  Topic modeling in marketing: recent advances and research opportunities.  _Journal of Business Economics_ , 89(3):327–356.
  * Röder et al. (2015) Michael Röder, Andreas Both, and Alexander Hinneburg. 2015.  Exploring the space of topic coherence measures.  In _Proceedings of the eighth ACM international conference on Web search and data mining_ , pages 399–408.
  * Sia et al. (2020) Suzanna Sia, Ayush Dalmia, and Sabrina J Mielke. 2020.  Tired of topic models? clusters of pretrained word embeddings make for fast and good topics too!  _arXiv preprint arXiv:2004.14914_.
  * Srivastava and Sutton (2017) Akash Srivastava and Charles Sutton. 2017.  Autoencoding variational inference for topic models.  _arXiv preprint arXiv:1703.01488_.
  * Stammbach et al. (2023) Dominik Stammbach, Vilém Zouhar, Alexander Hoyle, Mrinmaya Sachan, and Elliott Ash. 2023.  Re-visiting automated topic model evaluation with large language models.  _arXiv preprint arXiv:2305.12152_.
  * Tang et al. (2023) Liyan Tang, Zhaoyi Sun, Betina Idnay, Jordan G Nestor, Ali Soroush, Pierre A Elias, Ziyang Xu, Ying Ding, Greg Durrett, Justin F Rousseau, et al. 2023.  Evaluating large language models on medical evidence summarization.  _npj Digital Medicine_ , 6(1):158.
  * Thoppilan et al. (2022) Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng-Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, et al. 2022.  Lamda: Language models for dialog applications.  _arXiv preprint arXiv:2201.08239_.
  * Touvron et al. (2023a) Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023a.  Llama: Open and efficient foundation language models.  _arXiv preprint arXiv:2302.13971_.
  * Touvron et al. (2023b) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023b.  Llama 2: Open foundation and fine-tuned chat models.  _arXiv preprint arXiv:2307.09288_.
  * Wallach et al. (2009) Hanna M Wallach, Iain Murray, Ruslan Salakhutdinov, and David Mimno. 2009.  Evaluation methods for topic models.  In _Proceedings of the 26th annual international conference on machine learning_ , pages 1105–1112.
  * Wang et al. (2023) Jiaan Wang, Yunlong Liang, Fandong Meng, Beiqi Zou, Zhixu Li, Jianfeng Qu, and Jie Zhou. 2023.  Zero-shot cross-lingual summarization via large language models.
  * Yang et al. (2023) Xiaohao Yang, He Zhao, Dinh Phung, and Lan Du. 2023.  Towards generalising neural topical representations.  _arXiv preprint arXiv:2307.12564_.
  * Yi and Allan (2009) Xing Yi and James Allan. 2009.  A comparative study of utilizing topic models for information retrieval.  In _Advances in Information Retrieval: 31th European Conference on IR Research, ECIR 2009, Toulouse, France, April 6-9, 2009. Proceedings 31_ , pages 29–41. Springer.
  * Zhang et al. (2024) Tianyi Zhang, Faisal Ladhak, Esin Durmus, Percy Liang, Kathleen McKeown, and Tatsunori B Hashimoto. 2024.  Benchmarking large language models for news summarization.  _Transactions of the Association for Computational Linguistics_ , 12:39–57.
  * Zhao et al. (2021) He Zhao, Dinh Phung, Viet Huynh, Yuan Jin, Lan Du, and Wray Buntine. 2021.  Topic modelling meets deep neural networks: A survey.  _arXiv preprint arXiv:2103.00498_.
  * Zhao et al. (2020) He Zhao, Dinh Phung, Viet Huynh, Trung Le, and Wray Buntine. 2020.  Neural topic model via optimal transport.  _arXiv preprint arXiv:2008.13537_.


# Spatial Feature Calibration and Temporal Fusion for Effective
One-stage Video Instance Segmentation

Minghan Li1,2 Shuai Li1,2 Lida Li1 and Lei Zhang1,2111Corresponding author.
This work is supported by the Hong Kong RGC RIF grant (R5001-18).
1The HongKong Polytechnic University  2DAMO Academy  Alibaba Group
<EMAIL_ADDRESS>{ csshuaili, cslli<EMAIL_ADDRESS>

###### Abstract

Modern one-stage video instance segmentation networks suffer from two
limitations. First, convolutional features are neither aligned with anchor
boxes nor with ground-truth bounding boxes, reducing the mask sensitivity to
spatial location. Second, a video is directly divided into individual frames
for frame-level instance segmentation, ignoring the temporal correlation
between adjacent frames. To address these issues, we propose a simple yet
effective one-stage video instance segmentation framework by spatial
calibration and temporal fusion, namely STMask. To ensure spatial feature
calibration with ground-truth bounding boxes, we first predict regressed
bounding boxes around ground-truth bounding boxes, and extract features from
them for frame-level instance segmentation. To further explore temporal
correlation among video frames, we aggregate a temporal fusion module to infer
instance masks from each frame to its adjacent frames, which helps our
framework to handle challenging videos such as motion blur, partial occlusion
and unusual object-to-camera poses. Experiments on the YouTube-VIS valid set
show that the proposed STMask with ResNet-50/-101 backbone obtains 33.5 % /
36.8 % mask AP, while achieving 28.6 / 23.4 FPS on video instance
segmentation. The code is released online https://github.com/MinghanLi/STMask.

##  1 Introduction

Video instance segmentation aims to obtain the pixel-level segmentation mask
for individual instances of all classes over the entire frames of a video,
which heavily depends on spatial position-sensitive features to localize
frame-level objects and redundant temporal information to track instances
across frames. Following object detection and image instance segmentation
works, modern video instance segmentation approaches usually adopt the top-
down framework of first detecting and segmenting objects frame by frame and
then linking instance masks across frames.

(a) Original anchors and features

(b) Calibrated anchors and features

(c) Adaptive features

(d) Aligned features

Figure 1:  Spatial calibration for anchor boxes and bounding boxes. (a) and
(b) display anchors and features in original and calibrated one-stage networks
respectively. (c) and (d) demonstrate adaptive and aligned features extracted
from predicted bounding boxes. Purple, blue and green rectangles denote
anchors, predicted and ground-truth bounding boxes respectively, where
coloured areas indicate the receptive filed of their convolutional features.

Top-down video instance segmentation approaches can be divided into two-stage
and one-stage methods. By adding a tracking branch to Mask R-CNN [18], two-
stage video instance segmentation methods [43, 2, 16] first predict region-of-
interests (RoIs) around ground-truth bounding boxes, and then feed aligned
features via RoIPooling [32] or RoIAlign [18] to segment frame-level object
masks and to track cross-frame instances. To obtain better location-sensitive
features for mask predictor, many spatial feature calibration strategies for
RoIs have been continuously proposed in recent years like Deformable RoI [12]
and Hybrid Task Cascade [8]. For temporal information exploration, recently
proposed MaskProp [2] utilizes temporal features propagated from all frames of
a video clip for clip-level instance tracking. Obviously, two-stage methods
have recognized the importance of spatial feature localisation and temporal
feature tracking for video instance segmentation. One-stage instance
segmentation networks [11, 42, 5, 35, 6, 24, 25], which focus more on real-
time speed, usually employ a fully convolutional network structure to directly
predict the final mask for instances. Without RoIs of two-stage methods for
localisation, early one-stage methods have to introduce extra position-
sensitive information to improve segmentation performance like position-
sensitive score maps [11] or semantic features [9]. In image domain, recently
proposed anchor-based one-stage methods like Yolact [5] and CondInst [35]
decompose instance segmentation as the linear combination between instance-
specific mask coefficients and instance-independent prototypes. Furthermore,
SipMask [6], introducing a tracking branch in Yolact[5], achieves real-time
speed but inferior performance in video instance segmentation task.

Analysing these anchor-based one-stage instance segmentation methods, we
observe that, as shown in Fig.1 (a), multiple anchors of different shapes at
each spatial position (purple rectangles) share same convolutional features
(yellow area), which are neither aligned with pre-defined anchor boxes nor
with ground-truth bounding boxes. This fact does violate that instances
segmentation is a spatial location-sensitive task. On the other hand, one-
stage video instance segmentation methods directly divide a video into
separate frames to perform image instance segmentation frame by frame and then
track them across frames, which completely ignores high temporal correlation
between adjacent frames. This may fail to handle those challenging videos with
motion blur, partial occlusion, or unconventional object-to-camera poses. In
other words, modern one-stage video instance segmentation methods achieve
real-time speed at the cost of discarding spatial feature calibration and
temporal feature correlation.

To address the issues, we propose a simple yet effective one-stage video
instance segmentation framework, named STMask. Firstly, we design a feature
calibration strategy for anchor boxes and ground-truth bounding boxes to
obtain more precise spatial features. Specifically, as shown in Fig.1 (b), to
enable each anchor box can extract its own specific features, we first design
multiple convolutional kernels at each spatial position, and then generate
anchors according to the receptive field of these convolutional kernels. To
improve feature presentation for objects segmentation and tracking, we first
predict regressed bounding boxes around ground-truth bounding boxes by
regression branch, and then extract features from them to segment and track
instances. As shown in Fig.1 (c) and (d), we provide two strategies to extract
features from regressed bounding boxes, including adaptive features by a
single $1\times 1$ convolutional layer and aligned features by mathematical
derivation. Finally, we explore temporal correlation between video frames by
adding a temporal fusion module to infer instance masks from adjacent frames,
thereby improving the performance of objects detection, segmentation and
tracking for those challenging videos.

##  2 Related Work

Video instance segmentation directly benefits from advances in image instance
segmentation and video object detection field. Thus this section consists of
three parts.

Image Instance Segmentation. Existing image instance segmentation methods
either follow bottom-up or top-down paradigms. The bottom-up methods [9, 36]
widely employ multiple stages to first perform semantic segmentation and then
identify the specific location of each instance by boundary detection [20],
pixel clustering [25], or pixels embedding loss [30, 22, 31], position-
sensitive pooling [11, 24]. The top-down instance segmentation approaches
first predict bounding boxes by object detectors and then perform mask
segmentation within the predicted boxes. Mask R-CNN [18] extends Faster R-CNN
[32] by adding a mask segmentation branch on each Region of Interest (RoI),
where RoIAlign operator is introduced to obtain ‘repool’ features of each
proposal for better mask prediction. Follow-up works try to improve its
accuracy by aligning spatial features [12], enriching the FPN features [28] or
addressing the incompatibility between a mask’s confidence score and its
localization accuracy [19]. One-stage instance segmentation approaches [42,
34, 7, 45, 38, 5, 40, 21] recently are proposed to keep the trade-off between
speed and performance. Yolact-based methods [5, 35, 6] break up instance
segmentation into two parallel subtasks: generating a set of instance-
independent prototypes and predicting instance-specific mask coefficients.

Figure 2: An overview architecture of STMask, including frame-level instance
segmentation (blue area) and cross-frame instance segmentation (yellow area).
Specifically, we first use Yolact with spatial calibration to obtain frame-
level instance masks in time $t-1$ and $t$ respectively, then feed FPN
features of two adjacent frames and the output of correlation operation to
TemporalNet to infer the displacement of instances from time $t-1$ to $t$, and
finally merge frame-level and cross-frame instance masks to obtain final
instance masks.

Video Object Detection. For handling challenging videos such as motion blur
and occlusion, optical flow [46, 39], correlation operation [15], deformable
convolutional networks [3, 41] and relation networks [14, 10] are the popular
technologies to propagate or align features across frames. In addition,
methods like[13, 33] try to utilise semantic similarity between frames to
assist object detection in videos.

Video Instance Segmentation. To jointly perform the detection, segmentation
and tracking tasks simultaneously, most of video instance segmentation methods
extend Mask R-CNN [18] by adding a new branch for tracking. For example,
MaskTrack R-CNN [43] predicts an extra embedding vector for each instance and
use an external memory to store them for tracking across frames. MaskProp [2]
introduces a mask propagation branch to propagate frame-level instance masks
from each video frame to all the other frames in a video clip and then match
clip-level instance masks for tracking, providing state-of-the-art instance
segmentation performance and quite limited speed. Besides, [26] proposes a
modified variation autoencoder (VAE) architecture built on the top of Mask
R-CNN [18]. Recently, SipMask [6] also introduces a tracking branch (same as
[43]) in the one-stage image instance segmentation network Yolact [5] to
obtain inferior performance but real-time speed. Besides, inspired by Guided
Anchor [38, 23], SipMask further aligns feature with regressed bounding boxes
to improve feature representation for classification and mask coefficients
generation. Existing spatial feature calibration for one-stage approaches only
focus on the misalignment between features and regressed bounding boxes.

##  3 STMask

The overall architecture of STMask shown in the Fig. 2 consists of frame-level
instance segmentation with spatial calibration and cross-frame instance
segmentation by temporal fusion module.

###  3.1 Spatial Calibration

Our goal is to align features with anchors and ground-truth bounding boxes
respectively for one-stage instance segmentation methods. For spatial feature
calibration on anchors, multiple convolutional kernels shown in Fig. 3 are
introduced to alleviate the misalignment between features and anchors. For
spatial feature calibration on bounding boxes, shown in the pink rectangle of
Fig. 2, we first individually predict regressed bounding boxes around ground-
truth bounding boxes, and then extract features from those regressed bounding
boxes to classify, segment and track instances.

####  3.1.1 Feature Calibration for Anchors (FCA)

One-stage anchor-based object detectors usually sample a large number of
regions in the input image, determine whether these regions contain objects of
interest, and adjust the edges of the regions so as to more accurately predict
the ground-truth bounding boxes of the targets. In general, the sliding-window
fashion to generate anchors is the most popular method, which generates
multiple boxes with different scales and aspect ratios while centring on each
pixel. As shown in Fig. 1(a), for multiple anchors with different shapes, one-
stage detectors directly employ a $3\times 3$ convolution on the central point
of anchors to extract features. In practice, the receptive field of
convolution should be positively related to the size of anchor boxes. For
example, those larger anchor boxes should have larger receptive fields, while
those smaller boxes should have smaller ones. To address this issue, thus, we
adopt multiple convolution kernels with different aspect ratios on each FPN
layer. For example, we replace the single $3\times 3$ convolution with three
new aspect ratios of convolutions, $3\times 3,\ 3\times 5,\ 5\times 3$
respectively. The FCA architecture implemented on the bounding box regression
branch is shown in Fig. 3. To further ensure the calibration between
convolutional features and anchors, we keep the scales unchanged while
changing the anchor aspect ratios from [1, 1/2, 2] to [1, 3/5, 5/3], which are
same as the aspect ratios of convolutions. The simple feature calibration for
anchors maintains position sensitivity for object detection and segmentation.

Figure 3:  Bounding Box Regression Architecture. Taking the regression branch
as example, we design multiple convolutions for calibrating features with pre-
defined anchor boxes, where $a\\!=\\!3$ is the number of anchors.

####  3.1.2 Feature Calibration for Bounding Boxes (FCB)

After the feature calibration with anchors in last part, features for
classification, mask coefficients and embedding vectors of tracking branch
still fail to align with the regressed bounding boxes. To further address the
issue, we adopt the prediction architecture of first predicting regressed
bounding boxes and then extracting features from the regressed bounding boxes
to segment and track objects.

Let $\\{(P_{i},G_{i})\\}_{i=1,...,N}$ denote the pairs of anchor boxes and
ground-truth bounding boxes. Note that subscript $i$ is only added when
necessary. $P=(P_{x},P_{y},P_{w},P_{h})$ specifies the pixel coordinates of
the centre and the width and height of the anchor box. The same goes for
ground-truth bounding box $G=(G_{x},G_{y},G_{w},G_{h})$. The bounding-box
regression aims to learn a transformation $d$ that maps an anchor box to its
ground-truth bounding box:

$\displaystyle\vspace{-1mm}d=[d_{x},d_{y},d_{w},d_{h}].$ (1)

After that, its regressed bounding box $B_{i}$ can be calculated by applying
the transformation[17]

$\displaystyle B_{x}=P_{w}d_{x}+P_{x},\quad\ \ B_{y}=P_{h}d_{y}+P_{y};$
$\displaystyle B_{w}=P_{w}\exp(d_{w}),\quad B_{h}=P_{h}\exp(d_{h}).$ (2)

Based on the transformation between anchor boxes and predicted bounding boxes,
we can introduce a 2D deformable convolution to calibrate convolutional
features from anchor boxes to predicted bounding boxes. For easier
understanding, we take a $3\times 3$ convolution with dilation 1 as an example
to explicate the process of spatial feature calibartion for bounding boxes.
For each location $p_{0}$ on the output map $\mathbf{g}$, the 2D deformable
convolution can be formulated as

$\displaystyle\mathbf{g}(p_{0})=\sum_{p_{n}\in\mathcal{R}}w(p_{n})\cdot\mathbf{f}(p_{0}+p_{n}+\Delta
p_{n}),\vspace{-1mm}$ (3)

where $\Delta p_{n}$ is the offsets on the position $p_{n}\in\mathcal{R}$,
which is a point of the grid

$\vspace{-1mm}\mathcal{R}=\left\\{\begin{array}[]{lr}(-1,-1)\ (-1,0)\
(-1,1)\\\ (\ \ 0,-1)\ \ (\ \ 0,\ 0)\ \ (\ 0,\ 1)\\\ (\ \ 1,-1)\ \ (\ \ 1,\ 0)\
\ (\ 1,\ 1)\\\ \end{array}\right\\},$ (4)

To extract convolutional features from predicted bounding boxes, the offsets
$\mathcal{O}$ should be dominated by the transforamtion $d$. Thus, this paper
provides two strategies to obtain offsets: adding a single $1\times 1$
convolution layer to predict adptive offsets, or directly deriving the aligned
offsets through mathematical geometric knowledge.

(a) Scale-invariant Translation

(b) Scale Transformation

Figure 4:  Illustration of aligned features from anchor boxes to predicted
bounding boxes.

Adaptive Features on Bounding Boxes. Inspired by the anchor-guided feature
adaption module [38], we also transform the feature at each individual
location based on the underlying anchor transformation:

$\displaystyle\vspace{-2mm}\mathcal{O}=\mathcal{N}_{O}(d),\vspace{-1mm}$ (5)

where $\mathcal{N}_{O}$ is a $1\times 1$ convolutional layer to predict
offsets according to the bounding-box regression transformation $d$. When the
predicted offsets $\mathcal{O}\in R^{2\times 3\times 3}$ is fed into a
deformable convolutional layer to produce adaptive features shown in blue
points in Fig. 1(c). For such regression-dependent offsets, each regressed
bounding box can learn its own adaptive features to perform further objects
classification, segmentation and tracking.

Aligned Features on Bounding Boxes. For the regression transformation $d$ in
terms of four functions in Eq. 1, the first two specify a scale-invariant
translation of the centre of anchor box, while the second two specify log-
space translations of the width and height of anchor box. According to the
four parameters of transformation $d$, the derivation process of generating
the offsets can also be divided into two steps: scale-invariant translation
and scale transformation, shown in Fig. 4 (a) and (b) respectively. On one
hand, Fig. 4 (a) demonstrates that all sampling points on the grid
$\mathcal{R}$ of the anchor box have the same scale-invariant translation as
the centre point. On the other hand, Fig. 4 (b) shows that the absolute scale
transformation of the width and height on the grid $\mathcal{R}$ is related to
their own coordinate position. Overall, the derived offsets for all points on
the grid should be

$\vspace{-1mm}\mathcal{O}=(\Delta y,\Delta x)I+(\Delta h,\Delta
w)\mathcal{R}.$ (6)

where $I$ is a matrix with all elements as 1, $\Delta x,\Delta y,\Delta h$ and
$\Delta w$ are listed below:

$\displaystyle\Delta x=k_{w}d_{x},\ \Delta w=\exp(d_{w})-1,$ (7)
$\displaystyle\Delta y=k_{h}d_{y},\ \Delta h\ =\exp(d_{h})-1.\vspace{-1mm}$
(8)

where $k=(k_{w},k_{h})$ is the kernel size on the width and the height. Due to
the limit of space, the detailed formula derivation process is provided in the
supplementary materials. In fact, the mathematical derivation for the offset
in this part is equivalent to a special case of RoIAlign operation in two-
stage networks, where each bin only takes the central point as its output
value.

###  3.2 Temporal Fusion Module

Compared with image instance detection and segmentation, video instance
segmentation task faces more challenges such as partial occlusion, unusual
view, motion blur. To address the issue, thus, we further set up a temporal
fusion module for cross-frame bounding boxes regression and mask segmentation.
The yellow area of Fig. 2 shows the schematic diagram of temporal fusion
module between two adjacent frames, which concatenate features of FPN between
two adjacent frames to infer the displacement of instances from the previous
frame to the current frame, thereby obtaining cross-frame instance masks from
the previous frame to the current frame. Afterthat, we only need to merge
frame-level detected instance masks and cross-frame tracked instance masks to
get the final instances masks. Such a double guarantee architecture does
improve the accuracy of detection and segmentation in video domain.

Frame-level Instance Segmentation. Same as Yolact [5], STMask decomposes
instance segmentation into the linear combination between instance-independent
prototypes $P\in R^{H^{\prime}\times W^{\prime}\times k}$ and instance-
specific mask coefficients $C\in R^{k\times n}$. Essentially, the process of
learning prototypes $P$ is equivalent to learn the online basis set, also
called dictionary [29], when each instance $i$ can find a linear combination
$C_{i}\in R^{k\times 1}$ of a ’few’ atoms from prototypes that is ‘close’ to
the ground-truth mask $M_{i}\in R^{H^{\prime}\times W^{\prime}}$. The each
instance segmentation can be implemented efficiently using a single matrix
multiplication and the activation function (sigmoid function $\sigma$):

$\vspace{-1mm}M_{i}=\mathbf{Crop}(\sigma(PC_{i}),\ B_{i}).$ (9)

where the final masks $M_{i}$ is also cropped by the predicted bounding box
$B_{i}$.

Figure 5:  Temporal Network Architecture.

Cross-frame Instance Segmentation. Given a pair of frames $I^{t-1},I^{t}$ from
a video $\mathcal{V}\in R^{H\times W\times V}$, the goal of proposed temporal
fusion module is to track those object masks from time $t\\!-\\!1$ to $t$,
noted as $M^{t-1,t}$. Since the prototypes are instance-independent, according
to Eq. (9), the temporal fusion module intuitively needs to predict the mask
coefficients $C^{t-1,t}$ and bounding boxes $B^{t-1,t}$ of instances from time
$t\\!-\\!1$to $t$.

Inspired by RoI tracking process in video object detection task [15], the
temporal fusion module of STMask should include a bounding boxes regressor and
a mask coefficient predictor. To be specific, as shown in the yellow area of
Fig. 2, we first adopt a correlation operation on FPN features of $x^{t-1}\in
R^{H^{\prime}\times W^{\prime}\times f}$ and $x^{t}\in R^{H^{\prime}\times
W^{\prime}\times f}$ to embed motion information of instances, noted as
$x_{corr}^{t-1,t}\in R^{H^{\prime}\times W^{\prime}\times d^{2}}$, where $d$
is the side length of local square. Then the concatenation of features
$\\{x^{t-1},x^{t},x_{corr}^{t-1,t}\\}$ is transported to the temporal network,
shown in Fig. 5, to infer the displacement of bounding boxes $d^{t-1,t}$ and
mask coefficients $\Delta C^{t-1,t}$ of instances from time $t-1$ to $t$,
where

$\displaystyle\vspace{-1mm}d^{t-1,t}=\\{d_{x}^{t-1,t},d_{y}^{t-1,t},d_{w}^{t-1,t},d_{h}^{t-1,t}\\}.$
(10)

Thus, the cross-frame bounding boxes $B^{t-1,t}$ from time $t\\!-\\!1$ to $t$
can be formulated as

$\displaystyle
B_{x}^{t\\!-\\!1,t}\\!=\\!B_{w}^{t\\!-\\!1}d_{x}^{t\\!-\\!1,t}\\!+\\!B_{x}^{t-1},\
B_{y}^{t\\!-\\!1,t}\\!=\\!B_{h}^{t-1}d_{y}^{t\\!-\\!1,t}\\!+\\!B_{y}^{t-1},$
(11) $\displaystyle
B_{w}^{t\\!-\\!1,t}\\!=\\!B_{w}^{t\\!-\\!1}exp(d_{w}^{t\\!-\\!1,t}),\
B_{h}^{t\\!-\\!1,t}\\!=\\!B_{h}^{t-1}exp(d_{h}^{t\\!-\\!1,t}).$ (12)

where the predicted bounding box coordinates $B^{t-1}$ acts as an anchor. On
the other hand, due to the similarity of the prototypes between adjacent
frames, the mask coefficients of each instance on different frames should also
be ’close’ to each other. Based on the mask coefficients $C^{t-1}$ in the
$t\\!-\\!1$ frame, the cross-frame mask coefficients $C^{t-1,t}$ inferred from
time $t\\!-\\!1$ to $t$ can be obtained by

$\displaystyle C^{t-1,t}=C^{t-1}+\Delta C^{t-1,t}.$ (13)

Finally, the instance masks $M^{t-1,t}$ inferred from time $t\\!-\\!1$ to $t$
can be conducted by

$\vspace{-1mm}M^{t-1,t}=\mathbf{Crop}(\sigma(P^{t}C^{t-1,t}),B^{t-1,t}).$ (14)

Merge Frame-level and Cross-frame Instance Segmentation. We denote the set of
video-level instance IDs as $\mathcal{Y}$, which is incrementally built, and
$N_{t}$ is the number of instances predicted by frame-level instance
segmentation of Eq. (9) at time $t$. At the beginning, we assign IDs
$\mathcal{Y}=\\{1,\dots,N_{1}\\}$ for all frame-level object segmentations in
the first frame of a video. For following video frames, we first perform
frame-level instance segmentations of Eq. (9) to predict frame-level instance
masks $M^{t}$, and then execute the temporal fusion module of Eq. (14) to
infer cross-frame instance masks $M^{t-1,t}$ from time $t-1$ to $t$.
Obviously, the cross-frame instance masks $M^{t-1,t}$ are naturally given the
same embedding vectors $E^{t-1}$ and the same instance IDs as $M^{t-1}$. Thus,
the matching scores between frame-level masks $M^{t}$ and cross-frame masks
$M^{t-1,t}$ consist of two components: the cosine similarity of embedding
vectors and mask IoU:

$\displaystyle s_{ij}^{t}=\alpha\frac{E^{t}_{i}\cdot E^{t-1}_{j}}{\parallel
E^{t}_{i}\parallel\parallel E^{t-1}_{j}\parallel}+\beta\
\mbox{MIoU}(M^{t}_{i},M^{t-1,t}_{j})\vspace{-2mm}$ (15)

where $i\in\\{1,\dots N_{t}\\},j\in\mathcal{Y}$, and $\alpha,\beta$ are
hyperparameters to balance the effect of different components. For each frame-
level instance mask $M^{t}_{i}$, let
$s_{i}^{t}=\max_{j\in\mathcal{Y}}s_{ij}^{t}$ be the maximum score among all
instances IDs $j\in\mathcal{Y}$. If $s_{i}^{t}$ is greater than a certain
threshold $\epsilon$, the frame-level instance mask $M^{t}_{i}$ will be
assigned the instance ID with the highest score. Otherwise, it will be
considered as a new one and will be assigned an instance ID $|\mathcal{Y}|+1$.
Overall, the frame-level instance masks $M^{t}_{i}$ will be assigned instance
IDs by:

$y_{i}^{t}=\left\\{\begin{array}[]{ll}\arg\max_{j\in\mathcal{Y}}\
s_{ij}^{t},&\mbox{if}\ s_{i}>\epsilon,\\\
|\mathcal{Y}|+1,&\mbox{otherwise}.\end{array}\right.$ (16)

We denote $\mathcal{Y}^{t}=\\{y_{i}^{t}|i\in\\{1,\dots N_{t}\\}\\}$ as the set
including all frame-level instance IDs at time $t$. If there are instance IDs
that do not appear in the set of frame-level instance IDs $\mathcal{Y}^{t}$
but appear in the set of cross-frame instance IDs $\mathcal{Y}$, in other
words,

${\mathcal{Y}-\mathcal{Y}\cap\mathcal{Y}^{t}\neq\emptyset},$

these cross-frame instance masks
$M^{t-1,t}_{\mathcal{Y}-\mathcal{Y}\cap\mathcal{Y}^{t}}$ will be supplemented
as the missing instances in the $t$ frame. Finally, the merged masks of all
instances in the $t$-th frame should be

$\bar{M}^{t}=M^{t}\cup
M^{t-1,t}_{\mathcal{Y}-\mathcal{Y}\cap\mathcal{Y}^{t}}.$ (17)

##  4 Experimental Results

Training. We conduct video instance segmentation experiments on YouTube-VIS
[43] dataset using the standard metrics. In training, the YouTube-VIS train
set is split as two subsets: train-sub and valid-sub set. One is used for
training and the other is used as the valid set during training. We train all
models on pairs of frames, where the second frame in a pair is randomly
selected with a time gap $\delta\in$ [-5, 5] relative to the first frame,
using pre-trained models on MS COCO dataset [27]. Similar as MaskTrack
R-CNN[43], we use the input size $360\times 640$ for training. All model is
trained with batch size 16 for 160k iterations and divide the learning rate at
66K and 133K. Training takes 1-2 days on four NVIDIA 2080Ti.

Inference. Given an input image, we forward it through the frame-level network
to obtain the outputs including classification scores, bounding box, embedding
vectors, mask coefficients and the prototypes. Box-based Non-maximum
suppression (NMS) with the threshold being 0.5 is used to remove duplicated
detections and then the top 100 bounding-boxes are used to compute frame-level
masks. After that, we forward the features of the previous frame and the
current frames through temporal fusion module to first infer masks from the
previous frame to current frame, then assig IDs for frame-level masks of the
current frame and supplement the missing masks. We take the hyperparameters as
$\alpha=1,\beta=1$ in temporal fusion module.

###  4.1 Design Choices

In this section, we discuss the design choices of spatial calibration and
temporal fusion module for our proposed one-stage video instance segmentation
framework STMask. Similar as MaskTrack R-CNN [43], we extend Yolact [5] by
adding a tracking branch to predict embedding vectors for tracking. For this
part, we adopt ResNet101 with deformable convolution layers (interval=3 ) as
the backbone of Yolact [5]. Note that all experiments do not include other
improvements in Yolact++ [4] such as more anchors and semantic segmentation
loss.

Design Choices of FCA. Our baseline Yolact [5] has three anchors with aspect
ratios [1, 1/2, 2] and a $3\times 3$ convolution in prediction head. As shown
in Table 1 (1st row), it achieves 28.9% in mask AP. After adding FCA strategy,
the setting of aspect ratios with smaller convolutions $\\{3\times 3,3\times
2,2\times 3\\}$ still brings the gain of 1% mask AP, and the setting with
larger convolutions of aspect ratios $\\{3\times 3,3\times 5,5\times 3\\}$
brings a significant improvement of 1.9 % mask AP. Note that, for fair
comparison, we set the kernel size to nearly $3\times 3$ to ensure that the
performance gain does come from feature calibration on anchors rather than a
larger receptive field. For all experiments below, we choose the latter
setting by default.

Design choices of FCB. To analyse the impact of the feature calibration for
regression bounding boxes (FCB) on classification, mask coefficients and
embedding vectors of tracking branch, we discuss some possible combinations
among them on the prediction head. As shown in Table 2, our baseline with FCA
(1st row) achieves 30.8% mask AP. Overall, all experiments adding FCB with
adaptive and aligned features strategy on all possible combination among
branches of the prediction head obtain significantly improved results on all
metrics, around 1-3 %. Besides, all experiments with aligned features
calibration is relatively lower than that of adaptive features, because the
latter may be more adaptable than the former. By default, FCB (ada) and FCB
(ali) denote FCB with adaptive features on all three branches, and that with
aligned features on mask coefficient and tracking branches, respectively.

Table 1:  Design choices for FCA on YouTube-VIS valid set. Aspect ratios |  AP |  AP50 |  AP75 |  APS |  APM |  APL
---|---|---|---|---|---|---
{3:3, 3:3, 3:3} |  28.9 |  45.8 |  30.1 |  9.8 |  26.9 |  40.3
{3:3, 3:2, 2:3} |  29.9 |  48.9 |  31.2 |  8.0 |  27.8 |  40.3
{3:3, 3:5, 5:3} |  30.8 |  49.8 |  31.0 |  11.8 |  29.7 |  44.6
|  |  |  |  |  |
Table 2:  Design choices for FCB with adaptive and aligned features on YouTube-VIS valid set. Baseline+FCA |  +Class |  +Mask |  +Track |  AP(ada) |  AP(ali)
---|---|---|---|---|---
✓ |  |  |  |  30.8 |  30.8
✓ |  ✓ |  ✓ |  ✓ |  33.7 |  31.7
✓ |  ✓ |  ✓ |  |  34.4 |  31.7
✓ |  ✓ |  |  ✓ |  34.3 |  32.4
✓ |  |  ✓ |  ✓ |  32.5 |  33.1
|  |  |  |  |

Design Choices of Temporal Fusion Module. We first conduct experiments on
different FPN layers and the side length of local square for the correlation
operation. As shown in Table 3, the P4 FPN layer with the side length $11$
obtains the better performance for temporal fusion module.

Table 3:  Design choices for temporal fusion module. Layers |  d |  AP |  AP50 |  AP75 |  AR1 |  AR10
---|---|---|---|---|---|---
P3 |  13 |  33.8 |  51.3 |  35.8 |  32.1 |  39.6
P4 |  11 |  35.0 |  53.0 |  38.7 |  32.8 |  40.1
|  |  |  |  |  |

###  4.2 Ablation Study

We conduct the ablation study on YouTube-VIS valid set using Yolact with
ResNet101-DCN backbone[4] as our basline. For fair comparison, the baseline of
all experiments adopts three anchors and does not include semantic
segmentation loss. Table 4 shows the impact of progressively integrating our
different components: feature calibration with anchor boxes (FCA), adaptive /
aligned feature calibration with anchor boxes (FCB), and temporal fusion
module (TF) to the baseline. The four components progressively increase
segmentation performance to 36.8 % mask AP. Besides, to better explore the
individual improvements for mask segmentation brought by each single
component, we also conduct the experiments adding the different components
individually to the baseline shown in Table 5, where FCA, FCB(ada), FCB(ali)
and TF component respectively contribute to the gains of 1.9 % , 2.3 % , 2.4 %
and 5.2 % mask AP. Among these components, the FCB(ada) and TF provide the
most improvement in accuracy over the baseline. These results suggest that
each of our components individually contributes towards improving the final
performance.

We also perform more ablation study for STMask on COCO dataset to verify the
effectiveness of spatial feature calibration on image instance segmentation
task, which are provided in the supplementary material.

Table 4:  Ablation study on YouTube-VIS valid: progressively integrating different components into the baseline.  Baseline |  +FCA |  +FCB |  +TF |  AP |  AP50 |  AP75 |  FPS
---|---|---|---|---|---|---|---
|  (ada / ali) |  |  |  |
✓ |  |  |  |  28.9 |  45.8 |  30.1 |  31.2
✓ |  ✓ |  |  |  30.8 |  49.8 |  31.0 |  29.5
✓ |  ✓ |  ✓ / |  |  34.4 |  53.0 |  34.6 |  28.3
✓ |  ✓ |  / ✓ |  |  33.1 |  51.4 |  35.2 |  26.7
✓ |  ✓ |  ✓ / |  ✓ |  36.8 |  56.8 |  38.0 |  23.4
✓ |  ✓ |  / ✓ |  ✓ |  36.3 |  55.2 |  39.9 |  22.1
|  |  |  |  |  |  |
Table 5:  Ablation study on YouTube-VIS valid: individually integrating different components into the baseline. Methods |  Impro. |  AP |  AP50 |  AP75 |  APS |  APM |  APL
---|---|---|---|---|---|---|---
Baseline |  - |  28.9 |  45.8 |  30.1 |  9.8 |  26.9 |  40.3
\+ FCA |  +1.9 |  30.8 |  49.8 |  31.0 |  11.8 |  29.7 |  44.6
\+ FCB(ada) |  +2.3 |  31.2 |  52.1 |  31.2 |  11.1 |  32.8 |  43.6
\+ FCB(ali) |  +2.4 |  31.3 |  50.0 |  32.8 |  10.3 |  31.7 |  43.3
\+ TF |  +5.2 |  34.1 |  51.9 |  36.0 |  11.4 |  30.3 |  46.5
|  |  |  |  |  |  |
Table 6:  Quantitative performance comparison on YouTube-VIS valid set, where a, b, c, and d respectively refer to semantic segmentation, high-solution mask refinement, temporal information, and multi-scale training. Type |  Methods |  Backbone |  Frames |  Anchors |  Others |  FPS |  |  AP |  AP50 |  AP75 |  AR1 |  AR10
---|---|---|---|---|---|---|---|---|---|---|---|---
VOS baseline |  OSMN [44] |  R50-FPN |  3 |  3 |  - |  - |  |  27.5 |  45.1 |  29.1 |  28.6 |  33.1
|  FEELVOS [37] |  R50-FPN |  3 |  3 |  - |  - |  |  26.9 |  42.0 |  29.7 |  29.9 |  33.4
|  STEm-Seg [1] |  R50-FPN |  - |  - |  - |  - |  |  30.6 |  50.7 |  33.5 |  31.6 |  37.1
|  STEm-Seg [1] |  R101-FPN |  - |  - |  - |  2.1 |  |  34.6 |  55.8 |  37.9 |  34.4 |  41.6
Two-stage methods |  MaskTrack [43] |  R50-FPN |  2 |  3 |  - |  18.4 |  |  30.3 |  51.1 |  32.6 |  31.0 |  35.5
|  MaskProp [2] |  R50-FPN |  13 |  3 |  b,c,d |  - |  |  40.0 |  - |  42.9 |  - |  -
|  MaskProp [2] |  R101-FPN |  13 |  3 |  b,c,d |  - |  |  42.5 |  - |  45.6 |  - |  -
One-stage methods |  SipMask++ [6] |  R50-DCN |  2 |  - |  - |  30.0 |  |  32.5 |  53.0 |  33.3 |  33.5 |  38.9
|  SipMask++ [6] |  R101-DCN |  2 |  - |  - |  27.8 |  |  35.0 |  56.1 |  35.2 |  36.0 |  41.2
|  STMask (ada) |  R50-DCN |  2 |  9 |  c |  28.6 |  |  33.5 |  52.1 |  36.9 |  31.1 |  39.2
|  STMask (ada) |  R101-DCN |  2 |  3 |  c |  23.4 |  |  36.8 |  56.8 |  38.0 |  34.8 |  41.8
|  STMask (ali) |  R101-DCN |  2 |  3 |  c |  22.1 |  |  36.3 |  55.2 |  39.9 |  33.7 |  42.0
|  |  |  |  |  |  |  |  |  |  |  |

Figure 6:  Segmentation visual comparison on two videos with small objects and
occlusion. The first two rows show the original frames and the instance masks
produced by MaskTrack R-CNN [43] respectively, while the last two rows are the
results obtained by our STMask with only spatial calibration and with both
spatial calibration and temporal fusion module, respectively.

Figure 7:  Segmentation visual comparison on a video with uncommon camera-to-
object view.

###  4.3 Mask Results

We compare the proposed STMask with state-of-the-arts on YouTube-VIS valid
set. The results are shown in Table 6. The ResNet50 backbone is used by the
competing methods. The speed is computed on a single 2080Ti GPU.

From Table 6, we can see that among existing fast video object segmentation
(VOS) methods, OSMN [44] with ‘track-by-detect’ and FEELVOS [37] achieve
relatively lower mask AP scores. The bottle-up method STEm-Seg with ResNet 50
and ResNet 101 backbone reaches 30.6 % and 34.6 % mask AP respectively. The
recently introduced two-stage methods MaskTrack R-CNN [43] and MaskProp [2]
obtain mask AP scores of 30.3 % and 40.0 %, respectively. However, MaskProp
requires a video clip (more than 13 frames) to process an image, resulting in
very slow speed. For the one-stage method using ResNet50 Backbone, SipMask [6]
and our proposed STMask obtain 32.5 % and 33.5 % mask AP with a speed of 30.0
and 28.6 FPS respectively. In addition, STMask with ResNet101-DCN backbone
achieves 36.8 % and 36.3 % mask AP on adaptive and aligned FCB settings,
respectively, without any other extra tricks. Although STMask has relatively
lower mask AP than MaskProp, it does achieve a better trade-off between
accuracy and speed.

Figs. 6 and 7 visualise the instance segmentation masks of our STMask on
challenging videos with small objects, occlusion and uncommon camera-to-object
view. We can see that the temporal fusion module can indeed reduce the missed
detection across frames, compared with frame-level instance segmentation.

##  5 Conclusion

We observe that one-stage instance segmentation approaches underestimate the
importance of spatial feature calibration and temporal redundancy information
between video frames for video instance segmentation. To address the issue, we
first propose a simple spatial feature calibration to detect and segment
object masks frame-by-frame, and further introduce a temporal fusion module to
track instance across video frames to effectively reduce missed instances on
challenging videos like motion blur, partial occlusion and unusual object-to-
camera poses. Overall, on YouTube-VIS valid set, our proposed STMask with
ResNet-50/-101 backbone obtains 33.5% / 36.8 % mask AP, while achieving 28.6 /
23.4 FPS on video instance segmentation, which does keep the trade-off between
accuracy and speed.

## References

  * [1] Ali Athar, Sabarinath Mahadevan, Aljosa Osep, Laura Leal-Taixé, and Bastian Leibe.  Stem-seg: Spatio-temporal embeddings for instance segmentation in videos.  In European Conference on Computer Vision, pages 158–177. Springer, 2020.
  * [2] Gedas Bertasius and Lorenzo Torresani.  Classifying, segmenting, and tracking object instances in video with mask propagation.  In IEEE Conf. Comput. Vis. Pattern Recog., pages 9739–9748, 2020\.
  * [3] Gedas Bertasius, Lorenzo Torresani, and Jianbo Shi.  Object detection in video with spatiotemporal sampling networks.  In Eur. Conf. Comput. Vis., pages 331–346, 2018.
  * [4] Daniel Bolya, Chong Zhou, Fanyi Xiao, and Yong Jae Lee.  Yolact++: Better real-time instance segmentation.  arXiv preprint arXiv:1912.06218, 2019.
  * [5] Daniel Bolya, Chong Zhou, Fanyi Xiao, and Yong Jae Lee.  Yolact: Real-time instance segmentation.  In Int. Conf. Comput. Vis., pages 9157–9166, 2019.
  * [6] Jiale Cao, Rao Muhammad Anwer, Hisham Cholakkal, Fahad Shahbaz Khan, Yanwei Pang, and Ling Shao.  SipMask: Spatial information preservation for fast image and video instance segmentation.  arXiv preprint arXiv:2007.14772, 2020.
  * [7] Hao Chen, Kunyang Sun, Zhi Tian, Chunhua Shen, Yongming Huang, and Youliang Yan.  BlendMask: Top-down meets bottom-up for instance segmentation.  In IEEE Conf. Comput. Vis. Pattern Recog., pages 8573–8581, 2020\.
  * [8] Kai Chen, Jiangmiao Pang, Jiaqi Wang, Yu Xiong, Xiaoxiao Li, Shuyang Sun, Wansen Feng, Ziwei Liu, Jianping Shi, Wanli Ouyang, et al.  Hybrid task cascade for instance segmentation.  In IEEE Conf. Comput. Vis. Pattern Recog., pages 4974–4983, 2019\.
  * [9] Liang-Chieh Chen, Alexander Hermans, George Papandreou, Florian Schroff, Peng Wang, and Hartwig Adam.  MaskLab: Instance segmentation by refining object detection with semantic and direction features.  In IEEE Conf. Comput. Vis. Pattern Recog., pages 4013–4022, 2018\.
  * [10] Yihong Chen, Yue Cao, Han Hu, and Liwei Wang.  Memory enhanced global-local aggregation for video object detection.  In IEEE Conf. Comput. Vis. Pattern Recog., pages 10337–10346, 2020\.
  * [11] Jifeng Dai, Kaiming He, and Jian Sun.  Instance-aware semantic segmentation via multi-task network cascades.  In IEEE Conf. Comput. Vis. Pattern Recog., pages 3150–3158, 2016\.
  * [12] Jifeng Dai, Haozhi Qi, Yuwen Xiong, Yi Li, Guodong Zhang, Han Hu, and Yichen Wei.  Deformable convolutional networks.  In Int. Conf. Comput. Vis., pages 764–773, 2017.
  * [13] Hanming Deng, Yang Hua, Tao Song, Zongpu Zhang, Zhengui Xue, Ruhui Ma, Neil Robertson, and Haibing Guan.  Object guided external memory network for video object detection.  In Int. Conf. Comput. Vis., pages 6678–6687, 2019.
  * [14] Jiajun Deng, Yingwei Pan, Ting Yao, Wengang Zhou, Houqiang Li, and Tao Mei.  Relation distillation networks for video object detection.  In Int. Conf. Comput. Vis., pages 7023–7032, 2019.
  * [15] Christoph Feichtenhofer, Axel Pinz, and Andrew Zisserman.  Detect to track and track to detect.  In Int. Conf. Comput. Vis., pages 3038–3046, 2017.
  * [16] Qianyu Feng, Zongxin Yang, Peike Li, Yunchao Wei, and Yi Yang.  Dual embedding learning for video instance segmentation.  In Int. Conf. Comput. Vis. Worksh., pages 0–0, 2019.
  * [17] Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik.  Rich feature hierarchies for accurate object detection and semantic segmentation.  In IEEE Conf. Comput. Vis. Pattern Recog., pages 580–587, 2014\.
  * [18] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick.  Mask R-CNN.  In Int. Conf. Comput. Vis., pages 2961–2969, 2017.
  * [19] Zhaojin Huang, Lichao Huang, Yongchao Gong, Chang Huang, and Xinggang Wang.  Mask scoring R-CNN.  In IEEE Conf. Comput. Vis. Pattern Recog., pages 6409–6418, 2019\.
  * [20] Alexander Kirillov, Evgeny Levinkov, Bjoern Andres, Bogdan Savchynskyy, and Carsten Rother.  InstanceCut: from edges to instances with multicut.  In IEEE Conf. Comput. Vis. Pattern Recog., pages 5008–5017, 2017\.
  * [21] Alexander Kirillov, Yuxin Wu, Kaiming He, and Ross Girshick.  PointRend: Image segmentation as rendering.  In IEEE Conf. Comput. Vis. Pattern Recog., pages 9799–9808, 2020\.
  * [22] Shu Kong and Charless C Fowlkes.  Recurrent pixel embedding for instance grouping.  In IEEE Conf. Comput. Vis. Pattern Recog., pages 9018–9028, 2018\.
  * [23] Shuai Li, Lingxiao Yang, Jianqiang Huang, Xian-Sheng Hua, and Lei Zhang.  Dynamic anchor feature selection for single-shot object detection.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 6609–6618, 2019.
  * [24] Yi Li, Haozhi Qi, Jifeng Dai, Xiangyang Ji, and Yichen Wei.  Fully convolutional instance-aware semantic segmentation.  In IEEE Conf. Comput. Vis. Pattern Recog., pages 2359–2367, 2017\.
  * [25] Xiaodan Liang, Liang Lin, Yunchao Wei, Xiaohui Shen, Jianchao Yang, and Shuicheng Yan.  Proposal-free network for instance-level object segmentation.  IEEE Trans. Pattern Anal. Mach. Intell., 40(12):2978–2991, 2017\.
  * [26] Chung-Ching Lin, Ying Hung, Rogerio Feris, and Linglin He.  Video instance segmentation tracking with a modified vae architecture.  In IEEE Conf. Comput. Vis. Pattern Recog., pages 13147–13157, 2020\.
  * [27] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick.  Microsoft COCO: Common objects in context.  In Eur. Conf. Comput. Vis., pages 740–755. Springer, 2014.
  * [28] Shu Liu, Lu Qi, Haifang Qin, Jianping Shi, and Jiaya Jia.  Path aggregation network for instance segmentation.  In IEEE Conf. Comput. Vis. Pattern Recog., pages 8759–8768, 2018\.
  * [29] Julien Mairal, Francis Bach, Jean Ponce, and Guillermo Sapiro.  Online dictionary learning for sparse coding.  pages 689–696, 2009.
  * [30] Davy Neven, Bert De Brabandere, Marc Proesmans, and Luc Van Gool.  Instance segmentation by jointly optimizing spatial embeddings and clustering bandwidth.  In IEEE Conf. Comput. Vis. Pattern Recog., pages 8837–8845, 2019\.
  * [31] Alejandro Newell, Zhiao Huang, and Jia Deng.  Associative embedding: End-to-end learning for joint detection and grouping.  In Adv. Neural Inform. Process. Syst., pages 2277–2287, 2017.
  * [32] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun.  Faster R-CNN: Towards real-time object detection with region proposal networks.  In Adv. Neural Inform. Process. Syst., pages 91–99, 2015.
  * [33] Mykhailo Shvets, Wei Liu, and Alexander C Berg.  Leveraging long-range temporal relationships between proposals for video object detection.  In Int. Conf. Comput. Vis., pages 9756–9764, 2019.
  * [34] Konstantin Sofiiuk, Olga Barinova, and Anton Konushin.  AdaptIS: Adaptive instance selection network.  In Int. Conf. Comput. Vis., pages 7355–7363, 2019.
  * [35] Zhi Tian, Chunhua Shen, and Hao Chen.  Conditional convolutions for instance segmentation.  arXiv preprint arXiv:2003.05664, 2020.
  * [36] Jonas Uhrig, Eike Rehder, Björn Fröhlich, Uwe Franke, and Thomas Brox.  Box2Pix: Single-shot instance segmentation by assigning pixels to object boxes.  In IEEE Intell. Veh. Symp., pages 292–299. IEEE, 2018.
  * [37] Paul Voigtlaender, Yuning Chai, Florian Schroff, Hartwig Adam, Bastian Leibe, and Liang-Chieh Chen.  FEELVOS: Fast end-to-end embedding learning for video object segmentation.  In IEEE Conf. Comput. Vis. Pattern Recog., pages 9481–9490, 2019\.
  * [38] Jiaqi Wang, Kai Chen, Shuo Yang, Chen Change Loy, and Dahua Lin.  Region proposal by guided anchoring.  In IEEE Conf. Comput. Vis. Pattern Recog., pages 2965–2974, 2019\.
  * [39] Shiyao Wang, Yucong Zhou, Junjie Yan, and Zhidong Deng.  Fully motion-aware network for video object detection.  In Eur. Conf. Comput. Vis., pages 542–557, 2018.
  * [40] Xinlong Wang, Tao Kong, Chunhua Shen, Yuning Jiang, and Lei Li.  SOLO: Segmenting objects by locations.  arXiv preprint arXiv:1912.04488, 2019.
  * [41] Fanyi Xiao and Yong Jae Lee.  Video object detection with an aligned spatial-temporal memory.  In Eur. Conf. Comput. Vis., pages 485–501, 2018.
  * [42] Enze Xie, Peize Sun, Xiaoge Song, Wenhai Wang, Xuebo Liu, Ding Liang, Chunhua Shen, and Ping Luo.  PolarMask: Single shot instance segmentation with polar representation.  In IEEE Conf. Comput. Vis. Pattern Recog., pages 12193–12202, 2020\.
  * [43] Linjie Yang, Yuchen Fan, and Ning Xu.  Video instance segmentation.  In Int. Conf. Comput. Vis., pages 5188–5197, 2019.
  * [44] Linjie Yang, Yanran Wang, Xuehan Xiong, Jianchao Yang, and Aggelos K Katsaggelos.  Efficient video object segmentation via network modulation.  In IEEE Conf. Comput. Vis. Pattern Recog., pages 6499–6507, 2018\.
  * [45] Rufeng Zhang, Zhi Tian, Chunhua Shen, Mingyu You, and Youliang Yan.  Mask encoding for single shot instance segmentation.  In IEEE Conf. Comput. Vis. Pattern Recog., pages 10226–10235, 2020\.
  * [46] Xizhou Zhu, Yujie Wang, Jifeng Dai, Lu Yuan, and Yichen Wei.  Flow-guided feature aggregation for video object detection.  In Int. Conf. Comput. Vis., pages 408–417, 2017.