# Improving Neural Response Diversity with Frequency-Aware Cross-Entropy Loss

Shaojie Jiang  University of AmsterdamAmsterdamThe Netherlands<EMAIL_ADDRESS>,  Pengjie Ren  University of AmsterdamAmsterdamThe Netherlands<EMAIL_ADDRESS>,
Christof Monz  University of AmsterdamAmsterdamThe Netherlands<EMAIL_ADDRESS>and  Maarten de Rijke  University of AmsterdamAmsterdamThe Netherlands
<EMAIL_ADDRESS>
(2019)

###### Abstract.

Sequence-to-Sequence (Seq2Seq) models have achieved encouraging performance on
the dialogue response generation task. However, existing Seq2Seq-based
response generation methods suffer from a _low-diversity problem_ : they
frequently generate generic responses, which make the conversation less
interesting. In this paper, we address the low-diversity problem by
investigating its connection with _model over-confidence_ reflected in
predicted distributions. Specifically, we first analyze the influence of the
commonly used Cross-Entropy (CE) loss function, and find that the CE loss
function prefers high-frequency tokens, which results in low-diversity
responses. We then propose a Frequency-Aware Cross-Entropy (FACE) loss
function that improves over the CE loss function by incorporating a weighting
mechanism conditioned on token frequency. Extensive experiments on benchmark
datasets show that the FACE loss function is able to substantially improve the
diversity of existing state-of-the-art Seq2Seq response generation methods, in
terms of both automatic and human evaluations.

Chatbot; Dialogue system; Sequence-to-sequence model

††journalyear: 2019††copyright: iw3c2w3††conference: Proceedings of the 2019
World Wide Web Conference; May 13–17, 2019; San Francisco, CA, USA††booktitle:
Proceedings of the 2019 World Wide Web Conference (WWW’19), May 13–17, 2019,
San Francisco, CA, USA††doi: 10.1145/3308558.3313415††isbn:
978-1-4503-6674-8/19/05††ccs: Information systems Users and interactive
retrieval

##  1\. Introduction

Recently, dialogue response generation has attracted a lot of attention due to
its potential for applications, e.g., within intelligent customer service
agents and personal assistants. Most state-of-the-art approaches to this task
are based on Sequence-to-Sequence (Seq2Seq) frameworks (Vinyals and Le, 2015;
Li et al., 2016; Sordoni et al., 2015; Serban et al., 2016, 2017; Zhao et al.,
2017). Current Seq2Seq frameworks suffer from a _low-diversity problem_. As a
result, these approaches frequently generate generic responses such as “ _I
don’t know_ ” or “ _I’m sorry_ ” (Li et al., 2016; Serban et al., 2016, 2017;
Zhao et al., 2017).

To address the issue of low response diversity, previous studies have produced
several hypotheses about the cause of the low-diversity problem with
corresponding solutions. For example, Li et al. (2016) argue that the usual
max a posteriori (MAP) objective function might favor frequent responses.
Instead, they propose to use a Maximum Mutual Information (MMI) objective
function to encourage responses with higher mutual information regarding the
user’s utterance. The downside of this method is that it relies heavily on
beam search and/or an inverse model that is trained by swapping inputs and
outputs. Tao et al. (2018) assume that the single attention layer commonly
used in Seq2Seq models is only able to focus on a single semantic aspect of
the input sequence, so they propose a Multi-Head Attention Mechanism (MHAM) to
allow the decoder to generate more diverse responses. There are also studies
that try to improve response diversity by introducing randomness (Serban et
al., 2017; Zhao et al., 2017).

Recently, Jiang and de Rijke (2018) have shown that there is a strong
connection between the low-diversity problem and what they call _model over-
confidence_ following Pereyra et al. (2017), the phenomenon that a model
incorrectly assigns most probability to only a few tokens. However, the cause
of this phenomenon remains unknown. In §2, we investigate and conclude that
the model over-confidence problem is caused by an imbalanced training of
tokens with variable frequencies, which favors frequent tokens and results in
low-diversity. To correct for this, we propose a Frequency-Aware Cross-Entropy
(FACE) loss function that improves the traditional Cross-Entropy (CE) loss
function by taking token frequency into consideration. More specifically, we
first analyze the influence of the commonly used CE loss function, and find
that it prefers high-frequency tokens, which results in model over-confidence
and low-diversity responses. Then we propose a FACE loss function that
improves over the CE loss function by incorporating a weighting mechanism
conditioned on token frequency.

The primary differences between FACE and previous studies in addressing the
low-diversity problem are two-fold:  (1) The diversity improvements brought by
FACE do not rely on beam search or randomness; and  (2) FACEdoes not introduce
new layers or hyper-parameters to Seq2Seq models.

We perform extensive experiments on two benchmark datasets, namely
OpenSubtitles database (OSDb) and Twitter. Compared with deterministic
Seq2Seq-based methods, like MMI (Li et al., 2016) and MHAM (Tao et al., 2018),
FACE achieves the highest diversity performance and it does so with minimum
modifications to the original Seq2Seq model structure and existing hyper-
parameters.111Source code for both FACE and the baselines, together with the
validation and test sets used in our experiments can be found at
https://github.com/ShaojieJiang/FACE.

The main contributions we make in this paper are:

  * •

We examine the influence of token frequency on model over-confidence and
response diversity.

  * •

We propose a Frequency-Aware Cross-Entropy (FACE) loss function to balance the
per-token loss, which alleviates model over-confidence and, hence, improves
response diversity.

  * •

We investigate two token frequency calculation methods and corresponding
frequency-based weighting mechanisms for FACE.

##  2\. Low Diversity, Model Over-confidence and Loss Imbalance

As illustrated by Jiang and de Rijke (2018), model over-confidence (Pereyra et
al., 2017) can cause Seq2Seq-based conversation models to have low response
diversity. In this section, we show that model over-confidence and low-
diversity are statistical and empirical symptoms of the same problem.
Imbalanced training is the actual underlying reason.

Table 1. Frequency ranks of leading tokens and their percentage (%). Validation rank and Test rank are for the validation and test model responses, Training rank is for the ground-truth responses in the training data. | i | the | you | we | he | it
---|---|---|---|---|---|---
Validation rank | 1 (74) |  12 (5) | 3 (5) | 4 (3) | 5 (3) | 6 (3)
Test rank | 1 (72) |  12 (7) | 3 (5) | 5 (3) | 6 (3) | 4 (2)
Training rank | 1 (14) | 7 (12) | 3 (6) | 6 (3) | 8 (3) | 4 (3)

Existing Seq2Seq-based conversation models tend to be over-confident during
prediction and place most probability on only a few tokens. We define a
_leading_ token in a response to be a token that appears first. From Table 1,
we can see that all of the most frequent leading tokens in model responses are
actually very likely to appear first in training ground-truth responses, but
the percentages of the top-ranked token “i” are much higher in the validation
and test sets. This suggests that the model is over-fitting for frequent
leading tokens. We also find that the frequency of some subsequent tokens
(e.g., _’m_ and _not_) is much higher in model responses than in the training
ground truth, as illustrated in Table 2. This is also due to model over-
fitting, since these tokens are _relatively_ more frequent compared to other
tokens. For example, given that the previous token is “i”, the frequency of
the following token “’m” is much higher than others in the training data.

To understand this phenomenon and introduce our solution, we first look into
the commonly used max a posteriori (MAP) objective function of Seq2Seq models.
Given a dataset of message-response pairs $(X,Y)$, where
$X=(x_{1},x_{2},\dots,x_{|X|})$ and $Y=(y_{1},y_{2},\dots,y_{|Y|})$ are the
input and output sequences, respectively, the goal of Seq2Seq training is to
maximize the conditional probability $P(Y|X)$. Since the decoder Recurrent
Neural Network (RNN) can only give one output at each time step $t$, and to
generate grammatical responses, we are actually maximizing token-wise
probability at each time step during training:

(1) $\textstyle\max P(Y|X)=\max\prod_{t=1}^{|Y|}P(y_{t}|y_{<t},X),$

where $y_{<t}=(y_{1},y_{2},\dots,y_{t-1})$ are tokens generated in previous
time steps. At test time, the response is generated with respect to:

(2) $\textstyle\hat{Y}=\operatorname*{arg\,max}_{Y}P(Y|X).$

In practice, we usually maximize the aforementioned conditional probability by
minimizing the prediction loss at each step $t$, which is actually the Cross-
Entropy (CE) loss:

(3)
$\textstyle\operatorname*{CE}(y_{t})=-\sum_{i=1}^{N}\delta_{i}(y_{t})\log(P(c_{i}|y_{<t},X)),$

where $(c_{1},c_{2},\dots,c_{N})$ is the search space of $y_{t}$,
$\delta_{i}(y_{t})=1$ if $y_{t}=c_{i}$ and $0$ otherwise; $P(c_{i}|\cdot)$ is
the predicted probability of candidate token $c_{i}$ and is calculated using
the softmax function:

(4)
$P(c_{i}|y_{<t},X)=\frac{\exp(f_{\theta}(h_{t-1}^{\mathit{dec}},y_{t-1},c_{i},X))}{\sum_{j=1}^{N}\exp(f_{\theta}(h_{t-1}^{\mathit{dec}},y_{t-1},c_{j},X))},$

where $f_{\theta}(\cdot)$ is a non-linear scoring function with parameters
$\theta$; $f_{\theta}(\cdot)$ takes the hidden state $h_{t-1}^{\mathit{dec}}$,
last generation $y_{t-1}$ and $X$ as inputs, and calculates a score for each
possible candidate $c_{i}$.

Table 2. Frequency ranks of example tokens and their percentage (%) in validation and test model outputs, and ground truth outputs of the training data. | . | i | ’m | not | n’t
---|---|---|---|---|---
Validation rank | 1 (11) | 2 (10) |  13 (5) |  14 (5) |  18 (3)
Test rank | 1 (10) | 21 (9) |  13 (5) |  14 (5) |  18 (3)
Training rank | 1 1(7) | 41 (3) | 35 (1) | 32 (1) | 13 (1)

During training, the loss calculated using Eq. (3) is back-propagated through
the whole network. The effect is that $f_{\theta}$ will assign higher scores
for $c_{i}=y_{t}$ in the future, so that the loss in Eq. (3) will decrease,
and meanwhile the predicted probability in Eq. (4) will increase. The ultimate
effect is to maximize the probabilities of ground-truth outputs given input
sequences, as illustrated in Eq. (1). The model is frequently penalized by a
small number of frequent tokens, as a result of which the total loss (TL) for
these tokens is higher than for less-frequent ones:

(5)
$\textstyle\operatorname*{TL}(c_{i})=\sum_{t=1}^{N_{t}}\operatorname*{CE}(y_{t}=c_{i}),$

where $t$ denotes the time step, with maximum training steps $N_{t}$:

(6) $\textstyle N_{t}=\sum_{i=1}^{N}\operatorname*{freq}(c_{i}).$

Here, $\operatorname*{freq}(c_{i})$ represents the frequency that $c_{i}$
appears in the training ground truth. Assume that the expected value
$\mathbb{E}[\operatorname*{CE}(c_{i})]$ is roughly the same for $c_{i},\forall
i\in\\{1,2,\dots,N\\}$, then tokens with a higher frequency will have a larger
total loss during training. We refer to this phenomenon as _loss imbalance_.

Due to loss imbalance, a Seq2Seq model favors frequent tokens and thus is
over-confident about them. This is especially true for the leading token as
observed in Table 1, since the decoder language model does not have a strong
effect in the beginning of prediction. When a frequent token is selected as
leading token, the search space for subsequent tokens is hugely restricted by
the language model, and this will likely result in a frequent _generic_
response.

##  3\. Frequency-Aware Cross-Entropy Loss

Now that we have identified _loss imbalance_ as a cause of low-diversity
problems, we propose to balance the total loss for each token by applying a
weight factor to TL:

(7)
$\textstyle\operatorname*{WTL}(c_{i})=w_{i}\sum_{t=1}^{N_{t}}CE(y_{t}=c_{i}),$

where $w_{i}$ is the weight corresponding to $c_{i}$. By absorbing $w_{i}$
into the CE loss function, we obtain the FACE loss function:

(8)
$\textstyle\operatorname*{FACE}(y_{t})=-\sum_{i=1}^{N}w_{i}\delta_{i}(y_{t})\log(P(c_{i}|y_{<t},X)).$

Our key solution to the low-diversity problem lies in the weight factors
$w_{i}$. Based on the analysis in §2, one straightforward way to learning
$w_{i}$ is to take advantage of the token frequency
$\operatorname*{freq}(c_{i})$, so that frequent tokens will have lower
weights. Below we propose two methods to estimate
$\operatorname*{freq}(c_{i})$: _ground-truth or GT frequency_ and _output
frequency_.

#### GT frequency: Token frequency in the ground-truth responses.

As illustrated in Eq. (6), the number of training steps $N_{t}$ equals the sum
of frequencies of tokens in the training ground-truth responses, so it is
intuitive to use these token frequencies to adjust the weight in Eq. (8).
However, during training, the model is given data sequentially in random
order. As a result, the real-time token frequency seen by the model is likely
to differ from that of the entire training data. Our solution is to calculate
the _batch_ token frequency instead:

(9)
$\textstyle\operatorname*{freq}_{b}(c_{i})=\operatorname*{freq}_{b-1}(c_{i})+\operatorname*{freq}_{o}(c_{i}),$

where $b$ is the number of training batches seen so far and $o$ represents the
newly _observed_ batch.

#### Output frequency: Token frequency in model responses.

Alternatively, we can employ a train-and-refine strategy: the responses of a
pre-trained Seq2Seq model can reflect which tokens the model is already
overfitted for; by directly penalizing those tokens with a fine-tuning
procedure, we can improve the response diversity without retraining it from
scratch. The output frequency may have a more obvious effect on improving
response diversity than the GT frequency, because diversity is directly
exhibited by model outputs.

###  3.1. Weight calculation

Given the frequency of each token, we introduce the following two methods to
calculate the weight factor.

####  3.1.1. Pre-weight.

This method derives the weight factor $w_{i}$ prior to seeing new training
examples, i.e., _pre-weight function_ :

(10) $w_{i}=a\times RF_{i}+1.$

Here we formulate it as a linear function of _relative frequency_ :
$RF_{i}=\operatorname*{freq}(c_{i})/\sum_{j}\operatorname*{freq}(c_{j})$, and
$a=-1/\max RF_{j},\forall j\in\\{1,\dots,N\\}$, is the slope, and the bias is
$1$ so that $w_{i}$ falls in $[0,1]$. We then normalize
$\\{w_{1},w_{2},\dots,w_{N}\\}$ to have a mean of $1$. The pre-weight function
can make sure that tokens with a higher $RF$ value will get lower weights. In
other words, the influence of high frequency tokens will be penalized by using
the pre-weight function.

####  3.1.2. Post-weight.

This method tries to penalize the model’s _conservativeness_ : if the output
token $y_{t}$ has a higher frequency than the ground truth $c_{i}$, which
indicates the model conservatively picked a “safe” token, then its loss will
be scaled up by $w_{i}>1$, otherwise $w_{i}=1$ due to ReLU activations:

(11)
$w_{i}=1+\frac{ReLU(\operatorname*{freq}(y_{t})-\operatorname*{freq}(c_{i}))}{\sum_{j}^{N}\operatorname*{freq}(c_{j})}.$

Since we can only apply this weighting function after obtaining the model
outputs, we refer to it as the _post-weight function_.

###  3.2. Empowered by confidence penalty

As illustrated in (Jiang and de Rijke, 2018), Confidence Penalty (CP) methods
can alleviate the low-diversity problem. In the experiments of this paper, we
also test the performance of the Confidence Penalty (CP) function (Jiang and
de Rijke, 2018):

(12) $\operatorname*{CP}(y_{t})=\operatorname*{CE}(y_{t})-\beta
H(p(y_{t}|y_{<t},X)),$

where $p(y_{t}|y_{<t},X)$ is the predicted distribution at $t$, and $H(\cdot)$
is its entropy. However, during experiments we found that the parameter
$\beta$ needs to be carefully chosen, otherwise the loss
$\operatorname*{CP}(y_{t})$ will be negative, which is counter-intuitive since
losses should be greater than 0. Instead, we propose a _parameter-free_ CP
function:

(13)
$\operatorname*{CP}\nolimits_{\mathit{free}}(y_{t})=\operatorname*{CE}(y_{t})+\frac{1}{H(p(y_{t}|y_{<t},X))}.$

FACE and CP can be easily combined to further improve response diversity. A
trivial combination is to replace the CE loss function with the
$\operatorname*{FACE}$ loss function in Eq. (12) and Eq. (13). We also propose
a CP weighting function using the entropy in Eq. (13):

(14) $w=1+\frac{1}{H(p(y_{t}|y_{<t},X))},$

where $w$ (without subscript) is assumed to be independent of $c_{i}$, which
is different from that in Eq. (10) and Eq. (11). By using $w$ as the weight of
FACE in Eq. (8), we can penalize the model confidence by adjusting the weight
of FACE.

##  4\. Experimental Setup

To prove the effectiveness of our proposed methods, we design experiments to
answer the following questions:  (Q1) Which combination of our proposed
frequency methods (GT frequency and output frequency) and weighting functions
(pre- and post-weight) performs best?  (Q2) Does FACE improve the diversity of
Seq2Seq conversation models?  (Q3) Does CP improve the diversity of Seq2Seq
conversation models?  (Q4) Does the combination of FACE and CP further improve
performance?  (Q5) Does FACE improve the response quality besides diversity?

###  4.1. Baselines and datasets

Table 3. Components of different models. Model | Greedy | BS | #Attn | Extras
---|---|---|---|---
Seq2Seq | Yes | No | 1 | No
MMI | No | Yes | 1 | Reverse222The MMI-bidi method needs a reverse model which is pre-trained using inverse training examples: (response, message).
MHAM | Yes | No | 5 | No
FACE | Yes | No | 1 | No
CP | Yes | No | 1 | No

The first baseline is a vanilla Seq2Seq model with _general_ attention (Luong
et al., 2015) as implemented within the ParlAI platform (Miller et al., 2017).
We also choose the MMI models proposed in (Li et al., 2016) and the MHAM
models proposed in (Tao et al., 2018). These models are all deterministic
methods. In Table 3, we list the main differences between our methods and the
baselines in terms of greedy decoder, beam search (BS), number of attention
layers (#Attn), and other mandatory components (Extras). To allow for a fair
comparison, we implement all our methods and baselines using ParlAI. We choose
two publicly available benchmark datasets: OSDb and Twitter, for evaluating
the baselines and our proposed methods. We follow Li et al. (2016); Tao et al.
(2018) to use short-history conversations; see below for details.

####  4.1.1. OSDb.

The OSDb dataset (Tiedemann, 2009) is an online-available corpus of movie
subtitles.333http://www.opensubtitles.org/ Here, we use the 2011 version and
set aside $\sim$60M lines, which constitutes $\sim$30M message-response pairs
for training. Following Li et al. (2016), we randomly select 2K pairs from the
IMSDB dataset (Walker et al., 2011) for validation and test sets,
respectively, and we filter out pairs whose responses or messages are shorter
than 6 tokens.

####  4.1.2. Twitter.

Twitter is a commonly used source of data in dialogue generation research. For
ease of reproducibility of the results reported in this paper, we did not
follow Sordoni et al. (2015); Li et al. (2016) who used $\sim$130M context-
message-response triples before pre-processing. Instead, we use the version
released by (Serban et al., 2017) for training, which contains $\sim$4M tweet
IDs in total and can be scraped in 3 days.444On July 11 2018, when we finished
scraping, only 2.6M IDs in total are still valid. After formatting the tweets
as _context-message-response_ triples, we have 904K training examples and we
concatenate context-message as input. For the training, validation and test
sets, we exclude all overlapping IDs. Then we restrict the sequence length in
both validation and test sets to the range [6, 18], resulting in 18,162
validation and 1,897 test triples. Furthermore, since tuning the MMI models is
quite time-consuming, we randomly select 2K triples from the validation set
for hyper-parameter selection.

###  4.2. Evaluation

####  4.2.1. Automatic evaluation.

Although reported not to correlate well with human judgments (Liu et al.,
2016), we still report BLEU scores (Papineni et al., 2002) for fair
comparisons with baselines. To evaluate response diversity, we use the _d-1_
and _d-2_ metrics proposed in (Li et al., 2016) that are calculated as the
number of distinct uni- and bigrams, divided by the total number of tokens
generated.

####  4.2.2. Human evaluation.

Following Sordoni et al. (2015), we recruit crowd-source workers to perform
pairwise qualitative comparisons. Since the conversation history of the OSDb
dataset is only a single turn, which is hard for human annotators to judge, we
choose the Twitter dataset for human evaluation, which has two-turn histories.
We randomly select 1K test examples from the Twitter test set, and paired
model responses (_FACE_ vs.  _baseline_) are shown to 3 evaluators in random
order. Evaluators are told to choose a better response in terms of _relevance_
, _interestingness_ and _grammar_. Ties are allowed. We then carry out Welch’s
t-test on the human preferences obtained in this manner.

###  4.3. Implementation details

For the OSDb and Twitter datasets, we keep the most frequent 25,000 tokens and
replace other tokens with __UNK__. For the OSDb corpus, we use a 4-layer Long
Short-Term Memory (LSTM) network for both encoder and decoder. The hidden size
(HS) and word embedding size are set to 1,000. On the smaller Twitter corpus,
we use a smaller network with a 2-layer LSTMs for encoder and decoder. The HS
and word embedding sizes are set to 512 and 200, respectively. For both
networks, we randomly initialize the model parameters from a uniform
distribution $\mathcal{U}(-\sqrt{1/\acs{HS}},\sqrt{1/\acs{HS}})$ and the word
embeddings from a normal distribution $\mathcal{N}(0,1)$. We also employ
dropout (Srivastava et al., 2014) with drop ratio $p$ = 0.1. We use Adam
(Kingma and Ba, 2015) as our optimization method. For the hyper-parameters of
the Adam optimizer, we set the learning rate $\alpha=0.001$, two momentum
parameters $\beta 1=0.9$ and $\beta 2=0.999$, respectively, and $\epsilon$ =
$10^{-8}$. We also clip gradients (Pascanu et al., 2013) to $5$ during
training to avoid gradient explosion. To speed up training and convergence, we
use mini-batches of size 256. Since the model chosen with minimum training
loss usually has very low diversity, we choose _d-1_ as the early stopping
criterion. A scheduler is used to reduce the learning rate by a factor of
$0.5$ when a _d-1_ plateau is detected with $patience=3$. $\beta$ in Eq. (12)
is set to 0.01.

##  5\. Results and Analyses

###  5.1. FACE variants

To answer Q1, we first identify the best performing variant of FACE to be used
in later experiments. In Tables 4 and 5, we list the scores for four variants
of FACE, on the OSDb and Twitter datasets, respectively: Output token
frequency & PRe-weight (_FACE-OPR_), Output token frequency & POst-weight
(_FACE-OPO_), GT frequency & PRe-weight (_FACE-GPR_) and GT frequency & POst-
weight (_FACE-GPO_).

Table 4. Performance (%) on the OSDb dataset of different variants of FACE. Highest scores in bold face. Model | d-1 | d-2 | BLEU
---|---|---|---
_FACE-OPR_ | 4.32 | 20.47 | 8.03
_FACE-OPO_ | 4.56 | 14.96 | 6.76
_FACE-GPR_ | 2.87 | 10.51 | 7.56
_FACE-GPO_ | 5.03 | 19.66 | 6.92
Table 5. Performance (%) on the Twitter dataset of different variants of FACE. Highest scores are in bold face. Model | d-1 | d-2 | BLEU
---|---|---|---
_FACE-OPR_ | 6.23 | 24.18 | 8.33
_FACE-OPO_ | 5.73 | 17.95 | 8.80
_FACE-GPR_ | 4.13 | 15.03 | 7.99
_FACE-GPO_ | 5.69 | 17.78 | 8.72

On the OSDb dataset, we can see from Table 4 that _FACE-OPR_ and _FACE-GPO_
perform better than the other two in terms of diversity scores. Similar
results are shown on the Twitter dataset in Table 5. By comparing the results
in Table 4 and 5, we find that the performance of _FACE-OPR_ is very stable.
We therefore choose _FACE-OPR_ as our primary model and refer to it as _FACE_
in the following sections. In contrast, _FACE-GPR_ performs worst on both
datasets, which indicates that GT frequency does not work well with pre-
weight. The BLEU scores in Table 4 and 5 show no obvious patterns.

It is worth noting that all variants reported in Table 4 and 5 are using the
train-and-refine strategy: first training the Seq2Seq model with CE, then
fine-tuning it with FACE. We also tried training from scratch using FACE.
However, the performance did not show consistent improvements, which is
probably because our _equal expected loss_ assumption (under Eq. (6)) is
violated in the early training stages.

###  5.2. Automatic evaluation

We turn to Q2, Q3, and Q4. The automatic evaluation results are shown in
Tables 6 and 7. _Seq2Seq_ is vanilla-Seq2Seq with _general_ attention
mechanism (Luong et al., 2015). _Seq2Seq-refine_ is a fine-tuned version of
_Seq2Seq_ with a smaller batch size of 30. _MMI-antiLM_ adjusts the _Seq2Seq_
prediction probability using the decoder language model, while _MMI-bidi_
utilizes a reverse model. Both methods use beam size 200. _MHAM_ projects
encoder hidden-states to 5 different semantic spaces, and _CMHAM_ forces those
5 spaces to be perpendicular to each other. All of our models are fine-tuned
_Seq2Seq_ models with a smaller batch size of 30. _FACE_ is fine-tuned using
the FACE function and output token frequency. _CP_ and _CP free_ are fine-
tuned using Eq. (12) and (13), respectively. _FACE-CP_ is a linear combination
of FACE and CP. _FACE-CP free_ is the multiplicative combination of FACE and
the CP weighting function in Eq. (14).

Table 6. Results (%) on the OSDb dataset. Highest scores in each column are highlighted using bold face. Model | d-1 | d-2 | BLEU
---|---|---|---
_Seq2Seq_ | 2.70 |  28.63 | 7.34
_Seq2Seq-refine_ | 3.61 | 14.16 | 7.61
_MMI-antiLM_ | 2.73 | 10.68 | 7.83
_MMI-bidi_ | 3.06 | 12.19 | 7.01
_MHAM_ | 3.03 |  29.47 | 7.13
_CMHAM_ | 4.10 | 12.92 | 6.88
_FACE_ | 4.32 | 20.47 | 8.03
_CP_ | 4.18 | 15.59 | 7.27
_CP free_ | 5.48 | 18.59 | 7.08
_FACE-CP_ | 4.63 | 18.32 | 7.89
_FACE-CP free_ | 4.69 | 18.67 | 6.98

As shown in Table 6, all the methods proposed in this paper outperform the
baselines in terms of diversity metrics on the OSDb dataset. _FACE_ achieves
the highest d-2 score (increase of 6.3%). _FACE-CP_ and _FACE-CP free_ achieve
slightly higher d-1 and lower d-2 scores than _FACE_ , which can be viewed as
the trade-off between _FACE_ and _CP_ ’s. Although the penalty strength
$\beta$ of _CP_ is carefully selected, it is interesting to see that our
hyper-parameter free method _CP free_ outperforms _CP_ by a large margin and
achieves the highest d-1 score (1.9% improvement), which demonstrates the
effectiveness of Eq. (13). _FACE_ performs best in terms of BLEU scores, which
suggests that _FACE_ can generate higher-quality responses.

Although the MMI-based and MHAM-based methods all exhibit various improvements
over _Seq2Seq_ in terms of d-1 and d-2 scores, they mostly perform worse than
_Seq2Seq-refine_. This suggests that on the OSDb dataset, carefully designed
MMI-based and MHAM-based methods are not able to outperform a fine-tuned
_Seq2Seq_ baseline.

Table 7. Results (%) on the Twitter dataset. Highest scores in each column are highlighted using bold face. Model | d-1 | d-2 | BLEU
---|---|---|---
_Seq2Seq_ | 5.87 | 17.73 | 8.77
_Seq2Seq-refine_ | 5.69 | 17.54 | 8.82
_MMI-antiLM_ | 6.23 | 18.21 | 6.51
_MMI-bidi_ | 5.42 | 15.16 | 8.20
_MHAM_ | 5.52 | 17.04 | 8.96
_CMHAM_ | 4.99 | 14.91 | 8.71
_FACE_ | 6.23 | 24.18 | 8.33
_CP_ | 5.97 | 18.15 | 8.84
_CP free_ | 6.00 | 18.67 | 8.82
_FACE-CP_ | 6.07 | 23.50 | 8.25
_FACE-CP free_ | 5.89 | 17.81 | 8.85

Figure 1. Word cloud showing top-200 frequent tokens of model responses on the
OSDb dataset. Left: the larger the font, the higher the frequency. Right: the
larger the font, the higher the weight (pre-weight used). (Best viewed in
color.)

Similar to Table 6, Table 7 shows that all our methods can increase the
diversity of _Seq2Seq_ on the Twitter dataset. _FACE_ achieves the highest d-1
score (0.4% increase) together with _MMI-antiLM_ , but the highest d-2 score
of _FACE_ (6.5% improvement) demonstrates that our method fares better.

In Table 7 however, _Seq2Seq-refine_ decreases the diversity of _Seq2Seq_ ,
indicating that fine-tuning does not help to address the low-diversity problem
on the Twitter dataset. This is probably because the diversity of _Seq2Seq_ is
already relatively high on this dataset, and there is limited space for
further improvements.

While _MMI-bidi_ performs better than _MMI-antiLM_ on the OSDb dataset, on the
Twitter dataset, however, _MMI-bidi_ performs much worse and degrades the
diversity of the _Seq2Seq_ model. The reason for this phenomenon is probably
because of the reverse model of _MMI-bidi_ : the input consisting of Twitter
triples contains two turns from different speakers (_context-message_), thus
given the _response_ only, the prediction probability of _context-message_ is
very unreliable.

Although it achieves the highest BLEU score, _MHAM_ slightly hurts diversity.
Similarly, _CMHAM_ degrades the diversity even more. Closer inspection reveals
that most of the attention weights of both methods are on the first several
tokens of the input sequence (i.e., _context_). It is unlikely to be able to
properly learn the relation between _context-message_ and _response_ by only
paying attention to the _context_.

To illustrate the effectiveness of the weighting function, we display the
frequencies and the corresponding weights of some tokens in Figure 1. From
this figure, we can see that the most frequent tokens have very small weights,
and less frequent tokens receive larger weights. Please note that the tokens
with the largest weights are in the scope of top-200 frequent tokens.

###  5.3. Human evaluation

Table 8. Results of the pairwise human evaluation (%) on the Twitter dataset. “Win”, “Lose” and “Gain” correspond to “FACE wins”, “Baseline wins” and their difference ($Win-Lose$), respectively. Highest scores are highlighted in bold face, and ${}^{*},^{**},^{***}$ symbols indicate significant improvements with $p\mbox{-}value<0.05,<0.01,<0.005$, respectively. Comparison | Win | Lose | Gain
---|---|---|---
_FACE vs Seq2Seq_ | 38.61∗∗∗ | 21.54 | 17.07
_FACE vs MMI-antiLM_ | 51.30∗∗∗ | 19.35 | 31.95
_FACE vs MMI-bidi_ | 61.91∗∗∗ | 20.92 | 40.99
_FACE vs MHAM_ | 50.93∗∗ | 42.56 | 8.37
_FACE vs CMHAM_ | 43.75∗ | 38.85 | 4.90

We now turn to Q5. Human evaluation results are reported in Table 8. We can
see that _FACE_ is significantly better than all baselines. This means that by
increasing response diversity, FACE can improve relevance and interestingness
of responses, without sacrificing grammatical accuracy. Specifically, by
penalizing frequent tokens, _FACE_ gives more opportunities to less frequent
tokens, so the interestingness is higher than _Seq2Seq_. While high-frequency
tokens may be appropriate responses in more situations than low-frequency
ones, it is for the same reason that higher frequency tokens convey less
information, thus making the responses seem generic. Therefore, by encouraging
low-frequency tokens, _FACE_ has the potential to increase relevance of
responses. By looking at the human evaluation scores in Table 8 together with
the BLEU scores in Table 7, we can re-confirm that BLEU scores do not
correlate well with human qualitative evaluation (Liu et al., 2016).

Since _FACE_ learns a language model relying entirely on training data, the
grammatical accuracy of _FACE_ is as least as good as that of _Seq2Seq_. The
examples in Table 9 show that _FACE_ can actually generate more relevant and
interesting responses. In contrast, although _MMI-antiLM_ improves response
diversity, it increases the risk of grammatical errors since it penalizes the
language model. For example, the response of _MMI-antiLM_ in the second
example of Table 9 is ungrammatical and irrelevant to the conversation
history. Similarly, _MMI-bidi_ uses a reverse model to adjust the model
outputs, in the hope that the responses can be more diverse and relevant.
However, as explained earlier, the reverse model is highly unreliable, making
many of the responses look weird, see, e.g., the third row of Table 9.

Although _MHAM_ and _CMHAM_ can improve response diversity to some extent, it
is not guaranteed that the multiple attention heads are well distributed
throughout the whole history. In fact, there are many cases where the
attention weights are on the Context part of the history, making the responses
irrelevant to the Message, such as the fourth and fifth examples in Table 9.

Table 9. Example responses from different models. History |  Responses
---|---
Context: i need 4 mo followers .. |  _FACE_ : no i do n’t think so .. i ’m watching it now ..
Message: u ever watched criminal minds suspect behavior?? |  _Seq2Seq_ : i do n’t know what to say ..
Context: dont you dare steal my idea. i call shenanigans |  _FACE_ : well then you should have said it like that .
Message: Totally not stealing. Everyone collaborated on that idea. |  _MMI-antiLM_ : come on . come on go back to you*
Context: I never watched Spongebob’s movie tho, was it good? You know Hey Arnold? |  _FACE_ : haha , i ’m watching it now , it ’s so good : d
Message: yes it is good u should watch :D haha hey arnould with footballegg face :P |  _MMI-bidi_ : oh , that
Context: perhaps a recipe is in order? I have room in my fridge… ;) |  _FACE_ : sounds good ! i ’ll be sure to check it out ! thanks for sharing the mocha cakes with the butter cakes ; )
Message: trying this one for 1st time LOL http://bit.ly/nrFMIU start from bottom recipe & work up |  _MHAM_ : i ’m sure you ’ll find a recipe for me !
Context: Don’t laugh at my picture! That’s not nice. |  _FACE_ : i ’m not a fan of the picture but it ’s not that bad , it ’s just a pic of me saying “ i ’m a girl ” haha : - )
Message: i’m laughing because you actually posted the pic. I believed you:-) lol |  _CMHAM_ : i ’m not laughing at you .

##  6\. Conclusion

In this work, we have proposed a Frequency-Aware Cross-Entropy (FACE) function
for tackling the low-diversity problem of Seq2Seq-based conversation models.
Experiments on short-history conversation datasets demonstrate that the FACE
loss function can effectively improve the diversity and quality of responses.
FACE achieves the improvements with minimum modifications to the original
Seq2Seq model, which makes it flexible to extend. We also propose a hyper-
parameter free CPfree, which exhibits better performance than the original
parameter-dependent CP.

A limitation of FACE is that the learning procedure is not as stable as CE,
which increases the difficulty of training. In future work, we would like to
investigate this phenomenon in depth. We also hope to test FACE in a long-
history conversation setting by applying FACE to hierarchical Seq2Seq (Serban
et al., 2016). Besides, we also plan to apply FACE to stochastic models
(Serban et al., 2017; Zhao et al., 2017) and examine how FACE can be used in
an adversarial dialogue generation setup (Li et al., 2019).

## Acknowledgments

This research was supported by the China Scholarship Council, Ahold Delhaize,
the Association of Universities in the Netherlands (VSNU), and the Innovation
Center for Artificial Intelligence (ICAI). All content represents the opinion
of the authors, which is not necessarily shared or endorsed by their
respective employers and/or sponsors.

## References

  * (1)
  * Jiang and de Rijke (2018) Shaojie Jiang and Maarten de Rijke. 2018.  Why are Sequence-to-Sequence Models So Dull? Understanding the Low-Diversity Problem of Chatbots. In _Proceedings of the 2018 EMNLP Workshop SCAI: The 2nd International Workshop on Search-Oriented Conversational AI_ _(SCAI ’18)_.
  * Kingma and Ba (2015) Diederik P Kingma and Jimmy Ba. 2015.  Adam: A Method for Stochastic Optimization. In _International Conference on Learning Representations_ _(ICLR ’15)_.
  * Li et al. (2016) Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. 2016.  A Diversity-Promoting Objective Function for Neural Conversation Models. In _Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ _(NAACL-HLT ’16)_. 110–119.
  * Li et al. (2019) Ziming Li, Julia Kiseleva, and Maarten de Rijke. 2019\.  Dialogue generation: From imitation learning to inverse reinforcement learning. In _AAAI 2019: 33rd AAAI Conference on Artificial Intelligence_. AAAI.
  * Liu et al. (2016) Chia-Wei Liu, Ryan Lowe, Iulian Serban, Michael Noseworthy, Laurent Charlin, and Joelle Pineau. 2016\.  How NOT To Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response Generation. In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_ _(EMNLP ’16)_. 2122–2132.
  * Luong et al. (2015) Thang Luong, Hieu Pham, and Christopher D. Manning. 2015\.  Effective Approaches to Attention-based Neural Machine Translation. In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_ _(EMNLP ’15)_. 1412–1421.
  * Miller et al. (2017) Alexander Miller, Will Feng, Dhruv Batra, Antoine Bordes, Adam Fisch, Jiasen Lu, Devi Parikh, and Jason Weston. 2017\.  ParlAI: A Dialog Research Software Platform. In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_ _(EMNLP ’17)_. 79–84.
  * Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002\.  BLEU: A Method for Automatic Evaluation of Machine Translation. In _Proceedings of the 40th Annual Meeting on Association for Computational Linguistics_ _(ACL ’02)_. 311–318.
  * Pascanu et al. (2013) Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. 2013\.  On the Difficulty of Training Recurrent Neural Networks. In _Proceedings of the 30th International Conference on Machine Learning_ _(ICML ’13)_. 1310–1318.
  * Pereyra et al. (2017) Gabriel Pereyra, George Tucker, Jan Chorowski, Łukasz Kaiser, and Geoffrey Hinton. 2017.  Regularizing Neural Networks by Penalizing Confident Output Distributions. In _International Conference on Learning Representations_ _(ICLR ’17)_.
  * Serban et al. (2016) Iulian Vlad Serban, Alessandro Sordoni, Yoshua Bengio, Aaron C Courville, and Joelle Pineau. 2016\.  Building End-To-End Dialogue Systems Using Generative Hierarchical Neural Network Models. In _Proceedings of the 30th AAAI Conference on Artificial Intelligence_ _(AAAI ’16)_. 3776–3784.
  * Serban et al. (2017) Iulian Vlad Serban, Alessandro Sordoni, Ryan Lowe, Laurent Charlin, Joelle Pineau, Aaron C Courville, and Yoshua Bengio. 2017\.  A Hierarchical Latent Variable Encoder-Decoder Model for Generating Dialogues. In _Proceedings of the 31st AAAI Conference on Artificial Intelligence_ _(AAAI ’17)_. 3295–3301.
  * Sordoni et al. (2015) Alessandro Sordoni, Michel Galley, Michael Auli, Chris Brockett, Yangfeng Ji, Margaret Mitchell, Jian-Yun Nie, Jianfeng Gao, and Bill Dolan. 2015.  A Neural Network Approach to Context-Sensitive Generation of Conversational Responses. In _Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ _(NAACL-HLT ’15)_. 196–205.
  * Srivastava et al. (2014) Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014.  Dropout: A Simple Way to Prevent Neural Networks from Overfitting.  _The Journal of Machine Learning Research_ 15, 1 (2014), 1929–1958.
  * Tao et al. (2018) Chongyang Tao, Shen Gao, Mingyue Shang, Wei Wu, Dongyan Zhao, and Rui Yan. 2018\.  Get The Point of My Utterance! Learning Towards Effective Responses with Multi-Head Attention Mechanism. In _Proceedings of the 27th International Joint Conference on Artificial Intelligence_ _(IJCAI ’18)_. 4418–4424.
  * Tiedemann (2009) Jörg Tiedemann. 2009\.  News from OPUS-A Collection of Multilingual Parallel Corpora with Tools and Interfaces. In _Recent Advances in Natural Language Processing_ _(RANLP ’09)_ , Vol. 5. 237–248.
  * Vinyals and Le (2015) Oriol Vinyals and Quoc V. Le. 2015.  A Neural Conversational Model. In _ICML Deep Learning Workshop_.
  * Walker et al. (2011) Marilyn A Walker, Ricky Grant, Jennifer Sawyer, Grace I Lin, Noah Wardrip-Fruin, and Michael Buell. 2011\.  Perceived or Not Perceived: Film Character Models for Expressive NLG. In _International Conference on Interactive Digital Storytelling_ _(ICIDS ’09)_. 109–121.
  * Zhao et al. (2017) Tiancheng Zhao, Ran Zhao, and Maxine Eskenazi. 2017.  Learning Discourse-level Diversity for Neural Dialog Models using Conditional Variational Autoencoders. In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics_ _(ACL ’17)_. 654–664.

  *[Seq2Seq]: Sequence-to-Sequence
  *[CE]: Cross-Entropy
  *[FACE]: Frequency-Aware Cross-Entropy
  *[MAP]: max a posteriori
  *[MMI]: Maximum Mutual Information
  *[MHAM]: Multi-Head Attention Mechanism
  *[OSDb]: OpenSubtitles database
  *[RNN]: Recurrent Neural Network
  *[TL]: total loss
  *[CP]: Confidence Penalty
  *[LSTM]: Long Short-Term Memory
  *[HS]: hidden size
  *[LSTMs]: Long Short-Term Memory


# Rethinking Amodal Video Segmentation from Learning Supervised Signals with
Object-centric Representation

Ke Fan1,$\dagger$, Jingshi Lei1,$\dagger$, Xuelin Qian1,$\ast$, Miaopeng Yu1,
Tianjun Xiao2,$\ast$, Tong He2
Zheng Zhang2, Yanwei Fu1
1Fudan University 2Amazon Web Service
<EMAIL_ADDRESS><EMAIL_ADDRESS>
{tianjux, htong<EMAIL_ADDRESS>$\dagger$ co-first authors; $\ast$
corresponding authors.

###### Abstract

Video amodal segmentation is a particularly challenging task in computer
vision, which requires to deduce the full shape of an object from the visible
parts of it. Recently, some studies have achieved promising performance by
using motion flow to integrate information across frames under a self-
supervised setting. However, motion flow has a clear limitation by the two
factors of moving cameras and object deformation. This paper presents a
rethinking to previous works. We particularly leverage the supervised signals
with object-centric representation in real-world scenarios. The underlying
idea is the supervision signal of the specific object and the features from
different views can mutually benefit the deduction of the full mask in any
specific frame. We thus propose an Efficient object-centric Representation
amodal Segmentation (EoRaS). Specially, beyond solely relying on supervision
signals, we design a translation module to project image features into the
Bird’s-Eye View (BEV), which introduces 3D information to improve current
feature quality. Furthermore, we propose a multi-view fusion layer based
temporal module which is equipped with a set of object slots and interacts
with features from different views by attention mechanism to fulfill
sufficient object representation completion. As a result, the full mask of the
object can be decoded from image features updated by object slots. Extensive
experiments on both real-world and synthetic benchmarks demonstrate the
superiority of our proposed method, achieving state-of-the-art performance.
Our code will be released at https://github.com/kfan21/EoRaS.

##  1 Introduction

Deep learning has demonstrated remarkable success in various computer vision
tasks. Nevertheless, neural networks are limited to learning visible patterns
in the data, and are typically challenged in reasoning about the broader and
unseen components. Currently, most researches in object detection and
segmentation tasks concentrate on enhancing the visible part’s performance,
leaving few studies on inferring occluded information. Conversely, humans
possess an innate ability to imagine and extrapolate, enabling us to easily
complete an occluded part of an image based on prior knowledge. This critical
capacity is instrumental in advanced deep learning models for real-world
scenarios, such as medical diagnosis and autonomous driving. Thereby, the
central issue addressed in this paper is the video amodal segmentation task,
which aims to deduce an object’s complete mask, whether it is partially
obscured or not.

Figure 1: Illustrations of the difference between view prior, shape prior, and
our model. While SaVos [35] draws support from the optical flow to realize the
view prior, image-level amodal segmentation algorithms typically just utilize
the shape prior brought in by the supervision signals. Consequently, they are
limited by camera motion and complicated object types, respectively. Unlike
the previous methods, beyond the mergence of those two priors, EoRaS utilizes
view prior by object-centric learning and further introduces the BEV space
where obstruction doesn’t exist, which enables our EoRaS to easily handle
complex scenarios.

Prior studies on image amodal segmentation [22, 30, 32] are over-reliance on
prior knowledge, which actually hampers the model’s generalization abilities,
resulting in limited improvements under complex circumstances. For video
amodal, Yao et al. [35] proposed that the occluded part of the current frame
may appear in other frames, and therefore, information from all frames should
be collected to fill in the occluded regions of any specific frame. While this
method achieves promising results under the self-supervised setting, it fails
when camera motion exists, as 2D warping is used to make connections within
different frames, leading to distorted signals.

This paper aims to propose a better approach for video amodal segmentation by
rethinking the importance of using supervised signals with object-centric
representation. Such object-centric representations reflect the compositional
nature of our world, and potentially facilitate supporting more complex tasks
like reasoning about relations between objects. While signals such as motion
flow and shape priors have shown promising results, they are limited by moving
cameras and complicated object types respectively. In contrast, recent
advances [6, 13, 21] in video object segmentation produce highly accurate
object masks that are less sensitive to moving cameras, making them better
suited as supervision signals. Surprisingly, such video object masks have not
been fully exploited before.

To this end, we propose a novel approach that learns video amodal segmentation
not only from observed object supervision signals in the current frame (shape
prior) but also from integrated information of object features under different
views (view prior). Our motivation is clearly shown in Fig. 1. By using visual
patterns of other views to explain away occluded object parts in the current
frame [31], our approach gets rid of optical flow and eliminates the
shortcomings of mere reliance on shape priors. Our model is highly effective,
even in complex scenarios.

In this paper, we propose a novel supervised method for the video amodal
segmentation task that leverages a multi-view fusion layer based temporal
module and a Bird’s-Eye View (BEV) feature translation network. Rather than
relying on warping the amodal prediction into the next frame using optical
flow or using shape priors alone, we enhance the current frame features by
incorporating feature information from different viewpoints and leveraging the
supervision signals simultaneously. Specifically, we first extract front-view
features from the videos using FPN50 [18]. Then, we employ a translation
network to transform these front-view features into bird’s-eye view features,
which bring in 3D information through the usage of the intrinsic matrix. In
contrast to some related work [28] extracting object-centric 3D representation
by object reconstruction, the acquisition of BEV feature is simpler, faster,
and easier to train. As each frame is equivalent to a unique view, features
from both different frames and the BEV space, which carry shape information
about the occluded part, are further utilized. We repurpose the vanilla
object-centric representations [19] – object slots to integrate those
information, which is accomplished by our novel multi-view fusion layer.
Finally, we refine the front-view features using the updated object slots
containing object information from multiple views and decode the full mask.
Compared to previous methods [35], our model can handle scenarios with 3D
viewing angle changes or complex object shapes better by leveraging shape
knowledge and integrating information across multiple views simultaneously.

To evaluate our method, we conduct extensive experiments on real-world and
synthetic amodal benchmarks. The results demonstrate that our model achieves
outstanding performance compared to comparable models and effectively
demonstrates the efficacy of our architecture.

In summary, our main contributions are listed below. (1) Our contribution lies
in formulating the video amodal segmentation task using supervised signals for
the first time. Our model efficiently learns the shape and view priors,
enabling it to handle complex scenarios with ease. (2) We propose a novel
approach to learning object-centric representations through a multi-view
fusion layer based temporal module equipped with a set of object slots, which
achieves significant improvement in the correlation of information from
different views. (3) We introduce the novel concept of bird’s-eye view
features in our amodal task, which provides front-view features with 3D
information, resulting in consistent benefits. (4) By utilizing the bird’s-eye
view generator and multi-view fusion layer based temporal module, our
algorithm achieves remarkable improvement on both real-world and synthetic
amodal benchmarks, highlighting the novelty of our approach.

##  2 Related Work

Amodal segmentation is a more challenging task than instance segmentation
because it requires predicting the full shape of occluded objects through the
visible parts. While previous literature has focused on using shape priors
effectively through multi-level coding [22], variational autoencoder [14],
shape prior memory codebook construction [32], mixing feature decoupling [16]
or Bayesian model [29], relying solely on shape priors can lead to poor
empirical performance due to distribution shifts between training data and
real scenarios. To address this issue, [35] leverages spatiotemporal
consistency and dense object motion to explain away occlusion. Although their
work has made progress in video amodal segmentation, optical flow can cause
object deformation in the presence of camera motion. In contrast, our proposed
architecture introduces a novel approach that does not require optical flow
and utilizes bird’s-eye view features to bring in 3D information that enhances
the learning of front-view features.

Figure 2: A schematic illustration of our method. The novelty of this
architecture mainly lies in the BEV translation network and the multi-view
fusion layer. The SP and SR represent the shape provider and receiver (see
Section 3.1 for detail), respectively.

Object-centric learning aims at identifying all the objects from raw input for
better understanding the complex scenes. Existing object-centric learning
methods can be categorized into unsupervised and supervised methods. While
unsupervised methods use image/scene reconstruction to extract object
representations from images/scenes [2, 19, 25], supervised methods represent
each object as a query embedding and pay much attention to obtaining a great
initialization [3, 4, 7, 9, 13, 34]. Our EoRaS is more related to the
supervised method in terms of constructing a set of learnable queries as an
information container.

BEV maps generation requires to generate semantic maps in bird’s-eye view
space. Due to a lack of high-quality annotated data, most of the early work
adopts weak supervision by utilizing stereo information [20, 21] or obtaining
pseudo label [27]. Others directly translate semantic segmentation maps from
image space into bird’s-eye view space [6, 26]. With the advent of large-scale
annotated datasets, research on supervised methods has also made some
progress. [23] and [24] respectively take advantage of dense transformer layer
and 1D sequence-to-sequence translations to learn a map representation. [1]
and [17] instead blend features from multi-camera images to construct BEV map.
In our EoRaS, the bird’s-eye view feature is utilized to integrate 3D
information into the front-view feature. To the best of our knowledge, it’s
the first attempt to incorporate the BEV translation module in the amodal
segmentation task.

##  3 Methodology

This paper focuses on the video amodal segmentation task. Specifically, given
a video sequence $\\{I_{t}\\}_{t=1}^{T}$ with $K$ objects, EoRaS aims to
predict the full mask $\\{M_{t}^{k}\\}$ of each object in all frames, where
$k$ is the index of objects. In our EoRaS, visible masks $\\{V_{t}^{k}\\}$
also serves as supervision but will not be utilized at the test phase.

###  3.1 Architecture

The overall architecture of EoRaS is shown in Figure 2. Our EoRaS is mainly
comprised of four modules: (i) the feature encoding module which extracts the
front-view feature $f_{t}^{k}$ from the input frames; (ii) the BEV translation
network which converts the front-view features into bird’s-eye view angle
$b_{t}^{k}$ using the camera intrinsic matrix $K$ and neural network; (iii)
the multi-view fusion layer based temporal module which utilizes the object
slots updated through the forward and backward streams to integrate the
feature information from different views and fulfill the completion of each
front-view feature; and (iv) the deconvolution network that estimates the full
masks and visible masks of the current frame simultaneously.

Feature Encoding Module In this module, FPN50 [18] pretrained on ImageNet [5]
is used to extract features from the input frames. These features are obtained
from a frontal perspective and capture a lot of information but will fail to
make inferences about the missing parts of the objects.

$f_{t}^{k}=FPN(I_{t}^{k})$ (1)

(a) BEV Translation Network

(b) Object Attention Layer

Figure 3: (a) A three-dimensional cuboid is built and rasterized in the camera
coordinate. For each voxel, we use the intrinsic matrix to obtain its
coordinates in the plane system, and use bilinear interpolation on the front-
view features to obtain its feature. Then, a convolutional network is used to
obtain the bev map. (b) Our object attention layer in the multi-view fusion
layer is stacked by self-attention, cross-attention, and feedforward network.
This layer is designed for shape information fusion and takes two variables as
input. We nominate the variable to be updated as shape receiver (SR) and
another one as shape provider (SP).

BEV Translation Network The features from the bird’s-eye view (BEV) are widely
used and work well in autonomous driving research. Recall that features from
different perspectives are likely to contain the missing part information and
contribute to the full mask deduction of the current frame. As obstruct
doesn’t exist in BEV space unless objects are stacked on top of each other, it
is reasonable to introduce BEV as a special perspective to promote the
completion of the front-view feature. Consider a horizontally placed camera,
for each frame, a 3D volume feature $V_{3D}$ is constructed in the camera
coordinate system. As the BEV feature generation just involves the current
frame, we omit the subscript of time $t$ and the object index $k$ for
simplicity.

Denote the camera intrinsic matrices as $K$, we first focus on a single point
$(x,y,z)$ in the camera coordinate space. By utilizing intrinsic matrices,
this point can be easily projected into the image/feature plane and we denote
its coordinate as $(u,v)$:

$\left(\begin{array}[]{c}\lambda u\\\ \lambda v\\\
\lambda\end{array}\right)=K\left(\begin{array}[]{c}x\\\ y\\\
z\end{array}\right)$ (2)

We use bilinear interpolate to obtain the feature at $(u,v)$ from the
corresponding front-view feature $f$. The obtained value at $(u,v)$ will act
as the volume feature at $(x,y,z)$.

As shown in Figure 3(a), the 3D volume in the camera coordinate will be
rasterized into a group of points $p_{ijk}=(x_{i},y_{j},z_{k})$, where $1\leq
i\leq m,1\leq j\leq n,1\leq k\leq h$ and $x_{i},y_{j},z_{k}$ are three
predefined 1D grid. $x,y,z$ represent the direction of width, depth, and
height, respectively. The value of $V_{3D}$ will be gotten by simply repeating
the above process for each point. Further, by stacking the feature of the
volume obtained from different channels together, we will get
$V_{3D}\in\mathbb{R}^{c\times m\times n\times h}$. Since our goal is to
acquire BEV features, $V_{3D}$ is rearranged to $\mathbb{R}^{ch\times m\times
n}$ and sent to a lightweight CNN for compression along height dimension:

$b_{k}^{t}=\mathrm{CNN}(V_{3D}.reshape(ch,m,n))$ (3)

Multi-view Fusion Layer based Temporal Encoder As the occluded part of a
specific view may potentially appear in other frames, we can make full use of
the information in each frame (equivalent to different perspectives) to refine
the completion of the object shape. Specially, inspired by DETR [3] and Slot
Attention [19], we would like to generate an object-centric feature utilizing
both front-view and BEV representations.

A direct method is to follow [19], which uses ConvGRU to aggregate temporal
information. However, the cost of nested recurrent slot computation to gather
the object information from each frame is expensive when processing videos.
Here, we propose a more efficient attention-based encoder architecture named
Multi-view Fusion Layer. Generally, in such a layer, three $N$-layer object
attentions which is a non-recurrent variant of slot attention are carefully
designed and closely connected. And features from different views and object
slots serve as the inputs.

In particular, as shown in Figure 3(b), each object attention layer
($\operatorname{ObjAttention}(SP,SR)$) is stacked by self-attention, cross-
attention, and feed-forward networks and serves as information fusion network.
The variable absorbing the missing shape information during the fusion process
is named shape receiver (SR), and another one is dubbed shape provider (SP) as
it offers extra shape patches. The total forward process in
$\operatorname{ObjAttention}$ is formulated as,

$\hat{SR}=SR+\operatorname{Attention}(SR,SR,SR)$ (4)
$\widetilde{SR}=\hat{SR}+\operatorname{Attention}(SP,\hat{SR},SP)$ (5)
$output=\operatorname{MLP}(\widetilde{SR})$ (6)

where $\operatorname{Attention}(K,Q,V),\operatorname{MLP}(\cdot)$ denotes
multi-head attention module and two-layer feedforward network, respectively.
And we omit all normalization layers. $\operatorname{ObjAttention}$ first
enhances the SR representation by renewing information contained in itself,
then extracts fresh properties from the SP.

On the other hand, similar to [3], a set of object slots $S_{0}\in
R^{n_{s}\times d}$ is initialized before the videos enter. $n_{s}$ denotes the
number of slots and $d$ is the feature dimension. In our model, $S_{0}$ is set
to be learnable and serves as a container that gathers shape information from
various views.

With the above preparations, we now go to the detail of our multi-view fusion
layer. For each frame, we take advantage of the $S_{t-1}$ from the last frame
which includes object shape information from previous frames, and provide it
with the fresh characters from the front-view $f_{t}^{k}$ and BEV feature
$b_{t}^{k}$ under current perspective at first:

$S_{t}^{\prime}=\operatorname{ObjAttention}(SR=S_{t-1},SP=b_{t}^{k})$ (7)
$S_{t}=\operatorname{ObjAttention}(SR=S_{t}^{\prime},SP=f_{t}^{k})$ (8)

Then, the updated slots will provide clues about the occluded part and help
complete the front-view features of the current frame by setting the front-
view features as SR in the object attention layer. Thus, we inversely enhance
the front-view feature using the object slots by:

$\hat{f}_{t}^{k}=\operatorname{ObjAttention}(SR=f_{t}^{k},SP=S_{t})$ (9)

Deconvolution Network The deconvolution network (DeConv) is served as the mask
predictor and takes the updated front-view features as input since it shares
the same perspective with the full mask to be predicted. In our experiments,
we just construct several de-convolutional layers for this module.

$\hat{M}_{t}^{k},\hat{V}_{t}^{k}=\mathrm{DeConv}(\hat{f}_{t}^{k})$ (10)

where $\hat{M}_{t}^{k}$ and $\hat{V}_{t}^{k}$ are the full and visible mask
predictions of the current frame, respectively.

Bi-directional Prediction Cold start problem exists under the above framework
since the first few frames may not be informative enough. Thus, backward
prediction is added to solve this problem. We simply concatenate the forward
and backward features, and send them to the final deconvolution network.

###  3.2 Loss Function for EoRaS

Our EoRaS is designed as an end-to-end framework and trained with the focal
loss ($\mathrm{Focal}()$) using both full mask and visible mask as supervision
signals. Note that the discard of the visible mask loss will not heavily
damage the model performance, as shown in Tab. 4. The overall loss function is

$\mathcal{L}_{full}=\sum_{t=1}^{T}\sum_{k=1}^{K}\mathrm{Focal}(\hat{M}_{t}^{k},{M}_{t}^{k})$
(11)
$\mathcal{L}_{vis}=\sum_{t=1}^{T}\sum_{k=1}^{K}\mathrm{Focal}(\hat{V}_{t}^{k},{V}_{t}^{k})$
(12)

$\mathcal{L}=\mathcal{L}_{full}+\lambda\cdot\mathcal{L}_{vis}.$ (13)

##  4 Experiments

To fully evaluate our model, we conduct extensive experiments on both real-
world and synthetic amodal segmentation benchmarks, including Movi-B, Movi-D,
and KITTI datasets, with the visualization in Fig. 4.

Figure 4: Visualization of datasets. The first and second rows show images
from the Movi-B and Movi-D, respectively. The remaining four images belong to
the KITTI.

Movi Dataset [11] is a synthetic dataset consisting of random scenes and
objects created by Kubric [11]. In our experiments, we consider two datasets
(Movi-B and Movi-D) with different objects and different levels of occlusions.
We extract the amodal information during generation of the two datasets. The
objects in Movi-B and Movi-D are from the CLEVR [15], which consists of 11
relatively regular object shapes, and Google Scanned Objects [8], which
contains 1030 realistic objects, respectively. Both datasets use the
background from Poly Haven. To create situations with serious occlusion, all
objects are set to be static and stacked closely together. Videos are created
by setting the camera to rotate around the objects. Overall, compared with
Movi-B, Movi-D has a more complex object shape and lower camera viewing angle
with more serious occlusion.

Figure 5: Qualitative comparison between our EoRaS and competitors. From top
to down, the three rows are from Movi-B, Movi-D, and KITTI, respectively.

KITTI Dataset [10] is currently the largest real-world autonomous driving
evaluation dataset. It has been widely used in many vision tasks, such as
object detection and optical flow prediction. [22] annotated some images in
KITTI with amodal information and [35] matched these images to its original
video frame. Note that since these videos are not sufficiently annotated, it
is a weakly supervised scenario. For a fair comparison, we follow the same
data split in [35]. The visible masks and object tracks are extracted by
PointTrack [33]. It is noteworthy that only the car category is annotated in
this dataset.

DATASET | METHODS | Metrics
---|---|---
mIoUfull |  mIoUocc
Movi-B | VM | 59.19 | -
Convex | 64.21 | 18.42
PCNET | 65.79 | 24.02
AISFormer | 77.34 | 43.53
SaVos-sup. | 70.72 | 33.61
BiLSTM | 77.93 | 46.21
EoRaS (Ours) | 79.22 | 47.89
Movi-D | VM | 56.92 | -
Convex | 60.18 | 16.48
PCNET | 64.35 | 27.31
AISFormer | 67.72 | 33.65
SaVos-sup. | 60.61 | 22.64
BiLSTM | 68.43 | 36.00
EoRaS (Ours) | 69.44 | 36.96
KITTI | VM | 74.75 | -
Convex | 78.62 | 8.29
PCNET | 81.58 | 17.90
AISFormer | 86.42 | 51.04
SaVos-sup. | 83.09 | 37.33
BiLSTM | 86.68 | 49.95
EoRaS (Ours) | 87.07 | 52.00
Table 1: The performance of EoRaS on real-world and synthetic video amodal
benchmarks.

###  4.1 Competitors and Settings

Competitors We compare our method with the following related methods: (1) VM
(Visible Mask), directly use the ground truth visible mask as amodal
prediction; (2) Convex, take the convex hull of the visible mask as the amodal
mask; (3) PCNET[36], a self-supervised image-level amodal completion method by
in turn recovering occlusion ordering and completing amodal masks and content;
(4) AISFormer [30], an image-level amodal segmentation model equipped with a
transformer-based mask head and achieves the new state-of-the-art recently;
(5) Savos [35], a recent state-of-the-art method in the field of self-
supervised video amodal segmentation and is modified to supervised version by
removing the 2D warping and bringing the supervised signal for fair comparison
(We also did additional experiments involving warping operation, but the
experiment results are quite inferior); (6) BiLSTM [12], a variant of our
proposed method for which we keep the same FPN50 backbone but utilize BiLSTM
to aggregate temporal information across frames.

Implementations Results on all datasets are reported in terms of mIOU metrics
for both full mask and occluded regions. Since most amodal segmentation
algorithms use the visible mask or the bounding boxes of the visible part as
model input, the estimation results of the visible area may be more confident,
and the mIOU of the occluded area can better reflect the model performance. On
all datasets, the mIOU metric of the occluded part is only computed on those
partially occluded objects. We use AdamW as optimizer with batch size 4 for 50
epochs. The learning rate is set to $1e-5$ on Movi datasets and $1e-4$ on the
KITTI dataset. Exponential learning rate decay is used where the decay rate is
0.95. The weight decay is $5e-4$. And the $\gamma$ in focal loss is set to 2.
We set $\lambda=1$, $n_{s}=8$, $N=2$ and train our model on four Tesla T4 GPUs
using PyTorch.

No. | Designs | Metrics
---|---|---
Temporal | Bi-direction | BEV |  mIoUfull |  mIoUocc
1 | $\times$ | $\times$ | $\times$ | 76.93 | 44.55
2 | ✓ | ✓ | $\times$ | 78.66 | 46.83
3 | ✓ | $\times$ | ✓ | 78.42 | 46.51
4 | ✓ | ✓ | ✓ | 79.22 | 47.89

No. | Designs | Metrics
---|---|---
Temporal | Bi-direction | BEV |  mIoUfull |  mIoUocc
5 | $\times$ | $\times$ | $\times$ | 66.70 | 33.42
6 | ✓ | ✓ | $\times$ | 69.08 | 36.39
7 | ✓ | $\times$ | ✓ | 68.56 | 35.54
8 | ✓ | ✓ | ✓ | 69.44 | 36.96

Table 2: Ablation study of our temporal and bev modules on Movi-B (left) and
Movi-D (right) dataset.

###  4.2 Results on Movi Datasets

As shown in Table 1, compared with supervised SaVos, our EoRaS achieves
extremely significant performance improvements on both Movi datasets. In
particular, by applying our algorithm, the prediction of the full mask of the
objects in the two datasets is improved by 8.50% and 8.83%, respectively. The
improvements are more remarkable in the prediction of occluded parts. For the
performance on the occluded part, our EoRaS achieves 14.28% improvement on the
Movi-B over the baseline SaVos, and surprisingly improves by 14.32% on the
Movi-D. Moreover, the performance of EoRaS also exceeds the recent state-of-
the-art image-level algorithm AISFormer by a clear margin on both datasets.
And it’s noteworthy that EoRaS outperforms the combination of FPN50 and
BiLSTM/Transformer by at least 1% in plenty of experiments, showing the
effectiveness of introducing the BEV module. Additionally, despite the usage
of ground truth visible mask in Convex and PCNET, EoRaS still exhibits amazing
power. We also present the qualitative results in Figure 5. Obviously, the
full masks deduced by EoRaS are the closest to the original object shape among
all the competitors. Above all, EoRaS is more suitable for solving the video
amodal segmentation task, and leads to the new state-of-the-art.

Figure 6: Visualizations derived from models with/without proposed temporal or
BEV module. Clearly, the complete model deduces the best full mask, and both
the temporal and BEV module can bring consistent benefits.

###  4.3 Results on KITTI Dataset

The experiment results on the KITTI dataset are shown in Table 1. For objects
in real scenes, our EoRaS can still exceed all the current state-of-the-art
methods. Compared with the image-level baseline, we achieve 0.65% and 0.96%
improvement for the full and occluded mask prediction, respectively. For the
supervised SaVos, EoRaS achieves enormous promotion, $\sim$4% on the full
shape and $\sim$15% on the missing part. Furthermore, other video-level
baselines consistently underperform our EoRaS by $\sim$2% on the deduction of
the occluded part. Qualitative comparison in Figure 5 clearly exhibits the
great precision of EoRaS. The above evidence is sufficient enough to prove the
effectiveness of EoRaS under weakly supervised settings.

Dataset | # Slots |  mIoUfull |  mIoUocc
---|---|---|---
Movi-B | 8 | 79.22 | 47.89
16 | 79.22 | 47.79
32 | 79.29 | 47.88
64 | 79.22 | 47.78
128 | 79.19 | 47.73
256 | 79.20 | 47.75
Movi-D | 8 | 69.44 | 36.96
16 | 69.42 | 36.92
32 | 69.50 | 37.27
64 | 69.38 | 37.01
128 | 69.47 | 37.02
256 | 69.45 | 37.06
Table 3: Sensitivity Analysis of Slot Number. Despite the diverse settings of
slot number, the performance of EoRaS just changes slightly, demonstrating the
robustness against hyper-parameter $n_{s}$.

##  5 Further Analysis

Effectiveness of Temporal and BEV Modules As shown in Table 2, on Movi-B
dataset, our temporal module brings about $\sim$2.3% performance improvement
in occluded part prediction. After plugging in the BEV module, the occluded
mIOU is further improved by 1.06%. Additionally, Bi-direction prediction also
plays an important role in our model as it brings 1.38% performance
improvement for the missing part deduction. On the Movi-D dataset, the
improvements brought in by those modules are also significant, as presented in
the right table. Some visualizations derived from different architectures are
presented in Figure 6. It’s clear that both temporal and BEV modules are
capable of improving the smoothness and shape similarity of full masks. These
experiments fully prove the effectiveness of the modules proposed in this
paper, and also verify the correctness of our hypothesis that feature
information from different perspectives can benefit the completion of object
shape in any specific frame/view.

Sensitivity Analysis of Slot Number To analyze the sensitivity to the choice
of $n_{s}$, we conduct experiments by widely tuning the slot number. The
results are presented in Table 3 and indicate that the number of slots has
almost no impact on the performance of our model. This phenomenon demonstrates
the robustness of our model against the diverse choices of slot numbers.

Dataset | $\lambda$ |  mIoUfull |  mIoUocc
---|---|---|---
Movi-B | 0.0 | 78.93 | 47.55
0.25 | 79.14 | 47.80
0.5 | 79.21 | 47.90
0.75 | 79.20 | 47.85
1.0 | 79.22 | 47.89
Movi-D | 0.0 | 68.68 | 36.39
0.25 | 69.26 | 37.06
0.5 | 69.42 | 36.99
0.75 | 69.38 | 36.96
1.0 | 69.44 | 36.96
Table 4: Performance of EoRaS under different $\lambda$.  Methods | Target | Metrics
---|---|---
mIoUfull |  mIoUocc
AISFormer [30] | Movi-D | 62.94 | 28.65
KITTI | 71.36 | 29.84
SaVos-Sup. [35] | Movi-D | 57.19 | 25.85
KITTI | 65.49 | 21.82
EoRaS (Ours) | Movi-D | 63.98 | 31.22
KITTI | 71.73 | 31.35
Table 5: Open set segmentation on Movi-D and KITTI datasets. We use EoRaS pretrained on the Movi-B dataset and conduct transfer learning experiments without finetuning. Our EoRaS achieves the highest performance, indicating its great generalization ability.  Dataset | Methods | Metrics
---|---|---
mIoUfull |  mIoUocc
Movi-B | EoRaS | 79.22 | 47.89
+PP∗ | 79.38 | 47.66
+PP | 81.20 | 47.89
+SG | 81.76 | 49.39
Movi-D | EoRaS | 69.44 | 36.96
+PP∗ | 69.95 | 36.81
+PP | 72.76 | 36.96
+SG | 74.10 | 38.33
Table 6: The performance of EoRaS while using GTVM at test phase on Movi
dataset. PP∗ and PP means the predicted and ground truth visible mask are used
in post-process, respectively. And SG represents the model trained with the
concatenation of images and visible masks.

Different choices of $\lambda$ We conduct experiments to analyze the effect of
$\lambda$ on the performance of our model, and the results are presented in
Table 4. First of all, the utilization of visible masks in supervision signals
will benefit the model training as also shown in previous amodal segmentation
algorithms. But the way that EoRaS differs lies in the insensitivity to the
choice of $\lambda$ once the visible mask is added, which demonstrates the
superiority of EoRaS.

Open Set Segmentation To evaluate the capacity of out-of-distribution
generalization, we conduct open set segmentation experiments on Movi-D and
KITTI datasets. Models are pretrained on the relatively simple Movi-B dataset.
As presented in Table 5, EoRaS achieves the best accuracy among all
competitors. Concretely, compared with supervised SaVos, EoRaS outperforms by
at least 6%, showing strong dominance. Again, the image-level SOTA algorithm
underperforms EoRaS by $\sim$2% on the occluded part deduction, indicating
that the integration of information from different views indeed benefits the
generalization ability.

Test-time Assistance by Ground Truth Visible Mask (GTVM) The same as SaVos and
PCNET, we explore the utilization of GTVM at the test phase. On the one hand,
the post-processing (PP), including taking the intersection of the predicted
full mask and GTVM, is feasible. On the other hand, containing partial shape
information, GTVM may be capable of serving as a shape guidance (SG) for mask
completion. To this end, we simply train our model with the concatenation of
images and visible masks. The experimental results are presented in Table 6.
Overall, the introduction of GTVM brings in huge benefits, which is inline
with [35, 36]. Despite the usage of GTVM in those algorithms, our EoRaS still
outperforms them by a large margin (see Table 1), suggesting the powerful
function.

##  6 Conclusion

In this paper, we proposed a brand-new pipeline named EoRaS to cope with the
video amodal segmentation task. Based on the assumption that both the
supervision signals (shape prior) and the features from different perspectives
(view prior) will benefit the deduction of the full mask under any specific
view, the multi-view fusion layer based temporal encoder and BEV translation
network are designed to integrate 3D information and front-view shape patches
from different frames respectively in an object-centric pattern. Utilizing
those modules, our EoRaS eliminates the optical flow usage and the over-
reliance on shape priors, achieving high efficiency even in complex scenarios.
We conduct experiments on both real-world and synthetic video amodal
benchmarks, including Movi-B, Movi-D, and KITTI datasets. The empirical
results demonstrate that our EoRaS achieves the new state-of-the-art
performance.

Acknowledgements. This work is supported by China Postdoctoral Science
Foundation (2022M710746). Yanwei Fu is with the School of Data Science,
Shanghai Key Lab of Intelligent Information Processing, Fudan University, and
Fudan ISTBI—ZJNU Algorithm Centre for Brain-inspired Intelligence, Zhejiang
Normal University, Jinhua, China.

## References

  * [1] Maxim Berman, Amal Rannen Triki, and Matthew B Blaschko.  The lovász-softmax loss: A tractable surrogate for the optimization of the intersection-over-union measure in neural networks.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4413–4421, 2018.
  * [2] Christopher P Burgess, Loic Matthey, Nicholas Watters, Rishabh Kabra, Irina Higgins, Matt Botvinick, and Alexander Lerchner.  Monet: Unsupervised scene decomposition and representation.  arXiv preprint arXiv:1901.11390, 2019.
  * [3] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko.  End-to-end object detection with transformers.  In European conference on computer vision, pages 213–229. Springer, 2020.
  * [4] Bowen Cheng, Ishan Misra, Alexander G Schwing, Alexander Kirillov, and Rohit Girdhar.  Masked-attention mask transformer for universal image segmentation.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1290–1299, 2022.
  * [5] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee, 2009.
  * [6] Liuyuan Deng, Ming Yang, Hao Li, Tianyi Li, Bing Hu, and Chunxiang Wang.  Restricted deformable convolution-based road scene semantic segmentation using surround view cameras.  IEEE Transactions on Intelligent Transportation Systems, 21(10):4350–4362, 2019.
  * [7] Bin Dong, Fangao Zeng, Tiancai Wang, Xiangyu Zhang, and Yichen Wei.  Solq: Segmenting objects by learning queries.  Advances in Neural Information Processing Systems, 34:21898–21909, 2021.
  * [8] Laura Downs, Anthony Francis, Nate Koenig, Brandon Kinman, Ryan Hickman, Krista Reymann, Thomas B McHugh, and Vincent Vanhoucke.  Google scanned objects: A high-quality dataset of 3d scanned household items.  In 2022 International Conference on Robotics and Automation (ICRA), pages 2553–2560. IEEE, 2022.
  * [9] Yuxin Fang, Shusheng Yang, Xinggang Wang, Yu Li, Chen Fang, Ying Shan, Bin Feng, and Wenyu Liu.  Instances as queries.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 6910–6919, 2021.
  * [10] Andreas Geiger, Philip Lenz, Christoph Stiller, and Raquel Urtasun.  Vision meets robotics: The kitti dataset.  International Journal of Robotics Research (IJRR), 2013.
  * [11] Klaus Greff, Francois Belletti, Lucas Beyer, Carl Doersch, Yilun Du, Daniel Duckworth, David J Fleet, Dan Gnanapragasam, Florian Golemo, Charles Herrmann, Thomas Kipf, Abhijit Kundu, Dmitry Lagun, Issam Laradji, Hsueh-Ti (Derek) Liu, Henning Meyer, Yishu Miao, Derek Nowrouzezahrai, Cengiz Oztireli, Etienne Pot, Noha Radwan, Daniel Rebain, Sara Sabour, Mehdi S. M. Sajjadi, Matan Sela, Vincent Sitzmann, Austin Stone, Deqing Sun, Suhani Vora, Ziyu Wang, Tianhao Wu, Kwang Moo Yi, Fangcheng Zhong, and Andrea Tagliasacchi.  Kubric: a scalable dataset generator.  2022\.
  * [12] Sepp Hochreiter and Jürgen Schmidhuber.  Long short-term memory.  Neural computation, 9(8):1735–1780, 1997.
  * [13] Jie Hu, Liujuan Cao, Yao Lu, ShengChuan Zhang, Yan Wang, Ke Li, Feiyue Huang, Ling Shao, and Rongrong Ji.  Istr: End-to-end instance segmentation with transformers.  arXiv preprint arXiv:2105.00637, 2021.
  * [14] Won-Dong Jang, Donglai Wei, Xingxuan Zhang, Brian Leahy, Helen Yang, James Tompkin, Dalit Ben-Yosef, Daniel Needleman, and Hanspeter Pfister.  Learning vector quantized shape code for amodal blastomere instance segmentation.  arXiv preprint arXiv:2012.00985, 2020.
  * [15] Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick.  Clevr: A diagnostic dataset for compositional language and elementary visual reasoning.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2901–2910, 2017.
  * [16] Lei Ke, Yu-Wing Tai, and Chi-Keung Tang.  Deep occlusion-aware instance segmentation with overlapping bilayers.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4019–4028, 2021.
  * [17] Zhiqi Li, Wenhai Wang, Hongyang Li, Enze Xie, Chonghao Sima, Tong Lu, Yu Qiao, and Jifeng Dai.  Bevformer: Learning bird’s-eye-view representation from multi-camera images via spatiotemporal transformers.  In European conference on computer vision, pages 1–18. Springer, 2022.
  * [18] Tsung-Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie.  Feature pyramid networks for object detection.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2117–2125, 2017.
  * [19] Francesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexey Dosovitskiy, and Thomas Kipf.  Object-centric learning with slot attention.  Advances in Neural Information Processing Systems, 33:11525–11538, 2020.
  * [20] Chenyang Lu, Marinus Jacobus Gerardus van de Molengraft, and Gijs Dubbelman.  Monocular semantic occupancy grid mapping with convolutional variational encoder–decoder networks.  IEEE Robotics and Automation Letters, 4(2):445–452, 2019.
  * [21] Bowen Pan, Jiankai Sun, Ho Yin Tiga Leung, Alex Andonian, and Bolei Zhou.  Cross-view semantic segmentation for sensing surroundings.  IEEE Robotics and Automation Letters, 5(3):4867–4873, 2020.
  * [22] Lu Qi, Li Jiang, Shu Liu, Xiaoyong Shen, and Jiaya Jia.  Amodal instance segmentation with kins dataset.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3014–3023, 2019.
  * [23] Thomas Roddick and Roberto Cipolla.  Predicting semantic map representations from images using pyramid occupancy networks.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11138–11147, 2020.
  * [24] Avishkar Saha, Oscar Mendez, Chris Russell, and Richard Bowden.  Translating images into maps.  In 2022 International Conference on Robotics and Automation (ICRA), pages 9200–9206. IEEE, 2022.
  * [25] Mehdi SM Sajjadi, Daniel Duckworth, Aravindh Mahendran, Sjoerd van Steenkiste, Filip Pavetić, Mario Lučić, Leonidas J Guibas, Klaus Greff, and Thomas Kipf.  Object scene representation transformer.  arXiv preprint arXiv:2206.06922, 2022.
  * [26] Timo Sämann, Karl Amende, Stefan Milz, Christian Witt, Martin Simon, and Johannes Petzold.  Efficient semantic segmentation for visual bird’s-eye view interpretation.  In International Conference on Intelligent Autonomous Systems, pages 679–688. Springer, 2018.
  * [27] Samuel Schulter, Menghua Zhai, Nathan Jacobs, and Manmohan Chandraker.  Learning to look around objects for top-view representations of outdoor scenes.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 787–802, 2018.
  * [28] Prafull Sharma, Ayush Tewari, Yilun Du, Sergey Zakharov, Rares Ambrus, Adrien Gaidon, William T Freeman, Fredo Durand, Joshua B Tenenbaum, and Vincent Sitzmann.  Seeing 3d objects in a single image via self-supervised static-dynamic disentanglement.  arXiv preprint arXiv:2207.11232, 2022.
  * [29] Yihong Sun and Adam Kortylewski.  Amodal segmentation through out-of-task and out-of-distribution generalization with a bayesian model. cvpr. 2022.  In IEEE Conference on Computer Vision and Pattern Recognition, 2022.
  * [30] Minh Tran, Khoa Vo, Kashu Yamazaki, Arthur Fernandes, Michael Kidd, and Ngan Le.  Aisformer: Amodal instance segmentation with transformer.  arXiv preprint arXiv:2210.06323, 2022.
  * [31] ZHUOWEN TU, XIANGRONG CHEN, Alan Yuille, and Song-Chun Zhu.  Image parsing: Unifying segmentation, detection, and recognition.  In IJCV, 2015.
  * [32] Yuting Xiao, Yanyu Xu, Ziming Zhong, Weixin Luo, Jiawei Li, and Shenghua Gao.  Amodal segmentation based on visible region segmentation and shape prior.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 2995–3003, 2021.
  * [33] Zhenbo Xu, Wei Zhang, Xiao Tan, Wei Yang, Huan Huang, Shilei Wen, Errui Ding, and Liusheng Huang.  Segment as points for efficient online multi-object tracking and segmentation.  In Proceedings of the European Conference on Computer Vision (ECCV), 2020.
  * [34] Bin Yan, Yi Jiang, Jiannan Wu, Dong Wang, Ping Luo, Zehuan Yuan, and Huchuan Lu.  Universal instance perception as object discovery and retrieval.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15325–15336, 2023.
  * [35] Jian Yao, Yuxin Hong, Chiyu Wang, Tianjun Xiao, Tong He, Francesco Locatello, David Wipf, Yanwei Fu, and Zheng Zhang.  Self-supervised amodal video object segmentation.  arXiv preprint arXiv:2210.12733, 2022.
  * [36] Xiaohang Zhan, Xingang Pan, Bo Dai, Ziwei Liu, Dahua Lin, and Chen Change Loy.  Self-supervised scene de-occlusion.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3784–3792, 2020.

##  Appendix A Qualitative comparison between EoRaS and competitors

In the supplementary part, we show some qualitative comparisons of our model
and competitors.

###  A.1 Qualitative comparison in Movi-B and Movi-D

Figure 7 provides a comprehensive qualitative comparison between our EoRaS and
competitors across two datasets, Movi-B and Movi-D. The left column displays
images from Movi-B, and the right column displays images from Movi-D, with the
numbers in the upper-left corner indicating the source frame of each image.
For example, 17-3 indicates this image is from the $3^{rd}$ frame of the
$17^{th}$ video. Notably, we also highlight the objects with the largest
predicted mask difference by framing them for ease of comparison.

When analyzing the images from Movi-B, our model outperforms competitors in
many cases. For example, in the first image (17-3), our prediction for the
green cylinder is superior to those of our competitors. Specifically,
AISFormer predicts a full mask that extends beyond the ground truth, while
SaVos predicts an incomplete mask. In the last image (26-11), only our EoRaS
model accurately predicts the spout of the teapot.

Examining the Movi-D dataset, we note that AISFormer over-completes the
predictions for the objects in the first two images (4-1 and 4-5), while SaVos
delivers incomplete masks. Conversely, EoRaS accurately predicts the full
masks of the books in the third (34-13) and fourth (34-19) images, while
AISFormer and SaVos provide incomplete masks.

Figure 7:  Qualitative comparison between our EoRaS and competitors in the
Movi-B and Movi-D datasets. The images in the left column are from Movi-B, and
those in the right column are from Movi-D. The numbers in each upper-left
corner indicate where these images come from. For example, 17-3 indicates this
image is from the $3^{rd}$ frame of the $17^{th}$ video. For convenience, we
also put frames on those objects with the largest predicted mask difference.

###  A.2 Qualitative comparison in KITTI

In addition, we showcase the performance of our EoRaS model in the KITTI
dataset (Figure 8). Given the sparsely annotated nature of the KITTI dataset,
only a few frames have annotations, with no full ground truth masks available
for the selected images. Nevertheless, we observe that our model outperforms
competitors in certain cases. In the upper-right image (22-160), AISFormer
gives a weirdly shaped mask, while SaVos gives an over-completed mask.

To add that, we also noticed that for the cases when there is no occlusion in
front of one object, EoRaS can give a more accurate mask than our competitors,
as shown in the yellow mask of the lower-right image (22-402), which further
shows the robustness of our model.

Overall, the results presented in Figure 7 and Figure 8 suggest that our EoRaS
model outperforms competitors in terms of accuracy, completeness, and
robustness across various datasets.

Figure 8:  Qualitative comparison between our EoRaS and competitors in the
KITTI dataset. The numbers in each upper-left corner indicate where these
images come from. For example, 22-142 indicates this image is from the
$142^{nd}$ frame of the $22^{nd}$ video. For convenience, we also put frames
on those objects with the largest predicted mask difference. Due to the sparse
labeling of the KITTI dataset, many images do not have ground-truth full
masks.