# Automatic Portrait Video Matting via Context Motion Network

Qiqi Hou1 , Charlie Wang 2 This project is done during Qiqi Hou’s internship
at Intel.

# Automatic Portrait Video Matting via Context Motion Network

Qiqi Hou1 , Charlie Wang 2 This project is done during Qiqi Hou’s internship
at Intel.

###### Abstract

Automatic portrait video matting is an under-constrained problem. Most state-
of-the-art methods only exploit the semantic information and process each
frame individually. Their performance is compromised due to the lack of
temporal information between the frames. To solve this problem, we propose the
context motion network to leverage semantic information and motion
information. To capture the motion information, we estimate the optical flow
and design a context-motion updating operator to integrate features between
frames recurrently. Our experiments show that our network outperforms state-
of-the-art matting methods significantly on the Video240K SD dataset.

## Introduction

Portrait video matting aims to recover the foreground portraits from a
portrait video. It has attracted extensive interest due to its wide
applications in visual effects, video editing, and video meeting. It is an
under-constrained problem.

To solve the matting problem, most existing methods rely on extra inputs to
provide external information about the foreground or background areas. Trimap-
based methods (Xu et al. 2017; Hou and Liu 2019; Lu et al. 2019; Li and Lu
2020; Forte and Pitié 2020) require a trimap image as the extra input, where
users specify known foreground and background areas, as well as an undefined
area. These methods recover alpha matte values in the undefined area by
propagating the alpha values from known area. These methods achieve promising
performance. However, obtaining accurate trimaps often requires a vast amount
of user effort. Alternatively, background-based methods (Sengupta et al. 2020;
Lin et al. 2020) assume that the background is relatively static, and users
can capture the background. The matting problem is reduced to estimate the
matte values with a known background. These methods adopt an end-to-end
network to estimate the alpha values directly and produce high-quality
results. However, it is challenging to capture background for videos with
moving backgrounds, which impedes their applications to more general
scenarios.

Our research is inspired by recent portrait image matting without any extra
inputs (Shen et al. 2016; Zhu et al. 2017). These methods incorporate the
semantic information into the matting network to automatically estimate the
alpha maps for portraits. For instance, Shen et al.  (Shen et al. 2016)
employs a FCN (Long, Shelhamer, and Darrell 2015) as a part of the whole
network to estimate the trimap. They estimate the matte values based on the
trimap and achieve promising results. However, these methods are designed for
images. Their performance on videos might be compromised due to the lack of
temporal information. For instance, as shown in Figure LABEL:fig::figone, the
hand in the semantic segmentation is inaccurate, while the optical flow can
provide useful temporal information for the hand. Furthermore, adopting these
methods to each frame individually sometimes leads to undesirable temporal
incoherence, such as flickering.

Our idea for portrait video matting is to make use of not only semantic
information but also motion information from optical flow. There are several
advantages of introducing motion information to the portrait video matting.
First, the motions of foreground and background are typically different. It
will be easier to find the edges between foreground and background. Second, if
the foreground moves across the background, some portions of the background
will be covered or revealed. We can construct a more complete background to
matte against.

We accordingly developed a context motion network dedicated to portrait video
matting. Our network takes a sequence of frames as inputs and estimates the
alpha maps and foregrounds for each frame. Specifically, our network contains
four parts: the feature extraction network, the optical flow estimator, the
context motion updating operator, and the upsampler. We first extract the
context features for each frame and estimate the optical flow between
consecutive frames. The optical flow and context features are fed into the
context motion updating operator. To get the motion features, the context
motion updating operator backwarps features from neighboring frames using
optical flow and calculate the correlation between the current features and
the backwarped features. Our operator also encodes the optical flow directly.
We employ ConvGRUs to recurrently integrate features from consecutive frames.
We then feed the resulting features to upsamplers to predict alpha maps and
foreground. Our network is trained in an end-to-end manner. Our experiments
show that our method outperforms the state-of-the-art trimap-based and
background-based methods significantly.

This paper contributes to the research on automatic portrait video matting by
leveraging semantic information and temporal information. First, we propose a
novel context motion network. It is the first deep learning based method to
utilize optical flow to capture the motion information in the automatic
portrait video matting. Second, we design a context motion updating operator
to fuse the context and temporal information and recurrently update features
across consecutive frames. Third, the proposed method achieves state-of-the-
art performances compared to the trimap-based and background-based methods.

## Related Work

Matting is a classic computer vision problem. For an image $\mathbf{I}$, it
assumes that $\mathbf{I}$ is a linear composition of a foreground $\mathbf{F}$
and a background $\mathbf{B}$ based on an alpha map
$\boldsymbol{\alpha}\in[0,1]$,

$\mathbf{I}=\boldsymbol{\alpha}\mathbf{F}+(1-\boldsymbol{\alpha})\mathbf{B},$
(1)

where $\mathbf{F},\mathbf{B}$, and $\boldsymbol{\alpha}$ are unknown. Matting
aims to recover the foreground object, which is an under-constrained problem.
In this part, we discuss the most relevant works on image matting and video
matting. Due to the space limit, we refer readers to a comprehensive survey
article Wang et.al. (Wang, Cohen et al. 2008).

Image matting. To solve the matting problem, most of the current methods rely
on the extra inputs for external information. Trimap-based methods require a
user-provided trimap or stokes which specify the foreground, background, and
unknown areas. To recover the matte values in the unknown area, there are
three categories of trimap-based matting methods: color sampling methods,
propagation methods, and deep learning methods. Color sampling methods (Chuang
et al. 2001; Wang and Cohen 2005; He et al. 2011) are based on the observation
that neighboring pixels’ alpha values are often similar if their colors are
similar. However, the performance of color sampling methods declines sharply
when the color distributions of foreground and background have a large
overlap. Propagation methods (Aksoy, Ozan Aydin, and Pollefeys 2017; Chen, Li,
and Tang 2013; Chen et al. 2013; He, Sun, and Tang 2010; Lee and Wu 2011;
Levin, Lischinski, and Weiss 2008; Levin, Rav-Acha, and Lischinski 2008; Sun
et al. 2004; Levin, Lischinski, and Weiss 2008; Aksoy, Ozan Aydin, and
Pollefeys 2017; Chen, Li, and Tang 2013) propagate alpha values from known to
unknown pixels through various affinities between neighboring pixels. However,
it is challenging to obtain accurate affinities because of color ambiguity.
Deep learning methods use a CNN and estimate the alpha map from the source
images (Shen et al. 2016; Xu et al. 2017; Cho, Tai, and Kweon 2016, 2019; Hou
and Liu 2019; Lu et al. 2019; Tang et al. 2019; Li and Lu 2020) or finetune
the results from traditional methods (Cho, Tai, and Kweon 2016, 2019). Instead
of using hand-crafted features, these methods learned the features which are
more robust to the challenging scenarios. These methods achieved promising
results. However, trimap-based methods require an amount of user effort in
creating the trimap. The extensive requirements of accurate trimaps pose a
challenge, especially when running these methods on videos.

Alternatively, background-based methods (Sengupta et al. 2020; Lin et al.
2020) assume that the background image is relatively static, and users can
capture the background image. The matting problem is reduced to estimate the
foreground objects with a known background. These methods take the image as
well as the background image as inputs and estimate the alpha mattes with an
end-to-end trained network, which have shown particularly effective and high
efficiency when the background is relatively static. However, their
performance drops sharply when it has a large mismatch between the user-
captured background and the real background. Furthermore, for the videos with
a moving background, it is difficult to capture the accurate background.

The semantic-based portrait matting methods (Shen et al. 2016; Zhu et al.
2017) leverage the semantic information to estimate the matte values for the
portrait matting without any user interaction. For instance, Shen et al.
accordingly proposed an end-to-end trained network (Shen et al. 2016). It
first estimated the trimap. Then they designed a matting layer to estimate the
matte values based on the trimap. Their method achieved promising results for
images. Compared to these methods, our method improves the accuracy of
automatic matting with temporal information.

Recently, some methods (Li, Xu, and Lu 2020; Qiao et al. 2020) estimate the
matte values for any objects without any extra inputs. However, the
generalization capability of these methods is compromised.

Video matting. Most traditional video matting methods solve the video matting
problem in two steps (Chuang et al. 2002; Apostoloff and Fitzgibbon 2004; Lee,
Yoon, and Lee 2010; Bai, Wang, and Simons 2011). In the first step, these
methods employ semantic segmentation methods to estimate a binary segmentation
image of each frame independently. Based on the binary segmentation, these
methods generate trimaps for each frame by performing spatio-temporal
optimizations. In the second step, these methods apply image matting
algorithms to estimate the matte values for each frame using trimaps generated
in the first step. While these methods have achieved promising results for
some scenes, the quality and stability of predictions from these methods
decline when they are applied to challenging scenarios because these methods
heavily rely on hand-crafted features. Our method addresses this challenging
problem by learning the features with an end-to-end trained network.

Recently, there are deep-learning based methods for video matting (Ke et al.
2020; Zhang et al. 2021; Sun et al. 2021). For instance, Ke et al. first
estimates the alpha maps for each frame and employs a self-supervised strategy
to preserve the temporal coherence (Ke et al. 2020). Zhang et al. employs an
attention-based temporal aggregation module to recurrently predicts the alpha
maps for videos (Zhang et al. 2021). Sun et al. fuses the features from
multiple frames to estimate the alpha maps for the target frame (Sun et al.
2021). Compared to these methods, our method use optical flow to capture the
motion information.

There is also a concurrent work by by Lin et al.  (Lin et al. 2021) that also
applies temporal information for video matting. But it targets efficient video
matting, while our method focuses on the high quality video matting. To
facilitate comparisons between these two methods, we both compare to a
previous baseline method BGMV2 (Lin et al. 2020).

## Method

Figure 1: The architecture of our network. Our network takes a sequence of
frames as inputs and estimate the alpha maps and foreground image for each
frame.

Our method takes a sequence of portrait frames
$\\{\mathbf{I}_{t}\\}_{t=1}^{N}$ as inputs and aims to estimate their
corresponding alpha maps $\\{\boldsymbol{\alpha}_{t}\\}_{t=1}^{N}$ as well as
their foregrounds $\\{\mathbf{F}_{t}\\}_{t=1}^{N}$, where $N$ indicates the
number of frames. Our network leverages not only semantic information but also
temporal information.

As shown in Figure 1, we develop a new deep network architecture for automatic
portrait video matting. It recurrently estimates the alpha maps and foreground
images. Our network can be distilled into four parts: (1) feature extraction,
(2) optical flow estimation, (3) context-motion updating operator, and (4)
upsampler. All parts are differentiable, and we train the network in an end-
to-end manner. Given an input frame $\mathbf{I}_{t}$, our network first
extracts the context features $\mathbf{Fea}_{t}$ from an encoder pretrained on
the image classification task. We capture the motion information by estimating
the backward optical flow $\mathbf{f}_{t\rightarrow t-1}$ from the current
frame $\mathbf{I}_{t}$ to the previous frame $\mathbf{I}_{t-1}$ by the PWC-Net
(Sun et al. 2018). To fuse the context information and the motion information,
we develop a context-motion updating operator, which integrates context
features, optical flow as well as the features from the previous frame.
Finally, we upscale the resulting features $\mathbf{H}_{t}$ and predict the
alpha maps $\boldsymbol{\alpha}_{t}$ and the foreground $\mathbf{F}_{t}$.
Below we describe the network in detail.

Feature extraction. We extract the features from the input images using a
convolutional network. The feature encoder network is applied to all the input
frames. It maps the input images to feature maps at a lower resolution as
follows

$\mathbf{Fea}_{t}=g_{encoder}(\mathbf{I}_{t}),$ (2)

where $g_{encoder}$ indicates the encoder network. We adopt the modified
ResNet50 (He et al. 2016) pretrained on the ImageNet dataset. Our encoder
outputs features at $1/16$ resolution of the source frame as we empirically
find that the large receptive field is critical for the semantic information.
Aside from the context features, we also extract the multi-scale intermediate
features from the encoder to capture the fine structures. We use $3\times 3$
convolutional layers to map these intermediate feature maps to 32 channels.
The weights of the feature extraction network are shared for all the input
frames.

Optical flow estimation. We calculate the optical flow between the current
frame $\mathbf{I}_{t}$ and the previous frame $\mathbf{I}_{t-1}$, which
indicates the per-pixel motion between these two frames. Inspired by the
success of state-of-the-art optical flow networks (Sun et al. 2018; Zhao et
al. 2020; Teed and Deng 2020), we leverage the PWC-Net (Sun et al. 2018) as a
sub-network in our network to estimate the optical flow. It takes two
consecutive video frames as input and output the backward optical flow as
follows,

$\mathbf{f}_{t\rightarrow t-1}=g_{flow}(\mathbf{I}_{t},\mathbf{I}_{t-1}),$ (3)

where $g_{flow}$ indicates the optical flow network. $\mathbf{f}_{t\rightarrow
t-1}\in\mathbb{R}^{H\times W\times 2}$ indicates the backward optical flow. We
initialize the the optical flow subnetwork using weights pretrained on the
Sintel dataset (Butler et al. 2012). During our training, we train the whole
network in an end-to-end manner.

Context-motion updating operator. Context-motion updating operator is designed
to fuse semantic information and temporal information. The context-motion
updating operator is recurrent. The features from the previous frames are
utilized in the context-motion updating operator to provide temporal
information.

Given the frame feature $\mathbf{Fea}_{t}$, $\mathbf{Fea}_{t-1}$ as well as
the backward optical flow $\mathbf{f}_{t\rightarrow t-1}$, our goal is to
capture the motion information. Inspired by the design of PWCNet (Sun et al.
2018), we backwarp the features $\mathbf{Fea}_{t-1}$ from previous frame to
the current frame by

$\mathbf{B}_{t-1}=g_{backwarp}(\mathbf{Fea}_{t-1},\mathbf{f}_{t\rightarrow
t-1}),$ (4)

where $g_{backwarp}(\cdot)$ indicates the operator of backwarping.
$\mathbf{B}_{t-1}$ indicates the backwarped features. We calculate the
correlation between the current feature $\mathbf{Fea}_{t}$ and the backwarped
feature $\mathbf{B}_{t-1}$

$\mathbf{C}_{t}=g_{corr}([\mathbf{Fea}_{t},\mathbf{B}_{t-1}]),$ (5)

where $[\cdot]$ indicates the concenating operator. $g_{corr}$ indicates the
operator to calculate the correlation. Our network adopts two $3\times 3$
convolutional layers with 32 channels.

Our network also encodes the optical flow to capture the motion information
from optical flow directly. We first bilinear downsample the optical flow size
to the size of $\mathbf{Fea}_{t}$. The optical flow is encoded as

$\mathbf{Flo}_{t}=g_{flow\\_encode}(\mathbf{f}_{t\rightarrow t-1}),$ (6)

where $g_{flow\\_encode}$ indicates the operator for optical flow encoding.
Our network adopts two $7\times 7$ convolutional layers with 32 channels.

We fuse the correlation $\mathbf{C}_{t}$ and optical flow feature
$\mathbf{Flo}_{t}$ to get the motion features

$\mathbf{M}_{t}=g_{motion}([\mathbf{C}_{t},\mathbf{Flo}_{t}]),$ (7)

where $g_{flow\\_encode}(\cdot)$ indicates the operator for motion encoding.
Our network adopts a $3\times 3$ convolutional layer with 62 channels.

By concatenating the semantic feature $\mathbf{Fea}_{t}$, motion feature
$\mathbf{M}_{t}$, as well as the optical flow $\mathbf{f}_{t\rightarrow t-1}$,
our method gets the fused feature $\mathbf{Fus}_{t}$,

$\mathbf{Fus}_{t}=[\mathbf{Fea}_{t},\mathbf{M}_{t},\mathbf{f}_{t\rightarrow
t-1}].$ (8)

Inspired by RAFT (Teed and Deng 2020), our network adopts separable ConvGRUs
to leverage the features from the previous frame. It replaces the fully
connected layers in the GRU with separable convolutions. It can be represented
as

$\mathbf{z}_{t}=g_{z}([\mathbf{H}_{t-1},\mathbf{Fus}_{t}]),$ (9)
$\mathbf{r}_{t}=g_{r}([\mathbf{H}_{t-1},\mathbf{Fus}_{t}])$ (10)
$\widetilde{\mathbf{H}_{t}}=g_{h}([\mathbf{r}_{t}\cdot\mathbf{H}_{t-1},\mathbf{Fus}_{t}])$
(11)
$\mathbf{H}_{t}=(1-\mathbf{z}_{t})\cdot\mathbf{H}_{t-1}+\mathbf{z}_{t}\cdot\widetilde{\mathbf{H}_{t}},$
(12)

where $g_{z}(\cdot)$, $g_{r}(\cdot)$, and $g_{h}(\cdot)$ indicate the gated
activation units. Specifically, $g_{z}(\cdot)$ and $g_{r}(\cdot)$ employs
$sigmoid(\cdot)$ as their activation function. $g_{h}(\cdot)$ uses
$tanh(\cdot)$. Our network adopts two ConvGRUs. In the first ConvGRU,
$g_{z}(\cdot)$, $g_{r}(\cdot)$ and $g_{h}(\cdot)$ are one $1\times 5$
convolutional layer. And in the second ConvGRU, they are $5\times 1$
convolutional layers. It can increase the receptive field while keeps the
model size small.

Upsampler. As shown in Figure 1, our network contains two decoders: one for
the alpha maps and the other for the foreground. They have the identity
network architecture. We first bilinearly upsample the features by a factor of
2. Our network concatenates the upsampled features with the intermediate
features from the encoder. Our network fuses them by two $3\times 3$
convolutional layers with 32 channels. We repeat the process until the
resolution of upsampled features is the same as the input frame. The last
layer of the alpha decoder has two convolutional layers. We use the activation
function $sigmoid()$ for the second convolutional layer. The last layer of the
foreground decoder also has two convolutional layers. The second convolutional
layer direct outputs the foreground image without activation function.

Loss functions. For each frame of the input video, we compute the loss over
the alpha maps and the foreground images. For the alpha maps, we use the
standard $\ell_{1}$ to measure the difference between the predicted alpha maps
$\boldsymbol{\alpha}_{t}$ and the ground truth
$\widehat{\boldsymbol{\alpha}}_{t}$ as follows

$\mathcal{L}_{1,t}^{\alpha}=\|\boldsymbol{\alpha}_{t}-\widehat{\boldsymbol{\alpha}}_{t}\|_{1}.$
(13)

Following Context Matting (Hou and Liu 2019), we also employ the Laplacian
loss for the alpha maps. Instead of measuring the difference from the image
directly, it measures the differences of two Laplacian pyramid as follows,

$\mathcal{L}_{lap,t}^{\alpha}=\sum_{i=1}^{5}2^{i-1}\|L^{i}(\boldsymbol{\alpha}_{t})-L^{i}(\widehat{\boldsymbol{\alpha}}_{t})\|_{1},$
(14)

where $L^{i}(\cdot)$ indicates the $i^{th}$ level of the Laplacian pyramid.

For the foreground image, we also employ the standard $\ell_{1}$ to measure
the difference between the predicted foreground image $\mathbf{F}_{t}$ and
$\widehat{\mathbf{F}}_{t}$. We only calculate the loss where the foreground is
visible

$\mathcal{L}_{1,t}^{fg}=\|\mathbb{I}(\widehat{\boldsymbol{\alpha}}_{t}>0)\cdot(\mathbf{F}_{t}-\widehat{\mathbf{F}}_{t})\|_{1},$
(15)

where $\mathbb{I}(\cdot)$ indicates a binary function whose value is 1 if the
condition is true and 0 otherwise.

We get the total loss as

$\mathcal{L}_{total}=\sum_{t=1}^{N}\mathcal{L}_{1,t}^{\alpha}+\mathcal{L}_{lap,t}^{\alpha}+0.1\mathcal{L}_{1,t}^{fg},$
(16)

where $N$ indicates the length of the input video.

Training. We use PyTorch to train our neural network. The feature extraction
network is initialized with the official weights pretrained on the ImageNet
dataset. And the optical flow network is initialized with the weights
pretrained on the Sintel dataset. The other modules are initialized with
random weights. We adopt AdamW (Loshchilov and Hutter 2017) as our optimizer.
Our initial learning rate is set to $10^{-4}$. We use the “OneCycleLR”
learning rate policy (Smith and Topin 2019) in training, which we empirically
find it converges faster than other policies. Furthermore, we clip gradients
in the range $[-1,1]$.

We train our network for 300k iterations on a single NVIDIA Titan RTX GPU. In
the first 150k iterations, we freeze the parameters of the optical flow
network. We empirically find that freezing the parameters of the optical flow
network in the early stages can make the training process more stable. Our
batchsize is set to 4. In the following 150k iterations, we unfreeze the
parameters of the optical flow network and train the network in an end-to-end
manner. We set the batchsize to 2 due to the GPU memory limitation. We set the
length of frames as 3 for training. Furthermore, our network freezes the
parameters of batch norm layers during the whole training process.

Figure 2: Examples from our background video dataset. It contains various
indoor scenarios.

Dataset. We train our network using the video portrait matting dataset,
Video240K SD, shared by (Lin et al. 2020). This dataset contains 484 training
videos with the corresponding alpha maps and the foreground images, including
479 videos for training and 5 videos for testing. The resolution of videos is
$768\times 432$. The Video240K SD dataset only releases the background images,
while our method is designed for the videos. We collected 82 indoor videos
from YouTube as the background videos. These videos are under Common Creative
Licenses. The resolution of these videos is higher than or equal to 1080p.
Most videos last more than 10 minutes, and the frame rate is higher than or
equal to 24fps. We randomly extracted frame sequences from these videos. Each
frame sequence contains 50 consecutive frames. We also downsampled the frames
to 720p. We manually removed the sequences with cut shots, visual effects, and
extremely fast motion. We also manually removed the sequences which contain
humans. This culling process reduced the total number of sequences to 5636,
consisting of 281,800 individual video frames with a typical resolution of
$1080\times 720$. We split them into two parts: 5586 from 68 videos as the
training background and 50 from 14 videos as the testing background. There is
no overlap scene among them. As shown in Figure 2, our dataset covers various
indoor scenarios. For training, we created the training samples by compositing
the foreground video onto a randomly selected background sequences. For the
testing, we composite each testing foreground video onto 10 randomly selected
background videos. There are 50 testing videos in total.

Data augmentation. As discussed in (Hou and Liu 2019), the foreground videos
and background videos often contain different artifacts. Directly training the
network on the composited videos without augmentation might compromise the
generalization capability of the trained network. Following (Hou and Liu
2019), we leveraged the JPEG noise to the composited videos. We keep the
quality of the composited videos in the range of [70%, 100%]. Aside from the
augmentations of images, we also introduced the H264 noise to our composited
videos, which is commonly used in video compression. Our method also uses
standard augmentations. Specifically, we randomly crop the composited videos
to patches of size $384\times 384$, as the large cropping size is critical for
semantic information. We randomly flip the images horizontally. We employ the
color jitter for the foreground video to increase the color diversity. We set
the brightness in a range of [0.85, 1.15], the contrast in the range of [0.85,
1.15], the hue in the range of [0.9, 1.1], the saturation in the range of
[0.7, 1.3]. We also employ the gamma transformation for the alpha maps. We set
the gamma value in the range of [0.2, 2].

## Experiment

Method |  Extra | Alpha |  Foreground
---|---|---|---
|  input |  SAD |  MSE |  Grad |  Conn |  MSE
Context Matting |  $\mathbf{T}_{seg}$ |  5.39 |  13.38 |  18.09 |  5.34 |  5.17
|  $\mathbf{T}_{gt}$ |  1.41 |  1.78 |  4.24 |  1.31 |  4.26
Index Matting |  $\mathbf{T}_{seg}$ |  4.32 |  10.32 |  13.03 |  4.22 |  -
|  $\mathbf{T}_{gt}$ |  1.47 |  1.57 |  2.74 |  1.28 |  -
GCA Matting |  $\mathbf{T}_{seg}$ |  4.33 |  10.91 |  14.14 |  4.24 |  -
|  $\mathbf{T}_{gt}$ |  1.49 |  2.32 |  3.29 |  1.37 |  -
FBA Matting |  $\mathbf{T}_{seg}$ |  4.17 |  10.18 |  12.36 |  4.10 |  2.50
|  $\mathbf{T}_{gt}$ |  1.02 |  0.87 |  1.84 |  0.91 |  2.25
BGMV2 |  $\mathbf{B}_{first}$ |  83.56 |  247.03 |  16.79 |  - |  3.79
|  $\mathbf{B}_{gt}$ |  1.07 |  1.41 |  2.47 |  0.94 |  4.01
Ours |  - |  0.44 |  0.27 |  0.38 |  0.26 |  0.88
Table 1: Comparison on the Video240K SD dataset. $\mathbf{T}_{seg}$ indicates
the trimap generated from the semantic segmentation method (Chen et al. 2018).
$\mathbf{T}_{gt}$ indicates the trimap generated from the ground truth alpha
maps. $\mathbf{B}_{first}$ indicates the background image from the first
frame. $\mathbf{B}_{gt}$ indicates the ground truth background image for each
frame in the video.

Frame Ground Truth Alpha Context Matting (Hou and Liu 2019) Index Matting (Lu
et al. 2019) GCA Matting (Li and Lu 2020) FBA Matting (Forte and Pitié 2020)
BGMV2 (Lin et al. 2020) Ours

Frame Ground Truth Alpha Context Matting (Hou and Liu 2019) Index Matting (Lu
et al. 2019) GCA Matting (Li and Lu 2020) FBA Matting (Forte and Pitié 2020)
BGMV2 (Lin et al. 2020) Ours

Figure 3:  Visual comparison on the Video240K SD dataset. Our method generates
more plausible results. We adopt the trimaps $\mathbf{T}_{gt}$ generated from
the ground truth alpha maps for the trimap-based methods, including Index
Matting (Lu et al. 2019), Context Matting (Hou and Liu 2019), GCA Matting (Li
and Lu 2020) and FBA Matting (Forte and Pitié 2020). For the BGMV2 (Lin et al.
2020), we adopt the ground truth background $\mathbf{B}_{gt}$ for it.

We conduct our experiments on the Video240K SD dataset. To compare with the
trimap-based methods, we generate two versions of trimaps. The first version
$\mathbf{T}_{gt}$ follows the Adobe Matting Dataset (Xu et al. 2017), which is
generated by dilating the ground truth alpha mattes. For each video, we
randomly select the size of dilation in the range of $[5,15]$. Since manually
creating the trimaps will take a lot of user effort, some users automatically
create the trimaps by dilating the results from the semantic segmentation
networks. As a consequence, we generate the second version $\mathbf{T}_{seg}$
by dilating the semantic segmentation using the Deeplab V3+ (Chen et al.
2018). We set the size of dilation as 10 pixels.

Evaluation metrics. For the alpha maps, we adopt various metrics, including
SAD, MSE, Gradient (Grad), and Connectivity (Conn) (Xu et al. 2017). For the
foreground videos, we evaluate the MSE of the foreground video. In our
experiments, we employ the official evaluation codes from the authors of the
Video240K SD dataset (Lin et al. 2020). We scale MSE, Grad, Conn by $10^{3}$,
$10^{-3}$ and $10^{-3}$, respectively. Please note that the evaluation codes
only measure results of unknown areas on the trimaps, while our method doesn’t
require trimaps. For a fair comparison, we evaluate the whole image for all
comparing methods. Besides, Conn metric fails on some results, and some
methods do not estimate the foreground image. We use “-” to denote them.

### Comparison with state-of-the-art methods

We compare our methods with the state-of-the-art trimap-based methods,
including Index Matting (Lu et al. 2019), Context Matting (Hou and Liu 2019),
GCA Matting (Li and Lu 2020) and FBA Matting (Forte and Pitié 2020). We also
compare with the state-of-the-art background-based method BGMV2 (Lin et al.
2020). We used the code / model shared by their authors.

Table 1 reports the quantitative results on the Video240K SD dataset. For each
trimap-based method, we report their results on the trimap $\mathbf{T}_{seg}$
which is generated from the semantic segmentation, and $\mathbf{T}_{gt}$ which
is generated from the ground truth alpha maps. For the background-based
method, we assume that users can capture the background image for the first
frame and adopt it as the background for the whole video, whose result is
denoted as $\mathbf{B}_{first}$. We also report the results with the ground
truth background for each frame, which is denoted as $\mathbf{B}_{gt}$. As
shown in Table 1, our method outperforms both trimap-based and background-
based approaches by a large margin. Specifically, our method wins 0.58 on SAD,
0.60 on MSE, 1.46 on Grad, and 0.65 on Conn for the alpha maps. Our method
wins 1.37 on MSE for the foreground. Figure 3 shows several visual examples on
the Video240k SD dataset. Our results contain more fine structures. In the
first example, the girl’s hair is our result is much more accurate than
others. In the second example, the stethoscope and arms in our result preserve
more details.

### Ablation study

In this section, we examine several key components of our method.

Method | Alpha |  Foreground
---|---|---
|  SAD |  MSE |  Grad |  Conn |  MSE
MobileNetV2 |  0.63 |  0.54 |  0.80 |  0.41 |  1.32
ResNet18 |  0.48 |  0.27 |  0.48 |  0.29 |  1.51
ResNet50 |  0.44 |  0.27 |  0.38 |  0.26 |  0.88
Table 2: The effectiveness of the feature extraction network on the Video240K
SD dataset.

Feature extraction network. We examine how the feature extraction network
affects the final results. In this experiment, we change the feature
extraction network while fixing the other parts of the network. Table 2
reports the results of MobileNetV2 (Sandler et al. 2018), ResNet18 (He et al.
2016) and ResNet50 (He et al. 2016). Compared with other backbones, ResNet50
is deeper and has higher computational complexity. It achieves the best
performance. We believe that these improvements come from the more
representative information of ResNet50, which is consistent with the finding
of the previous semantic segmentation methods (Chen et al. 2017, 2018) and
image matting methods (Forte and Pitié 2020).

Method | Alpha |  Foreground
---|---|---
|  SAD |  MSE |  Grad |  Conn |  MSE
MaskFlownet |  0.73 |  0.82 |  0.94 |  0.52 |  1.61
PWC-Net |  0.63 |  0.54 |  0.80 |  0.41 |  1.32
Table 3: The effectiveness of the optical flow network on the Video240K SD
dataset.

Optical flow network. We examine how the optical flow network affects the
final results. In this experiment, we use the MobileNetV2 (Sandler et al.
2018) as our feature extraction network. We compare two models: one with PWC-
Net (Sun et al. 2018) as the optical flow network and the other one with
MaskFlownet (Zhao et al. 2020) as the optical flow network. Although
MaskFlownet achieves better performance in the optical flow estimation task
(Zhao et al. 2020), PWC-Net gets better matting results. As shown in Table 3,
PWC-Net wins 0.10 on SAD, 0.28 on MSE, 0.14 on Grad, and 0.09 on Conn on the
alpha maps. It also wins 0.29 on the foreground MSE. This is consistent with
many other works that PWC-Net has great generalization capability (Niklaus and
Liu 2018, 2020; Bao et al. 2019).

Context motion updating operator. We also investigate the contribution of each
component in the context motion updating operator. In this experiment, we
adopt the MobileNetV2 (Sandler et al. 2018) as the feature extraction and PWC-
Net (Sun et al. 2018) as the optical flow network. In particular, we study the
impact of context motion updating operator by removing it from the whole
network. For the model with “Baseline”, we remove the context motion updating
operator. For the model with “+ConvGRU”, we only keep the ConvGRU while
removing the other components from the context motion updating operator. For
the model with “+Motion”, we keep the context motion updating operator. The
performance of these models can be found in Table 4. We can find that ConvGRU
can improve all the evaluation metrics on the alpha maps while slightly
increasing the foreground MSE. Particularly, the motion plays a crucial role,
which improves the results by 0.20 on SAD, 0.42 on MSE, 0.25 on Grad, and 0.16
on Conn. It also improves the foreground MSE by 0.51. It verifies our
motivation to introduce the motion information into the video matting network.

Method | Alpha |  Foreground
---|---|---
|  SAD |  MSE |  Grad |  Conn |  MSE
Baseline |  0.95 |  1.45 |  1.22 |  0.75 |  1.66
\+ ConvGRU |  0.83 |  0.96 |  1.05 |  0.57 |  1.83
\+ Motion |  0.63 |  0.54 |  0.80 |  0.41 |  1.32
Table 4: The effectiveness of the context motion updating operator on the
Video240K SD dataset. We add the components cumulatively from top to bottom.

## Limitation and Future Work

There are several limitations and drawbacks of the proposed automatic portrait
video matting method.

Feature extraction. Our method employs ResNet50 as our feature extraction
network to capture semantic information. It will take too much GPU memory on
high-resolution videos, e.g., videos of 4k resolution. In training, the
cropping size of training examples on the high-resolution videos needs to be
large to capture the semantic information. Besides, the optical flow network
also requires large memory. The high computational and memory requirements
impede our training on high-resolution videos.

Optical flow. Our method currently relies on PWC-Net (Sun et al. 2018) to
estimate the optical flow between frames. However, PWC-Net might not produce
reliable optical flow for the challenging scenarios, e.g., videos with large
motion. In these videos, our method will fail to produce correct matting
results. An interesting future direction is to design an optical flow network
specifically for video matting.

## Conclusion

This paper presented an automatic portrait video matting method. We designed a
context motion network for this task. Our network takes a sequence of frames
as inputs and does not require extra inputs. Our network leverages context
information and temporal information. Our method first extracts the context
features for each frame. It also extracts the optical flow between consecutive
frames. This is followed by a context motion updating operator to fuse the
context and motion features recurrently. Our experiments showed that our
method is able to generate high-quality alpha maps and foreground images. Our
experiments also showed that the context motion updating operator is helpful
to generate high-quality results.

## References

  * Aksoy, Ozan Aydin, and Pollefeys (2017) Aksoy, Y.; Ozan Aydin, T.; and Pollefeys, M. 2017.  Designing effective inter-pixel information flow for natural image matting.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 29–37.
  * Apostoloff and Fitzgibbon (2004) Apostoloff, N.; and Fitzgibbon, A. 2004.  Bayesian video matting using learnt image priors.  In _Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004._ , volume 1, I–I. IEEE.
  * Bai, Wang, and Simons (2011) Bai, X.; Wang, J.; and Simons, D. 2011.  Towards temporally-coherent video matting.  In _International Conference on Computer Vision/Computer Graphics Collaboration Techniques and Applications_ , 63–74. Springer.
  * Bao et al. (2019) Bao, W.; Lai, W.-S.; Ma, C.; Zhang, X.; Gao, Z.; and Yang, M.-H. 2019.  Depth-aware video frame interpolation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 3703–3712.
  * Butler et al. (2012) Butler, D. J.; Wulff, J.; Stanley, G. B.; and Black, M. J. 2012.  A naturalistic open source movie for optical flow evaluation.  In A. Fitzgibbon et al. (Eds.), ed., _European Conf. on Computer Vision (ECCV)_ , Part IV, LNCS 7577, 611–625. Springer-Verlag.
  * Chen et al. (2017) Chen, L.-C.; Papandreou, G.; Schroff, F.; and Adam, H. 2017.  Rethinking atrous convolution for semantic image segmentation.  _arXiv preprint arXiv:1706.05587_.
  * Chen et al. (2018) Chen, L.-C.; Zhu, Y.; Papandreou, G.; Schroff, F.; and Adam, H. 2018.  Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation.  In _ECCV_.
  * Chen, Li, and Tang (2013) Chen, Q.; Li, D.; and Tang, C.-K. 2013.  KNN matting.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 35(9): 2175–2188.
  * Chen et al. (2013) Chen, X.; Zou, D.; Zhiying Zhou, S.; Zhao, Q.; and Tan, P. 2013.  Image matting with local and nonlocal smooth priors.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 1902–1907.
  * Cho, Tai, and Kweon (2016) Cho, D.; Tai, Y.-W.; and Kweon, I. 2016.  Natural image matting using deep convolutional neural networks.  In _European Conference on Computer Vision_ , 626–643. Springer.
  * Cho, Tai, and Kweon (2019) Cho, D.; Tai, Y.-W.; and Kweon, I. S. 2019.  Deep convolutional neural network for natural image matting using initial alpha mattes.  _IEEE Transactions on Image Processing_ , 28(3): 1054–1067.
  * Chuang et al. (2002) Chuang, Y.-Y.; Agarwala, A.; Curless, B.; Salesin, D. H.; and Szeliski, R. 2002\.  Video matting of complex scenes.  In _Proceedings of the 29th annual conference on Computer graphics and interactive techniques_ , 243–248.
  * Chuang et al. (2001) Chuang, Y.-Y.; Curless, B.; Salesin, D. H.; and Szeliski, R. 2001.  A bayesian approach to digital matting.  In _Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001_ , volume 2, II–II. IEEE.
  * Forte and Pitié (2020) Forte, M.; and Pitié, F. 2020.  $F$, $B$, Alpha Matting.  _arXiv preprint arXiv:2003.07711_.
  * He et al. (2011) He, K.; Rhemann, C.; Rother, C.; Tang, X.; and Sun, J. 2011.  A global sampling method for alpha matting.  In _CVPR 2011_ , 2049–2056. IEEE.
  * He, Sun, and Tang (2010) He, K.; Sun, J.; and Tang, X. 2010.  Fast matting using large kernel matting laplacian matrices.  In _2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition_ , 2165–2172. IEEE.
  * He et al. (2016) He, K.; Zhang, X.; Ren, S.; and Sun, J. 2016.  Deep residual learning for image recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 770–778.
  * Hou and Liu (2019) Hou, Q.; and Liu, F. 2019.  Context-Aware Image Matting for Simultaneous Foreground and Alpha Estimation.  In _Proceedings of the IEEE International Conference on Computer Vision_ , 4130–4139.
  * Ke et al. (2020) Ke, Z.; Li, K.; Zhou, Y.; Wu, Q.; Mao, X.; Yan, Q.; and Lau, R. W. 2020.  Is a Green Screen Really Necessary for Real-Time Portrait Matting?  _arXiv preprint arXiv:2011.11961_.
  * Lee and Wu (2011) Lee, P.; and Wu, Y. 2011.  Nonlocal matting.  In _CVPR 2011_ , 2193–2200. IEEE.
  * Lee, Yoon, and Lee (2010) Lee, S.-Y.; Yoon, J.-C.; and Lee, I.-K. 2010.  Temporally coherent video matting.  _Graphical Models_ , 72(3): 25–33.
  * Levin, Lischinski, and Weiss (2008) Levin, A.; Lischinski, D.; and Weiss, Y. 2008.  A closed-form solution to natural image matting.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 30(2): 228–242.
  * Levin, Rav-Acha, and Lischinski (2008) Levin, A.; Rav-Acha, A.; and Lischinski, D. 2008.  Spectral matting.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 30(10): 1699–1712.
  * Li and Lu (2020) Li, Y.; and Lu, H. 2020.  Natural image matting via guided contextual attention.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 34, 11450–11457.
  * Li, Xu, and Lu (2020) Li, Y.; Xu, Q.; and Lu, H. 2020.  Hierarchical opacity propagation for image matting.  _arXiv preprint arXiv:2004.03249_.
  * Lin et al. (2020) Lin, S.; Ryabtsev, A.; Sengupta, S.; Curless, B.; Seitz, S.; and Kemelmacher-Shlizerman, I. 2020.  Real-Time High-Resolution Background Matting.  _arXiv_ , arXiv–2012.
  * Lin et al. (2021) Lin, S.; Yang, L.; Saleemi, I.; and Sengupta, S. 2021.  Robust High-Resolution Video Matting with Temporal Guidance.  arXiv:2108.11515.
  * Long, Shelhamer, and Darrell (2015) Long, J.; Shelhamer, E.; and Darrell, T. 2015.  Fully convolutional networks for semantic segmentation.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 3431–3440.
  * Loshchilov and Hutter (2017) Loshchilov, I.; and Hutter, F. 2017.  Decoupled weight decay regularization.  _arXiv preprint arXiv:1711.05101_.
  * Lu et al. (2019) Lu, H.; Dai, Y.; Shen, C.; and Xu, S. 2019.  Indices matter: Learning to index for deep image matting.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 3266–3275.
  * Niklaus and Liu (2018) Niklaus, S.; and Liu, F. 2018.  Context-aware synthesis for video frame interpolation.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 1701–1710.
  * Niklaus and Liu (2020) Niklaus, S.; and Liu, F. 2020.  Softmax Splatting for Video Frame Interpolation.  In _IEEE Conference on Computer Vision and Pattern Recognition_.
  * Qiao et al. (2020) Qiao, Y.; Liu, Y.; Yang, X.; Zhou, D.; Xu, M.; Zhang, Q.; and Wei, X. 2020.  Attention-guided hierarchical structure aggregation for image matting.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 13676–13685.
  * Sandler et al. (2018) Sandler, M.; Howard, A.; Zhu, M.; Zhmoginov, A.; and Chen, L.-C. 2018.  MobileNetV2: Inverted Residuals and Linear Bottlenecks.  In _CVPR_.
  * Sengupta et al. (2020) Sengupta, S.; Jayaram, V.; Curless, B.; Seitz, S.; and Kemelmacher-Shlizerman, I. 2020.  Background Matting: The World is Your Green Screen.  In _Computer Vision and Pattern Regognition (CVPR)_.
  * Shen et al. (2016) Shen, X.; Tao, X.; Gao, H.; Zhou, C.; and Jia, J. 2016.  Deep automatic portrait matting.  In _European Conference on Computer Vision_ , 92–107. Springer.
  * Smith and Topin (2019) Smith, L. N.; and Topin, N. 2019.  Super-convergence: Very fast training of neural networks using large learning rates.  In _Artificial Intelligence and Machine Learning for Multi-Domain Operations Applications_ , volume 11006, 1100612. International Society for Optics and Photonics.
  * Sun et al. (2018) Sun, D.; Yang, X.; Liu, M.-Y.; and Kautz, J. 2018.  Pwc-net: Cnns for optical flow using pyramid, warping, and cost volume.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 8934–8943.
  * Sun et al. (2004) Sun, J.; Jia, J.; Tang, C.-K.; and Shum, H.-Y. 2004.  Poisson matting.  In _ACM SIGGRAPH 2004 Papers_ , 315–321.
  * Sun et al. (2021) Sun, Y.; Wang, G.; Gu, Q.; Tang, C.-K.; and Tai, Y.-W. 2021.  Deep Video Matting via Spatio-Temporal Alignment and Aggregation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 6975–6984.
  * Tang et al. (2019) Tang, J.; Aksoy, Y.; Oztireli, C.; Gross, M.; and Aydin, T. O. 2019.  Learning-based sampling for natural image matting.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 3055–3063.
  * Teed and Deng (2020) Teed, Z.; and Deng, J. 2020.  Raft: Recurrent all-pairs field transforms for optical flow.  In _European conference on computer vision_ , 402–419. Springer.
  * Wang and Cohen (2005) Wang, J.; and Cohen, M. F. 2005.  An iterative optimization approach for unified image segmentation and matting.  In _Tenth IEEE International Conference on Computer Vision (ICCV’05) Volume 1_ , volume 2, 936–943. IEEE.
  * Wang, Cohen et al. (2008) Wang, J.; Cohen, M. F.; et al. 2008.  Image and video matting: a survey.  _Foundations and Trends® in Computer Graphics and Vision_ , 3(2): 97–175.
  * Xu et al. (2017) Xu, N.; Price, B.; Cohen, S.; and Huang, T. 2017.  Deep image matting.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2970–2979.
  * Zhang et al. (2021) Zhang, Y.; Wang, C.; Cui, M.; Ren, P.; Xie, X.; Hua, X.-s.; Bao, H.; Huang, Q.; and Xu, W. 2021.  Attention-guided Temporal Coherent Video Object Matting.  _arXiv preprint arXiv:2105.11427_.
  * Zhao et al. (2020) Zhao, S.; Sheng, Y.; Dong, Y.; Chang, E. I.-C.; and Xu, Y. 2020.  MaskFlownet: Asymmetric Feature Matching with Learnable Occlusion Mask.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * Zhu et al. (2017) Zhu, B.; Chen, Y.; Wang, J.; Liu, S.; Zhang, B.; and Tang, M. 2017.  Fast deep matting for portrait animation on mobile phone.  In _Proceedings of the 25th ACM international conference on Multimedia_ , 297–305.