Yijie<EMAIL_ADDRESS>Bo<EMAIL_ADDRESS>Lv
<EMAIL_ADDRESS>Hao<EMAIL_ADDRESS>Shouhong
<EMAIL_ADDRESS>Youtu Lab,
Tencent,
Shanghai, China Computer Vision Lab,
ETH Zurich,
Switzerland Independent Researcher Highly Efficient Natural Image Matting

# Highly Efficient Natural Image Matting 111 Bo Li is equal contribution to
the first author. Corresponding authors are Bo Li and Lv Tang.

###### Abstract

Over the last few years, deep learning based approaches have achieved
outstanding improvements in natural image matting. However, there are still
two drawbacks that impede the widespread application of image matting: the
reliance on user-provided trimaps and the heavy model sizes. In this paper, we
propose a trimap-free natural image matting method with a lightweight model.
With a lightweight basic convolution block, we build a two-stages framework:
Segmentation Network (SN) is designed to capture sufficient semantics and
classify the pixels into unknown, foreground and background regions; Matting
Refine Network (MRN) aims at capturing detailed texture information and
regressing accurate alpha values. With the proposed cross-level fusion Module
(CFM), SN can efficiently utilize multi-scale features with less computational
cost. Efficient non-local attention module (ENA) in MRN can efficiently model
the relevance between different pixels and help regress high-quality alpha
values. Utilizing these techniques, we construct an extremely light-weighted
model, which achieves comparable performance with 1% parameters (344k) of
large models on popular natural image matting benchmarks.

##  1 Introduction

The natural image matting is an important task in computer vision, which
serves as a prerequisite for a broad set of applications, including image or
video editing, compositing and film post-production [Tang et al.(2019b)Tang,
Aksoy, Öztireli, and Gross, Sengupta et al.(2020)Sengupta, Jayaram, and
Curless, Lin et al.(2020)Lin, Ryabtsev, and Sengupta, Tang et al.(2020c)Tang,
Bai, Zhang, Torr, and Sebe, Tang et al.(2020d)Tang, Xu, Yan, Torr, and Sebe,
Tang et al.(2019a)Tang, Xu, Sebe, Wang, Corso, and Yan, Tang et
al.(2021a)Tang, Liu, Xu, Torr, and Sebe, Tang et al.(2020a)Tang, Bai, and
Sebe, Tang et al.(2020b)Tang, Bai, Torr, and Sebe]. Following [Porter and
Duff(1984)], the natural image $I$ is defined as a convex combination of
foreground image $F$ and background image $B$ at each pixel $i$ as:

$I_{i}=\alpha_{i}F_{i}+(1-\alpha_{i})B_{i},\alpha_{i}\in[0,1],$ (1)

where $\alpha_{i}$ is the alpha value at pixel $i$ that denotes the opacity of
the foreground object. Mathematically, image matting requires expressing pixel
colors in the transition regions

Figure 1: Comparison with SOTA methods. Best viewed by zooming in. Smaller SAD
and FLOPs mean better performance and less computational cost.

from foreground to background as a convex combination of their underlying
foreground and background colors. Since neither the foreground and background
colors nor the opacities are known, estimating the opacity values is a highly
ill-posed problem. To alleviate the difficulty of this problem, typically a
trimap is provided in addition to the original image [Aksoy et al.(2017)Aksoy,
Aydin, and Pollefeys]. The trimap can be provided by user or generated from
the network. The trimap is a rough segmentation of the input image into
foreground($\alpha_{i}=1$), background($\alpha_{i}=0$), and regions with
unknown opacity($\alpha_{i}\in(0,1)$) which like other segmentation tasks [Li
et al.(2019b)Li, Sun, Li, Wu, and Hu, Li et al.(2019a)Li, Sun, and Guo, Li et
al.(2019d)Li, Sun, Tang, Sun, and Shi, Li et al.(2019e)Li, Sun, Wang, and Li,
Li et al.(2020)Li, Sun, Xu, Wang, and Yu, Tang and Li(2020), Li et
al.(2017)Li, Sun, Hu, and Xu, Tang et al.(2021c)Tang, Li, Zhong, Ding, and
Song]. From then on, many traditional methods [Chuang et al.(2001)Chuang,
Curless, Salesin, and Szeliski, Sun et al.(2004)Sun, Jia, and Tang, Levin et
al.(2006)Levin, Lischinski, and Weiss, Wang and Cohen(2007), He et
al.(2011)He, Rhemann, Rother, Tang, and Sun, Shahrian et al.(2013)Shahrian,
Rajan, and Price, Aksoy et al.(2017)Aksoy, Aydin, and Pollefeys] are proposed
for matting task. These methods leverage color features to predict alpha
mattes, which will produce obvious artifacts when foreground and background
share similar colors.

Deep learning has recently demonstrated success in many computer vision
applications. Recent researches use deep neural networks to improve the
performance of natural image matting. Compared to traditional methods, deep-
learning matting methods [Cho et al.(2016)Cho, Tai, and Kweon, Xu et
al.(2017)Xu, Price, Cohen, and Huang, Tang et al.(2019b)Tang, Aksoy, Öztireli,
and Gross, Cai et al.(2019)Cai, Zhang, and Fan, Hou and Liu(2019), Lu et
al.(2019)Lu, Dai, Shen, and Xu, Li and Lu(2020)] have achieved remarkable
performance and further advance the matting field. However, the widespread
application of image matting still faces two major challenges: first, most
methods rely on user-provided trimaps as assistant input, which is not easy to
use for novice users without expertise in digital matting; second, their heavy
model sizes make them resource-hungry and hardly applicable on low-power
devices with limited storage/computational capability.

To address first problem, [Zhang and Gong(2019), Qiao et al.(2020)Qiao, Liu,
Yang, and Zhou] design trimap-free natural image matting models. [Zhang and
Gong(2019)] investigated semantic segmentation variant for foreground and
background weight map fusion to obtain alpha values. [Qiao et al.(2020)Qiao,
Liu, Yang, and Zhou] explores advanced semantics and appearance cues
synthetically to achieve high-quality alpha values. However, to capture
sufficient semantics, these two methods utilizate large backbones
(DenseNet-201 and ResNeXt101), which make it difficult to apply these methods
to some devices with limited storage/computational capability. To address
second problem, [Yoon and Park(2020)] designs a light-weighted natural image
matting model with the help of knowledge distillation. However, this method
also needs extra trimaps, which is not user-friendliness. Hence, designing a
trimap-free natural image matting method with a light-weighted model is barely
explored.

Without the user-provided trimap, there are two basic tasks for automatic
natural image matting: identify the foreground, background and unknown regions
in the image; predict the precise alpha values of unknown regions. The first
one is a classic classification task while the second task is typical
regression task. As discussed in [Zhang and Gong(2019)], it is hard to handle
the two tasks simultaneously with a single structure, we thus propose a two-
stage end-to-end deep learning framework for trimap-free light-weighted
matting task. The first stage is designed to capture sufficient semantics to
identify the foreground, background, and unknown regions. Then under the
guidance of the first stage, the second stage can give the precise alpha
values of unknown regions. Although such two-stage network design has appeared
before (e.g. [Zhu et al.(2017)Zhu, Chen, Wang, and Liu], [Hu and Clark(2019)],
[Chen et al.(2018)Chen, Ge, Xu, Zhang, Yang, and Gai]), it is an intutitive
idea. The most important is that [Zhu et al.(2017)Zhu, Chen, Wang, and Liu]
designs redundant structures besides low-resolution input ($128\times 128$),
and the strong portrait prior limit their application prospects. Then the core
problem lies in how to accomplish these two tasks with light-weighted network
structures.

We approach this core problem from two perspectives: a light-weighted backbone
and efficient feature aggregation/enhancement strategies of each stage. To be
specific, we first build a light-weighted basic block with some popular
efficient convolutional operations. Then we stack these basic blocks to
construct light-weighted U-Net like backbones for two stages. The first stage
is segmentation network (SN), and the second stage is matting refine network
(MRN). In the first stage, we design a cross-level fusion module (CFM) to
directly fuse the high-level semantic feature with low-level feature, which
can improve the discriminability of feature representation. In the second
stage, we design a novel efficient non-local attention module (ENA) to model
the relevance between different pixels in encoder/decoder feature and image
feature extracted from octave block (OCBlock), which can help regress high-
quality alpha values. Moreover, in matting task, modeling the relevance of
different pixels in long-range is not as important as modeling the relevance
of pixels in short-range. So the proposed ENA first calculates long-range
relevance with sampling strategy, then directly calculates short-range
relevance. This design meets the principle of a light-weighted model. As can
be seen in Fig.1, compared with LateFusion [Zhang and Gong(2019)], our
proposed method can predict a better result with accurate semantics and alpha
value.

Our major contributions can be summarized as:

  * •

We make one of the earliest effort on designing a light-weighted trimap-free
natural image matting method, which expands matting to much wider application
scenarios.

  * •

We construct two U-Net like backbone with light-weighted basic blocks for the
proposed two-stage framework and propose two corresponding efficient feature
aggregation/enhancement strategies CFM and ENA for two stages to better
identify the unknown regions and give precise alpha values.

  * •

We perform extensive experiments to demonstrate the proposed method refreshes
the SOTA performance on Composition-1k and Distinctions-646 testing sets with
1% parameters (344k) of SOTA large models.

Figure 2: The framework of the our trimap-free light-weighted natural image
matting model.

##  2 Method

In this section, we first describe the overall architecture of the proposed
trimap-free light-weighted natural image matting network, then introduce our
proposed light-weighted backbone, SN and MRN. The architecture of the proposed
approach is illustrated in Fig.2.

###  2.1 Network Overview

SN has a simple U-Net like Encoder-Decoder architecture, and the backbone is
formed by stacking our proposed OCBlock. Similar to ResNet-50 [He et
al.(2016)He, Zhang, Ren, and Sun], the proposed backbone has four levels,
named as En1-1, En1-2, En1-3, En1-4. The number of OCBlock in each level is 3,
4, 6, 4. Because En1-1 is too close to the input and its resolution is too
high, to save computational cost, we only use the last three levels features
for the following process. Cross-level fusion module (CFM) is added between
En1-4 and En1-2 to improve the discriminability of feature representations. SN
has 3 decoders, and each decoder is stacked by 1 OCBlock. Decoder fuses the
processed features from the encoders and the upsampled features from the
previous decoder level in a bottom-up manner. The output of each decoder is
defined as De1-4, De1-3, and De1-2. Finally, the network generates the
3-channel classification logits $T$ by a $3\times 3$ convolution.

MRN has a basic Encoder-Decoder architecture, and the input of MRN is
predicted logits $T$ and an image. The backbone is also stacked by our
proposed OCBlock, and all encoder levels contain 2 OCBlocks, respectively. To
accurately regress alpha values in unknown regions, we stick two efficient
non-local attention (ENA) module to the encoder and decoder symmetrically in
our stem. Moreover, image features extracted from OCBlocks contain more
detailed texture information, which can help regress alpha values. Thus, we
utilize the image features extracted from OCBlocks to guide the information
flow on Encoder/Decoder features. MRN has 4 decoders, and each decoder is
stacked by 1 OCBlock. Decoder fuses the features in each encoder and the
upsampled features from the previous level in a bottom-up manner. Finally,
together with the unknown and foreground maps predicted by SN, MRN can regress
a 1-channel result $\alpha$.

###  2.2 Light-weighted backbone

Recently, some researchers have designed various light-weighted backbones
[Howard et al.(2017)Howard, Zhu, and Chen, Chollet(2017), Zhang et
al.(2018)Zhang, Zhou, and Lin, Li et al.(2019c)Li, Sun, Tang, and Hu, Tang et
al.(2021b)Tang, Li, Wu, Xiao, and Ding] with special convolution operations. A
classic example is MobileNet which constructs a light-weighted backbone by
using the depthwise convolution. However, these light-weighted models are
designed for classification, it is not appropriate to directly use them for
matting task. Unlike classification task, matting needs multi-scale
information to capture sufficient semantics for pixel-level prediction. Some
researchers try to perform extra multi-scale feature extraction after
backbone, like atrous spatial pyramid pooling (ASPP), global convolutional
network (GCN) [Peng et al.(2017)Peng, Zhang, and Yu], which need more
computational cost. Inspire by MobileNet, we want to use depthwise convolution
to build our lightweight backbone. But only using depthwise convolution cannot
capture sufficient multi-scale information. To address this problem, by
leveraging the OctConv [Chen et al.(2019)Chen, Fan, and Xu], we design our
lightweight OCBlock, and stack OCBlocks to construct a light-weighted
backbone. To be specific, we replace a depthwise convolution with a vanilla
OctConv. The vanilla OctConv [Chen et al.(2019)Chen, Fan, and Xu] is presented
to conduct the convolution operation on features across high/low scales, as
shown in Fig.2(a). In OctConv, features from different scales are fused to
generate output. The proposed OCBlock is shown in Fig.2(b), which combines the
advantages of depthwise convolutions and OctConv to achieve a better
weight/performance balance.

###  2.3 Segmentation Network

SN plays the role of capturing sufficient semantics and classifying the pixels
into unknown, foreground, and background regions with light-weighted backbone.
Thus, learning discriminant feature representations with less computational
cost is essential. To achieve this goal, we first utilizate our proposed
light-weighted backbone to obtain multi-scale feature representations.
Moreover, we utilize CFM module to leverage the advantages of features at
high-level to learn more discriminant feature representations.
Cross-Level Fusion Module. High-level feature from En1-4 contains much
semantic information, which can make the network better distinguish the
appearance difference between unknown, foreground, and background regions.
However, our proposed SN is a U-like architecture, when information is
progressively returned from the high-level to the low-level, the high-level
semantic information is gradually diluted. Thus, we directly fuse low-level
feature En1-2 with high-level feature En1-4, to propagate the high-level
semantic information to the low-level. As different channels of feature in
high-level response to different semantics, it is unwise to treat all channels
without distinction. To alleviate the interference of the irrelevant semantic
information, we use channel-wise attention on En1-4 to assign larger weights
to channels which are more important for matting task. The operations in
channel-wise attention are with low time complexity. Therefore, it demands
negligible computational cost.

###  2.4 Matting Refine Network

MRN aims at regressing the alpha values of pixels in unknown regions. The MRN
takes the concatenation of 3-channel images and the 3-channel segmentation
results from SN as 6-channel input. For easy remembering, we named unknown
regions as $U$. We design an efficient non-local attention module which can
help better regress the alpha values with less computational cost.
Efficient Non-Local Attention. Modeling the relevance of different pixels can
help regress the accurate alpha value. Because it can help the network
effectively group pixels with similar texture, then pay more attention to
these pixels. Inspired by [Wang and Girshick(2018)], we use non-local
attention to achieve this goal. Common non-local attention can be defined as:

$A=Sim(Q,K)V,$ (2)

where $Q,K$ and $V$ mean $query$, $key$ and $value$ feature, and $Sim$ is the
similarity between query and key feature.

According to traditional methods, pixels with almost identical texture should
have similar opacity. Hence, pixels that share similar texture information
should have similar opacity features. However, as the network layers deepen,
the feature representations of each pixel will be blurred, which will affect
the regression of alpha value. Therefore, we use detailed image features to
guide the information flow on Encoder/Decoder features, which improves the
regression accuracy of alpha values. In our proposed work, the $query$ feature
is the feature from Encoder/Decoder, named as $X_{Q}$. The $key/value$ feature
is the feature extracted from the input image, named as $X_{K}$ and $X_{V}$.

Figure 3: The illustration of the proposed ENA. The query feature is obtained
from the multiplication of encoder/decoder features and unknown region. Image
is passed through OCBlocks to obtain the key and value feature. Our proposed
ENA is a cross-level non-local attention, but for the sake of simplicity, we
have only shown the long-range and short-range attention operations on query
feature $X_{Q}$. In this figure, we set $k=4$ and $n=16$.

Because MRN only cares about regressing the alpha values in unknown region, we
multiply $X_{Q}$ with $U$ to make $query$ feature more focused on the unknown
region. It can be formulated as:

$X_{Q}=X_{Q}*U.$ (3)

Directly using conventional non-local attention in our work is time-consuming,
which is not in line with the design principle of light-weighted models.
Moreover, motivated by [Chen et al.(2012)Chen, Li, and Tang], modeling the
relevance of different pixels in long-range is not as important as modeling
the relevance of pixels in short-range, and sometimes will lead to some noise
because of irrelevant matches. Thus, our proposed ENA is the combination of
long-range attention with sampling strategy and short-range attention. The
main point of the long-range attention is to apply the non-local attention to
the subsets of positions that have long spatial interval distances, which can
not only help reduce the probability of irrelevant matches between long-range
pixels, but also save computational cost. The details of long-range attention
can be seen in Fig.3(a). The input feature $X\in\\{X_{Q},X_{K},X_{V}\\}$ are
first divided into $k$ partitions, then sampling these pixels in different
partitions at intervals $\sqrt{k}$ into new subsets. In our work, we use
permute operation on behalf of sampling. So each subset contains $\frac{n}{k}$
pixels, where $n$ represents the size of input features. Thus the new subsets
are defined as: $X_{L}=[X^{1},X^{2},\cdots,X^{k}]$, where
$X^{k}\in\\{X_{Q}^{k},X_{K}^{k},X_{V}^{k}\\}$, and the shape of $X^{k}$ is
$\mathbb{R}^{\frac{n}{k}\times C}$. $C$ means the channel numbers. Then we
apply the non-local attention on $X^{k}$ independently:

$A_{G}^{k}=Sim(X_{Q}^{k},X_{K}^{k})X_{V}^{k},$ (4)

which is shown in Fig.3(b). Finally, we merge all the $A_{G}^{k}$ to get the
output $A_{G}$. For short-range attention, we permute the $A_{G}$ to group the
originally nearby positions together, which is the $query$ of short-range
attention. So the input of short-range attention is
$Z\in\\{A_{G},X_{K},X_{V}\\}$. Similarly, We divide the into $\frac{n}{k}$
partitions and each partition contains $k$ neighboring positions, defined as:
$Z_{S}=[Z^{1},Z^{2},\cdots,Z^{\frac{n}{k}}]$, where
$Z^{\frac{n}{k}}\in\\{A_{G}^{\frac{n}{k}},X_{K}^{\frac{n}{k}},X_{V}^{\frac{n}{k}}\\}$,
and the shape of $Z^{\frac{n}{k}}$ is $\mathbb{R}^{k\times C}$. Then we apply
the non-local attention on each $Z^{\frac{n}{k}}$ independently. Accordingly,
we can get $A_{L}^{\frac{n}{k}}$ and merge all of them to get the output
$A_{L}$. Finally, we directly fuse the $A_{L}$ with Encoder/Decoder features
to get the refined features. In our proposed model, we set $k=16$. We analyze
the computation/memory cost of the common non-local attention mechanism and
our ENA in supplementary materials Section 3.
MRN Output. Together with unknown maps, MRN will output the refined alpha
values $\alpha_{r}$ in unknown regions. The final alpha matte result $\alpha$
can be written as:

$\alpha=F+\alpha_{r}.$ (5)

###  2.5 Loss Function

To supervise our proposed network, the loss function contains two parts:
$L_{SN}$ and $L_{MRN}$. Let $\alpha^{gt}$ be the groundtruth of final
prediction $\alpha$.
The prediction of SN is 3-channel classification logits $T$. Following [Cai et
al.(2019)Cai, Zhang, and Fan], we generate its groundtruth $T^{gt}$ according
to $\alpha^{gt}$, which is defined as:

$T^{gt}(x,y)=\begin{cases}0,\ &{\alpha^{gt}(x,y)=0}\\\ 1,\
&{\alpha^{gt}(x,y)=1}\\\ 2,\ &{0<\alpha^{gt}(x,y)<1}\end{cases}$ (6)

where $(x,y)$ stands for each pixel location on the image. The value
$\\{0,1,2\\}$ means the pixel is in background, foreground or unknown region.
We use Softmax cross-entropy loss to supervise SN, which is defined as:

$\displaystyle L_{SN}=\sum_{l}^{l=3}CE(T_{l},T_{gt}).$ (7)

As show in Fig.2, we use multi-level supervision as an auxiliary loss to
facilitate sufficient training. $T_{l}$ means the prediction result in
$l-$level.
To supervise MRN, following [Zhang and Gong(2019)], we adopt the weighted
$L_{1}$ loss and gradient loss $L_{g}$. The weighted $L_{1}$ loss is defined
as:

$L_{1}=\sum_{p}w_{p}\cdot|\alpha_{p}-\alpha_{p}^{gt}|,$ (8)

where $\alpha_{p}$ means predicted alpha value at pixel $p$. The weight
$w_{p}$ is set to 1 when $0<\alpha_{p}<1$ and set to 0.1 when $\alpha_{p}=1$
or $\alpha_{p}=0$. This operation can make the network pay attention to
unknown region. The gradient loss $L_{g}$ is used to remove the over-blurred
alpha matte, which is defined as:

$L_{g}=|\nabla_{x}(\alpha)-\nabla_{x}(\alpha^{gt})|+|\nabla_{y}(\alpha)-\nabla_{y}(\alpha^{gt})|.$
(9)

Thus $L_{MRN}$ can be written as: $L_{MRN}=L_{1}+L_{g}.$ Note that all parts
of our proposed network are trained jointly, so the overall loss function is
given as:

$L=L_{SN}+L_{MRN}.$ (10)

##  3 Experiments

###  3.1 Dataset and Evaluation Metrics

Dataset. We evaluate our proposed network on the public Adobe Composition-1k
[Xu et al.(2017)Xu, Price, Cohen, and Huang]. The training set consists of 431
foreground objects with the corresponding ground truth alpha mattes. Each
foreground image is combined with 100 background images from MS COCO dataset
[Lin et al.(2014)Lin, Maire, Belongie, Hays, and Perona] to composite the
input images. For the test set, the Composition-1k contains 50 foreground
images as well as the corresponding alpha mattes, and 1000 background images
from PASCAL VOC2012 dataset [Everingham et al.(2010)Everingham, Gool, and
Williams]. The training and test sets were synthesized through the algorithm
provided by [Xu et al.(2017)Xu, Price, Cohen, and Huang].

For the data augmentation, following [Li and Lu(2020)], the foreground object
and the alpha image will be resized to 640 $\times$ 640 images with a
probability of 0.25 to make the network see the whole foreground image instead
of a cropped snippet. Then, a random affine transformation is used.
Subsequently, we randomly crop one 512 $\times$ 512 patch from each
foreground. All of the patches are centered on an unknown region according to
the alpha images. The foreground images are then converted to HSV space, and
different jitters are imposed to the hue, saturation, and value. During
testing, the resolution of input image is $512\times 512$.
Evaluation metrics. There are four metrics used in the evaluations: SAD (sum
of absolution difference), MSE (mean square error), gradient, and connectivity
defined in [Xu et al.(2017)Xu, Price, Cohen, and Huang]. The lower values of
the metrics, the better the predicted alpha matte is. The details of gradient
and connectivity metrics can be found in [Rhemann et al.(2009)Rhemann, Rother,
and Wang], they are used to reflect the visual quality of the alpha matte when
observed by a human.
Implementation Details. We use Pytorch to implement our model. The whole
network is trained in an end-to-end manner. For loss optimization, we use the
Adam optimizer with $\beta_{1}=0.5$ and $\beta_{2}=0.999$. The learning rate
is initialized to $4\times 10^{-4}$. Warmup and cosine decay [Loshchilov and
Hutter(2017), He et al.(2019)He, Zhang, Zhang, and Zhang] are applied to the
learning rate. The network is trained for 70 epochs with a batch size of 12.
We set the weight for $L_{SN}$ to 1, for $L_{g}$ to 0.5, and for $L_{1}$ to 5.
If not specially mentioned, we use this as the default settings. Our model can
be trained from scratch without other datasets.

Figure 4: The visual comparisons on the Composition-1k testing dataset. Zoom
in for the best view.

###  3.2 Evaluation on Composition-1k Test Set

Here we compare our network with 5 traditional matting methods: Shared Matting
[Gastal and Oliveira(2010)], Learning Based [Zheng and Kambhamettu(2009)],
Global Matting [He et al.(2011)He, Rhemann, Rother, Tang, and Sun], ClosedForm
[Levin et al.(2006)Levin, Lischinski, and Weiss], KNN Matting [Chen et
al.(2012)Chen, Li, and Tang], and 7 deep learning based methods: DCNN [Cho et
al.(2016)Cho, Tai, and Kweon], DIM [Xu et al.(2017)Xu, Price, Cohen, and
Huang], AlphaGAN [Lutz et al.(2018)Lutz, Amplianitis, and Smolic], IndexNet
[Lu et al.(2019)Lu, Dai, Shen, and Xu], Late Fusion [Zhang and Gong(2019)],
HAttMatting [Qiao et al.(2020)Qiao, Liu, Yang, and Zhou], GCA [Li and
Lu(2020)]. LateFusion, HAttMatting and our proposed network are trimap-free
matting methods. For other methods, we feed RGB images and trimaps produced by
25 pixels random dilation refer to [Xu et al.(2017)Xu, Price, Cohen, and
Huang]. We compute the metrics at $800\times 800$ for all methods. The testing
settings in our work are following two trimap-free methods [Zhang and
Gong(2019), Qiao et al.(2020)Qiao, Liu, Yang, and Zhou]. The qualitative
results are illustrated in Fig.4 and quantitative comparisons are reported in
Tab.1. Because we cannot obtain the executable code or results of HAttMatting,
we do not show their qualitative results and directly cite quantitative
results from their original paper.

Table 1: Quantitative and complexity comparison on the Composition-1k testing
dataset. ’DIM-Trimap-less’ denotes the results of the DIM method without
trimap as input during training. The underline indicates the best of the
trimap-based methods. The bolded numbers represent the best performace in
trimap-free methods.

Model | Complexity | Composition-1k
---|---|---
#PARM. | FLOPs | SAD | MSE | Grad | Conn
Shared Matting | - | - | 115.20 | 0.031 | 139.88 | 121.35
Learning Based | - | - | 100.51 | 0.027 | 107.83 | 104.74
Global Matting | - | - | 121.46 | 0.034 | 125.11 | 133.23
Closed-form | - | - | 118.98 | 0.032 | 130.77 | 118.17
KNN Matting | - | - | 133.99 | 0.051 | 140.29 | 134.03
DCNN | - | - | 122.40 | 0.030 | 129.57 | 121.80
DIM-Trimap-less | 130.55M | 443.56G | 103.07 | 0.108 | 73.76 | 106.64
DIM | 130.55M | 443.56G | 33.64 | 0.008 | 30.23 | 31.92
IndexNet | 5.72M | 71.41G | 14.97 | 0.006 | 12.02 | 14.58
AlphaGAN | 40.3M | 496.36G | 20.28 | 0.010 | 21.11 | 20.46
GCA | 25.16M | 89.41G | 13.54 | 0.006 | 11.39 | 12.96
Late Fusion | 37.91M | 74.85G | 58.29 | 0.011 | 36.58 | 59.63
HAttMatting | 59.2M | 159.7G | 44.01 | 0.007 | 29.26 | 46.41
Ours | 344.82K | 0.93G | 19.59 | 0.006 | 18.70 | 20.34

As can be seen in Tab.1, our proposed method outperforms all the traditional
image matting methods by a large margin. Compared to trimap-based deep-
learning methods, our method is slightly inferior to GCA and IndexNet, and is
better than DIM and AlphaGAN. However, trimaps are strongly required during
their training and inference phase, which restricts their effectiveness in
practical applications. Compared to trimap-free methods LateFusion and
HAttMatting, our proposed method can already outperform these two methods.
Moreover, our method achieves SOTA performance with fewer parameters
($0.3\%\thicksim 7.8\%$ of large models) and FLOPs ($0.2\%\thicksim 1\%$ of
large models) than all of the above methods which will upgrade high
practicality.

In Fig.4, compared to LateFusion, our method is able to find a more complete
foreground of the matte objects since our network can exploit multi-scale
features to better capture the semantics. Because of our proposed ENA, our
method can regress more accurate alpha values in the unknown region.

###  3.3 Ablation Study

Table 2: Ablation study on CFM and ENA. We also replaced our backbone with the
existing lightweight backbone.

Backbone | Configurations | Model | Metrics
---|---|---|---
CFM | ENA | # Param. | FLOPs | SAD | MSE | Grad | Conn
Ours | $\times$ | $\times$ | 285.70K | 0.63G | 56.86 | 0.019 | 36.81 | 48.16
$\times$ | $\checkmark$ | 342.70K | 0.93G | 34.33 | 0.021 | 28.24 | 32.98
$\checkmark$ | $\times$ | 287.83K | 0.63G | 28.90 | 0.013 | 24.27 | 26.35
$\checkmark$ | $\checkmark$ | 344.93K | 0.93G | 19.59 | 0.006 | 18.70 | 20.34
MobileNetV2 | $\checkmark$ | $\checkmark$ | 37.76M | 27.02G | 47.21 | 0.014 | 30.28 | 36.46
ShuffleNetV2 | $\checkmark$ | $\checkmark$ | 30.22M | 20.52G | 48.72 | 0.015 | 33.84 | 38.21

The core idea of our method contains three parts: light-weighted backbone,
cross-level fusion module (CFM), and Efficient Non-Local Attention (ENA). To
validate the effectiveness of our proposed light-weighted backbone, we replace
it with other light-weighted backbones, including MobileNet and ShuffleNet,
and results are shown in Tab.2. Compared to these two light-weighted
backbones, our proposed backbone can achieve better performance with fewer
parameters and FLOPs. Because our backbone can achieve multi-scale feature
representations for better capturing sufficient semantics. Moreover, only
using CFM or ENA can already heavily improve the performance. Better
performance has been achieved through the combination of these two
architectures.

To further verify the validity and reasonableness of our designed efficient
non-local attention module. We trained our model with different types of
attention: common non-local attention and efficient non-local attention model.
As shown in Tab.3, compared to common-local attention, our proposed ENA is not
only more efficient but also more usable for matting tasks. Using only short-
range attention (S) or long-range attention (L) can improve model performance
because they can model the relevance between different pixels, which can help
regress alpha values. As can be seen in Tab.3, different sampling intervals
$\sqrt{k}$ lead to different results. When $\sqrt{k}=1,2$, ENA cannot fully
exploit its convenience and effectiveness, because the sampling is too
intensive. And the long-range attention will lead to some artifacts because of
irrelevant matches. Experimentally, we set $\sqrt{k}$ to 4 in all experiments.
For more comprehensive analyses of our proposed method, please refer to the
supplementary materials. Fig.5 shows the matting results for internet images
for further testing of our method.

Table 3: Ablation study for different modalities using attention. $S$ and $L$
means short- and long-range attentions respectively.

Configurations | Model | Metrics
---|---|---
# Param. | FLOPs | SAD | MSE | Grad | Conn
Common Non-Local | 421.69K | 1.33G | 25.13 | 0.010 | 20.89 | 22.67
Efficient Non-local(Ours) | S | 325.15K | 0.82G | 25.98 | 0.011 | 21.93 | 23.02
L($\sqrt{k}$=4) | 292.19K | 0.72G | 27.01 | 0.013 | 23.94 | 24.31
S+L($\sqrt{k}$=1) | 424.11K | 1.34G | 25.01 | 0.010 | 20.31 | 22.96
S+L($\sqrt{k}$=2) | 371.43K | 1.02G | 23.27 | 0.009 | 19.37 | 21.61
S+L($\sqrt{k}$=4) | 344.83K | 0.93G | 19.59 | 0.006 | 18.70 | 20.34
S+L($\sqrt{k}$=8) | 340.27K | 0.91G | 19.90 | 0.008 | 19.05 | 21.35

Figure 5: The results on real world images and the unknown region predicted by
our method.

##  4 Conclusion

In this paper, we design a two-stage trimap-free light-weighted natural image
matting model, which makes one of the earliest effort on this area. SN is
designed to capture sufficient semantics and classify the pixels into unknown,
foreground and background regions. MRN aims at capturing detailed texture
information and regressing accurate alpha values. With proposed CFM, SN can
efficiently utilize multi-scale features with less computational cost. ENA in
MRN can efficiently model the relevance between different pixels and help
regress high-quality alpha values. Experiment results on Composition-1k
testing set demonstrate that our method outperforms state-of-the-art (SOTA)
approaches in natural image matting with 1% parameters (344k) of large models.
We believe our proposed backbone, CFM and ENA can contribute to other light-
weighted computer vision tasks in the future.

## References

  * [Aksoy et al.(2017)Aksoy, Aydin, and Pollefeys] Yagiz Aksoy, Tunç Ozan Aydin, and Marc Pollefeys.  Designing effective inter-pixel information flow for natural image matting.  In _CVPR_ , 2017.
  * [Cai et al.(2019)Cai, Zhang, and Fan] Shaofan Cai, Xiaoshuai Zhang, and Haoqiang Fan.  Disentangled image matting.  In _ICCV_ , 2019.
  * [Chen et al.(2012)Chen, Li, and Tang] Qifeng Chen, Dingzeyu Li, and Chi-Keung Tang.  KNN matting.  In _CVPR_ , 2012.
  * [Chen et al.(2018)Chen, Ge, Xu, Zhang, Yang, and Gai] Quan Chen, Tiezheng Ge, Yanyu Xu, Zhiqiang Zhang, Xinxin Yang, and Kun Gai.  Semantic human matting.  In _ACM MM_ , 2018.
  * [Chen et al.(2019)Chen, Fan, and Xu] Yunpeng Chen, Haoqi Fan, and Bing Xu.  Drop an octave: Reducing spatial redundancy in convolutional neural networks with octave convolution.  In _ICCV_ , 2019.
  * [Cho et al.(2016)Cho, Tai, and Kweon] Donghyeon Cho, Yu-Wing Tai, and In-So Kweon.  Natural image matting using deep convolutional neural networks.  In _ECCV_ , 2016.
  * [Chollet(2017)] François Chollet.  Xception: Deep learning with depthwise separable convolutions.  In _CVPR_ , 2017.
  * [Chuang et al.(2001)Chuang, Curless, Salesin, and Szeliski] Yung-Yu Chuang, Brian Curless, David Salesin, and Richard Szeliski.  A bayesian approach to digital matting.  In _CVPR_ , 2001.
  * [Everingham et al.(2010)Everingham, Gool, and Williams] Mark Everingham, Luc Van Gool, and Christopher K. I. Williams.  The pascal visual object classes (VOC) challenge.  _IJCV._ , 2010.
  * [Gastal and Oliveira(2010)] Eduardo Simoes Lopes Gastal and Manuel M. Oliveira.  Shared sampling for real-time alpha matting.  _CGF_ , 29(2), 2010.
  * [He et al.(2011)He, Rhemann, Rother, Tang, and Sun] Kaiming He, Christoph Rhemann, Carsten Rother, Xiaoou Tang, and Jian Sun.  A global sampling method for alpha matting.  In _CVPR_ , 2011.
  * [He et al.(2016)He, Zhang, Ren, and Sun] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In _CVPR_ , pages 770–778, 2016.
  * [He et al.(2019)He, Zhang, Zhang, and Zhang] Tong He, Zhi Zhang, Hang Zhang, and Zhongyue Zhang.  Bag of tricks for image classification with convolutional neural networks.  In _CVPR_ , 2019.
  * [Hou and Liu(2019)] Qiqi Hou and Feng Liu.  Context-aware image matting for simultaneous foreground and alpha estimation.  In _ICCV_ , 2019.
  * [Howard et al.(2017)Howard, Zhu, and Chen] Andrew G. Howard, Menglong Zhu, and Bo Chen.  Mobilenets: Efficient convolutional neural networks for mobile vision applications.  _CoRR_ , abs/1704.04861, 2017.
  * [Hu and Clark(2019)] Guanqing Hu and James J. Clark.  Instance segmentation based semantic matting for compositing applications.  In _CRV_ , 2019.
  * [Levin et al.(2006)Levin, Lischinski, and Weiss] Anat Levin, Dani Lischinski, and Yair Weiss.  A closed form solution to natural image matting.  In _CVPR_ , 2006.
  * [Li et al.(2017)Li, Sun, Hu, and Xu] Bo Li, Zhengxing Sun, Jiagao Hu, and Junfeng Xu.  Co-saliency detection via sparse reconstruction and co-salient object discovery.  In _PCM_ , 2017.
  * [Li et al.(2019a)Li, Sun, and Guo] Bo Li, Zhengxing Sun, and Yuqi Guo.  Supervae: Superpixelwise variational autoencoder for salient object detection.  In _AAAI_ , 2019a.
  * [Li et al.(2019b)Li, Sun, Li, Wu, and Hu] Bo Li, Zhengxing Sun, Qian Li, Yunjie Wu, and Anqi Hu.  Group-wise deep object co-segmentation with co-attention recurrent neural network.  In _ICCV_ , 2019b.
  * [Li et al.(2019c)Li, Sun, Tang, and Hu] Bo Li, Zhengxing Sun, Lv Tang, and Anqi Hu.  Two-b-real net: Two-branch network for real-time salient object detection.  In _ICASSP_ , 2019c.
  * [Li et al.(2019d)Li, Sun, Tang, Sun, and Shi] Bo Li, Zhengxing Sun, Lv Tang, Yunhan Sun, and Jinlong Shi.  Detecting robust co-saliency with recurrent co-attention neural network.  In _IJCAI_ , 2019d.
  * [Li et al.(2019e)Li, Sun, Wang, and Li] Bo Li, Zhengxing Sun, Quan Wang, and Qian Li.  Co-saliency detection based on hierarchical consistency.  In _ACM MM_ , 2019e.
  * [Li et al.(2020)Li, Sun, Xu, Wang, and Yu] Bo Li, Zhengxing Sun, Junfeng Xu, Shuang Wang, and Peiwen Yu.  Saliency based multiple object cosegmentation by ensemble MIML learning.  _MTAP_ , 2020.
  * [Li and Lu(2020)] Yaoyi Li and Hongtao Lu.  Natural image matting via guided contextual attention.  In _AAAI_ , 2020.
  * [Lin et al.(2020)Lin, Ryabtsev, and Sengupta] Shanchuan Lin, Andrey Ryabtsev, and Soumyadip Sengupta.  Real-time high-resolution background matting.  _CoRR_ , abs/2012.07810, 2020.
  * [Lin et al.(2014)Lin, Maire, Belongie, Hays, and Perona] Tsung-Yi Lin, Michael Maire, Serge J. Belongie, James Hays, and Pietro Perona.  Microsoft COCO: common objects in context.  In _ECCV_ , 2014.
  * [Loshchilov and Hutter(2017)] Ilya Loshchilov and Frank Hutter.  SGDR: stochastic gradient descent with warm restarts.  In _ICLR_ , 2017.
  * [Lu et al.(2019)Lu, Dai, Shen, and Xu] Hao Lu, Yutong Dai, Chunhua Shen, and Songcen Xu.  Indices matter: Learning to index for deep image matting.  In _ICCV_ , 2019.
  * [Lutz et al.(2018)Lutz, Amplianitis, and Smolic] Sebastian Lutz, Konstantinos Amplianitis, and Aljosa Smolic.  Alphagan: Generative adversarial networks for natural image matting.  In _BMVC_ , 2018.
  * [Peng et al.(2017)Peng, Zhang, and Yu] Chao Peng, Xiangyu Zhang, and Gang Yu.  Large kernel matters - improve semantic segmentation by global convolutional network.  In _CVPR_ , 2017.
  * [Porter and Duff(1984)] Thomas K. Porter and Tom Duff.  Compositing digital images.  In _SIGGRAPH_ , 1984.
  * [Qiao et al.(2020)Qiao, Liu, Yang, and Zhou] Yu Qiao, Yuhao Liu, Xin Yang, and Dongsheng Zhou.  Attention-guided hierarchical structure aggregation for image matting.  In _CVPR_ , 2020.
  * [Rhemann et al.(2009)Rhemann, Rother, and Wang] Christoph Rhemann, Carsten Rother, and Jue Wang.  A perceptually motivated online benchmark for image matting.  In _(CVPR)_ , 2009.
  * [Sengupta et al.(2020)Sengupta, Jayaram, and Curless] Soumyadip Sengupta, Vivek Jayaram, and Brian Curless.  Background matting: The world is your green screen.  In _CVPR_ , 2020.
  * [Shahrian et al.(2013)Shahrian, Rajan, and Price] Ehsan Shahrian, Deepu Rajan, and Brian L. Price.  Improving image matting using comprehensive sampling sets.  In _CVPR_ , 2013.
  * [Sun et al.(2004)Sun, Jia, and Tang] Jian Sun, Jiaya Jia, and Chi-Keung Tang.  Poisson matting.  _ACM Trans. Graph._ , 2004.
  * [Tang et al.(2019a)Tang, Xu, Sebe, Wang, Corso, and Yan] Hao Tang, Dan Xu, Nicu Sebe, Yanzhi Wang, Jason J Corso, and Yan Yan.  Multi-channel attention selection gan with cascaded semantic guidance for cross-view image translation.  In _CVPR_ , 2019a.
  * [Tang et al.(2020a)Tang, Bai, and Sebe] Hao Tang, Song Bai, and Nicu Sebe.  Dual attention gans for semantic image synthesis.  In _Proceedings of the 28th ACM International Conference on Multimedia_ , pages 1994–2002, 2020a.
  * [Tang et al.(2020b)Tang, Bai, Torr, and Sebe] Hao Tang, Song Bai, Philip HS Torr, and Nicu Sebe.  Bipartite graph reasoning gans for person image generation.  In _BMVC_ , 2020b.
  * [Tang et al.(2020c)Tang, Bai, Zhang, Torr, and Sebe] Hao Tang, Song Bai, Li Zhang, Philip HS Torr, and Nicu Sebe.  Xinggan for person image generation.  In _ECCV_ , 2020c.
  * [Tang et al.(2020d)Tang, Xu, Yan, Torr, and Sebe] Hao Tang, Dan Xu, Yan Yan, Philip HS Torr, and Nicu Sebe.  Local class-specific and global image-level generative adversarial networks for semantic-guided scene generation.  In _CVPR_ , 2020d.
  * [Tang et al.(2021a)Tang, Liu, Xu, Torr, and Sebe] Hao Tang, Hong Liu, Dan Xu, Philip HS Torr, and Nicu Sebe.  Attentiongan: Unpaired image-to-image translation using attention-guided generative adversarial networks.  _IEEE Transactions on Neural Networks and Learning Systems_ , 2021a.
  * [Tang et al.(2019b)Tang, Aksoy, Öztireli, and Gross] Jingwei Tang, Yagiz Aksoy, Cengiz Öztireli, and Markus H. Gross.  Learning-based sampling for natural image matting.  In _CVPR_ , 2019b.
  * [Tang and Li(2020)] Lv Tang and Bo Li.  CLASS: cross-level attention and supervision for salient objects detection.  In _ACCV_ , 2020.
  * [Tang et al.(2021b)Tang, Li, Wu, Xiao, and Ding] Lv Tang, Bo Li, Yanliang Wu, Bo Xiao, and Shouhong Ding.  Fast: Feature aggregation for detecting salient object in real-time.  In _ICASSP_ , 2021b.
  * [Tang et al.(2021c)Tang, Li, Zhong, Ding, and Song] Lv Tang, Bo Li, Yijie Zhong, Shouhong Ding, and Mofei Song.  Disentangled high quality salient object detection.  In _ICCV_ , October 2021c.
  * [Wang and Cohen(2007)] Jue Wang and Michael Cohen.  Optimized color sampling for robust matting.  In _CVPR_ , 2007.
  * [Wang and Girshick(2018)] Xiaolong Wang and Ross B. Girshick.  Non-local neural networks.  In _CVPR_ , 2018.
  * [Xu et al.(2017)Xu, Price, Cohen, and Huang] Ning Xu, Brian L. Price, Scott Cohen, and Thomas S. Huang.  Deep image matting.  In _CVPR_ , 2017.
  * [Yoon and Park(2020)] Donggeun Yoon and Jinsun Park.  Lightweight deep CNN for natural image matting via similarity-preserving knowledge distillation.  _SPL_ , 2020.
  * [Zhang et al.(2018)Zhang, Zhou, and Lin] Xiangyu Zhang, Xinyu Zhou, and Mengxiao Lin.  Shufflenet:an extremely efficient convolutional neural network for mobile devices.  In _CVPR_ , 2018.
  * [Zhang and Gong(2019)] Yunke Zhang and Lixue Gong.  A late fusion CNN for digital matting.  In _CVPR_ , 2019.
  * [Zheng and Kambhamettu(2009)] Yuanjie Zheng and Chandra Kambhamettu.  Learning based digital matting.  In _ICCV_ , 2009.
  * [Zhu et al.(2017)Zhu, Chen, Wang, and Liu] Bingke Zhu, Yingying Chen, Jinqiao Wang, and Si Liu.  Fast deep matting for portrait animation on mobile phone.  In _ACM MM_ , 2017.