11institutetext: 1DISI, University of Trento; 2LTCI, Télécom Paris, Institut
polytechnique de Paris; 3Snap Inc.; 4Fondazione Bruno Kessler; 5Huawei
Technologies Ireland

# Motion-supervised Co-Part Segmentation

Aliaksandr Siarohin 1*1* Subhankar Roy 11 4* 4* Stéphane Lathuilière 22 Sergey
Tulyakov 33 Elisa Ricci 11 4 4 Nicu Sebe 11 5 5

###### Abstract

††* denotes equal contribution

Recent co-part segmentation methods mostly operate in a supervised learning
setting, which requires a large amount of annotated data for training. To
overcome this limitation, we propose a self-supervised deep learning method
for co-part segmentation. Differently from previous works, our approach
develops the idea that motion information inferred from videos can be
leveraged to discover meaningful object parts. To this end, our method relies
on pairs of frames sampled from the same video. The network learns to predict
part segments together with a representation of the motion between two frames,
which permits reconstruction of the target image. Through extensive
experimental evaluation on publicly available video sequences we demonstrate
that our approach can produce improved segmentation maps with respect to
previous self-supervised co-part segmentation approaches.

###### Keywords:

co-part segmentation; generative modeling; self-supervised learning

##  1 Introduction

Discovering objects and object parts in images is one of the fundamental steps
towards semantic understanding of visual scenes. In computer vision this
problem is referred to as semantic segmentation and is approached within a
machine learning framework as a dense labeling task, where the goal is to
assign a categorical label to each pixel of an image. In this paper we address
a more challenging problem and a special case of semantic segmentation,
referred to as co-part segmentation. The task of the co-part segmentation
problem is to identify segments corresponding to different parts within a
single object. For instance, in a human body the relevant parts correspond to
hands, legs, head and torso. Such parts are of special interest for the
automatic analysis of visual scenes since they constitute intermediate
representations, which are robust to sensor changes and appearance variations.

(a) Self-supervised training on video frames

(b) Test on images

Figure 1: Co-part segmentation via a self-supervised approach. We leverage
motion information to train a segmentation network without annotation. At
training time (a), we use frame pairs (_source_ and _target_) extracted from
the same video. The key idea consists in predicting segments from the target
frame that can be combined with a motion representation between the two frames
in order to reconstruct the target frame. At inference time (b) only the
trained segmentation model is used to predict object parts segments on a test
image.

Recently, co-part segmentation algorithms have gained popularity as they are
key-enabling components for image editing and animations tools. Several works
use automatically computed object parts for virtual try-on [43], pose-guided
generation [9], face-swap and re-enactment [27]. The vast majority of co-part
segmentation methods operates in a supervised setting [5, 22], thus requiring
a large amount of human annotated data. To overcome this limitation some works
have focused on the challenging problem of unsupervised co-part segmentation
[7, 15], demonstrating that object parts can be inferred without relying on
labeled data. While effective, these approaches are inherently limited by the
fact that only appearance-based features are exploited to compute segments. In
contrast we argue and experimentally demonstrate that more precise information
about object parts can be extracted by leveraging motion information from vast
amounts of unlabeled videos. Thus, in this paper we propose a novel video-
based self-supervised co-part segmentation method. Once trained, our proposed
framework takes a single image as input and outputs the segmentation map
indicating different object parts. Differently from previous methods [15, 7],
we assume that at training time, a large collection of videos depicting
objects within the same category is available.

Our approach is inspired by previous works on self-supervised keypoints
estimation [17, 47, 23]. The main idea behind these methods is to disentangle
the semantic and appearance representation of an object by imposing a
reconstruction objective. Specifically, these approaches operate by
considering two images of the same object in different poses, typically
referred to as source and target images, obtained through synthetic
deformations or from video sequences. In summary, they compute the appearance
representation from the source image and the semantic representation from the
target and then attempt to reconstruct the target image combining these two
representations. However, in order for these approaches to work and to
successfully extract the semantic representation, the network should have a
tight information bottleneck. If this condition is not met, the semantic
representation will be contaminated by the appearance information and the
disentanglement will be poor. We call this phenomenon leaking. Because of
_leaking_ , most self-supervised methods [17, 47, 23] are inherently limited
to focus on low-dimensional object representation, i.e. keypoints. Whereas, in
our case this bottleneck semantic representation are segmentation maps. In
fact, compared to the keypoints, the semantic segmentation maps lie in a much
higher dimensional space. Consequently, naive usage of the above methods will
lead to heavily _leaking_ models that use the segmentation map to encode the
target appearance rather than predicting representative segmentation maps.

This paper tackles this problem and introduces a novel deep architecture that
predicts meaningful semantic representations without _leaking_. Our approach
predicts segments associated with the target frame and uses them, combined
with a per-segment motion representation, to reconstruct the target image
(Fig. 1). Specifically, we propose a part-based network that operates by
deforming each part from the source image in order to match the corresponding
part in the target image. This deformation is derived from the predicted
motion representation for each part. In addition, we incorporate background
visibility mask in order to achieve better background-foreground separation.
We tested our method on two datasets: Tai-Chi-HD [32] and VoxCeleb [26].
Through an in-depth ablation study and an extensive comparison with previous
methods on co-part segmentation [7, 15], we demonstrate the effectiveness of
our approach and the rationale behind our architectural choices.

To summarize, our contribution is twofold. We introduce the problem of video-
based co-part segmentation, showing that motion information can and should be
adopted for inferring meaningful object parts. We hope that our work will
stimulate future work in this new research direction. Additionally, we propose
a novel deep architecture for co-part segmentation. Our approach advances the
state-of-the-art for self-supervised object parts discovery, demonstrating
that not only object landmarks but also complex segmentation maps can be
inferred through disentanglement within a reconstruction framework. We make
our source code publically
avalaible111https://github.com/AliaksandrSiarohin/motion-cosegmentation.

##  2 Related Works

Figure 2: Overview of our method. It consist of two modules: Segmentation
Module and Reconstruction Module. On the one hand, the Segmentation Module
takes as input a source frame $\boldsymbol{X}_{S}$ and a target frame
$\boldsymbol{X}_{T}$, sampled from the same video clip. The Segmentation
Module predicts the segmentation maps $\boldsymbol{Y}_{S}$ and
$\boldsymbol{Y}_{T}$, along with the affine motion parameters. On the other
hand, the Reconstruction Module, is in charge of reconstructing the target
images from the source image and the segmentation module outputs. First, it
computes a background visibility mask $\boldsymbol{V}$ and an optical flow
$\mathcal{F}$ from the segmentation maps and the affine motion parameters.
Finally, it reconstructs the target frame $\boldsymbol{X}_{T}$ by warping the
features of the source frame $\boldsymbol{X}_{S}$ and masking occluded
features.

Self-supervision. Self-supervised learning is a form of unsupervised learning
where supervision is attained by solving a pretext task. Recently, it has
emerged as an efficient way to learn rich image or video representations. A
popular strategy consists in modifying the input data and asking the network
to automatically recover the applied modification. Depending on the
application, a different family of transformations is applied such as
rotations [1, 10, 11], or patch shuffling [28]. Another successful approach
trains a network on a prediction task on partially observed data. The task
then consists in either predicting an unobserved part [37, 24] or the
relationship between different data parts [8]. Self-supervision can also be
based on the principle of auto-encoding [14, 35, 46]. This auto-encoding
approach is especially popular when it comes to learning disentangled
representations [31, 17, 25]. Our method also falls under this category.

Landmark Discovery. Recently, several methods have been proposed to learn
landmarks in a self-supervised manner. Thewlis et al. [39] proposed to learn a
keypoint detector relying on geometric priors, where their training loss
enforces the network to be equivariant to affine and thin spline
transformations. Zhang et al. [47] extended this approach using an auto-
encoder architecture where the bottleneck information is the landmark
locations. A similar auto-encoding strategy is used in [17]. In the context of
video generation, Siarohin et al. [31, 32] proposed animation methods that
discover landmarks specifically designed to encode motion information.
Similarly to Siarohin et al. [31, 32], our method is also based on the motion
information. However, while animation methods assumes similar part based
motion model, those methods require at least 2 frames to predict keypoint
neighbourhoods, even during the inference. This makes predictions of Siarohin
et al. [31, 32] highly dependent on the other frame in a pair. For example (i)
if there is not enough motion between 2 frames then these methods will not
predict any neighbourhoods; (ii) image animations methods may include a lot of
background motion in the motion prediction. In contrast our approach encodes
more semantically meaningful parts by making independent frame-based
predictions.

Unsupervised Co-Part Segmentation. Most co-part segmentation methods operate
in a supervised setting [5, 22]. However, here we only review previous works
on unsupervised co-part segmentation. Recent methods propose to study the
internal feature representations derived from pre-trained ConvNets to find
object part information [2, 12]. Collins et al. [7] described an approach to
estimate segments corresponding to object parts by applying Non-negative
Matrix Factorization [36] (NMF) on features computed from a pretrained
ConvNet. This requires a costly optimization process at inference time and
also do not incorporate geometric priors. Xu et al. [42] proposed a deep model
to discover object parts and the associated hierarchical structure and
dynamical model from unlabeled videos. However, they assume that precomputed
motion information is available. Differently, our method estimates the motion
within the same architecture. Recently, Hung et al. [15] proposed a self-
supervised deep learning approach for co-part segmentation from static images.
A network is trained using several losses in order to impose geometric,
equivariance and semantic consistency constraints. Hung et al. [15] rely on a
large collection of unlabelled images of the same object category. In this
work, we also adopt a self-supervised learning strategy for co-part
segmentation. However, rather than considering static images, we propose to
train our model on a collection of unlabelled videos. Our motivation is to
leverage motion information with the purpose of obtaining segments that
correspond to group of pixels associated to object parts moving together.

As a final remark, we would like to point out that co-part segmentation should
not be confused with co-image and co-video segmentation [21, 6]. In co-image
and co-video segmentation the task is to discover pixels corresponding to
common foreground objects of the same class within images and videos,
respectively. In both cases, the final goal is to segment a single foreground
object and not to discover object parts.

##  3 Method

In this work, we are interested in training a deep neural network that at
inference time takes a single image as input and outputs co-part segments. At
training time, we assume a large collection of videos containing objects of
the same category is available. The proposed model is trained using pairs of
frames from the same video: $\boldsymbol{X}_{S}\in\mathbb{R}^{3\times H\times
W}$ and $\boldsymbol{X}_{T}\in\mathbb{R}^{3\times H\times W}$ of spatial
dimension $H\times W$ and are referred to as _source_ and _target_ frames,
respectively. Note that the pair of frames are randomly sampled from the
entire video sequence.

The overall pipeline is illustrated in Fig. 2. Our framework is composed of
two main modules. The first module, named _Segmentation Module_ , is in charge
of extracting semantic segments and the motion corresponding to those segments
from the input frames. We employ a neural network to independently segment the
frames $\boldsymbol{X}_{S}$ and $\boldsymbol{X}_{T}$ into $K\\!+\\!1$ parts,
i.e. $K$ segments for the foreground and one for the background. The predicted
segments are represented using $K\\!+\\!1$ channel tensors, i.e.
$\boldsymbol{Y}_{S}\in[0,1]^{K+1\times H^{\prime}\times W^{\prime}}$ and
$\boldsymbol{Y}_{T}\in[0,1]^{K+1\times H^{\prime}\times W^{\prime}}$
associated to the source and the target frames, respectively. Note that, the
segmentation source $\boldsymbol{Y}_{S}$ is predicted independently from the
target segmentation and, consequently, the proposed model can be used for
single image segmentation at inference time. To reduce time for computing the
segmentation masks we use a smaller resolution $H^{\prime}\times
W^{\prime}$.We use a U-Net-like architecture with a channel-wise softmax layer
at the output to obtain tensors that can be interpreted as part-assignment
confidence maps.

In our self-supervised setting, we also aim at training a generator network
that reconstructs the target frame from the source frame and the target
segmentation. However, reconstructing the target frame directly from its
segmentation maps tends to provide pathological solutions that use
$\boldsymbol{Y}_{T}$ to encode $\boldsymbol{X}_{T}$, rather than predicting
desired semantic segmentation. In contrast to unsupervised landmark detection,
this problem appears particularly severe in the case of segmentation because
of the relatively high dimension of $\boldsymbol{Y}_{T}$ (typically
$64\\!\times\\!64$). In this case, $\boldsymbol{Y}_{T}$ is too large to serve
as the information bottleneck allowing the information about the target frame
to _leak_ to the generator. Therefore, we use the estimated segmentation to
reconstruct the target frame in an indirect way. More precisely, we propose to
reconstruct the target frame by modeling the motion between the source and the
target frames and we include the segmentation $\boldsymbol{Y}_{T}$ into our
model of motion. To this end, we introduce the segment motion that describes
the motion between the source and the target frames in each segment. The
details are provided in Sec. 3.1.

Finally, the second module, named Reconstruction Module, is in charge of
reconstructing the target frame from the source frame and the segmentation
module outputs. With the help of segment motion we estimate optical flow
between source and target frames. Importantly, we do not use an external
optical flow estimator, and rely solely on segmentation maps and segment
motions. Optical flow
$\mathcal{F}\\!:\mathbb{R}^{2}\\!\rightarrow\\!\mathbb{R}^{2}$ maps each pixel
location in $\boldsymbol{X}_{T}$ to its corresponding location in
$\boldsymbol{X}_{S}$. In other words, $\mathcal{F}$ is the estimated backward
optical flow between the source and the target frames. Based on segmentation
maps we also estimate a background visibility mask
$\boldsymbol{V}\in\mathbb{R}^{H\times W}$. Background visibility mask
indicates parts of the background that were occluded in the source frame.
Details concerning optical flow and background visibility mask are provided in
Sec. 3.2. The target frame is reconstructed from the source frame using an
encoder-decoder network. To this aim, we align the intermediate feature
representation of the source frame with the pose of the object in the target
frame using the estimated optical flow $\mathcal{F}$. The overall architecture
is trained by minimizing a reconstruction loss between the target and
reconstructed target frames. The reconstruction module and the training
procedure are described in Sec. 3.3.

###  3.1 Segmentation Module

In this section, we detail how we model the motion of each segment. We design
our model with the aim of obtaining segments that group together pixels
corresponding to object parts that move together. The motion of the pixels
within each segment is modeled by an affine transformation. In other words,
the optical flow $\mathcal{F}$ can be approximated by an affine transformation
on the support corresponding to each segmented part. The encoder network
should provide segments such that our affine model is valid. Otherwise, it
would lead to wrong optical flow $\mathcal{F}$ and, subsequently, to high
reconstruction loss.

Formally, let $\boldsymbol{Y}_{T}^{k}\in[0,1]^{H^{\prime}\times W^{\prime}}$
be the $k^{th}$ channel of the segmentation $\boldsymbol{Y}_{T}$. Even if
$\boldsymbol{Y}_{T}$ and $\boldsymbol{Y}_{S}$ are tensors with continuous
values, for the sake of notation, here we assume that they are binary tensors
(this point is further discussed in Sec. 3.2). We define $\mathcal{Y}_{T}^{k}$
as the set of locations associated to the segment $k$,
$\mathcal{Y}_{T}^{k}=\\{z\in[1,H^{\prime}]\times[1,W^{\prime}]~{}|~{}\boldsymbol{Y}_{T}^{k}[z]=1\\}$.
In a similar way, we introduce $\mathcal{Y}_{S}^{k}$ for the source
segmentation. Assuming that the motion of each segment follows an affine
model, this implies that there exists $\boldsymbol{A}\in\mathbb{R}^{2\times
2}$ and $\beta\in\mathbb{R}^{2}$ such that:

$\vspace{-0.1cm}\forall
z\in\mathcal{Y}_{T}^{k},~{}\mathcal{F}(z)=\boldsymbol{A}z+\beta.$ (1)

In our context, we need to explicitly estimate the affine parameters
$\boldsymbol{A}$ and $\beta$ in order to obtain $\mathcal{F}$. To this end, we
rewrite Eq. (1) as follows:

$\vspace{-0.2cm}\forall
z\in\mathcal{Y}_{T}^{k},~{}\mathcal{F}(z)=p_{S}^{k}+\boldsymbol{A}_{S}^{k}{\boldsymbol{A}_{T}^{k}}^{-1}\big{(}z-p_{T}^{k}\big{)}$
(2)

where $\boldsymbol{A}_{S}^{k}\in\mathbb{R}^{2\times 2}$,
${\boldsymbol{A}_{T}^{k}}^{-1}\in\mathbb{R}^{2\times 2}$,
$p_{S}^{k}\in\mathbb{R}^{2}$ and $p_{T}^{k}\in\mathbb{R}^{2}$ are estimated by
the segmentation network, see Sup. Mat. A for details. The motivation for this
re-writing is to separate terms that are estimated only from the source (i.e.
$p_{S}^{k}$ and $\boldsymbol{A}_{S}^{k}$) from those estimated only from the
target (i.e. $p_{T}^{k}$ and $\boldsymbol{A}_{T}^{k}$), see [32].

###  3.2 Flow Model

We now detail how optical flow $\mathcal{F}$ is obtained by combining the
segment motion representations and the predicted segmentation. For each
segment, we employ the model given by Eq. (2) to obtain $K$ optical flow
fields $\boldsymbol{F}^{k}\in\mathbb{R}^{2\times H^{\prime}\times W^{\prime}}$
of $\mathcal{F}$ in each respective segment $\mathcal{Y}_{F}^{k}$. The tensor
$\boldsymbol{F}^{k}$ can be interpreted as the partial optical flow for the
pixels of the segment $\mathcal{Y}_{T}^{k}$ between the source and target
frames. Note that the last optical flow field $\boldsymbol{F}^{K+1}$ is
associated to the background. We assume a static background and consequently
set $\boldsymbol{F}^{K+1}=z$. We consider that each object part corresponds to
a segment in $\boldsymbol{Y}_{T}$ and that its motion can be encoded by its
corresponding segment motion descriptor. Consequently, we obtain the following
discretized estimator $\hat{\boldsymbol{F}}$ of the field $\mathcal{F}$:

$\vspace{-0.2cm}\hat{\boldsymbol{F}}=\sum_{k=1}^{K+1}\boldsymbol{Y}_{F}^{k}\otimes\boldsymbol{F}^{k}$
(3)

where $\otimes$ denotes the element-wise product. In Eq. (3), the segmentation
mask $\boldsymbol{Y}_{T}^{k}$ is used to assign a partial optical flow field
$\boldsymbol{F}^{k}$ to each location. In other words, the encoder network
must output segmentation masks that allows to correctly assign partial optical
flow field $\boldsymbol{F}^{k}$ to each location $z$. Because of this, the
segmentation mask must gather together locations that move in the same way. In
practice, $\boldsymbol{Y}_{T}^{k}$ are continuous tensors. Consequently, Eq.
(3) can be interpreted as a soft assignment model that selects motion vector
for each pixel location. The use of continuous tensors facilitates gradient-
based optimization.

Background visibility mask. Additionally we estimate the background visibility
mask $\boldsymbol{V}\in[0,1]^{H^{\prime}\times W^{\prime}}$. It indicates
which pixels of the background are occluded by the foreground object in the
source frame. Later, information about the occluded parts of the background
will be suppressed in the _Reconstruction Module_. Usage of the background
visibility mask enforces superior background/foreground separation. Poor
segmentation maps will lead to poor background visibility maps and this will
affect the final reconstruction quality. $\boldsymbol{V}$ is computed from the
source and target segmentations.

We estimate the foreground object locations in the source frame by merging the
K object parts
$\boldsymbol{O}_{S}\\!=\\!\sum_{k=1}^{K}\boldsymbol{Y}_{S}^{k}$. Occluded
locations are then defined as locations corresponding to the background in the
target ($\boldsymbol{Y}_{T}^{K+1}\\!=\\!1$) but to the foreground in the
source frame. Formally, the background visibility mask is:

$\vspace{-0.3cm}\boldsymbol{V}=1-(\boldsymbol{Y}_{T}^{K+1}\otimes\boldsymbol{O}_{S})$
(4)

Note that the background visibility provides an important regularization
necessary for reducing parts of the background incorrectly classified as the
foreground, which we later refer as _false fg_. Indeed, a large portion of
_false fg_ in the source frame means significant amount of information useful
for background reconstruction will be suppressed. In turn, optimization of the
reconstruction error will lead to shrinkage of _false fg_. Compared to the
occlusion map of [32] that specifies the parts that need to be inpainted thus
improving the quality of reconstruction, our background visibility does not
improve reconstruction. In fact, it is even harms it. We note that, providing
background visibility to the generator is be problematic since information
about the target frame will _leak_ through $\boldsymbol{Y}_{T}^{K+1}$. To this
end, we prevent the gradient flow through the $\boldsymbol{Y}_{T}^{K+1}$.

###  3.3 Reconstruction and Training

In this section, we describe how we make use of the estimated optical flow to
reconstruct the target frame. Then, we provide the details of our training
procedure. We employ a generator network $G$ that reconstructs the target
frame by warping the source frame features according to the estimated optical
flow. Similar strategy has been adapted in several recent pose-guided
generation frameworks [44, 33, 45, 31, 13]. $G$ has an encoder-decoder
architecture with a deformation layer in its bottleneck. This deformation
layer takes as input a feature map $\boldsymbol{\xi}\in\mathbb{R}^{C\times
H^{\prime}\times W^{\prime}}$, the estimated optical flow
$\hat{\boldsymbol{F}}\in\mathbb{R}^{2\times H^{\prime}\times W^{\prime}}$ and
the background visibility mask ${\boldsymbol{V}}$. We use ${\boldsymbol{V}}$
to mask out the feature map locations not visible according to the
segmentation maps. Thus, we reduce the impact of the features located in the
non-visible parts. Note that if the features in that part were useful for
reconstruction (this happens in the case of _false fg_), the source
segmentation will be penalized. The deformation layer output is defined as
follows:

$\vspace{-0.1cm}\boldsymbol{\xi}^{\prime}=\boldsymbol{V}\odot\mathcal{W}(\boldsymbol{\xi},\hat{\boldsymbol{F}})$
(5)

where $\mathcal{W}(\cdot,\cdot)$ denotes the back-warping operation and
$\odot$ denotes the Hadamard product. The warping operation is implemented in
a differentiable manner using the bilinear sampler of Jaderberg et al. [16].

Training Losses. We train our model in an end-to-end fashion. The
reconstruction loss is our main driving loss. It is based on the perceptual
loss of Johnson et al.  [18] and uses a pre-trained VGG-19 [34] network to
asses reconstruction quality. Given the input target frame
$\boldsymbol{X}_{T}$ and its corresponding reconstructed frame
$\bar{\boldsymbol{X}_{T}}$, the reconstruction loss is defined as:

$\vspace{-0.2cm}\mathcal{L}_{rec}(\bar{\boldsymbol{X}_{T}},\boldsymbol{X}_{T})=\sum_{i=1}^{I}\left|\Phi_{i}(\bar{\boldsymbol{X}_{T}})-\Phi_{i}(\boldsymbol{X}_{T})\right|,$
(6)

where $\Phi_{i}(\cdot)$ denotes the $i^{th}$ channel feature extracted from a
specific VGG-19 [34] layer and $I$ denotes the number of feature channels in
this layer. In addition, this loss is computed at several resolutions,
similarly to MS-SSIM [40, 38].

Furthermore, following previous unsupervised keypoint detection methods [17,
48] we employ the equivariance loss to force the network to predict segment
motion representations that are consistent with respect to known geometric
transformations. As in [32], we impose the equivariance loss with respect to
affine transformations defined by $p^{k}$ and $\boldsymbol{A}^{k}$, we refer
to this loss as $\mathcal{L}_{eq}$. Finally, the overall loss function is the
summation of $\mathcal{L}_{rec}$ and $\mathcal{L}_{eq}$ To facilitate the
convergence and improve the performance we start with the pre-trained weights
of [32], but we believe any unsupervised keypoint detection method [31, 23,
17] can be used. We also experimented with training from scratch. We observed
that the model does learn meaningful segments, however, their precision is
sub-optimal.

##  4 Experiments

_Naive_ |  |  |  |
---|---|---|---|---
_Shift-only_ |  |  |  |
_Affine-only_ |  |  |  |
_$\boldsymbol{O}$ -back-prop_ |  |  |  |
_Full_ |  |  |  |

Figure 3: Visual comparison of our method and baselines. In the even columns
segmentation mask is depicted, while in the odd columns original image with
overlayed segmentation is shown. Columns one through four corresponds to _Tai-
Chi-HD_ dataset, and columns five through eight show _VoxCeleb_ dataset.

Table 1: Comparison of _landmark regression MAE_ and _foreground segmentation
IoU_ scores between our and competitors methods (a). Comparison between our
and baselines methods (b). Comparison of _Full_ model with a different number
of segments on _VoxCeleb_ dataset (c).

|  | _Tai-Chi-HD_ |  | _VoxCeleb_
---|---|---|---
Method | MAE $\downarrow$ | IoU $\uparrow$ |  | MAE $\downarrow$ | IoU $\uparrow$
DFF [7] | 494.48 | - |  | 1254.25 | -
SCOPS [15] | 411.38 | 0.5485 |  | 663.04 | 0.5045
Ours | 389.78 | 0.7686 |  | 424.96 | 0.9135
_Tai-Chi-HD_ _VoxCeleb_ Model MAE $\downarrow$ IoU $\uparrow$ MAE $\downarrow$
IoU $\uparrow$ _Naive_ 629.10 0.1956 2245.81 0.7896 _Shift-only_ 365.56 0.6698
525.92 0.8944 _Affine-only_ 363.13 0.6666 433.80 0.8915 _$\boldsymbol{V}$
-back-prop_ 411.80 0.1956 600.27 0.7896 _Full_ 389.78 0.7686 424.96 0.9135 (b)
(a)
|  | _Number of segments_
---|---
| 1 | 5 | 10 | 15
MAE $\downarrow$ | 2177.42 | 656.63 | 424.96 | 355.38
IoU $\uparrow$ | 0.6834 | 0.9036 | 0.9135 | 0.9087
(c) |

Datasets. We evaluate our method on two datasets: _Vox-Celeb_ and _Tai-Chi-
HD_. _VoxCeleb_ is a large scale video dataset of human faces, extracted from
YouTube video interviews. We use the preprocessing described in Siarohin et
al. [32]. After the preprocessing, we obtain 12331 training videos and 444
test videos. The length of each video varies from 64 to 1024 frames. _Tai-Chi-
HD_ dataset is a collection of videos. Each video depicts a person performing
Tai-Chi actions. This dataset consists of 3049 training videos and 285 test
videos, with length varying from 128 to 1024 frames. These datasets are
selected because we require many videos containing objects of the same class
but with diverse appearances.

Evaluation Protocol. Unsupervised part segmentation cannot be trivially
evaluated with annotated segmentation or with segmentation obtained using a
supervised approach. Indeed, when assigning ground-truth labels to pixels, the
definition of each co-part is very subjective. Specially, depending on the
application, we may prefer to define the co-part segments in a different way.
Consequently, the segmentation maps obtained by the unsupervised method may
not correspond to the segmentation masks that a human would draw. Therefore,
in this work we follow the evaluation protocol indicated by Hung et al. [15]
and we adopt two proxy metrics: _landmark regression MAE_ and _foreground
segmentation IoU_.

The aim of the _landmark regression MAE_ is to evaluate whether the extracted
semantic parts are meaningful and consistent among different images. In more
details, we compute the center of mass $M^{k}$ for each foreground segment
$k$. Then we fit a linear regression model from $M^{k}$ to the ground truth
landmarks and compute the mean average error (MAE). Following Hung et al.
[15], we use 5000 images for fitting the regression model, and 300 other
images for computing the MAE. Since the datasets we consider come without any
annotations, we use as ground truth, the landmarks computed with the method of
Bulat et al. [3] for _VoxCeleb_ and Cao et al. [4] for _Tai-Chi-HD_.

The other metric, _Foreground segmentation IoU_ , evaluates whether extracted
parts separate the object from the background. For _VoxCeleb_ we extract the
corresponding ground truth foreground using the face parser of Lie et al.
[20], while for _Tai-Chi-HD_ , we use Detectron2 [41] (masks corresponding to
person class). From the predicted co-part, we estimate the foreground with
$\boldsymbol{O}_{T}\\!=\\!\sum_{k=1}^{K}\boldsymbol{Y}_{S}^{k}$ and compute
intersection over union (IoU) with the ground truth foreground.

Baselines. We compare our method with two state-of-the-art methods for co-part
image segmentation: SCOPS [15] and DFF [7]. To the best of our knowledge,
these baselines are the only methods on unsupervised co-part segmentation
based on deep architectures222Please note that we could not compare with [42]
since their training code was not available at the time of the submission..
Concerning SCOPS [15], we use the implementation provided by the authors
considering the same hyper-parameters provided by the authors, except the
number of parts that we set to $K=10$. We train SCOPS model on _Tai-Chi-HD_
and _VoxCeleb_ by treating all the frames of all the videos as images.
Importantly, SCOPS [15] relies on saliency maps that are obtained using the
unsupervised method described in [49]. However, the SCOPS authors’
implementation only provides pre-computed saliency maps for CelebA [22]. To
obtain the saliency for our datasets, we estimate the saliency maps using
[49]. Additional details provided in Sup. Mat. A.1.

To compare with DFF [7], we use the official implementation and employ the
default parameters except for the number of parts that we set to $K=10$. One
of the significant limitations of the DFF [7] is the fact that matrix
factorization requires all the images at once without distinction between
training and test. This leads to two consequences. First, the dataset must be
sub-sampled to fit in the memory. Second, train and test images should be
combined for matrix factorization, thereby making it impossible to apply on a
single test image. For these reasons we use all $5000+300$ images for DFF [7]
matrix factorization. See additional details in Sup. Mat. A.1.

###  4.1 Ablation Study

We first show the results of our ablation study which demonstrate the
importance of the architectural choices made in the design of our network.

First, we implement a _Naive_ baseline where we try to directly reconstruct
the target image from the target segmentation map $\boldsymbol{Y}_{T}$ and
source image $\boldsymbol{X}_{S}$. In this case, the segmentation map
$\boldsymbol{Y}_{T}$ is provided to the Reconstruction Module via
concatenation with the source image feature in the bottleneck layer. Then, we
consider another baseline to evaluate a simplified motion representation,
where transformations are only shifting and no background visibility masks are
used. In this case, we consider that $\boldsymbol{A}_{S}=\boldsymbol{I}$,
$\boldsymbol{A}_{T}=\boldsymbol{I}$ ($\boldsymbol{I}$ is the identity matrix)
and $\boldsymbol{V}=1$. We refer to this model as _Shift-only_. The third
model uses a more complex motion representation since it includes the affine
term in Eq. (1). However it does not use background visibility mask to
reconstruct the frame. This model is referred to as _Affine-only_. Finally we
introduce two baselines that use background visibility masks. In
_$\boldsymbol{V}$ -back-prop_, we propagate the gradient through the
background visibility mask at the training time. Conversely, in _Full_ , we
stop gradient propagation trough the background visibility mask as explained
in Sec. 3.3.

DFF [7] |  |  |  |
---|---|---|---|---
SCOPS [15] |  |  |  |
Ours |  |  |  |

Figure 4: Visual results on _VoxCeleb_ dataset. In the even columns the
segmentation mask is depicted, while in the odd columns the overlayed
segmentation is shown. Our method produces more meaningful face segments that
cover most part of the face, when compared to the state-of-the-art methods.

The quantitative results associated to our ablation study are reported in Tab.
1 (b), while qualitative results can be found in Fig. 3. First, from Fig. 3,
we clearly see that the _Naive_ model outputs really poor segmentation maps
that do not encode any semantic information. To better understand this result,
we inspect the reconstruction loss on the training set. We observe that the
obtained reconstruction loss is significantly lower than the one obtained by
our full model (i.e. 39.63 vs 87.70 in __VoxCeleb__ dataset). This indicates
that, in the case of Naive, appearance information _leaks_ into the semantic
map leading to really good reconstruction of the target image but very poor
segmentation quality.

DFF [7] |  |  |  |
---|---|---|---|---
SCOPS [15] |  |  |  |
Ours |  |  |  |

Figure 5: Visual results on _Tai-Chi-HD_ dataset. In the even columns the
segmentation mask is depicted, while in the odd columns the overlayed
segmentation is shown. Our method produces more meaningful and seamless body
segments when compared to the state-of-the-art methods.

Compared to _Naive_ , we observe that _Shift-only_ predicts segments with
clearer semantic correspondences. This shows the effectiveness of our part
based motion decomposition. Nevertheless, because of simplified motion
representation, we notice that the _Shift-only_ model can not predict precise
segments (for e.g. the legs in _Tai-Chi-HD_ , see Fig. 3). Moreover, the
borders of the segments are not smooth (see the red segment in the Fig. 3 for
_VoxCeleb_). This critical observation is confirmed by the quantitative
evaluation since we observe a relatively poor _foreground segmentation IoU_
score (see Tab. 1 (a)). When we employ a richer motion representation,
corresponding to _Affine-only_ baseline, the extracted segments are more
accurate (see the legs in the second _Tai-Chi-HD_ , Fig. 3). Quantitatively,
we notice better _landmark MAE_ scores on both datasets.

Regarding the use of background visibility masks, we observe that the addition
of the background visibility masks ($\boldsymbol{V}$_-back-prop_) leads to the
same issue as in the _Naive_. Low-level information _leaks_ into the semantic
map. Indeed, when computing the reconstruction loss, we observe that
$\boldsymbol{V}$_-back-prop_ can reconstruct the image more accurately than
our full model (46.30 vs 87.70). Finally, the _Full_ model corrects this issue
and leads to significant improvements in the localization of the foreground
object (see _foreground segmentation IoU_ in Tab. 1 (a)).

In the second series of experiments, we evaluate the impact of the number of
segments K. Quantitative evaluation is reported in Tab. 1 (c). We observe that
when we introduce more segments, more fine-grained details appear in the
estimated segmentations. Moreover, as more parts are used, the _landmark
regression MAE_ becomes lower. This behaviour is reasonable since predicting
more segments leads to a richer description of the object structure. However,
concerning _foreground segmentation IoU_ , we observe that the model with 5
segments already acceptably separates foreground and background and the
_foreground segmentation IoU_ does not improve significantly when more
segments are added (see Tab. 1 (c)). For visual results please refer to Sup.
Mat. B.

###  4.2 Comparison with State of the Art

We compare our method with two previous approaches for co-part segmentation:
SCOPS [15] and DFF [7]. Quantitative results are reported in Tab. 1 (a), while
qualitative comparison is shown in Fig. 4 and Fig. 5. From the figures it is
easy to observe that DFF is not able to localize the foreground object.
Furthermore, many predicted segmentation maps are fragmented. Compared to DFF,
our method produces semantic parts which are more consistent over different
object and where the main object is clearly separated from the background.
Note that since DFF [7] has no explicit way of handling the background, we do
not report _foreground segmentation IoU_ for it. In terms of _landmark
regression MAE_ , the benefit of our approach is clearly evident (see Tab. 1
(b)).

Concerning SCOPS [15], in case of _Tai-Chi-HD_ , the predicted parts are not
consistent in the different images and large portions of the background are
misclassified. Differently in our method, only small portions of the
background near to the object boundaries are included in the foreground
segments. Note however, that performance of SCOPS [15] heavily depends on
unsupervised saliency prediction method [49]. Overall, the segments from our
method are much more consistent and semantically meaningful. The better
performance of our method are also confirmed by quantitative results (see Tab.
1 (b)). For both datasets and both metrics, we obtain consistently better
scores. The gain of our method is especially evident on the _VoxCeleb_
dataset. For additional visual results please refer to Sup. Mat. C and our
supplementary video333https://youtu.be/RJ4Nj1wV5iA for more qualitative
result.

|  |
---|---|---
_5 segments_ |  |
|  |
_10 segments_ |  |

Figure 6: Visual results of video-editing. In the odd rows the target frames
from video sequences are depicted, alongside the masks of interest (in the
right bottom corners) intended to be swapped. In the even columns the source
image is shown, followed by the generated frames containing the indicated
parts swapped from the source. Results achieved with models for K=5 and K=10
are depicted. Best viewed with digital zoom.

###  4.3 Application: Face Part Swapping

Unsupervised discovery of body parts allows us to use our model for
interesting media applications such as video-editing, face-swapping, face-
editing and so on, without having to rely on externally trained parsers,
unlike [20]. Specifically, in video-editing the user can select one or
multiple semantic parts in a video sequence and swap them with the
corresponding semantic parts derived from another source image. In Fig. 6 we
show the results for video-editing obtained with our method, by operating in a
completely unsupervised setting.

In details, the video-editing results are generated on a frame-by-frame basis,
wherein our segmentation module predicts the semantic masks for both the
target frame and source image, along with the flow $\mathcal{F}$ for each
mask. This part-based $\mathcal{F}$ is then used to align the source features
with the corresponding target features, by deforming the former. The mask of
interest to be swapped (e.g. lower jaw) is then utilized to discard the lower
jaw features from the target frame and fill the void with the relevant
features from the source image. The resulting intermediate features are then
decoded by our reconstruction module to obtain the resulting image where the
lower jaw appears to be swapped from the source image (see Fig. 6 top right).
Since the results of video-editing depends on the application of interest, we
tested with two models that are trained to predict semantic parts with K=5 and
K=10, respectively. We use K=5 when we want to change large facial parts such
as blonde hair $\rightarrow$ black hair and clean shave $\rightarrow$ goatee
beard (see Fig. 6 top). On the other hand, we accomplish a fine-grained
editing by using a model with K=10 where we alter only small parts of the
face, as in red lips $\rightarrow$ purple lips and black eyes $\rightarrow$
turquoise eyes (see Fig. 6 bottom). Note that, we used continuous masks for
video-editing application to obtain a smoother transition between regions and
to avoid an abrupt change in skin tone. Please see Sup. Mat. D and
supplementary video for additional video-editing results.

##  5 Conclusions

We presented a novel self-supervised approach for co-part segmentation that
leverages motion information automatically extracted from video streams in
order to predict better segmentation maps from still images. We discussed how
the critical issue for developing a self-supervised reconstruction-based model
for segmentation lies into the _leaking_ issues, and thus, we introduced a
novel part-based motion formulation that permits to overcome this problem.
Additionally, we add background visibility mask to our model, thus promoting
the better separation of the object with respect to the background. Through an
extensive experimental evaluation and comparisons with state of the art
methods, we demonstrate the validity and superiority of our approach.

## References

  * [1] Agrawal, P., Carreira, J., Malik, J.: Learning to see by moving. In: CVPR (2015)
  * [2] Bau, D., Zhou, B., Khosla, A., Oliva, A., Torralba, A.: Network dissection: Quantifying interpretability of deep visual representations. In: CVPR (2017)
  * [3] Bulat, A., Tzimiropoulos, G.: How far are we from solving the 2d & 3d face alignment problem? (and a dataset of 230,000 3d facial landmarks). In: ICCV (2017)
  * [4] Cao, Z., Simon, T., Wei, S.E., Sheikh, Y.: Realtime multi-person 2d pose estimation using part affinity fields. In: CVPR (2017)
  * [5] Chen, X., Mottaghi, R., Liu, X., Fidler, S., Urtasun, R., Yuille, A.: Detect what you can: Detecting and representing objects using holistic models and body parts. In: CVPR (2014)
  * [6] Chiu, W.C., Fritz, M.: Multi-class video co-segmentation with a generative multi-video model. In: CVPR (2013)
  * [7] Collins, E., Achanta, R., Susstrunk, S.: Deep feature factorization for concept discovery. In: ECCV (2018)
  * [8] Doersch, C., Gupta, A., Efros, A.A.: Unsupervised visual representation learning by context prediction. In: CVPR (2015)
  * [9] Dong, H., Liang, X., Gong, K., Lai, H., Zhu, J., Yin, J.: Soft-gated warping-gan for pose-guided person image synthesis. In: NIPS (2018)
  * [10] Feng, Z., Xu, C., Tao, D.: Self-supervised representation learning by rotation feature decoupling. In: CVPR (2019)
  * [11] Gidaris, S., Singh, P., Komodakis, N.: Unsupervised representation learning by predicting image rotations. In: ICLR (2018)
  * [12] Gonzalez-Garcia, A., Modolo, D., Ferrari, V.: Do semantic parts emerge in convolutional neural networks? IJCV (2018)
  * [13] Grigorev, A., Sevastopolsky, A., Vakhitov, A., Lempitsky, V.: Coordinate-based texture inpainting for pose-guided image generation. In: CVPR (2019)
  * [14] Hinton, G.E., Salakhutdinov, R.R.: Reducing the dimensionality of data with neural networks. Science (2006)
  * [15] Hung, W.C., Jampani, V., Liu, S., Molchanov, P., Yang, M.H., Kautz, J.: Scops: Self-supervised co-part segmentation. In: CVPR (2019)
  * [16] Jaderberg, M., Simonyan, K., Zisserman, A., et al.: Spatial transformer networks. In: NIPS (2015)
  * [17] Jakab, T., Gupta, A., Bilen, H., Vedaldi, A.: Unsupervised learning of object landmarks through conditional image generation. In: NIPS (2018)
  * [18] Johnson, J., Alahi, A., Fei-Fei, L.: Perceptual losses for real-time style transfer and super-resolution. In: ECCV (2016)
  * [19] Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. In: ICLR (2014)
  * [20] Lee, C.H., Liu, Z., Wu, L., Luo, P.: Maskgan: Towards diverse and interactive facial image manipulation. arXiv preprint arXiv:1907.11922 (2019)
  * [21] Li, W., Jafari, O.H., Rother, C.: Deep object co-segmentation. In: Asian Conference on Computer Vision. pp. 638–653. Springer (2018)
  * [22] Liu, Z., Luo, P., Wang, X., Tang, X.: Deep learning face attributes in the wild. In: ICCV (2015)
  * [23] Lorenz, D., Bereska, L., Milbich, T., Ommer, B.: Unsupervised part-based disentangling of object shape and appearance. In: CVPR (2019)
  * [24] Luo, Z., Peng, B., Huang, D.A., Alahi, A., Fei-Fei, L.: Unsupervised learning of long-term motion dynamics for videos. In: CVPR (2017)
  * [25] Ma, L., Sun, Q., Georgoulis, S., Van Gool, L., Schiele, B., Fritz, M.: Disentangled person image generation. In: CVPR (2018)
  * [26] Nagrani, A., Chung, J.S., Zisserman, A.: Voxceleb: a large-scale speaker identification dataset. In: INTERSPEECH (2017)
  * [27] Nirkin, Y., Keller, Y., Hassner, T.: Fsgan: Subject agnostic face swapping and reenactment. In: CVPR (2019)
  * [28] Noroozi, M., Favaro, P.: Unsupervised learning of visual representations by solving jigsaw puzzles. In: ECCV (2016)
  * [29] Robinson, J.P., Li, Y., Zhang, N., Fu, Y., Tulyakov, S.: Laplace landmark localization. In: ICCV (2019)
  * [30] Ronneberger, O., Fischer, P., Brox, T.: U-net: Convolutional networks for biomedical image segmentation. In: MICCAI (2015)
  * [31] Siarohin, A., Lathuilière, S., Tulyakov, S., Ricci, E., Sebe, N.: Animating arbitrary objects via deep motion transfer. In: CVPR (2019)
  * [32] Siarohin, A., Lathuilière, S., Tulyakov, S., Ricci, E., Sebe, N.: First order motion model for image animation. In: NeurIPS (2019)
  * [33] Siarohin, A., Sangineto, E., Lathuilière, S., Sebe, N.: Deformable gans for pose-based human image generation. In: CVPR (2018)
  * [34] Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. In: ICLR (2015)
  * [35] Song, J., Zhang, H., Li, X., Gao, L., Wang, M., Hong, R.: Self-supervised video hashing with hierarchical binary auto-encoder. TIP (2018)
  * [36] Sra, S., Dhillon, I.S.: Generalized nonnegative matrix approximations with bregman divergences. In: NIPS (2006)
  * [37] Srivastava, N., Mansimov, E., Salakhudinov, R.: Unsupervised learning of video representations using lstms. In: ICML (2015)
  * [38] Tang, H., Xu, D., Wang, W., Yan, Y., Sebe, N.: Dual generator generative adversarial networks for multi-domain image-to-image translation. In: ACCV (2018)
  * [39] Thewlis, J., Bilen, H., Vedaldi, A.: Unsupervised learning of object landmarks by factorized spatial embeddings. In: ICCV (2017)
  * [40] Wang, Z., Simoncelli, E.P., Bovik, A.C.: Multiscale structural similarity for image quality assessment. In: ACSSC (2003)
  * [41] Wu, Y., Kirillov, A., Massa, F., Lo, W.Y., Girshick, R.: Detectron2. https://github.com/facebookresearch/detectron2 (2019)
  * [42] Xu, Z., Liu, Z., Sun, C., Murphy, K., Freeman, W.T., Tenenbaum, J.B., Wu, J.: Unsupervised discovery of parts, structure, and dynamics (2019)
  * [43] Yu, R., Wang, X., Xie, X.: Vtnfp: An image-based virtual try-on network with body and clothing feature preservation. In: CVPR (2019)
  * [44] Zablotskaia, P., Siarohin, A., Zhao, B., Sigal, L.: Dwnet: Dense warp-based network for pose-guided human video generation. In: BMVC (2019)
  * [45] Zanfir, M., Oneata, E., Popa, A.I., Zanfir, A., Sminchisescu, C.: Human synthesis and scene compositing. arXiv preprint arXiv:1909.10307 (2019)
  * [46] Zhang, R., Isola, P., Efros, A.A.: Split-brain autoencoders: Unsupervised learning by cross-channel prediction. In: CVPR (2017)
  * [47] Zhang, Y., Guo, Y., Jin, Y., Luo, Y., He, Z., Lee, H.: Unsupervised discovery of object landmarks as structural representations. In: CVPR (2018)
  * [48] Zhang, Y., Guo, Y., Jin, Y., Luo, Y., He, Z., Lee, H.: Unsupervised discovery of object landmarks as structural representations. In: CVPR (2018)
  * [49] Zhu, W., Liang, S., Wei, Y., Sun, J.: Saliency optimization from robust background detection. In: CVPR (2014)

##  A Implementation details

As stated in the main documents, there are 2 main modules in our method:
Segmentation module and Reconstruction module.

For the Segmentation module, we employ the U-Net [30] architecture. Our
Segmentation Module operates on resolution
$H^{\prime}\\!\times\\!W^{\prime}\\!=\\!64\\!\times\\!64$ and is composed of
five $conv_{3\times 3}$ \- $bn$ \- $relu$ \- $avg-pool_{2\times 2}$ blocks and
five $upsample_{2\times 2}$ \- $conv_{3\times 3}$ \- $bn$ \- $relu$ blocks. As
explained in the main paper, Segmentation module estimate shift parameters
$p^{k}$ and affine parameters $\boldsymbol{Y}^{k}$, along with segmentation
map $\boldsymbol{Y}^{k}$. The shift parameters $p^{k}$ can be understood as
anchor keypoints associated to the segments $\boldsymbol{Y}^{k}$. Practically,
the anchor keypoints are estimated by the encoder network that outputs $K$
additional channels, one per keypoint. From each channel, the anchor keypoint
location is estimated via soft-argmax as in [31, 29, 32]. On the other hand,
$\boldsymbol{A}^{k}$ is computed by using four additional outputs. From these
4-channel tensors, we obtain the coefficients of the matrix
$\boldsymbol{A}^{k}$ by computing spatial weighted average using as weights
the confidence map of the corresponding keypoint as in [32]. In the end, the
encoder outputs a total of $6K\\!+\\!1$ channels: $K\\!+\\!1$ for the
segmentation, $K$ for the anchor keypoints and $4K$ for the
$\boldsymbol{A}^{k}$ matrices.

For the Reconstruction Module we use the architecture of Johnson et al. [18]
that contains two down-sampling blocks, six residual blocks and two up-
sampling blocks. At train time the reconstruction loss is computed at four
different resolutions $256\times 256$, $128\times 128$, $64\times 64$ and
$32\times 32$ as in [38, 32]. We train the system using Adam [19] optimizer
with learning rate 2$e$-4 and batch size 20 for approximately 10k iterations
on 2 TitanX GPUs. In all the experiments we use $K=10$.

###  A.1 State-of-the-art comparison

As explained in the main paper, we compare our method with two state-of-the-
art methods for co-part image segmentation: SCOPS [15] and DFF [7].

Concerning SCOPS [15], we use the implementation provided by the
authors444https://github.com/NVlabs/SCOPS. We use the hyper-parameters
provided by the authors, except the number of parts that we set to $K=10$. We
train SCOPS model on _Tai-Chi-HD_ [32] and _VoxCeleb_ [26] by treating all the
frames of all the videos as images. Importantly, SCOPS [15] relies on saliency
maps that are obtained using the unsupervised method described in [49].
However, the SCOPS authors’ implementation only provides pre-computed saliency
maps for CelebA [22]. To obtain the saliency for our datasets, we estimate the
saliency maps using the public
implementation555https://github.com/yhenon/pyimgsaliency of [49].

With regards to DFF [7], we use the official implementations provided by the
authors666https://github.com/edocollins/DFF. We also employed the default
parameters except for the number of parts that we set to $K=10$. One of the
significant limitations of the DFF [7] is the fact that matrix factorization
requires all the images at once without distinction between training and test.
This leads to two consequences. First, the dataset must be sub-sampled to fit
in the memory. Second, train and test images should be combined for matrix
factorization, thereby making it impossible to apply on a single test image.
For these reasons we use all $5000+300$ (see main paper Sec. 4) images for DFF
[7] matrix factorization.

In our preliminary experiments, we also implemented a video-based version of
DFF [7] that performs matrix factorization using all the frames from a single
video. The motivation for this experiment is to compare our method with an
approach that uses video information. Nevertheless, this comparison is not
completely fair since this video-based DFF estimates segmentation in the test
frames by combining information from several frames. We include this video-
based implementation of DFF for the comparison in the supplementary video.
Despite evaluation biased in favour of DFF, we can observe that our method
produces clearly better visual results than this video-based DFF. Our approach
obtains much more consistent segments with seamless boundaries.

Furthermore, video-based DFF must be run independently for each video to
prevent memory issues. This leads to segmentation that are not consistent
among videos, as we can see in the supplementary video. Consequently, this DFF
version could not be included in our quantitative evaluation that uses
landmark-base metrics. Indeed, to compute this metric, we train and evaluate a
linear landmark regression model on two different subsets without identity
overlap. In the case of the video-based DFF, this procedure should be applied
for every run of the method, ie. every video. For this reason, subsets without
overlapping identities cannot be used.

##  B Qualitative results: number of segments

In this section, we report qualitative results for the ablation study where we
evaluate the impact of the number of segments (see Fig. 7).

_1 segment_ |  |  |  |
---|---|---|---|---
_5 segments_ |  |  |  |
_10 segments_ |  |  |  |
_15 segments_ |  |  |  |

Figure 7: Visual comparison of our method with different number of segments on
the _VoxCeleb_ dataset. In the even columns, predicted segmentations are
depicted, while in the odd columns, the original images with overlayed
segmentation are shown.

##  C Qualitative comparison with the state-of-the-art

In this section, we extend the qualitative comparison with state of the art
presented in the main paper (see Fig. 4 of the main paper). In Figs. 8, 9, we
provide an additional comparison of the predicted co-part segmentation
obtained with DFF, SCOPS and our methods on both the _Tai-Chi-HD_ and the
_VoxCeleb_ datasets. These results are well in-line with the results reported
in the main paper. We observe that on both datasets, our method outputs
segments much more consistent across different images. Furthermore, the
boundaries between the segments are cleaner and the foreground is better
separated from the background. Similar observations can also be made from the
video attached to this supplementary material. In this video, we also observe
that our method outputs stable (over time) segmentations even given that these
segmentations are estimated independently for each frame.

Input | DFF [7] | SCOPS [15] | Ours
---|---|---|---
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |

Figure 8: Visual comparison of our method and the state-of-the-art baselines
on _Tai-Chi-HD_ dataset. First column is an input. Next columns depict DFF
[7], SCOPS [15] and our method. For every method segmentation mask and image
with overlayed segmentation is shown.

Input | DFF [7] | SCOPS [15] | Ours
---|---|---|---
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |

Figure 9: Visual comparison of our method and the state-of-the-art baselines
on _VoxCeleb_ dataset. First column is an input. Next columns depict DFF [7],
SCOPS [15] and our method. For every method segmentation mask and image with
overlayed segmentation is shown.

##  D Additional part-swap results

In this section, we provide additional results for part-swap application, that
was introduced in Sec. 4.3 of the main paper. In Figs. 10, 11 we demonstrate
an additional visual results using the model trained on _VoxCeleb_ dataset. In
Fig. 10 we swap the hair (top rows) and the top part of the face (bottom
rows), using $K=5$ segment model. On the other hand in Fig. 11, we show hair
swaps with $K=10$ segments model. Interestingly, we observe that for persons
with short hair the best results are archived when we use 4 segments from the
source image, while for persons with long hair the best result archived when
we use 5 segments. Finally in Fig. 12 we demonstrate results of our method on
_Tai-Chi-HD_ dataset. In top rows we change the upper part of the body, while
in bottom 2 we alter an appearance of the legs. Overall, we observe that
without fine-tuning or special training our model can perform a range of
different part-swap operations.

---

Figure 10: Visual results of video-editing for VoxCeleb. In the odd rows the
target frames from video sequences are depicted, alongside the masks of
interest (in the right bottom corners) intended to be swapped. In the even
rows the source image is shown, followed by the generated frames containing
the indicated parts swapped from the source. Results achieved with model for
K=5 are depicted. Please note that the source images are downloaded from
Google Images. Best viewed with digital zoom.

---

Figure 11: Visual results of video-editing for VoxCeleb. In the first and the
fifth rows the target frames from video sequences are depicted, alongside the
masks of interest (in the right bottom corners) intended to be swapped. In the
rest of the rows the source images are shown, followed by the generated frames
containing the indicated parts swapped from the source. Results achieved with
model for K=10 are depicted. Please note that the source images are downloaded
from Google Images. Best viewed with digital zoom.

---

Figure 12: Visual results of video-editing for Tai-Chi-HD. In the odd rows the
target frames from video sequences are depicted, alongside the masks of
interest (in the right bottom corners) intended to be swapped. In the even
rows the source image is shown, followed by the generated frames containing
the indicated parts swapped from the source. Results achieved with model for
K=10 are depicted. Best viewed with digital zoom.