# Pose Metrics: A New Paradigm for Character Motion Edition

Léon Victor  0000-0003-1694-7189 Univ Lyon, INSA Lyon, LIRIS, UMR CNRS
5205France ,  Alexandre Meyer  0000-0002-0249-1048 alexandre.meyer@univ-
lyon1.fr Univ Lyon, Université Claude Bernard Lyon 1, LIRIS, UMR CNRS
5205France and  Saïda Bouakaz  0000-0001-7091-7859 Univ Lyon, Université
Claude Bernard Lyon 1, LIRIS, UMR CNRS 5205France

###### Abstract.

In animation, style can be considered as a distinctive layer over the content
of a motion, allowing a character to achieve the same gesture in various ways.
Editing existing animation to modify the style while keeping the same content
is an interesting task, which can facilitate the re-use of animation data and
cut down on production time. Existing animation edition methods either work
directly on the motion data, providing precise but tedious tools, or
manipulate semantic style categories, taking control away from the user. As a
middle ground, we propose a new character motion edition paradigm allowing
higher-level manipulations without sacrificing controllability. We describe
the concept of pose metrics, objective value functions which can be used to
edit animation, leaving the style interpretation up to the user. We then
propose an edition pipeline to edit animation data using pose metrics.

character animation, animation editing, animation style, neural networks

††conference: ArXiv; ; ††price: ††ccs: Computing methodologies Motion
processing††ccs: Computing methodologies Machine learning

##  1\. Introduction

Virtual characters are present everywhere in computer assisted media;
populating virtual worlds, embodying players, telling stories. Producing
qualitative motion for them is crucial to achieve engaging animation and raise
interest in the viewer. Character animation software provide tools and
interfaces which help artists do so. Many of such tools have been developed
over the years, striving to find balance between precision, ease of use, and
controllability. (Lasseter, 1987; Catmull, 1978)

The human eye perceives a lot of information through motion: it is a complex
behavior through which character convey intent, emotions or personality. A
same gesture can be performed in an wide variety of ways, even by a same
character. Editing an existing animation to change the way it is performed,
but not the gesture itself, is then an interesting research topic. Many
techniques have been proposed to tackle the problem of animation style
edition, but most of them act from the highest level, and lack the fine-
grained control users look for.

This paper presents a new framework for character motion edition using pose
metrics as a middle-ground between raw bone manipulation and high-level
”style” labels. We introduce these intermediate-level descriptors, then
propose a neural-networks-based edition pipeline which allows animators to
edit a character’s motion by manipulating the metrics.

##  2\. Related work

Virtual characters are traditionally represented by a mesh on which various
textures and materials are applied. Its motion is driven by an underlying
skeleton on which the mesh is fitted (”skinned”). The role of an animator is
to design the character’s motion by editing the skeleton’s poses at specific
time stamps, called key-frames. Instead of key-framing a new animation from
the ground up this way, it is also possible to capture the motion of real-life
actors through dedicated motion capture (MoCap) setups. MoCap data is a good
source of realistic motion, but the resulting animation still often needs to
be cleaned up and edited to fit the animator’s vision.

Previous research on character animation investigates tools, algorithms and
interfaces to help make authoring and editing motion a simpler process.
Increased availability and quality of natural motion acquired by MoCap systems
has led to raising interest in data-oriented methods. Meaningful information
can be extracted from ground truth data and put to use to provide animators
with more efficient tooling.

###  2.1. Animation edition

Whether originally key-framed by an an animator or recorded in a MoCap
setting, animation data is set to change for any number of reasons, ranging
from tweaking details to fit an artist’s vision, to re-purposing it to be used
on a different character. This paper builds upon a large body of work whose
common goal is to provide animators with tools facilitating the edition
process. The common animation edition pipeline can be split in two phases:
posing and interpolation control. During the first, animators manipulate the
skeleton to define key poses at specified key frames. In the second, they
manipulate the interpolation process which is used to fill the gaps between
key-frames.

#### Posing.

In most posing techniques the skeleton is conceptualized as a set of kinematic
chains (Magnenat-Thalmann et al., 1988). With forward kinematics (FK), the
modification of a joint’s position or rotation trickles down, modifying joints
down the chain. The opposite technique, inverse kinematics (IK) (Wang and
Chen, 1991; Aristidou and Lasenby, 2011), find a suitable parameterization of
the chain given a target end-effector, allowing the user to pose the character
much like a puppet. These two methods are at the core of most modern animation
software and lead to precise results, but can be tedious to use from an
artist’s perspective. Other techniques and interfaces have then been studied
to alleviate this problem.

Some describe alternative frameworks in which the character’s pose is derived
from a line drawn by the user, much like a paintbrush stroke (Guay et al.,
2013; Choi et al., 2016). This method has also been extended to virtual
reality, tracking the user’s hand in 3 dimensions and interpreting their
motion as strokes (Garcia et al., 2019). On the data-oriented side, Wu et al.
(Wu et al., 2011) use learned clustering techniques to extract from a motion
database a pose which most closely match the user’s input. Victor et al.
(Victor et al., 2021) train deep neural networks to solve the IK problem,
learning skeleton constraints from existing data to lift some of the burden of
respecting them from the user. Finally, some work has been dedicated to
designing physical devices to facilitate the posing process (Yoshizaki et al.,
2011; Jacobson et al., 2014).

#### Interpolations.

Once key poses have been set, different methods exist to control how the
software will interpolate between them to produce motion. Animators usually
provide the interpolation factor used for each joint’s position and rotation
values over the elapsed time between key frames by manipulating animation
curves through control points (Catmull and Rom, 1974). In the same veins,
optimization-based methods have been proposed to improve the curve edition
process (Koyama and Goto, 2018; Ciccone et al., 2019). In the family of data-
oriented solutions, Chai et al. (Chai and Hodgins, 2007) propose to learn a
low-dimensional space of motion, and solve the same kind of optimization
problems within it. Recently, Harvey et al. (Harvey et al., 2020) trained deep
neural networks to learn to produce inbetweening motion from key frames.

Creating high-quality, appealing motion is a complex task. Animating a
character according to fit one’s vision, while depicting the desired action
and conveying the right emotion requires an advanced knowledge of anatomy and
physics rules. All the previously mentioned work allow animators to edit
animation, but they all focus on controlling the character’s body at bone-
level. The process, while facilitated, is still a complex one. In order to
provide animators and the greater public access to fast an expressive
animation tools, higher-level concepts and tools are needed.

###  2.2. Motion style manipulation and transfer

Stemming from the need of methods to reuse existing animation, another body of
work has taken interest on editing animation by manipulating the motion style.
The notion of _style_ is most commonly used as opposed to the notion of
_content_. Given an animation sequence, the terms are used as a way to
separate _what_ the character is doing (the content) from _how_ it is doing it
(the style). For example, several animations sequence could represent a
character walking in various ways: fast, slow, making shorter or largers
strides, swinging its arms… In this case they all share the same _content_ ,
walking, but the various _styles_ set them apart.

Being able to generate stylized animation, and to modify the style of an
existing one, is then a desirable goal. Such control would allow the user to
easily experiment with different styles, and it would be possible to generate
a wide array of variations from an existing sequence.

Early solutions to the style edition problem use hand-made labels of style and
content to categorize existing motion, and interpolate between categories to
produce new motion. Rose et al. (Rose et al., 1998) use such a method, coupled
with an interface using a notion of verbs and adverbs to parameterize the
generated motion. Brand et al. (Brand and Hertzmann, 2000) describe a learned
state machine, in which motion sequences are organized by shared style
characteristics. More recently, Rodriguez et al. (Rodriguez et al., 2019)
stray away from the usual style paradigm and propose to edit animation trough
high level trajectories parameterization.

Some other methods leverage the work produced in related fields, such as dance
theory, to categorize motion style. Durupinar et al. (Durupinar et al., 2016)
build upon Laban Movement Analysis (von Laban and Ulmann, 1971) to attempt to
formulate a link between personality and body motion. Aristidou et al.
(Aristidou et al., 2017) map the Laban features to a system of emotions
coordinates, which can be interpolated in to edit and existing animation
style.

Drawing inspiration from the success achieved by style transfer methods on
images, Holden et al. (Holden et al., 2016) train a deep convolutional neural
network to extract ”style” features from of an animation and apply them to
another. In the same vein, other types of neural networks have been used, such
as variational autoencoders (Du et al., 2019), adversarial learning (Wang et
al., 2020) or generative flow (Wen et al., 2021).

On the opposite end of the spectrum from the posing methods discussed above,
these style edition approaches share the common drawback of being _too_ high-
level. Because the notion of animation style is still loosely defined (and
this shortcoming is discussed further in the next section), tools using styles
as control parameters could be met with incomprehension on the user’s end when
reaching broader audiences.

##  3\. Problem formulation

The previously defined usage of _style_ as a separation from an animation’s
_content_ is commonly accepted in the computer animation literature. The
limits of what the term itself entails are however not clearly defined.

Some associate an animation’s style to emotions experienced by the character,
such as ”joy”, ”anger” or ”sadness” (Fourati and Pelachaud, 2014). Some link
it to existing motion analysis systems, such as Laban’s (von Laban and Ulmann,
1971; Durupinar et al., 2016), or to psychological concepts such as the
arousal/valence scale (Ekman and Friesen, 1967; Randhavane et al., 2019). Some
others use arbitrary categories, such as ”gorilla”, ”childish”, ”zombie”, or
”neutral” (Xia et al., 2015).

Figure 1. Examples of walk animations in exaggerated styles from existing
motion databases (Holden et al., 2015; Xia et al., 2015). (1) ”zombie”; (2)
”old man”; (3) ”childish”. Distinguishing each style with no context is not
straight-forward.

We argue that this categorization is too subjective to be considered as an
expressive parameter to provide to users. Various factors can introduce bias
in style categories, leading edition tools based on them to fail to meet the
user’s expectation.

The bias lies in how the data used to calibrate the style categories is
recorded. It is usually not captured in the wild, but rather played by actors
(Xia et al., 2015). The source sequence are thus often overplayed, exaggerated
representations of the actor’s and director’s vision of the categories (see
Fig. 1). In the most grounded acquisition procedures, emotions states are
elicited in a subject by exposing him to a pre-determined scenario. The bias
is however not totally avoided, possibly induced by the design of the
experiment (McCambridge et al., 2014; Henrich et al., 2010), or even by the
divergent interpretations of a common emotion in different cultures (Elfenbein
and Ambady, 2002).

The main goal of animation edition research is to provide tools to animators
so that they can transform their mental representation of a character motion
into tangible animation data. If low-level methods are fine-grained but
tedious to manipulate, and if high-level ones are likely biased and take too
much of the control out of the user’s hands, finding a middle ground is an
important open problem. Efficient tooling, allowing easy manipulation for the
naïve user and controllable enough for the professional one, are yet to be
developed.

The parameterized edition paradigm, as seen in (Rodriguez et al., 2019), offer
interesting perspective in working around the bias problem while still
allowing higher-level control. Following this line of thought, this paper
proposes an alternative solution, introducing a method to manipulate style
from higher-level, but still objective, controls: pose metrics.

##  4\. Method

We propose a new character animation edition paradigm in which the animator
manipulates intermediate-level parameters called pose metrics, leveraging
neural networks to learn and automatically account for complex morphological
constraints. We first introduce this new concept, then describe the neural
networks pipeline we use in our experiments. We finally describe the method we
use to edit whole animation sequences according to a user’s input.

In the subsequent sections the term _skeleton_ is used to define the virtual
armature of a character composed of a hierarchy of joints and bones (Magnenat-
Thalmann et al., 1988). We call _pose_ the data vector describing the
concatenated position and/or orientation of each joint in the skeleton at a
single time frame. An _animation sequence_ is then a sequence of key poses,
which represent the character’s motion.

In equations the notation is as follow: we denote $A$ an animation sequence
containing $N$ frames, with $p=A_{n}$ the pose at frame $n$. In a skeleton
composed of $J$ joints, the pose can be indexed to access a specific joint’s
data: $p_{head}$.

###  4.1. Pose metrics

Being able to edit an animation sequence by manipulating higher-level
parameters is a desirable capacity, but the literature is polarized between
low level controls (operating on crude pose data such as bone position or
orientation using forward and inverse kinematics) and high level ones, using
_style_ labels relating to emotions or behavior (i.e. ”angry”, ”childish”,
etc. ) (Wang et al., 2020; Du et al., 2019; Aberman et al., 2020b). These
style controllers lack control in the sense that they produce a stylized
animation at once based on a generalized style definition, which users might
not share.

As a compromise, filling the gap between these poles, we introduce _pose
metrics_ : higher-level, objective, style agnostic values, that can be
directly computed from a single pose and do not rely on any assumed universal
human perceptive behavior.

The vision for these metrics is to be as flexible as possible, so as not to
restrict the user. All characters are different, and the concept must make as
little assumption as possible on the way they move. That way the user is
responsible for designing his own metrics, mentally link them to his own
definition of a particular style, and edit the animation through them.

For the sake of an example, imagine an artist editing an animation sequence of
a character walking, expressing cold rage. In his mind, the character should
be taking short, fast steps, his arms tense, stretched toward the floor. With
previous style edition methods, he might be able to hit an ”angry” button,
maybe use a slider to determine how angry the result should be. Unfortunately,
it might be that the system was calibrated with another definition of angry in
mind: the character now walks faster, the arms bent and swinging, with the
fists close to the torso. Using metrics would allow the user to circumvent
this problem. Creating a few metrics such as the angle formed by the elbows,
distance between feet, or between hands and the floor, and editing the
animation through them, we avoid relying on assumed style characteristics.

The only requirement for a new metric is to be computable from a single pose,
and to provide a function to do so. Given a function calculating such a user-
provided metric, the proposed associated framework is extendable enough to
accommodate it and edit animation sequences accordingly.

Figure 2. Visual depiction of the three metrics used in this paper’s examples.

#### Experimental pose metrics.

As previously explained pose metrics are intended to be provided by the user.
In order to demonstrate the application of our system, the following
experiments use three sample metrics applicable to a human skeleton:
spine_flexion, shoulders_openness and legs_spread.

All three represent the angle formed at the intersection of two vectors
$\vec{u}$ and $\vec{v}$ obtained from the pose. Formally the angle at the
intersection is computed following Eq. 1.

(1)
$\theta(\vec{u},\vec{v})=\arccos\frac{\vec{u}\cdot\vec{v}}{\lVert\vec{u}\rVert\cdot\lVert\vec{v}\rVert}$

The spine_flexion metric represents the angle between $\vec{y}$ the world ”up”
vector and the spine axis. It is computed as shown in eq. 2. It could be used
to bend the character forward when in a hurry, or backward when relaxed.

(2) $SF(p)=\theta(p_{neck}-p_{pelvis},\vec{y})$

Next the metric shoulders_openness (Eq. 3) describes how open the character’s
torso/shoulders area is. Example usages might be to raise the character
shoulders to denote anxiety or shyness, and to open up to denote a more
relaxed or proud behavior.

(3) $SO(p)=\theta(p_{spine1}-p_{rshoulder},p_{lshoulder}-p_{spine1})$

Finally, legs_spread (Eq.4) denotes how close the character’s knees are to
each other. Possible usages include restricting walking strides, or tweaking
whether the character appears in balance.

(4) $LS(p)=\theta(p_{pelvis}-p_{rknee},p_{lknee}-p_{pelvis})$

In this paper’s equations, the value $m$ of a metric $M$ for a pose $p$ is
denoted as the function $m=M(p)$.

###  4.2. Learned metric pipeline

This section introduces a pipeline through which an animation sequence can be
edited using pose metrics. It is designed with modularity in mind, as we want
users to be able to decide on which metric should be used at any time, as well
as extending it with custom metrics if need be. We first describe the global
layout of the pipeline, then each of its pieces separately. We also explain
how the pipeline can be extended by the user, and set up to satisfy any
combination of metrics. Finally, we propose a procedure to edit whole
animation clips, by taking advantage of our architecture to propagate changes
made to a selected frame to the rest of the sequence.

#### Pipeline layout.

Our pipeline (Fig. 3) relies on several small neural networks. We opt for
these networks because of their low memory footprint and fast run time. Using
smaller independent networks also allows us to easily configure the pipeline
by adding, removing or modifying modules.

The outer part of the pipeline is an encoder-decoder model pair whose purpose
is to learn a latent pose space, in which a pose representation is more
compact and easier to manipulate. As the decoder can only map latent codes to
the real pose space, there is no risk of accidentally generating a broken pose
when averaging and interpolating samples. The edition itself is done by the
metrics network, which operates only on the latent space. An instance of the
metric network is trained to modify a latent pose given a _single_ metric
value. The reduced size of the network and the fact that we re-use the already
trained encoder-decoder, mean that training times are fairly short. This in
turn further facilitates the possibility for users to design their own
metrics, train a network, and incorporate it in the pipeline.

Figure 3. Overview of the proposed pipeline with its pose metric module. The
process modifies an input pose $p$ according to an arbitrary number of user-
provided target metric values $\tilde{m}_{0...i}$. An encoder-decoder pair
($E$ and $D$) maps poses to and from a latent pose space, on which the other
modules operate. Each metric is processed by a dedicated network $M_{0..i}$.
Their resulting latent samples $\hat{z}_{0..i}$ are averaged, then mapped back
to the pose space.

#### Encoder-Decoder.

The encoder $E$ and decoder $D$ networks are trained together to map a pose
$p$ to a latent representation $z$, as shown in Eq. 5. In practice they are
trained to minimize the squared reconstruction error between $p$ and its
reconstructed version $\hat{p}$. Both networks are simple 1-layer perceptrons,
using 512 units in the hidden layer and rectified linear unit (ReLU)
activation functions. In our experiment, the latent size dimension is set to
64.

(5) $\begin{split}z&=E(p)\\\ \hat{p}&=D(z)\\\
\mathcal{L_{\textrm{ED}}}&=\frac{1}{J}\sum_{j=0}^{J}{\big{(}p_{j}-\hat{p}_{j}\big{)}^{2}}\end{split}$

#### Metric network(s)

. A metric network $N_{m}$ learns to produce a modified latent pose
$\hat{z}_{m}$, taking as input a starting one $z$ concatenated with a target
metric value $\tilde{m}$ (see Eq. 6).

(6) $\hat{z}_{m}=N_{m}\big{(}z\oplus\tilde{m}\big{)}$

To reflect this objective during training, we sample two poses from the same
original animation sequence: an input $p=A_{t}$ and a target
$\tilde{p}=A_{t+n}$. Sampling from a neighboring frame prevents the target
pose from being too far from the starting one and ensures the network learns
to _edit_ a pose rather than to generate an unrelated one. In our experiments,
$n$ is sampled randomly for each iteration in $n\in[-10,10]$. As we target an
interactive edition context, aiming for smaller changes in metric is
reasonable. The input target metric is computed from $\tilde{p}$ and fed to
the network along with $z=E(p)$. The resulting modified latent pose $\hat{z}$
is mapped back to the pose space using the decoder, i.e. $\hat{p}=D(\hat{z})$.
Finally the networks are trained to minimize the mean squared error between
the target pose and the modified one, as shown in the loss function in Eq. 7.

(7)
$\mathcal{L}=\frac{1}{J}\sum_{j=0}^{J}{\big{(}\tilde{p_{j}}-\hat{p}_{j}\big{)}^{2}}$

Keeping with the idea of smaller networks we opt for a single layer perceptron
with 126 hidden units and ReLU activation. The input layer’s number of units
is equal to the size of the auto-encoder’s latent dimension plus one (to
account for the metric value). The output is the same size as the latent
dimension. We empirically find that this architecture suits our needs,
larger/deeper networks yielding no perceptible value. If a users deemed a
different one more suitable for one of his metrics, the only requirements
would be on the size of the input and output layers.

#### Data

. We use a pose dataset of about 1.5 million poses as described by Victor et
al. (Victor et al., 2021). The dataset is obtained by merging several pre-
existing animation databases(Fourati and Pelachaud, 2014; University, 2003;
Holden et al., 2016). Each animation clip is retargeted to a common skeleton
(Holden et al., 2015). The poses data is composed of the positions of each
joint relative to the projection of the pelvis joint on the floor. When used
to train the networks, the dataset is normalized by subtracting the mean and
dividing by the standard deviation of each component value.

#### Training parameters

. The encoder-decoder and metrics networks are trained separately, but using
the same hyper-parameters: using the Adam optimizer (Kingma and Ba, 2015) with
a learning rate of $0.0001$, and batches of 1024 pose pairs until convergence,
which takes around 15 minutes on an NVIDIA Quadro T1000.

#### Pipeline modularity

In cases where it is desirable to edit multiple metrics at once, it is
possible to combine the results of a set of metric networks. As the output of
the network is a latent representation of a pose we can combine them. In
(Victor et al., 2021), the authors ran multiple networks sequentially. We
instead suggest averaging multiple latent poses, so that we don’t have to pick
an order for the modules.

As explicited in Eq. 8, the source pose $p$ is first mapped to the latent
space, then each specialized metric network $M{0..i}$ produces modified latent
result. These intermediary results are all averaged, and finally mapped back
to pose space by the decoder.

(8)
$\hat{p}=D\Big{(}\frac{1}{n}\sum_{i=0}^{n}{M_{i}\big{(}E(p)\big{)}}\Big{)}$

This method operating on single key-frames, can be used conjointly with other
posing solutions, and integrated with interpolation methods, to offer as much
control to the user as possible. Using a shared latent space also allow us to
incorporate other modules from the literature into the pipeline. For example,
the IK solver from (Victor et al., 2021) operates on a similar latent space,
and, if trained on the pipeline’s one, its output can be averaged with the
metric network ones.

###  4.3. Animation edition

The pose pipeline described above only operates on a single pose at once. In a
real-world scenario however animators typically edit a specific key frame,
expecting their changes to be gradually applied to the neighboring ones. To
meet these expectations we propose a method to edit an entire animation
sequence through a single set of target metrics.

Our approach (Fig. 4) resembles the traditional animation curve paradigm,
using a user provided weight curve $W$ to control how much each key pose will
be impacted by a modification. We once again leverage the good interpolation
properties of the latent space.

Figure 4. Visual overview of the proposed method to echo modifications made to
a key pose $A_{n}$ of an animation clip. The blend operation occurs between
latent poses (source $z_{n}$ and modified by the metric module
$\overline{z}_{n}$), and use a factor $W(n)$ sampled from the user-provided
curve.

To edit an animation $A$, the user selects a key frame $A_{n}$ and provides an
arbitrary number of target metrics values $\tilde{m}_{0..i}$ as well as a
weight curve $W$. For each frame $A_{t}$ in the sequence, a latent
representation $z_{t}$ is encoded. Each of them is then processed by the pose
metric module, resulting in an intermediary latent pose $\overline{z}_{t}$.
The final modified latent pose $\hat{z}_{t}$ is given by blending the between
$z_{t}$ and $\overline{z}_{t}$ by the $W(t)$ factor (Eq. 9).

(9) $\hat{z}_{t}=\big{(}1-W(t)\big{)}\cdot z_{t}+W(t)\cdot\overline{z}_{t}$

In our experiments $W$ is a hat function peaking at $1$ at the selected frame,
and reaching $0$ at the clips extremities. This weight function is however an
external parameter: experiments with a sinusoidal yield comparable results,
and the curve could be edited by animators to fit their expectations, much
like existing animation curves. We also note that each one of the clip’s poses
is processed independently from the others. The edition can thus happen in
parallel, maintaining real time interactivity.

##  5\. Results

This section presents visual animation edition results of the pipeline in
different scenarios. More examples can be found in the accompanying video.

###  5.1. Pose edition

Figure 5. Incremental modification of a pose by providing a target
$spine\\_flexion$ input. On each step, the user’s input is a $0.1$ increase
over the computed metric value.

In Fig. 5, a single pose is modified by a user by providing gradually
increasing target spine_flexion metric values. The results respect the desired
changes as shown by the measured metric values below each pose. While it could
be argued that changing the orientation angle of the pelvis bone would produce
similar results, a closer look to the resulting poses illustrates the
interesting properties of the latent pose space. At every iteration, the
skeleton’s arms are slightly raised, and its knees bent, so as to compensate
the loss of balance induced by the flexion of the spine.

###  5.2. Modular pipeline

Figure 6. Sample results of the pipeline using multiple modules. In (1),
shoulders_openness (so) and spine_flexion are used simultaneously. In (2) we
incorporate the latent inverse kinematic solver from (Victor et al., 2021)
into the pipeline, using it along spine_flexion.

Fig. 6 presents some results obtained by averaging the latent outputs from
multiple modules. The first example show the result of modifying the
spine_flexion and shoulders_openness metrics at the same time. Both
modifications can be seen on the skeleton: its spine is straightened, and its
shoulders lowered. The rest of the pose is also slightly edited to accommodate
for those change: the hands are lowered, and knees bent a little further.

The second one shows the result of using the spine_flexion metric module along
with the neural IK method from (Victor et al., 2021). Averaging the latent
outputs leads our method to find a compromise between each provided
constraint. Here, the skeleton’s back is straightened, and the IK targets for
the hands are reached for.

###  5.3. Animation edition

Figure 7. Editing an animation clip according to the user input. The output is
produced by the pipeline given a selected frame $A_{n}$, a target value for
the legs_spread metric, and a weight curve.

Fig. 7 showcases the result of an animation $A$ being edited using the
pipeline described earlier. The user selects a frame $A_{n}$, provides a
target leg_spread metric value, and and interpolation factor curve, a hat
function peaking at one on the $A_{n}$ and reaching down to 0 on both sides at
$A_{n-3}$ and $A_{n+3}$. The remaining frames use a factor of 0.

Each frame pose is processed separately by the network, and the result is
given by interpolating the latent representations of the input and output
poses, using the frame-specific factor.

As a result, the output animation $\hat{A}$ is modified, with the impact being
most important on the selected frame and gradually lowering on each side. At
frame $n$ the output pose is modified the most, and the reduced legs angle is
very visible. At frames $n-2$ and $n+2$ this impact is barely visible, while
further frames are not modified at all, as expected.

Interpolating between original and modified poses allow the whole sequence to
keep its temporal coherence even though the pipeline processes one frame at a
time, with no knowledge of the past and future frame context. We also take
advantage of the properties of latent spaces: any latent interpolation result
is mapped back to the real pose space, thus avoiding artifacts such as
skeleton interpenetration.

##  6\. Conclusion

In order to avoid the pitfalls of categorizing animation styles, we introduce
the concept of pose metrics, values computed on a single pose that user can
define themselves to fit their _own_ perception of styles. We propose a
pipeline to edit an animation by manipulating these metrics, which uses a
modular set of tiny neural networks, running in real time. A user can extend
the pipeline by designing a custom metric and training an associated metric
network in a short 15 minutes.

The main shortcoming of this approach is that it is limited to a single
standardized humanoid skeleton, with which the networks are trained. This is a
common problem with data-driven methods: in order to adapt to a new skeleton,
the network would need to be retrained. This could be difficult if access to
existing animation data for the new skeleton is limited, but recent progress
in animation retargeting (Aberman et al., 2020a) could help by allowing more
accessible data augmentation. The modularity of metrics helps here as well, as
different skeleton will require a new set of metrics.

Future work will focus on grounding the proposed pipeline: securing the
process by automatically detecting overlapping metrics, and conducting user
studies to evaluate the usability of the method.

## References

  * (1)
  * Aberman et al. (2020a) Kfir Aberman, Peizhuo Li, Dani Lischinski, Olga Sorkine-Hornung, Daniel Cohen-Or, and Baoquan Chen. 2020a.  Skeleton-Aware Networks for Deep Motion Retargeting.  _ACM Transactions on Graphics_ 39, 4 (July 2020), 62:62:1–62:62:14.  https://doi.org/10.1145/3386569.3392462
  * Aberman et al. (2020b) Kfir Aberman, Yijia Weng, Dani Lischinski, Daniel Cohen-Or, and Baoquan Chen. 2020b.  Unpaired Motion Style Transfer from Video to Animation.  _ACM Transactions on Graphics_ 39, 4 (July 2020), 64:64:1–64:64:12.  https://doi.org/10.1145/3386569.3392469
  * Aristidou and Lasenby (2011) Andreas Aristidou and Joan Lasenby. 2011.  FABRIK: A Fast, Iterative Solver for the Inverse Kinematics Problem.  _Graphical Models_ 73, 5 (Sept. 2011), 243–260.  https://doi.org/10.1016/j.gmod.2011.05.003
  * Aristidou et al. (2017) Andreas Aristidou, Qiong Zeng, Efstathios Stavrakis, KangKang Yin, Daniel Cohen-Or, Yiorgos Chrysanthou, and Baoquan Chen. 2017\.  Emotion Control of Unstructured Dance Movements. In _Proceedings of the ACM SIGGRAPH / Eurographics Symposium on Computer Animation_ _(SCA ’17)_. Association for Computing Machinery, New York, NY, USA, 1–10.  https://doi.org/10.1145/3099564.3099566
  * Brand and Hertzmann (2000) Matthew Brand and Aaron Hertzmann. 2000.  Style Machines. In _Proceedings of the 27th Annual Conference on Computer Graphics and Interactive Techniques_ _(SIGGRAPH ’00)_. ACM Press/Addison-Wesley Publishing Co., New York, NY, USA, 183–192.  https://doi.org/10.1145/344779.344865
  * Catmull (1978) Edwin Catmull. 1978\.  The Problems of Computer-assisted Animation. In _Proceedings of the 5th Annual Conference on Computer Graphics and Interactive Techniques_ _(SIGGRAPH ’78)_. ACM, New York, NY, USA, 348–353.  https://doi.org/10.1145/800248.807414
  * Catmull and Rom (1974) Edwin Catmull and Raphael Rom. 1974.  A Class of Local Interpolating Splines.  In _Computer Aided Geometric Design_ , ROBERT E. Barnhill and RICHARD F. Riesenfeld (Eds.). Academic Press, 317–326.  https://doi.org/10.1016/B978-0-12-079050-0.50020-5
  * Chai and Hodgins (2007) Jinxiang Chai and Jessica K. Hodgins. 2007.  Constraint-Based Motion Optimization Using a Statistical Dynamic Model. In _ACM SIGGRAPH 2007 Papers_ _(SIGGRAPH ’07)_. Association for Computing Machinery, New York, NY, USA, 8–es.  https://doi.org/10.1145/1275808.1276387
  * Choi et al. (2016) Byungkuk Choi, Roger Blanco i Ribera, J. P. Lewis, Yeongho Seol, Seokpyo Hong, Haegwang Eom, Sunjin Jung, and Junyong Noh. 2016\.  SketchiMo: Sketch-based Motion Editing for Articulated Characters.  _ACM Trans. Graph._ 35, 4 (July 2016), 146:1–146:12.  https://doi.org/10.1145/2897824.2925970
  * Ciccone et al. (2019) Loïc Ciccone, Cengiz Öztireli, and Robert W. Sumner. 2019\.  Tangent-Space Optimization for Interactive Animation Control.  _ACM Transactions on Graphics_ 38, 4 (July 2019), 1–10.  https://doi.org/10.1145/3306346.3322938
  * Du et al. (2019) Han Du, Erik Herrmann, Janis Sprenger, Klaus Fischer, and Philipp Slusallek. 2019.  Stylistic Locomotion Modeling and Synthesis Using Variational Generative Models. In _Motion, Interaction and Games_ _(MIG ’19)_. ACM, New York, NY, USA, 32:1–32:10.  https://doi.org/10.1145/3359566.3360083
  * Durupinar et al. (2016) Funda Durupinar, Mubbasir Kapadia, Susan Deutsch, Michael Neff, and Norman I. Badler. 2016.  PERFORM: Perceptual Approach for Adding OCEAN Personality to Human Motion Using Laban Movement Analysis.  _ACM Trans. Graph._ 36, 1 (Oct. 2016).  https://doi.org/10.1145/2983620
  * Ekman and Friesen (1967) P. Ekman and W. V. Friesen. 1967.  Head and Body Cues in the Judgment of Emotion: A Reformulation.  _Perceptual and motor skills_ 24, 3 (June 1967), 711–724.  https://doi.org/10.2466/pms.1967.24.3.711
  * Elfenbein and Ambady (2002) Hillary Anger Elfenbein and Nalini Ambady. 2002.  On the Universality and Cultural Specificity of Emotion Recognition: A Meta-Analysis.  _Psychological Bulletin_ 128, 2 (2002), 203–235.  https://doi.org/10.1037/0033-2909.128.2.203
  * Fourati and Pelachaud (2014) Nesrine Fourati and Catherine Pelachaud. 2014.  Emilya: Emotional Body Expression in Daily Actions Database. In _Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC-2014)_. European Languages Resources Association (ELRA), Reykjavik, Iceland, 3486–3493.
  * Garcia et al. (2019) Maxime Garcia, Remi Ronfard, and Marie-Paule Cani. 2019\.  Spatial Motion Doodles: Sketching Animation in VR Using Hand Gestures and Laban Motion Analysis. In _Motion, Interaction and Games_ _(MIG ’19)_. Association for Computing Machinery, New York, NY, USA, 1–10.  https://doi.org/10.1145/3359566.3360061
  * Guay et al. (2013) Martin Guay, Marie-Paule Cani, and Rémi Ronfard. 2013\.  The Line of Action: An Intuitive Interface for Expressive Character Posing.  _ACM Transactions on Graphics_ 32, 6 (Nov. 2013), 1–8.  https://doi.org/10.1145/2508363.2508397
  * Harvey et al. (2020) Félix G. Harvey, Mike Yurick, Derek Nowrouzezahrai, and Christopher Pal. 2020.  Robust Motion In-Betweening.  _ACM Transactions on Graphics_ 39, 4 (July 2020), 60:60:1–60:60:12.  https://doi.org/10.1145/3386569.3392480
  * Henrich et al. (2010) Joseph Henrich, Steven J. Heine, and Ara Norenzayan. 2010\.  The Weirdest People in the World?  _Behavioral and Brain Sciences_ 33, 2-3 (June 2010), 61–83.  https://doi.org/10.1017/S0140525X0999152X
  * Holden et al. (2016) Daniel Holden, Jun Saito, and Taku Komura. 2016.  A Deep Learning Framework for Character Motion Synthesis and Editing.  _ACM Transactions on Graphics_ 35, 4 (July 2016), 1–11.  https://doi.org/10.1145/2897824.2925975
  * Holden et al. (2015) Daniel Holden, Jun Saito, Taku Komura, and Thomas Joyce. 2015\.  Learning Motion Manifolds with Convolutional Autoencoders. In _SIGGRAPH ASIA 2015 Technical Briefs on - SA ’15_. ACM Press, Kobe, Japan, 1–4.  https://doi.org/10.1145/2820903.2820918
  * Jacobson et al. (2014) Alec Jacobson, Daniele Panozzo, Oliver Glauser, Cédric Pradalier, Otmar Hilliges, and Olga Sorkine-Hornung. 2014.  Tangible and Modular Input Device for Character Articulation.  _ACM Transactions on Graphics_ 33, 4 (July 2014), 82:1–82:12.  https://doi.org/10.1145/2601097.2601112
  * Kingma and Ba (2015) Diederik P. Kingma and Jimmy Ba. 2015.  Adam: A Method for Stochastic Optimization. In _arXiv:1412.6980 [Cs]_. San Diego, California, US.  https://doi.org/10.48550/ARXIV.1412.6980 arXiv:1412.6980 [cs]
  * Koyama and Goto (2018) Yuki Koyama and Masataka Goto. 2018.  OptiMo: Optimization-Guided Motion Editing for Keyframe Character Animation. In _Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems - CHI ’18_. ACM Press, Montreal QC, Canada, 1–12.  https://doi.org/10.1145/3173574.3173735
  * Lasseter (1987) John Lasseter. 1987\.  Principles of Traditional Animation Applied to 3D Computer Animation.  _ACM SIGGRAPH Computer Graphics_ 21, 4 (Aug. 1987), 35–44.  https://doi.org/10.1145/37402.37407
  * Magnenat-Thalmann et al. (1988) Nadia Magnenat-Thalmann, Richard Laperrire, and Daniel Thalmann. 1988.  Joint-Dependent Local Deformations for Hand Animation and Object Grasping. In _Proceedings on Graphics Interface ’88_ _(GI ’88)_. Canadian Man-Computer Communications Society, Edmonton, Alberta, Canada, 26–33.  https://doi.org/10.20380/gi1988.04
  * McCambridge et al. (2014) Jim McCambridge, John Witton, and Diana R. Elbourne. 2014\.  Systematic Review of the Hawthorne Effect: New Concepts Are Needed to Study Research Participation Effects.  _Journal of Clinical Epidemiology_ 67, 3 (March 2014), 267–277.  https://doi.org/10.1016/j.jclinepi.2013.08.015
  * Randhavane et al. (2019) Tanmay Randhavane, Uttaran Bhattacharya, Kyra Kapsaskis, Kurt Gray, Aniket Bera, and Dinesh Manocha. 2019\.  Identifying Emotions from Walking Using Affective and Deep Features.  https://doi.org/10.48550/arXiv.1906.11884
  * Rodriguez et al. (2019) Alba M. Rios Rodriguez, Steven Poulakos, Maurizio Nitti, Mattia Ryffel, and Robert Sumner. 2019\.  Parameterized Animated Activities. In _Motion, Interaction and Games_ _(MIG ’19)_. ACM, New York, NY, USA, 11:1–11:9.  https://doi.org/10.1145/3359566.3360062
  * Rose et al. (1998) Charles Rose, Michael F. Cohen, and Bobby Bodenheimer. 1998\.  Verbs and Adverbs: Multidimensional Motion Interpolation.  _IEEE Computer Graphics and Applications_ 18, 5 (Sept. 1998), 32–40.  https://doi.org/10.1109/38.708559
  * University (2003) Carnegie Mellon University. 2003\.  CMU Graphics Lab Motion Capture Database.
  * Victor et al. (2021) Léon Victor, Alexandre Meyer, and Saïda Bouakaz. 2021\.  Learning-Based Pose Edition for Efficient and Interactive Design.  _Computer Animation and Virtual Worlds_ 32, 3-4 (2021), e2013.  https://doi.org/10.1002/cav.2013
  * von Laban and Ulmann (1971) Rudolf von Laban and Lisa Ulmann. 1971.  _The Mastery of Movement_.
  * Wang and Chen (1991) Li-Chun Tommy Wang and Chih Cheng Chen. Aug./1991.  A Combined Optimization Method for Solving the Inverse Kinematics Problems of Mechanical Manipulators.  _IEEE Transactions on Robotics and Automation_ 7, 4 (Aug./1991), 489–499.  https://doi.org/10.1109/70.86079
  * Wang et al. (2020) Qi Wang, Thierry Artières, Mickael Chen, and Ludovic Denoyer. 2020.  Adversarial Learning for Modeling Human Motion.  _The Visual Computer_ 36, 1 (Jan. 2020), 141–160.  https://doi.org/10.1007/s00371-018-1594-7
  * Wen et al. (2021) Yu-Hui Wen, Zhipeng Yang, Hongbo Fu, Lin Gao, Yanan Sun, and Yong-Jin Liu. 2021\.  Autoregressive Stylized Motion Synthesis With Generative Flow. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 13612–13621.  https://doi.org/10.1109/CVPR46437.2021.01340
  * Wu et al. (2011) Xiaomao Wu, Maxime Tournier, and Lionel Reveret. 2011\.  Natural Character Posing from a Large Motion Database.  _IEEE Computer Graphics and Applications_ 31, 3 (May 2011), 69–77.  https://doi.org/10.1109/MCG.2009.111
  * Xia et al. (2015) Shihong Xia, Congyi Wang, Jinxiang Chai, and Jessica Hodgins. 2015\.  Realtime Style Transfer for Unlabeled Heterogeneous Human Motion.  _ACM Transactions on Graphics_ 34, 4 (July 2015), 119:1–119:10.  https://doi.org/10.1145/2766999
  * Yoshizaki et al. (2011) Wataru Yoshizaki, Yuta Sugiura, Albert C. Chiou, Sunao Hashimoto, Masahiko Inami, Takeo Igarashi, Yoshiaki Akazawa, Katsuaki Kawachi, Satoshi Kagami, and Masaaki Mochimaru. 2011.  An Actuated Physical Puppet as an Input Device for Controlling a Digital Manikin. In _Proceedings of the SIGCHI Conference on Human Factors in Computing Systems_ _(CHI ’11)_. Association for Computing Machinery, New York, NY, USA, 637–646.  https://doi.org/10.1145/1978942.1979034


# Stereo Superpixel Segmentation Via Decoupled Dynamic Spatial-Embedding
Fusion Network

Hua Li∗, Junyan Liang$*$, Ruiqi Wu, Runmin Cong, Wenhui Wu, Sam Tak Wu Kwong,
Fellow, IEEE ∗These authors contributed equally. Ruiqi Wu proposed the paper
idea.The preliminary work of this paper was published in ICME 2021 [1], which
is accepted as oral presentation.Hua Li is with the School of Computer Science
and Technology, Hainan University, Hainan 570228, China (e-mail:
lihua@hainanu.edu.cn).Junyan Liang is with the School of Computer Science and
Technology, Hainan University, Hainan 570228, China (e-mail:
liangjunyan@hainanu.edu.cn).Ruiqi Wu is with the School of Computer Science
and Artificial Intelligence, Wuhan University of Technology, Wuhan 430070,
China (e-mail: wuruiqi0722@gmail.com).Runmin Cong is with the Institute of
Information Science, Beijing Jiaotong University, Beijing 100044, China, and
also with the Beijing Key Laboratory of Advanced Information Science and
Network Technology, Beijing Jiaotong University, Beijing 100044, China
(e-mail: rmcong@bjtu.edu.cn).Wenhui Wu is with the College of Electronics and
Information Engineering, Shenzhen University, Shenzhen 518060, China (e-mail:
wuwenhui@szu.edu.cn).Sam Tak Wu Kwong is with the Department of Computer
Science, City University of Hong Kong, Kowloon Hong Kong 999077, China, and
also with the City University of Hong Kong Shenzhen Research Institute,
Shenzhen 518057, China (e-mail: cssamk@cityu.edu.hk).

###### Abstract

Stereo superpixel segmentation aims at grouping the discretizing pixels into
perceptual regions through left and right views more collaboratively and
efficiently. Existing superpixel segmentation algorithms mostly utilize color
and spatial features as input, which may impose strong constraints on spatial
information while utilizing the disparity information in terms of stereo image
pairs. To alleviate this issue, we propose a stereo superpixel segmentation
method with a decoupling mechanism of spatial information in this work. To
decouple stereo disparity information and spatial information, the spatial
information is temporarily removed before fusing the features of stereo image
pairs, and a decoupled stereo fusion module (DSFM) is proposed to handle the
stereo features alignment as well as occlusion problems. Moreover, since the
spatial information is vital to superpixel segmentation, we further design a
dynamic spatiality embedding module (DSEM) to re-add spatial information, and
the weights of spatial information will be adaptively adjusted through the
dynamic fusion (DF) mechanism in DSEM for achieving a finer segmentation.
Comprehensive experimental results demonstrate that our method can achieve the
state-of-the-art performance on the KITTI2015 and Cityscapes datasets, and
also verify the efficiency when applied in salient object detection on NJU2K
dataset. The source code will be available publicly after paper is accepted.

###### Index Terms:

Stereo image, superpixel segmentation, stereo corresponding capturing,
spatiality embedding.

##  I Introduction

Superpixel segmentation aims at grouping the discretizing pixels into some
high-level correlative units as input primitives in a variety of subsequent
computer vision tasks, e.g., salient object detection [2, 3, 4, 5], image
dehazing [6], image classification [7], object recognition [8], adversarial
attack [9]. Nowadays, dual cameras have been widely used in extensive
industrial applications, such as assistant driving and mobile phones. Compared
with single images, stereo image pairs can obtain complementary information
from the second viewpoint, which is beneficial to scene representation and
object modeling [10]. However, how to effectively utilize complementary and
correspondence information to generate superpixels for stereo images is still
a challenging task.

Figure 1: A simple illustration of the comparison between the state-of-the-art
superpixel segmentation methods and our method. Figure 2: Overall framework of
the proposed method.

For stereo superpixel segmentation, stereo image pairs are generally segmented
separately as single views by traditional methods. By this way, the
complementarity and correlation between the left and right views of stereo
image pairs are ignored and cannot be explored sufficiently [11, 12].
Therefore, these methods cannot be regarded as an real implementation of
stereo superpixel segmentation, since the intrinsic characteristics of stereo
images are neglected. To take the collaborative relationship between left and
right views into consideration, Li et al. [10] propose a collaborative
optimization scheme to generate stereo superpixels with the parallax
consistency, which is the first attempt to devise a specific superpixel
segmentation method for stereo image pairs. The method first match the
corresponding regions between the left and right view of a stereo image pair.
Superpixels are initialized and matched in the corresponding regions. Then,
the superpixels in the left and right views are refined simultaneously via a
collaborative optimization strategy. Experimental results demonstrate it
outperforms the methods that segment stereo image pairs separately.
Nevertheless, this method extracts handcrafted feature instead of deep feature
by an unsupervised way, which leads to the limitation of the performance.

Most recently, Wu et al. [1] propose an end-to-end dual attention fusion
network (StereoDFN) for stereo superpixel segmentation, which extracts the
deep features of stereo image pairs by convolutional neural networks instead
of handcrafted features. Then it models the correspondence between the left
and right views via the parallax attention module to integrate the
complementary information of stereo image pairs and has achieved impressive
performance. However, the existing superpixel segmentation methods like
StereoDFN and superpixel sampling network (SSN) [13] utilize the five-
dimensional features (including two-dimensional spatial features XY, and
three-dimensional color features Lab) as input, which aims to extract high-
dimensional deep features. For stereo superpixel segmentation task, the
disparity information is often adopted to emphasize the correspondence between
left and right views. If the spatial information is input before the feature
fusion, this may result in strong constraints on spatial information, since
the stereo features are coupled with spatial features, which will lead to
degradation of image boundaries.

The presented work significantly extends StereoDFN with a decoupling mechanism
of spatial information to only use three-dimensional color features (Lab)
instead of five-dimensional features (XYLab) in modeling the correspondence
between both views, thereby decoupling the stereo features (color features)
and spatial features (XY) to relax the constrain of spatial information.
Considering the importance of spatial information in superpixel segmentation,
we further design a Dynamic Spatiality Embedding Module (DSEM) to re-add
spatial information, the weighting of spatial information will be adaptively
adjusted through the Dynamic Fusion (DF) mechanism in DSEM to fit images of
different sizes, thereby obtaining a more accurate representation of spatial
information and achieving a better performance. As the simple comparison shown
in Fig. 1, we can see that our proposed method adhere to the object boundaries
better than FCN [14] and StereoDFN.

In this paper, we improve upon our previous work in ICME [1]. The main
contributions of the proposed work can be summarized as follows:

  1. 1.

We propose a stereo superpixel segmentation method with a decoupling mechanism
of spatial information to generate superpixels for stereo image pairs, which
can integrate the correspondence from both left and right viewpoints to take
the stereo features alignment and occlusion problems into account.

  2. 2.

Since the coupling of stereo features and spatial features may impose strong
constraints on spatial information while modeling the correspondence between
stereo image pairs, we develop a spatial decoupling mechanism to model the
correspondence with relaxed spatial constraint by decoupling the stereo
features and spatial features, and postponing the embedding of spatial
information after stereo features have been fused.

  3. 3.

We design a Dynamic Spatiality Embedding Module (DSEM) to re-add spatial
information for achieving a finer segmentation. The weighting of spatial
information can be adaptively adjusted via the Dynamic Fusion (DF) mechanism
in DSEM to fit images of different sizes, thereby achieving a better
performance.

  4. 4.

Our method achieves the state-of-the-art performance compared with previous
works both quantitatively and qualitatively. Extensive ablation studies
validate the effectiveness of the proposed strategy. With application in
salient object detection, we also demonstrate that our method can achieve
superior performance in downstream task.

The article is organized as follows. We briefly introduce the related work
about existing superpixel segmentation algorithms in Section II. Then, we
propose our model and detail each key component in Section III. The
qualitative and quantitative experimental results and analyses are presented
in Section IV. In Section V, we present the application of the proposed method
in salient object detection. Finally, Section VII concludes this article.

##  II Related Work

The concept of “Superpixel” is first introduced in [15], which is an over-
segmentation of images and generated by grouping pixels similar in low-level
properties. Existing superpixel segmentation algorithms can be simply divided
into two categories: unsupervised superpixel segmentation methods and
supervised superpixel segmentation methods.

###  II-A Unsupervised Superpixel Segmentation Methods

Simple linear iterative clustering (SLIC) [16] is one of the most widely used
unsupervised methods, which employs k-means clustering approach to generate
superpixels efficiently by grouping nearby pixels based on five-dimensional
color and position features of the images. Due to SLIC has fast runtime and
impressive performance, many superpixel-based applications commonly use SLIC
for superpixel segmentation. Linear spectral clustering (LSC) [17] generates
superpixels based on kernel function instead of using the traditional eigen,
which is not only able to produce compact superpixel, but also with low
computational costs. Considering the irregular structure of superpixels, Li et
al. [18] propose approximately structural superpixels (ASS), they regard
superpixel segmentation as a square-wise asymmetric partition problem and
generate ASS by an asymmetrically square-wise superpixel segmentation way,
which can preserve semantics better and largely reduces data amount.

###  II-B Supervised Superpixel Segmentation Methods

In recent years, inspired by the success of deep learning techniques in a wide
variety of computer vision tasks, some works try to use deep learning
techniques for superpixel segmentation. Jampani et al. [13] propose the first
deep learning-based end-to-end trainable superpixel segmentation network
(SSN), which is enlightened by the SLIC method. To simplify the generation of
superpixels, Yang et al. [14] propose a lightweight fully convolutional
networks (FCN) that based on encoder-decoder structure, which generates
superpixels efficiently by predicting the probability map between pixels and
superpixels. More recently, Wu et al. propose an dual attention fusion network
(StereoDFN), they attempt to take the collaborative relationship between
stereo image pairs into consideration by modeling the correspondence between
them, which is based on parallax attention mechanism.

##  III Proposed Method

In this work, we propose a stereo superpixel segmentation method with a
decoupling mechanism of spatial information, the framework is illustrated in
Fig. 2. In general, the proposed method can be divided into the following
steps: First, stereo image pairs with Lab color space are input into fully
convolutional network to extract the deep features. Then, the deep features of
left and right views are fed to the Decoupled Stereo Fusion Module (DSFM),
which integrates the features from both views. Moreover, Dynamic Spatiality
Embedding Module (DSEM) is proposed to adaptively combine the spatial
information with deep features. Finally, a soft clustering algorithm [13] is
adopted to generate the superpixels.

In what follows, we detail the main components of the proposed method, which
are feature extractor, Decoupled Stereo Fusion Module (DSFM), Dynamic
Spatiality Embedding Module (DSEM) and loss functions respectively.

###  III-A Feature Extractor

A pair of weight-shared Convolutional Neural Networks (CNNs) is adopted to
extract the deep feature of stereo image. The basic block is a ‘Conv-BN-ReLU’
block, which is composed of a convolution layer with $3\times 3$ kernel size
and $64$ output channels, a batch-normalization layer and a ReLU activation
function. Each of the two modules will be followed by a max-pooling layer for
downsampling. For features captured from Block2, Block4 and Block6, we
upsample them into the same resolution as the input image and concatenate them
together. Block7 will fuse them and generate the final output. Through this
way, the networks can effectively learn more multi-level and multi-scale
features, which is benefit for both superpixel segmentation and capturing the
correspondence of stereo image pairs. The schematic illustration of feature
extractor has been shown in Fig. 3.

###  III-B Decoupled Stereo Fusion Module

Decoupled Stereo Fusion Module (DSFM) is the key component for fusing the
stereo features. Considering the most significant problems in stereo features
fusion, such as features alignment and occlusion problem, the proposed DSFM
try to solve them via parallax attention mechanism and valid mask.

Figure 3: The schematic illustration of Feature Extractor.

(a) Previous method. (b) Proposed method.

Figure 4: Differences between our method and StereoDFN. We only use Lab color
information as input to decouple the stereo features and spatial features.
After Decoupled Stereo Fusion Module (DSFM), we design a Dynamic Spatiality
Embedding Module (DSEM) to re-add spatial information.

Spatial Decoupling Mechanism. Considering the coupling of stereo features and
spatial features may impose strong constraints on spatial information while
modeling the correspondence between stereo image pairs, thereby interfering
with superpixels to adhere to the object boundaries. The spatial decoupling
mechanism is proposed to model the correspondence between stereo image pairs
with relaxed spatial constraint.More specifically, we remove the spatial
information of input items for relaxing the constrain of spatial information.
The input of StereoDFN is a five-dimensional features (XYLab), while the input
of our proposed method is a three-dimensional features (Lab), this is the
essential difference between our proposed method and StereoDFN, we also
present the schematic illustration of the difference in Fig. 4. Furthermore,
the effectiveness of our spatial decoupling mechanism has been shown in Fig.
5, benefiting from decoupling stereo features and spatial features, our method
eliminates the interference of spatial information on modeling, and the
boundary information is much more clearly.

Stereo Features Alignment. Since the corresponding pixels in stereo image
pairs are located at different positions, it is extremely difficult to fuse
the stereo features directly. Therefore, aligning the stereo features is
necessary before fusion.

(a) Original image

(b) Coupled

(c) Decoupled

Figure 5: The visualization of deep features after modeling the correspondence
between stereo image pairs. Note that (b) is generated by StereoDFN, while (c)
is generated by our method. Figure 6: The schematic illustration of the
difference between self attention and parallax attention mechanism. (a) and
(b) are self attention mechanism and our parallax attention mechanism
respectively. We can find the computational complexity of parallax-attention
mechanism is much smaller by comparing (a) and (b).

Inspired by [19], the parallax attention mechanism is utilized to model the
correspondence between stereo image pairs. Since the left and right views of
the images are only translated horizontally, while aligning in the vertical
direction, the matching pixels in two views must be on the same horizontal
line. For this reason, as the schematic illustration shown in Fig. 6(a) and
Fig. 6(b), the parallax attention mechanism is only consider the correlation
of pixels on the same horizontal line, instead of all the pixels in the image
like traditional self attention mechanism. In this way, the computation
complexity of attention can be largely reduced.

As the schematic illustration of aligning the features from the right view
with the left view features shown in Fig. 2, for a pair of deep features
$\mathcal{F}_{L}$ and $\mathcal{F}_{R}\in\mathcal{R}^{H\times W\times C}$, we
can get $A$ and $B$ from a convolution layer with $1\times 1$ kernel size.
Then, the parallax attention map $\mathcal{M}\in\mathcal{R}^{H\times W\times
W}$ will be generated by:

$\mathcal{M}_{R\to L}=softmax(A\otimes B^{T}),$ (1) $\mathcal{M}_{L\to
R}=softmax(B\otimes A^{T}),$ (2)

where $\otimes$ denotes the batch-wise multiplication, and $T$ denotes the
batch-wise transposition. $\mathcal{M}_{L\to R}(i,j,k)$ and $\mathcal{M}_{R\to
L}(i,j,k)$ represent the contribution of the position $(i,j)$ in one view to
position $(i,j)$ in another view. In this way, the supplementary information
of one view can be obtained through another view, which can be formulated as
follows:

$\hat{\mathcal{F}}_{L}=\mathcal{M}_{R\to L}\otimes\mathcal{F}_{right},$ (3)
$\hat{\mathcal{F}}_{R}=\mathcal{M}_{L\to R}\otimes\mathcal{F}_{left},$ (4)

where $\hat{\mathcal{F}}_{L}$ and $\hat{\mathcal{F}}_{R}$ denote the aligned
features.

Occlusion Problem. Occlusion always exists in stereo images due to violent
disparity variation, which will lead to an inaccurate stereo features fusion.
Therefore, we further add the occlusion handling part in DSFM to take
occlusion problem into consideration.

Taking the example of handling the occlusion in the right image. Assuming that
a pixel located at $(i,j)$ is in the occlusion region, for any $k\in[1,W]$,
$\mathcal{M}_{L\to R}(i,j,k)$ is an extremely low value since any pixel on the
same horizontal line in the left view is not relevant to it. Thus, the valid
mask $O_{L\to R}$ can be generated from parallax attention map
$\mathcal{M}_{L\to R}$, which can be formulated as Eq. (5):

$O_{L\to R}(i,j)=\begin{cases}1,&{\sum_{k\in[1,W]}\mathcal{M}_{L\to
R}(i,k,j)\textgreater\tau}\\\ 0,&{\sum_{k\in[1,W]}\mathcal{M}_{L\to
R}(i,k,j)\leq\tau},\end{cases}$ (5)

where $\tau$ is a threshold set to $0.1$. Then, the occlusion handling part
will fuse the stereo features. The left fused features
$\tilde{\mathcal{F}}_{L}$ can be obtained as Eq. (6):

$\tilde{\mathcal{F}}_{L}=Concat(\hat{\mathcal{F}}_{L}\circ O_{L\to
R}+\mathcal{F}_{L}\circ(1-O_{L\to R}),\mathcal{F}_{L}),$ (6)

where $Concat(,)$ represents the concatenate operation on channel dimension,
while $\circ$ represents the Hadamard product. Finally,
$\tilde{\mathcal{F}}_{L}$ will be fed to a ‘Conv-BN-ReLU’ block for reducing
the channel size to the size of origin features.

###  III-C Dynamic Spatiality Embedding Module

To prevent the spatial information influence the stereo correspondence
modeling, the spatial information has been removed in the input items.
However, spatial information is indispensable for superpixel segmentation
method to adhere to object boundaries more accurately, which is vital to
achieve a better performance. Therefore, spatial information is re-added via
the Dynamic Spatiality Embedding Module (DSEM) to take both conditions into
consideration simultaneously, so that we can not only eliminate the
disadvantage of spatial information in modeling the correspondence, but also
utilize the advantage of spatial information. DSEM consists of two parts,
which are Spatiality Embedding (SE) and Dynamic Fusion (DF). The architecture
of DSEM can be seen in Fig. 2.

Figure 7: (a) denotes the input images with $300\times 300$ resolution, (b)
the deep features, (c) the features after adding the spatial information with
a value domain of $(0,1)$, and (d) the features after adding the spatial
information zoomed in the manner of [16, 13]. We can see our embedding
strategy preserves more details than (d).

Spatiality Embedding. A reliable superpixel segmentation algorithms require
the ability to handle images with different resolutions. However, the value of
spatial information can be extremely large for a high-resolution image, which
will pollute the image feature representation if the spatial information is
embedded directly. In order to avoid such a disadvantage, we normalize the
spatial information $X$ and $Y$ as Eq. (7):

$\hat{X}=\frac{X}{max(X)},~{}\hat{Y}=\frac{Y}{max(Y)},$ (7)

where $X$ and $Y$ are spatial information on horizontal and vertical
direction, respectively.

After normalizing, $\hat{X}$ and $\hat{Y}$ is added to fused features and get
$\tilde{\mathcal{F}_{X}},\tilde{\mathcal{F}_{Y}}$. Then, a convolution layer
with $1\times 1$ kernel size is followed to embed the spatial information. In
this way, we can prevent an over-consideration of spatial information.
Finally, $\tilde{\mathcal{F}_{X}},~{}\tilde{\mathcal{F}_{Y}}$ is concatenated
with input features and send them to dynamic fusion part. Fig. 7 shows the
effectiveness of our embedding strategy.

Figure 8: The schematic illustration of Dynamic Fusion mechanism.

(a) Performance on the KITTI2015 dataset.

(b) Performance on the validation set of Cityscapes dataset.

Figure 9: Quantitative comparison of the proposed method and other state-of-
the-art methods.

Dynamic Fusion. Although spatial information is indispensable in superpixel
segmentation task, it does not always play the same important role of
different regions in one image. For example, for regions with sparse textures,
spatial information should be considered more to generate regular and compact
superpixels. On the other hand, for regions with dense edges and complex
contents, spatial information is relatively less important. Therefore, to
achieve consistently excellent performance in different conditions, a dynamic
fusion mechanism is designed to adaptively adjust the weighting of spatial
information during fusion phase.

The dynamic fusion mechanism employs a channel-attention [20] way to
adaptively aggregate and refine features. More specifically, we first use a
‘Conv-BN-ReLU’ block to fuse the features coarsely. Then, a global average
pooling layer with another ‘Conv-BN-ReLU’ is followed to generate the global
feature map. Finally, a series operations are utilized to produce the
weighting map, which can be formulated as follows:

$\mathcal{W}=g\cdot\sigma(C(ReLU(LN(C(g))))),$ (8)

where $\mathcal{W}$ is the weighting map and $g$ is the global feature map.
$\sigma,C,ReLU$ and $LN$ represents sigmoid function, a convolution layer with
$1\times 1$ kernel size, ReLU activation function and layer-normalization,
respectively. In this way, a more effective representation of spatial
information with the guidance of weighting map can be obtained. Finally, the
weighting map is added to the input features and fed to the third ‘Conv-BN-
ReLU’ block to generate the adjusted features. Fig. 8 presents the details of
the Dynamic Fusion (DF) mechanism.

###  III-D Loss Functions

We design two loss functions for optimizing our model.

Semantic Loss. This function facilitates the superpixel adhere to semantic
boundaries, which utilize the cross-entropy loss function $SE$ to measure the
loss:

$\mathcal{L}_{sem}=CE(S,S^{*}),$ (9)

where $S$ denotes the one-hot semantic label of ground truth and $S^{*}$ is
the reconstructed semantic label.

Stereo Loss. This loss function is designed to constrain the model to
correctly estimate stereo correspondence. We also add valid mask to eliminate
the problems caused by occlusion. Stereo loss is defined as:

$\displaystyle\mathcal{L}_{stereo}=$ $\displaystyle\left\|O_{L\to
R}\circ(I_{L}-\mathcal{M}_{R\to L}\otimes I_{R})\right\|_{1}+$ (10)
$\displaystyle\left\|O_{R\to L}\circ(I_{R}-\mathcal{M}_{L\to R}\otimes
I_{L})\right\|_{1},$

where $I_{L},~{}I_{R}$ denotes the left and right image, respectively. $\circ$
denotes Hadamard product.

The total loss is the sum of these two functions:

$\mathcal{L}_{total}=\mathcal{L}_{sem}+\lambda\mathcal{L}_{stereo},$ (11)

where $\lambda$ is empirically set to $1.0$ for balancing the scales of
different losses.

All above are the basic definitions of the metrics we used for evaluating,
more details can be seen in [21].

##  IV Experiments and results

###  IV-A Experimental Setup

Implementation Details. We apply a batch-mode learning method with a batch
size of $8$ to train our model for 20K iterations. The Adam with default
parameters ($\beta_{1}=0.9,~{}\beta_{2}=0.999$) is utilized to optimize the
network. In addition, the initial learning rate is $2\times 10^{-4}$ and
decreases by half every 2k iterations. After 8k iterations, the learning rate
is fixed to $2\times 10^{-5}$. During training phase, we randomly crop the
images into size $200\times 200$ for augmenting the training data. Following
[16, 13], stereo image pairs with Lab color space is used as input, and the
Lab color space information is zoomed by multiplying a coefficient
$\beta=\eta\max(m_{w}/n_{w},m_{h}/n_{h})$, where $m$ and $n$ represent the
number of superpixels and pixels, $\eta$ is equal to $2.5$. All experiments
are implemented by PyTorch framework on a PC with NVIDIA RTX A4000 GPU.

Figure 10: Qualitative comparison of the proposed method and other state-of-
the-art methods in KITTI2015.

Datasets. Following the experiment settings in [1], we use KITTI2015 [22] and
Cityscapes[23] datasets to train and test our model. KITTI2015 contains 200
stereo image pairs with semantic annotations of left images, we select 150 for
training and 50 for testing. Moreover, to further indicate the superiority of
the proposed method, we also use the Cityscapes dataset for evaluation.
Cityscapes is a larger and more challenging dataset, which contains extensive
stereo image pairs captured with diverse scenes, weathers and illumination
conditions. Since the test set of Cityscapes is not public available, we use
the validation set for comparing, which is consist of 500 stereo images.
Furthermore, the image of Cityscapes has been scaled to quarter-resolution for
convenience.

Evaluation Metrics. In our experiments, we use three widely used metrics to
evaluate the performance of our model, which are achievable segmentation
accuracy (ASA), undersegmentation error (UE), and boundary recall (BR). For
superpixel map $S=\\{S_{i}\\}$ and ground truth of semantic label
$G=\\{G_{j}\\}$, The detailed definitions of these metrics are as follows:

Achievable segmentation accuracy (ASA): ASA is a metric for evaluating the
upper bound on the achievable segmentation accuracy, which can be formulated
as:

$ASA(S,G)=\frac{\sum_{i}max_{j}|S_{i}\cap G_{j}|}{\sum_{j}|G_{j}|}.$ (12)

Undersegmentaion error (UE): UE essentially measures the error of superpixel
segmentation with respect to the ground truth. The UE is defined as:

$UE(S,G)=\frac{1}{|G|}\sum_{G_{j}}\frac{(\sum_{S_{i}\cap
G_{j}}|S_{i}|)-|G_{j}|}{|G_{j}|}.$ (13)

Boundary recall (BR): This is a metric of how well the superpixel adhere to
image boundaries. We use a coefficient $r$ to divide all pixels into two
categories. $TP(S,G)$ is the boundary pixels in $G$ for which there is a
boundary pixel in $S$ in range $r$, and $FN(S,G)$ is the opposite of it. Then
BR can be formulated as:

$BR(S,G)=\frac{TP(S,G)}{TP(S,G)+FN(S,G)}.$ (14)

Figure 11: Qualitative comparison of the proposed method and other state-of-
the-art methods in Cityscapes.

###  IV-B Comparison with State-of-the-Arts

In this part, we compare our method with some state-of-the-art methods,
including SSN[13], FCN[14], LSC[24], StereoDFN[1]. All of the compared methods
are adopted the parameters setting of original works and implemented by the
code released by [21] or the original authors.

Quantitative Comparison. Fig. 9(a) and Fig. 9(b) shows the quantitative
comparison results of our proposed method and other state-of-the-art methods
on KITTI2015 and Cityscapes, respectively. We can see that our method achieves
the top score on KITTI2015 and comparaZble performance to FCN and StereoDFN on
Cityscapes. Taking 700 superpixels for example, in terms of ASA, UE and BR,
our method achieves the minimum percentage gain (computed with the highest
score of other methods) on KITTI2015 is 0.4$\%$, 9.3$\%$, 1.1$\%$, while the
maximum percentage gain is 1.4$\%$, 33.8$\%$, 4.8$\%$, respectively. On
Cityscapes, our method achieves minimum and maximum percentage gain of BR is
2.4$\%$ and 9.8$\%$, respectively, and also achieves a comparable performance
to FCN and StereoDFN in terms of ASA and UE.

Qualitative Comparison. As the qualitative comparison results shown in Fig. 10
and Fig. 11, it is clear that our method achieves the best visual performance
since it can adhere to object boundaries and preserve texture better. More
specifically, on KITTI2015, we can see that our method can adhere to the
boundary of various lane lines more accurately and capture the detail of the
warning sign in low light while other methods cannot. For Cityscapes, only our
method can capture the details on the warning sign and green light on the
traffic light while adhering the image boundaries well.

In conclusion, through the quantitative comparison results based on standard
evaluation criteria, we can see that our method outperforms other methods in
most cases, and achieves the best visual performance in qualitative
comparison. The impressive performance of our method also verifies the
superiority of the proposed spatial decoupling mechanism.

###  IV-C Ablation Experiments

In order to validate the effectiveness of each component in our proposed
network, we perform extensive ablation experiments on KITTI2015. There are
three types of ablation models including T0, T1 and T2, T0 denotes the full
model. For baselines in T1, they can indicate our DSEM can combine the spatial
information better, while baselines in T2 show that our method makes a good
use of information from another viewpoint and improve the performance of
superpixel segmentation. In addition, T1 and T2 contain 3 ablation models
respectively, More specifically,

  * •

B1 denotes the ablation model without SE and DF modules.

  * •

B2 stands for the ablation model with spatial information (XY) without SE and
DF modules.

  * •

B3 represents the ablation model without DF module.

  * •

B4 means that the ablation model without stereo loss, and does not consider
stereo features alignment and occlusion problem.

  * •

B5 refers to the ablation model without considering occlusion problem.

  * •

B6 is the ablation model without stereo loss

All of the ablation experiments are trained for 20K iterations. The specific
structure of each ablation model also has been shown in TABLE I.

Figure 12: Visual comparison of SOD results with different superpixel segmentation methods. Note that our method can preserve more details than others. Type | ID | Input | Component | Stereo Loss
---|---|---|---|---
SFA | OH | SE | DF
T0 | B0 | Stereo | ✓ | ✓ | ✓ | ✓ | ✓
| B1 | Stereo | ✓ | ✓ |  |  | ✓
T1 | B2 | Stereo+XY | ✓ | ✓ |  |  | ✓
| B3 | Stereo | ✓ | ✓ | ✓ |  | ✓
| B4 | Single |  |  | ✓ | ✓ |
T2 | B5 | Stereo | ✓ |  | ✓ | ✓ | ✓
| B6 | Stereo | ✓ | ✓ | ✓ | ✓ |
TABLE I: Detailed setup for ablation experiments. B$n$ and T$n$ denote the
$n$th baseline and the $n$th type respectively.

Figure 13: Ablation studies on KITTI2015. The top figure shows the
contributions of each component through ASA score, while the bottom figure
through BR score.

Effectiveness of Each Component. Fig. 13 reports the quantitative comparison
results of ablation models on KITTI2015. We can see that adding spatial
information directly can not make full use of it. However, the spatial
information can be embedded into the network better through our SE and DF
modules, resulting in higher performance gains. In addition, the DSFM module
and Stereo Loss also play an important role, which can solve the stereo
features alignment and occlusion problem and constrain the model to correctly
model stereo correspondence, respectively.

Influence of Spatial Information. From Fig. 13, we can observe that model with
SE module tends to have a larger performance improvement than the model
without SE module, which proves that the spatial information is helpful to
generate regular and compact superpixels. Furthermore, adjusting the weighting
of the spatial information adaptively through the DF module can make better
use of it to further improve the performance.

##  V Application on Salient Object Detection

Salient object detection (SOD) has attracted increasing interest in recent
years, since it plays a significant role in many popular computer vision
tasks, including object recognition and detection [25, 26], image retargeting
[27, 28], semantic segmentation [29, 30], etc. To improve the performance of
salient object detection, Zhu et al. [31] propose an superpixel-based salient
object detection method, they treat the saliency object detection problem as a
saliency value optimization problem for all superpixels in an image. Moreover,
they observe that background regions are more connected to image boundaries
than salient object regions. Therefore, they propose a measure called boundary
connectivity, which is utilized to perform salient object detection in their
proposed method. The boundary connectivity is defined as follows:

$BndCon(R)=\frac{|\\{p\mid p\in R,p\in Bnd\\}|}{\sqrt{|\\{p\mid p\in R\\}|}},$
(15)

where $p$ is a patch of an image and $Bnd$ is the set of all image boundary
patches.

Method |  SSN |  StereoDFN |  Ours
---|---|---|---
MAE $\downarrow$ |  0.1783 |  0.1794 |  0.1777
E-measure $\uparrow$ |  0.6489 |  0.6427 |  0.6498
TABLE II: Results on NJU2K benchmark. $\uparrow$ denotes the higher is the
better, and $\downarrow$ is contrary.

To indicate our method can perform better in downstream task, we use three
state-of-the-art methods, which are our proposed method, StereoDFN[1] and
SSN[13] to replace the default SLIC[16] as the superpixel segmentation method
of [31]. In our experiments, we use NJU2K [32] for evaluation. NJU2K is a
large dataset widely used for salient object detection of stereo images, which
contains 2000 stereo image pairs, involving various objects and scenarios of
different difficulty levels. Moreover, we select the first 400 images of NJU2K
and resize all of them to size $400\times 400$ for the ease of
experimentation.

Following [31], we choose the mean absolute error (MAE) [33] to evaluate each
method quantitatively, which is a metric to measure the average difference
between the binary ground truth and saliency prediction map. However, MAE only
focus on pixel-wise error. To consider structure cues, we also introduce
Enhanced-alignment measure (E-measure) [34] as our evaluation metric.

The results of the quantitative evaluation are shown in Table II, we can see
that our method achieves the best performance in terms of MAE and E-measure.
In addition, the visual comparison results in Fig. 12 also illustrate the
saliency map generated based on our method can focus on more details than
other state-of-the-art methods, which validates that our method can perform
well in downstream task both qualitatively and quantitatively.

##  VI Conclusion

Previously, stereo superpixel segmentation methods neglect the coupling of
stereo features and spatial features, which may impose strong constraints on
spatial information while modeling the correspondence between stereo image
pairs. To solve this problem, we have presented an end-to-end stereo
superpixel segmentation network with a decoupling mechanism of spatial
information to eliminate such negative influence. In addition, spatial
information is adjusted adaptively through our dynamic fusion mechanism in
dynamic spatiality embedding module to generate regular and compact
superpixels. Extensive experiments on several popular datasets have shown that
our proposed method achieves the state-of-the-art performance and performs
well in downstream task. The effectiveness of the components handling the
spatial information and stereo features have also been verified in our
ablation studies.

## References

  * [1] R. Wu, Y. Du, H. Li, and Y. Dai, “Stereo superpixel segmentation via dual-attention fusion networks,” in _Proc. Int. Conf. Multimedia Expo_ , 2021, pp. 1–6.
  * [2] R. Cong, J. Lei, H. Fu, J. Hou, Q. Huang, and S. Kwong, “Going from rgb to rgbd saliency: A depth-guided transformation model,” _IEEE Trans. Cybern._ , vol. 50, no. 8, pp. 3627–3639, 2020.
  * [3] R. Cong, J. Lei, H. Fu, F. Porikli, Q. Huang, and C. Hou, “Video saliency detection via sparsity-based reconstruction and propagation,” _IEEE Trans. Image Process._ , vol. 28, no. 10, pp. 4819–4831, 2019.
  * [4] R. Cong, J. Lei, H. Fu, Q. Huang, X. Cao, and C. Hou, “Co-saliency detection for rgbd images based on multi-constraint feature matching and cross label propagation,” _IEEE Trans. Image Process._ , vol. 27, no. 2, pp. 568–579, 2018.
  * [5] M. Xu, B. Liu, P. Fu, J. Li, and Y. H. Hu, “Video saliency detection via graph clustering with motion energy and spatiotemporal objectness,” _IEEE Trans. Multimedia_ , vol. 21, no. 11, pp. 2790–2805, 2019.
  * [6] M. Yang, J. Liu, and Z. Li, “Superpixel-based single nighttime image haze removal,” _IEEE Trans. Multimedia_ , vol. 20, no. 11, pp. 3008–3018, 2018\.
  * [7] C. Shi and C.-M. Pun, “Multiscale superpixel-based hyperspectral image classification using recurrent neural networks with stacked autoencoders,” _IEEE Trans. Multimedia_ , vol. 22, no. 2, pp. 487–501, 2019.
  * [8] C. Wang, Z. Liu, and S.-C. Chan, “Superpixel-based hand gesture recognition with kinect depth camera,” _IEEE Trans. Multimedia_ , vol. 17, no. 1, pp. 29–39, 2014.
  * [9] X. Dong, J. Han, D. Chen, J. Liu, H. Bian, Z. Ma, H. Li, X. Wang, W. Zhang, and N. Yu, “Robust superpixel-guided attentional adversarial attack,” in _Proc. Conf. Comput. Vision Pattern Recognit._ , 2020, pp. 12 892–12 901.
  * [10] H. Li, R. Cong, S. Kwong, C. Chen, Q. Xu, and C. Li, “Stereo superpixel: An iterative framework based on parallax consistency and collaborative optimization,” _Inf. Sci._ , vol. 556, pp. 209–222, 2021.
  * [11] F. Cheng, H. Zhang, M. Sun, and D. Yuan, “Cross-trees, edge and superpixel priors-based cost aggregation for stereo matching,” _Pattern Recognit._ , vol. 48, no. 7, pp. 2269–2278, 2015.
  * [12] L. Liu, W. Tao, and H. Liu, “Complementary saliency driven co-segmentation with region searching and hierarchical constraint,” _Inf. Sci._ , vol. 372, pp. 72–83, 2016.
  * [13] V. Jampani, D. Sun, M. Liu, M. Yang, and J. Kautz, “Superpixel sampling networks,” in _Proc. Eur. Conf. Comput. Vis._ , 2018, pp. 363–380.
  * [14] F. Yang, Q. Sun, H. Jin, and Z. Zhou, “Superpixel segmentation with fully convolutional networks,” in _Proc. Conf. Comput. Vision Pattern Recognit._ , 2020, pp. 13 961–13 970.
  * [15] Ren and Malik, “Learning a classification model for segmentation,” in _Proc. Int. Conf. Comput. Vis._ , 2003, pp. 10–17 vol.1.
  * [16] R. Achanta, A. Shaji, K. Smith, A. Lucchi, P. Fua, and S. Süsstrunk, “Slic superpixels compared to state-of-the-art superpixel methods,” _IEEE Trans. Pattern Anal. Mach. Intell._ , vol. 34, no. 11, pp. 2274–2282, 2012.
  * [17] J. Chen, Z. Li, and B. Huang, “Linear spectral clustering superpixel,” _IEEE Trans. Image Process._ , vol. 26, no. 7, pp. 3317–3330, 2017.
  * [18] H. Li, S. Kwong, C. Chen, Y. Jia, and R. Cong, “Superpixel segmentation based on square-wise asymmetric partition and structural approximation,” _IEEE Trans. Multimedia_ , vol. 21, no. 10, pp. 2625–2637, 2019.
  * [19] L. Wang, Y. Wang, Z. Liang, Z. Lin, J. Yang, W. An, and Y. Guo, “Learning parallax attention for stereo image super-resolution,” in _Proc. Conf. Comput. Vision Pattern Recognit._ , 2019, pp. 12 242–12 251.
  * [20] J. Hu, L. Shen, S. Albanie, G. Sun, and E. Wu, “Squeeze-and-excitation networks,” _IEEE Trans. Pattern Anal. Mach. Intell._ , vol. 42, no. 8, pp. 2011–2023, 2020.
  * [21] M. Wang, X. Liu, Y. Gao, X. Ma, and S. Nouman, “Superpixel segmentation: A benchmark,” _Signal Process.-Image Commun._ , vol. 56, pp. 28–39, 2017.
  * [22] M. Menze, C. Heipke, and A. Geiger, “Joint 3d estimation of vehicles and scene flow,” in _Proc. ISPRS Workshop Image Sequence Anal._ , 2015.
  * [23] M. Cordts, M. Omran, S. Ramos, T. Rehfeld, M. Enzweiler, R. Benenson, U. Franke, S. Roth, and B. Schiele, “The cityscapes dataset for semantic urban scene understanding,” in _Proc. Conf. Comput. Vision Pattern Recognit._ , 2016, pp. 3213–3223.
  * [24] J. Chen, Z. Li, and B. Huang, “Linear spectral clustering superpixel,” _IEEE Trans. Image Process._ , vol. 26, no. 7, pp. 3317–3330, 2017.
  * [25] Z. Ren, S. Gao, L.-T. Chia, and I. W.-H. Tsang, “Region-based saliency detection and its application in object recognition,” _IEEE Trans. Circuits Syst. Video Technol._ , vol. 24, no. 5, pp. 769–779, 2013.
  * [26] D. Zhang, D. Meng, L. Zhao, and J. Han, “Bridging saliency detection to weakly supervised object detection based on self-paced curriculum learning,” in _Proc. Int. Joint Conf. Artif. Intell._ , 2016.
  * [27] Y. Ding, J. Xiao, and J. Yu, “Importance filtering for image retargeting,” in _Proc. Conf. Comput. Vision Pattern Recognit._ , 2011, pp. 89–96.
  * [28] J. Sun and H. Ling, “Scale and object aware image retargeting for thumbnail browsing,” in _Proc. IEEE Int. Conf. Comput. Vis._ , 2011, pp. 1511–1518.
  * [29] Y. Wei, X. Liang, Y. Chen, X. Shen, M.-M. Cheng, J. Feng, Y. Zhao, and S. Yan, “Stc: A simple to complex framework for weakly-supervised semantic segmentation,” _IEEE Trans. Pattern Anal. Mach. Intell._ , vol. 39, no. 11, pp. 2314–2320, 2017.
  * [30] X. Wang, S. You, X. Li, and H. Ma, “Weakly-supervised semantic segmentation by iteratively mining common object features,” in _Proc. Conf. Comput. Vision Pattern Recognit._ , 2018, pp. 1354–1362.
  * [31] W. Zhu, S. Liang, Y. Wei, and J. Sun, “Saliency optimization from robust background detection,” in _Proc. Conf. Comput. Vision Pattern Recognit._ , 2014, pp. 2814–2821.
  * [32] R. Ju, Y. Liu, T. Ren, L. Ge, and G. Wu, “Depth-aware salient object detection using anisotropic center-surround difference,” _Signal Process.-Image Commun._ , vol. 38, pp. 115–126, 2015.
  * [33] F. Perazzi, P. Krähenbühl, Y. Pritch, and A. Hornung, “Saliency filters: Contrast based filtering for salient region detection,” in _Proc. Conf. Comput. Vision Pattern Recognit._ , 2012, pp. 733–740.
  * [34] D.-P. Fan, C. Gong, Y. Cao, B. Ren, M.-M. Cheng, and A. Borji, “Enhanced-alignment measure for binary foreground map evaluation,” in _Proc. Int. Joint Conf. Artif. Intell._ , 7 2018, pp. 698–704.