# Learning to Segment Rigid Motions from Two Frames

Gengshan Yang1 Code will be available at github.com/gengshan-y/rigidmask. Deva
Ramanan1,2
1Carnegie Mellon University  2Argo AI
{gengshay<EMAIL_ADDRESS>
###### Abstract

Appearance-based detectors achieve remarkable performance on common scenes,
benefiting from high-capacity models and massive annotated data, but tend to
fail for scenarios lack of training data. Geometric motion segmentation
algorithms, however, generalize to novel scenes, but have yet to achieve
comparable performance to appearance-based ones, due to noisy motion
estimations and degenerate motion configurations. To combine the best of both
worlds, we propose a modular network, whose architecture is motivated by a
geometric analysis of what independent object motions can be recovered from an
egomotion field. It takes two consecutive frames as input and predicts
segmentation masks for the background and multiple rigidly moving objects,
which are then parameterized by 3D rigid transformations. Our method achieves
state-of-the-art performance for rigid motion segmentation on KITTI and
Sintel. The inferred rigid motions lead to a significant improvement for depth
and scene flow estimation. At the time of submission, our method ranked 1st on
KITTI scene flow leaderboard, out-performing the best published method (SF
error: 4.89% vs 6.31%).

Figure 1: (a) Many data-driven segmentation methods heavily rely on appearance
cues, and fail for novel test scenes. For instance, PointRend [27] trained on
MSCOCO fails to detect coral reef fishes even with a low confidence threshold
of 0.1. (b) On the other hand, geometric motion segmentation [5, 60]
generalizes to novel appearance, but fails due to noisy flow inputs and
degenerate motion configurations. (c)-(e) We propose a neural architecture
powered by geometric reasoning that decomposes a scene into a rigid background
and multiple moving rigid bodies, parameterized by 3D rigid transformations.
It demonstrates generalization ability to novel scenes and robustness to noisy
inputs as well as motion degeneracies. The inferred rigid motions
significantly improve depth and scene flow accuracy.

##  1 Introduction

Autonomous agents such as self-driving cars need to be able to navigate safely
in dynamic environments. Static environments are far easier to process because
one can make use of geometric constraints (SFM/SLAM) to infer scene structure
[16]. Dynamic environments require the fundamental ability to both segment
moving obstacles and estimate their depth and speed [2]. Popular solutions
include object detection or semantic segmentation [29]. While one can build
accurate detectors for many categories of objects that are able to move,
“being able to move” is not equivalent to “moving”. For example, there is a
profound difference between a parked car and an ever-so-slightly moving car
(that is pulling out of parked location), in terms of the appropriate response
needed from a nearby autonomous agent. Secondly, class-specific detectors rely
heavily on appearance cues and categories present in a training set. Consider
a trash can that falls on the street; current closed-world detectors will
likely not be able to model all types of moving debris. This poses severe
implications for safety in the open-world that a truly autonomous agent must
operate [4].

Problem formulation: We follow historic work on motion-based perceptual
grouping [24, 45, 52, 58, 63] and segment moving objects without relying on
appearance cues. Specifically, we focus on segmenting rigid bodies from two
frames. We focus on two-frame because it is the minimal set of inputs to study
the problem of motion segmentation, and in practice, perception-for-autonomy
needs to respond immediately to dynamic scenes, e.g., an animal that appears
from behind an occlusion. We focus on rigid body and its 3D motion
parameterizations because it’s directly relevant for an autonomous agent
acting in a 3D world. While dynamic scenes often contain nonrigid objects such
as people, we expect that deformable objects may be modeled as a rigid body
over short time scales, or decomposed into rigidly-moving parts [1, 8]. One
example of our method decomposing a flying dragon into multiple rigid parts is
shown in Fig. 2.

Figure 2: Results on Sintel sequence temple_2, frame 17-18. (a)-(b) Our method
segments rigid motions and fits 3D rigid transformations over two frames. The
blue and red arrows indicate the estimated motion of the rigid background and
parts respectively. (c)-(d) An initial depth is refined by triangulating
optical flow within each rigid motion mask. Note that the tower in the cyan
rectangle is leaning in the initial MiDaS [39] depth, but “rectified” by our
method. (e)-(f) Our method segments rigid objects more reliably than the prior
two-frame rigidity estimation method [32] and generalizes to novel appearance
compared to appearance-based detectors [21].

Challenges: Earlier work on rigid motion segmentation often makes use of
geometric constraints arising from epipolar geometry and rigid
transformations. However, there are several fundamental difficulties that
plague geometric motion segmentation. First, epipolar constraints fail when
camera motion is close to zero [63]. Second, points moving along epipolar
lines cannot be distinguished from the rigid background [68], which we discuss
at length in Sec. 3.1. Third, geometric criteria are often not robust enough
to noisy motion correspondences and camera egomotion estimates, which can lead
to catastrophic failures in practice.

Figure 3: Collinear motion ambiguity. (a) The input scene contains a dynamic
object (the car in the lower left) moving parallel to camera translational
direction. (b) One can triangulate motion correspondences assuming overall
rigidity that places the moving car at an elevated height, which illustrates
both (1) the commonality of this degenerate case [68] in urban navigation, and
(2) one solution of using structural scene priors that do not allow for
floating objects above the ground. (c) Due to such ambiguities, the 2D motion
of the moving car is consistent with the camera egomotion, leaving it
indistinguishable under classic motion segmentation metrics such as Sampson
error [20].

Method: We theoretically analyze ambiguities in 3D rigid motion segmentation,
and resolve such ambiguities by exploiting recent techniques for upgrading 2D
motion observation to 3D with optical expansion [65] and monocular depth cues
[39]. To deal with noisy motion correspondences and degenerate scene motion,
we design a convolutional architecture that segments the rigid background and
an arbitrary number of rigid bodies from a given motion field. Finally, we
parameterize the 3D motion of individual rigid bodies by fitting 3D rigid
transformations.

Contributions: (1) We provide a geometric analysis for ambiguities in 3D rigid
motion segmentation from 2D motion fields, and introduce solutions to deal
with such ambiguities. (2) We propose a geometry-aware architecture for 3D
rigid motion segmentation from two RGB frames, which is generalizable to novel
appearance, resilient to different motion types and robust to noisy motion
observations. (3) Our method achieves state-of-the-art (SOTA) performance of
rigid motion segmentation on KITTI/Sintel. The inferred rigidity masks
significantly improve the performance of downstream depth and scene flow
estimation tasks.

##  2 Related Work

Geometric Motion Segmentation: The problem of clustering motion
correspondences into groups that follow a similar 3D motion model has been
extensively studied in the past [49, 50, 52, 54, 55, 63, 68]. However, prior
methods either focus on theoretical analysis with noisy-free data, or assume
relatively simple scenes where long-term motion trajectories can be obtained
by point tracking algorithms. Some recent work [5, 7, 60] tackles more complex
scenarios with two-frame optical flow inputs, where geometric constraints,
such as motion angle and plane plus parallax (P+P) [44] are considered as cues
of “moving versus static”. However, such geometric constraints are sensitive
to noise in optical flow even under a robust estimation framework [17].
Moreover, as we shall see in Sec. 3.1, the prior two-frame solutions do not
deal with several degenerate cases, including co-planar/co-linear motion [68]
and camera motion degeneracy [50]. We address these problems by encoding
geometric constraints into a modular neural network.

Learning-Based Video Object Segmentation: Segmenting salient objects from
videos historically stems from the problem of image salient object detection
[36, 37], where existing methods often rely either on appearance features or
on salient motions from 2D optical flow [25, 31, 47, 48, 66, 70]. Oftentimes,
optical flow is interpreted as a color image [25, 70], where geometric
information, such as camera egomotion, is ignored. Close to our methodology,
Motion Angle Network (MoA-Net) [6], analytically reduces the effect of camera
rotation and uses the “rectified” flow angle as input features to a binary
segmentation network. Our approach further incorporates 3D flow and depth cues
and segments multiple 3D rigid motions.

Instance Scene Flow: Scene flow is the problem of resolving dense 3D scene
motion from an ego-camera [35, 53], which is inherently challenging due to the
lack of visual evidence for correspondence matching in the local patch, for
example, occluded or specular surfaces. Prior approaches often utilize scene
rigidity priors to resolve such ambiguities, such as piecewise rigidity prior
[35, 56] and semantic rigidity prior [3, 33]. Then, scene flow is
parameterized as the 3D motion of each rigid segment. However, it is risky to
segment the scene purely relying on semantics – object that is able to move is
not the same as objects that are moving. Furthermore, such high-level cues do
not generalize to an open-world, where algorithms are required to be robust to
never-before-seen categories [4]. Instead, we exploit _motion rigidity_ for
scene flow estimation, which decomposes the scene into multiple rigidly moving
segments while preserving the completeness of individual rigid bodies.

Figure 4: When can a scene point ${\bf P}$ be identified as moving? Assuming
camera rotation has been removed, we have 3D scene flow (defined as motion
relative to the camera) ${\bf T_{sf}}={\bf T_{bg}}+{\bf T_{o}}$, where ${\bf
T_{bg}}=-{\bf T_{c}}$ is the rigid background motion induced by the camera
motion, and ${\bf T_{o}}$ is the independent object motion. Case (I): Assuming
a rigid scene point ${\bf P}$ with zero independent motion ${\bf T_{o}}={\bf
0}$, the 2D motion projected by ${\bf T_{sf}}$ must lie on the epipolar line.
Case (II): In other words, if the 2D motion deviates from the epipolar line,
$|{\alpha}|>{\alpha}_{0}$, ${\bf P}$ must be moving, analagous to Sampson
error [20]. Case (III): However, the inverse does not hold. If 2D flow is
consistent with the background motion ($|{\alpha}|<{\alpha}_{0}$), ${\bf P}$
might still be moving in the epipolar plane. However, if the angular direction
of 3D flow ${\bf T_{sf}}$ – computable from optical expansion [65] – differs
from ${\bf T_{bg}}$ ($|{\beta}|>{\beta}_{0}$), ${\bf P}$ must be moving. Case
(IV): If the 3D flow direction is consistent with background motion
($|{\beta}|<{\beta}_{0}$), ${\bf P}$ could still be moving in the direction of
${\bf T_{bg}}$, making it unrecoverable without knowing the scale (or relative
depth).

##  3 Approach

In this section, we first analyze degeneracies in motion segmentation that
arise when dynamic motion is indistinguishable from the camera motion, and
what information is required to resolve the ambiguities. We then design a
neural architecture for rigid instance motion segmentation that builds on this
geometric analysis, producing a pipeline for two-frame rigid motion
segmentation.

###  3.1 Two-Frame 3D Motion Segmentation

Problem setup: Given two-frame motion correspondences $({\bf p_{0}},{\bf
p_{1}})\in\mathbb{R}^{4}$, observed by a monocular camera with known camera
intrinsics $({\bf K_{0}},{\bf K_{1}})$, we are interested in detecting points
whose 3D motion cannot be described by the camera motion ${\bf R_{c}}\in{\bf
SO}(3)$, ${\bf T_{c}}\in\mathbb{R}^{3}$, such that

$\footnotesize({\bf R_{c}}{\bf P_{1}}+{\bf T_{c}})-{\bf P_{0}}\neq{\bf
0},\qquad\text{(transform of coordinates)}$ (1)

where ${{\bf P_{0}}=Z_{0}{\bf K_{0}}^{-1}{\bf\tilde{p}_{0}}}$ and ${\bf
P_{1}}=Z_{1}{\bf K_{1}}^{-1}{\bf\tilde{p}_{1}}$ are corresponding 3D points
observed in the camera coordinate systems of both frames, given corresponding
depth $(Z_{0},Z_{1})$ and homogeneous coordinates
$({\bf\tilde{p}_{0}},{\bf\tilde{p}_{1}})$. To gain more geometric insights, we
re-arrange Eq. (1) into

$\footnotesize\begin{split}&{\bf T_{sf}}={\bf K_{0}}^{-1}(Z_{1}{\bf
H_{R}}{\bf\tilde{p}_{1}}-Z_{0}{\bf\tilde{p}_{0}})\neq-{\bf T_{c}},\\\
&(\text{``rectified'' 3D scene flow $\neq$ negative camera
translation})\end{split}$ (2)

where ${\bf T_{sf}}={\bf R_{c}}{\bf P_{1}}-{\bf P_{0}}$ is the “rectified” 3D
scene flow, with the motion induced by camera rotation ${\bf R_{c}}$ removed
through “rectifying” the second frame coordinate system to have the same
orientation as the first frame; and ${\bf H_{R}}={\bf K_{0}}{\bf R_{c}}{\bf
K_{1}}^{-1}$ is the rotational homography that “rectifies” the second image
plane into the same orientation as the first image plane, removing the effect
of camera rotation from the 2D motion fields. Eq. (LABEL:eq:2D-rt) states that
the rectified 3D scene flow of a moving point ${\bf P}$ will not equal the
negative camera translation. However, assuming the camera motion is known,
there are still two crucial degrees of freedom that are undetermined: depth
$Z_{0}$ and $Z_{1}$.

Coplanar motion degeneracy: Solving for $Z_{0}$ and $Z_{1}$ equates to
estimating the depth and 3D scene flow, which itself is challenging [35]. To
remove such dependencies, classic geometric motion segmentation segments
points whose _2D motion_ is inconsistent with the camera motion, measured
either by Sampson distance to the epipolar line [20, 49] or plane plus
parallax (P+P) [44] representations that factor out camera rotation, allowing
one to evaluate the angular deviation of the 2D motion to the epipole [5, 24].
However, is 2D motion sufficient to segment points moving in 3D? The answer is
no (Fig. 3). Formally, 3D points that translate within the epipolar plane
defined by the camera translation vector ${\bf T}_{c}$ will project to the
epipolar line, making them "appear" as stationary points, as shown in Fig. 4
Case (II).

To detect such co-planar motion, we make use of optical expansion cues that
upgrade 2D flow to 3D as suggested by recent work [65]. Optical expansion,
measured by the scale change of overlapping image patches, approximates the
relative depth $\tau=\frac{Z_{1}}{Z_{0}}$ for non-rotating scene elements
under scaled orthographic projection [65]. We derive a 3D motion angle
criterion that does not require depth, but removes the ambiguity of points
moving within the epipolar plane. Normalizing Eq. (LABEL:eq:2D-rt) by depth
$Z_{0}$, we have

$\footnotesize\begin{split}&{\bf\tilde{T}_{sf}}={\bf K_{0}}^{-1}(\tau{\bf
H_{R}}{\bf\tilde{p}_{1}}-{\bf\tilde{p}_{0}})\not\sim-{\bf T_{c}},\\\
&(\text{rectified 3D flow direction $\neq$ neg. camera translation
direction})\end{split}$ (3)

where ${\bf\tilde{T}_{sf}}=\frac{{\bf{T}_{sf}}}{Z_{0}}$ is the rectified and
normalized 3D flow and $\not\sim$ indicates two vectors are different in their
directions. Eq. (LABEL:eq:nsf-rt) states that a point is moving if the
direction of its rectified 3D scene flow is not consistent with the direction
of the camera translation, as shown in Fig. 4 Case (III).

Collinear motion degeneracy: However, there is still a remaining ambiguity
that cannot be resolved. If point ${\bf P}$ moves in the opposite direction of
the camera translation, both classic criteria and Eq. (LABEL:eq:nsf-rt) would
fail, as shown in Fig. 4 Case (IV). Such ambiguity remains even given multiple
frames [68], but is common in many real-world applications, e.g., two cars
passing each other (Fig. 3). To identify moving points in such cases, one
could use depth $Z_{0}$ to recover the metric scale of normalized rectified
scene flow ${\bf\tilde{T}_{sf}}$, and compare it with camera translation ${\bf
T_{c}}$. However, in a monocular setup, we neither know the scale of ${\bf
T_{c}}$ nor trust the overall scale of $Z_{0}$ [20]. Instead, we derive a
depth contrast criterion, inspired by an observation that dynamic scene points
triangulated from flow assuming overall rigidity will appear “abnormal” in the
3D reconstruction, such as the floating car in Fig. 3 (b). To do so, we
contrast the flow-derived depth $Z^{flow}_{0}$ with a depth prior
$Z_{0}^{prior}$,

$\footnotesize Z_{0}^{flow}\neq\gamma Z^{prior}_{0},\quad(\text{flow-
triangulated depth $\neq$ depth prior})$ (4)

where $Z^{flow}_{0}$ can be computed efficiently using midpoint or DLT
triangulation algorithm [20], depth prior $Z^{prior}_{0}$ can be represented
by a data-driven monocular depth network, and the scale factor $\gamma$ that
globally aligns $Z^{prior}_{0}$ to $Z^{flow}_{0}$ can be determined by robust
least squares [46].

Egomotion degeneracy: Furthermore, when the camera translation is small, ${\bf
T_{c}}$ is notoriously difficult to estimate due to small motion parallax. In
such cases, rigid-background motion (and objects that deviate from it) is
easier to model with a rotational homography model [49, 51].

Figure 5: We detect and estimate rigid motions in three steps: First, initial
3D scene flow is computed using off-the-shelf networks (M) and camera motion
is estimated by epipolar geometry (P) given two frames. Then, rigidity cost
maps and rectified scene flow are computed (G) and fed into a two-stream
network that produces the segmentation masks of a rigid background and an
arbitrary number of rigidly moving instances. Finally, we fit a rigid
transformation for the background and each rigid instance to update their
depth and 3D scene flow.

###  3.2 Learning to Segment Rigid Motions

We now operationalize our motion analysis into a deep network for rigid motion
segmentation (Fig. 5). At its heart, our network learns to transform motion
measurements (noisy 3D scene flow) into pixel-level masks of rigid background
and instances. To do so, we construct motion cost maps designed to address the
motion degeneracies described earlier. Given such input maps and raw scene
flow measurements, we use a two-stream network architecture that separately
regresses the rigid background and rigid instance masks.

Motion estimation: First, we extract the camera and relative scene motion
given two frames. We apply existing methods to estimate optical flow, optical
expansion and monocular depth [39, 65]. To estimate camera motion, we fit and
decompose essential matrices from flow maps using the five-point algorithm
with a differentiable and parallel RANSAC [9].

Rigidity cost-maps inputs: We construct rigidity cost maps tailored to camera-
object motion configurations analysed in Sec. 3.1, including (1) an epipolar
cost for general configurations, computed as per-pixel Sampson error [20]; (2)
a homography cost to deal with small camera translations, implemented as per-
pixel reprojection error according to the rotational homography [15] in Eq.
(LABEL:eq:2D-rt); (3) a 3D P+P cost for coplanar motion configurations
computed as Eq. (LABEL:eq:nsf-rt); and (4) a depth contrast cost to deal with
colinear motion ambituity, computed as Eq. (4).

Rectified scene flow inputs: Besides rigidity cost-maps, we find it helpful to
also input raw scene flow measurements, represented as an 8-channel feature
map, containing the first frame 3D scene points ${\bf P_{0}}$, rectified
motion fields ${\bf T_{sf}}$, and uncertainty estimations of flow and optical
expansion $(\sigma_{1},\sigma_{2})$. To compute ${\bf P_{0}}$, we back-project
the first frame pixel coordinates given monocular depth inputs; to compute
${\bf T_{sf}}$, we upgrade optical flow using optical expansion $\tau$,

$\footnotesize\begin{split}&{\bf\tilde{T}_{sf}}={\bf K_{0}}^{-1}(\tau{\bf
H_{R}}{\bf\tilde{p}_{1}}-{\bf\tilde{p}_{0}}),\end{split}$ (5)

where the second coordinate frame is rectified by rotational homography ${\bf
H_{R}}={\bf K_{0}}{\bf R_{c}}{\bf K_{1}}^{-1}$ to remove the effect of camera
rotation. Finally, the uncertainty of optical flow and optical expansion are
computed as out-of-range confidence score and Gaussian variance respectively
[23, 64]. Such rectified scene flow inputs are more effective than 2D optical
flow, as empirically tested in ablation study (Tab. LABEL:tab:aba).

Architecture and losses: We use a two-stream architecture: (1) a lightweight
U-Net [42] architecture to predict binary labels for pixels belonging to the
(rigid) background and (2) a CenterNet [71] architecture to predict pixel
instance masks. Inspired by the single-shot segmentation head proposed in
PolarMask [61], stream (2) outputs a heatmap representing object centers and a
regression map of $K=36$ polar distances at evenly distributed angles. The
overall architecture is end-to-end differentiable and can be trained with
standard loss functions,

$\displaystyle
L=\alpha_{1}L_{\text{binary}}+\alpha_{2}L_{\text{center}}+\alpha_{3}L_{\text{polar}}$
(6)

where $L_{\text{binary}}$ is binary cross-entropy loss with label balancing,
$L_{\text{center}}$ is the focal loss and $L_{\text{polar}}$ is an $L_{1}$
regression loss evaluated at each ground-truth center location. Weights are
empirically chosen as $\alpha_{1}=1^{-3}$, $\alpha_{2}=1^{-3}$ and
$\alpha_{3}=1^{-8}$. Intuitively, stream (2) generates coarse instance-level
masks that are refined by pixel-accurate background masks from stream (1).
Specifically, pixels where rigid background and instance predictions disagree
are not used for rigid body fitting below, and marked as wrong prediction in
evaluation.

Rigid body scene flow: Given rigid motion segmentations, scene flow of the
background and N rigid instances are parameterized by their rotations,
translations $\\{({\bf R}_{i},{\bf T}_{i}),i=0,\cdots,N\\}$ and the first
frame 3D scene points ${\bf P_{0}}$. To find the best fit of rotations and up-
to-scale translations, we estimate and decompose essential matrices over flow
correspondences with a least median of squares estimator [20]. To obain a more
reliable 3D reconstruction than back-projected monocular depth, we triangulate
flow using rigid motion estimations for each rigid body, and determine their
scales by aligning each triangulated depth map to monocular depth inputs with
robust least squares [46]. With the above parameterization, the second frame
coordinates are computed as,

$\footnotesize{\bf P_{1}}=\textstyle\sum_{i=0}^{N}{\bf S}_{i}({\bf R_{i}}{\bf
P_{0}}+{\bf T_{i}}),$ (7)

where ${\bf S}_{i}$ is a one-hot rigid motion segmentation vector.

Figure 6: Comparison on KITTI-SF, image 137. The prefix of each method
indicates the test-time inputs: (1)Single frame. (2)Multi-frame with
appearance features. (3)Multi-frame without appearance. Our best appearance-
based segmentation baseline, (1)Mask R-CNN [21] detects all the moving
vehicles, but also reports parked cars in the background. (2)CC [40] correctly
detects moving cars but also reports the edge of the road as moving objects.
(3)Geometric segmentation algorithm [5, 60] fails on the approaching car due
to the colinear motion ambiguity, and reports false positives at the
background due to the noisy flow estimation. In contrast, (3)our method
correctly segments both the moving vehicles and the rigid background. Rigid
motions are estimated within each mask and applied to depth and scene flow
estimation.

##  4 Experiments

Our method is quantitatively compared with state-of-the-art rigidity
estimation algorithms on KITTI and Sintel in Sec. 4.1, and then applied to the
depth and scene flow estimation tasks in Sec. 4.2. In Sec. 4.3 we conclude
with an ablation study.

Table 1:  Rigidity estimation on KITTI (K) and Sintel (S) without fine-tuning. (1)Single frame. (2)Multi-frame with appearance features. (3)Multi-frame without appearance. The best result under each metric (IoU in %) is bolded. ∗:For methods only estimating background masks, we use connected components to obtain object masks. ‡:Methods trained on target dataset. MR-Flow-S (K) is trained on KITTI, and MR-Flow-S (S) is trained on Sintel. | Method |  K: obj $\uparrow$ |  K: bg $\uparrow$ |  S: bg $\uparrow$
---|---|---|---|---
(1) |  Mask R-CNN [59] | 88.20 | 96.42 | 81.98
U2 (Saliency) [38] |  64.80∗ | 93.34 | 82.01
MR-Flow-S (K) [60] |  75.59∗ |  94.70‡ | 76.11
MR-Flow-S (S) [60] |  11.11∗ | 84.72 |  92.64‡
(2) |  FSEG [25] |  85.08∗ | 96.27 | 80.22
MAT-Net [70] |  68.40∗ | 93.08 | 77.95
COSNet [31] |  66.67∗ | 93.03 | 80.86
CC [40] |  50.87∗ | 85.50 | ✗
RTN [32] |  34.29∗ | 84.44 | 64.86
(3) |  FSEG-Motion [25] | 61.29 | 89.41 | 78.25
CC-Motion [40] | 42.99 | 74.06 | ✗
Flow angle [5, 60] | 25.83 | 85.52 | 74.23
Ours | 90.71 | 97.05 | 86.72

Dataset: We use KITTI-SF (sceneflow) and Sintel for quantitative analysis.
KITTI-SF [18, 35] features an urban driving scene with multiple rigidly moving
vehicles. Sintel [11] is a synthetic movie dataset that features a highly
dynamic environment. It contains viewpoints and objects (such as dragons) that
are rare in existing datasets. KITTI provides ground-truth segmentation masks
for the rigid background and moving car instances, where the remaining dynamic
objects (such as pedestrians) are manually removed. For Sintel, computing
rigid instances masks is an ill-posed problem since most objects are nonrigid.
Instead, we obtain ground-truth rigid background segmentation from MR-Flow
[60]. Both datasets also provide ground-truth depth and scene flow as well as
camera intrinsics.

Setup I (monocular): In the monocular setup, depth and scene flow is predicted
from two consecutive frames. We use MiDaS [39], a state-of-the-art monocular
depth estimator to acquire imprecise, up-to-scale depth of the first frame as
inputs. The remaining networks are trained without target domain data: optical
flow and optical expansion networks are trained using FlyingChairs, SceneFlow,
VIPER, and HD1K [14, 28, 34, 41]; the rigid motion segmentation network is
trained on SceneFlow [34] with other networks being fixed. Finally, the
performance of rigid motion segmentation and scene flow is evaluated on KITTI-
SF and Sintel.

Setup II (stereo): Our method is able to take advantage of reliable depth
sensors, such as stereo cameras, to produce better segmentation and scene flow
estimation. Different than the monocular setup, two consecutive stereo pairs
are used as inputs. Specifically, we use GA-Net [69] to acquire metric depth
information from the first stereo pair as inputs to the segmentation network.
To take advantage of accurate depth estimation in the first frame, we refine
the estimated rigid transformations by solving a Perspective-n-Point problem
given first frame depth and flow using non-linear optimization. To estimate
optical flow and optical expansion, we use an off-the-shelf network pre-
trained on KITTI-SF [65]. To train the rigid motion segmentation network, we
mix KITTI-SF and SceneFlow. The performance of stereo scene flow is reported
on the KITTI benchmark.

###  4.1 Two-frame Rigid Motion Segmentation

Metrics: Following prior works, we compute background IoU [32, 40] for rigid
background segmentation and object F-measure [13] for rigid instance
segmentation. Only the rigid background segmentation metric is reported on
Sintel due to the lack of rigid bodies ground-truth rigid motion segmentation
masks.

Baselines: We group baselines according to test inputs.

(1)Single frame methods. Mask R-CNN with ResNeXt-101+FPN backone is the most
accurate model on MSCOCO provided by Detectron2 [21, 30, 59, 62]; U2Net [38]
is a state-of-the-art salient object detector; and MR-Flow-S [60] is a
semantic rigidity estimation network fine-tuned separately on KITTI and
Sintel.

(2)Multi-frame with appearance features. FusionSeg [25] is a two-stream
architecture that fuses the appearance and optical flow features, and we
provide SOTA optical flow on KITTI and Sintel as motion input. COSNet [31] and
MATNet [70] are SOTA video objection segmentation methods on DAVIS [37]. CC
[40] combines “flow-egomotion consensus score” (similar to our epipolar costs)
with a foreground probability regressed from five consecutive frames, which is
then thresholded to obtain the background mask. RTN [32] uses a CNN to predict
rigid background masks given two RGBD images. For Sintel, we use the ground-
truth depth as input; for KITTI, since the ground-truth depth is sparse, we
use MonoDepth2 [19] instead.

(3)Two-frame without appearance. We separately evaluate the motion stream of
FSEG and the flow-egomotion consensus results of CC. Following prior work [5,
60], we implement a classic motion segmentation pipeline that combines the
motion angle and motion residual criteria.

Besides CC, RTN, and the classic pipeline, all baselines are trained or pre-
trained on large-scale manually annotated segmentation datasets that contain
common objects and scenes, while ours is not.

Performance analysis: We show qualitative comparison in Fig. 6 and report
results in Tab. 1. On KITTI, our method outperforms the most accurate
baseline, Mask R-CNN, in terms of both rigid instance segmentation and
background segmentation. Although Mask R-CNN is trained on common scenes
(including driving), it cannot tell whether an object is moving or static,
similar to other single-frame methods. Therefore, our method compares
favorably to Mask R-CNN on rigid motion segmentation task. On Sintel, our
method outperforms all the baselines except MR-Flow-S (S), which uses the
first half of all Sintel sequences for training. If we compare to MR-Flow-S
(K), which is not fine-tuned on Sintel, our method is better. Finally, among
the motion-based segmentation methods, our method is the best on both
datasets, because of our robustness to degenerate motion configurations as
well as noisy flow inputs.

Table 2: Monocular depth and scene flow results on KITTI (K) and Sintel (S). D1: first frame disparity (inverse depth) error. SF: scene flow error (%). The best result is underlined, and further bolded if not trained on the target domain data. On Sintel, we evaluate on 330 frame pairs with average flow magnitude greater than 5 pixel. Method |  K: D1 $\downarrow$ |  K:SF $\downarrow$ |  S: D1 $\downarrow$ |  S:SF $\downarrow$
---|---|---|---|---
CC [40] | 36.20 | 51.80 | ✗ | ✗
SSM [22] | 31.25 | 47.05 | ✗ | ✗
Mono-SF [10] | 16.72 | 21.60 | ✗ | ✗
MiDaS+OE [65] | 37.27 | 44.87 | 49.89 | 55.43
MiDaS+Mask | 17.33 | 22.47 | 39.60 | 47.40
MiDaS+Ours | 16.98 | 22.19 | 38.29 | 46.05
Table 3: Stereo scene flow results on KITTI benchmark. D1 and D2: first and second frame disparity error. Fl: optical flow error. SF: scene flow error. Metrics are errors in percentage and top results are bolded. ∗First frame disparity is not refined by our method. Method |  D1∗ $\downarrow$ |  D2 $\downarrow$ |  Fl $\downarrow$ |  SF $\downarrow$
---|---|---|---|---
PRSM [56] | 4.27 | 6.79 | 6.68 | 8.97
OpticalExp [65] | 1.81 | 4.25 | 6.30 | 8.12
DRISF [33] | 2.55 | 4.04 | 4.73 | 6.31
Ours Mask R-CNN | 1.89 | 3.42 | 4.26 | 5.61
Ours Rigid Mask | 1.89 | 3.23 | 3.50 | 4.89

###  4.2 Rigid Depth and Scene Flow Estimation

The rigid motion masks are then applied to the two-frame depth and scene flow
estimation task under both monocular and stereo setups. In both cases, we fit
a 3D rigid transformation for pixels within each rigid body mask, which is
used to update the depth and scene flow estimation.

Metrics: To evaluate depth and scene flow estimation performance on KITTI and
Sintel, we report disparity error on both frames (D1, D2), optical flow error
(Fl) and scene flow error (SF) following KITTI [35]. In the monocular setup,
to remove the overall scale ambiguity, we take an extra step to align the
overall scale of the predictions to the ground-truth with their medians [39,
57].

Figure 7: Rigidity vs semantic-based segmentation for instance scene flow.
Given instance segmentation masks, scene flow can be optimized by fitting
rigid body transforms within each mask. While semantic segmentation fails to
improve scene flow estimation on the parked cars (in red circle), our rigid
motion mask groups the parked car together with the rigid background and
successfully reduces the scene flow error.

Setup I (monocular): We compare against state-of-the-art monocular scene flow
baselines. CC [40] and SSM [22] are representative methods for self-supervised
monocular depth and scene flow estimation that does not make use of
segmentation priors at inference time. Mono-SF [10] trains a monocular depth
network with KITTI ground-truth, and solve an optimization problem given
semantic instance segmentation provided by Mask R-CNN. The above three methods
are trained on KITTI and the results are taken from their papers. OE (optical
expansion) [65] learns to predict relative depth from dense optical expansion,
which together with optical flow, directly yields 3D motion. It is trained on
the synthetic SceneFlow dataset, and we use MiDaS to provide the scale. We
also implement a baseline (MiDaS+Mask) that predicts instance segmentation
masks by Mask R-CNN, and follows the same rigid body fitting procedure as
ours.

Performance analysis: We report results on KITTI-SF and Sintel in Tab.
LABEL:tab:depth. First, it is noted our method reduces the disparity error of
MiDaS by more than 50% on KITTI, and 20% on Sintel. Compared to OE, which uses
the same monocular depth input as ours, we are better in all metrics. (SF:
22.19% vs 44.87%), which demonstrates the effectiveness of our rigid motion
mask. Our method also outperforms the other methods that do not use
segmentation priors (CC and SSM). Compared to Mono-SF, which is trained with
ground-truth KITTI depth maps, and uses a semantic segmentation prior, our
method is slightly worse on KITTI. Compared to Midas-Mask, our method is
strictly better on both KITTI and Sintel, indicating the benefit of using our
rigid motion masks versus appearance-based masks.

Setup II (stereo): Our method segments rigid motions based on two-frame
rigidity and fits rigid body transformations over depth-flow correspondences,
which is used to update the second frame depth and flow estimations. Among the
baselines, OE [65] uses the same architecture (as in the monocular setup)
fine-tuned on KITTI to upgrade optical flow to 3D scene flow. Same as ours,
GA-Net stereo and VCN optical flow are used as inputs. PRSM [56] segments an
image into superpixels, and fits rigid motions to estimate piece-wise rigid
scene flow. Given semantic instance segmentation [21], depth, and optical
flow, DRISF [33] casts scene flow estimation as an energy minimization problem
and finds the best rigid transformation for each semantic instance. The key
difference between our method and DRISF is that we use rigid motion
segmentation masks.

Performance analysis: As reported in Tab. LABEL:tab:sf-test, our method
demonstrates state-of-the-art performance on KITTI scene flow benchmark (SF:
4.89 vs 6.31). If we replace the segmentation masks with semantic instance
segmentation, i.e., Mask R-CNN, the performance drops noticeably (SF: 4.89% to
5.61%). As illustrated in Fig. 7, our method successfully groups the static
objects (e.g. parked cars) with the rigid background, which effectively
improves scene flow accuracy by optimizing the whole background as one rigid
body, while semantic instance segmentation methods fail to do so.

Table 4: Diagnostics of rigid body motion segmentation on KITTI-SF. Dignostics in the second group are sequential. Method |  K: obj $\uparrow$ |  K: bg $\uparrow$ |  S: bg $\uparrow$
---|---|---|---
Reference | 89.53 | 97.22 | 84.63
(1)w/o cost maps | 88.66 | 96.59 | 76.81
(2)w/o uncertainty | 85.09 | 95.72 | 77.25
(3)w/o monocular depth | 84.46 | 94.84 | 76.14
(4)w/o expansion (MoA [6]) | 81.28 | 95.50 | 77.00
(5)w/o learning [5, 60] | 25.83 | 85.52 | 74.23

###  4.3 Diagnostics

We ablate critical components of our approach and retrain networks. Results
are shown in Tab. LABEL:tab:aba. We validate the design choices of using
(1)explicitly computed rigidity cost-maps inputs, (2)uncertainty estimation
inputs, (3)monocular depth inputs, (4)optical expansion that upgrades 2D
optical flow to 3D, and (5) our rigid motion segmentation network. (1)Removing
rigidity cost-maps leads to a slight drop of accuracy on KITTI, and a
significant drop on Sintel (84.63% to 76.81%). This indicates the cost map
features are crucial for Sintel, possibly due to complex camera and object
motion configurations, in which cases explicit geometric priors are helpful.
(2)Removing uncertainty inputs leads to a noticible drop of performance on
KITTI (88.66% to 85.09%). We posit uncertainty estimation contains rich
information about motion distribution, and is therefore useful for
segmentation. (3)Further removing monocular depth inputs leads to an accuracy
drop on all metrics, especially on KITTI, which shows the importance of using
depth cues to deal with collinear motions in autonomous driving scenes.
(4)After further removing optical expansion, our method degrades to MoA-Net
[6]. The performance drops noticeably on KITTI rigid instance segmentation
metric (84.46% to 81.28%), which indicates optical expansion is useful for
segmenting foreground objects. (5)Lastly, if we directly apply the rigidity
cost maps with manually-tuned thresholds to decide the background region
without the neural architecture and learning, the method becomes worse in all
metrics due to the lost of robustness to noisy inputs and degenerate motion.

##  5 Conclusion

We investigate the problem of two-frame rigid body motion segmentation in an
open environment. We analyze the degenerate cases in geometric motion
segmentation and introduce novel criteria and inputs to resolve such
ambiguities. We further propose a modular neural architecture that is robust
to noisy observations as well as different motion types, which demonstrates
state-of-the-art performance on rigid motion segmentation, depth and scene
flow estimation tasks.

Acknowledgements: This work was supported by the CMU Argo AI Center for
Autonomous Vehicle Research.

## References

  * [1] Gerald J Agin and Thomas O Binford.  Computer description of curved objects.  IEEE Transactions on Computers, (4):439–449, 1976.
  * [2] Ioan Andrei Bârsan, Peidong Liu, Marc Pollefeys, and Andreas Geiger.  Robust dense mapping for large-scale dynamic environments.  In ICRA, 2018.
  * [3] Aseem Behl, Omid Hosseini Jafari, Siva Karthik Mustikovela, Hassan Abu Alhaija, Carsten Rother, and Andreas Geiger.  Bounding boxes, segmentations and object coordinates: How important is recognition for 3d scene flow estimation in autonomous driving scenarios?  In ICCV, 2017.
  * [4] Abhijit Bendale and Terrance Boult.  Towards open world recognition.  In CVPR, 2015.
  * [5] Pia Bideau and Erik Learned-Miller.  It’s moving! a probabilistic model for causal motion segmentation in moving camera videos.  In ECCV, 2016.
  * [6] Pia Bideau, Rakesh R Menon, and Erik Learned-Miller.  Moa-net: self-supervised motion segmentation.  In ECCVW, 2018.
  * [7] Pia Bideau, Aruni RoyChowdhury, Rakesh R Menon, and Erik Learned-Miller.  The best of both worlds: Combining cnns and geometric constraints for hierarchichal motion segmentation.  In CVPR, 2018.
  * [8] Irving Biederman.  Geon theory as an account of shape recognition in mind and brain.  The Irish Journal of Psychology, 14(3):314–327, 1993.
  * [9] Eric Brachmann and Carsten Rother.  Neural- Guided RANSAC: Learning where to sample model hypotheses.  In ICCV, 2019.
  * [10] Fabian Brickwedde, Steffen Abraham, and Rudolf Mester.  Mono-SF: Multi-view geometry meets single-view depth for monocular scene flow estimation of dynamic traffic scenes.  In CVPR, 2019.
  * [11] Daniel J Butler, Jonas Wulff, Garrett B Stanley, and Michael J Black.  A naturalistic open source movie for optical flow evaluation.  In ECCV, 2012.
  * [12] Qi Cai, Yuanxin Wu, Lilian Zhang, and Peike Zhang.  Equivalent constraints for two-view geometry: pose solution/pure rotation identification and 3d reconstruction.  IJCV, 2019.
  * [13] Achal Dave, Pavel Tokmakov, and Deva Ramanan.  Towards segmenting everything that moves.  In ICCVW, 2019.
  * [14] A. Dosovitskiy, P. Fischer, E. Ilg, P. Häusser, C. Hazırbaş, V. Golkov, P. v.d. Smagt, D. Cremers, and T. Brox.  Flownet: Learning optical flow with convolutional networks.  In ICCV, 2015.
  * [15] Elan Dubrofsky.  Homography estimation.  Diplomová práce. Vancouver: Univerzita Britské Kolumbie, 2009.
  * [16] Hugh Durrant-Whyte and Tim Bailey.  Simultaneous localization and mapping: part i.  IEEE robotics & automation magazine, 13(2):99–110, 2006.
  * [17] Martin A Fischler and Robert C Bolles.  Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography.  Communications of the ACM, 24(6):381–395, 1981.
  * [18] Andreas Geiger, Philip Lenz, and Raquel Urtasun.  Are we ready for autonomous driving? the kitti vision benchmark suite.  In CVPR, 2012.
  * [19] Clément Godard, Oisin Mac Aodha, Michael Firman, and Gabriel J. Brostow.  Digging into self-supervised monocular depth prediction.  In ICCV, 2019.
  * [20] Richard Hartley and Andrew Zisserman.  Multiple view geometry in computer vision.  Cambridge university press, 2003.
  * [21] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick.  Mask r-cnn.  In ICCV, 2017.
  * [22] Junhwa Hur and Stefan Roth.  Self-supervised monocular scene flow estimation.  In CVPR, 2020.
  * [23] Eddy Ilg, Ozgun Cicek, Silvio Galesso, Aaron Klein, Osama Makansi, Frank Hutter, and Thomas Brox.  Uncertainty estimates and multi-hypotheses networks for optical flow.  In ECCV, 2018.
  * [24] Michal Irani and P Anandan.  A unified approach to moving object detection in 2d and 3d scenes.  PAMI, 1998.
  * [25] Suyog Dutt Jain, Bo Xiong, and Kristen Grauman.  Fusionseg: Learning to combine motion and appearance for fully automatic segmentation of generic objects in videos.  In CVPR, 2017.
  * [26] Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  arXiv preprint arXiv:1412.6980, 2014.
  * [27] Alexander Kirillov, Yuxin Wu, Kaiming He, and Ross Girshick.  PointRend: Image segmentation as rendering.  2019\.
  * [28] Daniel Kondermann, Rahul Nair, Katrin Honauer, Karsten Krispin, Jonas Andrulis, Alexander Brock, Burkhard Gussefeld, Mohsen Rahimimoghaddam, Sabine Hofmann, Claus Brenner, et al.  The hci benchmark suite: Stereo and flow ground truth with uncertainties for urban autonomous driving.  In CVPRW, 2016.
  * [29] Shih-Chieh Lin, Yunqi Zhang, Chang-Hong Hsu, Matt Skach, Md E Haque, Lingjia Tang, and Jason Mars.  The architectural implications of autonomous driving: Constraints and acceleration.  In Proceedings of the Twenty-Third International Conference on Architectural Support for Programming Languages and Operating Systems, pages 751–766, 2018.
  * [30] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick.  Microsoft coco: Common objects in context.  In ECCV. Springer, 2014.
  * [31] Xiankai Lu, Wenguan Wang, Chao Ma, Jianbing Shen, Ling Shao, and Fatih Porikli.  See more, know more: Unsupervised video object segmentation with co-attention siamese networks.  In CVPR, 2019.
  * [32] Zhaoyang Lv, Kihwan Kim, Alejandro Troccoli, Deqing Sun, James Rehg, and Jan Kautz.  Learning rigidity in dynamic scenes with a moving camera for 3d motion field estimation.  In ECCV, 2018.
  * [33] Wei-Chiu Ma, Shenlong Wang, Rui Hu, Yuwen Xiong, and Raquel Urtasun.  Deep rigid instance scene flow.  In CVPR, 2019.
  * [34] N. Mayer, E. Ilg, P. Häusser, P. Fischer, D. Cremers, A. Dosovitskiy, and T. Brox.  A large dataset to train convolutional networks for disparity, optical flow, and scene flow estimation.  In CVPR, 2016.
  * [35] Moritz Menze and Andreas Geiger.  Object scene flow for autonomous vehicles.  In CVPR, 2015.
  * [36] Peter Ochs, Jitendra Malik, and Thomas Brox.  Segmentation of moving objects by long term video analysis.  PAMI, 2013.
  * [37] F. Perazzi, J. Pont-Tuset, B. McWilliams, L. Van Gool, M. Gross, and A. Sorkine-Hornung.  A benchmark dataset and evaluation methodology for video object segmentation.  In CVPR, 2016.
  * [38] Xuebin Qin, Zichen Zhang, Chenyang Huang, Masood Dehghan, Osmar Zaiane, and Martin Jagersand.  U2-net: Going deeper with nested u-structure for salient object detection.  Pattern Recognition, 2020.
  * [39] René Ranftl, Katrin Lasinger, David Hafner, Konrad Schindler, and Vladlen Koltun.  Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer.  arXiv:1907.01341, 2019.
  * [40] Anurag Ranjan, Varun Jampani, Lukas Balles, Kihwan Kim, Deqing Sun, Jonas Wulff, and Michael J Black.  Competitive collaboration: Joint unsupervised learning of depth, camera motion, optical flow and motion segmentation.  In CVPR, 2019.
  * [41] Stephan R Richter, Zeeshan Hayder, and Vladlen Koltun.  Playing for benchmarks.  In ICCV, pages 2213–2222, 2017.
  * [42] Olaf Ronneberger, Philipp Fischer, and Thomas Brox.  U-net: Convolutional networks for biomedical image segmentation.  In International Conference on Medical image computing and computer-assisted intervention, pages 234–241. Springer, 2015.
  * [43] Peter J Rousseeuw.  Least median of squares regression.  Journal of the American statistical association, 79(388):871–880, 1984.
  * [44] Harpreet S Sawhney.  3d geometry from planar parallax.  In CVPR, 1994.
  * [45] Jianbo Shi and Jitendra Malik.  Motion segmentation and tracking using normalized cuts.  In ICCV, 1998.
  * [46] Deqing Sun, Stefan Roth, and Michael J Black.  Secrets of optical flow estimation and their principles.  In CVPR, 2010.
  * [47] Pavel Tokmakov, Karteek Alahari, and Cordelia Schmid.  Learning motion patterns in videos.  In CVPR, 2017.
  * [48] Pavel Tokmakov, Cordelia Schmid, and Karteek Alahari.  Learning to segment moving objects.  IJCV, 2019.
  * [49] Philip HS Torr.  Geometric motion segmentation and model selection.  Philosophical Transactions of the Royal Society of London. Series A: Mathematical, Physical and Engineering Sciences, 356(1740):1321–1340, 1998.
  * [50] Philip HS Torr, Andrew W Fitzgibbon, and Andrew Zisserman.  The problem of degeneracy in structure and motion recovery from uncalibrated image sequences.  IJCV, 1999.
  * [51] Philip HS Torr, Andrew Zisserman, and Stephen J Maybank.  Robust detection of degenerate configurations while estimating the fundamental matrix.  CVIU, 1998.
  * [52] Roberto Tron and René Vidal.  A benchmark for the comparison of 3-d motion segmentation algorithms.  In CVPR, 2007.
  * [53] Sundar Vedula, Simon Baker, Peter Rander, Robert Collins, and Takeo Kanade.  Three-dimensional scene flow.  In ICCV, 1999.
  * [54] René Vidal, Yi Ma, Stefano Soatto, and Shankar Sastry.  Two-view multibody structure from motion.  IJCV, 2006.
  * [55] René Vidal and Shankar Sastry.  Optimal segmentation of dynamic scenes from two perspective views.  In CVPR, 2003.
  * [56] Christoph Vogel, Konrad Schindler, and Stefan Roth.  3D scene flow estimation with a piecewise rigid scene model.  IJCV, 2015.
  * [57] Chaoyang Wang, José Miguel Buenaposada, Rui Zhu, and Simon Lucey.  Learning depth from monocular videos using direct methods.  In CVPR, pages 2022–2030, 2018.
  * [58] Joseph Weber and Jitendra Malik.  Rigid body segmentation and shape description from dense optical flow under weak perspective.  TPAMI, 19(2):139–143, 1997.
  * [59] Yuxin Wu, Alexander Kirillov, Francisco Massa, Wan-Yen Lo, and Ross Girshick.  Detectron2.  https://github.com/facebookresearch/detectron2, 2019.
  * [60] Jonas Wulff, Laura Sevilla-Lara, and Michael J. Black.  Optical flow in mostly rigid scenes.  In CVPR, 2017.
  * [61] Enze Xie, Peize Sun, Xiaoge Song, Wenhai Wang, Xuebo Liu, Ding Liang, Chunhua Shen, and Ping Luo.  Polarmask: Single shot instance segmentation with polar representation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12193–12202, 2020.
  * [62] Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He.  Aggregated residual transformations for deep neural networks.  In CVPR, 2017.
  * [63] Xun Xu, Loong Fah Cheong, and Zhuwen Li.  3d rigid motion segmentation with mixed and unknown number of models.  PAMI, 2019.
  * [64] Gengshan Yang and Deva Ramanan.  Volumetric correspondence networks for optical flow.  In NeurIPS, 2019.
  * [65] Gengshan Yang and Deva Ramanan.  Upgrading optical flow to 3d scene flow through optical expansion.  In CVPR, 2020.
  * [66] Zhao Yang, Qiang Wang, Luca Bertinetto, Weiming Hu, Song Bai, and Philip HS Torr.  Anchor diffusion for unsupervised video object segmentation.  In CVPR, 2019.
  * [67] Fisher Yu, Dequan Wang, Evan Shelhamer, and Trevor Darrell.  Deep layer aggregation.  In CVPR, pages 2403–2412, 2018.
  * [68] Chang Yuan, Gerard Medioni, Jinman Kang, and Isaac Cohen.  Detecting motion regions in the presence of a strong parallax from a moving camera by multiview geometric constraints.  PAMI, 2007.
  * [69] Feihu Zhang, Victor Prisacariu, Ruigang Yang, and Philip HS Torr.  GA-Net: Guided aggregation net for end-to-end stereo matching.  In CVPR, 2019.
  * [70] Tianfei Zhou, Shunzhou Wang, Yi Zhou, Yazhou Yao, Jianwu Li, and Ling Shao.  Motion-attentive transition for zero-shot video object segmentation.  In AAAI, 2020.
  * [71] Xingyi Zhou, Dequan Wang, and Philipp Krähenbühl.  Objects as points.  In arXiv preprint arXiv:1904.07850, 2019.

##  6 Appendix

###  6.1 Rigidity cost maps

In Sec. 3.2, we briefly motivated the design choice of using rigidity cost-
maps inputs, and we expand the particular cost functions here. Given motion
correspondences $({\bf p_{0}},{\bf p_{1}})\in R^{2}$, camera intrinsics $({\bf
K_{0}},{\bf K_{1}})$, and camera motion ${\bf R_{c}}\in SO(3)$, ${\bf
T_{c}}\in R^{3}$, we construct four geometric motion cost maps that are
tailored to particular motion configurations, including 1) an epipolar cost,
2) a homography cost, 3) a 3D P+P cost, and 4) a depth contrast cost.

1) Epipolar costs are applied to detect general moving objects, computed as
the classic Sampson error [20] per-pixel. We include it here for completeness:

$c_{\text{epi}}=\frac{({\bf\tilde{p}_{1}}^{T}{\bf
F}{\bf\tilde{p}_{0}})^{2}}{({\bf F}{\bf\tilde{p}_{0}})_{1}^{2}+({\bf
F}{\bf\tilde{p}_{0}})_{2}^{2}+({\bf F}^{T}{\bf\tilde{p}_{1}})_{1}^{2}+({\bf
F}^{T}{\bf\tilde{p}_{1}})_{2}^{2}+\epsilon},$ (8)

where ${\bf F}={\bf K_{1}}^{-T}{\bf R}[{\bf t}]_{\times}{\bf K_{0}}^{-1}$ is
the fundamental matrix and $({\bf\tilde{p}_{0}},{\bf\tilde{p}_{1}})$ are
motion correspondences in the homogeneous coordinates. $\epsilon=10^{-9}$ is a
constant value added for numerical stability.

2) Homography costs are applied to deal with motion degeneracies in epipolar
geometry [50], when it becomes difficult to estimate camera translation, but
not rotation [12]. A visual comparison between epipolar costs and homography
costs can be found in Fig. 8. The homography cost is implemented as per-pixel
symmetric transfer error [15] with regard to the rotational homography, ${\bf
H_{R}}={\bf K_{0}}{\bf R_{c}}{\bf K_{1}}^{-1}$,

$c_{\text{hom}}=d({\bf\tilde{p}_{0}},{\bf
H_{R}}{\bf\tilde{p}_{1}})^{2}+d({\bf\tilde{p}_{1}},{\bf
H_{R}}^{-1}{\bf\tilde{p}_{0}})^{2},$ (9)

where $d(\cdot,\cdot)$ is the Euclidian image distance between two points.

3) 3D P+P costs are applied to detect the coplanar motion, where points are
moving along the epipolar line (not detectable by the epipolar costs, as
analyzed in Sec. 3.1. Our 3D P+P cost is extended from the 2D residual error
of [5],

$c_{\text{3D}}=||{\bf\tilde{T}_{sf}}||\cdot|\sin\beta|,$ (10)

where $\beta=|\angle({\bf\tilde{T}_{sf}},-{\bf T_{c}})|$ is the measured angle
between the normalized scene flow ${\bf\tilde{T}_{sf}}$ (as computed through
optical expansion using the method of [65]) and negative camera translation
${\bf-T_{c}}$, capped to $\frac{\pi}{2}$. A visual comparison is shown in Fig.
9.

4) Depth contrast costs are applied to address the colinear motion ambiguity,
where points are moving opposite to the camera translation direction in 3D,
and therefore not detectable by the above costs, as shown in Fig. 10. The
depth contrast cost is implemented as:

$c_{\text{depth}}=|\log(\frac{Z^{\text{flow}}}{\gamma Z^{\text{prior}}})|,$
(11)

where the flow-triangulated depth $Z^{flow}_{0}$ can be computed efficiently
using midpoint or DLT triangulation algorithm [20], the monocular depth prior
$Z^{prior}_{0}$ can be represented by a data-driven monocular depth network
[19], and the scale factor $\gamma$ that globally aligns $Z^{prior}_{0}$ to
$Z^{flow}_{0}$ can be determined by robust least squares [46]. A visual
comparison between the flow-triangulated depth and monocular depth prior is
shown in Fig. 11.

###  6.2 Training details

The details for training optical flow, optical expansion and rigid motion
segmentaion networks are shown in Tab. LABEL:tab:hp.

Table 5: Details for network training. C: FlythingChairs [14]. T: FlythingThings [34]. SF: SceneFlow [34]. V: VIPER [41]. The optical flow network is trained sequentially on C, T, and C+SF+V. Parameter | Value
---|---
Optical flow
Network architecture |  VCN [64]
Optimizer |  Adam [26]
Learning rate | $1\times 10^{-3}$
Batch size / iterations on C | 16 image pairs / 70k
Batch size / iterations on T | 16 image pairs / 70k
Batch size / iterations on C+SF+V | 12 image pairs / 70k
Optical expansion
Network backbone |  U-Net [42, 65]
Optimizer |  Adam [26]
Learning rate | $1\times 10^{-3}$
Batch size / iterations on SF | 12 image pairs / 70k
Rigid motion segmentation
Network backbone |  U-Net+DLA-34 [42, 65, 67, 71]
Optimizer |  Adam [26]
Learning rate | $5\times 10^{-4}$
Batch size / iterations on SF | 12 image pairs / 70k

###  6.3 Details of rigid body scene flow

In Sec. 3.2, we describe rigid body scene flow that (1) fits 3D rigid motions
per rigid body, and (2) updates depth as well as flow measurements. More
details are provided here.

Overalll, our goal is to select _high-quality flow correspondences_ for model
fitting, and update the rigid bodies with _large enough motion_. To do so, we
first define “valid pixels” as pixels with flow confidence (in range 0-1,
estimated by VCN [64]) greater than 0.5. During fitting, we use flow
correspondences of valid pixels from each rigid motion mask to fit a essential
matrix through a least median of squares estimator [43]. Then, each essential
matrix is decomposed to four rotations and up-to-scale translations, where
only one is feasible through cheirality check [20]. To determine the scale of
translation, we triangulate flow correspondences at vailid pixels and align it
with the initial depth input by a scale factor through RANSAC [17]. To take
advantage of accurate depth estimation in the stereo case, we refine the
estimated rigid transformations by solving a Perspective-n-Point problem given
first frame depth and flow that minimizes re-projection errors with
Levenberg–Marquardt algorithm [20].

Finally, we update depth and flow estimations according to the estimated 3D
rigid motions. Rigid bodies whose average parallax flow magnitude (defined as
“rectified” optical flow after rotation removal in Sec.3.1) is lower than 4px,
or has fewer than 30% valid pixels are not updated.

Table 6: Ablation study of stereo scene flow on KITTI-SF images. D1 and D2: first and second frame disparity error. Fl: optical flow error. all: evaluated on all pixels. fg: evaluated on foreground pixels only. SF: scene flow error. $\Delta$: percentage of error reduction after refinement. ∗First frame disparity does not change during refinement. |  ∗D1 (%) | D2 (%) | Fl (%) | SF (%)
---|---|---|---|---
Method |  _all_ $\downarrow$ |  _fg_ $\downarrow$ |  _all_ $\downarrow$ |  _fg_ $\downarrow$ |  _all_ $\downarrow$ |  $\Delta$-all $\uparrow$ |  _fg_ $\downarrow$ |  $\Delta$-fg $\uparrow$ |  _all_ $\downarrow$ |  $\Delta$-all $\uparrow$ |  _fg_ $\downarrow$ |  $\Delta$-fg $\uparrow$
Baseline OE [65] | 1.41 | 0.76 | 2.45 | 0.91 | 4.02 | 0 | 2.50 | 0 | 5.12 | 0 | 3.07 | 0
Ours Mask R-CNN | 1.41 | 0.76 | 2.11 | 1.99 | 3.53 | 12.1 | 4.34 | -73.6 | 4.02 | 21.5 | 4.86 | -36.8
Ours Rigid Mask | 1.41 | 0.76 | 2.04 | 1.05 | 3.32 | 17.4 | 2.16 | 15.7 | 3.86 | 24.6 | 2.78 | 10.4

###  6.4 Ablation study of rigid body scene flow

We study the effect of rigid motion parameterization for scene flow estimation
and report results on 200 images of KITTI-SF as shown in Tab. LABEL:tab:sf-
aba. Without rigid motion parameterization, our method is equivalent to
optical expansion [65], which upgrades 2D flow fields to 3D, but does not
refine the first frame disparity as well as optical flow. In contrast, the
proposed method reduces the overall scene flow error by 24.6% through rigid
body refinement. Replacing the proposed rigid motion masks with appearance-
based masks produced by Mask R-CNN leads to a noticible accuracy drop. Our
rigid body parameterization also lead to constant improvement of scene flow
accuracy for both foreground and background regions.

###  6.5 Qualitative comparison

We provide additional visual comparison with prior approaches on KITTI and
Sintel in Fig. 12 and Fig. 13. Compared to appearance-based methods for
segmenting rigid motions, our method is able to correctly segment the static
objects as part of the rigid background, and generalizes to novel appearance.
Compared to geometric motion segmentation methods, our method is more robust
to degenerate motion configurations and noisy flow as well as camera inputs.

Figure 8: Epipolar costs vs homography costs. Top: Gray-scale costs values.
Bottom: Binary segmentations after thresholding the costs. This scene features
two moving foreground cars and a static camera that causes motion degeneracy
in epipolar geometry (e.g., the low-cost but moving region in the Sampson cost
map, marked by the red circle). In such cases, epipolar line is not well-
defined, and the homography model is more suitable for motion segmentation.
Figure 9: 3D P+P costs. Top: Gray-scale costs. Bottom: Binary segmentations
after thresholding the costs. This scene contains a moving camera and nonrigid
dynamic objects, where pixels that move along the epipolar line are not
recoverable under classic motion segmentation criteria (e.g., the low-cost but
moving region in the Sampson cost map, marked by the red circle). In such
cases, our 3D P+P cost is more suitable. Figure 10: Depth contrast costs. Top:
Gray-scale costs. Bottom: Binary segmentations after thresholding the costs.
This scene contains a moving camera and a rigid body (car) moving along the
negative direction of camera translation, which is not recoverable under
classic motion segmentation criteria (e.g., the low-cost but moving region in
the Sampson cost map, marked by the red circle) as well as the 3D P+P cost. In
such cases, our depth contrast cost is more suitable. Figure 11: Flow-
triangulated depth vs monocular depth prior. The flow-triangulated depth
$Z^{flow}_{0}$ (middle) is the triangulation of motion correspondences
assuming overall rigidity. The monocular depth prior $Z^{prior}_{0}$ (right)
can be represented by a data-driven monocular depth network [39]. In this
example, the left vehicle is moving opposite to the camera translation
direction, and cannot be detected by epipolar constraints, as shown in Fig. 2
of the main text. However, it appears abnormal (floating above the ground) in
the flow-triangulated reconstruction. To detect such collinearly moving
objects, we globally align $Z^{prior}_{0}$ to $Z^{flow}_{0}$ by a scale factor
$\gamma$ with robust least squares [46], which reveals the floating (moving)
car that is inconsistent with the monocular depth prior. Figure 12:
Illustration of coplanar motion ambiguity on KITTI. Rigid background are
indicated by the white color. Points moving along the epipolar line, for
example the roof of the cars, yields small Sampson error, and therefore are
estimated as background in classic geometric pipelines. We make use of optical
expansion, which reveals the relative depth change, to resolve such ambiguity.
Compared to prior motion-based segmentation method, ours is more robust to
noise. Figure 13: Results on Sintel market sequence. Prior single frame or
video motion segmentation methods fail due to unusual view-point (viewing from
the ground) and never-before-seen objects (dragon, wood carts). Our method
accurately segments novel moving objects.