Erez<EMAIL_ADDRESS>Rami<EMAIL_ADDRESS>Ben-Gurion
University
of the Negev Ben-Gurion University
of the Negev

# Motion Segmentation Using Locally
Affine Atom Voting

###### Abstract

We present a novel method for motion segmentation called LAAV (Locally Affine
Atom Voting). Our model’s main novelty is using sets of features to segment
motion for all features in the scene. LAAV acts as a pre-processing pipeline
stage for features in the image, followed by a fine-tuned version of the
state-of-the-art Random Voting (RV) method. Unlike standard approaches, LAAV
segments motion using feature-set affinities instead of pair-wise affinities
between all features; therefore, it significantly simplifies complex scenarios
and reduces the computational cost without a loss of accuracy. We describe how
the challenges encountered by using previously suggested approaches are
addressed using our model. We then compare our algorithm with several state-
of-the-art methods. Experiments shows that our approach achieves the most
accurate motion segmentation results and, in the presence of measurement
noise, achieves comparable results to the other algorithms.

##  1 Introduction

A high-level problem in computer vision is to differentiate motions of bodies
in a video sequence. It is an essential building block for purposes such as
traffic monitoring, moving object segmentation and motion analysis. In
general, motion can be considered to be a transformation of an object in space
and time. This transformation could be represented using epipolar geometry as
a projective transformation between two views [Zhang(1996)] or any other
parametric representation. In recent years, many algorithms have been
developed to segment motion, they can be divided into two main groups -
Multiview-based and Two-view-based.

The Input for the Multiview-based approaches are Point trajectories [Rao et
al.(2008)Rao, Tron, Vidal, and Ma, Costeira and Kanade(1998), Shi and
Malik(2000), Vidal et al.(2005)Vidal, Ma, and Sastry, Flores-Mangas and
Jepson(2013), Liu and Yan(2011), Keuper et al.(2015)Keuper, Andres, and Brox,
Zappella et al.(2010)Zappella, Provenzi, Lladó, and Salvi, Elhamifar and
Vidal(2012)]. Point trajectories are subsets consisting of selected feature
points that are tracked along a sequence. Point trajectories, can be
considered as a representation of motion of objects moving in the scene. In
order to obtain a segmentation result, Multiview-based methods primarily use
local information around each trajectory in order to create affinities between
trajectories, which can subsequently be segmented using spectral clustering
techniques [Shi and Malik(2000)]. Multiview-based methods generally work well
and are robust to noise because they handle the entire sequence at once. These
methods usually compute pairwise affinities between all detected features over
all frames. Consequently, complicated scenarios are accompanied by a high
computational cost.

The alternate fundamental approach for motion segmentation are Two-view-based
methods. These methods [Li(2007), Poling and Lerman(2013), Vidal et
al.(2002)Vidal, Soatto, Ma, and Sastry, Jian and Chen(2010), Jung et
al.(2014)Jung, Ju, and Kim, Torr(1998)] ], usually focus on the epipolar
constraint. Although these approaches are efficient, they only process a small
part of the information available [Li(2007), Poling and Lerman(2013), Vidal et
al.(2002)Vidal, Soatto, Ma, and Sastry, Jian and Chen(2010), Jung et
al.(2014)Jung, Ju, and Kim, Torr(1998)]. For example, an object that had moved
slowly or did not move significantly between two frames, would not be differ
easily from the background. A further constraint on these methods is the use
of random initialization. In practice, this means that convergence is not
guaranteed every time and usually its labeling result is not-consistent.

In this paper, we propose an initialization stage for the state-of-the-art
algorithm presented at CVPR called Random Voting (RV) [Jung et al.(2014)Jung,
Ju, and Kim]. We focus on extending its performance to deal with the following
issues:

$\bullet$

Expedite convergence - as RV is an iterative algorithm, performance depends on
the number of iterations for convergence 2

$\bullet$

Minimize accuracy distribution rates - even though accuracy can be high, it
can vary as random initialization by RV [Jung et al.(2014)Jung, Ju, and Kim]
yields non-consistent accuracy rates

$\bullet$

Improve accuracy with noise-free data and increase robustness to noise

By employing an algorithm based on RV [Jung et al.(2014)Jung, Ju, and Kim] and
focusing on the above issues, we have developed the Locally Affine Atom Voting
(LAAV) method. The LAAV is a fast and accurate motion segmentation algorithm
that incorporates two stages. The first stage is a pre-processing stage based
on a novel representation of objects called Piecewise-Smooth. The pre-
processing stage provides a hard-assignment for motion features, followed by a
pair-wise fine-tuning using RV in the segmentation stage. As a result of the
initialization stage, the RV algorithm becomes more robust to complex
scenarios, is more accurate and less variant. Our algorithm’s robustness to
noise is comparable to other state-of-the-art algorithms.

The paper is organized as follows: Section 2 discusses relevant past work. In
section 3 we present LAAV method and discuss its flow and components. Section
4 describes experiments done: comparison of performance and an extensive
evaluation of LAAV in conjunction with leading state-of-the-art RV. We
conclude our findings in section 5.

##  2 Related Work

As described above, there are two major groups of approaches to motion
segmentation: Multiview and Two-view.

The Two-view approach is derived merely from the relative camera poses from
multiple views, called relative-pose constraints, without any additional
assumptions of the scene. The epipolar constraint is such a constraint between
two views [Luong and Faugeras(1996)].

Random Voting (RV) [Jung et al.(2014)Jung, Ju, and Kim], which is considered
as the leading geometric method for motion segmentation partly because of its
robustness to noise, has shown particularly successful results with a low
computational cost. The algorithm, based on epipolar geometry, is an iterative
process of randomized feature selection between two frames, estimating a
fundamental matrix from the selected features and vote scores for the rest of
the remaining features to be associated with a certain motion model. Since the
method uses random initialization, it never loses any information even when
the selected features do not represent a motion model. However, this approach
only works well when the independent moving object is big enough, such that it
consists of enough features to properly estimate the object’s motion. In
addition, objects in the scene need to be in a certain size so that the
background/object features ratio is not too high in order for the object’s
features to be selected in the randomized features selection. Finally, its
accuracy rate results can vary due to the random initialization.

The Multiview approach utilizes the trajectory of the feature points. PAC
[Zappella et al.(2010)Zappella, Provenzi, Lladó, and Salvi] and SSC [Elhamifar
and Vidal(2012)] methods have quite accurate results in multiple motion cases
in a sequence and are also robust to noise. However, those algorithms are
extremely slow. Latent low-rank representation-based method (LatLRR) [Liu and
Yan(2011)] is faster and more accurate, but this method becomes degraded in
extremely noisy environments. The ICLM-based approach [Flores-Mangas and
Jepson(2013)] is very fast, but has lower accuracy than other state-of-the-art
approaches. In addition, while Multiview approaches are more accurate than
Two-view approaches, they do not have good performance when there are only a
few frames.

|  |
---|---|---
(a) | (b) | (c)
|  |
(d) | (e) | (f)
Figure 1: Overall flow of the proposed method. (a) The Video obtained from
[Tron and Vidal(2007)] with its feature points detected and tracked along the
sequence. (b) Visualization of atoms structure obtained using object’s piece-
wise smooth representation, all feature confined by the red circle would get
joined to an atom. (c) shows segmentation results based on atoms trajectories
as a minimum cost multicut problem. (d) shows LAAV output result with over
segmented independent objects. (e) Fine-To-Coarse output stage; groups of the
same moving object are joined for an object motion segmentation. (f) Final
segmentation after a moderate Random Voting(RV) to segment features that had
not been joined to an atom.

##  3 Motion Segmentation Using LAAV

In this section, we discuss the structure of the proposed algorithm comprised
from blocks presented in Figure 1.

Using the proposed method, all tracked 2D feature points along a video
sequence are segmented into either background or independent moving objects.
We denote a 2D feature point in a single frame $l$ as $y_{k}^{l}$ where
$k=1,.,.,K$, with $K$ as the total number of features and $l=1,…,L$, with $L$
as the total number of frames. $y_{k}^{l}$ is represented in homogeneous
coordinates.

Objects Piecewise-Smooth Representation \- We present a novel representation
of an object’s motion which relies on the fact that object regions are often
piecewise-smooth, i.e. can be represented as a linked overlapped patches
(referred to as atoms) with gradually differing affine transformations. By
explicitly incorporating the unique object’s piecewise-smooth quality, atoms
are robustly estimated from the matched feature points. Unlike standard
approaches for motion segmentation, the presented algorithm uses atoms instead
of pairwise affinities between all features; thus, significantly simplifies
complex scenarios and reduces the computational cost without the loss of
accuracy. Using this representation, we can estimate a fine [Krause et
al.(2013)Krause, Stark, Deng, and Fei-Fei] segmentation of the motion models
in the scene. i.e. non-rigid bodies could hold multiple motion models.

The LAAV input is a set of all feature points that are tracked along the
sequence and their trajectories.

In order to simplify the complexity of the scenario, neighbored feature
trajectories sharing similar affine transformations are joined to atoms
denoted as $x_{i}$ where $i=1,...,N$ and $N$ is the total number of atoms. The
set $X={x_{1},x_{x},...,x_{N}}$ consists of all atoms in a sequence. Figure 1
(b) visualizes the constructed atoms in their designed structure, all features
inside the red circle belong to an atom and could be joined to other atoms
while features inside the green circle are only belong to a single atom.

Fine 2D motion estimation  \- Non-rigid bodies can consist of multiple
independent parts moving in different directions. Therefore, the number of
motion models in every sequence could be considered as bigger than the number
of the independent moving objects. In this stage, each atom along a video
sequence is classified into either background or independent motion, resulting
in a fine-grained [Krause et al.(2013)Krause, Stark, Deng, and Fei-Fei]
description of all the independent moving objects in the sequence. For that
purpose we formulate the segmentation of a video sequence based on atom
trajectories as a minimum cost multicut problem [Keuper et al.(2015)Keuper,
Andres, and Brox], as shown in Figure 1 (c). Next, the set of atoms
$X={x_{1},x_{x},...,x_{N}}$ are segmented to an over-estimate of motion models
when given the true number of independent moving objects in the sequence,
denoted as C. We found empirically that the number $2C$ of motion models is
suitable for over-estimation of all motion models under the Hopkins155 dataset
[Tron and Vidal(2007)] which was used to evaluate our algorithm.

The LAAV result is the set $X$ and is a fine-grained description of all
independent moving objects in the sequence. Each atom $x_{i}$ in the set $X$
is being labeled as $S_{c}$ where $c = 1,...,2C$. i.e. $S_{c}$ express motion
models labels or the set of feature trajectories belonging to the $c^{th}$
motion model. Figure 1 (d) shows the segmentation results of LAAV.

Fine-To-Coarse 2D motion estimation  \- In order to reconstruct independent
motions models to independent moving objects the next stage is applied. Here,
all independent motions are joined in order to describe motion in a coarse-
grained [Krause et al.(2013)Krause, Stark, Deng, and Fei-Fei] mode. i.e. to
assemble independent moving objects from all motion models. The $2C$ motion
models from previous stage are joined down to $C$ motion models. Figure 1 (e)
shows the segmentation results of the Fine-To-Coarse procedure resulting with
the set $X$ being labeled as $S_{c}$.

Fine-tuning Randomized Voting \- Finally, this stage utilizes a moderate
version of RV [Jung et al.(2014)Jung, Ju, and Kim] in order to label features
that had not yet been joined to atoms marked in black in Figure 1(a)-(e), and
were not segmented to any independent moving object. Figure 1 (f) shows the
final motion segmentation results.

###  3.1 Atom Construction

Perspective projection is often used in order to model motion of rigid objects
between pairs of frames. However, alternative geometric relationships that
facilitate parameter computation have also been proven useful for this purpose
[Morel and Yu(2009)]. For instance, in small patches within images the
perspective transformation can effectively be approximated by an affine
transformation [Morel and Yu(2009)]. Furthermore, examining small patches of
non-rigid bodies can be worthwhile such that any non-rigid body can be
represented as a combination of multiple small rigid bodies.

Embracing this representation, we can define the term piecewise-smooth object.
Piecewise-smooth objects are non-rigid bodies where the temporal
transformation between two frames are often slow in small regions. On
piecewise-smooth objects, the affine transformation for neighboring patches is
similar. Hence, a group of small patches where all features correspond to the
same affine transformation can be grouped and denote as atoms.

We define atoms as feature sets that comply with the following conditions:

Condition 1 for every atom $x_{i}\in X$ the minimum number of features should
be no less than 3

Affine transformation has 6 degrees of freedom; translation + rotation + scale
+ aspect ratio + shear, so in order to estimate an affine transformation,
atoms should consist of a minimal 3 features. The atoms construction is as
follows: atoms are constructed sequentially. Iteratively, two frames $l,r$ are
selected randomly, with enough frame separation so that a minimum movement can
be detected. Then, a feature is randomly selected and a minimum of 3 neighbors
are located.

Condition 2 each atom $x_{i}$ should be comprised out of a set of features
$Y$, bounded by the external radius $R_{2}$

The atom’s bounding area is imperative so that the affine transformation
approximation is maintained. In order to reduce computational cost, we denote
an additional bounding radius $R_{1}$. All features that had been joined to an
atom, with a Euclidean distance from the origin less than $R_{1}$ are excluded
from being joined to other atoms in the followed iterations. Next, the set of
features $Y$ is tested for consensus using RANSAC [Fischler and Bolles(1981)]
under affine transformation. If there is a consensus, an atom is created. The
process continues for the rest of the features that had not yet been joined to
an atom. There is a trade-off for selecting the radius $R_{2}$: A radius that
is too big means more atoms will be comprised, but the affine transformation
premise will no longer be valid; choosing a $R_{2}$ radius that is too small
will comply under the affine transformation premise, but fewer atoms will be
comprised.

Condition 3 $x_{i}\cap x_{j} \neq \emptyset$, Where $X$ is a joint set where
atoms overlap

Condition 3 is imperative so that independent motion objects can be
represented as piecewise-smooth objects. When applying overlap between atoms
we can represent the transformation of a piecewise-smooth object as a minimum
cut problem [Keuper et al.(2015)Keuper, Andres, and Brox]. The minimum cut
problem is the problem of decomposing a graph $G=(V,E)$ into an optimal number
of segments such that the overall cost in terms of edge weights $w_{e}$ is
minimized.

This node labeling problem can equivalently be formulated as a binary edge
labeling problem

$\displaystyle\underset{x\in\left\\{0,1\right\\}^{E}}{min}\sum_{e\in
E}w_{e}x_{e}$ (1) $\displaystyle subject\,to\,x\in MinimumCut$

We build the graph $G$ such that every atom is represented by a vertex $v\in
V$. If every vertex is connected by an edge $e\in E$ to its nearest neighbors,
all solutions to the minimum cut problem yield a segmentation into connected
components. The weights of the edges $e\in E$ define how similar two atoms
are. We determine atom affinities using the forward-backward error evaluated
using the affine transformation of an atom between two frames on a different
atom.

The minimum cost multi-cut computed on $G$ yields our desired segmentation
into the optimal number of motion models. Since objects consist of multiple
independent moving parts, the optimal number of segments is selected to be
larger than the true number. The process yields a fine-grained segmentation
later to be coarsened.

The fine-grained segmentation result by LAAV show the advantage of the
presented approach. LAAV is not restricted to just a rigid body segmentation
algorithm but also can describe more complicated motions. For example in
Figure 1 (c), the motion of the cylinder’s upper face and middle face are
segmented separately. We later coarsen the segmentation results for the
purpose of comparison with other state-of-the-art algorithms and to enhance
run-time.

###  3.2 Fine-To-Coarse

To merge the atoms, we suggest a Fine-To-Coarse inference strategy that
utilizes motion segmentation with epipolar geometry - the intrinsic projective
geometry between two views. The fundamental matrix F represents the epipolar
geometry, which satisfies the following condition:

$y_{2}^{T}Fy_{1}=0$ (2)

Where the fundamental matrix $F$ is a 3x3 matrix of rank 2, $y_{1}$ and
$y_{2}$ are features in homogeneous coordinates;
$y_{1}=[y_{1_{x}},y_{1_{y}},1]^{T}$ ; and $y_{2}$ =
$[y_{2_{x}}^{\prime},y_{2_{y}}^{\prime},1]^{T}$. under Equation 2 we can infer
the following properties:

Property 1 all atoms on the same moving rigid object have the same fundamental
matrix $F$.

Property 2 the corresponding feature lies on a moving object in the other
frame and must lie on the epipolar line. Where $F_{y_{1}}$ represents an
epipolar line in the other frame and $y_{2}$ lies on the line.

In Fine-To-Coarse, we adopt a scoring scheme of all motion models estimated in
the previous step. Our approach randomly selects 2 atoms from one motion model
$X_{S_{j}}=\left\\{x_{j1},x_{j2}\in S_{j}\right\\}$ and 2 atoms from another
motion model $X_{S_{k}}=\left\\{x_{k1},x_{k2}\in S_{k}\right\\}$ and $j\neq
k$. Under Property 1, if both motion models are on the same moving body their
fundamental matrix $F$ should be the same. We estimate the fundamental matrix
$F$ from the 4 atoms. Once the fundamental matrix $F$ is estimated, we use the
Sampson distance to measure the distance between atoms and the epipolar line.
Under Property 2, a small distance between a corresponding feature to the
epipolar line means that the fundamental matrix $F$ represents the motion
model accordingly. In order to evaluate the Sampson distance on atoms we
measure the distance of all features comprising the atoms and the epipolar
line. We use it as follows:

$SD(X_{S_{j}},X_{S_{k}},F_{S_{j},S_{k}})=\frac{X_{S_{j}}^{T}F_{S_{j},S_{k}}X_{S_{k}}}{(F_{S_{j},S_{k}}FX_{S_{j}})^{2}+(F_{S_{j},S_{k}}X_{S_{k}})^{2}}$
(3)

We measure the distance
$d_{j,k}=SD(X_{S_{j}}^{(l)},X_{S_{k}}^{(r)},F_{S_{j}S_{k}}^{(l,r)})$ between
the epipolar line from the matrix $F_{S_{j}S_{k}}^{(l,r)}$, and two
corresponding atoms between frames ( $l,r$ ) from the motion models $s_{k}$
and $s_{j}$ . The distance between the two motion models can be used to vote
for two motion models belonging to the same motion. Therefore, every atom’s
$x_{i}\in\\{X_{S_{j}},X_{S_{k}}\\}$ distance from the epipolar line is voted
by $e^{-\lambda d_{i}}$ where the parameter $\lambda$ controls the voting
strength. If the value of $\lambda$ is small, then it gives a large voting
value to the atoms and vice-versa. As a result of this process, $d_{j,k}$ is
now a vector containing the distances from all features points in frame $r$ to
the epipolar line.

$d_{j,k}^{epipolar}=max(avg(d_{j,k}),avg(d_{k,j}))\,$ (4)

$d_{j,k}^{epipolar}$ epipolar is the total distance between the atoms chosen
randomly. In other words, if the distance value is large, the affinity between
the two selected motion models is small. In contrast to [Jung et
al.(2014)Jung, Ju, and Kim] we do not change the motion model labeling during
accumulation, only at the end of the process. Since we do not deal with
pairwise labeling, we do not need to worry about misclassification. In
addition, this process is not iterative, only a small number of 7 frames are
selected. The frames are selected randomly with enough distance between them
so that different motions will be well separated.

We also compute motion models’ affinities determined by the spatial distance
between trajectories. Similarly, to [Keuper et al.(2015)Keuper, Andres, and
Brox], we define the motion difference of two motion models at time $t$ as
$d_{t}^{motion}(A,B)=\frac{\|\partial A-\partial B\|}{\sigma_{t}}$, where
$\partial_{t}A$ and $\partial_{t}B$ are the partial derivatives of the
averaged features trajectories of atoms $A$ and $B$. $A$, $B$ belong to the
motion model $j,k$ respectively. $\sigma_{t}$ is the variation of the optical
flow [Brox et al.(2009)Brox, Bregler, and Malik]. We build this problem as a
superposition of both the spatial distance and the epipolar distance.

$Z_{j,k}=Z_{j,k}+exp(-\lambda\cdot d_{j,k}^{epipolar}\cdot
d_{t}^{motion}(S_{j},S_{k}))^{-1}$ (5)

where $\lambda$ is a forgetting-factor controlling the weight of every sample.
In order to group the motion model, we utilize a spectral clustering method
[Ng et al.(2002)Ng, Jordan, and Weiss] on $Z$, and $Z$ is the $(2C\,x\,2C)$
affinity matrix.

###  3.3 Fine Tuning

The Fine-Tuning stage is designated to segment sparse features that had not
been segmented to an atom and consequently was not segmented to a motion
model. To this end we use the randomized voting (RV) method [Jung et
al.(2014)Jung, Ju, and Kim] which performs the best up-to-date motion
segmentation and is robust to noise. RV also requires prior knowledge of the
motion models $C$. In contrast to RV, we do not use random initialization for
the groups, rather use the features that had been labeled in LAAV and Fine-To-
Coarse stages for initialization. At this stage there is a high confidence in
the features labeling, therefore an additional weight is added to the voting
histogram so that RV will affect the unlabeled ones more than the labeled
ones. The algorithm is similar to RV; for all groups the fundamental matrices
are estimated using the initial labels. Next, we update the labels using the
fundamental matrices estimated from the current features segmented to the
group, and the process is repeated. For each iteration the algorithm selects
$m$-features and estimates the fundamental matrix $F$ using the selected
features. Then, the Sampson distance is calculated using the matrix $F$ for
all the features. A score is added to the histogram for a feature as a
function of the Sampson distance from the epipolar line. Finally, the labels
of the features are updated, where labels are assigned according to the
maximum score. However, the randomized voting could drift to an incorrect
result due to bad fitting of the forgetting-factor $\alpha$, or a small voting
strength parameter $\lambda$. In this case, we use a large voting strength and
forgetting-factor. Due to using these parameters we do need to worry about
termination conditions, because the prior labeling was complete with the
exception of minor fine-tuning.

##  4 Results

We evaluated our algorithm using Hopkins155 dataset [Tron and Vidal(2007)].
The Hopkins155 dataset has been the standard evaluation metric for the problem
of motion segmentation. It contains 155 sets of features point trajectories of
two and three motion models from 50 videos, along with the corresponding
ground truth segmentation for comparison. We compared our proposed algorithm’s
performance with the state-of-the-art methods with emphasis on [Jung et
al.(2014)Jung, Ju, and Kim] as it showed best results so far. For evaluating
the quality of the methods, we have two crucial criteria, the accuracy with
noise-free data and with added artificial noise. The following methods were
compared: generalized principal component analysis (GPCA) [Vidal et
al.(2005)Vidal, Ma, and Sastry]; SCC presented in [Chen and Lerman(2009)]; The
spectral clustering (SC) method [Lauer and Schnorr(2009)]; Ranking of locally
sampled subspaces (RLS) [Dimitriou and Delopoulos(2012)]; Linear combination
of views based algorithm(LCV)[Zografos and Nordberg(2011)] ; SSC [Elhamifar
and Vidal(2012)]; PAC [Zappella et al.(2010)Zappella, Provenzi, Lladó, and
Salvi]; Random Voting(RV)[Jung et al.(2014)Jung, Ju, and Kim];LatLRR [Liu and
Yan(2011)], and ICLM [Flores-Mangas and Jepson(2013)].

Table 1: Total Errors. The first row represents average error rates for all
(155 sequences). The second and third rows are two (120 sequences) and three
(35 sequences) motions, respectively. n/a means the value is not presented in
the corresponding paper.

Figure 2: Accuracy with Artificial Noise using Hopkins155 with different
levels of Gaussian noise

Figure 3: Number Of Iteration for Accurate motion segmentation for
$\sigma_{n}\in\left\\{0,0.5,1,2\right\\}$ .

Figure 4: Accuracy Distribution for both methods for
$\sigma_{n}\in\left\\{0,0.5,1,2\right\\}$

###  4.1 Performance Comparison

In Table 1 we present the average misclassification error over two-motions,
three-motions, and both. In all three cases, the performance of our algorithm
showed the lowest error rate when compared to the other algorithms. Our
algorithm achieved total misclassification average error of 0.55%. In two-
motions and three-motions separately we achieved 0.37% and 1.12% respectively.
Another experimental setup is designated to test the effects of noise of
different magnitudes on the classification accuracy. Following [Jung et
al.(2014)Jung, Ju, and Kim], we used three Gaussians noise with a zero mean
and different diagonal covariance matrix
$\sum_{n}=\sigma_{n}^{2}I\>,\sigma_{n}\in\left\\{0.5,1,2\right\\}$ ,where, $I$
is a $2X2$ identity matrix. In the test we computed over 150 repetitions.
Results are shown in Figure 4, and indicate that our algorithm showed a
comparable performance to the other state-of-the-art methods. Like other
algorithms, large noise variances comprise classification accuracy, and so, in
the same manner the contribution of LAAV is limited. Large noise variances
result in only a small portion of atoms constructed, leaving a large portion
of features non-initialized. In this scenario the algorithmic approach is to
randomly initialize the remained features as in [Jung et al.(2014)Jung, Ju,
and Kim].

It can be seen that, from all the state-of-the-art methods, the iterative
algorithm RV outperforms all. The result of the randomized voting is reliable
when the number of parameters for estimation is small and the amount of
feature points is large. Thus, convergence can be considered as definite.
However, if in other cases the process does not converge the results are less
reliable, for example, when the number of features is too large. We show that
our proposed method consistently outperforms RV on Hopkins155 dataset in Table
1 and in Figure 4.

###  4.2 Proposed vs. Random Voting

Two criteria are relevant when comparing iterative methods, the accuracy
performance and variance, and the process convergence, it’s distribution as a
function of iteration number and it’s probability. Using the Hopkins155 [Tron
and Vidal(2007)] dataset we compared the distribution of the iteration number
when the process converged between our proposed method in the Fine-Tuning Step
and RV. Figure 4 shows the distribution of the iteration number for
convergence and convergence percentage under different levels of artificial
noise. In order to generate statistical significance, we repeated the test 150
times both for noise-free data and for added artificial noise. Following [Jung
et al.(2014)Jung, Ju, and Kim] we used three Gaussians noise with a zero mean
and different diagonal covariance matrix $∑_{n}$. In the RV algorithm, the
iteration termination criteria is set by the number of trials. In RV all
trials are independent and under the assumption that for enough trials the
algorithm is bounded to converge. RV process is divided into trials and
iterations. Every trial consists of multiple iterations. A trial is terminated
when the iteration termination criteria is reached, then another trial starts.
The calculation of RV iteration number was performed as follows: for every
trial, following [Jung et al.(2014)Jung, Ju, and Kim] the maximum iterations
termination criteria by default is 150. For every trial that does not converge
the iteration number increases by 150. Following [Jung et al.(2014)Jung, Ju,
and Kim] The maximum trial termination criteria by default is 10. We excluded
sequences that did not converge and estimated the ratio between the converged
sequence with the ones that did not converge. It can be seen from Figure 4
that our method converges with significantly less iterations. Also, in all
noise levels the number of non-converged sequences overall in our approach is
far smaller.

In order to compare the accuracy variation between RV and the proposed method
we repeated the test 150 times for each of the non-contaminated or
contaminated data with three Gaussians noise with a zero mean and different
diagonal covariance matrix $∑_{n}$. Results are shown in Figure 4. We show the
accuracy percentage error with 4 different values of added noise. The
Gaussians shown indicate that our method comprises accuracy for large levels
of noise, while still showing better accuracy and small variation than RV.

##  5 Conclusions

We introduced a computationally efficient motion segmentation algorithm for
trajectory data. Efficiency comes from the use of a simple but powerful
representation of motion as atom trajectories, built from small patches of
objects. We showed that an effective initialization for RV based on affine
transformation of atoms is more accurate and robust to noise. As a result, our
algorithm achieved excellent performances with total average error rates of
0.55%. In conclusion, our algorithm achieved, within a reasonable time, the
highest performance of all other state-of the art algorithms, and also
achieved comparable accuracy under noisy environmental conditions.

## References

  * [Brox et al.(2009)Brox, Bregler, and Malik] T. Brox, C. Bregler, and J. Malik.  Large displacement optical flow.  In _2009 IEEE Conference on Computer Vision and Pattern Recognition_ , pages 41–48, June 2009.  10.1109/CVPR.2009.5206697.
  * [Chen and Lerman(2009)] Guangliang Chen and Gilad Lerman.  Motion segmentation by SCC on the hopkins 155 database.  _CoRR_ , abs/0909.1608, 2009.  URL http://arxiv.org/abs/0909.1608.
  * [Costeira and Kanade(1998)] João Paulo Costeira and Takeo Kanade.  A multibody factorization method for independently moving objects.  _Int. J. Comput. Vision_ , 29(3):159–179, September 1998.  ISSN 0920-5691.  10.1023/A:1008000628999.  URL http://dx.doi.org/10.1023/A:1008000628999.
  * [Dimitriou and Delopoulos(2012)] N. Dimitriou and A. Delopoulos.  Improved motion segmentation using locally sampled subspaces.  In _2012 19th IEEE International Conference on Image Processing_ , pages 309–312, Sept 2012.  10.1109/ICIP.2012.6466857.
  * [Elhamifar and Vidal(2012)] Ehsan Elhamifar and R. Vidal.  Sparse subspace clustering: Algorithm, theory, and applications.  _CoRR_ , abs/1203.1005, 2012.  URL http://arxiv.org/abs/1203.1005.
  * [Fischler and Bolles(1981)] Martin A. Fischler and Robert C. Bolles.  Random sample consensus: A paradigm for model fitting with applications to image analysis and automated cartography.  _Commun. ACM_ , 24(6):381–395, June 1981.  ISSN 0001-0782.  10.1145/358669.358692.  URL http://doi.acm.org/10.1145/358669.358692.
  * [Flores-Mangas and Jepson(2013)] Fernando Flores-Mangas and Allan D. Jepson.  Fast rigid motion segmentation via incrementally-complex local models.  In _CVPR_ , pages 2259–2266. IEEE Computer Society, 2013.
  * [Jian and Chen(2010)] Yong-Dian Jian and Chu-Song Chen.  Two-view motion segmentation with model selection and outlier removal by ransac-enhanced dirichlet process mixture models.  _International Journal of Computer Vision_ , 88(3):489–501, 2010.  ISSN 1573-1405.  10.1007/s11263-010-0317-y.  URL http://dx.doi.org/10.1007/s11263-010-0317-y.
  * [Jung et al.(2014)Jung, Ju, and Kim] Heechul Jung, Jeongwoo Ju, and Junmo Kim.  Rigid motion segmentation using randomized voting.  In _Computer Vision and Pattern Recognition (CVPR), 2014 IEEE Conference on_ , June 2014.  10.1109/CVPR.2014.158.
  * [Keuper et al.(2015)Keuper, Andres, and Brox] M. Keuper, B. Andres, and T. Brox.  Motion trajectory segmentation via minimum cost multicuts.  In _IEEE International Conference on Computer Vision (ICCV)_ , 2015\.  URL http://lmb.informatik.uni-freiburg.de//Publications/2015/KB15b.
  * [Krause et al.(2013)Krause, Stark, Deng, and Fei-Fei] J. Krause, M. Stark, J. Deng, and L. Fei-Fei.  3d object representations for fine-grained categorization.  In _2013 IEEE International Conference on Computer Vision Workshops_ , pages 554–561, Dec 2013.  10.1109/ICCVW.2013.77.
  * [Lauer and Schnorr(2009)] F. Lauer and C. Schnorr.  Spectral clustering of linear subspaces for motion segmentation.  In _2009 IEEE 12th International Conference on Computer Vision_ , pages 678–685, Sept 2009.  10.1109/ICCV.2009.5459173.
  * [Li(2007)] H. Li.  Two-view motion segmentation from linear programming relaxation.  In _2007 IEEE Conference on Computer Vision and Pattern Recognition_ , pages 1–8, June 2007.  10.1109/CVPR.2007.382975.
  * [Liu and Yan(2011)] Guangcan Liu and Shuicheng Yan.  Latent low-rank representation for subspace segmentation and feature extraction.  In _2011 International Conference on Computer Vision_ , pages 1615–1622. IEEE, 2011.
  * [Luong and Faugeras(1996)] Quan-Tuan Luong and Olivier D. Faugeras.  The fundamental matrix: Theory, algorithms, and stability analysis.  _International Journal of Computer Vision_ , 17(1):43–75, Jan 1996.  ISSN 1573-1405.  10.1007/BF00127818.  URL https://doi.org/10.1007/BF00127818.
  * [Morel and Yu(2009)] J. Morel and G. Yu.  Asift: A new framework for fully affine invariant image comparison.  _SIAM Journal on Imaging Sciences_ , 2(2):438–469, 2009.  10.1137/080732730.  URL https://doi.org/10.1137/080732730.
  * [Ng et al.(2002)Ng, Jordan, and Weiss] Andrew Y. Ng, Michael I. Jordan, and Yair Weiss.  On spectral clustering: Analysis and an algorithm.  In T. G. Dietterich, S. Becker, and Z. Ghahramani, editors, _Advances in Neural Information Processing Systems 14_ , pages 849–856. MIT Press, 2002.  URL http://papers.nips.cc/paper/2092-on-spectral-clustering-analysis-and-an-algorithm.pdf.
  * [Poling and Lerman(2013)] Bryan Poling and Gilad Lerman.  A new approach to two-view motion segmentation using global dimension minimization.  _CoRR_ , abs/1304.2999, 2013.  URL http://arxiv.org/abs/1304.2999.
  * [Rao et al.(2008)Rao, Tron, Vidal, and Ma] S. R. Rao, R. Tron, R. Vidal, and Yi Ma.  Motion segmentation via robust subspace separation in the presence of outlying, incomplete, or corrupted trajectories.  In _2008 IEEE Conference on Computer Vision and Pattern Recognition_ , pages 1–8, June 2008.  10.1109/CVPR.2008.4587437.
  * [Shi and Malik(2000)] Jianbo Shi and Jitendra Malik.  Normalized cuts and image segmentation.  _IEEE Trans. Pattern Anal. Mach. Intell._ , 22(8):888–905, August 2000.  ISSN 0162-8828.  10.1109/34.868688.  URL http://dx.doi.org/10.1109/34.868688.
  * [Torr(1998)] P. H. Torr.  Geometric motion segmentation and model selection [and discussion].  _Philosophical Transactions: Mathematical, Physical and Engineering Sciences_ , 356(1740):1321–1340, 1998.  ISSN 1364503X.  URL http://www.jstor.org/stable/54848.
  * [Tron and Vidal(2007)] R. Tron and R. Vidal.  A benchmark for the comparison of 3-d motion segmentation algorithms.  In _2007 IEEE Conference on Computer Vision and Pattern Recognition_ , pages 1–8, June 2007.  10.1109/CVPR.2007.382974.
  * [Vidal et al.(2005)Vidal, Ma, and Sastry] R. Vidal, Yi Ma, and S. Sastry.  Generalized principal component analysis (gpca).  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 27(12):1945–1959, Dec 2005.  ISSN 0162-8828.  10.1109/TPAMI.2005.244.
  * [Vidal et al.(2002)Vidal, Soatto, Ma, and Sastry] Rene Vidal, Stefano Soatto, Yi Ma, and Shankar Sastry.  Segmentation of dynamic scenes from the multibody fundamental matrix, 2002\.
  * [Zappella et al.(2010)Zappella, Provenzi, Lladó, and Salvi] Luca Zappella, Edoardo Provenzi, Xavier Lladó, and Joaquim Salvi.  Adaptive motion segmentation algorithm based on the principal angles configuration.  In _Asian Conference on Computer Vision_ , pages 15–26. Springer, 2010.
  * [Zhang(1996)] Zhengyou Zhang.  Determining the epipolar geometry and its uncertainty: A review.  _International Journal of Computer Vision_ , 27:161–195, 1996.
  * [Zografos and Nordberg(2011)] Vasileios Zografos and Klas Nordberg.  Fast and accurate motion segmentation using linear combination of views.  In _BMVC 2011 :_ , pages 12.1–12.11, 2011.


# DDLNet: Boosting Remote Sensing Change Detection with Dual-Domain Learning
††thanks: This work was supported by the Public Welfare Science and Technology
Plan of Ningbo City (2022S125), the Key Research and Development Plan of
Zhejiang Province (2021C01031) and the Science and Technology Innovation 2025
Major Project of Ningbo City (Grant No. 2022Z032).

1st Xiaowen Ma† School of Software Technology
Zhejiang University
Hangzhou, China
<EMAIL_ADDRESS>2nd Jiawei Yang† School of Software Technology
Zhejiang University
Hangzhou, China
<EMAIL_ADDRESS>3rd Rui Che  School of Software Technology
Zhejiang University
Hangzhou, China
<EMAIL_ADDRESS>4th Huanting Zhang  School of Software Technology
Zhejiang University
Hangzhou, China
<EMAIL_ADDRESS>5th Wei Zhang∗ † Equal contributions. * Corresponding
author. School of Software Technology
Zhejiang University
Hangzhou, China
Innovation Center of Yangtze River DeltaZhejiang University
Jiaxing Zhejiang, 314103, China
<EMAIL_ADDRESS>
###### Abstract

Remote sensing change detection (RSCD) aims to identify the changes of
interest in a region by analyzing multi-temporal remote sensing images, and
has an outstanding value for local development monitoring. Existing RSCD
methods are devoted to contextual modeling in the spatial domain to enhance
the changes of interest. Despite the satisfactory performance achieved, the
lack of knowledge in the frequency domain limits the further improvement of
model performance. In this paper, we propose DDLNet, a RSCD network based on
dual-domain learning (i.e., frequency and spatial domains). In particular, we
design a Frequency-domain Enhancement Module (FEM) to capture frequency
components from the input bi-temporal images using Discrete Cosine Transform
(DCT) and thus enhance the changes of interest. Besides, we devise a Spatial-
domain Recovery Module (SRM) to fuse spatiotemporal features for
reconstructing spatial details of change representations. Extensive
experiments on three benchmark RSCD datasets demonstrate that the proposed
method achieves state-of-the-art performance and reaches a more satisfactory
accuracy-efficiency trade-off. Our code is publicly available at
https://github.com/xwmaxwma/rschange.

###### Index Terms:

Change Detection, Dual-domain Learning, Frequency-domain Enhancement, Spatial-
domain Recovery

##  I Introduction

RSCD is a crucial task to identify the change of interest, represented by
spectral variations, by analyzing multi-temporal remote sensing images
captured over the same geographic region. As a result, image pixels are
classified into a change map to determine whether each part of the region
involves the change of interest. This analytical process reveals the local
development patterns from both natural and socio-economic perspectives, and
possesses significant implications for urban planning, environmental
monitoring, and disaster assessment [1].

The rapid advancement of deep learning techniques has significantly propelled
the area of RSCD, particularly with the progress of Convolutional Neural
Networks (CNNs) in extracting hierarchical features [2]. For instance, LGPNet
[3] incorporates the attention mechanism to enhance the expressiveness of
change representations. The Siamese-based STANet [4] adopts the BAM to
leverage global spatio-temporal relations for discriminative features and the
PAM to aggregate multi-scale attentional representations for the details on
fine-grained objects. DASNets [5] employs a dual-attention mechanism to
capture long-range dependencies and obtain discriminative feature
representations for change detection. SNUNet [6] integrates a Siamese neural
network with the U-Net architecture to preserve high-level semantic
information and detailed features through jump connections. ChangeFormer [7]
introduces a Transformer-based Siamese neural network, which exploits the
Transformer model for spatial relationship learning in temporal sequences.
Reaching the considerable improvements in the performance, these methods
primarily focus on solving the problem of RSCD only in the spatial domain,
which leaves room for further advancements.

More recently, extensive attempts from the perspective of the frequency domain
have been undertaken in the areas of target detection and semantic
segmentation. For example, incorporating the features in the frequency domain
features after transforming images into CNNs has been validated to improve the
performance of image analysis tasks [8]. AFFormer [9] learns local descriptive
representations of clustering prototypes from a frequency perspective,
eliminating the need for a complex decoder. Simultaneously, it incorporates a
Transformer with linear complexity, simplifying semantic segmentation to
conventional classification. These studies have shown that the introduction of
frequency information contributes to the extraction of discriminative features
for semantic recognition. Therefore, we are inspired to explore from the
frequency domain to improve the performance of RSCD.

Figure 1: Architecture of the proposed DDLNet, which consists of four
components: a Siamese backbone, FEM, SRM and a lightweight change detection
header. DS_Conv refers to the depth-wise separable convolution, which is
employed to reduce the number of parameters and computational cost while
minimizing the compromise in the performance.

In this paper, we propose DDLNet, a RSCD network based on dual-domain
learning, which involves Frequency-domain Enhancement Modules (FEM) and
Spatial-domain Recovery Modules (SRM). Given the feature maps extracted by a
Siamese neural network from the input bi-temporal images, the FEM captures the
frequency components using the Discrete Cosine Transform (DCT) to enhance the
changes of interest. Meanwhile, SRM at different scales facilitate the
recovery of spatial details through the fusion of coarse and refined
representations on the change of interest in a cascaded manner. In addition,
we employ a lightweight decoder to interconnect the representations across all
scales to generate the change map.

The primary contributions of this paper can be summarized as follows:

  * •

we introduce frequency-domain enhancement that extracts frequency components
from bi-temporal image feature maps and enhances the changes of interest,

  * •

we enable spatial-domain recovery that employs cross-time and cross-scale
mechanisms for feature fusion and reconstructs spatial details in change
representations,

  * •

we propose a novel RSCD method that achieves state-of-the-art performance on
three change detection datasets while reaching a more satisfactory trade-off
between accuracy and efficiency.

##  II Method

Fig.1 illustrates the architecture of our DDLNet that consists of four key
components on feature extraction, frequency-domain enhancement, spatial-domain
recovery, and decoding. Given the input bi-temporal images $T_{1}$ and
$T_{2}$, a Siamese neural network (i.e, two-stream ResNet-18) is utilized to
extract bi-temporal feature maps at four different scales. At each scale, the
bi-temporal feature maps from the corresponding residual block are enhanced
regarding the changes of interest between $T_{1}$ and $T_{2}$ in the frequency
domain using a FEM, which leverages the DCT to effectively capture frequency
components. The frequency-enhanced feature maps at the fourth scale
experiences an initial feature fusion, followed by being processed by the SRM,
which focuses on recovering the spatial details on the change of interest,
along with the feature maps at each other scale. The feature maps at all
scales are upsampled and concatenated along the channel direction; the number
of channels count are reinstated through convolution. Finally, an upsampling
yields a map on the change of interest of the same size as $T_{1}$ and
$T_{2}$.

###  II-A Frequency-domain Enhancement Module

Conventional RSCD methods concentrate on exploiting the spatial domain and
leave a significant space to explore the potential of the frequency domain. A
wide spectrum of previous studies have demonstrated the effectiveness of
considering the frequency domain in extracting discriminative features and
achieving accurate semantic recognition. Inspired by AFFormer [9] and Fcanet
[10], we design the FEM that adopts the DCT to efficiently capture the
frequency components within features and thus enhance the distinguishness of
the changes of interest in the input images.

The two-dimensional DCT can be formulated as follows:

$B_{h,w}^{i,j}=cos(\frac{\pi h}{H}(i+\frac{1}{2}))\cos(\frac{\pi
w}{W}(j+\frac{1}{2})),$ (1)

$f_{h,w}^{2d}=\sum_{i=0}^{H-1}\sum_{j=0}^{W-1}x_{i,j}^{2d}B_{h,w}^{i,j},$ (2)

where $f^{2d}\in\mathbb{R}^{H\times W}$ denotes the 2D DCT frequency spectrum,
$x^{2d}\in\mathbb{R}^{H\times W}$ represents the input, $H$ and $W$ refer to
the height and the width of $x^{2d}$, respectively. Additionally, in this
formula, $h\in\\{0,1,\cdots,\textit{H}-1\\}$,
$w\in\\{0,1,\cdots,\textit{W}-1\\}$, $i\in\\{0,1,\cdots,\textit{H}-1\\}$, and
$j\in\\{0,1,\cdots,\textit{W}-1\\}$.

Our FEM divides the input image $X$ along the channel dimension into parts
$[X^{0},X^{1},\cdots,X^{n-1}]$, where $X^{i}\in\mathbb{R}^{C^{\prime}\times
H\times W},i\in\\{0,1,\cdots,n-1\\}$ and $C^{\prime}=\frac{C}{n}$ under the
assumption that $C$ is divisible by $n$. The corresponding 2D DCT frequency
component is assigned to each part $X^{i}$, so as to obtain the frequency
vector via concatenation as follows:

$Freq=\operatorname{cat}([Freq^{0},Freq^{1},\cdots,Freq^{n-1}]),$ (3)

where each frequency component is formulated as follows:

$Freq^{i}=\mathrm{DCT}^{u_{i},v_{i}}(X^{i})=\sum_{h=0}^{H-1}\sum_{w=0}^{W-1}X_{h,w}^{i}B_{h,w}^{u_{i},v_{i}},$
(4)

where $\textit{Freq}^{i}\in\mathbb{R}^{C^{\prime}}$ is a
$C^{\prime}$-dimensional vector, $[u_{i},v_{i}]$ denote the frequency
component’s 2D indices for $X^{i}$, and $i\in\\{0,1,\cdots,\textit{H}-1\\}$.
The FEM’s output is then calculated as follows:

$\mathcal{Z}=Sigmoid(fc(Freq)).$ (5)

in which $fc$ represents the one-dimensional convolution, and $\mathcal{Z}$
denotes the frequency-enhanced feature map. As described by Equation 3 and 5,
multiple frequency components are obtained so as to expedite the effective
enrichment of the representation on the changes of interest for feature
enhancement.

Furthermore, we need to select frequency component indices [$u_{i}$, $v_{i}$]
for each part $X^{i}$. Specifically, we first determine the importance of each
frequency component and then investigate the impact of using different numbers
of frequency components. Following the previous work [10], we conduct pre-
training on ImageNet to ascertain the significance of different frequency
components for classification accuracy. We selected the top-n values as the
target frequencies. The specific effects of different numbers of frequency
components will be discussed in the experimental section. In addition, it is
noteworthy we omit the normalization factor for simplicity, as shown in
Equation 2, which however barely affects the performance of the proposed
method.

###  II-B Spatial-domain Recovery Module

Our SRM achieves feature fusion through cross-time and cross-scale mechanisms.
Specifically, following the frequency enhancement of features at each
hierarchical level, the deepest-layer features undergo element-wise
subtraction and are concatenated with the image captured at time T1, resulting
in a representation of deep-level changes. Subsequently, the representation of
deep-level changes is independently fused with shallow-level features across
scales. This enables the utilization of higher-level change representations to
guide context modeling for lower-level representations, enhancing semantic
information in the lower-level features and improving the recovery of spatial
details for the change representations.

Given the frequency-enhanced feature maps $\mathcal{Z}_{1}$ and
$\mathcal{Z}_{2}$ output by the FEM from $T_{1}$ and $T_{2}$ at each of the
first three scales, SRM first conducts an initial feature fusion to reach a
coarse representation $\mathcal{Z}_{c}$ on the change of interest as follows:

$\mathcal{Z}_{c}=\psi((\mathcal{Z}_{1}\ominus\mathcal{Z}_{2})\oplus\mathcal{Z}_{1}),$
(6)

where $\ominus$ denotes element-wise subtraction, and $\psi$ refers to depth-
wise separable convolution. The weight map $\mathcal{W}_{c}$ of
$\mathcal{Z}_{c}$ are calculated as follows:

$\mathcal{W}_{1}=Sigmoid(\phi(\mathcal{P}_{mean}(\mathcal{Z}_{c})\oplus\mathcal{P}_{max}(\mathcal{Z}_{c})),$
(7)

where $\mathcal{P}_{mean}$ and $\mathcal{P}_{max}$ respectively denote average
pooling and max pooling, and $\phi$ represents $1\times 1$ convolution. The
weight map is then applied to the coarse representation $\mathcal{C}_{4}$ at
the fourth scale, which reaches the refined representation on the change of
interest $\mathcal{C}_{i}$ as follows:

$\mathcal{C}_{i}=(\mathcal{W}_{1}\otimes\mathcal{C}_{4})\oplus\mathcal{C}_{i+1},$
(8)

where $\mathcal{C}_{i+1}$ represents the refined representation on the change
of interest from the previous SRM.

TABLE I:  Comparison of our DDLNet and representative RSCD methods regarding
performance on three datasets. Highest scores are in bold. All scores are
reported in percentage.

Method | WHU-CD | LEVIR-CD | CLCD
---|---|---|---
F1 | Pre. | Rec. | IoU | OA | F1 | Pre. | Rec. | IoU | OA | F1 | Pre. | Rec. | IoU | OA
FC-EF [11] | 72.01 | 77.69 | 67.10 | 56.26 | 92.07 | 83.40 | 86.91 | 80.17 | 71.53 | 98.39 | 48.64 | 73.34 | 36.29 | 32.14 | 94.30
FC-Siam-Di [11] | 58.81 | 47.33 | 77.66 | 41.66 | 95.63 | 86.31 | 89.53 | 83.31 | 75.92 | 98.67 | 44.10 | 72.97 | 31.60 | 28.29 | 94.04
FC-Siam-Conc [11] | 66.63 | 60.88 | 73.58 | 49.95 | 97.04 | 83.69 | 91.99 | 76.77 | 71.96 | 98.49 | 54.48 | 68.21 | 45.22 | 37.35 | 94.35
IFNet [12] | 83.40 | 96.91 | 73.19 | 71.52 | 98.83 | 88.13 | 94.02 | 82.93 | 78.77 | 98.87 | 48.65 | 49.96 | 47.41 | 32.14 | 92.55
DTCDSCN [13] | 71.95 | 63.92 | 82.30 | 56.19 | 97.42 | 87.67 | 88.53 | 86.83 | 78.05 | 98.77 | 60.13 | 62.98 | 57.53 | 42.99 | 94.32
BIT [14] | 83.98 | 86.64 | 81.48 | 72.39 | 98.75 | 89.31 | 89.24 | 89.37 | 80.68 | 98.92 | 57.13 | 64.39 | 51.34 | 39.99 | 94.27
LGPNet [3] | 79.75 | 89.68 | 71.81 | 66.33 | 98.33 | 89.37 | 93.07 | 85.95 | 80.78 | 99.00 | 55.42 | 62.98 | 49.49 | 38.33 | 94.10
SNUNet [6] | 83.50 | 85.60 | 81.49 | 71.67 | 98.71 | 88.16 | 89.18 | 87.17 | 78.83 | 98.82 | 60.54 | 65.63 | 56.19 | 43.41 | 94.55
DMATNet [15] | 85.07 | 89.46 | 82.24 | 74.98 | 95.83 | 89.97 | 90.78 | 89.17 | 81.83 | 98.06 | 66.56 | 72.74 | 61.34 | 49.87 | 95.41
ChangeStar(FarSeg) [16] | 87.01 | 88.78 | 85.31 | 77.00 | 98.70 | 89.30 | 89.88 | 88.72 | 80.66 | 98.90 | 60.75 | 62.23 | 59.34 | 43.63 | 94.30
ChangeFormer [7] | 81.82 | 87.25 | 77.03 | 69.24 | 94.80 | 90.40 | 92.05 | 88.80 | 82.48 | 99.04 | 58.44 | 65.00 | 53.07 | 41.28 | 94.38
USSFC-Net [17] | 88.40 | 90.67 | 86.24 | 79.21 | 98.96 | 90.11 | 88.88 | 89.35 | 80.37 | 98.89 | 63.04 | 64.83 | 61.34 | 46.03 | 94.42
DDLNet (Ours) | 90.56 | 91.56 | 90.03 | 82.75 | 99.13 | 90.60 | 91.72 | 89.41 | 82.49 | 99.10 | 72.44 | 78.93 | 66.90 | 56.78 | 95.94

###  II-C Loss Function

Considering the renowned imbalance problem on the distribution of changed and
unchanged pixels in RSCD, we devise a hybrid loss function integrating both
focal loss and dice loss, whose formulation is as follows:

$\mathcal{L}=\mathcal{L}_{focal}+\mathcal{L}_{dice}.$ (9)

Specifically, the focal loss is formulated as follows:

$\mathcal{L}_{\mathrm{focal}}=-\alpha(1-\hat{p})^{\gamma}log(\hat{p}),$ (10)

$\left.\hat{p}=\left\\{\begin{array}[]{ll}p,&\text{if}\quad y=1\\\
1-p,&\text{otherwise}\end{array}\right.\right.,$ (11)

where $\alpha$ and $\gamma$ represent two hyperparameters governing the
contribution of positive and negative samples and the method’s attention to
the challenging samples, respectively (We set $\alpha$ to 0.2 and $\gamma$ to
2 in our experiments), $p$ denotes the probability, and $y$ refers to the
binary label for unchanged pixels (0) and changed pixels (1). Besides, the
dice loss is formulated as follows:

TABLE II:  Comparison of our DDLNet and its variants regarding RSCD performance on the WHU dataset. Highest scores are in bold. All scores are reported in percentage.  Model | F1 | Pre. | Rec. | IoU | OA
---|---|---|---|---|---
Base | 87.96 | 88.01 | 88.03 | 78.73 | 98.69
Base+FEM | 89.17 | 89.96 | 88.41 | 80.46 | 98.98
Base+SRM | 89.38 | 88.75 | 90.01 | 80.79 | 98.97
DDLNet (Ours) | 90.56 | 91.56 | 90.03 | 82.75 | 99.13

$\mathcal{L}_{dice}=1-\frac{2\cdot E\cdot
Softmax(E^{\prime})}{E+Softmax(E^{\prime})},$ (12)

$E^{\prime}=\left\\{e_{k}^{\prime}\in\mathbb{R}^{2}|k=1,2,\ldots,H\times
W\right\\},$ (13)

where $E$ denotes the ground truth, and $E^{\prime}\in\mathbb{R}^{H\times
W\times 2}$ represents the change map including labeled pixels
$\\{e_{k}^{\prime}\\}$.

##  III Experimental

###  III-A Datasets and Evaluation Metrics

We compare the proposed DDLNet on the three RSCD datasets by five commonly
used metrics: F1 score (F1), recall (Rec), precision (Pre), intersection over
union (IoU), and overall accuracy (OA).

The WHU architectural dataset [18] is a public CD dataset containing a pair of
bit-time aerial images of 32507×15354 dimensions. Following previous work
[14], we crop the images into patches of 256×256 size and randomly divide them
into a training set (6096 images), a validation set (762 images), and a test
set (762 images).

The LEVIR-CD dataset [4] is a CD dataset containing 637 pairs of high-
resolution dual-temporal remote sensing images. we crop the images into non-
overlapping patches of 256 × 256 size, and we randomly divide them into a
training set (7120 images), a validation set (1024 images), and a test set
(2048 images).

TABLE III:  The performance of DDLNet on the WHU-CD and LEVIR-CD datasets when using different frequency components. $n$ represents the number of frequency components.  n | WHU-CD |  LEVIR-CD
---|---|---
F1 | IoU | F1 | IoU
4 | 90.20 | 82.16 | 90.30 | 82.31
8 | 88.03 | 78.62 | 90.27 | 82.27
16 | 90.56 | 82.75 | 90.60 | 82.46
32 | 89.11 | 80.35 | 90.15 | 82.07

The CLCD dataset [19] consists of 600 pairs of sample images of farmland
changes with a size of 512 × 512 and it is randomly divided into a training
set (360 images), a validation set (120 images) and a test set (120 images).

###  III-B Implementation Details

We implement the proposed DDLNet using Python and Pytorch on a workstation
equipped with two NVIDIA GTX A5000 graphics cards (48 GB of GPU memory in
total.) We set the base learning rate to 0.05 and optimize the model using
stochastic gradient descent (SGD) with a momentum of 0.9. and weight decay to
5e-5. Also, during training, we update the learning rate using a multi-step
learning rate decay strategy with a gamma value of 0.1. In the experiments,
the WHU-CD dataset and LEVIR-CD dataset batch sizes are set to 8, and the CLCD
dataset batch size is set to 4. We perform data augmentation, such as rotation
and flipping, on the input training data.

TABLE IV:  Comparison of our DDLNet and representative RSCD methods regarding the number of parameters (Params) in million (M) and the number of Flops in Giga (G). Highest scores are in bold. The second-highest scores are underscored. All scores are reported in percentage.  Model |  Params (M) |  Flops (G)
---|---|---
IFNet [12] |  50.44 |  82.26
DTCDSCN [13] |  41.07 |  14.42
LGPNet [3] |  70.99 |  125.79
SNUNet [6] |  12.03 |  54.88
DMATNet [15] |  13.27 |  -
ChangeFormer [7] |  41.03 |  202.79
DDLNet (Ours) |  12.67 |  7.35

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

Figure 2: Comparison of our DDLNet (h) and representative RSCD methods: (d)
FC-Siam-Di, (e) LGPNet, (f) SNUNet and (g) USSFC-Net, regarding visualized
RSCD results, along with (a) $T_{1}$, (b) $T_{2}$ and (c) ground truth, on the
WHU and LEVIR-CD test sets. Pixels are colored differently for better
visualization (i.e., white for true positive, black for true negative, red for
false positive, and green for false negative).

###  III-C Comparison and Analysis

We conduct a comprehensive evaluation of our DDLNet on three benchmark
datasets: WHU-CD, LEVIR-CD, and CLCD. Table I presents the performance
comparison with current state-of-the-art methods, revealing that DDLNet
achieves F1 scores of 90.2% (WHU-CD), 90.6% (LEVIR-CD), and 72.44% (CLCD),
showcasing significant improvements across all datasets. This performance
superiority can be attributed to the effective feature optimization by our FEM
and SRM modules, utilization of multiple frequency components for enhanced
changes of interest, improved feature representation in pixel space, and fine-
grained information capture through feature fusion, facilitating the recovery
of spatial details in change representations.

Furthermore, we conduct ablation experiments on WHU-CD datasets. Table II
presents that the base model without FEM and SRM modules achieve F1 scores of
87.96% on WHU-CD dataset. The introduction of the FEM or SRM module
significantly improve performance, with our complete DDLNet achieving the
highest F1 of 90.2%, confirming the effectiveness of these modules in
enhancing change detection performance.

Additionally, as shown in Table III, we validate the results using $n$
frequency components, where we set $n$ to be 4, 8, 16, and 32. It can be
observed that the configuration with 16 frequency components achieve optimal
performance. Furthermore, as depicted in Table IV, DDLNet, with 12.67M
parameters and 7.35G Flops, demonstrate superior efficiency, requiring fewer
parameters and boasting the lowest computational cost compared to other
methods.

We visually validate the change detection results on three constructed
datasets in Fig.2. The comparison plots illustrate that DDLNet outperforms
other models by effectively leveraging the FEM and SRM modules to extract
high-quality change information and enhance change object boundaries.

## Conclusion

In this paper, we propose a DDLNet for change detection in remote sensing
images. The proposed DDLNet is based on dual-domain learning in the frequency
and spatial domains. Our DDLNet captures effective frequency components in
features using the discrete cosine transform, thereby enhancing the changes of
interest between dual temporal images. Additionally, through the spatial-
domain recovery module, it fuses spatiotemporal features to restore spatial
details of the change representation. We evaluate DDLNet on three public
change detection datasets, and the method achieves state-of-the-art
performance on all benchmark datasets, demonstrating its effectiveness in
remote sensing change detection.

## References

  * [1] Yuchao Feng, Honghui Xu, Jiawei Jiang, and Jianwei Zheng, “Compact intertemporal coupling network for remote sensing change detection,” in IEEE ICME. IEEE, 2023, pp. 144–149.
  * [2] Kelvin Xu, J Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhutdinov, R Zemel, and Yoshua Bengio, “Neural image caption generation with visual attention,” in Proc. ICML, 2015, pp. 2048–2057.
  * [3] Tongfei Liu, Maoguo Gong, Di Lu, Qingfu Zhang, Hanhong Zheng, Fenlong Jiang, and Mingyang Zhang, “Building change detection for vhr remote sensing im- ages via local–global pyramid network and cross-task transfer learning strategy,” IEEE TGARS, vol. 60, pp. 1–17, 2021.
  * [4] Hao Chen and Zhenwei Shi, “A spatial-temporal attention-based method and a new dataset for remote sensing image change detection,” Remote Sensing, vol. 12, no. 10, pp. 1662, 2020.
  * [5] Jie Chen, Ziyang Yuan, Jian Peng, Li Chen, Haozhe Huang, Jiawei Zhu, Yu Liu, and Haifeng Li, “Das- net: Dual attentive fully convolutional siamese networks for change detection in high-resolution satellite images,” IEEE J-STARS, vol. 14, pp. 1194–1206, 2020.
  * [6] Sheng Fang, Kaiyu Li, Jinyuan Shao, and Zhe Li, “Snunet-cd: A densely connected siamese network for change detection of vhr images,” IEEE GRSL, vol. 19, pp. 1–5, 2021.
  * [7] Wele Gedara Chaminda Bandara and Vishal M Patel, “A transformer-based siamese network for change de- tection,” in IEEE IGARSS. IEEE, 2022, pp. 207–210.
  * [8] Kai Xu, Minghai Qin, Fei Sun, Yuhao Wang, Yen- Kuang Chen, and Fengbo Ren, “Learning in the fre- quency domain,” in CVPR, 2020, pp. 1740–1749.
  * [9] Dong Bo, Wang Pichao, and Fan Wang, “Afformer: Head-free lightweight semantic segmentation with lin- ear transformer,” in AAAI, 2023.
  * [10] Zequn Qin, Pengyi Zhang, Fei Wu, and Xi Li, “Fcanet: Frequency channel attention networks,” in ICCV, 2021, pp. 783–792.
  * [11] Rodrigo Caye Daudt, Bertr Le Saux, and Alexandre Boulch, “Fully convolutional siamese networks for change detection,” in ICIP. IEEE, 2018, pp. 4063–4067.
  * [12] Chenxiao Zhang, Peng Yue, Deodato Tapete, Liangcun Jiang, Boyi Shangguan, Li Huang, and Guangchao Liu, “A deeply supervised image fusion network for change detection in high resolution bi-temporal remote sensing images,” ISPRS Journal of Photogrammetry and Re- mote Sensing, vol. 166, pp. 183–200, 2020.
  * [13] Yi Liu, Chao Pang, Zongqian Zhan, Xiaomeng Zhang, and Xue Yang, “Building change detection for re- mote sensing images using a dual-task constrained deep siamese convolutional network model,” IEEE GRSL, vol. 18, no. 5, pp. 811–815, 2020.
  * [14] Hao Chen, Zipeng Qi, and Zhenwei Shi, “Remote sens- ing image change detection with transformers,” IEEE TGARS, vol. 60, pp. 1–14, 2021.
  * [15] Xinyang Song, Zhen Hua, and Jinjiang Li, “Remote sensing image change detection transformer network based on dual-feature mixed attention,” IEEE TGARS, vol. 60, pp. 1–16, 2022.
  * [16] Zhuo Zheng, Ailong Ma, Liangpei Zhang, and Yanfei Zhong, “Change is everywhere: Single-temporal su- pervised object change detection in remote sensing im- agery,” in ICCV, 2021, pp. 15193–15202.
  * [17] Tao Lei, Xinzhe Geng, Hailong Ning, Zhiyong Lv, Maoguo Gong, Yaochu Jin, and Asoke K Nandi, “Ultra- lightweight spatial–spectral feature cooperation network for change detection in remote sensing images,” IEEE TGARS, vol. 61, pp. 1–14, 2023.
  * [18] Shunping Ji, Shiqing Wei, and Meng Lu, “Fully con- volutional networks for multisource building extraction from an open aerial and satellite imagery data set,” IEEE TGARS, vol. 57, no. 1, pp. 574–586, 2018.
  * [19] Mengxi Liu, Zhuoqun Chai, Haojun Deng, and Rong Liu, “A cnn-transformer network with multiscale context aggregation for fine-grained cropland change detection,” IEEE J-STARS, vol. 15, pp. 4297–4306, 2022.