# Single-particle shell strengths near the doubly magic nucleus 56Ni and the
56Ni($p$,$\gamma$)57Cu reaction rate in explosive astrophysical burning

D. Kahl<EMAIL_ADDRESS>P. J. Woods  T. Poxon-Pearson  F. M. Nunes  B. A.
Brown  H. Schatz  T. Baumann  D. Bazin  J. A. Belarge  P. C. Bender  B. Elman
A. Estrade  A. Gade  A. Kankainen  C. Lederer-Woods  S. Lipschutz  B.
Longfellow  S.-J. Lonsdale  E. Lunderberg  F. Montes  W. J. Ong  G. Perdikakis
J. Pereira  C. Sullivan  R. Taverner  D. Weisshaar  R. Zegers  School of
Physics & Astronomy, University of Edinburgh, Edinburgh EH9 3FD, United
Kingdom  National Superconducting Cyclotron Laboratory, Michigan State
University, East Lansing, MI 48824, USA  Department of Physics & Astronomy,
Michigan State University, East Lansing, MI 48824, USA  JINA Center for the
Evolution of the Elements, Michigan State University, East Lansing, MI 48824,
USA  Central Michigan University, Mount Pleasant, MI 48859, USA  University of
Jyväskylä, P.O. Box 35, FI-40014 University of Jyväskylä, Finland

###### Abstract

Angle-integrated cross-section measurements of the 56Ni($d$,$n$) and ($d$,$p$)
stripping reactions have been performed to determine the single-particle
strengths of low-lying excited states in the mirror nuclei pair 57Cu$-$57Ni
situated adjacent to the doubly magic nucleus 56Ni. The reactions were studied
in inverse kinematics utilizing a beam of radioactive 56Ni ions in conjunction
with the GRETINA $\gamma$-array. Spectroscopic factors are compared with new
shell-model calculations using a full $pf$ model space with the GPFX1A
Hamiltonian for the isospin-conserving strong interaction plus Coulomb and
charge-dependent Hamiltonians. These results were used to set new constraints
on the 56Ni($p$,$\gamma$)57Cu reaction rate for explosive burning conditions
in x-ray bursts, where 56Ni represents a key waiting point in the
astrophysical $rp$-process.

###### keywords:

X-ray bursts, shell model, transfer reactions, radioactive beams

††journal: Physics Letters Blowellaltlowellaltfootnotetext: Present Address:
Department of Physics and Applied Physics, University of Massachusetts Lowell,
Lowell, MA 01854, USA.

Doubly magic nuclei represent special cornerstones in our understanding and
exploration of nuclear structure (see, e.g., [1, 2]). These nuclei, and the
nuclei in their immediate vicinity, should be well described by very pure
shell-model configurations. In the case of the doubly magic, self-conjugate
nucleus 56Ni, the major shell closure at $N=Z=28$ arises from the spin-orbit
splitting of the $0f_{5/2}$ and $0f_{7/2}$ orbitals. There is evidence that
56Ni has a relatively soft core with significant configuration mixing in the
shell structure [3, 4, 5]. It exists on the cusp between stability, having a
(terrestrial) half-life of 6.08 d, and particle instability; the neighboring
nucleus, 57Cu, is only proton bound in its ground state. This special location
means 56Ni can form a key waiting point in explosive astrophysical burning
scenarios such as x-ray bursters, impeding the rate of flow of material
further along the proton drip-line via the 56Ni($p$,$\gamma$)57Cu reaction in
the $rp$ (rapid proton capture) process [6, 7]. 56Ni is the longest-lived
waiting point nucleus and historically was originally thought to represent the
termination point of the $rp$-process [6]. A direct measurement of the
56Ni($p$,$\gamma$) breakout reaction is not currently feasible with existing
radioactive beam intensities. Therefore, to constrain the
56Ni($p$,$\gamma$)57Cu reaction rate used in astrophysical models, an indirect
approach is mandated. This requires a knowledge of the shell structure and
properties of states in 57Cu involved in the explosive astrophysical
temperature burning range from $T\sim 0.5$–$2.0$ GK.

The mirror nucleus of 57Cu, 57Ni, has a ground-state spin-parity of $3/2^{-}$,
corresponding to a $1p_{3/2}$ neutron shell-model configuration, and $5/2^{-}$
and $1/2^{-}$ excited states corresponding to the occupation $0f_{5/2}$ and
$1p_{1/2}$ neutron shells, respectively [8]. A pioneering study of the
56Ni($d$,$p$)57Ni transfer reaction, measuring the differential cross section
in inverse kinematics, found these three lowest-lying states were described
with relatively pure single-neutron configurations with spectroscopic factors
$C^{2}S\sim 0.9$, with an approximate factor of two uncertainties [9]. This
information was then used to estimate the resonance contributions of the two
low-lying analog excited states in 57Cu [9, 10] for the 56Ni($p$,$\gamma$)57Cu
reaction rate, assuming isospin symmetry. Subsequently, an attempt was made to
measure the single-particle strengths of these states directly in 57Cu using
the 56Ni(3He,$d$)57Cu proton transfer reaction; however, due to the limited
resolution of $\sim\\!700$ keV in excitation energy and low statistics,
definite conclusions could not be drawn [11]. Two higher-lying $5/2^{-}$ and
$7/2^{-}$ states in 57Cu are expected to dominate the 56Ni($p$,$\gamma$)57Cu
reaction rate for burning temperatures $T>1$ GK (see, e.g., ref. [12]) but
only limits were set for the experimental $C^{2}S$ values in 57Ni [9].
Calculations predict the combined resonant capture reaction rate on the four
lowest-lying excited states in 57Cu is expected to dominate (by 3 orders of
magnitude) over the direct-capture contribution in explosive astrophysical
burning conditions [12, 7].

Here, we present a first study of the 56Ni($d$,$n$)57Cu proton transfer
reaction aimed at the first direct experimental determinations of the proton
single-particle strengths of the key low-lying excited states in 57Cu by
angle-integrated cross-section measurements. The method exploits the high
resolution and efficiency of the GRETINA (Gamma-Ray Energy Tracking In-beam
Nuclear Array) device [13] and an intense 56Ni radioactive beam produced in
flight by the National Superconducting Cyclotron Laboratory (NSCL). This new
approach has been shown to work successfully in determining spectroscopic
factors, including key astrophysical resonances [14, 15], and is described for
the present application in detail below. In addition, we have performed a
measurement of the 56Ni($d$,$p$)57Ni transfer reaction to study analog states
in the mirror nucleus 57Ni, and explore evidence for isospin symmetry breaking
effects. Furthermore, the 56Ni($d$,$p$) data were also used to measure for the
first time the single-particle strength of a high-lying excited state in 57Ni
whose analog likely determines the explosive astrophysical burning rate above
$T\sim 1$ GK in the 56Ni($p$,$\gamma$)57Cu reaction.

The 56Ni($d$,$n$)57Cu and 56Ni($d$,$p$)57Ni reaction studies were performed in
inverse kinematics. A 33.6 MeV/u 56Ni28+ beam was produced by in-flight
fragmentation of a 28-pnA 160-MeV/u primary beam of 58Ni27+ ions which
impinged upon a 1316-mg/cm2 thick 9Be production target. The A1900 fragment
separator [16] selected the ions of interest based on their magnetic rigidity
and used a 150-mg/cm2 thick Al achromatic wedge to provide isotopic separation
at its focal plane. The resulting 56Ni28+ beam had a purity of 47%
(contaminated mainly by 55Co with traces of 52Mn and 51Cr ions) and an average
intensity of $3\times 10^{5}$ 56Ni particles per second. The 56Ni beam
impinged on a 10.7(8)-mg/cm2 thick deuterated polyethylene target, (CD2)n,
which was surrounded by the GRETINA detectors [13] positioned in two rings at
laboratory angles of $58^{\circ}$ and $90^{\circ}$ with respect to the beam
direction. Beam-like residues were collected and analyzed with the S800
spectrograph [17] positioned at $0^{\circ}$ scattering angle. The analysis
line to the S800 was operated in achromatic mode to obtain a total acceptance
of nearly 100% for 57Cu29+ and 50(10)% for 57Ni28+ run settings (the lower
value for 57Ni reflects the need to block out scattered 56Ni28+ ions from part
of the focal plane). Using the S800, we measured the intensity of
56Ni28+,27+,26+ species after the CD2 target relative to the number of
incident 56Ni28+ beam ions, where we observed that 80% of the incident ions
emerged in the fully-stripped charge state. Given the similar properties of
energy, mass, and nuclear charge of the 56Ni beam and heavy residues 57Cu and
57Ni, we estimated an 80(5)% and 40(11)% collection and detection efficiency
for 57Cu29+ and 57Ni28+ ions, respectively. To account for reactions on carbon
producing the residues of interest, measurements were also performed for
approximately half the duration of the CD2 runs with an 8.8(15)-mg/cm2 thick
(CH2)n target.

Figure 1:  Doppler-reconstructed $\gamma$-ray spectra gated in coincidence
with (a) 57Cu and (b) 57Ni recoils ($v/c\approx 0.25$). The spectra were
produced using the CD2 data minus the (scaled) CH2 data. The location of known
states are indicated.

The data acquisition system was triggered either by a residue-$\gamma$
coincidence or downscaled (by a factor of 5) residue singles events. The
GRETINA absolute singles efficiency was calibrated with 56Co and 152Eu sources
as described in ref. [18]. An efficiency of 5.5% for $E_{\gamma}=1332$ keV was
achieved for the nine-module setup employed here. The in-beam GRETINA
efficiency was boosted by a factor 1.06(5) compared to a stationary source.
Doppler-adjusted $\gamma$-ray energy spectra gated on 57Cu and 57Ni ions are
shown in Figs. 1a) and b), respectively. Events induced by reactions on carbon
atoms have already been subtracted, following the procedure in refs. [14, 15].
In the 57Cu spectrum, two strong peaks are observed at energies of 1028(1) and
1109(2) keV, which are assigned to the $\gamma$ decays to the ground state of
the first and second excited states at 1028(4) and 1106(4) keV observed by
Zhou et al. in a study of the 1H(58Ni,57Cu)2$n$ reaction [10]. There is no
evidence for the $\gamma$ decay at 2398(10) keV reported in the same study, or
for previously unobserved $\gamma$ decays from the excited state at 2525(17)
keV [19, 20]. In Fig. 1b, three peaks are observed at energies of 768(1),
1122(5), and 2579(4) keV. The first two are assigned to the known decays of
the $5/2^{-}$ and $1/2^{-}$ levels at 769 and 1113 keV in 57Ni [8, 9],
respectively. The third peak is assigned to the known decay of the $7/2^{-}$
state at 2578 keV [8]. Angle-integrated cross sections for the four lowest
excited states in 57Cu and 57Ni are shown in Table 1 assuming 100% $\gamma$
branches to the ground states (consistent with nuclear compilation data for
57Ni [8]), and insignificant $\gamma$-decay feeding from higher-lying excited
states (there are no other $\gamma$-ray lines observed). The overall errors
for the cross sections in Table 1 were obtained from a combination of
statistical uncertainties for individual transitions, and an uncertainty of
20% (28%) for 57Cu (57Ni) estimated by combining contributions to the
uncertainty from target thicknesses, Doppler-corrected $\gamma$-ray detection
efficiency, residual charge-state distributions, and the momentum acceptance,
in quadrature.

Table 1:  Angle-integrated experimental ($\sigma_{\rm exp}$) and theoretical
($\sigma_{\rm th}$) reaction cross sections and derived spectroscopic factors
($C^{2}S_{(d,n)}$, $C^{2}S_{(d,p)}$). Comparisons are made with shell-model
(SM) calculations. 56Ni($d$, $n$)57Cu
---
$E_{\rm ex}$ | $J^{\pi}$ | $\ell$ |  $\sigma_{\rm exp}$ (mb) |  $\sigma_{\rm th}$ (mb) | $C^{2}S_{(d,n)}$ | $C^{2}S_{\rm SM}$
1.028 |  5/2- | 3 | 2.00(40) | 2.62 | 0.76(28) | 0.75
1.109 |  1/2- | 1 | 0.28(6) | 0.45 | 0.62(22) | 0.71
2.398 |  5/2- | 3 |  $<$0.2 | 2.61 | $<8\times 10^{-2}$ | $1.8\times 10^{-3}$
2.525 |  7/2- | 3 |  $<$0.2 | 14.5 | — | $3.9\times 10^{-2}$
56Ni($d$, $p$)57Ni
$E_{\rm ex}$ | $J^{\pi}$ | $\ell$ |  $\sigma_{\rm exp}$ (mb) |  $\sigma_{\rm th}$ (mb) | $C^{2}S_{(d,p)}$ | $C^{2}S_{\rm SM}$
0.768 |  5/2- | 3 | 2.10(60) | 2.77 | 0.77(31) | 0.74
1.122 |  1/2- | 1 | 0.50(15) | 0.68 | 0.73(31) | 0.69
2.443 |  5/2- | 3 |  $<$0.4 | 2.61 | $<0.1$ | $3\times 10^{-4}$
2.579 |  7/2- | 3 | 1.24(36) | 14.9 | $8(3)\times 10^{-2}$ | $4.1\times 10^{-2}$

The theoretical angle-integrated single-particle ($C^{2}S=1$) cross sections
for the 56Ni($d$,$n$)57Cu and 56Ni($d$,$p$)57Cu reactions shown in Table 1
were calculated using the finite-range adiabatic approximation [21], which
incorporates deuteron breakup. A laboratory beam energy of 32 MeV/u was used
corresponding to the approximate center-of-target energy. Nucleon-target
interactions used the CH89 [22] optical potential and the nucleon-nucleon
interaction from ref. [23]. The $n(p)\,+^{56}$Ni final states were described
by a real Woods-Saxon potential with central and spin-orbit terms. The radius
and diffuseness of these potentials were set to 1.23 fm and 0.67 fm,
respectively. The spin-orbit term was given the standard depth of $V_{\rm
so}=6$ MeV. For the ($d$,$p$) calculations, the central potential depth was
adjusted to reproduce the final bound-state binding energies. For the
($d$,$n$) calculations, the final states are resonances. Here, we applied a
bound-state approximation in which we adjusted the depth of the central
potential to produce a final state bound by just $E=0.001$ MeV, as was done in
[14]. For low-lying resonances, this approximation introduces less than 1%
error, but for the two higher-lying resonances, this approximation can
introduce an error of $\sim\\!6$%. Repeating the calculation with the
Becchetti-Greenlees potential [24] changed the total cross section by about
15%, dominating the error introduced by the bound-state approximation. Based
on this and other studies [25], we estimate an error of up to 30% in the total
cross section calculations. The effective adiabatic potentials for ($d$,$p$)
and ($d$,$n$) were computed with twofnr [26] and the transfer calculations
were performed with the reaction code, fresco [27]. Shell-model wavefunctions
for 56,57Ni and 57Cu were obtained in the full $pf$ model space with the
GPFX1A Hamiltonian [28] for the isospin-conserving strong interaction plus the
Coulomb and charge-dependent Hamiltonians from ref. [29]. The spectroscopic
factors (shown in Table 1) were derived from the overlap of these
wavefunctions.

Table 2:  Resonance parameters used in the 56Ni($p$, $\gamma$)57Cu reaction rate calculation. See the text for details.  $E_{\rm ex}$ (keV) |  $E_{\rm r}$ (keV) | $J^{\pi}$ |  $\Gamma_{p}$ (eV) |  $\Gamma_{\gamma}$ (eV) |  $\omega\gamma$ (eV)
---|---|---|---|---|---
1028(1) | 338 |  5/2- | $5.7\times 10^{-12}$ | $1.9\times 10^{-4}$ | $1.7\times 10^{-11}$
1108(2) | 418 |  1/2- | $1.9\times 10^{-7}$ | $8.6\times 10^{-3}$ | $1.9\times 10^{-7}$
2398(10) | 1708 |  5/2- | $5.5\times 10^{-3}$ | $9.0\times 10^{-3}$ | $1.0\times 10^{-2}$
2525(17) | 1835 |  7/2- | $5.3\times 10^{-1}$ | $6.8\times 10^{-3}$ | $2.7\times 10^{-2}$

Experimental spectroscopic factors for the two lowest-lying excited states in
57Cu are reported here for the first time. The values agree very well with
shell-model calculations and are consistent with strong single particle
states. A level hierarchy with the $5/2^{-}$ level below the $1/2^{-}$ is
confirmed by the relative difference in cross sections for these states, which
are consistent with our reaction theory calculations. Previously this
hierarchy had been assumed largely on the basis of mirror energy level shift
arguments [10, 12]. This relative difference in cross section is also
reflected for the analog states in 57Ni. The spectroscopic factors for these
states in 57Ni are found to be very similar to the analog states in 57Cu, and
show no evidence of significant isospin symmetry breaking. The 57Ni $C^{2}S$
values are lower, and more precise, than those first reported by Rehm et al.
[9], but agree well within errors. A clear difference between the spectra in
Fig. 1 is the strong presence of the decay of the $7/2^{-}$ state in 57Ni
which is not present for 57Cu. From this we can infer that proton decay
represents the dominant branch of the $7/2^{-}$ state in 57Cu (we therefore do
not present a $C^{2}S$ value for this state in 57Cu in Table 1). The
$C^{2}S(\ell=3)$ value for the $7/2^{-}$ state in 57Ni represents a first
measurement (rather than an upper limit) and is an interesting test of the
shell-model calculations. It is found to be much weaker in single particle
strength than the lower-lying states and is broadly consistent with the shell-
model prediction. The shell-model calculations suggest the neighboring
$5/2^{-}$ state has a smaller single-particle strength consistent with its
non-observation here in both the 57Ni and 57Cu data. Interestingly, the
$\gamma$ decay of this state was clearly observed in the 1H(58Ni,57Cu)2$n$
reaction [10], which suggests the $\gamma$ branch for this state in 57Cu is at
least comparable to the proton branch, otherwise it would not have been seen
in that study.

Table 2 shows resonance parameters for the four lowest-lying excited states in
57Cu used to calculate the 56Ni($p$,$\gamma$)57Cu astrophysical reaction rate.
Shell-model calculations of the $\gamma$-decay widths used the M1 and E2
effective operators from ref. [30]. The uncertainty in the $\gamma$-decay
widths is about a factor of two [30]. Proton-decay widths were calculated from
$\Gamma=C^{2}S\cdot P\cdot 2W$, where $P$ are the penetration factors obtained
from the Coulomb wavefunction, and $W$ is the Wigner single-particle width.
The $\ell$-dependent radius for the evaluation of $P$ was chosen to reproduce
the single-particle width obtained for proton scattering from a Woods-Saxon
potential. The Woods-Saxon parameters were consistent with the geometry used
in the cross section calculations and were chosen to reproduce the proton
separation energy of 56Ni to 55Co and the rms charge radius of 58Ni. The
potential depth for the scattered proton was adjusted to give a resonance
$Q$-value of 1 MeV. The uncertainty in these calculations of the single-
particle proton-decay widths is $\sim\\!20$%. A proton separation energy of
690.3(4) keV [31] was used for 57Cu to calculate the resonance energies, which
incorporate the new more precise excitation energy values reported here for
the two lowest-lying excited states. The proton widths for these states were
calculated for the first time using the experimentally constrained $C^{2}S$
values obtained for 57Cu in the present work. For the $7/2^{-}$ state we
derive the proton width value using the $C^{2}S(\ell=3)$ value reported here
for the first time for the analog state in 57Ni. For the proton width of the
2398-keV $5/2^{-}$ level, we take the $C^{2}S$ value from the shell-model
calculation which is compatible with the experimental observational limit in
the present 57Ni data (see Table 1).

Figure 2:  56Ni($p$,$\gamma$) astrophysical reaction rate and uncertainty
bounds calculated using the resonance parameters listed in Table 2 (shown in
orange). The total rates from refs. [7, 10, 9] are shown for comparison;
additionally, as the uncertainty bounds were presented in ref. [10], these are
also depicted. See text for details.

The present 56Ni($p$,$\gamma$)57Cu reaction rate and uncertainty bounds
incorporating our new experimental results and shell-model calculations is
shown in Fig. 2. Our rate for $T<1$ GK is dominated by the $\ell=1$ capture
resonance at 417.7(1.8) keV for which we derive a resonance strength,
$\omega\gamma=1.9(8)\times 10^{-7}$ eV. For the region with $T>1$ GK, both the
$\ell=3$ captures on the higher-lying $5/2^{-}$ and $7/2^{-}$ states are found
to contribute to, and collectively dominate, the reaction rate. The $5/2^{-}$
state is the only lower-lying resonance for which we have only an experimental
upper limit on the spectroscopic factor. If this value is significantly lower
than the shell-model calculation this would reduce its strength/significance
relative to the $7/2^{-}$ state. A much higher value for $C^{2}S$, and
therefore the proton partial width, is deemed unlikely as $\gamma$ decay has
been observed from this $5/2^{-}$ state in 57Cu [10]. Figure 2 also shows some
previous reaction rate calculations for comparison [7, 10, 9]. At lower
temperatures, the van Wormer theoretical calculation is orders of magnitude
lower and reflects the then unknown resonance energies in 57Cu. The later rate
of Zhou et al. used known experimental energies but the proton and $\gamma$
widths were entirely estimated from theory; the large reaction rate
difference, particularly for $T<1$ GK, is caused by the relatively low proton
partial width derived for the $1/2^{-}$ resonance in that estimate [10]. The
rate of Rehm et al. [9] incorporates the results for the spectroscopic factors
for the two lowest-lying excited states in 57Ni. This central rate agrees well
at low temperatures, as the same proton width and resonance strength values
are obtained for the $1/2^{-}$ state. One should note that the spectroscopic
factor used for the calculation in ref. [9] was based on the analog state in
57Ni, and had a higher $C^{2}S$ value (and uncertainty) than that measured
directly in the present study on 57Cu. For the two higher-lying resonance
contributions refs. [9] and [10] adopted the same partial width values so
their total rates for $T>1$ GK are nearly the same. Our result is
significantly lower in this region, reflecting the lower partial widths
obtained (particularly for the $5/2^{-}$ level) in the present shell-model
calculations.

In summary, we report angle-integrated cross-section measurements for the
56Ni($d$,$n$)57Cu and 56Ni($d$,$p$)57Ni transfer reactions. Comparisons with
reaction theory calculations allow definitive assignments of the first two
excited states in 57Cu to their analogs in the mirror partner, 57Ni. First
measurements of the spectroscopic factors for these two states in 57Cu show
they have a strong single-particle character with values agreeing well with
the new shell-model calculations obtained using a full $pf$ model space with
the GPFX1A interaction plus Coulomb and charge-dependent Hamiltonians. From a
comparison with their analog states in 57Ni we find no evidence for
significant isospin symmetry breaking effects. The spectroscopic factor of a
high-lying $7/2^{-}$ state is determined for the first time in 57Ni and is
found to have a much weaker single particle character in reasonable agreement
with the new shell-model calculations. We use these new results to re-evaluate
and significantly constrain the 56Ni($p$,$\gamma$)57Cu astrophysical reaction
rate required for modeling of explosive burning in x-ray bursts where the
astrophysical $rp$-process is thought to occur.

## Acknowledgments

This work was supported by the U.S. National Science Foundation (NSF) under
Cooperative Agreement No. PHY-1102511 (NSCL) and No. PHY-1430152 (JINA-CEE).
GRETINA was funded by the DOE, Office of Science. Operation of the array at
NSCL was supported by the DOE under Grants No. DE-SC0014537 (NSCL) and No. DE-
AC02-05CH11231 (LBNL). BAB kindly acknowledges funding from an NSF Grant No.
PHY-1811855. The Edinburgh group is appreciative of funding from the UK STFC
under Grant No. ST/L005824/1.

## References

  * Hinke  _et al._ [2012] C. B. Hinke, M. Böhmer, P. Boutachkov, T. Faestermann, H. Geissel, J. Gerl, R. Gernhäuser, M. Górska, A. Gottardo, H. Grawe, J. L. Grȩbosz, R. Krücken, N. Kurz, Z. Liu, L. Maier, F. Nowacki, S. Pietri, Z. Podolyák, K. Sieja, K. Steiger, K. Straub, H. Weick, H.-J. Wollersheim, P. J. Woods, N. Al-Dahan, N. Alkhomashi, A. Ataç, A. Blazhev, N. F. Braun, I. T. Čeliković, T. Davinson, I. Dillmann, C. Domingo-Pardo, P. C. Doornenbal, G. de France, G. F. Farrelly, F. Farinon, N. Goel, T. C. Habermann, R. Hoischen, R. Janik, M. Karny, A. Kaşkaş, I. M. Kojouharov, T. Kröll, Y. Litvinov, S. Myalski, F. Nebel, S. Nishimura, C. Nociforo, J. Nyberg, A. R. Parikh, A. Procházka, P. H. Regan, C. Rigollet, H. Schaffner, C. Scheidenberger, S. Schwertel, P.-A. Söderström, S. J. Steer, A. Stolz, and P. Strmeň, Nature 486, 341 (2012).
  * Auranen  _et al._ [2018] K. Auranen, D. Seweryniak, M. Albers, A. D. Ayangeakaa, S. Bottoni, M. P. Carpenter, C. J. Chiara, P. Copp, H. M. David, D. T. Doherty, J. Harker, C. R. Hoffman, R. V. F. Janssens, T. L. Khoo, S. A. Kuvin, T. Lauritsen, G. Lotay, A. M. Rogers, J. Sethi, C. Scholey, R. Talwar, W. B. Walters, P. J. Woods, and S. Zhu, Phys. Rev. Lett. 121, 182501 (2018).
  * Otsuka  _et al._ [1998] T. Otsuka, M. Honma, and T. Mizusaki, Phys. Rev. Lett. 81, 1588 (1998).
  * Yurkewicz  _et al._ [2004] K. L. Yurkewicz, D. Bazin, B. A. Brown, C. M. Campbell, J. A. Church, D. C. Dinca, A. Gade, T. Glasmacher, M. Honma, T. Mizusaki, W. F. Mueller, H. Olliver, T. Otsuka, L. A. Riley, and J. R. Terry, Phys. Rev. C 70, 054319 (2004).
  * Cocolios  _et al._ [2009] T. E. Cocolios, A. N. Andreyev, B. Bastin, N. Bree, J. Büscher, J. Elseviers, J. Gentens, M. Huyse, Y. Kudryavtsev, D. Pauwels, T. Sonoda, P. van den Bergh, and P. van Duppen, Phys. Rev. Lett. 103, 102501 (2009).
  * Wallace and Woosley [1981] R. K. Wallace and S. E. Woosley, Astrophys. J. Supp. 45, 389 (1981).
  * van Wormer  _et al._ [1994] L. van Wormer, J. Görres, C. Iliadis, M. Wiescher, and F.-K. Thielemann, Astrophys. J. 432, 326 (1994).
  * Bhat [1998] M. R. Bhat, Nucl. Data Sheets 85, 415 (1998).
  * Rehm  _et al._ [1998] K. E. Rehm, F. Borasi, C. L. Jiang, D. Ackermann, I. Ahmad, B. A. Brown, F. Brumwell, C. N. Davids, P. Decrock, S. M. Fischer, J. Görres, J. Greene, G. Hackmann, B. Harss, D. Henderson, W. Henning, R. V. Janssens, G. McMichael, V. Nanal, D. Nisius, J. Nolen, R. C. Pardo, M. Paul, P. Reiter, J. P. Schiffer, D. Seweryniak, R. E. Segel, M. Wiescher, and A. H. Wuosmaa, Phys. Rev. Lett. 80, 676 (1998).
  * Zhou  _et al._ [1996] X. G. Zhou, H. Dejbakhsh, C. A. Gagliardi, J. Jiang, L. Trache, and R. E. Tribble, Phys. Rev. C 53, 982 (1996).
  * Jiang  _et al._ [2009] C. L. Jiang, K. E. Rehm, D. Ackermann, I. Ahmad, J. P. Greene, B. Harss, D. Henderson, W. F. Henning, R. V. F. Janssens, J. Nolen, R. C. Pardo, P. Reiter, J. P. Schiffer, D. Seweryniak, A. Sonzogni, J. Uusitalo, I. Wiedenhöver, A. H. Wuosmaa, F. Brumwell, G. McMichael, M. Paul, and R. E. Segel, Phys. Rev. C 80, 044613 (2009).
  * Forstner  _et al._ [2001] O. Forstner, H. Herndl, H. Oberhummer, H. Schatz, and B. A. Brown, Phys. Rev. C 64, 045801 (2001).
  * Paschalis  _et al._ [2013] S. Paschalis, I. Y. Lee, A. O. Macchiavelli, C. M. Campbell, M. Cromaz, S. Gros, J. Pavan, J. Qian, R. M. Clark, H. L. Crawford, D. Doering, P. Fallon, C. Lionberger, T. Loew, M. Petri, T. Stezelberger, S. Zimmermann, D. C. Radford, K. Lagergren, D. Weisshaar, R. Winkler, T. Glasmacher, J. T. Anderson, and C. W. Beausang, Nucl. Instrum. Methods Phys. Res. A 709, 44 (2013).
  * Kankainen  _et al._ [2016] A. Kankainen, P. J. Woods, F. Nunes, C. Langer, H. Schatz, V. Bader, T. Baugher, D. Bazin, B. A. Brown, J. Browne, D. T. Doherty, A. Estrade, A. Gade, A. Kontos, G. Lotay, Z. Meisel, F. Montes, S. Noji, G. Perdikakis, J. Pereira, F. Recchia, T. Redpath, R. Stroberg, M. Scott, D. Seweryniak, J. Stevens, D. Weisshaar, K. Wimmer, and R. Zegers, Eur. Phys. J. A 52, 6 (2016).
  * Kankainen  _et al._ [2017] A. Kankainen, P. J. Woods, H. Schatz, T. Poxon-Pearson, D. T. Doherty, V. Bader, T. Baugher, D. Bazin, B. A. Brown, J. Browne, A. Estrade, A. Gade, J. José, A. Kontos, C. Langer, G. Lotay, Z. Meisel, F. Montes, S. Noji, F. Nunes, G. Perdikakis, J. Pereira, F. Recchia, T. Redpath, R. Stroberg, M. Scott, D. Seweryniak, J. Stevens, D. Weisshaar, K. Wimmer, and R. Zegers, Phys. Lett. B 769, 549 (2017).
  * Morrissey  _et al._ [2003] D. J. Morrissey, B. M. Sherrill, M. Steiner, A. Stolz, and I. Wiedenhoever, Nucl. Instrum. Methods Phys. Res. B 204, 90 (2003).
  * Bazin  _et al._ [2003] D. Bazin, J. A. Caggiano, B. M. Sherrill, J. Yurkon, and A. Zeller, Nucl. Instrum. Methods Phys. Res. B 204, 629 (2003).
  * Weisshaar  _et al._ [2017] D. Weisshaar, D. Bazin, P. C. Bender, C. M. Campbell, F. Recchia, V. Bader, T. Baugher, J. Belarge, M. P. Carpenter, H. L. Crawford, M. Cromaz, B. Elman, P. Fallon, A. Forney, A. Gade, J. Harker, N. Kobayashi, C. Langer, T. Lauritsen, I. Y. Lee, A. Lemasson, B. Longfellow, E. Lunderberg, A. O. Macchiavelli, K. Miki, S. Momiyama, S. Noji, D. C. Radford, M. Scott, J. Sethi, S. R. Stroberg, C. Sullivan, R. Titus, A. Wiens, S. Williams, K. Wimmer, and S. Zhu, Nucl. Instrum. Methods Phys. Res. A 847, 187 (2017).
  * Stiliaris  _et al._ [1987] E. Stiliaris, H. G. Bohlen, X. S. Chen, B. Gebauer, A. Miczaika, W. von Oertzen, W. Weller, and T. Wilpert, Z. Phys. A 326, 139 (1987).
  * Jokinen  _et al._ [2002] A. Jokinen, A. Nieminen, J. Äystö, R. Borcea, E. Caurier, P. Dendooven, M. Gierlik, M. Górska, H. Grawe, M. Hellström, M. Karny, Z. Janas, R. Kirchner, M. La Commara, G. Martinez-Pinedo, P. Mayet, H. Penttilä, A. Plochocki, M. Rejmund, E. Roeckl, M. Sawicka, C. Schlegel, K. Schmidt, and R. Schwengner, Eur. Phys. J. Direct 4, 1 (2002).
  * Johnson and Tandy [1974] R. C. Johnson and P. C. Tandy, Nucl. Phys. A 235, 56 (1974).
  * Varner  _et al._ [1991] R. L. Varner, W. J. Thompson, T. L. McAbee, E. J. Ludwig, and T. B. Clegg, Phys. Rep. 201, 57 (1991).
  * Reid [1968] R. V. Reid, Jr., Ann. Phys. 50, 411 (1968).
  * Becchetti and Greenlees [1969] F. D. Becchetti and G. W. Greenlees, Physical Review 182, 1190 (1969).
  * Nunes  _et al._ [2011] F. M. Nunes, A. Deltuva, and J. Hong, Phys. Rev. C 83, 034610 (2011).
  * [26] M. Tostevin, M. Igarashi, and N. Kishida, “University of Surrey modified version of the code TWOFNR,” Private Communication.
  * Thompson [1988] I. J. Thompson, Comput. Phys. Rep. 7, 167 (1988).
  * Honma  _et al._ [2005] M. Honma, T. Otsuka, B. A. Brown, and T. Mizusaki, Eur. Phys. J. A Supp. 25, 499 (2005).
  * Ormand and Brown [1989] W. E. Ormand and B. A. Brown, Nucl. Phys. A 491, 1 (1989).
  * Honma  _et al._ [2004] M. Honma, T. Otsuka, B. A. Brown, and T. Mizusaki, Phys. Rev. C 69, 034335 (2004).
  * Wang  _et al._ [2017] M. Wang, G. Audi, F. G. Kondev, W. J. Huang, S. Naimi, and X. Xu, Chin. Phys. C 41, 030003 (2017).


# Unsupervised CD in satellite image time series by contrastive learning and
feature tracking

Yuxing Chen, Lorenzo Bruzzone, Y. Chen and L. Bruzzone are with the Department
of Information Engineering and Computer Science, University of Trento, 38122
Trento, Italy
(e-mail:yuxing.chen@unitn.it;lorenzo.bruzzone@unitn.it).Corresponding author:
L. Bruzzone

###### Abstract

While unsupervised change detection using contrastive learning has been
significantly improved the performance of literature techniques, at the
present, it only focuses on the bi-temporal change detection scenario.
Previous state-of-the-art models for image time-series change detection often
use features obtained by learning for clustering or training a model from
scratch using pseudo labels tailored to each scene. However, these approaches
fail to exploit the spatial-temporal information of image time-series or
generalize to unseen scenarios. In this work, we propose a two-stage approach
to unsupervised change detection in satellite image time-series using
contrastive learning with feature tracking. By deriving pseudo labels from
pre-trained models and using feature tracking to propagate them among the
image time-series, we improve the consistency of our pseudo labels and address
the challenges of seasonal changes in long-term remote sensing image time-
series. We adopt the self-training algorithm with ConvLSTM on the obtained
pseudo labels, where we first use supervised contrastive loss and contrsative
random walks to further improve the feature correspondence in space-time. Then
a fully connected layer is fine-tuned on the pre-trained multi-temporal
features for generating the final change maps. Through comprehensive
experiments on two datasets, we demonstrate consistent improvements in
accuracy on fitting and inference scenarios.

###### Index Terms:

Contrastive Learning, Feature Tracking, Multi-temporal, Change Detection,
Remote Sensing.

##  I Introduction

Detection of changes in multi-temporal remote sensing (RS) images has been
extensively studied in the post decades [1]. Early approaches to change
detection in bi-temporal RS images include image algebra, image transformation
and image classification methods [2]. These methods have limitations, such as
relying on empirical feature extraction algorithms or being sensitive to
classification results, which limit their application in change detection.
Image algebra methods directly compare image values, such as in the case of
change vector analysis (CVA)-based methods [2, 3, 4, 5] that provide spectral
change information in terms of magnitude and direction of the spectral change
vectors. On the other hand, image transformation methods map images into the
same feature space for comparison. The most common transformation methods
include principal component analysis (PCA) [6], slow feature analysis (SFA)
[7], and canonical correlation analysis (CCA) [8]. Supervised image
classification methods project image values into different classes at each
date and comparedirectly class labels. This approach, known as post-
classification change detection [7], is widely used in large-scale land-cover
change detection. In general, image algebra and transformation methods heavily
rely on empirical feature extraction algorithms, while post-classification
methods are sensitive to the classification results of each image and to error
propagation. These limitations hinder the application of conventional change
detection methods.

Deep learning methods have been shown to significantly improve the performance
of conventional change detection methods by using deep neural networks [9] and
stochastic gradient descent [10]. One common approach is direct
classification, where models are trained using pre-defined labels and then
used to classify change and unchanged pixels. For example, Rodrigo et al. [11]
presented three Unet-based convolution neural network (CNN) architectures for
detecting binary changes between pairs of registered RGB images. In the
absence of ground truth, pseudo labels from conventional change detection
methods can be used to train models in a self-training paradigm. Zhou et al.
[12] proposed a self-training algorithm based on pseudo labels for change
detection, where the pseudo labels are generated by the traditional CVA
approach and used to train a new network end-to-end. The image transformation
approach has also been improved using deep learning, where deep neural
networks are utilized to extract discriminative features. Many new techniques
have been developed for extracting discriminative features from bi-temporal RS
images, such as generative [13] and discriminative [14] models. For the
generative model, Luppino et al. [15] combined domain-specific affinity
matrices and autoencoders (AEs) to align related pixels from multimodal
images. Chen et al. [16, 17] explored the use of discriminative models in
change detection, proposing the use of contrastive learning at pixel-level and
patch-level in multi-temporal and multisensor scenarios. The application of
deep learning in post-classification change detection can follow two main
directions. One is to use a deep learning-based segmentation approach to
classify the object of interest on bi-temporal images and then compare them.
For example, Nemoto et al. [18] first segmente buildings in an urban area and
then compare the building maps at two different times to detect changes.
Another approach is to perform binary change detection and segmentation of
both images simultaneously. Ding et al. [19] proposed combining post-
classification and direct classification methods using a bi-temporal semantic
reasoning network, where the network produces both a change map and two
classification maps. These approaches demonstrate the ability of deep learning
in deriving changes from image pairs.

The challenge of detecting changes in remote sensing (RS) images time-series
is compounded by the presence of seasonal noise, which can be difficult to
distinguish from true changes. One approach to addressing this challenge is to
use graph-based methods [20], which present detected spatiotemporal phenomena
as evolution graphs composed of spatiotemporal entities belonging to the same
geographical location in multiple timestamps. Deep learning methods have also
been applied to RS image time-series change detection, using techniques such
as recurrent neural networks (RNNs) [21] to extract discriminative features
from image sequences. However, supervised methods often require a large number
of labelled training samples, which can be difficult to obtain for long image
time-series. In this context, self-training approaches such as self-supervised
and pseudo-label learning have become popular, where networks are trained on a
pretext task such as image restoration using 3D CNN [22, 23] and predict the
correct order of shuffled image sequences [24]. For example, Kalincheva et al.
[25] proposed a framework combining a graph model and pseudo labels, which
associates changes in consecutive images with different spatial objects using
a gated recurrent unit (GRU) AE-based model. Meshkini et al. [22] further
proposed the use of a pre-trained 3D CNN to extract spatial-temporal
information from long satellite image time-series, where they can detect the
times and locations of changes in image sequences. However, pseudo labels
often have a high level of noise and do not consider temporal information, and
the pre-trained model can not adapt to various changes.

In this work, we propose the use of contrastive learning [26] and feature
tracking [27] to address these challenges and improve the performance of
change detection in RS image time-series. We leverage contrastive learning
methods to get good pre-trained features for pseudo label generation and
reduce the overfitting that results in incorrect pseudo labels when
considering supervised contrastive learning [26] and contrastive random walks
[28]. Additionally, by incorporating a feature tracking-based pseudo label
generation task and a convolutional long short-term memory network (ConvLSTM)
[29], we are able to extract time-series change maps from image time-series
and further train a new model from scratch. In detail, the pseudo-label
generation is based on the pre-trained model using contrastive learning. The
change detection model is trained from pseudo labels by the joint use of Unet
[30] and ConvLSTM network. We first extract pseudo labels from change pair
time-series and then use them with images to train the proposed network, which
outputs change maps relative to the first image in the sequence. During the
training, supervised contrastive loss, contrastive random walk loss and cross-
entropy loss are used to optimize the parameters of the feature encoder and
the last classifier, respectively. The supervised contrastive loss is used to
mitigate the noise in pseudo labels, while the contrastive random walk loss
improves the quality of the consecutive change results. Finally, we
demonstrate the effectiveness of our approach on two data sets.

In this paper, we propose the following main novel contributions:

  * •

To generate time-related pseudo labels for network training, we propose to use
feature tracking to extract reliable change pixels in image sequences that are
insensitive to seasonal changes.

  * •

To ensure the robustness and consistency of change maps, we propose to use
supervised contrastive loss and contrastive random walk loss on change feature
learning. These losses encourage the pixels in the same class to have a closer
feature representation.

  * •

To extend the approach to arbitrary long time-series, we jointly use Unet and
ConvLSTM as the model architectures. To verify the performance of the proposed
approach, we provide a comparison with state-of-the-art methods and an
ablation study. Our experiments show that our method obtains competitive
results on the datasets.

The remainder of this paper is organized as follows. Section II introduces the
related works. In Section III, we introduce our proposed approach, including
the network architecture, the supervised contrastive loss, the contrastive
random walk loss and the feature tracking-based pseudo label generation. In
Section IV, we present the experimental results obtained on two datasets and
compare our approach to state-of-the-art methods. In this section, we also
include an ablation study in the discussion. Finally, in Section V, we draw
conclusions and discuss future work.

##  II Related Works

###  II-A Self-supervised Learning in Change Detection

Self-supervised learning is a method of representation learning that does not
require human intervention for data annotation, as opposed to supervised
learning. It has been successful in remote sensing image change detection,
thanks to its ability to obtain good representative features. Self-supervised
learning has two main streams: generative and discriminative methods [14].

Generative models often rely on autoencoders, generative adversarial networks
[31], and diffusion models [32]. Denoising autoencoders [33], a classical
generative self-supervised learning model, are a type of autoencoder that
reconstructs one temporal image from another. Bergamasco et al. [34] proposed
the use of a multilayer convolutional denoising autoencoder for unsupervised
change detection in multi-temporal Sentinel-1 images. In addition to
autoencoders, generative adversarial networks have also been used for change
detection tasks. For example, Gong et al. [35] treated change detection as a
generative learning procedure that connects bi-temporal images and generates
change maps. As for diffusion models, Gedara et al. [36] used a pre-trained
denoising diffusion probabilistic model to extract feature representations
from unlabeled remote sensing images, and then train a lightweight change
detection classifier to detect changes from the learned features.

In contrast, discriminative models are mostly based on contrastive learning,
which learns a representation that helps distinguishing one object from
another. The objective of contrastive loss is to learn a representation where
semantically similar features are brought closer together and dissimilar
features are pushed apart. Unlike fine-tuning on downstream tasks,
unsupervised change detection using well pre-trained features relies on
thresholding approaches and only considers bi-temporal images. Discriminative
models used in self-supervised change detection include pre-defined tasks and
contrastive methods. In [37], Leenstra et al. used predefined tasks for
feature representation learning and trained a discriminative model to extract
features from bi-temporal images for change detection. An early attempt to use
contrastive methods in change detection is [17], where the authors proposed
contrastive learning for change detection in multi-view remote sensing images
(including multi-temporal and multi-sensor images). In [16], they further
proposed a pixel-wise contrastive approach to distil the features to alleviate
seasonal effects in bi-temporal change detection. One challenge these works
face in using contrastive approaches to unsupervised pretraining is that they
push apart samples that should belong to the same class, making it harder for
the classifier to later categorize them correctly or create accurate decision
boundaries.

###  II-B Change Detection in RS Image Time-series Using ConvLSTM

Change detection is often associated with sequential data, making it necessary
to evaluate temporal dynamics. The computer vision community has addressed the
modelling of temporal relationships among features using recurrent neural
networks, which have proven effective for a wide range of applications such as
object tracking and action recognition. Long short-term memory networks (LSTM)
are particularly effective for such problems, as they mitigate the vanishing
gradient problem when dealing with long-term dependencies. The combination of
recurrent neural networks and deep learning architectures has also been used
for time-series tasks, aiming to produce more useful feature representations
by extracting both spatial and temporal information during the learning
process. Recent RS image time-series change detection tasks have extensively
integrated LSTM techniques. In [38], an LSTM is integrated into a CNN to
consider both spatial and temporal features in an end-to-end framework. Sefrin
et al. [39] proposed combining FCN and LSTM to study land-cover changes using
Sentinel-2 images. For high-resolution image change detection, Sun et al. [40]
proposed using atrous Unet-ConvLSTM to better model multiscale spatial
information. For unsupervised approaches, Saha et al. [24] treated change
detection as an anomaly detection problem, using an LSTM network to learn a
representation of the image time-series. In this method, they used a pretext
task of reordering the image sequence. However, the predefined task cannot
resist the influence of seasonal noise, which leads to many pseudo-changes in
the results. Some researchers have shown that pseudo-labels can help solve
this problem. Kalinicheva et al. [25] proposed a new framework that combines a
graph model and pseudo-labels, using a gated recurrent unit (GRU) AE-based
model to associate the changes of consecutive images with different spatial
objects. Yang et al. [41] proposed an unsupervised time-distance-guided
convolutional recurrent neural network (UTRnet) for change detection in
irregularly collected images, using a weighted pre-change detection to obtain
reliable training samples.

Figure 1: Overview of the proposed network architecture for RS image time-
series change detection. The proposed network is based on the Unet and Bi-
ConvLSTM. The change pairs are given as input to the Unet and then the output
features are given as input to the Bi-ConvLSTM for temporal relationship
modelling. At the end of the network, the output features of MLP are used as
the inputs to the supervised contrastive loss, contrastive random walk loss
and cross-entropy loss in the feature learning and finetuning stages.

##  III Methodology

In this section, we present the proposed two-stage RS image time-series change
detection framework. It includes a feature tracking-based pseudo label
generation module and a self-training change detection module that follows the
training setting of supervised contrastive learning. We first get the pixel-
wise feature representation of each image in the image sequence using the pre-
trained model [16] and then get the pseudo change maps using the thresholding
approach. Then, the feature tracking approach is used to get the final pseudo
change labels based on the feature representation and the threshold-based
change map. Afterwards, the pseudo change labels are used to learn the
representation of change maps using a supervised contrastive loss and the
contrastive random walk loss. Finally, a fully connected layer is fine-tuned
on learned change map representation using cross-entropy loss with a weighted
supervised contrastive loss for final generating the change maps . In the
following subsections, we will describe the network architecture of the
proposed framework, the supervised contrastive loss, the contrastive random
walk loss and the feature tracking-based pseudo label generation.

###  III-A Network Architecture

The proposed approach uses an Unet-ConvLSTM network architecture, which
consists of two components: ResUnet and Bi-ConvLSTM. For the Unet, we adopt a
similar architecture as the FC-Siam-conc [11]. It consists of two encoders,
one bridge, one decoder, and skip connections between the downsampling and
upsampling paths. The decoder part has three blocks, each of which consists of
a convolution layer (Conv), batch normalization (BN), ReLU, and upsampling. A
$1\times 1$ Conv is used after the last block to reconstruct the learned
representations. We changed the padding type of all blocks to ”same” padding.
The parameters and channel size of each unit are presented in Table I. Each
convolution unit ($[\cdot]$) includes a convolutional layer, a BN layer, and a
ReLU activation layer. Each residual block (ResBlk) in the encoding path has
two residual units, each of which consists of two convolution units and an
identity mapping.

TABLE I: Structure of the proposed network. | Encoder 1 & 2 | Decoder |
---|---|---|---
| Conv1
---
Maxpool
| [3$\times$3, 32], stride 2
---
3$\times$3, stride 2
| Cat. ResBlk2
---
[3$\times$3, 128]
upsampling 2
| DecBlk1
---
stride 1
| ResBlk1
---
stried 1
$\left[\begin{array}[]{l}3\times 3,~{}32\\\ 3\times 3,~{}32\end{array}\right]\times 2$ |  | Cat. ResBlk1
---
[3$\times$3, 128]
upsampling 2
| DecBlk2
---
stride 1
| ResBlk2
---
stride 2
$\left[\begin{array}[]{l}3\times 3,64\\\ 3\times 3,64\end{array}\right]\times 2$ |  | Cat. Conv1
---
[3$\times$3, 128]
upsampling 2
| DecBlk3
---
stride 1
| ResBlk3
---
stride 2
$\left[\begin{array}[]{l}3\times 3,128\\\ 3\times 3,128\end{array}\right]\times 2$ |  | [3$\times$3, 128]
---
Bi-ConvLSTM
| LSTM Blk
---
| Bridge
---
stride 1
| [3$\times$3, 256]
---
upsampling 2
| [1$\times$1, 8/2]
---
| MLP
---

The output features of time-series change pairs are given in input the Bi-
ConvLSTM layer. Differently from the standard LSTM, ConvLSTM uses convolution
operations in the input-to-state and state-to-state transitions to improve the
modeling of the spatial correlation among sequence images. It consists of an
input gate $i_{t}$, an output gate $o_{t}$, a forget gate $f_{t}$, and a
memory cell $\mathcal{C}_{t}$. The input, output and forget gates act as
controlling gates to access, update, and clear memory cell. ConvLSTM can be
formulated as follows (for convenience we remove the subscript and subscript
from the parameters):

$\begin{array}[]{l}i_{t}=\sigma\left(\mathbf{W}_{xi}*\mathcal{X}_{t}+\mathbf{W}_{hi}*\mathcal{H}_{t-1}+\mathbf{W}_{ci}*\mathcal{C}_{t-1}+b_{i}\right)\\\
f_{t}=\sigma\left(\mathbf{W}_{xf}*\mathcal{X}_{t}+\mathbf{W}_{hf}*\mathcal{H}_{t-1}+\mathbf{W}_{cf}*\mathcal{C}_{t-1}+b_{f}\right)\\\
\mathcal{C}_{t}=f_{t}\circ\mathcal{C}_{t-1}+i_{t}\tanh\left(\mathbf{W}_{xc}*\mathcal{X}_{t}+\mathbf{W}_{hc}*\mathcal{H}_{t-1}+b_{c}\right)\\\
o_{t}=\sigma\left(\mathbf{W}_{xo}*\mathcal{X}_{t}+\mathbf{W}_{ho}*\mathcal{H}_{t-1}+\mathbf{W}_{co}\circ\mathcal{C}_{t}+b_{c}\right)\\\
\mathcal{H}_{t}=o_{t}\circ\tanh\left(\mathcal{C}_{t}\right)\end{array}$

where $*$ and $o$ denote the convolution and Hadamard functions, respectively.
$\mathcal{X}_{t}$ is the input tensor, $\mathcal{H}_{t}$ is the hidden state
tensor, and, $\mathcal{W}_{x*}$ and $\mathcal{W}_{h*}$ are 2D convolution
kernels corresponding to the input and hidden state, respectively, and
$b_{i}$, $b_{f}$, $b_{o}$ and $b_{c}$ are the bias terms.

In this study, we employ Bi-ConvLSTM [42, 43] to encode the features of time-
series change pairs. It was proposed to use both past and future information
to model sequential data. Bi-ConvLSTM uses two ConvLSTMs to process the input
data in both forward and backward directions, and then makes a decision for
the current input by taking into account the data dependencies in both
directions. It has been shown that analyzing both forward and backward
temporal perspectives improves predictive performance. Each of forward and
backward ConvLSTM can be considered as a standard one, with two sets of
parameters for backward and forward states. The output of Bi-ConvLSTM is
calculated as follows:

$\mathbf{Y}_{t}=\tanh\left(\mathbf{W}_{y}^{\overrightarrow{\mathcal{H}}}*\overrightarrow{\mathcal{H}}_{t}+\mathbf{W}_{y}^{\overleftarrow{\mathcal{H}}}\overleftarrow{\mathcal{H}}_{t}+b\right)$
(1)

where $\overrightarrow{\mathcal{H}}_{t}$ and $\overleftarrow{\mathcal{H}}_{t}$
denote the hidden state tensors for forward and backward states, respectively,
$b$ is the bias term, and $\mathbf{Y}_{t}$ indicates the final output
considering bidirectional spatio-temporal information. The hyperbolic tangent
(tanh) is used to combine the output of both forward and backward states in a
non-linear manner. After the last layer of Bi-ConvLSTM, an MLP block is used
to reconstruct output features at the feature learning stage and predict the
binary change maps at the finetuning stage.

###  III-B Loss Function

During training, each image $I_{i}(i>0)$ is used to construct a change pair
anchored at the initial image ($I_{i=0}$). The proposed network then predicts
the change features of each change pair and the final change map, capturing
the temporal changes related to the first image rather than the cumulated
changes of the image sequence. The training process uses a teacher-student
paradigm and the exponential moving average (EMA) algorithm [44]. The input of
the student network are the original time-series image pairs, while the
teacher network uses the same time-series image pairs with color jitter.

According to supervised contrastive learning, the training process consists of
the feature learning and finetuning stages. In the feature learning phase, we
use supervised contrastive loss with the contrastive random walk loss. The
loss can be written as:

$\mathcal{L}_{\text{feat }}=\mathcal{L}_{sc}+\lambda_{1}\mathcal{L}_{crw}$ (2)

In the finetuning stage, we use the cross-entropy loss with the weighted
supervised contrastive loss. The total loss is calculated as:

$\mathcal{L}_{\text{map }}=\mathcal{L}_{ce}+\lambda_{2}\mathcal{L}_{sc}$ (3)

The $\mathcal{L}_{ce}$ is the cross-entropy loss, the $\mathcal{L}_{sc}$ is
the supervised contrastive loss, and $\mathcal{L}_{crw}$ is the contrastive
random walk loss. The hyper-parameters $\lambda_{1}$ and $\lambda_{2}$ are
used to tune the losses, and values of $\lambda_{1}=0.1$ and
$\lambda_{2}=0.0005$ generally performed well in our experiments. In the
following, we provide details on the supervised contrastive loss and the
contrastive random walk loss.

####  III-B1 Contrastive loss

The proposed approach uses a supervised contrastive loss [26] to differentiate
representations between changed and unchanged pixels in time-series change
pairs. This loss is calculated by sampling over the pixel features in the
constructed time-series pairs. The pixel feature pairs at the same location in
the output of the teacher and student networks are called positive pairs,
while pixel features from different locations are called negative pairs. Given
a positive feature pair $(v_{1}^{i},v_{2}^{i})$ and a pixel feature
$v_{2}^{j}$ taken from another location, the contrastive loss can be
formulated as $\mathcal{L}_{\text{contrast}}$:

$\mathcal{L}_{\text{contrast}}=-\underset{S}{\mathbb{E}}{\left[\log\frac{e^{sim(v_{1}^{i},v_{2}^{i})}}{{\sum_{j=1}^{N}}e^{sim(v_{1}^{i},v_{2}^{j})}}\right]}$
(4)

where $sim$ is a similarity function (i.e., cosine similarity),
$(v_{1}^{i},v_{2}^{i})$ is the normalized latent representation of pixel $i$,
$(v_{1}^{i},v_{2}^{j}|j\geq i)$ is the normalized latent representation of
negative pair and $S=\\{s_{1}^{1},s_{2}^{1},s_{2}^{2},\cdots,s_{2}^{N-1}\\}$
is a set that contains $N-1$ negative samples and one positive sample. One
limitation of self-supervised contrastive learning is that, since the class
labels of the inputs are ignored, samples from the same class may end up being
treated as negative pairs, which can affect the training performance. To avoid
this limitation and enable the contrastive loss to learn in a supervised
fashion, Khosla et al. [26] extended the approach to account for input labels.
Following the original supervised contrastive learning method, we randomly
sample $N$ pixel features in each change pair from the teacher-student
network, generating two data views $\\{(x_{i},y_{i})\\}_{i=1}^{2N}$, where
$i\in I=[2N]$ is the index of an arbitrary sample. Given
$P=\\{P_{i,j}|y_{i}=y_{j},(x_{i},y_{i}),(x_{j},y_{j})\\}$, we perform
supervised contrastive learning with sampled pixel features:

$\displaystyle\mathcal{L}_{i}^{\sup}=\sum_{i\in
I}\frac{-1}{|P(i)|}\cdot\sum_{p\in
P(i)}\log\frac{e^{\operatorname{sim}\left(x_{i},y_{p}\right)/\tau}}{\sum_{b\in
B(i)}e^{\operatorname{sim}\left(x_{i},y_{b}\right)/\tau}}$ (5)

where $B(i)$ means the set of indices excluding $i$, i.e.,
$B(i)=I\backslash{i}$; $P(i)=\\{P_{i,j}|j\in B(i)\\}$ is the positive set
distinct from sample $i$ and $|\cdot|$ stands for cardinality. In this case,
the labels are binary pseudo labels. The use of the supervised contrastive
objective function improve the exploitation of the binary change information
with respect to only use the cross-entropy loss.

####  III-B2 Contrastive Random Walk Loss

Image time-series change detection is often treated as a simple extension of
bi-temporal change detection in time. However, the incorporation of temporal
information to mitigate seasonal noise poses a significant challenge, because
the change depicted at position $(x,y)$ in the frame $t$ might not have any
relation to what we find at the same location $x,y$ in frame $t+k$. To
overcome this limitation, the contrastive random walk method leverages
pathfinding on a space-time graph and associates features across space and
time. This method establishes nodes shared by neighboring frames, thereby
formulating correspondence. By converting image time-series into palindromes,
the walk step can be put through a contrastive learning problem. Moreover, the
walker’s destination offers guidance, allowing for the integration of whole
chains of intermediate comparisons.

This works builds upon the contrastive random walk framework by et al. [28],
where an image time-series is treated as a direct graph, composed of pixel
feature vectors forming the nodes and weighted edges connecting neighboring
frames. Given an input image time-series with $k$ frames, we choose $N$
feature vectors $q_{t}$ within a small patch from the frame $t$, which serve
as vertices of a graph. The graph connects all feature vectors within the
small patch in temporally adjacent frames. A random walker then steps through
the graph, moving forward in time from frames $1,2,\cdots,k$, and then
backward in time from $k-1,k-2,\cdots,1$, with transition probabilities
determined by the similarity of learned representations. Pairwise similarities
are converted into non-negative affinities by applying a softmax function
(with temperature $\tau$) over edges departing from each node. This process
generates the stochastic affinity matrix for the graph at each timestep.

$A_{t,t+1}(i,j)=\frac{exp(sim(q_{t}^{i},q_{t+1}^{j})/\tau)}{\sum_{l=1}^{N}exp(sim(q_{t}^{i},q_{t+1}^{j})/\tau)}$
(6)

for a pair of frames $t$ and $t+1$, where $q_{t}\in R^{N\times d}$ is the
matrix of $d-$dimensional embedding vectors, $\tau$ is a small constant, and
the softmax is performed along each row. The local affinity between patches of
two video frames, $q_{t}$ and $q_{t+1}$, is captured by this process. The
affinity matrix for the entire graph, which relates all nodes in the video as
a Markov chain, can be considered as a composition of local affinity matrices.
The established spatio-temporal connectivity in the graph and propose a
walking strategy for a random walker that can perform tracking by contrasting
similarity of neighboring nodes. Let $X_{t}$ to denote the state of the walker
at time $t$, with transition probabilities
$A_{t,t+1}(i,j)=P(X_{t+1}=j|X_{t}=i)$, where $P(X_{t}=i)$ is the probability
of being at node $i$ at time $t$. With this view, we can formulate long-range
correspondence as walking multiple steps along the graph:

$\overline{A}_{t,t+k}=\prod_{i=0}^{k-1}A_{t+i,t+i+1}=P(X_{t+k}|X_{t})$ (7)

The likelihood of cycle consistency is maximized by training the model to
achieve the event of the walker returning to its starting point.

$L_{CRW}=-tr(log(\overline{A}_{t,t+k}\overline{A}_{t+k,t}))$ (8)

where $\overline{A}_{t,t+k}$ are the transition probabilities from frame $t$
to $t+k$: $\overline{A}_{t,t+k}=\prod_{i=t}^{t+k-1}A_{i,i+1}$.

###  III-C Pseudo Labels Extraction

The pseudo change maps can be made less noisy by propagating the threshold-
based pseudo labels to each change pair using the label propagation algorithm
[27]. This algorithm considers both spatial and temporal neighbours, using a
queue of the $N_{T}$ most correlated change pairs for temporal neighbours, and
a spatial neighbourhood of the query node for spatial neighbours. The labels
of target nodes are determined by computing the matrix of transitions between
target nodes and source nodes, considering only top$-k$ transitions, and
multiplying it by the labels of the source nodes. For every feature embedding
in a frame, we compute its cosine similarity with the features in the queue,
and select the $N_{k}=10$ feature embeddings with the highest similarity. We
then use these embeddings to compute a weighted sum of the label predictions
at these locations, which is added to the label context. This process is
repeated for all change pairs in the image sequence, updating the label and
embedding contexts using the $N_{T}$ most correlated change pairs of each
change pair.

In detail, we obtain the embedding for change pair $t$, denoted as $h_{t}\in
R^{h,w,n}$, where $h$, $w$ are the spatial dimensions and $n$ is the dimension
of the feature embedding, using a pre-trained network. The embedding context
defined as
$\mathcal{E}_{t}=\left\\{h_{0},h_{t-N_{T}+1},\ldots,h_{t-1}\right\\}$
maintains embedding of each change pair. Similarly, we define the label
context as $M_{t}=\left\\{m_{0},m_{t-N_{T}+1},\cdots,m_{t-1}\right\\}$
obtained by using threholding method on pre-trained features. We also define
the predicted labels as
$L_{t}=\left\\{l_{0},l_{t-N_{T}+1},\cdots,l_{t-1}\right\\}$. We compute the
cosine similarity of embedding $h_{t}(i,j)$ with all embeddings in queue
$\mathcal{E}$, restricted to a spatial-temporal neighbourhood of size
$N_{T}\times P\times P$, centred on location $(i,j)$, which we denote as
$N_{p}(i,j)$. We then obtain the nearest-neighbour set $N_{p}^{(k)}(i,j)$ by
selecting $N_{k}$ embedding locations from $N_{P}(i,j)$ with the highest
cosine similarity and compute the local weight as follows:

$s_{i,j}^{t}\left(t^{*},l,n\right)=\frac{\exp\left(-d_{i,j}^{t}\left(t^{*},l,n\right)/\tau\right)}{\sum_{\left(t^{\prime},l^{\prime},n^{\prime}\right)}\exp\left(-d_{i,j}^{t}\left(t^{\prime},l^{\prime},n^{\prime}\right)/\tau\right)}$
(9)

The three coordinates $(t,l,n)$ for the temporal (first) and the spatial
(second and third) dimensions specify the neighbour locations in $N_{p}(i,j)$.
$d_{i,j}^{t}(t^{*},l,n)$ is the cosine similarity between embeddings
$h_{t}(i,j)$ and $h_{t^{*}}(l,n)$ from $N_{p}^{(k)}(i,j)$; $(t^{*},l,n)$ and
$(t^{\prime},l^{\prime},n^{\prime})\in N_{p}^{(k)}(i,j)$ , and $\tau$ is the
temperature hyperparameter set to 0.005. We compute the label $l_{t}$ as a
weighted sum of the label predictions in $N_{p}^{(k)}(i,j)$ as

$l^{t}(i,j)=\sum_{\left(t^{*},l,n\right)\in
N_{P}^{(k)}(i,j)}s_{i,j}^{t}\left(t^{*},l,n\right)m^{t^{*}}(l,n)$ (10)

where $m^{t^{*}}(l,n)$ comes from the label context $M$. We repeat this
process for the remaining change pairs in the image sequence.

##  IV Experiments

In this section, we first describe the datasets used in our experiments and
then introduce the related experiment setting on the network training and the
feature-tracking-based pseudo-label generation. Finally, we present the
results of the proposed approach and the comparison methods. We also present
an ablation study of each component of the proposed approach.

###  IV-A Description of Datasets

We conducted experiments on two multi-spectral datasets, one from the
Sentinel-2 satellite constellation and the other from the Landsat-8 satellite.

####  IV-A1 Sentinel-2 dataset

The Multi-temporal Urban Development (MUDS) dataset [45] was designed to
monitor urbanization by tracking changes in building construction from 2017 to
2020. It is an open-source dataset that includes native Planet 4-meter
resolution imagery and Sentinel-2 multi-spectral images with irregular
observation intervals across six continents. However, the original Sentinel-2
imagery often contains clouds and missing values. To improve the utility of
this dataset, we selected only 74 locations with a minimum of 12 clean images
and resized each image to 512 x 512 pixels. Of these 74 locations, we labelled
the significant change pair of 30 scenes for future evaluation, with all
change pairs referenced to the first image. We used only four bands in this
work, all of them with a spatial resolution of 10 meters. Due to the
unsupervised nature of this dataset, we only considered three types of
changes: built-up, bare land, and water.

####  IV-A2 Landsat-8 dataset

The UTRnet dataset [41] was specifically designed for validation of the UTRnet
model. The dataset consists of the satellite image time-series collected by
Landsat-8 from 2013 to 2021, with a spatial resolution of 30 meters. Six
spectral bands covering the visible to the shortwave infrared region are used,
including blue, green, red, near-infrared, and two shortwave infrared bands.
The dataset includes nine typical scenes located in different cities in China,
each with a different land cover type. For each scene, ten cloud-free
Landsat-8 images were selected to cover different seasons. The image size for
each scene is 400 x 400 pixels. The ground truth includes three classes:
changed pixels, unchanged pixels, and unlabeled pixels. The changed and
unchanged pixels are labelled using Google Earth images. In this study,
unlabeled pixels are treated as unchanged pixels to validate the influence of
seasonal noise. Due to the temporal limitations of high-resolution image
labelling, the labels only include the longest interval pairs. The change maps
include city expansion, water change, and soil change.

###  IV-B Experiment Settings

#### Evaluation Metrics

In order to evaluate the effectiveness of different methods in binary change
detection, this paper employs five evaluation metrics: precision (Pre), recall
(Rec), overall accuracy (OA), F1 score (F1), and Cohen’s kappa score (Kap).

#### Implementation Details

In the process of generating pseudo labels, we first derived pseudo labels of
each change pair using a thresholding approach on pre-trained features. Then
propagate the threshold-based labels to each change pair using the feature
tracking approach. In the setting of feature tracking parameters, the spatial
neighbours $P$ are set to 10, the temporal neighbours $N_{T}$ are set to 3
most correlated change pairs, and the value of $N_{k}$ is set to 10. In the
self-training algorithm, the proposed approach uses a two-layer Bi-ConvLSTM.
We choose the Adam optimizer with an initial learning rate of $3e^{-4}$ at
feature learning stage, which is decreased using step scheduling without
restarts. The batch size is set to 2 and the model is trained for 200 epochs.
For the finetuning, we use SGD optimizer with a learning rate of 0.01, a mini-
batch size of 10 and a number of epochs equal to 50. To evaluate the proposed
approach, it is compared with the state-of-the-art method UTRnet in fitting
and unseen scenarios. UTRnet is an improved LSTM-based self-training approach
that uses CVA to generate pseudo labels. Unlike the proposed approach, UTRnet
is not designed to generalize to unseen scenarios and requires fitting a
separate model for each scene. In the evaluation, we choose the fitting
evaluation on the Landsat-8 dataset due to the lack of training data while
choosing the inference evaluation on the Sentinel-2 dataset. For the fitting
scenarios of the Landsat-8 dataset, we chose scene 3, scene 5 and scene 7 as
evaluation sets. For the unseen scenarios of the Sentinel-2 dataset, we chose
scene $T1286\\_2921\\_13$, scene $T1736\\_3318\\_13$ and scene
$T6730\\_3430\\_13$ as the evaluation set. In addition to the comparison with
UTRnet, this paper also conducts extensive ablation experiments on the
labelled 30 scenes of the Sentinel-2 dataset to evaluate the impact of
different components of the proposed approach and different pseudo-label
generation algorithms. In particular, the proposed approach is compared with
versions that do not use the contrastive random walk loss or only use the
cross-entropy loss. It should be noted that only the change pair with the most
significant change is labelled as ground truth for evaluating the performance
of different approaches.

Figure 2: Examples of change detection results on three scenes for the
Landsat-8 dataset. Row 1: image time-series; Row 2: change maps of one-scene
fitting obtained by UTRnet; Row 3: change maps of all-scene fitting obtained
by UTRnet; Row 4: change maps of all-scene fitting obtained by the proposed
approach. Col. 1 of Row 2, 3, 4 in each scene is the most significant change
map versus the ground truth (Green: TP, White: TN, Blue: FN, Red: FP).

###  IV-C Experimental Results on Landsat-8 image time-series

In this study, the effectiveness of the proposed approach is evaluated using
the Landsat-8 dataset. The performance of the proposed approach is compared
with the state-of-the-art approach UTRnet, which has been validated by fitting
on each scene in the dataset. In order to evaluate the generalizability of the
proposed approach, results are provided for fitting on all scenes, while
UTRnet results are provided for both fitting on each scene and fitting on all
scenes. Quantitative evaluation is performed using the most significant change
in the time-series of images, due to the challenges of differentiating changes
in continuous change scenarios. The results of the proposed approach and
UTRnet are presented in Table II. In the one-scene fitting setting, UTRnet
achieves an OA of 89.66% and a Cohen’s kappa score of 0.56, underperforming
the results obtained from the pseudo labels. However, for the all-scene
fitting setting, UTRnet fails to differentiate changed and unchanged pixels,
achieving an OA of 48.50% and a Cohen’s kappa score of 0.09. In contrast, the
proposed approach achieves significantly better results than UTRnet in both
settings, with an OA of 91.30% and a Cohen’s kappa score of 0.64. Comparing
the results of the pseudo labels and the models, we can see that self-training
approaches further improve the results of the corresponding pseudo labels. It
is worth noting that the pseudo labels acquired by weight-CVA are even better
than those of the proposed approach in the significant change pairs, but they
result in a worse model performance due to the lack of enough change pairs for
training.

Besides the quantitative analysis, we also provide a visual comparison of the
results obtained from the proposed approach and the UTRnet method. We present
the results of UTRnet obtained by fitting on each scene, and the results of
UTRnet obtained by fitting on all scenes as well as the results of the
proposed approach. We also present the significant change maps in each first
column of the change maps in each scene in Fig 2, where true positives, true
negatives, false negatives, and false positives are colored in green, white,
blue, and red, respectively. From the visual comparison of the most
significant change map, we can see that the change map obtained by UTRnet
using all-scene fitting is noisy and contains a high number of false alarms.
In contrast, the change maps obtained by the other two settings are more
accurate and have less noise. Additionally, the proposed approach is able to
successfully detect most of the changed pixels and suppress the effects of
seasonal changes. When comparing the change map time-series, we can see that
the change maps obtained by UTRnet (one-scene fitting) have more false alarms
that are affected by historical changes. In contrast, the change maps obtained
by the proposed approach are robust to seasonal changes and only focus on real
changes happened at each time. While the one-scene fitting UTRnet still
achieves good results on all test scenes, the all-scene fitting UTRnet can
still perform well in a few scenes with less seasonal noise, but its results
are heavily influenced by the imbalanced training samples.

TABLE II: Quantitative evaluations of different approaches applied to the fitting test set on the Landsat-8 dataset. Method | Pre(%) | Rec(%) | OA(%) | F1 | Kap
---|---|---|---|---|---
Weighted CVA | 61.89 | 67.80 | 91.02 | 0.647 | 0.596
Feature Tracking | 53.11 | 76.01 | 88.07 | 0.625 | 0.557
UTRnet (One-Scene) | 60.86 | 62.57 | 89.66 | 0.617 | 0.557
UTRnet (All-Scene) | 17.50 | 77.19 | 48.50 | 0.285 | 0.087
Proposed.(All-Scene) | 64.89 | 74.76 | 91.30 | 0.695 | 0.644
TABLE III: Quantitative evaluations of different approaches applied to the inference test set on the Sentinel-2 dataset. Method | Pre(%) | Rec(%) | OA(%) | F1 | Kap
---|---|---|---|---|---
Weighted CVA | 91.02 | 51.99 | 92.54 | 0.662 | 0.623
Feature Tracking | 55.18 | 92.97 | 88.02 | 0.693 | 0.624
UTRnet (One-Scene) | 85.25 | 25.17 | 88.89 | 0.388 | 0.347
UTRnet (Inference) | 46.64 | 64.07 | 84.67 | 0.540 | 0.451
Proposed. (Inference) | 76.12 | 84.46 | 93.90 | 0.801 | 0.765
Figure 3: Examples of change detection results on three scenes for the
Sentinel-2 dataset. Row 1: image time-series; Row 2: change maps of one-scene
fitting obtained by UTRnet; Row 3: change maps on infer setting obtained by
UTRnet; Row 4: change maps on infer setting obtained by the proposed approach.
Col. 1 of Row 2, 3, 4 in each scene is the most significant change map versus
the ground truth (Green: TP, White: TN, Blue: FN, Red: FP).

###  IV-D Experimental Results on the Sentinel-2 image time-series

The Sentinel-2 dataset is characterized by its diversity of land-cover scenes
and a larger number of training samples. In contrast to the results obtained
on the Landsat-8 dataset, we present the results of different pseudo labels,
one-scene fitting UTRnet, and inference on unseen scenarios based on models
trained on all training samples. Similar to the evaluation on the Landsat-8
dataset, we only consider the most significant change map in each scene to
assess its quantitative performance (Table III). As one can observe, the one-
scene fitting UTRnet achieves worse results than those obtained on the
Landsat-8 dataset, with an OA of 88.89% and a Cohen’s kappa score of 0.35. The
possible reason is that the Sentinel-2 dataset contains more seasonal changes
such as snow. However, its performance is improved when inferred to unseen
scenarios. Nevertheless, it still shows significant improvements compared to
the all-scene fitting setting on the Landsat-8 dataset, which is largely due
to the increased number and diversity of training samples. On the other hand,
the inference results obtained from the proposed approach are significantly
better than those obtained by UTRnet in both the one-scene fitting and
inference on unseen scenarios settings. Across all five performance metrics,
the proposed approach achieves the best performance in most cases, except for
precision, achieving an OA of 93.9% and a Cohen’s kappa score of 0.77. This
indicates that the proposed approach not only outperforms the state-of-the-art
approach on trained samples, but also on unseen samples. In the proposed
approach the improvement is more pronounced when using a larger and more
diverse set of training samples. Similarly to the experiments on the Landset-8
dataset, UTRnet underperforms the results of the pseudo labels. In this case,
two pseudo-label generation approaches achieve comparable accuracy.

In addition to the quantitative analysis, we also provide a visual comparison
of the most significant change map and the change map time-series in each
scene. Fig. 3 shows a comparison of all methods on the Sentinel-2 test set.
The true positive, true negative, false negative, and false positive pixels of
the significant change map are colored green, white, blue, and red,
respectively. We first analyze the performance of the most significant change
map in each scene. As shown in the figure (first column of change maps in each
scene), the proposed approach successfully detects most changed pixels and
suppresses seasonal changed areas while the results of UTRnet contain more
false alarms and missing detections. For the change map time-series, one can
see that the results obtained by one-scene fitting UTRnet contain many missed
detection in particular related to the cultivated errors in the image
sequence. As for the inference results, UTRnet fails to suppress most seasonal
changes and presents more false alarms, but gets big improvements in noise
reduction compared with its performance on the Landset-8 dataset. This issue
is well addressed in the proposed approach, where abrupt changes and
continuous changes are both well detected.

Figure 4: Examples of change detection results on the Sentinel-2 ablation test
set. Row 1: image time-series; Row 2: pseudo labels obtained by thresholding
approach; Row 3: pseudo labels obtained by feature tracking; Row 4: change
maps obtained by the proposed approach only using cross-entropy loss; Row 5:
change maps obtained by the proposed approach only using contrastive loss; Row
6: change maps obtained by the proposed approach trained on threshold-based
pseudo labels; Row 7: change maps obtained by the proposed approach trained on
feature tracking-based pseudo labels. Col. 1 of Row 2-7 is the most
significant change map versus the ground truth (Green: TP, White: TN, Blue:
FN, Red: FP).

###  IV-E Discussion

In this section, we conduct extensive ablation studies on the proposed
approach to analyze the contribution of different components. To better
understand the proposed approach, we choose the scene $T4780\\_3377\\_13$ with
significant vegetation changes over time for visualization. However, the
quantitative evaluation was implemented on the selected 30 scenes of the
Sentinel-2 dataset as the ablation test set.

####  IV-E1 Pseudo labels

Many unsupervised change detection approaches employ a thresholding approach
for change detection. However, determining a reasonable threshold is often a
challenging task. To demonstrate the effectiveness of the proposed pseudo-
label generation approach, we present the pseudo-labels obtained by
thresholding and feature tracking methods, individually. Then, we train the
proposed approach using these two sets of pseudo-labels. Finally, we evaluate
the performance of the trained models on the ablation test set. Fig. 4 shows
the details of the pseudo-labels and the results obtained by the trained
models. As one can see, the thresholding approach produces more false alarms
in the pseudo change maps with the shorter time interval change pair. In
contrast, the feature tracking approach can mitigate the effect of this type
of seasonal changes while maintaining the most significant changes in the
change map time-series. Similarly, the model trained on threshold-based labels
produces more missing detections due to this type of noise, while the model
trained on feature tracking-based labels significantly reduces the false
alarms. Table IV presents all five metrics on the ablation test set for the
two trained models. Among these results, the model trained on feature
tracking-based labels provides the best result in almost all metrics,
including the highest overall accuracy of 94.41% and the highest kappa
coefficient of 0.599. Compared to the threshold method-based labels, the OA
and Kappa on the feature tracking-based labels are further improved by about
2% and 0.05. This indicates that the feature tracking-based pseudo-label
generation can more accurately detect reliable changes in RS image time-series
and thus benefit the self-training.

####  IV-E2 supervised contrastive loss and contrastive random walk loss

To verify the effectiveness of the contrastive and contrastive random walk
loss, we set up experiments with training on the proposed pseudo-labels.
Specifically, we trained models using both supervised contrastive and
contrastive random walk losses, and only using supervised contrastive loss, as
well as only using cross-entropy loss, respectively. The same ablation test
set defined before is used for evaluation. Fig. 4 and Table IV compare the
results obtained using three different models. Results show that the
supervised contrastive loss and contrastive random walk loss achieve
significant improvements in noise reduction and maintain the consistency of
changes in the time-series. The only use of the contrastive loss achieves an
OA of 93.67% and a kappa of 0.553, which are slightly lower than the values
obtained by using two loss functions. In addition, the use of both loss
functions increases by about 6% and 0.16 on OA and Kappa, respectively, with
respect to the only use of the cross-entropy loss. This demonstrates that the
joint use of contrastive loss and contrastive random walk loss can further
improve the performance of the self-training paradigm.

TABLE IV: Quantitative evaluations of different approaches applied to the Sentinel-2 test set in the ablation study. Method | Pre(%) | Rec(%) | OA(%) | F1 | Kap
---|---|---|---|---|---
threshold-based | 38.30 | 19.18 | 90.78 | 0.256 | 0.212
feature-tracking | 40.27 | 17.13 | 91.07 | 0.240 | 0.201
only cross-entropy | 37.94 | 74.17 | 87.86 | 0.502 | 0.441
only contrastive | 63.48 | 54.68 | 93.67 | 0.588 | 0.553
self-training thres. | 53.72 | 66.22 | 92.51 | 0.593 | 0.552
self-training feature. | 69.53 | 57.47 | 94.41 | 0.629 | 0.599

##  V Conclusion

In this work we have proposed a new framework for detecting changes in RS
image time-series without any manually annotated training data. Our framework
jointly uses an architecture based on Unet and ConvLSTM and adopts a self-
training algorithm. We first extract pseudo labels using the feature-tracking
method and then further improve the results by training a model from scratch.
Feature-tracking-based pseudo label generation in RS image time-series detect
the significant changes more accurately while alleviate the presence of
seasonal changes. The self-training combines the use of supervised contrastive
loss, contrastive random walk loss and cross-entropy loss following the two-
stage setting of supervised contrastive learning. This mitigates the effects
of the noise in pseudo labels and keep the consistency of change map time-
series. Our experiments on two different datasets demonstrate the
effectiveness of our approach compared to state-of-the-art methods. It is
worth noting that the proposed approach can also generalize well to unseen
scenarios. Although our method is demonstrated in the context of multi-
spectral images, it can be applied to other sensors, such as synthetic
aperture radar and RGB images. In future work, we plan to extend our method to
detect different types of changes using prior information from multi-spectral
images.

## References

  * [1] S. Liu, D. Marinelli, L. Bruzzone, and F. Bovolo, “A review of change detection in multitemporal hyperspectral images: Current techniques, applications, and challenges,” _IEEE Geoscience and Remote Sensing Magazine_ , vol. 7, no. 2, pp. 140–158, 2019.
  * [2] F. Bovolo and L. Bruzzone, “A theoretical framework for unsupervised change detection based on change vector analysis in the polar domain,” _IEEE Transactions on Geoscience and Remote Sensing_ , vol. 45, no. 1, pp. 218–236, 2006\.
  * [3] F. Bovolo, S. Marchesi, and L. Bruzzone, “A framework for automatic and unsupervised detection of multiple changes in multitemporal images,” _IEEE Transactions on Geoscience and Remote Sensing_ , vol. 50, no. 6, pp. 2196–2212, 2011.
  * [4] S. Liu, L. Bruzzone, F. Bovolo, M. Zanetti, and P. Du, “Sequential spectral change vector analysis for iteratively discovering and detecting multiple changes in hyperspectral images,” _IEEE transactions on geoscience and remote sensing_ , vol. 53, no. 8, pp. 4363–4378, 2015.
  * [5] M. Zanetti, F. Bovolo, and L. Bruzzone, “Rayleigh-rice mixture parameter estimation via em algorithm for change detection in multispectral images,” _IEEE Transactions on Image Processing_ , vol. 24, no. 12, pp. 5004–5016, 2015.
  * [6] T. Celik, “Unsupervised change detection in satellite images using principal component analysis and $k$-means clustering,” _IEEE geoscience and remote sensing letters_ , vol. 6, no. 4, pp. 772–776, 2009.
  * [7] C. Wu, B. Du, X. Cui, and L. Zhang, “A post-classification change detection method based on iterative slow feature analysis and bayesian soft fusion,” _Remote Sensing of Environment_ , vol. 199, pp. 241–255, 2017.
  * [8] L. Zhang, M. Liao, L. Yang, and H. Lin, “Remote sensing change detection based on canonical correlation analysis and contextual bayes decision,” _Photogrammetric Engineering & Remote Sensing_, vol. 73, no. 3, pp. 311–318, 2007.
  * [9] I. Goodfellow, Y. Bengio, and A. Courville, _Deep learning_. MIT press, 2016.
  * [10] L. Bottou _et al._ , “Stochastic gradient learning in neural networks,” _Proceedings of Neuro-Nımes_ , vol. 91, no. 8, p. 12, 1991.
  * [11] R. C. Daudt, B. Le Saux, and A. Boulch, “Fully convolutional siamese networks for change detection,” in _2018 25th IEEE International Conference on Image Processing (ICIP)_. IEEE, 2018, pp. 4063–4067.
  * [12] Y. Zhou and X. Li, “Unsupervised self-training algorithm based on deep learning for optical aerial images change detection,” _arXiv preprint arXiv:2010.07469_ , 2020.
  * [13] A. Oussidi and A. Elhassouny, “Deep generative models: Survey,” in _2018 International Conference on Intelligent Systems and Computer Vision (ISCV)_. IEEE, 2018, pp. 1–8.
  * [14] X. Liu, F. Zhang, Z. Hou, L. Mian, Z. Wang, J. Zhang, and J. Tang, “Self-supervised learning: Generative or contrastive,” _IEEE Transactions on Knowledge and Data Engineering_ , 2021.
  * [15] L. T. Luppino, M. A. Hansen, M. Kampffmeyer, F. M. Bianchi, G. Moser, R. Jenssen, and S. N. Anfinsen, “Code-aligned autoencoders for unsupervised change detection in multimodal remote sensing images,” _IEEE Transactions on Neural Networks and Learning Systems_ , 2022.
  * [16] Y. Chen and L. Bruzzone, “A self-supervised approach to pixel-level change detection in bi-temporal rs images,” _IEEE Transactions on Geoscience and Remote Sensing_ , vol. 60, pp. 1–11, 2022.
  * [17] Chen, Yuxing and Bruzzone, Lorenzo, “Self-supervised change detection in multiview remote sensing images,” _IEEE Transactions on Geoscience and Remote Sensing_ , vol. 60, pp. 1–12, 2021.
  * [18] K. Nemoto, R. Hamaguchi, M. Sato, A. Fujita, T. Imaizumi, and S. Hikosaka, “Building change detection via a combination of cnns using only rgb aerial imageries,” in _Remote Sensing Technologies and Applications in Urban Environments II_ , vol. 10431. SPIE, 2017, pp. 107–118.
  * [19] L. Ding, H. Guo, S. Liu, L. Mou, J. Zhang, and L. Bruzzone, “Bi-temporal semantic reasoning for the semantic change detection in hr remote sensing images,” _IEEE Transactions on Geoscience and Remote Sensing_ , vol. 60, pp. 1–14, 2022.
  * [20] F. Guttler, D. Ienco, J. Nin, M. Teisseire, and P. Poncelet, “A graph-based approach to detect spatiotemporal dynamics in satellite image time series,” _ISPRS Journal of Photogrammetry and Remote Sensing_ , vol. 130, pp. 92–107, 2017.
  * [21] H. Lyu, H. Lu, and L. Mou, “Learning a transferable change rule from a recurrent neural network for land cover change detection,” _Remote Sensing_ , vol. 8, no. 6, p. 506, 2016.
  * [22] K. Meshkini, F. Bovolo, and L. Bruzzone, “A 3d cnn approach for change detection in hr satellite image time series based on a pretrained 2d cnn,” _The International Archives of Photogrammetry, Remote Sensing and Spatial Information Sciences_ , vol. 43, pp. 143–150, 2022.
  * [23] Meshkini, Khatereh and Bovolo, Francesca and Bruzzone, Lorenzo, “An unsupervised change detection approach for dense satellite image time series using 3d cnn,” in _2021 IEEE International Geoscience and Remote Sensing Symposium IGARSS_. IEEE, 2021, pp. 4336–4339.
  * [24] S. Saha, F. Bovolo, and L. Bruzzone, “Change detection in image time-series using unsupervised lstm,” _IEEE Geoscience and Remote Sensing Letters_ , 2020\.
  * [25] E. Kalinicheva, D. Ienco, J. Sublime, and M. Trocan, “Unsupervised change detection analysis in satellite image time series using deep learning combined with graph-based approaches,” _IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing_ , vol. 13, pp. 1450–1466, 2020\.
  * [26] P. Khosla, P. Teterwak, C. Wang, A. Sarna, Y. Tian, P. Isola, A. Maschinot, C. Liu, and D. Krishnan, “Supervised contrastive learning,” _Advances in Neural Information Processing Systems_ , vol. 33, pp. 18 661–18 673, 2020\.
  * [27] N. Araslanov, S. Schaub-Meyer, and S. Roth, “Dense unsupervised learning for video segmentation,” _Advances in Neural Information Processing Systems_ , vol. 34, pp. 25 308–25 319, 2021.
  * [28] A. Jabri, A. Owens, and A. Efros, “Space-time correspondence as a contrastive random walk,” _Advances in neural information processing systems_ , vol. 33, pp. 19 545–19 560, 2020.
  * [29] X. Shi, Z. Chen, H. Wang, D.-Y. Yeung, W.-K. Wong, and W.-c. Woo, “Convolutional lstm network: A machine learning approach for precipitation nowcasting,” _Advances in neural information processing systems_ , vol. 28, 2015.
  * [30] Z. Zhang, Q. Liu, and Y. Wang, “Road extraction by deep residual u-net,” _IEEE Geoscience and Remote Sensing Letters_ , vol. 15, no. 5, pp. 749–753, 2018.
  * [31] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, “Generative adversarial networks,” _Communications of the ACM_ , vol. 63, no. 11, pp. 139–144, 2020.
  * [32] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer, “High-resolution image synthesis with latent diffusion models,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 10 684–10 695.
  * [33] E. Riz, B. Demir, and L. Bruzzone, “Domain adaptation based on deep denoising auto-encoders for classification of remote sensing images,” in _Image and Signal Processing for Remote Sensing XXII_ , vol. 10004. SPIE, 2016, pp. 197–204.
  * [34] L. Bergamasco, S. Saha, F. Bovolo, and L. Bruzzone, “Unsupervised change-detection based on convolutional-autoencoder feature extraction,” in _Image and Signal Processing for Remote Sensing XXV_ , vol. 11155. International Society for Optics and Photonics, 2019, p. 1115510.
  * [35] M. Gong, X. Niu, P. Zhang, and Z. Li, “Generative adversarial networks for change detection in multispectral imagery,” _IEEE Geoscience and Remote Sensing Letters_ , vol. 14, no. 12, pp. 2310–2314, 2017.
  * [36] W. Gedara Chaminda Bandara, N. Gopalakrishnan Nair, and V. M. Patel, “Remote sensing change detection (segmentation) using denoising diffusion probabilistic models,” _arXiv e-prints_ , pp. arXiv–2206, 2022.
  * [37] M. Leenstra, D. Marcos, F. Bovolo, and D. Tuia, “Self-supervised pre-training enhances change detection in sentinel-2 imagery,” _arXiv preprint arXiv:2101.08122_ , 2021.
  * [38] L. Mou, L. Bruzzone, and X. X. Zhu, “Learning spectral-spatial-temporal features via a recurrent convolutional neural network for change detection in multispectral imagery,” _IEEE Transactions on Geoscience and Remote Sensing_ , vol. 57, no. 2, pp. 924–935, 2018.
  * [39] O. Sefrin, F. M. Riese, and S. Keller, “Deep learning for land cover change detection,” _Remote Sensing_ , vol. 13, no. 1, p. 78, 2020.
  * [40] S. Sun, L. Mu, L. Wang, and P. Liu, “L-unet: An lstm network for remote sensing image change detection,” _IEEE Geoscience and Remote Sensing Letters_ , 2020.
  * [41] B. Yang, L. Qin, J. Liu, and X. Liu, “Utrnet: An unsupervised time-distance-guided convolutional recurrent network for change detection in irregularly collected images,” _IEEE Transactions on Geoscience and Remote Sensing_ , vol. 60, pp. 1–16, 2022.
  * [42] A. Graves, S. Fernández, and J. Schmidhuber, “Bidirectional lstm networks for improved phoneme classification and recognition,” in _International conference on artificial neural networks_. Springer, 2005, pp. 799–804.
  * [43] H. Song, W. Wang, S. Zhao, J. Shen, and K.-M. Lam, “Pyramid dilated deeper convlstm for video salient object detection,” in _Proceedings of the European conference on computer vision (ECCV)_ , 2018, pp. 715–731.
  * [44] A. Tarvainen and H. Valpola, “Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results,” _Advances in neural information processing systems_ , vol. 30, 2017\.
  * [45] A. Van Etten, D. Hogan, J. M. Manso, J. Shermeyer, N. Weir, and R. Lewis, “The multi-temporal urban development spacenet dataset,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 6398–6407.