# Context-aware Difference Distilling for Multi-change Captioning

Yunbin Tu1, Liang Li2111Corresponding authors, Li Su1111Corresponding authors,
Zheng-Jun Zha3,
Chenggang Yan4,5, Qingming Huang1
1University of Chinese Academy of Sciences, Beijing, China
2Key Lab of Intell. Info. Process., Inst. of Comput. Tech., CAS, Beijing,
China
3University of Science and Technology of China, Hefei, China
4Hangzhou Dianzi University, Hangzhou, China
5Lishui Institute of Hangzhou Dianzi University, Lishui, China
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Multi-change captioning aims to describe complex and coupled changes within an
image pair in natural language. Compared with single-change captioning, this
task requires the model to have higher-level cognition ability to reason an
arbitrary number of changes. In this paper, we propose a novel context-aware
difference distilling (CARD) network to capture all genuine changes for
yielding sentences. Given an image pair, CARD first decouples context features
that aggregate all similar/dissimilar semantics, termed common/difference
context features. Then, the consistency and independence constraints are
designed to guarantee the alignment/discrepancy of common/difference context
features. Further, the common context features guide the model to mine locally
unchanged features, which are subtracted from the pair to distill locally
difference features. Next, the difference context features augment the locally
difference features to ensure that all changes are distilled. In this way, we
obtain an omni-representation of all changes, which is translated into
linguistic sentences by a transformer decoder. Extensive experiments on three
public datasets show CARD performs favourably against state-of-the-art
methods. The code is available at https://github.com/tuyunbin/CARD.

##  1 Introduction

Change captioning aims to describe differences between a pair of similar
images, which enables many important applications, such as automatic report
generation about change conditions of surveillance areas Hoxha et al. (2022)
and pathological changes between medical images Liu et al. (2021). On the
other hand, this task is more challenging than image captioning Yang et al.
(2023); Rotstein et al. (2024); Zhao and Xiong (2024). This is because
machines need to understand the contents of two images simultaneously, and
further reason and caption all genuine changes between them, while resisting
irrelevant viewpoint/illumination changes.

Figure 1: Three examples about multi-change captioning. (a) includes certain
object changes; (b) consists of object and background changes; (c) shows both
object changes and irrelevant viewpoint change. These changes are shown in
colored boxes.

Recently, single-change captioning has made remarkable progress Tu et al.
(2021b); Hosseinzadeh and Wang (2021); Yao et al. (2022); Yue et al. (2023);
Tu et al. (2024). In a dynamic environment, however, the changes are usually
the many-in-one, where multiple changes exist in an image pair. For instance,
there are multiple object/background changes (Figure 1 (a) (b)). In other
cases, object and viewpoint changes simultaneously appear (Figure 1 (c)). In
above cases, unchanged objects commonly mingle with changed ones and even
appear position misalignment under viewpoint changes. Such distractors pose a
great challenge to identify and caption the genuine changes.

There are a few attempts to address multi-change captioning. The pioneer work
Jhamtani and Berg-Kirkpatrick (2018) computed pixel differences of two images,
which is sensitive to noise. Latest works tried to capture differences at
representation space: some of them Hoxha et al. (2022); Liu et al. (2022,
2023a) computed difference features by subtraction, while the others Qiu et
al. (2021); Chang and Ghamisi (2023) built the correlations between the
patches of two images to model the change features for caption generation.

Despite progress, the above endeavors in multi-change captioning have several
limitations. (1) Direct subtraction between two images generalizes poorly to
unaligned image pairs under viewpoint changes (Figure 1 (c)). (2) Directly
correlating two images fails to sufficiently mine locally unchanged features
as multiple objects change, because such features might mingle with the
features of changed objects. (3) These methods focus on modeling locally
difference features, which are useful to catch conspicuous changes.
Nevertheless, certain local changes with weak features might be overlooked,
_e.g.,_ the car occluded by its shadows in Figure 1 (a). These limitations
would result in obtaining unreliable difference features for the language
decoder.

We notice that the above methods capture differences between two images only
based on local features, while neglecting the use of more comprehensive
features. We argue that, to learn locally unchanged/changed features of two
images, the model should first encapsulate their context features of
commonality and difference. Such context features aggregate all
similar/dissimilar semantics, termed common/difference context features. The
former can help correlate and mine locally common features for deducing
locally difference features, while the latter can augment the locally
difference features to ensure all changes are distilled.

In this paper, we propose a Context-Aware DiffeRence Distilling (CARD) network
to learn the robust difference features under multi-change scenes.
Specifically, given the featuers of two images, we first build intra-image
interaction to help the model understand each image content of the pair. Then,
we use CARD to decouple the common/difference context features from the image
pair. Herein, the common context features of two images summarize joint
semantics in between; the difference context feature in each image provides an
independent space to preserve its all changed semantics. Besides, the
consistency and independence constraints are designed to enforce the alignment
and discrepancy of common and difference context features, respectively. Next,
guided by the common context features, CARD models inter-image interaction to
mine locally common features, which are removed from the pair to distill
locally difference features. Subsequently, CARD augments the locally
difference features via the difference context features, so as to construct an
omni-representation of all changes, for generating descriptions by a
transformer decoder.

Our key contributions are: (1) We propose CARD to first decouple common and
difference context features, and then use them to facilitate modeling an omni-
representation of all changes for multi-change captioning. (2) The consistency
and independence constraints are customized to guarantee the alignment and
discrepancy of decoupled common and difference context features. (3) Extensive
experiments show our method achieves the state-of-the-art results on three
public datasets.

##  2 Related Work

Change captioning is an emerging task in the community of multi-modal learning
Cong et al. (2022, 2023); Tu et al. (2022). In the following, we introduce the
relevant works about single-change captioning and multi-change captioning,
respectively.

Single-change Captioning has been widely studied by most existing methods. The
prior work Park et al. (2019) collects a dataset about geometric objects under
viewpoint changes. This work computes the difference representation by direct
subtraction, which generalizes poorly between two unaligned images. To remedy
this limitation, M-VAM Shi et al. (2020), VACC Kim et al. (2021) and R3Net Tu
et al. (2021a) match local feature to predict difference features, which has
been a classic paradigm. The latest work SCORER+CBR Tu et al. (2023c) further
improves this paradigm by maximizing cross-view contrastive alignment between
two images, so as to learn a more stable difference representation. In
addition, these works Hosseinzadeh and Wang (2021); Kim et al. (2021); Tu et
al. (2023c); Yue et al. (2024) introduce the idea of cross-modal consistency
constraint to improve captioning quality. Besides improving architecture,
recent works Yao et al. (2022); Guo et al. (2022) propose the strategy of pre-
training to fine-tuning for facilitating change location and caption. However,
it is seldom that only one change appears in a dynamic environment, so a
powerful model should has the capability to describe multiple changes.

Figure 2: The overall architecture of our method, which consists of (a) Image
Pair Encoding (Sec. 3.1), (b) Context-Aware DiffeRence Distilling (CARD) (Sec.
3.2), and (c) Caption Generation (Sec. 3.3). Herein, CARD is the major
component to learn the robust difference features by context features
decoupling and context-aware difference distilling. $S^{*}$ stands for ground-
truth sentences.

Multi-change Captioning has been explored by a few attempts. The pioneer work
Jhamtani and Berg-Kirkpatrick (2018) proposes to describe multiple changes
between the image pairs from the surveillance cameras, where it captures
changes at pixel level. Recent works Hoxha et al. (2022); Liu et al. (2022,
2023a); Chang and Ghamisi (2023) propose to caption changes between remote
sensing images, where they first compute the difference features and then use
them for change detection and description. Nevertheless, the above methods
only describe differences between two well-aligned images, while ignoring the
cases of unaligned image pairs under varied viewpoints. To this end, Qiu _et
al._ Qiu et al. (2021) collect a dataset to caption multiple changes under
viewpoint changes, where they adopt the classic paradigm of local feature
matching to detect changes for caption generation. However, such a matching
paradigm may fail to mine the fine-grained common features and thus learn
error-prone difference features. Meanwhile, the current methods focus on
modeling local difference features, which risks ignoring certain local changes
with weak features. On the other hand, Qiu _et al._ Qiu et al. (2023) develop
a new synthetic dataset to describe the multiple changes and their orders.
However, recording the order of changed objects is laborious in the real
world, making it hard to test such a capability in a real-world scene.

In short, different from previous methods computing difference features only
based on local features, CARD first decouples common and difference context
features from an image pair. The common context features guide the model to
fully extract locally unchanged features for computing the features of local
differences, while the difference context features augment the locally
difference features to construct an omni-representation of all changes, for
generating accurate sentences.

##  3 Methodology

The overall architecture of our method is shown in Figure 2. Given a pair of
images, our method is to generate linguistic sentences that detail all of the
changes. Architecture-wise, our method contains three components: (a) image
pair encoding; (b) context-aware difference distilling; (c) caption
generation. We provide an overview of two basic components (a) and (c) in Sec.
3.1 and Sec. 3.3, while elaborating our key ingredient (b) in Sec. 3.2.

###  3.1 Image Pair Encoding

Formally, given a pair of “before” $I_{bef}$ and “after” $I_{aft}$ images, we
first leverage an off-the-shell encoder (_e.g.,_ ResNet-101 He et al. (2016))
to extract $n$ local features for each of them, and then introduce a trainable
[CLS] feature to represent the global content of each image:
$X_{o}=\left\\{x_{cls}^{o},x_{1},x_{2},...,x_{n}\right\\}$, where
$o\in(bef,aft)$ and $x_{i}\in\mathbb{R}^{d}$. Besides, the trainable position
encodings are added into the features of each image to help the model perceive
the relative position changes of objects. Next, we exploit a multi-head self-
attention layer Vaswani et al. (2017) to capture the relationships among the
features of each image, which helps the model sufficiently understand the
image content of the pair. Through the above manner, we can obtain the
relation-embedded features for each image, denoted as
$\tilde{X}_{o}=\left\\{\tilde{x}_{cls}^{o},\tilde{x}_{1},\tilde{x}_{2},...,\tilde{x}_{n}\right\\}$.

###  3.2 Context-Aware Difference Distilling

####  3.2.1 Context Feature Decoupling

To obtain common and difference context features, we first devise a common
encoder $\mathcal{CE}\left(\cdot;\theta_{\mathcal{C}}\right)$ and two
difference encoders $\mathcal{DE}_{o}\left(\cdot;\theta_{o}\right)$, where
three encoders are based on the linear projection and $o\in(bef,aft)$. The
common encoder $\mathcal{CE}\left(\cdot;\theta_{\mathcal{C}}\right)$ shares
the parameters $\theta_{\mathcal{C}}$ between two images, while the difference
encoders $\mathcal{DE}_{o}\left(\cdot;\theta_{o}\right)$ learn the parameters
$\theta_{o}$ for each image. Then, we feed $\tilde{x}_{cls}^{o}$ into these
encoders to decouple the common and difference context features, respectively:

$\displaystyle
C_{o}=\mathcal{CE}\left(\tilde{x}_{cls}^{o};\theta_{\mathcal{C}}\right),$ (1)
$\displaystyle
D_{o}=\mathcal{DE}_{o}\left(\tilde{x}_{cls}^{o};\theta_{o}\right),$

where $C_{bef}$, $C_{aft}$, $D_{bef}$, and $D_{aft}\in\mathbb{R}^{d}$.

Consistency Constraint. To make two common context features $C_{bef}$ and
$C_{aft}$ embedded in a shared space, we tailor the consistency constraint
based on contrastive learning. Given a training batch, we sample $B$ pairs of
common context features. For the common context feature in the $k$-th “before”
image ${C}_{k}^{bef}$, the common context feature in the $r$-th “after” image
${C}_{r(r=k)}^{aft^{+}}$ is its positive, while common context features in the
other “after” images ${C}_{r(r\neq k)}^{aft^{-}}$ will be the negatives in
this batch. Then, we project these positive/negative pairs into a shared
embedding space, normalize them by $L_{2}$-normalization, and compute their
similarity. Next, we introduce the InfoNCE loss Oord et al. (2018) to optimize
their contrastive alignment, _i.e.,_ pulling semantically close pairs of
${C}_{k}^{bef}$ and ${C}_{r(r=k)}^{aft^{+}}$ together and pushing away non-
related pairs:

$\begin{gathered}\mathcal{L}_{b2a}=-\frac{1}{B}\sum_{k}^{B}\log\frac{e^{\left(\text{sim}\left({C}_{k}^{bef},{C}_{r(r=k)}^{aft^{+}}\right)/\tau\right)}}{\sum_{r}^{B}e^{\left(\text{sim}\left({C}_{k}^{bef},{C}_{r}^{aft}\right)/\tau\right)}},\\\
\mathcal{L}_{a2b}=-\frac{1}{B}\sum_{k}^{B}\log\frac{e^{\left(\text{sim}\left({C}_{k}^{aft},{C}_{r(r=k)}^{bef^{+}}\right)/\tau\right)}}{\sum_{r}^{B}e^{\left(\text{sim}\left({C}_{k}^{aft},{C}_{r}^{bef}\right)/\tau\right)}},\\\
\mathcal{L}_{con}=\frac{1}{2}(\mathcal{L}_{b2a}+\mathcal{L}_{a2b}),\end{gathered}$
(2)

where “sim” is the dot-product function to measure the similarity between two
context features. $\tau$ is the temperature hyper-parameter. Through this
consistency constraint, we enforce the common context features of two images
to be projected into a shared semantic space with aligned distributions.

Independence Constraint. Each decoupled difference context feature can learn
the unique characteristics of its corresponding image within the image pair.
These unique characteristics represent the semantic differences between the
two images, so each difference context feature should be distinct from the
other. To this end, we design an independence constraint, which ensures that
the two difference context features are projected into separate feature
spaces. Here, we opt for the Hilbert-Schmidt Independence Criterion (HSIC)
Song et al. (2007), a proven method for testing feature independence, to
design the loss of the independence constraint. A lower HSIC score between the
two difference context features indicates a higher independence between them:
each difference context feature adequately encapsulates the semantic changes
in each image.

Concretely, we first project the difference context feature of each image into
a separate space, normalize each by $L_{2}$-normalization, and define the
independence (HSIC) constraint between $D_{bef}$ and $D_{aft}$ as:

$\text{HSIC}\left(D_{bef},D_{aft}\right)=(B-1)^{-2}\operatorname{tr}\left(PK_{bef}PK_{aft}\right),$
(3)

where $K_{bef}$, $K_{aft}\in\mathbb{R}^{B\times B}$ are the Gaussian kernel
matrices with $k_{bef,ij}=k_{bef}\left(D_{bef}^{i},D_{bef}^{j}\right)$ and
$k_{aft,ij}=k_{aft}\left(D_{aft}^{i},D_{aft}^{j}\right)$. $B$ is batch size.
$D_{bef}^{i}$ refers to the difference context feature in the $i$-th “befor”
image. $P=\mathbf{I}-\frac{1}{B}ee^{T}$, where $P\in\mathbb{R}^{B\times B}$,
$\mathbf{I}$ is an identity matrix and $e$ is an all-one column vector. If the
HSIC score between $D_{bef}$ and $D_{aft}$ is lower, their disparity is more
significant. We define the independence loss as:

$\mathcal{L}_{ind}=\text{HSIC}\left(D_{bef},D_{aft}\right).$ (4)

####  3.2.2 Difference Distilling

With the local features of each image
$\tilde{L}_{o}=\left\\{\tilde{x}_{1},\tilde{x}_{2},...,\tilde{x}_{n}\right\\}$,
where $o\in(bef,aft)$, we first broadcast the common context feature of each
image $C_{o}\in\mathbb{R}^{d}$ to $C_{o}\in\mathbb{R}^{n\times d}$. Then, we
concatenate it with $\tilde{L}_{o}$ on the channel dimension to obtain
$\tilde{X}^{\prime}_{o}\in\mathbb{R}^{n\times 2d}$, where $C_{o}$ can guide
inter-image interaction to mine locally unchanged features. Next, we transform
$\tilde{X}^{\prime}_{o}\in\mathbb{R}^{n\times 2d}$ to
$\tilde{X}^{\prime}_{o}\in\mathbb{R}^{n\times d}$ by a non-linear function
with ReLU activation. Further, we compute the locally common features on each
image by the multi-head cross-attention (MHCA) mechanism (Vaswani et al.,
2017):

$\begin{gathered}\tilde{X}_{bef}^{c}=\operatorname{MHCA}\left(\tilde{X^{\prime}}_{bef},\tilde{X^{\prime}}_{aft},\tilde{X^{\prime}}_{aft}\right),\\\
\tilde{X}_{aft}^{c}=\operatorname{MHCA}\left(\tilde{X^{\prime}}_{aft},\tilde{X^{\prime}}_{bef},\tilde{X^{\prime}}_{bef}\right).\end{gathered}$
(5)

Subsequently, we respectively subtract each $\tilde{X}_{o}^{c}$ from
$\tilde{L}_{o}$ to compute the locally difference features of each image.
These locally difference features are further augmented by difference context
feature of each image, so as to distill all of the genuine changes in each
image:

$\begin{gathered}\tilde{X}_{bef}^{d}=[\tilde{L}_{bef}-\tilde{X}_{bef}^{c};D_{bef}],\\\
\tilde{X}_{aft}^{d}=[\tilde{L}_{aft}-\tilde{X}_{aft}^{c};D_{aft}],\end{gathered}$
(6)

where [;] is a concatenation operation. Both $\tilde{X}_{bef}^{d}$ and
$\tilde{X}_{aft}^{d}$ are then concatenated as an omni-representation of all
changes between two images, which is implemented by a non-linear
transformation with the ReLU function:

$\tilde{X}_{d}=\operatorname{ReLU}\left(\left[\tilde{X}_{bef}^{d};\tilde{X}_{aft}^{d}\right]W_{c}+b_{c}\right).$
(7)

###  3.3 Caption Generation

After obtaining the omni-representation $\tilde{X}_{d}\in\mathbb{R}^{hw\times
d}$, we use a transformer decoder (Vaswani et al., 2017) to decode it into
sentences. First, we obtain the embedding features of all $m$ words of these
sentences, where each sentence is separated by a special token [SEP]. Then, we
use the masked self-attention to model relationships among these word
features. Next, we model the interaction between the word features and omni-
representation by cross-attention, so as to locate the most related difference
features during word generation. Subsequently, we feed the selected features
into a feed-forward network to obtain the enhanced difference representation,
denoted as $\hat{H}\in\mathbb{R}^{m\times d}$. Finally, the probability
distributions of words in these sentences are calculated via a single hidden
layer:

$S=\operatorname{Softmax}\left(\hat{H}W_{s}+{b}_{s}\right),$ (8)

where $W_{s}\in\mathbb{R}^{d\times u}$ and $b_{s}\in\mathbb{R}^{u}$ are the
learnable parameters. $u$ is the dimension of vocabulary size.

###  3.4 Joint Training

Our method is trained in an end-to-end manner by maximizing the likelihood of
the observed word sequence. Given the ground-truth words
$\left(s_{1}^{*},\ldots,s_{m}^{*}\right)$, we minimize the negative log-
likelihood loss:

$\mathcal{L}_{cap}(\theta)=-\sum_{t=1}^{m}\log p_{\theta}\left(s_{t}^{*}\mid
s_{<t}^{*}\right),$ (9)

where $p_{\theta}\left(s_{t}^{*}\mid s_{<t}^{*}\right)$ is computed by Eq.
(8), and $\theta$ are all the learnable parameters. Our method is also self-
supervised by the losses of consistency and independence constraints. Hence,
the total loss is optimized as follows:

$\mathcal{L}=\mathcal{L}_{cap}+\lambda_{c}(\mathcal{L}_{con}+\mathcal{L}_{ind}),$
(10)

where $\lambda_{c}$ is a trade-off parameter to balance the contribution
between the caption generator and constraints, which is discussed in the
appendix.

##  4 Experiments

###  4.1 Datasets

CLEVR-Multi-Change Dataset Qiu et al. (2021) is about basic object scene with
multiple changes. Since original dataset has not been released, we regenerate
this dataset based on the authors’ released code. The regenerated dataset has
45,044 valid image pairs/captions with viewpoint changes. Based on the
official split, we split it into training, validation, and testing with a
ratio of 4:1:1.

LEVIR-CC Dataset Liu et al. (2022) is about remote sensing scene, which
contains 10,077 pairs of bi-temporal images and 50,385 ground-truth captions.
We use the official split with 6,815 image pairs for training, 1,333 for
validation, and 1,929 for testing, respectively.

Spot-the-Diff Dataset Jhamtani and Berg-Kirkpatrick (2018) has 13,192 image
pairs from surveillance cameras, and on an average there are 1.86 ground-truth
sentences per image pair. According to the official split, we split it into
training, validation, and testing with a ratio of 8:1:1.

###  4.2 Evaluation Metrics

We follow the existing methods of multi-change captioning to use the five
metrics for evaluating the generated sentences: BLEU-4 (B) (Papineni et al.,
2002), METEOR (M) (Banerjee and Lavie, 2005), ROUGE-L (R) (Lin, 2004), CIDEr
(C) (Vedantam et al., 2015) and SPICE (S) (Anderson et al., 2016). We compute
all the results by the Microsoft COCO evaluation server (Chen et al., 2015).

###  4.3 Implementation Details

For fair-comparison, we follow previous multi-change captioning methods to use
a pre-trained ResNet-101 (He et al., 2016) to extract the local features of a
pair of images, with the dimension of 1024 $\times$ 14 $\times$ 14\. We
project them into a lower dimension of 512, while the dimension of trainable
[CLS] features is also set to 512. The hidden size of the model and word
embedding size are set to 512 and 300. Temperature $\tau$ in Eq. (2) is set to
0.07. We train the model to converge with 10K iterations in total. We use Adam
optimizer (Kingma and Ba, 2014) to minimize the negative log-likelihood loss
of Eq. (10). More details are shown in the appendix.

Table 1: Comparison with the SOTA methods on CLEVR-Multi-Change. The main metric CIDEr on this dataset is highlighted. Method | B | M | R | S | C
---|---|---|---|---|---
DUDA | 41.8 | 36.2 | 53.9 | 64.7 | 283.5
M-VAM | 37.1 | 34.0 | 51.5 | 62.2 | 242.9
MCCFormers-S | 55.9 | 44.8 | 56.8 | 76.6 | 378.6
MCCFormers-D | 56.2 | 44.8 | 57.3 | 76.6 | 383.2
VARD-Trans | 48.1 | 41.8 | 55.5 | 72.1 | 344.2
SCORER+CBR | 56.4 | 44.9 | 57.1 | 76.7 | 388.0
CARD (Ours) | 56.7 | 45.2 | 57.4 | 76.9 | 391.6

###  4.4 Performance Comparison

Results on CLEVR-Multi-Change. We compare CARD with the following SOTA
methods: DUDA Park et al. (2019), M-VAM Shi et al. (2020), MCCFormers-S /
MCCFormers-D Qiu et al. (2021), VARD-Trans Tu et al. (2023a), and SCORER+CBR
Tu et al. (2023c). On this regenerated dataset, we re-implement the above
methods based on their papers and released codes.

The results are shown in Table 1. Our CARD performs favourably against these
SOTA methods on all metrics, showing that CARD can better describe multiple
changes under viewpoint changes. In addition, our CARD outperforms
MCCFormers-D and MCCFormers-S by a large margin, which are classic match-based
methods to directly capture inner/inter-patch correlations between two image
representations. On the caption-specific metric CIDEr, CARD significantly
surpasses both methods, in particular with increases of 2.2% and 3.4%.

Table 2: Comparison with the SOTA methods on LEVIR-CC.  Method | B | M | R | C
---|---|---|---|---
DUDA | 57.8 | 37.2 | 71.0 | 124.3
MCCFormers-S | 56.7 | 36.2 | 69.5 | 120.4
MCCFormers-D | 56.4 | 37.3 | 70.3 | 124.4
RSICCFormer | 62.8 | 39.6 | 74.1 | 134.1
PSNet | 62.1 | 38.8 | 73.6 | 132.6
Prompt-CC (soft) | 62.4 | 38.6 | 73.4 | 135.3
Prompt-CC (hard) | 63.5 | 38.8 | 73.7 | 136.4
Chg2Cap | 64.4 | 40.0 | 75.1 | 136.6
CARD (Ours) | 65.4 | 40.0 | 74.6 | 137.9

Results on LEVIR-CC. We compare CARD with the SOTA methods: DUDA Park et al.
(2019), MCCFormers-S/D Qiu et al. (2021), RSICCFormer Liu et al. (2022), PSNet
Liu et al. (2023a), Prompt-CC (soft/hard) Liu et al. (2023b), and Chg2Cap
Chang and Ghamisi (2023).

The experimental results are shown in Table 2. Our CARD achieves the best
results on all metrics. This indicates that it can detect whether there are
semantic changes and what have changed between two remote sensing images.
Besides, we notice that the match-based methods (MCCFormers-D / MCCFormers-S)
cannot generalize well in this remote sensing scenario. Our conjecture is that
since there are usually most areas changed (_e.g.,_ Fig. 1 (b)), directly
matching two images might fail to extract fine-grained unchanged objects and
thus capture the difference features with noise.

Table 3: Comparison with the SOTA methods on Spot-the-Diff.  Method | B | M | R | S | C
---|---|---|---|---|---
DDLA | 6.2 | 10.8 | 26.0 | - | 29.7
DUDA | 5.4 | 10.6 | - | 12.9 | 24.8
MCCFormers-S | 5.8 | 10.5 | - | 10.1 | 18.2
MCCFormers-D | 6.2 | 10.2 | - | 17.8 | 28.8
VARD-Trans | 4.1 | 11.4 | 22.2 | 11.5 | 13.4
SCORER+CBR | 5.1 | 9.3 | 23.0 | 11.9 | 20.9
CARD (Ours) | 6.6 | 10.8 | 26.9 | 17.8 | 32.4
Table 4: Ablation of common/difference context features (CCF/DCF) on CLEVR-Multi-Change and LEVIR-CC. |  |  | CLEVR-Multi-Change | LEVIR-CC
---|---|---|---|---
Ablative Variants | CCF | DCF | B | M | R | S | C | B | M | R | C
Baseline | $\times$ | $\times$ | 54.7 | 43.6 | 56.7 | 75.6 | 362.3 | 60.7 | 36.3 | 69.7 | 120.0
Baseline | $\checkmark$ | $\times$ | 56.5 | 45.1 | 57.1 | 76.8 | 385.8 | 63.5 | 38.5 | 72.3 | 130.4
Baseline | $\times$ | $\checkmark$ | 56.5 | 45.0 | 57.1 | 77.0 | 385.7 | 60.6 | 37.6 | 71.0 | 125.9
Baseline | $\checkmark$ | $\checkmark$ | 56.7 | 45.2 | 57.4 | 76.9 | 391.6 | 65.4 | 40.0 | 74.6 | 137.9
Table 5: Ablation of consistency/independence constraint (CC/IC) on CLEVR-Multi-Change and LEVIR-CC. |  |  | CLEVR-Multi-Change | LEVIR-CC
---|---|---|---|---
Ablative Variants | CC | IC | B | M | R | S | C | B | M | R | C
CARD | $\times$ | $\times$ | 54.6 | 44.1 | 57.2 | 75.8 | 363.7 | 55.9 | 35.6 | 72.3 | 132.2
CARD | $\checkmark$ | $\times$ | 56.2 | 44.8 | 57.1 | 76.8 | 384.2 | 56.2 | 35.8 | 72.6 | 137.6
CARD | $\times$ | $\checkmark$ | 56.5 | 45.1 | 57.2 | 77.0 | 389.9 | 60.6 | 37.7 | 72.5 | 133.0
CARD | $\checkmark$ | $\checkmark$ | 56.7 | 45.2 | 57.4 | 76.9 | 391.6 | 65.4 | 40.0 | 74.6 | 137.9

Results on the Spot-the-Diff Dataset. Most previous works Hosseinzadeh and
Wang (2021); Huang et al. (2022); Tu et al. (2023b); Yue et al. (2023) tested
the models based on single-change setup, where the models are only required to
randomly describe one of the changes. Different from them, we require the
model to caption all changes.

Under this setting, the compared SOTA methods are: DDLA Jhamtani and Berg-
Kirkpatrick (2018), DUDA Park et al. (2019), MCCFormers-S / MCCFormers-D Qiu
et al. (2021), VARD-Trans Tu et al. (2023a), and SCORER+CBR Tu et al. (2023c).
The experimental results are shown in Table 3. Our method achieves the best
results on most metrics, particularly with an increase of 9.1% on CIDEr. As
shown in Fig.1 (a), the changed objects are not well captured by surveillance
cameras and are even occluded by shadows. Our method still achieves
encouraging performance, which validates its good generalization in
surveillance scenes.

Performance Analysis. On the three datasets, our CARD outperforms the existing
methods by a large margin, which shows its good generalization. This
superiority benefits from that decoupled context features facilitate learning
difference features. Instead, the compared methods only compute the
differences based on the local features, which fails to mine fine-grained
common objects and thus disregards inconspicuous changes.

###  4.5 Ablation Study and Analysis

To figure out the contribution of each component, we conduct ablation studies
on two large-scale datasets: CLEVR-Multi-Change and LEVIR-CC. The image pairs
on CLEVR-Multi-Change contain basic geometric objects and are unaligned due to
viewpoint changes, while the pairs on LEVIR-CC are well-aligned and from the
real world.

####  4.5.1 Ablation Study for Context Features

We study the effectiveness of decoupled common and difference context
features, denoted as CCF and DCF, respectively. The results are shown in Table
4. Baseline directly matches two image features to extract the locally common
features and difference features for caption generation.

We find that on the both datasets, 1) the model’s performance is enhanced when
it is augmented by either CCF or DCF; 2) when we augment the model with both
context features, the model’s performance is significantly boosted, especially
the CIDEr score is enhanced from 362.3 to 391.6. These show that 1) CCF guides
the model to sufficiently interact and mine locally common features for
computing locally difference features, while DCF augments the locally
difference features to ensure all changes are distilled; 2) each kind of
context feature not only plays its unique role, but also supplements each
other for better reasoning genuine changes.

Figure 3: Visualization of context features on CLEVR-Multi-Change and LEVIR-
CC. The red and green colors indicate common context features in “before” and
“after” images, while blue and purple colors denote difference context
features in “before” and “after” images.  Figure 4: Qualitative examples on
the three datasets. For each example, we visualize the captions generated by
the SOTA method MCCFormers-D Qiu et al. (2021) and our CARD, as well as the
change localization of CARD. The successful cases of CARD are shown in the
green box, while the sub-optimal case is shown in the red box.  Figure 5:
Visualization of alignment of common objects (shown in yellow boxes) on the
three datasets, where the results are obtained by MCCFormers-D and our CARD.

####  4.5.2 Ablation Study for Constrains

To study the effect of the consistency constraint (CC) and independence
constraint (IC), we make the ablation study in Table 5. First, we enforce CC
or IC respectively on CARD, and find that each of them improves the
performance of CARD. Then, we impose both constraints and observe that the
results are enhanced significantly on two datasets. The increased results
indicate that both constraint losses are essential to learn the context
features.

We further visualize the common/difference context features decoupled
without/with the constraints, which are shown in Figure 3. Without
constraints, the common/difference context features cannot be well learned on
two datasets. With both constraints, the common context features are blended
on CLEVR-Multi-Change, while the difference context features are learned
better on LEVIR-CC. The results show that consistency constraint helps align
distributions of shared properties, while independence constraint makes
difference context features of two images more separable.

####  4.5.3 Generalization to unaligned image pairs

To verify the generalization to unaligned image pairs under viewpoint changes,
we compare CARD with two ablative variants on the CLEVR-Multi-Change dataset.
(1) Direct Subtraction first performs direct subtraction between the features
of two images to compute the locally difference features, which are then fed
into a transformer decoder for multi-change captioning. (2) Feature Matching
directly matches two image features to extract the locally common features and
difference features for multi-change captioning. The experimental results are
shown in Table 6.

Table 6: Verifying generalization to unaligned image pairs on CLEVR-Multi-Change. Model | B | M | R | C | S
---|---|---|---|---|---
Direct Subtraction | 53.3 | 42.5 | 56.3 | 350 | 74.6
Feature Matching | 54.7 | 43.6 | 56.7 | 362.3 | 75.6
CARD | 56.7 | 45.2 | 57.4 | 391.6 | 76.9

It is noted that the performance of Feature Matching is better than that of
Direct Subtraction on every metric, which validates the generalization of
feature matching paradigm to the unaligned image pairs under viewpoint
changes. Our proposed CARD outperforms both models by a large margin, which
not only indicates a better generalization of our method to unaligned image
pairs, but also verifies the effectiveness of context-aware difference
distilling, which helps capture genuine changes.

###  4.6 Qualitative Analysis

To obtain an overall evaluation of our method, we conduct qualitative analysis
on the three datasets. The compared method MCCFormers-D Qiu et al. (2021)
performs well on the three datasets, which directly correlates two images to
predict locally common and difference features. In Figure 5, we visualize the
alignment of common properties between two images, to validate whether context
features help mine locally common information. We find that MCCFormers-D fails
to align the common properties and even misjudges changed objects as unchanged
objects. Instead, our CARD can better match the joint objects. For instance,
in Figure 5 (a), the unchanged object is only the brown object. MCCFormers-D
wrongly identifies some changed objects as unchanged ones. By contrast, our
CARD can pinpoint the unchanged brown object. This superiority benefits from
the guidance of decoupled common context features, during the matching of two
image features.

In Figure 4, we further visualize the captions yielded by MCCFormers-D and
CARD, as well as the change localization results from CARD. In the first three
cases, MCCFormers-D either only describes one of the changes or misidentifies
changed objects. Contrarily, our CARD can accurately locate and describe all
changed objects. Particularly, we notice that our method performs better in
detecting subtle changes. For instance, in Figure 4 (c) that is from
surveillance scene, the moved car and disappeared person are very tiny, and
the car is occluded by the shadow of building. In this hard case, MCCFormers-D
fails to recognize the moved car in the “after” image, thus generating a wrong
sentence. By contrary, our CARD can locate and describe this tiny change. For
the failure reason of MCCFormers-D, our conjecture is that it directly
interacts two images, which cannot sufficiently identify locally unchanged
features and compute the locally difference features. Besides, MCCFormers-D
captures differences between two images only based on local features, which
risks overlooking certain tiny changes with weak features. Compared with
MCCFormers-D, the superiority of our method is mainly attributed to the
guiding and augmenting of common and difference context features. Guided by
the common context features, CARD models inter-image interaction to
sufficiently mine locally common features and compute locally difference
features. Further, the difference context features augment the locally
difference features to ensure that all changes are distilled. Through this
manner, the model can learn genuine changes for caption generation. More
qualitative examples are shown in the appendix.

##  5 Conclusion

In this paper, we propose the CARD to reason and describe genuine changes
under various multi-change scenarios. CARD first decouples the common and
difference context features from the image pair. Then, two kinds of
constraints are designed to ensure the alignment and discrepancy of the common
and difference context features, respectively. Further, we use the common
context features to guide the mining of locally common features for deducing
locally difference features. In addition, we leverage the difference context
features to augment the locally difference features, thereby constructing an
omni-representation of all changes for multi-change captioning. Extensive
experiments conducted on the three datasets show that the proposed CARD
outperforms the current state-of-the-art methods by a large margin.

## Limitations

The last case in Figure 4 shows that the trees are replaced by a cross road
with villas. Our CARD successfully locates the changed objects, which
validates the effectiveness of context-aware difference distilling. However,
it yields a sub-optimal sentence that does not well express the change
process: trees are removed and a cross road with villas built along. A more
proper sentence should be: trees are removed and a cross road with villas are
built. In the future work, we will try to introduce linguistic knowledge
(e.g., syntactic dependencies between words) that regularizes the process of
sentence generation, in order to generate the optimal sentences well
elaborating the change process.

## Acknowledgements

This work was supported in part by National Key R&D Program of China under
Grant (2023YFB4502800), National Natural Science Foundation of China:
62322211, 61931008, 62236008, 62336008, U21B2038, 62225207, Fundamental
Research Funds for the Central Universities (E2ET1104), “Pionee” and “Leading
Goose” R&D Program of Zhejiang Province (2024C01023, 2023C01030).

## References

  * Anderson et al. (2016) Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould. 2016.  Spice: Semantic propositional image caption evaluation.  In _ECCV_ , pages 382–398. Springer.
  * Ba et al. (2016) Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. 2016.  Layer normalization.  _arXiv preprint arXiv:1607.06450_.
  * Banerjee and Lavie (2005) Satanjeev Banerjee and Alon Lavie. 2005.  Meteor: An automatic metric for mt evaluation with improved correlation with human judgments.  In _Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization_ , pages 65–72.
  * Chang and Ghamisi (2023) Shizhen Chang and Pedram Ghamisi. 2023.  Changes to captions: An attentive network for remote sensing change captioning.  _IEEE Transactions on Image Processing_ , 32:6047–6060.
  * Chen et al. (2015) Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C Lawrence Zitnick. 2015.  Microsoft coco captions: Data collection and evaluation server.  _arXiv preprint arXiv:1504.00325_.
  * Cong et al. (2022) Gaoxiang Cong, Liang Li, Zhenhuan Liu, Yunbin Tu, Weijun Qin, Shenyuan Zhang, Chengang Yan, Wenyu Wang, and Bin Jiang. 2022.  Ls-gan: iterative language-based image manipulation via long and short term consistency reasoning.  In _ACM MM_ , pages 4496–4504.
  * Cong et al. (2023) Gaoxiang Cong, Liang Li, Yuankai Qi, Zheng-Jun Zha, Qi Wu, Wenyu Wang, Bin Jiang, Ming-Hsuan Yang, and Qingming Huang. 2023.  Learning to dub movies via hierarchical prosody models.  In _CVPR_ , pages 14687–14697.
  * Guo et al. (2022) Zixin Guo, Tzu-Jui Wang, and Jorma Laaksonen. 2022.  Clip4idc: Clip for image difference captioning.  In _AACL_ , pages 33–42.
  * He et al. (2016) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016.  Deep residual learning for image recognition.  In _CVPR_ , pages 770–778.
  * Hosseinzadeh and Wang (2021) Mehrdad Hosseinzadeh and Yang Wang. 2021.  Image change captioning by learning from an auxiliary task.  In _CVPR_ , pages 2725–2734.
  * Hoxha et al. (2022) Genc Hoxha, Seloua Chouaf, Farid Melgani, and Youcef Smara. 2022.  Change captioning: A new paradigm for multitemporal remote sensing image analysis.  _IEEE Transactions on Geoscience and Remote Sensing_ , 60:1–14.
  * Huang et al. (2022) Qingbao Huang, Yu Liang, Jielong Wei, Yi Cai, Hanyu Liang, Ho-fung Leung, and Qing Li. 2022.  Image difference captioning with instance-level fine-grained feature representation.  _IEEE Transactions on Multimedia_ , 24:2004–2017.
  * Jhamtani and Berg-Kirkpatrick (2018) Harsh Jhamtani and Taylor Berg-Kirkpatrick. 2018.  Learning to describe differences between pairs of similar images.  In _EMNLP_ , pages 4024–4034.
  * Kim et al. (2021) Hoeseong Kim, Jongseok Kim, Hyungseok Lee, Hyunsung Park, and Gunhee Kim. 2021.  Agnostic change captioning with cycle consistency.  In _ICCV_ , pages 2095–2104.
  * Kingma and Ba (2014) Diederik P Kingma and Jimmy Ba. 2014.  Adam: A method for stochastic optimization.  _arXiv preprint arXiv:1412.6980_.
  * Lin (2004) Chin-Yew Lin. 2004.  Rouge: A package for automatic evaluation of summaries.  In _Text summarization branches out_ , pages 74–81.
  * Liu et al. (2023a) Chenyang Liu, Jiajun Yang, Zipeng Qi, Zhengxia Zou, and Zhenwei Shi. 2023a.  Progressive scale-aware network for remote sensing image change captioning.  _IGARSS_.
  * Liu et al. (2022) Chenyang Liu, Rui Zhao, Hao Chen, Zhengxia Zou, and Zhenwei Shi. 2022.  Remote sensing image change captioning with dual-branch transformers: A new method and a large scale dataset.  _IEEE Transactions on Geoscience and Remote Sensing_ , 60:1–20.
  * Liu et al. (2023b) Chenyang Liu, Rui Zhao, Jianqi Chen, Zipeng Qi, Zhengxia Zou, and Zhenwei Shi. 2023b.  A decoupling paradigm with prompt learning for remote sensing image change captioning.  _IEEE Transactions on Geoscience and Remote Sensing_ , 61:1–18.
  * Liu et al. (2021) Fenglin Liu, Changchang Yin, Xian Wu, Shen Ge, Ping Zhang, and Xu Sun. 2021.  Contrastive attention for automatic chest x-ray report generation.  In _Findings of ACL_ , pages 269–280.
  * Oord et al. (2018) Aaron van den Oord, Yazhe Li, and Oriol Vinyals. 2018.  Representation learning with contrastive predictive coding.  _arXiv preprint arXiv:1807.03748_.
  * Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002.  Bleu: a method for automatic evaluation of machine translation.  In _ACL_ , pages 311–318.
  * Park et al. (2019) Dong Huk Park, Trevor Darrell, and Anna Rohrbach. 2019.  Robust change captioning.  In _ICCV_ , pages 4624–4633.
  * Paszke et al. (2019) Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. 2019.  Pytorch: An imperative style, high-performance deep learning library.  In _NeurIPS_ , pages 8024–8035.
  * Qiu et al. (2023) Yue Qiu, Yanjun Sun, Fumiya Matsuzawa, Kenji Iwata, and Hirokatsu Kataoka. 2023.  Graph representation for order-aware visual transformation.  In _CVPR_ , pages 22793–22802.
  * Qiu et al. (2021) Yue Qiu, Shintaro Yamamoto, Kodai Nakashima, Ryota Suzuki, Kenji Iwata, Hirokatsu Kataoka, and Yutaka Satoh. 2021.  Describing and localizing multiple changes with transformers.  In _ICCV_ , pages 1971–1980.
  * Rotstein et al. (2024) Noam Rotstein, David Bensaïd, Shaked Brody, Roy Ganz, and Ron Kimmel. 2024.  Fusecap: Leveraging large language models for enriched fused image captions.  In _WACV_ , pages 5689–5700.
  * Shi et al. (2020) Xiangxi Shi, Xu Yang, Jiuxiang Gu, Shafiq Joty, and Jianfei Cai. 2020.  Finding it at another side: A viewpoint-adapted matching encoder for change captioning.  In _ECCV_ , pages 574–590.
  * Song et al. (2007) Le Song, Alex Smola, Arthur Gretton, Karsten M Borgwardt, and Justin Bedo. 2007.  Supervised feature selection via dependence estimation.  In _ICML_ , pages 823–830.
  * Tu et al. (2023a) Yunbin Tu, Liang Li, Li Su, Junping Du, Ke Lu, and Qingming Huang. 2023a.  Viewpoint-adaptive representation disentanglement network for change captioning.  _IEEE Transactions on Image Processing_ , 32:2620–2635.
  * Tu et al. (2022) Yunbin Tu, Liang Li, Li Su, Shengxiang Gao, Chenggang Yan, Zheng-Jun Zha, Zhengtao Yu, and Qingming Huang. 2022.  I2 transformer: Intra-and inter-relation embedding transformer for tv show captioning.  _IEEE Transactions on Image Processing_ , 31:3565–3577.
  * Tu et al. (2023b) Yunbin Tu, Liang Li, Li Su, Ke Lu, and Qingming Huang. 2023b.  Neighborhood contrastive transformer for change captioning.  _IEEE Transactions on Multimedia_ , pages 1–12.
  * Tu et al. (2024) Yunbin Tu, Liang Li, Li Su, Zheng-Jun Zha, and Qingming Huang. 2024.  Smart: Syntax-calibrated multi-aspect relation transformer for change captioning.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_.
  * Tu et al. (2023c) Yunbin Tu, Liang Li, Li Su, Zheng-Jun Zha, Chenggang Yan, and Qingming Huang. 2023c.  Self-supervised cross-view representation reconstruction for change captioning.  In _ICCV_ , pages 2805–2815.
  * Tu et al. (2021a) Yunbin Tu, Liang Li, Chenggang Yan, Shengxiang Gao, and Zhengtao Yu. 2021a.  R^3Net:relation-embedded representation reconstruction network for change captioning.  In _EMNLP_ , pages 9319–9329.
  * Tu et al. (2021b) Yunbin Tu, Tingting Yao, Liang Li, Jiedong Lou, Shengxiang Gao, Zhengtao Yu, and Chenggang Yan. 2021b.  Semantic relation-aware difference representation learning for change captioning.  In _Findings of ACL_ , pages 63–73.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  In _NeurIPS_ , pages 5998–6008.
  * Vedantam et al. (2015) Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. 2015.  Cider: Consensus-based image description evaluation.  In _CVPR_ , pages 4566–4575.
  * Yang et al. (2023) Xu Yang, Jiawei Peng, Zihua Wang, Haiyang Xu, Qinghao Ye, Chenliang Li, Songfang Huang, Fei Huang, Zhangzikang Li, and Yu Zhang. 2023.  Transforming visual scene graphs to image captions.  In _ACL_ , pages 12427–12440.
  * Yao et al. (2022) Linli Yao, Weiying Wang, and Qin Jin. 2022.  Image difference captioning with pre-training and contrastive learning.  In _AAAI_.
  * Yue et al. (2024) Shengbin Yue, Yunbin Tu, Liang Li, Shengxiang Gao, and Zhengtao Yu. 2024.  Multi-grained representation aggregating transformer with gating cycle for change captioning.  _ACM Transactions on Multimedia Computing, Communications and Applications_.
  * Yue et al. (2023) Shengbin Yue, Yunbin Tu, Liang Li, Ying Yang, Shengxiang Gao, and Zhengtao Yu. 2023.  I3n: Intra- and inter-representation interaction network for change captioning.  _IEEE Transactions on Multimedia_ , pages 1–14.
  * Zhao and Xiong (2024) Kai Zhao and Wei Xiong. 2024.  Cooperative connection transformer for remote sensing image captioning.  _IEEE Transactions on Geoscience and Remote Sensing_.

##  Appendix A Appendix

In this appendix, we provide more details about the caption generation and
more experimental results, as well as the qualitative analyses.

###  A.1 Caption Generation

After obtaining the representation of all changes
$\tilde{X}_{d}\in\mathbb{R}^{hw\times d}$, we use a transformer decoder to
decode it into target sentences. First, we obtain the embedding features of
all $m$ words of these sentence $E[S]\in\mathbb{R}^{m\times d}$, where each
sentence is separated by a special token [SEP]. Then, we use the masked self-
attention Vaswani et al. (2017) to model relationships among these word
features, which is defined as:

${\hat{E}[S]}=\text{ LN }(E[S]+\text{ MHSA }(E[S],E[S],E[S])),$ (11)

where LN is short for layer normalization Ba et al. (2016). Next, we model the
interaction between these word features and difference representation
$\tilde{X}_{d}$ by multi-head cross-attention (MHCA) Vaswani et al. (2017), so
as to locate the most related difference features $\tilde{H}$:

$\tilde{H}=\text{ LN }({E[\hat{S}]}+\text{ MHCA
}({E[\hat{S}]},\tilde{X}_{d},\tilde{X}_{d})).$ (12)

Subsequently, we feed the selected features $\tilde{H}$ into a feed-forward
network to obtain the enhanced difference representation, denoted as
$\hat{H}\in\mathbb{R}^{m\times d}$.

$\hat{H}=\text{ LN}((\tilde{H}+\operatorname{FFN}(\tilde{H})).$ (13)

Finally, the probability distributions of words in these sentences are
calculated via a single hidden layer:

$S=\operatorname{Softmax}\left(\hat{H}W_{s}+{b}_{s}\right),$ (14)

where $W_{s}\in\mathbb{R}^{d\times u}$ and $b_{s}\in\mathbb{R}^{u}$ are the
learnable parameters. $u$ is the dimension of vocabulary size.

###  A.2 Experiments

####  A.2.1 Implementation Details

For fair-comparison, we follow the previous multi-change captioning methods
Jhamtani and Berg-Kirkpatrick (2018); Qiu et al. (2021); Liu et al. (2022) to
use a pre-trained ResNet-101 (He et al., 2016) to extract the local features
of a pair of images, with the dimension of 1024 $\times$ 14 $\times$ 14\. We
project them into a lower dimension of 512, while the dimension of trainable
[CLS] features is also set to 512. The hidden size of overall model and word
embedding size are set to 512 and 300, respectively. Temperature $\tau$ in Eq.
(2) of main paper is set to 0.07. The attention layers in CARD is set to 1 on
the CLEVR-Multi-Change and Spot-the-Diff datasets; and 3 on the LEVIR-CC
dataset. We train the model with PyTorch Paszke et al. (2019) on a single RTX
3090 GPU, and use Adam optimizer (Kingma and Ba, 2014) to minimize the
negative log-likelihood loss of Eq. (10) in the main paper. The training
details about batch size and learning rate are shown in Table 7. The used
training resources about time and GPU memory are shown in Table 8. We find
that training CARD does not require much more time and GPU memory. Hence, it
would be easily re-implemented by other researchers and be a strong baseline
for the future works.

Table 7: The training details of CARD on the three datasets. | batch size | learning rate
---|---|---
CLEVR-Multi-Change | 128 | 2 $\times$ $10^{-4}$
LEVIR-CC | 64 | 1 $\times$ $10^{-4}$
Spot-the-Diff | 32 | 2 $\times$ $10^{-4}$
Table 8: The used training resources of CARD on the three datasets. | Training Time | GPU Memory
---|---|---
CLEVR-Multi-Change | 71 minutes | 9.2 GB
LEVIR-CC | 120 minutes | 8 GB
Spot-the-Diff | 20 minutes | 3.9 GB
Table 9: Effects of $\lambda_{c}$ on CLEVR-Multi-Change. Model | $\lambda_{c}$ | B | M | R | S | C
---|---|---|---|---|---|---
CARD | 0 | 56.2 | 45.3 | 57.3 | 76.9 | 387.8
CARD | 0.1 | 56.6 | 45.2 | 57.4 | 77.0 | 391.2
CARD | 0.2 | 56.6 | 45.3 | 57.3 | 77.0 | 390.7
CARD | 0.3 | 56.7 | 45.2 | 57.4 | 76.9 | 391.6
CARD | 0.4 | 56.5 | 45.2 | 57.0 | 76.8 | 388.8
CARD | 0.5 | 56.6 | 45.3 | 57.4 | 76.9 | 390.7
Table 10: Effects of $\lambda_{c}$ on the LEVIR-CC dataset. Model | $\lambda_{c}$ | B | M | R | C
---|---|---|---|---|---
CARD | 0 | 55.9 | 35.6 | 72.3 | 132.2
CARD | 0.1 | 65.4 | 40.0 | 74.6 | 137.9
CARD | 0.2 | 63.7 | 39.3 | 73.3 | 132.9
CARD | 0.3 | 54.8 | 34.4 | 72.7 | 137.5
CARD | 0.4 | 63.2 | 38.3 | 73.6 | 137.5
CARD | 0.5 | 59.5 | 36.9 | 72.9 | 135.1
Table 11: Effects of $\lambda_{c}$ on the Spot-the-Diff dataset. Model | $\lambda_{c}$ | B | M | R | S | C
---|---|---|---|---|---|---
CARD | 0 | 4.3 | 10.6 | 23.3 | 13.0 | 15.8
CARD | 0.001 | 6.6 | 10.8 | 26.9 | 17.8 | 32.4
CARD | 0.002 | 4.3 | 9.8 | 23.9 | 15.1 | 23.0
CARD | 0.003 | 6.2 | 9.5 | 25.7 | 15.7 | 28.4
CARD | 0.004 | 5.0 | 11.0 | 24.4 | 15.2 | 20.5
CARD | 0.005 | 5.1 | 11.4 | 24.0 | 15.6 | 18.3

###  A.3 Study on the Trade-off Parameter $\lambda_{c}$

We discuss the influence of trade-off parameter $\lambda_{c}$ in Eq. (10). The
results under varied values are shown in Table 9-11. Note that $\lambda_{c}$=0
means there are no constraints upon the decoupled results. We observe that
imposing the constraint losses does improve the model’s performance. Besides,
there is a trade-off between the captioning loss and constraint losses,
because too large $\lambda_{c}$ may lead to deterioration in caption
performance. Based on the results, we choose the value as 0.3 on CLEVR-Multi-
Change, 0.1 and 0.001 on the LEVIR-CC and Spot-the-Diff, respectively.

Figure 6: Visualization of common objects matching on the CLEVR-Multi-Change
dataset under one-to-four changes. For each example, we visualize the matching
results by the state-of-the-art method MCCFormers-D Qiu et al. (2021) and our
CARD. The common objects are shown in the yellow boxes.  Figure 7:
Visualization of common objects matching on the LEVIR-CC and Spot-the-Diff
datasets under varied changes. For each example, we visualize the matching
results by the state-of-the-art method MCCFormers-D Qiu et al. (2021) and our
CARD. The common objects are shown in the yellow boxes.  Figure 8: Qualitative
examples on the CLEVR-Multi-Change dataset under one-to-four changes. For each
example, we visualize the captions generated by the state-of-the-art method
MCCFormers-D Qiu et al. (2021) and our CARD, as well as the change
localization of CARD. The changed objects are shown in the colored boxes.
Figure 9: Qualitative examples on the LEVIR-CC and Spot-the-Diff datasets
under varied changes. For each example, we visualize the captions generated by
the state-of-the-art method MCCFormers-D Qiu et al. (2021) and our CARD, as
well as the change localization of CARD. The changed objects are shown in the
colored boxes.

###  A.4 Qualitative Analysis

In this appendix, we will show more qualitative examples on the CLEVR-Multi-
Change, LEVIR-CC, and Spot-the-Diff datasets, which are shown in Figure 6-9.
To intuitively understand whether the common context features help mine
reliable common properties, we visualize the alignment of common properties
between two images on the three datasets, which are shown in Figure 6-7. The
compared method is MCCFormers-D Qiu et al. (2021), which has the stable
performance on the three datasets. From these examples, we can observe that
MCCFormers-D is unable to align the common properties properly and even
misjudges changed objects as unchanged objects. Compared with it, the proposed
CARD can better match the common objects, so as to validate the effectiveness
of the decoupled common context features.

Further, in Figure 8-9, we visualize the captions yielded by MCCFormers-D and
CARD, as well as the change localization results from CARD on the three
datasets. In Figure 8, on the CLEVR-Multi-Change dataset under one-to-four
changes, we find that MCCFormers-D either only describes partial changes or
misidentifies changed objects. Instead, the proposed CARD is able to
accurately locate and describe all changed objects. In Figure 9, on the LEVIR-
CC and Spot-the-Diff datasets, it is noted that MCCFormers-D fails to describe
all the changes within each image pair that is from the real-word environment.
Different it, our CARD is capable of locating and describing all changed
objects, which show a good generalization and robustness of our method. The
superiority benefits from that the proposed CARD can provide the model with a
overview of potential changed/unchanged semantics within an image pair, which
helps learn genuine changes for caption generation.


* [199] Seth A. Jacobson et al.  “Formation, Stratification, and Mixing of the Cores of Earth and Venus”  In _Earth and Planetary Science Letters_ 474, 2017  DOI: 10.1016/j.epsl.2017.06.023
  * [200] Robin Ramstad and Stas Barabash  “Do Intrinsic Magnetic Fields Protect Planetary Atmospheres from Stellar Winds? Lessons from Ion Measurements at Mars, Venus, and Earth”  In _Space Science Reviews_ 217 Springer, 2021, pp. 1–39
  * [201] DA Brain et al.  “Atmospheric escape from unmagnetized bodies”  In _Journal of Geophysical Research: Planets_ 121.12 Wiley Online Library, 2016, pp. 2364–2385
  * [202] Herbert Gunell et al.  “Why an intrinsic magnetic field does not protect a planet against atmospheric escape”  In _Astronomy & Astrophysics_ 614 EDP Sciences, 2018, pp. L3
  * [203] Chuanfei Dong, Zhenguang Huang and Manasvi Lingam  “Role of planetary obliquity in regulating atmospheric escape: G-dwarf versus M-dwarf Earth-like exoplanets”  In _The Astrophysical Journal Letters_ 882.2 IOP Publishing, 2019, pp. L16
  * [204] VS Airapetian et al.  “Impact of space weather on climate and habitability of terrestrial-type exoplanets”  In _International Journal of Astrobiology_ 19.2 Cambridge University Press, 2020, pp. 136–194
  * [205] H.. Urey  “The planets: Their origin and development”  In _Mrs. Hepsa Ely Silliman Memorial Lectures_ , 1952
  * [206] T.. Donahue and J.. Pollack  “Origin and evolution of the atmosphere of Venus.”  In _IN: Venus (A83-37401 17-91). Tucson, AZ, University of Arizona Press, 1983, p. 1003-1036. NASA-supported research._ University of Arizona Press, 1983, pp. 1003–1036
  * [207] K Hans Wedepohl  “The composition of the continental crust”  In _Geochimica et cosmochimica Acta_ 59.7 Elsevier, 1995, pp. 1217–1232
  * [208] Christophe Lécuyer, Laurent Simon and François Guyot  “Comparison of carbon, nitrogen and water budgets on Venus and the Earth”  In _Earth and Planetary Science Letters_ 181.1, 2000, pp. 33–40  DOI: https://doi.org/10.1016/S0012-821X(00)00195-3
  * [209] Jens Hartmann et al.  “The geochemical composition of the terrestrial surface (without soils) and comparison with the upper continental crust”  In _International Journal of Earth Sciences_ 101.1 Springer, 2012, pp. 365–376
  * [210] R.. Berner, A.. Lasaga and R.. Garrels  “The carbonate-silicate geochemical cycle and its effect on atmospheric carbon dioxide over the past 100 million years”  In _American Journal of Science_ 283, 1983, pp. 641–683  DOI: 10.2475/ajs.283.7.641
  * [211] N.. Sleep and K. Zahnle  “Carbon dioxide cycling and implications for climate on ancient Earth”  In _Journal of Geophysical Research: Planets_ 106, 2001, pp. 1373–1400  DOI: 10.1029/2000JE001247
  * [212] Joshua Krissansen-Totton, Ryan Garland, Patrick Irwin and David C. Catling  “Detectability of Biosignatures in Anoxic Atmospheres with the James Webb Space Telescope: A TRAPPIST-1e Case Study”  In _AJ_ 156.3, 2018, pp. 114  DOI: 10.3847/1538-3881/aad564
  * [213] J… Walker, P.. Hays and J.. Kasting  “A negative feedback mechanism for the long-term stabilization of the earth’s surface temperature”  In _Journal of Geophysical Research: Planets_ 86, 1981, pp. 9776–9782  DOI: 10.1029/JC086iC10p09776
  * [214] Lee R. Kump, Susan L. Brantley and Michael A. Arthur  “Chemical Weathering, Atmospheric CO2, and Climate”  In _Annual Review of Earth and Planetary Sciences_ 28, 2000, pp. 611–667  DOI: 10.1146/annurev.earth.28.1.611
  * [215] Richard E. Zeebe and Peter Westbroek  “A simple model for the CaCO3 saturation state of the ocean: The “Strangelove,“ the “Neritan,” and the “Cretan” Ocean”  In _Geochemistry, Geophysics, Geosystems_ 4.12, 2003, pp. 1104  DOI: 10.1029/2003GC000538
  * [216] Peter B. Kelemen and Craig E. Manning  “Reevaluating carbon fluxes in subduction zones, what goes down, mostly comes up”  In _Proceedings of the National Academy of Science_ 112.30, 2015, pp. E3997–E4006  DOI: 10.1073/pnas.1507889112
  * [217] David C Catling and James F Kasting  “Atmospheric evolution on inhabited and lifeless worlds”  Cambridge University Press, 2017
  * [218] Carl Sagan and George Mullen  “Earth and Mars: Evolution of Atmospheres and Surface Temperatures”  In _Science_ 177.4043, 1972, pp. 52–56  DOI: 10.1126/science.177.4043.52
  * [219] David C. Catling and Kevin J. Zahnle  “The Archean atmosphere”  In _Science Advances_ 6.9, 2020, pp. eaax1420  DOI: 10.1126/sciadv.aax1420
  * [220] N.. Bergman  “COPSE: A new model of biogeochemical cycling over Phanerozoic time”  In _American Journal of Science_ 304.5, 2004, pp. 397–437  DOI: 10.2475/ajs.304.5.397
  * [221] Dennis Höning  “The Impact of Life on Climate Stabilization Over Different Timescales”  In _Geochemistry, Geophysics, Geosystems_ 21.9, 2020, pp. e2020GC009105  DOI: 10.1029/2020GC009105
  * [222] G. Colbourn, A. Ridgwell and T.. Lenton  “The time scale of the silicate weathering negative feedback on atmospheric CO2”  In _Global Biogeochemical Cycles_ 29.5, 2015, pp. 583–596  DOI: 10.1002/2014GB005054
  * [223] Bradford J. Foley  “The Role of Plate Tectonic-Climate Coupling and Exposed Land Area in the Development of Habitable Climates on Rocky Planets”  In _ApJ_ 812.1, 2015, pp. 36  DOI: 10.1088/0004-637X/812/1/36
  * [224] K. Maher and C.. Chamberlain  “Hydrologic Regulation of Chemical Weathering and the Geologic Carbon Cycle”  In _Science_ 343.6178, 2014, pp. 1502–1504  DOI: 10.1126/science.1250770
  * [225] Bradford J. Foley and Andrew J. Smye  “Carbon Cycling and Habitability of Earth-Sized Stagnant Lid Planets”  In _Astrobiology_ 18.7, 2018, pp. 873–896  DOI: 10.1089/ast.2017.1695
  * [226] Dennis Höning, Nicola Tosi and Tilman Spohn  “Carbon cycling and interior evolution of water-covered plate tectonics and stagnant-lid planets”  In _A &A_ 627, 2019, pp. A48  DOI: 10.1051/0004-6361/201935091
  * [227] Joshua Krissansen-Totton and David C. Catling  “Constraining climate sensitivity and continental versus seafloor weathering using an inverse geological carbon cycle model”  In _Nature Communications_ 8, 2017, pp. 15423  DOI: 10.1038/ncomms15423
  * [228] Matthew J. Winnick and Kate Maher  “Relationships between CO2, thermodynamic limits on silicate weathering, and the strength of the silicate weathering feedback”  In _Earth and Planetary Science Letters_ 485, 2018, pp. 111–120  DOI: 10.1016/j.epsl.2018.01.005
  * [229] Kaustubh Hakim et al.  “Lithologic Controls on Silicate Weathering Regimes of Temperate Planets”  In _The planetary science journal_ 2.2, 2021, pp. 49  DOI: 10.3847/PSJ/abe1b8
  * [230] R.. Graham and Ray Pierrehumbert  “Thermodynamic and Energetic Limits on Continental Silicate Weathering Strongly Impact the Climate and Habitability of Wet, Rocky Worlds”  In _ApJ_ 896.2, 2020, pp. 115  DOI: 10.3847/1538-4357/ab9362
  * [231] Daniel E. Ibarra et al.  “Differential weathering of basaltic and granitic catchments from concentration-discharge relationships”  In _Geochimica et Cosmochimica Acta_ 190, 2016, pp. 265–293  DOI: 10.1016/j.gca.2016.07.006
  * [232] Keith D. Putirka and John C. Rarick  “The composition and mineralogy of rocky exoplanets: A survey of ¿4000 stars from the Hypatia Catalog”  In _American Mineralogist_ 104.6, 2019, pp. 817–829  DOI: 10.2138/am-2019-6787
  * [233] Mikhail Yu. Zolotov  “Gas–Solid Interactions on Venus and Other Solar System Bodies”  In _Reviews in Mineralogy and Geochemistry_ 84.1, 2018, pp. 351–392  DOI: 10.2138/rmg.2018.84.10
  * [234] Wallace S. Broecker and Tsung-Hung Peng  “The role of CaCO3 compensation in the glacial to interglacial atmospheric CO2 change”  In _Global Biogeochemical Cycles_ 1.1, 1987, pp. 15–29  DOI: 10.1029/GB001i001p00015
  * [235] Kaustubh Hakim, Meng Tian, Dan J. Bower and Kevin Heng  “Diverse Carbonates in Exoplanet Oceans Promote the Carbon Cycle”  In _ApJL_ 942.1, 2023, pp. L20  DOI: 10.3847/2041-8213/aca90c
  * [236] Laurence A. Coogan and Kathryn M. Gillis  “Evidence that low-temperature oceanic hydrothermal systems play an important role in the silicate-carbonate weathering cycle and long-term climate regulation”  In _Geochemistry, Geophysics, Geosystems_ 14.6, 2013, pp. 1771–1786  DOI: 10.1002/ggge.20113
  * [237] Edwin S. Kite and Eric B. Ford  “Habitability of Exoplanet Waterworlds”  In _ApJ_ 864.1, 2018, pp. 75  DOI: 10.3847/1538-4357/aad6e0
  * [238] S. Kadoya and E. Tajika  “Outer Limits of the Habitable Zones in Terms of Climate Mode and Climate Evolution of Earth-like Planets”  In _ApJ_ 875.1, 2019, pp. 7  DOI: 10.3847/1538-4357/ab0aef
  * [239] Jun Yang, Gwenaël Boué, Daniel C. Fabrycky and Dorian S. Abbot  “Strong Dependence of the Inner Edge of the Habitable Zone on Planetary Rotation Rate”  In _ApJL_ 787.1, 2014, pp. L2  DOI: 10.1088/2041-8205/787/1/L2
  * [240] E.. Wolf and O.. Toon  “Delayed onset of runaway and moist greenhouse climates for Earth”  In _Geophysical Research Letters_ 41.1, 2014, pp. 167–172  DOI: 10.1002/2013GL058376
  * [241] E.. Wolf and O.. Toon  “The evolution of habitable climates under the brightening Sun”  In _Journal of Geophysical Research (Atmospheres)_ 120.12, 2015, pp. 5775–5794  DOI: 10.1002/2015JD023302
  * [242] M.. Way et al.  “Climates of Warm Earth-like Planets. I. 3D Model Simulations”  In _The Astrophysical Journal Supplement Series_ 239.2, 2018, pp. 24  DOI: 10.3847/1538-4365/aae9e1
  * [243] Jacob L. Bean, Dorian S. Abbot and Eliza M.-R. Kempton  “A Statistical Comparative Planetology Approach to the Hunt for Habitable Exoplanets and Life Beyond the Solar System”  In _ApJL_ 841.2, 2017, pp. L24  DOI: 10.3847/2041-8213/aa738a
  * [244] M. Turbet  “Two examples of how to use observations of terrestrial planets orbiting in temperate orbits around low mass stars to test key concepts of planetary habitability”  In _SF2A-2019: Proceedings of the Annual meeting of the French Society of Astronomy and Astrophysics_ , 2019, pp. Di
  * [245] G.. Simpson  “The distribution of terrestrial radiation”  In _Quarterly Journal of the Royal Meteorological Society_ 55.229, 1929, pp. 73–73  DOI: 10.1002/qj.49705522908
  * [246] M. Komabayasi  “Discrete Equilibrium Temperatures of a Hypothetical Planet with the Atmosphere and the Hydrosphere of One Component-Two Phase System under Constant Solar Radiation”  In _Journal of the Meteorological Society of Japan. Ser. II_ 45.1, 1967, pp. 137–139  DOI: 10.2151/jmsj1965.45.1˙137
  * [247] Andrew P. Ingersoll  “The Runaway Greenhouse: A History of Water on Venus.”  In _Journal of Atmospheric Sciences_ 26.6, 1969, pp. 1191–1198  DOI: 10.1175/1520-0469(1969)026¡1191:TRGAHO¿2.0.CO;2
  * [248] Shinichi Nakajima, Yoshi-Yuki Hayashi and Yutaka Abe  “A study on the ’runaway greenhouse effect’ with a one-dimensional radiative-convective equilibrium model”  In _Journal of Atmospheric Sciences_ 49.23, 1992, pp. 2256–2266  DOI: 10.1175/1520-0469(1992)049¡2256:ASOTGE¿2.0.CO;2
  * [249] Dennis Höning et al.  “Early Habitability and Crustal Decarbonation of a Stagnant-Lid Venus”  In _Journal of Geophysical Research (Planets)_ 126.10, 2021, pp. e06895  DOI: 10.1029/2021JE006895
  * [250] Owen R. Lehmer, David C. Catling and Joshua Krissansen-Totton  “Carbonate-silicate cycle predictions of Earth-like planetary climates and testing the habitable zone concept”  In _Nature Communications_ 11, 2020, pp. 6153  DOI: 10.1038/s41467-020-19896-2
  * [251] Martin Schlecker et al.  “Bioverse: The Habitable Zone Inner Edge Discontinuity as an Imprint of Runaway Greenhouse Climates on Exoplanet Demographics”  In _arXiv e-prints_ , 2023, pp. arXiv:2309.04518  DOI: 10.48550/arXiv.2309.04518
  * [252] H. Rauer et al.  “The PLATO 2.0 mission”  In _Experimental Astronomy_ 38.1-2, 2014, pp. 249–330  DOI: 10.1007/s10686-014-9383-4
  * [253] R.. Graham, Tim Lichtenberg and Raymond T. Pierrehumbert  “CO2 Ocean Bistability on Terrestrial Exoplanets”  In _Journal of Geophysical Research (Planets)_ 127.10, 2022, pp. e2022JE007456  DOI: 10.1029/2022JE007456
  * [254] Kevin Zahnle et al.  “Emergence of a habitable planet”  In _Space Science Reviews_ 129.1-3 Springer, 2007, pp. 35–78
  * [255] Harold P Klein  “The Viking mission and the search for life on Mars”  In _Reviews of Geophysics_ 17.7 Wiley Online Library, 1979, pp. 1655–1662
  * [256] Michael H Carr  “Water on early Mars”  In _Ciba Foundation Symposium 202-Evolution of Hydrothermal Ecosystems on Earth (And Mars?) Evolution of Hydrothermal Ecosystems on Earth (And Mars?): Ciba Foundation Symposium 202_ , 2007, pp. 249–272  Wiley Online Library
  * [257] Jean-Pierre Bibring et al.  “Global mineralogical and aqueous Mars history derived from OMEGA/Mars Express data”  In _science_ 312.5772 American Association for the Advancement of Science, 2006, pp. 400–404
  * [258] Steven Squyres  “Roving Mars: Spirit, Opportunity, and the exploration of the red planet”  Hachette UK, 2005
  * [259] Robin D Wordsworth  “The climate of early Mars”  In _Annual Review of Earth and Planetary Sciences_ 44 Annual Reviews, 2016, pp. 381–408
  * [260] Janice L Bishop et al.  “Surface clay formation during short-term warmer and wetter conditions on a largely cold ancient Mars”  In _Nature Astronomy_ 2.3 Nature Publishing Group UK London, 2018, pp. 206–213
  * [261] Samuel P Kounaves et al.  “Identification of the perchlorate parent salts at the Phoenix Mars landing site and possible implications”  In _Icarus_ 232 Elsevier, 2014, pp. 226–231
  * [262] Christopher R Webster et al.  “Mars methane detection and variability at Gale crater”  In _Science_ 347.6220 American Association for the Advancement of Science, 2015, pp. 415–417
  * [263] Renyu Hu et al.  “Hypotheses for near-surface exchange of methane on Mars”  In _Astrobiology_ 16.7 Mary Ann Liebert, Inc. 140 Huguenot Street, 3rd Floor New Rochelle, NY 10801 USA, 2016, pp. 539–550
  * [264] W Bruce Banerdt et al.  “Initial results from the InSight mission on Mars”  In _Nature Geoscience_ 13.3 Nature Publishing Group UK London, 2020, pp. 183–189
  * [265] Richard Mattingly and Lisa May  “Mars sample return as a campaign”  In _2011 Aerospace Conference_ , 2011, pp. 1–13  IEEE
  * [266] David W Beaty et al.  “The potential science and engineering value of samples delivered to Earth by Mars sample return: International MSR Objectives and Samples Team (iMOST)”  In _Meteoritics & Planetary Science_ 54 Wiley Online Library, 2019, pp. S3–S152
  * [267] Joseph O’Rourke et al.  “Venus, the planet: introduction to the evolution of Earth’s sister planet”  In _Authorea Preprints_ Authorea, 2022
  * [268] Michael J Way and Anthony D Del Genio  “Venusian habitable climate scenarios: Modeling Venus through time and applications to slowly rotating Venus-like exoplanets”  In _Journal of Geophysical Research: Planets_ 125.5 Wiley Online Library, 2020, pp. e2019JE006276
  * [269] Martin Turbet et al.  “Day–night cloud asymmetry prevents early oceans on Venus but not on Earth”  In _Nature_ 598.7880 Nature Publishing Group, 2021, pp. 276–280
  * [270] Sanjay S Limaye et al.  “Venus, an astrobiology target”  In _Astrobiology_ 21.10 Mary Ann Liebert, Inc., publishers 140 Huguenot Street, 3rd Floor New …, 2021, pp. 1163–1185
  * [271] Jane S. Greaves et al.  “Phosphine gas in the cloud decks of Venus”  In _Nature Astronomy_ 5, 2021, pp. 655–664  DOI: 10.1038/s41550-020-1174-4
  * [272] L. Trompet et al.  “Phosphine in Venus’ atmosphere: Detection attempts and upper limits above the cloud top assessed from the SOIR/VEx spectra”  In _Astronomy & Astrophysics_ 645, 2021, pp. L4  DOI: 10.1051/0004-6361/202039932
  * [273] T. Encrenaz et al.  “A stringent upper limit of the PH3 abundance at the cloud top of Venus”  In _Astronomy I & Astrophysics_ 643, 2020, pp. L5  DOI: 10.1051/0004-6361/202039559
  * [274] I… Snellen et al.  “Re-analysis of the 267 GHz ALMA observations of Venus. No statistically significant detection of phosphine”  In _Astronomy & Astrophysics_ 644, 2020, pp. L2  DOI: 10.1051/0004-6361/202039717
  * [275] M.. Thompson  “The statistical reliability of 267-GHz JCMT observations of Venus: no significant evidence for phosphine absorption”  In _Monthly Notices of the Royal Astronomical Society: Letters_ 501.1, 2021, pp. L18–L22  DOI: 10.1093/mnrasl/slaa187
  * [276] G.. Villanueva et al.  “No evidence of phosphine in the atmosphere of Venus from independent analyses”  In _Nature Astronomy_ 5, 2021, pp. 631–635  DOI: 10.1038/s41550-021-01422-z
  * [277] Fredric W Taylor, Håkan Svedhem and James W Head  “Venus: the atmosphere, climate, surface, interior and near-space environment of an Earth-like planet”  In _Space Science Reviews_ 214 Springer, 2018, pp. 1–36
  * [278] Paul K Byrne et al.  “Venus tesserae feature layered, folded, and eroded rocks”  In _Geology_ 49.1 Geological Society of America, 2021, pp. 81–85
  * [279] Jennifer Whitten et al.  “Venus Tesserae: The importance of Venus tesserae and remaining open questions”  In _Bulletin of the American Astronomical Society_ 53.4, 2021, pp. 012
  * [280] Michael J Way et al.  “Synergies Between Venus & Exoplanetary Observations: Venus and Its Extrasolar Siblings”  In _Space Science Reviews_ 219.1 Springer, 2023, pp. 13
  * [281] Christopher P McKay and Heather D Smith  “Possibilities for methanogenic life in liquid methane on the surface of Titan”  In _Icarus_ 178.1 Elsevier, 2005, pp. 274–276
  * [282] Christopher F Chyba and Cynthia B Phillips  “Possible ecosystems and the search for life on Europa”  In _Proceedings of the National Academy of Sciences_ 98.3 National Acad Sciences, 2001, pp. 801–804
  * [283] Laura M Barge and Laura E Rodriguez  “Life on Enceladus? It depends on its origin”  In _Nature Astronomy_ 5.8 Nature Publishing Group UK London, 2021, pp. 740–741
  * [284] Steve Vance, Mathieu Bouffard, Mathieu Choukroun and Christophe Sotin  “Ganymede s internal structure including thermodynamics of magnesium sulfate oceans in contact with ice”  In _Planetary and Space Science_ 96 Elsevier, 2014, pp. 62–70
  * [285] Mark A Sephton, Jack Hunter Waite and Tim G Brockwell  “How to detect life on icy moons”  In _Astrobiology_ 18.7 Mary Ann Liebert, Inc. 140 Huguenot Street, 3rd Floor New Rochelle, NY 10801 USA, 2018, pp. 843–855
  * [286] Olivier Grasset et al.  “JUpiter ICy moons Explorer (JUICE): An ESA mission to orbit Ganymede and to characterise the Jupiter system”  In _Planetary and Space Science_ 78 Elsevier, 2013, pp. 1–21
  * [287] Cynthia B Phillips and Robert T Pappalardo  “Europa Clipper mission concept: Exploring Jupiter’s ocean moon”  In _Eos, Transactions American Geophysical Union_ 95.20 Wiley Online Library, 2014, pp. 165–167
  * [288] Ralph D Lorenz et al.  “Dragonfly: A rotorcraft lander concept for scientific exploration at Titan”  In _Johns Hopkins APL Technical Digest_ 34.3, 2018, pp. 14
  * [289] Jason W Barnes et al.  “Science goals and objectives for the Dragonfly Titan rotorcraft relocatable lander”  In _The Planetary Science Journal_ 2.4 IOP Publishing, 2021, pp. 130
  * [290] Sara Seager and Drake Deming  “Exoplanet Atmospheres”  In _ARAA_ 48, 2010, pp. 631–672  DOI: 10.1146/annurev-astro-081309-130837
  * [291] Giovanna Tinetti et al.  “A chemical survey of exoplanets with ARIEL”  In _Experimental Astronomy_ 46.1, 2018, pp. 135–209  DOI: 10.1007/s10686-018-9598-x
  * [292] Michaël Gillon et al.  “Seven temperate terrestrial planets around the nearby ultracool dwarf star TRAPPIST-1”  In _Nature_ 542.7642, 2017, pp. 456–460  DOI: 10.1038/nature21360
  * [293] Thomas P. Greene et al.  “Thermal emission from the Earth-sized exoplanet TRAPPIST-1 b using JWST”  In _Nature_ 618.7963, 2023, pp. 39–42  DOI: 10.1038/s41586-023-05951-7
  * [294] Sebastian Zieba et al.  “No thick carbon dioxide atmosphere on the rocky exoplanet TRAPPIST-1 c”  In _arXiv e-prints_ , 2023, pp. arXiv:2306.10150  DOI: 10.48550/arXiv.2306.10150
  * [295] Jacob Lustig-Yaeger, Victoria S. Meadows and Andrew P. Lincowski  “The Detectability and Characterization of the TRAPPIST-1 Exoplanet Atmospheres with JWST”  In _AJ_ 158.1, 2019, pp. 27  DOI: 10.3847/1538-3881/ab21e0
  * [296] Dániel Apai et al.  “A Thousand Earths: A Very Large Aperture, Ultralight Space Telescope Array for Atmospheric Biosignature Surveys”  In _AJ_ 158.2, 2019, pp. 83  DOI: 10.3847/1538-3881/ab2631
  * [297] David J. Des Marais et al.  “Remote Sensing of Planetary Properties and Biosignatures on Extrasolar Terrestrial Planets”  In _Astrobiology_ 2.2, 2002, pp. 153–181  DOI: 10.1089/15311070260192246
  * [298] M. Kasper et al.  “PCS — A Roadmap for Exoearth Imaging with the ELT”  In _The Messenger_ 182, 2021, pp. 38–43  DOI: 10.18727/0722-6691/5221
  * [299] I. Snellen et al.  “Combining high-dispersion spectroscopy with high contrast imaging: Probing rocky planets around our nearest neighbors”  In _A &A_ 576, 2015, pp. A59  DOI: 10.1051/0004-6361/201425018
  * [300] The LUVOIR Team  “The LUVOIR Mission Concept Study Final Report”  In _arXiv e-prints_ , 2019, pp. arXiv:1912.06219  DOI: 10.48550/arXiv.1912.06219
  * [301] B. Gaudi et al.  “The Habitable Exoplanet Observatory (HabEx) Mission Concept Study Final Report”  In _arXiv e-prints_ , 2020, pp. arXiv:2001.06683  DOI: 10.48550/arXiv.2001.06683
  * [302] Y. Feng et al.  “Characterizing Earth Analogs in Reflected Light: Atmospheric Retrieval Studies for Future Space Telescopes”  In _AJ_ 155.5, 2018, pp. 200  DOI: 10.3847/1538-3881/aab95c
  * [303] Sascha P. Quanz et al.  “Direct detection of exoplanets in the 3-10 $\mu$m range with E-ELT/METIS”  In _International Journal of Astrobiology_ 14.2, 2015, pp. 279–289  DOI: 10.1017/S1473550414000135
  * [304] R. Bowens et al.  “Exoplanets with ELT-METIS. I. Estimating the direct imaging exoplanet yield around stars within 6.5 parsecs”  In _A &A_ 653, 2021, pp. A8  DOI: 10.1051/0004-6361/202141109
  * [305] Sascha P. Quanz et al.  “Atmospheric characterization of terrestrial exoplanets in the mid-infrared: biosignatures, habitability, and diversity”  In _Experimental Astronomy_ 54.2-3, 2022, pp. 1197–1221  DOI: 10.1007/s10686-021-09791-z
  * [306] S.. Quanz et al.  “Large Interferometer For Exoplanets (LIFE). I. Improved exoplanet detection yield estimates for a large mid-infrared space-interferometer mission”  In _A &A_ 664, 2022, pp. A21  DOI: 10.1051/0004-6361/202140366
  * [307] Jens Kammerer, Sascha P. Quanz, Felix Dannert and LIFE Collaboration  “Large Interferometer For Exoplanets (LIFE). VI. Detecting rocky exoplanets in the habitable zones of Sun-like stars”  In _A &A_ 668, 2022, pp. A52  DOI: 10.1051/0004-6361/202243846
  * [308] Michael Line et al.  “The Importance of Thermal Emission Spectroscopy for Understanding Terrestrial Exoplanets”  In _Bull. Am. Astron. Soc._ 51.3, 2019, pp. 271
  * [309] D. Kitzmann et al.  “Clouds in the atmospheres of extrasolar planets. II. Thermal emission spectra of Earth-like planets influenced by low and high-level clouds”  In _A &A_ 531, 2011, pp. A62  DOI: 10.1051/0004-6361/201014343
  * [310] B.. Konrad et al.  “Large Interferometer For Exoplanets (LIFE). III. Spectral resolution, wavelength range, and sensitivity requirements based on atmospheric retrieval analyses of an exo-Earth”  In _A &A_ 664, 2022, pp. A23  DOI: 10.1051/0004-6361/202141964
  * [311] Eleonora Alei et al.  “Large Interferometer For Exoplanets (LIFE). V. Diagnostic potential of a mid-infrared space interferometer for studying Earth analogs”  In _A &A_ 665, 2022, pp. A106  DOI: 10.1051/0004-6361/202243760
  * [312] Edward W. Schwieterman et al.  “Exoplanet Biosignatures: A Review of Remotely Detectable Signs of Life”  In _Astrobiology_ 18.6, 2018, pp. 663–708  DOI: 10.1089/ast.2017.1729
  * [313] Damanveer S Grewal, Rajdeep Dasgupta and Alexandra Farnell  “The speciation of carbon, nitrogen, and water in magma oceans and its effect on volatile partitioning between major reservoirs of the Solar System rocky bodies”  In _Geochimica et Cosmochimica Acta_ 280 Elsevier, 2020, pp. 281–301