[affiliation=1]HelinWang [affiliation=1,2]JesúsVillalba
[affiliation=1]LaureanoMoro-Velazquez [affiliation=3]JiaruiHai
[affiliation=1]ThomasThebaud [affiliation=1,2]NajimDehak

# Noise-robust Speech Separation with Fast Generative Correction

###### Abstract

Speech separation, the task of isolating multiple speech sources from a mixed
audio signal, remains challenging in noisy environments. In this paper, we
propose a generative correction method to enhance the output of a
discriminative separator. By leveraging a generative corrector based on a
diffusion model, we refine the separation process for single-channel mixture
speech by removing noises and perceptually unnatural distortions. Furthermore,
we optimize the generative model using a predictive loss to streamline the
diffusion model's reverse process into a single step and rectify any
associated errors by the reverse process. Our method achieves state-of-the-art
performance on the in-domain Libri2Mix noisy dataset, and out-of-domain WSJ
with a variety of noises, improving SI-SNR by 22-35% relative to SepFormer,
demonstrating robustness and strong generalization capabilities.

###### keywords:

speech separation, noisy environments, generative model, diffusion model

##  1 Introduction

Speech separation, also known as the cocktail party problem, is a foundational
problem in speech and audio processing, garnering significant attention in
research [1]. This paper focuses on single-channel two-speaker speech
separation in noisy environments, which is much more challenging than clean
speech separation [2].

In recent years, deep learning models have emerged as powerful tools for
addressing speech separation, exhibiting notable performance gains over
traditional methods [3]. Hershey et al. introduced a clustering method
leveraging trained speech embeddings for separation [4]. Yu et al. proposed
Permutation Invariant Training (PIT) at the frame level for source separation
[5]. Luo et al. pioneered an influential deep learning method for speech
separation in the time domain, employing an encoder, separator, and decoder
[6]. Following this structure, various discriminative models have been
developed, such as WaveSplit [7], DPTNet [8], SepFormer [9], TF-GridNet [10]
and MossFormer [11]. These models directly optimize the Scale-Invariant
Signal-to-Noise Ratio (SI-SNR) metric [4], and the performance on WSJ0-2mix
[4] has been largely saturated [12]. However, they cannot generalize well to
speech coming from a wider range of speakers and recorded in slightly
different conditions, such as noisy and reverberant environments [13, 14].
Furthermore, the SI-SNR metric does not perfectly align with human perception,
as models trained using it may introduce perceptually unnatural distortions
that could adversely affect downstream tasks [15].

In contrast to discriminative models, generative models aim to learn a prior
distribution of the data. Recently, score-based generative models [16], also
referred to as diffusion models, have been introduced for various speech
processing tasks, such as speech enhancement and dereverberation [17, 18, 19],
target speech extraction [20, 21] and speech separation [22, 23, 24, 25].
While these methods are capable of generating audio samples with good
perceived quality and have the potential to generalize well to out-of-domain
data, they struggle with the permutation problem of speakers [5] and often
yield inferior results on reference-based measurements [23]. Additionally,
such diffusion models require multiple reverse steps, which can significantly
increase computational time for inference.

To leverage the strengths of both discriminative and generative models, we
introduce a method called Generative Correction (GeCo111Source code and
pretrained models are available at: https://github.com/WangHelin1997/Fast-
GeCo. Audio Samples are available at: https://fastgeco.github.io/Fast-GeCo/. )
for noise-robust speech separation. GeCo utilizes a corrector based on a
diffusion model to refine the output of a discriminative separator. The
separator predicts the initial separation and solves the permutation of
speakers during training, while the corrector removes noises and perceptually
unnatural distortions introduced by the separator. Furthermore, to reduce the
reverse steps of the diffusion model during inference and mitigate the
discretization error resulting from the reverse process, we propose to fine-
tune the GeCo model by optimizing a predictive loss, transforming the reverse
process into a single step (referred to as Fast-GeCo). Previous works by
Lutati et al. [24] and Hirano et al. [23] also proposed refiners based on
diffusion models for speech separation. However, these methods suffer from
slow inference speed and did not demonstrate advancements in noisy speech
separation. We evaluate our proposed method on the Libri2Mix noisy dataset
[14], and the results demonstrate a significant improvement in separation
performance, surpassing other state-of-the-art methods. In addition,
experiments on different out-of-domain noisy data show notable enhancements in
perceptual quality with GeCo and Fast-GeCo.

(a) Discriminative separator.

(b) Generative corrector.

(c) Fast generative corrector.

Figure 1: Illustration of the discriminative separator, generative corrector
and fast generative corrector. Here, we use two-speaker separation as an
example, and (b) and (c) depict an example of the first speaker. Three models
are trained in order. During inference, we can use the discriminative
separator followed by either the generative corrector or the fast generative
corrector.

##  2 Background

###  2.1 Notation and Signal Model

The task of noisy speech separation is to extract different speakers’
utterances and route them to different output signals
$\bm{s}_{k}\in\mathbb{R}^{N}$ from a noisy mixture $\bm{y}\in\mathbb{R}^{N}$,

$\bm{y}=\bm{n}+\sum_{k=1}^{K}\bm{s}_{k}$ (1)

where $\bm{n}\in\mathbb{R}^{N}$ is backgound noise and $N$ is the number of
data points. $K$ is the number of sources, which is 2 in this paper.

###  2.2 Separator

A discriminative separator $f_{\text{sep}}$ is trained to directly minimize
the difference between the estimated signals and the ground-truth signals.
Following [6], the SI-SNR loss is used to optimize the parameters of the
separator.

$\mathcal{L}_{\text{sep}}:=-10\sum_{k=1}^{K}\log_{10}\frac{\|\frac{\langle\hat{\mathbf{s}}_{k},\mathbf{s}_{k}\rangle\mathbf{s}_{k}}{\|\mathbf{s}_{k}\|^{2}}\|^{2}}{\|\hat{\mathbf{s}}_{k}-\frac{\langle\hat{\mathbf{s}}_{k},\mathbf{s}_{k}\rangle\mathbf{s}_{k}}{\|\mathbf{s}_{k}\|^{2}}\|^{2}}$
(2)

where
$\hat{\mathbf{s}}_{k}=f_{\text{sep}}(\bm{y};\theta_{\text{sep}})\in\mathbb{R}^{N}$
is the estimated $k$-th clean signal, $\theta_{\text{sep}}$ are trainable
parameters of the separator, and
$\|\mathbf{s}\|^{2}=\langle\mathbf{s},\mathbf{s}\rangle$ denotes the signal
power. Scale invariance is maintained by normalizing $\hat{\mathbf{s}}_{k}$
and $\bm{s}_{k}$ to zero mean before computation. During training, utterance-
level Permutation Invariant Training (uPIT) is employed to resolve the source
permutation issue [26].

###  2.3 Stochastic Differential Equations (SDE)

Different from discriminative models, generative models approximate complex
data distributions, often showing better generalization ability to out-of-
domain data and producing more natural sounding speech [24, 17]. Initial
experiments reveal that traditional discriminative separators do not
generalize well to out-of-domain noisy data and may introduce perceptually
unnatural artifacts. To achieve better separation, a diffusion probabilistic
model is applied to modify the output of the separator.

Following Song et al. [16], we devise a stochastic diffusion process
$\left\\{\mathbf{x}_{t}\right\\}_{t=0}^{T}$ indexed by a continuous time
variable $t\in[0,T]$, which is represented as the solution to a linear SDE.

$\mathrm{d}\mathbf{x}_{t}=\mathbf{f}\left(\mathbf{x}_{t},\hat{\mathbf{s}}_{1}\right)\mathrm{d}t+g(t)\mathrm{d}\overrightarrow{\mathbf{w}}$
(3)

Here, $\overrightarrow{\mathbf{w}}$ is the standard Wiener process (a.k.a.,
Brownian motion),
$\mathbf{f}\left(\mathbf{x}_{t},\hat{\mathbf{s}}_{1}\right):\mathbb{R}^{N}\rightarrow\mathbb{R}^{N}$
is the drift coefficient of $\mathbf{x}_{t}$, and
$g(\cdot):\mathbb{R}\rightarrow\mathbb{R}$ is the diffusion coefficient of
$\mathbf{x}_{t}$. $\mathbf{x}_{0}$ denotes the initial state of the speech,
i.e., $\mathbf{s}_{1}$ (we use the first clean speech signal as an example),
and $\mathbf{x}_{T}$ denotes the final state, i.e., the estimated speech
signal by the separator $\hat{\mathbf{s}}_{1}$. Thus, this SDE diffuses the
clean sample $\mathbf{s}_{1}$ into the noisy sample $\hat{\mathbf{s}}_{1}$,
estimated by the separator.

We take the definition of Brownian Bridge SDE [27, 28]. The BBED drift and
diffusion coefficients are given by

$\mathbf{f}\left(\mathbf{x}_{t},\hat{\mathbf{s}}_{1}\right)=\frac{\hat{\mathbf{s}}_{1}-\mathbf{x}_{t}}{1-t}$
(4) $g(t)=cv^{t}$ (5)

where $c,v>0$. To avoid division by zero in (4), $T$ is set slightly smaller
than 1. By solving (3) with (4) and (5), we find that the process state
$\mathbf{x}_{t}$ follows a Gaussian perturbation kernel,

$\displaystyle\mathbf{x}_{t}$
$\displaystyle=\bm{\mu}\left(\mathbf{x}_{0},\hat{\mathbf{s}}_{1},t\right)+\sigma(t)\mathbf{z}_{t}$
(6) $\displaystyle\bm{\mu}\left(\mathbf{x}_{0},\hat{\mathbf{s}}_{1},t\right)$
$\displaystyle=(1-t)\mathbf{x}_{0}+t\hat{\mathbf{s}}_{1}$ (7)

where $\mathbf{z}_{t}\sim\mathcal{N}(\mathbf{0},\mathbf{I})$ is random
Gaussian noise. Following [27],

$\displaystyle\sigma(t)$
$\displaystyle=\sqrt{(1-t)c^{2}\left[\left(v^{2t}-1+t\right)+\log\left(v^{2v^{2}}\right)(1-t)E\right]}$
(8) $\displaystyle E$
$\displaystyle=\mathrm{Ei}[2(t-1)\log(v)]-\operatorname{Ei}[-2\log(v)]$ (9)

where $\text{Ei}(x)=\int_{-\infty}^{x}\frac{e^{t}}{t}dt$ is the exponential
integral function [29].

Following Song et al. [16], the SDE in (3) has an associated reverse SDE,
which allows us to re-generate the clean signal $\mathbf{s}_{1}$ given the
separator estimate $\hat{\mathbf{s}}_{1}$,

$\mathrm{d}\mathbf{x}_{t}=\left[-\mathbf{f}\left(\mathbf{x}_{t},\hat{\mathbf{s}}_{1}\right)+g(t)^{2}\nabla_{\mathbf{x}_{t}}\log
p_{t}\left(\mathbf{x}_{t}\mid\hat{\mathbf{s}}_{1}\right)\right]\mathrm{d}t+g(t)\mathrm{d}\overleftarrow{\mathbf{w}}$
(10)

where $\overleftarrow{\mathbf{w}}$ is a backward Wiener process through the
diffusion time. In particular, the reverse process starts at $t=T$ and ends at
$t=0$.

##  3 GeCo and Fast-GeCo

###  3.1 GeCo

The score function $\nabla_{\mathbf{x}_{t}}\log
p_{t}\left(\mathbf{x}_{t}\mid\hat{\mathbf{s}}_{1}\right)$ is approximated by a
neural network called score model
$f_{\text{diff}}\left(\mathbf{x}_{t},\hat{\mathbf{s}}_{1},\mathbf{y},t;\theta_{\text{diff}}\right)$,
which is conditioned by the signal at the current state $\mathbf{x}_{t}$, the
estimated signal from the separator $\hat{\mathbf{s}}_{1}$, the mixture noisy
audio $\mathbf{y}$ and the time state $t$ , parameterized by
$\theta_{\text{diff}}$. Following [30], the score model is fit to the score of
the perturbation kernel,

$\mathcal{L}_{\text{diff}}:=\|f_{\text{diff}}\left(\mathbf{x}_{t},\hat{\mathbf{s}}_{1},\mathbf{y},t;\theta_{\text{diff}}\right)+\frac{\mathbf{z}_{t}}{\sigma(t)}\|_{2}^{2}$
(11)

where $\|\cdot\|_{2}^{2}$ is the $l2$-norm. This is called denoising score
matching (DSM) [30]. For each training step, we sample the estimated separated
signal for one of the speakers $\mathbf{s}_{1}$, given the discriminative
separator network, the corresponding signal, the original mixture
$\mathbf{y}$, and a timestep $t$ uniformly from $\left[t_{\epsilon},T\right]$,
where $t_{\epsilon}>0$ is a small value that assures numerical stability [18].
Then, we obtain $\mathbf{x}_{t}$ using (6) and minimize the DSM loss with
(11).

Once $\theta_{\text{diff}}$ is available, we can generate an estimate of clean
speech signals from $\hat{\mathbf{s}}_{1}$ by solving the reverse SDE in (10).
To find numerical solutions for SDEs, the Euler-Maruyama (EuM) first-order
method [16] can be employed to reduce the number of iterations. The interval
$[0,T]$ is partitioned into M equal subintervals of width $\Delta t=T/M$,
which approximates the continuous formulation into the discrete reverse
process $\left\\{\mathbf{x}_{T},\mathbf{x}_{T-\Delta
t},\ldots,\mathbf{x}_{0}\right\\}$. More specifically, the reverse process
starts with
$\mathbf{x}_{T}\sim\mathcal{N}\left(\hat{\mathbf{s}}_{1},\sigma\left(T\right)^{2}\mathbf{I}\right)$
and iteratively computes

$\displaystyle\hat{\mathbf{x}}_{(i-1)\Delta t}=\mathbf{x}_{i\Delta
t}+g\left(i\Delta t\right)\sqrt{\Delta t}\mathbf{z}_{i\Delta t}$ (12)
$\displaystyle+\left[-\mathbf{f}\left(\mathbf{x}_{i\Delta
t},\hat{\mathbf{s}}_{1}\right)+g\left(i\Delta
t\right)^{2}f_{\text{diff}}\left(\mathbf{x}_{i\Delta
t},\hat{\mathbf{s}}_{1},\mathbf{y},i\Delta
t;\theta_{\text{diff}}\right)\right]\Delta t$

where $i=M,M-1,\dots,2,1$. The last iteration outputs $\hat{\mathbf{x}}_{0}$
approximating the clean speech signal. The reverse starting point
$0<T^{\prime}\leq T$ can be used for trading performance for computational
speed. Using a small $T^{\prime}$ may degrade the performance but can reduce
the number of iterations.

###  3.2 Fast-GeCo

There are several drawbacks of the discretization reverse process in Section
3.1. First, there are discretization errors both from the EuM method and the
noise schedule. Second, as the reverse process starts with $T^{\prime}$
instead of $T$, there is a prior mismatch between the terminating forward
distribution and the initial reverse distribution [27]. Third, $M$ reverse
steps make the inference much slower than a discriminative model. In addition,
GeCo is trained to estimate the noise added between two steps, which is
different from the goal of extracting the clean signal from the noise mixture.
Therefore, it is hard to use loss objectives such as SI-SNR loss to optimize
the signal quality.

To address the above issues, we propose a fast generative correction method
that finetunes the GeCo model into a single reverse step during inference. The
idea is to optimize a single reverse process directly from the start step
$\mathbf{x}_{T^{\prime}}$ to the initial step $\mathbf{x}_{0}$ with an SI-SNR
loss. To be more specific, the one-step output is obtained by

$\displaystyle\hat{\mathbf{x}}_{0}$
$\displaystyle=\mathbf{x}_{T^{\prime}}+g\left(T^{\prime}\right)\sqrt{T^{\prime}}\mathbf{z}_{T^{\prime}}$
$\displaystyle+T^{\prime}\left[-\mathbf{f}\left(\mathbf{x}_{T^{\prime}},\hat{\mathbf{s}}_{1}\right)+g\left(T^{\prime}\right)^{2}f_{\text{diff}}\left(\mathbf{x}_{T^{\prime}},\hat{\mathbf{s}}_{1},\mathbf{y},T^{\prime};\theta_{\text{diff}}\right)\right]$
(13)

which is (LABEL:eq11), with $M=1$ and $\Delta t=T^{\prime}$.

The SI-SNR loss function is then used to fine-tune the corrector parameters
$\theta_{\text{diff}}$,

$\mathcal{L}_{\text{cor}}:=-10\log_{10}\frac{\|\frac{\langle\hat{\mathbf{x}}_{0},\mathbf{s}_{1}\rangle\mathbf{s}_{1}}{\|\mathbf{s}_{1}\|^{2}}\|^{2}}{\|\hat{\mathbf{x}}_{0}-\frac{\langle\hat{\mathbf{x}}_{0},\mathbf{s}_{1}\rangle\mathbf{s}_{1}}{\|\mathbf{s}_{1}\|^{2}}\|^{2}}$
(14)

##  4 Experiments

###  4.1 Datasets

To evaluate the performance of the proposed methods, we trained models on the
Libri2Mix noisy dataset [14], which is simulated by the WHAM$!$ noise data [2]
and the Librispeech utterances [31]. It contains two training subsets with 212
hours (train-360) and 58 hours (train-100) of audio respectively. The dev set
with 11 hours of audio was used for validation. We then evaluated our models
on the test set, containing 11 hours of audio.

Furthermore, we evaluated these trained models on out-of-domain data.
Utterances from the Wall Street Journal (WSJ) corpus were used as the speech
source, while noise audio from WHAM$!$, MUSAN [32] and DEMAND [33] were used
as the noise source for simulation. Each of these test sets has a duration of
5 hours. To maintain consistency with the wsj0-2mix dataset [4], we ensured
that the relative levels between the two speakers were preserved. Following
the approach in [2], noise was introduced by sampling a random Signal-to-Noise
Ratio (SNR) value from a uniform distribution ranging from -6 to +3 dB. In
addition, we generated a minimum-length version of the simulated data by
removing any leading and trailing noise and truncating it to match the length
of the shorter of the two speakers' utterances. All the generated mixtures
were resampled to 8 kHz as done in the previous works [6, 34].

###  4.2 Baseline Methods

Apart from GeCo and Fast-GeCo, we implemented the following baseline methods
under the same experimental settings.
SepFormer [9]: is a discriminative model providing state-of-the-art results in
Libri2Mix dataset, with one inference step.
SepFormer-SE: consists of a SepFormer and an enhancement model (an advanced
model called SGMSE+ [18] was used here). SepFormer and SGMSE+ were trained
independently. SGMSE+ was trained on the WHAM! dataset and needs 60 steps
during the inference.
SE-SepFormer: is different from SepFormer-SE that SGMSE+ is used before
SepFormer.
DiffSep [22]: is a fully generative separation method based on diffusion with
30 inference steps.
Refiner [23]: is a combination of discriminative model and generative model.
As the number of inference steps is not clarified in the original paper, we
chose an optimal number of 30 from our experiments.
SpeechFlow [25] is a generative model followed by a discriminative model which
requires a pre-training stage with 60k hours of audio data.

###  4.3 Metrics

We evaluated the performance on reference-based perceptual metrics, including
Perceptual Evaluation of Speech Quality improvement (PESQi) [35], Extended
Short-Time Objective Intelligibility improvement (ESTOi) [36], and SI-SNR
improvement (SI-SNRi). We also introduced a reference-free metric (non-
intrusive speech quality assessment (NISQA) [37], which utilizes a deep neural
network to estimate the mean opinion score (MOS) of a target signal without
any reference signal.

Table 1: Speech separation results on different test sets. All models are
trained on Libri2Mix training set. $\star$ are the results of models
reproduced by us. "SE" denotes a speech enhancement model. Values indicate
mean and standard deviation.

Method | Libri2Mix | WHAM! | MUSAN | DEMAND
---|---|---|---|---
PESQi | ESTOIi | SI-SNRi | PESQi | ESTOIi | SI-SNRi | PESQi | ESTOIi | SI-SNRi | PESQi | ESTOIi | SI-SNRi
SepFormer$\star$ [9] | 0.59$\pm$0.25 | 0.23$\pm$0.06 | 10.58$\pm$2.63 | 0.40$\pm$0.26 | 0.20$\pm$0.08 | 9.91$\pm$3.72 | 0.50$\pm$0.36 | 0.20$\pm$0.10 | 9.06$\pm$4.83 | 0.67$\pm$0.34 | 0.21$\pm$0.09 | 9.89$\pm$4.89
SepFormer-SE$\star$ | 0.72$\pm$0.30 | 0.24$\pm$0.07 | 9.80$\pm$2.44 | 0.45$\pm$0.26 | 0.23$\pm$0.10 | 9.85$\pm$3.88 | 0.61$\pm$0.37 | 0.21$\pm$0.08 | 8.89$\pm$4.25 | 0.82$\pm$0.30 | 0.25$\pm$0.10 | 9.98$\pm$4.70
SE-SepFormer$\star$ | 0.46$\pm$0.22 | 0.17$\pm$0.04 | 7.60$\pm$2.05 | 0.29$\pm$0.18 | 0.14$\pm$0.05 | 7.44$\pm$2.90 | 0.35$\pm$0.19 | 0.13$\pm$0.05 | 7.20$\pm$2.77 | 0.55$\pm$0.25 | 0.18$\pm$0.06 | 8.02$\pm$4.02
DiffSep$\star$ [22] | 0.50$\pm$0.22 | 0.22$\pm$0.05 | 8.90$\pm$2.30 | 0.33$\pm$0.22 | 0.18$\pm$0.07 | 8.71$\pm$3.33 | 0.41$\pm$0.22 | 0.19$\pm$0.08 | 8.44$\pm$4.40 | 0.62$\pm$0.30 | 0.20$\pm$0.08 | 9.47$\pm$4.53
Refiner$\star$ [23] | 0.64$\pm$0.25 | 0.23$\pm$0.05 | 10.10$\pm$2.70 | 0.42$\pm$0.26 | 0.21$\pm$0.09 | 9.74$\pm$3.66 | 0.51$\pm$0.35 | 0.20$\pm$0.10 | 8.83$\pm$4.80 | 0.60$\pm$0.30 | 0.20$\pm$0.09 | 9.55$\pm$4.54
SpeechFlow [25] | - | 0.33 | 10.46 | - | - | - | - | - | - | - | - | -
GeCo (ours) | 0.77$\pm$0.28 | 0.28$\pm$0.06 | 10.88$\pm$2.71 | 0.48$\pm$0.28 | 0.26$\pm$0.09 | 10.41$\pm$4.13 | 0.71$\pm$0.39 | 0.25$\pm$0.10 | 9.97$\pm$4.91 | 0.88$\pm$0.37 | 0.25$\pm$0.10 | 9.85$\pm$4.77
GeCo-1step (ours) | -0.26$\pm$0.11 | -0.08$\pm$0.03 | -3.11$\pm$1.30 | -0.20$\pm$0.10 | -0.07$\pm$0.03 | -3.85$\pm$1.15 | -0.28$\pm$0.09 | -0.10$\pm$0.06 | -4.20$\pm$1.44 | -0.15$\pm$0.10 | -0.10$\pm$0.04 | -3.55$\pm$1.81
Fast-GeCo (ours) |  0.86$\pm$0.31 |  0.34$\pm$0.07 |  12.98$\pm$2.81 |  0.46$\pm$0.28 |  0.32$\pm$0.08 |  12.56$\pm$3.59 |  0.75$\pm$0.42 |  0.30$\pm$0.11 |  12.29$\pm$5.26 |  0.92$\pm$0.33 |  0.29$\pm$0.10 |  13.31$\pm$4.91

Table 2: NISQA results on different test sets.

Method | Libri2Mix | WHAM! | MUSAN | DEMAND
---|---|---|---|---
Mixture | 1.15 | 0.96 | 1.64 | 1.99
Clean | 3.47 | 3.24 | 3.24 | 3.24
SepFormer [9] | 2.05 | 1.79 | 2.23 | 2.35
DiffSep [22] | 2.44 | 2.13 | 2.30 | 2.98
Refiner [23] | 3.46 | 3.60 | 3.33 | 3.75
GeCo (ours) | 3.65 | 3.58 | 3.63 | 3.61
Fast-GeCo (ours) | 3.68 | 3.76 | 3.59 | 3.87

###  4.4 Implementation Details

SepFormer was used as our discriminative separator, following the same
architecture as [9]. We set the first convolutional stride as 80, which
significantly speeded training up with minimal impact on performance. We
trained SepFormer on the Libri2Mix train-360 for 200 epochs set using Adam
optimizer with 16 batch-size, and initial learning rate of 0.00015 which was
halved after 2 epochs without improvement.

We further trained the GeCo and Fast-GeCo on the Libri2Mix train-100 set. We
employed the same noise conditional score network (NCSN++) architecture for
the score matching function $f_{\text{diff}}$, as in SGMSE+ [18]. NCSN++ is
based on multi-resolution U-Net structure with two input/output channels for
the real and imaginary parts of the complex-valued Short-Time Fourier
Transform (STFT). Using the complex STFT allows us to obtain a more
computationally efficient model compared to alternative diffusion models based
on wave samples. For GeCo, based on [27], we set $T=0.999$,
$t_{\epsilon}=0.03$, $k=2.6$, $c=0.51$ and the reverse starting point
$T^{\prime}=0.5$. We take $M=30$ steps for the inference. We used the Adam
optimizer with a learning rate of 0.0001, 100 epochs, and a batch size of 32.
An exponential moving average with a decay of 0.999 is used for the weights of
the network [38]. For Fast-GeCo, we trained for 50 epochs with the Adam
optimizer with a learning rate of 0.0001 and batch size of 32.

All models were trained on two NVIDIA A100 GPUs, each equipped with 40 GB of
GPU memory. We choose the best models and hyper-parameters based on the SI-SNR
metrics from the validation set.

##  5 Results and Analysis

Speech separation results with reference-based metrics are presented in Table
1. On the Libri2Mix test set, the discriminative model, namely SepFormer,
achieved a mean SI-SNRi of $10.58$. Introducing an enhancement model at the
end of the separation model (SepFormer-SE) improved PESQi and ESTOi metrics
while using the enhancement model in front of the separation model led to a
drop in performance. This suggests that the outputs of the enhancement model
may introduce severe artifacts, which the separation model is not able to
handle, or that the enhancement model is not suitable for overlapped audio.

Compared to SepFormer, previous diffusion-based methods (DiffSep, Refiner, and
SpeechFlow) did not significantly improve the metrics, particularly SI-SNRi.
DiffSep, being a fully-generative model, struggles with the problem of speaker
permutations during training, resulting in the worst performance. SpeechFlow
requires a pre-training stage with 60k hours of English speech but failed to
enhance the SI-SNRi metric. Refiner combines SepFormer with a denoising
diffusion restoration model conditioned by SepFormer's output, yet it did not
significantly improve the separation metrics, possibly because the original
mixture speech is not considered as the conditional information of the
diffusion model.

Similar to Refiner, our proposed method, GeCo, introduces a discriminative
model for initial separation, addressing the issue of speaker permutations
during training. Additionally, GeCo utilizes the original mixture speech as
the conditional information of the diffusion reverse process, solving the
Brownian Bridge SDE, and achieving better PESQi and SI-SNRi metrics than
state-of-the-art methods.

However, these diffusion-based methods require multiple reverse steps for the
inference stage, making them slower than SepFormer. Furthermore, as described
in Section 3.2, discretization errors in the reverse process influence their
performance, hindering significant metric improvements. By simply taking one
reverse step in GeCo (GeCo-1step in Table 1), the results were suboptimal. To
address these issues, we fine-tuned GeCo into one reverse step with an SI-SNR
loss, resulting in the proposed Fast-GeCo, which outperformed all other
methods.

For out-of-domain data (WHAM!, MUSAN, and DEMAND), similar trends are
observed, with GeCo and Fast-GeCo achieving superior results compared to
previous models. Fast-GeCo demonstrates state-of-the-art performance in all
experimental settings, indicating robust noise handling and generalization to
out-of-domain data. Compared to SepFormer, Fast-GeCo improved PESQi by 15-50%,
ESTOIi by 38-60% and SI-SNRi by 22-35%.

It is also crucial to consider reference-free metrics since the generative
model's output may not align perfectly with the ground truth. NISQA results in
Table 2 show significant improvements for all generative methods, with Fast-
GeCo achieving the best results. Processed examples in Figure 2 illustrate
enhanced harmonics structure clarity and sufficient suppression of non-speech
ambient noise compared to the original SepFormer output. Generative methods
can even outperform clean references in NISQA, likely due to light background
noises in the reference audios, consistent with the cleaner output of
generated samples shown in Figure 2.

Figure 2: Mel spectorgram separated examples of Libri2Mix. (a) Reference. (b)
Output of Sepformer. (c) Output of GeCo. (d) Output of Fast-GeCo.

##  6 Conclusions

In this paper, we proposed a generative corrector (GeCo) based on a Brownian
Bridge SDE diffusion model to enhance the output of a discriminative speech
separator achieving noise-robust speech separation. In order to speed up the
multi-step diffusion reverse process, this corrector was further finetuned
into a single-step reverse process (Fast-GeCo). The proposed Fast-GeCo
outperformed GeCo and other robust baselines achieving state-of-the-art
results in both reference-based and reference-free metrics on the Libri2Mix
noisy dataset. It also showed great generalization capabilities on the WSJ
out-of-domain data.

We observed a limitation of this work, i.e., GeCo may encounter difficulties
when processing outputs from SepFormer that exhibit significant confusion
between the two speakers. In future work, we will (i) design more effective
and efficient backbones for the separator and the corrector, (ii) test the
method in other tasks like target speech extraction, (iii) explore the method
in the settings of more speakers.

## References

  * [1] Y. Wang, A. Narayanan, and D. Wang, ``On training targets for supervised speech separation,'' _IEEE/ACM transactions on audio, speech, and language processing_ , vol. 22, no. 12, pp. 1849–1858, 2014.
  * [2] G. Wichern, J. Antognini, M. Flynn, L. R. Zhu, E. McQuinn, D. Crow, E. Manilow, and J. Le Roux, ``Wham!: Extending speech separation to noisy environments,'' in _Proc. Interspeech_ , 2019.
  * [3] D. Wang and J. Chen, ``Supervised speech separation based on deep learning: An overview,'' _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , vol. 26, no. 10, pp. 1702–1726, 2018.
  * [4] J. R. Hershey, Z. Chen, J. Le Roux, and S. Watanabe, ``Deep clustering: Discriminative embeddings for segmentation and separation,'' in _IEEE international conference on acoustics, speech and signal processing (ICASSP)_. IEEE, 2016, pp. 31–35.
  * [5] D. Yu, M. Kolbæk, Z.-H. Tan, and J. Jensen, ``Permutation invariant training of deep models for speaker-independent multi-talker speech separation,'' in _IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2017, pp. 241–245.
  * [6] Y. Luo and N. Mesgarani, ``Conv-tasnet: Surpassing ideal time–frequency magnitude masking for speech separation,'' _IEEE/ACM transactions on audio, speech, and language processing_ , vol. 27, no. 8, pp. 1256–1266, 2019.
  * [7] N. Zeghidour and D. Grangier, ``Wavesplit: End-to-end speech separation by speaker clustering,'' _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , vol. 29, pp. 2840–2849, 2021.
  * [8] J. Chen, Q. Mao, and D. Liu, ``Dual-Path Transformer Network: Direct Context-Aware Modeling for End-to-End Monaural Speech Separation,'' in _Proc. Interspeech_ , 2020, pp. 2642–2646.
  * [9] C. Subakan, M. Ravanelli, S. Cornell, M. Bronzi, and J. Zhong, ``Attention is all you need in speech separation,'' in _IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2021, pp. 21–25.
  * [10] Z.-Q. Wang, S. Cornell, S. Choi, Y. Lee, B.-Y. Kim, and S. Watanabe, ``Tf-gridnet: Integrating full-and sub-band modeling for speech separation,'' _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , 2023.
  * [11] S. Zhao and B. Ma, ``Mossformer: Pushing the performance limit of monaural speech separation using gated single-head transformer with convolution-augmented joint self-attentions,'' in _IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2023, pp. 1–5.
  * [12] S. Lutati, E. Nachmani, and L. Wolf, ``Sepit approaching a single channel speech separation bound,'' _arXiv preprint arXiv:2205.11801_ , 2022.
  * [13] M. Maciejewski, G. Wichern, E. McQuinn, and J. Le Roux, ``Whamr!: Noisy and reverberant single-channel speech separation,'' in _IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2020, pp. 696–700.
  * [14] J. Cosentino, M. Pariente, S. Cornell, A. Deleforge, and E. Vincent, ``Librimix: An open-source dataset for generalizable speech separation,'' 2020.
  * [15] J. Le Roux, S. Wisdom, H. Erdogan, and J. R. Hershey, ``Sdr–half-baked or well done?'' in _IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2019, pp. 626–630.
  * [16] Y. Song, J. Sohl-Dickstein, D. P. Kingma, A. Kumar, S. Ermon, and B. Poole, ``Score-based generative modeling through stochastic differential equations,'' in _International Conference on Learning Representations_ , 2021.
  * [17] Y.-J. Lu, Z.-Q. Wang, S. Watanabe, A. Richard, C. Yu, and Y. Tsao, ``Conditional diffusion probabilistic model for speech enhancement,'' in _IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2022, pp. 7402–7406.
  * [18] J. Richter, S. Welker, J.-M. Lemercier, B. Lay, and T. Gerkmann, ``Speech enhancement and dereverberation with diffusion-based generative models,'' _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , 2023.
  * [19] J.-M. Lemercier, J. Richter, S. Welker, and T. Gerkmann, ``Storm: A diffusion-based stochastic regeneration model for speech enhancement and dereverberation,'' _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , 2023.
  * [20] N. Kamo, M. Delcroix, and T. Nakatani, ``Target Speech Extraction with Conditional Diffusion Model,'' in _Proc. INTERSPEECH_ , 2023, pp. 176–180.
  * [21] J. Hai, H. Wang, D. Yang, K. Thakkar, N. Dehak, and M. Elhilali, ``Dpm-tse: A diffusion probabilistic model for target sound extraction,'' in _ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2024, pp. 1196–1200.
  * [22] R. Scheibler, Y. Ji, S.-W. Chung, J. Byun, S. Choe, and M.-S. Choi, ``Diffusion-based generative speech source separation,'' in _IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2023, pp. 1–5.
  * [23] M. Hirano, K. Shimada, Y. Koyama, S. Takahashi, and Y. Mitsufuji, ``Diffusion-based signal refiner for speech separation,'' 2023.
  * [24] S. Lutati, E. Nachmani, and L. Wolf, ``Separate and diffuse: Using a pretrained diffusion model for better source separation,'' in _The Twelfth International Conference on Learning Representations_ , 2024.
  * [25] A. H. Liu, M. Le, A. Vyas, B. Shi, A. Tjandra, and W.-N. Hsu, ``Generative pre-training for speech with flow matching,'' in _The Twelfth International Conference on Learning Representations_ , 2024.
  * [26] M. Kolbæk, D. Yu, Z.-H. Tan, and J. Jensen, ``Multitalker speech separation with utterance-level permutation invariant training of deep recurrent neural networks,'' _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , vol. 25, no. 10, pp. 1901–1913, 2017.
  * [27] B. Lay, S. Welker, J. Richter, and T. Gerkmann, ``Reducing the prior mismatch of stochastic differential equations for diffusion-based speech enhancement,'' _arXiv preprint arXiv:2302.14748_ , 2023.
  * [28] Z. Qiu, M. Fu, F. Sun, G. Altenbek, and H. Huang, ``Se-bridge: Speech enhancement with consistent brownian bridge,'' _arXiv preprint arXiv:2305.13796_ , 2023.
  * [29] C. M. Bender and S. A. Orszag, _Advanced mathematical methods for scientists and engineers I: Asymptotic methods and perturbation theory_. Springer Science & Business Media, 2013.
  * [30] P. Vincent, ``A connection between score matching and denoising autoencoders,'' _Neural computation_ , vol. 23, no. 7, pp. 1661–1674, 2011.
  * [31] V. Panayotov, G. Chen, D. Povey, and S. Khudanpur, ``Librispeech: an asr corpus based on public domain audio books,'' in _IEEE international conference on acoustics, speech and signal processing (ICASSP)_. IEEE, 2015, pp. 5206–5210.
  * [32] D. Snyder, G. Chen, and D. Povey, ``Musan: A music, speech, and noise corpus,'' _arXiv preprint arXiv:1510.08484_ , 2015.
  * [33] E. Hadad, F. Heese, P. Vary, and S. Gannot, ``Multichannel audio database in various acoustic environments,'' in _International Workshop on Acoustic Signal Enhancement (IWAENC)_. IEEE, 2014, pp. 313–317.
  * [34] Y. Luo, Z. Chen, and T. Yoshioka, ``Dual-path rnn: efficient long sequence modeling for time-domain single-channel speech separation,'' in _IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2020, pp. 46–50.
  * [35] A. W. Rix, J. G. Beerends, M. P. Hollier, and A. P. Hekstra, ``Perceptual evaluation of speech quality (pesq)-a new method for speech quality assessment of telephone networks and codecs,'' in _IEEE international conference on acoustics, speech, and signal processing (ICASSP)_. IEEE, 2001, pp. 749–752.
  * [36] J. Jensen and C. H. Taal, ``An algorithm for predicting the intelligibility of speech masked by modulated noise maskers,'' _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , vol. 24, no. 11, pp. 2009–2022, 2016.
  * [37] G. Mittag, B. Naderi, A. Chehadi, and S. Möller, ``NISQA: A Deep CNN-Self-Attention Model for Multidimensional Speech Quality Prediction with Crowdsourced Datasets,'' in _Proc. Interspeech_ , 2021, pp. 2127–2131.
  * [38] Y. Song and S. Ermon, ``Improved techniques for training score-based generative models,'' _Advances in neural information processing systems_ , vol. 33, pp. 12 438–12 448, 2020.


# VIXEN: Visual Text Comparison Network for Image Difference Captioning

Alexander Black1, Jing Shi2, Yifei Fan2, Tu Bui1, John Collomosse1,2

###### Abstract

We present VIXEN – a technique that succinctly summarizes in text the visual
differences between a pair of images in order to highlight any content
manipulation present. Our proposed network linearly maps image features in a
pairwise manner, constructing a soft prompt for a pretrained large language
model. We address the challenge of low volume of training data and lack of
manipulation variety in existing image difference captioning (IDC) datasets by
training on synthetically manipulated images from the recent InstructPix2Pix
dataset generated via prompt-to-prompt editing framework. We augment this
dataset with change summaries produced via GPT-3. We show that VIXEN produces
state-of-the-art, comprehensible difference captions for diverse image
contents and edit types, offering a potential mitigation against
misinformation disseminated via manipulated image content. Code and data are
available at http://github.com/alexblck/vixen

Figure 1: Visual change summarization produced by VIXEN for original-
manipulated image pairs. VIXEN is able to observe both background (left) and
main subject (mid) changes as well as generalize to other datasets (right).

## Introduction

Image manipulation often forms the basis for fake news and misinformation.
This threat may be countered by tools that encourage users to reflect upon the
provenance and content of images. Given the reactionary nature of sharing,
such tools should be intuitively comprehensible to enable users to make fast,
informed trust decisions (Gregory 2019).

This paper contributes VIXEN – a method for intuitively summarizing the visual
change between a pair of images using a short passage of text. Emerging open
standards (_e.g_. C2PA (Coalition for Content Provenance and Authenticity
2023)) describe provenance frameworks that match images circulating in the
wild to a federated database of originals using perceptual hashing methods
(Black et al. 2021b; Nguyen et al. 2021; Black et al. 2021a; Pizzi et al.
2022). VIXEN presents a comprehensible way to review any image manipulation
evidenced by such a matching (Fig. 1).

Image difference captioning (IDC) is typically addressed by representations
that seek to model the spatial-semantic distribution of concepts present in a
scene – for example, the relative positions of objects in CCTV footage
(Jhamtani and Berg-Kirkpatrick 2018), or of primitive geometric shapes
(Johnson et al. 2017). More complex kinds of manipulations require expertise
to construct and thus can not be easily scaled up in volume (Tan et al. 2019).
To this end, we make three technical contributions:

  1. 1.

Cross-modal image differencing. We present a novel image differencing concept
comprising a 2-branch GPT-J architecture to embed and compare facts derived
from the image pair using CLIP-based image encoding. The model generates text
conditioned on that comparison to explain salient changes between the image
pair. We show our textual explanations to be succinct and comprehensible to
non-expert users and to be quantitatively closer to ground-truth edit captions
than state-of-the-art captioning methods.

  2. 2.

Synthetic Edit Training. We propose a synthetic pair-wise training framework
for our VIXEN leveraging recent prompt2prompt (P2P) and language-based image
editing (LBIE) approaches to supervise fine-tuning on generative image
content, showing good generalization to unseen content.

  3. 3.

Augmented IP2P Dataset. We release an augmentation of the recent
InstructPix2Pix (IP2P) dataset with synthetic change captions generated via
GPT-3 as a basis for training and evaluating VIXEN.

We demonstrate that VIXEN achieves higher performance than prior image
difference captioning methods and is able to generalize to multiple datasets.

## Related Work

Image difference captioning (IDC) is closely related to image captioning and
visual question answering, both requiring a visual understanding system to
model images and a language understanding system capable of generating
syntactically correct captions. The revolution of IDC in recent years depends
heavily on the advent of visual and text modeling approaches, together with
cross-domain learning techniques that bridge the representation gap between
them.

Early visual content modeling approaches employ global CNN features such as
VGG (Donahue et al. 2015) and ResNet (Rennie et al. 2017) as input signals to
the text generation models, leveraging the semantically rich and compact
representations deliverable from these models. To better capture multi-object
representations and their relation, regional modeling methods are developed
(Lu et al. 2017; Gu et al. 2018; Anderson et al. 2018; Huang et al. 2019). In
some, images are gridded into non-overlapping patches upon which CNN features
are extracted, others instead use outputs from an early layer of a pretrained
ResNet model to effectively capture spatial features in a grid fashion. In
contrast, (Cornia et al. 2020; Anderson et al. 2018; Huang et al. 2019) employ
Region Proposal Network (RPN) to extract features from potential candidate
objects, offering better alignment with semantic objects mentioned in the
paired captions. Alternative approaches include graph-based (Yang et al. 2019)
and tree-based networks (Yao et al. 2019) to capture the relations of objects
at multiple levels of granularity.

For a long time RNN/LSTM (Graves and Graves 2012) have been used to model text
due to its inherent sequential properties. Single-layer RNN (Vinyals et al.
2015; Mao et al. 2015) or double-layer LSTM (Donahue et al. 2015; Anderson et
al. 2018; Yao et al. 2019) are employed along with various techniques to
integrate image features deeper into the recurrent process, including additive
attention (Stefanini et al. 2022). During inference, captions are generated in
an autoregressive fashion – the prediction of a word is conditioned on all
previous words. While this improves linguistic coherence, RNN/LSTM-based
approaches struggle in modeling long captions. This problem is levitated in
recent transformer-based approaches thanks to its full-attention mechanism
(Luo et al. 2021; Wang et al. 2021; Cornia et al. 2020). More advanced
transformer-based approaches such as BERT (Devlin et al. 2018), GPT (Brown et
al. 2020) and LLaMA (Touvron et al. 2023) have been successfully applied in
various visual-language tasks (Hu et al. 2022; Mokady, Hertz, and Bermano
2021; Gao et al. 2023; Zhang et al. 2021; Li et al. 2020).

Visual language modeling aims to bridge the gap between image/video and text
representations for specific tasks such as joint embedding (_e.g_. CLIP
(Radford et al. 2021) and LIMoE (Mustafa et al. 2022) for cross-domain
retrieval), text-to-image (_e.g_. Stable Diffusion (Rombach et al. 2022) for
text-based image generation, InstructPix2Pix (Brooks, Holynski, and Efros
2022) for image editing) and image-to-text (_e.g_. visual question answering
(Alayrac et al. 2022; Wang et al. 2021), visual instructions (Gao et al. 2023;
Driess et al. 2023)). In the context of image captioning, image-text mapping
strategies can be categorized into two research strands. The first strand
involves the early fusion of image and text features for better alignment
between image objects and words (Tsimpoukelli et al. 2021; Mokady, Hertz, and
Bermano 2021; Wang et al. 2021; Li et al. 2020). These methods adopt BERT-like
training strategies to input a pair of image and masked caption to the masked
words. At inference, the input caption is simply replaced by a start token or
a prefixed phrase _e.g_. ‘A picture of’. The second research strand focuses on
learning a direct transformation from image to text embedding. Early CNN-based
approaches feed image features as the hidden states of the LSTM text modules
(Donahue et al. 2015; Vinyals et al. 2015; Yao et al. 2019; Karpathy and Fei-
Fei 2015; Rennie et al. 2017) while later transformer-based methods favor
cross-attention (Luo et al. 2021; Cornia et al. 2020). Recently in both
research strands, there has been a trend of leveraging powerful pretrained
large language and vision models to learn a simple mapping between two domains
(Merullo et al. 2022; Eichenberg et al. 2021; Li et al. 2023; Tsimpoukelli et
al. 2021; Mokady, Hertz, and Bermano 2021).

Image difference captioning is a form of image captioning in which the caption
would ideally ignore common objects between images and rather highlight subtle
changes between them. As the first work addressing IDC, Spot-the-Diff
(Jhamtani and Berg-Kirkpatrick 2018) identifies potential change clusters and
models them using an LSTM-based network. Their work relies on the difference
between two input images at the pixel level, therefore sensitive to noises and
geometric transformations. DUDA (Park, Darrell, and Rohrbach 2019) instead
computes image difference at CNN semantic level, improving the robustness
against slight global changes. In M-VAM (Shi et al. 2020) and VACC (Kim et al.
2021), a view-point encoder is proposed to mitigate potential view-point
difference and VARD (Tu et al. 2023a) proposes a viewpoint invariant
representation network to explicitly capture the change. Meanwhile, (Sun et
al. 2022) uses bidirectional encoding to improve change localization and NCT
(Tu et al. 2023b) aggregates neighboring features with a transformer. These
methods mostly focus on image modality and take advantage of benchmark-
specific properties, such as near-identical views in Spot-the-Diff (Jhamtani
and Berg-Kirkpatrick 2018) or synthetic scenes with limited objects and change
types (color, texture, add, drop, remove) in CLEVR (Park, Darrell, and
Rohrbach 2019). More recently, IDC-PCL (Yao, Wang, and Jin 2022) and CLIP4IDC
(Guo, Wang, and Laaksonen 2022) adopt BERT-like training strategies to model
difference captioning language, achieving state-of-art performance.

Figure 2: Model architecture and data captioning augmentation pipeline
diagram. We use a pre-trained image encoder network $\mathcal{E}$ to produce a
representation of two images. Both of these are projected into the input space
of a large language model (LM) by a trained linear projection layer
$\mathcal{P}$. Frozen layers are marked in blue, trainable in red.

## Methodology

Our proposed method relies on synthetically generated image pairs and
associated difference captions. We describe the creation process of visual and
textual components of the dataset, details of the architecture of our proposed
approach and training details necessary to reproduce the results.

### Data Generation

To train our proposed approach, we require a large dataset of image pairs,
each annotated with a summary of the changes between them. We propose using
images generated by stable diffusion (Rombach et al. 2022) and edited with
prompt-to-prompt (Hertz et al. 2022) using the pipeline presented in
InstructPix2Pix (Brooks, Holynski, and Efros 2022) (IP2P). One of our
contributions is the introduction of difference summary captions to IP2P
images, generated using GPT-3 (Brown et al. 2020) in a few-shot learning
fashion.

The InstructPix2Pix dataset is generated using the prompt-to-prompt editing
framework, which provides text-based editing capabilities for synthesized
images by injecting the attention maps associated with a specific word in the
prompt to control the attention maps of the edited image. Therefore, all that
is required to generate an image pair is two textual prompts with slight
differences. IP2P uses a fine-tuned GPT-3 language model to generate plausible
edits based on real input captions from LAION (Schuhmann et al. 2022). In
addition to the image pairs and captions the dataset also contains an
instruction that describes what edits have to be applied in order to generate
the output image.

While these instructions are sufficient for the original InstructPix2Pix task
of text-based image editing, they often omit the information regarding the
input content. For example, for the prompt pair ”a photo of a cat”/”a photo of
a dog”, the edit instruction might be ”as a dog” or ”turn it into a dog”. We
aim to summarize the changes by referencing both the original and edited image
contents, therefore the desirable edit summarization caption would be ”the cat
has been replaced by a dog”. To achieve this, we use GPT-3 language model in a
few-shot learning fashion by including several examples of input-output-
instruction-summary quadruplets where summary captions are constructed
manually. While IP2P uses a fine-tuned GPT-3 to generate the instruction and
second image captions, we have found the fine-tuning unnecessary in our case.
Since our task does not require creativity from the model, but rather
summarization of the input information, the pre-trained ’davinci’ version of
GPT-3 is enough to produce the captions needed.

Figure 3: Image-caption pairs with an average correspondence score of 3
(left): may contain global changes when only local ones are expected (top) or
fail to produce desired edits due to vague captioning (bot); 4 (mid):
partially satisfy the caption, occasionally only some properties are realized
correctly (top) or an existing object is replaced rather than added to the
background (bot); 5 (right): mostly faithful to the depicted edits.

### Architecture

Our image captioning approach is inspired by (Merullo et al. 2022), which uses
a trainable linear mapping between the image encoder and a large language
model. However, instead of passing the projected embeddings of a single image
to the language model, we project the embeddings of two images and concatenate
them before feeding them into the language model. This architecture is
illustrated in Figure 2. Given a source image $I$ and its edited version
$I^{\prime}$ we use an image encoder $\mathcal{E}$ to extract image feature
maps

$f=\mathcal{E}(I);\;f^{\prime}=\mathcal{E}(I^{\prime})\in\mathbb{R}^{k\times
h},$ (1)

where $h$ is the size of feature maps and $k$ is the prompt sequence length.
We use a fully-connected layer $\mathcal{P}$ to linearly project the image
features into dimensionality of a language model input $e$, creating a soft
prompt $s^{v}$:

$s^{v}=[\mathcal{P}(f),\mathcal{P}(f^{\prime})]\in\mathbb{R}^{2k\times e},$
(2)

where $[,]$ denotes concatenation. Finally, we append a prefix $s_{t}$ made of
embedding of tokens for ”The differences between the images are as follows:
”/”Edit instructions:” to the visual prompt $s^{v}$ to obtain the final prompt
$s=[s^{v},s^{t}]$ used for generating the summarization text.

We explore two options for $\mathcal{E}$. Firstly, following (Merullo et al.
2022; Eichenberg et al. 2021), we use CLIP RN50x16 as $\mathcal{E}$. The
feature map before the pooling layer has dimensions $12\times 12\times 3072$,
flattened to $k\times h=144\times 3072$. Secondly, we use ViT-g, followed by a
Q-Former from BLIP-2 (Li et al. 2023). In this case sequence length $k=257$.
We refer to CLIP and Q-Former versions of VIXEN as VIXEN-C and VIXEN-Q,
respectively. For the language model, we use GPT-J(Wang and Komatsuzaki 2021),
which has input space dimensionality $l=4096$. Consequently, for both
configurations of $\mathcal{E}$, our linear projection layer $\mathcal{P}$ has
input and output dimensions $h=3072$ and $l=4096$, respectively. The loss for
the captioning task objective is defined as

$\mathcal{L}=-\sum^{m}_{i=1}l(s^{v},s^{t}_{1},\ldots,s^{t}_{i}),$ (3)

where $m$ is a variable token length and $l$ is next-token log-probability
conditioned on the previous sequence elements

$l(s^{v},s^{t}_{1},\ldots,s^{t}_{i})=\log p(t_{i}|x,t_{1},\ldots,t_{i-1}).$
(4)

### Training

During training, we may provide distractor image pairs with no changes present
by providing the same image as both inputs $I=I^{\prime}$. The frequency of
the presence of distractor images is determined by probability $p_{d}$. In
such cases, the target difference summary text is chosen at random from a list
of pre-defined sentences, all synonymous with ”there is no difference”. For
all our models we first train with $p_{d}=0$ for two epochs, followed by two
more epochs with $p_{d}=0.5$. Total training time is approximately 100 hours
on a single A100 GPU. We use gradient accumulation to train with an effective
batch size of 2048 and optimize the loss using AdamW optimizer with
$\beta_{1}=0.9,\beta_{2}=0.98$ and weight decay 0.05. For baseline approaches
CLIP4IDC and IDC, we implemented dataloaders for our dataset, precomputed all
necessary supporting data (e.g., ResNet-101 features, negative samples, and a
vocabulary dictionary for IDC) and followed their standard two-step training
pipeline with default hyperparameters specified in the GitHub repos.

## Experiments

Method | MPNet | B@4 | C | M | R
---|---|---|---|---|---
Instruct Pix2Pix
| @3 | @4 | @5 | @3 | @4 | @5 | @3 | @4 | @5 | @3 | @4 | @5 | @3 | @4 | @5
VIXEN-Q (ours) | 56.9 | 59.1 | 62.3 | 16.5 | 18.5 | 20.8 | 80.3 | 93.9 | 134.9 | 17.1 | 18.4 | 20.6 | 38.0 | 40.1 | 42.2
VIXEN-C (ours) | 59.3 | 61.4 | 61.5 | 16.8 | 18.2 | 19.2 | 96.6 | 107.0 | 126.1 | 17.6 | 18.6 | 19.6 | 39.2 | 40.8 | 39.3
CLIP4IDC | 56.8 | 58.3 | 60.7 | 15.8 | 17.3 | 17.7 | 58.8 | 71.0 | 114.7 | 20.9 | 22.5 | 23.3 | 33.3 | 35.1 | 34.0
IDC | 38.3 | 38.6 | 37.4 | 8.2 | 8.8 | 7.7 | 4.4 | 5.0 | 5.6 | 16.0 | 16.8 | 16.5 | 29.1 | 30.0 | 27.7
PSBattles
VIXEN-Q (ours) | 45.1 | 5.8 | 7.5 | 11.0 | 22.2
VIXEN-C (ours) | 40.3 | 4.5 | 7.7 | 9.5 | 20.5
CLIP4IDC | 32.7 | 3.2 | 5.0 | 10.1 | 21.7
IDC | 27.0 | 1.0 | 0.7 | 9.2 | 19.5
Image Editing Request
VIXEN-Q (ours, FT) | 50.1 | 7.9 | 35.4 | 14.4 | 33.5
VIXEN-C (ours, FT) | 52.5 | 8.6 | 38.1 | 15.4 | 42.5
VARD | - | 10.0 | 35.7 | 14.8 | 39.0
CLIP4IDC | - | 8.2 | 32.2 | 14.6 | 40.4
NCT | - | 8.1 | 34.2 | 15.0 | 38.8
BiDiff | - | 6.9 | 27.7 | 14.6 | 38.5
DUDA | - | 6.5 | 27.8 | 12.4 | 37.3
rel-att | - | 6.7 | 26.4 | 12.8 | 37.4

Table 1: Image difference captioning performance on IP2P, PSBattles and Image Editing Request datasets. Evaluated on semantic similarity (MPNet), BLEU-4 (B@4), CIDEr (C), METEOR (M) and ROUGE-L (R). For IP2P, performance is reported for subsets at image-caption correspondence thresholds of 3, 4, 5. Method | MPNet | B@4 | C | M | R
---|---|---|---|---|---
Instruct Pix2Pix
VIXEN-C (ours) | 59.3 | 16.8 | 96.6 | 17.6 | 39.2
VIXEN-C p=0 | 54.4 | 15.4 | 88.5 | 16.1 | 35.9
PSBattles
VIXEN-C (ours) | 40.3 | 4.5 | 7.7 | 9.5 | 20.5
VIXEN-C p=0 | 37.8 | 4.2 | 7.2 | 8.9 | 19.2
Table 2: Impact of distractor images on performance of the model evaluated on
semantic similarity (MPNet), BLEU-4 (B@4), CIDEr (C), METEOR (M) and ROUGE-L
(R).

### Data

We perform our main evaluation on a subset of the InstructPix2Pix (Brooks,
Holynski, and Efros 2022) dataset, unseen by models during training. To ensure
a high quality of the synthetically generated image-caption pairs, we score
their correspondence via a user study. Additionally, we crowd-source
annotations for a subset of images from the PSBattles (Heller, Rossetto, and
Schuldt 2018) dataset and fine-tune and evaluate on Image Editing Request (Tan
et al. 2019).

InstructPix2Pix dataset presents challenges due to its synthetically generated
nature, as some of the edit summarization captions fail to accurately describe
the changes made to the image pairs. This is mainly due to prompt-to-prompt
occasionally generating images that do not depict the desired change
accurately enough. This is further discussed in the limitations section below
and illustrated in Figure 5 (mid). To ensure a reliable evaluation, we
conducted a user study using Amazon Mechanical Turk (MTurk) on a sample of
5,000 images from the dataset. This results in a 837,466/93,052/5,000
train/validation/test splits. The study involved three participants per image-
caption pair (95 unique participants) and aimed to rate the degree of
correspondence between the image pair and its associated caption, using a
scoring system from 1 (low) to 5 (high). The distribution of scores is 1:
$5\%$, 2: $13\%$, 3: $26\%$, 4: $33\%$, 5: $24\%$ . Figure 3 shows random
samples of the image-caption pairs for different score threshold values.

PSBattles is a dataset of images edited in Adobe PhotoshopTM, collected from
the ‘Photoshopbattles’ subreddit. The dataset contains 10k original images,
paired with several manipulated variants. There are 102k variants in total
contributed by 31k artists. We randomly sample 100 image pairs for crowd-
sourced annotation on MTurk and collect captions from 3 participants per image
pair.

Image Editing Request is a dataset of realistic photographs, paintings and
illustrations paired with instructions written by humans. It contains 4k
images-annotations pairs and incorporates a wide variety of edits, including
affine edits and crops that are not present in the other datasets.

### Metrics

We evaluate the performance of difference captioning methods using both
traditional N-gram-based metrics (BLEU-4 (Papineni et al. 2002), CIDEr
(Vedantam, Lawrence Zitnick, and Parikh 2015), METEOR (Banerjee and Lavie
2005) and ROUGE-L (Lin 2004)), as well as semantic similarity metric based on
a language transformer model. We have found that due to a larger diversity of
images and edits, the generated captions need to encompass a significantly
larger vocabulary to accurately describe the changes. As a result, there are
instances where the captions may not align word for word with the actual image
differences, but they still convey a similar meaning. To account for this, we
use a semantic textual similarity metric. We define semantic textual
similarity $S_{\mathrm{sim}}$ between the target $c$ and generated
$c^{\prime}$ summarizations

$S_{\mathrm{sim}}=\cos(E(c),E(c^{\prime})),$ (5)

where $\cos(,)=\frac{\mathbf{A}\cdot\mathbf{B}}{\|\mathbf{A}\|\|\mathbf{B}\|}$
denotes cosine similarity and $E$ is a sentence transformer. We use MPNet
(Song et al. 2020) as the best-performing sentence transformer to map
sentences to 768-dimensional normalized embeddings.

We also assess the quality of captions via a crowd-sourced study on Amazon
Mechanical Turk (MTurk). Participants are presented with both the original and
edited images. For each image pair, participants are tasked to choose one of
the 4 captions, arranged in a random order. In case all four captions do not
summarize the differences well enough, participants may choose the ’none of
the above’ option. Each task is performed by 3 unique participants. The
preference is considered to be given to a particular method if two or more
participants have voted for it.

(a) InstructPix2Pix

(b) PSBattles

Figure 4: Examples of edit summarizations for global changes, object
replacement and material changes produced by VIXEN and CLIP4IDC on
InstructPix2Pix (a) and PSBattles (b) datasets. Failure case marked with a
dashed red box.

### Results

For the proposed datasets, we compare the performance of VIXEN against two
baselines, IDC (Yao, Wang, and Jin 2022) and CLIP4IDC (Guo, Wang, and
Laaksonen 2022). We train both of them on our augmented IP2P dataset,
following the author’s guidelines. For the IER dataset, we fine-tune on IER
training set and compare against reported numbers of multiple baselines.

We report the evaluation results of evaluating both the proposed method as
well as baselines in Table 1, with examples shown in Figure 4. Our methods
achieve a higher score in all metrics, except METEOR (IP2P), where CLIP4IDC
scores higher than both proposed architectures. This indicates that VIXEN is
more tuned towards precision, rather than recall of n-grams as METEOR heavily
favors recall. For IP2P, results are reported at three different
correspondence thresholds. For lower threshold values, the best results are
obtained by VIXEN-C. VIXEN-Q seems to benefit the most from threshold increase
and outperforms other methods on pairs with a correspondence score of 5.

While all methods suffer significant performance drops when evaluated on a
dataset from a different domain, VIXEN-Q shows a better ability to generalize
to new data by scoring the highest on the PSBattles dataset. After fine-tuning
the model on Image Editing Request, VIXEN-C outperforms previous methods on
most metrics, except B@4 of VARD(Tu et al. 2023a).

The results of the crowd-sourced user preference study, shown in Figure 6,
demonstrate that the users prefer difference captions generated by VIXEN more
often than others. For the IP2P dataset, captions generated by VIXEN-Q and
VIXEN-C obtained a majority vote in 32% and 26% of the cases, respectively,
followed by CLIP4IDC and IDC with 24% and 15%. For the PSBattles dataset, the
highest preference score is achieved by VIXEN-C with 15% of the votes.
Participants chose the ’None of the above’ option in 75% of the cases for
PSBattles, as opposed to just 2% in IP2P. This indicates that generalization
to new data domains remains a challenging task.

Figure 5: Limitations of the proposed method. Left: image captioning instead of difference captioning in case of unidentified edit. Middle: mismatch between target text-image pair and LM runoff. Right: edit described in reverse order. Figure 6: User preference study results. Study participants are shown an image pair and captions generated by four methods on IP2P and PSBattles datasets. Fusion method | B@4 | C | M | R
---|---|---|---|---
Instruct Pix2Pix
Concatenation | 16.8 | 96.6 | 17.6 | 39.2
Subtraction | 16.4 | 93.7 | 16.9 | 36.8
Addition | 16.2 | 90.8 | 17.3 | 37.5
Multiplication | 14.4 | 82.7 | 15.3 | 35.9
Mean | 10.7 | 63.9 | 12.4 | 33.6
Table 3: Image feature fusion ablation of VIXEN-C

During inference we assume an input where one image is an edited version of
the other, but we demonstrate the benefits of having distractor same image
pairs during training. The possibility of no edits case makes it harder for
the model to guess the right answer by memorizing the most frequent edits
within the dataset. Table 2 shows that setting the probability $p=0$ of the
same image pairs during training of VIXEN-C yields worse results on both IP2P
and PSBattles datasets.

Table 3 shows performance results for different feature fusion strategies that
redefine $s^{v}$ in Eq 2. We have observed that concatenation leads to
slightly better performance than subtraction, addition or multiplication and
taking the mean of two features causes a significant performance drop. This
shows that retaining the information of both image features without
degradation is important for the task.

### Limitations

In Figure 5 we show examples of VIXEN’s failure cases. We identify and discuss
three main challenges. Left shows an example of a very minor difference
between the two images. In such cases, VIXEN occasionally resorts to
captioning the image content instead of summarizing the differences. Mid shows
a mismatch between the summary and generated images: an image pair with a
slightly changed book cover, but the target caption assumes that the style of
the whole image has been changed to that of a comic book. As with other LLMs,
VIXEN exhibits LM runoff: having identified a concept (”cartoon character”),
it might continue generating a text with a strong linguistic prior (”big eyes
and exaggerated features”), absent in the images. Right shows that
occasionally VIXEN may describe the differences between the images in a
reversed order.

## Conclusion

We presented VIXEN – an image difference captioning approach that provides
textual descriptions of the manipulations applied to an image. We have
augmented the InstructPix2Pix dataset of generated images with difference
summarization captions generated by GPT-3 in order to train and evaluate
VIXEN. We have shown that VIXEN achieves higher performance than other image
difference captioning methods. We have also demonstrated that, while VIXEN
shows better generalizability to other datasets, there is still a performance
gap when switching from synthetic to real data. Future works might alleviate
this by including a varied spread of manipulations types into the training
set, including insertion, deletion and text edits, which the current
generative pipelines struggle with.

## References

  * Alayrac et al. (2022) Alayrac, J.-B.; Donahue, J.; Luc, P.; Miech, A.; Barr, I.; Hasson, Y.; Lenc, K.; Mensch, A.; Millican, K.; Reynolds, M.; et al. 2022.  Flamingo: a visual language model for few-shot learning.  _NeurIPS_ , 35: 23716–23736.
  * Anderson et al. (2018) Anderson, P.; He, X.; Buehler, C.; Teney, D.; Johnson, M.; Gould, S.; and Zhang, L. 2018.  Bottom-up and top-down attention for image captioning and visual question answering.  In _Proc. CVPR_ , 6077–6086.
  * Banerjee and Lavie (2005) Banerjee, S.; and Lavie, A. 2005.  METEOR: An automatic metric for MT evaluation with improved correlation with human judgments.  In _Proc. ACL WS Intr. Extr. Eval. Measures Machine Trans. Summarization_ , 65–72.
  * Black et al. (2021a) Black, A.; Bui, T.; Jenni, S.; Swaminathan, V.; and Collomosse, J. 2021a.  Vpn: Video provenance network for robust content attribution.  In _Proc. CVMP_ , 1–10.
  * Black et al. (2021b) Black, A.; Bui, T.; Jin, H.; Swaminathan, V.; and Collomosse, J. 2021b.  Deep Image Comparator: Learning to Visualize Editorial Change.  In _Proc. CVPR WS_ , 972–980. IEEE.
  * Brooks, Holynski, and Efros (2022) Brooks, T.; Holynski, A.; and Efros, A. A. 2022.  Instructpix2pix: Learning to follow image editing instructions.  _arXiv preprint arXiv:2211.09800_.
  * Brown et al. (2020) Brown, T.; Mann, B.; Ryder, N.; Subbiah, M.; Kaplan, J. D.; Dhariwal, P.; Neelakantan, A.; Shyam, P.; Sastry, G.; Askell, A.; et al. 2020.  Language models are few-shot learners.  _NeurIPS_ , 33: 1877–1901.
  * Coalition for Content Provenance and Authenticity (2023) Coalition for Content Provenance and Authenticity. 2023.  Technical Specification 1.3.  Technical report, C2PA.
  * Cornia et al. (2020) Cornia, M.; Stefanini, M.; Baraldi, L.; and Cucchiara, R. 2020.  Meshed-memory transformer for image captioning.  In _Proc. CVPR_ , 10578–10587.
  * Devlin et al. (2018) Devlin, J.; Chang, M.-W.; Lee, K.; and Toutanova, K. 2018.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_.
  * Donahue et al. (2015) Donahue, J.; Anne Hendricks, L.; Guadarrama, S.; Rohrbach, M.; Venugopalan, S.; Saenko, K.; and Darrell, T. 2015.  Long-term recurrent convolutional networks for visual recognition and description.  In _Proc. CVPR_ , 2625–2634.
  * Driess et al. (2023) Driess, D.; Xia, F.; Sajjadi, M. S. M.; Lynch, C.; Chowdhery, A.; Ichter, B.; Wahid, A.; Tompson, J.; Vuong, Q.; Yu, T.; Huang, W.; Chebotar, Y.; Sermanet, P.; Duckworth, D.; Levine, S.; Vanhoucke, V.; Hausman, K.; Toussaint, M.; Greff, K.; Zeng, A.; Mordatch, I.; and Florence, P. 2023.  PaLM-E: An Embodied Multimodal Language Model.  In _arXiv preprint arXiv:2303.03378_.
  * Eichenberg et al. (2021) Eichenberg, C.; Black, S.; Weinbach, S.; Parcalabescu, L.; and Frank, A. 2021.  MAGMA–Multimodal Augmentation of Generative Models through Adapter-based Finetuning.  _arXiv preprint arXiv:2112.05253_.
  * Gao et al. (2023) Gao, P.; Han, J.; Zhang, R.; Lin, Z.; Geng, S.; Zhou, A.; Zhang, W.; Lu, P.; He, C.; Yue, X.; et al. 2023.  LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Model.  _arXiv preprint arXiv:2304.15010_.
  * Graves and Graves (2012) Graves, A.; and Graves, A. 2012.  Long short-term memory.  _Supervised sequence labelling with recurrent neural networks_ , 37–45.
  * Gregory (2019) Gregory, S. 2019.  Ticks or it didn’t happen.  https://lab.witness.org/ticks-or-it-didnt-happen/.  Accessed: 2024-01-20.
  * Gu et al. (2018) Gu, J.; Cai, J.; Wang, G.; and Chen, T. 2018.  Stack-captioning: Coarse-to-fine learning for image captioning.  In _Proc. AAAI_ , volume 32.
  * Guo, Wang, and Laaksonen (2022) Guo, Z.; Wang, T.-J.; and Laaksonen, J. 2022.  CLIP4IDC: CLIP for Image Difference Captioning.  In _Proc. Conf. Asia-Pacific Chapter Assoc. Comp. Linguistics and Int. Joint Conf. NLP_ , 33–42.
  * Heller, Rossetto, and Schuldt (2018) Heller, S.; Rossetto, L.; and Schuldt, H. 2018.  The PS-Battles Dataset – an Image Collection for Image Manipulation Detection.  _CoRR_ , abs/1804.04866.
  * Hertz et al. (2022) Hertz, A.; Mokady, R.; Tenenbaum, J.; Aberman, K.; Pritch, Y.; and Cohen-Or, D. 2022\.  Prompt-to-prompt image editing with cross attention control.  _arXiv preprint arXiv:2208.01626_.
  * Hu et al. (2022) Hu, X.; Gan, Z.; Wang, J.; Yang, Z.; Liu, Z.; Lu, Y.; and Wang, L. 2022.  Scaling up vision-language pre-training for image captioning.  In _Proc. CVPR_ , 17980–17989.
  * Huang et al. (2019) Huang, L.; Wang, W.; Xia, Y.; and Chen, J. 2019.  Adaptively aligned image captioning via adaptive attention time.  _NeurIPS_ , 32.
  * Jhamtani and Berg-Kirkpatrick (2018) Jhamtani, H.; and Berg-Kirkpatrick, T. 2018.  Learning to Describe Differences Between Pairs of Similar Images.  In _Proc. Conf. Empirical Methods NLP_ , 4024–4034.
  * Johnson et al. (2017) Johnson, J.; Hariharan, B.; Van Der Maaten, L.; Fei-Fei, L.; Lawrence Zitnick, C.; and Girshick, R. 2017.  Clevr: A diagnostic dataset for compositional language and elementary visual reasoning.  In _Proc. CVPR_ , 2901–2910.
  * Karpathy and Fei-Fei (2015) Karpathy, A.; and Fei-Fei, L. 2015.  Deep visual-semantic alignments for generating image descriptions.  In _Proc. CVPR_ , 3128–3137.
  * Kim et al. (2021) Kim, H.; Kim, J.; Lee, H.; Park, H.; and Kim, G. 2021.  Agnostic change captioning with cycle consistency.  In _Proc. ICCV_ , 2095–2104.
  * Li et al. (2023) Li, J.; Li, D.; Savarese, S.; and Hoi, S. 2023.  Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models.  _arXiv preprint arXiv:2301.12597_.
  * Li et al. (2020) Li, X.; Yin, X.; Li, C.; Zhang, P.; Hu, X.; Zhang, L.; Wang, L.; Hu, H.; Dong, L.; Wei, F.; et al. 2020.  Oscar: Object-semantics aligned pre-training for vision-language tasks.  In _Proc. ECCV_ , 121–137. Springer.
  * Lin (2004) Lin, C.-Y. 2004.  Rouge: A package for automatic evaluation of summaries.  In _Text summarization branches out_ , 74–81.
  * Lu et al. (2017) Lu, J.; Xiong, C.; Parikh, D.; and Socher, R. 2017.  Knowing when to look: Adaptive attention via a visual sentinel for image captioning.  In _Proc. CVPR_ , 375–383.
  * Luo et al. (2021) Luo, Y.; Ji, J.; Sun, X.; Cao, L.; Wu, Y.; Huang, F.; Lin, C.-W.; and Ji, R. 2021\.  Dual-level collaborative transformer for image captioning.  In _Proc. AAAI_ , volume 35, 2286–2293.
  * Mao et al. (2015) Mao, J.; Xu, W.; Yang, Y.; Wang, J.; Huang, Z.; and Yuille, A. 2015.  Deep captioning with multimodal recurrent neural networks (m-rnn).  In _Proc. ICLR_.
  * Merullo et al. (2022) Merullo, J.; Castricato, L.; Eickhoff, C.; and Pavlick, E. 2022.  Linearly mapping from image to text space.  _arXiv preprint arXiv:2209.15162_.
  * Mokady, Hertz, and Bermano (2021) Mokady, R.; Hertz, A.; and Bermano, A. H. 2021.  Clipcap: Clip prefix for image captioning.  _arXiv preprint arXiv:2111.09734_.
  * Mustafa et al. (2022) Mustafa, B.; Ruiz, C. R.; Puigcerver, J.; Jenatton, R.; and Houlsby, N. 2022.  Multimodal Contrastive Learning with LIMoE: the Language-Image Mixture of Experts.  In _NeurIPS_.
  * Nguyen et al. (2021) Nguyen, E.; Bui, T.; Swaminathan, V.; and Collomosse, J. 2021.  Oscar-net: Object-centric scene graph attention for image attribution.  In _Proc. ICCV_ , 14499–14508.
  * Papineni et al. (2002) Papineni, K.; Roukos, S.; Ward, T.; and Zhu, W.-J. 2002.  Bleu: a method for automatic evaluation of machine translation.  In _Proc. Assoc. Comp. Linguistics_ , 311–318.
  * Park, Darrell, and Rohrbach (2019) Park, D. H.; Darrell, T.; and Rohrbach, A. 2019.  Robust change captioning.  In _Proc. ICCV_ , 4624–4633.
  * Pizzi et al. (2022) Pizzi, E.; Roy, S. D.; Ravindra, S. N.; Goyal, P.; and Douze, M. 2022.  A self-supervised descriptor for image copy detection.  In _Proc. CVPR_ , 14532–14542.
  * Radford et al. (2021) Radford, A.; Kim, J. W.; Hallacy, C.; Ramesh, A.; Goh, G.; Agarwal, S.; Sastry, G.; Askell, A.; Mishkin, P.; Clark, J.; et al. 2021.  Learning transferable visual models from natural language supervision.  In _Proc. ICML_ , 8748–8763. PMLR.
  * Rennie et al. (2017) Rennie, S. J.; Marcheret, E.; Mroueh, Y.; Ross, J.; and Goel, V. 2017.  Self-critical sequence training for image captioning.  In _Proc. CVPR_ , 7008–7024.
  * Rombach et al. (2022) Rombach, R.; Blattmann, A.; Lorenz, D.; Esser, P.; and Ommer, B. 2022.  High-resolution image synthesis with latent diffusion models.  In _Proc. CVPR_ , 10684–10695.
  * Schuhmann et al. (2022) Schuhmann, C.; Beaumont, R.; Vencu, R.; Gordon, C.; Wightman, R.; Cherti, M.; Coombes, T.; Katta, A.; Mullis, C.; Wortsman, M.; Schramowski, P.; Kundurthy, S.; Crowson, K.; Schmidt, L.; Kaczmarczyk, R.; and Jitsev, J. 2022.  LAION-5B: An open large-scale dataset for training next generation image-text models.  In _NeurIPS_ , volume 35, 25278–25294.
  * Shi et al. (2020) Shi, X.; Yang, X.; Gu, J.; Joty, S.; and Cai, J. 2020.  Finding it at another side: A viewpoint-adapted matching encoder for change captioning.  In _Proc. ECCV_ , 574–590. Springer.
  * Song et al. (2020) Song, K.; Tan, X.; Qin, T.; Lu, J.; and Liu, T.-Y. 2020.  Mpnet: Masked and permuted pre-training for language understanding.  _NeurIPS_ , 33: 16857–16867.
  * Stefanini et al. (2022) Stefanini, M.; Cornia, M.; Baraldi, L.; Cascianelli, S.; Fiameni, G.; and Cucchiara, R. 2022.  From show to tell: A survey on deep learning-based image captioning.  _IEEE TPAMI_ , 45(1): 539–559.
  * Sun et al. (2022) Sun, Y.; Li, L.; Yao, T.; Lu, T.; Zheng, B.; Yan, C.; Zhang, H.; Bao, Y.; Ding, G.; and Slabaugh, G. 2022.  Bidirectional difference locating and semantic consistency reasoning for change captioning.  _IJIS_ , 37(5): 2969–2987.
  * Tan et al. (2019) Tan, H.; Dernoncourt, F.; Lin, Z.; Bui, T.; and Bansal, M. 2019.  Expressing Visual Relationships via Language.  arXiv:1906.07689.
  * Touvron et al. (2023) Touvron, H.; Lavril, T.; Izacard, G.; Martinet, X.; Lachaux, M.-A.; Lacroix, T.; Rozière, B.; Goyal, N.; Hambro, E.; Azhar, F.; et al. 2023.  Llama: Open and efficient foundation language models.  _arXiv preprint arXiv:2302.13971_.
  * Tsimpoukelli et al. (2021) Tsimpoukelli, M.; Menick, J. L.; Cabi, S.; Eslami, S.; Vinyals, O.; and Hill, F. 2021.  Multimodal few-shot learning with frozen language models.  _NeurIPS_ , 34: 200–212.
  * Tu et al. (2023a) Tu, Y.; Li, L.; Su, L.; Du, J.; Lu, K.; and Huang, Q. 2023a.  Viewpoint-Adaptive Representation Disentanglement Network for Change Captioning.  _IEEE Transactions on Image Processing_ , 32: 2620–2635.
  * Tu et al. (2023b) Tu, Y.; Li, L.; Su, L.; Lu, K.; and Huang, Q. 2023b.  Neighborhood Contrastive Transformer for Change Captioning.  arXiv:2303.03171.
  * Vedantam, Lawrence Zitnick, and Parikh (2015) Vedantam, R.; Lawrence Zitnick, C.; and Parikh, D. 2015.  Cider: Consensus-based image description evaluation.  In _Proc. CVPR_ , 4566–4575.
  * Vinyals et al. (2015) Vinyals, O.; Toshev, A.; Bengio, S.; and Erhan, D. 2015.  Show and tell: A neural image caption generator.  In _Proc. CVPR_ , 3156–3164.
  * Wang and Komatsuzaki (2021) Wang, B.; and Komatsuzaki, A. 2021.  GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model.  https://github.com/kingoflolz/mesh-transformer-jax.  Accessed: 2024-01-20.
  * Wang et al. (2021) Wang, Z.; Yu, J.; Yu, A. W.; Dai, Z.; Tsvetkov, Y.; and Cao, Y. 2021.  Simvlm: Simple visual language model pretraining with weak supervision.  In _Proc. ICLR_.
  * Yang et al. (2019) Yang, X.; Tang, K.; Zhang, H.; and Cai, J. 2019.  Auto-encoding scene graphs for image captioning.  In _Proc. CVPR_ , 10685–10694.
  * Yao, Wang, and Jin (2022) Yao, L.; Wang, W.; and Jin, Q. 2022.  Image difference captioning with pre-training and contrastive learning.  In _Proc. AAAI_ , volume 36, 3108–3116.
  * Yao et al. (2019) Yao, T.; Pan, Y.; Li, Y.; and Mei, T. 2019.  Hierarchy parsing for image captioning.  In _Proc. ICCV_ , 2621–2629.
  * Zhang et al. (2021) Zhang, P.; Li, X.; Hu, X.; Yang, J.; Zhang, L.; Wang, L.; Choi, Y.; and Gao, J. 2021\.  Vinvl: Revisiting visual representations in vision-language models.  In _Proc. CVPR_ , 5579–5588.