# Unsupervised Anomaly and Change Detection with Multivariate Gaussianization

José A. Padrón-Hidalgo, Valero Laparra, and Gustau Camps-Valls Paper published
in IEEE Transactions on Geoscience and Remote Sensing, vol. 60, pp. 1-10,
2022, Art no. 5513010, doi: 10.1109/TGRS.2021.3116186. Image Processing
Laboratory (IPL)
Universitat de València, Catedrático A. Escardino - 46980 Paterna, València
(iSpain). E-mail<EMAIL_ADDRESS>Research funded by the European Research
Council (ERC) under the ERC-CoG-2014 SEDAL project (grant agreement 647423)
and the Spanish Ministry of Economy, Industry and Competitiveness under the
‘Network of Excellence’ program (grant code TEC2016-81900-REDT) and the
projects TEC2016-77741-R, DPI2017-89867-C2-2-R, RTI2018-096765-A-I00 and
PID2019-109026RB-I00. Jose A. Padrón was supported by the Grisolia grant from
Generalitat Valenciana (GVA) with code GRISOLIA/2016/100.

###### Abstract

Anomaly detection is a field of intense research in remote sensing image
processing. Identifying low probability events in remote sensing images is a
challenging problem given the high-dimensionality of the data, especially when
no (or little) information about the anomaly is available a priori. While
plenty of methods are available, the vast majority of them do not scale well
to large datasets and require the choice of some (very often critical)
hyperparameters. Therefore, unsupervised and computationally efficient
detection methods become strictly necessary, especially now with the data
deluge problem. In this paper, we propose an unsupervised method for detecting
anomalies and changes in remote sensing images by means of a multivariate
Gaussianization methodology that allows to estimate multivariate densities
accurately, a long-standing problem in statistics and machine learning. The
methodology transforms arbitrarily complex multivariate data into a
multivariate Gaussian distribution. Since the transformation is
differentiable, by applying the change of variables formula one can estimate
the probability at any point of the original domain. The assumption is
straightforward: pixels with low estimated probability are considered
anomalies. Our method can describe any multivariate distribution, makes an
efficient use of memory and computational resources, and is parameter-free. We
show the efficiency of the method in experiments involving both anomaly
detection and change detection in different remote sensing image sets. For
anomaly detection we propose two approaches. The first using directly the
Gaussianization transform and the second using an hybrid model that combines
Gaussianization and the Reed-Xiaoli (RX) method typically used in anomaly
detection. For change detection we take advantage of the Gaussianization
transform and attribute the change to pixels with low probability compared to
the first image, instead of those with high difference value typically
employed in remote sensing. Results show that our approach outperforms other
linear and nonlinear methods in terms of detection power in both anomaly and
change detection scenarios, showing robustness and scalability to
dimensionality and sample sizes.

###### Index Terms:

Change Detection (CD), Anomaly detection, Extremes, Gaussianization, principal
component analysis, information, deep learning, probability density
estimation.

##  I Introduction

Remote Sensing (RS) has become a powerful tool to develop applications for
Earth monitoring [1, 2, 3]. Earth observation (EO) satellite missions, such as
Sentinels-2 and Landsat-8 are able to replace the hard and costly work of the
man on the ground. Also, the use of very high resolution (VHR) satellite
imagery (e.g. QuickBird and the Worldview constellation) is becoming
increasingly important for remote sensing applications, and it makes possible
the detection of dangerous events such as extreme precipitations, heat waves,
latent fires, droughts, floods or urbanization. The vast amount of data
available from different sensors makes it urgent to have automatic methods to
detect these events. A good and quite standard approach nowadays to tackle
this problem considers statistical models that allow us to detect anomalies
and changes on the Earth cover.

Figure 1: Illustration of the RBIG procedure on toy data. Original data
($i=0$) is transformed by using multiple iterations ($i=1,2,3,4...$). After
enough iterations ($i=M$) the data follows a Gaussian distribution, in this
particular case $M=150$. This example has been performed using the toolbox
that can be found in [4].

Statistical methods for anomaly detection (AD) focus on detecting small
portions of the image which do not belong to the background of the scene [5].
Anomalies are considered a group of weird (low probability) pixels which
significantly differ from their neighbors. AD is a challenging task and many
variants have been proposed in the literature, such as neighbor based,
clustering, classification, etc [6, 7, 8, 9, 10, 11]. However, among all of
them, the Reed-Xiaoli (RX) approach [12] is still the most widely used method
for AD since the Gaussian distribution assumption is a reasonable approach in
several cases, it is unsupervised, fast and easy to implement. The RX method
allows us to detect the anomalous samples compared to the background using the
well-known Mahalanobis distance. RX can be used as either a local or global
approach [13]. In this study, all the methods are used in a global setting for
the sake of simplicity. However, the proposed method is general enough and
could be easily adapted to both local and dual-window approaches. Since the
Gaussian assumption is not flexible enough in most cases, variants of the RX
has been developed to cope with higher-order feature relations. One option
which obtains good results is based on the theory of reproducing kernels in
Hilbert spaces, which extend the RX approach to the kernel RX (KRX) [14, 15,
16]. However, the KRX algorithm has not been widely adopted in practice
because, being a kernel method, the memory and computational cost increase
with the number of pixels cubically and quadratically respectively, and more
importantly the selection of the kernel parameters is critical to achieve a
good performance. While unsupervised approaches to fit the kernel parameter
exist, they achieve a sub-optimal performance and hence supervised approaches
have to be used. In this manuscript we propose to use a different, more
straightforward approach to the problem of anomaly detection based on
multivariate Gaussianization transformation. The proposed method is able to
provide an estimation of the probability of multidimensional data. The basic
idea is simple: the method finds an invertible and differentiable
transformation that converts the multivariate distribution of the input data
into a multivariate Gaussian and applies the change of variable formula under
transformations to estimate the density in the input domain. Therefore, the
probability of each data point in the original domain can be estimated by
using two ingredients: the probability of the data in the transform domain and
the determinant of the Jacobian of the transformation applied to the point. It
is important to stress the fact that both ingredients are needed to estimate
the density in the input domain. That is, the method is not transforming the
data to a Gaussian domain to estimate the probability there, but uses the
Gaussian domain as a convenient intermediate stage to estimate the density in
the original one. Actually, the probability in the Gaussian domain must be
multiplied by the determinant of the Jacobian. As we will see, the operations
involved to estimate these transformations are easy and fast, the method
scales well with data dimensionality and does not require additional
information to fit any parameter [17, 18]. The parameters required for RBIG
computation are the ones to fit the marginal density estimation. In our case,
we use the simplest case based on computing histograms with the default
parameters in the toolbox (see [4]) and no extra fitting is used in any case.
Note that the proposed method estimates the data density using all available
data. In the AD setting, a naive application would incidentally use the
anomalies too, which is not desirable. To address this issue, we propose a
two-step procedure that finds a trade-off between the Gaussian assumption
(very rigid) aimed to discard the potential anomalies, and the proposed method
(very flexible) aimed to better estimate the background density. This two-step
approach improves the results of the two components when used separately, and
yields more robust density estimates, which ultimately leads to a powerful,
automatic, unsupervised, algorithm for anomaly detection.

Detecting changes in images automatically is extremely important because it
allows us to improve predictions and our understanding of events. The usual
approach to change detection (CD) in remote sensing is based on arithmetic
operations over the image before and after the possible change [19, 9]. The
most usual approach to detect changes is based on thresholding or classifying
the difference image. This approach relies on previous (and quite critical)
stages like atmospheric correction or image co-registration. Here we instead
illustrate our method on the CD approach of finding statistical
(probabilistic) differences between the signals before and after the event
[20]. Other settings for change detection with RBIG could be defined by
exploiting differences, ratios or other convenient transformations of the pair
of images. Our approach can be considered as particular case of the anomaly
detection problem, where the change class is the target class to be detected.

As for AD, a similar problem occurs when using statistical methods for CD [21,
22]: one aims to learn the distribution of the original image and analyze the
statistical differences of the pixels in the new incoming image. Likewise, the
RX and KRX extensions have been proposed to deal with CD problems. However,
they show the same drawbacks as in AD. Here we describe how the multivariate
Gaussianization can be used also in CD problems. Note that in both cases, AD
and CD, the proposed statistical method is used to evaluate the pixel’s
probability so that one can classify them as anomalies or changes,
respectively.

The remainder of the manuscript is organized as follow. Section II summarizes
the Gaussianization transformation in general, and how to adapt it to anomaly
and change detection. In section III, we illustrate its performance in three
experiments, involving simulated anomalies, real AD and CD examples with a
database of multi- and hyperspectral images. Results show that the proposed
approach is robust and flexible enough to be applied in different AD and CD
scenarios, and obtains better performance than other simple and robust methods
(like the RX) and more flexible and adaptable ones, like the KRX. Section IV
concludes the paper with some remarks and further work.

##  II Multivariate Gaussianization for Detection

Here we use the rotation-based iterative Gaussianization (RBIG), a
nonparametric method for density estimation of multivariate distributions
proposed in [17, 18]. RBIG is rooted in the idea of Gaussianization,
introduced in the seminal work by [23] and further developed in [24], which
consists of seeking for a transformation $G_{x}$ that converts a multivariate
dataset $\mathbf{X}\in\mathbb{R}^{\ell\times d}$ in domain $X$ to a domain
where the mapped data $\mathbf{Y}\in\mathbb{R}^{\ell\times d}$ follows a
multivariate normal distribution in domain $Y$, i.e.
$p_{Y}(\mathbf{y})\sim\mathcal{N}({0},{I})$:

$\displaystyle\begin{array}[]{llll}G_{x}:&\mathbf{x}\in\mathbb{R}^{d}&\mapsto&\mathbf{y}\in\mathbb{R}^{d}\\\
&\sim
p_{X}(\mathbf{x})&&p_{Y}(\mathbf{y})\sim{\mathcal{N}}({0},{I}_{d}),\end{array}$
(3)

where inputs and mapped data points have the same dimensionality
$\mathbf{x},\mathbf{y}\in\mathbb{R}^{d}$, ${0}$ is a vector of zeros (for the
means) and ${I}_{d}$ is the identity matrix for the covariance of dimension
$d$. Using the change of variable formula one can estimate the probability of
a point $\mathbf{x}$ in the original domain:

$p_{X}(\mathbf{x})=p_{Y}(\mathbf{y})|J_{G_{x}}(\mathbf{y})|,$ (4)

where $p_{X}(\mathbf{x})$ is the probability distribution of the original data
point $\mathbf{x}$, $p_{Y}(\mathbf{y})$ is the probability of the data point
transformed to the Gaussian domain (note that by definition
$p_{Y}\sim{\mathcal{N}}({0},{I}_{d})$ and can be easily computed), and
$|J_{G_{x}}(\mathbf{y})|$ is the determinant of the Jacobian of the
transformation $G_{x}$ in the point $\mathbf{y}$.

It is important to stress the important role of the term
$|J_{G_{x}}(\mathbf{y})|$: Note that one would be tempted to see the
Gaussianization as a sort of pre-processing step that converts data to a
convenient (Gaussian) domain where one could then apply methods that assume
Gaussian distribution. This is, however, incorrect since the term
$|J_{G_{x}}(\mathbf{y})|$ has to be included in the density estimation, and it
is computed for each data point. For this formula to work, $G_{x}$ has to be
differentiable, i.e. the $|J_{G_{x}}(\mathbf{y})|>0,\forall\mathbf{y}$. The
Gaussianization method we propose in this paper, RBIG, obtains a
transformation $G_{x}$ that fulfills this property, cf. [17]. Therefore, RBIG
can be easily applied to estimate the probability of data points in the
original domain, $p_{X}(\mathbf{x})$.

RBIG is an iterative algorithm, where in each iteration, $n$, two steps are
applied: 1) a set of $d$ marginal Gaussianizations to each of the variables,
$\boldsymbol{\Psi}=[\Phi_{1},\ldots,\Phi_{d}]$, and 2) a linear rotation,
${R}\in\mathbb{R}^{d\times d}$:

$\mathbf{x}[i+1]={R}[i]\cdot\boldsymbol{\Psi}[i](\mathbf{x}[i]),~{}~{}~{}i=1,\ldots,M$
(5)

where $M$ is the number of steps (iterations) in the sequence, $i=1,\ldots,M$.
The final transformation $G_{x}$ is the composition of all performed
transformations through iterations. This procedure is illustrated with a toy
example in Fig. 1. In [17] we showed that with enough iterations $M$ the
method converges and the transformed data ultimately follows a standardized
Gaussian, i.e. $p_{Y}(\mathbf{y})\sim\mathcal{N}({0},{I}_{d})$, taking
$\mathbf{y}=\mathbf{x}[M]$.

The computational load can be divided on two steps: the first one when the
method is trained on a representative training set, and the second when is
applied to unseen test dataset. On the one hand, the training time will depend
on the the number of iterations and the employed type of linear transformation
(see [25] for a discussion on the different options). A reasonable number of
iterations cannot be set before hand since it will be depend on the non-
gaussianity of the data. However, it can be automatically found with the
stopping criteria proposed in [25]. On the other hand, when applying the
method, the computation of the Jacobian will depend only on the number of
iterations. It is important to note that, since the linear transformations are
restricted to be rotations, the determinant of the Jacobian for them is always
equal to $1$. Therefore the only part to be computed is the Jacobian of the
marginal transformation, which is equal to the probability of the data before
applying the transformation [25]. Therefore, the application of the method is
in principle computationally easier than the training step. However, note that
if one wants to obtain the full Jacobian, not only the determinant as it is
needed to compute the probability, the amount of data to be store is huge
which can make the method difficult to apply. The use of RBIG in the framework
of Normalizing Flows or density destructors can be useful in this sense since
it is devoted to find transformation with easily computable Jacobians [26]. An
implementation of RBIG in this framework can be found in [4].

An illustration of how RBIG can be adapted to describe the distribution of
remote sensing data is shown in Fig. 2. In this example we take data from the
Sentinel-2 image Australia (see Table I for details), which has $d=12$ bands,
and use RBIG to Gaussianize its pixel’s distribution. We can see that the
Gaussianized data follows a Gaussian distribution. Besides we apply the
inverse of the learned Gaussianization transformation to randomly generated
Gaussian points obtaining synthetic new data that follows a deemed similar
distribution as the original one. This illustrates the invertibility property
of RBIG, which allows us to estimate densities in the original domain and use
the well-known relation between probability and anomaly to derive unsupervised
density-based anomaly and change detectors. Now we describe how
Gaussianization can be applied to both anomaly detection (AD) and change
detection problems (CD). The point-wise density estimate can be trivially
linked to a degree of anomaly in a particular image; the higher the
probability the lower the anomalousness. Likewise, the point-wise density
estimate can be used to assess the probability of change between images; ones
uses the first image to estimate the background, and applies the transform to
the second image.

Real Image | Original Data
---|---
|
Gaussianized data | Synthesized data
|
Figure 2: Illustration of synthesized data using the RBIG methodology in a
real Sentinel-2 image. Top-left: RGB composite of the original image. Top-
right: representation of the first two bands of the original image. Bottom-
left: first two dimensions of the Gaussianized data. Bottom-right: first two
bands of the original image data (blue) and randomly generated data inverted
using the learned Gaussianization transformation (orange).

###  II-A RBIG for Detection of Anomalies

One of the most successful methods applied to the problem of anomaly detection
is the Reed-Xiaoli (RX) method [12], a successful type of matched filter. The
idea behind the RX method can be interpreted in probabilistic terms [15];
intuitively, a data point is more anomalous when it has less probability to
appear:

$A(\mathbf{x})\propto\frac{1}{p_{X}(\mathbf{x})}.$ (6)

Actually, when the distribution is assumed to be Gaussian, $p_{X}\sim p_{G}$,
this relation defines the RX method anomaly detector, i.e. $A(\mathbf{x})\sim
A_{\text{RX}}(\mathbf{x})$. Actually $A_{\text{RX}}(\mathbf{x})$ is equivalent
to the Mahalanobis distance between the data point and the mean, i.e.
$A_{\text{RX}}(\mathbf{x})=(\mathbf{x}-\mu)^{\top}\boldsymbol{\Sigma}^{-1}(\mathbf{x}-\mu)$,
where $p(\mathbf{x})\sim{\mathcal{N}}({0},\boldsymbol{\Sigma})$.

While RX has been widely used, it has the limitations inherent to the Gaussian
distribution assumption. The use of kernel methods has been proposed to
generalize the RX method to the nonlinear and non-Gaussian case [14, 15].
Conceptually, the kernel RX (KRX) method can be understood as a way to propose
a Gaussian distribution where the covariance is defined in a higher
dimensional Hilbert feature space. Taking as a reference the RX method, this
translates into replacing the covariance matrix with a kernel matrix that
estimates the similarities between samples [27, 28]. In practice, this implies
that correlation is substituted by a non-linear (kernel) similarity measure.
The anomaly detected using the KRX method can be formulated as:

$A_{\text{KRX}}(\mathbf{x})\propto\frac{1}{p_{K}(\mathbf{x})},$ (7)

where $p_{K}(\mathbf{x})$ is the distribution induced by using the kernel
function instead of the covariance. The KRX is an elegant extension of the RX,
yet it has the problem of fitting kernel parameters and the high computational
cost (as one has to invert a kernel matrix, which has cubic cost with the
number of points $\ell$). Whereas some heuristics exist in the literature to
fit the kernel parameters, in practice one only achieves the full potential of
the KRX approach by fitting the parameters after cross-validation [15]. This
requires having access to labeled data as anomalous versus non-anomalous
classes, which is not a very realistic and not even practical setting. In this
work, we approach the more useful and practical, yet more challenging, problem
of unsupervised anomaly detection (i.e. no labeled data available), and
therefore in our comparisons we will fit the kernel method parameter using the
most successful (and sensible) heuristic to set the Gaussian kernel
lengthscale $\sigma$ as the average of all distances among $\mathbf{X}$.

As an alternative to linear measures of anomalousness like in RX, or nonlinear
yet implicit feature transformations with parameters to tune like in KRX, we
here propose a more straightforward approach to estimate the probability
density function with RBIG (sec. II). Here we stick to the idea of assuming a
Gaussian distribution in a transform domain. In particular, RBIG forces the
true distribution in the transform domain to be a multivariate Gaussian.
However, note that the distribution in the original domain is estimated not
only assuming Gaussian distribution in the transform domain but also including
the determinant of the Jacobian of the transformation (see eq. 4). This gives
us a non-parametric, parameter-free and efficient estimation of the data
distribution in the original domain. RBIG is an unsupervised method by
construction so it does not require labeled data, and scales linearly with the
data. By using RBIG to compute $p_{X}$, we obtain the method proposed in this
work:

$A_{\text{RBIG}}(\mathbf{x})\propto\frac{1}{p_{\text{RBIG}}(\mathbf{x})}.$ (8)

An important aspect to take into account is the intrinsic characteristics of
the data used to estimate the density, which has implications in the quality
of the estimation. When the distribution contains even a moderate number of
anomalies, an accurate density estimate will cast anomalies as regular points,
i.e. non-anomalous. This vastly depends on the flexibility of the class of
models used. When the model is rigid like in the RX case, this is not a
problem since it cannot be adapted to the anomalies. For the KRX one can
control this effect by tuning the kernel lengthscale and the regularization
term, but as explained before requires labeled data. This is an important
aspect to take into account mostly in the anomaly detection scenario, where
all data (included the anomalous samples) are used to estimate the density.
Therefore we propose to use an hybrid model that combines the (too rigid) RX
model with the (too flexible) RBIG model. First, we apply RX to discard data
more likely to be anomalous. The remaining points are then used to apply RBIG.
This tries to avoid using anomalous data in RBIG, which after all is intended
to learn the background or pervasive data distribution. The number of data
points selected as non-anomalous in the first step will define the trade-off
between flexibility and rigidity.

###  II-B RBIG for change detection

Remote sensing change detection algorithms have mainly relied on thresholding
or classifying the difference (or ratio) between a pair of previously co-
registered images [9]. Multitemporal registration and radiometric and
atmospheric corrections are thus very important aspects that strongly impact
the detection performance. A minimum error in image co-registration for
example leads to false detections. In this paper we illustrate and motivate
the use of RBIG following an alternative probabilistic approach. Basically,
the idea is to characterize statistically the original image probability, and
attribute the change to pixels in the second image with low probability. When
applied pixel-wise (as we do here) this has the drawback of not detecting some
changes that are not statistically noticeable. For instance, in two images
with the same car in different locations of a road, the method would not
detect the car as a change. Such unfortunate cases could be however addressed
by extending the idea easily by working in a sliding window instead of a
pixel-wise approach. The statistical approach has the advantage of reducing
the importance of perfect co-registration since the method operates in the
geometric space defined by the image, not in the spatial domain explicitly.
Thus, small errors of co-registration would not penalize the performance of
the proposed approach.

The idea to exploit RBIG for change detection is using data coming from the
first image $\mathbf{X}_{1}$ only to estimate the probability model and then
evaluating the probability (or change score, $C$) for each point in the second
image $\mathbf{X}_{2}$, as follows:

$C(\mathbf{x}_{2})\propto\frac{1}{p_{X_{1}}(\mathbf{x}_{2})}.$ (9)

As for the anomaly detection case, we can use different models to estimate
$p_{X_{1}}$. The most widely used is the Gaussian model. As in the previous
section, when assuming a Gaussian distribution for the input data, the RX
method can be used here too, i.e. $C_{\text{RX}}(\mathbf{y})$.

Likewise, kernel methods have been proposed to alleviate the strict assumption
of Gaussian distribution [15] . While different configurations were proposed
in order to take into account only the anomalous changes, here we use the
configuration designed for change detection. Following the idea in equation
(9), the data of the first image ($\mathbf{X}_{1}$) is used to estimate the
kernel and then the method is evaluated in the second image:

$C_{\text{KRX}}(\mathbf{x}_{2})\propto\dfrac{1}{p_{\text{K}(X_{1})}(\mathbf{x}_{2})}.$
(10)

Equivalently, we can use RBIG to estimate the probability of the first image
and evaluate the probability in the second one:

$C_{\text{RBIG}}(\mathbf{x}_{2})\propto\dfrac{1}{p_{\text{RBIG}(X_{1})}(\mathbf{x}_{2})}.$
(11)

It is important to note that, in this case, the data used to estimate the
probability density does not contain anomalies (changes in this setting) so
the hybrid model is not needed here.

##  III Experimental Results

This section analyzes the performance of the proposed RBIG method for anomaly
and change detection. In order to assess the robustness we performed tests in
both simulated and real scenes of varying dimensionality and sample size. We
evaluate the detection power of the methods quantitatively through the
Receiver Operating Characteristic (ROC) and Precision-Recall (PR) curves,
along with the Area Under the Curve (AUC) scores. Besides, we provide examples
of detection maps of each method to evaluate their quality by visual
inspection.

We have performed three experiments. The first experiment is designed to
illustrate the effect of the evaluated in an anomaly detection (AD) toy
example. The second experiment deals with AD problem in different real
scenarios: detection of air planes, latent fires, vehicles, and urbanization
(roofs). The third experiment is related to evaluate the methods in change
detection (CD) problems involving floods, fires and droughts. Table I
summarizes the different data sets used in the experiments.

We added two standard methods in the experimental results for comparison: (1)
a kernel-based method known as the support vector domain descriptor (SVDD)
[29], where one seeks to embrace all data points into a hypersphere in Hilbert
space, and (2) the classical kernel density estimation (KDE), where a Gaussian
kernel is used to define the distribution. All methods were used in the same
way to the anomaly detection (Eq. (6)) and the change detection problems (Eq.
(9)). In order to ease the reproducibility, we provide MATLAB code
implementations of the all methods, as well as database with the labeled
images used in the second and third experiments in [4].

###  III-A Experiment 1: Simulated Anomalies

The aim of this experiment is to illustrate the behavior of the proposed
methods in challenging distributions exhibiting highly nonlinear feature
relations. We designed a two-dimensional dataset where the non-anomalous data
is in a circumference and the anomalous data in the middle. Figure 3 shows the
performance of the different methods. The RX method assumption does not hold
(the data is clearly non-Gaussian), hence it shows poor performance. The
performance of KRX is better than RX but some false detections emerge in the
outer circle, mainly related to the difficulty to select a reasonable kernel
parameter. The direct application of RBIG easily identifies the anomalous
points since they are far from the more dense (most probable) region. The
proposed hybrid model further refines the detection since the density is
estimated from pervasive data yielded by RX only.

|
---|---
|

(a) RX

(b) KRX

(c) RBIG

(d) HYBRID

Figure 3: Synthetic experiment to illustrate the methods performance when
detecting anomalies. The color bar shows the intensity in terms of anomaly
score from dark blue (less) to yellow (more). The image (a) correspond to RX
detector, image (b) is the kernel version of RX, (c) represent the RBIG method
and (d) showcase the hybrid model.

###  III-B Experiment 2: Anomaly Detection in Real Scenarios

We performed tests in four real examples. Table I summarizes relevant
attributes of the datasets such as sensors, spatial and spectral resolution.

####  III-B1 Data collection

We collected multispectral and hyperspectral images acquired by the AVIRIS and
ROSIS-03 sensors. Figure 4 showcases the scenes used in the experiments. The
AD scenarios consider anomalies related to a diversity of problems: airplane,
latent fires, urbanization and vehicle detection [30, 31, 32].

TABLE I: Image attributes used for the experiments of anomaly detection (AD) and change detection (CD).  Images | Sensor | Size | Bands | SR [m]
---|---|---|---|---
AD
Cat-Island | AVIRIS | 150$\times$150 | 188 | 17.2
WTC | AVIRIS | 200$\times$200 | 224 | 1.7
Texas-Coast | AVIRIS | 100$\times$100 | 204 | 17.2
GulfPort | AVIRIS | 100$\times$100 | 191 | 3.4
CD
Texas | Cross-Sensor | 301$\times$201 | 7 | 30
Argentina | Sentinel-2 | 1257$\times$964 | 12 | 10-60
Chile | Landsat-8 | 201$\times$251 | 12 | 10-60
Australia | Sentinel-2 | 1175$\times$2031 | 12 | 10-60
|  |  |  |  |
---|---|---|---|---|---
Cat-Island | GT | RX (0.96) | KDE (0.97) | RBIG (0.99) | HYBRID (0.99)
|  |  |  |  |
WTC | GT | RX (0.95) | KDE (0.95) | RBIG (0.95) | HYBRID (0.95)
|  |  |  |  |
Texas-Coast | GT | RX (0.99) | KDE (0.99) | RBIG (0.94) | HYBRID (0.99)
|  |  |  |  |
GulfPort | GT | RX (0.90) | KRX (0.95) | RBIG (0.95) | HYBRID (0.95)
Figure 4: Anomaly detection predictions in four images (one per row). First
column: Cat-Island, World Trade Center (WTC), Texas Coast and Pavia original
datasets with anomalies outlined in green. Second column: represent the
reference maps of each image. From third column to the last column: activation
maps and the AUC values (in parenthesis) for the RX, the best performance
method among KRX, KDE and SVDD, RBIG and the HYBRID model, respectively.

(a) Cat-Island

(b) WTC

(c) Texas Coast

(d) GulfPort

Figure 5: Anomaly detection ROC curves in linear scale for all scenes. Numbers
in legend display the AUC values for each method.

(a) Cat-Island (b) WTC (c) Texas Coast (d) GulfPort

Figure 6: Anomaly detection results of the bootstrap experiment for 1000
experiments. AUC values and standard deviation for each method are shown as
boxplot, red line represent the median value, the blue box contains 95% of the
values, black lines represent the maximum and minimum values.

The Cat-Island dataset corresponds to the airplane captured flying over the
beach and it is considered a strange object when compared to the rest of the
image (a white spot in the middle of a beach) and the percentage of anomalies
represent the 0.09% of the scene. The World Trade Center (WTC) image was
collected by the Airborne Visible Infra-Red Imaging Spectrometer (AVIRIS) over
the WTC area in New York on 16 September 2001 (after the collapse of the
towers in NY). The data set covered the hot spots corresponding to latent
fires at the WTC, which can be considered as anomalies and it represent the
0.23% of the scene. In the Texas Coast dataset, the anomalies represent the
0.67% of the scene and the image contains roofs built on a wooded site and
bright spots that reflect light which can be considered an anomaly. The
GulfPort dataset correspond to a battery of airplanes taxied on the runway and
the pecentage of anomalies represent the 0.60% of the scene.

|  |  |  |
---|---|---|---|---
Texas ($t_{1}$) | Texas ($t_{2}$) | RX(0.91) | KSVDD(0.95) | RBIG(0.98)
|  |  |  |
Argentina ($t_{1}$) | Argentina ($t_{2}$) | RX (0.94) | K-RX (0.93) | RBIG (0.97)
|  |  |  |
Chile ($t_{1}$) | Chile ($t_{2}$) | RX (0.64) | KRX (0.66) | RBIG (0.72)
|  |  |  |
Australia ($t_{1}$) | Australia ($t_{2}$) | RX (0.86) | KSVDD (0.93) | RBIG (0.93)
Figure 7: Change detection results for different images. First two columns
show the images before and after the change, with the changed region
highlighted in green. Columns three to five show the prediction maps for the
different methods, the amount of change detected in each pixel is colored from
white (less) to red (more). AUC values are given in parenthesis. The changed
region is outlined in black to facilitate the visual inspection.

####  III-B2 Numerical and Visual Comparison

It is important to take into consideration that KRX requires the selection of
some hyperparameters, being the kernel parameter the most critical one. In
order to perform a fair comparison while staying in an unsupervised learning
setting, we use the standard RBF kernel function,
$k({a},{b})=\exp(-\|{a}-{b}\|^{2}/(2\sigma^{2}))$ and set the lengthscale
parameter $\sigma$ to the median distance between all examples.

TABLE II: AUC results for Anomaly Detection images. The value for the best method for each image is in bold. METHODS | RX | K-RX | SVDD | KDE | RBIG | HYBRID
---|---|---|---|---|---|---
Cat-Island | 0.96 | 0.70 | 0.70 | 0.97 | 0.99 | 0.99
WTC | 0.95 | 0.82 | 0.67 | 0.95 | 0.95 | 0.95
Texas-Coast | 0.99 | 0.86 | 0.77 | 0.99 | 0.94 | 0.99
GulfPort | 0.90 | 0.95 | 0.83 | 0.90 | 0.95 | 0.95

(a) Texas

(b) Argentina

(c) Chile

(d) Australia

Figure 8: ROC (top row) and Precision-Recall (bottom row) curves for change
detection problems.

A visual comparison of the results in terms of activation maps for all methods
is given in Fig. 4. They display the predictions given to each sample. The
prediction maps show a binary representation between change and non-change
samples obtained from the model subject to a threshold. Results in all scenes
demonstrate that (1) RX is a competitive method for detection, (2) KRX
struggles to obtain reasonable results mainly due to the problem of
hyperparameter tuning, (3) RBIG alone excels in all cases, while the hybrid
approach (i.e. RX followed by RBIG) refines the results and yields clearer
activation maps with sharper spatial detections.

Additionally, for a quantitative assessment of the results, it is customary to
provide the ROC curves and to derive scores like the AUC from it. Figure 5
shows the ROC curves and Table II summarizes all AUC values for all images and
methods. For each experiment, we performed 1000 runs for testing the
significance of the methods based on the ROC profiles. The results are shown
in Figure 6. Although the RBIG model achieves good results, RX model is able
to compete and achieve results as good as RBIG for some images. The HYBRID
model is able to keep the properties of the above mentioned models obtaining
results equal or better than any other method. While KRX obtains a reasonable
performance in some images, it clearly fails in some situations like the Cat-
Island image. The low standard deviations show that all methods but the KRX
are clearly robust with a little bit bigger standard deviation for the RX
method in most cases.

###  III-C Experiment 3: Real and Natural Changes

This section reports an experiment to analyze the performance of the proposed
methods in change detection problems. The database is composed of different
scenes with natural changes, whose characteristics are summarized in Table I.

####  III-C1 Data collection

We collected pairs of multispectral images in such a way that they coincide at
the same spatial resolution but at different acquisition time, the images are
co-registered. We selected the images in such a way that an anomalous change
happened between the two acquisition times. We manually labeled all the images
finding the changed pixels. Labeling considered avoiding shadows, changes in
lighting and natural changes in vegetation which could compromise results
evaluation. All images contain changes of different nature, which allows us to
analyze and study how the algorithms perform in heterogeneous realistic
scenarios. The Texas wildfire dataset is composed by a set of four images
acquired by different sensors over Bastrop County, Texas (USA), and is
composed by a Landsat 5 TM as the pre-event image and a Landsat 5 TM plus an
EO-1 ALI and a Landsat 8 as post-event images. This phenomenon is considered
the most destructive wildland-urban interface wildfire in Texas history and
the interest region represent the 19.54%. The Argentina image represents an
area burned between the months of July and August 2016 due to the high
temperatures in these crop areas, the change region representing the 7.5% of
the whole scene. The Chile dataset represents the Aculeo lake in central part
of this country, which has now dried up completely. These images contrast the
lake in 2014, when it still contained substantial water, and 2019, when it
consisted of dried mud and green vegetation. Scientists attribute the lake’s
decline to an unusual decade-long drought, coupled with increased water
consumption from a growing population, and the changed region represents a
relevant 10.81% of the whole scene. The last dataset labeled as Australia
shows the natural floods caused by Cyclone Debbie in Australia 2017. Storm
damage resulted from both the high winds associated with the cyclone, and the
very heavy rain that produced major riverine floods. The change samples
represent an important portion of the scene, the 17.35% of pixels affected.
Since our RBIG approach only takes the time

$t_{1}$

image, these big changes do not have a critical impact on method’s
performance.

####  III-C2 Numerical and Visual Comparison

Figure 7 shows the RGB composites of the pairs of images, the corresponding
reference map and activation maps obtained. RBIG obtains clearly better
results than the other methods in all cases; very good performance in three
out of the four scenarios and a clear advantage in the most difficult one
(Chile image).

When dealing with highly skewed datasets, PR curves give a more informative
picture of an algorithm’s performance compared to ROC. Figure 8 shows both the
ROC and the PR curves results for all methods and all the images. In all cases
RBIG outperforms the other methods largely, thus suggesting the suitability of
adopting a more direct approach of density estimation in the change detection
problems too. A summary of the AUC values of all methods and scenarios is
shown in Table III. The RBIG approach is to able to estimate the change
samples with a high accuracy overtaking in 7%, 3%, 6% and 5% respectively with
respect the second best method. While AUC returns an overall measure of
method’s robustness, in (remote sensing) anomaly detection problems one
typically cares about the low false alarm rate regime. We study the
performance of the methods looking at different false positive rates (FPRs) in
Table IV. The proposed RBIG approach consistently reports the best
performance, especially in very low FPR regimes.

TABLE III: AUC results for Change Detection images. The best value for each image are in bold  METHODS | RX | K-RX | K-SVDD | KDE | RBIG
---|---|---|---|---|---
Texas | 0.91 | 0.80 | 0.95 | 0.91 | 0.98
Argentina | 0.94 | 0.93 | 0.87 | 0.77 | 0.97
Chile | 0.64 | 0.66 | 0.61 | 0.63 | 0.72
Australia | 0.86 | 0.88 | 0.93 | 0.83 | 0.93
TABLE IV: Low false alarm rate regime. First row correspond to false positive rate (FPR) and columns represent the AUC values from rigorous to moderate FPR.  FPR | 0.1 | 0.2 | 0.3 | 0.1 | 0.2 | 0.3
---|---|---|---|---|---|---
| Texas | Argentina
RX | 0.78 | 0.87 | 0.91 | 0.93 | 0.96 | 0.97
KRX | 0.46 | 0.65 | 0.76 | 0.85 | 0.92 | 0.95
SVDD | 0.90 | 0.93 | 0.95 | 0.37 | 0.84 | 0.97
KDE | 0.78 | 0.86 | 0.90 | 0.20 | 0.48 | 0.75
RBIG | 0.96 | 0.98 | 0.99 | 0.95 | 0.96 | 0.97
| Chile | Australia
RX | 0.21 | 0.37 | 0.49 | 0.76 | 0.81 | 0.83
KRX | 0.09 | 0.34 | 0.50 | 0.79 | 0.83 | 0.86
SVDD | 0.07 | 0.08 | 0.45 | 0.84 | 0.89 | 0.93
KDE | 0.06 | 0.23 | 0.47 | 0.71 | 0.76 | 0.78
RBIG | 0.38 | 0.72 | 0.73 | 0.84 | 0.89 | 0.93

##  IV Conclusions

We introduced a novel detector based on multivariate Gaussianization. The
methodology copes with anomaly and change detection problems in remote sensing
image processing, and meets all requirements of the problems: is an
unsupervised method with no parameters to fit, it can deal with large amount
of data, and it is more accurate to competing approaches. The model assumption
is based on detecting anomalies by estimating probabilities of pixels. The
proposed method excelled quantitatively (AUC, ROC and PR curves) and
qualitative based on visual inspection over the rest of the implementations,
in both anomaly and change detection. The evaluation considered a wide range
of remote sensing images, in a diversity of problems, dimensionality and
number of examples. We also suggested a hybrid approach where the
Gaussianization method is applied after a regular anomaly detector: this
facilitates the density estimation and improves the results notably. Future
work will consider exploiting the information-theoretic properties of RBIG
[33] which opens alternatives to identify changes in image time series.

## References

  * [1] G. Camps-Valls, D. Tuia, L. Bruzzone, and J. A. Benediktsson,  “Advances in hyperspectral image classification: earth monitoring with statistical learning methods,”  IEEE Signal Processing Magazine, vol. 31, no. 1, pp. 45–54, 2014\.
  * [2] A.Plaza, J. A. Benediktsson, and J. W. Boardman,  “Recent advances in techniques for hyperspectral image processing,”  Remote Sensing of Environment, vol. 113, pp. S110 – S122, 2009\.
  * [3] G. Camps-Valls, Devis Tuia, Luis Gómez-Chova, Sandra Jiménez, and Jesús Malo,  Remote sensing image processing,  Morgan & Claypool Publishers, 1st edition, 2011.
  * [4] J. A. Padrón-Hidalgo, V. Laparra, and G. Camps-Valls,  “RBIG for anomaly and change detection in remote sensing,” https://isp.uv.es/RBIG4AD.html, 2021.
  * [5] D. W. J. Stein, S. G. Beaven, L. E. Hoff, E. M. Winter, A. P. Schaum, and A. D. Stocker,  “Anomaly detection from hyperspectral imagery,”  IEEE Signal Processing Magazine, vol. 19, no. 1, pp. 58–69, Jan 2002.
  * [6] S. Matteoli, M. Diani, and G. Corsini,  “A tutorial overview of anomaly detection in hyperspectral images,”  IEEE Aerospace and Electronic Systems Magazine, vol. 25, no. 7, pp. 5–28, 2010.
  * [7] M. Ben Salem, K. Saheb Ettabaa, and M. S. Bouhlel,  “An adaptive spatial and spectral neighborhood for the RX anomaly detector,”  in IGARSS 2018 - 2018 IEEE International Geoscience and Remote Sensing Symposium, 2018, pp. 8484–8487.
  * [8] Y. E. Sahın, S. Arisoy, and K. Kayabol,  “Anomaly detection with bayesian gauss background model in hyperspectral images,”  in 2018 26th Signal Processing and Communications Applications Conference (SIU), 2018, pp. 1–4.
  * [9] S. Liu, D. Marinelli, L. Bruzzone, and F. Bovolo,  “A review of change detection in multitemporal hyperspectral images: current techniques, applications, and challenges,”  IEEE Geoscience and Remote Sensing Magazine, vol. 7, no. 2, pp. 140–158, 2019.
  * [10] Y. Qu, R. Guo, W. Wang, H. Qi, B. Ayhan, C. Kwan, and S. Vance,  “Anomaly detection in hyperspectral images through spectral unmixing and low rank decomposition,”  in 2016 IEEE International Geoscience and Remote Sensing Symposium (IGARSS), 2016, pp. 1855–1858.
  * [11] W. Wang, S. Li, H. Qi, B. Ayhan, C. Kwan, and S. Vance,  “Identify anomaly component by sparsity and low rank,”  in 2015 7th Workshop on Hyperspectral Image and Signal Processing: Evolution in Remote Sensing (WHISPERS), 2015, pp. 1–4.
  * [12] I. S. Reed and X. Yu,  “Adaptive multiple-band CFAR detection of an optical pattern with unknown spectral distribution,”  IEEE Transactions on Acoustics, Speech, and Signal Processing, vol. 38, pp. 1760–1770, 1990.
  * [13] Y. Qu, W. Wang, R. Guo, B. Ayhan, C. Kwan, S. Vance, and H. Qi,  “Hyperspectral anomaly detection through spectral unmixing and dictionary-based low-rank decomposition,”  IEEE Transactions on Geoscience and Remote Sensing, vol. 56, no. 8, pp. 4391–4405, 2018.
  * [14] H. Kwon and N. M. Nasrabadi,  “Hyperspectral anomaly detection using kernel RX-algorithm,”  in 2004 International Conference on Image Processing, 2004. ICIP ’04., 2004, vol. 5, pp. 3331–3334 Vol. 5.
  * [15] J. A. Padron-Hidalgo, A. Perez-Suay, F. Nar, and G. Camps-Valls,  “Efficient Nonlinear RX Anomaly Detectors,”  IEEE Geoscience and Remote Sensing Letters, pp. 1–5, 2020.
  * [16] J. Zhou, C. Kwan, B. Ayhan, and M. T. Eismann,  “A novel cluster kernel rx algorithm for anomaly and change detection using hyperspectral images,”  IEEE Transactions on Geoscience and Remote Sensing, vol. 54, no. 11, pp. 6497–6504, 2016.
  * [17] V. Laparra, G. Camps-Valls, and J. Malo,  “Iterative Gaussianization: From ICA to Random Rotations,”  IEEE Transactions on Neural Networks, vol. 22, pp. 537–549, 2011\.
  * [18] J. Emmanuel Johnson, Valero Laparra, Maria Piles, and Gustau Camps-Valls,  “Gaussianizing the Earth: Multidimensional information measures for earth data analysis,”  IEEE Geoscience and Remote Sensing Magazine, 2021,  https://arxiv.org/abs/2010.06476.
  * [19] D. Lu Corresponding author, P. Mausel, E. Brondízio, and E. Moran,  “Change detection techniques,”  International Journal of Remote Sensing, vol. 25, no. 12, pp. 2365–2401, 2004.
  * [20] G. J. J. Van den Burg and C. K. I. Williams,  “An evaluation of change point detection algorithms,”  arXiv preprint arXiv:2003.06222, 2020.
  * [21] D. Lu, P. Mausel, E. Brondízio, and E. Moran,  “Change detection techniques,”  International Journal of Remote Sensing, vol. 25, no. 12, pp. 2365–2401, 2004.
  * [22] C. Wu, B. Du, and L. Zhang,  “A subspace-based change detection method for hyperspectral images,”  IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, vol. 6, no. 2, pp. 815–830, 2013.
  * [23] J. H. Friedman,  “Exploratory Projection Pursuit,”  Journal of the American Statistical Association, vol. 82, pp. 249–266, 1987.
  * [24] S. S. Chen and R. A. Gopinath,  “Gaussianization,”  in NIPS, 2000.
  * [25] V. Laparra, G. Camps-Valls, and J. Malo,  “Iterative gaussianization: From ica to random rotations,”  IEEE Transactions on Neural Networks, vol. 22, no. 4, pp. 537–549, 2011.
  * [26] J.E. Johnson, V. Laparra, G. Camps-Valls, R. Santos-Rodríguez, and J. Malo,  “Information theory in density destructors,”  CoRR, vol. abs/2012.01012, 2020.
  * [27] G. Camps-Valls and L. Bruzzone,  Kernel methods for Remote Sensing Data Analysis,  Wiley & Sons, UK, Dec 2009.
  * [28] J.L. Rojo-Álvarez, M. Martínez-Ramón, J. Muñoz-Marí, and G. Camps-Valls,  Digital Signal Processing with Kernel Methods,  Wiley & Sons, UK, Apr 2018.
  * [29] D.M.J. Tax and R.P.W. Duin,  “Support vector domain description,”  Pattern Recogn. Lett., vol. 20, nov 1999.
  * [30] Q. Guo, R. Pu, and J. Cheng,  “Anomaly detection from hyperspectral remote sensing imagery,”  Geosciences, vol. 6, no. 4, 2016.
  * [31] X. Kang, X. Zhang, S. Li, K. Li, J. Li, and J. A. Benediktsson,  “Hyperspectral anomaly detection with attribute and edge-preserving filters,”  IEEE Transactions on Geoscience and Remote Sensing, vol. 55, no. 10, pp. 5600–5611, 2017.
  * [32] J. A. Padrón-Hidalgo, V. Laparra, N. Longbotham, and G. Camps-Valls,  “Kernel anomalous change detection for remote sensing imagery,”  IEEE Transactions on Geoscience and Remote Sensing, vol. 57, no. 10, pp. 7743–7755, Oct 2019.
  * [33] V. Laparra, J. E. Johnson, G. Camps-Valls, R. Santos-Rodriguez, and J. Malo,  “Information theory measures via multidimensional gaussianization,” 2020,  Available at arxiv:2010.03807.

José A. Padrón Hidalgo received the B.Sc degree in Telecommunications and
Electronics from the University of Pinar del Río, Cuba, in 2015. He received
the Ph.D. degree in electronics engineering from the Universitat de València,
Valencia, in 2021.

His current research interest includes developing algorithms in order to
detect anomaly and anomalous changes for Remote Sensing imagery with the Image
and Signal Processing Group with the Universitat de València.

Valero Laparra was born in Valencia, Spain, in 1983. He received the B.Sc
degree in telecommunications engineering and the B.Sc. degree in the
electronics engineering from the Universitat de València, Valencia, in 2005
and 2007, respectively, the B.Sc. degree in mathematics from the Universidad
Nacional de Educación a Distancia, Madrid, Spain, in 2010, and the Ph.D.
degree in computer science and mathematics from the Universitat de València,
in 2011. He is currently an Assistant Professor with the Escuela Técnica
Superior de Ingenería, Universitat de València, where he is also a Researcher
with the Image Processing Laboratory

Gustau Camps-Valls (IEEE Fellow’18, ELLIS Fellow, IEEE Distinguished lecturer,
PhD in Physics) is a Full professor in Electrical Engineering and head of the
Image and Signal Processing (ISP) group, http://isp.uv.es, at the Universitat
de València. He is interested in the development of machine learning
algorithms for geosciences and remote sensing data analysis. He is an author
of around 250 journal papers, more than 300 conference papers, 25
international book chapters, and editor of 6 books on kernel methods and deep
learning. He holds a Hirsch’s index h=75 (Google Scholar), entered the ISI
list of Highly Cited Researchers in 2011, and Thomson Reuters ScienceWatch
identified one of his papers on kernel-based analysis of hyperspectral images
as a Fast Moving Front research. He received two European Research Council
(ERC) grants: an ERC Consolidator grant on ”Statistical learning for Earth
observation data analysis” (2015) and an ERC Synergy grant on ”Understanding
and Modelling the Earth system with machine learning” (2019). In 2016 he was
included in the prestigious IEEE Distinguished Lecturer program of the GRSS.


# Weakly-supervised Automated Audio Captioning via text only training

###### Abstract

In recent years, datasets of paired audio and captions have enabled remarkable
success in automatically generating descriptions for audio clips, namely
Automated Audio Captioning (AAC). However, it is labor-intensive and time-
consuming to collect a sufficient number of paired audio and captions.
Motivated by the recent advances in Contrastive Language-Audio Pretraining
(CLAP), we propose a weakly-supervised approach to train an AAC model assuming
only text data and a pre-trained CLAP model, alleviating the need for paired
target data. Our approach leverages the similarity between audio and text
embeddings in CLAP. During training, we learn to reconstruct the text from the
CLAP text embedding, and during inference, we decode using the audio
embeddings. To mitigate the modality gap between the audio and text embeddings
we employ strategies to bridge the gap during training and inference stages.
We evaluate our proposed method on Clotho and AudioCaps datasets demonstrating
its ability to achieve a relative performance of up to $83\%$ compared to
fully supervised approaches trained with paired target data. 111This work was
conducted in the framework of the PREMIERE project (No. 101061303) that is
funded by the European Union. Our code is available at:
https://github.com/zelaki/wsac

Index Terms—  Automated audio captioning, multi-modal learning, contrastive
learning.

##  1 Introduction

Audio-Language tasks have recently gained the attention of the audio community
with the introduction of Automated Audio Captioning and Language-Based Audio
Retrieval in the DCASE Challenge and the release of publicly available Audio-
Language datasets such as Clotho [1] and AudioCaps [2]. The intrinsic
relationship between Audio and Language presents an opportunity for the
development of models that can effectively establish a shared semantic space
for the two modalities. Such an approach has recently achieved great success
with models like COALA [3], AudioClip [4], and CLAP [5, 6, 7]. These models
use parallel audio-text data to train a joint representation, where the
embeddings of audio-text pairs are similar. Such models achieve high accuracy
in a zero-shot setting in a variety of tasks including Sound Event
Classification, Music tasks, and Speech-related tasks [5].

Automated Audio Captioning (AAC) is a multimodal task that aims to generate
textual descriptions for a given audio clip. In order to generate meaningful
descriptions, a method needs to capture the sound events present in an audio
clip and generate a description in natural language. Training audio captioning
models requires large datasets of audio-caption pairs, and these are
challenging to collect. While great effort has been done, the data scarcity
issue of audio captioning still withholds. The common datasets in AAC,
AudioCaps and Clotho, contain together 50k captions for training, whereas 400k
captions are provided in COCO caption [8] for image captioning. Kim et al. [9]
observe that due to the limited data, prior arts design decoders with shallow
layers that fail to learn generalized language expressivity and are fitted to
the small-scaled target dataset. Due to this issue, their performance
radically decreases when tested on out-of-domain data. Motivated by these
limitations we present an approach to AAC that only requires a pre-trained
CLAP model and unpaired captions from a target domain. This alleviates the
need for paired audio-text data, and also allows for simple and efficient
domain adaptation.

Our approach is inspired by recent advances in zero-shot image captioning [10,
11], that leverage the aligned multi-modal latent space provided by CLIP [12]
obviating the need for image data during training and by the recent success of
Contrastive Language-Audio models such as CLAP [5] in many downstream tasks.
We train a lightweight decoder model to reconstruct texts from their
respective CLAP embeddings, and at inference use this decoder to decode the
audio embeddings. Our findings align with prior studies in image captioning
suggesting that such an approach is suboptimal due to the presence of a
phenomenon known as modality gap [13].

The modality gap suggests that embeddings from different data modalities are
located in two completely separate regions of the embedding space of multi-
modal contrastive models [13]. To mitigate this issue we employ strategies
that have been shown to effectively condense the gap in CLIP embeddings [10,
11] and show that they can be effectively utilized for CLAP models. These
strategies can be divided into two categories, strategies that condense the
gap during training and during inference.

Experiments on Clotho and AudioCaps datasets show that our weakly-supervised
approach can achieve comparable performance to prior fully supervised arts,
without requiring any target audio data during training. Our contributions can
be summarized as follows: (1) We propose WSAC: Weakly-Supervised Audio
Captioning an AAC approach that requires no auditory in-domain data for
training, (2) we demonstrate that the modality gap phenomenon is present in
CLAP models, and (3) employ methods that effectively mitigate it.

Figure 1: Overview of our proposed approach. Left: An illustration of the CLAP
training paradigm. The encoders are trained to map semantically similar audio-
caption pairs to similar embeddings in a joint representation space. Middle:
Our proposed weakly supervised training. A frozen CLAP text encoder embeds a
caption and a decoder learns to reconstruct the caption from its embedding.
Right: At inference, we decode the audio embedding extracted from a frozen
CLAP audio encoder, using the trained decoder.

##  2 Text-only training

Our goal is to learn a model that produces a caption for a given audio clip.
Unlike fully supervised approaches, during training we only assume that we
have access to a set of target domain captions $\mathcal{C}$. We further
assume a pre-trained CLAP model with an audio encoder $\mathcal{A}_{clap}$ and
a text encoder $\mathcal{T}_{clap}$ trained to project semantically similar
audio-text pairs into similar embeddings in a shared embedding space as
presented in Fig. 1 (Left). Given an audio clip $x_{a}$ and text $x_{t}$ let
$\mathbf{z_{a}}=\mathcal{A}_{clap}(x_{a})\in\mathbb{R}^{d}$ and
$\mathbf{z_{t}}=\mathcal{T}_{clap}(x_{t})\in\mathbb{R}^{d}$ be their
embeddings.

First we extract text embeddings $\mathbf{z}_{t}$ for all
$x_{t}\in\mathcal{C}$, keeping $\mathcal{T}_{clap}$ frozen. During training,
our goal is to learn a network that inverts the CLAP text encoder
$\mathcal{T}_{clap}$. We use a textual decoder $D$ consisting of a mapping
network $f$ and an auto-regressive language model, to reconstruct the original
text $x_{t}$ from the CLAP text embedding $\mathbf{z_{t}}$. Following recent
work [9], we train our decoder using the prefix language modeling paradigm.
Specifically, after passing the text embedding through the mapping network $f$
we regard $\mathbf{p}=f(\mathbf{z_{t}})$ as a prefix to the caption. Given a
text $t=\\{w_{1},w_{2},...,w_{T}\\}$, our objective is to minimize the
autoregressive cross-entropy loss:

$\mathcal{L}=-\sum_{i=1}^{T}\log D(w_{i}|w_{<i},\mathbf{p})$ (1)

Since the CLAP text embedding is optimized to be similar to the CLAP audio
embedding, we can directly infer the text decoder using the audio embeddings
$\mathbf{z}_{a}$ without any pairwise training on the target dataset. The
training and inference stages are presented in Fig. 1 (middle) and (right)
respectively.

##  3 Stradegies to bridge the modality gap

Directly employing the audio embeddings to infer $D$ is not optimal due to the
presence of the modality gap. Fig. 2 is a visualization of generated
embeddings from the pre-trained CLAP model from the Clotho training set.
Paired inputs are fed into the pre-trained model and the embeddings are
visualized in 2D using T-SNE [14]. This visualization clearly demonstrates the
presence of the modality gap phenomenon, as a noticeable gap separates the
paired audio and text embeddings. To address this issue, we utilize strategies
that have demonstrated success in bridging the modality gap in CLIP embedding
space [10, 11, 13]. We show that these strategies can be adopted for CLAP and
show their effectiveness in mitigating the modality gap. These approaches can
be divided into two categories: Bridging the gap either during the training
phase or during the inference phase.

###  3.1 Training strategies

Attempting to reduce the modality gap during training we adopt the following
strategies: (a) Noise injection [10], and Embedding Shift [13]. These
strategies aim to narrow the disparity between the modality used to train the
decoder, which is text, and the target modality, which is audio.

####  3.1.1 Noise injection

In [10], the authors show that injecting the text embedding with Gaussian
noise during training has the effect of creating a region in the embedding
space that will map to the same caption. This method assumes that the
corresponding audio embedding is more likely to be inside this region.
Following [10], we add zero-mean Gaussian noise of standard deviation $\sigma$
to the text embedding before feeding it to the decoder. We set $\sigma$ to the
mean $L_{inf}$ norm of embedding differences between five captions that
correspond to the same audio. Since we assume no access to target audio data
we estimate $\sigma$ using 50 audio-caption pairs from the WavCaps dataset
[7]. Thus the prefix in Eq. 1 becomes
$\mathbf{p}=f(\mathbf{z_{t}}+\mathbf{n})$, where $\mathbf{n}\in\mathbb{R}^{d}$
is a random standard Gaussian noise with standard deviation $\sigma$.

Figure 2: Visualization of audio and text embedding pairs randomly sampled
from the Clotho training set. The modality gap phenomenon is present as the
audio and text modalities are embedded in two completely separate regions.

####  3.1.2 Embedding shift

Building upon the findings of [13], who investigated the impact of shifting
embeddings in various multi-modal contrastive learning models on downstream
tasks, we propose a method to align the text embeddings with the audio
embeddings during training. First, we define the modality gap following [13],
as the difference between the center of audio embeddings and text embeddings:

$\mathbf{\Delta_{gap}}=\frac{1}{n}\sum_{i=1}^{n}\mathbf{z_{a_{i}}}-\frac{1}{n}\sum_{i=1}^{n}\mathbf{z_{t_{i}}}$
(2)

Then, we shift every text embedding toward closing the modality gap, and thus
the prefix in Eq. 1 becomes
$\mathbf{p}=f(\mathbf{z_{t}}+\mathbf{\Delta_{gap}})$.

###  3.2 Inference strategies

At inference, we adopt two training-free strategies proposed in [11], and map
an audio embedding extracted from the CLAP audio encoder $\mathcal{A}_{clap}$
into the text embedding space. For both strategies, we will assume a decoder
$D$ trained on some target data as described in Section 2 and a set of text
embeddings obtained from the target training set that we will refer to as
Memory,
$\mathcal{M}=\\{\mathbf{z_{t}^{1}},\mathbf{z_{t}^{2}},...\mathbf{z_{t}^{N}}\\}$,
where $N$ is the size of the training set.

####  3.2.1 Nearest-neighbor decoding

A straightforward strategy that can be adopted at inference time to mitigate
the modality gap is to use the nearest text embedding as the prefix, instead
of the audio embedding. We calculate the cosine similarity between the audio
embedding $\mathbf{z_{a}}$ and the text embeddings in $\mathcal{M}$ and decode
with the most similar:

$\mathbf{p}=\mathbf{z_{i}}\;\;|\;\;i=\underset{\mathbf{z_{t}}\in\mathcal{M}}{argmax}\;sim(\mathbf{z_{a}},\mathbf{z_{t}})$
(3)

Where
$sim(\mathbf{x},\mathbf{y})=\frac{\mathbf{x}\cdot\mathbf{y}}{\|\mathbf{x}\|\cdot\|\mathbf{y}\|}$.
Since the decoder is trained to reconstruct the original text conditioned on
the text embedding, nearest-neighbor decoding can be successful if a
sufficiently similar text embedding is present in $\mathcal{M}$.

####  3.2.2 Projection-based decoding

A better approach is to project the audio embedding into the text embedding
space. This involves obtaining the representation of the audio embedding, by
combining the embeddings in $\mathcal{M}$ through a weighted combination.

$\mathbf{p}=\sum_{i=1}^{|\mathcal{M}|}w_{i}*\mathbf{z_{t_{i}}}$ (4)

The weights $w_{i}$ for these text embeddings are determined by calculating
the cosine similarity between the audio embedding $\mathbf{z_{a}}$ and each
embedding in $\mathcal{M}$. Following [11] the similarity is then scaled by a
temperature parameter $\tau$ and normalized using a softmax function:

$w_{i}=\frac{\exp(sim(\mathbf{z_{a}},\mathbf{z_{t_{i}}})/\tau)}{\sum_{j=1}^{|\mathcal{M}|}\exp(sim(\mathbf{z_{a}},\mathbf{z_{t_{j}}})/\tau)}$
(5)

##  4 Experimens

###  4.1 Data

We conduct experiments using two benchmarks, AudioCaps and Clotho. AudioCaps
contains $\leavevmode\nobreak\ 50$k, 10-second audio clips sourced from
Audioset [15]. Each audio is annotated with one caption in the training set
and five captions in the evaluation set. Clotho consists of 4981 audio samples
of 15 to 30 seconds duration. Each audio is annotated with five captions. We
follow the standard recipes of training, validation, and test splits on each
dataset for our experiments. To adhere to a weakly-supervised setting we
assume no access to audio data in the training and validation sets.

###  4.2 Experimental setup

To extract audio and text embeddings we employ a frozen CLAP
model222https://github.com/XinhaoMei/WavCaps/tree/master trained on WavCaps
[7]. The audio encoder is a CNN14 from Pre-trained Audio Neural Networks
(PANNs) [16], and the text encoder is a BERT-based model [17]. We choose this
model as the embedding extractor because AudioCaps and Clotho datasets were
not included in its training set. This choice is made under the assumption
that target audio data are unavailable for training purposes. The decoder $D$
consists of a mapping network $f$ which is a 2-layered MLP, and the language
model which is a 4-layer Transformer [18] with 4 attention heads. The size of
the hidden state is 768. The decoder $D$ is trained from scratch on the target
captions. The noise variance for Noise Injection training is set to
$\sigma^{2}=0.013.$ We train the proposed model for 30 epochs using Adam
optimizer [19] and a batch size of 64. The learning rate is linearly increased
to $2\times 10^{-5}$ in the first five epochs using warm-up, which is then
multiplied by 0.2 every 10 epochs. We use greedy search for decoding.

Table 1: Results on AudioCaps and Clotho. We report results for fully
supervised methods trained on audio-caption pairs, and our proposed methods
trained only on captions. WSAC is our baseline approach presented in Section
2. We refer to Noise injection as NI, Embedding shift as ES, Nearest-
neighborhood decoding as NND and, Projection-based decoding as PD. We
highlight the best results for fully and weakly supervised methods with
underline and bold respectively.

Dataset | Supervision | Method | BLEU1 | BLEU2 | BLEU3 | BLEU4 | METEOR | ROUGEL | CIDEr | SPICE | SPIDEr
---|---|---|---|---|---|---|---|---|---|---|---
Audiocaps |  Audio-Caption Pairs |  Mei et al. [20] | $0.647$ | $0.488$ | $0.356$ | $0.252$ | $0.222$ | $0.468$ | $0.679$ | $0.160$ | $0.420$
Kim et al. [9] | $0.713$ | $0.552$ | $0.421$ | $0.309$ | $0.240$ | $0.503$ | $0.733$ | $0.177$ | $0.455$
Gontier et al. [21] | $0.699$ | $0.523$ | $0.380$ | $0.266$ | $0.241$ | $0.493$ | $0.753$ | $0.176$ | $0.465$
Mei et al. [7] | $\underline{0.707}$ | - | - | $\underline{0.283}$ | $\underline{0.250}$ | $\underline{0.507}$ | $\underline{0.787}$ | $\underline{0.182}$ | $\underline{0.485}$
Captions Only | WSAC | $0.574$ | $0.398$ | $0.267$ | $0.167$ | $0.222$ | $0.426$ | $0.493$ | $0.155$ | $0.324$
WSAC+NI | $0.662$ | $0.477$ | $0.328$ | $0.216$ | $0.223$ | $0.46$ | $0.579$ | $0.155$ | $0.367$
WSAC+ES | $0.653$ | $0.458$ | $0.300$ | $0.185$ | $0.214$ | $0.451$ | $0.540$ | $0.154$ | $0.347$
WSAC+NND | $0.643$ | $0.457$ | $0.312$ | $0.198$ | $0.231$ | $0.454$ | $0.548$ | $0.166$ | $0.357$
WSAC+PD | $\mathbf{0.698}$ | $\mathbf{0.511}$ | $\mathbf{0.357}$ | $\mathbf{0.232}$ | $\mathbf{0.241}$ | $\mathbf{0.479}$ | $\mathbf{0.633}$ | $\mathbf{0.173}$ | $\mathbf{0.403}$
Clotho |  Audio-Caption Pairs |  Xu et al. [22] | $0.556$ | $0.363$ | $0.242$ | $0.159$ | $0.169$ | $0.368$ | $0.377$ | $0.115$ | $0.246$
Koh et al. [23] | $0.551$ | $0.369$ | $0.252$ | $0.168$ | $0.165$ | $0.373$ | $0.380$ | $0.111$ | $0.246$
Kim et al. [9] | $0.560$ | $0.376$ | $0.253$ | $0.160$ | $0.170$ | $0.378$ | $0.392$ | $0.118$ | $0.255$
Mei et al. [7] | $\underline{0.601}$ | - | - | $\underline{0.180}$ | $\underline{0.185}$ | $\underline{0.400}$ | $\underline{0.488}$ | $\underline{0.133}$ | $\underline{0.310}$
Captions Only | WSAC | $0.462$ | $0.282$ | $0.173$ | $0.102$ | $0.166$ | $0.343$ | $0.265$ | $0.113$ | $0.189$
WSAC+NI | $0.525$ | $0.314$ | $0.193$ | $0.118$ | $0.164$ | $0.352$ | $0.315$ | $0.113$ | $0.214$
WSAC+ES | $\mathbf{0.546}$ | $\mathbf{0.332}$ | $\mathbf{0.203}$ | $\mathbf{0.120}$ | $0.159$ | $0.353$ | $0.301$ | $0.109$ | $0.205$
WSAC+NND | $0.498$ | $0.294$ | $0.179$ | $0.106$ | $0.166$ | $0.338$ | $0.332$ | $0.113$ | $0.222$
WSAC+PD | $0.532$ | $0.324$ | $0.200$ | $0.118$ | $\mathbf{0.174}$ | $\mathbf{0.354}$ | $\mathbf{0.371}$ | $\mathbf{0.123}$ | $\mathbf{0.247}$

###  4.3 Compared methods and evaluation metrics

Since no previous work has addressed AAC in similar supervision settings we
compare our methods against fully supervised approaches trained on paired
data. Koh et al. [23] use a latent space similarity objective and train a
model with a PANNs encoder and a transformer decoder. Xu et al. [22] design a
GRU for the decoder. Mei et al. [20] propose a full transformer encoder-
decoder architecture. Gontier et al. [21] utilize a pre-trained language model
based on BART [21], and finetune it for AAC using guidance from Audioset tags.
Kim et al. [9] propose prefix tuning for AAC learning a prefix to guide the
caption generation of a frozen GPT-2 [24]. Mei et al. [7] utilize a CLAP audio
encoder pre-trained on WavCaps and a BART decoder achieving state-of-the-art
results in both Clotho and AudioCaps. All the methods in this work are
evaluated by the metrics widely used in the captioning tasks, including BLEU
[25], METEOR [26], ROUGE-L [27], CIDEr [28], SPICE [29], and SPIDEr [30].

###  4.4 Results and Discussion

In this section, we present the results of our proposed methods on the
performance metrics and compare them with fully supervised arts. Additionally,
we illustrate the effectiveness of each strategy in reducing the modality gap.
As shown in Table 1 our methods demonstrate comparable performance to prior
state-of-the-art models despite never encountering in-domain audio data during
training. We present the results of our baseline approach described in Section
2 and the results of the baseline approach in conjunction with the strategies
presented in Section 3. It is evident that all the strategies boost the
performance of our baseline approach in both evaluation sets. Interestingly
the inference strategies outperform the training strategies in most cases. We
hypothesize that this is because they utilize the Memory $\mathcal{M}$ which
consists of in-domain text embeddings in order to bridge the modality gap. Our
best-performing method, namely Projection-based decoding achieves 80% and 83%
of the SPIDEr performance of the current fully supervised state-of-the model
in Clotho and AudioCaps evaluation sets respectively. Additionally Projection-
based decoding matches the performance of the of fully-supervised approaches
proposed by Kim et al. [9]. Koh et al. [23] and Xu et al. [22] in the Clotho
evaluation set.

Visualization of embeddings: To further examine the effectiveness of the
proposed strategies we illustrate the embeddings in 2D space using t-SNE in
Fig. 3. In Fig. 3a and 3b we randomly sample audio and text embeddings from
the Clotho training set after applying Noise Injection and Embedding Shift to
the text embeddings. Fig. 3c and 3d illustrate randomly selected text
embeddings from the Clotho evaluation set, alongside the embeddings utilized
for decoding, namely the nearest neighbors and the projections, rather than
the paired audio embeddings. It is evident that all strategies are effective
in condensing the modality gap showcased in Fig. 2, where the audio and text
modalities are embedded at arm’s length in their shared representation space.

(a) Noise Injection

(b) Embedding Shift

(c) Nearest-neighbor decoding

(d) Projection-based decoding

Figure 3: TSN-E visualizations of the embedding space after applying the
strategies presented in Section 3.

##  5 Conclusion and Feature Work

In this work, we propose a weakly-supervised approach for Automated Audio
Captioning that requires a pre-trained CLAP model and only additional text
data to train on a target domain. Our method alleviates the necessity of
paired data in a target domain, which are hard to collect. We demonstrate that
by leveraging the shared embedding space of CLAP we can learn to reconstruct
the text from the CLAP text embedding and during inference decode using the
audio embeddings. We show that such an approach is suboptimal due to the
presence of a modality gap and adopt strategies that effectively mitigate it.
Our best-performing method achieves comparable results to prior arts trained
in a fully supervised manner. For future work, we plan to study the
effectiveness of our proposed approach on other tasks, such as Music
Captioning and Audio Question Answering. We further aim to train a mapping
network to learn the gap between the two modalities in a supervised manner.

## References

  * [1] K. Drossos, S. Lipping, and T. Virtanen, “Clotho: an audio captioning dataset,” _ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pp. 736–740, 2019.
  * [2] C. D. Kim, B. Kim, H. Lee, and G. Kim, “AudioCaps: Generating captions for audios in the wild,” in _In Proc. NAACL_. Minneapolis, Minnesota: Association for Computational Linguistics, June 2019, pp. 119–132. [Online]. Available: https://aclanthology.org/N19-1011
  * [3] X. Favory, K. Drossos, T. Virtanen, and X. Serra, “Coala: Co-aligned autoencoders for learning semantically enriched audio representations,” _arXiv preprint arXiv:2006.08386_ , 2020.
  * [4] A. Guzhov, F. Raue, J. Hees, and A. R. Dengel, “Audioclip: Extending clip to image, text and audio,” _ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pp. 976–980, 2021.
  * [5] B. Elizalde, S. Deshmukh, M. A. Ismail, and H. Wang, “Clap learning audio concepts from natural language supervision,” in _ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 2023, pp. 1–5.
  * [6] Y. Wu, K. Chen, T. Zhang, Y. Hui, T. Berg-Kirkpatrick, and S. Dubnov, “Large-scale contrastive language-audio pretraining with feature fusion and keyword-to-caption augmentation,” _In Proc. ICASSP_ , vol. abs/2211.06687, 2022.
  * [7] X. Mei, C. Meng, H. Liu, Q. Kong, T. Ko, C. Zhao, M. . Plumbley, Y. Zou, and W. Wang, “Wavcaps: A chatgpt-assisted weakly-labelled audio captioning dataset for audio-language multimodal research,” _ArXiv_ , vol. abs/2303.17395, 2023.
  * [8] X. Chen, H. Fang, T. Lin, R. Vedantam, S. Gupta, P. Dollár, and C. L. Zitnick, “Microsoft COCO captions: Data collection and evaluation server,” _CoRR_ , vol. abs/1504.00325, 2015. [Online]. Available: http://arxiv.org/abs/1504.00325
  * [9] M.-K. Kim, K. Sung‐Bin, and T.-H. Oh, “Prefix tuning for automated audio captioning,” _In Proc. ICASSP 2023_ , vol. abs/2303.17489, 2023.
  * [10] D. Nukrai, R. Mokady, and A. Globerson, “Text-only training for image captioning using noise-injected clip,” in _Conference on Empirical Methods in Natural Language Processing_ , 2022.
  * [11] W. Li, L. Zhu, L. Wen, and Y. Yang, “Decap: Decoding clip latents for zero-shot captioning via text-only training,” _In Proc. ICLR_ , vol. abs/2303.03032, 2023.
  * [12] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever, “Learning transferable visual models from natural language supervision,” in _In Proc ICML_ , 2021.
  * [13] W. Liang, Y. Zhang, Y. Kwon, S. Yeung, and J. Y. Zou, “Mind the gap: Understanding the modality gap in multi-modal contrastive representation learning,” _ArXiv_ , vol. abs/2203.02053, 2022.
  * [14] L. van der Maaten and G. Hinton, “Visualizing data using t-SNE,” _Journal of Machine Learning Research_ , vol. 9, pp. 2579–2605, 2008. [Online]. Available: http://www.jmlr.org/papers/v9/vandermaaten08a.html
  * [15] J. F. Gemmeke, D. P. W. Ellis, D. Freedman, A. Jansen, W. Lawrence, R. C. Moore, M. Plakal, and M. Ritter, “Audio set: An ontology and human-labeled dataset for audio events,” in _2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 2017, pp. 776–780.
  * [16] Q. Kong, Y. Cao, T. Iqbal, Y. Wang, W. Wang, and M. D. Plumbley, “Panns: Large-scale pretrained audio neural networks for audio pattern recognition,” _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , vol. 28, pp. 2880–2894, 2019.
  * [17] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “BERT: Pre-training of deep bidirectional transformers for language understanding,” in _In Proc. ACL, Volume 1 (Long and Short Papers)_ , June 2019.
  * [18] A. Vaswani, N. M. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, “Attention is all you need,” in _NIPS_ , 2017\.
  * [19] I. Loshchilov and F. Hutter, “Fixing weight decay regularization in adam,” _ArXiv_ , vol. abs/1711.05101, 2017.
  * [20] X. Mei, X. Liu, Q. Huang, M. D. Plumbley, and W. Wang, “Audio captioning transformer,” in _DCASE Workshop_ , 2021.
  * [21] F. Gontier, R. Serizel, and C. Cerisara, “Automated audio captioning by fine-tuning bart with audioset tags,” in _DCASE Workshop_ , 2021.
  * [22] X. Xu, H. Dinkel, M. Wu, Z. Xie, and K. Yu, “Investigating local and global information for automated audio captioning with transfer learning,” _In Proc. ICASSP_ , pp. 905–909, 2021.
  * [23] A. Koh, X. Fuzhao, and C. E. Siong, “Automated audio captioning using transfer learning and reconstruction latent space similarity regularization,” in _In Proc. ICASSP_. IEEE, 2022, pp. 7722–7726.
  * [24] A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever, “Language models are unsupervised multitask learners,” 2019.
  * [25] K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu, “Bleu: a method for automatic evaluation of machine translation,” in _ACL_. Philadelphia, Pennsylvania, USA: Association for Computational Linguistics, July 2002, pp. 311–318. [Online]. Available: https://aclanthology.org/P02-1040
  * [26] S. Banerjee and A. Lavie, “METEOR: An automatic metric for MT evaluation with improved correlation with human judgments,” in _ACL_. Ann Arbor, Michigan: Association for Computational Linguistics, June 2005, pp. 65–72. [Online]. Available: https://aclanthology.org/W05-0909
  * [27] C.-Y. Lin, “ROUGE: A package for automatic evaluation of summaries,” in _Text Summarization Branches Out_. Barcelona, Spain: Association for Computational Linguistics, July 2004, pp. 74–81. [Online]. Available: https://aclanthology.org/W04-1013
  * [28] R. Vedantam, C. L. Zitnick, and D. Parikh, “Cider: Consensus-based image description evaluation,” in _In Proc. CVPR_ , 2015, pp. 4566–4575.
  * [29] P. Anderson, B. Fernando, M. Johnson, and S. Gould, “Spice: Semantic propositional image caption evaluation,” in _In Proc ECCV_. Springer, 2016, pp. 382–398.
  * [30] S. Liu, Z. Zhu, N. Ye, S. Guadarrama, and K. P. Murphy, “Optimization of image description metrics using policy gradient methods,” _ArXiv_ , vol. abs/1612.00370, 2016.