Liwen Tan1,2, Yin Cao3, Yi Zhou1,2,∗††thanks: *Corresponding author

# EDTC: ENHANCE DEPTH OF TEXT COMPREHENSION IN AUTOMATED AUDIO CAPTIONING

###### Abstract

Modality discrepancies have perpetually posed significant challenges within
the realm of Automated Audio Captioning (AAC) and across all multi-modal
domains. Facilitating models in comprehending text information plays a pivotal
role in establishing a seamless connection between the two modalities of text
and audio. While recent research has focused on closing the gap between these
two modalities through contrastive learning, it is challenging to bridge the
difference between both modalities using only simple contrastive loss. This
paper introduces Enhance Depth of Text Comprehension (EDTC), which enhances
the model's understanding of text information from three different
perspectives. First, we propose a novel fusion module, FUSER, which aims to
extract shared semantic information from different audio features through
feature fusion. We then introduced TRANSLATOR, a novel alignment module
designed to align audio features and text features along the tensor level.
Finally, the weights are updated by adding momentum to the twin structure so
that the model can learn information about both modalities at the same time.
The resulting method achieves state-of-the-art performance on AudioCaps
datasets and demonstrates results comparable to the state-of-the-art on Clotho
datasets.

Index Terms: Audio captioning, Feature fusion, Feature alignment, Contrastive
learning

##  1 Introduction

Figure 1: Overview of the methodology presented in this paper. Audio encoder
uses PANNs[5], HTSAT[6], ConvNeXt[17] for extracting audio features.
INSTRUCTOR-base[18] is used as Text Encoder for extracting text features and
BART is used as Decoder for generating captions. All encoder weights are
frozen.

Automated Audio Captioning (AAC) is a multimodal generation task that
integrates audio and text to generate descriptive captions for audio[1]. The
applicability of AAC research is diverse, encompassing areas such as aiding
the hearing-impaired through audio-to-text conversion and building intelligent
content-oriented human-computer interactions[2]. AAC research has received a
dramatic increase in attention and made substantial progress in recent years,
thanks to the DCASE challenge and the release of datasets such as Clotho[3]
and AudioCaps[4].

The current mainstream AAC methods employ encoder-decoder system
architectures. Specifically, an audio encoder such as PANNs[5], HTSAT[6],
etc., is utilized to extract the acoustic features of log-mel energies or Mel-
Frequency Cepstral Coefficients (MFCCs), and then the text decoder generates
the textual output. Text decoders generally use sequence-to-sequence design.
In recent years, with the development of Transformers architecture, more and
more excellent decoders such as GPT2[7], BART[8], etc., have gradually become
the first choice for text decoders.

In previous studies, many researchers have explored AAC research through
various perspectives. Some researchers have improved system performance by
introducing additional feature embeddings (such as sound event detection[9] or
keyword information[10]). Other researchers directly optimize the metrics to
enhance the quality of generated subtitles by using reinforcement learning
algorithms[11]. In order to solve the problem of scarce training data, Mei et
al. proposed WavCaps[12] dataset, and Wu et al. proposed LAION-Audio-630K
dataset[15], which greatly solved this problem. Despite these approaches, a
fundamental challenge remains unresolved — the huge disparity between the
audio and text modalities.

With the popularity of Contrastive Language-Image Pretraining (CLIP)[13] in
image captioning, contrastive learning has gradually gained attention from
researchers. CLIP extracts the features of two modalities through two
independent encoders and then closes the gap between the two modalities by
using a contrastive loss. Elizalde et al. proposed Contrastive Language-Audio
Pretraining (CLAP) to apply CLIP to audio-text domain by replacing the image
encoder in CLIP with an audio encoder[14]. Wu et al. applied CLAP to AAC
research by large scale training[15]. Chang et al. used a similar architecture
but used InfoNCE loss to close the gap between the two modalities[16]. Despite
the fact that all the referenced studies yielded outstanding outcomes, they
uniformly employed a contrastive loss function to facilitate the learning of
text features by the audio encoder, derived from the text encoder, without
fully capitalizing on the information extracted by the text encoder. This
raises a critical question: Is it feasible for a mere loss function to
adequately bridge the substantial gap between the audio and text modalities?

Previous research often encountered a significant challenge: directly feeding
audio features into the text decoder in an attempt to directly interpret audio
features through the text decoder, a task that proves to be exceedingly
difficult. Consider an analogy where the audio information represents "meow"
and the text information signifies "bark"; even with the most intelligent cats
and dogs, is mutual comprehension truly achievable? To bridge the gap between
these two distinct modalities, it is essential to employ auxiliary techniques
that facilitate the convergence of the audio and text modalities within the
model.

In this paper, we propose to use three different approaches in order to
address the disparity between the audio and text modalities. The details of
each method and the reason why it was chosen are described in Section 3. Our
model, shown in Figure 1, is divided into audio and text branches. The audio
branch first extracts audio features by multiple different audio encoders with
frozen weights, and then a feature fusion module FUSER fuses the extracted
audio features to extract the shared semantic information. Then, the fused
features are fed into an alignment module TRANSLATOR for feature alignment in
the tensor level. Finally, the output features are fed into a text decoder to
get the generated captions. The text branch consists of a frozen text encoder
and a twinned TRANSLATOR module. First, the text encoder extracts text
features from the original captions, and then the extracted text features are
passed through a Sim TRANSLATOR module with the same structure as the audio
branch, the TRANSLATOR modules of the two branches share the weights and use
momentum to update the weights in such a way that the TRANSLATOR module of the
audio branch is also able to learn the text features directly. Finally, a
contrastive loss assist model bridge the difference between both audio and
text modalities. In general, the contributions of this paper are as follows:

  * •

We propose a novel feature fusion module, FUSER, which can extract shared
semantic information from the output features of multiple different audio
encoders and act as a plug-and-play feature extractor to extract higher
dimensional semantic information when input to multiple identical encoders.

  * •

We propose a new plug-and-play alignment module, TRANSLATOR, to further bridge
the difference between the two modalities by aligning audio features with text
features in the tensor level.

  * •

We propose to use a set of twin TRANSLATOR structures, combined with the
momentum update weight and the addition of auxiliary contrastive loss to
enable the model to directly learn the information of the two modalities
simultaneously.

##  2 Related work

This section will provide a brief overview of the research areas related to
this paper, including audio captioning, feature fusion, contrastive learning.

###  2.1 Audio captioning

The aim of audio captioning is to generate descriptive captions for audio.
Many researchers currently employ transfer learning methods to allow models to
learn more samples in order to enhance model performance. Reinforcement
learning is also used in this field for direct optimization of metrics,
However, it will cause the generated captions to be not fluent. Wu et al.
applied CLAP to AAC research by large scale training[15], and Chen et al.
proposed CLIP-AAC by a similar approach[19], where by extracting the features
of their respective modalities through two different modality encoders, and
then through clip loss (a kind of contrastive loss) to bridge the difference
between the two modalities. Chang and Wu et al. adopted a similar
architecture, but used BAETs + Conformer as the audio encoder and employed
InfoNCE loss instead of clip loss[16]. Deshmukh et al. attempted training the
model using only text information based on this architecture[20], which proves
that this architecture can indeed bridge the difference between both
modalities effectively. The method proposed in this paper is to learn
information about both modalities simultaneously through a set of twinned
structures, instead of bridging the difference between both modalities with
only a single contrastive loss. Furthermore, we propose to further aid the
model in understanding the disparity between modalities through feature fusion
and tensor level alignment.

###  2.2 Feature fusion

The aim of feature fusion is to integrate the information of different
features, which can be usually divided into the same modality feature fusion
and multi-modal feature fusion. Based on the model architecture, it can be
classified into the use of a unified architecture model and the use of a non-
unified architecture model. Using a unified architecture model for fusion[22,
23, 24], that is, different features are fused with the same architecture
model. This kind of method can effectively alleviate the problem of limited
data amount by using different kinds of data (such as audio, text, image,
etc.) for training. Which can effectively extract the common information
between features, but the unique information of each feature is often ignored.
The method of using no unified architecture model[25, 26, 27], that is, first
using a separate model for each feature to extract features, and then
performing some series of operations such as averaging or concatenation or
cross-attention on the extracted features to obtain the final features. Which
can effectively extract the unique information of various features, but
exhibits limited capability in capturing common information. Our proposed
method integrates both approaches. Firstly, audio features are extracted
through different feature encoders, because different encoders can extract
features from different perspectives for the same audio sample (encoders based
on CNN architecture pay more attention to local features, encoders based on
Transformer architecture are better at extracting global information).
Subsequently, these features are fed into a fusion module, where common and
unique information is separately extracted during the fusion process. The
amalgamation of these pieces of information generates higher-dimensional
semantic information, thus narrowing the gap between the two modalities of
audio and text.

###  2.3 Contrastive learning

Contrastive learning has long captivated researchers owing to its advantages
of self-supervised learning, and its goal is to extract useful information by
learning the differences and similarities between samples. In previous
studies, researchers obtained different views of the same image by applying
different data enhancement methods to the same image[28, 29]. Then compare the
different views to get useful information. In recent studies, a series of
studies on contrastive learning, mainly represented by CLIP[13], have
gradually become popular in multimodal tasks. There are also many researchers
in the field of AAC have done similar work[15, 19, 20], they basically use the
same architecture. The model is divided into two branches, each branch has an
encoder for extracting features, and then the contrastive loss is used to
bridge the difference between the features of the two modalities. While this
approach is straightforward, it presents some challenges. As mentioned
earlier, the large disparity between the two modalities is difficult to
accommodate with a single loss function. In addition, this will inevitably
introduce an additional text encoder to extract text features. If only the
text features and audio features are used to calculate the loss, this will not
make full use of text features and cause a waste of resources. In this study,
we add a set of twin modules to the audio branch and the text branch, and
update the weight by momentum, so that the modules of the two branches can
share the weight and learn the features of their respective branches, so as to
effectively use the text features. This also allows the audio encoder to learn
the text features directly, instead of just bridging the difference between
two features via the loss function.

##  3 Method

In this section, we will provide a comprehensive description of the
methodology proposed in this paper, including a detailed rationale for its
selection.

###  3.1 Encoders and decoder

There are many excellent encoders that have been applied to AAC in previous
studies[5, 6, 17]. These encoders are based on different architectures, which
gives them different perspectives when extracting features. For example,
PANNs[5] is based on CNN architecture, and its extracted features are more
local detailed features, while HTSAT[6] based on Transformer architecture will
focus more on global information due to the application of attention
mechanism. The proposed method aims to extract common semantic information by
integrating features from different views.

####  3.1.1 Audio encoders

We choose three widely used Pre-trained audio encoders, PANNs[5], HTSAT[6] and
ConvNeXt[17], to extract audio features under various views. PANNs[5] is based
on CNN architecture and consists of 14 convolutional layers. HTSAT[6] is an
audio encoder based on the Swin-Transformer architecture. ConvNeXt[17] is also
based on CNN architecture, but different from PANNs, it uses larger
convolution kernels and reference part of Transformer structure settings. All
three audio encoders are frozen with weights and do not participate in weight
updates.The audio feature tensors extracted by each audio encoder are resized
to the same shape of 32 × 768.

####  3.1.2 Text encoder

We adopt INSTRUCTOR-base[18] as the text encoder to extract text features from
captions. Which is a pre-trained T5 model based on instruction fine-tuning.
During training, the weights of INSTRUCTOR are frozen and use "Represent the
audio caption:" as the instruction to generate word-level text embeddings. The
tensor shape out of the final extracted text embedding is adjusted to the same
shape as the audio encoders.

####  3.1.3 Text decoder

We adopt BART[8] as the text decoder, which is a Transformer decoder with
bidirectional encoder and autoregressive decoder (BERT + GPT). Use the same
parameter settings as BART-base in Hugging Face, which have 6 layers of
encoder and 6 layers of decoder, and have a 5K vocabulary size. Unlike the
encoder part, BART takes part in the training and updates the weights.

Cross-entropy loss is the main loss function of the model, which is obtained
by calculating the output distribution of BART and the true captions. The
cross-entropy loss can be expressed as follows:

$L_{ce}=-\frac{1}{N}\sum_{n=1}^{N}\log{p\left(y_{n}\mid y_{1:n-1};x\right)}$

Where $x$ is the audio sample, $y$ is the audio caption, $y_{n}$ is the
$n^{th}$ word in the caption, and $N$ is the batch size.

###  3.2 FUSER

Figure 2: The statement about the reason for selecting FUSER, where the orange
dialog box represents the audio sample, and the blue dialog box represents the
text corresponding to each piece of audio.

This section describes the FUSER module structure in detail and why it was
chosen.

Figure 3: Detail diagram of the FUSER module. The module consists of an Input
layer for the pre-fusion and a Fusion layer for the L layers. The input of the
module is the audio features extracted for each encoder, and the output is the
final fusion features.

####  3.2.1 Why choose FUSER

To put it another way, the goal of AAC task is to make the text decoder
understand the audio features extracted by the audio encoder. If the
interaction between an audio encoder and a text decoder is compared to a
conversation between two people, as shown in Figure 2, where one person (the
audio encoder) speaks English and the other (the text decoder) speaks French,
it is obvious that no matter how the English speaker condense his expression,
it will be difficult for the other person to understand because these are two
different languages (modalities) anyway. To solve this problem, now let three
different people (three different audio encoders) express the same
information, and let three different people (the fusion structure) each
understand all the different expressions. Although a single person still
cannot understand each other, from different expressions and different
understandings, some common points can always be found. These common points
are the so-called semantic information, which does not belong to any single
modality. In this way, a bridge can be built between the two modalities to
help them understand each other. FUSER, proposed in this paper, aims to
extract semantic information from audio features by this method.

####  3.2.2 FUSER specific details

The specific structure diagram of FUSER is shown in Figure 3, which consists
of an Input Layer and multiple repeated Fusion layers. The inputs of the FUSER
are audio features extracted from three different audio encoders. First, these
features are fed into an Input Layer, which consists of three Encoder Fusion
blocks (EFB), a Channel Fusion Block (CFB) and a Feed Forward Network (FFN).
Each EFB will take one audio feature as the baseline, compute the cross-
attention and sum it with the other two features. The outputs of each EFB are
then concatenated together and use a convolutional layer with a kernel size of
7 for dimension reduction. Finally, the output of each EFB and the output of
the CFB are fed into a shared weight FFN to obtain the output. Through the
Input Layer, the features are fused in advance before deep fusion to improve
the fusion quality.

The structure of the Fusion Layer is similar to that of the Input Layer, but
the EFB module is replaced by Shared Feature Extract Block (SFEB) and Private
Feature Extract Block (PFEB). PFEB is used to extract the unique information
of each feature. This kind of information tends to be local features. We use a
3-layer convolution architecture to extract these local features, and add
Batch Normalization between each layer to reduce overfitting. SFEB is used to
extract the common information of each feature, which is usually global
information. We extract the global information by calculating the self-
attention of the channel dimension and the feature vector dimension, and the
weight is shared between each SFEB. In order to reduce memory consumption, the
linear layer is used to reduce the dimension from 768 to 256 before
calculating the self-attention of the feature vector dimension, and then the
dimension is re-increased to 768 by the linear layer after the calculation.
The features will be deeply fused through the L-layer Fusion Layer to finally
obtain the fused features. We experimentally find that L = 3 performs best.

It is worth noting that the input of FUSER can be replaced with the features
extracted by any of the three encoders. And when the model structure has only
one encoder, three same features can also be used as the input of FUSER. At
this time, the situation is similar to one person express and three people
understand, and they also have the ability to extract semantic information
from the same feature.

###  3.3 TRANSLATOR

This section describes the TRANSLATOR module structure in detail and why it
was chosen.

Figure 4: The statement about the reason for selecting TRANSLATOR, where
time_step is the time step, n_mel is the number of mel filters, seq_len is the
text length, d_model denotes the encoding dimension of each word.

####  3.3.1 Why choose TRANSLATOR

Generally, in the construction of the model, different modules can be spliced
together by just adjusting the input and output tensors to the same shape, but
this completely ignores the specific significance of each dimension of the
tensor. As shown in Figure 4, in AAC task, audio features are basically
obtained by extracting the corresponding log mel spectrum, and the shape of
audio feature tensor is (bs, time_step, n_mel), where bs stands for batch
size, time_step stands for time step, and n_mel stands for the number of mel
filters. The text features is represented by a tensor of shape (bs, seq_len,
d_model), where bs is the batch size, seq_len is the text length, and d_model
is the encoding dimension of each word. By comparison, we can see that only
the first dimension of the two tensors represents the same meaning, and the
last dimension should also represent a similar meaning through complex changes
of the model, but the second dimension is difficult to establish a
corresponding relationship. Let's imagine that each time step of the audio
features can really correspond to every word in the text?

It can be imagined that the correspondence between each time step and each
word is not a simple one-to-one relationship, but a many-to-many relationship.
For text information, each word contains not only its own information, but
also more context information, while for audio samples, each time step more
focuses on the current time node information. By making the features at each
time step contain more context information, we align the two features in the
tensor level to reduce the gap between the two modalities, which is the
specific role of the TRANSLATOR module.

Figure 5: Detail diagram of the TRANSLATOR module, which is composed of an
M-layer Encoder Layer and an N-layer Decoder Layer. The module input is split
into individual time steps, and then the output of each time step is
concatenated into a complete output after passing through the module.

####  3.3.2 TRANSLATOR specific details

As shown in Figure 5, TRANSLATOR is composed of M-layer encoder and N-layer
decoder, whose input is a tensor of shape (bs, time_step, d_model), and splits
it in the time_step dimension. Each split tensor has the shape (bs, d_model)
and is used as input for each time step.

The encoder consists of a Context Attention Block (CAB), a Global Attention
Block (GAB) and a standard Feed Forward Network (FFN). Inside CAB, the input
is first extracted by a Conv1D with a kernel size of 45 to extract the current
time node features. Through this step, the input which is originally a whole
can be subdivided into a single time step instead of simply splitting the
tensor. Using a large convolution kernel can reduce the influence of
convolution on the original information at each time step by expanding the
receptive field. After the convolutional layer, the cross-attention
information of the current time step and the hidden state of the previous time
step are calculated. Finally the output of the CAB module is obtained by layer
normalization through a residual structure. The feature of the current time
step through the CAB module will carry the information of all previous time
steps. After that, each time step feature passes through a GAB module, which
has the same internal structure as CAB, but computes cross-attention with the
original input that was not split. The value obtained through the calculation
is passed to the next time step as the hidden state of the current time step.
Through the GAB module, the feature at the current time step further have
global context information. Finally, the feature are linearly transformed by
an FFN to obtain the output of the encoder at the current time step.

The structure of decoder is similar to encoder, but a part of the structure
between GAB module and FFN is added to calculate the cross-attention of
encoder feature and decoder feature at the same time step, so that it can have
more abundant global information. The calculated value is used as the hidden
state of decoder at the current time step. In addition, the last hidden state
of the last layer of the encoder is used as the initial hidden state at time
step 0 of each layer of the decoder. Finally, the output of each time step of
decoder is spliced together as the final output of TRANSLATOR. We
experimentally find that M = 3, N = 2, and kernel_size = 45 perform best.

The output of the TRANSLATOR $y_{tran}$ can be expressed as follows:

${\Huge x_{0},\ x_{1},\cdots\ x_{T}=Split\left(X\right)}$
$y_{tran}=Concat_{n=0}^{T}\ f\left(x_{n}\ \odot\ h_{n-1}\right)$

Where $X$ represents the TRANSLATOR's input, $x_{n}$ represents the input at
$n^{th}$ time step, $h_{n-1}$ represents the hidden state at
$\left(n-1\right)^{th}$ time step, $T$ represents the total number of time
steps, $f\left(\cdot\right)$ represents the TRANSLATOR's calculation process,
$split\left(\cdot\right)$ represents the tensor split along the time
dimension, $Concat\left(\cdot\right)$ represents the tensor resplicing,
$\odot$ stands for computing cross-attention. In summary, TRANSLATOR aligns
the text information in the tensor level by performing an identity
transformation on the audio features. It can be added as a plug-and-play
alignment module to almost all models in the AAC field.

###  3.4 Twin structure and momentum update

This section describes the twin structure and momentum update in detail and
why it was chosen.

####  3.4.1 Why choose this structures

As a multi-modal task, how to make full use of text information has always
been one of the key points to improve the performance of AAC system. A most
direct approach is to use audio and text as the input of the model at the same
time. In this way, the information of audio and text can be directly used at
the same time. while since only audio information can be used in the inference
process, this method will make the model miss part of the input during
inference. Previous studies almost bridge the difference between both
modalities by using contrastive loss[14, 15, 16, 19, 20]. Although this method
is simple and easy to use, it is an indirect use of text information. Isn't
there a way to use text information directly? In order to use text information
directly, a core problem must be overcome, which is how to make the model
learn how to extract text features while retaining the ability to extract
audio features.

Table 1: Performance in Clotho and AudioCaps[4] evaluation split, where Extra
training data denotes the use of additional data during the training phase. AC
denotes AudioCaps and WC denotes WavCaps[12]. Bold indicates the best result
for each group.

Dataset | Methods | Extra training data | BLEU1 | BLEU4 | ROUGEl | METEOR | CIDER | SPICE | SPIDEr | SPIDEr_FL
---|---|---|---|---|---|---|---|---|---|---
Clotho |  Zhang et al.[39] | - | 0.566 | 0.161 | 0.375 | 0.176 | 0.409 | 0.121 | 0.265 | -
Ye et al.[40] | - | 0.577 | 0.174 | 0.377 | 0.174 | 0.419 | 0.119 | 0.269 | -
Komatsu et al.[41] | ESC-50 | 0.601 | 0.166 | 0.388 | 0.194 | 0.454 | 0.127 | 0.291 | 0.289
Wu et al.[16] (SOTA) | AC(ChatGPT assist) | - | - | - | 0.187 | 0.474 | 0.134 | 0.304 | 0.303
Ours | - | 0.597 | 0.177 | 0.391 | 0.188 | 0.483 | 0.135 | 0.309 | 0.304
Mei et al.[12] | WC | 0.601 | 0.180 | 0.400 | 0.185 | 0.488 | 0.133 | 0.310 | -
Wu et al.[16] \+ Rerank | AC(ChatGPT assist) | - | - | - | 0.193 | 0.506 | 0.146 | 0.326 | 0.326
AudioCaps |  Liu et al.[42] | - | 0.698 | 0.281 | 0.494 | 0.237 | 0.711 | 0.172 | 0.442 | -
M. Kim et al.[43] | - | 0.713 | 0.309 | 0.503 | 0.240 | 0.733 | 0.177 | 0.455 | -
Deshmukh et al.[44] | - | 0.691 | 0.253 | 0.482 | 0.232 | 0.752 | 0.182 | 0.467 | -
E. Kim et al.[45] | - | 0.700 | 0.289 | 0.502 | 0.242 | 0.769 | 0.181 | 0.475 | -
Mei et al.[12] (SOTA) | WC | 0.707 | 0.283 | 0.507 | 0.249 | 0.787 | 0.182 | 0.485 | -
Ours | - | 0.702 | 0.280 | 0.507 | 0.254 | 0.806 | 0.180 | 0.493 | 0.484

####  3.4.2 specific details

In the proposed method, the model is divided into audio branch and text
branch. The detail of the model is shown in Figure 1. Firstly, each of the two
branches extracts the features of the corresponding modality. In the audio
branch, the combination of Encoders + FUSER is used to extract audio features.
In the text branch, a frozen language model INSTRUCTOR-base is used to extract
word-level features. Then, the features of each branch are input into a twin
TRANSLATOR structure, which is composed of two identical TRANSLATOR modules
with shared weights, and features of the two modalities are learned through
this identical network structure. Although this method allows the TRANSLATOR
to learn the features of two modalities at the same time, it still cannot
solve the problem of missing text data in the inference stage. Therefore, we
add two methods to solve this problem: (1) Updating the weight with momentum.
(2) Adding contrastive loss.

Assuming that the weight of the TRANSLATOR in the audio branch is $W$, and the
weight of the TRANSLATOR in the text branch is $W_{sim}$, the momentum update
weight can be expressed as follows:

$W=\beta\cdot W\ +\ \left(1-\beta\right)\cdot W_{sim}$

Where $\beta$ is the momentum parameter, which is used to control the
proportion of updates at each step. Each time the model updates the weights,
the twin TRANSLATOR structure is updated by using the momentum to update the
weights. Specifically, after each weight update, the weight of the TRANSLATOR
module in the audio branch is composed of two branch weights combined in a
certain proportion. By increasing the weight proportion of the audio branch
($\beta>0.9$), the ability of the audio branch TRANSLATOR to extract audio
features is guaranteed while slowly learn text features. After the weights are
updated, the weights of the TRANSLATOR in the audio branch are directly copied
to the TRANSLATOR in the text branch. This is to ensure that the weights of
the two branches are consistent, so as to prevent the weights of the two
branches from being completely uncorrelated due to long training, which will
lead to twin structure loses the ability to learn two modalities information
at the same time. We experimentally find that $\beta=0.95$ is best.

There are two purposes for applying contrastive loss: (1) By distinguishing
positive and negative sample pairs in each batch, the audio branch module can
indirectly learn the text features extracted by the text branch module. (2) By
distinguishing the positive and negative sample pairs in each batch, the
TRANSLATOR modules in the two branches are forced to learn the information of
their respective modalities, so as to realize the goal of learning the
information of two modalities at the same time. In the proposed method, the
last hidden state of the TRANSLATOR in each branch are used to compute the
contrastive loss. The last hidden state is the hidden state of the last time
step of the TRANSLATOR's last layer decoder, which has the information of all
time steps simultaneously. Compared with the dimension reduction of the two
modalities features by linear transformation, the direct use of last hidden
state can avoid the information loss caused by dimension reduction. The way to
calculate the contrastive loss is similar to CLIP[13], which can be expressed
as follows:

$L_{a\rightarrow
c}=\frac{1}{N}{\textstyle\sum_{i=1}^{N}}\log{\frac{\exp\left(h_{i}^{a}\cdot
h_{i}^{c}\right)}{{\textstyle\sum_{j=1}^{N}}\exp\left(h_{j}^{a}\cdot
h_{j}^{c}\right)}}$ $L_{c\rightarrow
a}=\frac{1}{N}{\textstyle\sum_{i=1}^{N}}\log{\frac{\exp\left(h_{i}^{a}\cdot
h_{i}^{c}\right)}{{\textstyle\sum_{j=1}^{N}}\exp\left(h_{j}^{c}\cdot
h_{j}^{a}\right)}}$ $L_{cl}=\frac{1}{2}\left(L_{a\rightarrow c}\ +\
L_{c\rightarrow a}\right)$

Where $N$ denotes the batch size, $\left(h^{a},\ h^{c}\right)$ denotes the
TRANSLATOR's last hidden state in the audio branch and text branch,
respectively. The total loss of the model can be expressed as follows:

$L=L_{ce}\ +\ L_{cl}$

##  4 Results

This section will introduce the experimental procedure, training strategy and
evaluation metrics in detail

###  4.1 Dataset

We verify the feasibility of the proposed method in Clotho V2[3] and
AudioCaps[4], respectively.

Table 2: Results of ablation experiments on the Clotho[3] evaluation split.
The effectiveness of the method is verified by adding modules one by one on
the basis of baseline. Where Encoders means that three audio encoders
(CNN14[5], HTSAT[6], ConvNeXt[17]) are used at the same time, and (Sim)
TRANSLATOR means that twin TRANSLATOR structure is used. All encoder weights
are frozen during training.

Adding modules | Module composition | METEOR | CIDER | SPICE | SPIDEr | SPIDEr_FL
---|---|---|---|---|---|---
baseline | CNN14 + BART | 0.169 | 0.405 | 0.115 | 0.260 | 0.257
HTSAT + BART | 0.178 | 0.426 | 0.126 | 0.276 | 0.274
ConvNeXt + BART | 0.172 | 0.398 | 0.122 | 0.260 | 0.256
FUSER | Encoders + FUSER + BART | 0.185 | 0.463 | 0.131 | 0.297 | 0.293
HTSAT + FUSER + BART | 0.184 | 0.454 | 0.133 | 0.293 | 0.292
TRANSLATOR | Encoders + FUSER + TRANSLATOR + BART | 0.186 | 0.470 | 0.135 | 0.302 | 0.297
HTSAT + TRANSLATOR + BART | 0.182 | 0.450 | 0.133 | 0.290 | 0.288
Twin structure | HTSAT + (Sim) TRANSLATOR + BART | 0.182 | 0.453 | 0.132 | 0.293 | 0.286
Encoders + FUSER + (Sim) TRANSLATOR + BART | 0.188 | 0.483 | 0.135 | 0.309 | 0.304

####  4.1.1 Clotho V2

Clotho V2[3] is split into development, Evaluation, validation, and test
splits. In addition to the test split, the other three splits are all composed
of audio samples with a duration of 15 to 30 seconds, and each audio sample
contains five captions of length 5-10. As the official dataset for the DCASE
challenge, there are a total of 6974 audio samples with 34870 captions.

####  4.1.2 AudioCaps

The AudioCaps[4] data consists of 50K audio samples picked by Audioset[30] and
split into training, validation, and test splits. The length of each audio
sample is 10s. Each audio sample corresponds to one caption in training
splits, and each audio sample corresponds to five captions in validation and
test splits.

###  4.2 Evaluation metrics

The evaluation metrics for AAC tasks can be divided into two categories, which
are based on n-gram overlap and based on semantic similarity[1]. BLEU-n[31]
evaluates the grammatical match between the predicted value and the ground
truth value by calculating n-grams, but there are problems where short
captions are more likely to obtain high scores. ROUGE-L[32] is similar to BLEU
but focuses more on recall. METEOR[33] is more focused on calculating the F1
score and recall for individual words. CIDEr[34] weights n-grams by
calculating TF-IDF scores. SPICE[35] is used to evaluate how well the
predicted values match the ground truth semantically. SPIDEr[36] is the
average of SPICE and CIDEr. SPIDEr-FL is a newly proposed metric, which is
obtained by calculating the average of SPIDEr and FENSE[37].

Since the start of the DCASE 2023 challenge, METEOR, CIDEr, SPICE, SPIDEr, and
SPIDEr-FL have been used as metrics for AAC tasks. We use the same metrics as
DCASE to evaluate the proposed method, and other metrics will also be listed
for reference.

###  4.3 Experimental environment and parameter Settings

We use hyperparameter settings that have been extensively validated in
previous studies. AdamW is used as the optimizer in the training process with
batch size 32. The initial learning rate was $3\times 10^{-5}$, and the cosine
annealing algorithm was used to update the learning rate. In a cycle of 4
epochs, the benchmark learning rate first decreased by 10 times and then
recovered in a cycle. Each epoch is evaluated three times, each time comparing
the CIDEr[34] to determine whether the weight is saved. In addition, label
smooth is applied to reduce the problem of model overconfidence. Use teacher
forced methods to shorten the training time. Each audio encoder uses
SpecAugment[38] for data augmentation.

We train for 20 epochs on a single NVIDIA V100 (32GB) GPU, halve the baseline
learning rate every 4 epochs. The strategy of early stopping is used to end
the training when the metrics are not updated for 4 consecutive epochs.

In the inference phase, the beam search method with num_beams = 4 is used to
generate captions, and data augmentation is disabled.

###  4.4 Comparison with other methods

We compare the proposed method with the current best performing method, as
shown in Table 1. Even though BLEU[31] and ROUGE[32] have been removed from
the final results since the DCASE 2023 challenge, for the sake of fairness, we
still list them for reference.

Our method outperforms the current state of the art methods on the
AudioCaps[4] dataset and achieves competitive results with the SOTA method on
the Clotho[3] dataset. Notably, Mei et al.[12] 's method underperforms our
method on the AudioCaps dataset even pre-trained on the WavCaps[12] dataset
(which is almost 60 times larger than Clotho), while it achieves only a slight
advantage $\left(\leq 0.005\right)$ on the Clotho dataset.

We also specifically compare with the method proposed by Wu et al.[16], as
shown in Table 3, which achieves the state of the art performance on the
Clotho dataset. Wu et al.[16] 's method uses the AudioCaps dataset for pre-
training and ChatGPT to assist in integrating captions in the dataset, which
essentially uses additional data for training. The results show that our
method, both without and with text embeddings, performs better on the Clotho
dataset than the method proposed by Mei et re[16], despite not training with
additional data. And the number of parameters is only about 35% of Wu et
al.[16]

It is important to point out that our approach is not fine-tune with
reinforcement learning.[11] Because it will improve the metrics by repeatedly
generating keywords which will reduce the fluency of the generated captions.

###  4.5 Ablation Study

We conduct an ablation study of the proposed method on the Clotho[3] dataset
by adding a single module to the baseline model benchmark. The specific
results are shown in Table 2. The architecture of a single audio encoder and
using BART[8] as the text decoder is the baseline model. During training, the
weights of all encoders are frozen.

Lines 1-3 show that HTSAT[6] performs best when BART is used as the text
decoder. We tried using other audio encoders (BEATs[46], etc.) but didn't get
better results. Lines 4-5 show that adding the FUSER module proposed in this
paper can achieve significant results. This means that the FUSER module can
effectively combine different audio features. The results in line 5 show that
efficient results are still achieved when using a single audio encoder as
input to the FUSER module, but the performance is not as good as using
multiple audio encoders. This indicates that the FUSER module can also propose
higher dimensional semantic information from a single audio features.

The results in lines 6-9 show that both the TRANSLATOR module and the twin
structure proposed in this paper can achieve effective results, but the boost
is less than that of the FUSER module. It is worth noting that under the
proposed architecture, the TRANSLATOR module must be used when using the twin
structure. We tried using InfoNCE[28] loss as the contrastive loss, but it
didn't perform as well as CLIP loss.

Table 3: Detailed comparison with SOTA[16]. The baseline representation only
uses audio embeddings. text embs means adding text embeddings to the baseline
model. Rerank represents the reordering of captions using additional methods.

Modules |  | METEOR | CIDER | SPICE | SPIDEr | SPIDEr_FL
---|---|---|---|---|---|---
baseline | Ours | 0.186 | 0.470 | 0.135 | 0.302 | 0.297
SOTA | 0.186 | 0.458 | 0.134 | 0.296 | 0.294
+ text embs | Ours | 0.188 | 0.483 | 0.135 | 0.309 | 0.304
SOTA | 0.187 | 0.474 | 0.134 | 0.304 | 0.303
+ Rerank | Ours | - | - | - | - | -
SOTA | 0.193 | 0.506 | 0.146 | 0.326 | 0.326

##  5 Conclusion and future work

In this work, in order to overcome the disparity between the two modalities,
three perspectives are proposed to jointly assist the model to understand the
text information. Firstly, FUSER is proposed to extract common semantic
information from different audio features by feature fusion. Then, by
proposing TRANSLATOR, audio features are aligned to text features from the
tensor level. Finally, through the twin structure, the model in the audio
branch can retain its ability to extract audio features, and learn text
information from text features at the same time, so as to make use of text
information as efficiently as possible. The FUSER module and TRANSLATOR module
proposed in this study can be used as plug-and-play semantic feature
extractors and applied to almost models in AAC research.

In future work, self-supervised methods may be combined to try not to use
additional text encoders, which will fundamentally solve the problem of
resource waste due to the introduction of additional encoders.

## References

  * [1] X. Mei, X. Liu, M. D. Plumbley, and W. Wang, ``Automated audio captioning: an overview of recent progress and new challenges,'' _EURASIP journal on audio, speech, and music processing_ , vol. 2022, no. 1, pp. 1–18, 2022.
  * [2] X. Xu, M. Wu, and K. Yu, ``A comprehensive survey of automated audio captioning,'' _arXiv preprint arXiv:2205.05357_ , 2022.
  * [3] K. Drossos, S. Lipping, and T. Virtanen, ``Clotho: An audio captioning dataset,'' in _ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2020, pp. 736–740.
  * [4] C. D. Kim, B. Kim, H. Lee, and G. Kim, ``Audiocaps: Generating captions for audios in the wild,'' in _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , 2019, pp. 119–132.
  * [5] Q. Kong, Y. Cao, T. Iqbal, Y. Wang, W. Wang, and M. D. Plumbley, ``Panns: Large-scale pretrained audio neural networks for audio pattern recognition,'' _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , vol. 28, pp. 2880–2894, 2020.
  * [6] K. Chen, X. Du, B. Zhu, Z. Ma, T. Berg-Kirkpatrick, and S. Dubnov, ``Hts-at: A hierarchical token-semantic audio transformer for sound classification and detection,'' in _ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2022, pp. 646–650.
  * [7] A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever _et al._ , ``Language models are unsupervised multitask learners,'' _OpenAI blog_ , vol. 1, no. 8, p. 9, 2019.
  * [8] M. Lewis, Y. Liu, N. Goyal, M. Ghazvininejad, A. Mohamed, O. Levy, V. Stoyanov, and L. Zettlemoyer, ``Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension,'' _arXiv preprint arXiv:1910.13461_ , 2019.
  * [9] Z. Xie, X. Xu, M. Wu, and K. Yu, ``Enhance temporal relations in audio captioning with sound event detection,'' _arXiv preprint arXiv:2306.01533_ , 2023.
  * [10] A. Ö. Eren and M. Sert, ``Automated audio captioning with topic modeling,'' _IEEE Access_ , vol. 11, pp. 4983–4991, 2023.
  * [11] X. Mei, Q. Huang, X. Liu, G. Chen, J. Wu, Y. Wu, J. Zhao, S. Li, T. Ko, H. L. Tang _et al._ , ``An encoder-decoder based audio captioning system with transfer and reinforcement learning,'' _arXiv preprint arXiv:2108.02752_ , 2021.
  * [12] X. Mei, C. Meng, H. Liu, Q. Kong, T. Ko, C. Zhao, M. D. Plumbley, Y. Zou, and W. Wang, ``Wavcaps: A chatgpt-assisted weakly-labelled audio captioning dataset for audio-language multimodal research,'' _arXiv preprint arXiv:2303.17395_ , 2023.
  * [13] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark _et al._ , ``Learning transferable visual models from natural language supervision,'' in _International conference on machine learning_. PMLR, 2021, pp. 8748–8763.
  * [14] B. Elizalde, S. Deshmukh, M. Al Ismail, and H. Wang, ``Clap learning audio concepts from natural language supervision,'' in _ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2023, pp. 1–5.
  * [15] Y. Wu, K. Chen, T. Zhang, Y. Hui, T. Berg-Kirkpatrick, and S. Dubnov, ``Large-scale contrastive language-audio pretraining with feature fusion and keyword-to-caption augmentation,'' in _ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2023, pp. 1–5.
  * [16] S.-L. Wu, X. Chang, G. Wichern, J.-w. Jung, F. Germain, J. L. Roux, and S. Watanabe, ``Improving audio captioning models with fine-grained audio features, text embedding supervision, and llm mix-up augmentation,'' _arXiv preprint arXiv:2309.17352_ , 2023.
  * [17] Z. Liu, H. Mao, C.-Y. Wu, C. Feichtenhofer, T. Darrell, and S. Xie, ``A convnet for the 2020s,'' in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2022, pp. 11 976–11 986.
  * [18] H. Su, W. Shi, J. Kasai, Y. Wang, Y. Hu, M. Ostendorf, W.-t. Yih, N. A. Smith, L. Zettlemoyer, and T. Yu, ``One embedder, any task: Instruction-finetuned text embeddings,'' _arXiv preprint arXiv:2212.09741_ , 2022.
  * [19] C. Chen, N. Hou, Y. Hu, H. Zou, X. Qi, and E. S. Chng, ``Interactive audio-text representation for automated audio captioning with contrastive learning,'' _arXiv preprint arXiv:2203.15526_ , 2022.
  * [20] S. Deshmukh, B. Elizalde, D. Emmanouilidou, B. Raj, R. Singh, and H. Wang, ``Training audio captioning models without audio,'' _arXiv preprint arXiv:2309.07372_ , 2023.
  * [21] Z. Zhao, H. Bai, J. Zhang, Y. Zhang, S. Xu, Z. Lin, R. Timofte, and L. Van Gool, ``Cddfuse: Correlation-driven dual-branch feature decomposition for multi-modality image fusion,'' in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2023, pp. 5906–5916.
  * [22] H. Jung, Y. Kim, H. Jang, N. Ha, and K. Sohn, ``Unsupervised deep image fusion with structure tensor representations,'' _IEEE Transactions on Image Processing_ , vol. 29, pp. 3845–3858, 2020.
  * [23] H. Xu, J. Ma, Z. Le, J. Jiang, and X. Guo, ``Fusiondn: A unified densely connected network for image fusion,'' in _Proceedings of the AAAI conference on artificial intelligence_ , vol. 34, no. 07, 2020, pp. 12 484–12 491.
  * [24] H. Xu, J. Ma, J. Jiang, X. Guo, and H. Ling, ``U2fusion: A unified unsupervised image fusion network,'' _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 44, no. 1, pp. 502–518, 2020.
  * [25] C.-W. Kuo and Z. Kira, ``Haav: Hierarchical aggregation of augmented views for image captioning,'' in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2023, pp. 11 039–11 049.
  * [26] L. Huang, W. Wang, J. Chen, and X.-Y. Wei, ``Attention on attention for image captioning,'' in _Proceedings of the IEEE/CVF international conference on computer vision_ , 2019, pp. 4634–4643.
  * [27] W. Jiang, L. Ma, Y.-G. Jiang, W. Liu, and T. Zhang, ``Recurrent fusion network for image captioning,'' in _Proceedings of the European conference on computer vision (ECCV)_ , 2018, pp. 499–515.
  * [28] K. He, H. Fan, Y. Wu, S. Xie, and R. Girshick, ``Momentum contrast for unsupervised visual representation learning,'' in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2020, pp. 9729–9738.
  * [29] T. Chen, S. Kornblith, M. Norouzi, and G. Hinton, ``A simple framework for contrastive learning of visual representations,'' in _International conference on machine learning_. PMLR, 2020, pp. 1597–1607.
  * [30] J. F. Gemmeke, D. P. Ellis, D. Freedman, A. Jansen, W. Lawrence, R. C. Moore, M. Plakal, and M. Ritter, ``Audio set: An ontology and human-labeled dataset for audio events,'' in _2017 IEEE international conference on acoustics, speech and signal processing (ICASSP)_. IEEE, 2017, pp. 776–780.
  * [31] K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu, ``Bleu: a method for automatic evaluation of machine translation,'' in _Proceedings of the 40th annual meeting of the Association for Computational Linguistics_ , 2002, pp. 311–318.
  * [32] L. Chin-Yew, ``Rouge: A package for automatic evaluation of summaries,'' in _Proceedings of the Workshop on Text Summarization Branches Out, 2004_ , 2004.
  * [33] S. Banerjee and A. Lavie, ``Meteor: An automatic metric for mt evaluation with improved correlation with human judgments,'' in _Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization_ , 2005, pp. 65–72.
  * [34] R. Vedantam, C. Lawrence Zitnick, and D. Parikh, ``Cider: Consensus-based image description evaluation,'' in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2015, pp. 4566–4575.
  * [35] P. Anderson, B. Fernando, M. Johnson, and S. Gould, ``Spice: Semantic propositional image caption evaluation,'' in _Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part V 14_. Springer, 2016, pp. 382–398.
  * [36] S. Liu, Z. Zhu, N. Ye, S. Guadarrama, and K. Murphy, ``Improved image captioning via policy gradient optimization of spider,'' in _Proceedings of the IEEE international conference on computer vision_ , 2017, pp. 873–881.
  * [37] Z. Zhou, Z. Zhang, X. Xu, Z. Xie, M. Wu, and K. Q. Zhu, ``Can audio captions be evaluated with image caption metrics?'' in _ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2022, pp. 981–985.
  * [38] D. S. Park, W. Chan, Y. Zhang, C.-C. Chiu, B. Zoph, E. D. Cubuk, and Q. V. Le, ``Specaugment: A simple data augmentation method for automatic speech recognition,'' _arXiv preprint arXiv:1904.08779_ , 2019.
  * [39] Y. Zhang, H. Yu, R. Du, Z.-H. Tan, W. Wang, Z. Ma, and Y. Dong, ``Actual: Audio captioning with caption feature space regularization,'' _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , 2023.
  * [40] Z. Ye, H. Wang, D. Yang, and Y. Zou, ``Improving the performance of automated audio captioning via integrating the acoustic and semantic information,'' _arXiv preprint arXiv:2110.06100_ , 2021.
  * [41] T. Komatsu, Y. Fujita, K. Takeda, and T. Toda, ``Audio difference learning for audio captioning,'' _arXiv preprint arXiv:2309.08141_ , 2023.
  * [42] X. Liu, Q. Huang, X. Mei, H. Liu, Q. Kong, J. Sun, S. Li, T. Ko, Y. Zhang, L. H. Tang _et al._ , ``Visually-aware audio captioning with adaptive audio-visual attention,'' _arXiv preprint arXiv:2210.16428_ , 2022.
  * [43] M. Kim, K. Sung-Bin, and T.-H. Oh, ``Prefix tuning for automated audio captioning,'' in _ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2023, pp. 1–5.
  * [44] S. Deshmukh, B. Elizalde, R. Singh, and H. Wang, ``Pengi: An audio language model for audio tasks,'' _arXiv preprint arXiv:2305.11834_ , 2023.
  * [45] E. Kim, J. Kim, Y. Oh, K. Kim, M. Park, J. Sim, J. Lee, and K. Lee, ``Exploring train and test-time augmentations for audiolanguage learning,'' _arXiv preprint arXiv:2210.17143_ , 2023.
  * [46] S. Chen, Y. Wu, C. Wang, S. Liu, D. Tompkins, Z. Chen, and F. Wei, ``Beats: Audio pre-training with acoustic tokenizers,'' _arXiv preprint arXiv:2212.09058_ , 2022.