# Audio-Language Datasets of Scenes and Events:
A Survey

Gijs Wijngaard1, Elia Formisano2, Michele Esposito2 and Michel Dumontier1  1
Department of Advanced Computing Sciences, Faculty of Science and Engineering,
Maastricht University
2 Department of Cognitive Neuroscience, Faculty of Psychology and
Neuroscience, Maastricht University

###### Abstract

Audio-language models (ALMs) process sounds to provide a linguistic
description of sound-producing events and scenes. Recent advances in computing
power and dataset creation have led to significant progress in this domain.
This paper surveys existing datasets used for training audio-language models,
emphasizing the recent trend towards using large, diverse datasets to enhance
model performance. Key sources of these datasets include the Freesound
platform and AudioSet that have contributed to the field’s rapid growth.
Although prior surveys primarily address techniques and training details, this
survey categorizes and evaluates a wide array of datasets, addressing their
origins, characteristics, and use cases. It also performs a data leak analysis
to ensure dataset integrity and mitigate bias between datasets. This survey
was conducted by analyzing research papers up to and including December 2023,
and does not contain any papers after that period.

###### Index Terms:

Audio-Language Learning, Multimodal Datasets

##  I Introduction

Audio-Language learning, also referred to as Audio-Text learning, is an
emerging field of research that focuses on processing, understanding and
description of sounds. It is driven by the advancement of machine learning
technologies and the increasing availability of datasets that combine sounds
with their corresponding textual descriptions. These models fall into the
broader category of Multimodal Large Language Models, which are capable of
understanding and generating responses based on multiple types of input data
[1]. This field has recently experienced significant growth with the emergence
of large language models (LLMs), inspiring new research directions [2, 3].
Transformer models [4] have revolutionized natural language processing and
have been adapted for audio-language learning in various settings [5, 6, 7, 8,
9]. Contrastive learning and transfer learning training techniques have
demonstrated remarkable performance on various tasks [10, 11, 12, 13].

In the audio domain, the increase in available datasets has been driven by two
main sources: the release of the Freesound platform in 2013 [14] and AudioSet
in 2017 [15]. These collections have been used to train models for a wide
range of sounds. Several research studies on audio-language datasets have
followed [16, 17, 12, 18], with datasets being released and refined to meet
the needs of specific audio-language models [19, 10, 20].

Recent studies indicate that scaling laws are important to develop foundation
models[21, 22]. In training these models, a correlation can be shown between
model performance and the amount of data and compute. In audio-language
learning, for example, the ESC-50 audio classification dataset requires around
2 million audio-text data points for a zero-shot model to match human parity
[23]. An issue in the retrieval of audio-language data is that the amount of
training audio data available is inevitably less than that for natural
language processing tasks [24]. Current audio datasets are based on sound
effect databases, YouTube, or Freesound for their audio, whereas LLMs are
trained on a large portion of the open Internet [25]. Having a limited set of
training data presents the issue of distributional shift: a mismatch between
training data and production data, which makes a model poorly equipped for
real-world data [26]. Models trained with contrastive learning (CLAP) [12,
11], a method to distinguish between similar and dissimilar data pairs,
benefit from large and diverse datasets for better performance in audio-
language domains [27, 28, 29, 30, 31].

This survey aims to introduce researchers to the various datasets available in
the audio-language domain. To the best of our knowledge, it is the most
comprehensive survey on audio datasets to date. The specific contributions are
twofold:

  1. 1.

It provides a detailed overview of the datasets used in the field of audio-
language models. This includes relevant statistics, presented in Table I, such
as the average length of audio and text (captions), total hours, and their
sources and domains.

  2. 2.

In Section IV-B, a data leak analysis is performed to assess the extent of
data leakage and to gain insight into the overlap between datasets. This
information is relevant for combining datasets and developing new ones.

Given the substantial scope of the audio-language field, this article focuses
on two specific aspects. First, we examine the domain of sound and event
machine listening, excluding speech and music modeling from this survey (see,
for example, [32, 33, 9, 34]). Second, we distinguish between different types
of machine learning tasks. Traditional tasks, such as sound event detection
(SED) and Audio Tagging (AT) are limited to predicting within the scope of the
labeled data on which they were trained. For example, a model trained on the
AudioSet dataset can only predict labels out of 527 classes given an audio
input. In contrast, more recent approaches utilize an encoder-decoder setup,
also known as audio-language models. Audio-language models consist of one
model for audio understanding and representation and another one for text
understanding and representation. This survey paper focuses on these models.
There are several tasks in the field of audio-language modeling, including
Automated Audio Captioning (AAC), Audio-Text Retrieval (ATR), Automated Audio
Generation (AAG) and Text-to-Audio Retrieval (ATR). This survey paper briefly
explains all these tasks in Section II.

This survey begins by introducing the field and its models in the background
section (Section II). Afterwards, it reviews related work on audio-language
research and describes the research methodology used. Then, it provides a
comprehensive overview of the identified datasets in the field (Section III).
The survey concludes with an analysis on data leakage (Section IV-B), followed
by a discussion on bias and data quality in audio datasets (Section V), and a
conclusion (Section VI) of data overlap and its use for future research.

##  II Background

This section serves as an introductory overview of Audio-Language Learning,
delineating the field into two primary task categories: Audio-to-Language and
Language-to-Audio. Within these categories, we describe the main tasks and
briefly touch upon some prominent Audio-Language training techniques. For
readers interested in specific tasks, we highly recommend consulting the
survey papers of Xu et al. [6] and Li et al. [8].

###  II-A Audio-to-Language

The Audio-to-Language category comprises models that convert audio input into
textual output, either through generation by a model or retrieval from a
dataset. This category encompasses several subfields:

  1. 1.

Audio Captioning: This involves generating natural language descriptions for
audio events.

  2. 2.

Audio-Text Retrieval: This focuses on using a database of audio-language pairs
to find texts corresponding to given audio inputs.

  3. 3.

Audio-Question Answering: This involves answering questions asked in natural
language about specific audio content.

  4. 4.

Audio Difference Captioning: This focuses on generating descriptions that
highlight content differences between pairs of similar audio clips (e.g. the
addition of a sound to another sound).

####  II-A1 (Automated) Audio Captioning (AAC)

AAC, as originally introduced by Drossos et al. [35] represents the task of
converting audio signals into natural language descriptions. Prominent
datasets within this domain include Clotho [17] and AudioCaps [16] (see
Section III). AAC models predominantly utilize an encoder-decoder mechanism.
The encoder in an audio captioning model processes audio signals to create a
machine representation. The decoder translates this encoded representation
into textual descriptions. Encoders are frequently trained on datasets such as
AudioSet [15] in an audio classification context [36, 37] to facilitate
transfer learning capabilities, as demonstrated in various studies [38, 39,
40]. Common architectures for encoders include Convolutional Neural Networks
(CNNs) [15, 41] and Transformer encoders [42], where decoders typically employ
Transformers [43, 44].

The task of Audio Logging (AL) extends the concept of AAC by focusing on
longer, paragraph-length descriptions. For example, Bai et al. [45] introduced
a method that uses LLM to describe audio recordings up to 3 minutes long.

####  II-A2 Audio-to-Text Retrieval (ATR)

ATR involves identifying the relevant textual descriptions corresponding to an
audio query. This task presents significant challenges, due to the inherent
differences between audio and text modalities. These discrepancies require the
development of models that can effectively bridge these two forms of content.
Recent studies in this field have concentrated on developing methodologies for
cross-modal alignments between audio and text [46, 47], to make effective
bridging of these content forms possible.

####  II-A3 Audio-Question Answering (AQA)

AQA is a multimodal task that involves a system receiving both an audio input
and a related question, to provide accurate answers about the audio content.
This task requires the integration of audio signal processing and natural
language understanding to produce precise responses. This field has seen
notable progression with the introduction of attention-based neural network
models. These models employ self-attention mechanisms to create
representations from both audio and text and use cross-attention techniques to
map relevant audio features to textual queries [48, 49].

Datasets utilized in Audio Question Answering (AQA) differ significantly from
those used in Audio Captioning and Audio-Text Retrieval tasks. To train AQA
models, specific datasets such as the Clotho-AQA dataset [50] and the OpenAQA
dataset [19] have been proposed.

####  II-A4 Audio Difference Captioning (ADC)

ADC tackles an issue in traditional audio captioning where models generate
similar descriptions for similar sounds, without adequately distinguishing the
differences in their content. ADC models utilize annotations highlighting
similar captions’ differences, allowing better differentiation between audio
clips. Research by Bhosale et al. [51] shows that incorporating ADC techniques
improves performance in standard audio captioning tasks. Furthermore, Tsubaki
et al. [52] have successfully applied ADC to identify small anomalies in
machine sounds to showcase its usefulness in real-world scenarios.

Examples of datasets in ADC include: (1) the MIMII-Change dataset [52] that
focuses on language descriptions of machine-sound anomalies. (2) The
AudioDiffCaps dataset [53] with artificially mixed background and foreground
sounds from existing environmental sound datasets.

A similar technique for training named Audio Difference Learning is described
by Komatsu et al. [54], particularly for pre-training a model for Audio
Captioning purposes. In this paradigm, two audios - input and reference - are
mixed and encoded, the encoded version is subtracted against the encoded
reference, which is then fed to a decoder to create the caption. This avoids
the need to create a custom dataset for ADC.

###  II-B Language-to-Audio

The Language-to-Audio category encompasses fields where language input is used
to produce or retrieve audio. Three distinct fields are distinguished within
this category.

  1. 1.

Audio Generation: This field involves creating audio based on natural language
prompts.

  2. 2.

Text-To-Audio Retrieval: In this category, natural language is used to find
matching audio from a database containing audio-language pairs.

  3. 3.

Language-Queried Audio Source Separation: This area focuses on isolating
specific sounds through queries made in natural language.

####  II-B1 Audio Generation (AG)

Audio generation, also known as Audio Synthesis, focuses on creating synthetic
audio that can mimic sounds, speech, or music. These audio generation models
typically take text inputs known as prompts, with a wide variety in length,
and are capable of generating diverse audio content. There are at least two
main categories of language-to-audio generation models:

  1. 1.

Language-model based methods, such as AudioGen [55] and AudioLM [56]. Types of
autoregressive models that are trained on text-audio pairs to be able to
generate audio by prompting.

  2. 2.

Methods based on diffusion models, such as Diffsound [57] and AudioLDM [58],
which can generate audio from scratch or can be conditioned on input audio or
text.

####  II-B2 Text-to-Audio Retrieval (TAR)

Text-to-audio retrieval, also called language-based audio retrieval, involves
searching and retrieving audio clips that closely match a given text query
[59]. The task is similar to Audio-to-Text Retrieval, where the challenge is
to map audio and text into a common embedding space for effective cross-modal
retrieval. Methods to achieve this include pre-training models using
contrastive learning [60, 61], or employing training objectives such as
triplet loss [62, 63].

####  II-B3 Language queried audio source separation (LASS)

Language-queried audio source separation (LASS) uses descriptions of natural
language to pinpoint (and isolate) specific sound sources from a mixture of
sounds. Other names for this field are Text-to-Audio Grounding [64, 65],
Language-assisted Sound Event Detection (SED-LM) [66], Language-Queried Target
Sound Extraction [67], and query-based sound separation (QSS) [68].

The prominent models in this field include LASSNet [69], SoundWords [70],
CLIPSep [71], and AudioSep [68]. In 2024, LASS was introduced as a new
challenge in the DCASE competition 111dcase.community/challenge2024/task-
language-queried-audio-source-separation.

###  II-C Audio Captioning Model Overview

Audio captioning models are trained on datasets comprised of pairs
$(\tau,\alpha)$, where models learn to infer text $\tau$ from audio $\alpha$.
In training these models, several types of model training methods come into
play (Fig. 1). Historically, these models have leveraged an encoder-decoder
architecture, with the encoder processing one modality to produce embeddings
that a text decoder then interprets. Today, a variety of approaches exist,
including multimodal fusion with cross-attention, joint integration of audio
and language using audio embeddings as prefixes for the decoder, and a
combination of off-the-shelf models without any training (zero-shot). In this
section, we discuss each of these three topics.

(a) An encoder-decoder model

(b) A prefix model

(c) A zero-shot model

Figure 1: Three different ways an audio-language model can be trained.

####  II-C1 Cross-attention

Cross-attention mechanisms are used in a large number of audio captioning
models, particularly to align audio features with textual outputs in a
decoder. In this setup, the encoder, which processes audio through models such
as CNN or Transformers, outputs features that serve as keys and queries for
the attention mechanism (Fig. 1(a)). These are distinct from the inputs, which
are typically derived from the decoder itself. In cross-attention models, both
the audio encoder and the text decoder can be static (or frozen). An important
aspect of cross-attention is the mapping network that is often employed to
adapt the encoder’s output to be effectively utilized by the decoder. This
approach allows for sophisticated integration of audio and text data, even
when direct learning from one to the other is not feasible. Common models used
as audio encoders include CNNs, with models such as VGGish [72, 73] or PANNs
[74, 39], and Transformers such as the Vision Transformer (e.g. [37]) [75].

####  II-C2 Jointly training with audio prefixes

In prefix-based models (Fig. 1(b)), the output of an audio encoder is mapped
to the start of the decoder sequence [76, 77]. This alignment requires that
the embedding size of the audio prefix match those of the decoder’s input.
Unlike approaches that use cross-attention, this method employs self-attention
to integrate audio embeddings directly into the language model, which is often
frozen during training to maintain stability.

During generation, the language model predicts each subsequent token based on
both the preceding tokens and the audio input. A notable variation in this
approach is retrieval-augmented (RAG) audio captioning, where additional
information is retrieved from a datastore and used as a prefix [78]. This
method also finds use in audio retrieval tasks [79, 80].

Furthermore, training with audio prefixes serves as an effective pre-training
mechanism for foundational audio-language models. It allows integrating a
variety of audio types, using a unified encoder [20] or multiple encoders
[81]. This training technique also allows scenarios where a model is trained
exclusively on text while employing a frozen encoder trained by contrastive
learning to interpret audio cues [80, 82].

####  II-C3 Off-the-shelf models

Off-the-shelf models (Fig. 1(c)) refer to those not specifically trained for
audio captioning but adapted from pre-trained models to function in a zero-
shot audio captioning context. These models leverage existing multimodal
networks and LLMs to generate captions without direct training on audio
captioning tasks. For example, Shaharabany et al. [83] uses a multimodal
network with a text description to generate a sentence with an LLM, followed
by a model that assesses the relevance of the sentence to the audio content.
Similarly, Salewski et al. [84] use encoder models to extract relevant
keywords, which then guide an LLM to predict the next token based on a
combination of text predictions and audio-text alignment, assessed by a pre-
trained audio-language model. Another approach of Miao et al. [85] involves a
CLAP model that compares the similarity between inference audio and its
training set, associating them with pre-learned semantic concepts from text
descriptions.

##  III Audio-language datasets

Audio-language datasets play an important role in training models for audio
captioning tasks. However, most existing audio datasets are relatively small
and domain-specific [86, 87]. In annotating these datasets, human authors
often omit background details or common sense relationships [88]. In addition,
researchers face challenges in the time-consuming collection process of
existing audio-language datasets [18].

Some parts of the datasets used in the training of audio models, particularly
those extracted from platforms such as YouTube, exhibit varying levels of
noise and reliability [89]. Clips derived from these platforms are mostly
recorded using consumer devices in real-life environments, and audio events in
these clips can include a range of background sounds, such as human speech,
music, and environmental sounds like wind, which may not be indicated in the
captions. The primary trade-off in using these datasets is balancing dataset
size against data quality. Although larger datasets can be obtained by
including noisier and less curated content, this does not necessarily improve
the performance of language models [90].

In the development of audio-language models, it is not uncommon to explore
datasets originally designed for visual tasks (e.g., [91, 92]). A significant
issue with using these visual-centric datasets is that their captions often
include details that are visually observable but not audible. For example,
captions might describe the colors of clothing, such as “the man is wearing a
blue sweater”. Or sounds in a video may be produced by objects off the screen
[93]. This discrepancy can introduce bias when training audio models, as the
model may learn to predict irrelevant visual information.

Due to these concerns, we have chosen to exclude such datasets from our
primary analysis. However, it should be noted that these datasets could still
be valuable for training purposes in scenarios where the inclusion of visual
descriptions is acceptable or desired, and therefore are reported in the
Appendix -B.

In this section, we first highlight the aspects that distinguish our survey
from other recent related surveys. Next, we describe the methodology used to
identify datasets utilized or potentially applicable in the audio-language
learning domain. Finally, we provide detailed descriptions of the identified
dataset, including their features and use cases.

###  III-A Related work

To the best of our knowledge, there are five other previous surveys within the
audio-language domain. Three focus on automated audio captioning (AAC) [5, 6,
7], one on audio retrieval [8], and one on the usage of LLMs in audio
processing [9].

Mei et al. [5] offer insights into fundamental concepts, as well as training
and evaluation methodologies in the field. Xu et al. [6] also address AAC,
focusing on dataset augmentation, and models with additional guidance.
Moreover, Xu et al. [6] highlights future directions based on current
limitations, such as controllable and diverse captioning. Both surveys
especially tackle training techniques and supplementary strategies for
improvements.

In Nag [7], a literature review is conducted in thesis form on visual and
audio captioning, describing various existing methods, models, popular
datasets, and evaluation metrics, with a particular focus on the various
captioning methods in both domains.

Li et al. [8] provide a review of cross-modal retrieval, including a taxonomy
of machine learning paradigms, mechanisms and models. It also discusses
benchmarks, metrics, performances and future directions. Latif et al. [9]
examine the application of large language models in audio processing,
including speech and music, reviewing recent developments and challenges.

Our survey distinguishes itself by providing a detailed exploration of audio
datasets, alongside a discussion of gaps and deficiencies in these areas. This
focus on audio datasets is driven by the consideration that quality and
diversity are as crucial as the models and training techniques themselves, as
the datasets form the foundation on which these models are trained. Through a
thorough analysis, we identify shortcomings in current research on audio data.
Recognizing what remains unexplored or inadequately covered in existing
datasets helps pinpoint areas where future research can make significant
contributions.

###  III-B Method

The articles and datasets selected for this survey were identified through a
manual review of publications listed in Google Scholar from 2019 to 2023. We
initiated this survey with publications that cited at least one known task and
one dataset within the field. Using the exact search term in Google Scholar
via Publish or Perish 222harzing.com/resources/publish-or-perish, we retrieved
a total of 409 papers: (“Audio Language” OR “Audio Captioning” OR “Audio-Text
Retrieval” OR “Audio-Question Answering” OR “Language-to-Audio” OR “Audio
Generation” OR “Text-to-Audio Retrieval“) AND (“Clotho” OR “AudioCaps” OR
“MACS” OR “Auto-ACD” OR “AudioCaption” OR “WavText5K” OR “SoundDescs” OR
“WavCaps”). For a detailed overview of these datasets, including statistics
and characteristics, refer to Table I.

Please note that this survey was conducted by analyzing research papers up to
and including December 2023, and does not contain any papers after that
period. Consequently, the information presented in this survey may be
outdated.

###  III-C Datasets

We categorize the audio captioning datasets into several groups based on their
sources:

  * •

YouTube-Based: Includes datasets derived from YouTube, such as AudioSet and
AudioCaps, which originates from AudioSet.

  * •

Freesound-Based: Clotho is a primary example, sourced from the Freesound
database.

  * •

Sound-Effect Datasets: Examples include WavText5K and the BBC Sound Effects
Library.

  * •

Other Categories: Encompasses datasets that do not fit the previous
categories, including those based on field recordings and television shows.

  * •

Accumulated Datasets: A compilation of datasets based on the above categories,
referred to as Audio-Language Gathered Datasets and Audio-Speech-Music
Language Gathered Datasets.

Additionally, we explore datasets tailored to specific audio-language tasks
such as Audio-Question-Answering and Audio Difference Captioning.

TABLE I: Comparative overview of 19 audio-language datasets. Only single-source datasets that were possible to obtain were included. Captions = the total number of captions. Length = the average amount of words of the captions with its standard deviation. Words = the unique number of words in these captions. Sounds = the total number of sounds. Hours = the total number of hours that all sounds encompass. Size = an indication of how much storage the dataset requires, which can vary depending on the encoding and properties of the audio. Acc = is the data easily accessible to download by clicking a link (e.g., without scraping)? Source = Who generated the text data: Human means that the data were annotated and curated by humans. Online indicates that the data was scraped from the Internet. GPT implies that the data was generated by a generative model, such as ChatGPT. Generated signals indicates that the data were mechanistically created, e.g. by overlaying two separate audio files on top of each other. Domain = the data source domain from which the data originated. Options are YouTube, FreeSound, television shows, studio / field recordings or sound effects (SFX) libraries. License = the data license. * = CC BY 3.0, CC PD-Mark, CC BY-NC-ND 3.0, CC BY-NC-SA 3.0, CC BY-ND 3.0, CC BY-SA 1.0 Dataset | Captions | Length | Words | Sounds | Hours | Size (GB) | Acc | Source | Domain | License
---|---|---|---|---|---|---|---|---|---|---
AnimalSpeak [94] | 895,190 |  5.89$\pm$5.04 | 42,968 | 878,359 | 11,841.12 | 1,131.02 | No | GPT | Field | CC BY-NC-SA 4.0
AudioCaps [16] | 57,188 |  8.80$\pm$4.30 | 6,623 | 36,706 | 100.55 | 21.58 | No | Human | YouTube | CC BY 4.0, MIT
AudioCaption [95, 96] | 29,117 |  16.24$\pm$7.80 | 8,542 | 7,336 | 21.43 | 12.96 | Yes | Human | TV Shows | MIT
AudioDiffCaps [53] | 37,488 |  7.83$\pm$3.55 | 973 | 7,756 | 21.43 | 12.68 | No | Generated | FreeSound | CC BY-NC-ND
AudioGrounding [64] | 4,750 |  9.80$\pm$4.54 | 1,715 | 4,588 | 12.71 | 2.73 | Yes | Human | YouTube | CC BY 4.0, MIT
AudioSet [15] | 2,084,317 |  2.88$\pm$2.43 | 1,067 | 1,955,135 | 5,383.11 | 1,155.23 | No | Human | YouTube | CC BY 4.0
CAPTDURE [97] | 8,034 |  13.11$\pm$7.51 | 6,195 | 2,088 | 4.9 | 2.67 | Yes | Human | Studio | CC BY-NC-SA 4.0
Clotho [17] | 29,645 |  11.34$\pm$2.78 | 8,168 | 6,972 | 43.64 | 12.91 | Yes | Human | FreeSound | CC BY, CC BY-NC
Clotho-Detail [98] | 3,939 |  6.83$\pm$2.16 | 4,407 | 3,939 | 24.55 | 7.26 | Yes | GPT | FreeSound | BSD 3-clause, CC BY
ClothoAQA [50] | 35,838 |  6.83$\pm$2.16 | 4407 | 1,991 | 12.46 | 3.68 | Yes | Human | FreeSound | CC BY, CC BY-NC, MIT
DAQA [99] | 598,878 |  14.33$\pm$3.73 | 271 | 100,000 | 2,089.36 | 224.17 | No | Generated | Field,YouTube | CC BY 4.0
FAVDBench [100] | 9,643 |  14.30$\pm$8.47 | 6,433 | 10,000 | 21.55 | 13.87 | No | Human | YouTube | CC BY-NC 4.0
FSD50k [86] | 51,197 |  32.60$\pm$38.07 | 86,349 | 51,197 | 108.59 | 32.12 | Yes | Online | FreeSound |  | CC BY, CC BY-NC,
---
CC Sampling+, CC0
MACS [101, 102] | 17,275 |  9.25$\pm$3.89 | 2,775 | 14,400 | 10.92 | 39.56 | Yes | Human | Field | CC BY-NC
mClothoAQA [103] | 203,082 |  6.35$\pm$2.13 | 27,522 | 1,991 | 12.46 | 3.68 | Yes | Generated | FreeSound |  | CC BY, CC BY-NC,
---
CC0 1.0, MIT
MULTIS [104] | 8,835 |  22.48$\pm$8.45 | 5,156 | 7,645 | 21.07 | 4.52 | No | GPT | YouTube | BSD 3-clause
SoundDescs [79] | 32,979 |  15.84$\pm$7.75 | 26,362 | 32,836 | 1,056.63 | 621.77 | No | Online | SFX Libraries | Apache, RemArc
SoundingEarth [105] | 51,058 |  33.30$\pm$29.93 | 156,983 | 51,078 | 3,453.36 | 400.69 | Yes | Online | Field | *
WavText5K [80] | 4,348 |  13.02$\pm$8.76 | 8,096 | 4,525 | 25.48 | 14.43 | No | Online | SFX Libraries | CC BY 3.0, CC0 1.0, MIT

###  III-D YouTube-based datasets

YouTube-based datasets, such as AudioSet and VGGSound, have been instrumental
in the development of audio captioning models. However, using these datasets
comes with specific challenges due to YouTube’s policies. Content creators can
license their videos under the Creative Commons CC BY license
333support.google.com/youtube/answer/2797468, allowing reuse if they have the
rights to the material. Despite this, YouTube’s Terms of Service restrict
content download unless explicitly authorized or permitted by YouTube
444youtube.com/static?template=terms.

As a result, researchers cannot directly share audio files. Instead, YouTube-
derived datasets typically include only video links with the corresponding
start and end timestamps. Researchers also provide scripts for scraping these
videos. Consequently, anyone looking to replicate the models must scrape the
data themselves.

A further complication arises from link rot, where video owners may delete
their videos, or YouTube may remove them for policy violations. This leads to
the unavailability of some portions of the dataset, which complicates efforts
to reproduce research results or train new models.

####  III-D1 AudioSet

Launched by Google in 2017 [15], AudioSet is one of the first large-scale
datasets for audio classification, containing 2,085,544 audio clips, each
approximately 10 seconds in duration. AudioSet features 527 categories of
audio events, each manually annotated using Mechanical Turk (MTurk)
555mturk.com, providing multi-label annotations. For creating audio-language
models, various researchers concatenate the audio event labels together to
form a caption.

The categories are organized into an ontology with top-level labels such as
Human sounds, Source-ambiguous sounds, Animals, Sounds of things, Music,
Natural sounds, and Environmental and background sounds. AudioSet is divided
into balanced and unbalanced training sets, with the balanced set comprising
about 1% of the total and designed to ensure at least 59 examples per
category. The unbalanced set contains the majority of the examples,
particularly dominated by categories such as music and speech.

Due to the transient nature of YouTube content, the exact composition of
AudioSet varies in research, as videos are sometimes removed from the
platform. This variation can affect the size of the dataset used in different
studies by up to $\pm 5\%$, potentially influencing outcomes and reported
metrics such as mean average precision (mAP) [89].

Several other derivative datasets have been created from AudioSet. A direct
derivative is AudioSet Strong [106]. AudioSet Strong provides enhanced
annotations with strong labels, which indicate the exact start and end times
of each sound event within the clips. This subset allows for more precise
training on sound event detection tasks.

####  III-D2 AudioCaps

AudioCaps is a substantial dataset consisting of 57,190 audios, each 10
seconds long, sourced from AudioSet [16]. This dataset is divided into
training, validation, and test sets. Due to the nature of its online source
YouTube, the dataset’s size has diminished over time. As of December 2023, it
includes 45,187 of the original 49,898 training audios, 444 of the 495
validation audios, and 883 of 975 test audios. The validation and test sets
each provide five captions per audio clip, whereas the training set includes
only one caption per audio. These captions were generated through tasks
completed by workers in Amazon Mechanical Turk.

A notable issue within AudioCaps is the presence of duplicate samples. Using
clustering techniques, researchers identified 257 audio clips with repetitive
elements, such as identical introductory audio effects. This redundancy could
potentially affect the training and performance evaluations of models
developed using the dataset [107].

####  III-D3 TextToAudioGrounding

AudioGrounding (v2) [64] is a dataset specifically designed for Text-To-Audio
Grounding, a task that aims to predict the onsets and offsets of sound events
described by natural language (see Section II-B3). Similarly to AudioCaps, it
includes 4,994 audios and 13,985 captions, with one caption per audio in the
training set and five captions in the validation and test sets. The dataset
features caption-oriented sound event tagging for each audio file and provides
segmentation timestamps for each identified sound event. It was created by
initially extracting sound event phrases from AudioSet and subsequently
refining these through manual phrase merging.

####  III-D4 FAVDBench

FAVDBench dataset is a specialized dataset to support the fine-grained audible
video description task (FAVD) [100]. This dataset includes video clips, each
accompanied by a summary, 4-6 visual descriptions, and 1-2 audio-related
descriptions. Although visual descriptions are primarily tailored for visual
understanding and are not ideal for audio-language learning, audio-related
descriptions are specifically crafted to capture sounds and auditory events
that are discernible without visual context.

These videos were sourced from YouTube and are available exclusively under the
Creative Commons license. The dataset was manually annotated in Chinese by a
team of 60 people and subsequently translated into English. The annotation
process for each video involved one primary annotator and two reviewers. In
addition, each sentence in the descriptions was required to comprise more than
five Chinese characters.

####  III-D5 Take It Seriously

The Take It Seriously dataset, compiled by de Gail et al. [108], consists of a
substantial collection of 1.7 million audio-language pairs, extracted from
YouTube videos that are licensed under a Creative Commons license. This
dataset specifically uses captions that are enclosed in square brackets,
parentheses, and asterisks, which are often used to indicate sound
descriptions or actions within video subtitles.

These subtitles underwent a cleaning process using filtering heuristics.
Captions containing pronouns, numeric, and punctuation signs, or any terms
from a custom blacklist, were discarded to enhance the dataset’s relevance and
accuracy for audio-language tasks. In addition, the name of the person within
the captions was standardized to “man”, “woman”, or “person” depending on the
gender of the name that is most commonly associated, based on statistical
analysis. To ensure comprehensive audio coverage, the audio clips were
downloaded with an additional three seconds of leading and trailing audio.

The dataset was used for a DCASE Challenge on Language-Based Audio Retrieval
to improve the author’s previous performance on the same challenge.
Unfortunately, the dataset is not available for download.

####  III-D6 SoundWords Dataset

SoundWords is a joint embedding model of natural language and sounds by Google
[70], trained with contrastive learning using a large set of audio clips with
natural language descriptions. This model is trained on a diverse collection
of audio clips paired with natural language descriptions from three primary
data sources.

  * •

Internet Video Clips: A vast dataset comprising 50 million 10-second audio
clips extracted from random internet videos. The associated text includes the
name and natural language descriptions of knowledge graph entities related to
each video.

  * •

AudioSet: Used in two formats: the standard set featuring 527 sound
categories, and an enhanced version incorporating human-generated natural
language sound descriptions, as well as the AudioCaps dataset for richer
contextual understanding.

  * •

Pro Sound Effects Library: Consists of 110,000 sound event recordings, each
paired with a detailed textual description. This library is particularly
valuable for its high-quality, professionally curated sound effects and the
precise language used to describe them.

Unfortunately, this dataset is not made public. If that were the case, it
would exceed all other datasets in size.

###  III-E Freesound-based datasets

Freesound 666freesound.org is a widely used resource in the audio domain,
where users can upload sound clips they have created or found, attaching
relevant metadata such as filenames, keywords, and descriptions. Originally
developed by Font Corbera et al. [14], this platform hosts a wide range of
audio files. Although user-generated metadata can vary in quality and
structure, often including extraneous information or featuring audio clips of
impractical lengths, the raw data typically require significant preprocessing
to make it suitable for machine learning applications.

One of the main advantages of using Freesound-based datasets is the licensing.
Freesound allows for the broad utilization of its sounds under various
licenses, making it a valuable source for training datasets in audio-related
machine learning tasks. This section will explore several Freesound-derived
datasets, detailing how they have been processed and used for specific
research purposes.

####  III-E1 Clotho

Clotho [17] is one of the most widely used datasets in the field of audio
captioning. It features 3,839 development items, along with 1,045 validation
and 1,045 evaluation audio samples, each ranging from 15 to 30 seconds in
length. Every audio file in Clotho is paired with five captions created using
Amazon Mechanical Turk. In total, the dataset includes approximately 37 hours
of audio, accompanied by 29,645 captions, each consisting of 8 to 20 words.

Additionally, Clotho includes a test set of 1,043 audio samples without
accompanying captions hosted on a separate Zenodo page, specifically designed
for inference purposes. This dataset is also prominently used in the DCASE
challenges for Automated Audio Captioning
777dcase.community/challenge2024/task-automated-audio-captioning.

A study by Xie et al. [109] involved a crowd-sourcing task on MTurk to
evaluate the precision of 200 Clotho captions. Participants were asked to
score how well the sound content matched the given caption and other relevant
captions not attributed to the audio. From this study, which collected 18,204
responses from 340 workers, it was found that approximately 60% of the audio
content in Clotho accurately matches its given caption.

Several researchers have developed revised or extended versions of the Clotho
dataset to address specific challenges and enhance its utility for training
audio captioning models. Key developments include:

  * •

Revised-Clotho [110] : This dataset is a modified version of Clotho, created
to address issues with captions that poorly represent audio content. Using
GPT-3.5, captions were evaluated based on the audio file name tags and revised
if deemed inaccurate.

  * •

Clotho-Detail [98]: Designed for instruction tuning, this dataset comprises
3,938 audio-text pairs, with each caption averaging 52.7 words. It was
generated automatically using GPT-4, which consolidated the five original
captions per audio into a single comprehensive caption in a few-shot setting.

  * •

ClothoV2_GPT [111] : Using GPT-3.5, this version involves generating new
captions for the Clotho dataset. This process used both the keywords and the
original captions to produce five distinct captions for each audio file,
resulting in a total of 96,000 new captions.

####  III-E2 FSD50k

FSD50k [86] is primarily a sound event classification dataset, consisting of
audio samples and event classes. Although it does not include captions,
researchers have adapted it for the training of audio-language models [11].
This adaptation involves concatenating the title and description provided in
the metadata for each audio file. Although this method generates long
descriptions, it introduces a degree of noisiness due to the unstructured
nature of the metadata.

####  III-E3 Freesound 500K

Freesound 500K [112] is a dataset consisting of 500K audio samples sourced
together from the Freesound website. Each sample in the dataset is accompanied
by tags and descriptions. Tang et al. [112] used the dataset together with
AudioSet, AudioCaps, Freesound, BBC Sound Effects, and Soundnet to train a
Audio Diffuser.

###  III-F Sound effect datasets

Sound effects (SFX), originally designed for multimedia enhancements, have
found significant utility in the audio domain. These datasets are invaluable
not just for their primary purpose - adding audio layers to visual media like
comedy series or action movies - but also for their structured captions, which
facilitate quick and accurate retrieval of specific sounds used when searching
for specific content on SFX search websites. This structured metadata is a key
advantage for training audio models as it allows for use as a caption
describing the sound in these datasets.

A notable subset of SFX is the Foley sounds [113, 114]. Foley artists
typically use everyday objects to mimic the sounds associated with actions or
objects in a scene. For example, the sound of breaking bones might be
simulated by snapping celery sticks. This creative process not only produces
similar sound effects but also provides data for training audio models to
recognize and replicate these sounds. However, some Foley sounds could be
sonically different sounds with the same textual representation [115].

In this section, we will explore popular datasets that are based on these SFX
sounds.

####  III-F1 WavText5k

WavText5k [80], is a collection of 4,525 sound effects sourced from two
prominent SFX websites, BigSoundBank 888bigsoundbank.com and SoundBible
999soundbible.com. Unlike other datasets, WavText5k is distinguished by its
captions, which are notably more descriptive and typically focus on a single
event occurring within the audio clip. This specificity makes WavText5k
particularly useful for tasks that require detailed detection and description
of sound events.

WavText5k, created by Microsoft, was originally used for audio retrieval and
the training of a model using contrastive learning. The dataset does not share
the audio files directly. Instead, it provides links to the original sounds
along with detailed metadata for each entry.

####  III-F2 BBC Sound Effects

BBC Sound Effects 101010sound-effects.bbcrewind.co.uk database offers a
collection of 33,066 audio clips spread over 23 categories, such as nature,
transport, and machines. This library features historical sounds from the BBC
Radiophonic Workshop, the Blitz in London, various BBC TV and radio
productions, and recordings from the BBC Natural History Unit archive. The
accompanying captions are noisy and typically begin with nouns that identify
the sound source or scene, followed by more detailed descriptions, which are
often incomplete.

SoundDescs [79] serves as an audio retrieval benchmark that includes 32,979
audio files. It contains a wide range of audio content and vocabulary in the
descriptions. In particular, 109 files exceed 10 minutes in duration. The
dataset is organized into training (70%), validation and test splits (15%
each), covering 23 categories such as nature, clocks, and fire. Recent
evaluations have identified data leakage in SoundDescs, including issues with
duplicates, reprocessed recordings, and partial overlaps, and proposed a new
split of the dataset to eliminate these issues [116].

####  III-F3 Pro Sound Effects

The Pro Sound Effects Library 111111prosoundeffects.com, a commercial
resource, was the first dataset to be used in an Automated Audio Captioning
setting [35]. The library consists of 175,000 sound recordings, spread over
501 categories. After an initial automatic curation process, the dataset was
refined to 152,619 samples, each accompanied by captions ranging from 1 to 60
words. This curated set contains a vocabulary of 11,478 unique words totaling
1.16 million words.

Subsequent studies have used this dataset for specialized applications.
Kilgour et al. [70] utilized 110,000 samples for the text-driven separation of
sounds, a technique that improves the clarity and isolation of specific audio
elements. Furthermore, Miao et al. [85] used samples from this library to
advance research in bioacoustics detection of wildlife.

###  III-G Other audio-language datasets

This section explores a range of audio-language datasets that were not derived
from common sources such as YouTube, Freesound, or standard sound effect
websites. Instead, these datasets originate from unique and specialized
resources. These sources include bioacoustics websites such as AnimalSpeak.
Adaptations of existing datasets from related domains, such as MACS. Content
from other video platforms, including SAM-S and AudioCaption. Self-recorded
datasets, such as CAPTDURE, and collections from other online sources, such as
SoundingEarth.

####  III-G1 MACS

MACS [101, 102], a human-annotated audio dataset, features 3,930 files, each
annotated with an average of 4.4 captions. Created from the TAU Urban Acoustic
Scenes 2019 Dataset [117], MACS focuses specifically on urban environments,
encapsulating three distinct sound categories: airport, public square, and
park sounds. Each audio file is precisely 10 seconds long, with captions
averaging 9.247 words in length, contributed by 133 annotators.

####  III-G2 Hospital & Car Dataset (AudioCaption)

The AudioCaption project encompasses two distinct datasets focused on specific
scenes: one in a hospital setting and the other inside a car. Both datasets
were originally annotated in Mandarin and later translated into English using
automated tools. Due to their shared origins and annotation methodologies,
these datasets are often used in conjunction.

  * •

The Hospital dataset [95], consists of 3,707 audio clips, each 10 seconds in
length. These clips were sourced from Chinese video platforms Youku, Iqiyi and
Tencent, predominantly featuring content from TV shows. The annotation process
involved first asking annotators four questions about the sounds they heard,
the sources of these sounds, their characteristics, and the sound’s location.
Responses to these queries formed the basis for the three captions assigned to
each audio clip, culminating in a total of 11,121 captions. The dataset is
divided into 3,337 training and 371 test clips.

  * •

The Car dataset [96] comprises 3,602 audio clips. Each clip in this dataset is
annotated with five captions, making for a total of 18,010 captions. These
captions were generated by five Mandarin speakers per audio clip.

####  III-G3 AnimalSpeak

The AnimalSpeak dataset [94], is a comprehensive collection of bioacoustic
sounds aimed at improving bioacoustic research. This dataset was initially
compiled using unstructured labels with the assistance of ChatGPT, and further
refined through screening with a language model. Captions that did not pass
the screening were revised and recaptioned to ensure accuracy and relevance.

AnimalSpeak consists of several subsets, each sourced from distinct databases:

  * •

iNaturalist 121212inaturalist.org: This platform offers a crowd-sourced
collection of animal observations, contributing 300,267 samples to the
dataset.

  * •

Xeno-Canto 131313xeno-canto.org: Known for sharing wildlife sounds,
particularly birds, and recently expanding to include grasshoppers and bats,
this website adds 679,676 samples.

  * •

Watkins Marine Mammal Database 141414go.whoi.edu/marine-mammal-sounds: An
open-access repository of marine mammal recordings collected throughout the
career of William Watkins, providing 15,567 samples.

  * •

Animal Sound Archive 151515animalsoundarchive.org: Maintained by the Museum
für Naturkunde Berlin, this database initially contributed 21,473 samples.
Unfortunately, most of this subset was lost due to a cyberattack.

  * •

AudioCaps: see Section III-D2.

Each audio clip in the dataset, except those from AudioCaps, includes two
captions: one referencing the animal species by its common name and the other
by its scientific (Latin) name. In total, AnimalSpeak contains 1,062,983
samples, making it one of the largest bioacoustic datasets available.

####  III-G4 CAPTDURE

CAPTDURE [97], is a manually created dataset comprised of 1,044 audio
recordings captured by researchers in 14 different environments, each
annotated with 3-5 captions to provide a comprehensive description of the
sound. It consists of single-source sounds and multiple-source sounds, each
with captions and appropriateness scores for each caption. Originally crafted
in Japanese and translated into English. The annotation was carried out by
crowd workers who not only provided captions, but also participated in a
quality control process by evaluating the captions submitted by their peers.

####  III-G5 SAM-S

The Subtitle-Aligned Movie Sounds (SAM-S) dataset [118] offers a comprehensive
collection of sound-specific data extracted from closed-caption transcripts of
430 Hollywood movies released between 2014 and 2018. This dataset consists of
116,000 captions that were originally enclosed in brackets, a method used to
denote sound descriptions in film subtitles.

These captions provide detailed information about the sounds, their sources,
and the quality of the audio events depicted in the movies. Of the total
captions, 21,000 were identified as unique, highlighting the diversity of
audio events captured in contemporary films. The breakdown of these captions
reveals a difference in length: 1,500 consist of a single word, 11,000 contain
two words, and the remaining captions extend to three words or more.

####  III-G6 SoundingEarth

The SoundingEarth dataset [105] is a comprehensive collection of 50,454
geotagged ambient sound recordings, collectively accumulating over 3,500 hours
of audio. This dataset is sourced from the Radio Aporee project
161616aporee.org, a community-driven project that aims to archive as many
field recordings around the world as possible, including the location where
the sound was heard.

Each audio file in the dataset is accompanied by the corresponding images from
Google Earth, aligning the visual context with the ambient sounds. This
feature is particularly valuable for studies in audiovisual representation
learning, where synchronization between audio and visual data can lead to more
robust models capable of interpreting environmental contexts. The recordings
vary significantly in length, with a median duration of approximately 3
minutes. However, some of the longest samples in the dataset extend beyond 30
minutes, offering extended captures of environmental sounds.

###  III-H Audio-Question Answering Datasets

Audio-question answering datasets represent a distinct category within the
broader field of audio-language research. Unlike typical audio-language
datasets, which generally consist of audio-text pairs, audio-question
answering datasets are structured as triplets. Each triplet includes an audio
file, a text-based question on the audio, and a corresponding text-based
answer.

This structure supports more interaction models, where the focus is not just
on understanding or transcribing the audio but also on interpreting and
responding to specific queries related to the audio content. The question and
answer components are both textual, and the answers are often designed to fit
within a classification framework. This is due to the limited range of
potential answers which, in some datasets, are as straightforward as yes or
no. For analytical purposes, we have concatenated the question and answer
components into a single text string (see Table I).

####  III-H1 ClothoAQA

ClothoAQA [50], is derived from the Clotho dataset to be specifically used to
answer audio questions. ClothoAQA comprises of 1,991 audio samples, each
accompanied by six questions and three corresponding answers, thus 18
question-answer pairs. The dataset was crowd-sourced using Amazon Mechanical
Turk (MTurk), with each pair of question-answers per audio annotated by a
different worker to ensure diversity and reliability in the responses. The
questions are structured to vary in difficulty: two per audio file are ‘yes-
no’ questions designed for binary responses, while the remaining four require
a single-word answer. The training, validation and test split are 1174, 344
and 473 audio files.

####  III-H2 mClothoAQA

The mClothoAQA [103], expands the audio question-answering framework of
ClothoAQA [50] into a multilingual context. This dataset supports audio
question-answering research in eight languages: English, French, Hindi,
German, Spanish, Italian, Dutch, and Portuguese. It includes 1,991 audio files
and 35,838 question-answers for each of the eight languages, totaling 286,704
pairs across the dataset. mClothoAQA was constructed by automatically
transcribing the English question-answering pairs of the ClothoAQA into seven
other languages using Google’s machine translation API.

####  III-H3 DAQA

The Diagnostic Audio Question Answering (DAQA) dataset [99] by Meta
(Facebook), is crafted to evaluate the temporal reasoning capabilities of
machine learning models engaged in Audio Question Answering (AQA). This
dataset comprises roughly 100,000 audio sequences that are synthesized from
400 distinct natural sound events. In addition, it includes about 600,000
questions and answers generated programmatically. The audio content is sourced
primarily from the AudioSet database, along with recordings made specifically
for this dataset.

####  III-H4 AQUALLM

The AQUALLM dataset [119], comprises three subsets: AQUALLM-AudioCaps,
AQUALLM-Clotho, and AQUALLM-MACS. These subsets were developed using an
automated framework designed to ensure breadth, quality, and diversity in the
content provided. In terms of scale, AQUALLM-AudioCaps includes 728,310
questions and 35,008 answers; AQUALLM-Clotho features 438,600 questions and
25,235 answers; while AQUALLM-MACS offers 268,875 questions and 11,874
answers.

####  III-H5 Audio Conversation 10k (MULTIS)

Zhao et al. [104] introduced the Audio Conversation 10k as part of the MULTIS
datasets, which focuses on multimodal instructional tasks. This particular
dataset is formatted for instruction tuning and includes audio-language pairs
aimed at enhancing the capabilities of the large language model in audio-
question-answering scenarios. An example interaction in this dataset features
a human asking “What is making the sound in the audio?” to which the model
responds, “The sound in the audio is made by a waterfall pouring into a
stream.” All audio clips in this dataset are sourced from AudioSet.

####  III-H6 LTU / OpenAQA-5M

The Listen Think Understand (LTU) model [120], and its speech-inclusive
counterpart, LTI-AS, are two models that focus on audio reasoning and audio
question answering. The LTU model is trained on the OpenAQA-5M dataset, which
combines eight prominent audio datasets into a unified format consisting of
(audio, question, answer) tuples. Specifically, it features 1.9 million
closed-ended and 3.7 million open-ended audio question-answer pairs, the
latter of which are generated using GPT. The datasets integrated into
OpenAQA-5M include AudioSet-Strong, AudioSet, VGGSound, FSD50K, AudioCaps,
FreeSound, Clotho, and Sound Bible.

###  III-I Audio Difference Captioning Datasets

####  III-I1 MIMII-Change

The MIMII-Change dataset [52] is a specialized collection derived from the
MIMII-DG dataset, which focuses on identifying malfunctions in industrial
machines. This dataset comprises 1,500 manually annotated pairs of sounds,
covering five different types of machine sounds. For each type of sound, there
are 300 pairs evenly divided between normal operating sounds and their
anomalous counterparts. The unique aspect of this dataset is the use of
onomatopoeia in the annotations, where the annotators describe the specific
changes in sound between each pair. Consequently, each pair of sounds is
accompanied by three distinct captions.

####  III-I2 AudioDiffCaps

The AudioDiffCaps dataset [53] consists of pairs of audio clips that are
similar but distinct. Each pair includes subtle differences, which are
highlighted through human-annotated descriptions. These pairs were crafted
using the Scaper library [121], a tool for soundscape synthesis and
augmentation, by mixing foreground event sounds with background noises sourced
from environmental sound datasets, namely FSD50K and ESC-50. Each audio clip
within the pairs is precisely 10 seconds in length. This dataset complements
the Audio Pair with Difference dataset introduced earlier by the same
researchers [122].

###  III-J Audio-Language Gathered Datasets

This section focuses on datasets that integrate multiple smaller datasets into
a single resource for research. These gathered datasets often serve as an
important resource for audio-language foundation models, providing a richer
and more diverse set of data points by combining various sources. Included
here are datasets that incorporate elements of the datasets discussed
previously, as well as others that have not been highlighted in individual
contexts. A dataset qualifies for inclusion in this section if it consists
predominantly of subsets that have been independently used in audio-language
model research.

####  III-J1 Auto-ACD

The Auto-ACD dataset [87], is a substantial collection that includes 1.9
million sounds sourced primarily from VGGSound [91] and AudioSet [15]. This
dataset improves the original keywords from these sources using rich
audiovisual information processed through six different models. Specifically,
four algorithms analyze the middle frame of the video as an image: (1) BLIP-2
for image captioning, (2) Grounding Dino for object detection, (3) OpenAI’s
CLIP model, which categorizes images into 1000 ImageNet categories using the
prompt “a photo of a label”, and (4) PlaceCNN for place recognition.

For audio processing, two algorithms are employed: (1) PANNs [41], which tags
audio clips using the AudioSet Ontology and assigns confidence scores to three
tags, and (2) AudioCaption [123], which provides a brief descriptive caption.
These six data points collectively inform a new comprehensive caption for each
sound, generated using ChatGPT (version GPT-3.5).

####  III-J2 ONE-PEACE Pretraining set

ONE-PEACE [124] is a representation model designed to handle a wide range of
modalities, with a particular focus on audio and visual tasks. In its audio-
text segment, the model integrates a substantial compilation of open-source
environmental sound datasets, all of which are publicly accessible. This
includes Epidemic Sound, AudioCaps, AudioSet, AudioStock, Clotho, FreeSound,
MACS, SoundDescs, and WavText5K, collectively forming approximately 2.4
million audio-text pairs. The combined content amounts to about 8,000 hours of
audio. In particular, for the Epidemic Sound and AudioSet datasets, the tags
are transformed into more descriptive captions using T5 augmentation.

####  III-J3 LAION-Audio-630k

The LAION-Audio-630k dataset [12], compiled for training a CLAP model,
includes 633,526 audio clips paired with captions, a total of 4,025.39 hours
of audio. The bulk of LAION-630k is drawn from Freesound, with the remainder
sourced from various sound effect providers, including paid platforms such as
Sonnis Game Effects, We Sound Effects, and Paramount Motion Sound Effects.

####  III-J4 MS-CLAP

Microsoft’s 2022 release of the CLAP model (MS-CLAP22) [11] uses 128,010
audio-text pairs collected from four primary datasets. FSD50K, ClothoV2,
AudioCaps, and MACS. In total, MS-CLAP22 trained a library of 2.2 million
files, which are used in 16 different tasks classified into seven distinct
groups. This version focuses on leveraging a diverse set of audio contexts to
improve the model’s general audio understanding and performance across various
tasks.

In 2023, researchers expanded the CLAP model to encompass a vastly larger and
more varied collection of 4.6 million audio-text pairs, sourced from a mix of
datasets and web archives (MS-CLAP23) [31]. This version aims to train a model
for general-purpose audio representations. The audio content spans a wide
range of sounds and scenarios, including human activities, environmental
noise, acoustic scenes, music, sound effects, speech emotions, and more.
Specific datasets used for MS-CLAP23 training include WavCaps, AudioSet,
FSD50K, Clotho, AudioCaps, MACS, WavText5k, SoundDesc, Nsynth, FMA, MOSI,
MELD, IEMOCAP, MOSEI, MSP-Podcast, CochlScene, LJspeech, EpicKitchen,
Kinectics, and findsounds.co.

####  III-J5 TangoPromptBank

The TangoPromptBank dataset [125] is a diverse collection designed for the
generation of text-to-audio. This dataset aggregates approximately 1.2 million
audio samples, each no longer than 10 seconds, sourced from a diverse set of
repositories. Included are WavCaps, AudioCaps, ESC, UrbanSound, MusicCaps,
GTZAN, and the Musical Instruments dataset. The audio content encompasses a
broad spectrum of sounds, from environmental noises to a wide array of musical
instruments and genres.

####  III-J6 WavCaps

WavCaps [18] is a weakly labeled dataset that brings together audio clips from
four diverse sources: FreeSound, BBC Sound Effects, SoundBible, and AudioSet
(strong labels). The dataset comprises 403,050 audio clips, which vary
significantly in length, from as short as one second to as long as 18 hours,
with an average duration of approximately 67.59 seconds. This variation
showcases the wide range of audio content in the dataset. Each clip is
accompanied by a caption, averaging 7.8 words, which is generated using
ChatGPT from raw descriptions. Additionally, WavCaps includes both
prefiltering and post-processing steps aimed at refining the dataset by
eliminating undesirable data.

####  III-J7 NetEase AAC Dataset

The NetEase AAC Dataset [126] was developed to train audio captioning models.
It aggregates data from multiple sources: FreeSound, Soundbible, SoundJay,
Zapsplat, and AudioCaps, in total 117,227 items. It employs heuristic rules
specifically designed to improve the quality of text annotations. These rules
include removing non-English words or those with special patterns and dropping
text containing numbers and pronouns. A version without AudioCaps was
collected by Han et al. [127].

###  III-K Audio-Speech-Music Language Gathered Datasets

This section is dedicated to datasets that are crucial in training multimodal
models capable of handling a wide range of audio tasks, in the everyday audio,
speech, and music domains. A notable feature often seen of models using these
datasets is the use of prompt prefixes which guide the model response
mechanism. These prompts could specify whether a caption should reflect spoken
content (for speech) or describe the ambient sounds (for non-speech
scenarios), which aids the model in understanding which content is being
processed.

####  III-K1 HEAR benchmark

The Holistic Evaluation of Audio Representations (HEAR) benchmark
171717hearbenchmark.com provides a framework for evaluating the
generalizability of audio embedding approaches [128]. Originally launched as a
NeurIPS competition in 2021, it is designed to assess how well these
embeddings perform across a broad spectrum of downstream tasks without the
need for fine-tuning. The primary objective of the HEAR benchmark is to
facilitate the development of a versatile, general-purpose audio
representation that can be used in various domains, including musical pitches,
human speech, and environmental sounds. Includes the following datasets: DCASE
2016 Task 2, NSynth Pitch, Speech Commands, Beehive States, Beijing Opera
Percussion, CREMA-D, ESC-50, FSD50K, Gunshot Triangulation, GTZAN, LibriCount,
MAESTRO, Mridangam, Vocal Imitations, and VoxLingua107.

####  III-K2 SALMONN

SALMONN is a versatile model designed for tasks such as speech recognition,
audio recognition, and captioning [81]. Benefits from a wide range of datasets
for training. Specifically, it incorporates data from sources including
LibriSpeech, GigaSpeech, CoVoST2-En2Zh, LibriMix, VoxCeleb1 and IEMOCAP for
speech-related tasks; AudioCaps, Clotho, and WavCaps for everyday sound-
related tasks; MusicCaps, MillionSong5 and MusicNet for music-related tasks.
In total, the model is trained on roughly 4400 hours of audio and 2.3 million
samples.

####  III-K3 Comp-A

The CompA-661K dataset [129] is part of the development of an improved version
of the CLAP model, which by 2023 exceeds previous versions in text-to-audio
and audio-to-text tasks, as tested on the AudioCaps and Clotho datasets. This
dataset comprises 661,299 audio-caption pairs, compiled from a variety of
sources. These sources include MACS, ESC-50, SONISS, Musical Instrument,
SoundBible, LibriTTS, Free Sound, Medley-solos, WavText5K, MusicCaps, GTZAN,
FSD50K, BBC, Urbansound8K, CompA-AudioSet, AudioCaps, and Clotho. The creation
of CompA-661K is part of a broader initiative aimed at overcoming the
shortcomings of current benchmarks by introducing CompA, which features two
expert-annotated benchmarks designed to assess compositional reasoning in
audio-language models.

####  III-K4 Qwen-Audio

Qwen-Audio is a large-scale audio-language model, capable of interpreting a
broad spectrum of audio tasks and types without the need for task-specific
fine-tuning [20]. It uses a unified multitask learning framework that
integrates speech, natural sounds, and music. Although the specific datasets
used are not disclosed, the authors provide detailed training data volumes for
each category of audio. Training is segmented into three main categories:
speech, sound, and music and song. Specifically, the speech category comprises
18 tasks with approximately 89,200 hours of audio; the sound category includes
5 tasks with about 16,800 hours of audio; and the music and song category
contains 8 tasks totaling around 40,500 hours of audio.

####  III-K5 Pengi

Pengi is an audio language model that utilizes transfer learning by framing
all audio tasks as challenges to text generation [10]. By simply processing an
audio recording and the corresponding text, it generates free-form text as
output, enabling it to handle various audio-language tasks without the need
for additional fine-tuning or task-specific modifications. It is trained
across a diverse set of datasets, tailoring each task’s label to fit the audio
context. For example, sound event classification might use the prompt “this is
a sound of {event}”. The datasets include AudioSet, FSD50k, CochlScene, MSP
Podcast, CMU MOSI, CMU MOSEI, MELD, NSynth, FMA, AudioCaps, Clotho, ClothoAQA,
WavText5k, SoundDescs, MACS, WavCaps, FreeSound, and FindSound.

Similarly, a related model [85] uses a comprehensive dataset to train CLAP
models specifically for wildlife bioacoustics detection. This model features
an HTS-AT based encoder and leverages over 2.1 million audio-text pairs. In
addition to the datasets used by Pengi, this model also incorporates
Kinetics700, BigSoundBank, SoundBible, IEMOCAP, and ProSoundEffects to enhance
its capabilities.

####  III-K6 Mix-185K

Audiobox, a model for audio generation [130], was trained using a variety of
datasets collectively called the Mix-185K dataset. This dataset is composed of
a diverse collection of audio samples, including 160,000 hours of multilingual
speech, 20,000 hours of music, and 6,000 hours of general sound samples. For
the specific training of Audiobox Speech, the SP-multi-100k dataset was
utilized. Regarding Audiobox Sound, the training involved around 6,000 hours
of audio, of which 150 hours were captioned audio samples (SD-cap-150). The
remaining audio data, referred to as SD-tag-6K, consisted of audio tags that
were used as pseudocaptions by concatenating these tags to simulate captions.

####  III-K7 Uniaudio

Uniaudio is an audio-language model designed to generate a variety of audio
types, such as speech, sounds, music, and singing [131]. It has been trained
in a comprehensive 165,000 hours of audio. This training includes datasets
like LibriLight, LibriTTS, MLS, AISHELL3 and PromptSpeech for speech, while
AudioSet, AudioCaps and WavCaps are used for sound. Several other domains are
included as well, such as the Million Song Dataset for music, OpenCPOP and
OpenSinger for singing, and openSLR26/openSLR28 for room impulse response. The
evaluation of Uniaudio’s performance is conducted using test datasets that
cover a broad spectrum of audio tasks, including LibriSpeech and VCTK for
speech, TUT2017 for noise, Clotho for sound and MusicCaps and M4Singer for
music and singing.

####  III-K8 AudioLDM

AudioLDM 2 is an innovative audio generation model developed to encapsulate a
general representation termed Language of Audio (LOA) [132]. This model
utilizes a latent diffusion process for self-supervised audio generation,
drawing on a diverse set of datasets for training, including AudioSet,
WavCaps, AudioCaps, VGGSound, Free Music Archive, Million Song Dataset,
LJSpeech, and GigaSpeech. Its predecessor, AudioLDM [58], focuses on learning
continuous audio representations from CLAP embeddings. AudioLDM employs a
mixture of datasets for this purpose, specifically AudioSet, AudioCaps,
FreeSound, and BBC Sound Effects.

####  III-K9 CLARA

CLARA is a multilingual audio understanding model that not only processes
audio data but also incorporates visual cues as a data augmentation technique
to enhance its training [30]. The model leverages a wide array of datasets to
support its diverse capabilities. These include spoken language databases such
as Common Voice, MSWC, YouTube Dataset, LibriSpeech, LJSpeech, CMU Arctic, and
Cambridge Dictionary, as well as specialized datasets for emotional or gender
speech recognition tasks such as CREMA-D, EMNS, EmoV DB, and RAVDESS.
Additionally, CLARA utilizes sound databases like AudioCaps, AudioSet, and
VGGSound, FSD50K, MACS, ESC50 and UrbanSound8K and two vocal imitation
datasets VocalSketch and Fine-grained Vocal Imitation Set.

##  IV Audio Dataset Evaluation

In this section, an evaluation of the primary audio datasets in this paper is
presented. First, visualizations are provided based on a large-scale principal
component analysis of audio- and text-embeddings. These visualizations
illustrate the relative distance between audio- and text- content in the
various datasets and their absolute numbers in terms of AudioSet top-level
categories. Second, a quantitative evaluation of data leakage is presented.
Data leakage occurs when information about the evaluation data is accessible
to the machine learning model during training.

###  IV-A Audio Dataset Analysis

For a subset of datasets in sections III-D, III-E, III-F, III-G, III-H and
III-I, where we could access the audio data, we generate visualizations of
CLAP embeddings for both audio and text (Fig. 2), and their absolute count in
the seven top-level categories of AudioSet (Fig. 4).

Figure 2: PCA decomposition of all the datasets on both the audio and the
language. The stars in the visualisations represent the centroid of the
dataset. Only the data from sections III-D, III-E, III-F, III-G, III-H and
III-I that we were able to receive the data of are shown.

(a) Pairwise distances between the audio embeddings

(b) Pairwise distances between the text embeddings

Figure 3: Distances of the datasets based on the average pairwise distance
between the points in the dataset. Figure 4: The distribution of the top-level
categories of AudioSet for both text and corresponding audio. The text is
classified by a LLM into one of 7 top-level categories of AudioSet.

The first set of visualizations (Fig. 2) represents the first and second
principal components of audio (left) and text (right) embeddings obtained
using the laion/larger_clap_general CLAP model hosted on HuggingFace. In this
set, the data points are color-coded based on the datasets to highlight the
acoustic and linguistic variability of each dataset, as well as their
centroids. These centroids are indicated by star glyphs and calculated by
averaging all CLAP embeddings for that dataset after reducing them to two
dimensions. For quantitative analysis of variability, we calculate distance
scores based on the pairwise Euclidean distance between each individual point
in the datasets (see Fig. 3).

From the distance scores, we demonstrate that AnimalSpeak and DAQA differ the
most from the others in terms of their audio embeddings (Fig. 3(a)).
AnimalSpeak, a dataset containing a large number of bioacoustic sounds
including many animal species, exhibits the greatest acoustic variability due
to its many different sounds in nature. DAQA is mechanistically generated and
differs from the rest of the audios due to its small sample dataset size on
which its audios are generated (400 examples). In the text domain, AnimalSpeak
and mAQA differ the most from all others in terms of their text embeddings
(Fig. 3(b)). This might be due to its multilingualism, since AnimalSpeak
contains Latin words for scientific names of animals, and mAQA contains 8
different languages.

In the second set of visualizations (Fig. 4), audios and texts are categorized
to their top-level category in AudioSet to highlight the relative distribution
of sounds in terms of general semantics. For this purpose, each text
corresponding to a sound was processed using the Llama 3 model by Meta
181818llama.meta.com/llama3 (specifically, the meta-llama/Meta-
Llama-3-8B-Instruct model hosted on HuggingFace). This model was prompted to
label each sentence and return one of the seven categories. Captions in which
the LLM was unable to return a category were omitted from this visualization.
In the visualizations (Fig. 4), music, human sounds, and the sound of things
represent the majority of sounds in the datasets, due to the large presence of
speech and music in Audioset.

###  IV-B Duplicate analysis

Previous research has highlighted the issue of duplicate data. In audio-
language research, several studies have highlighted overlaps between datasets,
which is particularly relevant for inference in a zero-shot setting. For
example, when training models on WavCaps [18], some data in FreeSound and
AudioSet were excluded to be able to perform zero-shot evaluation on Clotho
and AudioCaps, since some data from Clotho and AudioCaps also occur in
FreeSound and AudioSet, respectively. Similarly, while training LAION-CLAP
[12], researchers omitted samples from various datasets of sound events due to
overlaps with Clotho and overlaps between the AudioCaps test set and the
AudioSet training set. In the DCASE challenge 2024, a CSV is published
containing forbidden Freesound sound ids to mitigate the overlap between
Freesound and Clotho 191919dcase.community/challenge2024/task-automated-audio-
captioning. Our duplicate analysis is inspired by two papers that focus on
specific audio-language datasets. Bralios et al. [107] examined memorization
of training data in latent diffusion models, discovering numerous duplicated
audio clips in the AudioCaps database using CLAP embeddings and mel-
spectrogram analysis. Here, CLAP embeddings could be prone to covariate shift,
which can cause dissimilarity in values, as some of the datasets in the
comparison are also used in training the model, while others are not. Another
interesting approach was taken by Weck and Serra [116], who analyzed the
SoundDescs dataset [79], detecting duplicates or overlapping recordings, and
created a revised version without duplicates. To find overlaps, they applied
an audio fingerprinting algorithm. This audio fingerprinting algorithm is
$O(n^{2})$ and is infeasible to perform on all the datasets in this survey
paper.

Our approach for duplicate analysis uses mel-spectrograms from the audio files
and processes them to compute cosine similarity scores. This process involves
converting the audio files into mel-spectrograms with a sample rate of 16k, 16
bins, and a 128 ms window with 75% overlap. Files are padded to 10.242
seconds, files shorter than 5 seconds are omitted, so that the resulting mel-
spectrogram for each audio is a vector of 1712 dimensions. The mel-
spectrograms are then converted to decibel-valued embeddings and clipped.
Similarly to Bralios et al. [107], each of the mel-spectrogram is normalized
with a bias term before the distance is calculated with cosine similarity:

$\displaystyle
c(\alpha_{1},\alpha_{2})==\frac{\langle\alpha_{1},\alpha_{2}\rangle}{\|\alpha_{1}\|\cdot\|\alpha_{2}\|}$
(1)
$\displaystyle\hat{c}(\alpha_{1},\alpha_{2})=c(\alpha_{1},\alpha_{2})-\beta\cdot\operatorname{bias}(\alpha_{1})$
(2)
$\displaystyle\operatorname{bias}(\alpha)=\frac{1}{K}\sum_{k=1}^{K}\arg\min_{x^{\prime}_{1},\ldots,x^{\prime}_{k}}\sum_{i=1}^{k}\|\alpha-c(\alpha,\beta^{\prime}_{i})\|$
(3)

Where $\beta=0.5$, $k=5$, $\alpha$ are all audio data and $\beta^{\prime}$ is
a subset of 5000 audios from all audios representing the bias. All values
greater than $\tau=0.5005$ are kept. From this we get a total of 34,953,855
overlaps. We depict the overlap in the form of a heatmap (Fig. 5). In this
heatmap, we show the absolute numbers and the colors as relative percentage:
we divide the number of occurrences by the total product of the audios of
those datasets and multiply the result by 100. Overall, high intra-dataset
similarity is evident between some train and validation sets of the same
dataset, indicating that similar audios in the train set occur in the
validation set of various audio datasets. This behavior is most seen in DAQA,
where audios have been machine-generated, the validation and test split of
AudioGrounding, derivatives of the Clotho dataset, and the SoundDescs splits.
That within SoundDescs there is a large overlap of sounds, it was already
concluded by Weck and Serra [116].

High cross-dataset similarity can be seen across various datasets, such as
datasets that come from the same origin. This includes Clotho with
ClothoDetail and ClothoAQA. Datasets such as AnimalSpeak and Audioset have a
relatively high overlap with other datasets. AudioDiffCaps, MACS have
relatively few similar other datasets. Four datasets are not included, due to
no overlap with any other datasets, those include the FAVDBench training set,
the SoundingEarth training set, and the CAPTDURE single- and multisource
dataset.

Figure 5: Overlap of datasets and its splits based on mel-spectogram distance.
Numbers are absolute, color displayed in relative percentage, with all values
under 1e-04 removed.

For all datasets, we provide subsets of data that do not overlap with other
data in our Github repository 202020https://github.com/GlJS/audio-datasets. We
also provide instructions and easy-to-use bash scripts for downloading each of
the datasets in our analysis.

##  V Discussion

###  V-A Data Quality

The quality of the dataset significantly influences the performance of machine
learning models. However, as others have noted, finding high-quality training
data on public datasets is difficult. Thus, relying on large-scale noisy
datasets may be the only scalable approach [11]. However, as argued in the
AudioCaps paper, a noisy large-scale dataset may induce a distributional shift
[16].Mei et al. [18] showed that adopting a processing pipeline to filter out
noisy data and rewrite raw descriptions into captions led to state-of-the-art
performance using a smaller amount of higher quality data. Another issue is
the combination of datasets. Mei et al. [18] presented a notable performance
decrease in the AudioCaps dataset when the LAION-Audio-630K dataset, which is
larger but presumably of lower quality, was introduced into training. This
phenomenon was reversed by incorporating the WavCaps dataset into training,
resulting in significant improvements in the Clotho and AudioCaps datasets
despite utilizing fewer data.

Several updated datasets have been proposed as a recommendation for use
instead. Instead of using Clotho, Revised-Clotho as presented by Xiao et al.
[110] is a better filtered dataset (see Section • ‣ III-E1). Instead of using
SoundDescs, Weck and Serra [116] propose a better split without any data
leakage (see also III-F2).

###  V-B Bias in Audio Datasets

Various types of bias could arise from using audio-language models. One of the
most critical issues is dataset bias, which can significantly impact the
performance and fairness of these systems. This section explores various
facets of dataset bias, including cross-contamination of train/test splits,
the implications of visually grounded datasets on audio-only models, and
diversity bias.

####  V-B1 Cross-contamination

Cross-contamination occurs when the data used to train a model inadvertently
overlap with the data used to test the model. This can lead to overly
optimistic performance metrics, as the model is essentially being tested on
information it has already seen. In the context of audio-language learning,
this could mean that specific audio clips or closely related variations of
them appear in both the training and testing datasets. In particular, when
training on multiple datasets, it is possible to find data in one dataset that
are also in another dataset’s test set. This contamination can mask the true
generalizability of a model to new audio inputs, leading to models that
perform well on benchmark tests but not in real-world applications. Various
audio-language datasets overlap. For example, 17.6% of AudioCaps and 89% of
Clotho overlap with WavCaps [133].

####  V-B2 Visual-grounded datasets

Another point of issue is the large amount of datasets that are visually
grounded (see section III). These datasets are created with a focus on the
visual aspects of the data, often including labels or annotations related to
visual content. For example, in AudioCaps [16], a prominent Audio-Language
dataset, the corresponding video was available to the annotators and there was
the possibility that the annotators might use the video to write the caption.
With this, there is a risk of introducing bias, as it contains descriptions
that are visual but cannot be heard (i.e. the caption “A female instructor
giving a speech in front of a live audience” in the AudioCaps training set).

In addition, Xu et al. [93] concluded that the noise from visual pivoting in
audio-text alignment leads to limited improvement in various studies [23, 134,
135]. For these reasons, we omitted many datasets with visual contexts.

####  V-B3 Diversity bias

Diversity bias issues may also arise. Audio datasets can have imbalances,
which can potentially introduce biases that may impact underrepresented groups
[136]. Audio-language models trained on non-diverse datasets may perform
poorly on audio inputs in these groups. Similarly, audio retrieval systems
might exhibit lower accuracy when searching for content related to cultures or
languages not well represented in the dataset.

Related to this are audio-language models and datasets in other languages than
English. Almost all datasets consist only of English text, some notable
examples excluded: the AudioCaption dataset [95, 96] and the FAVDBench dataset
[100] both have original Mandarin captions that were translated to English.
Similarly, the CAPTDURE dataset was translated from Japanese to English [97].
Cousin et al. [137] used automatic translation to translate AudioCaps and
Clotho to German, French, and Spanish. However, no original dataset in other
languages exists. Kang et al. [138] build a Korean audio captioning system
training an audio captioning system in English and translating the caption
from English to Korean afterwards. This approach has also been scaled to 181
languages by Noriy et al. [30]. In this work, researchers use a language model
to translate captions into target languages. They found that this multilingual
data augmentation enriches the variability of languages and accents within
their training set, as its model is exposed to far greater linguistic
diversity, which enhances its generalization capabilities.

##  VI Conclusion

In conclusion, this survey on audio-language datasets provides a comprehensive
overview of the current landscape in the field of audio-language models, with
a particular focus on the datasets that are used in training and developing
these models. The survey highlights progress made due to advances in computing
power and the creation of large-scale datasets, such as AudioCaps and Clotho.
These developments helped to train more sophisticated models capable of
understanding a wide range of sounds.

This survey also underscores the importance of more data for training using
contrastive learning approaches, such as CLAP, which are highly effective in
training audio-language models. Data scarcity limits the development and
performance of more effective models in this regard. The need for larger and
more diverse datasets is evident, especially to tackle the problem of having
model biases of underrepresented groups.

This survey has briefly explained various audio-language tasks and the
different training techniques used for audio captioning models. Furthermore,
the survey has provided valuable insight into the design and availability of
datasets, which is an important aspect that is often overlooked in previous
surveys. By including dataset statistics and data overlap analysis, the survey
aims to show researchers how to prevent data leakage and guide future dataset
development efforts.

The survey has also identified gaps within the field, pointing out the lack of
datasets in certain areas and the need for more elaborate tasks beyond speech,
music, and sound event models.

In general, the survey serves as a resource for researchers and practitioners
in the field of audio-language modeling, providing a detailed account of the
datasets available, their applications, and the challenges that need to be
addressed to advance the state-of-the-art.

## Acknowledgment

This work was supported by the Dutch Research Council (NWO 406.20.GO.030 to
Prof. Elia Formisano), the Dutch national e-infrastructure with the support of
the SURF Cooperative using grant no. EINF-6190, Data Science Research
Infrastructure (DSRI; Maastricht University) and the Dutch Province of
Limburg.

## References

  * Yin et al. [2023] S. Yin, C. Fu, S. Zhao, K. Li, X. Sun, T. Xu, and E. Chen, “A Survey on Multimodal Large Language Models,” Jun. 2023.
  * Zhao et al. [2023a] W. X. Zhao, K. Zhou, J. Li, T. Tang, X. Wang, Y. Hou, Y. Min, B. Zhang, J. Zhang, Z. Dong, Y. Du, C. Yang, Y. Chen, Z. Chen, J. Jiang, R. Ren, Y. Li, X. Tang, Z. Liu, P. Liu, J.-Y. Nie, and J.-R. Wen, “A Survey of Large Language Models,” Nov. 2023.
  * Naveed et al. [2024] H. Naveed, A. U. Khan, S. Qiu, M. Saqib, S. Anwar, M. Usman, N. Akhtar, N. Barnes, and A. Mian, “A Comprehensive Overview of Large Language Models,” Apr. 2024.
  * Vaswani et al. [2017] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, “Attention Is All You Need,” Dec. 2017.
  * Mei et al. [2022a] X. Mei, X. Liu, M. D. Plumbley, and W. Wang, “Automated audio captioning: An overview of recent progress and new challenges,” _EURASIP journal on audio, speech, and music processing_ , vol. 2022, no. 1, pp. 1–18, 2022.
  * Xu et al. [2023a] X. Xu, Z. Xie, M. Wu, and K. Yu, “Beyond the Status Quo: A Contemporary Survey of Advances and Challenges in Audio Captioning,” Nov. 2023.
  * Nag [2021] I. Nag, “Systematic Literature Review of Deep Visual and Audio Captioning,” Ph.D. dissertation, Tampere University, Tampere, Finland, 2021.
  * Li et al. [2023a] F. Li, L. Zhu, T. Wang, J. Li, Z. Zhang, and H. T. Shen, “Cross-Modal Retrieval: A Systematic Review of Methods and Future Directions,” Oct. 2023.
  * Latif et al. [2023] S. Latif, M. Shoukat, F. Shamshad, M. Usama, Y. Ren, H. Cuayáhuitl, W. Wang, X. Zhang, R. Togneri, E. Cambria, and B. W. Schuller, “Sparks of Large Audio Models: A Survey and Outlook,” Sep. 2023.
  * Deshmukh et al. [2024] S. Deshmukh, B. Elizalde, R. Singh, and H. Wang, “Pengi: An Audio Language Model for Audio Tasks,” Jan. 2024.
  * Elizalde et al. [2022] B. Elizalde, S. Deshmukh, M. A. Ismail, and H. Wang, “CLAP: Learning Audio Concepts From Natural Language Supervision,” Jun. 2022.
  * Wu et al. [2023a] Y. Wu, K. Chen, T. Zhang, Y. Hui, T. Berg-Kirkpatrick, and S. Dubnov, “Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentation,” Apr. 2023.
  * Anupriya [2023] Anupriya, “Transfer Learning for Audio Processing: A Review of Recent Advances,” _Int. J. Mech. Eng._ , 2023.
  * Font Corbera et al. [2013] F. Font Corbera, G. Roma Trepat, and X. Serra, _Freesound Technical Demo_. ACM Association for Computer Machinery, 2013.
  * Gemmeke et al. [2017] J. F. Gemmeke, D. P. Ellis, D. Freedman, A. Jansen, W. Lawrence, R. C. Moore, M. Plakal, and M. Ritter, “Audio set: An ontology and human-labeled dataset for audio events,” in _2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2017, pp. 776–780.
  * Kim et al. [2019] C. D. Kim, B. Kim, H. Lee, and G. Kim, “AudioCaps: Generating Captions for Audios in The Wild,” in _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , J. Burstein, C. Doran, and T. Solorio, Eds. Minneapolis, Minnesota: Association for Computational Linguistics, Jun. 2019, pp. 119–132.
  * Drossos et al. [2020] K. Drossos, S. Lipping, and T. Virtanen, “Clotho: An Audio Captioning Dataset,” in _ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , May 2020, pp. 736–740.
  * Mei et al. [2023] X. Mei, C. Meng, H. Liu, Q. Kong, T. Ko, C. Zhao, M. D. Plumbley, Y. Zou, and W. Wang, “WavCaps: A ChatGPT-Assisted Weakly-Labelled Audio Captioning Dataset for Audio-Language Multimodal Research,” Mar. 2023.
  * Gong et al. [2023a] Y. Gong, H. Luo, A. H. Liu, L. Karlinsky, and J. Glass, “Listen, think, and understand,” _arXiv preprint arXiv:2305.10790_ , 2023.
  * Chu et al. [2023] Y. Chu, J. Xu, X. Zhou, Q. Yang, S. Zhang, Z. Yan, C. Zhou, and J. Zhou, “Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Models,” Dec. 2023.
  * Kaplan et al. [2020] J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Radford, J. Wu, and D. Amodei, “Scaling Laws for Neural Language Models,” Jan. 2020.
  * Hoffmann et al. [2022] J. Hoffmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford, D. d. L. Casas, L. A. Hendricks, J. Welbl, A. Clark, T. Hennigan, E. Noland, K. Millican, G. van den Driessche, B. Damoc, A. Guy, S. Osindero, K. Simonyan, E. Elsen, J. W. Rae, O. Vinyals, and L. Sifre, “Training Compute-Optimal Large Language Models,” Mar. 2022.
  * Zhao et al. [2022] Y. Zhao, J. Hessel, Y. Yu, X. Lu, R. Zellers, and Y. Choi, “Connecting the Dots between Audio and Text without Parallel Data through Visual Knowledge Transfer,” May 2022.
  * Takeuchi et al. [2020] D. Takeuchi, Y. Koizumi, Y. Ohishi, N. Harada, and K. Kashino, “Effects of Word-frequency based Pre- and Post- Processings for Audio Captioning,” Sep. 2020.
  * Raffel et al. [2023] C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu, “Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer,” Sep. 2023.
  * Chang et al. [2023] E. Chang, P.-J. Lin, Y. Li, S. Srinivasan, G. Le Lan, D. Kant, Y. Shi, F. Iandola, and V. Chandra, “In-Context Prompt Editing for Conditional Audio Generation,” in _ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2023, pp. 5320–5324.
  * Kim et al. [2023a] J. Kim, CB. Jeon, Y. Oh, JH. Bae, and K. Lee, “Overcoming Data Shortage in Audio-Text Multi-modal Retrieval: A Tech Report For DCASE 2023 Challenge,” _Detection and Classification of Acoustic Scenes and Events (DCASE) Challenge, Tech. Rep_ , 2023.
  * Liu et al. [2021] X. Liu, Q. Huang, X. Mei, T. Ko, HL. Tang, MD. Plumbley, and W. Wang, “CL4AC: A contrastive loss for audio captioning,” _Detection and Classification of Acoustic Scenes and Events (DCASE) Challenge, Tech. Rep_ , 2021.
  * Wu et al. [2023b] H.-H. Wu, O. Nieto, J. P. Bello, and J. Salamon, “Audio-Text Models Do Not Yet Leverage Natural Language,” Mar. 2023.
  * Noriy et al. [2023] K. A. Noriy, X. Yang, M. Budka, and J. J. Zhang, “CLARA: Multilingual Contrastive Learning for Audio Representation Acquisition,” Nov. 2023.
  * Elizalde et al. [2024] B. Elizalde, S. Deshmukh, and H. Wang, “Natural Language Supervision for General-Purpose Audio Representations,” Feb. 2024.
  * Mehrish et al. [2023] A. Mehrish, N. Majumder, R. Bharadwaj, R. Mihalcea, and S. Poria, “A review of deep learning techniques for speech processing,” _Information Fusion_ , vol. 99, p. 101869, Nov. 2023.
  * Prabhavalkar et al. [2024] R. Prabhavalkar, T. Hori, T. N. Sainath, R. Schlüter, and S. Watanabe, “End-to-End Speech Recognition: A Survey,” _IEEE/ACM Trans. Audio Speech Lang. Process._ , vol. 32, pp. 325–351, 2024.
  * Abhyankar et al. [2023] S. G. Abhyankar, S. S. Bharadwaj, G. S. Rani, P. G. Karigiri, S. Srikanth, and S. Gurugopinath, “A Survey on Music Genre Classification Using Multimodal Information Processing and Retrieval,” in _2023 International Conference on Recent Trends in Electronics and Communication (ICRTEC)_ , Feb. 2023, pp. 1–6.
  * Drossos et al. [2017] K. Drossos, S. Adavanne, and T. Virtanen, “Automated audio captioning with recurrent neural networks,” in _2017 IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA)_ , Oct. 2017, pp. 374–378.
  * Koh et al. [2021] A. Koh, F. Xue, and E. S. Chng, “Automated Audio Captioning using Transfer Learning and Reconstruction Latent Space Similarity Regularization,” Aug. 2021.
  * Mei et al. [2021a] X. Mei, X. Liu, Q. Huang, M. D. Plumbley, and W. Wang, “Audio Captioning Transformer,” Jul. 2021.
  * Won et al. [2021] H. Won, B. Kim, IY. Kwak, and C. Lim, “Transfer learning followed by Transformer for automated audio captioning,” _Detection and Classification of Acoustic Scenes and Events (DCASE) Challenge, Tech. Rep_ , 2021.
  * Mei et al. [2021b] X. Mei, Q. Huang, X. Liu, G. Chen, J. Wu, Y. Wu, J. Zhao, S. Li, T. Ko, H. L. Tang, X. Shao, M. D. Plumbley, and W. Wang, “An Encoder-Decoder Based Audio Captioning System With Transfer and Reinforcement Learning,” Aug. 2021.
  * Kadlčík et al. [2023] M. Kadlčík, A. Hájek, J. Kieslich, and R. Winiecki, “A Whisper transformer for audio captioning trained with synthetic captions and transfer learning,” May 2023.
  * Kong et al. [2020] Q. Kong, Y. Cao, T. Iqbal, Y. Wang, W. Wang, and M. D. Plumbley, “PANNs: Large-Scale Pretrained Audio Neural Networks for Audio Pattern Recognition,” Aug. 2020.
  * Gong et al. [2021] Y. Gong, Y.-A. Chung, and J. Glass, “AST: Audio Spectrogram Transformer,” Jul. 2021.
  * Radford et al. [2019] A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever, “Language Models are Unsupervised Multitask Learners,” _OpenAI blog_ , vol. 1, no. 8, p. 9, 2019.
  * Lewis et al. [2019] M. Lewis, Y. Liu, N. Goyal, M. Ghazvininejad, A. Mohamed, O. Levy, V. Stoyanov, and L. Zettlemoyer, “BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension,” Oct. 2019.
  * Bai et al. [2023] J. Bai, H. Yin, M. Wang, D. Shi, WS. Gan, J. Chen, and S. Rahardja, “AudioLog: LLMs-Powered long audio logging with hybrid token-semantic contrastive learning,” _arXiv preprint arXiv:2311.12371_ , 2023.
  * Mei et al. [2022b] X. Mei, X. Liu, J. Sun, M. D. Plumbley, and W. Wang, “On Metric Learning for Audio-Text Cross-Modal Retrieval,” Jun. 2022.
  * Luo et al. [2023] K. Luo, X. Zhang, J. Wang, H. Li, N. Cheng, and J. Xiao, “Contrastive Latent Space Reconstruction Learning for Audio-Text Retrieval,” Sep. 2023.
  * Li et al. [2023b] G. Li, Y. Xu, and D. Hu, “Multi-Scale Attention for Audio Question Answering,” _arXiv preprint arXiv:2305.17993_ , 2023.
  * Sudarsanam and Virtanen [2023] P. Sudarsanam and T. Virtanen, “Attention-Based Methods For Audio Question Answering,” May 2023.
  * Lipping et al. [2022] S. Lipping, P. Sudarsanam, K. Drossos, and T. Virtanen, “Clotho-AQA: A Crowdsourced Dataset for Audio Question Answering,” Jun. 2022.
  * Bhosale et al. [2022] S. Bhosale, R. Chakraborty, and S. K. Kopparapu, “Text-to-Audio Grounding Based Novel Metric for Evaluating Audio Caption Similarity,” Oct. 2022.
  * Tsubaki et al. [2023] S. Tsubaki, Y. Kawaguchi, T. Nishida, K. Imoto, Y. Okamoto, K. Dohi, and T. Endo, “Audio-change captioning to explain machine-sound anomalies,” in _Proceedings of the 8th Detection and Classification of Acoustic Scenes and Events 2023 Workshop (DCASE2023)_ , Tampere, Finland, Sep. 2023, pp. 201–205.
  * Takeuchi et al. [2023] D. Takeuchi, Y. Ohishi, D. Niizumi, N. Harada, and K. Kashino, “Audio Difference Captioning Utilizing Similarity-Discrepancy Disentanglement,” Aug. 2023.
  * Komatsu et al. [2023] T. Komatsu, Y. Fujita, K. Takeda, and T. Toda, “Audio difference learning for audio captioning,” _arXiv preprint arXiv:2309.08141_ , 2023.
  * Kreuk et al. [2023] F. Kreuk, G. Synnaeve, A. Polyak, U. Singer, A. Défossez, J. Copet, D. Parikh, Y. Taigman, and Y. Adi, “AudioGen: Textually Guided Audio Generation,” Mar. 2023.
  * Borsos et al. [2023] Z. Borsos, R. Marinier, D. Vincent, E. Kharitonov, O. Pietquin, M. Sharifi, D. Roblek, O. Teboul, D. Grangier, M. Tagliasacchi, and N. Zeghidour, “AudioLM: A Language Modeling Approach to Audio Generation,” Jul. 2023.
  * Yang et al. [2022] D. Yang, J. Yu, H. Wang, W. Wang, C. Weng, Y. Zou, and D. Yu, “Diffsound: Discrete Diffusion Model for Text-to-sound Generation,” Jul. 2022.
  * Liu et al. [2023a] H. Liu, Z. Chen, Y. Yuan, X. Mei, X. Liu, D. Mandic, W. Wang, and M. D. Plumbley, “AudioLDM: Text-to-Audio Generation with Latent Diffusion Models,” Sep. 2023.
  * Oncescu et al. [2021] A.-M. Oncescu, A. S. Koepke, J. F. Henriques, Z. Akata, and S. Albanie, “Audio Retrieval with Natural Language Queries,” Jul. 2021.
  * Wu et al. [2022a] Y. Wu, T. Zhang, and K. Chen, _Text-to-Audio Retrieval via Large-Scale Contrastive Training_. dcase.community, 2022.
  * Lai et al. [2022] Y. Lai, J. Pan, and B. Chen, “A RESNET-BASED CLIP TEXT-TO-AUDIO RETRIEVAL SYSTEM FOR DCASE CHALLENGE 2022 TASK 6B,” _Detection and Classification of Acoustic Scenes and Events (DCASE) Challenge, Tech. Rep_ , 2022.
  * Xie et al. [2022] H. Xie, S. Lipping, and T. Virtanen, “Language-based Audio Retrieval Task in DCASE 2022 Challenge,” Sep. 2022.
  * Pellegrini [2022] T. Pellegrini, “Language-based audio retrieval with textual embeddings of tag names,” _Detection and Classification of Acoustic Scenes and Events (DCASE) Challenge, Tech. Rep_ , Nov. 2022.
  * Xu et al. [2021] X. Xu, H. Dinkel, M. Wu, and K. Yu, “Text-to-Audio Grounding: Building Correspondence Between Captions and Sound Events,” Feb. 2021.
  * Xu et al. [2024] X. Xu, Z. Ma, M. Wu, and K. Yu, “Towards Weakly Supervised Text-to-Audio Grounding,” Jan. 2024.
  * Wang et al. [2023a] H. Wang, J. Mao, Z. Guo, J. Wan, H. Liu, and X. Wang, “Furnishing Sound Event Detection with Language Model Abilities,” Aug. 2023.
  * Xu et al. [2023b] X. Xu, Y. Zhang, Y. Yang, and W. Tu, “Leveraging sound local and global features for language-queried target sound extraction,” _International Conference on Neural Information Processing_ , 2023.
  * Liu et al. [2023b] X. Liu, Q. Kong, Y. Zhao, H. Liu, Y. Yuan, Y. Liu, R. Xia, Y. Wang, M. D. Plumbley, and W. Wang, “Separate Anything You Describe,” Oct. 2023.
  * Liu et al. [2022] X. Liu, H. Liu, Q. Kong, X. Mei, J. Zhao, Q. Huang, M. D. Plumbley, and W. Wang, “Separate What You Describe: Language-Queried Audio Source Separation,” Mar. 2022.
  * Kilgour et al. [2022] K. Kilgour, B. Gfeller, Q. Huang, A. Jansen, S. Wisdom, and M. Tagliasacchi, “Text-driven separation of arbitrary sounds,” _arXiv preprint arXiv:2204.05738_ , 2022.
  * Dong et al. [2023] H.-W. Dong, N. Takahashi, Y. Mitsufuji, J. McAuley, and T. Berg-Kirkpatrick, “CLIPSep: Learning Text-queried Sound Separation with Noisy Unlabeled Videos,” Mar. 2023.
  * Gontier et al. [2021] F. Gontier, R. Serizel, and C. Cerisara, “Automated audio captioning by fine-tuning bart with audioset tags,” in _DCASE 2021 - 6th Workshop on Detection and Classification of Acoustic Scenes and Events_ , Nov. 2021.
  * Koizumi et al. [2020] Y. Koizumi, Y. Ohishi, D. Niizumi, D. Takeuchi, and M. Yasuda, “Audio Captioning using Pre-Trained Large-Scale Language Model Guided by Audio-based Similar Caption Retrieval,” Dec. 2020.
  * Chen et al. [2020a] K. Chen, Y. Wu, Z. Wang, X. Zhang, F. Nian, S. Li, and X. Shao, “Audio captioning based on transformer and pre-trained CNN.” in _DCASE_ , 2020, pp. 21–25.
  * Zaman et al. [2023] K. Zaman, M. Sah, C. Direkoglu, and M. Unoki, “A Survey of Audio Classification Using Deep Learning,” _IEEE Access_ , vol. 11, pp. 106 620–106 649, 2023.
  * Schaumlöffel et al. [2023] T. Schaumlöffel, M. G. Vilas, and G. Roig, “PEACS: Prefix encoding for auditory caption synthesis,” in _Proc. Conf. Detection Classification Acoust. Scenes Events Challenge_ , 2023, pp. 1–3.
  * Kim et al. [2023b] M. Kim, K. Sung-Bin, and T.-H. Oh, “Prefix Tuning for Automated Audio Captioning,” in _ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , Jun. 2023, pp. 1–5.
  * Ghosh et al. [2023a] S. Ghosh, S. Kumar, C. K. R. Evuru, R. Duraiswami, and D. Manocha, “RECAP: Retrieval-Augmented Audio Captioning,” Sep. 2023.
  * Koepke et al. [2023] A. S. Koepke, A.-M. Oncescu, J. F. Henriques, Z. Akata, and S. Albanie, “Audio Retrieval with Natural Language Queries: A Benchmark Study,” _IEEE Trans. Multimedia_ , vol. 25, pp. 2675–2685, 2023.
  * Deshmukh et al. [2022] S. Deshmukh, B. Elizalde, and H. Wang, “Audio Retrieval with WavText5K and CLAP Training,” Sep. 2022.
  * Tang et al. [2023a] C. Tang, W. Yu, G. Sun, X. Chen, T. Tan, W. Li, L. Lu, Z. Ma, and C. Zhang, “SALMONN: Towards Generic Hearing Abilities for Large Language Models,” Oct. 2023.
  * Kouzelis and Katsouros [2023] T. Kouzelis and V. Katsouros, “Weakly-supervised Automated Audio Captioning via text only training,” Sep. 2023.
  * Shaharabany et al. [2023] T. Shaharabany, A. Shaulov, and L. Wolf, “Zero-Shot Audio Captioning via Audibility Guidance,” Sep. 2023.
  * Salewski et al. [2023] L. Salewski, S. Fauth, A. S. Koepke, and Z. Akata, “Zero-shot audio captioning with audio-language model guidance and audio context keywords,” Nov. 2023.
  * Miao et al. [2023] Z. Miao, B. Elizalde, S. Deshmukh, J. Kitzes, H. Wang, R. Dodhia, and JM. Lavista Ferres, “Zero-shot transfer for wildlife bioacoustics detection,” 2023.
  * Fonseca et al. [2022] E. Fonseca, X. Favory, J. Pons, F. Font, and X. Serra, “FSD50K: An Open Dataset of Human-Labeled Sound Events,” Apr. 2022.
  * Sun et al. [2023a] L. Sun, X. Xu, M. Wu, and W. Xie, “A Large-scale Dataset for Audio-Language Representation Learning,” Oct. 2023.
  * Betker et al. [2023] J. Betker, G. Goh, L. Jing, T. Brooks, J. Wang, L. Li, L. Ouyang, J. Zhuang, J. Lee, Y. Guo, W. Manassra, P. Dhariwal, C. Chu, Y. Jiao, and A. Ramesh, “Improving Image Generation with Better Captions,” _OpenAI blog_ , vol. 2, no. 3, p. 8, 2023.
  * Li [2023] J. Li, “Towards robust large-scale Audio/Visual learning,” Ph.D. dissertation, Carnegie Mellon University, Pittsburgh, 2023.
  * Anil et al. [2023] R. Anil, A. M. Dai, O. Firat, M. Johnson, D. Lepikhin, A. Passos, S. Shakeri, E. Taropa, P. Bailey, Z. Chen, E. Chu, J. H. Clark, L. E. Shafey, Y. Huang, K. Meier-Hellstern, G. Mishra, E. Moreira, M. Omernick, K. Robinson, S. Ruder, Y. Tay, K. Xiao, Y. Xu, Y. Zhang, G. H. Abrego, J. Ahn, J. Austin, P. Barham, J. Botha, J. Bradbury, S. Brahma, K. Brooks, M. Catasta, Y. Cheng, C. Cherry, C. A. Choquette-Choo, A. Chowdhery, C. Crepy, S. Dave, M. Dehghani, S. Dev, J. Devlin, M. Díaz, N. Du, E. Dyer, V. Feinberg, F. Feng, V. Fienber, M. Freitag, X. Garcia, S. Gehrmann, L. Gonzalez, G. Gur-Ari, S. Hand, H. Hashemi, L. Hou, J. Howland, A. Hu, J. Hui, J. Hurwitz, M. Isard, A. Ittycheriah, M. Jagielski, W. Jia, K. Kenealy, M. Krikun, S. Kudugunta, C. Lan, K. Lee, B. Lee, E. Li, M. Li, W. Li, Y. Li, J. Li, H. Lim, H. Lin, Z. Liu, F. Liu, M. Maggioni, A. Mahendru, J. Maynez, V. Misra, M. Moussalem, Z. Nado, J. Nham, E. Ni, A. Nystrom, A. Parrish, M. Pellat, M. Polacek, A. Polozov, R. Pope, S. Qiao, E. Reif, B. Richter, P. Riley, A. C. Ros, A. Roy, B. Saeta, R. Samuel, R. Shelby, A. Slone, D. Smilkov, D. R. So, D. Sohn, S. Tokumine, D. Valter, V. Vasudevan, K. Vodrahalli, X. Wang, P. Wang, Z. Wang, T. Wang, J. Wieting, Y. Wu, K. Xu, Y. Xu, L. Xue, P. Yin, J. Yu, Q. Zhang, S. Zheng, C. Zheng, W. Zhou, D. Zhou, S. Petrov, and Y. Wu, “PaLM 2 Technical Report,” Sep. 2023.
  * Chen et al. [2020b] H. Chen, W. Xie, A. Vedaldi, and A. Zisserman, “VGGSound: A Large-scale Audio-Visual Dataset,” Sep. 2020.
  * Sun et al. [2023b] L. Sun, X. Xu, M. Wu, and W. Xie, “A Large-scale Dataset for Audio-Language Representation Learning,” Oct. 2023.
  * Xu et al. [2023c] X. Xu, Z. Zhang, Z. Zhou, P. Zhang, Z. Xie, M. Wu, and K. Q. Zhu, “BLAT: Bootstrapping Language-Audio Pre-training based on AudioSet Tag-guided Synthetic Data,” Mar. 2023.
  * Robinson et al. [2023] D. Robinson, A. Robinson, and L. Akrapongpisak, “Transferable Models for Bioacoustics with Human Language Supervision,” Aug. 2023.
  * Wu et al. [2019] M. Wu, H. Dinkel, and K. Yu, “Audio Caption: Listen and Tell,” in _ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , May 2019, pp. 830–834.
  * Xu et al. [2020] X. Xu, H. Dinkel, M. Wu, and K. Yu, “Audio Caption in a Car Setting with a Sentence-Level Loss,” Oct. 2020.
  * Okamoto et al. [2023] Y. Okamoto, K. Shimonishi, K. Imoto, K. Dohi, S. Horiguchi, and Y. Kawaguchi, “CAPTDURE: Captioned Sound Dataset of Single Sources,” May 2023.
  * Zhao et al. [2023b] Y. Zhao, Z. Lin, D. Zhou, Z. Huang, J. Feng, and B. Kang, “BuboGPT: Enabling Visual Grounding in Multi-Modal LLMs,” Jul. 2023.
  * Fayek and Johnson [2019] H. M. Fayek and J. Johnson, “Temporal Reasoning via Audio Question Answering,” Nov. 2019.
  * Shen et al. [2023] X. Shen, D. Li, J. Zhou, Z. Qin, B. He, X. Han, A. Li, Y. Dai, L. Kong, M. Wang, Y. Qiao, and Y. Zhong, “Fine-grained Audible Video Description,” Mar. 2023.
  * Martin-Morato and Mesaros [2021] I. Martin-Morato and A. Mesaros, “What is the ground truth? Reliability of multi-annotator data for audio tagging,” Apr. 2021.
  * Martin Morato and Mesaros [2021] I. Martin Morato and A. Mesaros, _Diversity and Bias in Audio Captioning Datasets_. DCASE, Nov. 2021.
  * Behera et al. [2023a] S. R. Behera, P. B. Reddy, A. M. Tripathi, M. B. Rathod, and T. Karavadi, “Towards Multi-Lingual Audio Question Answering,” in _INTERSPEECH 2023_. ISCA, Aug. 2023, pp. 356–360.
  * Zhao et al. [2023c] Z. Zhao, L. Guo, T. Yue, S. Chen, S. Shao, X. Zhu, Z. Yuan, and J. Liu, “ChatBridge: Bridging modalities with large language model as a language catalyst,” _arXiv preprint arXiv:2305.16103_ , 2023.
  * Heidler et al. [2021] K. Heidler, L. Mou, D. Hu, P. Jin, G. Li, C. Gan, J.-R. Wen, and X. X. Zhu, “Self-supervised Audiovisual Representation Learning for Remote Sensing Data,” Aug. 2021.
  * Hershey et al. [2021] S. Hershey, D. P. W. Ellis, E. Fonseca, A. Jansen, C. Liu, R. C. Moore, and M. Plakal, “The Benefit Of Temporally-Strong Labels In Audio Event Classification,” May 2021.
  * Bralios et al. [2023] D. Bralios, G. Wichern, FG. Germain, Z. Pan, S. Khurana, Hori, and J. Le Roux, “Generation or replication: Auscultating audio latent diffusion models,” _arXiv preprint arXiv:2310.10604_ , 2023.
  * de Gail et al. [2023] T. L. de Gail, B. Zgórzynski, A. Ples, and K. Górzynski, “Take it seriously: Improving on last year with all sorts of tricks,” _Detection and Classification of Acoustic Scenes and Events (DCASE) Challenge, Tech. Rep_ , 2023.
  * Xie et al. [2023] H. Xie, K. Khorrami, O. Räsänen, and T. Virtanen, “Crowdsourcing and Evaluating Text-Based Audio Retrieval Relevances,” Aug. 2023.
  * Xiao et al. [2023] F. Xiao, Q. Zhu, H. Lan, W. Wang, and J. Guan, “Ensemble systems with contrastive language-audio pretraining and attention-based audio features for audio captioning and retrieval,” _etection Classification Acoust. Scenes Events Challenge, Tech. Rep_ , 2023.
  * Primus et al. [2023] P. Primus, K. Koutini, and G. Widmer, “Advancing Natural-Language Based Audio Retrieval with PaSST and Large Audio-Caption Data Sets,” Aug. 2023.
  * Tang et al. [2023b] Z. Tang, Z. Yang, C. Zhu, M. Zeng, and M. Bansal, “Any-to-Any Generation via Composable Diffusion,” May 2023.
  * De Götzen et al. [2013] A. De Götzen, E. Sikström, F. Grani, and S. Serafin, “Real, foley or synthetic? An evaluation of everyday walking sounds,” _Proceedings of SMC_ , 2013.
  * Ament [2014] V. T. Ament, _The Foley Grail: The Art of Performing Sound for Film, Games, and Animation_. Routledge, 2014.
  * Wilkins et al. [2023] J. Wilkins, J. Salamon, M. Fuentes, J. P. Bello, and O. Nieto, “Bridging High-Quality Audio and Video via Language for Sound Effects Retrieval from Visual Queries,” Aug. 2023.
  * Weck and Serra [2023] B. Weck and X. Serra, “Data leakage in cross-modal retrieval training: A case study,” in _ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , Jun. 2023, pp. 1–5.
  * Heittola et al. [2019] T. Heittola, A. Mesaros, and T. Virtanen, “TAU Urban Acoustic Scenes 2019, Development dataset,” Mar. 2019.
  * Hebbar et al. [2023] R. Hebbar, D. Bose, K. Somandepalli, V. Vijai, and S. Narayanan, “A dataset for Audio-Visual Sound Event Detection in Movies,” Feb. 2023.
  * Behera et al. [2023b] S. R. Behera, K. M. Injeti, J. S. K. Patibandla, P. K. Pokala, and B. R. Pailla, “AQUALLM: Audio Question Answering Data Generation Using Large Language Models,” Dec. 2023.
  * Gong et al. [2023b] Y. Gong, A. H. Liu, H. Luo, L. Karlinsky, and J. Glass, “Joint Audio and Speech Understanding,” Dec. 2023.
  * Salamon et al. [2017] J. Salamon, D. MacConnell, M. Cartwright, P. Li, and J. P. Bello, “Scaper: A library for soundscape synthesis and augmentation,” in _2017 IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA)_. IEEE, 2017, pp. 344–348.
  * Takeuchi et al. [2022] D. Takeuchi, Y. Ohishi, D. Niizumi, N. Harada, and K. Kashino, “Introducing Auxiliary Text Query-modifier to Content-based Audio Retrieval,” Jul. 2022.
  * Xu et al. [2022] X. Xu, Z. Xie, M. Wu, and K. Yu, “The SJTU system for DCASE2022 challenge task 6: Audio captioning with audiotext retrieval pre-training,” _Tech. Rep., DCASE2022 Challenge_ , 2022.
  * Wang et al. [2023b] P. Wang, S. Wang, J. Lin, S. Bai, X. Zhou, J. Zhou, X. Wang, and C. Zhou, “ONE-PEACE: Exploring One General Representation Model Toward Unlimited Modalities,” May 2023.
  * Ghosal et al. [2023] D. Ghosal, N. Majumder, A. Mehrish, and S. Poria, “Text-to-Audio Generation using Instruction-Tuned LLM and Latent Diffusion Model,” May 2023.
  * Yuan et al. [2021] W. Yuan, Q. Han, D. Liu, X. Li, and Z. Yang, _The DCASE 2021 Challenge Task 6 System: Automated Audio Captioning with Weakly Supervised Pre-Traing and Word Selection Methods_. dcase.community, 2021.
  * Han et al. [2021] Q. Han, W. Yuan, D. Liu, X. Li, and Z. Yang, “Automated audio captioning with weakly supervised pre-training and word selection methods.” in _DCASE_ , 2021, pp. 6–10.
  * Turian et al. [2022] J. Turian, J. Shier, H. R. Khan, B. Raj, B. W. Schuller, C. J. Steinmetz, C. Malloy, G. Tzanetakis, G. Velarde, K. McNally, M. Henry, N. Pinto, C. Noufi, C. Clough, D. Herremans, E. Fonseca, J. Engel, J. Salamon, P. Esling, P. Manocha, S. Watanabe, Z. Jin, and Y. Bisk, “HEAR: Holistic Evaluation of Audio Representations,” May 2022.
  * Ghosh et al. [2023b] S. Ghosh, A. Seth, S. Kumar, U. Tyagi, C. K. Evuru, S. Ramaneswaran, S. Sakshi, O. Nieto, R. Duraiswami, and D. Manocha, “CompA: Addressing the Gap in Compositional Reasoning in Audio-Language Models,” Oct. 2023.
  * Vyas et al. [2023] A. Vyas, B. Shi, M. Le, A. Tjandra, Y.-C. Wu, B. Guo, J. Zhang, X. Zhang, R. Adkins, W. Ngan, J. Wang, I. Cruz, B. Akula, A. Akinyemi, B. Ellis, R. Moritz, Y. Yungster, A. Rakotoarison, L. Tan, C. Summers, C. Wood, J. Lane, M. Williamson, and W.-N. Hsu, “Audiobox: Unified Audio Generation with Natural Language Prompts,” Dec. 2023.
  * Yang et al. [2023] D. Yang, J. Tian, X. Tan, R. Huang, S. Liu, X. Chang, J. Shi, S. Zhao, J. Bian, X. Wu, Z. Zhao, S. Watanabe, and H. Meng, “UniAudio: An Audio Foundation Model Toward Universal Audio Generation,” Dec. 2023.
  * Liu et al. [2023c] H. Liu, Q. Tian, Y. Yuan, X. Liu, X. Mei, Q. Kong, Y. Wang, W. Wang, Y. Wang, and M. D. Plumbley, “AudioLDM 2: Learning Holistic Audio Generation with Self-supervised Pretraining,” Sep. 2023.
  * Labbé et al. [2023] É. Labbé, T. Pellegrini, and J. Pinquier, “CoNeTTE: An efficient Audio Captioning system leveraging multiple datasets with Task Embedding,” Sep. 2023.
  * Guzhov et al. [2021] A. Guzhov, F. Raue, J. Hees, and A. Dengel, “AudioCLIP: Extending CLIP to Image, Text and Audio,” Jun. 2021.
  * Wu et al. [2022b] H.-H. Wu, P. Seetharaman, K. Kumar, and J. P. Bello, “Wav2CLIP: Learning Robust Audio Representations From CLIP,” Feb. 2022.
  * Roman et al. [2023] R. S. Roman, Y. Adi, A. Deleforge, R. Serizel, G. Synnaeve, and A. Défossez, “From Discrete Tokens to High-Fidelity Audio Using Multi-Band Diffusion,” in _Thirty-Seventh Conference on Neural Information Processing Systems (2023)_ , Nov. 2023.
  * Cousin et al. [2023] M. Cousin, É. Labbé, and T. Pellegrini, “Multilingual Audio Captioning using machine translated data,” Sep. 2023.
  * Kang et al. [2020] T. Kang, J. Kim, and J. Lee, “Development of korean audio caption system,” in _Proceedings of the Korean Society of Broadcast Engineers Conference_. The Korean Institute of Broadcast and Media Engineers, 2020, pp. 364–367.
  * Krishna et al. [2017] R. Krishna, K. Hata, F. Ren, L. Fei-Fei, and J. C. Niebles, “Dense-Captioning Events in Videos,” May 2017.
  * Chen and Dolan [2011] D. L. Chen and W. B. Dolan, “Collecting highly parallel data for paraphrase evaluation,” in _Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics (ACL-2011)_ , Portland, OR, Jun. 2011.
  * Xu et al. [2016] J. Xu, T. Mei, T. Yao, and Y. Rui, “MSR-VTT: A Large Video Description Dataset for Bridging Video and Language,” in _2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_. Las Vegas, NV, USA: IEEE, Jun. 2016, pp. 5288–5296.
  * Wang et al. [2020] X. Wang, J. Wu, J. Chen, L. Li, Y.-F. Wang, and W. Y. Wang, “VATEX: A Large-Scale, High-Quality Multilingual Dataset for Video-and-Language Research,” Jun. 2020.
  * Rohrbach et al. [2014] A. Rohrbach, M. Rohrbach, W. Qiu, A. Friedrich, S. Amin, M. Andriluka, M. Pinkal, and B. Schiele, “Coherent Multi-Sentence Video Description with Variable Level of Detail,” in _Pattern Recognition: 36th German Conference, GCPR 2014, M{\”u}nster, Germany, September 2-5, 2014, Proceedings 36_, vol. 8753. Springer, 2014, pp. 184–195.
  * Zhou et al. [2017] L. Zhou, C. Xu, and J. J. Corso, “Towards Automatic Learning of Procedures from Web Instructional Videos,” Nov. 2017.
  * Miech et al. [2019] A. Miech, D. Zhukov, J.-B. Alayrac, M. Tapaswi, I. Laptev, and J. Sivic, “HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips,” Jul. 2019.
  * Zhukov et al. [2019] D. Zhukov, J.-B. Alayrac, R. G. Cinbis, D. Fouhey, I. Laptev, and J. Sivic, “Cross-task weakly supervised learning from instructional videos,” Apr. 2019.
  * Sanabria et al. [2018] R. Sanabria, O. Caglayan, S. Palaskar, D. Elliott, L. Barrault, L. Specia, and F. Metze, “How2: A Large-scale Dataset for Multimodal Language Understanding,” Dec. 2018.
  * Tian et al. [2018] Y. Tian, J. Shi, B. Li, Z. Duan, and C. Xu, “Audio-Visual Event Localization in Unconstrained Videos,” Mar. 2018.
  * Lei et al. [2020] J. Lei, L. Yu, T. L. Berg, and M. Bansal, “TVR: A Large-Scale Dataset for Video-Subtitle Moment Retrieval,” Aug. 2020.
  * Tian et al. [2020] Y. Tian, D. Li, and C. Xu, “Unified Multisensory Perception: Weakly-Supervised Audio-Visual Video Parsing,” Jul. 2020.
  * Xue et al. [2022] H. Xue, T. Hang, Y. Zeng, Y. Sun, B. Liu, H. Yang, J. Fu, and B. Guo, “Advancing High-Resolution Video-Language Representation with Large-Scale Video Transcriptions,” in _2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_. New Orleans, LA, USA: IEEE, Jun. 2022, pp. 5026–5035.
  * Bain et al. [2022] M. Bain, A. Nagrani, G. Varol, and A. Zisserman, “Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrieval,” May 2022.
  * Zhu et al. [2024] B. Zhu, B. Lin, M. Ning, Y. Yan, J. Cui, H. Wang, Y. Pang, W. Jiang, J. Zhang, Z. Li, W. Zhang, Z. Li, W. Liu, and L. Yuan, “LanguageBind: Extending Video-Language Pretraining to N-modality by Language-based Semantic Alignment,” Jan. 2024.
  * Chen et al. [2023a] S. Chen, X. He, L. Guo, X. Zhu, W. Wang, J. Tang, and J. Liu, “VALOR: Vision-Audio-Language Omni-Perception Pretraining Model and Dataset,” Apr. 2023.
  * Chen et al. [2023b] S. Chen, H. Li, Q. Wang, Z. Zhao, M. Sun, X. Zhu, and J. Liu, “VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and Dataset,” Oct. 2023.
  * Lei et al. [2023] W. Lei, Y. Ge, K. Yi, J. Zhang, D. Gao, D. Sun, Y. Ge, Y. Shan, and M. Z. Shou, “ViT-Lens-2: Gateway to Omni-modal Intelligence,” Nov. 2023.
  * Rohrbach et al. [2017] A. Rohrbach, A. Torabi, M. Rohrbach, N. Tandon, C. Pal, H. Larochelle, A. Courville, and B. Schiele, “Movie Description,” _Int J Comput Vis_ , vol. 123, no. 1, pp. 94–120, May 2017.
  * Rohrbach et al. [2015] A. Rohrbach, M. Rohrbach, N. Tandon, and B. Schiele, “A Dataset for Movie Description,” Jan. 2015.
  * Torabi et al. [2015] A. Torabi, C. Pal, H. Larochelle, and A. Courville, “Using Descriptive Video Services to Create a Large Data Source for Video Annotation Research,” Mar. 2015.
  * Bae et al. [2023] J. Bae, S. Jeong, S. Kang, N. Han, J.-Y. Lee, H. Kim, and T. Kim, “Sound of Story: Multi-modal Storytelling with Audio,” Oct. 2023.
  * Aytar et al. [2016] Y. Aytar, C. Vondrick, and A. Torralba, “SoundNet: Learning Sound Representations from Unlabeled Video,” https://arxiv.org/abs/1610.09001v1, Oct. 2016.
  * Chatterjee et al. [2021] M. Chatterjee, J. L. Roux, N. Ahuja, and A. Cherian, “Visual Scene Graphs for Audio Source Separation,” Sep. 2021.

###  -A Example caption per dataset

TABLE II: Example caption per dataset: the first occurrence of the word dog in each dataset. Dataset | Example Caption
---|---
AnimalSpeak |  | The sound of a Magnolia Warbler singing from
---
dense dogwood foliage.
AudioCaps | A dog whimpers quietly
AudioCaption | A woman nurse and a female doctor talk about dogs.
AudioDiffCaps | remove sound of dog barking
AudioGrounding | a dog is growling and barking
AudioSet | Speech, Whimper (dog)
CAPTDURE |  | The sound of a dog on a leash “roaming” on an
---
inorganic floor.
Clotho |  | A dog jumped into the water and swim and then
---
came out panting, it shook its body
ClothoAQA | Is it a dog making the noise? no
ClothoDetail |  | In this audio, we hear a series of creaking and
---
squeaking sounds. These noises could be attributed
to someone leaning back and forth on an office
chair, or perhaps to a slowly creaking door opening.
Additionally, there are intermittent popping sounds
that eventually taper off. Background noises, such
as a dog barking, add additional context and
atmosphere to the scene.
DAQA |  | Was the sound of the car passing by louder than
---
the sound of the dog making noise? yes
FAVDBench |  | A man commands a dog. Occasionally, you may
---
hear a dog barking in the distance
FSD50k |  | 20061012\. a distant ambulance siren along with
---
some dog barks. The noise of the siren ends
abruptly, presumably upon arrival to destination.
ME62+AKGCK94 into iRiver H120 decoded to
midside stereo / una abulancia lejana con algun
ladrido de perro
MACS | adults speaking while a dog barking
MULTIS |  | What is happening to the dog? The dog sounds
---
like it is upset, possibly whimpering because of
hunger, pain, or fear.
SoundDescs |  | Common (Northern) Flicker (Colaptes Auratus) -
---
medium close-up calls. Calls from Killdeer flying
over lake, dogs barking in part and insects in
background.
SoundingEarth |  | Dandeli jungle camp, Pradhani, Karnataka 581363,
---
Inde - Dogs howling. India, Karnataka, Dandeli,
Pradhani, jungle, camp, night, dogs, insects,
Binaural, Fostex FR-2LE, MS-TFB-2 microphones
VGGSound | dog growling
WavCaps |  | People are speaking and dogs are barking, with
---
ticking sounds in the background.
WavText5K | barking dogs.

###  -B List of audio-visual datasets

  * •

Video Captioning and Dense Description: ActivityNet-Captions [139], MSVD
[140], MSR-VTT [141], VATEX [142], TaCos [143].

  * •

Instructional and Procedural Content Understanding: YouCook [144], HowTo100M
[145], CrossTask [146], How2 [147].

  * •

Event Localization and Action Recognition: AVE [148], TVC [149], LLP [150].

  * •

Large-Scale Multimodal Integration: HD-VILA-100M [151], Webvid10M [152],
VIDAL-10M [153], VALOR-1M [154], VAST-27M [155], ViT-Lens-2 [156].

  * •

Specialized Domains: LSMCDC/MPII-MD [157, 158], M-VAD [159], Sound Of Story
[160], SoundNet [161], ASIW [162]