[a]Robin Plantey

# Light scalars in the Weinberg 3HDM potential with spontaneous CP violation

O. M. Ogreid  P. Osland  M. N. Rebelo  M. Aa. Solberg

###### Abstract

The $\mathbb{Z}_{2}\times\mathbb{Z}_{2}$ -symmetric 3HDM potential, sometimes
called Weinberg’s 3HDM potential, can accommodate both explicit and
spontaneous CP violation as well as natural flavour conservation and is hence
relevant for studies of CP violation coming from a realistic extended Higgs
sector. The model has an interesting regime, controlled by a single parameter,
where it acquires an approximate global $U(1)\times U(1)$ symmetry which
causes CP violating effects to be small and two pseudo-Goldstone bosons to be
present in the scalar spectrum. From parameter space scans, we find that, in a
realistic implementation featuring an SM-like Higgs boson, this model
predominantely leads to the existence of light additional Higgs bosons with a
mass lower than the SM-like Higgs boson.

##  1 Introduction

Weinberg realized that one could secure strangeness and charm conservation
within a gauge theory, and yet have explicit CP violation while imposing
discrete symmetries on a three-Higgs-doublet model [1]. His proposal amounts
to imposing a $\mathbb{Z}_{2}\times\mathbb{Z}_{2}$ symmetry on the theory
allowing for complex coefficients in the potential. The most general
renormalizable potential for three doublets $\phi_{i}$, invariant under
$\mathbb{Z}_{2}\times\mathbb{Z}_{2}$, is $V=V_{2}+V_{4}$, with111We follow the
notation of Ivanov and Nishi [2].

$\displaystyle V_{2}$
$\displaystyle=-[m_{11}(\phi_{1}^{\dagger}\phi_{1})+m_{22}(\phi_{2}^{\dagger}\phi_{2})+m_{33}(\phi_{3}^{\dagger}\phi_{3})]$
(1)

and

$\displaystyle V_{4}$ $\displaystyle=V_{0}+V_{\text{ph}},$ (2) $\displaystyle
V_{0}$
$\displaystyle=\lambda_{11}(\phi_{1}^{\dagger}\phi_{1})^{2}+\lambda_{12}(\phi_{1}^{\dagger}\phi_{1})(\phi_{2}^{\dagger}\phi_{2})+\lambda_{13}(\phi_{1}^{\dagger}\phi_{1})(\phi_{3}^{\dagger}\phi_{3})+\lambda_{22}(\phi_{2}^{\dagger}\phi_{2})^{2}$
$\displaystyle+\lambda_{23}(\phi_{2}^{\dagger}\phi_{2})(\phi_{3}^{\dagger}\phi_{3})+\lambda_{33}(\phi_{3}^{\dagger}\phi_{3})^{2}$
$\displaystyle+\lambda^{\prime}_{12}(\phi_{1}^{\dagger}\phi_{2})(\phi_{2}^{\dagger}\phi_{1})+\lambda^{\prime}_{13}(\phi_{1}^{\dagger}\phi_{3})(\phi_{3}^{\dagger}\phi_{1})+\lambda^{\prime}_{23}(\phi_{2}^{\dagger}\phi_{3})(\phi_{3}^{\dagger}\phi_{2})$
(3) $\displaystyle V_{\text{ph}}$
$\displaystyle=\lambda_{1}(\phi_{2}^{\dagger}\phi_{3})^{2}+\lambda_{2}(\phi_{3}^{\dagger}\phi_{1})^{2}+\lambda_{3}(\phi_{1}^{\dagger}\phi_{2})^{2}+\text{
h.c.}$ (4)

With real coefficients CP is explicitly conserved by the potential. However
there is a region of parameter space [3] where the potential $V$ admits a CP
violating vacuum. In this work we always assume the potential to be real while
natural flavour conservation is achieved by letting the doublets and right-
handed fermions have the following $\mathbb{Z}_{2}\times\mathbb{Z}_{2}$
charges

$\displaystyle\phi_{1}:(+1,+1)$ $\displaystyle\phi_{2}:(-1,+1)$
$\displaystyle\phi_{3}:(+1,-1)$ (5a) $\displaystyle u_{R}:(+1,+1)$
$\displaystyle d_{R}:(-1,+1)$ $\displaystyle e_{R}:(+1,-1).$ (5b)

This choice is not unique. The important point is to allow each set of right-
handed fermions to couple to only one Higgs doublet.

##  2 CP content of the neutral scalars

In a CP violating model such as the one considered here, the physical neutral
scalars are in general not CP eigenstates. Nevertheless, their CP content,
i.e., how close they are to being CP odd or CP even, can be probed by studying
certain couplings, of which two examples are presented below. In this
analysis, the Higgs basis $5\times 5$ diagonalization matrix of the neutral
scalars mass matrix denoted $O$, plays a central role.

###  2.1 Gauge couplings: $h_{i}h_{j}Z$

####  2.1.1 General features

The trilinear gauge couplings involving the Z boson and two neutral scalars
probe the relative CP content of the two scalars. Indeed, in a CP conserving
theory, this coupling vanishes unless the two scalars have opposite CP.
Letting the $h_{i}h_{j}Z$ coupling, in units of $g/(2\cos\theta_{W})$ be
denoted $P_{ij}$, it is given in terms the Higgs basis diagonalization matrix
as [4]

$P_{ij}=(O_{i2}O_{j4}+O_{i3}O_{j5})-(i\leftrightarrow j).$ (6)

We will refer to this quantity as the "Z affinity" $P_{ij}$. A value of unity
would correspond to the strength of the $HAZ$ coupling in the CP-conserving
2HDM, whereas a value of zero corresponds to the vanishing $hAZ$ and $hHZ$
couplings in that model. On the other hand, in the CP non-conserving 2HDM it
is given by linear expressions in the rotation matrix [5], here
$P_{12}=e_{3}/v$, $P_{23}=e_{1}/v$, and P${}_{31}=e_{2}/v$, with
$e_{1}^{2}+e_{2}^{2}+e_{3}^{2}=v^{2}$.

For the 3HDM, it satisfies a similar sum rule. By orthogonality, it follows
that

$\sum_{j}P_{ij}^{2}=\sum_{j\neq i}O_{j1}^{2}=1-O_{i1}^{2},$ (7)

and thus $\sum_{ij}P_{ij}^{2}=5-1=4$. By antisymmetry, the 5 diagonal elements
$P_{ii}$ vanish, and for a random $O$ matrix the non-zero values of
$P_{ij}^{2}$ would have an average of $4/20=0.2$.

The quantity $P_{ij}$ can be either positive or negative. We have performed
scans over the potential parameters [4] and studied the "average" Z affinity
as the following root-mean-square

$\left(P_{ij}\right)_{\text{r.m.s}}=\sqrt{\frac{1}{N}\sum_{1}^{N}P_{ij}^{2}},$
(8)

where the sum runs over the $N$ points of the scan. When the affinity
$\left(P_{ij}\right)_{\text{r.m.s}}$ is large, the $h_{i}h_{j}$ pair has a
strong coupling to the $Z$, i.e., they have rather different CP content. On
the other hand, when the affinity $\left(P_{ij}\right)_{\text{r.m.s}}$ is low,
the two states are close in their CP profiles and have a suppressed coupling
to the $Z$. It should be noted however that the affinities $P_{ij}$ are not
transitive: If $P_{ab}=P_{ac}$, that is, $h_{b}$ and $h_{c}$ are equally “far”
from $h_{a}$ in terms of CP content, then it does not follow that $P_{bc}=0$,
i.e., $h_{b}$ and $h_{c}$ do not necessarily have the same CP content. The
origin of this property lies in the fact that $P_{ij}$ consists of terms each
involving both $i$ and $j$.

###  2.2 The regime of $U(1)\times U(1)$ symmetry

When the rephasing sensitive part of the potential $V_{\text{ph}}$ vanishes,
the $\mathbb{Z}_{2}\times\mathbb{Z}_{2}$ symmetry is enhanced to a $U(1)\times
U(1)$ symmetry. Whenever this symmetry is broken by the Higgs vacuum
expectation values (VEVs), there are two massless states in the scalar
spectrum, as expected from the Goldstone theorem [6, 7]. Moreover, when the
$U(1)\times U(1)$ symmetry is exact, at the Lagrangian level, the phases of
the VEVs can be rotated away and therefore spontaneous CP violation does not
occur. This model has the interesting property that close to this limit, in
general it is controlled by only one parameter, $\lambda_{1}$, since the other
couplings in $V_{\text{ph}}$, $\lambda_{2}$ and $\lambda_{3}$, can be related
to $\lambda_{1}$ by using the minimization conditions for the potential.

If the $U(1)\times U(1)$ symmetry is only approximate, i.e., if $\lambda_{1}$
is small, but finite with respect to all other quartic couplings, then the
spectrum contains two light states corresponding to two pseudo-Goldstone
bosons. The phenomenology of the model can be conveniently analysed using
$\lambda_{1}$ since it controls both the masses of the $U(1)\times U(1)$
pseudo-Goldstone bosons and the strength of the CP violating effects coming
from the Higgs sector.

In Fig. 1 we compare Z affinities for all pairs of neutral scalars, and for
two cases. In the left panel, we impose the “near $U(1)\times U(1)$ symmetry”
condition

$\max(|\lambda_{1}|,|\lambda_{2}|,|\lambda_{3}|)=0.01,$ (9)

whereas in the right panel we impose no such constraint, i.e., we do not
restrict the scan to the regime of near $U(1)\times U(1)$ symmetry. The left
panel shows a clear separation into two sets of states, $h_{1}$ and $h_{2}$
have low affinity to the $Z$, meaning they have similar CP content, as does
the other set, $h_{3}$, $h_{4}$ and $h_{5}$. It is natural to interpret this
as follows: Near the $U(1)\times U(1)$ limit we have two neutral states that
are approximately odd under CP, and three that are approximately even. This is
fully compatible with the expectations from the Goldstone theorem [6, 7],
since the Goldstone bosons in the $U(1)\times U(1)$ limit will be CP odd.

Figure 1: Average Z affinity $\left(P_{ij}\right)_{\text{r.m.s}}$ of states
$h_{i}$ and $h_{j}$. Left: the $U(1)\times U(1)$ limit, as defined by Eq. (9);
Right: no restriction on the lambdas.

We note that this limit also secures an upper bound on the amount of CP
violation in the scalar sector.

####  2.2.1 Alignment

In order to make contact with experimental reality, we have to make sure one
scalar has a coupling to the electroweak gauge bosons $W^{+}W^{-}$ that is
close to unity, in units of $gm_{W}$. This amounts to requiring $|O_{k1}|$
being close to unity, for some $k$. Let us first note that in the exact
alignment limit, with $|O_{k1}|=1$, then, by unitarity, all $O_{kj}=0$, for
$j\neq 1$, and all $O_{i1}=0$ for $i\neq k$. Thus,

$\text{Exact alignment: }\quad P_{ik}=P_{kj}=0,\quad\text{for all }i\text{ and
all }j.$ (10)

In this exact alignment limit, the row and column referring to $h_{k}$ in the
$\left(P_{ij}\right)_{\text{r.m.s}}$ plot would thus be zero, reflecting the
fact that $h_{j}h_{k}$ would not couple to $Z$ for any $j$.

We shall relax the alignment condition, replacing it by the experimental one
[8], obeyed at the $3\sigma$ level:

$\kappa_{V}-3\sigma<O_{k1}^{2}<\kappa_{V}+3\sigma,\quad\text{for some }k.$
(11)

Imposing this constraint on $h_{2}$ and $h_{3}$, the scan [4] yields the
average Z affinities shown in Fig. 2. Qualitatively, the features described
above are verified. Couplings of the $Z$ to the “SM state” are strongly
suppressed, whereas there can be strong couplings to $h_{i}h_{j}$, provided
$i$ and $j$ both differ from $k$.

Figure 2: Average Z affinity $\left(P_{ij}\right)_{\text{r.m.s}}$ of states
$h_{i}$ and $h_{j}$. Left: $h_{2}$ is assumed to satisfy the SM constraint,
Eq. (11); Right: $h_{3}$ satisfies the SM constraint.

###  2.3 Neutral mass spectrum

In the Weinberg potential with an SM-like Higgs boson state, there is a strong
tendency for one or two additional bosons to be lighter than $125$ GeV. In the
parameter scan performed in Ref. [4] the SM state $h_{k}$ is the second
lightest in 38% of the cases, and the third lightest in an additional 28% of
the cases. However, this does not necessarily rule out much of the parameter
space of the model as the light states may have escaped detection at LEP where
the dominant production channel would be the Bjorken mechanism. Indeed, the
cross section for this channel is proportional to the $h_{i}ZZ$ couplings
$O_{i1}$ which are suppressed for the BSM scalars by the orthogonality of the
mixing matrix.

###  2.4 Yukawa couplings

Even though this model would not generate a complex CKM matrix, for real
Yukawa couplings there would still be CP violation in the Yukawa sector coming
from Higgs interactions. Indeed, the neutral scalars $h_{i}$ do not have
definite CP and have both CP-even and CP-odd couplings to fermions. Hence
their Yukawa interactions are given by

$\displaystyle\mathcal{L}_{h_{i}ff}=\frac{m_{f}}{v}h_{i}(\kappa^{h_{i}ff}\bar{f}f+i\tilde{\kappa}^{h_{i}ff}\bar{f}\gamma_{5}f).$
(12)

One can define the angle $\alpha^{h_{i}ff}$ through

$\tan\alpha^{h_{i}ff}=\frac{\tilde{\kappa}^{h_{i}ff}}{\kappa^{h_{i}ff}}.$ (13)

which measures the relative strength of the CP-even and CP-odd coupling of
$h_{i}$ to a fermion $f$ and hence also measures the CP profile of $h_{i}$. In
this model, when $f$ couples to the doublet $\phi_{k}$, $\alpha^{h_{i}ff}$ can
be computed in terms of the matrix $\tilde{\cal R}$ [4] which rotates the
doublets to the Higgs basis and the rotation matrix $O$ which diagonalizes the
neutral mass-squared matrix in the Higgs basis

$\displaystyle\alpha^{h_{i}ff}=\arg(Z_{i}^{(k)})$ (14)

where

$\displaystyle Z_{i}^{(k)}$ $\displaystyle=\left(\tilde{\cal
R}^{T}\right)_{k1}O_{i1}+\left(\tilde{\cal
R}^{T}\right)_{k2}(O_{i2}+iO_{i4})+\left(\tilde{\cal
R}^{T}\right)_{k3}(O_{i3}+iO_{i5})$ $\displaystyle=\tilde{\cal
R}_{1k}O_{i1}+\tilde{\cal R}_{2k}(O_{i2}+iO_{i4})+\tilde{\cal
R}_{3k}(O_{i3}+iO_{i5}).$ (15)

However, it should be pointed out that a real CKM matrix is already ruled out
by experiment [9, 10]. The simplest way to fix this problem is to allow for
complex Yukawa couplings.

##  3 Conclusion

In most of the parameter space of the Weinberg potential, the scalar spectrum
contains at least one neutral scalar lighter than the observed Higgs boson.
This is not necessarily a problem for the model because the main production
channel for these light states would have been suppressed due to the
orthogonality of the scalar mass mixing matrix. In fact there has been recent
interest in light Higgs bosons due to observed excesses in Higgs searches
around $95$ GeV [11, 12, 13, 14]. It would be interesting to explore if a
model based on Weinberg’s 3HDM potential could fit these excesses, while
obeying current experimental constraints.

## References

  * [1] S. Weinberg, _Gauge Theory of CP Violation_ , _Phys. Rev. Lett._ 37 (1976) 657.
  * [2] I. P. Ivanov and C. C. Nishi, _Symmetry breaking patterns in 3HDM_ , _JHEP_ 01 (2015) 021, [1410.6139].
  * [3] G. C. Branco, _Spontaneous CP Nonconservation and Natural Flavor Conservation: A Minimal Model_ , _Phys. Rev. D_ 22 (1980) 2901.
  * [4] R. Plantey, O. M. Ogreid, P. Osland, M. N. Rebelo and M. A. Solberg, _Weinberg’s 3HDM potential with spontaneous CP violation_ , 2208.13594.
  * [5] B. Grzadkowski, O. M. Ogreid and P. Osland, _Measuring CP violation in Two-Higgs-Doublet models in light of the LHC Higgs data_ , _JHEP_ 11 (2014) 084, [1409.7265].
  * [6] J. Goldstone, _Field Theories with Superconductor Solutions_ , _Nuovo Cim._ 19 (1961) 154–164.
  * [7] J. Goldstone, A. Salam and S. Weinberg, _Broken Symmetries_ , _Phys. Rev._ 127 (1962) 965–970.
  * [8] Particle Data Group collaboration, P. Zyla et al., _Review of Particle Physics_ , _PTEP_ 2020 (2020) 083C01.
  * [9] F. J. Botella, G. C. Branco, M. Nebot and M. N. Rebelo, _New physics and evidence for a complex CKM_ , _Nucl. Phys. B_ 725 (2005) 155–172, [hep-ph/0502133].
  * [10] CKMfitter Group collaboration, J. Charles, A. Hocker, H. Lacker, S. Laplace, F. R. Le Diberder, J. Malcles et al., _CP violation and the CKM matrix: Assessing the impact of the asymmetric $B$ factories_, _Eur. Phys. J. C_ 41 (2005) 1–131, [hep-ph/0406184].
  * [11] LEP Working Group for Higgs boson searches, ALEPH, DELPHI, L3, OPAL collaboration, R. Barate et al., _Search for the standard model Higgs boson at LEP_ , _Phys. Lett. B_ 565 (2003) 61–75, [hep-ex/0306033].
  * [12] CMS collaboration, A. M. Sirunyan et al., _Search for a standard model-like Higgs boson in the mass range between 70 and 110 GeV in the diphoton final state in proton-proton collisions at $\sqrt{s}=$ 8 and 13 TeV_, _Phys. Lett. B_ 793 (2019) 320–347, [1811.08459].
  * [13] CMS collaboration, _Search for new resonances in the diphoton final state in the mass range between 80 and 115 GeV in pp collisions at $\sqrt{s}=8$ TeV_, .
  * [14] T. Biekötter, S. Heinemeyer and G. Weiglein, _Mounting evidence for a 95 GeV Higgs boson_ , 2203.13180.


# Audio Flamingo: A Novel Audio Language Model with Few-Shot Learning and
Dialogue Abilities

Zhifeng Kong† Arushi Goel† Rohan Badlani† Wei Ping† Rafael Valle† Bryan
Catanzaro†

###### Abstract

Augmenting large language models (LLMs) to understand audio – including non-
speech sounds and non-verbal speech – is critically important for diverse
real-world applications of LLMs. In this paper, we propose Audio Flamingo, a
novel audio language model with 1) strong audio understanding abilities, 2)
the ability to quickly adapt to unseen tasks via in-context learning and
retrieval, and 3) strong multi-turn dialogue abilities. We introduce a series
of training techniques, architecture design, and data strategies to enhance
our model with these abilities. Extensive evaluations across various audio
understanding tasks confirm the efficacy of our method, setting new state-of-
the-art benchmarks. †††NVIDIA. Correspondence to: Zhifeng Kong
(zkong@nvidia.com), Wei Ping (wping@nvidia.com), Rafael Valle
(rafaelvalle@nvidia.com).

Copyright 2024 by the author(s).

##  1 Introduction

The ability to understand sound is unarguably important and necessary for an
agent to interact with the world. While large language models (LLMs) have
shown an impressive ability to understand and reason about the world through
text, their understanding of sound remains limited to transcriptions of speech
(Lyu et al., 2023), thus making LLMs agnostic to important information in non-
speech sounds and non-verbal speech. Even though recent contributions have
improved their ability to understand sound (Gong et al., 2023c; Lyu et al.,
2023; Huang et al., 2023; Deshmukh et al., 2023; Chu et al., 2023; Tang et
al., 2023a), there exists no model that combines: i) strong audio
understanding ability on various tasks (Deshmukh et al., 2023), ii) the
ability to execute multi-turn dialogues (Duan et al., 2023), and iii) the
ability to quickly adapt to unseen tasks without fine-tuning, for example,
through in-context learning (Alayrac et al., 2022) and retrieval augmented
generation (Lewis et al., 2020).

Our contribution to expand LLM’s ability to understand sound is called Audio
Flamingo: a novel audio language model that supports in-context learning
(ICL), retrieval augmented generation (RAG), and multi-turn dialogues. It
achieves state-of-the-art results on multiple audio understanding tasks. 111We
will open-source the training and inference code. Our demo website is:
https://audioflamingo.github.io/.

Expanding LLM’s ability to understand sound is a challenging task. The first
challenge is extracting features from variable-length audio, and conditioning
the LM on the audio features. While prior works have designed representations
for audio of any length (Wu et al., 2023), they can lose temporal information.
In this work, we introduce an audio feature extractor with sliding window
based on Elizalde et al. (2023b), which we believe to capture temporal
information better. To condition the LM on audio inputs, previous models
prepended language tokens with audio tokens (Deshmukh et al., 2023; Chu et
al., 2023; Tang et al., 2023a). This approach may have excessive overhead
especially for long audio, as the complexity is quadratic with respect to the
number of audio tokens. In contrast, we use cross attentions to fuse audio
inputs into the LM in a similar way as Flamingo (Alayrac et al., 2022). In our
approach the complexity is linear in the number of audio tokens, thus making
Audio Flamingo able to generalize to many audio inputs efficiently.

The second challenge is collecting and training on highly heterogeneous data.
Prior works have collected and combined different datasets for training (Doh
et al., 2023; Deshmukh et al., 2023; Chu et al., 2023; Gong et al., 2023c). We
follow their approach and curate a heterogeneous dataset with approximately
5.9 million audio-text pairs. Prior works have also designed different
training curriculum, such as training on close-ended tasks first followed by
open-ended tasks (Gong et al., 2023c; Tang et al., 2023a). However, these
result in a trade-off between close-ended and open-ended tasks, limiting the
overall performance (Deshmukh et al., 2023; Tang et al., 2023a). We use a
different approach based on a widely adopted and stable method to train LLMs
(Ouyang et al., 2022). Specifically, we use two training stages: pre-training
and supervised fine-tuning (SFT), each with different subsets and training
techniques. These innovations make Audio Flamingo achieve the state-of-the-art
results on several audio understanding benchmarks with $<\frac{1}{3}$ number
of parameters as Chu et al. (2023) and Gong et al. (2023c).

Figure 1: Audio Flamingo versus previous state-of-the-art (Deshmukh et al.,
2023; Chu et al., 2023; Gong et al., 2023b, c; Tang et al., 2023a; Ghosh et
al., 2023) on a number of audio understanding benchmarks. The numbers are
normalized such that the maximum of all models is $100\%$ on each task. Audio
Flamingo sets the new state-of-the-art results on most of these tasks.

The third challenge is to give the audio language model the ability to quickly
adapt to new tasks without fine-tuning, for instance, via in-context learning
(ICL) (Brown et al., 2020) and retrieval. While recent audio language models
have shown zero-shot abilities (Deshmukh et al., 2023; Gong et al., 2023c),
they lack the ability to perform in-context few-shot learning to new tasks. In
this paper, we introduce a series of techniques to realize this ability. We
implement an efficient retrieval method, introduce an ICL template, and use
retrieved samples to create interleaved ICL datasets for training. We also
introduce a novel cross attention mask for interleaved samples. As a result,
Audio Flamingo can be quickly adapted to new tasks via ICL and retrieval
without task-specific fine-tuning. Our results confirm the efficacy of our
approach and set new state-of-the-art few-shot benchmarks.

The last challenge is to give the audio language model the ability to chat
with a user for many rounds. While prior methods have shown demos of dialogues
(Gong et al., 2023c; Chu et al., 2023), they lack systematic and quantitative
evidence. To address this challenge, we create two multi-turn dialogue
datasets with GPT-4 (Achiam et al., 2023) based on detailed annotations of two
datasets, with an emphasis on correlated context. We obtain a chat model by
fine-tuning Audio Flamingo on these datasets. Our results show that our chat
model has strong multi-turn dialogue ability and significantly outperforms
previous methods.

We evaluate Audio Flamingo on a large and diverse set of close and open-ended
benchmarks. A single Audio Flamingo model surpasses the previous state-of-the-
art on most benchmarks, and the chat version of Audio Flamingo significantly
outperforms baselines on dialogue benchmarks. Figure 1 summarizes the
benchmark results of Audio Flamingo. We also briefly discuss about the neural
architecture and hyper parameters in the experiments. Our key contributions
include:

  1. 1.

We propose Audio Flamingo: a Flamingo-based audio language model for audio
understanding with a series of innovations. Audio Flamingo achieves state-of-
the-art results on several close-ended and open-ended audio understanding
tasks.

  2. 2.

We design a series of methodologies for efficient use of ICL and retrieval,
which lead to the state-of-the-art few-shot learning results.

  3. 3.

We enable Audio Flamingo to have strong multi-turn dialogue ability, and show
significantly better results compared to baseline methods.

##  2 Related work

Multimodal LLMs. There has been tremendous progress in the area of multimodal
LLMs. In addition to text, these models take inputs from various modalities
such as vision (Tsimpoukelli et al., 2021; Alayrac et al., 2022; Yang et al.,
2023; Driess et al., 2023; Liu et al., 2023a; Li et al., 2023a), audio
(Deshmukh et al., 2023; Gong et al., 2023b; Rubenstein et al., 2023), or
multiple of them (Han et al., 2023; Tang et al., 2023b; Moon et al., 2023;
Zhao et al., 2023), and each has a different integration method. In the audio
modality, prior works have looked at speech tasks (Chen et al., 2023;
Rubenstein et al., 2023), general audio understanding (Deshmukh et al., 2023;
Gong et al., 2023c), music understanding (Gardner et al., 2023; Won et al.,
2023; Li et al., 2023c; Liu et al., 2023b; Doh et al., 2023), or a combination
of these (Gong et al., 2023b; Tang et al., 2023a; Chu et al., 2023). The focus
of our paper is audio understanding, which includes non-speech sound and
music, and non-verbal speech. Different from prior works, our model has
stronger audio understanding ability, and is the first audio understanding
model with i) in-context few-shot learning ability, ii) retrieval augmented
generation ability, and iii) strong multi-turn dialogue ability.

Audio encoders and representation. Many audio encoders extract audio features
from the spectrogram, including CNN-based method (Kong et al., 2020) and
Transformer-based methods (Gong et al., 2021; Chen et al., 2022; Défossez et
al., 2022; Radford et al., 2023; Gong et al., 2023a). These methods are
primarily targeted at solving a particular problem such as speech recognition
or event detection. Based on these encoders, many joint audio-language
embeddings have been proposed (Elizalde et al., 2023a, b; Wu et al., 2023;
Huang et al., 2022; Li et al., 2023c). These methods use contrastive learning
to map audio and language embeddings into the same space, and are often
trained on a large variety of audio and language. However, many of these
methods compute a single embedding for an audio and therefore may lose
temporal information. In this paper, we build an audio encoder with sliding
windows based on ClapCap (Elizalde et al., 2023b) to better capture long-range
and temporal information.

Data augmentation. Due to limited amount of high-quality human annotated
sounds besides speech transcriptions, many works have proposed to augment
textural description with existing LLMs such as GPT-4 (Achiam et al., 2023). A
common strategy is to provide an LLM with annotated tags, timestamps, and
other miscellaneous information, and then ask it to generate captions (Wu et
al., 2023; Doh et al., 2023; Mei et al., 2023; Gardner et al., 2023) or
question-answering data pairs (Gong et al., 2023c, b; Liu et al., 2023b). In
this paper, we leverage existing LLMs to generate two multi-turn dialogue
datasets based on detailed annotations, which enable our model strong dialogue
abilities.

In-context learning (ICL). In-context learning is a kind of few-shot learning
ability, where an LLM rapidly adapts to a desired task at inference time only
after looking at a few examples in the prompt (Brown et al., 2020). It has
widely shown success in natural language tasks (Wei et al., 2021) and visual-
language tasks (Alayrac et al., 2022; Yang et al., 2023). In the speech
domain, ICL has been shown to help speech-related tasks such as speech
recognition, translation, and processing (Gao et al., 2022; Wang et al., 2023;
Hsu et al., 2023; Chen et al., 2023). However, ICL for general audio
understanding is much less explored. In this paper, we propose the first audio
understanding model with ICL ability.

Retrieval-augmented generation (RAG). Retrieval-augmented generation for LLMs
is to improve generation quality by using external knowledge, for example from
an external database, which contains useful and related knowledge. It has been
widely applied in natural language tasks (Guu et al., 2020; Karpukhin et al.,
2020; Lewis et al., 2020; Borgeaud et al., 2022) and visual-language models
(Yang et al., 2023). In the audio-language domain, Ghosh et al. (2023)
proposed a retrieval method for audio captioning by prepending captions from
similar audios to the prompt. However, it does not provide the retrieved audio
to the model. Consequently, the model loses information on how similar the
retrieved audio is to the test audio. In contrast, we provide both the
retrieved audio and text to our model. The benefit of this approach is that
our model could determine when and how to use the retrieval based on the
similarity between test and retrieved audio. We provide comparisons in our
few-shot experiments.

##  3 Method

In this section, we introduce Audio Flamingo, an audio-understanding language
model with few-shot learning via ICL and RAG. In Section 3.1, we introduce the
architecture used in Audio Flamingo, including the audio feature extractor,
audio representation transformation layers, language model, and the
conditioning method. In Section 3.2, we introduce the training method of Audio
Flamingo, including the training objective, design of masks, and training
stages.

###  3.1 Architecture

Our neural architecture is composed of four components: i) an audio feature
extractor with sliding window, ii) audio representation transformation layers,
iii) a decoder-only language model, and iv) gated xattn-dense layers. Figure 2
summarizes the architecture.

Figure 2: Neural architecture of Audio Flamingo. It takes interleaved audio
and text as input and outputs free-form text.

i) Audio feature extractor with sliding window. We use ClapCap (Elizalde et
al., 2023b) as the audio feature extractor backbone, which we denote as
$\mathcal{E}$. ClapCap takes 7-second 44.1kHz raw audio as input, then
transforms the audio into Mel-spectrogram of hop length 320, window length
1024, 64 Mel bins, and finally outputs a 1024-dimensional vector
representation.

We consider each 7-second segment as a window and use sliding windows to
extract features for longer audio. The overlap between consecutive windows is
$7\times 0.75=5.25$ seconds. We use a maximum of $m=16$ sliding windows.
Formally, let $x(s\texttt{:}t)$ be the segment of $s$ to $t$ seconds in audio
$x$. Then, the extracted feature is
$\left[\mathcal{E}(x(0\texttt{:}7)),\mathcal{E}(x(\frac{7}{4}\texttt{:}\frac{7\times
5}{4})),\cdots,\mathcal{E}(x(\frac{7(m-1)}{4}\texttt{:}\frac{7(m+3)}{4})\right]$,
which supports a maximum of $33.25$ seconds. The goal of this design is to
capture long-range and temporal information that might be ignored in a single
fused representation vector (Wu et al., 2023). Long audio will be cropped and
short audio will be zero-padded. If an entire segment is zero-padded then we
will mask the corresponding embedding at cross attention. If the input is
interleaved data with $>1$ audio, we concatenate their sliding window
representations.

ii) Audio representation transformation layers. We increase model capacity by
further applying a few audio representation transformation layers to the
concatenated audio feature representations described earlier. It is comprised
of 3 self-attention layers (Vaswani et al., 2017), with 8 heads and inner
dimension 2048 each. This module is fully trainable.

iii) Language model. We use a decoder-only causal LM in our architecture. In
this paper, we use OPT-IML-MAX-1.3B (Iyer et al., 2022), a 1.3B parameter
model with 24 LM blocks. It has been instruction-tuned on many natural
language tasks.

iv) Conditioning LM on audio representations. We use the gated xattn-dense
layers from Flamingo (Alayrac et al., 2022) to achieve conditioning on audio
inputs. Each layer has two blocks: 1) a residual block with cross attention
and $\tanh$ gating, followed by 2) a residual block with dense layer and
$\tanh$ gating. These layers are prepended to each LM block.

###  3.2 Training Method

i) Training objective. Let $x$ be the mono-channel audio input,
$y_{\mathrm{ins}}$ be the instruction text (e.g. question), and
$y_{\mathrm{out}}$ be the output text. For conciseness we use
$z=(x,y_{\mathrm{ins}},y_{\mathrm{out}})$ to represent each training sample.

We use maximum likelihood estimation (MLE) to train our model. Let
$(y_{\mathrm{out}})_{t}$ be the $t$-th token and $(y_{\mathrm{out}})_{<t}$ the
first $t-1$ tokens of the output. For a non-interleaved sample
$z=(x,y_{\mathrm{ins}},y_{\mathrm{out}})$, the log-likelihood is

$\mathcal{L}(z)=\sum_{t=1}^{|y_{\mathrm{out}}|}\log
p_{\theta}\left((y_{\mathrm{out}})_{t}|x,y_{\mathrm{ins}},(y_{\mathrm{out}})_{<t}\right).$
(1)

For an interleaved training sample composed of $J$ samples
$z_{\mathrm{int}}=\\{z^{1},\cdots,z^{J}\\}$, where
$z^{j}=(x^{j},y_{\mathrm{ins}}^{j},y_{\mathrm{out}}^{j})$, the log-likelihood
is computed over all outputs:

$\begin{array}[]{l}\displaystyle\mathcal{L}_{\mathrm{int}}(z_{\mathrm{int}}=\\{z^{1},\cdots,z^{J}\\})=\\\
\displaystyle~{}\sum_{j=1}^{J}\sum_{t=1}^{|y_{\mathrm{out}}^{j}|}\log
p_{\theta}\left((y_{\mathrm{out}}^{j})_{t}|z^{<j},x^{j},y_{\mathrm{ins}}^{j},(y_{\mathrm{out}}^{j})_{<t}\right).\end{array}$
(2)

Note this interleaved data loss is different from prior models, which compute
losses only on the last output $y_{\mathrm{out}}^{J}$ (Yang et al., 2023), or
have either none or indirect conditioning on prior multimodal inputs $x^{<j}$
(Alayrac et al., 2022; Ghosh et al., 2023). We expect (2) can help the model
look at a various number (including zero when $j=1$) of ICL samples as well as
the associated audio, thus improving robustness and training efficiency in a
similar way as bucketing (Khomenko et al., 2016), especially when the ICL
samples are retrieved similar samples. The corresponding loss mask is shown on
the right-hand-side of Figure 3.

Let $\\{\mathcal{D}^{i},i\in\mathcal{I}\\}$ be all non-interleaved training
datasets, and
$\\{\mathcal{D}_{\mathrm{int}}^{i^{\prime}},{i^{\prime}}\in\mathcal{I}_{\mathrm{int}}\\}$
be all interleaved training datasets. The overall training objective is a
weighted mixture of losses on each dataset:

$L=-\sum_{i\in\mathcal{I}}\lambda_{i}\mathbb{E}_{z\sim\mathcal{D}^{i}}\mathcal{L}(z)-\sum_{i^{\prime}\in\mathcal{I_{\mathrm{int}}}}\lambda_{i^{\prime}}\mathbb{E}_{z_{\mathrm{int}}\sim\mathcal{D}_{\mathrm{int}}^{i^{\prime}}}\mathcal{L}_{\mathrm{int}}(z_{\mathrm{int}}),$
(3)

where $\lambda_{i}$’s are the weights for each dataset. The weights are
constant hyper-parameters and have a huge impact on the final model. They are
computed from the pre-defined number of epochs for each dataset (see Section
4.1 for the intuition, and Appendix A for details).

ii) Cross attention masks. We use block upper-triangular cross attention masks
for interleaved samples so that the likelihood of $j$-th output
$p_{\theta}(y_{\mathrm{out}}^{j})$ is conditioned only on the first $j$ audio
inputs $x^{\leq j}$. We expect this helps making the model to look at previous
audio. Figure 3 demonstrates the mask.

Figure 3: Left: the block upper-triangular cross attention masks between text
tokens and audio embeddings. Right: the loss mask of an interleaved training
sample.

iii) Two training stages. We divide training into pre-training and supervised
fine-tuning (SFT), a widely adopted and stable method in training LMs (Ouyang
et al., 2022). During pre-training we only train the audio representation
transformation layers and the gated xattn-dense layers. The purpose is to
obtain a good set of initialization weights for these layers. During SFT, we
unfreeze the entire LM, and train all modules but the audio encoder. 222In
initial experiments we found unfreezing the audio encoder led to much worse
results.

##  4 Data

###  4.1 Datasets

In this section, we introduce our data strategies, including dataset
collection, generation, and blending. We also introduce templates for each
type of dataset.

Dataset sources. We train our model on a variety of audio datasets that can be
roughly classified into three types: music, non-speech general sound, and non-
verbal speech. In this paper, we focus on these data given the immediate
availability of state-of-the-art speech transcription models. We look at three
types of tasks: (1) audio captioning (CAP), where we would like the model to
describe the audio in a sentence; (2) audio question-answering (AQA), where we
would like the model to answer questions regarding the audio, and (3) audio
classification (CLS), where we would like the model to classify the sound into
one or more labels corresponding to events, scenes, music genres, instruments,
qualities, and others. An overview of all training datasets is shown in Table
6 in the Appendix.

ICL datasets. In order to give our model in-context learning and retrieval
augmentation abilities, we construct ICL datasets for each of the raw datasets
based on $k$NN computed on audio embeddings. Let $\mathcal{D}^{i}$ be the
$i$-th training dataset. For each
$z=(x,y_{\mathrm{ins}},y_{\mathrm{out}})\in\mathcal{D}^{i}$, we find its
top-$k$ closest training samples in $\mathcal{D}^{i}$ excluding $z$, where the
distance function is $\ell_{2}$ in the fused LAION-CLAP embedding space (Wu et
al., 2023) for the audio part of the sample. We use Faiss-gpu (Johnson et al.,
2019) to accelerate searching. Figure 4 demonstrates this process.

Figure 4: Construction of ICL samples based on RAG. We use LAION-CLAP to find
top-$k$ most similar samples from the database, and use the retrieved audio
and text to construct an ICL training sample.

Dataset staging and blending. We use different datasets during the pre-
training stage and the supervised fine-tuning (SFT) stage. The selection is
based on data quality, diversity, source, and size as described below. 1) Data
quality: low quality datasets, including those with low-quality or noisy
audio, low-quality text, and inaccurate text annotation, are used for pre-
training. 2) Data diversity: datasets with less diversity or strong biases in
label distributions are used for pre-training. 3) Data sources: datasets
containing AI-generated contents are mostly used for pre-training, whereas
some high-quality subsets may be used for SFT. 4) Data sizes: very large
datasets may be used both for pre-training and SFT. 5) ICL datasets are used
in the SFT stage.

We assign different weights $\lambda_{i}$ when sampling from different
datasets based on their sizes, quality, and diversity. The weights are
computed from the number of epochs for each dataset. The details of staging
and weights can be found in Appendix A.

###  4.2 Templates

Our templates are based on OPT-IML’s template (Iyer et al., 2022) and
Flamingo’s multimodal template (Alayrac et al., 2022). For a non-interleaved
sample, the template is describe below.

<s>{task description}<audio>{instruction} Options:$\backslash$n-
option1$\backslash$n$\cdots$\- optionm <SEP>{output}<EOC></s>

In this template, <audio> is the special token that informs the language model
the location of audio in the context. The {task description} is natural
language that tells the language model which task it is handling, for example
“The task is event classification”. The {instruction} is the language
instruction such as a question. The options sentence is to tell the language
model all options for classification so that it can classify an audio by
outputting free-form text. The {output} is the ground truth output that will
be trained. The <SEP> token is a separator that indicates the end of
instruction, and <EOC> is the end-of-chunk token that indicates the end of a
sample. Below is the template for interleaved (ICL) samples with $k+1$ tuples
of (audio, instruction, output).

<s>{task description}Here are similar samples.
<audio>{instruction1}<SEP>{output1}<EOC> $\cdots$
<audio>{instructionk}<SEP>{outputk}<EOC> <audio>{instruction}
Options:$\backslash$n- option1$\backslash$n$\cdots$\- optionm
<SEP>{output}<EOC></s>

###  4.3 Multi-Turn Dialogue Dataset

We aim at giving our model stronger abilities in dealing with complicated
multi-turn dialogues. To achieve this, we use GPT-4 (Achiam et al., 2023) to
generate two multi-turn dialogue datasets. We construct these datasets based
on the strongly labeled AudioSet-SL (Hershey et al., 2021) and MusicCaps
(Agostinelli et al., 2023), with thresholding based on LAION-CLAP embeddings
(Wu et al., 2023) to filter undesirable samples. We name these two generated
datasets AF-Dialogue-AudioSetSL and AF-Dialogue-MusicCaps, respectively. The
detailed instructions, filtering method, dataset statistics, and examples are
in Appendix B. We use the following template for an $s$-turn dialogue data
sample.

<s>The task is dialogue.<audio> user: {instruction1} assistant:
<SEP>{output1}<EOC> $\cdots$ user: {instructions} assistant:
<SEP>{outputs}<EOC></s>

##  5 Experiments

In this section, we answer the following questions:
Q1. Does Audio Flamingo understand audio better than the state-of-the-art
baselines?
Q2. How significantly does the ICL-based RAG help Audio Flamingo adapt to new
tasks?
Q3. What is Audio Flamingo’s ability to have multi-turn dialogues with a user?
Q4. Which specific configuration of Audio Flamingo works the best overall?

###  5.1 Experimental Setup

We use 8 NVIDIA A100 GPUs to train our model. During pre-training, we use
batch size $=384$, AdamW optimizer (Loshchilov & Hutter, 2017) with learning
rate $=1\times 10^{-4}$ and weight decay $=0.1$. For efficiency, we use bf16
with automatic mixed precision. During supervised fine-tuning (SFT), we reduce
the batch size to 128, the learning rate to $2\times 10^{-5}$, and use fp32
for better numerical precision. We let the maximum number of interleaved
samples to be 8 unless specified. We set the maximum number of text tokens to
be 512.

We compare to the most recent state-of-the-art baselines on several
benchmarks. On each dataset, we choose the best score among all SOTA baselines
as the reference value. Unless specified, we report accuracy for question-
answering and single-label classification, F1 for multi-label classification,
and CIDEr (Vedantam et al., 2015) for captioning and dialogues. Note we use
free-form text output to evaluate our model at all times. We use a single
model to evaluate on all benchmarks except for dialogues, and a chat model on
dialogues.

For zero-shot and few-shot benchmarks, these datasets are excluded from the
pre-training sets and SFT sets. For those derived from a parent dataset (e.g.
AudioCaps audio are derived from AudioSet), we removed the training samples
from the parent set as well as other child sets derived from that parent set.

Table 1: Evaluation of Audio Flamingo versus SOTA baseline methods on in-distribution benchmarks. Reference values are the SOTA models for each task. Audio Flamingo shows strong audio understanding ability on captioning, question answering, and audio classification. Dataset | Task | Metric | Previous SOTA $\uparrow$ | Ours $\uparrow$
---|---|---|---|---
Clotho-v2 | CAP | CIDEr |  $0.441$ (Chu et al., 2023) | $\mathbf{0.465}$
ClothoAQAunanimous | AQA | ACC |  $74.9\%$ (Chu et al., 2023) | $\mathbf{86.9\%}$
ClothoAQAnon-binary | AQA | ACC |  $29.1\%$ (Deshmukh et al., 2023) | $\mathbf{49.5\%}$
ClothoAQAnumerical | AQA | ACC |  $26.2\%$ (Deshmukh et al., 2023) | $\mathbf{36.4\%}$
MusicAVQAaudio-only | AQA | ACC |  $\mathbf{72.1\%}$ (Chu et al., 2023) | $71.6\%$
CochlScene | CLS | ACC |  $\mathbf{91.6\%}$ (Deshmukh et al., 2023) | $83.0\%$
NonSpeech17k | CLS | ACC |  $79.0\%$ (Rashid et al., 2023) | $\mathbf{85.1\%}$
FSD50k | CLS | F1approx |  $65.6\%$ (Deshmukh et al., 2023) | $\mathbf{69.7\%}$
NSinstrument | CLS | ACC |  $\mathbf{78.8\%}$ (Chu et al., 2023) | $77.1\%$
NSquality | CLS | F1 |  $46.3\%$ (Deshmukh et al., 2023) | $\mathbf{66.7\%}$
NSsource | CLS | ACC |  $60.1\%$ (Deshmukh et al., 2023) | $\mathbf{78.7\%}$
Average (besides CAP) |  |  | $62.4\%$ | $\textbf{70.5}\%$
Table 2: Evaluation of Audio Flamingo versus SOTA baseline methods on zero-shot benchmarks. Reference values are the SOTA models for each task. Audio Flamingo shows strong zero-shot generalization ability. Dataset | Task | Metric | Previous SOTA (0-shot) $\uparrow$ | Ours (0-shot) $\uparrow$
---|---|---|---|---
AudioCaps (Kim et al., 2019) | CAP | CIDEr |  $0.281$ (Salewski et al., 2023) | $\mathbf{0.502}$
CREMA-D (Cao et al., 2014) | CLS | ACC |  $18.5\%$ (Deshmukh et al., 2023) | $\mathbf{26.5\%}$
Ravdess (Livingstone & Russo, 2018) | CLS | ACC |  $\mathbf{21.7\%}$ (Elizalde et al., 2023b) | $20.9\%$
US8K (Salamon et al., 2014) | CLS | ACC |  $71.9\%$ (Deshmukh et al., 2023) | $\mathbf{75.0\%}$
GTZAN (Sturm, 2013) | CLS | ACC |  $\mathbf{71.0\%}$ (Han et al., 2023) | $67.9\%$
Medley-solos-DB (Lostanlen et al., 2019) | CLS | ACC |  $61.3\%$ (Deshmukh et al., 2023) | $\mathbf{92.7\%}$
Average (besides CAP) |  |  | $48.9\%$ | $\mathbf{56.6}\%$

###  5.2 Q1: Strong Audio Understanding Ability

We evaluate our model on several in-distribution (train-test) benchmarks, and
compare to state-of-the-art audio language model baselines. The results are
shown in Table 1. Note that we define F1approx to measure inexact but similar
predicted labels in FSD50k, where we consider the prediction to be correct if
SentenceBERT_similarity(output, ground_truth) $>0.8$ (Reimers & Gurevych,
2019, 2020). This metric is applied to outputs from baselines as well.

Audio Flamingo can match or outperform SOTA baselines – many of which are much
larger LLMs (7B (Gong et al., 2023c, b; Chu et al., 2023) or 13B (Tang et al.,
2023a)) – on most tasks, indicating our proposed method has strong audio
understanding ability. Our model also listens to the audio better. On
ClothoAQA, our model has $10.4\%$ higher accuracy than baselines on numerical
question answering, indicating our model understands the number of occurrences
better. On NSynth, our model has $20.4\%$ higher F1 on quality prediction and
$18.6\%$ higher accuracy on source prediction, indicating our model
understands the overall quality of audio better. In Appendix C.2, we use
qualitative samples to show that our model understands the order of appearance
of different sounds, perceives loudness and its change over time, and
perceives the distances of sounds from different objects.

###  5.3 Q2: In-Context Few-Shot Learning

We aim to measure the effect of ICL-based RAG in Audio Flamingo when it is
evaluated on unseen datasets.

First, we report the results on several zero-shot benchmarks and comparison
with SOTA zero-shot methods in Table 2. The results indicate our method is
better on most tasks and has strong generalization ability.

We then apply ICL-based RAG to these benchmarks. We compare to our zero-shot
results and the SOTA baseline of audio captioning on AudioCaps. The results on
classification are shown in Table 3, and the comparison on retrieval-augmented
audio captioning is shown in Table 4. As expected, there is consistent
improvement over zero-shot results, with an average improvement over $10\%$
for classification. Our method also significantly outperforms the SOTA
retrieval-augmented audio captioning method on AudioCaps. In Appendix C.1, we
show Audio Flamingo can adapt to unseen labels. In Appendix C.3, we show Audio
Flamingo looks at related retrieval (e.g., by taking key words from retrieved
captions), and ignores noisy retrieval.

###  5.4 Q3: Multi-Turn Dialogues

We measure Audio Flamingo’s ability to answer questions in a multi-turn
dialogue setting. The context is more complex and strongly correlated between
rounds (e.g. there exist many pronouns and follow-up questions). We fine-tune
Audio Flamingo on the two sets that we generated (AF-Dialogue-AudioSetSL and
AF-Dialogue-MusicCaps) to obtain a chat model. We evaluate the chat model on
the test split of these two dialogue datasets. We take user instructions and
let the model generate answers turn-by-turn (where previous generated answers
become the chatting history for next generation). We compare to Qwen-Audio
(Chu et al., 2023) and LTU (Gong et al., 2023c) in Table 5. 333While the
baseline methods claimed to support multi-turn dialogues, we were unable to
find quantitative evidence. Our chat model achieves significantly better
results than baseline methods. In Appendix C.4, we use qualitative samples to
show that our chat model captures context such as prior information and
pronouns better.

Table 3: Evaluation of few-shot results of Audio Flamingo with ICL-based RAG. $\Delta$ is the absolute improvement of few-shot over zero-shot results in Table 2. ICL-based RAG leads to consistent improvement over zero-shot results. Dataset | Ours (8-shot) $\uparrow$ |  $\Delta$ $\uparrow$
---|---|---
CREMA-D | $31.8\%$ | $5.3\%$
Ravdess | $35.2\%$ | $14.3\%$
US8K | $94.7\%$ | $19.4\%$
GTZAN | $79.5\%$ | $11.6\%$
Medley-solos-DB | $95.7\%$ | $3.0\%$
Average | $67.4\%$ | $10.7\%$
Table 4: Evaluation of retrieval-augmented audio captioning on AudioCaps. We compare Audio Flamingo to the SOTA baseline RECAP (Ghosh et al., 2023). Audio Flamingo achieves significantly better results than RECAP. Method | RECAP | Ours | Ours
---|---|---|---
# Shots | $4$ | $4$ | $8$
CIDEr $\uparrow$ | $0.362$ | $0.518$ | $\mathbf{0.538}$
Table 5: Evaluation of Audio Flamingo versus baseline methods on the multi-turn dialogue test sets. A stands for AF-Dialogue-AudioSetSL and M stands for AF-Dialogue-MusicCaps. We report CIDEr, Bleu4 (Papineni et al., 2002), and Rouge-L (R-L) (Lin, 2004). Audio Flamingo significantly outperforms larger baseline models, indicating strong dialogue ability of our proposed model. Testset | Method | CIDEr $\uparrow$ | Bleu4 $\uparrow$ | R-L $\uparrow$
---|---|---|---|---
A | LTU | $0.580$ | $0.122$ | $0.324$
A | Qwen-Audio | $0.507$ | $0.060$ | $0.292$
A | Ours | $\mathbf{1.622}$ | $\mathbf{0.237}$ | $\mathbf{0.473}$
M | LTU | $0.168$ | $0.065$ | $0.217$
M | Ours | $\mathbf{1.143}$ | $\mathbf{0.142}$ | $\mathbf{0.417}$
Figure 5: Relative improvement of few-shot results over zero-shot results
under different number of ICL samples. Using more ICL samples consistently
improves few-shot results, and the benefit is dataset-dependent.

###  5.5 Q4: Ablation Studies

Effect of number of few-shot samples. We reduce the number of in-context few-
shot samples to 4 and evaluate on the few-shot benchmarks. In Figure 5, we
plot the relative improvements over zero-shot results:
$(\mathrm{ACC_{few\text{-}shot}}-\mathrm{ACC_{0\text{-}shot}})/\mathrm{ACC_{0\text{-}shot}}$.
We find adding more ICL samples consistently improves few-shot results, and
the benefit is dataset-dependent.

Effect of LM. We found using an instruction-tuned LLM to be key to the success
of audio understanding. In early experiments we investigated opt-1.3b/2.7b
(Zhang et al., 2022) and phi-1.5 (Li et al., 2023b) and found the audio
understanding results to be slightly but consistently worse.

Effect of audio encoder. In early experiments, we tested all variants of
LAION-CLAP (Wu et al., 2023) and Microsoft-CLAP (Elizalde et al., 2023a, b) as
the audio encoder backbone. We found ClapCap (Elizalde et al., 2023b) performs
the best overall – in terms of both evaluation results and computing speed –
whereas a large LAION-CLAP trained on more music data led to better accuracy
on music information retrieval.

##  6 Conclusion and Future Work

In this paper, we present Audio Flamingo, an audio language model with a
series of innovations that achieves the state-of-the-art results on several
close-ended and open-ended audio understanding tasks without task specific
fine-tuning. It also has strong ICL and RAG abilities, and has the state-of-
the-art few-shot learning results. Furthermore, we design a dataset generation
strategy and introduce two dialogue datasets, and enable Audio Flamingo to
chat with a user and achieve state-of-the-art results on dialogue benchmarks.

One important future direction is to investigate scaling strategies for using
larger LMs. Assuming that larger LMs could have better knowledge and stronger
ability to follow instructions, we believe that Audio Flamingo could benefit
from a larger LM. A second future direction is to investigate complex speech-
related tasks beyond transcription. A third future direction is to combine the
audio understanding abilities of our model with visual language models such
that one model could understand video with accompanied audio.

## Acknowledgement

We thank Siddharth Gururani, Zihan Liu, Mostofa Patwary, Shrimai Prabhumoye,
and Chen Zhu for helpful discussions. We thank Ke Chen and Yuan Gong for help
on sharing datasets.

## References

  * Achiam et al. (2023) Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al.  Gpt-4 technical report.  _arXiv preprint arXiv:2303.08774_ , 2023.
  * Adigwe et al. (2018) Adigwe, A., Tits, N., Haddad, K. E., Ostadabbas, S., and Dutoit, T.  The emotional voices database: Towards controlling the emotion dimension in voice generation systems.  _arXiv preprint arXiv:1806.09514_ , 2018.
  * Agostinelli et al. (2023) Agostinelli, A., Denk, T. I., Borsos, Z., Engel, J., Verzetti, M., Caillon, A., Huang, Q., Jansen, A., Roberts, A., Tagliasacchi, M., et al.  Musiclm: Generating music from text.  _arXiv preprint arXiv:2301.11325_ , 2023.
  * Alayrac et al. (2022) Alayrac, J.-B., Donahue, J., Luc, P., Miech, A., Barr, I., Hasson, Y., Lenc, K., Mensch, A., Millican, K., Reynolds, M., et al.  Flamingo: a visual language model for few-shot learning.  _Advances in Neural Information Processing Systems_ , 35:23716–23736, 2022.
  * Barros et al. (2018) Barros, P., Churamani, N., Lakomkin, E., Siqueira, H., Sutherland, A., and Wermter, S.  The omg-emotion behavior dataset.  In _2018 International Joint Conference on Neural Networks (IJCNN)_ , pp. 1–7. IEEE, 2018.
  * Bogdanov et al. (2019) Bogdanov, D., Won, M., Tovstogan, P., Porter, A., and Serra, X.  The mtg-jamendo dataset for automatic music tagging.  ICML, 2019.
  * Borgeaud et al. (2022) Borgeaud, S., Mensch, A., Hoffmann, J., Cai, T., Rutherford, E., Millican, K., Van Den Driessche, G. B., Lespiau, J.-B., Damoc, B., Clark, A., et al.  Improving language models by retrieving from trillions of tokens.  In _International conference on machine learning_ , pp. 2206–2240. PMLR, 2022.
  * Brown et al. (2020) Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al.  Language models are few-shot learners.  _Advances in neural information processing systems_ , 33:1877–1901, 2020.
  * Cao et al. (2014) Cao, H., Cooper, D. G., Keutmann, M. K., Gur, R. C., Nenkova, A., and Verma, R.  Crema-d: Crowd-sourced emotional multimodal actors dataset.  _IEEE transactions on affective computing_ , 5(4):377–390, 2014.
  * Cartwright et al. (2019) Cartwright, M., Mendez, A. E. M., Cramer, A., Lostanlen, V., Dove, G., Wu, H.-H., Salamon, J., Nov, O., and Bello, J.  Sonyc urban sound tagging (sonyc-ust): A multilabel dataset from an urban acoustic sensor network.  2019\.
  * Chen et al. (2022) Chen, K., Du, X., Zhu, B., Ma, Z., Berg-Kirkpatrick, T., and Dubnov, S.  Hts-at: A hierarchical token-semantic audio transformer for sound classification and detection.  In _ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pp. 646–650. IEEE, 2022.
  * Chen et al. (2023) Chen, Z., Huang, H., Andrusenko, A., Hrinchuk, O., Puvvada, K. C., Li, J., Ghosh, S., Balam, J., and Ginsburg, B.  Salm: Speech-augmented language model with in-context learning for speech recognition and translation.  _arXiv preprint arXiv:2310.09424_ , 2023.
  * Chu et al. (2023) Chu, Y., Xu, J., Zhou, X., Yang, Q., Zhang, S., Yan, Z., Zhou, C., and Zhou, J.  Qwen-audio: Advancing universal audio understanding via unified large-scale audio-language models.  _arXiv preprint arXiv:2311.07919_ , 2023.
  * Defferrard et al. (2016) Defferrard, M., Benzi, K., Vandergheynst, P., and Bresson, X.  Fma: A dataset for music analysis.  _arXiv preprint arXiv:1612.01840_ , 2016.
  * Défossez et al. (2022) Défossez, A., Copet, J., Synnaeve, G., and Adi, Y.  High fidelity neural audio compression.  _arXiv preprint arXiv:2210.13438_ , 2022.
  * Deshmukh et al. (2022) Deshmukh, S., Elizalde, B., and Wang, H.  Audio retrieval with wavtext5k and clap training.  _arXiv preprint arXiv:2209.14275_ , 2022.
  * Deshmukh et al. (2023) Deshmukh, S., Elizalde, B., Singh, R., and Wang, H.  Pengi: An audio language model for audio tasks.  _arXiv preprint arXiv:2305.11834_ , 2023.
  * Doh et al. (2023) Doh, S., Choi, K., Lee, J., and Nam, J.  Lp-musiccaps: Llm-based pseudo music captioning.  _arXiv preprint arXiv:2307.16372_ , 2023.
  * Driess et al. (2023) Driess, D., Xia, F., Sajjadi, M. S., Lynch, C., Chowdhery, A., Ichter, B., Wahid, A., Tompson, J., Vuong, Q., Yu, T., et al.  Palm-e: An embodied multimodal language model.  _arXiv preprint arXiv:2303.03378_ , 2023.
  * Drossos et al. (2020) Drossos, K., Lipping, S., and Virtanen, T.  Clotho: An audio captioning dataset.  In _ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pp. 736–740. IEEE, 2020.
  * Duan et al. (2023) Duan, H., Wei, J., Wang, C., Liu, H., Fang, Y., Zhang, S., Lin, D., and Chen, K.  Botchat: Evaluating llms’ capabilities of having multi-turn dialogues.  _arXiv preprint arXiv:2310.13650_ , 2023.
  * Elizalde et al. (2023a) Elizalde, B., Deshmukh, S., Al Ismail, M., and Wang, H.  Clap learning audio concepts from natural language supervision.  In _ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pp. 1–5. IEEE, 2023a.
  * Elizalde et al. (2023b) Elizalde, B., Deshmukh, S., and Wang, H.  Natural language supervision for general-purpose audio representations, 2023b.  URL https://arxiv.org/abs/2309.05767.
  * Engel et al. (2017) Engel, J., Resnick, C., Roberts, A., Dieleman, S., Eck, D., Simonyan, K., and Norouzi, M.  Neural audio synthesis of musical notes with wavenet autoencoders, 2017\.
  * Fonseca et al. (2021) Fonseca, E., Favory, X., Pons, J., Font, F., and Serra, X.  Fsd50k: an open dataset of human-labeled sound events.  _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , 30:829–852, 2021.
  * Foster et al. (2015) Foster, P., Sigtia, S., Krstulovic, S., Barker, J., and Plumbley, M. D.  Chime-home: A dataset for sound source recognition in a domestic environment.  In _2015 IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA)_ , pp. 1–5. IEEE, 2015.
  * Gao et al. (2022) Gao, H., Ni, J., Qian, K., Zhang, Y., Chang, S., and Hasegawa-Johnson, M.  Wavprompt: Towards few-shot spoken language understanding with frozen language models.  _arXiv preprint arXiv:2203.15863_ , 2022.
  * Gardner et al. (2023) Gardner, J., Durand, S., Stoller, D., and Bittner, R. M.  Llark: A multimodal foundation model for music.  _arXiv preprint arXiv:2310.07160_ , 2023.
  * Gemmeke et al. (2017) Gemmeke, J. F., Ellis, D. P., Freedman, D., Jansen, A., Lawrence, W., Moore, R. C., Plakal, M., and Ritter, M.  Audio set: An ontology and human-labeled dataset for audio events.  In _2017 IEEE international conference on acoustics, speech and signal processing (ICASSP)_ , pp. 776–780. IEEE, 2017.
  * Ghosh et al. (2023) Ghosh, S., Kumar, S., Evuru, C. K. R., Duraiswami, R., and Manocha, D.  Recap: Retrieval-augmented audio captioning.  _arXiv preprint arXiv:2309.09836_ , 2023.
  * Gong et al. (2021) Gong, Y., Chung, Y.-A., and Glass, J.  Ast: Audio spectrogram transformer.  _arXiv preprint arXiv:2104.01778_ , 2021.
  * Gong et al. (2023a) Gong, Y., Khurana, S., Karlinsky, L., and Glass, J.  Whisper-at: Noise-robust automatic speech recognizers are also strong general audio event taggers.  _arXiv preprint arXiv:2307.03183_ , 2023a.
  * Gong et al. (2023b) Gong, Y., Liu, A., Luo, H., Karlinsky, L., and Glass, J.  Joint audio and speech understanding.  In _IEEE Automatic Speech Recognition and Understanding Workshop_ , 2023b.
  * Gong et al. (2023c) Gong, Y., Luo, H., Liu, A. H., Karlinsky, L., and Glass, J.  Listen, think, and understand.  _arXiv preprint arXiv:2305.10790_ , 2023c.
  * Guu et al. (2020) Guu, K., Lee, K., Tung, Z., Pasupat, P., and Chang, M.  Retrieval augmented language model pre-training.  In _International conference on machine learning_ , pp. 3929–3938. PMLR, 2020.
  * Han et al. (2023) Han, J., Zhang, R., Shao, W., Gao, P., Xu, P., Xiao, H., Zhang, K., Liu, C., Wen, S., Guo, Z., et al.  Imagebind-llm: Multi-modality instruction tuning.  _arXiv preprint arXiv:2309.03905_ , 2023.
  * Hershey et al. (2021) Hershey, S., Ellis, D. P., Fonseca, E., Jansen, A., Liu, C., Moore, R. C., and Plakal, M.  The benefit of temporally-strong labels in audio event classification.  In _ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pp. 366–370. IEEE, 2021.
  * Hsu et al. (2023) Hsu, M.-H., Chang, K.-W., Li, S.-W., and Lee, H.-y.  An exploration of in-context learning for speech language model.  _arXiv preprint arXiv:2310.12477_ , 2023.
  * Huang et al. (2022) Huang, Q., Jansen, A., Lee, J., Ganti, R., Li, J. Y., and Ellis, D. P.  Mulan: A joint embedding of music audio and natural language.  _arXiv preprint arXiv:2208.12415_ , 2022.
  * Huang et al. (2023) Huang, R., Li, M., Yang, D., Shi, J., Chang, X., Ye, Z., Wu, Y., Hong, Z., Huang, J., Liu, J., et al.  Audiogpt: Understanding and generating speech, music, sound, and talking head.  _arXiv preprint arXiv:2304.12995_ , 2023.
  * Iyer et al. (2022) Iyer, S., Lin, X. V., Pasunuru, R., Mihaylov, T., Simig, D., Yu, P., Shuster, K., Wang, T., Liu, Q., Koura, P. S., et al.  Opt-iml: Scaling language model instruction meta learning through the lens of generalization.  _arXiv preprint arXiv:2212.12017_ , 2022.
  * James et al. (2018) James, J., Tian, L., and Watson, C.  An open source emotional speech corpus for human robot interaction applications.  _Interspeech 2018_ , 2018.
  * Jeong & Park (2022) Jeong, I.-Y. and Park, J.  Cochlscene: Acquisition of acoustic scene data using crowdsourcing.  In _2022 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC)_ , pp. 17–21. IEEE, 2022\.
  * Johnson et al. (2019) Johnson, J., Douze, M., and Jégou, H.  Billion-scale similarity search with GPUs.  _IEEE Transactions on Big Data_ , 7(3):535–547, 2019.
  * Karpukhin et al. (2020) Karpukhin, V., Oğuz, B., Min, S., Lewis, P., Wu, L., Edunov, S., Chen, D., and Yih, W.-t.  Dense passage retrieval for open-domain question answering.  _arXiv preprint arXiv:2004.04906_ , 2020.
  * Khomenko et al. (2016) Khomenko, V., Shyshkov, O., Radyvonenko, O., and Bokhan, K.  Accelerating recurrent neural network training using sequence bucketing and multi-gpu data parallelization.  In _2016 IEEE First International Conference on Data Stream Mining & Processing (DSMP)_, pp. 100–103. IEEE, 2016.
  * Kim et al. (2019) Kim, C. D., Kim, B., Lee, H., and Kim, G.  Audiocaps: Generating captions for audios in the wild.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pp. 119–132, 2019.
  * Kong et al. (2020) Kong, Q., Cao, Y., Iqbal, T., Wang, Y., Wang, W., and Plumbley, M. D.  Panns: Large-scale pretrained audio neural networks for audio pattern recognition.  _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , 28:2880–2894, 2020.
  * Lewis et al. (2020) Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W.-t., Rocktäschel, T., et al.  Retrieval-augmented generation for knowledge-intensive nlp tasks.  _Advances in Neural Information Processing Systems_ , 33:9459–9474, 2020.
  * Li et al. (2022) Li, G., Wei, Y., Tian, Y., Xu, C., Wen, J.-R., and Hu, D.  Learning to answer questions in dynamic audio-visual scenarios.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 19108–19118, 2022.
  * Li et al. (2023a) Li, J., Li, D., Savarese, S., and Hoi, S.  Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models.  _arXiv preprint arXiv:2301.12597_ , 2023a.
  * Li et al. (2023b) Li, Y., Bubeck, S., Eldan, R., Del Giorno, A., Gunasekar, S., and Lee, Y. T.  Textbooks are all you need ii: phi-1.5 technical report.  _arXiv preprint arXiv:2309.05463_ , 2023b.
  * Li et al. (2023c) Li, Y., Yuan, R., Zhang, G., Ma, Y., Chen, X., Yin, H., Lin, C., Ragni, A., Benetos, E., Gyenge, N., et al.  Mert: Acoustic music understanding model with large-scale self-supervised training.  _arXiv preprint arXiv:2306.00107_ , 2023c.
  * Lin (2004) Lin, C.-Y.  Rouge: A package for automatic evaluation of summaries.  In _Text summarization branches out_ , pp. 74–81, 2004.
  * Lipping et al. (2022) Lipping, S., Sudarsanam, P., Drossos, K., and Virtanen, T.  Clotho-aqa: A crowdsourced dataset for audio question answering.  In _2022 30th European Signal Processing Conference (EUSIPCO)_ , pp. 1140–1144. IEEE, 2022.
  * Liu et al. (2023a) Liu, H., Li, C., Wu, Q., and Lee, Y. J.  Visual instruction tuning.  _arXiv preprint arXiv:2304.08485_ , 2023a.
  * Liu et al. (2023b) Liu, S., Hussain, A. S., Sun, C., and Shan, Y.  Music understanding llama: Advancing text-to-music generation with question answering and captioning.  _arXiv preprint arXiv:2308.11276_ , 2023b.
  * Livingstone & Russo (2018) Livingstone, S. R. and Russo, F. A.  The ryerson audio-visual database of emotional speech and song (ravdess): A dynamic, multimodal set of facial and vocal expressions in north american english.  _PloS one_ , 13(5):e0196391, 2018.
  * Loshchilov & Hutter (2017) Loshchilov, I. and Hutter, F.  Decoupled weight decay regularization.  _arXiv preprint arXiv:1711.05101_ , 2017.
  * Lostanlen et al. (2019) Lostanlen, V., Cella, C.-E., Bittner, R., and Essid, S.  Medley-solos-DB: a cross-collection dataset for musical instrument recognition, February 2019.  URL https://doi.org/10.5281/zenodo.1344103.
  * Lotfian & Busso (2017) Lotfian, R. and Busso, C.  Building naturalistic emotionally balanced speech corpus by retrieving emotional speech from existing podcast recordings.  _IEEE Transactions on Affective Computing_ , 10(4):471–483, 2017.
  * Lyu et al. (2023) Lyu, C., Wu, M., Wang, L., Huang, X., Liu, B., Du, Z., Shi, S., and Tu, Z.  Macaw-llm: Multi-modal language modeling with image, audio, video, and text integration.  _arXiv preprint arXiv:2306.09093_ , 2023.
  * Martin Morato & Mesaros (2021) Martin Morato, I. and Mesaros, A.  Diversity and bias in audio captioning datasets.  2021\.
  * Mei et al. (2023) Mei, X., Meng, C., Liu, H., Kong, Q., Ko, T., Zhao, C., Plumbley, M. D., Zou, Y., and Wang, W.  Wavcaps: A chatgpt-assisted weakly-labelled audio captioning dataset for audio-language multimodal research.  _arXiv preprint arXiv:2303.17395_ , 2023.
  * (65) Mohanty, S. P.  Sound Of 114 Species Of Birds Till 2022.  URL https://www.kaggle.com/datasets/soumendraprasad/sound-of-114-species-of-birds-till-2022.
  * Moon et al. (2023) Moon, S., Madotto, A., Lin, Z., Nagarajan, T., Smith, M., Jain, S., Yeh, C.-F., Murugesan, P., Heidari, P., Liu, Y., et al.  Anymal: An efficient and scalable any-modality augmented language model.  _arXiv preprint arXiv:2309.16058_ , 2023.
  * Oncescu et al. (2021) Oncescu, A.-M., Koepke, A., Henriques, J. F., Akata, Z., and Albanie, S.  Audio retrieval with natural language queries.  _arXiv preprint arXiv:2105.02192_ , 2021.
  * Ouyang et al. (2022) Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al.  Training language models to follow instructions with human feedback.  _Advances in Neural Information Processing Systems_ , 35:27730–27744, 2022.
  * Papineni et al. (2002) Papineni, K., Roukos, S., Ward, T., and Zhu, W.-J.  Bleu: a method for automatic evaluation of machine translation.  In _Proceedings of the 40th annual meeting of the Association for Computational Linguistics_ , pp. 311–318, 2002.
  * Park et al. (2022) Park, J., Cho, Y., Sim, G., Lee, H., and Choo, J.  Enemy spotted: In-game gun sound dataset for gunshot classification and localization.  In _2022 IEEE Conference on Games (CoG)_ , pp. 56–63. IEEE, 2022\.
  * Pichora-Fuller & Dupuis (2020) Pichora-Fuller, M. K. and Dupuis, K.  Toronto emotional speech set (TESS), 2020.  URL https://doi.org/10.5683/SP2/E8H2MF.
  * Poria et al. (2018) Poria, S., Hazarika, D., Majumder, N., Naik, G., Cambria, E., and Mihalcea, R.  Meld: A multimodal multi-party dataset for emotion recognition in conversations.  _arXiv preprint arXiv:1810.02508_ , 2018.
  * Radford et al. (2023) Radford, A., Kim, J. W., Xu, T., Brockman, G., McLeavey, C., and Sutskever, I.  Robust speech recognition via large-scale weak supervision.  In _International Conference on Machine Learning_ , pp. 28492–28518. PMLR, 2023.
  * Rafii et al. (2019) Rafii, Z., Liutkus, A., Stöter, F.-R., Mimilakis, S. I., and Bittner, R.  Musdb18-hq - an uncompressed version of musdb18, August 2019.  URL https://doi.org/10.5281/zenodo.3338373.
  * Rashid et al. (2023) Rashid, M. M., Li, G., and Du, C.  Nonspeech7k dataset: Classification and analysis of human non-speech sound.  _IET Signal Processing_ , 17(6):e12233, 2023.
  * Reimers & Gurevych (2019) Reimers, N. and Gurevych, I.  Sentence-bert: Sentence embeddings using siamese bert-networks.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing_. Association for Computational Linguistics, 11 2019\.  URL https://arxiv.org/abs/1908.10084.
  * Reimers & Gurevych (2020) Reimers, N. and Gurevych, I.  Making monolingual sentence embeddings multilingual using knowledge distillation.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing_. Association for Computational Linguistics, 11 2020\.  URL https://arxiv.org/abs/2004.09813.
  * Rubenstein et al. (2023) Rubenstein, P. K., Asawaroengchai, C., Nguyen, D. D., Bapna, A., Borsos, Z., Quitry, F. d. C., Chen, P., Badawy, D. E., Han, W., Kharitonov, E., et al.  Audiopalm: A large language model that can speak and listen.  _arXiv preprint arXiv:2306.12925_ , 2023.
  * Salamon et al. (2014) Salamon, J., Jacoby, C., and Bello, J. P.  A dataset and taxonomy for urban sound research.  In _Proceedings of the 22nd ACM international conference on Multimedia_ , pp. 1041–1044, 2014.
  * Salewski et al. (2023) Salewski, L., Fauth, S., Koepke, A., and Akata, Z.  Zero-shot audio captioning with audio-language model guidance and audio context keywords.  _arXiv preprint arXiv:2311.08396_ , 2023.
  * Sturm (2013) Sturm, B. L.  The gtzan dataset: Its contents, its faults, their effects on evaluation, and its future use.  _arXiv preprint arXiv:1306.1461_ , 2013.
  * Tang et al. (2023a) Tang, C., Yu, W., Sun, G., Chen, X., Tan, T., Li, W., Lu, L., Ma, Z., and Zhang, C.  Salmonn: Towards generic hearing abilities for large language models.  _arXiv preprint arXiv:2310.13289_ , 2023a.
  * Tang et al. (2023b) Tang, Z., Yang, Z., Khademi, M., Liu, Y., Zhu, C., and Bansal, M.  Codi-2: In-context, interleaved, and interactive any-to-any generation.  _arXiv preprint arXiv:2311.18775_ , 2023b.
  * Tsimpoukelli et al. (2021) Tsimpoukelli, M., Menick, J. L., Cabi, S., Eslami, S., Vinyals, O., and Hill, F.  Multimodal few-shot learning with frozen language models.  _Advances in Neural Information Processing Systems_ , 34:200–212, 2021.
  * Vaswani et al. (2017) Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., and Polosukhin, I.  Attention is all you need.  _Advances in neural information processing systems_ , 30, 2017.
  * Vedantam et al. (2015) Vedantam, R., Lawrence Zitnick, C., and Parikh, D.  Cider: Consensus-based image description evaluation.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pp. 4566–4575, 2015.
  * Wang et al. (2023) Wang, S., Yang, C.-H. H., Wu, J., and Zhang, C.  Can whisper perform speech-based in-context learning.  _arXiv preprint arXiv:2309.07081_ , 2023.
  * Wei et al. (2021) Wei, J., Bosma, M., Zhao, V. Y., Guu, K., Yu, A. W., Lester, B., Du, N., Dai, A. M., and Le, Q. V.  Finetuned language models are zero-shot learners.  _arXiv preprint arXiv:2109.01652_ , 2021.
  * Won et al. (2023) Won, M., Hung, Y.-N., and Le, D.  A foundation model for music informatics.  _arXiv preprint arXiv:2311.03318_ , 2023.
  * Wu et al. (2023) Wu, Y., Chen, K., Zhang, T., Hui, Y., Berg-Kirkpatrick, T., and Dubnov, S.  Large-scale contrastive language-audio pretraining with feature fusion and keyword-to-caption augmentation.  In _ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pp. 1–5. IEEE, 2023.
  * Yang et al. (2023) Yang, Z., Ping, W., Liu, Z., Korthikanti, V., Nie, W., Huang, D.-A., Fan, L., Yu, Z., Lan, S., Li, B., et al.  Re-vilm: Retrieval-augmented visual language model for zero and few-shot image captioning.  In _EMNLP_ , 2023.
  * Zhang et al. (2022) Zhang, S., Roller, S., Goyal, N., Artetxe, M., Chen, M., Chen, S., Dewan, C., Diab, M., Li, X., Lin, X. V., et al.  Opt: Open pre-trained transformer language models.  _arXiv preprint arXiv:2205.01068_ , 2022.
  * Zhao et al. (2023) Zhao, Z., Guo, L., Yue, T., Chen, S., Shao, S., Zhu, X., Yuan, Z., and Liu, J.  Chatbridge: Bridging modalities with large language model as a language catalyst.  _arXiv preprint arXiv:2305.16103_ , 2023.

##  Appendix A Dataset Staging, Weights, and Templates

Table 6 includes an overview of datasets (by type) we use to train Audio
Flamingo.

Table 6: All datasets used to train our model. The total number of audio-text pairs is approximately 5.9 million. The total length of audio is approximately 18.1 thousand hours. Audio Type | Task | Datasets | #Audio-Text Pairs
---|---|---|---
General Sound | CAP |  WavCaps (Mei et al., 2023), Macs (Martin Morato & Mesaros, 2021), | $\sim$829 K
SoundDescs (Oncescu et al., 2021), Clotho-v2 (Drossos et al., 2020),
WavText5K (Deshmukh et al., 2022), LAION-630k (Wu et al., 2023)
AQA |  Clotho-AQA (Lipping et al., 2022), Open-AQA (Gong et al., 2023b) |  $\sim$1970 K
CLS |  AudioSet (Gemmeke et al., 2017), FSD50k (Fonseca et al., 2021), | $\sim$1091 K
CochlScene (Jeong & Park, 2022), NonSpeech7K (Rashid et al., 2023),
Chime-Home (Foster et al., 2015), Sonyc-UST (Cartwright et al., 2019)
Music | CAP |  LP-MusicCaps (Doh et al., 2023), MusicCaps (Agostinelli et al., 2023) |  $\sim$1389 K
AQA |  MusicQA (Liu et al., 2023b), MusicAVQA (Li et al., 2022) |  $\sim$94 K
CLS |  NSynth (Engel et al., 2017), MTG-Jamendo (Bogdanov et al., 2019), | $\sim$459 K
FMA (Defferrard et al., 2016), MusDB-HQ (Rafii et al., 2019),
Speech | CLS |  MSP-Podcast (Lotfian & Busso, 2017), Emov-DB (Adigwe et al., 2018) | $\sim$92 K
JL-Corpus (James et al., 2018), Tess (Pichora-Fuller & Dupuis, 2020),
MELD (Poria et al., 2018), OMGEmotion (Barros et al., 2018)

We construct instructions for each task and dataset. Below are all instruction
templates we use.

Audio Captioning: $\circ$ Describe the sound/music in a sentence. $\circ$
Describe the sound/music at length. Audio Question Answering: $\circ$
{question} $\circ$ Please answer this question: {question} $\circ$ Please
answer this question: {question}. Options:$\backslash$n- yes$\backslash$n- no
Audio Classification: $\circ$ Classify this sound. (Options: ...) $\circ$
Describe the sound in {number} words. $\circ$ What is the emotion of this
speech? (Options: ...) $\circ$ What is the instrument/genre of this music?
(Options: ...) $\circ$ This music note is produced by

The detailed pre-training datasets and their number of epochs are shown in
Table 7. The detailed SFT datasets and their number of epochs are shown in
Table 8.

Table 7: Pre-training datasets and epochs. Dataset | Audio Length | #Audio-Text Pairs | Epochs
---|---|---|---
OpenAQA | 693.2 hrs | 1959.8K | $1.0$
Laion630kBBCSoundEffects | 456.9 hrs | 15.1K | $5.0$
Laion630kFreesound | 2494.8 hrs | 306.5K | $1.0$
SoundDescs | 749.7 hrs | 23.1K | $1.0$
WavCaps | 3793.3 hrs | 402.6 K | $1.75$
AudioSet | 2617.8 hrs | 950.8K | $1.0$
WavText5K | 23.8 hrs | 4.3K | $3.0$
MSP-Podcast | 73.9 hrs | 45.1K | $1.2$
MELD | 8.7 hrs | 32.9K | $2.4$
MusicAVQAaudio-visual | 142.4 hrs | 17.9K | $3.0$
MusicQA | 62.9 hrs | 70K | $1.2$
LP-MusicCapsMSD | 5805.7 hrs | 1331.8K | $1.0$
NSynth | 321.3 hrs | 289.2K | $0.4$
MTG-Jamendo | 3768.9 hrs | 55.6K | $1.0$
Table 8: SFT datasets and epochs. Dataset | Audio Length | #Audio-Text Pairs | Epochs | ICL Dataset Epochs
---|---|---|---|---
ClothoAQA | 7.4 hrs | 9.7K | $3.5$ | $0.5$
OpenAQA | 693.2 hrs | 1959.8K | $0.1$ | -
Clotho-v2 | 24.0 hrs | 19.2K | $2.0$ | $0.5$
Laion630kEpidemic | 209.4 hrs | 40.7K | $0.8$ | $0.2$
MACS | 10.9 hrs | 17.3K | $0.8$ | $0.2$
FSD50k | 80.8 hrs | 41.0K | $0.9$ | $0.3$
CochlScene | 169.0 hrs | 60.9K | $1.2$ | $0.3$
NonSpeech 7k | 6.2 hrs | 6.3K | $2.4$ | $0.6$
Chime-home | 5.0 hrs | 4.5K | $1.5$ | $0.5$
Sonyc-UST | 34.9 hrs | 27.9K | $0.8$ | $0.2$
Emov-DB | 7.8 hrs | 6.8K | $1.6$ | $0.4$
JL-Corpus | 1.4 hrs | 2.4K | $6.0$ | $1.5$
Tess | 1.6 hrs | 2.8K | $2.0$ | $0.5$
OMGEmotion | 3.0 hrs | 1.7K | $3.0$ | -
MusicAVQAaudio-only | 77.1 hrs | 5.7K | $5.0$ | $1.0$
MusicQA | 62.9 hrs | 70K | $0.35$ | $0.05$
LP-MusicCapsMSD | 5805.7 hrs | 1331.8K | $0.025$ | $0.007$
LP-MusicCapsMTT | 126.4 hrs | 46.9K | $0.8$ | $0.2$
LP-MusicCapsMC | 7.4 hrs | 7.9K | $2.0$ | -
MusicCaps | 7.4 hrs | 2.6K | $6.0$ | -
NSynth | 321.3 hrs | 289.2K | $1.0$ | $1.0$
MTG-Jamendo | 3768.9 hrs | 55.6K | $0.1$ | -
MusDB-HQ | 29.1 hrs | 10.2K | $1.0$ | -
FMA | 860.7 hrs | 104.2K | $0.4$ | $0.1$

##  Appendix B Generated dialogue datasets

###  B.1 Overview

In this section, we introduce methods to generate our AF-Dialogue-AudioSetSL
and AF-Dialogue-MusicCaps datasets with GPT-4 (Achiam et al., 2023). AF-
Dialogue-AudioSetSL is generated based on the annotated events and timestamps
of strongly labeled AudioSet-SL (Gemmeke et al., 2017; Hershey et al., 2021).
There are 76k dialogues in the train split and 1.5k dialogues in test split.
AF-Dialogue-MusicCaps is generated based on tags and descriptions of MusicCaps
(Agostinelli et al., 2023). There are 3.3k dialogues in the train split and
1.4k dialogues in the test split. Each training sample has one to four rounds
(QA pairs), and each testing sample has three or four rounds.

###  B.2 Prompts

Below we show the prompts used to generate AF-Dialogue-AudioSetSL and AF-
Dialogue-MusicCaps.

sys prompt: Based on the sound events, create a dialogue between you (the
assistant) and a person (the user) about the events in the audio. Each
dialogue should consist of: 1\. A user examines the audio, and sends a
reasonable and creative message to the assistant. 2\. Once the audio is
provided, the assistant thoroughly perceives and comprehends them, responding
with helpful answers that provide comprehensive reasoning. Do not include
timestamps in the answer provided by the assistant. 3\. Considering the past
dialogue i.e. the question and the answer in the previous timestep, the user
chooses another question or further inquiry. 4\. The assistant provides a
helpful answer providing comprehensive reasoning regarding the audio content.
The audios are 10 seconds long and the sound events are segmented based on
where they occur in the audio. All the questions asked by the user should be
diverse and complex. Example 1: Sound Events: Sound of Ambulance (siren)
(High-pitched and wailing): [0.0s-1.0s]; Sound of Traffic noise, roadway noise
(Droning, loud and intrusive): [0.0s-10.0s]; Sound of Accelerating, revving,
vroom (Highpitched, short and intense): [2.0s-10.0s]; Sound of Generic impact
sounds (Loud and sharp): [6.7s-6.8s]. Dialogue: User: What does the audio
sound like? Assistant: An ambulance siren echoes while traffic noise fades,
and an engine revs. User: Can you describe in more detail about the engine
revs? Assistant: The engine revving is highpitched, short and intense. User:
From the highpitched sound, can you tell me what vehicle it was? Assistant: It
is hard to tell this from the information provided. User: Are there any other
events in the audio that might be relevant? Assistant: There is a sound of
impact which is very loud and sharp. Example 2: Sound Events: Sound of
Outside, rural or natural: [0.000s-10.000s]; Sound of Liquid: [0.474s-0.680s],
[4.249s-5.292s]; Sound of Gurgling, bubbling: [0.474s-0.680s],
[4.249s-5.292s]; Sound of Splash, splatter: [1.620s-3.116s]; Sound of
Laughter: [2.745s-4.029s], [5.299s-10.000s]; Sound of Chuckle, chortle:
[2.752s-4.029s]; Sound of Speech: [4.551s-4.935s]; Sound of Male speech, man
speaking: [4.557s-4.935s]; Sound of Belly laugh: [5.292s-10.000s] Dialogue:
User: What are the main events in the audio? Assistant: Liquid sounds,
laughter, speech, and a man speaking can be heard in a rural setting. User: Is
is he speaking to other people? Assistant: As there is sound of laughter
present from other people, the man might be speaking to others. User: Can you
explain in more detail about the first event? Assistant: The liquid sounds are
more of splashing, thrashing and splattering sounds. User: Are there other
people laughing with him? Assistant: Yes, there are other laughter voices
along with that of the man. Based on the above examples, design a four-turn
dialogue between the user and assistant for the example given below. The
question asked by the user should use pronouns such as ‘he’, ‘she’, ‘it’,
‘they’ and ‘them’ while referring to entities that have already occurred in
the past questions or answers. Format each QA pair in a single line as a JSON
dictionary (key “user” for question, and “assistant” for answer, wrapped with
{ and }). Do not include any other explanation. Example 3:

sys prompt: Based on the music, create a dialogue between you (the assistant)
and a person (the user) about the events in the music audio. Each dialogue
should consist of: 1\. A user examines the music, and sends a reasonable and
creative message to the assistant. 2\. Once the music audio is provided, the
assistant thoroughly perceives and comprehends them, responding with helpful
answers that provide comprehensive reasoning. Do not include timestamps in the
answer provided by the assistant. 3\. Considering the past dialogue i.e. the
question and the answer in the previous timestep, the user chooses another
question or further inquiry. 4\. The assistant provides a helpful answer
providing comprehensive reasoning regarding the music content. The more
complex and diverse the question, the better. You can use the list of
questions below while asking questions. List of questions:  Is it a live
recording or a studio recording? Is the guitar acoustic or electric? Is it
playing chords, a riff or a solo? What music style or genre is it? What key is
it in? How about the time signature? What’s the tempo? Slow, Medium or Fast?
Does it have vocals? Is it low pitched or high pitched? Is the person speaking
fast or slow? What is the language and accent? What emotion is conveyed by the
voice? What are the vocal characteristics of that emotion? How old is the
person likely to be given their vocal characteristics? Example 1: Music
description: House music with a thought provoking message in the beginning
that eventually builds up and can be used to create the sense of a social
revolution. Dialogue: User: What type of music is this? Assistant: It is house
music with synth and drums and a man speaking at the beginning. User: What is
his tone while speaking? Example 2: He has an intense and energetic tone in
his voice. User: Does he speak till the end? Assistant: No, he stops speaking
halfway and then it is followed by house music. User: What is the source of
this music? Assistant: This is electronic music. Example 2:  Music
description: Energetic bluesy song with a harmonica and horn section in
musical dialogue. Dialogue: User: What instruments are playing in this music?
Assistant: This is a jazzy tune with harmonica, trumpet and drums playing.
User: Which key is the music playing in? Assistant: The key is A Major. User:
How will you explain the tempo of this piece? Assistant: The music is bouncy
and positive with high beats per minute in the beginning and then changes
signature halfway through. User: What is the source of this music? Assistant:
This is fusion jazz as it has elements of bass and synthesizer. Based on above
examples, design a four-turn dialogue between the user and assistant for the
example given below. The question asked by the user should use pronouns such
as ‘he’, ‘she’, ’it’, ’they’ and ‘them’ while referring to entities that have
already occurred in the past questions or answers. Format each QA pair in a
single line as a JSON dictionary (key “user” for question, and “assistant” for
answer, wrapped with { and }). Do not include any other explanation. Example
3:

###  B.3 Dialogue filtering

The dialogues generated by GPT-4 as discussed in Section B.2 do not always
follow the prompts, resulting in answers that have phrases such as “does not
specify”, “cannot be determined”, “without additional context” and so on.
Hence, following Gardner et al. (2023), we filter such QA pairs to improve the
data quality and ensure desirable outputs from the model. Apart from the
manual filtering step, we also filter samples based on the similarity of the
answer generated by GPT-4 and the audio samples. Specifically, we compute the
cosine similarity between the LAION-CLAP text-embeddings and audio-embeddings
(Wu et al., 2023) for a given QA pair in each dialogue. The distributions of
similarities are shown in Figure 6. We remove samples if the similarity is
below 0.3.

Figure 6: Distributions of LAION-CLAP similarities before filtering.

###  B.4 Examples of generated dialogues

[Annotations: Sound of Mechanisms (Produces sound through mechanical means.):
[0.119s-10.000s]; Sound of Generic impact sounds (Short, sharp, and loud
bursts.): [0.217s-0.387s], [2.394s-2.727s], [2.889s-3.255s], [3.799s-4.132s],
[4.473s-4.766s], [4.928s-5.188s], [8.129s-10.000s]; Sound of Glass shatter
(High-pitched, sharp, sudden, loud crash.): [1.622s-4.652s]; Sound of Walk,
footsteps (Soft, rhythmic, echoing thuds.): [5.724s-6.106s], [6.455s-6.650s],
[7.024s-7.479s], [7.625s-7.901s]] User: Can you describe the main events in
the audio? Assistant: The audio mainly consists of mechanical sounds, sharp
impact sounds, the shattering of glass, and the sound of footsteps. User: What
can you tell me about the impact sounds? Assistant: The impact sounds are
short, sharp, and loud bursts. They occur at several points throughout the
audio, suggesting some kind of repetitive action or event. User: What about
the glass shattering, does it happen more than once? Assistant: The glass
shattering sound is a high-pitched, sharp, sudden, loud crash. It occurs once
in the audio, lasting for about three seconds.

[Annotations: This is the recording of a cowbell. The cowbell is being played
with the side and the tip of the drumstick repeatedly. This recording could be
sampled for use in beat-making especially for genres that have a heavy usage
of cowbells such as latin dance music.] User: What instrument is being played
in this audio? Assistant: The instrument being played in this audio is a
cowbell. User: How is it being played? Assistant: The cowbell is being played
with the side and the tip of the drumstick repeatedly. User: What is the tone
of the cowbell? Assistant: The tone of the cowbell is bright, resonant, and
penetrating.

##  Appendix C Additional Examples and Discussion of Audio Flamingo

We refer readers to visit our demo website: https://audioflamingo.github.io/
for comprehensive demos.

###  C.1 Additional Few-shot Experiments on Unseen Labels

Audio Flamingo can be adapted to classification with completely unseen labels
with the help of ICL-based RAG. In Table 9 we demonstrate Audio Flamingo can
be adapted to new classification tasks where none of the labels have been seen
during training. The labels are bird species and gun names, respectively. The
huge improvement from zero-shot to few-shot indicates Audio Flamingo’s strong
ability to follow retrieved in-context samples.

Table 9: Accuracy of Audio Flamingo on new classification tasks with unseen labels. Dataset | Zero-shot | Few-shot
---|---|---
BG-Gun-Sound (Park et al., 2022) | $1.6\%$ | $53.5\%$
Birds-Dataset-Till-2022 (Mohanty, ) | $0.0\%$ | $42.2\%$

The labels for BG-Gun-Sound are:
No gun, SLR, M4, VSS, AK, SKS, PP, P90, AUG, SCAR, UMP, QBZ, P92, M249, QBU,
DEAGLE, TOMY, PUMP, P1911, K2, DP, GRO, AWM, VEC, UZI, M24, KAR, VERL, MINI,
WIN, MK, R45, M16, P118C, G36C, S12K, DBS, R1895.

The labels for Birds-Dataset-Till-2022 are:
Brazilian Tinamou, Red-legged Tinamou, Hooded Tinamou, Chestnut-bellied Guan,
Dwarf Cassowary, Little Spotted Kiwi, Dwarf Tinamou, Baudo Guan, Little
Chachalaca, Pale-browed Tinamou, Lesser Rhea, Australian Brushturkey,
Berlepsch’s Tinamou, Spotted Nothura, Puna Tinamou, Brushland Tinamou, Andean
Guan, Black-billed Brushturkey, Blue-throated Piping Guan, Somali Ostrich,
Wattled Brushturkey, White-bellied Chachalaca, Barred Tinamou, Chaco
Chachalaca, Rufous-headed Chachalaca, Plain Chachalaca, Emu, Southern
Cassowary, White-bellied Nothura, Undulated Tinamou, Tawny-breasted Tinamou,
Thicket Tinamou, Bearded Guan, Quebracho Crested Tinamou, Scaled Chachalaca,
Micronesian Megapode, Nicobar Megapode, White-throated Tinamou, Malleefowl,
Crested Guan, Chestnut-winged Chachalaca, Speckled Chachalaca, Ornate Tinamou,
Curve-billed Tinamou, Tataupa Tinamou, White-crested Guan, Southern Brown
Kiwi, Sula Megapode, Biak Scrubfowl, Vanuatu Megapode, Patagonian Tinamou,
Moluccan Megapode, Variegated Tinamou, White-browed Guan, Marail Guan,
Chestnut-headed Chachalaca, Northern Cassowary, Bartlett’s Tinamou, Rusty-
margined Guan, Spix’s Guan, Band-tailed Guan, Andean Tinamou, Trinidad Piping
Guan, Elegant Crested Tinamou, North Island Brown Kiwi, Cinereous Tinamou,
White-winged Guan, Grey-headed Chachalaca, Okarito Kiwi, Rufous-bellied
Chachalaca, Red-winged Tinamou, Rufous-vented Chachalaca, Black-fronted Piping
Guan, Tanimbar Megapode, Choco Tinamou, Great Spotted Kiwi, Little Tinamou,
Slaty-breasted Tinamou, Tepui Tinamou, Greater Rhea, Chilean Tinamou, Cauca
Guan, Yellow-legged Tinamou, Black Tinamou, Lesser Nothura, Rusty Tinamou,
Grey-legged Tinamou, Dusky Megapode, Red-billed Brushturkey, Dusky-legged
Guan, Collared Brushturkey, Common Ostrich, Colombian Chachalaca, Melanesian
Megapode, Maleo, New Guinea Scrubfowl, Red-throated Piping Guan, Huayco
Tinamou, Buff-browed Chachalaca, Black-capped Tinamou, Philippine Megapode,
Great Tinamou, Orange-footed Scrubfowl, Brown Tinamou, Solitary Tinamou,
Taczanowski’s Tinamou, Grey Tinamou, Small-billed Tinamou, Tongan Megapode,
Highland Tinamou, West Mexican Chachalaca, East Brazilian Chachalaca, Red-
faced Guan, Darwin’s Nothura.

###  C.2 Audio Flamingo understands sound in depth

Audio Flamingo understands occurrences in audio. [Subjective description of
the audio: three people talking in order, with a lot of noise in the
recording.] Task: Audio Question Answering Instruction: How many different
voices are there? Ground Truth: Three. Baseline: One. Audio Flamingo: Three.

Audio Flamingo understands the order of sounds. [Subjective description of the
audio: a man speaks on a talk show, then the audience laugh and clap their
hands.] Task: Audio Captioning Instruction: Describe this sound. Ground Truth:
A man is speaking followed by applause. Baseline: A crowd of people are
speaking and applauding. Audio Flamingo: A man is speaking and then the
audience claps.

Audio Flamingo understands quality of audio. [Subjective description of the
audio: a beautiful and sad piece of instrumental music starting with keyboard
playing chords slowly followed by soft violin.] Task: Audio Question Answering
Instruction: What is the atmospheric quality of the audio? Ground Truth: The
atmospheric quality of the audio is its ability to create a sense of space and
depth, with a focus on creating a sense of mystery and wonder. Baseline: The
audio has a rich and dynamic quality, which suggests a lively and engaging
atmosphere, possibly associated with music or a musical performance, but it is
not specified in the given audio events. Audio Flamingo: The audio has a
melancholic and introspective atmosphere.

Audio Flamingo perceives loudness and its change over time. [Subjective
description of the audio: a man speaks in a small volume, then there is very
loud burst, and then the man laughs for very short time in a surprised way.]
Task: Audio Captioning Instruction: Describe the sound in a sentence. Ground
Truth: A man briefly talks followed by a loud burst then laughter. Baseline: A
person is laughing and a gun is fired. Audio Flamingo: A man is speaking and
then a loud bang occurs.

Audio Flamingo understands distances of sounds. [Subjective description of the
audio: a man speaks in the rain, then another man laughs very slightly,
followed by the sound of thunderstorm.] Task: Audio Captioning Instruction:
Describe this sound. Ground Truth: A man talking followed by another man
laughing as rain falls and thunder roars in the distance. Baseline: A man is
talking and rain is falling. Audio Flamingo: A man is talking in the rain with
thunder in the background.

###  C.3 Audio Flamingo looks at retrieved samples

Audio Flamingo borrows key words from retrieved samples. Task: Audio
Captioning Instruction: Describe this sound. Ground Truth: Paper is crinkling
and crumpling, and thumping occurs. Baseline: Movement of a person in a
kitchen with pots and pans on a counter. Retrieved samples: 1) Some material
is crumpled. 2) Plastic is being crumpled. 3) Crinkling of a paper with a
small step. Audio Flamingo: A person crumples up a sheet of paper and crinkles
it. Task: Audio Captioning Instruction: Describe this sound. Ground Truth: a
woman speaks, and faucet water runs. Baseline: A woman is talking to a child.
Retrieved samples: 1) A woman speaking and water running. 2) A women voice
followed by flowing water from a faucet. 3) Water sprays from a faucet, a
person speaks. Audio Flamingo: A woman speaks while water runs from a faucet.

Audio Flamingo can ignore noisy retrieved samples and make correct prediction.
Task: Music Genre Classification Instruction: What is the genre of this music?
Ground Truth: pop. Retrieved samples: 1) reggae. 2) jazz. 3) disco. Audio
Flamingo: pop. Task: Music Instrument Classification Instruction: What is the
instrument of this music? Ground Truth: flute. Retrieved samples: 1) clarinet.
2) clarinet. 3) clarinet. Audio Flamingo: flute.

###  C.4 Audio Flamingo has strong dialogue ability

[Audio annotations: Sound of Chirp, tweet (High-pitched, short, repetitive
bird sounds.): [0.000s-1.157s], [1.543s-2.055s], [3.220s-4.409s],
[4.748s-5.709s], [5.921s-6.780s], [7.094s-7.976s], [8.331s-8.819s],
[9.142s-10.000s]; Sound of Animal (Loud, diverse, and often rhythmic.):
[0.000s-1.157s], [1.528s-2.031s], [3.220s-4.386s], [4.764s-5.701s],
[5.929s-6.740s], [7.071s-7.976s], [8.331s-8.795s], [9.134s-10.000s]; Sound of
Bird vocalization, bird call, bird song (Melodic, rhythmic, varied pitch and
tone.): [0.000s-1.165s], [1.528s-2.039s], [3.213s-4.386s], [4.772s-5.693s],
[5.921s-6.701s], [7.047s-7.953s], [8.331s-8.780s], [9.150s-10.000s]; Sound of
Bird (High-pitched, melodic, rhythmic chirping sounds.): [0.000s-1.173s],
[1.535s-1.992s], [3.205s-4.362s], [4.780s-5.669s], [5.906s-6.669s],
[7.016s-7.929s], [8.323s-8.772s], [9.134s-10.000s]; Sound of Speech (S̈ound
waves produced by vocal cords.)̈: [2.024s-3.008s], [4.630s-5.283s],
[6.835s-8.134s], [8.701s-10.000s]; Sound of Female speech, woman speaking:
[2.047s-3.039s], [4.614s-5.315s], [6.843s-8.150s], [8.709s-10.000s]] User: Can
you briefly explain what you hear in the audio? Ground Truth: The audio
primarily consists of bird sounds, including chirping and bird vocalizations.
There are also sounds of other animals. Additionally, there is a woman
speaking at certain intervals. Baseline: Birds are chirping. Audio Flamingo:
The audio features a variety of sounds including bird vocalizations,
mechanical sounds, impact sounds, a man speaking, and a ticking sound. User:
Are the bird sounds continuous? Ground Truth: The bird sounds are not
continuous. They occur at different intervals throughout the audio, with some
overlap with the woman’s speech. Baseline: Yes, the bird sounds are
continuous. Audio Flamingo: No, the bird sounds are not continuous. They occur
at different intervals throughout the audio. User: Do the bird sounds and the
person’s speech overlap? Ground Truth: Yes, there are instances where the bird
sounds and the woman’s speech overlap. Baseline: Yes, the bird sounds and the
woman’s speech overlap. Audio Flamingo: Yes, the bird sounds and the person’s
speech do overlap at certain points in the audio.

[Audio annotations: Sound of Music (Sound produced by vibrating instruments.):
[0.000s-10.000s]; Sound of Rapping (Rhythmic speech over beats.):
[0.000s-10.000s]; Sound of Male singing (Deep, resonant, and powerful tones.):
[0.000s-10.000s]] User: Is the male voice rapping or singing? Ground Truth:
The male voice is doing both, rapping and singing in the audio. Baseline: The
male voice is rapping. Audio Flamingo: The male voice is rapping. User: What
kind of music is it? Ground Truth: The type of music isn’t specified, but it’s
accompanied by rapping and male singing. Baseline: The kind of music is hip-
hop. Audio Flamingo: The music is a rap song. User: Is the male voice the only
one present? Ground Truth: Yes, the male voice is the only one specified in
the audio, both rapping and singing. Baseline: Yes, the male voice is the only
one present. Audio Flamingo: Yes, the male voice is the only one present in
the audio.

[Audio annotations: Sound of Music (Sound produced by vibrating instruments.):
[0.000s-10.000s]; Sound of Male singing (Deep, resonant, and powerful tones.):
[1.341s-3.734s], [9.289s-10.000s]] User: What genre does this music belong to?
Ground Truth: This music belongs to the R&B/Disco genre. Baseline: It is not
specified in the given audio clip. Audio Flamingo: This music belongs to the
R&B genre. User: Can you tell me more about the vocals in this track? Ground
Truth: Sure, the vocals in this track are from a male singer who is singing in
a higher register. His voice has been modified with effects, making it sound
like a voice sample. Baseline: The vocals in this track are described as rich
and full-bodied, but no further details are given about the specific singer or
the lyrics being sung in the audio clip. Audio Flamingo: The vocals in this
track are by a male singer. His voice is deep, resonant, and powerful, adding
to the overall emotional impact of the song.