# Decision trees compensate for model misspecification

Hugh Panton∗ Gavin Leech∗ Laurence Aitchison

###### Abstract

The best-performing models in ML are not interpretable. If we can explain why
they outperform, we may be able to replicate these mechanisms and obtain both
interpretability and performance.

One example are decision trees and their descendent gradient boosting machines
(GBMs). These perform well in the presence of complex interactions, with tree
depth governing the order of interactions. However, interactions cannot fully
account for the depth of trees found in practice.

We confirm 5 alternative hypotheses about the role of tree depth in
performance in the _absence_ of true interactions, and present results from
experiments on a battery of datasets. Part of the success of tree models is
due to their robustness to various forms of mis-specification.

We present two methods for robust generalized linear models (GLMs) addressing
the composite and mixed response scenarios.

Machine Learning, interpretability, tree models, linear models

##  1 Introduction

Some models are inherently interpretable, for instance sparse linear models
e.g.

$\displaystyle\begin{split}\mathrm{price}&=\$3000-\$1000\times\mathrm{diesel}+\$50\times\mathrm{red}\\\
&\quad+\$100\times\mathrm{economy}-\$450\times\mathrm{years}\end{split}$ (1)

Why is this interpretable? There are a small number of parameters; the
parameters have a natural interpretation on a single scale; the model is
decomposable (i.e. each parameter can be interpreted independently) (Lipton,
2018); the response is a straightforward function of the parameters.

However, most models are less straightforward. A typical tree-based model, as
produced by the popular library XGBoost (Chen and Guestrin, 2016) with default
tree-depth 3 and default tree number 100, could be depicted in full:

Figure 1:  Two of the 100 trees in a typical GBM, depth $d=3$.

but only by also plotting the 98 other trees. This is not interpretable, in
several senses. The splits are not decomposable. It is not simulatable: no one
could understand the global behaviour even given this complete specification.
(See Muggleton et al. (2018) for an operationalisation of simulatability, as
‘comprehensibility’.) We are thus forced to use post-hoc model explanation to
obtain a (limited) sense of how they work.

The main obstacle to interpretability is interaction. If the component trees
had depth $d\\!=\\!1$, the ‘decision stumps’ of Iba and Langley (1992), then
partial dependence offers a complete feature-wise summary of the model’s
behaviour (Friedman, 2001). For $p$ features used in the splits, an array of
$p$ partial dependence plots are unlikely to be as simulatable as our linear
equation (1), but it remains possible to see exactly how each feature affects
the response.

However, in a complex system, the effects of variables are modified by other
variables. Increasingly, datasets are large enough and signals strong enough
that we actually can meaningfully model interactions (Efron and Hastie, 2016).
In these cases, models without the capacity to model interactions will
underperform those which can.

We give five scenarios in which problems besides interaction cause high-depth
models to outperform, and show how they could be addressed without using
$d>1$. These support the following hypotheses by giving existence proofs:

  1. 1.

These problems may be present where tree-based models are applied, and are at
least somewhat responsible for the performance gap between high- and low-tree-
depth models.

  2. 2.

For a given dataset, the gap between the performance of conventional tree-
based models and the best interpretable model may be smaller than a standard
grid-search suggests.

  3. 3.

For a given dataset, the maximum depth required for optimum performance by a
tree model may be lower than a standard grid-search suggests.

  4. 4.

Models which appear to benefit from $d>1$ might be matched by interpretable
models, once these problems are addressed.

The code for the generators, the illustrative GBMs, and our multiresp solution
is here.

##  2 Background

We use GBMs as an exemplar of less interpretable models, and GLMs as an
exemplar of interpretable models.

###  2.1 Gradient Boosting Machines

##  3 Related Work

Machine learning practitioners have found that high-depth tree ensembles
perform well; 10-depth ensembles are common, and often win Kaggle contests,
thus passing a relatively strict test of generalisation (Zhang, 2015). (Note
also the unproblematic 11-depth and 21-depth cases in Friedman (2001)’s
original GBM experiments.) Most accounts of tree success invoke interaction
modelling, or equivalently data segmentation (Efron and Hastie, 2016). This is
incongruous when we consider that this claims a ninth-order interaction, and
of sufficient power to have estimated this interaction well. There is a
classical relationship between interaction effect size, power, and sample
size: for a two-way interaction $x_{1}x_{2}$ with effect size half that of the
main effect, it takes 16 times the sample size to estimate the effect of
$x_{1}x_{2}$, maintaining precision (Gelman et al. (2020), chp 16.4). The
standard error compounds as the order of the interaction increases, and as the
size of the interaction effect decreases. Some very large datasets and some
unusually large interaction effects may pass this bar, but in most cases we
should not expect high-order interaction estimates to be justified. So the
ubiquity of high-depth tree models, even when datasets and interaction effects
are small, presents us with a puzzle.

Little past work has touched on our main topic, the robustness to mis-
specification of trees. Linear models can outperform trees in the absence of
mis-specification (Venkatasubramaniam et al., 2017). Some work has noted the
ability of tree learning to handle mis-specification (Tan and Roy, 2019);
Kreif et al. (2016), a medical case study showed that the CART algorithm
reduced bias relative to maximum likelihood in the presence of mis-
specification.

##  4 Methods & Results

What non-interaction reasons might a tree-based model have for high tree-
depth? We give 5 synthetic scenarios and show how tree depth can improve
performance. Code for the data generators and GBMs can be found here. Four of
the scenarios work with Poisson responses, since three of our hypotheses are
shown through multiplicative processes, and Poisson is the simplest model with
a canonical log-link. Since our data have known distributions, the results are
given without error bars.

###  4.1 High tree-depth can compensate for insufficient learning

Consider a data generating process (code):

$\displaystyle m$ $\displaystyle=10$ $\displaystyle n$ $\displaystyle=1000$
$\displaystyle\theta$ $\displaystyle=0.5$ $\displaystyle x_{i}$
$\displaystyle\sim\mathrm{Bern}(\theta)\quad\mathrm{for}\,\,i=1,...,m$
$\displaystyle y_{j}$
$\displaystyle\sim\sum^{m}_{i=0}x_{i}+\mathcal{N}(0,1)\quad\mathrm{for}\,\,j=1,...,n$

e.g. flipping ten fair coins a thousand times apiece. The explanatory
variables are the results of the ten coin flips; the response variable is the
number of heads in each set of ten flips, plus Gaussian noise.

We model the resulting data using XGBoost with varying $d$; the learning rate
and number of trees are specified, but otherwise we use default parameters.
Despite there being no interactions between variables, and despite the form of
this process being ideal for additive (unity-linked) Gaussians (e.g. decision
stumps, $d\\!=\\!1$), higher-depth models obtain consistently better test
scores (Table 1).

max_depth $d$ | MAE | RMSE
---|---|---
$d=1$ | $1.967$ | $2.382$
$d=2$ | $1.916$ | $2.304$
$d=3$ | $\mathbf{1.886}$ | $\mathbf{2.248}$
Table 1: Example results from Scenario 4.1, with 100 trees

When other hyperparameters result in undertraining, increasing the tree-depth
will make the model less underfit, even if there are no interactions to model.

Alternatively, if we are not underfitting, then adding spurious complexity
(such as increased depth) to our model will only make things worse. This is
elementary; the practical point is that subtle undertraining along any
hyperparameter could inflate the depth of the best model.

For example, trying to find the best tree-depth varying only tree-depth
entangles the search for optimal depth with the search for optimal complexity.
To actually find the best tree-depth requires a grid-search over depth and
other hyperparameters, and moreover a search over a range which actually
contains the optimum for each depth.

As a proxy for the degree of underfitting, we use learning rate (LR), where
low LR models are assumed to be underfit.

| $d=1$ | $d=2$ | $d=3$
---|---|---|---
$\mathrm{LR}=0.05$ | 1.0948 | 0.9323 | $\mathbf{0.8541}$
$\mathrm{LR}=0.1$ | 0.9053 | 0.8289 | $\mathbf{0.8164}$
$\mathrm{LR}=0.2$ | $\mathbf{0.7994}$ | 0.8124 | 0.8166
$\mathrm{LR}=0.3$ | $\mathbf{0.789}$ | 0.8099 | 0.8255
$\mathrm{LR}=0.4$ | $\mathbf{0.7883}$ | 0.813 | 0.825
$\mathrm{LR}=0.5$ | $\mathbf{0.7884}$ | 0.811 | 0.836
Best | $\mathbf{0.7883}$ | $0.8099$ | $0.8164$
Table 2: MAE for depth and learning rate, under Scenario 4.1

There are two relevant asymmetries. The first, shown in Table 2, is that
undertrained models tend to be worse than overtrained ones. The second is that
high-depth models are capable of modelling lower-depth effects, but low-depth
models cannot model higher-depth effects, so excessive depth will tend to be
less harmful than insufficient depth.

###  4.2 High tree-depth can compensate for incorrect linkages

Consider a multiplicative response with only two true features: the colour and
size of gemstones predicts their price. The true relationship is in Table 3.

| Size $=$ Small | Size $=$ Large
---|---|---
Color $=$ Red | $\$100$ | $\$300$
Color $=$ Blue | $\$200$ | $\$600$
Table 3: Hypothetical gemstone pricing

An additive (unity-linked) model would need tree-depth $>1$ to fully model
price. Start with $100, add $100 for blueness, add $200 for largeness, and add
a further $200 for the interaction of blueness and largeness. But our effects
are not additive - a multiplicative (log-linked) model would only need tree-
depth of 1: blueness multiplies price by 2, largeness multiplies price by 3,
and that exhausts the relation.

Real-world datasets outside the physical sciences are generally complex enough
that effects from any given feature will have both additive and multiplicative
structure (and more structure besides). This implies that models with a
conventional approach to linkage will never be perfect. But we can minimise
the impact of this – and hence the tree-depth required – by using the least
incorrect linkage.

###  4.3 Tree-depth can compensate for multiple responses

The following illustrates when the response is

Consider a composite multiplicative process (code):

$\displaystyle m$ $\displaystyle=4$ $\displaystyle\theta$ $\displaystyle=0.5$
$\displaystyle x_{i}$
$\displaystyle\sim\mathrm{Bern}(\theta)\quad\mathrm{for}\,\,i=1,...,m$
$\displaystyle d_{A}$
$\displaystyle=I[x_{1}=1]+I[x_{2}=1]+I[x_{3}=0]+I[x_{4}=0]$ $\displaystyle
h_{A}$ $\displaystyle=I[x_{1}=0]+I[x_{2}=0]+I[x_{3}=1]+I[x_{4}=1]$
$\displaystyle\lambda_{A}$ $\displaystyle=2^{d_{A}}0.5^{h_{A}}$ $\displaystyle
d_{B}$ $\displaystyle=I[x_{1}=1]+I[x_{2}=0]+I[x_{3}=1]+I[x_{4}=0]$
$\displaystyle h_{B}$
$\displaystyle=I[x_{1}=0]+I[x_{2}=1]+I[x_{3}=0]+I[x_{4}=1]$
$\displaystyle\lambda_{B}$ $\displaystyle=2^{d_{B}}0.5^{h_{B}}$ $\displaystyle
d_{C}$ $\displaystyle=I[x_{1}=1]+I[x_{2}=0]+I[x_{3}=0]+I[x_{4}=1]$
$\displaystyle h_{C}$
$\displaystyle=I[x_{1}=0]+I[x_{2}=1]+I[x_{3}=1]+I[x_{4}=0]$
$\displaystyle\lambda_{C}$ $\displaystyle=2^{d_{C}}0.5^{h_{C}}$ $\displaystyle
y$
$\displaystyle\sim\mathrm{Poisson}(\lambda_{A})+\mathrm{Poisson}(\lambda_{B})+\mathrm{Poisson}(\lambda_{C})$
$\displaystyle X$ $\displaystyle=(x_{1},x_{2},x_{3},x_{4})$

i.e. we flip four coins per row. Start with a value of 1; double it for every
heads in the first two coins or tails in the second two, and halve it for
every tails in the first two coins or heads in the second two. Use the result
as the $\lambda$ for a Poisson generator, and take the output as response A.
Repeat the process with the same coin results with different permutations of
the doubling and halving indicators to get response B and C. Your total
response, the observed $y$ is the sum of these component responses.

This is contrived, but the point holds for any situation where your response
variable can be decomposed into multiple processes which deserve their own
models, but all you have access to is the sum of outputs from those processes.

For example, say you are predicting staff injuries caused by lions at a zoo;
bites and scratches and headbutts should be modelled separately, but you’re
only given the total injury count per lion per year. Or an insurance model
predicting claims from customer data, where some claims are fraudulent and
some are not, but you only know how many claims each customer made in total.

If the effects of the explanatory features were additive instead of
multiplicative, they would combine simply, and no benefit to increased tree-
depth would be observed. But in our log-linked setup, despite no interactions
in the response generator, $d\\!=\\!3$ models consistently beat $d\\!=\\!1$
models.

| MAE | RMSE
---|---|---
$d=1,100$ trees | $2.9414$ | $4.0419$
$d=1,200$ trees | $2.9413$ | $4.0596$
$d=3,100$ trees | $\mathbf{1.9583}$ | $\mathbf{2.6376}$
Table 4: Example results

(Unlike Scenario 4.1, a gridsearch is not required to demonstrate our
hypothesis: doubling the tree count and getting no significantly better result
is sufficient to show that the improvement from increased tree-depth is not
just compensation for underfitting.)

In some sense the above process does have interactions - the component
features affect each other’s effects on the total output in ways that cannot
be captured by an appropriately-linked linear system. In the relevant sense it
does not; there are no interactions between the explanatory features in each
subresponse. If we ignore the randomness, we have a linear system:

$\displaystyle y$
$\displaystyle=\exp\left(k_{0a}+k_{1a}x_{1}+k_{2a}x_{2}+k_{3a}x_{3}+k_{4a}x_{4}\right)$
$\displaystyle+\exp\left(k_{0b}+k_{1b}x_{1}+k_{2b}x_{2}+k_{3b}x_{3}+k_{4b}x_{4}\right)$
$\displaystyle+\exp\left(k_{0c}+k_{1c}x_{1}+k_{2c}x_{2}+k_{3c}x_{3}+k_{4c}x_{4}\right)$

where $y$ is the composite response, $x_{i}$ are coin flips, and $k_{i}$ are
the coefficients which define the system. Let $S$ be the number of submodels
used; here $S=3$. This implies a much simpler algorithm for the above than
high-depth trees: we set up a model of that form, apply an appropriate error
distribution, and use gradient descent to learn $k_{i}$. Call this multiresp,
multiple response modelling.

This toy case yields a model defined by fifteen $k$-values, so that each
feature’s effect on $\hat{y}$ can still be seen at a glance. In the presence
of categorical and quantitative features (as well as binary ones), we could
still use partial dependence plots to fully explain each feature’s effect.

This is an analogue of mixture modelling, but for composite responses rather
than subpopulations in the features. A person fitting two sub-responses to
their lion injury data need not know that some are bites and some are
scratches; they can just set $S\\!=\\!2$ and let the algorithm do the rest.
Nor do they need to know $S\\!=\\!2$ apriori: it’s just another hyperparameter
to optimise.

If we compare the performance of our multiresp approach on our toy composite
data, we see that the reasoning does hold: high tree-depth and submodels do
equally well here.

| MAE | RMSE
---|---|---
$d=1,100$ trees | 2.9390 | 4.0973
$d=1,200$ trees | 2.9370 | 4.1085
$d=3,100$ trees | $\mathbf{2.0208}$ | $\mathbf{2.7222}$
Canonical model, $\mathrm{S}=1$ | 2.9370 | 4.1087
Canonical model, $\mathrm{S}=2$ | 2.4667 | 3.4335
Canonical model, $\mathrm{S}=3$ | $\mathbf{2.0207}$ | $\mathbf{2.7223}$
Table 5: Example results from tree-based & canonical models

multiresp scales well in features $m$; see Appendix C for further results.

###  4.4 Tree-depth can compensate for multiple populations

Consider a mixture of multiplicative processes:

$\displaystyle m$ $\displaystyle=12$ $\displaystyle x_{i}$
$\displaystyle\sim\mathrm{Bern}(0.5)\quad\mathrm{for}\,\,i=1,...,m$
$\displaystyle k$ $\displaystyle\sim\mathrm{Bern}(0.25)$ $\displaystyle d$
$\displaystyle=x_{1}+x_{2}+x_{3}$ $\displaystyle d_{A}$
$\displaystyle=x_{4}+x_{5}+x_{6}$ $\displaystyle d_{B}$
$\displaystyle=x_{7}+x_{8}+x_{9}$ $\displaystyle h$
$\displaystyle=x_{10}+x_{11}+x_{12}$ $\displaystyle f_{A}$
$\displaystyle=2^{2k-1}$ $\displaystyle f_{B}$ $\displaystyle=2-1.5k$
$\displaystyle\lambda$
$\displaystyle=2^{d}\,0.5^{h}\,f_{A}^{d_{A}}\,f_{B}^{d_{B}}$ $\displaystyle y$
$\displaystyle\sim\mathrm{Poisson}(\lambda)$

That is, for each row, roll a four-sided die ($k$) and flip twelve coins
$\\{x_{i}\\}$. Rolling a 1 makes it an A-row, and otherwise it’s a B-row:
don’t keep that information in the explanatory variables, but do keep all the
flips. The value for $\lambda$ starts at 1; each heads on flips 1-3 doubles
$\lambda$ for the row; each heads on flips 4-6 doubles $\lambda$ for an A-row
but halves it for a B-row; each heads on flips 7-9 halves it for an A-row but
doubles it for a B-row; and each heads on flips 10-12 halves it for any row.
Once the halvings and doublings are all applied, we generate our response $y$
with a Poisson generator.

Despite no interactions between the features provided, the dataset generated
will consistently yield better scores with $d=3$ than $d=1$.

| MAE | RMSE
---|---|---
$d=1,1000$ trees | $1.1661$ | $1.7386$
$d=1,2000$ trees | $1.1661$ | $1.7386$
$d=3,1000$ trees | $\mathbf{1.1425}$ | $\mathbf{1.6981}$
Table 6: Example results for Scenario 4.4

Scenario 4.3 addressed how multiple effects could be present in the same row;
the above addresses how multiple effects could be present across different
rows. This applies to any situation where the datapoints being modelled can’t
be approximated as homogenous.

Returning to our examples: let’s say we predict injuries caused by exotic cats
at a zoo, and some are lions and some tigers, and the species differences are
profound enough to merit different models, but none of your explanatory
variables indicate the species. Or, less fancifully, you’re building an
insurance model predicting claims from customer data, and some of your
customers are honest while others are frauds, but you have no label for this
class.

For this scenario, the system on its own terms looks like this (with two
parameter sets, one each for mixture component $a$ and $b$):

$y=\exp\left(k_{0a}+k_{1a}x_{1}+k_{2a}x_{2}+\cdots+k_{12a}x_{12}\right)$

with probability $f$, and

$y=\exp\left(k_{0b}+k_{1b}x_{1}+k_{2b}x_{2}+\cdots+k_{12b}x_{12}\right)$

with probability $(1-f)$

So we can use gradient descent to find the best $k$-values, and end up with
only twice the complexity of a normal linear model. Call this a mixed GLM.

| MAE | RMSE
---|---|---
Canonical model, $S=2,\mathrm{f}=0.05$ | 1.3554 | 2.5590
Canonical model, $S=2,\mathrm{f}=0.15$ | 1.3789 | 2.4893
Canonical model, $S=2,\mathrm{f}=0.25$ | $\mathbf{1.4160}$ | $\mathbf{2.4615}$
Canonical model, $S=2,\mathrm{f}=0.35$ | 1.5154 | 2.6799
Canonical model, $S=2,\mathrm{f}=0.45$ | 1.5386 | 2.7053
$d=1,1000$ trees | 1.4628 | 2.6322
$d=3,1000$ trees | $\mathbf{1.4181}$ | $\mathbf{2.4657}$
Table 7: Example results from tree-based and canonical models for the Scenario
4.4 generator. True $f=0.25$.

We find that $f$ is best treated as a hyperparameter rather than a parameter.
To fit ten subpopulations, $S=10$, we would need nine degrees of freedom,
i.e. nine $f$s to search. However, increasing the submodels as high as 10
would be counterproductive from the perspective of interpretability; making
users look at ten numbers per feature (or, for quantitative and categorical
analogues, ten graphs per feature) would only be a slight improvement over
making them scan hundreds of trees. If you find yourself modelling ten truly
distinct subpopulations, then there may be no inherently interpretable model.

Note that mixed GLMs cannot model the specific case of $f=0.5$ and
$\mu_{a}=\mu_{b}$.

###  4.5 Tree-depth can compensate for missing features

This scenario addresses the behaviour of an idealised log-linked dataset where
some of the explanatory variables are absent, but are correlated with those
present.

Consider a process:

$\displaystyle m$ $\displaystyle=6$ $\displaystyle h_{i}$
$\displaystyle\sim\mathrm{Bern}(1/2)\quad\mathrm{for}\,\,i=1,...,m$
$\displaystyle d_{i}$
$\displaystyle\sim\mathrm{Bern}(1/10)\quad\mathrm{for}\,\,i=1,...,m$
$\displaystyle a_{i}$ $\displaystyle=I[d_{i}\neq
1]h_{i}+I[d_{i}=1](1-h_{i})\quad\mathrm{for}\,\,i=1,...,m$
$\displaystyle\lambda$ $\displaystyle=2^{\sum^{m}a_{i}}$ $\displaystyle y$
$\displaystyle\sim\mathrm{Poisson}(\lambda)$ $\displaystyle\mathbf{x}$
$\displaystyle=\\{a_{i}\\}\quad\mathrm{for}\,\,i=1,2,3$

i.e. For each row, toss six coins and roll six ten-sided dice. Record the
opposite of the coin flip on rolling a 1, and record the true result on any
other result. For your dependent variable $y$, raise 2 to the power of the
number of heads you recorded, then feed the result into a Poisson random
number generator as in the last two scenarios. For your explanatory variables,
take only the first three recorded coin flips. When using these explanatory
variables to predict this dependent variable with an appropriately-linked and
distributed XGB model, $d=3$ outperforms $d=1$.

| MAE | RMSE
---|---|---
$d=1,100$ trees | $7.4127$ | $11.4690$
$d=1,200$ trees | $7.4125$ | $11.4692$
$d=3,100$ trees | $\mathbf{7.0407}$ | $\mathbf{11.0973}$
Table 8: Example results under missing features.

A natural idea for bringing this effect into interpretable models is to use
observed features to predict the posterior distribution of latent feature
values, and then use all features together to predict the response. This is
impractical; the problem of detecting missing features in general is clearly
ill-defined. But even even if it was implementable, the resulting models would
be difficult to interpret: it would require graphs to deduce missing features
from present features, and others using absent and present features to predict
the response. This would obscure the effect of each individual feature on the
final prediction, and so for our purposes it is not desirable.

The practical upshot is instead to emphasise investigating extra features when
modelling a log-linked system. Besides that, the effect is not obvious: that
using more features – even features highly correlated with those present – can
make models less complex is counter-intuitive.

###  4.6 Validating on real data

To illustrate the competitiveness of multiresp and mixed GLMs, we evaluate on
.

##  5 Conclusions

The above illustrate scenarios in which $d>1$ outperforms decision stumps,
even when there are no real interactions in the data generator. We find that
the learner increases depth to compensate for violated assumptions:

  * •

“High tree-depth can compensate for insufficient learning”: if you assume the
model needs much less training than it does, higher tree-depth will help.

  * •

“High tree-depth can compensate for incorrect linkages”: if you assume the
system has a linkage it doesn’t, higher tree-depth will help.

  * •

“High tree-depth can compensate for multiple responses”: if you assume single
responses in a log-linked system where response is composite, higher tree-
depth will help.

  * •

“High tree-depth can compensate for multiple populations”: if you assume
homogeneity in a heterogeneous population, higher tree-depth will help.

  * •

“High tree-depth can compensate for absent explanatory columns”: if you assume
you have all data needed to explain a log-linked system when you don’t, higher
tree-depth will help.

We note that all but the first of these are unsurprising, consequences of the
simplification inherent to modelling (outside mechanistically grounded
domains):

  * •

“High tree-depth can compensate for incorrect linkages”: All linkages are
approximations.

  * •

“High tree-depth can compensate for multiple responses”: All responses are
arbitrarily decomposable.

  * •

“High tree-depth can compensate for multiple populations”: All models elide
relevant heterogeneity.

  * •

“High tree-depth can compensate for absent explanatory columns”: All models
have absent explanatory columns.

This implies that, where these mis-specifications occur, there is scope to fix
them and thereby obtain less-complex interpretable models (e.g. a stump
ensemble or a GLM) that performs comparably to uninterpretable deep trees.
Note though that merely reducing tree depth may not improve matters much:
there’s a discontinuity at $d=1$ to $d=2$, at which point partial dependence
plots stop being a complete summary of model behaviour.

It is a good thing that a learning algorithm succeeds even when the model is
not correct, since misspecification is the norm in social science or
industrial contexts. However, when interpretability is a priority (as it
arguably should be for models that serve a mass market or policy decisions),
our results imply that there is an alternative to automatic compensation:
attempt to fix the mis-specification and return to interpretable models.

## References

  * Chen and Guestrin [2016] Tianqi Chen and Carlos Guestrin.  XGBoost.  _Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining_ , Aug 2016.  doi: 10.1145/2939672.2939785.  URL http://dx.doi.org/10.1145/2939672.2939785.
  * Efron and Hastie [2016] Bradley Efron and Trevor Hastie.  _Computer Age Statistical Inference_ , volume 5.  Cambridge University Press, 2016.
  * Friedman [2001] Jerome H Friedman.  Greedy function approximation: a gradient boosting machine.  _Annals of statistics_ , pages 1189–1232, 2001.
  * Gelman et al. [2020] Andrew Gelman, Jennifer Hill, and Aki Vehtari.  _Regression and Other Stories_.  Cambridge University Press, 2020.
  * Iba and Langley [1992] Wayne Iba and Pat Langley.  Induction of one-level decision trees.  In Derek Sleeman and Peter Edwards, editors, _Machine Learning Proceedings 1992_ , pages 233 – 240. Morgan Kaufmann, San Francisco (CA), 1992\.  ISBN 978-1-55860-247-2.  doi: https://doi.org/10.1016/B978-1-55860-247-2.50035-8.  URL http://www.sciencedirect.com/science/article/pii/B9781558602472500358.
  * Kreif et al. [2016] Noémi Kreif, Susan Gruber, Rosalba Radice, Richard Grieve, and Jasjeet S Sekhon.  Evaluating treatment effectiveness under model misspecification: a comparison of targeted maximum likelihood estimation with bias-corrected matching.  _Statistical methods in medical research_ , 25(5):2315–2336, 2016.
  * Lipton [2018] Zachary C Lipton.  The mythos of model interpretability.  _Queue_ , 16(3):31–57, 2018.
  * Muggleton et al. [2018] Stephen Muggleton, Ute Schmid, Christina Zeller, Alireza Tamaddoni-Nezhad, and Tarek Besold.  Ultra-strong machine learning: comprehensibility of programs learned with ILP.  _Machine Learning_ , pages 1119–1140, 2018.  https://link.springer.com/article/10.1007/s10994-018-5707-3.
  * Tan and Roy [2019] Yaoyuan Vincent Tan and Jason Roy.  Bayesian additive regression trees and the general BART model.  _Statistics in medicine_ , 38(25):5048–5069, 2019\.
  * Venkatasubramaniam et al. [2017] Ashwini Venkatasubramaniam, Julian Wolfson, Nathan Mitchell, Timothy Barnes, Meghan JaKa, and Simone French.  Decision trees in epidemiological research.  _Emerging themes in epidemiology_ , 14(1):11, 2017\.
  * Zhang [2015] Owen Zhang.  Learn Kaggle techniques from Kaggle.  https://www.youtube.com/watch?v=LgLcfZjNF44, 2015.

Appendices

##  Appendix A Trees in the limit

When all explanatory variables are binary, a saturated XGB model with
max_depth = $d$ approaches an unpenalized generalized linear model with every
d-way interaction.

i.e. if $d=2$ and there are three explanatory features, the model
$y=\mathrm{XGB}(x_{1},x_{2},x_{3})$ will give the same results as

$y=g^{-1}\left(c+k_{1}x_{1}+k_{2}x_{2}+k_{3}x_{3}+(k_{1,2}x_{1}x_{2})+(k_{1,3}x_{1}x_{3})+(k_{2,3}x_{2}x_{3})\right)$

with inverse link function $g^{-1}$.

This assertion was tested by running an XGB model with depth $d$ on a 2x2x2x2
hypercube grid containing the first 16 prime numbers, and comparing the
predictions of that grid to those made by a linear model with every $d$-way
interaction. For all $d<4$, the outputs were identical.

Every XGB model constructed in Section 3.3, Section 3.4, Section 3.5 and
Appendix C had an unpenalized generalized linear model of the same depth
created on the same data. In every case, the performance of the linear model
was very close to that of the XGB model. Crucially, the key patterns were
retained: the performance of the d=3 linear model exceeded that of the d=1
linear model, and the performance of the canonical model matched (for Section
3.3) or outperformed (for Section 3.4 and Appendix C) the best linear model.

This argues against the possibility that outperformance of XGB models by
canonical models was due to specifying the wrong hyperparameters for training
XGB: there is only one optimal unpenalized generalized linear model with every
d-way interaction for a given dataset and value of d.

###  A.1 Theoretical bounds on the utility of multiresp models and mixture
GLMs

Consider the dataset used in Section 3.3: one response variable, four
explanatory binary variables, generation process optimal for using multiresp
with $S=3$.

A $d=4$ linear model would need $2^{4}=16$ k-values to fully describe how the
responses predict y. However, the model created using multiresp only has
$3(4+1)=15$ k-values. Having one fewer degree of freedom makes it marginally
easier to interpret, and marginally less likely to overfit. (The d=3 linear
model actually used - optimal for this scenario because one column affects all
responses the same way - can be described with only $2^{4}-4C4=15$ k-values,
so there’s no improvement.)

Scaling quickly makes the benefit of canonical models more obvious. A d=8
linear model acting on a dataset with eight binary explanatory features (see
Appendix C) would need $2^{8}=256$ k-values to fully describe how the
responses predict y; multiresp with S=3 would only need $3(8+1)=27$ k-values.

Formalizing the above: an ideally multiresp-friendly dataset with $B$ binary
explanatory features (or, equivalently, explanatory variables conveying B bits
of information), in which no columns affect all sub-responses the same way,
can be usefully modelled with a multiresp model with S submodels when and only
when the resulting model would have fewer k-values than a $d=B$ linear model.

k-values in maximum max_depth linear model $=2^{B}$

k-values in multiresp model = $S(B+1)$

So when you have B binary features, it’s possible for multiresp to lower the
number of k-values when

$\displaystyle S(B+1)<2^{B}$ (2)

For various values of B, the maximum value of S for a multiresp model is
presented in Table 9.

B | Max useful S
---|---
1 | 0
2 | 1
3 | 1
4 | 3
5 | 5
6 | 9
7 | 15
8 | 28
9 | 51
10 | 93
Table 9: Relationship of number of covariates and maximum $S$ for multiresp
models

For the mixture GLMs from section 3.4, the calculation is slightly different,
because the $f$s are also a degree of freedom.

Let $k_{F}$ be the combined number of k-values and f-values in a mixture GLM:

$\displaystyle k_{F}=S(B+1)+(S-1)=S(B+2)-1$ (3)

Mixture GLMs can therefore be useful when

$\displaystyle S(B+2)-1<2^{B}$ (4)

For various values of $B$, the maximum value of S for a mixture GLM is Table
10.

B | Max useful S
---|---
1 | 0
2 | 1
3 | 1
4 | 2
5 | 4
6 | 8
7 | 14
8 | 25
9 | 46
10 | 85
Table 10: Relationship of number of covariates and maximum $N$ for mixture
GLMs

In practice, we would be unlikely to want to use an $S>3$ multiresp model or
mixture GLMs, and the typical dataset used in an industry modelling task has
hundreds of explanatory columns conveying millions of bits of information.

In other words: outside a small number of pathological cases unlikely to
appear in the real world, multiresp models and mixture GLMs are capable of
reducing the degrees of freedom used to model compared to a linear model, and
hence to an XGB model.

##  Appendix B Deriving the GLM objectives

###  B.1 Simple (one-population, basic response) case

PMF for Poisson is

$P=\frac{\lambda^{k}e^{-\lambda}}{k!}$ (5)

Per-row contribution to deviance is log of PMF, so:

$D=\ln(\lambda^{k}e^{-\lambda})-\ln(k!)=k\ln(\lambda)-\lambda-\ln(k!)$ (6)

Differentiate with respect to $\lambda$

$\frac{\mathrm{d}D}{\mathrm{d}\lambda}=\frac{k}{\lambda}-1=(k-\lambda)/\lambda$
(7)

Define $\lambda$ by its components in a log-linked model

$\lambda=\exp\left(c+m_{1}x_{1}+...+m_{N}x_{N}\right)$ (8)

Differentiate with respect to components

$\frac{\mathrm{d}\lambda}{\mathrm{d}c}=\exp\left(c+m_{1}x_{1}+...+m_{N}x_{N}\right)=\lambda$
(9) $\displaystyle\frac{\mathrm{d}\lambda}{\mathrm{d}m_{n}}=$
$\displaystyle=x_{n}\exp\left(c+m_{1}x_{1}+...+m_{N}x_{N}\right)$ (10)
$\displaystyle=\lambda x_{n}$ (11)

So

$\frac{\mathrm{d}D}{\mathrm{d}c}=\frac{\mathrm{d}D}{\mathrm{d}\lambda}\frac{\mathrm{d}\lambda}{\mathrm{d}c}=\frac{k-\lambda}{\lambda}\times\lambda=(k-\lambda)$
(12)
$\frac{\mathrm{d}D}{\mathrm{d}m_{n}}=\frac{\mathrm{d}D}{\mathrm{d}\lambda}\frac{\mathrm{d}\lambda}{\mathrm{d}m_{n}}=\frac{k-\lambda}{\lambda}\times\lambda
x_{n}=(k-\lambda)x_{n}$ (13)

So feed this into gradient descent.

###  B.2 Composite response case

The sum of two Poisson distributions is another Poisson, so the following
largely mirrors the above.

Assume two submodels (extending to $n$ is trivial).

$\lambda=\lambda_{A}+\lambda_{B}$ (14)

Differentiate and apply chain rule:

$\frac{\mathrm{d}\lambda}{\mathrm{d}\lambda_{A}}=\frac{\mathrm{d}\lambda}{\mathrm{d}\lambda_{B}}=1$
(15)

Define $\lambda_{A}$ (and by symmetry $\lambda_{B}$) by its components in a
log-linked model

$\lambda_{A}=\exp\left(c_{A}+m_{1A}x_{1}+...+m_{NA}x_{N}\right)$ (16)

Differentiate wrt components

$\frac{\mathrm{d}\lambda_{A}}{\mathrm{d}c_{A}}=\exp\left(c+m_{1A}x_{1}+...+m_{NA}x_{N}\right)=\lambda_{A}$
(17)
$\frac{\mathrm{d}\lambda_{A}}{\mathrm{d}m_{nA}}=x_{n}\exp(c+m_{1}x_{1}+...+m_{NA}x_{N})=\lambda_{A}x_{n}$
(18)

So

$\frac{\mathrm{d}D}{\mathrm{d}c_{A}}=\frac{\mathrm{d}D}{\mathrm{d}\lambda}\frac{\mathrm{d}\lambda}{\mathrm{d}\lambda_{A}}\frac{\mathrm{d}\lambda_{A}}{\mathrm{d}c_{A}}=\frac{k-\lambda}{\lambda}\lambda_{A}$
(19)
$\frac{\mathrm{d}D}{\mathrm{d}m_{nA}}=\frac{\mathrm{d}D}{\mathrm{d}\lambda}\frac{\mathrm{d}\lambda}{\mathrm{d}\lambda_{A}}\frac{\mathrm{d}\lambda_{A}}{\mathrm{d}m_{nA}}=\frac{k-\lambda}{\lambda}\lambda_{A}x_{n}$
(20)

###  B.3 Mixture of populations case

PMF for two Poisson processes, one with prevalence $f$ and one with prevalence
$(1-f)$, is

$P=f\times\frac{\lambda_{A}^{k}\exp(-\lambda_{A})}{k!}+(1-f)\times\frac{\lambda_{B}^{k}\exp(-\lambda_{B})}{k!}$
(21)

Per-row contribution to deviance is log of PMF, so:

$D=\ln\left(f\lambda_{A}^{k}\exp(-\lambda_{A})+(1-f)\lambda_{B}^{k}\exp(-\lambda_{B})\right)-\ln(k!)$
(22)

Differentiate with respect to $\lambda_{A}$

$\displaystyle\frac{\mathrm{d}D}{\mathrm{d}\lambda_{A}}$
$\displaystyle=\frac{f\times\left(k\lambda_{A}^{k-1}\exp\left(-\lambda_{A}\right)-\lambda_{A}^{k-1}\exp(-\lambda_{A})\right)}{f\times\lambda_{A}^{k}\exp(-\lambda_{A})+(1-f)\times\lambda_{B}^{k}\exp(-\lambda_{B})}$
(23)
$\displaystyle=\frac{f\times(k-\lambda_{A})\lambda_{A}^{k-1}\exp(-\lambda_{A})}{f\times\lambda_{A}^{k}\exp(-\lambda_{A})+(1-f)\times\lambda_{B}^{k}\exp(-\lambda_{B})}$
(25)

Equations (16), (17) and (18) still apply, so all that remains is to chain
them to (25):

$\displaystyle\frac{\mathrm{d}D}{\mathrm{d}m_{nA}}=\frac{\mathrm{d}D}{\mathrm{d}\lambda_{A}}\frac{\mathrm{d}\lambda_{A}}{\mathrm{d}m_{nA}}=x_{n}\times\frac{f\times(k-\lambda_{A})\lambda_{A}^{k}\exp(-\lambda_{A})}{f\times\lambda_{A}^{k}\exp(-\lambda_{A})+(1-f)\times\lambda_{B}^{k}\exp(-\lambda_{B})}$
(26)
$\displaystyle\frac{\mathrm{d}D}{\mathrm{d}c_{A}}=\frac{\mathrm{d}D}{\mathrm{d}\lambda_{A}}\frac{\mathrm{d}\lambda_{A}}{\mathrm{d}c_{A}}=\frac{f\times(k-\lambda_{A})\lambda_{A}^{k}\exp(-\lambda_{A})}{f\times\lambda_{A}^{k}\exp(-\lambda_{A})+(1-f)\times\lambda_{B}^{k}\exp(-\lambda_{B})}$
(27)

##  Appendix C Results from scaling multiresp

The multiresp model in section 3.3 only matches the performance of an XGB
model with depth 3, and does not exceed it. This is because we tried to use
the simplest synthetic dataset for which an XGB model with depth 3 would
consistently outperform an XGB model with depth 1. A slightly more complex
dataset, generated as in section 3.3 but with twice as many explanatory
variables, was used to compare multiresp and XGB.

| MAE | RMSE
---|---|---
$d=1,500$ trees | $11.6716$ | $25.8049$
$d=1,1000$ trees | $11.6717$ | $25.8051$
$d=2,500$ trees | $4.1651$ | $7.7415$
$d=3,500$ trees | $3.1354$ | $4.9433$
$d=4,500$ trees | $2.8113$ | $4.2739$
$d=5,500$ trees | $2.7685$ | $4.1910$
$d=6,500$ trees | $2.7713$ | $4.1932$
$d=7,500$ trees | $2.7712$ | $4.1923$
$d=8,500$ trees | $2.7712$ | $4.1924$
Canonical model, $S=1$ | $11.6717$ | $25.8051$
Canonical model, $S=2$ | $7.9865$ | $19.3870$
Canonical model, $S=3$ | $\mathbf{2.7442}$ | $\mathbf{4.1687}$
Table 11: Example results for Scenario 4.4

Note that here the best multiresp model outperforms the best XGB model. Note
also that performance for the XGB model stops improving around depth 6 because
only 6 of the 8 features behave differently between sub-responses.

##  Appendix D Failed approaches

For completeness, we describe two experiments which did not demonstrate the
tree depth - robustness connection.

###  D.1 Highly nonlinear effects

We did not find a generator for which higher treedepth caused improved
performance when modelling highly nonlinear effects from continuous features.
As expected, high-depth models learned the ‘inner’ parts of a w-shaped feature
effect at the same rate as the ‘outer’ parts, and lower-depth models fixated
on the edges. But no scenario was found where this learning dynamic had a
reliable impact on model performance.

###  D.2 Outliers and mis-specified errors

The presence of outliers (i.e. the use of incorrect error distribution)
appeared to lead to higher tree depth robustness, but the benefit was only
consistent for some loss metrics, and only when tree depth was already $d>1$.
That is: the MAE for $d=10$ would be consistently lower than for $d=5$ in the
presence of sufficiently severe outliers, but we did not find a scenario where
MAE was consistently lower than for $d=1$; moreover, extra complexity did not
systematically improve RMSE at all.

(The ability of extra model complexity to lower MAE but not RMSE – by enabling
it to ignore outliers – helps to explain the non-monotonic effect on RMSE in
Table 7.)


# Improving Audio Captioning Models with Fine-grained Audio Features, Text
Embedding Supervision, and LLM Mix-up Augmentation

###### Abstract

Automated audio captioning (AAC) aims to generate informative descriptions for
various sounds from nature and/or human activities. In recent years, AAC has
quickly attracted research interest, with state-of-the-art systems now relying
on a sequence-to-sequence (seq2seq) backbone powered by strong models such as
Transformers. Following the macro-trend of applied machine learning research,
in this work, we strive to improve the performance of seq2seq AAC models by
extensively leveraging pretrained models and large language models (LLMs).
Specifically, we utilize BEATs to extract fine-grained audio features. Then,
we employ Instructor LLM to fetch text embeddings of captions, and infuse
their language-modality knowledge into BEATs audio features via an auxiliary
InfoNCE loss function. Moreover, we propose a novel data augmentation method
that uses ChatGPT to produce caption mix-ups (i.e., grammatical and compact
combinations of two captions) which, together with the corresponding audio
mixtures, increase not only the amount but also the complexity and diversity
of training data. During inference, we propose to employ nucleus sampling and
a hybrid reranking algorithm, which has not been explored in AAC research.
Combining our efforts, our model achieves a new state-of-the-art 32.6 SPIDEr-
FL score on the Clotho evaluation split, and wins the 2023 DCASE AAC
challenge.

Index Terms—  AAC, BEATs, LLM, mix-up, InfoNCE

##  1 Introduction

Automated audio captioning (AAC) is a multimodal task whose goal is to
describe an input audio clip using text. The descriptions are not restricted
to a fixed set of class labels or tags, but are free-form sentences [1], which
allow better flexibility and expressivity. Research progress on AAC has
accelerated in recent years thanks to the yearly DCASE challenges, the
impressive performance of Transformer-based language models, and the release
of the open audio captioning datasets Clotho [2] and AudioCaps [3]. Recent
leading works in AAC [4, 5, 6, 7, 8, 9] all used the sequence-to-sequence
(seq2seq) modeling framework, where an audio encoder is used to extract
features from the input audio, and a text decoder learns to generate the
caption autoregressively based on the extracted audio features.

While our work is also seq2seq-based, we extensively leverage machine learning
models that are pretrained on large-scale datasets to improve AAC performance
from multiple aspects, namely, audio feature extraction, auxiliary training
objective, and data augmentation. We begin with revamping the audio encoder
(Section 2.1). While PANN [10], a convolution-based audio feature extractor,
has long been the tried-and-true choice for AAC research [4, 5, 6], many
recently proposed audio encoders [11, 12, 13] have the potential to further
improve AAC performance due to more advanced architectures, pretraining
objectives, and finer-grained output features. Specifically, we choose
Bidirectional Encoder representation from Audio Transformers (BEATs) [13], a
state-of-the-art multi-label audio tagging model pretrained on AudioSet [14],
as our audio encoder.

Next, witnessing the tremendous success of large language models (LLMs) in
representing and generating text [15, 16], we use text embeddings from the
Instructor Transformer [17] to provide additional supervision (Section 2.2),
and employ ChatGPT [18] to perform a novel mix-up data augmentation (Section
2.3). In previous AAC studies, to help linking audio features with concepts in
text, which is the output space of AAC tasks, [5] pretrained the PANN encoder
with an audio-caption InfoNCE [19] contrastive loss, while [6] used multitask
learning to predict keywords in the caption. In our work, we combine the
benefits of both representation learning and multitask training, and also
leverage the LLM’s rich knowledge of text. Particularly, we use Instructor to
obtain text embeddings for ground-truth captions, and apply an auxiliary
InfoNCE loss on a Conformer [20, 21] postencoder to refine/summarize the BEATs
audio features and align them with Instructor text embeddings.

For data augmentation, other than SpecAugment [22] used commonly in audio-
related tasks, researchers have leveraged the original mix-up [23] to linearly
combine audio/text embeddings from two unrelated samples [8, 24], synonym
substitution on ground-truth captions [7], and caption concatenation [7]. As
for LLM-based efforts, ChatGPT has been used to compile the large-scale
WavCaps [25, 7, 8] AAC dataset by rewriting tags or fragmented descriptions,
which are often associated with audio files on the web, to coherent sentences.
Integrating the ideas of mix-up and LLM augmentation methods, we prompt
ChatGPT to mix-up the captions of two audio clips, which produces more natural
combined captions than simple text concatenation [7]. The text mix-ups, when
paired with audio mix-ups (i.e., summations of waveforms), increase the
amount, complexity, and diversity of our training data. With all the
techniques above, we also discover that nucleus sampling decoding [26]
followed by hybrid reranking (Section 2.4) leads to a further performance
boost.

Our AAC model attains a state-of-the-art SPIDEr-FL score of 32.6 (on Clotho V2
[2] evaluation set) and is the winner of the 2023 DCASE AAC
Challenge.111Associated technical report at:
dcase.community/documents/challenge2023/technical_reports/DCASE2023_Wu_31_t6a.pdf
Despite the numerous components introduced to our model, we show in our
ablation study (Section 3.4) that every component is indispensable to its
great performance. We plan to release our codebase and the ChatGPT-generated
caption mix-ups upon paper publication.

Fig. 1: Overview of our Transformer-based captioning system. We utilize a
frozen BEATs [13] to extract audio features from the mel spectrogram. On top
of BEATs, we attach a Conformer [20] postencoder to further contextualize the
audio features. Then, a BART [27] text decoder cross-attends to the
contextualized audio features and generates the caption autoregressively. To
provide text-modality guidance to our encoder stack, we extract the captions’
sentence embeddings from an instruction-tuned large language model (LLM),
Instructor-XL [17], and apply an InfoNCE [19] auxiliary loss to train
Conformer’s output audio representation to mimic the corresponding caption’s
Instructor sentence embedding.

##  2 Method

###  2.1 Network Architecture and Main Loss Function

We utilize BEATs [13] as our main audio encoder. The BEATs module takes a 16
kHz audio waveform as input, converts the waveform into a mel spectrogram with
a 10-millisecond hop size, splits the spectrogram into 2D patches, and
transforms the patches into a sequence of representations through 12 self-
attention (i.e., Transformer) layers. Compared to PANN [10], which has been
popular in the AAC literature, BEATs comes with some key modifications:

  * •

Architecture: BEATs features a Transformer backbone, while PANN is based on a
convolutional neural network (CNN).

  * •

Pretraining objectives: While both BEATs and PANN are pretrained on AudioSet
[14], a general-domain, large-scale audio dataset, BEATs is first trained on
masked language modeling [28] of tokenized audio, and then on multilabel audio
classification. PANN is only trained on the latter.

  * •

Resolution: BEATs provides more fine-grained outputs at 50 Hz, compared to
PANN, which has 1-Hz outputs.

Due to these differences, and the better performance of BEATs on AudioSet
multilabel classification (50.6% vs. 43.9% mean average precision), BEATs
would likely be a more suitable selection for the AAC task. In our pilot
experiments, we tried either to finetune the BEATs module or to keep it
frozen. Both options led to similar SPIDEr-FL score, so we simply freeze BEATs
to reduce computation and memory footprint.

Given that the BEATs module is frozen, to enable further training on the audio
features (more details in Section 2.2), we attach a convolutional downsampling
layer, followed by a 2-layer222The number of layers is determined by
hyperparameter search. Conformer [20] postencoder on top of the BEATs module.
These additional layers further contextualize the audio features, and reduce
the text decoder’s workload for summarizing the audio features.

Following the recent trend of AAC models [4, 5], we adopt a 6-layer BART
Transformer decoder [27] to generate captions. We use the default BART text
tokenizer with a 50K vocabulary size, and train the BART’s weights from
scratch. The BART decoder cross-attends to the Conformer’s output
representations and self-attends to the historical caption tokens to generate
the next caption token autoregressively. The main loss function to our BEATs-
BART captioning model, applied on the BART’s output distributions, is the
negative log-likelihood (NLL) of audio captions, i.e.,

$\mathcal{L}_{\mathrm{NLL}}=\mathbb{E}_{(\bm{x},\bm{y})\in\mathcal{D_{\text{train}}}}\Big{[}\sum_{n=1}^{|\bm{y}|}-\log
p(y_{n}|\bm{y}_{1:n-1};\bm{x})\Big{]}\,,$ (1)

where $\mathcal{D_{\text{train}}}$ is the training dataset, $\bm{x}$ is the
input audio waveform, $\bm{y}$ is an audio caption, and $y_{n}$ is the
$n^{\text{th}}$ token in the caption. A schematic overview of our captioning
model is depicted in Fig. 1.

Table 1: Randomly chosen samples of ChatGPT mix-up augmentations. In general, ChatGPT is able to faithfully reflect all content in the two input captions grammatically, and sometimes exhibits creativity in sentence structuring and vocabulary choice. Clotho caption #1 |  Clotho caption #2 |  ChatGPT mix-up
---|---|---
water flowing over some rocks throughout a creek |  in the distance fireworks pop and crackle constantly as they are set off |  a serene creek babbles over rocks as distant fireworks pop and crackle in celebration
a muffled object is dragged along a surface in a room that echoes |  several dogs barking with many birds making noise in the background |  dogs bark in a room that echoes while a muffled object is dragged as birds chirp faintly in the background
a gate squeals as it sways while birds chirp in the background |  a machine is whirring loudly at first and then slowly shuts off |  as the gate sways and creaks a nearby machine loudly whirs before slowly powering down amidst chirping birds

###  2.2 Instructor Embedding Supervision

To infuse text-related knowledge into our audio (i.e., encoder) features, we
leverage an LLM, Instructor-XL333‘XL’ describes the size of the network—it has
1.5B parameters. Transformer [17], to fetch the text embeddings for ground-
truth captions and supervise our encoder stack with them. Instructor is based
on a pretrained T5 [29] text encoder, that is then finetuned using InfoNCE
loss [19] on a variety of natural language processing (NLP) tasks, such as
classification, reranking, summarization, and text quality evaluation, to
learn sentence-level text embeddings. Task- and domain-specific instructions
are prepended to the input text as conditions, e.g., “Represent the Medicine
statement for retrieval:”, hence the name Instructor. In the Massive Text
Embedding Benchmark (MTEB) [30], Instructor-XL is the state of the art on text
summarization and reranking tasks,444As of May 2023, when our project was
being carried out. which are closely related to audio captioning.

In our use case, we place “Represent the audio caption:” as the instruction to
the (frozen) Instructor to fetch sentence embeddings from ground-truth
captions. On our BEATs-Conformer encoder stack, we perform mean-pooling along
the timestep dimension to obtain a single audio embedding for the input
waveform. We denote the audio embedding and the Instructor caption embedding
by $\bm{a}$ and $\bm{c}$ respectively. An auxiliary InfoNCE loss is computed
using in-batch negative samples:

$\displaystyle\mathrm{sim}(\bm{a},\bm{c})$
$\displaystyle=\exp\Big{(}\frac{\bm{a}^{\top}\bm{c}}{||\bm{a}||\,||\bm{c}||}\cdot\frac{1}{\tau}\Big{)},$
(2) $\displaystyle\mathcal{L}_{\mathrm{InfoNCE{\textunderscore}a}}$
$\displaystyle=\mathbb{E}_{\mathcal{B}\subset\mathcal{D_{\text{train}}}}\Big{[}\sum_{i=1}^{|\mathcal{B}|}-\log\frac{\mathrm{sim}(\bm{a}_{i},\bm{c}_{i})}{\sum_{j=1}^{|\mathcal{B}|}\mathrm{sim}(\bm{a}_{j},\bm{c}_{i})}\Big{]},$
(3) $\displaystyle\mathcal{L}_{\mathrm{InfoNCE{\textunderscore}c}}$
$\displaystyle=\mathbb{E}_{\mathcal{B}\subset\mathcal{D_{\text{train}}}}\Big{[}\sum_{i=1}^{|\mathcal{B}|}-\log\frac{\mathrm{sim}(\bm{a}_{i},\bm{c}_{i})}{\sum_{j=1}^{|\mathcal{B}|}\mathrm{sim}(\bm{a}_{i},\bm{c}_{j})}\Big{]},$
(4) $\displaystyle\mathcal{L}_{\mathrm{InfoNCE}}$
$\displaystyle=\frac{1}{2}(\mathcal{L}_{\mathrm{InfoNCE{\textunderscore}a}}+\mathcal{L}_{\mathrm{InfoNCE{\textunderscore}c}})\,,$
(5)

where $\mathrm{sim(\cdot,\cdot)}$ is the exponentiated temperature-scaled
cosine similarity, $\tau$ is the temperature hyperparameter,555Generally
speaking, a higher temperature makes the contrastive objective more
challenging, as the distribution is made less peaky. We perform a search in
$\tau$ = {0.03, 0.07, 0.2, 0.5, 1.0} and find $\tau$ = 0.5 works the best.
$\mathcal{B}$ denotes a sampled mini-batch, and $i,j$ index samples in the
mini-batch. The multitask loss $\mathcal{L}$ used to train our model can hence
be written as:

$\mathcal{L}=\mathcal{L}_{\mathrm{NLL}}+\alpha\mathcal{L}_{\mathrm{InfoNCE}}\,,$
(6)

where $\alpha$ is a hyperparameter and we find $\alpha=1$ works well.

###  2.3 ChatGPT Mix-up Augmentation

As a novel data augmentation measure, we employ another LLM, ChatGPT [18], to
‘mix-up’ [23, 31] pairs of captions in the Clotho dataset, and create more
complex and diverse in-domain training data. Specifically, we mix-up captions
with different corresponding audio clips, rather than two ground-truth
captions for the same audio. The corresponding audio waveforms are also mixed
up to ensure consistency between audio and mixed-up captions.

We collect such mix-up augmentations using the public ChatGPT API. In the
prompt, we ask ChatGPT to “Generate a mix of the following two audio captions,
and keep the generation under 25 words:”, and then provide it with two
randomly sampled captions from Clotho [2]. We explicitly limit the number of
words to force ChatGPT to be more concise. We use the FENSE disfluency
detector [32] to filter out poor examples.666Less than 1% of ChatGPT mix-ups
are detected as disfluent. Mix-up of audio waveforms is more straightforward:
we follow the algorithm used in WavLM [33], scaling the two waveforms to
ensure their relative root-mean-square energy is within $\pm$5 dB before
adding them together.

Table 1 displays a few examples of ChatGPT-generated mix-ups. We try including
either 50K or 100K ChatGPT mix-ups, and using 50K yields a better performance.
The API cost for generating 50K mix-ups is roughly $8.50.

###  2.4 Sampling and Reranking

In past AAC research works, the most commonly used decoding algorithm has been
beam search [5, 6, 4]. However, we find that, after introducing all the
techniques in Section 2.1$\sim$2.3, around 1/3 of generations using nucleus
sampling [26], which is known to produce more diverse and informative
generations than beam search, score higher in terms of SPIDEr-FL than those
using beam search. This reveals the potential advantage of a sampling-then-
reranking approach.

To ‘pick the right sample’ with nucleus sampling, we propose a hybrid
reranking algorithm that utilizes again the knowledge of both our learned
audio encoder stack and our text decoder. The two reranking metrics we
consider are:

  * •

Caption log-likelihood: We feed the input waveform $\bm{x}$ and the generated
caption $\bm{\hat{y}}$ into our captioning model to directly compute $\log
p(\bm{\hat{y}}\,|\,\bm{x})=\sum_{n=1}^{|\bm{\hat{y}}|}\log
p(\hat{y}_{n}\,|\,\bm{\hat{y}}_{1:n-1};\bm{x})$ (cf. Eq. (1)). As the log-
likelihood is computed on decoder outputs, we call this decoder reranking.

  * •

Audio-caption representation similarity: We feed the generated caption
$\bm{\hat{y}}$ into the Instructor model to get its text embedding
$\bm{\hat{c}}$, and fetch the audio embedding $\bm{a}$ of the input waveform
$\bm{x}$ from our trained audio encoder stack. Then, we compute the cosine
similarity between the text and audio embeddings, i.e.,
$(\bm{a}^{\top}\bm{\hat{c}})\,/\,(||\bm{a}||\,||\bm{\hat{c}}||)$ (cf. Eq.
(2)). As the representation from the audio encoder is used here, we refer to
this as encoder reranking.

Candidate captions are ranked by the weighted sum of the two metrics above
(with weights tuned on some held-out validation data), and we return the
highest-scoring one as the final predicted caption.

Table 2: Ablation study results on Clotho [2] evaluation split. The results demonstrate that every component introduced in our AAC model, i.e., BEATs [13] audio encoder (Section 2.1), Instructor [17] sentence embedding supervision (Section 2.2), ChatGPT [18] mix-up augmentation (Section 2.3), and nucleus sampling [26] $\,+\,$reranking (Section 2.4), is beneficial to the performance.  | Model components |  Performance metrics (in %)
---|---|---
| Audio encoder |  Instructor emb. | ChatGPT mix-up | Decoding | Reranking | METEOR | CIDEr | SPICE | SPIDEr | SPIDEr-FL
Full model | BEATs | ✓ | ✓ | Sampling | Hybrid | 19.3 | 50.6 | 14.6 | 32.6 | 32.6
w/o hybrid rerank and/or sampling | BEATs | ✓ | ✓ | Sampling | Decoder only | 18.6 | 45.1 | 13.7 | 29.4 |  29.4 ($-$3.2)
| BEATs | ✓ | ✓ | Sampling | Encoder only | 18.0 | 43.7 | 13.4 | 28.5 |  28.5 ($-$4.1)
| BEATs | ✓ | ✓ | Beam search | n.a. | 18.7 | 47.4 | 13.4 | 30.4 |  30.3 ($-$2.3)
w/o ChatGPT mixup | BEATs | ✓ | ✗ | Sampling | Hybrid | 19.1 | 47.6 | 13.8 | 30.7 |  30.7 ($-$1.9)
| BEATs | ✓ | ✗ | Beam search | n.a. | 18.6 | 47.7 | 13.1 | 30.4 |  30.2 ($-$2.4)
w/o Instructor | BEATs | ✗ | ✓ | Beam search | n.a. | 18.6 | 45.8 | 13.4 | 29.6 |  29.4 ($-$3.2)
w/o BEATs | PANN | ✓ | ✓ | Sampling | Hybrid | 17.5 | 39.7 | 12.3 | 26.0 |  26.0 ($-$6.6)
| PANN | ✓ | ✓ | Beam search | n.a. | 17.4 | 41.6 | 12.2 | 26.9 |  26.7 ($-$5.9)
Table 3:  Performance comparison (on Clotho [2] evaluation split) with top-ranking methods in recent DCASE challenges. All models presented here are single models, not ensembles. ‘RL’ indicates the use of reinforcement learning [34] to directly optimize CIDEr score. For fair comparison, best results in each group (i.e., w/ or w/o RL) are bold-faced. Notice that RL can lead to a heavy punishment on SPIDEr-FL, the new DCASE official metric, due to fluency flaws.  |  |  Performance metrics (in %)
---|---|---
| RL | METEOR | CIDEr | SPICE | SPIDEr | SPIDEr-FL
Ours | ✗ | 19.3 | 50.6 | 14.6 | 32.6 | 32.6
Labbé et al., ’23 [8] | ✗ | 19.2 | 48.5 | 13.9 | 31.2 | 31.0
Cho et al., ’23 [7] | ✗ | 18.8 | 48.3 | 13.7 | 31.0 | 30.7
Ye et al., ’22 [6] | ✗ | 17.8 | 44.5 | 12.7 | 28.6 |  n.a. ($\leq$28.6)
Xu et al., ’22 [5] | ✗ | 17.9 | 42.1 | 12.7 | 27.4 |  n.a. ($\leq$27.4)
Cho et al., ’23 [7] | ✓ | 19.5 | 52.6 | 14.3 | 33.5 | 22.5
Ye et al., ’22 [6] | ✓ | 18.5 | 50.3 | 13.2 | 31.7 |  n.a. ($\leq$31.7)
Xu et al., ’22 [5] | ✓ | 18.6 | 50.9 | 12.0 | 31.5 |  n.a. ($\leq$31.5)

##  3 Experiments and Results

###  3.1 Training and Inference

We first pretrain the model on the combined dataset of AudioCaps777We filter
out captions with $<$6 words, leading to 35K remaining samples. [3] and 50K
ChatGPT mix-ups of samples from the better-curated but smaller Clotho [2]
dataset for 10 epochs (about 13K gradient steps), and then finetune it on
Clotho (development split, $\sim$4K samples) for 40 epochs (or 1.2K steps).
Teacher-forcing is applied on the BART decoder inputs. We adopt the AdamW
optimizer with a 2$\,\times\,$10-4 learning rate for the
‘AudioCaps$\,+\,$ChatGPT mix-up’ pretraining stage, and 2$\,\times\,$10-5 for
the Clotho finetuning stage.

As the Conformer attention (see Section 2.1) is the primary memory bottleneck
due to the long sequence length of audio features, there is a tradeoff between
the batch size that can be used and the downsampling rate for the Conv1D layer
between our BEATs and Conformer modules—using less downsampling gives the
model finer-grained audio features, but a smaller batch size would lead to
less reliable gradients and hamper contrastive learning [35]. Through
experiments, we settle on the 3x downsampling rate, which allows a batch size
of 32 and achieves the best performance.

We train the model on two NVIDIA A100 (40GB) GPUs, and the two training stages
take around 6 and 3 hours respectively. Next-token prediction accuracy on the
Clotho validation split is used as the checkpoint selection criterion.

At inference time, we experiment with generating {20, 50, 100} candidate
captions per test case with nucleus sampling,888Nucleus sampling
hyperparameters: temperature 0.5; cumulative distribution truncation point,
i.e., top-$p$, 0.95. and find that generating 50 strikes the best balance
between performance gain and compute efficiency. Additionally, we leverage the
FENSE evaluator [32] to filter out generations with fluency issues. We tune
the weights of the reranking metrics (see Section 2.4) on the Clotho
validation split and eventually pick {0.3, 0.7} respectively for decoder and
encoder reranking metrics.999Weights for ablated models (see Table 2) are
separately tuned to be fair.

###  3.2 Evaluation

The metrics used to evaluate the quality of generated captions are: METEOR,
CIDEr, SPICE, SPIDEr, and SPIDEr-FL. METEOR and CIDEr are both based on
$n$-gram overlap, with the former penalizing fragmentation between the ground-
truth and generated captions, and the latter promoting generating informative
words by weighting $n$-grams by their TF-IDF scores. SPICE focuses on the
overlap computed on semantic graphs constructed by objects, object attributes,
and relations. SPIDEr is the simple mean of CIDEr and SPICE, and it had been
the official evaluation metric in DCASE AAC challenges until 2022. In 2023,
the official metric was changed to SPIDEr-FL, which uses FENSE [32], i.e., a
BERT-based binary classifier, to penalize the SPIDEr score of disfluent
generations by 90%.

Our evaluation is done on the public ‘evaluation’ split101010DCASE challenge
uses the blind ‘test’ split to rank the submissions. of the Clotho [2]
dataset, which consists of 1,045 samples. The evaluation results of our full
model can be found in the 1${}^{\text{st}}$ row of Table 2.

###  3.3 Comparison with Past and Concurrent Works

We compare our model (i.e., winner of the DCASE 2023 AAC challenge) to other
top performers in the 2022 and 2023 challenges [5, 6, 7, 8]. We note that
while all the best-scoring systems for each participant were ensemble models,
we present the metrics for single models for fairness and practicality
reasons.111111Ensembles can contain a wildly different # of models (e.g.,
3$\sim$20), and the performance gain can seldom justify the extra compute
required. The comparison in Table 3 shows that our model is state-of-the-art
in terms of the new official metric, SPIDEr-FL, and performs competitively on
other metrics. Moreover, while optimizing CIDEr with reinforcement learning
has been popular among challenge submissions, the resulting disfluency issues
[36] get severely punished on SPIDEr-FL (see 6${}^{\text{th}}$ row in Table
3).

###  3.4 Ablation Study

To show that every component in our AAC model is indispensable to achieve the
best performance, we conduct a comprehensive ablation study that tries to
remove components one at a time. Table 2 presents the results that corroborate
the necessity of all of our model components—Each one of them gives at least a
2-point improvement on SPIDEr-FL,121212We cannot perform hybrid reranking with
the ‘w/o Instructor’ setting as the audio encoder is not trained to match the
caption text embedding. Thus, we simply use beam search as it outperforms
decoder-only reranking. with BEATs audio encoder (replacing the popular PANN)
being the most crucial one causing a 6-point difference.

Some intriguing additional findings are: (i) hybrid reranking is required to
outperform beam search (see rows 1$\sim$4 in Table 2), suggesting that decoder
and encoder reranking methods are strongly complementary and hence should be
used together when possible; (ii) the ‘sampling$\,+\,$reranking’ decoding
approach improves performance the most when both BEATs encoder and ChatGPT
mix-ups are used (see rows ‘1 vs. 4’, ‘5 vs. 6’, and ‘8 vs. 9’ in Table 2).

##  4 Conclusion and Future Work

In this work, we improved audio captioning models from multiple aspects with
an extensive use of pretrained models. We employed the BEATs Transformer to
extract more fine-grained audio features. We then utilized the Instructor text
embeddings for multitask learning to provide rich language-modality guidance.
ChatGPT was also leveraged to generate faithful and fluent caption mix-ups
which, when paired with the corresponding audio mix-ups, increased the size,
diversity, and complexity of our training data. Finally, nucleus sampling and
hybrid reranking were used to exploit our model’s capabilities to the fullest
extent. We accomplished a state-of-the-art 32.6 SPIDEr-FL score and
demonstrated via a thorough ablation study that all components are crucial to
our model’s success.

Future endeavors may explore audio feature extractors that are pretrained with
larger amounts of data [37] or multimodal supervision [38]. More advanced
reinforcement learning methods [39] can also be applied to optimize captioning
metrics that correlate well with human judgment [40] without introducing
disfluency issues.

## References

  * [1] K. Drossos, S. Adavanne and T. Virtanen,  “Automated audio captioning with recurrent neural networks,”  in Proc. WASPAA, 2017.
  * [2] K. Drossos, S. Lipping and T. Virtanen,  “Clotho: an audio captioning dataset,”  in Proc. ICASSP, 2020.
  * [3] C. D. Kim, B. Kim, H. Lee and G. Kim,  “AudioCaps: Generating captions for audios in the wild,”  in Proc. NAACL-HLT, 2019.
  * [4] W. Yuan, Q. Han, D. Liu et al.,  “The DCASE 2021 challenge task 6 system: Automated audio captioning with weakly supervised pre-training and word selection methods,”  Tech. Rep., DCASE Challenge, 2021.
  * [5] X. Xu, Z. Xie, M. Wu and K. Yu,  “The SJTU system for DCASE2022 challenge task 6: Audio captioning with audio-text retrieval pre-training,”  Tech. Rep., DCASE Challenge, 2022.
  * [6] Z. Ye, Y. Zou, F. Cui and Y. Wang,  “Automated audio captioning with multi-task learning,”  Tech. Rep., DCASE Challenge, 2022.
  * [7] J.-H. Cho, Y.-A. Park, J. Kim and J.-H. Chang,  “HYU submission for the DCASE 2023 task 6a: Automated audio captioning model using AL-MixGen and synonyms substitution,”  Tech. Rep., DCASE Challenge, 2023.
  * [8] E. Labbé, T. Pellegrini and J. Pinquier,  “IRIT-UPS DCASE 2023 audio captioning and retrieval system,”  Tech. Rep., DCASE Challenge, 2023.
  * [9] C. P. Narisetty, T. Hayashi, R. Ishizaki et al.,  “Leveraging state-of-the-art ASR techniques to audio captioning.,”  in Proc. DCASE, 2021.
  * [10] Q. Kong, Y. Cao, T. Iqbal et al.,  “PANNs: Large-scale pretrained audio neural networks for audio pattern recognition,”  IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2020.
  * [11] K. Chen, X. Du, B. Zhu et al.,  “HTS-AT: A hierarchical token-semantic audio transformer for sound classification and detection,”  in Proc. ICASSP, 2022.
  * [12] P.-Y. Huang, H. Xu, J. Li et al.,  “Masked autoencoders that listen,”  in Proc. NeurIPS, 2022.
  * [13] S. Chen, Y. Wu, C. Wang et al.,  “BEATs: Audio pre-training with acoustic tokenizers,”  arXiv preprint arXiv:2212.09058, 2022.
  * [14] J. F. Gemmeke, D. P. Ellis, D. Freedman et al.,  “Audio Set: An ontology and human-labeled dataset for audio events,”  in Proc. ICASSP, 2017.
  * [15] L. Ouyang, J. Wu, X. Jiang et al.,  “Training language models to follow instructions with human feedback,”  in Proc. NeurIPS, 2022.
  * [16] H. Touvron, T. Lavril, G. Izacard et al.,  “LLaMA: Open and efficient foundation language models,”  arXiv preprint arXiv:2302.13971, 2023.
  * [17] H. Su, J. Kasai, Y. Wang et al.,  “One embedder, any task: Instruction-finetuned text embeddings,”  arXiv preprint arXiv:2212.09741, 2022.
  * [18] J. Schulman, B. Zoph, C. Kim et al.,  “Introducing ChatGPT,”  OpenAI Blog, 2022.
  * [19] A. van den Oord, Y. Li and O. Vinyals,  “Representation learning with contrastive predictive coding,”  arXiv preprint arXiv:1807.03748, 2018.
  * [20] A. Gulati, J. Qin, C.-C. Chiu et al.,  “Conformer: Convolution-augmented transformer for speech recognition,”  in Proc. Interspeech, 2020.
  * [21] K. Miyazaki, T. Komatsu, T. Hayashi et al.,  “Convolution-augmented transformer for semi-supervised sound event detection,”  in Proc. DCASE, 2020.
  * [22] D. S. Park, W. Chan, Y. Zhang et al.,  “SpecAugment: A simple data augmentation method for automatic speech recognition,”  in Proc. Interspeech, 2019.
  * [23] H. Zhang, M. Cisse, Y. N. Dauphin and D. Lopez-Paz,  “mixup: Beyond empirical risk minimization,”  in Proc. ICLR, 2018.
  * [24] T. Kouzelis, G. Bastas, A. Katsamanis and A. Potamianos,  “Efficient audio captioning transformer with patchout and text guidance,”  Tech. Rep., DCASE Challenge, 2022.
  * [25] X. Mei, C. Meng, H. Liu et al.,  “WavCaps: A chatgpt-assisted weakly-labelled audio captioning dataset for audio-language multimodal research,”  arXiv preprint arXiv:2303.17395, 2023.
  * [26] A. Holtzman, J. Buys, L. Du et al.,  “The curious case of neural text degeneration,”  in Proc. ICLR, 2020.
  * [27] M. Lewis, Y. Liu, N. Goyal et al.,  “BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension,”  in Proc. ACL, 2020.
  * [28] J. Devlin, M.-W. Chang, K. Lee and K. Toutanova,  “BERT: Pre-training of deep bidirectional Transformers for language understanding,”  in Proc. NAACL-HLT, 2019.
  * [29] C. Raffel, N. Shazeer, A. Roberts et al.,  “Exploring the limits of transfer learning with a unified text-to-text transformer,”  Journal of Machine Learning Research, 2020.
  * [30] N. Muennighoff, N. Tazi, L. Magne and N. Reimers,  “MTEB: Massive text embedding benchmark,”  arXiv preprint arXiv:2210.07316, 2022.
  * [31] Y. Gong, Y.-A. Chung and J. Glass,  “PSLA: Improving audio tagging with pretraining, sampling, labeling, and aggregation,”  IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2021.
  * [32] Z. Zhou, Z. Zhang, X. Xu et al.,  “Can audio captions be evaluated with image caption metrics?,”  in Proc. ICASSP, 2022.
  * [33] S. Chen, C. Wang, Z. Chen et al.,  “WavLM: Large-scale self-supervised pre-training for full stack speech processing,”  IEEE Journal of Selected Topics in Signal Processing, 2022.
  * [34] S. J. Rennie, E. Marcheret, Y. Mroueh et al.,  “Self-critical sequence training for image captioning,”  in Proc. CVPR, 2017.
  * [35] T. Chen, S. Kornblith, M. Norouzi and G. Hinton,  “A simple framework for contrastive learning of visual representations,”  in Proc. ICML, 2020.
  * [36] X. Mei, Q. Huang, X. Liu et al.,  “An encoder-decoder based audio captioning system with transfer and reinforcement learning,”  in Proc. DCASE, 2021.
  * [37] B. Elizalde, S. Deshmukh, M. Al Ismail and H. Wang,  “CLAP: learning audio concepts from natural language supervision,”  in Proc. ICASSP, 2023.
  * [38] J. Schulman, B. Zoph, C. Kim et al.,  “ImageBind: holistic AI learning across six modalities,”  Meta AI Blog, 2023.
  * [39] G. O. dos Santos, E. L. Colombini and S. Avila,  “CIDEr-R: Robust consensus-based image description evaluation,”  in Proc. Workshop on Noisy User-generated Text, 2021.
  * [40] J. Hessel, A. Holtzman, M. Forbes et al.,  “CLIPScore: A reference-free evaluation metric for image captioning,”  in Proc. EMNLP, 2021.