#  Multitask learning in Audio Captioning: a sentence embedding regression
loss acts as a regularizer

Etienne Labbé  IRIT, Université de Toulouse
CNRS, UT3, Toulouse, France
<EMAIL_ADDRESS>Julien Pinquier  IRIT, Université de Toulouse
CNRS, UT3, Toulouse, France
<EMAIL_ADDRESS>Thomas Pellegrini  IRIT, Université de Toulouse
CNRS, UT3, Toulouse, France
<EMAIL_ADDRESS>
###### Abstract

In this work, we propose to study the performance of a model trained with a
sentence embedding regression loss component for the Automated Audio
Captioning task. This task aims to build systems that can describe audio
content with a single sentence written in natural language. Most systems are
trained with the standard Cross-Entropy loss, which does not take into account
the semantic closeness of the sentence. We found that adding a sentence
embedding loss term reduces overfitting, but also increased SPIDEr from 0.397
to 0.418 in our first setting on the AudioCaps corpus. When we increased the
weight decay value, we found our model to be much closer to the current state-
of-the-art methods, with a SPIDEr score up to 0.444 compared to a 0.475 score.
Moreover, this model uses eight times less trainable parameters. In this
training setting, the sentence embedding loss has no more impact on the model
performance.

###### Index Terms:

sound event description, multitask learning, audio language task, overfitting,
sentence embedding regression loss, semantic loss

##  I Introduction

In recent years, new machine learning systems have been significantly improved
for text processing, generation, and understanding, leading to the use of
natural language as a global interface between humans and machines. Free-form
text can contain much more information than a predefined set of classes, which
could improve the machine understanding of our world. In audio, most of the
tasks are focused on classification and localization of sound events.
Following this idea, the Automated Audio Captioning (AAC) task appeared in
2017 [1] and aims to create systems that generate a sentence written in
natural language that describes an audio file. The audio can contain various
sound events (human, natural, domestic, urban, music, effects…) of different
lengths, recorded with different devices and in different scenes. The
description can contain any kind of detail in the audio, with temporal or
spatial relations between them (followed by, in the background…) or different
characterizations (high-pitched, short, repetitive…). Since the descriptions
are written by humans, we need to consider different words used to describe
similar sounds (Birds are calling / chirping / singing / tweeting), different
sentence structures (A door that needs to be oiled / A door with squeaky
hinges), subjectivity (Man speaks in a foreign language), high-level
descriptions (A vulgar man speaks / Unintelligible conversation), and
vagueness (Someone speaks instead of A man gives a speech over a reverberating
microphone).

In AAC, most approaches use deep learning models trained with the standard
Cross-Entropy (CE) loss. However, this loss tends to generate repetitive and
generic content [2] and does not take into account synonyms, various sentences
structures or the semantic closeness. Several studies introduced another
criterion, the Self-Critical Sequence Training [3] (SCST) used in
reinforcement learning to fine-tune the model directly on a metric instead of
the loss. This technique relies on sampling the next word to generate a new
sentence. If this sentence has a higher score than the original one, the model
is rewarded and the outputs probabilities for this new sentence are
encouraged. However, this technique leads to degenerated sentences [4], with
repetitive n-grams without syntactical correctness.

Motivated by the limitations of CE and SCST, in this work, we attempted to add
a Sentence Embedding Regression (SER) loss used in [5] to improve our model.
We begin this paper by describing our baseline system and then explain how to
add SER loss. We present related work in which we compare and then describe
the detailed hyperparameters. Finally, we present the results and discuss the
differences.

##  II Baseline system description

We use an encoder-decoder architecture widely used in AAC systems, with an
encoder pre-trained on AudioSet [6] to extract a strong representation of
sounds events. More specifically, we used the CNN14_DecisionLevel_Att audio
encoder from the Pre-trained Audio Neural Networks study (PANN) [7], with the
pre-trained weights available on Zenodo111https://zenodo.org/record/3987831.
This architecture gives the best results on the classification of sound events
in the audio captioning dataset part when compared to the other PANN
architectures available. We have found that freezing weights does not decrease
performance while significantly speeding up the training process. This encoder
provides sequences of embeddings of dimension $31\times 2048$ for ten-second
long audio recordings. On top of that, we add a projection layer to get
256-dimensional embeddings to match the input dimension of the decoder
$d_{model}$.

The decoder is a standard transformer decoder [8] with 6 layers, 4 attention
heads per layer, a global embedding size $d_{model}$ set to 256 and a global
dropout probability of 0.2. We also used the GELU [9] activation layer in the
decoder.

The decoder is trained using teacher forcing, i.e. gives the ground truth
previous reference tokens to the model to predict the next one. The baseline
criterion is the standard CE loss over the whole sequence between the output
probabilities and the reference token classes.

During inference, we used the beam search algorithm with a beam size set to 2
since higher value does not bring improvements. We conditioned the sentence
generation to improve performance and overall caption quality: by limiting the
prediction length to a minimum of 3 tokens and a maximum of 30 tokens and by
forbidding the model to generate the same token twice, except for stop-word
tokens predefined in the Natural Language ToolKit (NLTK) [10] package. These
constraints reduce the number of invalid sentences and repetitions and give a
slight improvement in the performance of our model.

##  III Adding a sentence embedding regression loss

###  III-A Sentence-BERT model

The Sentence-BERT [11] (SBERT) model is a transformer-based model which
combines a BERT [12] model with a pooling and a projection layer to produce a
single embedding of a fixed size of 768 values for a given sentence.

Available SBERT models have been trained on two text databases: the Stanford
Natural Language Inference (SNLI) [13] and the Multi-Genre Natural Language
Inference (MultiNLI) [14]. These datasets contain pairs of two sentences
annotated with contradiction, entailment or neutral label. To learn sentence
semantic, two SBERT embedding are fed to a classification layer, which must
predicts the label of the pair.

###  III-B SER loss

To use the SBERT model to improve our model, we need to use the same token
units to have the same sequence size. BERT uses WordPiece tokens [15] instead
of words which form a vocabulary of 30522 different units in our experiments.

Fig. 1 resumes the whole procedure and layers used. During training phase, we
use audio features and the ground truth previous tokens to generate the next
token embeddings named $\hat{e}_{t}$. These embeddings are used in two
different parts of the model. First, they are projected to logits using a
classifier for the standard CE loss $\mathcal{L}_{t}$. The token embeddings
are also projected from 256 to 768-dimensional embedding to match the SBERT
embedding input shape. The resulting embedding $\hat{e}_{s}$ is used as input
with the ground truth embedding for the SER loss component $\mathcal{L}_{s}$.
In order to use the SBERT model to train our model, we need to remove the
first layer of SBERT which maps tokens IDs to embedding vectors (named ”Embed”
in the figure), since this layer is not differentiable. At inference time,
only the classifier branch is used.

Figure 1: Overview of our proposed training method. The blue boxes are the
pre-trained frozen layers, the purple ones the trainable layers and the orange
ones the functions. The SBERT block contains the SBERT model with its first
embedding layer removed.

We tried several regression criteria for the $\mathcal{L}_{s}$ loss:
CosineEmbeddingLoss, L1Loss, MSELoss and SmoothL1Loss. The best one that we
obtained is the SmoothL1Loss [16], a regression function which combines MSE
and L1Loss, described in equation (1):

$\mathcal{L}_{s}(\hat{e}_{s},e_{s})=\begin{cases}(\hat{e}_{s}-e_{s})^{2}\cdot\frac{1}{2\beta}&\text{if
}|\hat{e}_{s}-e_{s}|<\beta\\\
|\hat{e}_{s}-e_{s}|-\frac{\beta}{2}&\text{otherwise}\end{cases}$ (1)

The $\beta$ hyperparameter control whether MSE or L1Loss must be used. We kept
the standard CE as our first component $\mathcal{L}_{t}$ to help the model to
produce syntactically valid sentences. The final loss is given by equation (2)
and sum $\mathcal{L}_{t}$ and $\mathcal{L}_{s}$, weighted by a coefficient
$\lambda$:

$\mathcal{L}=\mathcal{L}_{t}+\lambda\cdot\mathcal{L}_{s}$ (2)

##  IV Related work

The current state-of-the-art on AudioCaps [17] is a full transformer
architecture named Audio Captioning Transformer (ACT) [18], pre-trained on
AudioSet like PANN models. The system uses mixup [19] to improve
generalization during training between audio waveform and spectrograms and
concatenate the corresponding captions. For inference, Gaussian noise and
SpecAugment [20] are used to produce several variants of the same sample and
are given to the model. The intermediate representations of the same example
are averaged to produce a better sentence.

In [21], the authors proposed to use a pre-trained transformer decoder named
BART [22] to generate better sentences. Their first encoder (YAMNet) predicts
the names of the predicted AudioSet classes to improve the audio
representation. These classes names are the inputs of the BART embedding layer
and are added to the PANN encoder audio embeddings. The decoder is the
pretrained BART transformer decoder part. We named their model BYP in the
table for shortening BART+YAMNet+PANNs.

An approach similar to ours has been proposed in [5]. There are notable
differences (the audio encoder, optimizer and hyperparameters for optimization
and generation) which provide a stronger baseline. Unlike them, we train our
model with only one phase (CE+SER losses) instead of two (CE then CE+SER
losses).

TABLE I: AAC results on AudioCaps testing subset with captioning metrics. The arrow $\uparrow$ indicates that a higher value in the column is better, while $\downarrow$ indicates that a lower value is better. Optim. and Wd. stand for Optimizer and Weight decay, respectively. Method | Optim. | Wd. | CIDEr-D | SPICE | SPIDEr | FENSE | SBERT | FluErr | #Words | Trainable | Frozen
---|---|---|---|---|---|---|---|---|---|---|---
$\uparrow$ | $\uparrow$ | $\uparrow$ | $\uparrow$ | $\uparrow$ | $\downarrow$ | $\uparrow$ | params | params
Cross-referencing | N/A | N/A | .901 | .217 | .559 | .680 | .682 | .005 | 952.2 | 0 | 0
Multi-TTA [17] | AdamW | $10^{-6}$ | .769 | .181 | .475 | N/A | N/A | N/A | N/A | 108M | 0
BYP [21] | N/A | .753 | .176 | .465 | 408M
CNN10-trans[5] | Adam | N/A | .573 | .158 | .365 | N/A | .545 | N/A | N/A | 14M | 0
+SER cosine loss | .573 | .166 | .370 | .555 | N/A
Our baseline |  |  | .628 | .165 | .397 | .595 | .601 | .034 | 485.8 | 12.4M | 79.7M
+SBERT tokens | AdamW | $10^{-6}$ | .659 | .167 | .413 | .602 | .607 | .034 | 445.0 | 25.7M | 79.7M
+SER loss |  |  | .665 | .170 | .418 | .607 | .614 | .027 | 465.6 | 25.9M | 146.7M
Our baseline |  |  | .712 | .176 | .444 | .619 | .621 | .004 | 387.2 | 12.4M | 79.7M
+SBERT tokens | AdamW | $2$ | .715 | .175 | .445 | .620 | .621 | .002 | 390.2 | 25.7M | 79.7M
+SER loss |  |  | .715 | .172 | .443 | .619 | .620 | .002 | 348.8 | 25.9M | 146.7M

##  V Experimental setup

###  V-A Dataset

We train and evaluate our models on the AudioCaps [23] dataset, which is the
largest known audio-language dataset with human generated captions. The audio
files are 10-second clips from AudioSet [6] and are extracted from YouTube
videos. Since some of the original videos are removed or unavailable, our
version of the dataset contains 46230 over 49838 files in training subset, 464
over 495 in validation subset and 912 over 975 files in testing subset. Each
audio is described by one caption in the training subset and five captions in
the validation and testing subsets.

###  V-B Metrics

We focused only on the captioning metrics and decided to dismiss the
translation metrics (BLEU, ROUGE, METEOR) which are mainly based on n-gram
overlapping. CIDEr-D [24] computes the cosine similarity of the TF-IDF scores
for common n-grams in candidates and references. SPICE [25] computes the
F1-score of the graph edges representing the semantic propositions extracted
from the sentences using a parser and grammar rules. SPIDEr [26] is the
average of CIDEr-D and SPICE and mainly used to rank AAC systems. Since we are
also studying a sentence similarity loss, we decided to add three model-based
metrics from [27]: SBERT, FluErr and FENSE. The SBERT metric correspond to the
cosine similarity of the sentence embedding extracted using a SBERT model.
FluErr is the fluency error rate detected by a model trained to detect common
errors made by captioning systems like incomplete sentence, repeated event,
repeated adverb, missing conjunction and missing verb. The FENSE metric is the
SBERT score for each sentence, unless an error in the FluErr metric is
detected, then it will be divided by 10. Finally, the last metric ”#Words” is
the number of unique words used in the candidates in the whole subset.

###  V-C Hyperparameters

Hyperparameters are crucial for training deep learning systems. We found that
the model can obtain drastically different scores when trained using different
sets of hyperparameters. We optimized our hyperparameters to maximize the
FENSE score on the validation subset of AudioCaps. We train our model for a
total of 100 epochs $K$ and with a batch size of 512 samples on a single GPU.
We used the AdamW optimizer [28] with a weight decay (wd) of $10^{-6}$ in the
first experiments and set to $2$ to limit overfitting in the second setting.
The weight decay is not applied to the bias weights of the network. We also
denote that the network does not converge when using the standard Adam [29]
optimizer with a large wd. The initial learning rate $\text{lr}_{0}$ is set to
$5\cdot 10^{-4}$ at the beginning of the training, and the values of
$\beta_{1}$ and $\beta_{2}$ are respectively set to 0.9 and 0.999. We used
cosine scheduler decay updated at the end of each epoch $k$ with the following
rule:
$\text{lr}_{k}=\frac{1}{2}\big{(}1+\cos(\frac{k\pi}{K})\big{)}\text{lr}_{0}$.
The captions are put in lowercase and all the punctuation characters are
erased. We clipp the gradient by l2-norm to 10 to stabilize training and add
label smoothing set to 0.1 to the CE loss component. To select our best model
among epochs, we used the highest FENSE score instead of using the CE loss on
the validation subset. Using FENSE allows you to choose a later training epoch
than with the loss of validation function, which gives better results.

For the sentence embedding regression method, we used the paraphrase-
TinyBERT-L6-v2 model222https://www.sbert.net/docs/pretrained_models.html,
since it is the one used in the metrics SBERT and FENSE. We also tried larger
models (all-mpnet-base-v2 and all-mpnet-base-v1), but it does not bring any
improvements. The chosen model contains 67M parameters and its weights are
frozen during training. We have set $\beta$ to $1$ in the $\mathcal{L}_{s}$
function. We tried several values for $\lambda$ (1, 10, 100, 1000, and 10000),
and found that 100 is the best parameter for sentence embedding regression.
Higher values decrease performance, while lower values have no impact on
multitasking compared to the baseline.

##  VI Results

[5pt]Loss with wd set to $10^{-6}$. [5pt]Loss with wd set to 2. [5pt]SBERT
with wd set to $10^{-6}$. [5pt]SBERT with wd set to 2.

Figure 2: CE losses and SBERT cosine similarities over epochs on validation.

We reported the scores in table I of our baseline method using a word
tokenizer, the method using the SBERT tokenizer (baseline+SBERT tokens) and
with the SER loss method (baseline+SBERT tokens+SER loss). All of our scores
are averaged over 5 seeds. We also added the other SER method scores named
CNN10-trans, the current state-of-the-art scores (Multi-TTA and BYP) and the
cross-referencing scores. Cross-referencing is performed by excluding one of
the five captions for each audio file and using it as a candidate sentence,
while the other four remain the ground truth references. This process is
repeated five times to compute an average human agreement score, which we call
”cross-references.”

###  VI-A Discussion

Using a small wd value, the SER loss shows a slight improvement in FENSE and
SPIDEr scores respectively increased from 0.595 to 0.607 and from 0.397 to
0.418. Moreover, when we introduce a large wd value value to prevent
overfitting, we can see a large improvement in FENSE and SPIDEr scores from
0.595 to 0.619 and from 0.397 to 0.445 respectively with our baseline.
Nevertheless, even if the SER loss also benefits from the use of a large wd
value, the resulting scores became very close to the new baseline which also
uses this wd value and the regularization effect given by the SER loss seems
no longer significant.

In Fig. 2, we can see that the increase in the validation learning curve is
reduced by the SER loss with a small wd, which means that it limits the
overfitting of our model and explains the gain obtained in the table. The
validation losses and SBERT cosine similarities in the figures show that the
regularization with a large wd works very well on the model.

The increase in the number of trainable parameters between our baseline and
our baseline+SBERT tokens (12.M to 25.7M) comes from the increase in
vocabulary (4724 to 30522) which drastically grows the number of parameters in
the classifier and the input embedding layer in the decoder part. The method
using SBERT tokens and large wd value has become our new best model according
to SPIDEr and FENSE, although it is closely followed by our methods using the
same decay. Our methods are also very close to the current state-of-the-art
method which obtain 0.475 compared to our best SPIDEr scores of 0.445 and
0.444, despite the fact that we have eight and four times fewer trainable
parameters for our baseline and baseline+SBERT tokens, respectively.

###  VI-B Qualitative analysis

Tables II and III show several examples of the sentences generated by our
model over different training procedures. The baseline system using small wd
value seems to try to use more synonyms but fail more often to provide a good
description, like in II. When using large wd value, the system uses even less
words and more generic sentence structures, despite being more accurate. In
this case, we also denote that the number of words used in the testing subset
decreased from an average of 485.8 words to 387.2 in our baselines systems
using small and large wd value, respectively. The reduction is even more
important when we add the SER loss, with only 348.8 words used on average.

TABLE II: Captions for an AudioCaps testing file (id: “ARFFw0e_jig”) using the wd value $10^{-6}$ for different methods. Candidates |  Method |  SPIDEr |  FENSE
---|---|---|---
a person belching |  Our baseline |  .070 |  .208
a person burps loudly |  SBERT tokens |  .135 |  .291
a person burps loudly several times |  SER loss |  .252 |  .398
References
loud burping and screaming
loud burping repeating
a loud distorted belch followed by a series of burping
several distorted belches followed by non-distorted burps
a series of distorted burps followed by non-distorted burps
TABLE III: Captions for an AudioCaps testing file (id: “JZloTOdIY_c”) with different weight decays of our baseline. Candidates |  wd |  SPIDEr |  FENSE
---|---|---|---
a horse neighs and breathes heavily |  $10^{-6}$ |  .284 |  .658
a horse is trotting |  $2$ |  .337 |  .452
References
horses growl and clop hooves
a horse neighs followed by horse trotting and snorting
horses neighing and snorting while trotting on grass
horses neighing then snorting and trotting on a dirt surface
horses neighing and stomping on the ground

##  VII Conclusions

In this work, we studied the addition of a sentence embedding regression loss
component to improve an Automated Audio Captioning system. We searched for the
most optimal configuration for our baseline by optimizing hyperparameters,
conditioning generation and using a stronger pre-trained encoder. We
discovered that the SER loss component seems to limit overfitting for AAC
systems, but it does not bring improvement anymore when combined with a
stronger regularization method like a large weight decay value. We also
noticed that even if the main metrics (SPIDEr, FENSE) are improved by the
regularization methods, they do not take into account the diversity of the
words used. The diversity of words used could be taken into account in future
captioning metrics, or directly by the model during learning like in [30].

## References

  * [1] K. Drossos, S. Adavanne, and T. Virtanen, “Automated audio captioning with recurrent neural networks,” in _2017 IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA)_ , 2017, pp. 374–378.
  * [2] A. Holtzman, J. Buys, L. Du, M. Forbes, and Y. Choi, “The curious case of neural text degeneration,” 2019. [Online]. Available: https://arxiv.org/abs/1904.09751
  * [3] S. J. Rennie, E. Marcheret, Y. Mroueh, J. Ross, and V. Goel, “Self-critical sequence training for image captioning,” 2016. [Online]. Available: https://arxiv.org/abs/1612.00563
  * [4] X. Mei, Q. Huang, X. Liu, G. Chen, J. Wu, Y. Wu, J. Zhao, S. Li, T. Ko, H. L. Tang, X. Shao, M. D. Plumbley, and W. Wang, “An encoder-decoder based audio captioning system with transfer and reinforcement learning,” 2021. [Online]. Available: https://arxiv.org/abs/2108.02752
  * [5] R. Mahfuz, Y. Guo, and E. Visser, “Improving audio captioning using semantic similarity metrics,” 2022. [Online]. Available: https://arxiv.org/abs/2210.16470
  * [6] J. F. Gemmeke, D. P. W. Ellis, D. Freedman, A. Jansen, W. Lawrence, R. C. Moore, M. Plakal, and M. Ritter, “Audio Set: An ontology and human-labeled dataset for audio events,” in _2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. New Orleans, LA: IEEE, Mar. 2017, pp. 776–780. [Online]. Available: http://ieeexplore.ieee.org/document/7952261/
  * [7] Q. Kong, Y. Cao, T. Iqbal, Y. Wang, W. Wang, and M. D. Plumbley, “PANNs: Large-Scale Pretrained Audio Neural Networks for Audio Pattern Recognition,” _arXiv:1912.10211 [cs, eess]_ , Aug. 2020, arXiv: 1912.10211. [Online]. Available: http://arxiv.org/abs/1912.10211
  * [8] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, “Attention is all you need,” 2017. [Online]. Available: https://arxiv.org/abs/1706.03762
  * [9] D. Hendrycks and K. Gimpel, “Gaussian error linear units (gelus),” 2016. [Online]. Available: https://arxiv.org/abs/1606.08415
  * [10] S. Bird, E. Klein, and E. Loper, _Natural language processing with Python: analyzing text with the natural language toolkit_. ” O’Reilly Media, Inc.”, 2009.
  * [11] N. Reimers and I. Gurevych, “Sentence-bert: Sentence embeddings using siamese bert-networks,” 2019. [Online]. Available: https://arxiv.org/abs/1908.10084
  * [12] J. Devlin, M. Chang, K. Lee, and K. Toutanova, “BERT: pre-training of deep bidirectional transformers for language understanding,” _CoRR_ , vol. abs/1810.04805, 2018. [Online]. Available: http://arxiv.org/abs/1810.04805
  * [13] S. R. Bowman, G. Angeli, C. Potts, and C. D. Manning, “A large annotated corpus for learning natural language inference,” in _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_. Lisbon, Portugal: Association for Computational Linguistics, Sep. 2015, pp. 632–642. [Online]. Available: https://aclanthology.org/D15-1075
  * [14] A. Williams, N. Nangia, and S. Bowman, “A broad-coverage challenge corpus for sentence understanding through inference,” in _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_. New Orleans, Louisiana: Association for Computational Linguistics, Jun. 2018, pp. 1112–1122. [Online]. Available: https://aclanthology.org/N18-1101
  * [15] Y. Wu, M. Schuster, Z. Chen, Q. V. Le, M. Norouzi, W. Macherey, M. Krikun, Y. Cao, Q. Gao, K. Macherey, J. Klingner, A. Shah, M. Johnson, X. Liu, L. Kaiser, S. Gouws, Y. Kato, T. Kudo, H. Kazawa, K. Stevens, G. Kurian, N. Patil, W. Wang, C. Young, J. Smith, J. Riesa, A. Rudnick, O. Vinyals, G. Corrado, M. Hughes, and J. Dean, “Google’s neural machine translation system: Bridging the gap between human and machine translation,” 2016. [Online]. Available: https://arxiv.org/abs/1609.08144
  * [16] R. B. Girshick, “Fast R-CNN,” _CoRR_ , vol. abs/1504.08083, 2015. [Online]. Available: http://arxiv.org/abs/1504.08083
  * [17] E. Kim, J. Kim, Y. Oh, K. Kim, M. Park, J. Sim, J. Lee, and K. Lee, “Improving audio-language learning with mixgen and multi-level test-time augmentation,” 2022\. [Online]. Available: https://arxiv.org/abs/2210.17143
  * [18] X. Mei, X. Liu, Q. Huang, M. D. Plumbley, and W. Wang, “Audio captioning transformer,” 2021. [Online]. Available: https://arxiv.org/abs/2107.09817
  * [19] H. Zhang, M. Cisse, Y. N. Dauphin, and D. Lopez-Paz, “mixup: Beyond empirical risk minimization,” 2018.
  * [20] D. S. Park, W. Chan, Y. Zhang, C.-C. Chiu, B. Zoph, E. D. Cubuk, and Q. V. Le, “SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition,” in _Proc. Interspeech 2019_ , 2019, pp. 2613–2617.
  * [21] F. Gontier, R. Serizel, and C. Cerisara, “Automated audio captioning by fine-tuning bart with audioset tags,” in _DCASE 2021 - 6th Workshop on Detection and Classification of Acoustic Scenes and Events_ , Virtual, Spain, Nov. 2021. [Online]. Available: https://hal.inria.fr/hal-03522488
  * [22] M. Lewis, Y. Liu, N. Goyal, M. Ghazvininejad, A. Mohamed, O. Levy, V. Stoyanov, and L. Zettlemoyer, “Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension,” 2019. [Online]. Available: https://arxiv.org/abs/1910.13461
  * [23] C. D. Kim, B. Kim, H. Lee, and G. Kim, “AudioCaps: Generating captions for audios in the wild,” in _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_. Minneapolis, Minnesota: Association for Computational Linguistics, Jun. 2019, pp. 119–132. [Online]. Available: https://aclanthology.org/N19-1011
  * [24] R. Vedantam, C. L. Zitnick, and D. Parikh, “CIDEr: Consensus-based Image Description Evaluation,” _arXiv:1411.5726 [cs]_ , Jun. 2015, arXiv: 1411.5726. [Online]. Available: http://arxiv.org/abs/1411.5726
  * [25] P. Anderson, B. Fernando, M. Johnson, and S. Gould, “SPICE: Semantic Propositional Image Caption Evaluation,” _arXiv:1607.08822 [cs]_ , Jul. 2016, arXiv: 1607.08822. [Online]. Available: http://arxiv.org/abs/1607.08822
  * [26] S. Liu, Z. Zhu, N. Ye, S. Guadarrama, and K. Murphy, “Improved Image Captioning via Policy Gradient optimization of SPIDEr,” _2017 IEEE International Conference on Computer Vision (ICCV)_ , pp. 873–881, Oct. 2017, arXiv: 1612.00370. [Online]. Available: http://arxiv.org/abs/1612.00370
  * [27] Z. Zhou, Z. Zhang, X. Xu, Z. Xie, M. Wu, and K. Q. Zhu, “Can Audio Captions Be Evaluated with Image Caption Metrics?” Jan. 2022, number: arXiv:2110.04684 arXiv:2110.04684 [cs, eess]. [Online]. Available: http://arxiv.org/abs/2110.04684
  * [28] I. Loshchilov and F. Hutter, “Decoupled weight decay regularization,” 2017. [Online]. Available: https://arxiv.org/abs/1711.05101
  * [29] D. P. Kingma and J. Ba, “Adam: A Method for Stochastic Optimization,” _arXiv:1412.6980 [cs]_ , Jan. 2017, arXiv: 1412.6980. [Online]. Available: http://arxiv.org/abs/1412.6980
  * [30] S. Welleck, I. Kulikov, S. Roller, E. Dinan, K. Cho, and J. Weston, “Neural text generation with unlikelihood training,” 2019. [Online]. Available: https://arxiv.org/abs/1908.04319


Eungbeom Kim1⋆, Jinhee Kim2⋆, Yoori Oh2, Kyungsu Kim2,
---
Minju Park2, Jaeheon Sim1, Jinwoo Lee2, Kyogu Lee1,2,3
††thanks: ⋆ Equal contribution.

# Exploring Train and Test-Time Augmentations for Audio-Language Learning

###### Abstract

In this paper, we aim to unveil the impact of data augmentation in audio-
language multi-modal learning, which has not been explored despite its
importance. We explore various augmentation methods at not only train-time but
also test-time and find out that proper data augmentation can lead to
substantial improvements. Specifically, applying our proposed audio-language
paired augmentation PairMix, which is the first multi-modal audio-language
augmentation method, outperforms the baselines for both automated audio
captioning and audio-text retrieval tasks. To fully take advantage of data
augmentation, we also present multi-level test-time augmentation (Multi-TTA)
for the test-time. We successfully incorporate the two proposed methods and
uni-modal augmentations and achieve 47.5 SPIDEr on audio captioning, which is
an 18.2% relative increase over the baseline. In audio-text retrieval, the
proposed methods also show an improvement in performance as well.

Index Terms: Data augmentation, Audio-language learning, Automated audio
captioning, Audio-text retrieval

##  1 Introduction

Data augmentation has shown meaningful performance gain in various domains.
Multi-modal learning is one of the areas where data augmentation is
prospective and various augmentation strategies are applicable. We can apply
different augmentations on each modality such as audio augmentations [1, 2]
and text augmentations [3, 4]. We can also augment multi-modalities jointly.
MixGen [5], for instance, is a simple way to generate image-text pairs by
applying mixup [6] on images and concatenating each corresponding text.

However, data augmentation on audio-language learning remains poorly
understood. Even worse, AudioCaps [7], the largest public audio-language
dataset, only includes about 40K audio clips and 46K captions in opposition to
the MS COCO Captions [8] which is a widely known vision-language dataset
containing 330K images and 1.5M captions. For this reason, we delve into data
augmentation on audio-language learning in this study.

We first argue that paired multi-modal data augmentation takes an important
role in audio-language learning. Therefore, as a train-time augmentation, we
propose PairMix which is a simple and effective audio-text paired
augmentation. PairMix generates audio-text pairs by randomly selecting either
waveform-level or mel spectrogram-level mixup with a certain probability for
each generated sample and concatenating their corresponding text captions, as
illustrated in Figure 1. To support our argument, we first analyze the uni-
modal augmentations for audio-language learning and the proposed multi-modal
augmentation method, PairMix. As a result, we observe the importance of multi-
modal augmentation in modeling the audio-language relationship. PairMix not
only outperforms other data augmentation methods but also helps uni-modal data
augmentation when uni-modal augmentation and PairMix are applied at the same
time. Since the audio modality has different design choices compared to the
image, we also compared PairMix with other audio augmentation design choices.
PairMix surpasses other designs for multi-modal augmentation such as waveform-
level mixup, mel spectrogram-level mixup, and audio concatenation.

Test-time augmentation (TTA) is another data augmentation approach that is
shown to increase performances across diverse tasks [9, 10]. However, TTA is
disregarded in most audio-language learning studies. In this paper, we
investigate the effect of TTA on audio-language learning. The problem with TTA
is that it has bounded performance, and fail to scale up with the size of
augmentation. To address this issue, we approach TTA from a new perspective,
the place of aggregation. We present a novel multi-level test-time
augmentation (Multi-TTA) methodology, which is a generalization of TTA over
multiple layers. Multi-TTA successfully outperforms the baselines and we
discover that there is still room for improvement using augmentations at test-
time.

Figure 1: Illustration of PairMix. PairMix mixup audio clips and concatenates
the corresponding text. Audio mixup can be implemented at waveform-level or
mel spectrogram-level. Figure 2: Frameworks for test-time augmentation (TTA).
With the augmented inputs at test-time, (a) Conventional TTA averages the
final predictions, (b) Mid-TTA averages the intermediate representations, and
(c) Multi-TTA averages the outputs of multiple layers at different levels.

We evaluate the proposed methods across automated audio captioning (AAC) and
audio-text retrieval on AudioCaps dataset. In AAC, PairMix with audio
augmentations achieves 46.6 SPIDEr, which is a 15.9% over the baseline without
modifying its architecture, and outperforms the state-of-the-art method with a
5x smaller model. Applying PairMix and Multi-TTA results in an extra
improvement and attains 47.5 SPIDEr. PairMix and Multi-TTA are also effective
in audio-text retrieval. Compared to the baseline, recall at rank 10 (R@10)
increases from 82.7% to 87.2% for audio-to-text retrieval, and from 81.3% to
83.2% for text-to-audio retrieval.

##  2 PairMix

In this section, we introduce PairMix, the first audio-text paired
augmentation, which is a simple but effective augmentation method that
generates audio-text pairs. Previous vision-language augmentation [5] simply
mixes randomly selected raw images and concatenates their corresponding texts.
We adapt this approach to the audio-language domain to generate audio-text
pairs while preserving the multi-modal relationship.

PairMix first randomly selects $N$ audio-text pairs
$\\{(a_{i},t_{i})\\}_{i=1}^{N}$ containing audio $a_{i}$ and text $t_{i}$ and
then mixes each modality separately to generate the new audio-text pair.
Unlike vision-language MixGen which simply implements mixup for images,
PairMix leverages the characteristic of audio modality by probabilistically
implementing either waveform-level or mel spectrogram-level audio mixup to
generate new mel spectrogram $\hat{s}$. Finally, PairMix generates a new
audio-text pair $(\hat{s},\hat{t})$ by concatenating text. It is formally
written as:

$\begin{split}&\hat{s}_{w}=M(\sum^{N}_{i=1}\lambda_{i}a_{i})\\\
&\hat{s}_{m}=\sum^{N}_{i=1}\lambda_{i}M(a_{i})\\\
&\hat{s}=\gamma\hat{s}_{w}+(1-\gamma)\hat{s}_{m}\\\
&\hat{t}=\text{Concat}(\\{t_{i}\\}_{i=1}^{N}),\end{split}$ (1)

where $\lambda_{i}\in[0,1]$ for $i=1,...,N$ is a hyperparameter such that
$\sum_{i=1}^{N}\lambda_{i}=1$, $M(\cdot)$ is a mel spectrogram transformation
from waveform, $\gamma$ is randomly sampled from Bernoulli distribution with
probability 0.5, and Concat$(\cdot)$ is a concatenation operator which
concatenates text inputs. $\hat{s}_{w}$ and $\hat{s}_{w}$ denote mel
spectrogram which is generated from waveform-level mixup and mel spectrogram-
level mixup, respectively. Probabilistically applying two types of mixup
allows us to generate more diverse samples, and we observe that this
randomness contributes to stable improvements over hyperparameters as shown in
Figure 4. Since PairMix can be easily applied with any other uni-modal
augmentations, it is possible to drastically scale up the size of the dataset
with a simple variant.

##  3 Multi-level Test-time Augmentation

Test-time augmentation (TTA) contributes to generalizing models by making
multiple predictions from augmented inputs and averaging the predictions. TTA
has no additional training cost because TTA is only implemented at test-time
as opposed to traditional augmentations which mainly focus on train-time. In
this paper, we discuss the effects of conventional TTA for audio-language
learning and present the generalized TTA method. As shown in Figure 2,
conventional TTA only averages outputs from augmented inputs. Although this
can contribute to performance, conventional TTA has limited performance
improvement bound as shown in Figure 4. To fully leverage TTA, we approach
this problem with the view of the place of aggregation, unlike previous
research [11, 12] that tends to focus on a selection of augmentation types and
a method of aggregation. We present middle-level TTA (Mid-TTA) that averages
intermediate representations and outputs a single prediction. Finally, we
propose a novel multi-level TTA (Multi-TTA) methodology by generalizing
conventional TTA and Mid-TTA. Unlike a single layer-based conventional TTA,
Multi-TTA selects multiple layers for TTA so that augmentations are aggregated
at various layers as shown in Figure 2 (c).

We define a Multi-TTA strategy $S=(\\{P_{h}\\}_{h=1}^{H},\tau)$ given a model
$f$ containing $H$ layers where $\tau$ is the total number of input
augmentations as follows. Let $P_{h}$ be a partition of
$\\{1,...,|P_{h-1}|\\}$. Each element of $P_{h}$ contains indices of the
previous layer's outputs that should be aggregated together. We define
$P_{h}[i]$ as an $i$-th element of $P_{h}$ for simplicity. For clarity,
$|P_{h}|$ decides the number of $h$-th layer's outputs, and $|P_{h}[i]|$
decides the number of required $h$-th layer's inputs for an $i$-th output of
$h$-th layer. A strategy $S$ satisfies $|P_{H}|=1$ to output a single
prediction and $\tau=\Pi_{h=1}^{H}|P_{h}|$. We define an $i$-th output of a
$h$-th layer $o_{h,i}$ for $i=1,...,|P_{h}|$ as follows:

$o_{h,i}=\dfrac{1}{|P_{h}[i]|}\sum_{j\in P_{h}[i]}f_{h}(o_{h-1,j}).$ (2)

From the notations defined above, conventional TTA becomes a special case of
Multi-TTA when $|P_{h}|=\tau$ for $1\leq h\leq H-1$ and $|P_{H}|=1$. Mid-TTA
is also a special case of Multi-TTA when $|P_{h}|=\tau$ for $1\leq h\leq
h^{\prime}$ and $|P_{h}|=1$ for $h>h^{\prime}$.

Table 1: Evaluation of the model performance in automated audio captioning. Method | BLEU1 | BLEU2 | BLEU3 | BLEU4 | METEOR | ROUGEL | CIDEr | SPICE | SPIDEr
---|---|---|---|---|---|---|---|---|---
Baseline [13] | 65.4 | 47.5 | 33.6 | 23.4 | 22.4 | 47.1 | 63.5 | 16.8 | 40.2
\+ PairMix | 69.3 | 52.9 | 38.9 | 28.3 | 24.1 | 49.9 | 75.5 | 17.7 | 46.6
\+ Multi-TTA | 70.0 | 53.4 | 39.5 | 28.9 | 24.2 | 50.2 | 76.9 | 18.1 | 47.5
Table 2: Evaluation of the model performance in audio-text retrieval. Method | text-to-audio | audio-to-text
---|---|---
R@1 | R@5 | R@10 | R@50 | meanR | R@1 | R@5 | R@10 | R@50 | meanR
Baseline [14] | 33.0 | 67.9 | 81.3 | 96.5 | 10.0 | 37.9 | 71.0 | 82.7 | 97.4 | 8.8
\+ PairMix | 34.4 | 69.6 | 83.1 | 97.5 | 8.2 | 40.0 | 73.1 | 86.8 | 97.3 | 6.6
\+ Multi-TTA | 34.7 | 70.3 | 83.2 | 97.5 | 8.0 | 40.2 | 74.0 | 87.2 | 97.6 | 6.3
Table 3: Comparison between uni-modal and multi-modal data augmentation. In modality, A means Audio augmentations and T means text augmentations. Augmentation All denotes using every audio augmentation with PairMix. Augmentation | Modality | SPIDEr |  T$\rightarrow$ A |  A$\rightarrow$ T
---|---|---|---|---
R@10 | R@10
- | - | 41.6 | 78.9 | 81.9
Noise | A | 40.9 | 79.1 | 81.1
Reverb | A | 41.1 | 80.4 | 81.0
SpecAugment [1] | A | 40.2 | 81.3 | 82.7
EDA [3] | T | - | 79.3 | 81.3
PairMix | AT | 43.0 | 82.2 | 84.9
All | AT | 46.6 | 83.1 | 86.8

##  4 Experiments

###  4.1 Dataset

AudioCaps is the largest audio-text paired dataset containing approximately
46K audio clips of 10 seconds extracted from AudioSet [15] and their
corresponding text descriptions. We investigate the capability of our proposed
methods on AudioCaps for automated audio captioning and audio-text retrieval.
We follow the train, validation, and test splits stated in [7] and reproduce
the baselines without unavailable audio clips.

###  4.2 Audio-Language Learning

####  4.2.1 Automated Audio Captioning

Audio Captioning Transformer (ACT) [13] has an encoder-decoder structure from
Transformer [16]. The encoder block of ACT is initialized with DeiT [17] which
was trained for image classification tasks and then pre-trained on an audio
tagging task using AudioSet. We utilize the pre-trained ACT
encoder111https://github.com/XinhaoMei/ACT and randomly initialize the ACT-m
decoder for a baseline of AAC, resulting in 108M parameters. For evaluation,
we employ BLEU [18], METEOR [19], and ROUGE-L [20] from machine translation
metrics. We also report CIDEr [21], SPICE [22], and SPIDEr [23]. Captions are
decoded with beam search up to size $3$.

Table 4: Experiment with different augmentation design choices for audio mixing in multi-modal augmentation. Design | SPIDEr |  T$\rightarrow$ A |  A$\rightarrow$ T
---|---|---|---
R@10 | R@10
concat | 45.0 | 81.9 | 80.8
waveform-only | 43.7 | 80.5 | 84.1
spectrogram-only | 44.6 | 82.3 | 83.3
PairMix | 45.8 | 82.9 | 84.7

####  4.2.2 Audio-Text Retrieval

Pre-trained encoders are employed and trained using contrastive loss for
audio-text retrieval, following the previous study [14]. We choose PANNs [24]
as the audio encoder, which has ResNet38 architecture trained on the audio
tagging task using AudioSet. For the text encoder, we initialize our model
with pre-trained BERT [25] and extract a text representation using the [CLS]
token. The final model has 187M trainable parameters. Retrieval models are
evaluated by Recall at rank k (R@k) and mean rank (meanR).

###  4.3 Experimental Setup

We experiment PairMix with different ratios $K\in\\{0.125,0.25,0.5,0.6\\}$
where $K$ is a ratio of #(generated samples) in a mini-batch to a size of
mini-batch. Also, we compare a fixed mixup ratio $\lambda=0.5$ to
$\lambda\sim$ Beta$(0.1,0.1)$. As models with $K=0.25$ and $\lambda\sim$
Beta$(0.1,0.1)$ show high and stable performance, we use this setting for
later experiments.

Additionally, we explore other design choices such as audio concatenation,
waveform-level-only mixup, and mel spectrogram-level-only mixup for mixing
audio in PairMix. We mix two audio-text pairs and load additional audio-text
pairs only for PairMix to avoid duplication within a mini-batch. To validate
the effectiveness and compatibility of PairMix, we compared PairMix with other
uni-modal augmentations. Gaussian noise and reverberation are employed as
waveform-level audio augmentation, with a probability of 0.5. SpecAugment [1],
which is already applied in the baseline model, was applied as a spectrogram-
level audio augmentation. For text augmentation, easy data augmentation (EDA)
[3] was used. Note that text augmentation is not applied in audio captioning
as it can harm the ground-truth captions.

We validate Multi-TTA strategies where TTA operation is implemented at an
intermediate layer $h_{E}$ and an output layer $H$. We set $h_{E}$ as the last
layer of the encoder for AAC and the last layer of the PANN audio encoder for
audio-text retrieval. We compare the results of conventional TTA, Mid-TTA, and
Multi-TTA with $\tau\in\\{10,25,50,100\\}$. For Multi-TTA, we employ the
strategy $S$ satisfying
$(\tau,|P_{h_{E}}[i],|P_{H}[j]|)\in\\{(10,2,5),(25,5,5),(50,5,10),\\\
(100,5,20)\\}$ for $\forall i,j$. As we already mentioned, $|P_{l}[k]|$
denotes the number of required $l$-th layer's inputs for a single $k$-th
output of layer $l$. Note that we set $|P_{l}[k]|$ for $\forall k$ as
identical value for simplicity of experiments and $S$ satisfies
$\tau=|P_{h_{E}}[i]*|P_{H}[j]|$. We stabilize a prediction by averaging the
output at which no augmentation is applied and the output of TTA. Types of
data augmentation on test-time are consistent with train-time. Multi-modal
augmentations are not included because generated samples are needless for
evaluation. In test-time, we halve the maximum width and height of SpecAugment
masking, reverberation degree, and the probability for applying audio
augmentations compared to train-time.

All models are trained for 30 epochs using AdamW optimizer of learning rate
10-4 with the weight decay of 10-6. SpecAugment is applied to every model.
Other hyperparameters follow [13] for AAC and [14] for audio-text retrieval.
Single NVIDIA GeForce RTX 3090 was used to train every model, taking 10 hours
and 9 hours to train the AAC and audio-text retrieval models, respectively.

Figure 3: Examples of generated captions in automated audio captioning.
PairMix correctly captures the overlapping events in a single audio clip,
unlike the others.

##  5 Results

Table 1 and 2 show an overview of the result. In AAC, using PairMix with the
uni-modal audio augmentations achieves 46.6 SPIDEr, which outperforms the
previous state-of-the-art performance with a 5x smaller model. In audio-text
retrieval, applying PairMix increases R@10 to 86.8% for audio-to-text
retrieval and 83.1% for text-to-audio retrieval. Furthermore, attaching Multi-
TTA to AAC achieves 47.5 SPIDEr, which is an +18.2% relative to the baseline.
Adding Multi-TTA gives additional gain to audio retrieval as well, R@10 of
audio-to-text retrieval and text-to-audio retrieval is 87.2% and 83.2%,
respectively.

###  5.1 PairMix

In Table 3, we compare the proposed PairMix to existing uni-modal
augmentations. As shown in the table, PairMix performs better than all the
other uni-modal augmentations in both AAC and audio-text retrieval. The
superiority of our PairMix demonstrates that paired audio-text data
augmentation is critical to audio-language learning. Furthermore, applying
PairMix with uni-modal augmentations leads to additional improvement.

Also, we explore other design choices of PairMix. Concat PairMix concatenates
audio clips instead of mixing, and waveform PairMix and spectrogram PairMix
only augments at waveform- and mel spectrogram-level, respectively. Table 4
demonstrates that our PairMix with probabilistic waveform- and spectrogram-
level audio mixup is the best strategy. We investigate the generated captions
of the baseline, PairMix, and concat PairMix to illustrate the difference. In
failure cases, the baseline and concat PairMix miss or generate inaccurate
explanations as shown in Figure 3. However, PairMix generates robust captions
with samples containing overlapped events.

Figure 4: Experiment on TTA with various $\tau$ in automated audio captioning.
Multi-TTA strategies are describes in the form of
$|P_{h_{E}}[i]|\times|P_{H}[j]|$. Overall, Multi-TTA surpasses conventional
TTA.

###  5.2 Multi-Level Test-Time Augmentation

Figure 4 demonstrates the relationship between the number of input
augmentations and the output performance. Only Multi-TTA enhances as the
number of input augmentations $\tau$ increases, even though they seem to be
similar with a small number of augmentations. This indicates that aggregating
augmentations in multiple layers enhances the bounded performance of
conventional single-layer TTA, allowing the model to fully use the benefit of
augmented samples. As a result, applying Multi-TTA to model with $\tau=100$,
$|P_{h_{E}}[i]|=5$, and $|P_{H}[j]|=20$ successfully improves every evaluation
metric in both AAC and audio-text retrieval.

There could be other hyperparameters to change like the selection of middle
layers. Especially, unlike AAC where we can easily select intermediate layers
to apply TTA as it has an encoder-decoder structure, audio-text retrieval
models have more freedom to select any layers inside the encoder. Future
research on automated layer search could further improve current results.

##  6 Conclusions

We demonstrate that appropriate augmentations improve audio-language learning
performance. Specifically, experimental results show that PairMix simply
generates new audio-text pairs effectively. Also, we generalize TTA and
propose Multi-TTA which enhances the efficiency of augmentations. Our proposed
methods surpass the state-of-the-art method in AAC and show competitive
results in audio-text retrieval. Since PairMix and Multi-TTA are flexible,
they can be simply incorporated into other architectures.

## References

  * [1] D. S. Park, W. Chan, Y. Zhang, C.-C. Chiu, B. Zoph, E. D. Cubuk, and Q. V. Le, ``SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition,'' in _Proc. Interspeech 2019_ , 2019, pp. 2613–2617.
  * [2] H. Wang, Y. Zou, and W. Wang, ``SpecAugment++: A Hidden Space Data Augmentation Method for Acoustic Scene Classification,'' in _Proc. Interspeech 2021_ , 2021, pp. 551–555.
  * [3] J. Wei and K. Zou, ``Eda: Easy data augmentation techniques for boosting performance on text classification tasks,'' in _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , 2019, pp. 6382–6388.
  * [4] S. Edunov, M. Ott, M. Auli, and D. Grangier, ``Understanding back-translation at scale,'' in _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , 2018, pp. 489–500.
  * [5] X. Hao, Y. Zhu, S. Appalaraju, A. Zhang, W. Zhang, B. Li, and M. Li, ``Mixgen: A new multi-modal data augmentation,'' in _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_ , 2023, pp. 379–389.
  * [6] H. Zhang, M. Cisse, Y. N. Dauphin, and D. Lopez-Paz, ``mixup: Beyond empirical risk minimization,'' in _International Conference on Learning Representations_ , 2018. [Online]. Available: https://openreview.net/forum?id=r1Ddp1-Rb
  * [7] C. D. Kim, B. Kim, H. Lee, and G. Kim, ``Audiocaps: Generating captions for audios in the wild,'' in _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , 2019, pp. 119–132.
  * [8] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick, ``Microsoft coco: Common objects in context,'' in _Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13_. Springer, 2014, pp. 740–755.
  * [9] N. Moshkov, B. Mathe, A. Kertesz-Farkas, R. Hollandi, and P. Horvath, ``Test-time augmentation for deep learning-based cell segmentation on microscopy images,'' _Scientific reports_ , vol. 10, no. 1, pp. 1–7, 2020\.
  * [10] A. Lyzhov, Y. Molchanova, A. Ashukha, D. Molchanov, and D. Vetrov, ``Greedy policy search: A simple baseline for learnable test-time augmentation,'' in _Conference on Uncertainty in Artificial Intelligence_. PMLR, 2020, pp. 1308–1317.
  * [11] I. Kim, Y. Kim, and S. Kim, ``Learning loss for test-time augmentation,'' _Advances in Neural Information Processing Systems_ , vol. 33, pp. 4163–4174, 2020.
  * [12] D. Shanmugam, D. Blalock, G. Balakrishnan, and J. Guttag, ``Better aggregation in test-time augmentation,'' in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2021, pp. 1214–1223.
  * [13] X. Mei, X. Liu, Q. Huang, M. D. Plumbley, and W. Wang, ``Audio captioning transformer,'' in _Detection and Classification of Acoustic Scenes and Events-DCASE 2021_ , 2021.
  * [14] X. Mei, X. Liu, J. Sun, M. Plumbley, and W. Wang, ``On Metric Learning for Audio-Text Cross-Modal Retrieval,'' in _Proc. Interspeech 2022_ , 2022, pp. 4142–4146.
  * [15] J. F. Gemmeke, D. P. Ellis, D. Freedman, A. Jansen, W. Lawrence, R. C. Moore, M. Plakal, and M. Ritter, ``Audio set: An ontology and human-labeled dataset for audio events,'' in _2017 IEEE international conference on acoustics, speech and signal processing (ICASSP)_. IEEE, 2017, pp. 776–780.
  * [16] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, ``Attention is all you need,'' _Advances in neural information processing systems_ , vol. 30, 2017.
  * [17] H. Touvron, M. Cord, M. Douze, F. Massa, A. Sablayrolles, and H. Jégou, ``Training data-efficient image transformers & distillation through attention,'' in _International Conference on Machine Learning_. PMLR, 2021, pp. 10 347–10 357.
  * [18] K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu, ``Bleu: a method for automatic evaluation of machine translation,'' in _Proceedings of the 40th annual meeting of the Association for Computational Linguistics_ , 2002, pp. 311–318.
  * [19] S. Banerjee and A. Lavie, ``Meteor: An automatic metric for mt evaluation with improved correlation with human judgments,'' in _Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization_ , 2005, pp. 65–72.
  * [20] C.-Y. Lin, ``Rouge: A package for automatic evaluation of summaries,'' in _Text summarization branches out_ , 2004, pp. 74–81.
  * [21] R. Vedantam, C. Lawrence Zitnick, and D. Parikh, ``Cider: Consensus-based image description evaluation,'' in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2015, pp. 4566–4575.
  * [22] P. Anderson, B. Fernando, M. Johnson, and S. Gould, ``Spice: Semantic propositional image caption evaluation,'' in _European conference on computer vision_. Springer, 2016, pp. 382–398.
  * [23] S. Liu, Z. Zhu, N. Ye, S. Guadarrama, and K. Murphy, ``Improved image captioning via policy gradient optimization of spider,'' in _Proceedings of the IEEE international conference on computer vision_ , 2017, pp. 873–881.
  * [24] Q. Kong, Y. Cao, T. Iqbal, Y. Wang, W. Wang, and M. D. Plumbley, ``Panns: Large-scale pretrained audio neural networks for audio pattern recognition,'' _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , vol. 28, pp. 2880–2894, 2020.
  * [25] J. D. M.-W. C. Kenton and L. K. Toutanova, ``Bert: Pre-training of deep bidirectional transformers for language understanding,'' in _Proceedings of NAACL-HLT_ , 2019, pp. 4171–4186.


#  Dissociative production of vector mesons at electron-ion colliders

D. Bendova  Faculty of Nuclear Sciences and Physical Engineering, Czech
Technical University in Prague, Czech Republic  J. Cepila  Faculty of Nuclear
Sciences and Physical Engineering, Czech Technical University in Prague, Czech
Republic  J. G. Contreras  Faculty of Nuclear Sciences and Physical
Engineering, Czech Technical University in Prague, Czech Republic

###### Abstract

We present predictions for the exclusive and dissociative production of vector
mesons off protons in an electron-ion collider. The computation is based on
the energy-dependent hot spot model that was shown to successfully describe
the available photoproduction data. We find that the model also describes
correctly all available electroproduction data. In addition, we find that the
cross section for dissociative production as a function of the center-of-mass
energy of the photon-proton system has a maximum, whose position depends on
the virtuality of the photon and the mass of the vector meson. We use these
maxima to define a geometrical saturation scale and find that it grows
linearly with energy as a function of the scale of the process. This
phenomenon can be studied at the proposed electron-ion colliders, JLEIC, eRHIC
and LHeC.

###### pacs:

12.38.-t,13.60.Le

##  I Introduction

Within perturbative Quantum Chromodynamics (pQCD), the structure of hadrons in
terms of its constituent partons evolves with energy, or equivalently with
Bjorken-$x$. Very precise measurements of the $F_{2}(x,Q^{2})$ structure
function of the proton performed at HERA with photons of virtuality $Q^{2}$
indicate that the gluon density grows steeply for decreasing $x$ Abramowicz
_et al._ (2015). According to pQCD this behavior changes at some point where
non-linear effects start to be important and the proton structure enters a
regime known as saturation; see for example Gelis  _et al._ (2010) and
references therein.

Exclusive vector meson production in electron-hadron colliders, depicted in
Fig. 1 (a), has been advocated as a tool to study the saturation phenomenon in
the facilities that are under design now, like the EIC or the LHeC Accardi
_et al._ (2016); Abelleira Fernandez  _et al._ (2012). In this process, the
incoming electron emits a photon which interacts with the proton to produce a
vector meson. The photon can be quasi-real ($\gamma$) or have a large
virtuality ($\gamma^{*}$); these cases are known as photo- or
electroproduction, respectively. Here, $W_{\gamma\mathrm{p}}$ is the center-
of-mass energy of the photon-proton system and $-t$ is the square of the
momentum transferred in the proton vertex. This process has been extensively
investigated at HERA and at the LHC. (For recent reviews see Newman and Wing
(2014) and Contreras and Tapia Takaki (2015), respectively.) These
measurements have been successfully described by a variety of models including
saturation effects; e.g. Kowalski  _et al._ (2006); Armesto and Rezaeian
(2014); Goncalves  _et al._ (2014). A recent study addresses in detail the
corresponding measurements at future electron-ion colliders Lomnitz and Klein
(2018).

A related process, shown schematically in Fig. 1 (b), that has recently
attracted renewed attention, is the production of a vector meson accompanied
by the dissociation of the scattered proton. In a Good-Walker approach Good
and Walker (1960); Miettinen and Pumplin (1978) this process can be related to
fluctuations of the partonic structure of the proton Mäntysaari and Schenke
(2016a, b). Specifically, it is related to the variance over the different
configurations of the partonic structure, and the main contribution to the
variance is given by fluctuations in the geometrical configurations in the
impact-parameter plane. Using a model with three so-called hot spots — regions
of high gluonic density —, the authors of Mäntysaari and Schenke (2016a)
showed that the measurement of the cross section for the dissociative
photoproduction of $\mathrm{J/}\psi$ as a function of $|t|$, at a fixed
$W_{\gamma\mathrm{p}}$, could be successfully described.

These ideas were extended in Cepila  _et al._ (2017) by the inclusion of an
energy dependence on the number of hot spots, which grows with decreasing $x$,
mimicking the expectations of pQCD. This model successfully describes all
available data on the energy dependence of both exclusive and dissociative
photoproduction of $\mathrm{J/}\psi$ off protons. Furthermore, it predicts
that the dissociative cross section grows with energy up to a maximum value
and then decreases steeply. These investigations were continued in Cepila  _et
al._ (2018a) and Cepila  _et al._ (2018b) to describe the production off
nuclear targets and of different vector mesons, respectively. In Cepila  _et
al._ (2018b), it was observed that the position of the maximum of the
dissociative cross section depends on the mass of the vector meson in
photoproduction processes.

In this article, we apply our model to the case of the dissociative
electroproduction of vector mesons. We find that this cross section has a
maximum, whose position depends on the virtuality of the photon and the mass
of the vector meson. We use these maxima to define a geometrical saturation
scale and find that it grows linearly with energy as a function of the scale
of the process, as reported in Fig 8. The rest of this contribution is
organized as follows. A brief description of the formalism is presented in
Sec. II. The model predictions are presented and compared to the available
data in Sec. III. Section IV introduces the geometrical saturation scale. We
close with a brief summary and outlook in Sec. V.

Figure 1:  Diagrams for exclusive ($a$) and dissociative ($b$) production of
vector mesons in an electron-ion collider. See text for details.

##  II Description of the formalism

###  II.1 The photon-proton scattering amplitude

The diffractive production of a vector meson when a virtual photon interacts
with a proton can be well described within the color dipole picture Mueller
(1994); Mueller and Patel (1994). In this case, the scattering amplitude takes
the following form (for a detailed derivation see e.g. Kowalski  _et al._
(2006)),

$\mathcal{A}_{T,L}(x,Q^{2},\vec{\Delta})=i\int\mathrm{d}\vec{r}\int\limits_{0}^{1}\frac{\mathrm{d}z}{4\pi}\int\mathrm{d}\vec{b}|\Psi_{\rm
V}^{*}\Psi_{\gamma^{*}}|_{T,L}\exp\left[-i\left(\vec{b}-(1-z)\vec{r}\right)\vec{\Delta}\right]\frac{\mathrm{d}\sigma_{q\bar{q}}}{\mathrm{d}\vec{b}},$
(1)

where the subscripts $T$ and $L$ denote the contribution from the
transversally, respectively longitudinally, polarized virtual photon.
$\Psi_{\rm V}$ is the wave function of the vector meson, $\Psi_{\gamma^{*}}$
is the wave function of a virtual photon, which fluctuates into a quark-
antiquark dipole, $\vec{r}$ is the transverse size of the color dipole, $z$ is
the fraction of the photon longitudinal momentum carried by the quark,
$\vec{b}$ is the impact parameter and $\vec{\Delta}^{2}\equiv-t$. The
Bjorken-$x$ of the exchanged pomeron is, under the assumption of large
$W_{\gamma\mathrm{p}}$, given by

$x=\frac{Q^{2}+M^{2}}{W_{\gamma\mathrm{p}}^{2}+Q^{2}},$ (2)

with $M$ being the invariant mass of the given vector meson. Finally,
$\mathrm{d}\sigma_{q\bar{q}}/\mathrm{d}\vec{b}$ is the cross section for the
interaction of the color dipole and the target.

In this formalism, the exclusive cross section to produce the vector meson V
is given by

$\frac{\mathrm{d}\sigma^{\gamma^{*}p\rightarrow{\rm
V}p}}{\mathrm{d}|t|}\bigg{|}_{T,L}=\frac{\left(R_{g}^{T,L}\right)^{2}}{16\pi}|\langle\mathcal{A}_{T,L}\rangle|^{2},$
(3)

while the cross section where the proton dissociates into a system $Y$ is

$\frac{\mathrm{d}\sigma^{\gamma^{*}p\rightarrow{\rm
V}Y}}{\mathrm{d}|t|}\bigg{|}_{T,L}=\frac{\left(R_{g}^{T,L}\right)^{2}}{16\pi}\left(\langle|\mathcal{A}_{T,L}|^{2}\rangle-|\langle\mathcal{A}_{T,L}\rangle|^{2}\right).$
(4)

In both cases, the total cross section is given by the sum of the transverse
and the longitudinal contributions. The factor $R_{g}^{T,L}$ is called the
skewedness correction Shuvaev  _et al._ (1999) and takes into account that
there are two values of $x$ involved in the interaction but only one appears
in Eq. (1).

There are two ingredients of Eq. (1) that need to be modeled: the wave
function to create a vector meson out of the quark-antiquark dipole and the
cross section for the interaction of the color dipole and the target. They are
discussed in the following.

###  II.2 Wave functions of vector meson

The wave functions of vector mesons are modeled assuming that the vector meson
is predominantly a $q\bar{q}$ pair with the same polarization structure as the
photon. The overlap of the photon-meson wave functions in Eq. (1) is given as

$|\Psi_{\rm V}^{*}\Psi_{\gamma^{*}}|_{T}=\hat{e}_{f}e\frac{N_{C}}{\pi
z(1-z)}\left[m_{f}^{2}K_{0}(\epsilon
r)\phi_{T}(r,z)-\left(z^{2}+(1-z)^{2}\right)\epsilon K_{1}(\epsilon
r)\partial_{r}\phi_{T}(r,z)\right],$ (5)

and

$|\Psi_{\rm
V}^{*}\Psi_{\gamma^{*}}|_{L}=\hat{e}_{f}e\frac{N_{C}}{\pi}2Qz(1-z)K_{0}(\epsilon
r)\left[M\phi_{L}(r,z)+\delta\frac{m_{f}^{2}-\nabla_{r}^{2}}{Mz(1-z)}\phi_{L}(r,z)\right],$
(6)

where $r\equiv|\vec{r}|$, $N_{C}$ is the number of colors, the (effective)
mass of the given flavor is $m_{f}$, and an effective charge is denoted by
$\hat{e}_{f}$. The parameter $\delta$ is a switch to include or not the
corresponding term; we set it equal to one, which corresponds to the boosted
Gaussian model Nemchik  _et al._ (1994, 1997); Forshaw  _et al._ (2004).
$K_{i}$ are Bessel functions and

$\epsilon=z(1-z)Q^{2}+m_{f}^{2}.$ (7)

The scalar part $\phi_{T,L}$ of the wave function is in general model-
dependent. In the boosted Gaussian model the scalar part is described by the
Gaussian distribution

$\phi_{T,L}(r,z)=N_{T,L}z(1-z)\exp\left(-\frac{m_{f}^{2}R^{2}}{8z(1-z)}-\frac{2z(1-z)r^{2}}{R^{2}}+\frac{m_{f}^{2}R^{2}}{2}\right).$
(8)

The parameters of the model are fixed using a normalization condition and the
measured electronic decay width (see, e.g. Kowalski  _et al._ (2006)) . For
the first excited state 2S, the scalar wave function has the form

$\phi^{\rm 2S}_{T,L}(r,z)=\Phi_{T,L}(r,z)\left(1+\alpha_{\rm
2S}\left(2+\frac{m_{f}^{2}R^{2}}{4z(1-z)}-\frac{4z(1-z)r^{2}}{R^{2}}-m_{f}^{2}R^{2}\right)\right).$
(9)

The condition that the 1S and 2S states are orthogonal, fixes the extra
parameter $\alpha_{\rm 2S}$.

We have recomputed the values of the parameters for the wave functions of all
vector mesons discussed in the following to match them to the measurements
gathered in the PDG of 2016 Patrignani  _et al._ (2016). The parameter values
are reported in Tab. 1.

Table 1:  Parameters for vector meson (V) wave functions: mass of the vector meson $M$, effective mass of the given flavor $m_{f}$, effective charge $\hat{e}_{f}$, scalar part parameters $N_{T}$, $N_{L}$, $R^{2}$ and $\alpha_{\rm 2S}$, fixed with the values reported in the 2016 PDG Patrignani  _et al._ (2016).  V |  $M\;[\mathrm{GeV}$] | $m_{f}\;[\mathrm{GeV}]$ | $\hat{e}_{f}\;[-]$ | $N_{T}\;[-]$ | $N_{L}\;[-]$ | $R^{2}\;[\mathrm{GeV^{-2}}]$ | $\alpha_{\rm 2S}\;[-]$
---|---|---|---|---|---|---|---
$\rho^{0}$ | 0.775260 | 0.14 | $1/\sqrt{2}$ | 0.909 | 0.853 | 12.75 | –
$\phi$ | 1.019461 | 0.14 | 1/3 | 0.918 | 0.823 | 11.3 | –
$\mathrm{J/}\psi$ | 3.09690 | 1.4 | 2/3 | 0.582 | 0.578 | 2.24 | –
$\psi(2\mathrm{S})$ | 3.686097 | 1.4 | 2/3 | 0.666 | 0.658 | 3.705 | -0.6225
$\Upsilon(1S)$ | 9.46030 | 4.2 | 1/3 | 0.478 | 0.478 | 0.585 | –
$\Upsilon(2S)$ | 10.02326 | 4.2 | 1/3 | 0.614 | 0.610 | 0.831 | -0.568

###  II.3 Dipole-target cross section

The cross section for the interaction between the color dipole with the proton
target is related, via the optical theorem, to the imaginary part of the
dipole-proton amplitude $N(x,\vec{r},\vec{b})$:

$\frac{\mathrm{d}\sigma_{q\bar{q}}}{\mathrm{d}\vec{b}}=2N(x,\vec{r},\vec{b}).$
(10)

In order to separate the effects of fluctuations of the proton structure in
the transverse plane from the energy dependence of the cross section we
proposed in Cepila  _et al._ (2017) to use the factorized form

$\frac{\mathrm{d}\sigma_{q\bar{q}}}{\mathrm{d}\vec{b}}=\sigma_{0}N(x,r)T_{p}(\vec{b}),$
(11)

where $T_{p}(\vec{b})$ decribes the proton profile in the impact-parameter
plane and $\sigma_{0}$ is a normalization parameter, which we fixed to
$\sigma_{0}=4\pi B_{p}$. The interpretation of $B_{p}$ is discussed below.

The dipole amplitude $N(x,r)$ can be obtained from various parameterizations
(for an overview see e.g. Kowalski  _et al._ (2006)) or as the solution of the
Balitsky-Kovchegov evolution equation Balitsky (1996); Kovchegov (1999). To
keep the model as simple as possible, we have chosen the form of the dipole
amplitude $N(x,r)$ given by the Golec-Biernat and Wusthoff model Golec-Biernat
and Wusthoff (1998, 1999),

$N(x,r)=\left[1-\exp\left(-\frac{r^{2}Q_{s}^{2}(x)}{4}\right)\right],$ (12)

where $Q_{s}(x)$ is the so-called saturation scale, which in this model is
given by

$Q_{s}^{2}(x)=Q_{0}^{2}\left(\frac{x_{0}}{x}\right)^{\lambda}.$ (13)

Since the proton is a quantum object, its structure changes from interaction
to interaction. To incorporate this effect we use a model of the proton as
constituted by hot spots ($hs$), which represent regions of high gluon
density. The positions of these hot spots in the transverse plane fluctuate
event-by-event and are described by the proton profile function
$T_{p}(\vec{b})$, which is defined as

$T_{p}(\vec{b})=\frac{1}{N_{hs}}\sum\limits_{i=1}^{N_{hs}}T_{hs}\left(\vec{b}-\vec{b}_{i}\right),$
(14)

where each hot spot is defined as

$T_{hs}(\vec{b}-\vec{b}_{i})=\frac{1}{2\pi
B_{hs}}\exp\left(-\frac{\left(\vec{b}-\vec{b}_{i}\right)^{2}}{2B_{hs}}\right).$
(15)

Each vector $\vec{b_{i}}$ is obtained from a two-dimensional Gaussian
distribution with width $B_{p}$ and centered at (0,0). Thus, the parameters
$B_{p}$ and $B_{hs}$ can be interpreted as half of the average of the squared
radius of the proton and of the hot spot, respectively. In this sense
$\sigma_{0}=4\pi B_{p}$ is a measure of the overall transverse area of the
proton.

The key feature of our model is the evolution of the number of hot spots with
energy. $N_{hs}$ is a random number drawn from a zero-truncated Poisson
distribution, where the Poisson distribution has a mean value

$\langle N_{hs}(x)\rangle=p_{0}x^{p_{1}}(1+p_{2}\sqrt{x}),$ (16)

where $p_{0}$, $p_{1}$ and $p_{2}$ are parameters.

The values of all parameters of our model were fixed in earlier publications
Cepila  _et al._ (2017, 2018a, 2018b) using $\mathrm{J/}\psi$ data from
photoproduction at HERA. The values are listed here for completeness:
$B_{p}=4.7$ GeV-2, $B_{hs}=0.8$ GeV-2, $p_{0}=0.011$, $p_{1}=-0.58$,
$p_{2}=300$, $\lambda=0.21$, $x_{0}=2\times 10^{-4}$ and $Q_{0}=1$ GeV. In
order to describe the normalization of the photoproduction of $\rho$ and of
$\phi$ we set $B_{p}=8$ GeV-2 as done in Cepila  _et al._ (2018b) and
consistent with the observations of the H1 Collaboration Aaron  _et al._
(2010). For the case of electroproduction discussed below we set $B_{p}=4.7$
GeV-2 for all vector mesons.

##  III Predictions and comparison with experimental data

Using the model described above we predict the energy dependence of the
exclusive and dissociative production of vector mesons off a proton target for
$\rho^{0}$, $\phi$, $\mathrm{J/}\psi$, $\psi(2\mathrm{S})$, $\Upsilon(1S)$ and
$\Upsilon(2S)$ at different virtualities of the exchanged photon. We compare
our predictions with data when available. For completeness we also show the
predictions for photoproduction that were presented in Cepila  _et al._
(2018b).

Figure 2:  (Color online) Comparison of the model predictions (solid lines)
with HERA Aaron  _et al._ (2010); Aid  _et al._ (1996); Adloff  _et al._
(1997); Breitweg  _et al._ (1998); Chekanov  _et al._ (2007) and CMS data CMS
(2018) for the $W_{\gamma\mathrm{p}}$ dependence of the exclusive (left) and
dissociative (right) photo- and electroproduction cross section of a
$\rho^{0}$ meson.

Figure 3:  (Color online) Comparison of the model predictions (solid lines)
with HERA data from H1 Aaron  _et al._ (2010); Adloff  _et al._ (1997) and
ZEUS Chekanov  _et al._ (2005) for the $W_{\gamma\mathrm{p}}$ dependence of
the exclusive (left) and dissociative (right) photo- and electroproduction
cross section of a $\phi$ meson.

The predictions for the $W_{\gamma\mathrm{p}}$ dependence of the exclusive and
dissociative cross section of the $\rho^{0}$ vector meson are presented in
Fig. 2. Predictions are compared with H1 Aid  _et al._ (1996); Adloff  _et
al._ (1997); Aaron  _et al._ (2010) and ZEUS data Breitweg  _et al._ (1998);
Chekanov  _et al._ (2007) for several values of $Q^{2}$ and also to the
preliminary CMS data CMS (2018) for photoproduction in p–Pb collisions at the
center-of-mass energy $\sqrt{s}=5.02$ TeV. The predictions for
electroproduction, both exclusive and dissociative, give a very good
description of the available data covering virtualities from 2.4 GeV2 to 35.6
GeV2. Recently, the H1 Collaboration released preliminary data (not shown in
the figure) on the energy dependence of $\rho^{0}$ dissociative
photoproduction. The predictions of our model are consistent with these
preliminary data, although a definitive comparison can only be done after the
measurement is published in its final form.

The predictions for the energy-dependence of the exclusive and dissociative
photo- and electroproduction cross sections of the $\phi$ vector meson are
compared with H1 Adloff  _et al._ (1997); Aaron  _et al._ (2010) and ZEUS data
Chekanov  _et al._ (2005) in Fig. 3. The description of the electroproduction
data is satisfactory, however it is not as good as for the case of the
$\rho^{0}$ meson.

Figure 4:  (Color online) Comparison of the model predictions (solid lines)
with H1 Aktas  _et al._ (2006); Alexa  _et al._ (2013) and ALICE data Abelev
_et al._ (2014); Acharya  _et al._ (2018) for the $W_{\gamma\mathrm{p}}$
dependence of the exclusive (left) and dissociative (right) photo- and
electroproduction cross section of a $\mathrm{J/}\psi$ meson.

Figure 5:  (Color online) Comparison of the model predictions (solid lines)
with H1 Adloff  _et al._ (2000), ZEUS Chekanov  _et al._ (2009), LHCb Aaij
_et al._ (2015) and CMS data Sirunyan  _et al._ (2018) for the
$W_{\gamma\mathrm{p}}$ dependence of the exclusive (left) and dissociative
(right) photo- and electroproduction cross section of a $\Upsilon(1S)$ meson.

The photoproduction of $\mathrm{J/}\psi$ has already been studied in Cepila
_et al._ (2017, 2018b). Here we show the same comparison of H1 Alexa  _et al._
(2013) and ALICE p–Pb data Abelev  _et al._ (2014) with the model predictions
in Fig. 4. Additionally, recent ALICE data Acharya  _et al._ (2018) is
included. These new photoproduction measurements are also correctly described
by the predictions. Electroproduction data from H1 Aktas  _et al._ (2006) is
also shown in the figure. The predictions for the exclusive and dissociative
cross sections show a good agreement with all these data.

The comparison between the predictions for the exclusive and dissociative
photoproduction cross section for the $\Upsilon(1S)$ vector meson and data has
been presented in Cepila  _et al._ (2018b). We present it here again in Fig. 5
to provide a comparison with the electroproduction predictions which are the
main topic of this work. The exclusive photoproduction cross section is
compared with ZEUS Adloff  _et al._ (2000) and H1 Chekanov  _et al._ (2009)
data from HERA. It is also compared with LHCb data taken in proton-proton
collisions at $\sqrt{s}=7$ TeV and $\sqrt{s}=8$ TeV at the LHC Aaij  _et al._
(2015). The last set of data we compare our predictions with was measured by
CMS in p–Pb collisions at $\sqrt{s}=5.02$ TeV Sirunyan  _et al._ (2018). The
data is correctly described by the predictions, although the current
uncertainty of the measurement does not allow us to extract strong conclusions
regarding the agreement between data and the model. Currently, to our
knowledge there is no electroproduction data for the exclusive nor for the
dissociative process. We expect these measurements to be performed at future
electron-ion colliders.

Figure 6:  (Color online) Model predictions (solid lines) for the
$W_{\gamma\mathrm{p}}$ dependence of the exclusive (left) and dissociative
(right) photo- and electroproduction cross section of a $\psi(2\mathrm{S})$
meson.

Figure 7:  (Color online) Model predictions (solid lines) for the
$W_{\gamma\mathrm{p}}$ dependence of the exclusive (left) and dissociative
(right) photo- and electroproduction cross section of a $\Upsilon(2S)$ meson.

To complete the set of our predictions we present predictions for the excited
states $\psi(2\mathrm{S})$ and $\Upsilon(2S)$ in Figs. 6 and 7. Currently,
there is no direct data for these particles, but there is photoproduction data
from H1 Adloff  _et al._ (2002) and electroproduction data at $Q^{2}=16$ GeV2
from ZEUS Abramowicz  _et al._ (2016) for the ratio of the exclusive
production of $\psi(2\mathrm{S})$ to that of $\mathrm{J/}\psi$. Our
predictions describe correctly the measured ratios, although, as in the case
of the $\Upsilon(1S)$, the current uncertainty of the measurement does not
allow us to extract strong conclusions regarding the agreement between data
and the model.

In summary, there is a good agreement between all existing data for the
exclusive and the dissociative photo- and electoproduction of vector mesons
and the predictions of our model.

##  IV Geometric saturation scale

###  IV.1 Introduction of the geometric saturation scale

As already noticed in Cepila  _et al._ (2017) for the case of
$\mathrm{J/}\psi$ photoproduction and confirmed in Cepila  _et al._ (2018b)
for the photoproduction of $\rho^{0}$ and $\Upsilon(1S)$, the behavior of the
dissociative cross section as a function of the photon-proton center-of-mass
energy is quite striking. At low energies, the cross section rises with
$W_{\gamma\mathrm{p}}$ to reach a maximum, after which it decreases steeply.
The same behavior is observed for the dissociative electroproduction of vector
mesons. Interestingly, the position of the maximum depends not only on the
mass of the vector meson, but also on the virtuality of the exchanged photon.

The interpretation of this behavior is given by the form of the cross section
shown in Eq. (4). The dissociative production measures the variance over the
different configurations into which the structure of the proton can fluctuate.
In our model, this is given by the different geometrical placements of the hot
spots in the impact-parameter plane. As the energy $W_{\gamma\mathrm{p}}$
increases, so it does the number of hot spots inside the proton as shown in
Eq. (16). As the hot spots have all the same transversal area, the more hot
spots there are, the more the proton area is filled. At some point, all the
possible configurations start to look alike, because all of them start filling
all the available area in the proton and overlap in a process reminiscent of
percolation Armesto  _et al._ (1996). From this energy onwards the variance
over configurations steeply decreases. The maximum of the dissociative cross
section defines a well defined energy at a well defined scale. We call this
point the geometric saturation scale (GSS) and in the following study some of
its properties.

###  IV.2 Energy dependence of the geometric saturation scale

Figure 8:  (Color online) Left: Position of the maxima of the dissociative
cross sections (markers) and an estimation of the related uncertainty (bars)
as a function of $Q^{2}+M^{2}$. The line is a fit to the line defined in Eq.
(17) and the band represents the one sigma contour. Right: the same data as in
the left panel, but translating $W_{\rm GSS}$ into $x_{\rm GSS}$ and plotting
them in logarithmic variables. The red line is the fit to Eq. (18) The
diagonal lines represent the kinematic reach of some of the proposed future
electron-ion colliders. See text for details.

For each of the vector mesons and for each of the virtualities we determine
the energy $W_{\rm GSS}$ at which the maximum is found. As the predictions are
based on a random process, the value at the maximum may fluctuate a bit, so we
chose a region containing the 1% largest values of the cross section to
determine the position of the maximum along with an estimation of the
associated uncertainty.

Figure 8 shows in the left panel the position of the maximum as a function of
$Q^{2}+M^{2}$, which is a measure of the scale of the process. The behavior
seems to be linear, so we fitted the extracted maxima to the functional form

$W_{\rm GSS}=a_{0}+a_{1}(Q^{2}+M^{2}).$ (17)

For the fit we considered only points with $Q^{2}+M^{2}$ larger than 2 GeV2.
The fit is good. The $\chi^{2}$ per degree-of-freedom is 0.41, the small value
reflecting the large assigned uncertainty on the position of the maxima. The
parameter values are $a_{0}=-21\pm 11$ GeV and the slope that is obtained is
$a_{1}=41.5\pm 1.8$ GeV-1.

Using Eq. (2) we can translate $W_{\rm GSS}$ into $x_{\rm GSS}$. The result is
shown in the right panel of Fig. 8. The behavior is also linear in the
logarithmic variables so we have fit the predictions to

$\ln(1/x_{\rm GSS})=c_{0}+c_{1}\ln((Q^{2}+M^{2})/{\rm GeV}^{2}).$ (18)

We found $c_{0}=7.2\pm 0.2$ and $c_{1}=1.04\pm 0.06$. The same figure shows
the kinematic limit of some of the proposed future electron-ion colliders.
This limit is obtained from

$xys=Q^{2},$ (19)

where the inelasticity of the collision is set to $y=1$ and the center-of-mass
energies $s$ of the accelerators are obtained from the energies of the proton,
$E_{p}$, and electron, $E_{e}$, beams taken from Tab. I of Lomnitz and Klein
(2018): $E_{e}=10$ GeV, $E_{p}=100$ GeV for JLEIC; $E_{e}=18$ GeV, $E_{p}=275$
GeV for eRHIC; $E_{e}=27.5$ GeV, $E_{p}=920$ GeV for HERA; and $E_{e}=60$ GeV,
$E_{p}=7$ TeV for LHeC.

It is interesting to notice that even for the collider with the lower energy,
one could measure this linear behavior using electroproduction of $\rho^{0}$
and of $\phi$ vector mesons at relatively small virtualities, but in all cases
at scales $Q^{2}+M^{2}$ above 1 GeV2. The detectors at the JLEIC and eRHIC are
still under development, but the envisaged capabilities would allow the
measurement of $\rho^{0}$ and $\phi$ as discussed in detail in Lomnitz and
Klein (2018). To investigate the positions of the maxima for $\mathrm{J/}\psi$
one needs the LHC and the LHeC for photo- and electroproduction cases,
respectively. The positions of the maxima for the Upsilon states seems to be
out of reach even for the LHeC.

##  V Summary and outlook

Using the energy-dependent hot spot model we have presented predictions for
the exclusive and dissociative electroproduction of vector mesons off proton
targets. We studied the production of $\rho^{0}$, $\phi$, $\mathrm{J/}\psi$,
$\psi(2\mathrm{S})$, $\Upsilon(1S)$ and $\Upsilon(2S)$ states. We found that
the dissociative cross section as a function of $W_{\gamma\mathrm{p}}$
presents a maximum and have used this maximum to define a geometrical
saturation scale. We found that the energy evolution of this scale is linear
in $Q^{2}+M^{2}$ and that this behavior can be studied at the planned JLEIC,
eRHIC and LHeC electron-ion colliders.

To be able to perform such measurements the detectors would have to be
instrumented in the forward rapidity regions in order to tag the presence of
the products from the dissociative state. Such a technique has been used at
HERA in the past; it is also used nowadays at the LHC to reject the
dissociative events when measuring the exclusive production channel, so it
seems to be feasible if planned in advance.

Mapping the energy evolution of the geometric saturation scale provides an
extra handle to investigate quantitatively the high-energy limit of QCD and to
study the phenomenon of gluon saturation in the proton.

## Acknowledgements

We thank Victor Gonçalves for useful discussions. This work has been partially
supported by the following grants: 17-04505S of the Czech Science Foundation
(GAČR), COST Action CA15213 THOR, LTC17038 of the INTER-EXCELLENCE program at
the Ministry of Education, Youth and Sports of the Czech Republic and the
European Regional Development Fund-Project ”Brookhaven National Laboratory -
participation of the Czech Republic” (No. CZ.02.1.01/0.0/0.0/16_013/0001569).

## References

  * Abramowicz  _et al._ (2015) H. Abramowicz _et al._ (ZEUS, H1), Eur. Phys. J. C75, 580 (2015), arXiv:1506.06042 [hep-ex] .
  * Gelis  _et al._ (2010) F. Gelis, E. Iancu, J. Jalilian-Marian, and R. Venugopalan, Ann.Rev.Nucl.Part.Sci. 60, 463 (2010), arXiv:1002.0333 [hep-ph] .
  * Accardi  _et al._ (2016) A. Accardi _et al._ , Eur. Phys. J. A52, 268 (2016), arXiv:1212.1701 [nucl-ex] .
  * Abelleira Fernandez  _et al._ (2012) J. Abelleira Fernandez _et al._ (LHeC Study Group), J.Phys. G39, 075001 (2012), arXiv:1206.2913 [physics.acc-ph] .
  * Newman and Wing (2014) P. Newman and M. Wing, Rev. Mod. Phys. 86, 1037 (2014), arXiv:1308.3368 [hep-ex] .
  * Contreras and Tapia Takaki (2015) J. G. Contreras and J. D. Tapia Takaki, Int. J. Mod. Phys. A30, 1542012 (2015).
  * Kowalski  _et al._ (2006) H. Kowalski, L. Motyka, and G. Watt, Phys. Rev. D74, 074016 (2006), arXiv:hep-ph/0606272 [hep-ph] .
  * Armesto and Rezaeian (2014) N. Armesto and A. H. Rezaeian, Phys. Rev. D90, 054003 (2014), arXiv:1402.4831 [hep-ph] .
  * Goncalves  _et al._ (2014) V. P. Goncalves, B. D. Moreira, and F. S. Navarra, Phys. Rev. C90, 015203 (2014), arXiv:1405.6977 [hep-ph] .
  * Lomnitz and Klein (2018) M. Lomnitz and S. Klein, (2018), arXiv:1803.06420 [nucl-ex] .
  * Good and Walker (1960) M. L. Good and W. D. Walker, Phys. Rev. 120, 1857 (1960).
  * Miettinen and Pumplin (1978) H. I. Miettinen and J. Pumplin, Phys. Rev. D18, 1696 (1978).
  * Mäntysaari and Schenke (2016a) H. Mäntysaari and B. Schenke, Phys. Rev. Lett. 117, 052301 (2016a), arXiv:1603.04349 [hep-ph] .
  * Mäntysaari and Schenke (2016b) H. Mäntysaari and B. Schenke, Phys. Rev. D94, 034042 (2016b), arXiv:1607.01711 [hep-ph] .
  * Cepila  _et al._ (2017) J. Cepila, J. G. Contreras, and J. D. Tapia Takaki, Phys. Lett. B766, 186 (2017), arXiv:1608.07559 [hep-ph] .
  * Cepila  _et al._ (2018a) J. Cepila, J. G. Contreras, and M. Krelina, Phys. Rev. C97, 024901 (2018a), arXiv:1711.01855 [hep-ph] .
  * Cepila  _et al._ (2018b) J. Cepila, J. G. Contreras, M. Krelina, and J. D. Tapia Takaki, Nucl. Phys. B934, 330 (2018b), arXiv:1804.05508 [hep-ph] .
  * Mueller (1994) A. H. Mueller, Nucl. Phys. B415, 373 (1994).
  * Mueller and Patel (1994) A. H. Mueller and B. Patel, Nucl. Phys. B425, 471 (1994), arXiv:hep-ph/9403256 [hep-ph] .
  * Shuvaev  _et al._ (1999) A. G. Shuvaev, K. J. Golec-Biernat, A. D. Martin, and M. G. Ryskin, Phys. Rev. D60, 014015 (1999), arXiv:hep-ph/9902410 [hep-ph] .
  * Nemchik  _et al._ (1994) J. Nemchik, N. N. Nikolaev, and B. G. Zakharov, Phys. Lett. B341, 228 (1994), arXiv:hep-ph/9405355 [hep-ph] .
  * Nemchik  _et al._ (1997) J. Nemchik, N. N. Nikolaev, E. Predazzi, and B. G. Zakharov, Z. Phys. C75, 71 (1997), arXiv:hep-ph/9605231 [hep-ph] .
  * Forshaw  _et al._ (2004) J. R. Forshaw, R. Sandapen, and G. Shaw, Phys. Rev. D69, 094013 (2004), arXiv:hep-ph/0312172 [hep-ph] .
  * Patrignani  _et al._ (2016) C. Patrignani _et al._ (Particle Data Group), Chin. Phys. C40, 100001 (2016).
  * Balitsky (1996) I. Balitsky, Nucl.Phys. B463, 99 (1996), arXiv:hep-ph/9509348 [hep-ph] .
  * Kovchegov (1999) Y. V. Kovchegov, Phys. Rev. D60, 034008 (1999), arXiv:hep-ph/9901281 [hep-ph] .
  * Golec-Biernat and Wusthoff (1998) K. J. Golec-Biernat and M. Wusthoff, Phys. Rev. D59, 014017 (1998), arXiv:hep-ph/9807513 [hep-ph] .
  * Golec-Biernat and Wusthoff (1999) K. J. Golec-Biernat and M. Wusthoff, Phys. Rev. D60, 114023 (1999), arXiv:hep-ph/9903358 [hep-ph] .
  * Aaron  _et al._ (2010) F. D. Aaron _et al._ (H1), JHEP 05, 032 (2010), arXiv:0910.5831 [hep-ex] .
  * Aid  _et al._ (1996) S. Aid _et al._ (H1), Nucl. Phys. B463, 3 (1996), arXiv:hep-ex/9601004 [hep-ex] .
  * Adloff  _et al._ (1997) C. Adloff _et al._ (H1), Z. Phys. C75, 607 (1997), arXiv:hep-ex/9705014 [hep-ex] .
  * Breitweg  _et al._ (1998) J. Breitweg _et al._ (ZEUS), Eur. Phys. J. C2, 247 (1998), arXiv:hep-ex/9712020 [hep-ex] .
  * Chekanov  _et al._ (2007) S. Chekanov _et al._ (ZEUS), PMC Phys. A1, 6 (2007), arXiv:0708.1478 [hep-ex] .
  * CMS (2018) _Exclusive $\rho(770)^{0}$ photoproduction in ultra-peripheral pPb collisions at $\sqrt{s_{NN}}=5.02~{}\mathrm{TeV}$ with the CMS experiment_, Tech. Rep. CMS-PAS-FSQ-16-007 (CERN, Geneva, 2018).
  * Chekanov  _et al._ (2005) S. Chekanov _et al._ (ZEUS), Nucl. Phys. B718, 3 (2005), arXiv:hep-ex/0504010 [hep-ex] .
  * Aktas  _et al._ (2006) A. Aktas _et al._ (H1), Eur. Phys. J. C46, 585 (2006), arXiv:hep-ex/0510016 [hep-ex] .
  * Alexa  _et al._ (2013) C. Alexa _et al._ (H1), Eur. Phys. J. C73, 2466 (2013), arXiv:1304.5162 [hep-ex] .
  * Abelev  _et al._ (2014) B. B. Abelev _et al._ (ALICE), Phys. Rev. Lett. 113, 232504 (2014), arXiv:1406.7819 [nucl-ex] .
  * Acharya  _et al._ (2018) S. Acharya _et al._ (ALICE), (2018), arXiv:1809.03235 [nucl-ex] .
  * Adloff  _et al._ (2000) C. Adloff _et al._ (H1), Phys. Lett. B483, 23 (2000), arXiv:hep-ex/0003020 [hep-ex] .
  * Chekanov  _et al._ (2009) S. Chekanov _et al._ (ZEUS), Phys. Lett. B680, 4 (2009), arXiv:0903.4205 [hep-ex] .
  * Aaij  _et al._ (2015) R. Aaij _et al._ (LHCb), JHEP 09, 084 (2015), arXiv:1505.08139 [hep-ex] .
  * Sirunyan  _et al._ (2018) A. M. Sirunyan _et al._ (CMS), Submitted to: Eur. Phys. J. (2018), arXiv:1809.11080 [hep-ex] .
  * Adloff  _et al._ (2002) C. Adloff _et al._ (H1), Phys. Lett. B541, 251 (2002), arXiv:hep-ex/0205107 [hep-ex] .
  * Abramowicz  _et al._ (2016) H. Abramowicz _et al._ (ZEUS), Nucl. Phys. B909, 934 (2016), arXiv:1606.08652 [hep-ex] .
  * Armesto  _et al._ (1996) N. Armesto, M. A. Braun, E. G. Ferreiro, and C. Pajares, Phys. Rev. Lett. 77, 3736 (1996), arXiv:hep-ph/9607239 [hep-ph] .