# Acoustic event detection for multiple overlapping similar sources

Dan Stowell and David Clayton
Queen Mary University of London
London, UK
<EMAIL_ADDRESS>
###### Abstract

Many current paradigms for acoustic event detection (AED) are not adapted to
the organic variability of natural sounds, and/or they assume a limit on the
number of simultaneous sources: often only one source, or one source of each
type, may be active. These aspects are highly undesirable for applications
such as bird population monitoring. We introduce a simple method modelling the
onsets, durations and offsets of acoustic events to avoid intrinsic limits on
polyphony or on inter-event temporal patterns. We evaluate the method in a
case study with over 3000 zebra finch calls. In comparison against a HMM-based
method we find it more accurate at recovering acoustic events, and more robust
for estimating calling rates.

##  1 Introduction

Acoustic event detection (AED) is useful for various purposes, such as
security monitoring, wildlife monitoring, and music transcription [1, 2, 3,
4]. Many approaches to AED assume that the acoustic scene is monophonic—having
no simultaneous or overlapping events—which is unrealistic but useful for some
applications. Approaches which allow for polyphonic scenes are more flexible,
but often assume that each stream is different in kind, allowing one
monophonic stream for each class of event considered [5, 6]. They may also
assume a fixed number of simultaneous streams, for example when source
separation is applied as a first step and then each separated channel is
treated as a monophonic scene [7].

In this paper we explore approaches for AED in cases with an unknown number of
similar sources. As an example, consider a sound recording in which a flock of
birds can be heard calling, all of the same species. This is representative of
practical scenarios in which AED might assist ecologists or conservation
organisations wishing to estimate the total number of individuals detected in
a recording, or alternatively the total number of calls in a recording. Both
such “point counts” are used (in most cases manually detected) for monitoring
trends in breeding populations [4].

Note the specific information need: for overall event counts, which is
distinct from the “transcription” task in which the objective is to recover a
list of the true events. An exact transcription would itself give us an exact
value for the event count; but an imperfect transcription may be an
inefficient or biased route to count estimation. In the abstract, event
counting tasks are regression problems, and may not require an estimated event
transcript at any point. Also important is that we wish to avoid placing
limits or inappropriate biases on the number of events that can be
simultaneously active. Methods that assume monophonic sequences per event
class are likely to be inappropriate, and other methods may bias estimation
due to assumptions implicit in their models. With this in mind, we briefly
consider some event detection paradigms in previous work.

To decompose an audio scene, one strand of research uses non-negative matrix
factorisation (NMF), in particular convolutive NMF which allows events to have
spectro-temporal structure [8]. However, these models are inflexible about the
temporal evolution within an event, depending on good matching of spectro-
temporal templates. This is particularly problematic for sounds with much
inherent within-class variability such as animal calls.

Hidden Markov models (HMMs) have been used in various systems for acoustic
event detection (e.g. [9, 5]). These can allow for variation in the temporal
evolution of events. However a typical HMM corresponds to a monophonic model
of events; extensions such as the factorial HMM extend this to a specific
fixed number of parallel sources, and thus retain strong limits on the level
of polyphony. In one example of polyphonic adapatation of HMM tracking, [5]
train and apply a standard HMM for event detection, where in their case each
state corresponds to a class of event. To achieve polyphonic detection they
perform multiple Viterbi decoding passes: after each Viterbi pass, the states
used are taken out of consideration for the subsequent passes. In this way a
transcription is obtained which allows multiple event classes to occur in
parallel. However it does not allow multiple simultaneous events of the same
class, and retains the fixed limit on polyphony.

In Section 3 we will describe an alternative way to adapt a HMM to multiple
detection scenario. However, that is primarily as a point of comparison
against the main model we wish to explore here, which uses an onset-duration-
offset model of acoustic events to allow for unbounded polyphony. We describe
this method in the next section. Then we will describe our alternative HMM
method, before evaluating both methods using a dataset of bird calls.

##  2 Onset-duration-offset model

Physiological studies indicate that biological auditory processing involves
early-stage “edge detectors” having separate auditory detection units for
onsets and for offsets, both in humans [10] and in songbirds [11]. The
information from these detectors is then combined in later processing for
cognition of “auditory objects” or events. Although there is no requirement
for our computational systems to mimic the organisation found in nature, this
suggests that a processing strategy starting with onset and offset detectors
and combining their outputs may be fruitful. We can combine onsets and offsets
with other information to yield posterior beliefs about the events observed
(Figure 1). If these components are based on the characteristics of individual
events, and not the temporal relationships between events, we should be able
to design a system that imposes few constraints or biases on the observable
event patterns.

Figure 1: Schematic diagram of onset-duration-offset event detection model.

The scheme just presented assumes that the onset and offset characteristics
are the reliable, relatively invariant characteristics of the events of
interest. However, it makes no strong assumptions on event durations, nor even
the signal content in the middle of the event, thus allowing for organic
variability. It also makes no strong assumptions on the temporal occurrence
patterns, and in particular the level of polyphony is unbounded: at any
particular time, if the system overall has detected $k$ more onsets than
offsets, then the current number of parallel active events is $k$.

Our approach is probabilistic: we will use onset/offset detectors that yield
detection probabilities at each time point, and our prior beliefs about event
duration will be expressed as a distribution over durations. To combine these
probabilities together, we characterise acoustic events in a two-dimensional
space indexed by onset time $t$ and duration $\tau$. We characterise the
conditional probability of an event at some point in that space as

$p_{\text{evt}}(t,\tau|\mathbf{y})\propto
p_{\text{on}}(t|\mathbf{y})\,p_{\text{off}}(t+\tau|\mathbf{y})\,p_{\text{dur}}(\tau)$

where $\mathbf{y}$ is the observation (the audio signal). The conditional
probabilities $p_{\text{on}}$ and $p_{\text{off}}$ come from the detectors,
and $p_{\text{dur}}$ is our duration prior. When dealing with discretised time
(as we do here), this Bernoulli model imposes a mild constraint: no two events
can have exactly the same onset time and duration. This constraint is very
mild, since events can co-occur in our scheme as long as they have slight
mutual differences in onset time or duration. (We will impose a slightly
stronger constraint to recover an event transcript, described shortly.)

Figure 2: Schematic diagram of how probabilities are combined in the onset-
duration-offset event detection model. Each source of probabilistic
information is a marginal with respect to a different direction in the [onset
$\times$ duration] space.

Each of our probabilistic sources of information (onsets, offsets, durations)
gives us information that acts as a one-dimensional marginal, when considered
in our two-dimensional [onset $\times$ duration] space (Figure 2). Note
particularly that offset detection probabilities are translated into [onset
$\times$ duration] space with an off-axis influence, since offset time is
equivalent to onset time plus duration. In this space, we assume that the
three types of probability information are conditionally independent and
multiply them together to produce a posterior “intensity” over possible events
(Figure 2). This could be thresholded to give a polyphonic event sequence, or
marginalised to give posterior beliefs about onsets and offsets. Such
posterior beliefs will be related to the raw onset/offset detections but
refined using the other information sources. Note that the posterior is not a
probability distribution: it does not sum up to one over our 2D space. It
represents a set of binomial probabilities; the sum over the 2D posterior
gives the expected number of events.

In the implementation we use here, for onset and offset detectors we use
random forest regression (cf. [12]). Our detectors take spectrogram patches as
input, the spectrograms having been treated with background noise reduction by
median-thresholding, and then first differencing in time. The trained random
forest outputs detection probabilities for each patch. Since the regression
makes an independence assumption for adjacent (overlapping) patches, the
outputs are liable to be correlated in time; to reduce this effect, after
training the random forest we then train an ordinary least squares regression
from a sliding window of 11 outputs from the detector onto the ground truth,
to recover “sharper” detections. To implement our prior on event durations, we
will train a Gaussian mixture model (GMM) on the durations observed in
training data.

We note some resemblances between our approach and that of [12]. Those authors
also use random forest regression as a recognition component that contributes
towards an eventual event segmentation. However, their method is fundamentally
different in that its elements for recognition are not the onsets/offsets, but
the frames “within” an event, which have been augmented with pointers to their
associated onset/offset. For this and other reasons their approach is limited
to monophonic event detection.

In the following we will refer to our onset-duration-offset model as ODO for
short.

###  2.1 Recovering an event transcript vs. event counts

To recover a definite set of events, our 2D posterior can be thresholded using
a threshold determined during the training phase. In practice, however, we
observe that this tends to yield a large number of duplicated events, since
any particular ground-truth event of duration $\tau$ will often be detected
with a relatively strong probability for duration $\tau+1$, $\tau-1$, etc.,
each of which is a separate position in our 2D posterior. This effect can be
reduced by imposing further assumptions: perhaps about the maximum polyphony,
or the temporal pattern. In the present work we wish to avoid imposing
assumptions that might strongly bias event counts and the like. We choose to
impose an assumption which is partly implicit in the detectors themselves: the
assumption that only one onset, and one offset, may happen in any time frame.
This assumption may bias detection in very dense audio recordings, but for
many densities encountered in practice this assumption holds almost always.
Hence to recover an event transcript, we keep only the events whose posterior
probability is stronger than all other events with matching onset time or
offset time. From these events we then select a threshold for discarding low-
probability events.

Taking Figure 2 as an example, in the posterior we see that the density due to
the second detected onset overlaps in 2D space with two possible offsets. In
this case we would keep no more than one of these events, namely that which
yields the strongest probability.

A straightforward way to count events is to count the number of items included
in an event transcript. However, as just described, producing an event
transcript requires making “hard” thresholding decisions, discarding some
information from the posterior. We can avoid the transcription step and simply
use the sum of the posterior, over the time range of interest, as the expected
event count in that time range. We will use this in our evaluation.

##  3 Hidden Markov model for event count

As an alternative to the method presented above, we also describe a hidden
Markov model (HMM) method for event detection. As described in Section 1, HMM
approaches to event detection impose limits on the possible polyphony, and
also may bias the durations and timing patterns of detected events. With those
caveats acknowledged, we wish to use the HMM as a point of comparison since it
is in widespread use. So in order to detect events of a single type, but with
potential polyphony, we apply a HMM but where the hidden states are not simply
‘on’ and ‘off’, but the count of currently-active events, i.e. the count of
events that have started but not yet ended. The state space is thus
$\\{0,1,2...K\\}$ where $K$ is the maximum number of simultaneous events
observed in the training data.

For modelling the observations, we train a separate Gaussian mixture model
(GMM) with ten components for each cardinality. Again we use spectral patches
to train this model, but without differencing them in time since in this case
we aim to model states rather than transitions.

To recover an event transcription from this HMM, we perform Viterbi decoding.
From the decoded sequence of cardinalities, we deduce onset and offset times,
and we associate onsets and offsets with each other in order of occurrence.
This transcript is then also used for event counting.

###  3.1 Combining the two models

The ODO and HMM models we have described offer two very different approaches
to event detection. We note that it is possible to combine the two, as
follows. We can expand the HMM state space to include not only the current
event cardinality, but also two binary indicators of whether the current frame
includes an onset and/or an offset. This expands the set of HMM states by a
factor of four. Not all state transitions are possible: e.g. a change in
cardinality from 3 to 4 can only occur when the onset state is 1. We do not
impose such limits manually but allow the system to learn them from the
transitions seen in training data.

Even with this expanded state space, HMM-based models inherit the limitations
already mentioned: cardinalities higher than seen in the training data will
not be correctly detected, and the HMM may bias timing patterns. However in
the following evaluation we will compare the empirical characteristics of the
methods we have described.

##  4 Evaluation

We recorded a set of four female zebra finches (Taeniopygia guttata) in an
indoor aviary. The birds exchanged contact calls at a rate of approx 40 calls
per minute. Birds were recorded for extended periods, and their calls were
transcribed as event sequences. Transcription was performed separately by two
human annotators, whose annotations were combined automatically, and any
discrepancies resolved by the first author.

This recording setup was designed for use in various studies; in the present
paper we use it as a case study for event detection. We took two 30-minute
recording sessions, recorded with the same birds but on separate days, and
used these two sessions for two-fold crossvalidation. The 30-minute sessions
contained 1663 and 1770 annotated calls in total. Here we use single-channel
omni mic recordings of the sessions.

The true polyphony in the original recordings ranged from zero to four. In
order to investigate heavier densities, as well as to investigate the effect
of density mismatches between training and test data, for each 30-minute
session we also created an artificial 10-minute mixture with the three
10-minute segments superimposed. All experiments thus used the same set of
calls, but in some cases the training or test data was “folded” down to a
denser 10-minute recording by superimposition.

As in other event detection evaluations [1, 2], for evaluating event
transcription we use the F-measure metric and we consider an event to be
correctly recovered if the onset matches within a fixed tolerance and the
duration matches within $50\%$ of the true duration. The typical event
duration in this data was approx 100 ms, so we chose $\pm 25$ ms as our
tolerance.

Separately, for evaluating event counts we divide the data into ten-second
windows and measure the RMS error between the true and estimated number of
events for each window. Note that both systems do exhibit some miscalibration,
in that their estimated counts even on the training data could exhibit a
multiplicative deviation from the truth. For the ODO system we believe this is
largely due to the independence assumption already mentioned in the underlying
detectors, and thus more sophisticated edge detectors might remedy this. To
account for this most basic aspect of miscalibration, during training of all
systems we used the training data to choose a multiplicative calibration
factor to apply to all event counts. Calibration did not make use of test
data. RMS error statistics are reported from the calibrated outputs.

We tested the following five configurations of event detector:

  * •

The full ODO system of Section 2.

  * •

The ODO system with a flat duration prior rather than the GMM. This
constrained events to lie within a reasonable duration but did not learn a
distribution on the duration, and allowed us to probe the extent of the
benefit provided by the GMM duration prior.

  * •

The HMM system of Section 3.

  * •

The combined ODO+HMM system of Section 3.1.

  * •

The raw output from the onset-detector component. This can be evaluated for
event counts only, not transcription, but indicates the extent to which the
detector component contributes to ODO performance.

Specific implementation details were as follows. Audio was recorded at 96 kHz,
and spectrograms calculated with frame length 2048, 50% overlap and Hann
windowing. Spectral information outside the range 0.5–20 kHz was discarded.
Noise reduction was implemented using spectral median-subtraction, with the
median calculated in a sliding ten-second window for each spectral band.
Spectral patches for the onset/offset detectors were taken from the time-
differenced spectrogram, of size five frames before and five frames after the
frame under consideration. The detectors were implemented using random forest
regression from the sklearn module, with 20 trees. Spectral patches for the
HMM-GMM modelling were taken from the non-time-differenced spectrogram, of
size five frames after the frame under consideration.

Figure 3: Event transcription results (F-measure), averaged over the two
crossvalidation folds. Error bars cover the range of results obtained within
individual folds.

Results for event transcription (Figure 3) show a number of tendencies.
Firstly the full ODO model consistently outperforms the ODO model with flat
duration prior, indicating that the learned prior on event durations adds
useful information. Secondly, the HMM system generally performs much worse
than the ODO system. This poor performance might be attributed to various
points of difference between the two systems, and so it is interesting to
observe that the combined ODO+HMM system improves on HMM but does not approach
ODO’s strong performance, despite making use of ODO’s onset/offset output as
part of its input observations.

The experiments with mismatched density in training and test give a general
performance degradation for all systems, indicating that there is still some
way to go to perform detection robust to very wide variation in event density.
The HMM-based systems perform relatively well in the experiment with increased
training density—an exception to the general pattern. Conversely, the poor
performance of the HMM-based systems in the experiment with increased testing
density matches expectations since the training did not encompass all the
event cardinalities found in the testing data.

In best conditions, our ODO system achieved an F-measure of around 37%.
Although quite distant from the ideal of 100%, it is on a similar scale as the
results reported for state-of-the-art methods for related tasks [1, 2].

Figure 4: Count estimation results (RMS error of counts in ten-second time
windows). The y-axis is inverted so that upwards means better performance, to
match Figure 3.

Results for event counting (Figure 4) show a slightly different picture.
Again, the mismatch in training and testing conditions has a general negative
impact on performance. In the main experiment, most of the systems perform at
very similar quality levels. This is except for the raw onset detector output,
included for comparison, which performs notably worse than the full systems
—illustrating that the ODO method performs much better than its underlying
detector. However, although the HMM and ODO+HMM systems achieve similar
performance as ODO in the main experiment, this is not the case in the
experiments with mismatched training and test densities, for which their
performance degrades further.

Taken together, these evaluations indicate that the ODO method is more
accurate and more robust than the HMM method for detecting or counting events
in polyphonic bird recordings such as those we have studied. The method can be
used for any data with events well-characterised by ‘landmarks’ such as onsets
and offsets, including animal and human sounds. However we note that all the
systems evaluated here showed quite some decay in performance when evaluated
with mismatched event densities. Improving these polyphonic detection
paradigms to be robust to these wide ranges of event density remains as future
work. Good detector components must be a key to strong performance: for the
present work we used simple detection methods using spectral patches as data;
improvements such as feature learning [13] could improve detection
performance. It also remains to evaluate systems on a wider range of event-
annotated audio recordings.

##  5 Acknowledgment

Our thanks to Maeve McMahon for help with bird management, and to Robert Jack
and Alex Wilson for data annotation.

## References

  * [1] D. Giannoulis, E. Benetos, D. Stowell, M. Rossignol, M. Lagrange, and M. D. Plumbley, “Detection and classification of acoustic scenes and events: an IEEE AASP challenge,” in _Proceedings of the Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA)_ , 2013.
  * [2] D. Stowell, D. Giannoulis, E. Benetos, M. Lagrange, and M. D. Plumbley, “Detection and classification of acoustic scenes and events,” _IEEE Transactions on Multimedia_ , 2015.
  * [3] R. Stiefelhagen, K. Bernardin, R. Bowers, J. Garofolo, D. Mostefa, and P. Soundararajan, “The CLEAR 2006 evaluation,” _Multimodal Technologies for Perception of Humans_ , pp. 1–44, 2007.
  * [4] T. A. Marques, L. Thomas, S. W. Martin, D. K. Mellinger, J. A. Ward, D. J. Moretti, D. Harris, and P. L. Tyack, “Estimating animal population density using passive acoustics,” _Biological Reviews_ , 2012.
  * [5] A. Diment, T. Heittola, and T. Virtanen, “Sound event detection for office live and office synthetic AASP challenge,” in _IEEE AASP Challenge on Detection and Classification of Acoustic Scenes and Events (WASPAA 2013 special session)_ , 2013, http://c4dm.eecs.qmul.ac.uk/sceneseventschallenge/abstracts/OL/DHV.pdf.
  * [6] S. Ewert, M. D. Plumbley, and M. Sandler, “A dynamic programming variant of non-negative matrix deconvolution for the transcription of struck string instruments,” in _Proc ICASSP 2015_ , 2015.
  * [7] T. Heittola, A. Mesaros, T. Virtanen, and A. Eronen, “Sound event detection in multisource environments using source separation,” in _Workshop on Machine Listening in Multisource Environments (CHiME 2011)_ , 2011, pp. 36–40. [Online]. Available: http://spandh.dcs.shef.ac.uk/projects/chime/workshop/papers/pS32_heittola.pdf
  * [8] P. D. O’Grady and B. A. Pearlmutter, “Convolutive non-negative matrix factorisation with a sparseness constraint,” in _Machine Learning for Signal Processing, 2006. Proceedings of the 2006 16th IEEE Signal Processing Society Workshop on_. IEEE, 2006, pp. 427–432.
  * [9] E. Benetos, M. Lagrange, and S. Dixon, “Characterisation of acoustic scenes using a temporally-constrained shift-invariant model,” in _Conference on Digital Audio Effects Conference (DAFx-12)_ , vol. 17, 2012, p. 21.
  * [10] M. Chait, D. Poeppel, and J. Z. Simon, “Auditory temporal edge detection in human auditory cortex,” _Brain research_ , vol. 1213, pp. 78–90, 2008.
  * [11] S. M. Woolley, P. R. Gill, T. Fremouw, and F. E. Theunissen, “Functional groups in the avian auditory system,” _The Journal of Neuroscience_ , vol. 29, no. 9, pp. 2780–2793, 2009.
  * [12] H. Phan, M. Maasz, R. Mazur, and A. Mertins, “Random regression forests for acoustic event detection and classification,” _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , vol. 23, no. 1, pp. 20–31, 2014.
  * [13] D. Stowell and M. D. Plumbley, “Automatic large-scale classification of bird sounds is strongly improved by unsupervised feature learning,” _PeerJ_ , vol. 2, p. e488, 2014.


# Snowmass 2021
Underground Facilities for the Cosmic Frontier
Topical Report

J. Cooley∗, S. Hertel, H. Lippincott∗, K. Ni, E. Pantic
∗Liaisons from other Snowmass Frontiers.

###### Contents

  1. 1 Direct detection of particle-like dark matter
  2. 2 Future experiments and their need
    1. 2.1 Noble liquids
    2. 2.2 Cryogenic bolometers
    3. 2.3 Other technologies
  3. 3 Conclusions

##  1 Direct detection of particle-like dark matter

Detecting dark matter directly using sensitive underground detectors is one of
the most promising ways to address the question of what is dark matter. During
the last decade, noble liquid detectors probed and excluded large parameter
spaces for heavy weakly interacting massive particle (WIMP) as the candidate
for dark matter and new technologies are being developed to probe light dark
matter with extremely low energy thresholds. Experiments have searches for not
only conventional nuclear recoils from dark matter nucleus scatterings, but
also from electronic recoils resulting from dark matter electron scattering
and absorption from other dark matter candidates such as the axion-like
particles.

The next generation underground experiments will search for dark matter in two
directions. One is to use larger target mass detectors to further probe
smaller cross sections in the WIMP parameter space and the other is to probe
lower mass dark matter which requires new technologies with extremely low
threshold. Both directions will eventually be limited by astrophysical
neutrino background. Once dark matter is detected, experiments with
directional sensitivities and experiments with longer exposure and larger
target masses are needed for the time-dependent signatures, such as annual
modulations, to verify their cosmic origin and to further study their
properties.

##  2 Future experiments and their need

The next generation underground dark matter search experiments fall into two
main detector classes: large noble liquids (liquid argon and liquid xenon)
detectors with the primary goal to probe lower WIMP cross sections and low-
threshold detectors (cryogenic bolometer experiments and other techniques) to
probe light dark matter candidates. Their needs for underground facilities are
described separately below. We also comment briefly on the need of other
technologies for dark matter searches.

###  2.1 Noble liquids

The next generation noble liquid experiments will use 50 to 300 tons of liquid
xenon or liquid underground argon (argon extracted from underground sources)
to probe heavy WIMPs into the “neutrino fog”. Future Projects at the planning
stage (from CF1 report) include:

  * •

The PandaX-xT experiment at CJPL using $\sim$50 ton liquid xenon.

  * •

A 50–100 ton liquid xenon observatory by the XLZD consortium, location
undetermined

  * •

A 300 ton ARGO liquid underground argon observatory at SNOLAB

There are also smaller scale noble liquid experiments at the conceptual stage,
such as the DarkSide-LowMass program using about 1.5-ton liquid argon.

Maintaining the large amount of cryogenic liquid in a stable cold liquid and
its supporting systems, such as distillation columns, requires a significant
continuous cooling power, which can result in heat-loads to the underground
facility which must be carefully considered. Liquid nitrogen cooling can be
supplied in addition to industrial cryogenic cooling. For example, the XENON1T
infrastructure features a 10 m3 LN2 tank underground, serving several
subsystems. A ten times larger LN2 tank supplying the need for the next
generation liquid xenon will be needed.

Many of the challenges in the coming noble liquid projects are due to the
scale and volume of the target material itself, and the even larger volumes of
surrounding active veto and shielding. The detector must be in an experiment
hall that is large both in floor space and ceiling height. Large noble liquid
experiments also require additional underground staging space for ‘cylinder
farms’ or other storage of large amounts of Xe or Ar at room temperature,
while waiting for the cosmogenic activated isotopes to decay away. At a
shallow depth, cosmogenic activation of certain isotopes, such as Xe-137, will
be a concern if the liquid xenon target will be used to search for 136Xe
$0\nu\beta\beta$ signals. Argon in the atmosphere contains cosmogenically
produced radioactive isotopes, mainly 39Ar but also 37Ar or 42Ar which can be
a significant background. The concentration of these thee isotopes is greatly
reduced in argon extracted from underground source, but the production of
cosmogenic radionuclides after extraction must be minimized. Typically, 37Ar
activation in liquid xenon or liquid argon at the surface will decay away in
several months of storage at underground. Further reduction of 37Ar in xenon
can be achieved using cryogenic distillation column as used in XENON1T/nT.

The future noble liquid dark matter experiments, both LAr and LXe, would
benefit greatly from on-site or near-site isotopic separation ability, to
mitigate for example a small air leak or cosmogenically-activated
radioisotopes at surface, introducing a trace amount of noble radioisotopes
(e.g. 37Ar,39Ar,or 85Kr), and to reduce the radioactive radon emanating from
the detector inner surfaces. This can be achieved using on-site cryogenic
distillation columns, such as use in XENON1T [2] and XENONnT. The XENON1T/nT
distillation column, requires 5.5 m of height, and releases some 10s of kW of
heat into the cavern. The XENONnT radon distillation column [6] is 3.8 m in
height and requires a few kW of cooling power. Both the height and heat load
requirement of a distillation column should be considered when siting future
large noble liquid DM experiments.

Muon-induced neutron background should be reduced to a negligible level
combining the deep underground rock overburden and sufficiently-thick active
shield. LZ and XENON1T/nT [7] liquid xenon detectors use water, while
DarkSide-20k liquid argon detector uses liquid argon as the shield for
external and muon-induced neutrons. Further neutron background from the
detector material is reduced by using liquid scintillator based veto detectors
with demonstrated high efficiency for neutron tagging. Since not all the
underground facilities are allowing usage of liquid scintillator, gadolinium
doped in the water, or a liquid argon veto coupled with gadolinium loaded
acrylic are new technologies employed. These veto’s neutron tagging efficiency
will need to be demonstrated with the current generation experiments. Safe
handling of these shielding liquids (gadolinium doped liquid scintillator or
water) shall be taken into consideration in regard of the environmental
concerns.

Assembling the large multiple-stage detector is a challenge in terms of
cleanliness (better than ISO-6), low-dust and low-radon requirement (better
than 100 mBq/m3). The current approach of transporting the inner detector
assembled in an above-ground clean room will not be feasible due to the large
future detector sizes. Large underground low-radon clean room with staging
areas is needed to assemble the inner detector parts. Sufficient vertical
space and cranes are needed to lift heavy detector parts during assembling.
The underground facilities will also need to balance the needs of continuing
experiments when performing any construction works such as new excavations.

Finally, the safety of handling the huge cryogen target, for both human safety
(asphyxiation etc.) and enormous expense of the target gas itself, needs to be
carefully taken into account during the design and operation of the
experiments.

Related: Large Bubble Chambers

Target liquid at pressure and in a superheated metastable state, they are the
500-kg to 5-ton scale PICO-500 and PICO-5T experiments, and a 1-ton scale
liquid argon Scintillation Bubble Chamber (SBC). These experiments are similar
to standard noble liquid experiments in facility requirements, in that
pressure vessel and cryogenic safety topics require host lab input and
interaction. Underground laboratories should provide recommendations on how to
handle these safety topics in next generation experiments.

###  2.2 Cryogenic bolometers

Many future experiments will focus on DM nuclear recoils of sub-keV energy,
with a signal either largely or entirely in the phonon or heat channel. Such
experiments rely on mK temperatures for this sensitivity, achieved via 3He/4He
dilution refrigeration. The 3He/4He dilution refrigeration technology enforces
a space constraint on such experiments, particularly in the vertical
direction. Even if the target masses themselves are small (e.g., $<$1 kg) the
experiment as a whole requires significant vertical space for the opening and
closing of the fridge and the shielding external to the cryostat. A vertical
space of at least 4m is highly beneficial. It is possible to configure a
dilution-refrigerator-based experiment to require less vertical space, but at
the cost of increased complexity and decreased cooling power.

The vibration environment is a second item of concern for many cryogenic
bolometer technologies. The target mass of such experiments are held in some
fashion to the support structure, and a slip in this holding, even a slip at
the atomic scale, can induce a visible ‘dark rate’ of signal into the phonon
system. While multiple groups are investigating mitigation methods either
through alternative target holding methods or vibrational isolation [3], the
environment itself must also be considered. A typical environmental goal may
be to keep these vibrations below 10${}^{-7}\mathrm{g\sqrt{Hz}}$ at all
frequencies.

Many bolometric sensors require a quiet electromagnetic environment as well.
Superconducting electronics (e.g. SQUID amplifiers) are highly sensitive to
noise across a wide range of frequencies, and are also sensitive to a DC
magnetic field. Faraday cage mitigations and cold filtering can help, but the
E&M environment should be kept in mind when selecting underground sites,
developing underground laboratory space, and evaluation of how experiments can
share a facility without cross-interference.

As the threshold of bolometric technologies continues to be pushed to lower
and lower energies, the reduction of backgrounds specific to sub-keV energies
is now an active, complementary research area. At these low energies,
backgrounds include Cherenkov or luminescence backgrounds from insulators near
the detectors [4], and also heat-only events which are still mysterious in
origin [5]. Given that very low-threshold experiments are currently dominated
by non-cosmogenic backgrounds, it is difficult to assess the depth requirement
of future low-threshold experiments until that R&D is more mature.

However, while depth requirements for _full experiments_ with cryogenic
bolometers are difficult to assess, the need for underground space for _R &D
associated with cryogenic systems_ is important for continued development
success. Specifically, the traditional strategy is to complete R&D above
ground, then deploy underground. In current and future generations of
cryogenic bolometer development, low-rate, ‘heat-only events’ from
instrumental effects are better constrained in the quiet and cosmic-ray
shielded environments of underground laboratories. Stated another way,
cryogenic experiments with long duration events suffer a challenge of
characterization due to event pile-up in above ground test systems. This is a
recognition of the value of underground space availability for developmental
and R&D-scale efforts in advancing cryogenic bolometer dark matter searches.

###  2.3 Other technologies

As described in the CF1 report, there are a multitude of other dark matter
detection technologies, including skipper CCD (OSCURA), Point-contact Ge
(CDEX-100/1T), Low-pressure gas detectors (directional detectors, CYGNUS),
Scintillator (SABRE, COSINE-200, others), and Superheated water (SNOWBALL). At
their current stages of development, each of these technologies, except CDEX,
is (for now) relatively compact in footprint, requiring underground space at a
level similar to or smaller than the cryogenic bolometer class. Also, each of
these technologies is relatively robust to lab environmental factors such as
vibration, etc. or would otherwise perform well if the needs of the above
noble liquid and cryogenic bolometer experiments are met.

##  3 Conclusions

Unraveling the properties of dark matter will continue being pursued through
direct detection at deep underground laboratories with larger targets (noble
liquids) and new experiments being planned. The facilities hosting the current
generation experiments are mostly full. There is a clear need for expanded
underground space, tailored to the needs beyond late 2020s.

These new underground space should specifically include: large spaces for
large experiments, specially noble liquids (liquid argon and liquid xenon) and
small spaces for smaller experiments (cryogenic bolometer & ‘other
technologies’); large radon-free clean rooms for detector assembly and
installation for the next generation experiments. In addition, large staging
areas are needed to store low-background detector materials, such as xenon or
underground argon gases, and utilities, such as cryogenic facilities.

## References

  * [1]
  * [2] Aprile, E., Aalbers, J., Agostini, F. et al. “Removing krypton from xenon by cryogenic distillation to the ppq level”, Eur. Phys. J. C 77, 275 (2017).
  * [3] R. Maisonobe, J. Billard, M. De Jesus, A. Juillard, D. Misiak, E. Olivieri, S. Sayah and L. Vagneron. “Vibration decoupling system for massive bolometers in dry cryostats”, 2018 JINST 13 T08009
  * [4] Peizhi Du, Daniel Egana-Ugrinovic, Rouven Essig, and Mukul Sholapurkar. Phys. Rev. X 12, 011009 – Published 13 January 2022
  * [5] Adari, et al. “EXCESS workshop: Descriptions of rising low-energy spectra”, https://arxiv.org/abs/2202.05097 2022
  * [6] M. Murra, D. Schulte, C. Huhmann and C. Weinheimer, “Design, construction and commissioning of a high-flow radon removal system for XENONnT,” [arXiv:2205.11492 [physics.ins-det]].
  * [7] E. Aprile et al. [XENON], Eur. Phys. J. C 77, no.12, 881 (2017) doi:10.1140/epjc/s10052-017-5326-3 [arXiv:1708.07051 [astro-ph.IM]].


# CrossVoice: Crosslingual Prosody Preserving Cascade-S2ST using Transfer
Learning

Medha Hira, Arnav Goel11footnotemark: 1 & Anubha Gupta
Indraprastha Institute of Information Technology
New Delhi - 110020, India
<EMAIL_ADDRESS>
Equal Contribution

###### Abstract

This paper presents CrossVoice, a novel cascade-based Speech-to-Speech
Translation (S2ST) system employing advanced ASR, MT, and TTS technologies
with cross-lingual prosody preservation through transfer learning. We
conducted comprehensive experiments comparing CrossVoice with direct-S2ST
systems, showing improved BLEU scores on tasks such as Fisher Es-En, VoxPopuli
Fr-En and prosody preservation on benchmark datasets CVSS-T and IndicTTS. With
an average mean opinion score of 3.6 out of 4, speech synthesized by
CrossVoice closely rivals human speech on the benchmark highlighting the
efficacy of cascade-based systems and transfer learning in multilingual S2ST
with prosody transfer.

##  1 Introduction

Transformer-based models (Vaswani et al., 2017) have revolutionized speech
processing, leading to significant advancements in automatic speech
recognition and text-to-speech technologies (Latif et al., 2023; Prabhavalkar
et al., 2023). This shift towards end-to-end systems has opened new avenues in
Speech-to-Speech Translation (S2ST) for translating speech across languages.
Our work introduces _CrossVoice_ , a cascade-based S2ST system utilizing the
latest open-source automatic speech recognition (ASR), machine translation
(MT), and text-to-speech (TTS) models unlike direct S2ST methods that bypass
MT. It is evaluated against state-of-the-art (SOTA) direct S2ST systems for
speech quality, cross-lingual prosody preservation, and translation accuracy
using BLEU (BiLingual Evaluation Understudy) score (Papineni et al., 2002).
Further, we investigate the performance of cascade-based vis-á-vis direct
approaches in S2ST and demonstrate how transfer learning can enhance prosody
transfer in cross-lingual settings.

##  2 Related Work

Current open-source systems for direct-S2ST involve various techniques such as
self-supervised learning (Lee et al., 2021b), using speech discrete units (Lee
et al., 2021a), text modalities (Zhang et al., 2023) and linguistic decoders
(Jia et al., 2022a). However, these systems often face challenges including
lower translation accuracy and inferior audio quality, particularly, in cross-
lingual prosody transfer (Bentivogli et al., 2021). In contrast, cascade-based
S2ST systems that integrate separate ASR, MT, and TTS models (Nakamura et al.,
2006) are criticized for high latency and subpar prosody transfer (Latif et
al., 2021).

Recent advancements in transfer-learning, such as voice cloning (Jia et al.,
2019) and transformer-based ASR and TTS, suggest the potential for more
efficient and effective prosody transfer in cascade-based systems (Huang et
al., 2023). Our study leverages these SOTA technologies in the proposed
cascade-based framework, _CrossVoice_ , and compares its performance with
direct S2ST systems on prosody transfer and overall efficiency.

Figure 1: Proposed Architecture for CrossVoice

##  3 Methodology

CrossVoice integrates state-of-the-art ASR, MT, and TTS techniques to
establish a baseline translation cascade: 1) Faster-
Whisper111https://github.com/SYSTRAN/faster-whisper for ASR (comparision of
other ASR models in A.2), which is a faster and batch-capable version of
Whisper-Large (Radford et al., 2022; Moslem et al., 2022); 2) Google’s NMT
model (Wu et al., 2016) for MT, which is known to reduce error rates
significantly; and 3) the Massive Multilingual Speech (MMS) model (Pratap et
al., 2023) based on VITS-TTS (Kim et al., 2021) for TTS, which is capable of
handling over 1000 languages with superior performance in linguistic diversity
and speech synthesis. CrossVoice uses transfer learning on a voice cloning
module (trained on the speaker identification task) for prosody preservation.
For this, a pre-trained speaker encoder generates X-vector embeddings
(Ravanelli et al., 2021; Snyder et al., 2018) and is coupled with FreeVC’s (Li
et al., 2023) voice conversion module to effectively transfer speaker prosody.

We conducted two sets of experiments to evaluate our system’s performance in
translation and speech synthesis. The first experiment evaluates synthesized
speech quality on the CVSS-T (Jia et al., 2022b) and IndicTTS benchmark
datasets (Kumar et al., 2023). We report mean opinion scores (MOS) from a
survey of 40 respondents, rating on a four-point scale with a 95% confidence
interval as per the protocol of Huang et al. (2023). MOS-h represents ratings
for natural human speech (called as Ground Truth or GT here), MOS-v for
baseline TTS audio without prosody transfer, and MOS-c for speech synthesized
by CrossVoice222Details about MOS calculations and the protocol are given in
the appendix A. The second experiment compares the BLEU performance of
CrossVoice with recent direct-S2ST SOTA systems discussed in Section 2 on the
translation tasks for which their superiority has been claimed over cascade-
based systems.

##  4 Results

Table 4 tabulates results of the first experiment on five translation tasks.
MOS-c score is almost the same as MOS-h (i.e., the GT) and also beats MOS-v
scores of the vanilla TTS considerably, by almost 40% on each task. Figures 2
and 3 (see A.5) highlight high BLEU scores of CrossVoice that averaged to 33.4
over all the languages of the chosen benchmark datasets.

Table-2 lists the BLEU scores of CrossVoice (BLEU-c) and SOTA methods
(BLEU-r). For calculating the BLEU scores, we employed Whisper (using the
_temperature setting of one_ and _greedy decoding_) for generating transcripts
of the speech generated using CrossVoice and SOTA methods. We sourced BLEU
scores from the original papers for the SOTA methods (reported as BLEU-r).
CrossVoice surpasses the claimed superior performance of direct S2ST systems
in their respective tasks, notably achieving almost a 19-point increase in
BLEU score in the VoxPopuli S2ST Fr-En task. This significant performance
boost is attributed to effective ASR and precise audio reconstruction through
voice cloning.

Table 1: MOS comparison on S2ST quality

Translation Task MOS-h ($\uparrow$) MOS-v ($\uparrow$)∘ MOS-c ($\uparrow$)∘
(GT) (Vanilla TTS) (CrossVoice) Spanish-English† 3.88 2.75 $\pm$ 0.12 3.76
$\pm$ 0.08 German-English† 3.83 2.64 $\pm$ 0.05 3.73 $\pm$ 0.11 Italian-
English† 3.75 2.89 $\pm$ 0.01 3.53 $\pm$ 0.10 Hindi-English⋆ 3.79 2.54 $\pm$
0.07 3.63 $\pm$ 0.02 English-Hindi⋆ 3.67 2.65 $\pm$ 0.03 3.34 $\pm$ 0.04
22footnotetext: †CVSS-T, ⋆Indic-TTS, ∘mean$\pm$std

Table 2: Comparison on S2ST-BLEU

Task (reported in BLEU-r ($\uparrow$) BLEU-c ($\uparrow$) SOTA method) (SOTA
method) (CrossVoice) Fisher Es-En 42.9 (Jia et al., 2022a) 45.6 Fisher Es-En
39.9 (Lee et al., 2021a) 45.6 MuST-C En-De 30.2 (Zhang et al., 2023) 39.7
MuST-C En-Fr 40.8 (Zhang et al., 2023) 46.5 VoxPopuli Fr-En 20.3 (Lee et al.,
2021b) 39.6

##  5 Conclusion and Future Work

CrossVoice effectively combines advanced ASR, MT, and TTS technologies,
establishing itself as a highly proficient cascade-based S2ST system with
strengths in cross-lingual prosody preservation and translation accuracy. Our
comprehensive experiments reveal that CrossVoice outperforms existing direct
S2ST systems, underscoring the effectiveness and reliability of cascade-based
systems with transfer learning for direct speech translation across languages.
Future work includes improving transfer of emphasis and intonation across
languages as reported in A.4.

#### URM Statement

The authors acknowledge that all the authors of this work meet the URM
criteria of ICLR 2024 Tiny Papers Track.

## References

  * Baevski et al. (2020) Alexei Baevski, Henry Zhou, Abdelrahman Mohamed, and Michael Auli.  wav2vec 2.0: A framework for self-supervised learning of speech representations, 2020.
  * Bentivogli et al. (2021) Luisa Bentivogli, Mauro Cettolo, Marco Gaido, Alina Karakanta, Alberto Martinelli, Matteo Negri, and Marco Turchi.  Cascade versus direct speech translation: Do the differences still make a difference?, 2021.
  * Chen et al. (2022) Sanyuan Chen, Chengyi Wang, Zhengyang Chen, Yu Wu, Shujie Liu, Zhuo Chen, Jinyu Li, Naoyuki Kanda, Takuya Yoshioka, Xiong Xiao, Jian Wu, Long Zhou, Shuo Ren, Yanmin Qian, Yao Qian, Jian Wu, Michael Zeng, Xiangzhan Yu, and Furu Wei.  Wavlm: Large-scale self-supervised pre-training for full stack speech processing.  _IEEE Journal of Selected Topics in Signal Processing_ , 16(6):1505–1518, October 2022.  ISSN 1941-0484.  doi: 10.1109/jstsp.2022.3188113.  URL http://dx.doi.org/10.1109/JSTSP.2022.3188113.
  * Di Gangi et al. (2019) Mattia A Di Gangi, Roldano Cattoni, Luisa Bentivogli, Matteo Negri, and Marco Turchi.  Must-c: a multilingual speech translation corpus.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pp. 2012–2017. Association for Computational Linguistics, 2019.
  * Huang et al. (2023) Wen-Chin Huang, Benjamin Peloquin, Justine Kao, Changhan Wang, Hongyu Gong, Elizabeth Salesky, Yossi Adi, Ann Lee, and Peng-Jen Chen.  A holistic cascade system, benchmark, and human evaluation protocol for expressive speech-to-speech translation, 2023.
  * Jia et al. (2019) Ye Jia, Yu Zhang, Ron J. Weiss, Quan Wang, Jonathan Shen, Fei Ren, Zhifeng Chen, Patrick Nguyen, Ruoming Pang, Ignacio Lopez Moreno, and Yonghui Wu.  Transfer learning from speaker verification to multispeaker text-to-speech synthesis, 2019.
  * Jia et al. (2022a) Ye Jia, Michelle Tadmor Ramanovich, Tal Remez, and Roi Pomerantz.  Translatotron 2: High-quality direct speech-to-speech translation with voice preservation, 2022a.
  * Jia et al. (2022b) Ye Jia, Michelle Tadmor Ramanovich, Quan Wang, and Heiga Zen.  CVSS corpus and massively multilingual speech-to-speech translation.  In _Proceedings of Language Resources and Evaluation Conference (LREC)_ , pp. 6691–6703, 2022b.
  * Kim et al. (2021) Jaehyeon Kim, Jungil Kong, and Juhee Son.  Conditional variational autoencoder with adversarial learning for end-to-end text-to-speech.  In _International Conference on Machine Learning_ , pp. 5530–5540. PMLR, 2021.
  * Kumar et al. (2023) Gokul Karthik Kumar, Praveen S V au2, Pratyush Kumar, Mitesh M. Khapra, and Karthik Nandakumar.  Towards building text-to-speech systems for the next billion users, 2023.
  * Latif et al. (2021) Siddique Latif, Inyoung Kim, Ioan Calapodescu, and Laurent Besacier.  Controlling prosody in end-to-end TTS: A case study on contrastive focus generation.  In Arianna Bisazza and Omri Abend (eds.), _Proceedings of the 25th Conference on Computational Natural Language Learning_ , pp. 544–551, Online, November 2021. Association for Computational Linguistics.  doi: 10.18653/v1/2021.conll-1.42.  URL https://aclanthology.org/2021.conll-1.42.
  * Latif et al. (2023) Siddique Latif, Aun Zaidi, Heriberto Cuayahuitl, Fahad Shamshad, Moazzam Shoukat, and Junaid Qadir.  Transformers in speech processing: A survey, 2023.
  * Lee et al. (2021a) Ann Lee, Peng-Jen Chen, Changhan Wang, Jiatao Gu, Sravya Popuri, Xutai Ma, Adam Polyak, Yossi Adi, Qing He, Yun Tang, et al.  Direct speech-to-speech translation with discrete units.  _arXiv preprint arXiv:2107.05604_ , 2021a.
  * Lee et al. (2021b) Ann Lee, Hongyu Gong, Paul-Ambroise Duquenne, Holger Schwenk, Peng-Jen Chen, Changhan Wang, Sravya Popuri, Yossi Adi, Juan Pino, Jiatao Gu, et al.  Textless speech-to-speech translation on real data.  _arXiv preprint arXiv:2112.08352_ , 2021b.
  * Li et al. (2023) Jingyi Li, Weiping Tu, and Li Xiao.  Freevc: Towards high-quality text-free one-shot voice conversion.  In _ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pp. 1–5. IEEE, 2023.
  * Moslem et al. (2022) Yasmin Moslem, Rejwanul Haque, and Andy Way.  Translation word-level auto-completion: What can we achieve out of the box?, 2022.
  * Nakamura et al. (2006) Satoshi Nakamura, Konstantin Markov, Hiromi Nakaiwa, Gen-ichiro Kikui, Hisashi Kawai, Takatoshi Jitsuhiro, J-S Zhang, Hirofumi Yamamoto, Eiichiro Sumita, and Seiichi Yamamoto.  The atr multilingual speech-to-speech translation system.  _IEEE Transactions on Audio, Speech, and Language Processing_ , 14(2):365–376, 2006.
  * Panayotov et al. (2015) Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur.  Librispeech: an asr corpus based on public domain audio books.  In _2015 IEEE international conference on acoustics, speech and signal processing (ICASSP)_ , pp. 5206–5210. IEEE, 2015.
  * Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu.  Bleu: a method for automatic evaluation of machine translation.  In Pierre Isabelle, Eugene Charniak, and Dekang Lin (eds.), _Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics_ , pp. 311–318, Philadelphia, Pennsylvania, USA, July 2002. Association for Computational Linguistics.  doi: 10.3115/1073083.1073135.  URL https://aclanthology.org/P02-1040.
  * Post et al. (2014) Matt Post, Gaurav Kumar, Adam Lopez, Damianos Karakos, Chris Callison-Burch, and Sanjeev Khudanpur.  Fisher and callhome spanish–english speech translation.  _LDC2014T23. Web Download. Philadelphia: Linguistic Data Consortium_ , 2014.
  * Prabhavalkar et al. (2023) Rohit Prabhavalkar, Takaaki Hori, Tara N. Sainath, Ralf Schlüter, and Shinji Watanabe.  End-to-end speech recognition: A survey, 2023.
  * Pratap et al. (2023) Vineel Pratap, Andros Tjandra, Bowen Shi, Paden Tomasello, Arun Babu, Sayani Kundu, Ali Elkahky, Zhaoheng Ni, Apoorv Vyas, Maryam Fazel-Zarandi, Alexei Baevski, Yossi Adi, Xiaohui Zhang, Wei-Ning Hsu, Alexis Conneau, and Michael Auli.  Scaling speech technology to 1,000+ languages, 2023.
  * Radford et al. (2022) Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever.  Robust speech recognition via large-scale weak supervision, 2022.
  * Ravanelli et al. (2021) Mirco Ravanelli, Titouan Parcollet, Peter Plantinga, Aku Rouhe, Samuele Cornell, Loren Lugosch, Cem Subakan, Nauman Dawalatabad, Abdelwahab Heba, Jianyuan Zhong, Ju-Chieh Chou, Sung-Lin Yeh, Szu-Wei Fu, Chien-Feng Liao, Elena Rastorgueva, François Grondin, William Aris, Hwidong Na, Yan Gao, Renato De Mori, and Yoshua Bengio.  SpeechBrain: A general-purpose speech toolkit, 2021.  arXiv:2106.04624.
  * Snyder et al. (2018) David Snyder, Daniel Garcia-Romero, Gregory Sell, Daniel Povey, and Sanjeev Khudanpur.  X-vectors: Robust dnn embeddings for speaker recognition.  In _2018 IEEE international conference on acoustics, speech and signal processing (ICASSP)_ , pp. 5329–5333. IEEE, 2018.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  _Advances in neural information processing systems_ , 30, 2017.
  * Wang et al. (2021) Changhan Wang, Morgane Rivière, Ann Lee, Anne Wu, Chaitanya Talnikar, Daniel Haziza, Mary Williamson, Juan Pino, and Emmanuel Dupoux.  Voxpopuli: A large-scale multilingual speech corpus for representation learning, semi-supervised learning and interpretation, 2021.
  * Wu et al. (2016) Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, Łukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, Nishant Patil, Wei Wang, Cliff Young, Jason Smith, Jason Riesa, Alex Rudnick, Oriol Vinyals, Greg Corrado, Macduff Hughes, and Jeffrey Dean.  Google’s neural machine translation system: Bridging the gap between human and machine translation, 2016.
  * Zhang et al. (2023) Yuhao Zhang, Chen Xu, Bojie Hu, Chunliang Zhang, Tong Xiao, and Jingbo Zhu.  Improving end-to-end speech translation by leveraging auxiliary speech and text data.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 37, pp. 13984–13992, 2023.

##  Appendix A Appendix

###  A.1 Acronyms used

  * •

MOS : Mean Opinion Score

  * •

S2ST : Speech to Speech Translation

  * •

ASR : Automated Speech Recognition

  * •

MT : Machine Translation

  * •

NMT : Neural Machine Translation

  * •

TTS : Text to Speech

  * •

GT : Ground Truth

  * •

SOTA : State-Of-The-Art

  * •

BLEU : Bilingual Evaluation Understudy

###  A.2 ASR Results

We compared various ASR models such as variants of Whisper, Wav2Vec2.0
(Baevski et al., 2020), WavLM (Chen et al., 2022) and Faster-Whisper on
multilingual datasets: Librispeech-test-clean (English), IndicTTS (Indian
accented English speech) (Kumar et al., 2023) and VoxPopuli - French, Spanish
and German. Results are shown in Table 3 and 4. Faster-Whisper clearly
performs very well on both WER and average latency metrics. We measured
average latency as the weighted average of the time taken to transcribe each
sample of the entire dataset.

Table 3: Results of different ASR Models on Librspeech-test-clean subset (Panayotov et al., 2015) Model | WER (%) | Average Latency (s)
---|---|---
Whisper - Tiny | 9.78 | 0.183
Whisper - Base | 6.94 | 0.234
Whisper - Small | 4.85 | 0.385
Whisper - Large | 3.63 | 1.145
Wav2Vec2.0 - Large | 3.20 | 0.415
WavLM - Large | 2.80 | 0.525
Faster-Whisper | 4.23 | 0.152
Table 4: WER benchmarking of models on various Datasets Model | IndicTTS-en | VoxPopuli-French | VoxPopuli-Spanish | VoxPopuli-German
---|---|---|---|---
Wav2Vec2.0 - XLSR | 15.65 | 25.34 | 21.34 | 24.73
WavLM - Large | 14.25 | 23.21 | 18.65 | 20.56
Whisper - Tiny | 10.74 | 31.53 | 19.63 | 25.24
Whisper - Base | 8.63 | 21.34 | 15.32 | 19.75
Whisper - Small | 5.28 | 13.24 | 12.18 | 13.32
Whisper - Large | 3.85 | 10.56 | 7.82 | 9.75
Faster-Whisper | 4.38 | 11.23 | 8.96 | 10.32

###  A.3 Translation Tasks

We benchmarked CrossVoice on 3 benchmark S2ST tasks and they are summarised as
follows:

  1. 1.

Fisher (Spanish-English) (Post et al., 2014): The Fisher Spanish dataset is a
collection of telephone speech conversations in Spanish, primarily involving
topics of daily life. It contains over 160 hours of recorded conversations,
involves more than 130,000 utterances and includes around 24,000 speakers.

  2. 2.

MuST-C (English to German & English to French) (Di Gangi et al., 2019): It is
a multilingual speech translation corpus with 273 hours of audio recorded for
the English to German task and 236 hours of audio recorded for the English to
French task.

  3. 3.

VoxPopuli French-English (Wang et al., 2021): French and English segments of
the VoxPopuli dataset are taken for translation with 211 and 543 hours of
transcribed audio. Same text segments from the dataset are taken for the S2ST
task.

For computing the BLEU-c scores, we randomly sampled 250 clips 10 times for
each task and tested our system. The reported BLEU-c score is the average of
these 10 iterations to ensure a fair and correct representation of our
results.

###  A.4 MOS Calculation Methodology and Protocol

Following the protocol laid out by (Huang et al., 2023), a survey was
conducted of 40 respondents, where each respondent was shown the same set of
translated clips along with their clips in the source language. This set
consisted of 15 voice clips of duration varying from 2 secs to 10 secs. The
following questions were asked from the respondents:

  1. 1.

Rate the similarity of the voice of the speaker to the original source clip :
1 - _Completely Different_ , 2 - _Some similarities but more differences_ , 3
- _Some differences but more similarities_ , 4 - _Perfectly similar_.

  2. 2.

Rate the quality and naturalness of the generated audio clip: 1 - _Extremely
poor / robotic_ , 2 - _Somewhat natural but more robotic / poor_ , 3 -
_Somewhat robotic/poor but more natural_ , 4 - _Perfectly natural_.

  3. 3.

Rate the similarity of the emphasis and intonation of the source clip and
synthesised clip: 1 - _Completely Different_ , 2 - _Some similarities but more
differences_ , 3 - _Some differences but more similarities_ , 4 - _Perfectly
similar_.

Respondents were allowed to rate “exactly in-between” for intermediary cases.
It was noted starkly that on the first two questions, a huge proportion of
respondents rated the synthesised speech for the five languages as close to 4.
However, on the last question, a lot of respondents rated the system between 2
and 3 indicating that while the speaker’s voice characteristics and prosody
are being transferred with quality, intonation and emphasis will need
improvement.

For calculating MOS-v, we employed our MMS TTS without using any voice
cloning. Similar surveys on a lesser number of clips were able to see the
Vanilla TTS system getting lower ratings compared to CrossVoice on all the
three questions. We referenced MOS-h scores from the official paper of (Jia et
al., 2022b).

###  A.5 Results on CVSS-T and IndicTTS

We conducted experiments on 11 languages from the CVSS-T dataset and Hindi
from IndicTTS dataset using CrossVoice. Figure 2 shows the results for these
12 languages when translated from any language $X\longrightarrow en$
(English), whereas Figure 3 shows the results for the 12 languages when
translated from (English) $en\longrightarrow X$. Notably, our system shows
higher BLEU scores on translating from English to any language because of low
WER of Whisper on English and NMT being extensively pre-trained on
$en\longrightarrow X$ tasks.

For calculating the results, we randomly took samples of 100 clips for each
language and calculated results for one sample. We repeated this process for
10 iterations to check for biases. We report the average BLEU score for each
language from the experiments. The standard deviation shown on all the tasks
ranged between $\in(0.5,1.5)$, thus, indicating lesser deviation.

Figure 2: BLEU scores on CVSS-T for X-en Translation Figure 3: BLEU scores on
CVSS-T for en-X Translation

###  A.6 Ethical Considerations

This study has been conducted and tested on standard open source datasets
(that are appropriately cited in the paper), widely used in the literature.

We recognize that voice cloning has the potential to be used for malicious
activities; however, the benefits of this technology may outweigh the
negatives. Our system is designed to encourage inclusivity and transcend the
language barrier in communication between individuals.

Further, we advocate for transparency in the use of voice cloning technology
and users should always be informed when they are interacting with a cloned
voice.

###  A.7 Limitations and Challenges

CrossVoice relies heavily on extensive datasets for training. Obtaining and
processing large, high-quality, and diverse datasets that cover a wide range
of languages and accents is a significant challenge and can limit the system’s
effectiveness and scalability. CrossVoice encounters challenges in accurately
transferring prosody, like intonation and stress patterns, across different
languages. This is a complex task due to the inherent differences in
linguistic structures and prosodic features among languages. This lack of
appropriate transfer of intonation and emphasis is also depicted by the MOS
score protocol A.4