# Genetic Bottleneck and the Emergence of High Intelligence by Scaling-out and
High Throughput ††thanks: This work is part of the Ph.D. thesis work of the
first author.

Arifa Khan1, Saravanan P.2, and Venkatesan S.K.3

(1Faculty Of Management, SRM Institute of Science and Technology,
Katankalattur, India 60320, Email<EMAIL_ADDRESS>
2Faculty Of Management, SRM Institute of Science and Technology,
Katankalattur, India 603203, Email<EMAIL_ADDRESS>
3CQRL Bits LLP,Vignesh avenue, Selaiyur, Chennai 600073,
Email<EMAIL_ADDRESS>
###### Abstract

We study the biological evolution of low-latency natural neural networks for
short-term survival, and its parallels in the development of low-latency high-
performance Central Processing Unit in computer design and architecture. The
necessity of accurate high-quality display of motion picture led to the
special processing units known as the GPU, just as how special visual cortex
regions of animals produced such low-latency computational capacity. The human
brain, especially considered as nothing but a scaled-up version of a primate
brain evolved in response to genomic bottleneck, producing a brain that is
trainable and prunable by society, and as a further extension, invents
language, writing and storage of narratives displaced in time and space. We
conclude that this modern digital invention of social media and the archived
collective common corpus has further evolved from just simple CPU-based low-
latency fast retrieval to high-throughput parallel processing of data using
GPUs to train Attention-based Deep Learning Neural Networks producing
Generative AI with aspects like toxicity, bias, memorization, hallucination,
with intriguing close parallels in humans and their society. We show how this
paves the way for constructive approaches to eliminating such drawbacks from
human society and its proxy and collective large-scale mirror, the Generative
AI of the LLMs.

Keywords: Genomic bottleneck, Scaling Out, Data driven parallelization,
Huang’s law, memorization.

## Introduction

In an earlier study [1] we showed how the collective social evolution of
writing, printing of manuscripts, books and libraries, grew as a response to
the limitations in human memory. We also show how digitization of books and
manuscripts under social media such a Wikipedia, Newspapers, Magazines etc.
have made possible the creation of complete corpus of human society and thus
its essence reflected in the emergence of Artificial Intelligence in Large
Language Neural Network models. In this study we consider these aspects in
more quantitative detail and also introduce additional aspects such as the
genomic bottleneck that enticed this scaled-out architecture of the neural
network.

In the first section we consider the important aspects of low-latency response
of insects, birds, reptiles and mammals vis-a-vis the evolution of computer
hardware architecture, especially the CPU and the support services around the
CPU such as the RAM, caches etc.

In the second section we consider the genomic bottleneck and how a scale-out
response of cluster of neurons like the brain developed in response to this.
We next consider the limitations in animal brain and in the fourth section we
consider how in the human case the brain grew to a maximum set of neurons and
synaptic connections that is trained and pruned through social interactions.
We further consider the limitations of human brain and how through social
extension of human brain through discovery of language, writing, storing,
reproducing narratives displaced in time and space that finally we are able to
achieve Generative Artificial Intelligence through Large Learning Models.

## Low latency as a response to the survival of the fittest

In this article we consider how the insects, birds, reptiles and mammals have
evolved to produce a low-latency natural neural network for short-term
survival. There is interplay between innate structures developed on birth and
the learning that takes place after birth. Many insects are born as worms at
birth have their own survival methods and they later metamorph into flying
insects through an intermediate stage of a pupa. All these are built into the
innate genetic structure of the insect with very little social parenting or
learning after birth, but even in this case it must be stated carefully as
there may be certain interplay between the environment and the innate
structures that may allow certain degree learning by the individual, however
limited it may be. Also, it has been proven that some social insects like the
honeybee learn dancing skills that indicate the direction of food source [2].
Of course, the ability to learn is an innate skill developed at birth. For
insects, whose life cycle is short, such an investment in after-birth learning
or social grooming is untenable as a cost. It could be also that their
survival strategy is mostly by very high rate of reproduction, which is
closely linked to the genome-based Darwinian evolution. But at the same time,
a mosquito or a fly has to escape predatory birds and bats, so it has also
developed low-latency neural networks with complex strategies of flight with a
quick effectiveresponse.

### Insect brain and it’s neural network.

One of the interesting aspects of nervous system of the insects is that some
insects can even live a normal life for several days even after their head has
been cut-off. This has made some to conclude that they don’t have a brain, but
as Charles Darwin [3] put it:

“It is certain that there may be extraordinary activity with an extremely
small absolute mass of nervous matter; thus, the wonderfully diversified
instincts, mental powers, and affections of ants are notorious, yet their
cerebral ganglia are not so large as the quarter of a small pin’s head. Under
this point of view, the brain of an ant is one of the most marvelous atoms of
matter in the world, perhaps more so than the brain of man.”

The natural neural networks of insects are not just information extractors
like LLM, but they also have endpoints for action nodes, more like the agents
in LLM that provide actionable decision points. So, they are quite complex. In
the case of some insects the brain seems to be not a critical component in
orchestration of events and that the rest of the neural systems seems to be
quite capable of orchestrating the insect without it. It seems that the
peripheral systems have their own fully functional backup neural capabilities.
In Table 1 we lists some insects along with their neural and genomic
capacities.

Table 1: Neural and Genomic infromation and processing capacities of some insects Species |  Neurons |  Base Pairs |  Parenting |  Size |  Lifespan |  No. of eggs
---|---|---|---|---|---|---
Culex pipiens (Mosquito) |  220,000 |  1.38 billion |  None. |  5 mm |  14 days |  1200 eggs
Tribolium ferrugineum (Rusty grain beetle) |  Around 100,000 [6] |  Appx 165 million [7] |  Female lays eggs in stored grain products [8] |  2-3 mm [9] |  1-2 years [9] |  400 eggs [10]
Cimex lectularius (Bed bug) |  Appx
250,000 [11] |  Appx 700 million [12] |  Female lays eggs in cracks and crevices near host animals [13] |  4-5 mm [14] |  6-12 months [14] |  500 eggs [15]
Musca domestica (House fly) |  Appx
100,000 [16] |  Appx 1.5 billion [17] |  Females lay eggs in decaying organic matter [18] |  6-9 mm [19] |  15-30 days [19] |  500 eggs [20]
Tribolium confusum (Confused flour beetle) |  Appx
100,000 [21] |  160 million [22] |  Female lays eggs in stored food products [23] |  3-4 mm [24] |  1-2 years
[24] |  400 eggs [25]
Tribolium castaneum (Red flour beetle) |  100,000 [26] |  163 million [27] |  Female lays eggs, typically in food sources like flour [28] |  2-3 mm [29] |  2-12 months [29] |  500 eggs [30]
Apis mellifera (Honeybee) |  Appx
960,000 [31] |  250 million [32] |  Social structure with various roles; queen lays eggs, workers care for them [33] |  12- 15 mm [34] |  Up to 5 weeks for workers, several years for queen [34] |  Queen lay up to 2,000 eggs per day [35]
Caenorhabditis elegans (Roundworm) |  Around 302 (hermaph-rodite) [36] |  100 million [37] |  Hermaph-rodites self-fertilize; male mating also occurs [38] |  1 mm [39] |  2-3 weeks [39] |  About 300 eggs per hermaph-rodite [40]
Drosophila melanogaster
(Fruit fly) |  Appx 100,000 [41] |  165 million [42] |  None (lay eggs) [43] |  2-3 mm [44] |  40-50 days [44] |  Varies, but can lay hundreds of eggs in its lifetime [45]

### Computer CPU and the microprocessor

The CPUs fetch, decode, execute units that implement it using instruction sets
that divide the task into control unit, arithmetic logic unit, address
generation unit and memory management unit. Although CPUs were initially
created as an independent system, the modern CPUs are part of the integrated
microprocessor circuit where multicore CPUs are embedded inside one circuit
board. They can process from several mega flops (floating point operations) to
nowadays several Giga flops of data. Another traditional way to measure speed
of CPU is through clock-cycle frequency of CPU which also typical runs from
several MegaHertz to several GigaHertz. If you have multiple CPU cores, then
parallelization is possible, and the throughput can be increased.

### Insect neural systems and the modern CPU

It is not straight forward to compare complex natural neural networks with the
computer CPU. The CPU is based on fetch, decode and execute sequence, where
quantitative estimators like number of Flops (floating point operations) or
clock-cycle may make sense. Even in the case of CPU there are other aspects
ALU, IO that may dominate in some use case where L1/L2/L3 cache sizes may be
of important. Parallelization is another aspect that may depend on particular
use cases where large throughput is required that can be orchestrated in
parallel. However, here we will try to make such drastic simplifications as a
starting point. Insects typically have about 200,000 neurons and assuming
synaptic connection of about 1000 connection per neuron we have about 200
million synaptic connections or Flops. This is roughly equivalent to Intel’s
Pentium Pro or Pentium II 350nm Klamath Core (233 and 266 MHz) that could do
200-250 million instructions per second or even better the 250nm Tonga and
Dixon cores but with limited L2 cache of 512KB. This is comparable to the case
of insects, which do have limited temporary memory. Table 2 lists the
evolution of Intel CPUs and Figure 1 illustrates the Moore’s law of CPU
development.

Table 2: Evolution of the Intel CPU Year | Microprocessor | Flops |  Clock Speed (GHz) | RAM (GB) | L1 Cache (KB) |  L2 Cache (MB)
---|---|---|---|---|---|---
1971 | Intel 4004 | - | 0.000074 | - | - | -
1972 | Intel 8008 | - | 0.0002 | - | - | -
1974 | Intel 8080 | - | 0.002 | - | - | -
1978 | Intel 8086 | None | 0.005-0.01 | 0.000064 | - | -
1981 | Intel 8088 | 0.33 KF | 0.00477 | 0.000016 | - | -
1982 | Intel 80286 | 1.2 KF | 0.004-0.012 | 0.000128 | - | -
1985 | Intel 386 | 5-10 KF | 0.012-0.033 | 0.000512 | - | -
1989 | Intel 486 | 20-54 KF | 0.02-0.1 | 0.001-0.016 | 0.008-0.016 | -
1993 | Intel Pentium | 90-125 KF | 0.05-0.3 | 0.002-0.256 | 8-16 | -
2000 | Intel Pentium 4 | 1.3-5.4 GF | 1.3-3.8 | 0.128-4 | 8-32 | 0.256-2
2006 | Intel Core 2 Duo | 8-20 GF | 1.06-2.66 | 0.5-8 | 32-64 | 2-6
2010 | Intel Core i7 | 40-107 GF | 1.06-3.6 | 4-32 | 64-128 | 4-12
2013 | Intel Core i7 | 83-186 GF | 2.4-3.5 | 8-64 | 128-256 | 4-20
2017 | Intel Core i9 | 218-331 GF | 3.0-4.3 | 16-128 | 256-512 | 8-24
2020 | Intel Core i9 | 426-695 GF | 2.9-5.3 | 32-128 | 512-1024 | 12-24
2022 | Intel Core i9 | 677-861 GF | 3.1-5.3 | 64-256 | 512-2048 | 16-32
Figure 1: Moore’s law of CPU.

## The genomic bottleneck and the brain as the centralized processing unit

Here we first explain here on what “genomic bottleneck” is [4] and how it
played a crucial role in the formation external information storage system
such as the brain. The information content of the DNA codes provides a rough
upper bound on the amount of information that can be transmitted from one
generation to another. A simple worm like C. elegans has about 108 base pairs,
so it could transmit 2 $\mathrm{\times}$ 108 bits. On the other hand, if the
C. elegans brain has a dense 3022 connection matrix that would be 9
$\mathrm{\times}$ 104 bits of storage and even allowing for a small factor
associated with the number of bits required for a synaptic weight, this would
be more than adequate to explicitly encode the highly stereotyped connectivity
among the 302 neurons. In the case of insects, the number of base pairs is
around 3 $\mathrm{\times}$ 108 bits [5]. But in the case of insects, they have
200,000 neurons, so associating 1000 synaptic connections per neuron, this
makes it around 2 $\mathrm{\times}$ 108 and multiplying by a factor for number
bits per connection for synaptic weights this is roughly on par with the
genomic capacities of an insect. So, this explains the incredible variety of
insects and niches they occupy. The rapid high rate of reproduction being the
Darwinian strategy they adopt to survive thus reinforcing the genomic
capacity, especially given their short life that necessitates a short-term
strategy.

We also see in the case of insects that although it has a very complex neural
network system that is comparable to Pentium II level microprocessor, its
dependence on the brain is limited as established by the fact that in some
insects even cutting-off the brain keeps the insect almost fully functional
for even several days. This shows that in the case of insects the brain has
not yet developed as a crucial centralized controlling unit as is the case in
organisms such as fishes, reptiles, birds and mammals.

However, due to an information genomic bottleneck [4], the natural
networksevolved to produce a centralized memory storage that was scaled-out to
produce a larger and larger brain.

Although, in terms of encephalization ratio, an insect or even a bird brain is
far superior to human brain in terms of range performances and activities that
it co-ordinates with such a small brain. In the case of larger animals, scale
seems to matter, especially in terms of narrating an incidence that occurred
in different space and time and to produce complex social interactions. We
have seen in the case of the honeybees, ants such complex social behavior
could also be achieved, but with limited capacity. In the case of birds like
corvidae s(the common crow), we have seen how they do have complex behavior
[6] with even a small-sized brain. However, unlike in the case of insects,
their brain is an essential vital central processing unit that leads to
completely dysfunctional organism as soon as the brain (head) is cut-off, as
it has crucial elements that orchestrate the organism. So, it seems as the
organisms grow and complexity, the necessity for centralized controlling unit
as the brain become vital (again there may be exceptions, but minor exceptions
more less provide statistical proof of the main argument). In this case, the
longer-life cycle of the animal and lower reproduction rates reduces the
Darwinian genomic pressures and thus expands on the trainable prunable neural
component with nascent capacities.

## Limitations of the animal brain and the evolution of
scaled-up human brain.

Larger egg-laying animals had to take care of their young ones by incubating
eggs as in the case of birds and by burying them underground by reptiles. The
reptiles did take some care but eventually the young ones that hatched out of
the eggs had to survive in a hostile environment as soon as they are born,
making innate genomic capacities important. The mammals (including kangaroos)
on the other hand developed a capacity to protect the young ones within their
womb. This gave them space to groom them after emerging out of the egg, inside
a protective cage, so that their neural systems get trained and pruned inside
a limited basic set of controlled (in-vitro) input conditions before it merges
from the womb. This does fine-tune and hardening of neural network to basic
life-sustaining functions (like regularizing heartbeat, immune response, and
cell-level functions) before allowing it to expand on other external stimuli
after birth. Longer gestation period allows for greater hardening to these
basic life-sustaining functions, especially quite useful in a larger animal
with longer lifespan. Extensive learning and pruning to external conditions
also begin after coming out of the womb. In the case of deer and ungulates,
the learning period is short as it is immediately under selection pressure
from carnivores and scavengers. From the camouflaged bed face, once weaned
from the mother, they can feed and run from predators in about couple of
months. Although the life-span of deer and a carnivore like a Tiger or a cat
(house cats are smaller and shorter life spans of around 15 years, just like
some smaller deer like the white-tailed deer) is roughly the same, in the case
of carnivores they take much longer to mature and become independent, a luxury
their predator status affords them. Also, the hunting skills require a certain
degree of training by the parent, which starts early by allowing it to play
with its tail as a proxy for a prey. Playing among themselves with one acting
as a proxy for the prey is also a means for training at grabbing prey like
deer and ungulates. These complex hand-skills require an innate capacity for
training and pruning fast growing neurons and their neural networks. Cats have
roughly 760 million neurons with quite dense network of $\mathrm{\sim}$1
$\mathrm{\times}$ 1013 synaptic connections, with $\mathrm{\sim}$2
$\mathrm{\times}$ 109 DNA base pairs. The humble mouse (an omnivore that is
closely related to humans) also has 71 million neurons and $\mathrm{\sim}$1
$\mathrm{\times}$ 1012 connections but with $\mathrm{\sim}$3 $\mathrm{\times}$
109 DNA base pairs, just 15% less than human genome. In these cases, the
genomic information capacity is of the order of 1000 factor smaller than
neural information capacity. In the case of humans, with $\mathrm{\sim}$8.6
$\mathrm{\times}$ 10${}^{10\ }$neurons and with $\mathrm{\sim}$1.5
$\mathrm{\times}$ 1014 synaptic connections again the genomic size is about
1000 factor smaller than neural information capacity. This leads the
development of dexterity of hand in the case of cats for hunting, for digging
holes and scavenging for rodents, and for climbing trees in the case tree
shrew and primates. For humans it develops further with the dexterous capacity
to balance on two legs by the age of two and to hold objects with hands to use
tools and to even make tools. Of course, in response to the dexterous
development of the hand, arises the sympathetic development of the larynx
which in turn develops the spoken language along with the stone tool society,
with probably the click sounds as in the sound made by stone tools, like some
of the ancient languages of Africa with a rich rapporteur of phonemes. The
collective syncretic convergence like the primordial music of the Ainur in the
Tolkien’s Silammarlion orchestra emerges the music and art, being the fertile
forerunners for the emergence of culture of Saussurean symbols and their
orchestration known as the language.

## Limitations of human brain

Human brain with 86 billion neurons and 150 trillion synaptic connections with
its corresponding complex society and its interactions, considered to be
potentially self-conscious being, is one of the astonishing miracles of the
universe no doubt. Of course, life itself seems like a miracle in the vast
barren lifeless universe, with self-conscious being as humans a miracle within
a miracle. Despite all these splendid achievements of Darwinian evolution
there remains limitations of the biological evolution. Unlike insects and
birds, humans cannot fly, but he has managed to invent a flying machine that
has overcome such an inability. Likewise, he has limitations of memory, of
retaining long narrations displaced in space and time. The complex web of
society he has developed for its continuous intellectual development requires
such a thread of historical narration. Many devices were invented for
memorization and easy recall like music and tonality in language with rhyme
and reason. Poetry was an ancient device of retention. Story telling was
another stimulating device used to improve recollection and capacity for
analysis. Ancient bards provided such devices like J.R.R. Tolkien’s abstract
stories and other ancient mythical stories expertly analyzed by the
anthropologist Lev Strauss. Once writing developed initially as
graphemes/phonemes and later as alphabets, there arose devices like clay
tablets, stone tablets, palm leaves, leather parchments, and the society was
well on its way to developing libraries recording its narrations displaced in
space and time, like the library of Alexandria and the Nalanda University of
the east. All those wonderful devices for memorization like poetry and rhymes
gave way to plain simple prose like the travelogues of European travelers like
Marco Polo, Vasco Da Gamma, and the Chinese scholars like the Yuvan Tsuang,
stinging ancient insular societies of Asia to adopt prose and modern
scientific developments of the West. The Arabs recovered the scientific
treasures of Greece banished to pagandom by Rome and added to it the decimal
number inventions of India, and with collapse of Roman empire these scientific
ideas transmitted to Europe developed fast producing the modern industrial
age.

The invention of printing by Guttenberg printing press and the books
andmanuscripts printed by mushrooming printing presses suddenly threw open to
a wide audience the historical continuity of inventions and narrations.
Scientific journals and societies helped produce the great inventions of the
modern society along with it the work of numerous laboratories, engineers and
technicians.

## Social extension of human brain – Internet, Wikipedia and the Large
Language Models

Due to limitations of brain, especially due to the lossy nature of the
information that is stored in memory, writing was discovered along with
methods to store it as books and manuscripts in large libraries. This took an
even bigger and effective form with searchable internet and social media
platforms such as Wikipedia.

The CPUs that were designed for low-latency was supplemented by high-
throughput GPUs to cause emergence of Deep Learning Attention-base Large
Language Transformer Models and especially the Generative AI. This extends
andimproves the trainable, scalable and prunable natural neural network of the
human brain.

With constant miniaturization like how raw stone tools of Paleolithic era gave
rise to sophisticated tools of Neolithic era, the initial large
unsophisticated electric diode and triode tubes gave way to Silicon based
semi-conductor transistors with CPU started growing in processing speed
according to Moore’s law that states that number of transistors in an
integrated circuit doubles every two years, so in about 20 years it would have
grown more than billion-fold. Now the 7nm-wide transistors have given way to
the smaller 5nm-wide transistors that are just 10 atoms wide, so atomic-level
quantum decoherence effects have started putting limits to
furtherminiaturization.

So, the development GPU with it’s high-throughput optimization rather than
low-latency optimization has produced much higher overall processing rates
1000 GB/s and more. The Moore’s law has given-way to Huang’s law of GPU
designed for high-throughput (Figure 2). Table 3 shows the stages in CPU/GPU
evolution, while Table 4 illustrates the rapid evolution of NVIDIA GPU cores
for handling the huge data parallelization requirements.

Figure 2: Heung’s law of GPU. Table 3: Stages of CPU/GPU evolution Year | Milestones | References
---|---|---
1970s |  First Micro Processor Introduction (Intel 4004, 8008) | [46]
1980s |  Intel 8088 CPU launched by IBM PC | [47]
1990s |  Intel Pentium series CPUs Introduction | [48]
|  3D Graphics cards like NVIDIA RIVA came into existence | [49]
2000s |  Intel Core series, multi-core CPUs Introduction | [50]
|  GPUs evolution for programmable architectures | [51]
|  GPGPU computing Introduction | [52]
2010s |  Demand for GPU acceleration because of Deep learning | [53]
|  CUDA and OpenCL came into existence for GPU programming | [54]
|  Specialized AI accelerators like TPUs development | [55]
Present |  Rigorous development of multi-core CPUs | [56]
|  To handle AI workloads GPUs with dedicated Tensor cores
development | [57]
Future |  Developments in heterogeneous computing designs | [58]

Evolution Of GPU, CUDA Cores, Memory Interface Width, Memory Bandwidth

Table 4: GPU model each year, CUDA Cores, memory Interface and Bandwidth

Year | Model | CUDA Cores |  Memory Interface Width(bit) |  Memory Bandwidth (GB/s)
---|---|---|---|---
2024 | RTX-4090 | 16384 | 384 | 1008
2023 | RTX-3080 Ti | 10240 | 384 | 912
2022 | RTX-3090 | 10496 | 384 | 936
2021 | RTX-3090 Ti | 10572 | 384 | 936
2020 | RTX-3080 | 8704 | 320 | 760
2019 | RTX-2080 Ti | 4352 | 352 | 616
2018 | RTX-2080 | 2944 | 256 | 448
2017 | GTX 1080 Ti | 3584 | 352 | 484
2016 | GTX 1080 | 2560 | 256 | 320
2015 | GTX 980 Ti | 2816 | 384 | 336.5
2014 | GTX 980 | 2048 | 256 | 224
2013 | GTX 780 Ti | 2880 | 384 | 336
2012 | GTX 690 | 3072 | 512 (256 per GPU) | 384
2011 | GTX 580 | 512 | 384 | 192
2010 | GTX 480 | 480 | 384 | 177.4
2009 | GTX 295 | 480 | 896 | 223.8
2008 | GTX 280 | 240 | 512 | 141.7
2007 | GTX 260 | 192 | 448 | 111.9
2006 | GTX 260 | 192 | 448 | 111.9

Ref: NVidia Graphics Card Specification Chart - Studio 1 Productions and David
Knarr

## Limitations of LLM, human society, and attempts to fix it

It has been well established that LLMs tend to reproduce racial, sex,
religious and geographical biases in the corpus produced by the society with
those biases. The biases also tend to multiply further as a reflection between
the digital access biases of the haves and have-nots, especially between the
North and South. Serious attempts have been made to fix this, especially
Facebook’s attempt at “Constitutional AI”. Reinforcement Learning with
Human/AI feedback have been effective at throttling some of the biases. It is
a trade-off between stifling bias vs. being non-informative in replying “I
cannot reply to that”. LLM’s tendency to reproduce text from the corpus ad-
nauseam, known as “Memorization,” is also a serious problem that leads to
legal violations of “copyright”. This can be fixed by dropping nodes randomly
to avoid over-fitting the data. Hallucination is another problem that is being
addressed by creating RAG implementations that prompt the LLM to follow
factual documents.

Humans are also born with biases and prejudices, and they are also trained by
educators to stifle or even outright ban those biases in campuses. Of course,
it is more important to modify those material conditions that produce those
biases and prejudices rather than fixing them after they have been created in
humans. The corpus text that reflects these biases and prejudices are high-
order effects of these fundamental problems of the society.

## Conclusion

Biological genomic bottleneck of animals was overcome by the development of
specialized brain in mammals and especially the cortex region of the humans
with trillions of synaptic connections enabled by the special evolution of the
homo sapiens with it’s scaled-up version of the primate brain. Many of the
limitations have been overcome by enabler technologies of the modern era,
including the limitations of memory and processing capacity of natural neural
network of humans, with especially the modern development of deep learning
attention-based Large Language Models with hundreds of billions of synaptic
connections that is getting quite close to the 200 trillion connections that
humans are estimated to have. The limitations of low-latency CPUs have been
overcome by the high-throughput GPUs that has circumvented limitations of
further miniaturization beyond the 5nm size by parallel architecture at the
transistor-level, especially for the narrow domain of matrix computations that
seems to dominate the training of huge LLM models.

Many aspects of human neural architecture and social principles are yet to be
mapped to the artificial neural network architecture and design. However, many
of the drawbacks like toxicity, bias, memorization, hallucination etc. have
already been identified with LLM, making interesting parallels with human
society and its collective social intelligence with its toxicity and biases.

## References

  * [1] Arifa Khan, et al., (2024) Social Evolution of Published Text and The Emergence of Artificial Intelligence Through Large Language Models and The Problem of Toxicity and Bias (unpublished)
  * [2] Lars Chittka, Natacha Rossi, Bees learn to dance, Science, 379, 6636, (985-986), (2023).
  * [3] Darwin, C., (1871). The Descent of Man, and Selection in Relation to Sex(London: John Murray).
  * [4] Koulakov, Alexei, et al. ”Encoding innate ability through a genomic bottleneck.” BiorXiv (2021): 2021-03.
  * [5] He K, Lin K, Wang G, Li F. Genome Sizes of Nine Insect Species Determined by Flow Cytometry and k-mer Analysis. Front Physiol. 2016 Nov 24;7:569.doi: 10.3389/fphys.2016.00569. PMID: 27932995; PMCID: PMC5121235.
  * [6] Tomoyasu, Y., Miller, S. C., Tomita, S., & Schoppmeier, M. (2021). Arthropod Neurogenetics: From Neural Development to Circuit Function and Behavior.Genetics, 217(2), iyaa018. [DOI: 10.1093/genetics/iyaa018]
  * [7] Richards, S., Gibbs, R. A., Weinstock, G. M., Brown, S. J., Denell, R., Beeman,R. W., … & Gibbs, R. A. (2008). The genome of the model beetle and pest Tribolium castaneum. Nature, 452(7190), 949-955. [DOI: 10.1038/nature06784]
  * [8] Edde, P. A., & Phillips, T. W. (2011). Adult cannibalism in the confused flour beetle, Tribolium confusum: effects of population density and delayed mating. Journal of Insect Science, 11(1), 27. [DOI: 10.1673/031.011.0127]
  * [9] Hamilton, R. L., & Dill, L. M. (2012). Sexual cannibalism in the wild:Antecedents, current events, and future directions. Journal of Zoology, 287(2), 77-79. [DOI: 10.1111/j.1469-7998.2012.00943.x]
  * [10] Edde, P. A., & Phillips, T. W. (2011). Adult cannibalism in the confused flour beetle, Tribolium confusum: effects of population density and delayed mating. Journal of Insect Science, 11(1), 27. [DOI: 10.1673/031.011.0127]
  * [11] Reinhardt, K., & Siva-Jothy, M. T. (2007). Biology of the Bed Bugs (Cimicidae). Annual Review of Entomology, 52(1), 351–374.
[DOI: 10.1146/annurev.ento.52.040306.133913]

  * [12] Benoit, J. B., Adelman, Z. N., Reinhardt, K., Dolan, A., Poelchau, M., Jennings, E. C., … Richards, S. (2016). Unique features of a global human ectoparasite identified through sequencing of the bed bug genome. Nature Communications, 7, 10165. [DOI: 10.1038/ncomms10165]
  * [13] Doggett, S. L., Dwyer, D. E., Penas, P. F., & Russell, R. C. (2012). Bed bugs: clinical relevance and control options. Clinical Microbiology Reviews, 25(1), 164–192. [DOI: 10.1128/CMR.05015-11]
  * [14] Potter, M. F. (2011). The history of bed bug management—with lessons from the past. American Entomologist, 57(1), 14–25. [DOI: 10.1093/ae/57.1.14]
  * [15] Reinhardt, K., & Siva-Jothy, M. T. (2007). Biology of the Bed Bugs (Cimicidae). Annual Review of Entomology, 52(1), 351–374.
[DOI: 10.1146/annurev.ento.52.040306.133913]

  * [16] Su, C. Y., Menuz, K., & Carlson, J. R. (2009). Olfactory perception: receptors, cells, and circuits. Cell, 139(1), 45-59. [DOI: 10.1016/j.cell.2009.09.015]
  * [17] Scott, J. G., Warren, W. C., Beukeboom, L. W., Bopp, D., Clark, A. G., Giers, S. D., … & Richards, S. (2014). Genome of the house fly, Musca domestica L., a global vector of diseases with adaptations to a septic environment. Genome Biology, 15(10), 466. [DOI: 10.1186/s13059-014-0466-3]
  * [18] Greenberg, B. (1973). Flies and Disease: Biology, Behavior, and Disease Transmission (Vol. 1). Princeton University Press.
  * [19] Miller, J. R., & Robinson, A. S. (1972). Factors influencing longevity of adult house flies in the laboratory. Environmental Entomology, 1(5), 544-548.
[DOI: 10.1093/ee/1.5.544]

  * [20] Greenberg, B. (1973). Flies and Disease: Biology, Behavior, and Disease Transmission (Vol. 1). Princeton University Press.
  * [21] Lovick, J. K., & Bishop, C. D. (2018). Serotonin immunoreactive interneurons in the brain of the larva of the flour beetle, Tribolium castaneum. Journal of Comparative Neurology, 526(5), 757-773. [DOI: 10.1002/cne.24373]
  * [22] Richards, S., Gibbs, R. A., Weinstock, G. M., Brown, S. J., Denell, R., Beeman,R. W., … & Gibbs, R. A. (2008). The genome of the model beetle and pest Tribolium castaneum. Nature, 452(7190), 949-955. [DOI: 10.1038/nature06784]
  * [23] Davies, T. G., Field, L. M., & Williamson, M. S. (2012). The re-emergence of the bed bug as a nuisance pest: implications of resistance to the pyrethroid insecticides. Medical and Veterinary Entomology, 26(3), 241-254.
[DOI: 10.1111/j.1365-2915.2011.00990.x]

  * [24] Greenberg, B. (1973). Flies and Disease: Biology, Behavior, and Disease Transmission (Vol. 1). Princeton University Press.
  * [25] Edde, P. A., & Phillips, T. W. (2011). Adult cannibalism in the confused flour beetle, Tribolium confusum: effects of population density and delayed mating. Journal of Insect Science, 11(1), 27. [DOI: 10.1673/031.011.0127]
  * [26] Lovick, J. K., & Bishop, C. D. (2018). Serotonin immunoreactive interneurons in the brain of the larva of the flour beetle, Tribolium castaneum. Journal of Comparative Neurology, 526(5), 757-773. [DOI: 10.1002/cne.24373]
  * [27] Richards, S., Gibbs, R. A., Weinstock, G. M., Brown, S. J., Denell, R., Beeman,R. W., … & Gibbs, R. A. (2008). The genome of the model beetle and pest Tribolium castaneum. Nature, 452(7190), 949-955. [DOI: 10.1038/nature06784]
  * [28] Drury, D. W., & Wade, M. J. (2007). Genetic variation and reproductive success in Tribolium castaneum: correlation between neutral marker loci and quantitative traits. Journal of Evolutionary Biology, 20(6), 2133-2143.
[DOI: 10.1111/j.1420-9101.2007.01422.x]

  * [29] Miller, J. R., & Robinson, A. S. (1972). Factors influencing longevity of adult house flies in the laboratory. Environmental Entomology, 1(5), 544-548.
[DOI: 10.1093/ee/1.5.544]

  * [30] Drury, D. W., & Wade, M. J. (2007). Genetic variation and reproductive success in Tribolium castaneum: correlation between neutral marker loci and quantitative traits. Journal of Evolutionary Biology, 20(6), 2133-2143.
[DOI: 10.1111/j.1420-9101.2007.01422.x]

  * [31] Brandes, C., Sugawa, M., Menzel, R., & Fiala, A. (2018). Wolf honeybee larvae affect the allocation of foragers among dance followers in a group. Journal of Experimental Biology, 221(2), jeb171355. [DOI: 10.1242/jeb.171355]
  * [32] Wallberg, A., Bunikis, I., Pettersson, O. V., Mosbech, M. B., Childers, A. K., Evans, J. D., … & Webster, M. T. (2019). A hybrid de novo genome assembly of the honeybee, Apis mellifera, with chromosome-length scaffolds. BMC Genomics, 20(1), 275. [DOI: 10.1186/s12864-019-5642-0]
  * [33] Winston, M. L. (1987). The biology of the honey bee. Harvard University Press.
  * [34] Delaplane, K. S., & Mayer, D. F. (2000). Crop pollination by bees. CABI.
  * [35] Winston, M. L. (1987). The biology of the honey bee. Harvard University Press.
  * [36] White, J. G., Southgate, E., Thomson, J. N., & Brenner, S. (1986). The structure of the nervous system of the nematode Caenorhabditis elegans. Philosophical Transactions of the Royal Society of London. B, Biological Sciences, 314(1165), 1–340. [DOI: 10.1098/rstb.1986.0056]
  * [37] C. elegans Sequencing Consortium. (1998). Genome sequence of the nematode C. elegans: a platform for investigating biology. Science, 282(5396), 2012–2018. [DOI: 10.1126/science.282.5396.2012]
  * [38] Félix, M. A., & Braendle, C. (2010). The natural history of Caenorhabditis elegans. Current Biology, 20(22), R965–R969. [DOI: 10.1016/j.cub.2010.09.050]
  * [39] Barrière, A., & Félix, M. A. (2005). High local genetic diversity and low outcrossing rate in Caenorhabditis elegans natural populations. Current Biology, 15(13), 1176–1184. [DOI: 10.1016/j.cub.2005.06.022]
  * [40] Félix, M. A., & Duveau, F. (2012). Population dynamics and habitat sharing of natural populations of Caenorhabditis elegans and C. briggsae. BMC Biology, 10, 59. [DOI: 10.1186/1741-7007-10-59]
  * [41] Technau, G. M., & Heisenberg, M. (1982). Neural reorganization during metamorphosis of the corpora pedunculata in Drosophila melanogaster. Nature,295(5850), 405–407. [DOI: 10.1038/295405a0]
  * [42] Adams, M. D., Celniker, S. E., Holt, R. A., Evans, C. A., Gocayne, J. D., Amanatides, P. G., … & Venter, J. C. (2000). The genome sequence of Drosophila melanogaster. Science, 287(5461), 2185–2195.
[DOI: 10.1126/science.287.5461.2185]

  * [43] Simon, A. F., Chou, M. T., Salazar, E. D., Nicholson, T., Saini, N., Metchev, S., & Krantz, D. E. (2012). A simple assay for investigating social behavior inDrosophila: assessing social space within a group. Genes, Brain and Behavior, 11(2), 243-252. [DOI: 10.1111/j.1601-183x.2011.00747.x]
  * [44] Partridge, L., & Fowler, K. (1993). Responses and correlated responses to artificial selection on thorax length in Drosophila melanogaster. Evolution, 47(1),213–226. [DOI: 10.1111/j.1558-5646.1993.tb01202.x]
  * [45] Bainbridge, S. P., & Bownes, M. (1981). Staging the metamorphosis of Drosophila melanogaster. Journal of Embryology and Experimental Morphology, 66, 57–80.
  * [46] Intel. (n.d.). Intel Corporate Timeline.
https://www.intel.com/content/www/us/en/history/museum-timeline.html

  * [47] IBM. (n.d.). IBM Archives: 1981.
https://www.ibm.com/ibm/history/history/year_1981.html

  * [48] Intel. (n.d.). Intel Pentium Processors.
https://www.intel.com/content/www/us/en/products/processors/pentium-
processors.html

  * [49] NVIDIA. (n.d.). NVIDIA’s History & Milestones.
https://www.nvidia.com/en-us/about-nvidia/nvidia-history/

  * [50] Intel. (n.d.). Intel Core Processors.
https://www.intel.com/content/www/us/en/products/processors/core.html

  * [51] NVIDIA. (n.d.). CUDA GPUs.
https://developer.nvidia.com/cuda-gpus

  * [52] NVIDIA. (n.d.). CUDA Toolkit Documentation.
https://docs.nvidia.com/cuda/

  * [53] LeCun, Y., Bengio, Y., & Hinton, G. (2015). Deep learning. Nature, 521(7553), 436–444. https://doi.org/10.1038/nature14539
  * [54] Khronos Group. (n.d.). OpenCL Overview. https://www.khronos.org/opencl/
  * [55] Google Cloud. (n.d.). Cloud TPUs. Retrieved from
https://cloud.google.com/tpu

  * [56] Intel. (n.d.). Intel Processors.
https://www.intel.com/content/www/us/en/products/processors.html

  * [57] NVIDIA. (n.d.). NVIDIA A100 Tensor Core GPU.
https://www.nvidia.com/en-gb/data-center/a100/

  * [58] AMD. (n.d.). Accelerated Processing Units.
https://www.amd.com/en/technologies/apu


# Compact Speech Translation Models via Discrete Speech Units Pretraining

Tsz Kin Lam  Alexandra Birch  Barry Haddow
School of Informatics, University of Edinburgh
{tlam, a.birch<EMAIL_ADDRESS>

###### Abstract

Using Self-Supervised Learning (SSL) as model initialization is now common to
obtain strong results in Speech Translation (ST). However, they also impose a
large memory footprint, hindering on-device deployment. In this paper, we
leverage the SSL models by pretraining smaller models on their Discrete Speech
Units (DSU). We pretrain encoder-decoder models on 1) Filterbank-to-DSU and 2)
DSU-to-Translation data, and take the encoder from 1) and the decoder from 2)
to initialise a new model, finetuning this on limited speech-translation data.
The final model becomes compact by using the DSU pretraining to distil the
knowledge of the SSL model. Our method has several benefits over using DSU as
model inputs, such as shorter inference pipeline and robustness over (DSU)
tokenization. In contrast to ASR pretraining, it does not require transcripts,
making it applicable to low-resource settings. Evaluation on CoVoST-2 X-En
shows that our method is >$0.5$ BLEU better than a ST model that directly
finetune the SSL model, given only half the model size, and on a par with ASR
pretraining.

Compact Speech Translation Models via Discrete Speech Units Pretraining

Tsz Kin Lam and Alexandra Birch and Barry Haddow School of Informatics,
University of Edinburgh {tlam, a.birch<EMAIL_ADDRESS>

##  1 Introduction

In Speech-to-Text Translation (ST), the use of SSL models, e.g., wav2vec2 and
HuBERT Baevski et al. (2020); Hsu et al. (2021), as model initialization is
now common to obtain SOTA result Agarwal et al. (2023). Nevertheless, using
these bulky models as initialisation imposes a large memory footprint to the
downstream ST, which limits flexibility in modelling choices. In addition, it
hinders on-device deployment that is crucial for privacy and useful in the
absence of internet connection.

Knowledge distillation (KD) Kim and Rush (2016); Inaguma et al. (2021) is the
standard approach to creating smaller models, but obtaining the high-quality
pseudo labels for training ST systems is very costly. Instead, we focus on
Discrete Speech Units (DSU) since they can be obtained cheaper, e.g., without
requiring a translation model, and are an intermediate representation between
speech and text. DSUs are K-means clusters of speech representations from
selected layers within a SSL model. It represents sequences of discrete
tokens, which are easier to model within a text processing architecture Polyak
et al. (2021); Chou et al. (2023). DSU sequences are far smaller than dense
Filterbank sequences, and they can be shortened with removing duplicates and
Byte Pair Encoding (BPE) Sennrich et al. (2016). Therefore, in ST, a
straightforward method to distill the SSL models is to replace the dense
Filterbank by the DSUs, aka the DSU-to-Translation (DSU-to-Trl) model.
Although using DSUs for training allows for transfer learning, using them at
inference requires large and slow models.

To address the above, we use DSU for pretraining rather than as model input to
make ST models more compact. Our method leverages large SSL models by
pretraining smaller models on the corresponding DSU. More specifically, it
pretrains encoder-decoder models on 1) Filterbank-to-DSU (Fbk-to-DSU) and 2)
DSU-to-Trl data, and takes the encoder from 1) and the decoder from 2) to
initialise a new model. After that, the new model is finetuned on limited
speech-translation data. Under this formulation, (1) the model is much more
compact than the SSL model (2) the compact model does not use DSU as inputs
which (a) avoids the lengthy SSL and clustering inference pipeline and (b)
minimizes its sensitivity to (DSU) tokenization. (3) Our method does not
require transcripts, unlike ASR pretraining, making it applicable to low-
resource settings. Similar to ST methods that use pretrained components, our
method could be limited by the _pretraining modality gap_ Liu et al. (2020);
Le et al. (2023). Motivated by prior works, we investigate mitigating it with
Connectionist Temporal Classification (CTC) Graves et al. (2006)
regularisation.

We evaluate our method on CoVoST-2 Wang et al. (2021) X-En (21 language
directions) with multilingual ST. By simply using a monolingual HuBERT model
to extract the DSU, our method has a gain of >$3$ BLEU with respect to
training from scratch, demonstrating DSU’s cross-lingual ability. Our method
is also >$0.5$ BLEU better than a ST model that directly finetuned HuBERT
despite having half model size. Our main contributions are:

  * •

We propose to distill knowledge from large SSL models to make ST models more
compact via DSU pretraining. Our method avoids the lengthy inference pipeline
in the DSU-to-Trl method.

  * •

We examine the effect of (DSU) tokenization on the ST models. Our method is
robust across different tokenizations, which is not the case for the DSU-to-
Trl method.

  * •

We examine applying CTC to the DSU in pretraining and on the translation in
finetuning to mitigate the modality gap, and we show that both improve
performance.

##  2 Related Work

There are a number of related works that use DSU to enhance ST. Fang and Feng
(2023) and Zhang et al. (2023b) use DSU to create more training data in a
back-translation fashion. Chang et al. (2023) and Zhang et al. (2023b) explore
the replacement of Filterbank by DSU as speech input. In contrast, we use DSU
and its translation model for pretraining, resulting in a better Fbk-to-Trl
model that has a shorter inference pipeline.

In the case of pretraining, Wu et al. (2023) use a single Speech-to-DSU model
in pretraining for general speech-to-text purposes whereas we tailor the use
for ST by using a pair of encoder-decoder models. Zhang et al. (2022b) also
decompose ST into speech-to-unit and unit-to-text tasks. Their training is
based on masked unit prediction, and it requires an extra unit-encoder module
in inference. In contrast, we resort to supervised training on the DSU in
acoustic pretraining and require no extra module in inference. More
importantly, our goal is to make (multilingual) ST more compact, aiming also
at low-resource settings where transcripts are not easily available, rather
than learning a joint semantic space for both transcripts and audios.

AST model (#Params) | High | Mid | Low | All
---|---|---|---|---
Scratch (52M) | $19.37$ | $7.91$ | $0.73$ | $5.99$
ASR-Pretraining (52M) | $26.53$ | $12.22$ | $1.82$ | $9.00$
Hu-Transformer (113M) | $24.31$ | $11.42$ | $2.18$ | $8.60$
DSU-Adapter (48M) | $26.54$ | $12.91$ | $1.76$ | $9.13$
Table 1: Results in BLEU on CoVoST-2 X-En. DSU-Adapter is 0.53 points better
than Hu-Transformer that is $2.3$ times larger and on a par with ASR-
Pretraining.

##  3 Method

Our method distills knowledge from the SSL representations via first DSU
pretraining (PT) and then finetuning (FT) on the paired ST data.

In PT, our method uses DSU from an SSL speech model to train a pair of
encoder-decoder models: 1) Fbk-to-DSU and 2) DSU-to-Trl. The Fbk-to-DSU model
takes the log Mel Filterbank as the encoder input and predicts the
corresponding DSU sequence. The model is trained by Maximum Likelihood
Estimation (MLE). In addition, we interpolate between MLE and CTC applied to
the last encoder layer:

$\mathcal{L}^{\text{Fbk-to-
DSU}}=(1-\lambda_{\alpha})\mathcal{L}_{\text{MLE}}(\mathbf{U}|\mathbf{F})+\lambda_{\alpha}\mathcal{L}_{\text{CTC}}(\mathbf{\tilde{U}}|\mathbf{F})$
(1)

where $\mathbf{F}\in\mathbb{R}^{T\text{x}D}$, $\mathbf{U}\in\mathcal{U}$ and
$\mathbf{\tilde{U}}\in\mathcal{\tilde{U}}=\\{\mathcal{U},\emph{blank}\\}$ are
the Fbk, DSU and the CTC label sequences respectively. The DSU-to-Trl model
learns via MLE to predict the translations $\mathbf{Y}$ given $\mathbf{U}$:
$\mathcal{L}^{\text{DSU-to-
Trl}}=\mathcal{L}_{\text{MLE}}(\mathbf{Y}|\mathbf{U})$. In essence, we use the
DSU to bridge the speech and text modalities for ST finetuning.

In FT, we use the encoder of the Fbk-to-DSU model and the decoder (and its
output layer) of the DSU-to-Trl model to initialise the compact model,
followed by finetuning on the paired speech-translation data using both MLE
and CTC loss Gaido et al. (2021); Zhang et al. (2023a):

$\mathcal{L}^{\text{FT}}=(1-\lambda_{\beta})\mathcal{L}_{\text{MLE}}(\mathbf{Y}|\mathbf{F})+\lambda_{\beta}\mathcal{L}_{\text{CTC}}(\mathbf{\tilde{Y}}|\mathbf{F})$
(2)

where
$\mathbf{\tilde{Y}}\in\mathcal{\tilde{Y}}=\\{\mathcal{Y},\emph{blank}\\}$.

#### Tokenization

As mentioned in section 1, we examine the tokenization effect, including using
separate (source/target) vocabularies or not. We denote each DSU token by a
hashtag with an index, e.g,. ’#456’. Following Ao et al. (2022), we reduce the
sequence length via merging the sequential repetitions, e.g, ’#1 #1 #1 #456
#456 #23’ becomes ’#1 #456 #23’. When BPE is applied on the DSU, the
whitespace characters are ignored, e.g., ’#1 #456 #23’ becomes ’#1#456#23’
which could be split into ’#1#45 6#2 3’.

Has BPE | $|\mathcal{V}|$ | DSU | Length | DSU-to-Trl | Enc-Init | EncDec-Init | DSU-Adapter
---|---|---|---|---|---|---|---
on DSU? |  | Length | Ratio |  $20$M to $27$M |  $52$M to $70$M |  $46$M to $64$M |  $48$M to $67$M
No |  $1$K-$8$K | $176$ | $12.9$ | $6.73$ | $7.70$ | $7.87$ | $8.54$
$1$K-$16$K | " | $14.1$ | $6.36$ | $7.50$ | $8.00$ | $8.43$
$1$K-$32$K | " | $14.9$ | $6.30$ | $7.23$ | $7.65$ | $7.94$
$8$K | " | $12.7$ | $6.88$ | $7.64$ | $8.05$ | $8.26$
$16$K | " | $14.0$ | $6.33$ | $7.41$ | $7.91$ | $8.17$
$32$K | " | $14.9$ | $6.26$ | $6.66$ | $7.41$ | $7.68$
Yes |  $1$K-$8$K | $221$ | $16.3$ | $4.52$ | $8.23$ | $8.44$ | $8.61$
$16$K-$8$K | $129$ | $9.5$ | $5.06$ | $8.51$ | $8.76$ | $8.95$
$32$K-$8$K | $115$ | $8.5$ | $4.43$ | $8.67$ | $9.02$ | $9.13$
$8$K | $150$ | $7.6$ | $7.02$ | $8.33$ | $8.51$ | $8.82$
$16$K | $133$ | $7.7$ | $6.50$ | $8.57$ | $8.61$ | $8.93$
$32$K | $118$ | $7.8$ | $5.07$ | $8.30$ | $8.44$ | $8.70$
Table 2: Tokenization effect on 4 types of ST methods. In general, darker
(brighter) cells refer to weaker (stronger) models. The best two models apply
both BPE on the DSU and separate vocabularies in PT (cells in yellow).

##  4 Experiments

###  4.1 Data Preprocessing

We follow standard practices to preprocess the CoVoST-2 X-En data, i.e., log-
Mel Filterbank of $80$ dimensions for speech inputs and BPE with $8$K
vocabulary on the English target, $16$K on the transcripts and $32$K on the
DSU, unless otherwise specified. We use an English HuBERT-Base model to
extract the DSU by first downsampling the CoVoST-2 audio to $16$KHz. Each
audio data utterance is then converted into the DSU, i.e., the clustering
indexes, by applying K-Means clustering (K=$1,000$) on its HuBERT
representation from the 6th layer Lakhotia et al. (2021). We sample training
data from each language pair to form a subset of $98$K instances to train the
K-Means model. Further details are in Appendix A.1.

###  4.2 Model Configuration

All models are based on Transformer Vaswani et al. (2017) with implementations
from fairseq Ott et al. (2019); Wang et al. (2020). In the Fbk-to-
token111Token refers to either transcriptions, DSU, or translations. models,
the encoder has convolutional layers to downsample the Fbk by a factor of 4.
There are 12-6 layers in the transformer encoder-decoder, whereas the
embedding and feed-forward network (FFN) dimensions are 256 and 4,096
respectively, unless otherwise specified. It is worth noting that: (1) The
Fbk-to-DSU model is not trained on the translations, so it is not directly
comparable to the ST models. Its effect to ST lies on its pretrained encoder
(Table 2). (2) The DSU-to-Trl model is a ST model, and it can also be used for
initialization.

Scratch is a ST model trained on the paired speech-translation data without
pretraining.

ASR Pretraining refers to a ST model whose encoder is initialized by a speech
recognition task with CTC regularisation on the transcripts.

DSU-to-Trl follows the Transformer used in text translation. We use 6-6 layers
in the encoder-decoder which the dimension of embedding and FFN is 256 and
2,048 respectively. Despite its smaller model size, its inference requires
storing and calling both HuBERT and K-Means models.

Hu-Transformer uses the entire HuBERT as the speech encoder initialization
Fang and Feng (2023). For comparison to our DSU-Adapter, its subsequent
encoder-decoder also has 1-6 layers.

DSU-Adapter is our proposed method. To better align the two pre-trained
components, we also experiment with adding an extra encoder layer222All
parameters are trainable in ST finetuning. after the pre-trained encoder.
Since its decoder is initialized by the DSU-to-Trl method, its decoder FFN
dimension is $2,048$.

Enc-Init is a ST model that has its encoder initialized by the Fbk-to-DSU
encoder. EncDec-Init is a DSU-Adapter model without the adapter layer.

###  4.3 Training and Inference

We leave the details in Appendix A.3. It is worth noting that we do not use
extra audio data, e.g., Libri-Light Kahn et al. (2020) in pretraining.

##  5 Results and Analysis

We divide the 21 language pairs by resource level into: 1) "High": {ca, de,
es, fr}, 2) "Mid": {fa, it, pt, ru and zh}, 3) "Low": {ar, cy, et, id, ja, lv,
mn, nl, sl, sv, ta, tr} and 4)"All": the 21 languages pairs. We report the
average BLEU over the test sets of each group using sacrebleu Post
(2018)333nrefs:1—case:mixed—eff:no—tok:13a—smooth:exp—version:2.3.1.

Table 1 compares our DSU-Adapter and the baselines. Our DSU-Adapter is $3$
BLEU points (in the group "All") higher than Scratch. Furthermore, it is
better than Hu-Transformer in spite of having half the parameters. For "Mid"
and "High", the improvement is $1.49$ and $2.23$ points respectively, but it
falls short by $0.42$ points for "Low". We also compare to ASR pretraining,
which is not always applicable, e.g., in low-resource setting or perhaps even
in an unwritten language Zhang et al. (2022a). Surprisingly, our adapter is on
a par with it, i.e., $0.13$ points better. The result remains consistent when
measured in chrF Popović (2015) and COMET444WMT22-COMET-DA Rei et al. (2022),
see Appendix A.4.

###  5.1 Tokenization and its Transfer

Table 2 presents the effect of tokenization. The 1st column indicates if BPE
is applied on the DSU, and the 2nd column "$|\mathcal{V}|$" shows the
vocabulary size, and if the model has a joint vocabulary or not in PT. For
example, "$1$K-$8$K" means that we use a vocabulary of size $1$K for DSU and
$8$K for English. These results are for all languages, please see Appendix A.4
for fine grained results.

#### Effect of BPE on DSU-to-Trl

When BPE is not applied on the DSU, the DSU-to-Trl models have a BLEU of
$6.48$ $\pm$ $0.26$ across the 6 vocabulary settings. Despite being smaller
(<$30$M), these models are better than the Scratch model of $5.99$ BLEU.

When BPE is applied, the DSU sequence length becomes a crucial factor. On one
hand, it delivers the best model ("$8$K": $7.02$ BLEU). On the other hand, the
result deteriorates a lot in the other 5 settings, i.e., a BLEU of $5.12$
$\pm$ $0.83$. More importantly, BPE’s correlation with the model performance
is not straightforward, neither is the length ratio. For an example, the
"$32$K" model (DSU length of $118$) is about $2$ BLEU points behind to the
"$8$K" model (DSU length of $150$).

#### DSU-Adapter is more tokenization robust

When BPE is applied on the DSU, DSU-Adapter has a higher BLEU of $8.86$ $\pm$
$0.19$, in contrast to $8.17$ $\pm$ $0.32$. This behaviour is opposed to the
DSU-to-Trl model that has a lower BLEU of $5.54$ $\pm$ $1.07$ (with also
larger variance), in contrast to $6.48$ $\pm$ $0.26$ when BPE is not used. The
better scores with smaller variance show that DSU-Adapter is more tokenization
robust. We see this as a benefit of introducing the DSU (SSL knowledge) via PT
rather than as model inputs. Besides, using separate vocabularies in PT is
preferred since it performs slightly better, and the DSU, which are not needed
in the ST output, would not occupy the target vocabulary.

#### Ablation

Having similar size, DSU-Adapter is better than EncDec-Init and Enc-Init. The
BLEU over their 12 settings is $8.51$ $\pm$ $0.44$, $8.22$ $\pm$ $0.51$, and
$7.99$ $\pm$ $0.61$ respectively. Encoder-initialization seems more crucial
than decoder-initialization, as reflected by the fact that the best DSU-
Adapter comes from a combination with the weakest DSU-to-Trl model of $4.43$
BLEU.

###  5.2 CTC regularisation helps

Has CTC in | High | Mid | Low | All
---|---|---|---|---
DSU PT? | ST FT?
No | No | $25.81$ | $9.91$ | $1.46$ | $8.14$
No | Yes | $26.10$ | $11.35$ | $1.69$ | $8.71$
Yes | No | $25.94$ | $10.82$ | $1.51$ | $8.44$
Yes | Yes | $26.12$ | $11.53$ | $1.73$ | $8.74$
Table 3: Effect of CTC on Fbk-to-DSU PT and/or ST FT to the DSU-Adapter
method.

Table 3555Owing to the large number of configurations, we can experiment on 4
vocabularies only and then report their average. presents an analysis of using
CTC in the Fbk-to-DSU PT and/or in the ST FT using the DSU-Adapter method. We
find that CTC helps on either stage, but the gain is 0.27 BLEU more in
finetuning. Using them jointly still helps, but the gain becomes marginal,
i.e, 0.03 BLEU.

##  6 Conclusion

We propose to distill knowledge from the large SSL model via pretraining on
its DSU to make ST models more compact for on-device deployment. Our compact
model, i.e., the DSU-Adapter, shows stronger result over finetuning the
corresponding (larger) HuBERT in multilingual ST task. In contrast to using
DSU as model input, our method requires less components in inference and shows
stronger robustness over the tokenization. Finally, our method is suitable for
the low-resource setting since it requires no transcripts.

##  7 Limitations

In this work, we propose to distill knowledge from the informative SSL
representations via DSU pretraining to make ST models more compact. In
addition, we also discuss advantages of our method over using DSU as model
inputs. In spite of its noticeable benefits, there are several factors that
are not thoroughly explored:

#### K-Means Clustering

We did not inspect the clustering size (fixed to $1,000$) and the number of
training instances (only fixed to $98,000$) used in training the K-Means
clustering model.

#### Other acoustic encoders

We did not experiment other acoustic encoders, such as conformer Gulati et al.
(2020); Papi et al. (2023) and E-Branchformer Peng et al. (2023). This
stronger encoders should provide further gains for our method since they also
enjoy the benefit of pretraining.

#### Self-Supervised Models

We did not experiment using other layers (only fixed to the 6th layer) from
the HuBERT model nor from stronger SSL models to extract the DSU. These two
factors potentially increase the DSU quality and potentially provide better
improvement for our method while maintaining the same model size.

## Acknowledgements

This work was funded by UK Research and Innovation (UKRI) under the UK
government’s Horizon Europe funding guarantee (grant number 10039436: UTTER).
The computations described in this research were performed using the
Baskerville Tier 2 HPC service (https://www.baskerville.ac.uk/). Baskerville
was funded by the EPSRC and UKRI through the World Class Labs scheme
(EP/T022221/1) and the Digital Research Infrastructure programme
(EP/W032244/1) and is operated by Advanced Research Computing at the
University of Birmingham.

## References

  * Agarwal et al. (2023) Milind Agarwal, Sweta Agrawal, Antonios Anastasopoulos, Luisa Bentivogli, Ondřej Bojar, Claudia Borg, Marine Carpuat, Roldano Cattoni, Mauro Cettolo, Mingda Chen, William Chen, Khalid Choukri, Alexandra Chronopoulou, Anna Currey, Thierry Declerck, Qianqian Dong, Kevin Duh, Yannick Estève, Marcello Federico, Souhir Gahbiche, Barry Haddow, Benjamin Hsu, Phu Mon Htut, Hirofumi Inaguma, Dávid Javorský, John Judge, Yasumasa Kano, Tom Ko, Rishu Kumar, Pengwei Li, Xutai Ma, Prashant Mathur, Evgeny Matusov, Paul McNamee, John P. McCrae, Kenton Murray, Maria Nadejde, Satoshi Nakamura, Matteo Negri, Ha Nguyen, Jan Niehues, Xing Niu, Atul Kr. Ojha, John E. Ortega, Proyag Pal, Juan Pino, Lonneke van der Plas, Peter Polák, Elijah Rippeth, Elizabeth Salesky, Jiatong Shi, Matthias Sperber, Sebastian Stüker, Katsuhito Sudoh, Yun Tang, Brian Thompson, Kevin Tran, Marco Turchi, Alex Waibel, Mingxuan Wang, Shinji Watanabe, and Rodolfo Zevallos. 2023.  FINDINGS OF THE IWSLT 2023 EVALUATION CAMPAIGN.  In _Proceedings of the 20th International Conference on Spoken Language Translation (IWSLT 2023)_ , pages 1–61, Toronto, Canada (in-person and online). Association for Computational Linguistics.
  * Ao et al. (2022) Junyi Ao, Ziqiang Zhang, Long Zhou, Shujie Liu, Haizhou Li, Tom Ko, Lirong Dai, Jinyu Li, Yao Qian, and Furu Wei. 2022.  Pre-Training Transformer Decoder for End-to-End ASR Model with Unpaired Speech Data.  In _Proc. Interspeech 2022_ , pages 2658–2662.
  * Baevski et al. (2020) Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. 2020.  wav2vec 2.0: A framework for self-supervised learning of speech representations.  _Advances in neural information processing systems_ , 33:12449–12460.
  * Chang et al. (2023) Xuankai Chang, Brian Yan, Kwanghee Choi, Jeeweon Jung, Yichen Lu, Soumi Maiti, Roshan Sharma, Jiatong Shi, Jinchuan Tian, Shinji Watanabe, et al. 2023.  Exploring speech recognition, translation, and understanding with discrete speech units: A comparative study.  _arXiv preprint arXiv:2309.15800_.
  * Chou et al. (2023) Ju-Chieh Chou, Chung-Ming Chien, Wei-Ning Hsu, Karen Livescu, Arun Babu, Alexis Conneau, Alexei Baevski, and Michael Auli. 2023.  Toward joint language modeling for speech units and text.  In _Findings of the Association for Computational Linguistics: EMNLP 2023_ , pages 6582–6593, Singapore. Association for Computational Linguistics.
  * Fang and Feng (2023) Qingkai Fang and Yang Feng. 2023.  Back translation for speech-to-text translation without transcripts.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 4567–4587, Toronto, Canada. Association for Computational Linguistics.
  * Gaido et al. (2021) Marco Gaido, Mauro Cettolo, Matteo Negri, and Marco Turchi. 2021.  CTC-based compression for direct speech translation.  In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_ , pages 690–696, Online. Association for Computational Linguistics.
  * Graves et al. (2006) Alex Graves, Santiago Fernández, Faustino J. Gomez, and Jürgen Schmidhuber. 2006.  Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks.  In _Machine Learning, Proceedings of the Twenty-Third International Conference (ICML 2006), Pittsburgh, Pennsylvania, USA, June 25-29, 2006_ , volume 148 of _ACM International Conference Proceeding Series_ , pages 369–376. ACM.
  * Gulati et al. (2020) Anmol Gulati, James Qin, Chung-Cheng Chiu, Niki Parmar, Yu Zhang, Jiahui Yu, Wei Han, Shibo Wang, Zhengdong Zhang, Yonghui Wu, and Ruoming Pang. 2020.  Conformer: Convolution-augmented Transformer for Speech Recognition.  In _Proc. Interspeech 2020_ , pages 5036–5040.
  * Hsu et al. (2021) Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. 2021.  Hubert: Self-supervised speech representation learning by masked prediction of hidden units.  _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , 29:3451–3460.
  * Inaguma et al. (2021) Hirofumi Inaguma, Tatsuya Kawahara, and Shinji Watanabe. 2021.  Source and target bidirectional knowledge distillation for end-to-end speech translation.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 1872–1881, Online. Association for Computational Linguistics.
  * Kahn et al. (2020) Jacob Kahn, Morgane Rivière, Weiyi Zheng, Evgeny Kharitonov, Qiantong Xu, Pierre-Emmanuel Mazaré, Julien Karadayi, Vitaliy Liptchinsky, Ronan Collobert, Christian Fuegen, Tatiana Likhomanenko, Gabriel Synnaeve, Armand Joulin, Abdelrahman Mohamed, and Emmanuel Dupoux. 2020.  Libri-light: A benchmark for ASR with limited or no supervision.  In _2020 IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2020, Barcelona, Spain, May 4-8, 2020_ , pages 7669–7673. IEEE.
  * Kim and Rush (2016) Yoon Kim and Alexander M. Rush. 2016.  Sequence-level knowledge distillation.  In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_ , pages 1317–1327, Austin, Texas. Association for Computational Linguistics.
  * Kingma and Ba (2015) Diederik P. Kingma and Jimmy Ba. 2015.  Adam: A method for stochastic optimization.  In _3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings_.
  * Kudo and Richardson (2018) Taku Kudo and John Richardson. 2018.  SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pages 66–71, Brussels, Belgium. Association for Computational Linguistics.
  * Lakhotia et al. (2021) Kushal Lakhotia, Eugene Kharitonov, Wei-Ning Hsu, Yossi Adi, Adam Polyak, Benjamin Bolte, Tu-Anh Nguyen, Jade Copet, Alexei Baevski, Abdelrahman Mohamed, and Emmanuel Dupoux. 2021.  On generative spoken language modeling from raw audio.  _Transactions of the Association for Computational Linguistics_ , 9:1336–1354.
  * Le et al. (2023) Phuong-Hang Le, Hongyu Gong, Changhan Wang, Juan Pino, Benjamin Lecouteux, and Didier Schwab. 2023.  Pre-training for speech translation: CTC meets optimal transport.  In _Proceedings of the 40th International Conference on Machine Learning_ , volume 202 of _Proceedings of Machine Learning Research_ , pages 18667–18685. PMLR.
  * Liu et al. (2020) Yuchen Liu, Junnan Zhu, Jiajun Zhang, and Chengqing Zong. 2020.  Bridging the modality gap for speech-to-text translation.  _arXiv preprint arXiv:2010.14920_.
  * Nguyen and Salazar (2019) Toan Q. Nguyen and Julian Salazar. 2019.  Transformers without tears: Improving the normalization of self-attention.  In _Proceedings of the 16th International Conference on Spoken Language Translation_ , Hong Kong. Association for Computational Linguistics.
  * Ott et al. (2019) Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019.  fairseq: A fast, extensible toolkit for sequence modeling.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations)_ , pages 48–53, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Papi et al. (2023) Sara Papi, Marco Gaido, Andrea Pilzer, and Matteo Negri. 2023.  When good and reproducible results are a giant with feet of clay: The importance of software quality in nlp.  _arXiv preprint arXiv:2303.16166_.
  * Park et al. (2019) Daniel S. Park, William Chan, Yu Zhang, Chung-Cheng Chiu, Barret Zoph, Ekin D. Cubuk, and Quoc V. Le. 2019.  SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition.  In _Proc. Interspeech 2019_ , pages 2613–2617.
  * Peng et al. (2023) Yifan Peng, Kwangyoun Kim, Felix Wu, Brian Yan, Siddhant Arora, William Chen, Jiyang Tang, Suwon Shon, Prashant Sridhar, and Shinji Watanabe. 2023.  A Comparative Study on E-Branchformer vs Conformer in Speech Recognition, Translation, and Understanding Tasks.  In _Proc. INTERSPEECH 2023_ , pages 2208–2212.
  * Polyak et al. (2021) Adam Polyak, Yossi Adi, Jade Copet, Eugene Kharitonov, Kushal Lakhotia, Wei-Ning Hsu, Abdelrahman Mohamed, and Emmanuel Dupoux. 2021.  Speech Resynthesis from Discrete Disentangled Self-Supervised Representations.  In _Proc. Interspeech 2021_ , pages 3615–3619.
  * Popović (2015) Maja Popović. 2015.  chrF: character n-gram F-score for automatic MT evaluation.  In _Proceedings of the Tenth Workshop on Statistical Machine Translation_ , pages 392–395, Lisbon, Portugal. Association for Computational Linguistics.
  * Post (2018) Matt Post. 2018.  A call for clarity in reporting BLEU scores.  In _Proceedings of the Third Conference on Machine Translation: Research Papers_ , pages 186–191, Brussels, Belgium. Association for Computational Linguistics.
  * Rei et al. (2022) Ricardo Rei, José G. C. de Souza, Duarte Alves, Chrysoula Zerva, Ana C Farinha, Taisiya Glushkova, Alon Lavie, Luisa Coheur, and André F. T. Martins. 2022.  COMET-22: Unbabel-IST 2022 submission for the metrics shared task.  In _Proceedings of the Seventh Conference on Machine Translation (WMT)_ , pages 578–585, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.
  * Sennrich et al. (2016) Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016.  Neural machine translation of rare words with subword units.  In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1715–1725, Berlin, Germany. Association for Computational Linguistics.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  _Advances in neural information processing systems_ , 30.
  * Wang et al. (2020) Changhan Wang, Yun Tang, Xutai Ma, Anne Wu, Dmytro Okhonko, and Juan Pino. 2020.  Fairseq S2T: Fast speech-to-text modeling with fairseq.  In _Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing: System Demonstrations_ , pages 33–39, Suzhou, China. Association for Computational Linguistics.
  * Wang et al. (2021) Changhan Wang, Anne Wu, Jiatao Gu, and Juan Pino. 2021.  CoVoST 2 and Massively Multilingual Speech Translation.  In _Proc. Interspeech 2021_ , pages 2247–2251.
  * Wu et al. (2023) Felix Wu, Kwangyoun Kim, Shinji Watanabe, Kyu Jeong Han, Ryan McDonald, Kilian Q. Weinberger, and Yoav Artzi. 2023.  Wav2seq: Pre-training speech-to-text encoder-decoder models using pseudo languages.  In _IEEE International Conference on Acoustics, Speech and Signal Processing ICASSP 2023, Rhodes Island, Greece, June 4-10, 2023_ , pages 1–5. IEEE.
  * Zhang et al. (2022a) Biao Zhang, Barry Haddow, and Rico Sennrich. 2022a.  Revisiting end-to-end speech-to-text translation from scratch.  In _International Conference on Machine Learning_ , pages 26193–26205. PMLR.
  * Zhang et al. (2023a) Biao Zhang, Barry Haddow, and Rico Sennrich. 2023a.  Efficient CTC regularization via coarse labels for end-to-end speech translation.  In _Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics_ , pages 2264–2276, Dubrovnik, Croatia. Association for Computational Linguistics.
  * Zhang et al. (2023b) Dong Zhang, Rong Ye, Tom Ko, Mingxuan Wang, and Yaqian Zhou. 2023b.  DUB: Discrete unit back-translation for speech translation.  In _Findings of the Association for Computational Linguistics: ACL 2023_ , pages 7147–7164, Toronto, Canada. Association for Computational Linguistics.
  * Zhang et al. (2022b) Ziqiang Zhang, Long Zhou, Junyi Ao, Shujie Liu, Lirong Dai, Jinyu Li, and Furu Wei. 2022b.  SpeechUT: Bridging speech and text with hidden-unit for encoder-decoder based speech-text pre-training.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 1663–1676, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.

##  Appendix A Appendix

###  A.1 Data preprocessing

#### log Mel Filterbank

The 80-D log Mel Filterbank is computed every 10ms with a 25ms window,
followed by feature normalisation using statistics, i.e., mean and variance,
computed over each channel.

#### BPE

We use the BPE implementation from sentencepiece Kudo and Richardson (2018)
and obtain vocabulary of size 8K on the English target, 16K on the transcripts
and 32K on the DSU, unless otherwise specified.

#### HuBERT

We use
hubert_base_ls960.pt666https://github.com/facebookresearch/fairseq/tree/main/examples/textless_nlp/gslm/speech2unit.

#### K-Means Clustering

We use MiniBatchKMeans from sklearn with a clustering size of $1,000$ and
default settings in other hyper-parameters. When training the clustering
model, we divide the 21 language pairs into three groups: 1) {ar, cy, et, id,
ja, lv, mn, sl, sv, ta, tr}, 2) {nl, pt, ru, zh} and 3) {ca, de, es, fa, fr,
it} for sampling. For each language pair in group 1), we sample $1$K
instances, whereas it becomes $3$K in group 2) and $12.5$K in group 3),
resulting in a total of $98$K instances.

###  A.2 Model Configuration

We use "pre" layer-normalization Nguyen and Salazar (2019) in DSU-to-Trl.

###  A.3 Training and Inference

In training,

  * •

we skip training data that are more than $3$K frames for Filterbank, $320$K
frames for HuBERT input or $1,024$ tokens.

  * •

we apply SpecAugment Park et al. (2019) with parameters:
{$F=30,T=40,m_{F}=2,m_{T}=2$} on Filterbank inputs.

  * •

we share the embedding weights when using a joint vocabulary in the DSU-to-Trl
model.

  * •

we set $\lambda_{\alpha}$ and $\lambda_{\beta}$ in CTC to $0.3$ and the
smoothing parameter in LS to $0.1$

  * •

we initialize the encoder (decoder) with the last (best) checkpoint from the
PT model.

All models are optimized by Adam Kingma and Ba (2015) with inverse square root
scheduler. For simplicity, we provide major training parameters for each model
in the format of {effective batch size, warm-up steps, peak learning rate,
maximum training steps}:

#### Fbk-to-Token:

{32K frames, 25K, 2e-3, 60K}

#### DSU-to-Trl:

{80K tokens, 10K, 5e-4, 50K}

#### Hu-Transformer:

{4M frames, 4K, 1e-4, 300K}

In inference, we average the last 5 checkpoints and use beam size of 5 in
generation. All experiments are run on Nvidia A100 GPUs. It takes about 1 day
for 2 A100 (40GB) GPUs to complete an experiment that uses Filterbank as
inputs.

###  A.4 Results and Analysis

In addition to BLEU (see Table 8 for the result of each language), we also
provide the result of Table 1 using
chrF777nrefs:1—case:mixed—eff:yes—nc:6—nw:0—space:no—version:2.3.1 and COMET.
In order to compute the COMET score, we feed the gold-reference transcripts as
the source sentence.

Table 4 and Table 6 show the average result in chrF and COMET respectively
over the four groups defined in section 5. Their results follow the findings
in BLEU: the DSU-Adapter is better than Hu-Transformer and on a par with ASR
pretraining. Further details, such as training data size (in terms of the
number of utterances rather than hours888”zh” has 10 hours of audio data,
whereas ”nl” has only 7 hours. Therefore, ”zh” is classified into ”Mid”
resource group even though it has slightly less number of utterances than
”nl”.) and per language performance, are shown on Table 5 for chrF and Table 7
for COMET.

#### CTC Regularisation

On Table 3, we average the results over this four configurations: 1) "No-BPE
1K-8K", 2) "BPE 8K", 3) "BPE 32K" and 4) "BPE 32K-8K".

AST model | Transcripts | Model size | High | Mid | Low | All
---|---|---|---|---|---|---
Scratch | No | 52M | $43.58$ | $27.22$ | $14.62$ | $23.10$
ASR Pretraining | Yes | 52M | $51.90$ | $32.82$ | $16.38$ | $27.10$
Hu-Transformer | No | 113M | $49.88$ | $31.94$ | $16.96$ | $26.80$
DSU-Adapter | No | 48M | $52.08$ | $33.94$ | $16.46$ | $27.40$
Table 4: Results of Table 1 in chrF score by resource group. Similar to the findings in BLEU, DSU-Adapter is far better than Hu-Transformer, in particular for "High" and "Mid", and on a par with ASR Pretraining. The best result in each group is denoted by ’_’. Languages | #Training data | Resources | Scratch | ASR-PT | Hu-Transformer | DSU-Adapter
---|---|---|---|---|---|---
ca | 95,854 | High | $42.5$ | $50.9$ | $49.8$ | $51.5$
de | 127,824 | $35.4$ | $45.1$ | $43.6$ | $45.4$
es | 79,013 | $48.1$ | $56$ | $52.8$ | $55.7$
fr | 207,372 | $48.3$ | $55.6$ | $53.3$ | $55.7$
fa | 53,949 | Mid | $15.6$ | $16.9$ | $19.9$ | $16.9$
it | 31,698 | $40.6$ | $50.2$ | $46.4$ | $50.3$
pt | 9,158 | $25$ | $31.3$ | $33.8$ | $32.3$
ru | 12,112 | $35.6$ | $42.6$ | $40$ | $47.7$
zh | 7,085 | $19.3$ | $23.1$ | $19.6$ | $22.5$
ar | 2,283 | Low | $12.5$ | $14.4$ | $13.8$ | $13.2$
cy | 1,241 | $14.5$ | $16.3$ | $15.8$ | $15.4$
et | 1,782 | $17.9$ | $21.3$ | $19$ | $19.9$
id | 1,243 | $13.8$ | $14.2$ | $14.9$ | $14.5$
ja | 1,119 | $14$ | $14$ | $13.7$ | $14.4$
lv | 2,337 | $13.9$ | $14.5$ | $15.9$ | $15.4$
mn | 2,067 | $14.5$ | $15.4$ | $14.3$ | $14.8$
nl | 7,108 | $17.8$ | $24.3$ | $30.8$ | $27$
sl | 1,843 | $13.9$ | $15.4$ | $15.3$ | $15.4$
sv | 2,160 | $13.2$ | $13.8$ | $17.3$ | $14.5$
ta | 1,358 | $12$ | $11.9$ | $12$ | $12.3$
tr | 3,966 | $17.4$ | $21.1$ | $20.7$ | $20.7$
Table 5: Results of Table 1 in chrF score for each language pair. The best
result in each group is denoted by ’_’.

AST model | Transcripts | Model size | High | Mid | Low | All
---|---|---|---|---|---|---
Scratch | No | 52M | $0.605$ | $0.498$ | $0.433$ | $0.481$
ASR Pretraining | Yes | 52M | $0.680$ | $0.537$ | $0.443$ | $0.511$
Hu-Transformer | No | 113M | $0.650$ | $0.522$ | $0.439$ | $0.499$
DSU-Adapter | No | 48M | $0.681$ | $0.548$ | $0.442$ | $0.513$
Table 6: Results of Table 1 in COMET score by resource group. Similar to the findings in BLEU, DSU-Adapter is far better than Hu-Transformer, in particular for "High" and "Mid", and on a par with ASR Pretraining. The best result in each group is denoted by ’_’.  Languages | #Training data | Resources | Scratch | ASR-PT | Hu-Transformer | DSU-Adapter
---|---|---|---|---|---|---
ca | 95,854 | High | $0.607$ | $0.681$ | $0.654$ | $0.684$
de | 127,824 | $0.535$ | $0.608$ | $0.588$ | $0.613$
es | 79,013 | $0.620$ | $0.696$ | $0.654$ | $0.692$
fr | 207,372 | $0.660$ | $0.736$ | $0.704$ | $0.735$
fa | 53,949 | Mid | $0.479$ | $0.494$ | $0.501$ | $0.493$
it | 31,698 | $0.557$ | $0.640$ | $0.594$ | $0.641$
pt | 9,158 | $0.468$ | $0.497$ | $0.510$ | $0.508$
ru | 12,112 | $0.551$ | $0.580$ | $0.563$ | $0.630$
zh | 7,085 | $0.434$ | $0.472$ | $0.440$ | $0.467$
ar | 2,283 | Low | $0.438$ | $0.465$ | $0.45$ | $0.445$
cy | 1,241 | $0.420$ | $0.448$ | $0.424$ | $0.426$
et | 1,782 | $0.413$ | $0.432$ | $0.407$ | $0.423$
id | 1,243 | $0.420$ | $0.406$ | $0.416$ | $0.412$
ja | 1,119 | $0.417$ | $0.412$ | $0.411$ | $0.422$
lv | 2,337 | $0.457$ | $0.453$ | $0.465$ | $0.461$
mn | 2,067 | $0.429$ | $0.441$ | $0.419$ | $0.437$
nl | 7,108 | $0.427$ | $0.460$ | $0.483$ | 0.477
sl | 1,843 | $0.430$ | $0.428$ | $0.433$ | $0.428$
sv | 2,160 | $0.454$ | $0.448$ | $0.472$ | $0.454$
ta | 1,358 | $0.413$ | $0.406$ | $0.403$ | $0.411$
tr | 3,966 | $0.480$ | $0.519$ | $0.484$ | $0.514$
Table 7: Results of Table 1 in COMET score for each language pair. The best result in each group is denoted by ’_’. Languages | #Training data | Resources | Scratch | ASR-PT | Hu-Transformer | DSU-Adapter
---|---|---|---|---|---|---
ca | 95,854 | High | $18.24$ | $25.66$ | $24.24$ | $25.98$
de | 127,824 | $13.97$ | $21.06$ | $19.28$ | $21.32$
es | 79,013 | $22.37$ | $29.8$ | $26.64$ | $29.34$
fr | 207,372 | $22.91$ | $29.6$ | $27.09$ | $29.52$
fa | 53,949 | Mid | $1.9$ | $3.07$ | $4.63$ | $2.84$
it | 31,698 | $15.1$ | $23.55$ | $19.98$ | $23.36$
pt | 9,158 | $7.01$ | $11.2$ | $13.13$ | $11.69$
ru | 12,112 | $11.97$ | $16.87$ | $14.95$ | $21$
zh | 7,085 | $3.58$ | $6.39$ | $4.4$ | $5.67$
ar | 2,283 | Low | $0.44$ | $1.96$ | $1.26$ | $0.48$
cy | 1,241 | $0.64$ | $1.24$ | $1.21$ | $0.72$
et | 1,782 | $0.49$ | $1.46$ | $0.61$ | $0.93$
id | 1,243 | $0.31$ | $0.67$ | $1.27$ | $0.79$
ja | 1,119 | $0.27$ | $0.67$ | $0.36$ | $0.62$
lv | 2,337 | $1.22$ | $1.65$ | $2.63$ | $2.06$
mn | 2,067 | $0.16$ | $0.44$ | $0.31$ | $0.19$
nl | 7,108 | $2.14$ | $6.95$ | $10.75$ | $8.46$
sl | 1,843 | $0.41$ | $1.83$ | $1.23$ | $1.81$
sv | 2,160 | $0.9$ | $1.28$ | $3.4$ | $1.76$
ta | 1,358 | $0.14$ | $0.14$ | $0.13$ | $0.12$
tr | 3,966 | $1.67$ | $3.5$ | $3.04$ | $3.14$
Table 8: Results of Table 1 in BLEU score for each language pair. The best
result in each group is denoted by ’_’.