# Enhancing Cooperation through Selective Interaction and Long-term
Experiences in Multi-Agent Reinforcement Learning

Tianyu Ren1 Xiao-Jun Zeng1 1 University of Manchester {tianyu.ren,
<EMAIL_ADDRESS>
###### Abstract

The significance of network structures in promoting group cooperation within
social dilemmas has been widely recognized. Prior studies attribute this
facilitation to the assortment of strategies driven by spatial interactions.
Although reinforcement learning has been employed to investigate the impact of
dynamic interaction on the evolution of cooperation, there remains a lack of
understanding about how agents develop neighbour selection behaviours and the
formation of strategic assortment within an explicit interaction structure. To
address this, our study introduces a computational framework based on multi-
agent reinforcement learning in the spatial Prisoner’s Dilemma game. This
framework allows agents to select dilemma strategies and interacting
neighbours based on their long-term experiences, differing from existing
research that relies on preset social norms or external incentives. By
modelling each agent using two distinct Q-networks, we disentangle the
coevolutionary dynamics between cooperation and interaction. The results
indicate that long-term experience enables agents to develop the ability to
identify non-cooperative neighbours and exhibit a preference for interaction
with cooperative ones. This emergent self-organizing behaviour leads to the
clustering of agents with similar strategies, thereby increasing network
reciprocity and enhancing group cooperation.

##  1 Introduction

††Proceedings of the Thirty-Third International Joint Conference on Artificial
Intelligence (IJCAI-24)

The emergence of cooperation is fundamental to human civilization and evident
in various biological and multi-agent systems Rand and Nowak (2013); Kramár et
al. (2022). However, maintaining cooperation in social dilemmas is challenging
due to the conflict between individual interests and collective welfare
Sigmund (2010). A key focus in cooperation research involves understanding the
conditions under which individuals favour altruism over self-interest, with
reciprocity identified as a crucial element. In this context, reciprocity is
examined through repeated interactions limited to direct neighbours due to
physical or social constraints, where the behaviours of individuals are shaped
by the actions of their counterparts Perc et al. (2013). Studies using
evolutionary game theory (EGT) demonstrate that network structure can promote
altruistic behaviour through spatial reciprocity, clustering similar
strategies and reducing exploitation by free-riders Santos et al. (2006); Wang
et al. (2013). Furthermore, considering individuals’ capability to interact
with neighbours selectively, recent research emphasizes the significance of
dynamic interaction mechanisms in the evolution of cooperation Su et al.
(2022); Rand et al. (2011); Sylwester and Roberts (2013). Such neighbour
selection behaviours notably transform learning dynamics, thereby allowing
agents to modify their interaction structures in response to evolving
cooperative scenarios.

Despite advancements, explanations for the spontaneous emergence of
cooperative behaviours and selective interactions remain limited. Traditional
game theory and EGT emphasize social learning, where agents imitate nearby
successful strategies Sigmund et al. (2010); Li et al. (2016), overlooking
learning through trial-and-error Metcalfe (2017). Additionally, empirical
studies employing experience-based learning face challenges in developing
enduring strategies, particularly due to the complexities encountered in large
population iterations McKee et al. (2023). To unravel the intricate
coevolutionary dynamics of agent behaviours, there is an increasing interest
in leveraging advanced deep reinforcement learning (RL) algorithms Perolat et
al. (2017); Du et al. (2023); Willis et al. (2023). These methods are not only
used to examine agents’ decision-making processes but also to investigate the
emergence of their behaviours Köster et al. (2022). However, RL agents
typically optimize personal policies, which may lead to a reduction in global
optimization Lowe et al. (2017). Although studies introducing mechanisms like
reputation Anastassacos et al. (2020) and moral rewards Tennant et al. (2023)
to address these issues, these specialized approaches exhibit restricted
applicability. Therefore, it is crucial to have a deeper understanding of the
learning dynamics of agent behaviours and encourage cooperation.

In this study, we construct a computational model using deep Q-learning (DQN)
Mnih et al. (2015) to explore how agents can simultaneously learn both
interaction and dilemma strategies from their long-term experiences. These
agents, modelled as artificial neural networks, learn behavioural policies and
obtain rewards in a spatial Prisoner’s Dilemma Game (PDG) setting within a
multi-agent reinforcement learning (MARL) environment. Unlike previous studies
that depended on predefined social norms or explicit external incentives, our
approach highlights the significance of temporal factors and historical
information in influencing agent decision-making. Initially, agents have no
prior knowledge regarding the actions or game states, hindering their ability
to assess the consequences of their actions and respond effectively.
Throughout the training process, they must learn the causality between their
actions, observations, and rewards from local environment observation. To aid
this learning process, we introduce a utility function that integrates self-
learning and social learning, reflecting the interplay of personal preferences
and the influence of others in shaping human behaviour.

The experimental results demonstrate that RL agents trained in our framework
effectively differentiate between cooperative neighbours and those who are
free-riders. Their preference for building connections with cooperators
bolsters network reciprocity, contributing to the formation of strategy
clusters in network-structured populations. This finding aligns with existing
EGT research, emphasizing the importance of strategy assortment in promoting
cooperation. Moreover, the trained agents achieve superior cooperation levels
and greater average payoff compared to the EGT baseline. Further, we observe
that increased efficiency in learning is correlated with the length of memory
experiences. For a detailed comparison between our RL model and EGT
approaches, refer to the Supplementary Information (SI)
LABEL:Supp:Hyperparameters.

Our work offers three key contributions. Firstly, it reveals the
coevolutionary dynamics of cooperation and interaction strategies within a
spatial PDG framework, demonstrating that RL agents can learn effective
interaction mechanisms to enhance network reciprocity and cooperation.
Secondly, it sheds light on how extensive long-term experiences positively
influence group cooperation. Finally, the MARL training environment we
developed sets the stage for future explorations into diverse aspects of pro-
social cooperative behaviour.

##  2 Related Works

###  2.1 Evolutionary Game Theory

EGT is crucial for exploring the evolution of cooperation among self-
interested individuals. It expands on traditional game theory by considering
extended interactions and strategy dynamics, exploring the emergence and
stability of cooperative behaviours. Notably, Nowak Nowak (2006) identifies
five mechanisms central to understanding cooperation evolution. One vital
aspect noted is the emergence of cooperation on network structures, wherein
individuals predominantly interact with their immediate neighbours Perc et al.
(2017).

Inspired by the dynamic nature of social interactions, numerous studies have
explored the coevolutionary dynamics of cooperation by integrating strategy
evolution with network changes. In this domain, the concept of network
assortativity has been highlighted, revealing that agents with similar
strategies often connect, thereby boosting group cooperation Tanimoto (2013);
Ren and Zheng (2021). Research like Su et al. Su et al. (2022) explores how
cooperative strategies evolve and spread, particularly in unidirectional
interactions, shedding light on shaping social interactions to promote
cooperation. However, these often neglect the self-learning aspect of agents,
which is crucial in real-life where directly copying strategies is
impractical. To address this gap and accurately depict cooperation emergence,
we applied the MARL framework, enabling agents to learn both dilemma and
interaction strategies through environmental observation independently.

###  2.2 Human Experiments

While EGT is essential for examining evolutionary trajectories and conditions
favouring cooperation, incorporating empirical data brings psychological
nuances to these models Köbis et al. (2019), which focus on imposed
interaction structures and psychological mechanisms beyond laboratory settings
Rand and Nowak (2013). Surprisingly, behavioural studies indicate that
participants in structured settings tend to randomly change strategies instead
of copying higher-payoff neighbours Traulsen et al. (2010), disrupting the
clustering process and rendering cooperation less advantageous. Addressing
this, Rand et al. Rand et al. (2011) found that dynamic interactions enhance
multilateral cooperation by encouraging links with cooperators over defectors,
promoting strategy clustering. However, such laboratory experiments often
encounter challenges in terms of scalability and struggle with complex,
larger-scale, or long-term scenarios Moffatt et al. (2009).

###  2.3 Multi-agent Reinforcement Learning

Recent RL applications also focus on understanding the emergence of
cooperation by integrating spatial and temporal dynamics relevant to realistic
scenarios Ren and Zeng (2023); Vinitsky et al. (2023); Tennant et al. (2023),
moving beyond traditional matrix games through the fusion of complex incentive
structures Leibo et al. (2017); Jaques et al. (2019); Abeywickrama et al.
(2023). Studies have applied the intrinsic trial-and-error learning
characteristic of RL to reformulate agent interaction strategies. For
instance, Anastassacos et al. Anastassacos et al. (2020) demonstrate that RL
agents can learn an interaction strategy akin to Tit-for-Tat when partner
selection is present, aiding in the maintenance of cooperation. Meanwhile,
McKee et al. McKee et al. (2023) used a graph neural network-based agent as a
social planner, demonstrating the ability of deep RL to foster coordination
and cooperation in a group.

Concurrently with our work, Ueshima Ueshima et al. (2023) employed two
distinct Q-networks for each agent, differentiating interaction and dilemma
strategies. However, our approach varies as follows: (1) we extend the model
to allow each agent to interact with four potential neighbours, unlike their
paired interaction focus; (2) we incorporate environments with an explicit
interaction structure, considering network reciprocity; (3) they consider
single-round observation input, while we take into account the agents’ long-
term experiences.

##  3 Background

###  3.1 Prisoner’s Dilemma Game

The PDG is a fundamental paradigm in EGT Rapoport et al. (1965), representing
a typical decision-making scenario where agents balance individual benefits
against collective welfare. Fundamentally, the PDG is characterized as a
symmetric matrix game, representing interactions between pairs of individuals
within a population. Each participant faces a choice: to cooperate ($C$),
incurring a cost $c$ while providing a benefit $b$ to others, or to defect
($D$), avoiding the cost while exploiting those who cooperate (with $b>c>0$).
The corresponding payoff matrix can be summarized as

$\mathcal{M}_{p}=\left[\begin{array}[]{ccc}R&S\\\ T&P\\\ \end{array}\right]$
(1)

where mutual cooperation yields a reward $R=b-c$, while mutual defection
result in $P=0$. Unilateral cooperation against a defector incurs a cost
$S=c$, wherease the defector gains $T=b$. This payoff matrix forms four
classical game structures Wang et al. (2015): PD, chicken, harmony and stag
hunt, each defined by specific payoff inequalities. In the PD, the conditions
$T>R>P>S$ and $2R>T+S$ hold. Our model employs a weak PDG with $T=b$ $(1\leq
b\leq 2)$, $R=1$, and $P=S=0$ Nowak and May (1993). Here, the parameter $b$
directly assesses the strength of the dilemma, given $c=0$. In a single-shot
PDG, defection is the dominant strategy, leading to a Nash Equilibrium of
defection, despite cooperation could yield a Pareto improvement. Considering
that players often engage in repeated iterations with the same counterparts,
the conventional PD can be extended to the iterated prisoner’s dilemma (IPD)
format. In this work, the dilemma strategy of agent $i$ at timestep $t$ is
represented by a two-dimensional unit vector $a_{d_{i}}(t)$, with
$a_{d_{i}}=[1,0]^{T}$ indicating cooperation and $a_{d_{i}}=[0,1]^{T}$
signifying defection.

###  3.2 MARL Markov Game

Within MARL, the IPD is conceptualized as a multi-agent extension of Markov
decision processes (MDPs), termed partially observable general-sum Markov
games Littman (1994). Here, agents have observations limited to their local
environment. Formally, a $N$-player MDP is defined by the tuple
$\mathcal{M}=\langle\mathcal{S},\mathcal{\\{A}_{i}\\}_{i\in{\mathcal{N}}},\mathcal{T},\gamma,\mathcal{R}\rangle$,
where $\mathcal{S}$ is a set of joint states for all agents,
$\mathcal{A}_{1},\dots,\mathcal{A}_{N}$ represent joint actions, and
$\mathcal{R}$ is the reward function. The function
$\mathcal{O}:\mathcal{S}\times\\{1\dots,N\\}\to\mathbb{R}^{d}$ maps each
player’s $d$-dimensional view of the state space. In a given state, each agent
$i$ selects an action from $\mathcal{A}_{i}$, and the dynamics of MDP are
determined by the stochastic transition function
$\mathcal{T}:\mathcal{S}\times\mathcal{A}_{1}\times\dots\times\mathcal{A}_{N}\to\triangle(\mathcal{S})$,
where $\triangle(\mathcal{S})$ represents the set of discrete probability
distributions over $\mathcal{S}$. The objective of each agent $i$ is to learn
a policy $\pi_{i}:\mathcal{O}_{i}\to\triangle(A_{i})$ to maximize its
extrinsic reward $r_{i}(s,a^{1},\dots,a~{}^{N})$ based on the agent’s own
observation, simplified as $\pi(a^{i}|o^{i})$. This optimization is conducted
while adhering to the joint policy of all agents. The long-term
$\gamma$-discounted payoff for agent $i$ under the joint policy
$\overrightarrow{\pi}=(\pi_{1},\dots,\pi_{N})$ from an initial state $s_{0}$
can be defined as:

$V_{i}^{\overrightarrow{\pi}}(s_{0})=\mathbb{E}_{\overrightarrow{a_{t}}\sim\overrightarrow{\pi}(\mathcal{O}(s_{t})),s_{t+1}\sim\mathcal{T}(s_{a},\overrightarrow{a}_{t})}[\sum^{T}_{t=0}\gamma^{t}r_{i}(s_{t},\overrightarrow{a}_{t})]$
(2)

where $\gamma\in[0,1]$ represents the temporal discount factor, and $T$
denotes the time horizon. Policies are optimized using trial-and-error
interactions within the MARL environment to maximise cumulative long-term
rewards.

###  3.3 Deep Q-Network

As an extension of Q-learning, DQN stands out as one of the most popular off-
policy Deep RL algorithms, which utilizes an independent deep neural network
to estimate Q-values Mnih et al. (2015). Departing from the traditional
tabular representation for Q-values of state-action pairs, DQN utilizes a
parametrized Q-function $Q_{\theta}(s,a)$ to approximate Q-values. Each
Q-network is parameterized by $\theta$, representing the weights of the neural
network. This approach incorporates the utilization of a replay memory buffer
$\mathcal{D}$ to store past experiences and a target Q function $\overline{Q}$
to mitigate the risk of overestimating Q-values. The learning process for the
optimal action-value function $Q^{*}$ involves minimizing the loss.

$\mathcal{L}_{\theta}=\mathbb{E}_{(s,a,r,s^{\prime})\sim\mathcal{D}}[(r+\gamma\max_{a^{\prime}}\overline{Q}(s^{\prime},a^{\prime})-Q(s,a))^{2}].$
(3)

DQN can be directly extended to a multi-agent setting by having each agent $i$
learn an independent Q function denoted as
$Q_{i}:\mathcal{O}_{i}\times\mathcal{A}_{i}\rightarrow\mathbb{R}$. In line
with the traditional Q-learning approach, DQN also adopts an $\epsilon$-greedy
policy to promote exploration. The policy for the $i$-th agent is
parameterized as:

$\pi_{i}(s)=\begin{cases}\arg\max_{a_{i}\in\mathcal{A}_{i}}Q_{i}(s,a)&\text{with
probability}\ 1-\epsilon\\\ \mathcal{U}(\mathcal{A}_{i})&\text{with
probability}\ \epsilon\end{cases}$ (4)

where $\mathcal{U}(\mathcal{A}_{i})$ signifies a sample drawn from the uniform
distribution over the action space $\mathcal{A}_{i}$.

##  4 Methodology

This model employs value-based optimization utilizing the DQN approach within
a MARL environment. Throughout the training, agents interact with neighbours,
exhibiting cooperative or defective behaviours across various episodes. As
illustrated in Fig. 1, each agent $i$ aims to learn a joint policy $\pi_{i}$
concerning dilemmas and selection actions, informed by their local
observations and long-term experiences. A comprehensive explanation of our
methodology follows. 111Code: https://github.com/itstyren/InteractionMARL-Coop

Figure 1: Training Framework for Developing Dilemma and Interaction
Strategies. Each iteration involves agent $i$ choosing dilemma strategy and
selecting neighbouring agents for PDG engagement. Each agent uses two
Q-networks: the dilemma policy network, which processes long-term actions in
dilemmas by the agent and its neighbours, and the interaction selection
network, which assesses neighbours’ dilemma actions alongside the agent’s
previous interactions. The agent calculates the utility of its actions based
on the rewards accumulated from past encounters.

###  4.1 Game Environment

Agents in our experiment are situated within an $L\times L$ square lattice
with periodic boundary conditions. They are placed at specific spatial
coordinates and can engage in interactions limited to their von Neumann
neighbourhood. The graphical representation employs vertices to denote agents
and edges to indicate the relationships between an agent and its four
neighbours. The action space for each agent encompasses two strategies: the
overall dilemma strategy and the specific interaction selection strategy. This
dual contributes to the formulation of the agent’s policy $\pi_{i}$ expressed
as:

$\pi_{i}=(\pi_{s_{i}},\pi_{d_{i}})$ (5)

where $\pi_{s_{i}}$ refers to the interaction selection policy dictating
whether to interact with its neighbours, while $\pi_{d_{i}}$ is the dilemma
policy that guides the agent in choosing between a cooperative or defective
strategy.

####  4.1.1 Interaction and Dilemma Actions

During each timestep of an episode, agents participate in multiple pairwise
interactions within the IPD game framework, consisting of two phases:
neighbour selection and dilemma strategy determination. In the first phase,
agent $i$ chooses game partners through an interaction selection action
denoted as $a_{s_{i}}\in\\{0,1\\}^{4}$, informed by previously observed game
experiences with its neighbours. This selection action $a_{s}^{i}$ is
represented by the following bitstring of size 4:

$a_{s_{i}}=(a_{s_{i}}^{1},a_{s_{i}}^{2},a_{s_{i}}^{3},a_{s_{i}}^{4}).$ (6)

Specifically, a bit value of $1$ for $a_{s_{i}}^{j}$ signifies agent $i$
choosing to interact with neighbour $j$, whereas a $0$ implies no interaction.
For example, a selection action of $a_{s}=(1,0,0,1)$ implies interaction only
with the first and last neighbours. Importantly, actual interaction occurs
only when both players decide to interact with each other, i.e., when
$a_{s_{i}}^{j}=a_{s_{j}}^{i}=1$.

In the second phase, each player selects either cooperation or defection as
their dilemma strategy. Following the interaction phase, paired co-players
engage in one round of PDG pairwise, utilizing the action pair
$[a_{d_{i}},a_{d_{j}}]$ where $a_{d}\in[C,D]$. Critically, the chosen binary
dilemma strategy remains consistent across all interacted neighbours. In other
words, a player cannot cooperate with one neighbour while defecting against
another neighbour simultaneously.

####  4.1.2 Game Formulation and Reward

According to their selected dilemma and interaction actions, agent $i$
receives an accumulative payoff at each timestep $t$ by participating in
multiple rounds of the PDGs with its currently interacted neighbours, as shown
by:

$r_{i}(t)=\sum_{j=0}^{n_{i}^{t}}a_{d_{i}}^{T}\mathcal{M}_{p}a_{d_{j}}$ (7)

where $n_{i}^{t}=\sum_{j\in\Omega_{i}}a^{j}_{s_{i}}\times a^{i}_{s_{j}}$
denotes the number of interacted neighbours for agent $i$ at timestep $t$, and
$\Omega_{i}$ is the set of neighbors. In this model, we incorporate learning
from previous interactions by employing a weighted moving average of past
payoffs Danku et al. (2019). Thus, the final payoff of agents $i$ at each
timestep not just based on the current round’s payoff but also includes
payoffs from the past $m$ rounds:

$R_{i}(t)=\frac{r_{i}^{t}+\sum^{M}_{m=1}\alpha^{m}R_{i,m}}{1+\sum^{M}_{m=1}\alpha^{m}}$
(8)

where $\alpha$ is a parameter that controls the rate of weight decay with
increasing $m$, indirectly determining the memory length $M$. To effectively
assess the emergent behaviours, $M$ is restricted by the condition
$M=\min\\{n\mid\alpha^{n}<0.01\\}$. With $\alpha=0$, agents focus only on the
current round, indicating short memory. Conversely, as $\alpha\to 1$, agent
memory extends to include all previous timesteps, capturing a comprehensive
history of experiences.

In EGT research, the Fermi rule is commonly used to model the dynamics of
strategy evolution Szabó and Tőke (1998), reflecting social learning where
neighbours tend to imitate the most successful policy observed. A detailed
description is elaborated in SI LABEL:Supp:EGT. To adapt these imitation
dynamics to the RL context, we have modified the utility function of agent $i$
to align with game payoffs, which is formulated as follows:

$U_{i}(t)=\frac{[\omega_{i}(a_{d}^{t})+1]R_{i}(a_{d_{i}}^{t},a_{s_{i}}^{t},s_{i}^{t})-\omega_{i}(\tilde{a}_{d}^{t})\overline{R}(\tilde{a}_{d}^{t},a_{s}^{t})}{\sum_{a_{d}\in
A_{d}}\omega_{i}(a_{d})+1}$ (9)

where $\tilde{a}_{d_{i}}$ represents a counterfactual dilemma action,
condition on the actual action $a_{d}$ taken by agent $i$. The function
$\omega_{i}(\cdot)$ returns the number of neighbours performing a specific
action. The term $\overline{R}(\tilde{a}_{d}^{t},a_{s}^{t})$ denotes the
average payoff associated with the counterfactual action $\tilde{a}_{d}^{t}$
across the population at timestep $t$. Essentially, the agent raises a
retrospective question: “Would a different past action have led to a more
advantageous outcome?” This setting integrates aspects of social learning and
RL, allowing agents to compare global information from group about the
performance of different actions with their own localized experiences.

###  4.2 Training Approach

In our multi-agent PDG framework, the training methodology aligns with the
well-established DQN approach, typically used in single-agent tasks. Our focus
is on formulating a joint policy $(\pi_{s},\pi_{d})$, which utilizes the
combined action utilities to calculate the Q-loss for each policy network,
guiding both dilemma strategy and interaction selection processes. See SI
LABEL:Supp:Algorithm for a detailed elucidation of the training procedure.

####  4.2.1 Network Architecture

Each agent in our independent MARL setup is equipped with a memory buffer,
storing experiences from the last $M$ rounds, encompassing a record of both
the agent’s own actions and those of adjacent agents. The agent updates its
memory at each timestep with recent feedback from the local environment,
ensuring an accurate representation of its state. The selection and dilemma
networks process inputs from long-term neighbour interactions and prior
dilemma strategies, respectively. Additionally, both networks consider the
dilemma strategies of four neighbouring agents, with actions represented
through one-hot encoding and sequenced together.

In the selection phase, agents evaluate the state $s_{s}\in\mathbb{R}^{2\times
16\times M}$ , and in the dilemma phase $s_{d}\in\mathbb{R}^{2\times 5\times
M}$. They operate with two Q-networks configurations, formulating policies
independently and without sharing parameters across agents. The architecture
of each Q-network includes a dual-layer perception with 32 hidden units and
employs the tanh activation function for nonlinear transformations.

####  4.2.2 Experiment Setup

During the training stage of our RL experiments, we employ a centralized
training with decentralized execution approach Lowe et al. (2017). This method
allows agents to access global information regarding the average payoff for
potential action in the PDG among the population, thereby facilitating an
effective evaluation of their action utilities.

The training involved 900 agents, each equipped with two neural networks,
ensuring a broad representation of cooperation dynamics at the population
level. To generate experiences for agents, $10$ parallel arenas were
established. In each arena during every experimental trial, agents engage in
interactions with their neighbours over $6,000$ episodes, each comprising $10$
timesteps, resulting in a total of $60,000$ steps. At the end of each episode,
sampled trajectories for agents were aggregated and subsequently forwarded to
the respective learner. We compute the gradient by using the Adam optimizer
Kingma and Ba (2014) with a linear annealing schedule of learning rate. To
enhance the efficiency of the Q-learner in learning from experience replay, we
also implement a common practice known as prioritized experience replay Schaul
et al. (2015) within the DQN framework. Additional details on hyperparameters,
please refer to the SI LABEL:Supp:Hyperparameters.

##  5 Results and Discussion

In this section, we present the outcomes of our experimental investigations,
which provide evidence supporting the hypothesis that incorporating the
interaction selection action with the dilemma action through RL promotes
network reciprocity and cooperation evolution in a spatial PDG setting.

###  5.1 Experimental Setup

Our experiment employs a square lattice setup, randomly assigning agents as
cooperators or defectors with equal likelihood. The primary evaluation metric
is the fraction of cooperative agents in the population, representing the
achieved level of overall cooperation. For robustness, we average the outcomes
of the final 10 episodes over the entire training duration. All experiments
were replicated five times to ensure replicability. Unless otherwise stated, a
memory weight of 0.6 is assigned, incorporating experiences from the previous
four rounds as network input.

###  5.2 Promotional Effect of RL on Cooperation

Figure 2: RL-based training approach (ours) promotes cooperation more
effectively than the EGT (baseline) method.  The EGT (orange) represents
agents solely calculating cumulative payoffs and adjusting dilemma actions
through social learning. In contrast, the implementation of effective dilemma
and selection policies, guided by RL (blue), has significantly enhanced the
level of cooperation within the population. Our RL-based method maintains full
cooperation in the population until dilemma strength exceeds $1.2$.

To evaluate the effectiveness of our proposed method, we first train a
population to learn a combined policy, including both dilemma and interaction
strategies under various dilemma strength conditions. Figure 2 demonstrates
that the application of RL in coordinating interaction and dilemma strategies
enables the population to sustain a high cooperation level successfully.
Notably, this approach proves robust, maintaining its efficacy even in
scenarios characterized by increased dilemma intensity. As shown, the MARL
system transitions from a complete cooperation phase to a mixed strategy phase
exclusively, when the dilemma strength $b>1.2$.

For comparison, the evolutionary outcomes of conceptually similar models from
existing literature are used as a benchmark Danku et al. (2019). Within EGT
framework, agents evaluate equivalent lengths of past payoffs and emulate the
most successful dilemma strategy observed in their neighbourhood (a detailed
description of EGT methodology, refer to the SI LABEL:Supp:EGT). It is
noteworthy that even when the dilemma intensity is reduced to $b=1.1$, only
$54\%$ EGT agents opt for cooperative strategies. Moreover, ablation
experiments detailed in SI LABEL:Supp:ablation demonstrate that agents
utilizing RL to learn dilemma strategies exclusively underperform in
comparison to the model proposed in this study. These results suggest that
lacking selective interaction in the PDG and the mere imitation of
neighbouring successful strategies are insufficient to achieve optimal
performance within the population. Additional MARL-based benchmarks are
reported in SI LABEL:Supp:More_benchmark.

###  5.3 Evolutionary Dynamics of Cooperation

Figure 3: The evolution of cooperation and associated payoffs across varying
dilemma strengths. In all scenarios, the fraction of cooperators first
decreases and then increases over time, coinciding with reduced average
individual payoffs and increased inequality as dilemma strength intensifies.
The evaluation encompasses evolutionary trajectories of (a) cooperation level
and (b) the Gini Coefficient, alongside metrics including (c) average group
payoffs and (d)-(e) payoffs for trained cooperators and defectors, with
dilemma strength varying from $b=1.20$ to $1.26$.

We next investigate the evolutionary dynamics and outcomes of overall dilemma
strategies within the population, focusing on the evolution trajectories and
average payoffs in four representative dilemma conditions. Figure 3(a) reveals
a rapid initial decline in the population’s cooperation level as the dilemma
intensity increases. However, RL agents employing additional interaction
strategies exhibit two evident phases in their learning process, aligning with
observations from previous EGT experiments Wang et al. (2013): the END period
and the EXP period. During the END period, cooperative agents resist the
invasion of defectors, and successful cooperators convert those defectors into
cooperators in the EXP period. In our experiments, the former period is
characterized by a rapid decrease in cooperation levels in the first $3,000$
training step, followed by a phase where these levels rise unless defectors
completely dominate in the early stages. Consequently, at the end of the EXP
period, the cooperation level in the population significantly decreases,
falling from $0.987$ to $0.294$ as $b$ rises from $1.20$ to $1.26$.

The dilemma strength also significantly affects the distribution of individual
payoffs, leading to a bifurcated process. As shown in Figure 3(b), the group
Gini Coefficient exhibits a temporal evolution, initially increasing and
subsequently decreasing, hinting at a correlation between payoff equality and
the evolution of cooperation. Notably, a large fraction of cooperators
contributes to high levels of group equality. In Figures 3(c)-(e), the focus
is on evaluating the average payoff for the population, as well as the
separate payoffs for cooperative and defective individuals. There is a general
decrease in the average payoff as the dilemma increases, which aligns with
expectations, given that cooperators are primarily the contributors to the
group payoff. Cooperative individuals, however, show greater resilience to
tougher dilemma conditions. Specifically, with an increase in dilemma strength
from $b=1.20$ to $1.22$, the average payoff per episode decreases from $2.67$
to $2.25$, but this can be lessened by adopting an effective interaction
mechanism, resulting in a smaller decrease in cooperative payoff from $3.52$
to $3.41$, compared to a substantial drop for defectors from $1.44$ to $1.10$.
This trend indicates the potential of RL agents to develop interaction
policies for dilemma scenarios, mitigating adverse effects on their payoff.
For detailed information on the average payoff from the trained population,
refer to Table LABEL:Supp_table:payoff_comparison in the SI.

###  5.4 Efficiency of Learned Interaction Patterns

Figure 4: Temporal evolution of strategy connectivity and actual link
interactions. RL agents demonstrate enhanced interaction capabilities,
increasing connections with cooperative neighbours. (a) The average
connectivity ratio for cooperators and defectors in the first half of the
total timestep. (b) The frequency of actual link connections between dilemma
strategies during the first ten episodes. The dilemma strength is set to
$b=1.20$.

To elucidate the role of interaction selection in network reciprocity and
cooperation, we analyze strategy connection and distribution patterns in
Figures 4 and 5. Through participation in PDG with selective neighbours, RL
agents develop policies for distinguishing cooperators from defectors, thereby
boosting spatial reciprocity and cooperation. Figure 4(a) depicts how the
disparity in average connections between cooperators and defectors increases
during the initial half of the total timestep. This leads to a preference for
forming connections with cooperators, irrespective of the dilemma strategy
chosen by agents. In Figure 4(b), we evaluate the actual link connections
across different link types during the first 10 training episodes, signifying
frequency where the chosen co-player reciprocally opts for interaction within
the same round. As illustrated, the occurrence of interaction between two
neighbouring individuals who both employ the defective strategy (DD link) is
merely 45.48%. In contrast, interactions between two cooperators (CC link) can
increase to as high as 77.29%. For measurement metrics of agent interactions,
see SI LABEL:Supp:_measurement.

Figure 5: Snapshots of the spatial evolution of strategies and their
connections. Cooperative individuals resist defector incursions by forming and
expanding clusters. Panels (a)-(d) depict strategy distributions; (e)-(h)
illustrate corresponding strategy connections at identical timesteps. Pixels
represent agents as cooperators (blue) and defectors (red), with strategy
connectivity ratio varying from 0 (shallow) to 1 (deep). The results are
obtained for $b=1.20$.

We next provide intuitive evidence regarding the previously described learned
interaction policy and its role in enhancing spatial reciprocity by
illustrating the spatial coevolution of the dilemma and interaction strategies
within the population over time. Initially, in the END phase depicted in
Figures 5(a) and (e), cooperators resist the invasion of defectors by forming
small clusters, yet lack an effective neighbour selection strategy. As
training progresses, RL agents learn to adapt their interaction strategies in
response to neighbouring dilemma strategies, enhancing the influence of
spatial reciprocity in promoting the evolution of cooperation. Figures 4 and 5
(f)-(h) show that, in the EXP phase, individuals within the cooperative
cluster show a higher tendency to engage in the PDG with nearby cooperators,
while those at the periphery are more likely to avoid interactions with
defectors. This selective interaction mechanism favours cooperators and limits
the payoff obtained from free-riding behaviours. Consequently, defectors
gradually switch their dilemma strategies, leading to the expansion of
cooperative clusters.

###  5.5 Role of Long-term Experiences

Figure 6: Average cooperation level and effective connection for cooperators
with varying experience lengths. Incorporating longer experiences as input
enhances group cooperation and cooperative interactions. (a) Post-training
cooperation levels in the population. (b) Frequency of cooperators
participating in PDG across the initial 20 episodes. These results are
obtained for four dilemma strength scenarios, with memory weight $\alpha$
ranging from 0 to 0.6.

In prior experiments, RL agents use experiences from the last four rounds as
network input. Finally, we investigated the influence of varying memory
lengths on their learning dynamics, with findings presented in Figure 6. Three
memory weights were assessed, with $\alpha$ varying from 0.6 to 0,
representing input scenarios based on observations from the last, second, and
fourth rounds, respectively. Notably, a positive correlation is observed
between the group cooperation level and the memory length of trained agents.
Figure 6(a) illustrates that, for instance, when dilemma strength increased
from 1.2 to 1.22, agents recalling four-time steps maintained group
cooperation effectively. Conversely, those relying solely on current
information saw a substantial decrease in cooperation levels, dropping from
0.64 to 0.19.

Furthermore, Figure 6(b) echoes the findings of Figure 4, demonstrating that
successful cooperation benefits from a preference for efficient communication
with neighbouring cooperators. Moreover, the efficiency of interaction
selection is also influenced by the length of agent memory. Longer observation
inputs are shown to improve the average interaction ratio among cooperators in
the END and initial EXP stages, which supports network reciprocity and
contributes to the formation and development of cooperative clusters.
Interestingly, our findings also reveal that RL individuals consistently and
effectively avoid interactions with defectors, irrespective of memory length
and dilemma intensity (detailed in SI LABEL:Supp:_interaction). Finally,
experiments conducted in SI LABEL:Supp:Q-Network demonstrate that the dual
Q-network configuration outperforms its single Q-network counterpart,
highlighting the advantage of our proposed dual network approach.

##  6 Conclusion

Our computational model enables agents to interact selectively with their
neighbours and protect cooperative behaviours from antisocial influences by
MARL and iterative trial-and-error in simulations. Unlike existing RL
cooperation studies, our approach does not rely on predefined social norms or
external incentives Ueshima et al. (2023); Tennant et al. (2023). By
integrating a spatial PDG into the training environment, we extend focus from
paired interactions to those within a spatial structure, enabling a deeper
analysis of how long-term learning impacts the coevolution dynamics of
cooperation and selective interaction.

Our findings reveal that trained agents are capable of differentiating between
neighbouring cooperators and defectors by observing information from their
immediate surroundings, which enhances the network reciprocity and the
associated group cooperation. Consistent with theoretical models Szolnoki and
Chen (2020), our findings suggest that the performance of the learned
interaction mechanism in promoting cooperation is attributed to its ability to
help populations form homogeneous strategic clusters. These clusters are
crucial for resisting invasions by defectors, especially in the early stages
of development. Additionally, we confirm a positive correlation between agent
memory length and the effectiveness of interaction selection, which in turn,
aids the evolution of cooperative behaviors Park et al. (2022).

In conclusion, we emphasize the significance of understanding the learning
dynamics in interaction selection and their contribution to fostering
cooperation. These insights offer new insights into the emergence of
cooperation within social dilemmas. Moreover, the computational framework
developed here has broader implications, providing a versatile tool for
investigating and examining mechanisms behind the evolution of cooperation in
spatially structured environments. Integrating psychological complexity in our
training framework emerges as a promising direction for future research.
Understanding these mechanisms may provide solutions to social dilemmas and
strengthen cooperation within both human societies and artificial intelligence
systems.

## Acknowledgments

The authors would like to thank anonymous reviews for their constructive
comments. We also appreciate the insightful discussions with Xiaomei Mi and
Yang Li related to this paper, and are grateful Haotong Zhang and Runqi Zhao
for their support with computational resources. Finally, we acknowledge the
assistance given by Research IT and the use of the Computational Shared
Facility at The University of Manchester.

## References

  * Abeywickrama et al. [2023] Dhaminda B Abeywickrama, Nathan Griffiths, Zhou Xu, and Alex Mouzakitis.  Emergence of norms in interactions with complex rewards.  In Proceedings of the 2023 International Conference on Autonomous Agents and Multiagent Systems, pages 2280–2282, 2023.
  * Anastassacos et al. [2020] Nicolas Anastassacos, Stephen Hailes, and Mirco Musolesi.  Partner selection for the emergence of cooperation in multi-agent systems using reinforcement learning.  In Proceedings of the AAAI Conference on Artificial Intelligence, pages 7047–7054, 2020.
  * Danku et al. [2019] Zsuzsa Danku, Matjaž Perc, and Attila Szolnoki.  Knowing the past improves cooperation in the future.  Scientific reports, 9(1):262, 2019.
  * Du et al. [2023] Yali Du, Joel Z Leibo, Usman Islam, Richard Willis, and Peter Sunehag.  A review of cooperation in multi-agent learning.  arXiv preprint arXiv:2312.05162, 2023.
  * Jaques et al. [2019] Natasha Jaques, Angeliki Lazaridou, Edward Hughes, Caglar Gulcehre, Pedro Ortega, DJ Strouse, Joel Z Leibo, and Nando De Freitas.  Social influence as intrinsic motivation for multi-agent deep reinforcement learning.  In International conference on machine learning, pages 3040–3049, 2019.
  * Kingma and Ba [2014] Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  arXiv preprint arXiv:1412.6980, 2014.
  * Köbis et al. [2019] Nils C Köbis, Bruno Verschuere, Yoella Bereby-Meyer, David Rand, and Shaul Shalvi.  Intuitive honesty versus dishonesty: Meta-analytic evidence.  Perspectives on Psychological Science, 14(5):778–796, 2019.
  * Köster et al. [2022] Raphael Köster, Dylan Hadfield-Menell, Richard Everett, Laura Weidinger, Gillian K Hadfield, and Joel Z Leibo.  Spurious normativity enhances learning of compliance and enforcement behavior in artificial agents.  Proceedings of the National Academy of Sciences, 119(3):e2106028118, 2022.
  * Kramár et al. [2022] János Kramár, Tom Eccles, Ian Gemp, Andrea Tacchetti, Kevin R McKee, Mateusz Malinowski, Thore Graepel, and Yoram Bachrach.  Negotiation and honesty in artificial intelligence methods for the board game of diplomacy.  Nature Communications, 13(1):7214, 2022.
  * Leibo et al. [2017] Joel Z Leibo, Vinicius Zambaldi, Marc Lanctot, Janusz Marecki, and Thore Graepel.  Multi-agent reinforcement learning in sequential social dilemmas.  In Proceedings of the 16th Conference on Autonomous Agents and MultiAgent Systems, pages 464–473, 2017.
  * Li et al. [2016] Jiaqi Li, Chunyan Zhang, Qinglin Sun, Zengqiang Chen, and Jianlei Zhang.  Changing the intensity of interaction based on individual behavior in the iterated prisoner’s dilemma game.  IEEE Transactions on Evolutionary Computation, 21(4):506–517, 2016.
  * Littman [1994] Michael L Littman.  Markov games as a framework for multi-agent reinforcement learning.  In Machine learning proceedings 1994, pages 157–163. Elsevier, 1994.
  * Lowe et al. [2017] Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, Pieter Abbeel, and Igor Mordatch.  Multi-agent actor-critic for mixed cooperative-competitive environments.  In Proceedings of the 31st International Conference on Neural Information Processing Systems, pages 6382–6393, 2017.
  * McKee et al. [2023] Kevin R McKee, Andrea Tacchetti, Michiel A Bakker, Jan Balaguer, Lucy Campbell-Gillingham, Richard Everett, and Matthew Botvinick.  Scaffolding cooperation in human groups with deep reinforcement learning.  Nature Human Behaviour, 7(10):1787–1796, 2023.
  * Metcalfe [2017] Janet Metcalfe.  Learning from errors.  Annual review of psychology, 68:465–489, 2017.
  * Mnih et al. [2015] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al.  Human-level control through deep reinforcement learning.  nature, 518(7540):529–533, 2015.
  * Moffatt et al. [2009] Peter Moffatt, Chris Starmer, Robert Sugden, Nicholas Bardsley, Robin Cubitt, and Graham Loomes.  Experimental economics: Rethinking the rules.  Princeton University Press, 2009.
  * Nowak and May [1993] Martin A Nowak and Robert M May.  The spatial dilemmas of evolution.  International Journal of Bifurcation and Chaos, 3(01):35–78, 1993.
  * Nowak [2006] Martin A Nowak.  Five rules for the evolution of cooperation.  science, 314(5805):1560–1563, 2006.
  * Park et al. [2022] Peter S Park, Martin A Nowak, and Christian Hilbe.  Cooperation in alternating interactions with memory constraints.  Nature Communications, 13(1):737, 2022.
  * Perc et al. [2013] Matjaž Perc, Jesús Gómez-Gardenes, Attila Szolnoki, Luis M Floría, and Yamir Moreno.  Evolutionary dynamics of group interactions on structured populations: a review.  Journal of the royal society interface, 10(80):20120997, 2013.
  * Perc et al. [2017] Matjaž Perc, Jillian J Jordan, David G Rand, Zhen Wang, Stefano Boccaletti, and Attila Szolnoki.  Statistical physics of human cooperation.  Physics Reports, 687:1–51, 2017.
  * Perolat et al. [2017] Julien Perolat, Joel Z Leibo, Vinicius Zambaldi, Charles Beattie, Karl Tuyls, and Thore Graepel.  A multi-agent reinforcement learning model of common-pool resource appropriation.  In Proceedings of the 31st International Conference on Neural Information Processing Systems, pages 3646–3655, 2017.
  * Rand and Nowak [2013] David G Rand and Martin A Nowak.  Human cooperation.  Trends in cognitive sciences, 17(8):413–425, 2013.
  * Rand et al. [2011] David G Rand, Samuel Arbesman, and Nicholas A Christakis.  Dynamic social networks promote cooperation in experiments with humans.  Proceedings of the National Academy of Sciences, 108(48):19193–19198, 2011.
  * Rapoport et al. [1965] Anatol Rapoport, Albert M Chammah, and Carol J Orwant.  Prisoner’s dilemma: A study in conflict and cooperation.  University of Michigan Press, 1965.
  * Ren and Zeng [2023] Tianyu Ren and Xiao-Jun Zeng.  Reputation-based interaction promotes cooperation with reinforcement learning.  IEEE Transactions on Evolutionary Computation, 2023.
  * Ren and Zheng [2021] Tianyu Ren and Junjun Zheng.  Evolutionary dynamics in the spatial public goods game with tolerance-based expulsion and cooperation.  Chaos, Solitons & Fractals, 151:111241, 2021.
  * Santos et al. [2006] Francisco C Santos, Jorge M Pacheco, and Tom Lenaerts.  Cooperation prevails when individuals adjust their social ties.  PLoS computational biology, 2(10):e140, 2006.
  * Schaul et al. [2015] Tom Schaul, John Quan, Ioannis Antonoglou, and David Silver.  Prioritized experience replay.  arXiv preprint arXiv:1511.05952, 2015.
  * Sigmund et al. [2010] Karl Sigmund, Hannelore De Silva, Arne Traulsen, and Christoph Hauert.  Social learning promotes institutions for governing the commons.  Nature, 466(7308):861–863, 2010.
  * Sigmund [2010] Karl Sigmund.  The calculus of selfishness.  Princeton University Press, 2010.
  * Su et al. [2022] Qi Su, Benjamin Allen, and Joshua B Plotkin.  Evolution of cooperation with asymmetric social interactions.  Proceedings of the National Academy of Sciences, 119(1):e2113468118, 2022.
  * Sylwester and Roberts [2013] Karolina Sylwester and Gilbert Roberts.  Reputation-based partner choice is an effective alternative to indirect reciprocity in solving social dilemmas.  Evolution and Human Behavior, 34(3):201–206, 2013.
  * Szabó and Tőke [1998] György Szabó and Csaba Tőke.  Evolutionary prisoner’s dilemma game on a square lattice.  Physical Review E, 58:69–73, 1998.
  * Szolnoki and Chen [2020] Attila Szolnoki and Xiaojie Chen.  Blocking defector invasion by focusing on the most successful partner.  Applied Mathematics and Computation, 385:125430, 2020.
  * Tanimoto [2013] Jun Tanimoto.  Difference of reciprocity effect in two coevolutionary models of presumed two-player and multiplayer games.  Physical Review E, 87(6):062136, 2013.
  * Tennant et al. [2023] Elizaveta Tennant, Stephen Hailes, and Mirco Musolesi.  Modeling moral choices in social dilemmas with multi-agent reinforcement learning.  In Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence, pages 317–325, 2023.
  * Traulsen et al. [2010] Arne Traulsen, Dirk Semmann, Ralf D Sommerfeld, Hans-Jürgen Krambeck, and Manfred Milinski.  Human strategy updating in evolutionary games.  Proceedings of the National Academy of Sciences, 107(7):2962–2966, 2010.
  * Ueshima et al. [2023] Atsushi Ueshima, Shayegan Omidshafiei, and Hirokazu Shirado.  Deconstructing cooperation and ostracism via multi-agent reinforcement learning.  arXiv preprint arXiv:2310.04623, 2023.
  * Vinitsky et al. [2023] Eugene Vinitsky, Raphael Köster, John P Agapiou, Edgar A Duéñez-Guzmán, Alexander S Vezhnevets, and Joel Z Leibo.  A learning agent that acquires social norms from public sanctions in decentralized multi-agent settings.  Collective Intelligence, 2(2):26339137231162025, 2023.
  * Wang et al. [2013] Zhen Wang, Satoshi Kokubo, Jun Tanimoto, Eriko Fukuda, and Keizo Shigaki.  Insight into the so-called spatial reciprocity.  Physical Review E, 88(4):042145, 2013.
  * Wang et al. [2015] Zhen Wang, Satoshi Kokubo, Marko Jusup, and Jun Tanimoto.  Universal scaling for the dilemma strength in evolutionary games.  Physics of Life Reviews, 14:1–30, 2015.
  * Willis et al. [2023] Richard Willis, Yali Du, Joel Z Leibo, and Michael Luck.  Resolving social dilemmas with minimal reward transfer.  arXiv preprint arXiv:2310.12928, 2023.


Yist Y. Lin∗, Tao Han∗, Haihua Xu∗, Van Tung Pham, Yerbolat Khassanov,
Tze Yuang Chong, Yi He, Lu Lu, Zejun Ma

# Random Utterance Concatenation Based Data Augmentation for
Improving Short-video Speech Recognition

###### Abstract

One of limitations in end-to-end automatic speech recognition (ASR) framework
is its performance would be compromised if train-test utterance lengths are
mismatched. In this paper, we propose an on-the-fly random utterance
concatenation (RUC) based data augmentation method to alleviate train-test
utterance length mismatch issue for short-video ASR task. Specifically, we are
motivated by observations that our human-transcribed training utterances tend
to be much shorter for short-video spontaneous speech ($\sim$3 seconds on
average), while our test utterance generated from voice activity detection
front-end is much longer ($\sim$10 seconds on average). Such a mismatch can
lead to suboptimal performance. Empirically, it's observed the proposed RUC
method significantly improves long utterance recognition without performance
drop on short one. Overall, it achieves 5.72% word error rate reduction on
average for 15 languages and improved robustness to various utterance length.

**footnotetext: Authors equally contributed to this work.

Index Terms: random utterance concatenation, data augmentation, short video,
end-to-end, speech recognition

##  1 Introduction

End-to-end (E2E) automatic speech recognition (ASR) [1, 2, 3, 4] framework has
been predominant in both academic and industrial areas[5, 6, 7], thanks to its
simplicity, compactness, as well as efficacy in modeling capacity. However,
there still remains unresolved problems for E2E ASR framework. One of the
problems is the learned E2E ASR models tend to overfit to the utterances that
have been seen during the training. This makes the models cannot well
generalize to the unseen utterances during the inference. Here, the so-called
``unseen" can refer to various noise conditions [8, 9], out-of-vocabulary or
tail words [10, 11], as well as sequence length mismatch problems [12, 13, 14,
15, 16], to mention a few.

To alleviate the overfitting problem, many effective solutions have been
proposed. One of the earlier pioneer works is dropout [17, 18, 19, 20], which
randomly drops some of the visible or hidden nodes during training. To date,
dropout is widely employed in deep learning to yield robust networks that can
alleviate generalization problems. Following that, another popular technique
developed to address train-test data mismatch problem is scheduled sampling
[21, 14, 22], which is crucial for autoregressive attention-based encoder-
decoder (AED) framework [23, 2, 24]. This is because ground-truth tokens are
used as prior knowledge during training stage, while during evaluation the
prior tokens might be erroneous.

Another group of works are focused on front-end data augmentation [25, 26, 27,
28, 29]. To let models see diverse data, speed perturbation [25], as well as
artificial reverberant noise corruptions [26] are applied. Inspired by
``cutout" in computer vision [30], the recently proposed SpecAugment [27],
which masks a small portion of Mel coefficients along both spectral and
temporal axes during training, has demonstrated significant efficacy for ASR
performance.

More recently, the problem of train-test utterance length mismatch has drawn
increased attention in both machine translation (MT) [31] and ASR communities
[12, 13, 15, 16]. Specifically, it is observed that the models trained with
short utterances can result in significantly degraded results (a lot of
deletions) when the input test utterances are much longer. To address this
problem in MT, [31] proposed a multi-sentence resampling (MSR) method to
concatenate sentences, yielding improved translation performance. In ASR task,
a series of recipes, such as passing random state to simulate utterance
concatenation [12], attention mechanism manipulation [23, 13], and overlapping
decoding [12, 15], have been proposed to improve long utterance recognition.

In this paper, we propose a random utterance concatenation (RUC) method to
address the train-test utterance length mismatch problem in short-video speech
recognition scenario111Short video refers to the one with duration of around
three minutes in this paper.. We are motivated from the observation that our
human transcribed utterances are too short ($\sim$3 seconds on average), while
the test utterances from the front-end voice activity detection (VAD) are much
longer ($\sim$10 seconds on average). The RUC method augments the training
data by randomly concatenating two or more utterances. The intention here is
to improve the robustness of ASR system by letting the E2E ASR model see more
diverse utterances in length.

Our main contributions lie in the following aspects: First to the best of our
knowledge, we are the first to explicitly apply on-the-fly RUC for ASR
training, and reveal the effectiveness of the proposed method as a front-end
data augmentation under ASR scenario. Secondly, we validate the efficacy of
the proposed method on the challenging short-video ASR tasks of 15 languages,
where the datasets are real, spontaneous, and noisy, and the amount of the
training data is in the range from $\sim$1,000 to $\sim$32,000 hours. Besides,
we have conducted extensive experiments for optimal concatenation settings, as
well as a series of analysis under a wide range of train-test length mismatch
conditions.

##  2 Relation to prior work

To alleviate the train-test utterance length mismatch problem in MT task, [31]
proposed a MSR-based sentence concatenation method that is very close to the
proposed RUC here. Although some analyses are conducted on Librispeech data
for the ASR task, no actual experiment indicating the efficacy for the ASR
task has been performed in [31]. Besides, compared with the MT task, utterance
concatenation for the ASR task is more complicated, since the concatenated
utterances are not only potentially heterogeneous in semantics, they might
also be acoustically irrelevant. One cannot know for sure whether it is
working without empirical study.

Train-test utterance length mismatch problem has been extensively studied in
ASR community [12, 13, 15] recently. However, the hallmark of the work is to
deal with very long-form speech to decode. For instance, the length of their
target utterances can be as long as several minutes. Here, our target test
utterances are shorter than 25 seconds. For such utterances, the effectiveness
of their methods has not been reported. Besides, all experiments in [12, 13,
15] are conducted on RNN-T ASR framework, while our ASR models belong to the
AED. Additionally, the proposed RUC is purely a data augmentation method, and
it is orthogonal to the prior approaches proposed in [12, 13, 15] that mainly
make efforts on model level.

##  3 Random utterance concatenation

The purpose of the proposed RUC method is to generate longer utterances on-
the-fly by randomly concatenating utterances during training. Algorithm 1
reveals the implementation details.

Algorithm 1 On-the-fly Random Utterance Concatenation

1:$S\leftarrow$ overall training steps

2:$B\leftarrow$ batch size of training data

3:$N\leftarrow$ maximum number of utterances to concatenate

4:for $s\leftarrow 1$ to $S$ do

5: $D\leftarrow$ randomly subset buffer of overall training set

6: $b\leftarrow\varnothing$

7: while $|b|<B$ do

8: $n\leftarrow$ random integer from 1 to $N$

9: new_trans $\leftarrow\varnothing$

10: new_feat $\leftarrow\varnothing$

11: for $i\leftarrow 1$ to $n$ do

12: (transcript, feature) $\leftarrow\text{random\\_sample}(D)$

13: new_trans $\leftarrow$ new_trans.concat(transcript)

14: new_feat $\leftarrow$ new_feat.concat(feature)

15: end for

16: b.append((new_trans, new_feat))

17: end while

18: do_one_step_training(b)

19:end for

In Algorithm 1, the length of concatenated utterances are constrained within
300 tokens, and the maximum duration is 25 seconds in time. The ``feature" in
Algorithm 1 can be either raw waveform or MEL spectral coefficients. In this
paper, all concatenations are performed on MEL feature level.

The whole RUC training is divided into two stages. The first stage is a normal
cross-entropy training with learning rate decay method for 200k steps. After
that, we employ the RUC method to fine-tune the model with constant learning
rate for another 50k steps.

##  4 Dataset

We employ the anonymized short-video dataset to verify the efficacy of the RUC
approach for ASR training. The data is rather challenging. Not only are they
spontaneous, the genres are also highly diverse. For instance, within a normal
short video, majority of audios are not speech, but music, ambient noise, and
other non-verbal sounds, such as laughter, coughs, breath, etc. Taking German
and Swedish as examples, we observe over 90 percent of videos contain only
around 3-5 short utterances that are transcribable.

Table 1 reports the overall training data statistics for 15 languages used in
this work. The dataset lengths are in the range from 1k to 32k hours. The
smallest training dataset is Swedish, with 1k hours. Following Swedish,
Japanese and Korean are also not that large, with 1.7k and 2.5k hours
respectively. While the largest is English that covers the data from different
countries, including US, UK, Canada, Australia, New Zealand, and South
African.

Table 1: Training data statistics of 15 languages Language (ID) | Hours (K) | Utterances (M)
---|---|---
Burmese (my) | 3.8 | 2.8
Dutch (nl) | 4.8 | 5.2
Filipino (fil) | 5.0 | 4.8
French (fr) | 7.0 | 8.4
German (de) | 9.5 | 10.6
Indonesian (id) | 9.8 | 9.4
Italian (it) | 9.6 | 7.9
Japanese (ja) | 1.7 | 2.2
Korean (ko) | 2.5 | 2.2
Polish (pl) | 9.6 | 9.5
Portuguese (pt) | 10.0 | 13.2
Russian (ru) | 9.0 | 6.2
Swedish (sv) | 1.0 | 1.1
Vietnamese (vi) | 9.7 | 11.7
English (en) | 32.7 | 24.4

Table 2 further presents the average utterance length statistics of training
and test sets in terms of both time and token units for all languages.

Table 2: Average utterance length statistics (mean $\pm$ standard deviation) in training and test sets for 15 languages ID | Training | Test
---|---|---
Duration (s) | #Tokens | Duration (s) | #Tokens
my | 4.62 |  $\pm$ | 3.49 | 13.47 |  $\pm$ |  11.87 | 10.67 |  $\pm$ | 4.51 | 29.2 |  $\pm$ | 18.9
nl | 3.19 |  $\pm$ | 2.63 | 9.12 |  $\pm$ |  7.76 | 10.93 |  $\pm$ | 4.72 | 24.2 |  $\pm$ | 17.1
fil | 3.51 |  $\pm$ | 2.82 | 9.88 |  $\pm$ |  8.46 | 9.98 |  $\pm$ | 4.26 | 21.3 |  $\pm$ | 13.3
fr | 2.82 |  $\pm$ | 2.44 | 11.30 |  $\pm$ |  10.08 | 10.47 |  $\pm$ | 4.64 | 34.1 |  $\pm$ | 23.1
de | 3.08 |  $\pm$ | 2.86 | 9.74 |  $\pm$ |  9.44 | 10.22 |  $\pm$ | 4.21 | 26.7 |  $\pm$ | 15.9
id | 3.57 |  $\pm$ | 3.15 | 9.09 |  $\pm$ |  8.23 | 11.35 |  $\pm$ | 4.54 | 20.8 |  $\pm$ | 13.9
it | 4.14 |  $\pm$ | 3.22 | 11.53 |  $\pm$ |  10.11 | 10.85 |  $\pm$ | 4.71 | 26.6 |  $\pm$ | 17.5
ja | 2.63 |  $\pm$ | 2.56 | 14.95 |  $\pm$ |  14.58 | 12.39 |  $\pm$ | 4.11 | 78.0 |  $\pm$ | 39.6
ko | 3.89 |  $\pm$ | 3.28 | 8.95 |  $\pm$ |  8.59 | 10.33 |  $\pm$ | 4.80 | 17.7 |  $\pm$ | 13.9
pl | 3.50 |  $\pm$ | 2.88 | 11.81 |  $\pm$ |  10.39 | 10.41 |  $\pm$ | 4.61 | 32.7 |  $\pm$ | 20.9
pt | 2.73 |  $\pm$ | 2.08 | 8.76 |  $\pm$ |  7.09 | 10.07 |  $\pm$ | 3.97 | 29.8 |  $\pm$ | 15.7
ru | 5.11 |  $\pm$ | 3.99 | 14.45 |  $\pm$ |  12.52 | 11.11 |  $\pm$ | 4.54 | 23.9 |  $\pm$ | 15.4
sv | 3.35 |  $\pm$ | 2.70 | 10.30 |  $\pm$ |  9.06 | 10.59 |  $\pm$ | 4.58 | 29.0 |  $\pm$ | 19.4
vi | 2.88 |  $\pm$ | 2.58 | 12.47 |  $\pm$ |  11.28 | 10.70 |  $\pm$ | 4.73 | 35.8 |  $\pm$ | 25.5
en | 4.84 |  $\pm$ | 4.23 | 15.64 |  $\pm$ |  15.29 | 11.74 |  $\pm$ | 3.73 | 39.8 |  $\pm$ | 17.9

From Table 2, we can see that the average utterance length of our training
data is much shorter than the test data in terms of both time and token units.
Taking time unit for instance, Russian training set has the longest
utterances, 5.11s on average, and with the ratio of test-train utterance
duration $r\approx 2.17$. Japanese has the shortest utterances, with 2.63s on
average and $r\approx 4.71$. The length mismatch here mainly attributes to two
reasons. On the one hand, as was mentioned earlier, our short-video data is
mostly non-speech, and human transcribers tend to have short utterances to
transcribe for training data preparation. On the other hand, it is related
with our specific test settings. Our incoming test data is an entire short
video, and we rely on a NN-based VAD front-end to first remove non-speech
parts, then split longer speech segments into smaller ones. Finally, they are
fed to the ASR engine. To avoid intra-speech segmentation or speech missing,
the VAD tends to output longer speech to the ASR engine, which results in
train-test length mismatch.

##  5 Experiments and results

###  5.1 Modeling

The E2E ASR models employed in this work are based on attention-based encoder-
decoder architecture, which are similar to the ones used in [32].
Specifically, the encoder is Transformer, while the decoder is LSTM.
Transformer's {layer, dim, head} parameters are {18, 512, 8}, and the feed-
forward network dimension is set to 2048 with the GLU activations. The LSTM
decoder has four layers with 1024 cells per layer. For robust training, we
employed both variational noise (VN) [33] and SpecAugment [34] methods. The VN
training is activated after 10k steps. The SpecAugment is activated after 2k
steps, and its frequency {F, mF} and temporal {W, mT, p} parameters are set to
{27, 2} and {100, 1, 0.1} respectively. During inference, the beam size is
fixed to 10, and the best length normalization factor [35] is selected for
each language independently in [0.0, 0.8] range. All ASR models employ word
piece model [36, 37] with the vocabulary size being 3-7k.

###  5.2 Results

Table 3 reports the WER and corresponding WER reduction (WERR) results (for
N={1, 4, 6, 8}) using the proposed RUC method on 15 languages. In table 3, we
report 4 categories of experiments, where N={4, 6, 8} means we randomly
concatenate up to 4, 6, 8 utterances for the proposed method. As a contrast,
N=1 refers to a continual training without any utterance concatenation. As
shown in Table 3, continual training without the proposed method gets only
0.62% WERR on average, while the proposed RUC data augmentation method makes
consistent significant WERR improvement with three concatenation settings. The
general trend is shown to be the larger N the bigger WERR. Specifically the
WERR is from 4.24% up to 5.72% on average.

Table 3: WER (%) and corresponding WERR (%) for 15 languages using the proposed RUC data augmentation method, where N=1 means 50k steps of continual training, and no utterance concatenation being performed ID | Baseline (WER) | N=1 | N=4 | N=6 | N=8
---|---|---|---|---|---
my | 20.65 | -4.00 | 5.94 | 8.36 | 10.15
nl | 23.90 | -2.30 | 2.73 | 1.94 | 1.10
fil | 26.27 | 1.54 | 2.74 | 2.17 | 2.98
fr | 19.35 | 1.29 | 5.13 | 6.08 | 6.87
de | 15.05 | 4.25 | 7.73 | 9.59 | 10.12
id | 21.79 | 2.94 | 5.03 | 6.06 | 5.58
it | 18.27 | 2.38 | 4.18 | 3.85 | 3.56
ja | 17.66 | -2.89 | -1.45 | 3.57 | 4.10
ko | 18.70 | 2.73 | 1.52 | 4.10 | 5.42
pl | 13.12 | 3.16 | 0.08 | -1.42 | 1.30
pt | 12.72 | 1.77 | 9.07 | 8.94 | 10.06
ru | 19.22 | -3.59 | -0.40 | 1.37 | 0.68
sv | 24.88 | 2.97 | 5.08 | 1.80 | 1.65
vi | 26.73 | 1.48 | 17.23 | 22.38 | 22.53
en | 9.62 | -2.39 | -1.04 | -0.49 | -0.31
Avg. | 19.20 | 0.62 | 4.24 | 5.22 | 5.72

In Table 3, the RUC has not achieved similar WERRs for each language. The best
WERRs are obtained on Vietnamese (vi), Portuguese (pt), German (de), and
Burmese (my) 4 languages, and their best WERRs are over 10%. Interestingly, it
gains 22.53% WERR on Vietnamese. We hypothesize such a huge WERR could be
related with too short utterances ($\sim 2.88s$) in the training dataset. On
the opposite side, there are almost no WERRs for English (en), Russian (ru),
Polish (pl), and Dutch (nl) 4 languages, and the average utterance length of
these 4 languages are over 3.0s. These might partially mean RUC is potential
useful for the training dataset that has a lot of shorter utterances.

Figure 1 plots the WERR versus train-test utterance duration ratio for each of
15 languages.

$0$$0.5$$1$$1.5$$2$$2.5$$0$$10$$20$Train-test utterance duration ratioWERR
(%)mynlfilfrdeiditjakoplptrusvvien Figure 1: WERR vs. train-test utterance
duration ratio ($R$) for all languages

Denoting $R$ as train-test average utterance duration ratio, we can observe
from Figure 1 that when $R\in[0.5,1.0]$, all language ASR models obtain
increased WERRs except for Korean, and Polish ASR models. Particularly, the
latter shows clear performance drop trend. Interestingly, as $R\in[1.0,2.0]$,
there is no obvious WERR deterioration for any language, and on the contrary,
the best performing languages in Table 3 still get further WERR. The result in
Figure 1 indicates employing RUC to produce training utterances as one to two
times long as test utterances ($R$ lies in [1.0, 2.0]) could yield improved
ASR performance.

##  6 Analysis

Train-test mismatch is unavoidable in real applications. To simulate such a
mismatch, we have attempted different test utterance length by VAD, namely,
15s, 12s, 10s, 7s222$N$s denotes the average utterance length produced with a
VAD setting is $N$ seconds., so as to examine the robustness of the proposed
method. We find after RUC training the performance fluctuation is considerably
reduced. Table 4 reveals the regarding efficacy via a contrast between ASR
models with RUC and without RUC training on 4 languages.

Table 4: WER (%) under different utterance length by VAD settings and their standard deviations (SD) before and after the proposed RUC training for 4 languages ID | Without RUC | With RUC
---|---|---
15s | 12s | 10s | 7s | SD | 15s | 12s | 10s | 7s | SD
pt | 16.61 | 16.54 | 15.72 | 15.37 | 0.61 | 16.02 | 16.12 | 15.72 | 15.59 | 0.25
vi | 19.76 | 20.24 | 17.21 | 15.12 | 2.38 | 15.47 | 15.61 | 13.62 | 13.68 | 1.09
ja | 15.04 | 15.11 | 13.93 | 13.40 | 0.84 | 13.59 | 13.57 | 12.96 | 13.66 | 0.33
ko | 12.19 | 12.27 | 11.93 | 12.31 | 0.17 | 11.93 | 11.77 | 11.82 | 12.24 | 0.21

We can observe from Table 4 not only is WER performance improved but the WER
standard deviations (SDs) with different VAD settings are also remarkably
reduced respectively. We note that the WER SDs of the Korean (ko) are changed
little since its WERs between different VAD settings are already rather small,
and it gets minor increased after RUC training, from 0.17 to 0.21.

One of the bad effects of train-test utterance length mismatch, particularly
training utterance being much shorter than test ones, is that many more
deletion errors will arise during inference [31]. To alleviate deletion
errors, one would simply employ length normalization therapy which basically
awards longer utterances. We use the length normalization method as advocated
in [35] in this work, and it is formulated as follows:

$\displaystyle s(\bm{y},\bm{x})=\sum_{i=1}^{m}\log P(y_{i}\ |\
y_{1:i},\bm{x})$ (1) $\displaystyle
s^{\prime}(\bm{y},\bm{x})=s(\bm{y},\bm{x})\mathbin{/}\frac{(5+|\bm{y}|)^{\alpha}}{(5+1)^{\alpha}}$
(2)

where $|\bm{y}|$ refers to the token length of the decoded hypothesis, and
$\alpha$ is the so-called length normalization hyper-parameter that controls
the dynamic range of the denominator, normally in the range of [0.0, 0.8] in
our work. Eq. 1 is the normal AED decoding formula, while Eq. 2 is the
implementation formula of Eq. 1 in practice. We found that after RUC training,
the ASR model is less reliant on normalization, that is, we found the WER gaps
between different normalization factors are much smaller after RUC training.
Figure 2 illustrates the WER dynamic range on two groups of languages. One
group of languages that obtains the least WERR from RUC training are English,
Polish, Russian, and Dutch (Figure 2(a) and 2(b)), while the other group that
has achieved best WERR are Portuguese, German, Burmese, and Vietnamese (Figure
2(c) and 2(d)) respectively.

enplrunl$10$$20$$30$WER (%) (a) Without RUC

enplrunl$10$$20$$30$ (b) With RUC

ptdemyvi$20$$30$$40$WER (%) (c) Without RUC

ptdemyvi$20$$30$$40$MeanMax & min (d) With RUC

Figure 2: Illustration of RUC efficacy on length normalization for inference
through contrasts between those with and without RUC training.

From Figure 2, we notice that the WER change dynamics are obviously smaller
after RUC training compared with the case of which no RUC training is
performed on either group of languages. This indicates RUC training brings
more robust ASR models, which would be beneficial when our ASR engine receives
diverse incoming speech.

Table 3 has clearly revealed the effectiveness of the proposed RUC training on
overall WERR, and now we are curious about if such a performance improvement
occurs on the overall test utterances with diverse utterance length
distribution. To take a closer look at what has happened with more details, we
perform RUC on a predefined Swedish dev set, and compare the results of
different ASR models with or without RUC training. Figure 3 plots the details
of WER versus utterance length on Swedish.

$20$$40$$60$$80$$10$$15$$20$$25$$30$$35$Utterance length (token)WER (%)Without
RUCWith RUC Figure 3: Illustration of WER versus test utterance length with
and without the proposed RUC method on Swedish.

From Figure 3, it is interesting to note that the RUC training method can
yield comparable WER results with the normal training method on shorter test
utterances whose length is below 20 tokens. As test utterances are getting
longer and longer, the gap between the proposed and conventional methods
appears and tends to enlarge. More specifically, the WER gaps begin to appear
for utterances in 20-40 token area, and the proposed RUC method prevails. As
the utterance length is in 40-60 token area, the gap enlarges obviously.
Finally, when test utterance length is over 60 tokens, the normal ASR models
without RUC method yields deteriorating WER results, while the RUC method
consistently gets improved WERR. This further demonstrates the effectiveness
of the proposed RUC data augmentation method since it is beneficial to the
overall test set with wide distribution of utterance length.

##  7 Conclusion

In this work, we proposed an on-the-fly random utterance concatenation method
as front-end data augmentation for improving short-video speech recognition.
Specifically, the proposed method addresses train-test utterance length
mismatch originated from the situation in which incoming test utterance length
is much longer than that in the training set. We demonstrated its efficacy
using diverse ASR models from 15 languages whose datasets are in the range of
1k to 32k hours. With the method, we achieved up to 5.72% WER reduction on
average for the overall languages. By further analysis, we found the proposed
data augmentation method can make the ASR model less sensitive to length
normalization, which potentially proves that the ASR models are more robust to
diverse environments. Moreover, the proposed method is beneficial to long
utterance decoding without any performance drop on short utterance.

## References

  * [1] A. Graves, ``Sequence transduction with recurrent neural networks,'' _arXiv:1211.3711_ , 2012.
  * [2] W. Chan, N. Jaitly, Q. Le, and O. Vinyals, ``Listen, attend and spell: A neural network for large vocabulary conversational speech recognition,'' in _ICASSP 2016_ , 2016, pp. 4960–4964.
  * [3] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, ``Attention is all you need,'' in _Proc. of the 31st NIPS_ , 2017, pp. 6000–6010.
  * [4] A. Gulati, J. Qin, C.-C. Chiu, N. Parmar, Y. Zhang, J. Yu, W. Han, S. Wang, Z. Zhang, Y. Wu, and R. Pang, ``Conformer: Convolution-augmented Transformer for Speech Recognition,'' in _Proc. Interspeech 2020_ , 2020, pp. 5036–5040. [Online]. Available: http://dx.doi.org/10.21437/Interspeech.2020-3015
  * [5] Y. He, T. N. Sainath, R. Prabhavalkar _et al._ , ``Streaming end-to-end speech recognition for mobile devices,'' in _ICASSP_. IEEE, 2019.
  * [6] T. N. Sainath, Y. He, B. Li _et al._ , ``A streaming on-device end-to-end model surpassing server-side conventional model quality and latency,'' in _Proc. of ICASSP_. IEEE, 2020.
  * [7] B. Li, T. Sainath, R. Pang, S.-Y. Chang, Q. Xu, T. Strohman, V. Chen, Q. Liang, H. Liu, Y. He, P. Haghani, and S. Bidichandani, ``A Language Agnostic Multilingual Streaming On-Device ASR System,'' in _Proc. Interspeech 2022_ , 2022, pp. 3188–3192.
  * [8] S. Kim, T. Hori, and S. Watanabe, ``Joint ctc-attention based end-to-end speech recognition using multi-task learning,'' in _ICASSP_. IEEE, 2017.
  * [9] S. Sun, P. Guo, L. Xie, and M.-Y. Hwang, ``Adversarial regularization for attention based end-to-end robust speech recognition,'' _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , vol. 27, no. 11, pp. 1826–1838, 2019.
  * [10] X. Zheng, Y. Liu, D. Gunceler, and D. Willett, ``Using synthetic audio to improve the recognition of out-of-vocabulary words in end-to-end asr systems,'' in _ICASSP 2021_ , 2021, pp. 5674–5678.
  * [11] C. Peyser, S. Mavandadi, T. N. Sainath, J. Apfel, R. Pang, and S. Kumar, ``Improving Tail Performance of a Deliberation E2E ASR Model Using a Large Text Corpus,'' in _Proc. Interspeech 2020_ , 2020, pp. 4921–4925. [Online]. Available: http://dx.doi.org/10.21437/Interspeech.2020-1465
  * [12] A. Narayanan, R. Prabhavalkar, C.-C. Chiu, D. Rybach, T. N. Sainath, and T. Strohman, ``Recognizing long-form speech using streaming end-to-end models,'' in _ASRU 2019_ , 2019, pp. 920–927.
  * [13] C.-C. Chiu, W. Han _et al._ , ``A comparison of end-to-end models for long-form speech recognition,'' in _ASRU_. IEEE, 2019.
  * [14] P. Zhou, R. Fan, W. Chen, and J. Jia, ``Improving generalization of transformer for speech recognition with parallel schedule sampling and relative positional embedding,'' _arXiv:1911.00203_ , 2019.
  * [15] C.-C. Chiu, A. Narayanan, W. Han, R. Prabhavalkar, Y. Zhang, N. Jaitly, R. Pang, T. N. Sainath, P. Nguyen, L. Cao, and Y. Wu, ``Rnn-t models fail to generalize to out-of-domain audio: Causes and solutions,'' in _SLT 2021_ , 2021, pp. 873–880.
  * [16] Z. Lu, Y. Pan _et al._ , ``Input length matters: Improving rnn-t and mwer training for long-form telephony speech recognition,'' _arXiv:2110.03841_ , 2021.
  * [17] N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov, ``Dropout: a simple way to prevent neural networks from overfitting,'' _The journal of machine learning research_ , vol. 15, no. 1, pp. 1929–1958, 2014.
  * [18] G. E. Dahl, T. N. Sainath, and G. E. Hinton, ``Improving deep neural networks for lvcsr using rectified linear units and dropout,'' in _Proc. of ICASSP_. IEEE, 2013.
  * [19] T. Moon, H. Choi, H. Lee, and I. Song, ``Rnndrop: A novel dropout for rnns in asr,'' in _Proc. of ASRU_. IEEE, 2015, pp. 65–70.
  * [20] G. Cheng, V. Peddinti, D. Povey, V. Manohar, S. Khudanpur, and Y. Yan, ``An exploration of dropout with lstms.'' in _INTERSPEECH_ , 2017.
  * [21] S. Bengio, O. Vinyals, N. Jaitly, and N. Shazeer, ``Scheduled sampling for sequence prediction with recurrent neural networks,'' _Advances in NIPS_ , vol. 28, 2015.
  * [22] C.-C. Chiu, T. N. Sainath, Y. Wu _et al._ , ``State-of-the-art speech recognition with sequence-to-sequence models,'' in _Proc. of ICASSP_. IEEE, 2018.
  * [23] J. K. Chorowski, D. Bahdanau, D. Serdyuk, K. Cho, and Y. Bengio, ``Attention-based models for speech recognition,'' _Advances in NIPS_ , vol. 28, 2015.
  * [24] D. Bahdanau, J. Chorowski, D. Serdyuk, P. Brakel, and Y. Bengio, ``End-to-end attention-based large vocabulary speech recognition,'' in _in Proc. of ICASSP_. IEEE, 2016.
  * [25] T. Ko, V. Peddinti, D. Povey, and S. Khudanpur, ``Audio augmentation for speech recognition,'' in _INTERSPEECH_ , 2015.
  * [26] T. Ko, V. Peddinti, D. Povey, M. L. Seltzer, and S. Khudanpur, ``A study on data augmentation of reverberant speech for robust speech recognition,'' in _ICASSP_. IEEE, 2017.
  * [27] D. S. Park, W. Chan, Y. Zhang, C.-C. Chiu, B. Zoph, E. D. Cubuk, and Q. V. Le, ``Specaugment: A simple data augmentation method for automatic speech recognition,'' in _Proc. of INTERSPEECH_ , 2019.
  * [28] A. Rosenberg, Y. Zhang _et al._ , ``Speech recognition with augmented synthesized speech,'' in _Proc. of ASRU_. IEEE, 2019.
  * [29] T.-S. Nguyen, S. Stueker, J. Niehues, and A. Waibel, ``Improving sequence-to-sequence speech recognition training with on-the-fly data augmentation,'' in _Proc. of ICASSP_. IEEE, 2020.
  * [30] T. DeVries and G. W. Taylor, ``Improved regularization of convolutional neural networks with cutout,'' _arXiv:1708.04552_ , 2017.
  * [31] I. Provilkov and A. Malinin, ``Multi-Sentence Resampling: A Simple Approach to Alleviate Dataset Length Bias and Beam-Search Degradation,'' in _Proc. EMNLP 2021_ , 2021, pp. 8612–8621. [Online]. Available: https://aclanthology.org/2021.emnlp-main.677
  * [32] Y. Liu, R. Ma _et al._ , ``Internal language model estimation through explicit context vector learning for attention-based encoder-decoder asr,'' in _Proc. INTERSPEECH_ , 2022.
  * [33] A. Graves, A.-r. Mohamed, and G. Hinton, ``Speech recognition with deep recurrent neural networks,'' in _2013 Proc. of ICASSP_. IEEE, 2013.
  * [34] D. S. Park, W. Chan, Y. Zhang, C.-C. Chiu, B. Zoph, E. D. Cubuk, and Q. V. Le, ``SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition,'' in _Proc. Interspeech 2019_ , 2019, pp. 2613–2617. [Online]. Available: http://dx.doi.org/10.21437/Interspeech.2019-2680
  * [35] Y. Wu, M. Schuster _et al._ , ``Google's neural machine translation system: Bridging the gap between human and machine translation,'' _arXiv:1609.08144_ , 2016.
  * [36] R. Sennrich, B. Haddow, and A. Birch, ``Neural machine translation of rare words with subword units,'' in _Proceedings of the 54th ACL_ , 2016.
  * [37] T. Kudo and J. Richardson, ``SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing,'' in _Proc. EMNLP 2018: System Demonstrations_ , 2018, pp. 66–71. [Online]. Available: https://aclanthology.org/D18-2012