# MARLIN: Soft Actor-Critic based Reinforcement Learning for Congestion
Control in Real Networks

Raffaele Galliera${\dagger}$, $\star$, Alessandro Morelli${\dagger}$, Roberto
Fronteddu${\dagger}$, Niranjan Suri${\dagger}$,$\star$,${\ddagger}$
${\dagger}$Florida Institute for Human & Machine Cognition (IHMC)
$\star$Department of Intelligent Systems & Robotics \- The University of West
Florida (UWF)
Pensacola, FL, USA  ${\ddagger}$US Army Research Laboratory (ARL)
Adelphi, MD, USA
{rgalliera, amorelli, rfronteddu<EMAIL_ADDRESS>
###### Abstract

Fast and efficient transport protocols are the foundation of an increasingly
distributed world. The burden of continuously delivering improved
communication performance to support next-generation applications and
services, combined with the increasing heterogeneity of systems and network
technologies, has promoted the design of Congestion Control (CC) algorithms
that perform well under specific environments. The challenge of designing a
generic CC algorithm that can adapt to a broad range of scenarios is still an
open research question. To tackle this challenge, we propose to apply a novel
Reinforcement Learning (RL) approach. Our solution, MARLIN, uses the Soft
Actor-Critic algorithm to maximize both entropy and return and models the
learning process as an infinite-horizon task. We trained MARLIN on a real
network with varying background traffic patterns to overcome the sim-to-real
mismatch that researchers have encountered when applying RL to CC. We
evaluated our solution on the task of file transfer and compared it to TCP
Cubic. While further research is required, results have shown that MARLIN can
achieve comparable results to TCP with little hyperparameter tuning, in a task
significantly different from its training setting. Therefore, we believe that
our work represents a promising first step towards building CC algorithms
based on the maximum entropy RL framework.

###### Index Terms:

Computer Networks, Communications Protocol, Machine Learning, Congestion
Control, Reinforcement Learning, Soft Actor-Critic.

©2023 IEEE. Personal use of this material is permitted. Permission from IEEE
must be obtained for all other uses, in any current or future media, including
reprinting/republishing this material for advertising or promotional purposes,
creating new collective works, for resale or redistribution to servers or
lists, or reuse of any copyrighted component of this work in other works.

##  I Introduction

Network communications are the backbone of an ever-increasingly distributed
digital world. Each day, technologies such as software, platform, and
infrastructure as a service over Cloud/Edge/Fog computing systems, the
Internet of Things, 5G networks, space communications and networks, peer-to-
peer networks, blockchain, ad-hoc and mesh networks enable countless
organizations and people to run their business and perform tasks that have
become part of our daily routine with just a tap on a screen. The different
characteristics of those technologies, combined with the requirements of very
diverse services built on top of them, present unique challenges for network
protocol designers and researchers.

Transport protocols that can achieve high network resource utilization while
maintaining congestion, i.e., end-to-end queueing delays, low in presence of
ever-changing network conditions and high churn of connections are the key
enabling technology that underlies modern distributed systems. Within
transport protocol design, this dual goal (high channel utilization and low
congestion) is the objective of Congestion Control (CC) algorithms. The
critical impact of CC on the performance of distributed applications and
services has led researchers and engineers to design many variants, each one
with different target scenarios and trade-offs in terms of aggressiveness,
responsiveness to loss and congestion, fairness, and friendliness [1, 2, 3].
However, the challenge of designing a generic CC algorithm that can provide
near-optimal performance in a broad range of network and traffic scenarios is
still open.

Recent advances in computational capacity, made possible by novel CPU, GPU,
and Application-Specific Integrated Circuit (ASIC) architectures, along with
the availability of low-cost, yet powerful, versions of such hardware
accelerators, have led to an explosion of successful applications of Machine
Learning (ML) technology in several domains, coming to meet the requirements
of resource constrained environments such as the edge [4]. This has sprung
renewed research interest towards the development of ML models, Reinfocement
Learning (RL) agents, algorithms, and applications that can face the unique
challenges of the real-world [5, 6, 7, 8].

The difficulty of designing an efficient generic CC algorithm, in contrast
with the relative ease of collecting data on the communication performance and
the small size of the action space, which typically focuses solely on
adjusting the size of the Congestion Window (CWND), has prompt researchers to
investigate the use of ML for CC optimization [9, 10]. While many studies have
already shown promising results, they still underperform when compared to
existing TCP CC algorithms in many scenarios. Reasons may include:
difficulties to train RL agents and/or ML models using real networks and
hardware; low fidelity of simulation and emulation environments; sub-optimal
environment representation (state), action space, and/or reward function
design; and delayed action effects on the state due to communications delay.
Additional research is needed to address these problems.

This paper introduces our initial work on MARLIN, a RL agent for CC. Our
approach is based on Soft Actor-Critic (SAC) [11], an off-policy, entropy-
regularized RL algorithm, that has seen successful application in numerous
real-world problems, especially robotics [7]. We describe the design and
architecture of MARLIN and discuss how we trained the agent, the
hyperparameters used, and the main assumptions and choices we made during its
design phase, comparing our work with other RL-based approaches to CC.
Furthermore, we discuss the future directions of this research, of which this
paper is just the first step. Finally, we present preliminary experimental
results obtained in a real networking scenario, where we investigate the
current performance of MARLIN and compare it to the Transmission Control
Protocol (TCP) in handling a file transfer task.

##  II Designing a Reinforcement Learning agent for Congestion Control

MARLIN is a RL agent for CC, trained in a real network by using continuous
actions to update the CWND. For the purpose of this research, we did not
consider fairness towards competing MARLIN flows and/or other protocols as one
of the optimization objectives of MARLIN, which we leave as a future research
question.

Within MARLIN, learning is based on SAC [11], which trains a stochastic, off-
policy, entropy-regularized agent. The agent interfaces with a custom
transport protocol, presented in Section III, via Remote Procedure Calls
(RPCs) in a non-blocking, bi-directional fashion, as shown in Figure 1. The
frequency of action taking depends on the latest Smoothed Round-Trip Time
(SRTT) estimated for the path. The reward function is shaped as a strictly
negative reward to intrinsically encode the principle that every step taken to
transfer the information is a step too much.

Although it is generally desirable that the agent completes its tasks as
quickly as possible, the RL setting proposed here does not aim for terminal
states. It is inconvenient for the agent to overfit its trajectories on a
fixed amount of bytes to transfer, or a fixed episode length, such that it
will try to find a terminal state at any cost in a certain number of
timesteps. Instead, we aim at designing RL agents able to work on "crowded"
links and achieve high channel utilization independently from the volume of
data transferred during training. For such reason, training is based on
infinite-horizon tasks, never encountering a terminal state and following the
Partial Episode Bootstrapping (PEB) principles for infinite-horizon training
presented in [12].

Figure 1: Agent-Protocol interface. Communication protocol and agent
communicate through a gRPC server. Figure 2: A dumbbell network is used for
training and testing. Background and agent traffic are transmitted from one
sub-network to the other over the bottleneck link. Senders and receivers are
all connected to a WiFi Access Point.

###  II-A Congestion Control as Markov Decision Process

Traditional CC algorithms implemented within transport protocols such as TCP
take actions, e.g., updating their CWND, in response to changes in the
environment they sense with the goal of achieving maximum throughput and
minimum congestion. Changes may include, for example, packet loss, duplicate
packets, or variations in the measured Round-Trip Time (RTT). Such sequential
decision making setting can be naturally posed as a Markov Decision Process
(MDP) [13].

The MDP framework formalizes with $\langle S,A,p,R\rangle$ the sequential
agent-environment interaction as the exchange of three different signals at
every time step $t$ of the process. The RL "learning from interaction"
fundamentals [14] stem from this abstraction: the action $a_{t}\in A(s_{t})$
taken by the agent at a certain step $t$ after sensing the state $s_{t}\in S$
leads to a consequent response of the environment as a reward signal
$r_{t+1}\in R\subset\mathbb{R}$ and transitions to a new state $s_{t+1}$. This
process follows the environment dynamics described by the probability function
$p(s_{t+1},r_{t+1}|s_{t},a_{t}):S\times R\times S\times
A\longrightarrow[0,1]$. Such formulation of the transition dynamics also
defines the Markov property, which requires the state to include all the
relevant information about past interactions that would impact the future, as
each possible values of $r_{t}\in R$ and $s_{t}\in S$ depend solely on the
previous state and action $s_{t-1},a_{t-1}$.

Communication networks are a subtle environment for a RL agent. Packets in
modern networks are often transmitted a few milliseconds apart, sometimes
microseconds. At the same time, running Stochastic Gradient Descent (SGD) to
update the model during training, or even simply computing a new action, can
take up a significantly greater amount of time. As a consequence, the
environment might have already changed significantly by the time the action is
taken, potentially making it obsolete or, even, wrong. This has been
identified as one of the main reasons why it is hard to maintain the
performance achieved in a simulated environment after moving to a real-world
deployment [15].

Furthermore, actions do not instantly affect the agent’s perception, as the
effects of changing the CWND require some time before they have an impact on
the network and that impact is propagated back to the source. As new packets
are transmitted at a different rate after an action has been taken, they first
need to reach their destination, then the receiver needs to generate and
transmit a new ACKnowledge (ACK) message for those packets, and finally that
ACK needs to arrive back at the source before any information about the impact
of the action taken can be inferred. These events will take, at the very
least, one RTT to complete. This raises a fundamental question for anybody who
wants to apply RL to the problem of CC: when should the next action be taken?
While other similar approaches, like [16, 17], gather statistics during a
fixed amount of time to compute the state and take the next action, MARLIN
implements a different heuristic algorithm. Every time the transport protocol
makes new network statistics available to MARLIN, if at least the last
reported SRTT has elapsed since the time the last action was taken, data
collected up to that time is refined, the reward $r_{t}$ with regards to
action $a_{t-1}$ is assigned, a new state $s_{t}$ is fed to the agent, and a
new action $a_{t}$ is taken. Waiting for at least one SRTT between actions
prevents that a new action is taken before the impact of the previous one can
be sensed by the agent.

Feature Description Statistic 1 Current CWND Current CWND 1 Last 2 KBs Sent
Amount of KB sent * 2 Mean 3 New KBs sent Amount of KB acked * 3 STD 4 Acked
KBs Amount of KB acked * 4 Min 5 Packets sent Packets sent * 5 Max 6
Retransmissions Number of packets retransmitted * 6 EMA 7 Instantaneous
Throughput Throughput * 7 Difference from Previous 8 Instantaneous Goodput
Goodput * 9 Unacked KBs Amount of KBs in flight 10 Last RTT Last RTT detected
* 12 Min RTT Min RTT * 12 Max RTT Max RTT * 13 SRTT Smoothed RTT * 14 VAR RTT
RTT variance * * During the last RTT timeframe

Table I: Features composing the state of the environment. The horizon of the
observation is also augmented with the previous 10 observations.

###  II-B Training Environment Setup

A key design point of MARLIN is to be trained and evaluated on real networks.
Utilizing real components allowed us to understand the challenges that the CC
domain brings to RL, avoiding the risk of failing a "sim-to-real"
transposition, which could be caused, by inaccurate emulated/simulated
patterns and weak assumptions, at the cost of a slower training time. Relying
on real hardware and protocol implementation intrinsically diversifies
training experience thanks to the stochasticity of the environment and it
prevents the agent from overfitting on artificial patterns that would not be
found in the real world.

To this end, we built the network shown in Fig. 2, which comprises two sources
of traffic and two receivers, two WiFi Access Point (AP), two network
switches, and one router connected in a dumbbell topology. Traffic sources are
connected to the WiFi AP located at one end of the dumbbell, while traffic
receivers are connected to the AP at the other end. The "Background Traffic
Source", on the left, is responsible for generating background and send it to
the "Background Traffic Receiver" node, on the right. The "Sender Application"
transmits traffic to the "Receiver Application" via a custom transport
protocol implementation that uses MARLIN as its CC algorithm.

The need for implementation flexibility led us to integrate MARLIN with
Mockets, a protocol originally designed for communication environments
characterized by limited bandwidth, typically found in tactical and wireless
sensor networks. Further details regarding Mockets are presented in Section
III. To avoid running into situations where MARLIN could underperform due to
the protocol implementation, we shaped the network scenario around the typical
Mockets use case for this first iteration. We set a Smart Queue policy on the
router to limit the maximum amount of traffic flowing between the two
subnetworks to 250 KB/s; we also used TC NetEm - Traffic Control Network
Emulator (TC-NETEM) on the "Sender Application" node to introduce 100 ms of
latency and 3% random packet loss that only affect the traffic generated by
the application. Note that the router’s manual reports that it effectively
limits the actual rate to 95% of the specified value when Smart Queue is
enabled [18].

The background traffic patterns generated can be divided into elephant flows,
i.e., long-lived data transfers that represent a large percentage of the total
traffic (imagine large file transfers or video streaming), and mice flows,
i.e., short-lived data transfers at low throughput (e-mail or RPC calls).
Roughly 87% of the background traffic in our testbed is made by elephant
flows, consisting of four different flows that alternate each other over the
link every two seconds, and two mice flows, which continuously generate very
short-lived (in the order of milliseconds) traffic bursts with intervals that
follow a Poisson distribution.

The Background Traffic Source generates traffic using Multi-Generator Network
Test Tool (MGEN) [19]. Elephant flows consist of two UDP communications
transferring data at 100 KB/s, one UDP communication transferring data at 50
KB/s, and one TCP connection producing 200 KB/s of traffic. One TCP and one
UDP mice flow introduce an extra 17 KB of traffic in average over the link
each second. Each elephant flow is assigned to a different temporal slot of a
2 second duration and they repeat every 8 seconds. Mice flows start with the
first elephant flow and continue until traffic generation is stopped.

###  II-C Agent Design

The interface between MARLIN and the transport protocol, shown in Figure 1, is
implemented using gRPC. The gRPC middleware sitting between the transport
protocol and MARLIN takes care of delivering observations and reward pairs to
the agent and actions back to the protocol.

At a given step $t$, the agent receives network statistics from the transport
protocol via the gRPC middleware. Statistics are then processed and stacked
with the previous 10 observations to form the state $s_{t}$. The agent can
take actions that will increase, maintain constant, or decrease the transport
protocol CWND by a chosen factor. Ideally, the agent should learn to maximize
the volume of data transmitted in the minimum amount of time, while being
watchful of growing queueing delays, which would manifest with an increase in
the measured RTT.

MARLIN is implemented on top of the RL Baselines3 Zoo (RL-Zoo3) [20]
framework, which follows best practices for using Stable-Baselines3 (SB3)
[21], a PyTorch [22]-based library that implements state-of-the-art RL
algorithms following the OpenAI gym interface [23].

####  II-C1 State

Table I describes the state encoded in MARLIN. 14 features are gathered during
the time frame that follows the action taken at step ${t-1}$. MARLIN then
augments the state space with 7 statistics, i.e., last, mean, standard
deviation, minimum, maximum, Exponential Moving Average (EMA), and difference
from the previous state $s_{t-1}$, that are computed for each of the 14
features. The previous $N$ states are also stacked together to form a history
of the previous observations, attempting to adhere to the Markov property. The
final state served to the agent will then have
$N\times|Features|\times|Stats|$ features. This totals up to 980 different
features.

$N$ is considered a hyperparameter of the problem; table II has a complete
list of all hyperparameters used for MARLIN. For the choice of $N$, we
followed the empirical considerations made in [24], and chose 10 as the length
of the history after some preliminary trials and evaluations. During the
training process, observations are processed and normalized through a moving
average by using the VecNormalize environment wrapper present in SB3.

####  II-C2 Actions

MARLIN takes continuous actions contained in the range $[-1,1]$, which
represent the percentage gain of the CWND size. For example, if the action
chosen by the agent is going to be $1$, the CWND is doubled; a value of $0$
means no change in the CWND; $-0.5$ reduces the window size by 50%. The
initial CWND size is set to 4KB at the beginning of the episode, as per the
transport protocol implementation default.

For the purpose of this study, we capped the CWND size to 50 KB, a value that,
if reached, would yield double the throughput that the network can accommodate
in the setup we used for training and evaluation. This choice only impacts the
very first phases of training, when the agent takes random steps to prime the
model. After this phase, which in MARLIN is set to last 10K steps, our
experiments have shown that the agent correctly never fills the CWND to 50 KB.

####  II-C3 Reward

We designed a reward function that gives higher rewards to the agent the
closer it gets to fully utilizing the available bandwidth:

$r_{t}=-\frac{target_{t}}{target_{t}+acked\\_kilobytes_{t}^{cumulative}}$ (1)

where $target_{t}$ represents the amount of bytes the agent should have
delivered up to step $t$ since the beginning of the episode in order to fully
utilize the link and $acked\\_kilobytes_{t}^{cumulative}$ represents the
number of kilobytes there were acknowledged by the receiver until step $t$.

A strictly negative reward function promotes the agent to accumulate the
smallest amount of penalties. The penalty received is much smaller the closer
the agent it is, at each step, to having utilized the link to the best of its
possibilities.

The careful reader might notice that such rewarding system encourages the
agent to accumulate acked bytes regardless explicit impact on the RTT, falling
into the risk of privileging actions that could produce more acked bytes in
the immediate future, with the drawback of causing undesired queuing delays.
To prevent such risk, we consider a second formulation of the reward function
that introduces a RTT-based penalty coefficient:

$r_{t}=-\frac{target_{t}}{target_{t}+acked\\_kilobytes_{t}^{cumulative}*(1-penalties)}$
(2)

The term $penalties$ depends on the difference between the current RTT and the
minimum EMA RTT and is defined as follows:

$penalties=\begin{cases}\alpha\frac{rtt_{diff}}{rtt_{min}^{ema}},&\mbox{if
}\frac{rtt_{diff}}{rtt_{min}^{ema}}<1\par\\\
0.99,&\mbox{otherwise}\end{cases}$ (3)

In Eq. 3, $\alpha$ depends on the magnitude of the difference between
$rtt_{diff}$ and $rtt_{min}^{ema}$ and it is defined as follows:

$\alpha=\begin{cases}1,&\mbox{if
}|\frac{rtt_{diff}}{rtt_{min}^{ema}}|>0.6\par\\\ 0.5,&\mbox{if
}0.1<|\frac{rtt_{diff}}{rtt_{min}^{ema}}|\leq 0.6\par\\\ 0.3,&\mbox{if
}0.05<|\frac{rtt_{diff}}{rtt_{min}^{ema}}|\leq 0.1\\\
0.1,&\mbox{otherwise}\end{cases}$ (4)

It is worth noticing that, in case $rtt_{diff}<0$, the $penalties$ term
becomes negative, thus rewarding the agent when RTT improvements are detected.

####  II-C4 RL Algorithm

MARLIN adopts SAC [11], an off-policy actor-critic algorithm based on the
maximum entropy RL framework. SAC augments the maximum reward objective,
foundation of RL settings, with an entropy maximization term. Such term acts
as a trade-off between exploration and exploitation, so that the agent aims to
maximize its return while also acting as random as possible. In circumstances
where multiple actions seem equally attractive, i.e. in the case of equal or
close Q-Values, the learned policy is encouraged to assign equal probability
mass to those actions.

In practice, the effects of the entropy, or temperature, term prompt the agent
to discard unsuitable trajectories in favor of more promising ones, as well as
to improve the learning speed. The entropy term can be either fixed or
optimized/learned as further steps are taken. However, the optimal entropy
coefficient varies depending on a series of factors, such as the nature of the
task or even the current policy. As a consequence, it is usually considered
good practice to avoid fixed values, preferring instead to update the term at
the same time actor, critic, and the target networks are optimized [7].

Figure 3: The rolling averaged CWND size is plotted against the time from the
beginning of the episode, while segment color represents the RTT observed
between two actions. This plot was obtained from the best run of the second
MARLIN model, i.e. 5(b), which completed the transfer in 24.84s.

###  II-D Future Directions

The work described above is the first step of a multi-year research project.
We have already identified future steps that follow naturally from our present
work. While we proceed describing them sequentially, in reality these steps
are much intertwined and we will likely addressed them in subsequent
iterations.

We think that MARLIN would benefit from a more expressive reward function. We
envision problem and reward formulations that truncate unpromising
trajectories that have moved too distant from the optimal. We believe this
could significantly speed up the solution convergence. Furthermore, we suspect
that MARLIN’s current state might present redundant information as well as
features of low relevance to the problem. To address this, we plan to
investigate smaller and more refined state representations, with the double
goal of lowering complexity and improving convergence. We plan to train this
new agent in diversified networking scenarios, which can capture different
traffic patterns and network technologies, to assess the degree of
generalizability. Finally, a thorough, automated hyperparameter tuning would
further enhance MARLIN’s performance and complete a first cycle of
improvements.

Further advancements will require an evolution in the agent’s design congruent
with specific problems that afflict CC. The literature has shown that ML
models can accurately distinguish between packet loss attributable to
congestion or channel errors. We plan to integrate a similar classifier within
MARLIN and investigate the feasibility of an analogous approach to identify
variations in end-to-end latency that are caused by changes in the path to
destination. The fundamental building block of CC algorithms is how and when
to change the CWND size. MARLIN currently actively controls the "how", leaving
the "when" to a heuristic. We will investigate learning-based approaches to
include such decision factor into MARLIN, with the goal of turning it into a
more comprehensive and reactive system, able to make rational decisions at its
own tempo.

Up until now, we shaped the problem of CC from the perspective of a single RL
agent. Nonetheless, the need for CC algorithms in transport protocols
originated from the lack of coordination in a multi-agent system, where single
entities were acting in a completely self-centered manner. Therefore, we
expect that the next step ahead in learning-based CC will come from the
application of advancing Multi-Agent Reinforcement Learning (MARL) algorithms
that can optimize cooperative and/or competitive agent behavior.

Hyperparameter | Value
---|---
Training steps | $1\times 10^{6}$
History length | $10$
Training episode length | $200$
Learning rate | $3\times 10^{-4}$
Buffer size | $5\times 10^{5}$
Warm-up (learning starts) |  $1\times 10^{4}$ steps
Batch size | 512
Tau | $5\times 10^{-3}$
Gamma | $0.99$
Training Frequency | $1/episode$
Gradient Steps |  $-1$ (same as episode length)
Entropy regularization coefficient | "auto" (Learned)
MLP policy hidden layers | $[400,300]$
Table II: Hyperparameters used in MARLIN.

##  III The Mockets Transport Protocol

One of the main design choices we faced was the transport protocol we would
use to train and evaluate MARLIN. The decision fell on a custom transport
protocol because it is much simpler to integrate with MARLIN than TCP.
Additionally, a custom protocol enables greater flexibility in terms of
retrieving the information required to encode the agent state.

For this study, we integrated MARLIN with Mockets, a message-based
communication middleware implemented on top of UDP, following a school of
thought similar to the one that drove the design of the QUIC protocol [25]
(for additional details on Mockets, the reader can refer to [26]). Mockets
implements a very aggressive CC algorithm whose purpose is to fully utilize
the available bandwidth in presence of variable communication latency and
elevated packet loss. While it presents several advantages over TCP and other
transport protocols in degraded environments [27], Mockets’ existing CC fails
to share link capacity with other communication flows going through the same
links and cannot adapt quickly to changes in the available bandwidth.
Therefore, Mockets could benefit significantly from MARLIN, which could make
it a viable choice also for traditional network scenarios.

To have an accurate representation of the state of the environment within
MARLIN, we improved the RTT estimation in Mockets. To do so, the sender keeps
track of the transmission time of each packet. When an ACK arrives, the sender
computes the RTT by calculating the difference between the current time and
the transmission time of the last packet received by the receiver before
generating the ACK message and then subtracting the ACK processing time from
it. To make this calculation possible, the receiver appends the identifier of
the last received packet (a strictly increasing unsigned integer) and the
processing time (in microseconds) to all ACK messages.

##  IV Experimental Results

###  IV-A Hardware Involved

The testbed described in Figure 2 is implemented using Commercial Off-The-
Shelf (COS) devices. The agent is trained on a workstation equipped with an
Intel i7-8700 CPU, 32GB DDR4 RAM, an NVIDIA GeForce RTX 2060 GPU, and an Intel
Wireless-AC 9560 Network Interface Card (NIC). The "Receiver Application"
resides on a 4GB Raspberry Pi 4 Model B (RPi4B). The two AP used are a TP-Link
EAP245 and a Netgear WAC505, connected to two Netgear GS108E switches. A 2015
Apple MacBook Pro with macOS Catalina v10.15.6 generates background traffic,
which is sent to the receiving application running on a Dell Latitude 7000
laptop (E7470). Both the desktop machine and E7470 run the Ubuntu 20.04.5 LTS
OS. All NIC-related optimizations, e.g., TCP Segmentation Offload (TSO) or
Large Receive Offload (LRO), are enabled, as per default, on all systems. A
Ubiquiti Networks EdgeRouter X with EdgeOS routes packets between the two
subnetworks.

###  IV-B Training

The agent is trained for 1M steps on an infinite-horizon task with partial
episodes lasting 200 steps. When the last step of a partial episode is
encountered, a truncated signal is emitted and PEB is performed, recalling the
agent that additional steps and rewards would actually be available thereafter
[12].

Due to time constraints, the optimization steps involved (for actor, critic,
target networks, and learned entropy coefficient) happen at the end of every
partial episode. Taking training steps between two actions on a real network
is a luxury we cannot afford, as we observed it would introduce delays in the
order of 800-1000ms. Training at the end of an episode allows us to avoid such
effect, limiting the overhead to the sole pipeline, which we report being
around 25 to 30 ms.

At the beginning of training, in order to increase exploration, actions coming
from a uniform random distribution are taken for 10K steps. Subsequently, SAC
starts its normal exploration-exploitation strategy, led by its entropy
coefficient.

We trained three different models for MARLIN. For the first one, we used Eq 1
(5(a)). For the second model, we used the reward function in Eq. 2 (5(b)). In
both cases, the background traffic pattern was fixed, with elephant flows
following this order: $[100\leavevmode\nobreak\
\text{UDP},200\leavevmode\nobreak\ \text{TCP},100\leavevmode\nobreak\
\text{UDP},50\leavevmode\nobreak\ \text{UDP}]$. For the last model (5(c)), we
changed the background traffic to use all permutations presented in Section
II-B.

Figure 4: Mean training reward over the last 100 partial episodes on the three
training runs.

The mean training reward shows a promisingly increasing trend in each of the
three setting, as shown in Figure 4. It is straightforward to interpret the
performance showed during training by taking as a reference our ideal target,
which, in this case, would result in -100 mean training reward.

###  IV-C Results

Following training, we tested all the three MARLIN models by transferring a
3MB file on the same network with the background traffic pattern
$[100\leavevmode\nobreak\ \text{UDP},200\leavevmode\nobreak\
\text{TCP},100\leavevmode\nobreak\ \text{UDP},50\leavevmode\nobreak\
\text{UDP}]$. It is important to note that file transfer is a significantly
different problem from the one that the agent faces during training: here, the
agent is evaluated on completing a task longer than the partial episodes seen
during training, which are limited to 200 steps, and a faster completion
coincides better results.

We repeated the experiment 100 times for each of the three trained agents.
Results are then compared to the performance of Secure Copy Protocol (SCP),
which in our system uses TCP CUBIC as the transport protocol, on the same file
transfer task. Figure 5 shows the optimal behaviour, computed from the
a-priori knowledge of the background traffic, along with the batch of results
obtained from the trained agents. The average, best, and worst performance of
TCP CUBIC are also shown under the same conditions after having repeated the
transfer 100 times as well. Figure 3 presents the variation of the CWND size
during MARLIN’s best run across all the three experiments. The color of each
segment represents the RTT of the communication measured during a period of
one SRTT immediately following the action that took the CWND size to the value
represented by the rightmost end of the segment.

Runs with significant performance degradation were aborted after 80 seconds,
with 5(a) reporting 4 aborted experiments, 5(b) reporting 20, and 5(c)
reporting 2. TCP CUBIC completes the file transfer in 45.03s in average, with
its fastest transfer achieved in 33.97s, and a worst performance of 70.78s.
Excluding aborted runs, MARLIN reached the target in 46.1s in average in 5(a),
50.31s in 5(b), and 46.9s in 5(c). All agents’ best performance was faster
than TCP CUBIC’s best. In 5(a), 48% of the experiments achieve a performance
equal or better than the average accomplished by TCP CUBIC, 31% in 5(b), and
47% in 5(c). The fastest transfer was completed by 5(b) in 24.84s, 27% faster
than the fastest SCP transfer and 45% faster than its average. For comparison,
the available bandwidth on the link allows file transfers to be completed in
25s (see Figure 5); faster transfers can still occur if the traffic injected
by MARLIN causes other flows to slow down.

###  IV-D Discussion

Despite the increased decision making complexity brought by continuous actions
in a problem as intricate as CC in a real network and a training budget of 1M
steps, which is modest when compared to other RL-based CC agents, results are
promising and already comparable to TCP CUBIC in our environment. We believe
that part of it is due to the sample efficiency of SAC. Another reason can be
found in the shaping of the CC problem as an infinite-horizon task with
strictly negative rewards, which enables the agent to exploit its acquired
experience in tasks longer than the partial episodes seen during training and
with different goals.

Permuting the order of the background traffic patterns during training did not
deteriorate the performance during evaluation. In fact, evaluation runs
exhibited lower variance, a better fastest transfer, and similar average
transfer time compared to the experiments shown in 5(a). Moreover, fewer
experiments had to be aborted. These results suggest that MARLIN’s robustness
might be further improved by feeding data from diverse scenarios to the
algorithm.

Figure 3 shows the CWND size during one of the evaluation runs. Note that most
of the time, following an increased RTT reading, the agent has learned to
respond by reducing the CWND size or decreasing its growth speed in the next
step. This behavior is compatible with many RTT-based CC algorithms.

Finally, although most trajectories obtained during evaluation express
promising and valuable results (Figure 5), they also present a significant
degree of instability, which warrants additional research. This behavior is
particularly evident in 5(b), where the model is trained using the function in
Eq. 2. This model has had several transfers significantly faster than the ones
in 5(a) and 5(c), as well as those obtained with SCP; nonetheless, the model
has proved considerably slower in average.

##  V Related Work

Research efforts on the design and optimization of TCP CC have historically
been very different from the approach discussed in this paper. Traditional CC
algorithms aim at achieving full bottleneck link utilization by applying
diverse heuristic strategies that increase the CWND size until a congestion
sign emerges, such as a packet loss, e.g., NewReno [28] and CUBIC [29] (the
default CC algorithm in modern Linux Kernels and recent Windows operating
systems), or an increase in latency, e.g., Vegas [30] and, more recently, BBR
[31]. These approaches have been shown to work well under specific network
conditions, but underperform or experience other types of issues in other
scenarios [32, 33, 34].

Due to the exceptional degree of heterogeneity, complexity, and intrinsic
dynamism of networking environments, and following the reinvigorated interest
in ML that captivated the scientific community in the last two decades,
several recent efforts have focused on learning-based transport protocol
optimization techniques. Some approaches focus on addressing very specific
problem of transport protocols, such as accurately identifying congestion
events, but do not aim at replacing CC algorithms. For instance, researchers
have successfully built ML-based models that can distinguish between losses
caused by congestion and losses due to channel errors in wireless networks
[35, 36] or losses caused by medium contention in optical burst switching
networks [37]. While we expect that our approach would benefit from such
solutions, as discussed above, the final goal of MARLIN is to train an agent
that can take on the tasks of CC algorithms efficiently in a number of
different network scenarios.

More interesting studies seek to apply ML models and RL agents directly to CC,
by learning policies that can optimally adjust the sending rate [38], the CWND
[39], or other parameters that tune the CC algorithm [40]. Comprehensive
surveys of applications of ML to CC are given in [9, 10]. Common problems that
emerged from the reviewed studies include parameter selection, computational
complexity, high memory consumption, low training efficiency, and
compatibility and fairness against existing CC heuristics.

The possibility of training a RL agent in a completely autonomous manner makes
the case for applying RL to CC optimization very compelling. The blocking
nature of widely used RL libraries such as OpenAI Gym, designed to solve RL
problems such as the ATARI games [41], which can be blocked while waiting for
the next action, constitutes one of the main challenges. Researchers have
circumvented this obstacle by training agents in simulated environments and
achieved very promising results, such as with DRL-CC [42] and Aurora [43].
However, reproducing those results in real-world networks has proved to be a
challenge. Solutions that rely on blocking while the agent computes the next
action introduce significant delays, which are detrimental to the overall
communication performance, especially in fast networks. In contrast, we
designed MARLIN to integrate asynchronously with Mockets, which avoids
blocking the transport protocol waiting for the agent to take the next action.

Other researchers propose the usage of non-blocking agents to optimize CC.
MVFST-RL [16] proposed a non-blocking agent based on IMPALA [44], a C++
implementation of the QUIC transport protocol, and Pantheon [39] for network
emulation. Communication between the agent and the system work in a similar
fashion to Park [17], a platform for experimenting with RL agents on computer
system problems based on RPCs. MARLIN follows a similar philosophy to avoid
the drawbacks of blocking systems. However, to the best of our knowledge,
MARLIN, is the only work that uses strictly negative rewards and investigates
the use of an off-policy and entropy-regularized RL algorithm, such as SAC,
with a continuous action space, trained on a real network in which real
background traffic flows compete for bandwidth access. Additionally, we
trained MARLIN on a infinite-horizon setting and evaluated the model on a
common real-world problem such as transferring a file over a shared link.

##  VI Conclusion

This work has shown how effective policies can be obtained by training a RL
agent based on a off-policy, entropy-regularized algorithm such as SAC. MARLIN
shapes the CC problem as a strictly negative rewarded task actuating on
continuous-actions in a real network with competing dynamic background
traffic.

We have also presented future research directions that we plan to pursue.
These include training in more heterogeneous environments, exploring MARL
settings, investigating more expressive reward functions, and designing an
agent able to autonomously decide when to take the next action.

((a)) Agent trained on a single traffic pattern.

((b)) Agent trained on a single traffic pattern with RTT penalties.

((c)) Agent trained on every permutation of the traffic flows.

Figure 5: Evaluation of each model on 100 testing experiments.

## References

  * [1] Josip Lorincz, Zvonimir Klarin and Julije Ožegović  “A Comprehensive Overview of TCP Congestion Control in 5G Networks: Research Challenges and Future Perspectives”  In _Sensors_ 21.13, 2021  DOI: 10.3390/s21134510
  * [2] Shiyao Ma, Jingjie Jiang, Wei Wang and Bo-chen Li  “Fairness of Congestion-Based Congestion Control: Experimental Evaluation and Analysis”  In _arXiv: Networking and Internet Architecture_ , 2017
  * [3] J. Widmer, R. Denda and M. Mauve  “A survey on TCP-friendly congestion control”  In _IEEE Network_ 15.3, 2001, pp. 28–37  DOI: 10.1109/65.923938
  * [4] Raffaele Galliera and Niranjan Suri  “Object Detection at the Edge: Off-the-shelf Deep Learning Capable Devices and Accelerators” 2022 International Conference on Military Communication and Information Systems (ICMCIS)  In _Procedia Computer Science_ 205, 2022, pp. 239–248  DOI: https://doi.org/10.1016/j.procs.2022.09.025
  * [5] Nikita Rudin, David Hoeller, Philipp Reist and Marco Hutter  “Learning to Walk in Minutes Using Massively Parallel Deep Reinforcement Learning”  arXiv, 2021  DOI: 10.48550/ARXIV.2109.11978
  * [6] OpenAI: Marcin Andrychowicz et al.  “Learning dexterous in-hand manipulation”  In _The International Journal of Robotics Research_ 39.1, 2020, pp. 3–20  DOI: 10.1177/0278364919887447
  * [7] Tuomas Haarnoja et al.  “Soft Actor-Critic Algorithms and Applications”  arXiv, 2018  DOI: 10.48550/ARXIV.1812.05905
  * [8] Tobias Jacob., Raffaele Galliera., Muddasar Ali. and Sikha Bagui.  “Marine Vessel Tracking using a Monocular Camera”  In _Proceedings of the 2nd International Conference on Deep Learning Theory and Applications - DeLTA,_ SciTePress, 2021, pp. 17–28  INSTICC  DOI: 10.5220/0010516000170028
  * [9] Wenting Wei, Huaxi Gu and Baochun Li  “Congestion Control: A Renaissance with Machine Learning”  In _IEEE Network_ 35 Institute of ElectricalElectronics Engineers Inc., 2021, pp. 262–269  DOI: 10.1109/MNET.011.2000603
  * [10] Huiling Jiang et al.  “When machine learning meets congestion control: A survey and comparison”  In _Computer Networks_ 192 Elsevier B.V., 2021  DOI: 10.1016/j.comnet.2021.108033
  * [11] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel and Sergey Levine  “Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor”  In _CoRR_ abs/1801.01290, 2018  arXiv: http://arxiv.org/abs/1801.01290
  * [12] Fabio Pardo, Arash Tavakoli, Vitaly Levdik and Petar Kormushev  “Time Limits in Reinforcement Learning”  arXiv, 2017  DOI: 10.48550/ARXIV.1712.00378
  * [13] Martin L. Puterman  “Markov Decision Processes: Discrete Stochastic Dynamic Programming”  USA: John Wiley & Sons, Inc., 1994
  * [14] Richard S. Sutton and Andrew G. Barto  “Reinforcement Learning: An Introduction”  Cambridge, MA, USA: A Bradford Book, 2018
  * [15] Lei Zhang et al.  “Reinforcement Learning Based Congestion Control in a Real Environment”  In _2020 29th International Conference on Computer Communications and Networks (ICCCN)_ , 2020, pp. 1–9  DOI: 10.1109/ICCCN49398.2020.9209750
  * [16] Viswanath Sivakumar et al.  “MVFST-RL: An Asynchronous RL Framework for Congestion Control with Delayed Actions”, 2019  URL: http://arxiv.org/abs/1910.04054
  * [17] Hongzi Mao et al.  “Park: An Open Platform for Learning-Augmented Computer Systems”  In _Advances in Neural Information Processing Systems_ 32 Curran Associates, Inc., 2019  URL: https://proceedings.neurips.cc/paper/2019/file/f69e505b08403ad2298b9f262659929a-Paper.pdf
  * [18] “Ubiquiti - EdgeOS”  In _EdgeOS User Guide_ Ubiquiti  URL: https://dl.ubnt.com/guides/edgemax/EdgeOS_UG.pdf
  * [19] Naval Research Laboratory (NRL) PROTocol Engineering Advanced Networking (PROTEAN) Research Group  “Multi-Generator (MGEN) Network Test Tool”  U.S. Naval Research Laboratory, https://www.nrl.navy.mil/Our-Work/Areas-of-Research/Information-Technology/NCS/MGEN/, 2021
  * [20] Antonin Raffin  “RL Baselines3 Zoo”  In _GitHub repository_ GitHub, https://github.com/DLR-RM/rl-baselines3-zoo, 2020
  * [21] Antonin Raffin et al.  “Stable-Baselines3: Reliable Reinforcement Learning Implementations”  In _Journal of Machine Learning Research_ 22.268, 2021, pp. 1–8  URL: http://jmlr.org/papers/v22/20-1364.html
  * [22] Adam Paszke et al.  “PyTorch: An Imperative Style, High-Performance Deep Learning Library”  In _Proceedings of the 33rd International Conference on Neural Information Processing Systems_ Red Hook, NY, USA: Curran Associates Inc., 2019
  * [23] Greg Brockman et al.  “OpenAI Gym”, 2016  eprint: arXiv:1606.01540
  * [24] Nathan Jay, Noga Rotman, Brighten Godfrey, Michael Schapira and Aviv Tamar  “A Deep Reinforcement Learning Perspective on Internet Congestion Control”  In _Proceedings of the 36th International Conference on Machine Learning_ 97, Proceedings of Machine Learning Research  PMLR, 2019, pp. 3050–3059  URL: https://proceedings.mlr.press/v97/jay19a.html
  * [25] Martin Thomson Jana Iyengar  “QUIC: A UDP-Based Multiplexed and Secure Transport”  IETF, Internet Requests for Comments, 2022  URL: https://datatracker.ietf.org/doc/rfc9000/
  * [26] Erika Benvegnù, Niranjan Suri, Mauro Tortonesi and Tomás Esterrich  “Seamless network migration using the Mockets communications middleware”  In _2010 - MILCOM 2010 MILITARY COMMUNICATIONS CONFERENCE_ , 2010, pp. 2298–2303  DOI: 10.1109/MILCOM.2010.5680364
  * [27] Alessandro Morelli, Michel Provosty, Roberto Fronteddu and Niranjan Suri  “Performance Evaluation of Transport Protocols in Tactical Network Environments”  In _MILCOM 2019 - 2019 IEEE Military Communications Conference (MILCOM)_ , 2019, pp. 30–36  DOI: 10.1109/MILCOM47813.2019.9021047
  * [28] Andrei Gurtov, Tom Henderson, Sally Floyd and Yoshifumi Nishida  “The NewReno Modification to TCP’s Fast Recovery Algorithm”, Request for Comments 6582  RFC Editor, RFC 6582, 2012  DOI: 10.17487/RFC6582
  * [29] Sangtae Ha, Injong Rhee and Lisong Xu  “CUBIC: A New TCP-Friendly High-Speed TCP Variant”  In _SIGOPS Oper. Syst. Rev._ 42.5 New York, NY, USA: Association for Computing Machinery, 2008, pp. 64–74  DOI: 10.1145/1400097.1400105
  * [30] Lawrence S. Brakmo, Sean W. O’Malley and Larry L. Peterson  “TCP Vegas: New Techniques for Congestion Detection and Avoidance”  In _SIGCOMM_ , 1994
  * [31] Neal Cardwell, Yuchung Cheng, C. Gunn, Soheil Hassas Yeganeh and Van Jacobson  “BBR: Congestion-Based Congestion Control”  In _ACM Queue_ 14, September-October, 2016, pp. 20–53  URL: http://queue.acm.org/detail.cfm?id=3022184
  * [32] Philipp Bruhn, Mirja Kuehlewind and Maciej Muehleisen  “Performance and Improvements of TCP CUBIC in Low-Delay Cellular Networks”  In _2022 IFIP Networking Conference (IFIP Networking)_ , 2022, pp. 1–9  DOI: 10.23919/IFIPNetworking55013.2022.9829781
  * [33] Marko Šošić and Vladimir Stojanović  “Resolving poor TCP performance on high-speed long distance links — Overview and comparison of BIC, CUBIC and Hybla”  In _2013 IEEE 11th International Symposium on Intelligent Systems and Informatics (SISY)_ , 2013, pp. 325–330  DOI: 10.1109/SISY.2013.6662595
  * [34] Phuong Ha, Minh Vu, Tuan-Anh Le and Lisong Xu  “TCP BBR in Cloud Networks: Challenges, Analysis, and Solutions”  In _2021 IEEE 41st International Conference on Distributed Computing Systems (ICDCS)_ , 2021, pp. 943–953  DOI: 10.1109/ICDCS51616.2021.00094
  * [35] Kimoon Han, Jae Yong Lee and Byung Chul Kim  “Machine-Learning based Loss Discrimination Algorithm for Wireless TCP Congestion Control”  In _2019 International Conference on Electronics, Information, and Communication (ICEIC)_ , 2019, pp. 1–2  DOI: 10.23919/ELINFOCOM.2019.8706382
  * [36] P. Geurts, I. El Khayat and G. Leduc  “A machine learning approach to improve congestion control over wireless computer networks”  In _Fourth IEEE International Conference on Data Mining (ICDM’04)_ , 2004, pp. 383–386  DOI: 10.1109/ICDM.2004.10063
  * [37] A. Jayaraj, T. Venkatesh and C. Ram Murthy  “Loss classification in optical burst switching networks using machine learning techniques: improving the performance of TCP”  In _IEEE Journal on Selected Areas in Communications_ 26.6, 2008, pp. 45–54  DOI: 10.1109/JSACOCN.2008.033508
  * [38] Mo Dong, Qingxi Li, Doron Zarchy, P. Godfrey and Michael Schapira  “PCC: Re-Architecting Congestion Control for Consistent High Performance”  In _Proceedings of the 12th USENIX Conference on Networked Systems Design and Implementation_ , NSDI’15  Oakland, CA: USENIX Association, 2015, pp. 395–408
  * [39] Francis Y. Yan et al.  “Pantheon: the training ground for Internet congestion-control research”  In _2018 USENIX Annual Technical Conference (USENIX ATC 18)_ Boston, MA: USENIX Association, 2018, pp. 731–743  URL: https://www.usenix.org/conference/atc18/presentation/yan-francis
  * [40] Keith Winstein and Hari Balakrishnan  “TCP Ex Machina: Computer-Generated Congestion Control”  In _SIGCOMM Comput. Commun. Rev._ 43.4 New York, NY, USA: Association for Computing Machinery, 2013, pp. 123–134  DOI: 10.1145/2534169.2486020
  * [41] Volodymyr Mnih et al.  “Playing Atari with Deep Reinforcement Learning”  In _CoRR_ abs/1312.5602, 2013  arXiv: http://arxiv.org/abs/1312.5602
  * [42] Zhiyuan Xu, Jian Tang, Chengxiang Yin, Yanzhi Wang and Guoliang Xue  “Experience-Driven Congestion Control: When Multi-Path TCP Meets Deep Reinforcement Learning”  In _IEEE Journal on Selected Areas in Communications_ 37.6, 2019, pp. 1325–1336  DOI: 10.1109/JSAC.2019.2904358
  * [43] Nathan Jay, Noga Rotman, Brighten Godfrey, Michael Schapira and Aviv Tamar  “A Deep Reinforcement Learning Perspective on Internet Congestion Control”  In _Proceedings of the 36th International Conference on Machine Learning_ 97, Proceedings of Machine Learning Research  PMLR, 2019, pp. 3050–3059  URL: https://proceedings.mlr.press/v97/jay19a.html
  * [44] Lasse Espeholt et al.  “IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures”  In _CoRR_ abs/1802.01561, 2018  arXiv: http://arxiv.org/abs/1802.01561


# Prak: An automatic phonetic alignment tool for Czech

Václav Hanžl1, Adléta Hanžlová2

###### Abstract

Labeling speech down to the identity and time boundaries of phones is a labor-
intensive part of phonetic research. To simplify this work, we created a free
open-source tool generating phone sequences from Czech text and time-aligning
them with audio.

Low architecture complexity makes the design approachable for students of
phonetics. Acoustic model ReLU NN with 56k weights was trained using PyTorch
on small CommonVoice data. Alignment and variant selection decoder is
implemented in Python with matrix library.

A Czech pronunciation generator is composed of simple rule-based blocks
capturing the logic of the language where possible, allowing modification of
transcription approach details.

Compared to tools used until now, data preparation efficiency improved, the
tool is usable on Mac, Linux and Windows in Praat GUI or command line,
achieves mostly correct pronunciation variant choice including glottal stop
detection, algorithmically captures most of Czech assimilation logic and is
both didactic and practical.

Keywords: phonetic alignment, segmentation, PyTorch, Czech, Praat

## 1\. Introduction

Labeling speech recordings and identifying phone boundaries is a significant
part of phonetic research. Even though there are software tools to automate
this process, many of them target only languages with great quantities of
speakers and most are also not freely available, their use requires complex
installation and can be restricted by license agreements. Tool choices are
even more limited for less common languages like Czech.

Praat’s [1] own integrated alignment tool is usable when aligning individual
words or short sentences, but its performance is significantly worsened when
the audio contains pauses. It also only works from the Praat View & Edit
window menu and doesn’t enable automatic alignment of larger datasets.

The Czech aligner used in academic research so far and preferred as the most
useful available is Prague Labeller [2, 3] which is based on HTK GMM models
and shows admirable longevity. Subsequent research used Kaldi [4]. Other ad-
hoc alignment solutions were never finalized as universally usable.

As identifying words and phones and their time boundaries in a recording is
very useful in phonetics not only for research purposes, but also for
educational needs and general better orientation in the audio, we find it
important that a cross-platform tool enabling the automation of such processes
and striving to be as precise as possible is widely available and easy to
access by phoneticians and students and requires no programming knowledge from
the user. In this paper, we present Prak – a tool we developed with this idea
in mind.

## 2\. Design goals and scope

Observing current practices when preparing Czech phone-aligned research data,
we had several goals for our new alignment tool:

  * •

an open-source tool free for any use – MIT license [5] for code, trained on
free audio data

  * •

functioning on Mac, Linux and Windows

  * •

easy to install – low dependencies, only reliabledependencies which are
(hopefully) here to stay

  * •

simple architecture, preferably building on techniques from phonetics
students’ curricula

  * •

usable from both GUI and command line

  * •

using explainable and modifiable logic (rather than a trained blackbox) where
possible

  * •

automatic pronunciation variant selection

The tool expects an audio recording and text transcript as an input. While we
can imagine doing the transcript via ASR, we deliberately left this task to an
external tool if so desired (manual transcription is a noticeably smaller task
than phone boundary adjustments). This way we can always use a state of the
art ASR and combine it with our phone alignment tool, without modifying the
ASR tool for phone alignment needs – merging these tasks into one tool used to
be a vital approach in the GMM HTK era, but modern ASR tool construction often
abandons the notion of phones altogether.

Figure 1: Output textgrid of Prak used in Praat.

The initial release supports Czech language only, as practical help to Czech
research was our primary goal. Nevertheless, we tried to make extensions to
more languages easy by using a multilingual training data source and by
modular design of the code.

## 3\. Design

### 3.1. Training data

Czech language has high quality speech corpora for acoustic model (AM)
training, however, many data sets are unsuitable for a free tool. Some are
commercial and costly (SpeechDAT [6], Speecon [7]), some free of cost, but for
research use only. There is no LibriSpeech [8] or TEDx for Czech. Corpus of
audio recordings from the Chamber of Deputies of the Parliament of the Czech
Republic available freely online [9, 10] is big and free, but contains
reverberations stronger than standard studio recordings. Some Czech TV
recordings are available, but have a low number of speakers.

We decided to first try the CommonVoice (CV) [11] dataset, the size of which
is marginal for Czech (14 hours of training data, 4 hours being silence), but
it has suitable recording conditions, contains a high number of speakers, has
an extremely permissive license and is available for many other languages. To
our surprise, CV itself allowed us to train an AM of sufficient quality, so we
initially released Prak with a model trained on CV alone.

We are also very grateful to the Institute of Phonetics, Charles University in
Prague, for providing their manually labeled recordings from which we selected
5 hours of easily usable data. We did not use this data in any training to
keep our tool free of additional dependencies, but we used 10% of this dataset
to evaluate the tool by comparison to human labeling made by potential users
themselves.

### 3.2. Software tools

Figure 2: Command line output showing phonetic transcription with
pronunciation variants.

Currently available free and precise deep learning ASR tools [12] naturally
come to mind as building blocks, however, the shift towards direct mapping
between text and audio omits the phone level, requiring significant
modifications of these complex tools to repurpose them for phone alignment.

From older ASR tools, Kaldi [13] is still close to the traditional HMM-based
approach of HTK and explicitly uses phones. It is however quite a complex
dependency in a software system. We therefore used a modern NN toolkit PyTorch
[14] for construction of the phone AM, but designed our ASR-like
infrastructure from scratch in Python [15] using a classical HMM approach. As
phone alignment is quite a limited subset of ASR, we expected this to be
feasible.

The matrix library in PyTorch also has a very close alternative in the even
more broadly used NumPy [16], allowing easy reimplementation of at least the
inference part, should such need ever arise.

### 3.3. Integration in Praat GUI

The Praat [1] integration of Prak is designed to be simple to use without any
programming knowledge. A Praat script which embeds the main Python tool is
easy to add to Praat dynamic menu, so users can align text with audio in just
one click – see fig. 1 for example result. The script also performs several
input file checks and adds additional features such as aligning multiple
sounds using one text source.

## 4\. Pronunciation generator

### 4.1. Phonetic alphabets

Alignment output in Praat TextGrid files uses SAMPA [17] to stay compatible
with established practice. Terminal (command line) output uses much more
readable IPA for Czech as used in [18], see fig. 2 for an example of generated
pronunciation and its variants. Internally (in the source code) the tool uses
programming-friendly encoding where each phone is represented by one Czech
character.

### 4.2. Text cleanup

Figure 3: Acoustic model with ReLU NN stack converting MFCC window to phone
probabilities.

Great care was taken to be able to process all usual texts on all the
supported platforms. Apart from notoriously known end-of-line encoding
differences [19], there are more subtle pitfalls like invisible Unicode BOM
characters, NFC and NFD forms of accented characters etc. We hopefully made it
safe to process all the variants happening in practice.

### 4.3. Foreign words

Users can add exceptions in the form of replacement rules. For each word, the
longest match in rules is replaced by all pronunciation variants from the rule
(e.g. rule "washington vošingtn" can do replacement in "Washingtonu").
Replaced text is not matched anymore, but parts of words before and after
replacement are subject to potential additional (shorter) matches. The
resulting text eliminates differences between foreign and native Czech words.

We find the longest match priority more practical than the approach used in
[3] where rules are tried in sequence, making it hard to identify the right
place for adding new rules among hundreds of old ones.

### 4.4. Assimilation logic in backward FSTs

Many Czech assimilations can be expressed as Finite State Transducer (FST)
with a small number of states (often just 2 or 3) processing the phone
sequence backwards, including rather far distance effects. For example, in the
sequence "vošingtnu" (made by the replacement rule above), processing [t]
changes FST state to devoicing and [g] turns into voiceless [k]. In another
FST, [k] or [g] forces state velarization and affected [n] becomes [N]. We use
a convolution of FSTs taking care of:

  * •

glottal stop [], intervocalic [j]

  * •

assimilation of voicing

  * •

consonant groups containing dtn/ďťň such as “ntní” [nti:/nci:/ci:]

  * •

bě/pě/vě/mě/fě [bjE, pjE, vjE, mE, fjE]

  * •

velarization in nk/ng [Nk/Ng]

FSTs provide multiple outputs (needed esp. on word boundaries), leaving the
final choice to the AM.

## 5\. Phone acoustic model

While transformers [20, 21] of various kinds or at least convolutional
structures come to mind as appropriate state of the art in speech processing,
our limited goal of only phone alignment seemed achievable with much simpler
architectures, significantly lowering the barrier for students trying to grasp
the inner workings of the tool. We therefore started with a simple stack of
layers composed of matrix multiplication and ReLU – Rectifying Linear Units,
$\max(x,0)$ – applied to each element of a vector which is passed between
layers (see fig. 3).

As input, the NN gets 19 consecutive MFCC frames plus a "speaker vector" which
is an average MFCC in 4 energy bands (frames are split to above and below
average energy and then sub-split again). We compute Kaldi-like MFCCs via
PyTorch.

The stack of ReLU layers only maps one position in the audio to one phone
hypothesis, not sequence to sequence. For this purpose, we stuck to a
classical HMM structure, replacing a GMM model with ReLU NN. Our training is a
cross between Baum-Welch reestimation and NN training by gradient descent,
alternating phases of time re-alignment of the target phone sequence and
gradient descent of the AM guessing these phones.

Figure 4: Phone alignments during training, rows show train epochs. An
optional second glottal stop is only found later, leading to [objEu…]

Phone sequences are guessed by Viterbi alignment using phone probabilities
estimated by NN AM, while the NN AM itself is trained by gradient descent,
using the phone sequences as training targets. Tools like HTK solve similar
circular dependencies in Baum-Welch reestimation by a wild initial alignment
guess. What we do is similar: For the initial alignment, every phone is
presupposed to be 30ms long with equal silences preceding and following speech
in a recording. The system is able to converge from this (mostly false)
initial alignment to a very good phone AM and very good phone alignments. Fig.
4 shows example training dynamic.

The only adjustment needed for convergence is an artificial boost of rare
phones. In Viterbi decoding, we adjusted phone probabilities according to the
current overall number of frames classified as this phone in the whole
training dataset.

## 6\. Evaluation of phone alignments

We compared the performance of Prak with Prague Labeller [3], using manually
aligned data as a reference. Direct comparison is not easy as phone sets
differ between aligners (e.g. Prak also detects glottal stops, uses both
voiced and voiceless "ř", considers assimilation at word boundaries etc.). The
manual alignment, while made with great effort and care, uses several slightly
different approaches and contains some phone identity errors. An important
parameter for practical users is the frequency of misalignments which must be
corrected by moving multiple phone boundaries, therefore we tried to detect
major phone boundary misplacements.

We compared pronunciations, counting phone insertions, deletions and
substitutions. At places of match, we measured phone center time shifts
against the manual reference (while the exact phone identity may be
questioned,11endnote: 1We manually checked the test set and corrected obvious
errors, however given its size (there are 20k phones in the 10% test subset of
the full 5 hour set), we certainly did not correct all errors even by this
additional pass of human work. the time positions are very exact in our manual
reference data). The count of time mismatches exceeding a threshold was used
as a quality measure, results are shown in tab. 1.

test | PL [3] | Prak
---|---|---
phone mismatch | $6.61$ | $1.88$
match but misplace 0.1s+ | $\mathbf{4.28}$ | $\mathbf{0.36}$
match but misplace 0.2s+ | $3.22$ | $0.09$
mismatch or misplace 0.1s+ | $10.89$ | $2.24$
Table 1: Percentage of phone mismatch and boundary misplacement, comparing
most used tool to ours.

## 7\. Future work

We highly emphasized simplicity, the only bigger tool being a deep learning
framework, only applied to a basic ReLU stack. We avoided FST toolkit (a basic
tool which Kaldi explicitly uses, unlike HTK where this functionality is
present but dissolved in higher layers) and used a simple "sausage" structure
to capture pronunciation variants. There are many wonderful ready-to-use
building blocks in current ASR toolkits which would be great to test in this
task.

While modern ASR tools abandoned explicit design of middle layer
representation like phonemes or phones, it would be possible to map text to a
phone sequence and train e.g. wav2vec2 [22] architecture on phones instead of
characters and extract time boundaries of phones using a method such as [23].

Another option would be to use more complex NN structures like transformers
for the AM. The question is how much would time boundaries be smeared by the
structure paying attention to too distant parts of audio. Such AM would
perhaps only be useful as a first layer alignment anchoring audio to text on
the level of words, followed by fine-grained alignment by a more local AM like
ours.

Phone boundaries could be further fine-tuned. Movements of boundaries towards
spectrum change might help for some phone pairs, different processing for
others – in fact, the challenge here is translating [18] into a programming
language.

Our speaker-vectors can certainly be replaced by i-vectors [13] or x-vectors
[24], trading better results for considerably increased complexity.

We hope that our alignment tool can serve as a useful framework and baseline
for such experiments.

## 8\. Conclusion

We made a practical Czech phone alignment tool which significantly outperforms
the best tool available so far in the boundary misalignment measure. Prak also
provides new features like automatic variant selection and is more resilient
to problematic input texts. We consider our biggest achievement that we gave
researchers and students an open-source tool with no restrictions and simple
to understand architecture, hopefully allowing others to build on it. Free
download of the source code is possible here:  github.com/vaclavhanzl/prak

## References

  * [1] P. Boersma and D. Weenink, _Praat: doing phonetics by computer [Computer program]. Version 6.1.56_ , http://www.praat.org/, 2021.
  * [2] P. Pollák, J. Volín, and R. Skarnitzl, “Influence of HMM´s parameters on the accuracy of phone segmentation - evaluation baseline,” in _Proceedings of the 16th Conference Joined with the 15th Czech-German Workshop "Speech Processing"_ , vol. 1, 2005, pp. 302–309.
  * [3] ——, “HMM-based phonetic segmentation in Praat environment,” in _The XII International Conference Speech and Computer - SPECOM_ , 2007, pp. 537–541.
  * [4] Z. Patc, P. Mizera, and P. Pollak, “Phonetic segmentation using KALDI and reduced pronunciation detection in causal Czech speech,” in _Text, Speech, and Dialogue_. Springer International Publishing, 2015, pp. 433–441.
  * [5] “The MIT license.” [Online]. Available: https://opensource.org/licenses/MIT
  * [6] P. Pollák, J. Černocký, J. Boudy, K. Choukri, J. Kochanina, W. Majewski, V. Ostroukhov, M. Rusko, J. Sadowski, R. Siemund, P. Staroniewicz, M. Trnka, H. Tropf, K. Vicsi, H. Heuvel, and A. Virag, “SpeechDat(E) - Eastern European telephone speech databases,” in _Proceedings LREC’2000 Satelite workshop XLDB - Very Large Telephone Speech Databases_ , 2000, pp. 20–25.
  * [7] “Czech Speecon database.” [Online]. Available: http://catalog.elra.info/en-us/repository/browse/ELRA-S0298/
  * [8] V. Panayotov, G. Chen, D. Povey, and S. Khudanpur, “Librispeech: An ASR corpus based on public domain audio books,” in _2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 2015, pp. 5206–5210.
  * [9] M. Kopp, V. Stankov, O. Bojar, B. Hladká, and P. Straňák, _ParCzech 3.0_. LINDAT/CLARIAH-CZ digital library at the Institute of Formal and Applied Linguistics (ÚFAL), Faculty of Mathematics and Physics, Charles University, 2021. [Online]. Available: http://hdl.handle.net/11234/1-3631
  * [10] “Společná česko-slovenská digitální parlamentní knihovna.” [Online]. Available: https://www.psp.cz/eknih/
  * [11] R. Ardila, M. Branson, K. Davis, M. Henretty, M. Kohler, J. Meyer, R. Morais, L. Saunders, F. M. Tyers, and G. Weber, “Common Voice: A massively-multilingual speech corpus,” _arXiv preprint arXiv:1912.06670_ , 2019.
  * [12] A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever, “Robust speech recognition via large-scale weak supervision,” _arXiv preprint arXiv:2212.04356_ , 2022.
  * [13] D. Povey, A. Ghoshal, G. Boulianne, L. Burget, O. Glembek, N. Goel, M. Hannemann, P. Motlicek, Y. Qian, P. Schwarz, J. Silovsky, G. Stemmer, and K. Vesely, “The Kaldi speech recognition toolkit,” in _IEEE 2011 Workshop on Automatic Speech Recognition and Understanding_ , Dec. 2011.
  * [14] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Kopf, E. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala, “PyTorch: An imperative style, high-performance deep learning library,” in _Advances in Neural Information Processing Systems 32_ , 2019, pp. 8024–8035.
  * [15] G. Van Rossum and F. L. Drake, _Python 3 Reference Manual_. CreateSpace, 2009.
  * [16] C. R. Harris, K. J. Millman, S. J. van der Walt, R. Gommers, P. Virtanen, D. Cournapeau, E. Wieser, J. Taylor, S. Berg, N. J. Smith, R. Kern, M. Picus, S. Hoyer, M. H. van Kerkwijk, M. Brett, A. Haldane, J. F. del Río, M. Wiebe, P. Peterson, P. Gérard-Marchant, K. Sheppard, T. Reddy, W. Weckesser, H. Abbasi, C. Gohlke, and T. E. Oliphant, “Array programming with NumPy,” _Nature_ , vol. 585, no. 7825, pp. 357–362, Sep. 2020.
  * [17] “Czech SAMPA.” [Online]. Available: https://www.phon.ucl.ac.uk/home/sampa/czech-uni.htm
  * [18] P. Machač and R. Skarnitzl, _Fonetická segmentace hlásek_. Epocha, 2010.
  * [19] “CR/LF issues and text line-endings.” [Online]. Available: https://portal.perforce.com/s/article/3096
  * [20] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. u. Kaiser, and I. Polosukhin, “Attention is all you need,” in _Advances in Neural Information Processing Systems_ , vol. 30, 2017.
  * [21] T. Lin, Y. Wang, X. Liu, and X. Qiu, “A survey of transformers,” _AI Open_ , vol. 3, pp. 111–132, 2022.
  * [22] A. Baevski, Y. Zhou, A. Mohamed, and M. Auli, “wav2vec 2.0: A framework for self-supervised learning of speech representations,” _Advances in Neural Information Processing Systems_ , vol. 33, pp. 12 449–12 460, 2020.
  * [23] M. Hira, “Forced alignment with Wav2Vec2.” [Online]. Available: https://pytorch.org/audio/main/tutorials/forced_alignment_tutorial.html#forced-alignment-with-wav2vec2
  * [24] D. Snyder, D. Garcia-Romero, G. Sell, D. Povey, and S. Khudanpur, “X-Vectors: Robust DNN embeddings for speaker recognition,” in _2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 2018, pp. 5329–5333.

###### Notes

  1. 1 We manually checked the test set and corrected obvious errors, however given its size (there are 20k phones in the 10% test subset of the full 5 hour set), we certainly did not correct all errors even by this additional pass of human work.