# Real-time Bandwidth Estimation from Offline Expert Demonstrations

Aashish Gottipati1, Sami Khairy2, Gabriel Mittag2, Vishak Gopal2, and Ross
Cutler2
1University of Texas at Austin, 2Microsoft

###### Abstract

In this work, we tackle the problem of bandwidth estimation (BWE) for real-
time communication systems; however, in contrast to previous works, we
leverage the vast efforts of prior heuristic-based BWE methods and synergize
these approaches with deep learning-based techniques. Our work addresses
challenges in generalizing to unseen network dynamics and extracting rich
representations from prior experience, two key challenges in integrating data-
driven bandwidth estimators into real-time systems. To that end, we propose
Merlin, the first purely offline, data-driven solution to BWE that harnesses
prior heuristic-based methods to extract an expert BWE policy. Through a
series of experiments, we demonstrate that Merlin surpasses state-of-the-art
heuristic-based and deep learning-based bandwidth estimators in terms of
objective quality of experience metrics, while generalizing beyond the offline
world to in-the-wild network deployments where Merlin achieves a $42.85\%$ and
$12.8\%$ reduction in packet loss and delay, respectively, when compared
against WebRTC in inter-continental videoconferencing calls. We hope that
Merlin’s offline-oriented design fosters new strategies for real-time network
control.

##  1 Introduction

Estimating the optimal rate of information flow is essential to ensuring
congestion-free network communication. The bottleneck link– the link with the
least available bandwidth– dictates the rate of information flow across the
network. Estimating the bottleneck link refers to the problem of bandwidth
estimation (BWE)– a challenging and active area of networking research. BWE is
fundamental to real-time communication (RTC) systems and lies at the heart of
network systems. Without accurate bandwidth estimates, seamless network
communication becomes nearly impossible.

The challenge of BWE emerges from the complex and dynamic nature of network
environments [52]. First, network flows change over time as devices come and
go and users change applications, resulting in a non-stationary environment
with the bottleneck link varying over time. Second, the bottleneck link often
lies beyond the first hop and cannot be probed instantaneously. Third, network
environments are partially observable. That is, many outside factors such as
cross-traffic impact the bottleneck link and cannot be directly controlled. To
chip away at these challenges, early bandwidth estimators in RTC relied mainly
on Real-time Transport Protocol (RTP) [46], which probes the network and
aggregates receive-side network statistics. By periodically probing the
network via RTP, the effects of non-stationarity and partial observability can
be mitigated when estimating the capacity of the bottleneck link. The
simplicity of RTP enables portability; however, it limits the quality of
estimates produced [28], leading to the widespread adoption of more
sophisticated heuristic-based methods [7].

Based on aggregated network statistics, rule-based estimators such as WebRTC
[4] leverage statistical models to estimate the available bandwidth. While
these methods have been widely adopted, increasing network heterogeneity and
complexity necessitates more sophisticated methods. For example, RTC
applications such as videoconferencing require high bandwidth and low latency
while passive internet-of-things (IoT) monitoring systems require low
bandwidth. Both of these flows, while disparate in their requirements, compete
for resources in the network core. Scaling to millions or even billions of
flows, resources quickly become scarce while flow interactions perturb the
network. To better serve users and meet growing resource demands, we require
fine-grain network control, i.e., instantaneous adaption to network changes.
However, prior rule-based methods tend to follow longer-term network trends to
promote smooth estimates [4]. Additionally, heuristic-based models were hand-
crafted based on extensive domain knowledge, making them difficult to adapt to
the ever-changing network landscape. Lastly, even with domain knowledge,
network complexity is quickly outstripping human intuition, e.g., configuring
a simple TCP session has dozens of free variables [9]. To enable future
network applications, we require methods that react to instantaneous network
changes, cope with the growing complexity of networks, and are easy to update.

Within recent years, deep learning-based models have demonstrated impressive
ability for real-time adaption under complex domains, while enabling ease of
updates through enhanced input data and fine-tuned objective functions [39].
Although these properties are desirable, real networks tend to be extremely
diverse, making data-driven adoption difficult. In contrast to other deep
learning methods, reinforcement learning (RL) seeks to learn a policy. The
learned policy incorporates environment dynamics, enabling the learned agent
to grasp not only which control actions are positive but which are negative as
well. RL utilizes exploration to search over control policies, enabling the
agent to try unconventional strategies and discover robust network control
policies [24]. Even so, RL agents are conventionally trained from a blank
slate in online environments, neglecting the vast amount of knowledge encoded
in previous heuristic-based methods, and requiring a large number of training
samples to converge to an acceptable policy. In the case of videoconferencing,
the large sample complexity translates to hundreds of thousands of
videoconferencing calls and extremely long convergence times [13]. Second, and
most crucially, in videoconferencing we seek to learn a policy to maximize
user quality of experience (QoE); however, defining a reward for subjective
user experience with objective network metrics is difficult and remains an
open research problem. Furthermore, without a well-defined reward function,
agents may exploit the reward definition and maximize the expected return
without learning the desired task– a phenomenon known as reward hacking [19].
To that end, we desire a method for real-time BWE that exhibits the benefits
of deep RL but leverages the vast efforts of prior domain experts; thus, we
turn towards offline imitation learning (IL) methods.

IL differs from RL in that it seeks to learn a policy from a known expert,
i.e., given a set of offline demonstrations (expert states and actions),
extract a policy that best fits the expert. IL builds upon the vast efforts of
previous domain knowledge encoded within heuristic-based experts and enables
many of the benefits of RL. However, there is no free lunch. Imitating an
expert for real-world BWE suffers from two distinct challenges. First,
extracting a policy from a finite set of expert demonstrations does not
necessarily result in the true generalizable, expert policy. Without
extracting the true policy, the agent is likely to introduce compounding
errors under unseen dynamics, severely degrading user QoE [60]. Secondly,
handcrafted expert feature representations may not necessarily translate
directly to data-driven methods [14]. Consequently, we propose Merlin, the
first purely offline, data-driven solution for BWE that harnesses the encoded
domain knowledge of prior methods to extract an expert BWE policy. Merlin is
trained to imitate an expert Unscented Kalman Filter (UKF) model strictly from
offline, simulated demonstrations via behavioral cloning (BC), a method that
reduces policy learning to supervised learning. We emphasize that no network
interactions are required to train Merlin and data is collected once before
training. Furthermore, as expert demonstrations are sampled strictly from
simulation, we require no specialized hardware or testbed environments to
generate data for our agent, democratizing access to learning-based network
control.

We rigorously evaluate Merlin in simulated, testbed, and wild environments.
The highlights of our evaluations are as follows. We find that our IL model
outperforms the state-of-the-art heuristic-based methods as well as the state-
of-the-art RL-based methods in terms of objective QoE metrics within our
emulated testbed. Furthermore, we demonstrate that our imitator is robust to
domain shifts and is capable of mimicking our expert UKF model with high
confidence in emulated environments. We further support our claims with in-
the-wild deployments, where Merlin achieves a $42.85\%$ and $12.8\%$ reduction
in packet loss and delay, respectively, while preserving a higher receive rate
in comparison to WebRTC in inter-continental videoconferencing calls. Lastly,
we observe that the reported receive rate and media type features are critical
to extracting a control policy. Increasing the number of demonstrations
appears to aid in mitigating domain shift but was not directly reflected in IL
loss metrics. Leveraging temporal correlations via recurrent models tends to
perform better than non-recurrent methods. Lastly, when expert demos are
abundant, BC tends to outperform more sophisticated IL methods.

In summary, our contributions are as follows:

  1. 1.

We demonstrate a new method for BWE that utilizes IL to extract expert
policies from purely offline data, leveraging the extensive domain expertise
encoded in prior hand-crafted network heuristics.

  2. 2.

We rigorously evaluate Merlin in simulated, emulated, and real-world
conditions to study the generalization of cloned policies. Our analysis
provides insights into achieving robustness to distribution shift, a key
challenge in adapting data-driven models.

  3. 3.

We conduct multiple ablation studies to uncover IL best practices for network
control. Our guidelines on features, demonstrations, and architectures provide
a recipe for future research at the intersection of machine learning and
networking.

  4. 4.

We discuss the broader potential of learned policies to transition toward
data-driven control for non-stationary networked systems. IL shows promise for
flexibly modeling complex network dynamics offline.

Overall, we position this work as advancing the integration of machine
learning techniques into networking systems.

##  2 Related Work

Figure 1: Learning from Offline Demonstrations.

Congestion control (CC) solutions broadly seek to promote packet flow,
reducing network congestion while encouraging maximum bandwidth utilization.
These approaches often employ BWE to measure the available link capacity and
set the send rate accordingly. Classical techniques for BWE have traditionally
relied on various forms of packet probing techniques [21]; however, these
methods are fundamentally limited by probing overhead which may itself
contribute to congestion. More modern approaches employed sophisticated
statistical techniques to estimate bandwidth [27, 56]. Most notably,
implementations of WebRTC [4] utilize a Kalman Filter for BWE and have become
the de facto standard for RTC. While widespread, heuristic-based methods tend
to be conservative, matching long-term trends rather than tracking
instantaneous network perturbations [13, 7].

Other methods have sought to take a broader approach to CC for RTC systems.
For example, enforcing a close coupling between video codecs and the transport
layer has been shown to reduce network congestion by jointly adapting the
encoding rate of video frames and the transport layer send rate [16, 69]. More
recent endeavors have shifted to machine learning and deep learning techniques
to estimate bandwidth by exploiting the structured signal present in network
data [25, 47, 55, 59]. While traditional deep learning techniques tend to
perform well for static tasks such as classification, they struggle to adapt
to more dynamic control tasks such as CC, which requires learned dynamics to
inform congestion decisions [1].

Incorporating network dynamics enables learning richer BWE and CC policies.
Accordingly, recent works have sought to apply reinforcement learning to CC
[67, 22, 5, 40, 43, 33, 57, 18, 58, 48]. Mao et al. [38] learn a policy to
dynamically tune the transmit rate of video packets and deploy their learned
model on Facebook’s web-based video streaming platform; however, their area of
focus is confined to video-on-demand and not RTC systems. On the other hand,
the first RL-based bandwidth predictor for RTC systems, R3Net [13], relied
exclusively on deep RL to produce bandwidth estimates, neglecting prior domain
expertise. Other novel RL-based approaches to CC exploit cross-layer features
to inform policy decisions [34, 37, 66, 30, 8, 35]; however, enforcing a
close-coupling between layers restricts architecture design and reduces
modularity.

In contrast, recent works have sought to leverage expert algorithms for model
training. Eagle [11] adopts an expert-oriented approach, seeking to match BBR
[6] via online RL. DuGu [23] utilizes an online IL approach to mimic a custom
CC oracle for short-term video uploads. Zhou et al. propose Concerto [68], a
BC-based method that leverages oracle estimates to select the best video
bitrate from a discrete set of encodings; however, we emphasize we focus on
learning to estimate the available network resources from offline domain-
expertise collected from prior heuristic-based methods.

Later works such as Gemini [61], HRCC [54], HybridRTS [65], BoB [3], SAFR
[62], OnRl [64], and Libra [10] build non-standalone estimators directly on
top of heuristic-based methods or utilize prior methods as fall-back
mechanisms to mitigate tail-end predictions of learned agents. Along similar
lines, Zhang et al. explore fusing cloned experts with online RL-based models
for video bitrate prediction [63]. In contrast to these works, we propose
Merlin a standalone, data-driven approach to BWE that does not rely on
auxiliary estimators such as Google’s Congestion Control (GCC) algorithm for
BWE.

Lastly, the most similar work to ours is Sage [60]. Sage builds upon the vast
efforts of prior methods for learning a purely, data-driven TCP-based CC
algorithm via offline RL. In contrast to Sage, we emphasize that we tackle the
problem of BWE for RTC systems rather than CC for TCP-based applications;
hence, the dynamics of our problem are dissimilar, e.g., TCP-based systems
maintain reliability while RTC systems exchange reliability for reduced
latency. Furthermore, no specialized testbed equipment nor even emulated
interfaces are required for model training. Merlin is trained completely from
offline simulated experience, enabling others to readily build on our method.

In summary, Merlin differs in fundamental ways from prior works. First, we do
not utilize RL or hybrid methods and rely on IL to learn a standalone BWE
policy. Second, as depicted in Figure 1, we train purely offline without ever
interacting with a network during training, generalizing from offline
simulated experience to both emulated and wild environments. Lastly, Merlin is
designed specifically for RTC systems, prioritizing latency over reliability.

(a) Bandwidth Overshooting MOS.

(b) Bandwidth Tracking MOS.

(c) Bandwidth Undershooting MOS.

Figure 2: Quality of Bandwidth Estimates vs. User Experience.

##  3 Bandwidth Estimates vs. User Experience

To illustrate the importance of BWE, we conducted a preliminary study on the
impact of bandwidth estimates on user QoE during RTC videoconferencing. We
benchmark three bandwidth estimators: our expert UKF model, an overshooter,
and an undershooter. We conduct approximately $100$ live videoconferencing
calls on real networks and report the video mean opinion score (MOS), a gold
standard metric for video QoE that ranges from one to five with one
corresponding to low QoE and five mapping to high QoE [41]. We report a subset
of our findings from network environments with stable $1$ Mbps links in Figure
2. In Figure 2(a), the bandwidth estimator overshoots the $1$ Mbps limit,
which causes the video MOS to severely degrade, oscillating between $0.0$ and
$2.0$ (a MOS of $0$ indicates an error, i.e., no video frames are received).
The poor bandwidth estimates severely degrade objective QoE and cause the
sender to flood the network with packets, resulting in increased queuing
delays and packet loss.

In contrast to overshooting, in Figure 2(c), the bandwidth estimator severely
underestimates the $1$ Mbps link. Although not as harsh as overshooting, the
video MOS degrades due to a lack of high-quality video frames, resulting in a
stable MOS of $2.0$. Specifically, the video codec leverages the bandwidth
estimates to choose an encoding rate for each frame. By underestimating the
available bandwidth, the video frames are encoded at a reduced bitrate to meet
bandwidth requirements, delivering lower-quality video streams to the
receiver. However, since the network is not congested, packets are still able
to flow freely and arrive at the receiver without being lost, leading to
better user experience and a higher MOS than in the overshoot case.

Lastly, we observe the best video MOS when the bandwidth estimator tracks the
bandwidth closely as demonstrated by our UKF expert in Figure 2(b). By
providing accurate resource estimates, packets are able to flow freely without
being dropped; additionally, video frames can be encoded at higher bitrates
due to the increase in estimated network resources. The increased video
encoding rate translates to higher-quality video streams being delivered to
the receiver which results in a stable MOS of $3.0$. In comparison to
undershooting and overshooting, tracking the available bandwidth closely leads
to a significant boost in objective QoE. We report the results on simple
stable cases to illustrate the impact of bandwidth estimates; however, BWE
becomes more challenging in live environments due to the partial observability
and non-stationary nature of real networks [52]. Thus, in this work, we seek
to tackle complex environments. We conduct several videoconferencing
experiments on live inter-continental links in section 5, and demonstrate that
we can preserve user QoE through high-quality BWE via purely offline IL.

##  4 Merlin

Design Goals. Conventional bandwidth estimators provide smooth estimates over
long horizons, making instantaneous network adaption difficult. In contrast,
new RL-based estimators react promptly to network perturbations and are “easy”
to update but often exhibit high sample complexity and require a well-defined
reward function to guide network control policy. Thus, we desire a method for
real-time BWE that exhibits the benefits of deep RL but leverages the vast
efforts of prior domain experts to bypass reward definitions; thus, we turn
towards offline IL. Specifically, we seek to leverage BC to extract an expert
BWE policy from offline expert demonstrations for real-time BWE.

Overview. For our work, we seek to mimic our expert UKF model, a rule-based
model constructed from extensive domain expertise. UKF, like WebRTC, adopts a
delay-based approach; that is, based on observed network delays, UKF smoothly
adapts its bandwidth estimates. More concretely, UKF utilizes an internal
dynamics model to represent the current network state and a set of static
functions to adapt its bandwidth estimate. In contrast to WebRTC, UKF was
designed specifically for videoconferencing. The estimates produced by UKF do
not follow an additive increase multiplicative decrease (AIMD) scheme, which
leads to smoother bandwidth estimates in comparison to WebRTC’s “sawtooth”
behavior. UKF has previously been deployed on Microsoft Teams; hence, it is a
battle-tested expert for real-time bandwidth estimation. Additionally, as
extensive domain expertise is required to adjust UKF, it is the perfect
candidate for our work. Thus, given a set of collected UKF demonstrations
$\Xi$, UKF states $S$, and UKF actions $\pi^{*}(s)$, we seek to learn the
expert policy $\pi^{*}$ in the following manner:

$\pi^{*}=\operatorname*{arg\,min}_{\pi}\sum_{\xi\in\Xi}\sum_{s\in
S}L(\pi(s),\pi^{*}(s))$ (1)

where $\pi$ corresponds to the policy of our imitator.

By reframing policy learning in the context of supervised learning, BC enables
agents to learn a control policy while benefiting from the stability and
convergence properties of supervised learning. Despite these benefits, BC
suffers from the problem of compounding error [53]. Supervised learning relies
on the i.i.d. assumption which may not hold during long-horizon tasks.
Trajectories contain a sequence of states and actions that often depend
temporally, breaking the i.i.d. assumption [45]. As a result, when a BC model
arrives at an unseen state, the newly executed action may be misaligned with
the true objective, diverging slightly from the expert trajectory. The
dependence between states causes the error to compound as the BC agent moves
along its trajectory, diverging more and more from the expert. Compounding
error has been shown to limit the robustness of BC models as the learned
agents are incapable of bridging to new unseen environments [60]. To tackle
real-world applications, IL methods must also overcome the distribution shift
between offline demonstrations and target environments.

Addressing Compounding Error. To combat compounding error and improve
generalization, we utilize a large number of expert demonstrations. We utilize
OpenNetLab’s gym environment to collect expert trajectories. Specifically, we
collect $100$k expert demonstrations from simulation, randomly varying the
call parameters to improve data diversity. By sampling the expert from a
diverse set of circumstances, Merlin is able to better observe the expert from
a variety of states, which enhances state space coverage and mitigates
compounding error. Furthermore, to mitigate the effects of compounding error
that may arise due to the sheer size of the output range, we restrict
bandwidth estimates to $10$ Kbps and $8$ Mbps, a range that supports audio-
only calls to high-definition videoconferencing calls with screen sharing. We
additionally limit our action space $\hat{b}_{log}$ to a real number between
$0$ and $1$ and employ a log transform to project Merlin’s output action into
bps,

Figure 3: Merlin’s Network Architecture.

$\hat{b}_{log}=\frac{\log(\hat{b})-\log(b_{min})}{\log(b_{max})-\log(b_{min})}$
(2)

where $\hat{b}$ represents the estimated bandwidth, $b_{min}$ corresponds to
the minimum bandwidth, $b_{max}$ is the maximum bandwidth, and all bandwidth
values are in Mbps. Limiting the action space to a number between $0$ and $1$
reduces the complexity of our action space and helps our model learn a more
robust policy. The impact of these design decisions is reflected in the
results detailed in section 5.

Extracting BWE Signals in Partially Observable Environments. To deal with
partial observability, state-of-the-art heuristic-based methods such as WebRTC
combine queuing theory with raw network metrics to model one-way delay
gradients for BWE [7]. In contrast, current RL methods start from a blank
slate and learn to extract complex BWE signals from raw network metrics. Both
methods, while performant, neglect the benefits of each other; that is,
heuristic-based methods are fundamentally limited by the capacity of domain
experts and cold-start RL overlooks prior efforts. Accordingly, Merlin is
designed to learn these domain-specific representations implicitly through
expert supervision via IL. Similarly, Merlin relies on its data-driven
architecture to extrapolate more complex signals; specifically, we incorporate
a Long Short Term Memory (LSTM) unit to maintain the history of previous
network behavior. The LSTM acts as a buffer for experience, and, over the
duration of a videoconferencing call, Merlin builds up its internal state
representation to learn temporally-dependent features that capture the non-
stationary nature of the network environment. The robustness to partial
observability is validated in section 5.

In addition to incorporating expert supervision and learned feature
representations, we conducted an exhaustive feature ablation study detailed in
section 5 to arrive at the best performing state representation (detailed in
Appendix A). Most notably, we experimented with including the five previous
bandwidth estimates which were sampled at $60$ ms granularity. However, as we
utilize offline expert trajectories for training, these estimates actually
correspond to our expert’s previous predictions. It was observed that these
features hindered Merlin’s ability to extract the expert policy. We
hypothesize that the previous estimates led to Merlin placing more weight on
these previous estimates; effectively, “cheating” by reusing UKF estimates for
bandwidth prediction. Thus, when generalizing to new environments, Merlin
would perform poorly in comparison to our expert UKF model. Pruning these
previous estimates greatly enhanced Merlin’s robustness to domain shift.

Tackling Domain Shift. As previously mentioned, prior heuristic methods were
constructed by domain experts; hence, updating these models is non-trivial.
The current process for updating these methods relies on a time-consuming
processing and entails collecting network statistics, manually hand-
engineering representations, and redeploying these heuristics, repeating until
an acceptable measure of success is achieved. As network heterogeneity
increases, specializing these models for individuals and adapting model
parameters to each new environment quickly becomes infeasible. On the other
hand, RL agents can be readily fine-tuned with new observations; however,
estimates tend to be noisier, reacting to instantaneous network perturbations.
In contrast, Merlin utilizes IL for expert supervision to mitigate overly
aggressive reactions to network perturbations, effectively regularizing
bandwidth estimates. In combination with expert supervision, Merlin’s state
construction utilizes both short term and long horizon features to promote
smooth bandwidth estimates. As a first step, IL enables learning a policy for
smooth expert bandwidth estimates, while the data-driven design facilitates a
personalized experience by fine-tuning on new observations. Merlin’s
resilience to domain shift is empirically validated by generalizing from
offline simulated observations to real, inter-continental videoconferencing
calls, an environment where state observations deviate significantly from
offline experience, in section 5. Merlin’s architecture and policy network are
described next.

Architecture Details. We choose to utilize an LSTM for our policy network to
exploit the temporal correlations present in network traffic and mitigate
compounding error. We benchmark against non-recurrent architectures in section
5. Our findings demonstrate that the recurrent structure of the LSTM enhances
network control performance by exploiting the temporal dependencies between
network control actions. Merlin accepts a $64$ length tensor of normalized
state observations. As depicted in Figure 3, Merlin first encodes state
observations with its LSTM and then leverages two fully-connected layers to
decode state observations into output actions. The first fully-connected layer
is followed by a ReLU activation function while the final output layer
utilizes a sigmoid activation function, limiting the model output to a real
number between $0$ and $1$. It is important to note that we utilize a small,
lightweight architecture for client-side deployment and real-time execution
($\approx 60$ ms granularity).

Figure 4: System overview of Merlin.

Training Procedure. As illustrated in Figure 5, we train Merlin from offline
UKF demonstrations. First, we deploy our expert UKF in a fork of OpenNetLab’s
gym environment [12]. The gym leverages the WebRTC stack and NS3 [44] to
simulate videoconferencing calls to promote data-driven BWE solutions. Network
behavior is simulated according to the provided trace workloads and network
parameters. Our calls are generated from a diverse set of production traces,
encompassing low bandwidth, high bandwidth, fluctuating bandwidth, burst loss,
and LTE workloads. During generation, we randomly vary call parameters such as
the proportion of video packets to audio packets, queuing delays, and the
video start times. At each step in the simulation, transport layer information
such as the send time, arrival time, and payload size is reported. The gym
environment provides tools for calculating relevant transport metrics such as
the delay, loss ratio, and receiving rate from the exposed packet level
information.

We record the observed packet level information and UKF bandwidth predictions
from $100$k unique calls. We apply the inverse of equation 2 to project UKF’s
bandwidth estimate into action space for training. It is important to note
that gym packets do not contain real video and audio payloads; hence, there is
a gap between real and simulated videoconferencing calls. The sim-to-real gap
limits the breadth of expert demonstrations; however, the programmatic nature
and customizability of call parameters enable us to collect expert
demonstrations from a diverse set of circumstances that may not be realizable
under real-world conditions. Although our expert is not observed in real-world
environments, we emphasize that the diverse set of observations enables our
model to bridge the gap from simulated networks to in-the-wild deployments.

Figure 5: From Offline Demos to Agent Deployment, Training Merlin via
Imitation Learning.

Before training, an offline dataset of state observations was constructed by
grouping packets received in $60$ ms windows and aggregating feature-level
information accordingly. Features were reconstructed based on the state
representation detailed in Appendix A. The sequence of states and actions from
one call was treated as a single training sample. To train Merlin, a batch
size of $256$ was utilized, i.e., $256$ calls were ingested per training step.
The mean squared error (MSE) between Merlin’s actions and UKF’s expert actions
was minimized over the course of $1000$ training epochs. Different loss
functions such as mean-absolute-error (MAE) were empirically tested,
indicating that the MSE loss objective produced the best imitation policy. To
update model parameters and dynamically tune the learning rate, Adam
optimization was employed with an initial learning rate of $0.001$. After each
training epoch, utilizing randomly generated videoconferencing workloads and
OpenNetLab’s gym, the MSE validation performance between Merlin and UKF was
reported (see Section 5 for more details on workload generation). Most
importantly, Merlin never interacted with a network nor was a single packet
transmitted during train time. Furthermore, training data was collected once–
prior to training.

Implementation. We implement Merlin’s LSTM architecture in Pytorch. We export
Merlin to ONNX format and utilize ONNX runtime for deployment. ONNX provides a
basic model wrapper that enables compatibility across operating systems and
applications. Merlin can be deployed directly into the Teams media stack via
ONNX runtime for receiver-side estimation. By extracting an expert policy from
offline experience, we succeed in building upon prior heuristic-based
estimators for BWE. That is, we utilize the encoded domain knowledge to learn
a generalizable, data-driven bandwidth estimator without ever transmitting a
single network packet.

##  5 Evaluation

In this section, we seek to answer the following questions:

  1. 1.

Is offline, simulated experience alone sufficient to produce expert bandwidth
estimates?

  2. 2.

How robust is the learned BWE policy to domain shift?

  3. 3.

Which features are most important to mimicking an expert estimator?

  4. 4.

How do different IL architectures and techniques impact bandwidth estimates?

  5. 5.

How does the quantity and quality of demonstrations affect BWE performance?

###  5.1 Methods

Environment Assumptions and Parameters. For our evaluations, we sample state
observations at $60$ ms granularity, enabling real-time BWE. Bandwidth
estimates are clipped to be between $10$ Kbps and $8$ Mbps, which supports a
broad range of RTC applications, ranging from audio-only calls to high-
definition, $30$ frames per second videoconferencing with interactive screen
sharing. Estimates below $10$ Kbps are below industry recommendations for
audio-only calls, while estimates above $8$ Mbps offer no added benefits.
Lastly, we restrict our evaluations to peer-to-peer, audio and video calls
(see Figure 6) and leave group calls for future work.

Benchmarks. We benchmark against two different state-of-the-art bandwidth
estimators. (1) WebRTC [4], the de facto standard for RTC applications that
utilizes GCC for BWE. GCC leverages a Kalman filter to tune bandwidth
estimates based on the estimated end-to-end one-way delay variation [7],
employing an AIMD scheme. WebRTC is the most widely utilized communication
stack for RTC applications. (2) R3Net v1.4, a variant of the online RL model
proposed by Fang et al. in [13]. R3Net v1.4 was previously benchmarked against
HRCC [54] and Gemini [61], the winners of MMSys ’21 Bandwidth Estimation
Challenge, and shown to outperform both models, achieving state-of-the-art
performance for RL-based BWE. In addition to baseline comparisons, we evaluate
Merlin’s imitation quality in relation to our expert UKF, a custom, hand-
engineered statistical model previously deployed in production on Microsoft
Teams.

Metrics. For our simulated evaluations, we track the MSE between Merlin and
UKF in action space as our sole key performance indicator. A small action MSE
indicates that the actions produced by our BC model closely match those
produced by UKF, while a larger difference indicates a deviation between the
imitator and the expert. While calls are simulated, the simulated packets do
not carry any meaningful payload, so we do not compute gold-standard metrics
such as the video MOS in this environment.

In contrast to our simulated environments, we track the following metrics in
our testbed and wild environments. (1) Video MOS. The MOS values range from
$1$ to $5$ with $1$ indicating poor QoE and $5$ indicating exceptional QoE.
Video MOS is the gold standard for QoE and is computed based on user feedback.
In our work, we utilize a vision-based model to produce a video MOS estimate
[41]. The estimates were shown to exhibit $99\%$ correlation with user visual
experience. (2) Audio MOS. Similar to the video MOS, the audio MOS also ranges
from $1$ to $5$ with $1$ indicating low audio QoE and $5$ indicating high QoE.
Audio MOS is the gold standard for audio-based QoE and is computed based on
user feedback. In our work, we utilize a signal-based model to produce an
audio MOS estimate. The estimates were internally shown to correlate
significantly with user audio experience. (3) Receiving rate. The receiving
rate is reported in Kbps and correlates with user experience. While a higher
receiving rate is preferred, delay and packet loss must be taken into account
as a high receiving rate can correlate with network congestion. (4) Packet
loss rate. The percentage of lost packets observed. A lower loss rate
indicates that packets are freely flowing through the network while a high
loss rate indicates network congestion and degradation in user QoE. (5) Delay.
The delay metric is reported in ms, with a lower delay indicating higher QoE
and a higher delay indicating network congestion. We choose to track the delay
mean as opposed to the delay MOS, as a notable shift in delay is required to
move the delay MOS score significantly, e.g., $3$ ms increase in delay
corresponds to an observed $0.001$ increase in delay MOS (a higher delay MOS
is worse). Furthermore, it is important to note that while multiple network
metrics correlate to user experience, the relationship is complex. Thus, to
understand the overall impact on user experience, we must analyze these
metrics collectively rather than individually.

Simulated Evaluation. We evaluate the performance of Merlin against UKF using
traces generated from production parameters within a fork of OpenNetLab [12].
We randomly generate traces for evaluation from a diverse set of network
workloads containing low bandwidth, high bandwidth, fluctuating bandwidth,
burst loss, and LTE. Call parameters such as the proportion of video packets
to audio packets, queue delay, and the video start time are randomly sampled
at runtime. Our evaluation consists of $480$ distinct simulated calls. We run
$1000$ validation runs, which corresponds to nearly $480,000$ simulated calls.
We report the best achieved performance. It is important to note that in real
videoconferencing calls, video packets are not received at the start of the
call. This is because audio packets tend to flow first, which leads to a sharp
change in bandwidth once video packets are received. Our randomly generated
traces capture the variation in delay of video streams. This shift in
bandwidth leads to more challenging traces for BWE.

Testbed Evaluation. We benchmark the performance of Merlin against WebRTC,
UKF, and R3Net v1.4 using production traces over emulated networks within our
testbed environment. We utilize $10$ different production traces which cover
low bandwidth, high bandwidth, fluctuating bandwidth, and burst loss settings.
Our testbed consists of two lab machines connected over a network link. The
network link is emulated to match production settings. Since we are
transmitting over real networks, other factors such as cross-traffic and
queuing delays influence the results of our evaluation. To mitigate noise, we
conduct hundreds of evaluation calls and utilize a Welch t-test to determine
whether our results are statistically significant. Our evaluation consists of
over $100$ emulated calls per model ($\approx 400$ calls in total at $10$ per
trace). We report the averaged metrics across each network profile. In
relation to UKF, we seek to accept the null hypothesis, that is, there is no
difference between the performance of the imitator and the expert within our
emulated environment. In contrast, we seek to outperform existing methods such
as WebRTC and R3Net v1.4.

In the Wild Evaluation. We measure the performance of Merlin against UKF and
WebRTC over real networks in the wild. Our setup consists of $20$ nodes
distributed across North America, Asia, and Europe. For each evaluation call,
we randomly sample $10$ pairs of nodes. We then conduct calls with UKF,
WebRTC, and Merlin. We conducted our experiments during the day and at night
over the course of a week. Similar to our emulated evaluation, we conducted
hundreds of evaluation runs and utilized a Welch t-test to determine whether
our results were statistically significant. Our evaluation consists of over
$700$ in the wild calls per model ($\approx 2100$ calls in total). We report
the averaged metrics across all runs. In relation to UKF, we seek to accept
the null hypothesis, that is, there is no difference between the performance
of the imitator and the expert within real deployments. In contrast, we seek
to improve upon WebRTC.

Ablation Studies. We experiment with different learning parameters and
evaluate our imitator against UKF using randomly generated workloads within
our simulated environment. The setup is identical to our simulated
evaluations. We report the best achieved performance across each parameter
setting. We experiment with different architectures, IL methods, input
features, demonstration numbers, and types of demonstrations. For an in-depth
review of the IL methods tested, we direct readers to Appendix E.

Key Findings. Through our evaluations, we demonstrate the following findings:

  1. 1.

Offline experience is sufficient to produce expert bandwidth estimates. Merlin
outperforms the state-of-the-art bandwidth estimators in terms of video MOS,
audio MOS, and the average receiving rate. We show that the change in video
MOS and receiving rate are statistically significant. Merlin also shows no
statistical movement in comparison to UKF on real networks.

  2. 2.

Our learned BC policy is robust to domain shift. We train on offline
experience from simulated UKF calls and generalize to both emulated and wild
networks. We further show that these results are statistically significant by
showing no movement in terms of video MOS and audio MOS.

  3. 3.

The most important features for mimicking an estimator and producing expert
bandwidth estimates are the receiving rate and the media type features.
Counter-intuitively, we find that the average packet loss rate and loss ratio
have little effect on the learned predictor. The best subset of features
contains all five input feature categories.

  4. 4.

Rather than using demonstrations drawn from our target environment, we find
that the richness and diversity of demonstrations contributes more to the
performance of the imitator; hence, using a diverse set simulated data is
sufficient for policy extraction.

###  5.2 Simulated Audio and Video Calls

We first assess the ability of Merlin to imitate our expert UKF bandwidth
estimator. Merlin achieves an MSE difference of $0.0012$ in action space in
comparison to UKF over $480$ randomly generated traces (see LSTM-BC in Figure
10(b)). While opaque, these results indicate that Merlin can closely mimic our
expert UKF estimator which is evident in our qualitative assessment in Figure
7. Most notably, in Figure 7(a), we see that Merlin inherits the same quirks
as our expert. UKF takes a more conservative approach to BWE to produce
smoother estimates over the duration of the videoconferencing call; as a
result, UKF avoids abrupt changes that are prominent in the fluctuating case.
Furthermore, our expert produces estimates based on the observed packets, and
since video packets tend to not flow immediately, our expert severely
undershoots the true bandwidth at the beginning of the high bandwidth call in
Figure 7(b). Since audio packets only require $10$ Kbps of bandwidth, both the
expert and imitator severely undershoot the true bandwidth at the beginning of
the call; however, as soon as bandwidth-hungry video packets begin to flow
across the network, both the imitator and UKF exhibit the same behavior of
smoothly ramping up to the bandwidth limit. Both quantitatively and
qualitatively, we demonstrate that our imitator is capable of mimicking our
expert from purely offline experience within simulation. Our simulation mainly
serves as a validation check that our imitator works as expected. We discuss
more rigorous evaluations in the coming evaluation sections.

Figure 6: Deploying estimators on videoconferencing clients.

(a) Fluctuating Bandwidth.

(b) High Bandwidth.

Figure 7: Imitating UKF in Simulation.

###  5.3 Testbed Videoconferencing

We compare the performance of our learned imitator against three benchmark
models: WebRTC, UKF, and R3Net v1.4 over controlled networks with emulated
links. We note that cross-traffic and other noise contributors are present
within the testbed environment. We test each estimator across a diverse set of
production network traces and aggregate statistics across $\approx 100$ calls
for each method. Our results are summarized in Table 1. In terms of video MOS
scores, Merlin outperforms both WebRTC and R3Net v1.4, two state-of-the-art
methods. Merlin achieves a $3\%$ improvement over WebRTC and a $0.3\%$
improvement against R3Net v1.4 in terms of video MOS. The movement in video
MOS is statistically significant. As for audio MOS, Merlin beats WebRTC and
R3Net v1.4 by $0.4\%$ and $1.7\%$ respectively. While Merlin attains a modest
improvement over state-of-the-art methods, we emphasize that Merlin is trained
completely from offline experience while R3Net v1.4 required millions of
online network interactions to converge and WebRTC involved extensive domain
knowledge and hand-engineered rules to attain comparable performance. It is
important to note that WebRTC, while the standard for RTC, is designed to be a
general purpose RTC stack as opposed to specializing in videoconferencing. Due
to its general purpose nature, UKF’s specialized estimates outperform WebRTC;
hence, as the imitator, Merlin in turn outperforms WebRTC. In contrast, the
performance gap between Merlin and R3Net v1.4 is likely due to limitations in
generalizability. For example, while heuristic-based models like UKF were
designed with domain expertise to ensure domain adaption, R3Net v1.4 was
trained stochastically to maximize its reward, which may not fully map to user
subjective experience across network settings.

The evaluations further indicate that Merlin regresses in terms of the packet
loss rate and delay mean against R3Net v1.4 and WebRTC respectively; however,
Merlin attains a higher receiving rate which translates to more packets
flowing across the network, triggering an increase in both the loss rate and
delay mean. First, when the estimator produces higher estimates, a greater
receiving rate can be attained. These estimates are fed to the codec and the
codec itself makes executive decisions such as assigning more parity packets
to the link to mitigate packet loss through forward error correction (FEC).
Thus, with a higher receiving rate, we can expect the loss rate and congestion
to increase which degrades the delay metric. However, we emphasize that the
$3$ ms delay regression against WebRTC is negligible as it translates to a
delay MOS increase of $\approx 0.001$. The movement in receiving rate and
delay are statistically significant. Despite these regressions, Merlin
enhances both video MOS and audio MOS, which are two established objective QoE
metrics, when compared with WebRTC and R3Net v1.4.

Lastly, we compare against UKF on our testbed environment to study how well
Merlin imitates our expert. Quantitatively, in terms of video MOS, audio MOS,
and the receiving rate, we observe no statistically significant movement
across our production traces (see Table 1), which indicates that these metrics
are statistically drawn from the same distribution. However, Merlin does incur
a degradation in terms of both the loss rate and delay in comparison to UKF.
While close, the small deviation in loss rate and delay may indicate that
Merlin fails to entirely mimic our expert. One possible explanation for this
regression is the domain shift between our simulated environment and the
testbed environment. Since Merlin is trained entirely from offline simulated
data, the delay metrics observed at train time may not fully match our target
environment. This is because networks fall under the umbrella of partially
observable environments as links outside the subnet cannot be probed directly.
As a result, many of these factors cannot be accurately simulated within our
simulated environment, leading to a divergence in delay metrics. One possible
resolution would be to incorporate real-world data into our offline dataset,
enabling Merlin to observe the shift in delay computation. Despite this
regression, in terms of our gold standard MOS metrics, Merlin outperforms the
state-of-the-art and demonstrates no statistical movement in comparison to the
expert, demonstrating that Merlin is capable of generalizing to new
environments from simulated, offline expert observations. Qualitative results
are presented in Appendix C.

###  5.4 Videoconferencing in the Wild

We compare Merlin against WebRTC and UKF over real network links. We deploy
and evaluate each estimator over real networks with links spanning multiple
continents. We aggregate statistics from $\approx 2100$ calls. Our results are
summarized in Table 2 and Figure 9. The results demonstrate that Merlin is
capable of generalizing from simulated to real-world environments, performing
competitively with our expert UKF and WebRTC. We observe competitive audio MOS
performance against both UKF and WebRTC, differing by $0.2\%$ and $0.1\%$
respectively. In terms of video MOS scores, Merlin regresses by $2.9\%$
against UKF; however Merlin outperforms WebRTC by $3.75\%$. Similarly, Merlin
leads to a $0.4\%$ boost against WebRTC and a $13\%$ reduction in comparison
to UKF in terms of the observed receiving rate. We note that Merlin achieves a
higher median video MOS and receiving rate than UKF; however, UKF’s video MOS
and receiving rate metrics exhibit higher variance compared to Merlin (see
Appendix D). Although Merlin regresses, we emphasize the two following
aspects. One, Merlin was trained purely from offline, simulated experience
with zero network interactions and is able to outperform WebRTC and compete
with UKF. Two, while we have not solved the problem of domain shift
completely, Merlin’s data-driven design enables the potential to reduce the
performance gap by incorporating more data and finetuning our objective
function.

In contrast, the evaluations further indicate that Merlin improves in terms of
the packet loss rate and delay against both UKF and WebRTC. The difference in
observed metrics translates to a $19\%$ and $42.85\%$ loss rate improvement
against UKF and WebRTC respectively. In terms of observed delay, Merlin
achieves a $4.9\%$ and $12.8\%$ gain over UKF and WebRTC respectively. Similar
to the previous evaluation, we emphasize that UKF achieves a higher receiving
rate, leading to more packets entering the network and potentially increasing
the packet loss and delay; however, in comparison to WebRTC, Merlin achieves a
higher receiving rate while reducing both packet loss and delay which
translates to improved QoE. We hypothesize that this performance regression is
a direct result of WebRTC’s general purpose nature, resulting in regressions
against more specialized methods (e.g., Merlin) for RTC applications such as
videoconferencing.

Lastly, we show how well Merlin’s extracted policy generalizes to a real-world
environment. Specifically, we observe no statistically significant movement of
Merlin against UKF within a real environment, indicating that Merlin is
retaining the expert policy obtained from simulated experience even within
environments that differ significantly from our simulation. Overall, Merlin
produces competitive bandwidth estimates with our expert UKF model, while
outperforming WebRTC across various settings. Our evaluations demonstrate that
Merlin is capable of generalizing to new environments from simulated, offline
expert observations.

Model | Video MOS | Audio MOS | Receiving Rate | Loss Rate | Delay Mean
---|---|---|---|---|---
Merlin | 2.9150 $\pm$ 0.1538** | 2.8598 $\pm$ 0.0821 | 849.9764 $\pm$ 104.4793** | 0.0353 $\pm$ 0.0144 | 30.8500 $\pm$ 4.3147
UKF | 2.9068 $\pm$ 0.2004 | 2.8401 $\pm$ 0.0795 | 829.7375 $\pm$ 127.6 | 0.0328 $\pm$ 0.0105 | 28.0160 $\pm$ 3.5423
WebRTC | 2.8519 $\pm$ 0.1217 | 2.8452 $\pm$ 0.0667 | 775.3979 $\pm$ 60.2634 | 0.0375 $\pm$ 0.0053 | 26.1996 $\pm$ 2.5462**
R3Net v1.4 | 2.9050 $\pm$ 0.2438 | 2.8094 $\pm$ 0.0897 | 847.9143 $\pm$ 148.3244 | 0.0328 $\pm$ 0.0163 | 42.2350 $\pm$ 7.4667
Table 1: Benchmarking on Emulated Links. **Statistical significance with
$p<0.05$ between Merlin, WebRTC, and R3Net v1.4.

(a) Low Bandwidth.

(b) Fluctuating Bandwidth.

(c) High Bandwidth.

Figure 8: Video MOS vs. Audio MOS performance across emulated network environments. Model | Video MOS | Audio MOS | Receiving Rate | Loss Rate | Delay Mean
---|---|---|---|---|---
Merlin | 4.069 $\pm$ 0.6204 | 4.811 $\pm$ 0.1344 | 1910.521 $\pm$ 705.7378 | 0.0021 $\pm$ 0.0136 | 6.856 $\pm$ 26.7283
UKF | 4.190 $\pm$ 0.3236 | 4.824 $\pm$ 0.1218 | 2159.809 $\pm$ 311.1544 | 0.0025 $\pm$ 0.0177 | 7.191 $\pm$ 15.5186
WebRTC | 3.919 $\pm$ 0.4873 | 4.817 $\pm$ 0.1674 | 1901.152 $\pm$ 435.089 | 0.0030 $\pm$ 0.0183 | 7.730 $\pm$ 28.2775
Table 2: Benchmarking on Wild Networks. Figure 9: Video MOS vs. Audio MOS on
Wild Networks.

###  5.5 Guidance for Bandwidth Estimation

Impact of Features. We seek to study the impact of different features on
learned BWE; specifically, handcrafted features with domain knowledge may
impede our learned estimators. We group features into five categories and
ablate on these groups: receiving rate, loss ratio, average number of lost
packets, queuing delay, and media type features. The media type features
report the probability mass of video packets, audio packets, and screen
sharing packets over the last three time steps. We then exhaustively retrain
Merlin on each feature subset and report the performance on generated
production traces within our simulated environment. We report our findings in
Figure 10(a). Our experiments indicate that the two most impactful features
are the receiving rate and the media type features. Surprisingly, we find that
the receiving rate feature alone is sufficient to mimic UKF with an action
space MSE of $\approx 0.0028$. The impact of the receiving rate is expected,
but the magnitude is unexpected. The receiving rate corresponds to the rate of
information that the receiving endpoint is capable of receiving; hence, we
would expect the receiving rate to correlate heavily with the throughput
achieved at the bottleneck link.

Similarly, the media type feature alone enables Merlin to achieve approximate
action space errors of $0.0037$ in relation to UKF; in contrast, while the
queuing delay feature bolsters our learned estimator, the queuing delay, loss
ratio, and average loss features individually are insufficient to mimic our
expert (see Appendix B). The ablation results further indicate that without
either the receiving rate or media type features, Merlin is unable to learn a
competitive BWE policy. One explanation is that both receive rate and media
type features encode the observed audio and video receive rates. The receive
rate feature reflects the combination of both audio and video receive rates,
while the media type features provide estimates of the relative proportion of
audio and video packets received over a given window. Depending on the
workload, the proportion skew heavily influences bandwidth estimates. Since
audio requires only $10$ Kbps of resources, the flow of audio packets remains
relatively constant; hence, when the video packet to audio packet proportion
is high, link resources correspondingly are abundant to support the flow of
high-quality video which is further corroborated by the impact of the video
start times detailed in section 5.2. In combination, both the audio receive
rate and video receive rate can be derived from these two features, which
together help improve the quality of bandwidth estimates.

While loss metrics provide the agent with auxiliary information, by themselves
they are noisy. We hypothesize that this discrepancy in performance stems
directly from partial observability. For example, while packet loss may result
from network congestion and exhausted link resources, loss may additionally
arise from physical factors, e.g., fading in wireless networks and link level
disconnections– both of which are indistinguishable from the agent’s
perspective and impact loss rates regardless of the bottleneck’s available
resources. Lastly, we find that the best performing subset of features
contains all five feature groups (see Appendix B for more details). It appears
that the receiving rate, packet type, and delay-based features serve as core
features and are crucial towards real-time BWE, while loss-based features
appear to bolster performance but operate as auxiliary information.

(a) Imitation Performance with Different Feature Sets: Receive Rate subsets
(RR); Media Type subsets (MT); No Receive Rate or Media Type features (No RR +
No MT).

(b) Performance of various Imitation Learning methods for bandwidth
estimation.

(c) Quantity and quality of demonstrations vs. imitation performance.

Figure 10: Ablation Study Results.

Learning Methods and Architecture. We evaluate the performance of different IL
methods and architectures for BWE within our gym environment. We compare three
different IL approaches: BC, Implicit Behavioral Cloning (IBC) [15], and
Generative Adversarial Imitation Learning (GAIL) [20] and two policy
architectures for BC: a multi-layer perceptron (MLP) and an LSTM. We report
our findings in Figure 10(b). We implement IBC and GAIL with MLP-based policy
networks. We adopt similar hyperparameters for each model and maintain the
same gym validation parameters across trials. We train GAIL with $16000$
expert state-action samples per training epoch. Our evaluations indicate that
BC with an LSTM policy network outperforms all other benchmarked methods,
achieving an action MSE error of $0.0016$ (see Figure 10(b)). The next best
performing approach is BC with an MLP-based policy network, followed by IBC
and then GAIL. We hypothesize that IBC and GAIL are more effective than BC in
environments with limited expert demonstrations due to their joint
representation of states and actions; however, in our case, we have an
abundance of rich expert demonstrations and are effectively able to better
cover our state space, mitigating the likelihood our imitator arrives at a
previously unseen state which in turn reduces the effect of compounding error.
Lastly, we ablate on the size of Merlin’s LSTM; however, we observe little
performance difference between model sizes (see Appendix B). Given that Merlin
produces a single output from $64$ inputs, the problem is relatively small;
hence, we would not expect model size to severely influence performance. In
essence, when expert demonstrations are readily available, BC provides
favorable results. Furthermore, exploiting temporal correlations present in
network control tasks with recurrent networks appears to enhance policy
performance.

Quantity and Quality of Demonstrations. We explore the impact of the quantity
and quality of demonstrations on our learned estimator. Specifically, we
compare with expert demonstrations drawn from two environments: our simulation
and our testbed environment. We retrain Merlin on four different datasets:
$1.5$k emulated trajectories, $10$k emulated trajectories, $10$k simulated
trajectories, and $100$k simulated trajectories. The emulated trajectories are
intentionally limited; that is, we do not randomize call parameters and
leverage parameters drawn directly from production traces. By limiting the
breadth of demonstrations from the target distribution, the relation between
demonstration diversity and data quality can be explored. Emulated data is of
higher quality as it reduces the domain shift between offline and online
samples.

Furthermore, we evaluate the implications of DAGGER [45] on our $10$k gym
dataset, doubling the dataset size to $20$k demonstrations by the end of the
training run. We report our findings in Figure 10(c). First, comparing the
performance of Merlin on $10$k emulated trajectories and $10$k simulated
trajectories, we observe greater gym performance with our simulated
trajectories rather than our emulated demonstrations. Merlin achieves an
action MSE error of $0.0016$ with the simulated dataset in comparison to
$0.0055$ action MSE error with the emulated demonstrations. Furthermore,
qualitatively, we observe that utilizing $10$k simulated demonstrations is
superior to training on $10$k emulated demonstrations even when evaluating on
our emulated platform (see Appendix C). We hypothesize that this robustness to
domain shift is due to the reduction of compounding error; specifically, since
our simulated dataset contains a richer set of trajectories, i.e.,
trajectories unlikely to be encountered in real deployments, Merlin is able to
observe our expert across a more diverse set of circumstances which bolsters
performance. Second, comparing the performance of $10$k gym samples, $20$k gym
samples collected with DAGGER, and $100$k gym samples, we observe little added
benefit within our gym environment. Our results indicate a modest improvement
of $0.0003$ in terms of action MSE with $80$k extra demonstrations compared to
DAGGER-enhanced training; however, when benchmarking each method on our
testbed, we observe that the model trained with $100$k demonstrations performs
the best overall. While we observe that $10$k demonstrations are sufficient to
learn a BWE policy, increasing the number of expert observations appears to
improve generalization. Furthermore, data diversity appears to impact
imitation performance more than demonstration quality. Thus, to learn network
control policies via offline expert demonstrations, we find that providing the
agent with a large number of diverse demonstrations is key to ensuring
robustness against domain shift which is corroborated in [60].

##  6 Conclusion

In our work, we tackle key challenges in adopting AI-based system optimization
and control such as domain shift. Through our evaluations, we demonstrate
Merlin as a data-driven solution that builds upon prior network control
heuristics. We provide preliminary results demonstrating the promise of
offline learning for learning real-time network control policies. Although
Merlin learns a robust BWE policy and outperforms state-of-the-art rule-based
and learning-based methods, Merlin is not the end for data-driven BWE. For
example, we evaluate only audio and video calls in peer-to-peer setups;
however, many videoconferencing calls consist of multiple endpoints
communicating over group video calls which involve a single centralized
server. As flows are concentrated at a single node, the complexity of BWE
increases as competing receiver flows may impede one another. Furthermore,
warm-starting RL agents with policies extracted from BC have been shown to
produce strong results [36] which may translate to improved network control
policies. We hope that Merlin’s offline-oriented design fosters new strategies
for real-time network control.

## Acknowledgements

We would like to thank Lili Qiu for her highly valuable feedback. We thank
Scott Inglis and Ezra Ameri for their help.

## References

  * [1] Soheil Abbasloo, Chen-Yu Yen, and H Jonathan Chao.  Classic meets modern: A pragmatic learning-based congestion control for the internet.  In Proceedings of the Annual conference of the ACM Special Interest Group on Data Communication on the applications, technologies, architectures, and protocols for computer communication, pages 632–647, 2020\.
  * [2] Pieter Abbeel and Andrew Y Ng.  Apprenticeship learning via inverse reinforcement learning.  In Proceedings of the twenty-first international conference on Machine learning, page 1, 2004.
  * [3] Abdelhak Bentaleb, Mehmet N Akcay, May Lim, Ali C Begen, and Roger Zimmermann.  Bob: Bandwidth prediction for real-time communications using heuristic and reinforcement learning.  IEEE Transactions on Multimedia, 2022.
  * [4] A Bergkvist, D Burnett, C Jennings, and A Narayanan.  Webrtc 1.0: Real-time communication between browsers. w3c working draft.  World Wide Web Consortium, 2012.
  * [5] Rajarshi Bhattacharyya, Archana Bura, Desik Rengarajan, Mason Rumuly, Bainan Xia, Srinivas Shakkottai, Dileep Kalathil, Ricky KP Mok, and Amogh Dhamdhere.  Qflow: A learning approach to high qoe video streaming at the wireless edge.  IEEE/ACM Transactions on Networking, 30(1):32–46, 2021.
  * [6] Neal Cardwell, Yuchung Cheng, C. Stephen Gunn, Soheil Hassas Yeganeh, and Van Jacobson.  Bbr: Congestion-based congestion control: Measuring bottleneck bandwidth and round-trip propagation time.  Queue, 14(5):20–53, oct 2016.
  * [7] Gaetano Carlucci, Luca De Cicco, Stefan Holmer, and Saverio Mascolo.  Analysis and design of the google congestion control for web real-time communication (WebRTC).  In Proceedings of the 7th International Conference on Multimedia Systems - MMSys ’16, pages 1–12, Klagenfurt, Austria, 2016\. ACM Press.
  * [8] Ke Chen, Han Wang, Shuwen Fang, Xiaotian Li, Minghao Ye, and H Jonathan Chao.  Rl-afec: adaptive forward error correction for real-time video communication based on reinforcement learning.  In Proceedings of the 13th ACM Multimedia Systems Conference, pages 96–108, 2022.
  * [9] R. Droms.  Automated configuration of tcp/ip with dhcp.  IEEE Internet Computing, 3(4):45–53, 1999.
  * [10] Zhuoxuan Du, Jiaqi Zheng, Hebin Yu, Lingtao Kong, and Guihai Chen.  A unified congestion control framework for diverse application preferences and network conditions.  In Proceedings of the 17th International Conference on emerging Networking EXperiments and Technologies, pages 282–296, 2021.
  * [11] Salma Emara, Baochun Li, and Yanjiao Chen.  Eagle: Refining congestion control by learning from the experts.  In IEEE INFOCOM 2020-IEEE Conference on Computer Communications, pages 676–685. IEEE, 2020.
  * [12] Jeongyoon Eo, Zhixiong Niu, Wenxue Cheng, Francis Y Yan, Rui Gao, Jorina Kardhashi, Scott Inglis, Michael Revow, Byung-Gon Chun, Peng Cheng, et al.  Opennetlab: Open platform for rl-based congestion control for real-time communications.  Proc. of APNet, 2022.
  * [13] Joyce Fang, Martin Ellis, Bin Li, Siyao Liu, Yasaman Hosseinkashi, Michael Revow, Albert Sadovnikov, Ziyuan Liu, Peng Cheng, Sachin Ashok, David Zhao, Ross Cutler, Yan Lu, and Johannes Gehrke.  Reinforcement learning for bandwidth estimation and congestion control in real-time communications.  2019\.  arXiv:1912.02222 [cs].
  * [14] Chelsea Finn, Sergey Levine, and Pieter Abbeel.  Guided cost learning: Deep inverse optimal control via policy optimization.  In International conference on machine learning, pages 49–58. PMLR, 2016.
  * [15] Pete Florence, Corey Lynch, Andy Zeng, Oscar A Ramirez, Ayzaan Wahid, Laura Downs, Adrian Wong, Johnny Lee, Igor Mordatch, and Jonathan Tompson.  Implicit behavioral cloning.  In Conference on Robot Learning, pages 158–168. PMLR, 2022.
  * [16] Sadjad Fouladi, John Emmons, Emre Orbay, Catherine Wu, Riad S Wahby, and Keith Winstein.  Salsify:$\\{$Low-Latency$\\}$ network video through tighter integration between a video codec and a transport protocol.  In 15th USENIX Symposium on Networked Systems Design and Implementation (NSDI 18), pages 267–282, 2018.
  * [17] Justin Fu, Katie Luo, and Sergey Levine.  Learning robust rewards with adversarial inverse reinforcement learning.  arXiv preprint arXiv:1710.11248, 2017.
  * [18] Benjamin Fuhrer, Yuval Shpigelman, Chen Tessler, Shie Mannor, Gal Chechik, Eitan Zahavi, and Gal Dalal.  Implementing reinforcement learning datacenter congestion control in nvidia nics.  In 2023 IEEE/ACM 23rd International Symposium on Cluster, Cloud and Internet Computing (CCGrid), pages 331–343, 2023.
  * [19] Dylan Hadfield-Menell, Smitha Milli, Pieter Abbeel, Stuart J Russell, and Anca Dragan.  Inverse reward design.  Advances in neural information processing systems, 30, 2017.
  * [20] Jonathan Ho and Stefano Ermon.  Generative adversarial imitation learning.  Advances in neural information processing systems, 29, 2016.
  * [21] Ningning Hu and Peter Steenkiste.  Estimating available bandwidth using packet pair probing.  Technical report, CARNEGIE-MELLON UNIV PITTSBURGH PA SCHOOL OF COMPUTER SCIENCE, 2002.
  * [22] Tianchi Huang, Rui-Xiao Zhang, Chao Zhou, and Lifeng Sun.  Qarc: Video quality aware rate control for real-time video streaming based on deep reinforcement learning.  In Proceedings of the 26th ACM international conference on Multimedia, pages 1208–1216, 2018.
  * [23] Tianchi Huang, Chao Zhou, Lianchen Jia, Rui-Xiao Zhang, and Lifeng Sun.  Learned internet congestion control for short video uploading.  In Proceedings of the 30th ACM International Conference on Multimedia, pages 3064–3075, 2022.
  * [24] Nathan Jay, Noga Rotman, Brighten Godfrey, Michael Schapira, and Aviv Tamar.  A deep reinforcement learning perspective on internet congestion control.  In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 3050–3059. PMLR, 09–15 Jun 2019.
  * [25] Nathan Jay, Noga Rotman, Brighten Godfrey, Michael Schapira, and Aviv Tamar.  A deep reinforcement learning perspective on internet congestion control.  In International Conference on Machine Learning, pages 3050–3059. PMLR, 2019.
  * [26] Wonseok Jeon, Seokin Seo, and Kee-Eung Kim.  A bayesian approach to generative adversarial imitation learning.  Advances in neural information processing systems, 31, 2018.
  * [27] Ingemar Johansson.  Self-clocked rate adaptation for conversational video in lte.  In Proceedings of the 2014 ACM SIGCOMM workshop on Capacity sharing workshop, pages 51–56, 2014.
  * [28] Sukhpreet Kaur Khangura and Sami Akin.  Online available bandwidth estimation using multiclass supervised learning techniques.  Computer Communications, 170:177–189, 2021.
  * [29] Akira Kinose and Tadahiro Taniguchi.  Integration of imitation learning using gail and reinforcement learning using task-achievement rewards via probabilistic graphical model.  Advanced Robotics, 34(16):1055–1067, 2020.
  * [30] Insoo Lee, Seyeon Kim, Sandesh Sathyanarayana, Kyungmin Bin, Song Chong, Kyunghan Lee, Dirk Grunwald, and Sangtae Ha.  R-fec: Rl-based fec adjustment for better qoe in webrtc.  In Proceedings of the 30th ACM International Conference on Multimedia, pages 2948–2956, 2022.
  * [31] Sergey Levine and Vladlen Koltun.  Continuous inverse optimal control with locally optimal examples.  arXiv preprint arXiv:1206.4617, 2012.
  * [32] Sergey Levine, Zoran Popovic, and Vladlen Koltun.  Nonlinear inverse reinforcement learning with gaussian processes.  Advances in neural information processing systems, 24, 2011.
  * [33] Haoyong Li, Bingcong Lu, Jun Xu, Li Song, Wenjun Zhang, Lin Li, and Yaoyao Yin.  Reinforcement learning based cross-layer congestion control for real-time communication.  In 2022 IEEE International Symposium on Broadband Multimedia Systems and Broadcasting (BMSB), pages 01–06, 2022.
  * [34] Haoyong Li, Bingcong Lu, Jun Xu, Li Song, Wenjun Zhang, Lin Li, and Yaoyao Yin.  Reinforcement learning based cross-layer congestion control for real-time communication.  In 2022 IEEE International Symposium on Broadband Multimedia Systems and Broadcasting (BMSB), pages 01–06. IEEE, 2022.
  * [35] Haoyong Li, Bingcong Lu, Jun Xu, Li Song, Wenjun Zhang, Lin Li, and Yaoyao Yin.  Reinforcement learning based cross-layer congestion control for real-time communication.  In 2022 IEEE International Symposium on Broadband Multimedia Systems and Broadcasting (BMSB), pages 01–06. IEEE, 2022.
  * [36] Jianxiong Li, Xiao Hu, Haoran Xu, Jingjing Liu, Xianyuan Zhan, and Ya-Qin Zhang.  Proto: Iterative policy regularized offline-to-online reinforcement learning.  arXiv preprint arXiv:2305.15669, 2023.
  * [37] Qiong Liu, Peng Yang, Feng Lyu, Ning Zhang, and Li Yu.  Multi-objective network congestion control via constrained reinforcement learning.  In 2021 IEEE Global Communications Conference (GLOBECOM), pages 1–6. IEEE, 2021.
  * [38] Hongzi Mao, Shannon Chen, Drew Dimmery, Shaun Singh, Drew Blaisdell, Yuandong Tian, Mohammad Alizadeh, and Eytan Bakshy.  Real-world video adaptation with reinforcement learning.  arXiv preprint arXiv:2008.12858, 2020.
  * [39] Hongzi Mao, Ravi Netravali, and Mohammad Alizadeh.  Neural Adaptive Video Streaming with Pensieve.  In Proceedings of the Conference of the ACM Special Interest Group on Data Communication - SIGCOMM ’17, pages 197–210, Los Angeles, CA, USA, 2017. ACM Press.
  * [40] Hongzi Mao, Ravi Netravali, and Mohammad Alizadeh.  Neural adaptive video streaming with pensieve.  In Proceedings of the conference of the ACM special interest group on data communication, pages 197–210, 2017.
  * [41] Gabriel Mittag, Babak Naderi, Vishak Gopal, and Ross Cutler.  Lstm-based video quality prediction accounting for temporal distortions in videoconferencing calls.  In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1–5. IEEE, 2023.
  * [42] Andrew Y Ng, Stuart Russell, et al.  Algorithms for inverse reinforcement learning.  In Icml, volume 1, page 2, 2000.
  * [43] Ali Rezagholizadeh.  Source rate control in videoconferencing application using state–action–reward–state–action temporal difference reinforcement learning.  PhD thesis, École de technologie supérieure, 2022.
  * [44] George F Riley and Thomas R Henderson.  The ns-3 network simulator.  In Modeling and tools for network simulation, pages 15–34. Springer, 2010.
  * [45] Stephane Ross, Geoffrey Gordon, and Drew Bagnell.  A reduction of imitation learning and structured prediction to no-regret online learning.  In Geoffrey Gordon, David Dunson, and Miroslav Dudík, editors, Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, volume 15 of Proceedings of Machine Learning Research, pages 627–635, Fort Lauderdale, FL, USA, 11–13 Apr 2011\. PMLR.
  * [46] Henning Schulzrinne, Stephen Casner, Ron Frederick, and Van Jacobson.  Rtp: A transport protocol for real-time applications.  Technical report, 2003.
  * [47] Sadia J Siddiqi, Faisal Naeem, Saud Khan, Komal S Khan, and Muhammad Tariq.  Towards ai-enabled traffic management in multipath tcp: A survey.  Computer Communications, 181:412–427, 2022.
  * [48] Viswanath Sivakumar, Olivier Delalleau, Tim Rocktäschel, Alexander H Miller, Heinrich Küttler, Nantas Nardelli, Mike Rabbat, Joelle Pineau, and Sebastian Riedel.  Mvfst-rl: An asynchronous rl framework for congestion control with delayed actions.  arXiv preprint arXiv:1910.04054, 2019.
  * [49] Mingfei Sun and Xiaojuan Ma.  Adversarial imitation learning from incomplete demonstrations.  arXiv preprint arXiv:1905.12310, 2019.
  * [50] Umar Syed, Michael Bowling, and Robert E Schapire.  Apprenticeship learning using linear programming.  In Proceedings of the 25th international conference on Machine learning, pages 1032–1039, 2008.
  * [51] Umar Syed and Robert E Schapire.  A game-theoretic approach to apprenticeship learning.  Advances in neural information processing systems, 20, 2007.
  * [52] Chen Tessler, Yuval Shpigelman, Gal Dalal, Amit Mandelbaum, Doron Haritan Kazakov, Benjamin Fuhrer, Gal Chechik, and Shie Mannor.  Reinforcement learning for datacenter congestion control.  SIGMETRICS Perform. Eval. Rev., 49(2):43–46, jan 2022.
  * [53] Faraz Torabi, Garrett Warnell, and Peter Stone.  Behavioral cloning from observation.  arXiv preprint arXiv:1805.01954, 2018.
  * [54] Bo Wang, Yuan Zhang, Size Qian, Zipeng Pan, and Yuhong Xie.  A hybrid receiver-side congestion control scheme for web real-time communication.  In Proceedings of the 12th ACM Multimedia Systems Conference, pages 332–338, 2021.
  * [55] Wenting Wei, Huaxi Gu, and Baochun Li.  Congestion control: A renaissance with machine learning.  IEEE Network, 35(4):262–269, 2021.
  * [56] Keith Winstein, Anirudh Sivaraman, and Hari Balakrishnan.  Stochastic forecasts achieve high throughput and low delay over cellular networks.  In 10th USENIX Symposium on Networked Systems Design and Implementation (NSDI 13), pages 459–471, 2013.
  * [57] Zhenchang Xia, Libing Wu, Fei Wang, Xudong Liao, Haiyan Hu, Jia Wu, and Dan Wu.  Glider: rethinking congestion control with deep reinforcement learning.  World Wide Web, 26(1):115–137, 2023.
  * [58] Xuedou Xiao, Mingxuan Yan, Yingying Zuo, Boxi Liu, Paul Ruan, Yang Cao, and Wei Wang.  From ember to blaze: Swift interactive video adaptation via meta-reinforcement learning.  arXiv preprint arXiv:2301.05541, 2023.
  * [59] Francis Y Yan, Hudson Ayers, Chenzhi Zhu, Sadjad Fouladi, James Hong, Keyi Zhang, Philip Levis, and Keith Winstein.  Continual learning improves internet video streaming.  arXiv preprint arXiv:1906.01113, 2019.
  * [60] Chen-Yu Yen, Soheil Abbasloo, and H. Jonathan Chao.  Computers can learn from the heuristic designs and master internet congestion control.  In Proceedings of the ACM SIGCOMM 2023 Conference, ACM SIGCOMM ’23, page 255–274, New York, NY, USA, 2023. Association for Computing Machinery.
  * [61] Tianrun Yin, Hongyu Wu, Runyu He, Shushu Yi, and Dingwei Li.  Gemini: An ensemble framework for bandwidth estimation in web real-time communications.
  * [62] Wenpei Yin, Bingcong Lu, Yan Zhao, Jun Xu, Li Song, and Wenjun Zhang.  Safr: A real-time communication system with adaptive frame rate.  In Proceedings of the 1st International Workshop on Networked AI Systems, pages 1–6, 2023.
  * [63] Huanhuan Zhang, Anfu Zhou, Yuhan Hu, Chaoyue Li, Guangping Wang, Xinyu Zhang, Huadong Ma, Leilei Wu, Aiyun Chen, and Changhui Wu.  Loki: improving long tail performance of learning-based real-time video adaptation by fusing rule-based models.  In Proceedings of the 27th Annual International Conference on Mobile Computing and Networking, pages 775–788, 2021.
  * [64] Huanhuan Zhang, Anfu Zhou, Jiamin Lu, Ruoxuan Ma, Yuhan Hu, Cong Li, Xinyu Zhang, Huadong Ma, and Xiaojiang Chen.  Onrl: improving mobile video telephony via online reinforcement learning.  In Proceedings of the 26th Annual International Conference on Mobile Computing and Networking, pages 1–14, 2020.
  * [65] Kaizhe Zhang, Zhiwen Wang, Hansen Ma, Haipeng Du, and Weizhan Zhang.  Hybridrts: A hybrid congestion control framework with rule and reinforcement learning for low-latency webrtc live video streaming.  In 2022 IEEE 24th Int Conf on High Performance Computing & Communications; 8th Int Conf on Data Science & Systems; 20th Int Conf on Smart City; 8th Int Conf on Dependability in Sensor, Cloud & Big Data Systems & Application (HPCC/DSS/SmartCity/DependSys), pages 1757–1764. IEEE, 2022.
  * [66] Lei Zhang, Yong Cui, Mowei Wang, Kewei Zhu, Yibo Zhu, and Yong Jiang.  Deepcc: Bridging the gap between congestion control and applications via multiobjective optimization.  IEEE/ACM Transactions on Networking, 2022.
  * [67] Lei Zhang, Kewei Zhu, Junchen Pan, Hang Shi, Yong Jiang, and Yong Cui.  Reinforcement learning based congestion control in a real environment.  In 2020 29th International Conference on Computer Communications and Networks (ICCCN), pages 1–9. IEEE, 2020.
  * [68] Anfu Zhou, Huanhuan Zhang, Guangyuan Su, Leilei Wu, Ruoxuan Ma, Zhen Meng, Xinyu Zhang, Xiufeng Xie, Huadong Ma, and Xiaojiang Chen.  Learning to coordinate video codec with transport protocol for mobile video telephony.  In The 25th Annual International Conference on Mobile Computing and Networking, pages 1–16, 2019.
  * [69] Xiaoqing Zhu and Rong Pan.  Nada: A unified congestion control scheme for low-latency interactive video.  In 2013 20th International Packet Video Workshop, pages 1–8. IEEE, 2013.
  * [70] Brian D Ziebart, J Andrew Bagnell, and Anind K Dey.  Modeling interaction via the principle of maximum causal entropy.  2010\.
  * [71] Brian D Ziebart, Andrew L Maas, J Andrew Bagnell, Anind K Dey, et al.  Maximum entropy inverse reinforcement learning.  In Aaai, volume 8, pages 1433–1438. Chicago, IL, USA, 2008.

##  Appendix A State Construction

See Table 3 for feature state construction.

Metric | Window Size
---|---
Receiving Rate | 5
Long Term Receiving Rate
Average Queue Time
Long Term Average Queue Time
Loss Ratio
Long Term Loss Ratio
Average Loss Packet
Long Term Average Loss Packet
Ratio of Video Packets | 3
Long Term Ratio of Video Packets
Ratio of Audio Packets
Long Term Ratio of Audio Packets
Ratio of Screenshare Packets
Long Term Ratio of Screenshare Packets
Ratio of Probe Packets
Long Term Ratio of Probe Packets
Table 3: Merlin State Construction. Short term metrics are sampled every $60$
ms, while longer-term metrics are sampled every $600$ ms.

##  Appendix B Simulated Evaluation

LSTM ablation results presented in Table 4. Feature set ablations detailed in
Figure 11.

LSTM Neurons | Action MSE
---|---
32 | 0.0016
64 | 0.0016
128 | 0.0014
256 | 0.0016
512 | 0.0016
1024 | 0.0016
Table 4: Impact of LSTM size on imitation performance.

(a) RR included and MT excluded.

(b) MT included and RR excluded.

(c) No RR or MT features included.

Figure 11: Imitation MSE per Feature Group: receive rate (rr), media type
(mt), delay (qd), loss rate (lr), average loss rate (alr). Note the difference
in axis scale.

##  Appendix C Emulated Evaluation

Qualitative imitation performance of Merlin on real networks with emulated
links presented in Figure 12. Impact of quality of demonstrations for BWE
illustrated in Figure 13.

(a) Merlin Bandwidth.

(b) UKF Bandwidth.

Figure 12: Imitating expert bandwidth predictions on emulated links with
$100$k training samples.

(a) Restricted $10$k Emulated.

(b) Diverse $10$k Simulated.

Figure 13: Quality of Demonstrations vs. Observed BWE Performance.

##  Appendix D Live Evaluation

Distribution of objective QoE metrics during inter-continental
videoconferencing calls presented in Figure 14.

(a) Video MOS Distribution.

(b) Audio MOS Distribution.

(c) Delay Distribution.

(d) Receive Rate Distribution.

(e) Loss Rate Distribution.

Figure 14: Distribution of Objective Metrics over Inter-continental Links.

##  Appendix E Reinforcement Learning and Learning from Demonstrations

Reinforcement Learning. RL seeks to learn a policy for acting in an
environment where the environment is represented by a Markov decision process
(MDP). Given a set of states $S$ and actions $A$, an agent seeks to learn a
mapping between states and actions $\pi:A\times S\rightarrow[0,1]$.
Furthermore, the transition model defines the probability of a taking an
action given the current state: $\pi(a,s)=Pr(a_{t}=a|s_{t}=s)$. We call the
combination of the mapping and the transition model the policy– a conditional
distribution of actions and states. To learn the desired policy, the agent
seeks to maximize a reward metric that encodes the task goal. Agents are
trained online for episodes of length $t$, estimating the expected reward at
each step. The expected return is represented as
$V_{\pi}(s)=E[R|s_{0}=s]=E[\sum_{t=0}^{\infty}\gamma^{t}{r_{t}}|s_{0}=s]$
where $\gamma$ is referred to as the discount rate, which prioritizes events
in the immediate future. By maximizing the expected return, the agent
prioritizes states with higher expected rewards, guiding the agent toward a
policy that accomplishes the desired task. However, mapping objective metrics
to subjective metrics is still an area of active research; hence, designing
comprehensive reward functions is hard. Accordingly, these challenges have
spurred research in IL where a policy is extracted purely from expert
demonstration. IL can loosely be categorized into inverse reinforcement
learning (IRL), apprenticeship learning, and BC.

Inverse Reinforcement Learning. Early motivations of IRL stemmed from tasks in
which reward functions were ill-defined, e.g., teaching a robot to pick up a
box. Rather than handcrafting a reward function, early works sought to
leverage expert demonstrations to derive a reward function via linear
programming [42]. However, in many cases, the set of functions that describe a
task is infinite, which makes derivation difficult, e.g., the trivial solution
of $f(x)=0$ satisfies most tasks [71]. In response to these difficulties,
researchers turned toward entropy-based methods [71, 70]. That is, rather than
searching over an unrestricted set of reward functions, the search space is
constrained to policies that are “close” to the expert. Early works sought to
match feature expectations, i.e., the agent arrives at similar states as the
expert [32]. To accomplish this, the authors maintain close proximity to the
expert policy via KL divergence and maximize the entropy of the learned policy
to combat noise and partial observability present in expert demonstrations.
Thus, the goal is to learn the worst policy that remains close to the expert.
It is important to note that many of these works rely on the presupposition
that the observed expert is optimal in regard to the observed task. Multiple
works build on this notion of maximum entropy to take into account contextual
information, e.g., previously visited states and actions, improving the
runtime complexity, and relaxing the constraint that demonstrations are
optimal [31, 70]. More recently, researchers have turned towards learned
feature representations to enable rich reward extraction [14]. Adversarial
Inverse Reinforcement Learning (AIRL) [17], the state-of-the-art IRL method,
builds on many of the previously mentioned ideas but utilizes adversarial
learning to derive and extract the expert reward. Although AIRL facilitates
offline learning and robust reward extraction, IRL relies on executing RL
within an "inner loop." That is, each sample corresponds to an entire training
episode which leads to extremely long runtimes and many environmental
interactions.

Apprenticeship Learning. In contrast to IRL, apprenticeship learning seeks to
learn a policy directly by matching feature expectations [2]. Proponents of
apprenticeship learning note the following insight: the reward function
depends on feature expectations; hence, the reward is encoded implicitly.
Early works sought to leverage various optimization techniques to extract an
expert policy [50, 51]. In contrast, similar to AIRL, the predominant
apprenticeship learning method is Generative Adversarial Imitation Learning
(GAIL) [20]. Unlike AIRL, GAIL is not an IRL technique, rather it learns a
policy directly through distribution matching. GAIL leverages two neural
networks in a GAN setup with the generator learning to produce a policy and
the discriminator learning to discriminate between the imitator’s policy
distribution and the expert’s. Many works have extended GAIL by handling
incomplete demonstrations, altering the objective metric, and adding
conditional goals [26, 49, 29]. GAIL remains a robust method in low data
regimes; however, in settings where expert demonstrations are abundant, GAIL
does not perform as well as competing methods such as BC [14].

Behavioral Cloning. BC diverges from both apprenticeship learning and IRL in
that it does not require iteratively running RL. BC runs purely from offline
data, reframing the IL problem as supervised learning. That is, given an
offline dataset of expert states and actions, BC learns an explicit mapping
between expert state and action pairs. Given that traditional BC does not
require querying an expert or interacting with an environment, training is
often less expensive in comparison to its IL counterparts. Similarly, BC
benefits from the well-studied discipline of supervised learning and inherits
many of the convergence guarantees [53]. However, there is no free lunch. BC,
although simple, suffers from the problem of compounding error [15]. That is,
supervised learning tends to rely on the i.i.d assumption that each data
observation is independent. However, when observing an expert complete a task,
trajectory samples are not independent and often exhibit temporal dependence.
As a result, when a BC model experiences an unfamiliar state, it produces an
error. These errors compound over the task trajectory due to the temporal
dependence between actions. To accommodate this, implicit methods have been
proposed to alleviate compounding error by learning a joint distribution of
states and actions [15]. Other methods such as DAGGER [45] have sought to
expand the demonstration dataset and increase state space coverage. All in
all, in low data regimes, BC tends to perform poorly due to compounding
errors.