# T$3$P: Demystifying Low-Earth Orbit Satellite Broadband

Shubham Tiwari1, Saksham Bhushan1, Aryan Taneja1, Mohamed Kassem2, Cheng Luo3,
Cong Zhou4, Zhiyuan He3, Aravindh Raman5, Nishanth Sastry2, Lili Qiu3, Debopam
Bhattacherjee1 1Microsoft Research India, 2University of Surrey, UK,
3Microsoft Research Asia,
4Shanghai Jiao Tong University, 5Telefonica Research, Spain

###### Abstract.

The Internet is going through a massive infrastructural revolution with the
advent of low-flying satellite networks, $5/6$G, WiFi$7$, and hollow-core
fiber deployments. While these networks could unleash enhanced connectivity
and new capabilities, it is critical to understand the performance
characteristics to efficiently drive applications over them. Low-Earth orbit
(LEO) satellite mega-constellations like SpaceX Starlink aim to offer broad
coverage and low latencies at the expense of high orbital dynamics leading to
continuous latency changes and frequent satellite hand-offs.

This paper aims to quantify Starlink’s latency and its variations and
components using a real testbed spanning multiple latitudes from the North to
the South of Europe. We identify tail latencies as a problem. We develop
predictors for latency and throughput and show their utility in improving
application performance by up to $25\%$. We also explore how transport
protocols can be optimized for LEO networks and show that this can improve
throughput by up to $115\%$ (with only a $5\%$ increase in latency). Also, our
measurement testbed with a footprint across multiple locations offers unique
trigger-based scheduling capabilities that are necessary to quantify the
impact of LEO dynamics.

††conference: ; ;

##  1\. Introduction

There has been a recent revolution in network connectivity with multiple big
players like SpaceX (Starlink) (SpaceX, 2023a), OneWeb (one, 2023), Telesat
(Lightspeed) (tel, 2023), Amazon (Kuiper) (kui, 2018), and others starting to
offer Internet services across the globe leveraging significant reductions
(Roberts, 2022) in satellite launch costs. Most of these constellations target
Low-Earth Orbit or LEO (Wikipedia, 2023f) heights ($<$$2$,$000$ km) to reduce
latencies by order(s) of magnitude compared to geosynchronous or GEO
(Wikipedia, 2023c) satellites which require an altitude of $35$,$786$ km.

While there is much excitement that LEO could leverage lower latencies to
deliver traffic for fast interactive networked applications, this
configuration also necessitates satellites flying at blazing speeds offering
connectivity windows of only a few minutes to ground terminals. Such transient
windows lead to frequent hand-offs at the ground terminal between satellites,
which can potentially have a negative effect on applications.

In this paper, we shed light on the network dynamics of the real SpaceX
Starlink LEO network and the ramifications of significant latency fluctuations
on transport and application layers. We found that although Starlink provides
high throughput at low latencies, the tail of the latency distribution can be
a problem, with the worst case latencies being 11-16x the median latencies,
which can greatly affect application performance.

We explore two parallel threads to address some of the performance bottlenecks
we find. First, we construct predictive models tailored to LEO networks,
capable of accurately estimating latency and throughput. We treat the
prediction task as a time series estimation and regression problem and develop
models using XGBoost and LSTM and features like the Starlink terminal
orientation (extracted from the telemetry data exposed by Starlink terminals
via a gRPC API), current satellite positions (from Celestrak (cel, 2023)), and
past telemetry data from our continuous experiments. The best-performing
latency (throughput, resp.) prediction model achieves a MAPE (Mean Average
Percentage Error) of $3.65\%$ ($19.2\%$). We augment a cutting-edge video
streaming Adaptive Bitrate (ABR) algorithm, RobustMPC (Yin et al., 2015), to
consume throughput predictions from our custom-built model. This integration
of LEO awareness improves video streaming Quality of Experience (QoE;
Pensieve) (Mao et al., 2017) by up to $25\%$ demonstrating how cross-layer
insights could help applications improve performance over these dynamic
networks.

In the second thread of exploration, we quantify and augment end-to-end
transport performance over Starlink LEO. Based on our understanding of the
impact of the network dynamics on transport performance, we pursue an
intuitive $2$-dimensional parameter space exploration of BBRv$2$ (Cardwell et
al., 2019) based on the expectation that (a) LEO latencies fluctuate due to
the continous motion of satellites as well as abruptly due to hand-offs
between satellites, thus necessitating the need to probe for the minimum
latency more often, and (b) congestion control should be robust to non-
congestive and higher loss (Kassem et al., 2022; Stevenson, 2023) over these
networks. Our parameter search identifies BBRv$2$ configurations that offer
$115\%$ higher throughput than the default setting with minimal inflation in
latency.

For running all our measurement and performance optimization experiments, we
built a testbed, called LEOScope, consisting of measurement clients behind
Starlink terminals, measurement servers on Azure, and an orchestrating service
deployed on Azure for scheduling experiments across nodes. LEOScope consists
of $5$ measurement clients (behind ‘pizza-box sized’ Starlink terminals)
spread across $2$ countries and measurement servers on Azure deployed
geographically close to the clients. Also, LEOScope offers two useful features
(discussed later), trigger-based scheduling and scavenger mode, focused on the
LEO dynamics and the scalability of the testbed. As part of this work, we have
released the LEOScope code (GitHub, 2023).

Our key contributions are as follows111The title of the paper, T$3$P, is
influenced by these key contributions. Note that T$3$P is a solution stack
discussed in §7.2.:

  * •

Telemetry: We conduct SpaceX Starlink performance measurement experiments
across different vantage points. We focus on measuring the latency to services
on the Web that are critical – DNS and CDN servers. Our latency dissection
analysis shows how the bent pipe (client to Starlink terminal to ground
station to the Internet and vice versa) contributes to the path latency.

  * •

Predictors: Our custom-built LEO performance predictors demonstrate useful QoE
enhancement opportunities when plugged into applications.

  * •

Transport: Our BBRv$2$ $2$-dimensional parameter tuning experiment
demonstrates performance improvement opportunities by making end-to-end
congestion control aware of the LEO dynamics. This opens up a research space
to explore clean-slate congestion control design over Starlink-like LEO
networks.

  * •

Triggers: We built a distributed testbed, LEOScope, to run our experiments
which offers trigger-based scheduling and scavenger mode useful for catching
LEO performance characteristics and scaling the testbed respectively. Triggers
could be used to probe for arbitrary events of interest – LEO latency
inflation, bad weather, terminal movements, satellite positions, etc.

##  2\. Background & Motivation

In this section, we discuss the current state, goals, and dynamics of LEO
broadband networks. SpaceX Starlink satellites deployed at $\sim$$550$ km
orbital heights fly at $7.5$ km/s (Bhattacherjee and Singla, 2019), more than
$20\times$ faster than sound. While we have seen limited mobility of
infrastructure and/or clients, thousands of routers/switches flying in the sky
at these incredible speeds make LEO networks interesting to study.

###  2.1. LEO Broadband Deployments

While previous efforts such as the Iridium constellation (Iridium
Communications Inc., 2023) also operated in the Low Earth Orbit, LEO broadband
did not take off due to the parallel cellular communication boom (Mellow,
2004) and the exorbitant cost (Roberts, 2022) of sending satellite payloads to
space at that time. Since then the space industry has seen multiple major
advances in technology – reusable boosters (Wikipedia, 2023g) reducing launch
costs by order(s) of magnitude, compact and cheap satellite design (Wikipedia,
2023b), inter-satellite link tracking and laser communication (Wikipedia,
2023d), etc. Such developments have now made it economically viable to launch
large Low-Earth orbit mega-constellations, consisting of thousands to tens of
thousands of satellites, as proposed/deployed by multiple players like SpaceX,
OneWeb, Amazon, Telesat, and others. These dense constellations, deployed at
much lower heights than GEO satellites, promise coverage across large areas of
the world, at significantly lower latencies than GEO. This approach requires a
high density of satellites to ensure that each user terminal across the globe
can connect to one at any point in time.

SpaceX Starlink is leading the front and has already deployed
$\sim$$4$,$000+$(Wikipedia, 2023h) LEO satellites and has started offering
Internet connectivity services in $58$ (Wikipedia, 2023h) countries across the
globe. Fig. LABEL:fig:trends shows how these numbers are increasing over time
since Jan’$21$. Interestingly, user adoption is growing super-linearly with
the number of Starlink users touching $1.5$ million (Wikipedia, 2023h)
recently. OneWeb has also deployed $500+$ (cel, 2023) LEO satellites already.

###  2.2. LEO Broadband Goals

Coverage goal SpaceX’s stated goal (Mauldin, 2019) is “to have the majority of
long distance Internet traffic go over this (Starlink) network”. While this is
yet to be achieved, as Fig. LABEL:fig:trends shows, the number of Starlink
users has increased at an incredibly fast pace, and Starlink offerings have
expanded across many countries in just a few years. Starlink has also engaged
to offer continued connectivity in conflict and disaster-stricken regions. It
is straightforward to install user terminals anywhere on earth and connect to
satellites overhead – LEO orbital geometry makes sure that providers need to
deploy enough satellites uniformly between servicing north and south latitudes
to offer continuous connectivity. Interestingly, this is a highly democratic
and inclusive design of a globally spanning network, enforced by the Earth’s
gravity and rotation.

Similar to SpaceX, OneWeb states (OneWeb, 2023) that it aims to “enable high-
speed, low latency connectivity for governments, businesses, and communities
everywhere around the world”. Amazon’s Kuiper aims (Amazon, 2023) to “provide
low-latency, high-speed broadband connectivity to unserved and underserved
communities around the world”.

Latency goal As can be seen in the above publicly stated goals, most of the
LEO broadband providers aim to offer ‘low-latency’ Internet connectivity.
SpaceX also claims the same (Elon Musk, 2021) for Starlink and argues that the
latencies should be ‘good enough’ (Brodkin, 2020) for fast online gaming.
Starlink currently offers bent-pipe connectivity with single satellites acting
as intermediate hops between user terminals and terrestrial Internet gateways.
SpaceX claims (Brodkin, 2020) to be working on reducing the latencies further.
With their latest generation satellites having multiple inter-satellite laser
terminals being deployed currently, long-distance Internet latencies are set
to improve (Bhattacherjee et al., 2018; Handley, 2018a) drastically compared
to fiber.

###  2.3. LEO Dynamicity

While LEO broadband aims to offer global low-latency connectivity, this comes
at a performance cost. At LEO heights, satellites move at breakneck speeds
($7.5$ km/s at $550$ km height (Bhattacherjee and Singla, 2019)) thus offering
a limited few-minute connectivity window to the user terminals underneath.
While thousands of satellites in a constellation make sure that most or all of
the user terminals get at least a few satellite options to connect to at any
point in time, such extreme dynamics lead to frequent hand-offs and latency
changes. While the core infrastructure itself is mobile unlike in terrestrial
cellular networks, LEO satellite trajectories are predefined. Based on this
expectation, we aim to predict latency/throughput changes and hand-offs, with
the goal of improving transport, traffic engineering, and application
performance over LEO networks.

###  2.4. LEO Performance Variations

LEO broadband network performance could vary due to multiple key reasons – LEO
orbital dynamics as discussed above, the geometry of the constellations,
weather having its toll on satellite-ground connectivity, continuous
deployments, local demand changes, GEO arc-avoidance (SpaceX FCC update,
2018), etc. To have a quick understanding of such variations, we mined social
interactions on r/Starlink (Reddit, 2023) subReddit. It is a thriving
community with $370+$ posts and $5.5K+$ comments on average per week between
Jan’$21$ and Dec’$22$222Newer data currently unavailable as Reddit is
upgrading their APIs (u/KeyserSosa, 2023). with redditors exhaustively
discussing SpaceX Starlink bandwidth, latency, and outage, and sharing speed-
test screenshots. We used Python Reddit API Wrapper (PRAW) (Boe, 2023) to
gather publicly available data from r/Starlink for the above period, got rid
of PII (Personally identifiable information), and used standard language and
computer vision tools like Azure’s Cognitive Services (ACS) (Microsoft, 2023)
and Natural Language Toolkit (NLTK) (NLTK Project, 2023) to extract relevant
information on network bandwidth, latency, loss, and outage from the posts,
comments, and screenshots (PII removed). Fig. LABEL:fig:redditspeed and
LABEL:fig:redditlatency plot the CDFs of reported downlink speeds and ping
latencies extracted from $\sim$$1.7K$ public speed-test screenshots shared by
Redditors during the period. The $95^{th}$ percentile speed (latency) is
$\sim$$3.4\times$($\sim$$4.5\times$) the median value showing the significant
variation in performance even though the constellation deployment is intended
to be roughly uniform (going by the FCC filings (SpaceX FCC update, 2018;
Telesat, 2020)) in the latitude bands being served. These observations call
for a deeper understanding of LEO network performance at scale across space
and time by running active experiments.

##  3\. Telemetry: Understanding LEO latency and its tail

We start the T$3$P journey by taking a bird’s-eye view of the latency
offerings in SpaceX Starlink’s LEO network. We quantify the latency to reach
critical Web infrastructure that users expect low latency from — public DNS
and CDN servers. We dig deeper to dissect the components that constitute end-
to-end latency and underscore the presence of persistent latency spikes, which
could likely be attributed to satellite handovers.

###  3.1. Experimental Setup

While we have a detailed discussion on the LEOScope testbed internals in §6,
here we briefly touch upon the experiment setup to not stray too far from the
focus of this work — T$3$P. As Triggers are an inherent offering of the
testbed, we discuss them later with LEOScope. Our experiments are run on
measurement clients behind Starlink terminals deployed in $2$ countries -
Spain ($2$$\times$; BCN-$1$ and BCN-$2$ in Barcelona) and UK ($3$$\times$; EDH
in Edinburgh, CMG in Cambridge, and LDN in London). These locations help us
get an interesting spatial diversity. Starlink’s first shell of satellites is
known to be deployed at $53\degree$ (SpaceX FCC update, 2018) inclination –
hence the density of satellites is highest around $53\degree$ N and S
latitudes. Our measurement clients are distributed in a way that they cover
the lower latitudes (BCN-$1$ & BCN-$2$: $41.4\degree$ N), the area of high
density (LDN: $51.5\degree$ N, CMG: $52.2\degree$ N), and latitudes further
north of this dense ring (EDH: $55.9\degree$ N). The measurement servers are
publicly known servers like DNS and CDN, or servers hosted by us on the public
cloud geographically close to the measurement client deployments. All clients
(servers, resp.) deployed by us are Standard Linux $5$.$15$.$0$ machines (VMs)
with $2$ ($8$) v/CPUs, $8$ ($32$) GB of memory, and high-bandwidth network
interfaces not rate limiting Starlink capacity. All measurements were run in
June’$23$. The experiments were scheduled with LEOScope’s orchestrator on
Azure. Note that the same setup is used for running all experiments in this
work.

###  3.2. Latency to popular services

First, we quantify Starlink’s latency offering by reaching out to popular
public Internet services like DNS and CDN from the measurement clients and
gathering latency telemetry data. For this, we run ping and hping (TCP SYN-
SYN/ACK handshake) latency probes from the measurement clients to public DNS
and CDN servers, which are known to be well-distributed, at a cadence of $1$
second for $2$ hours, $3$ times a day for $7$ days. As shown in past work
(Bozkurt et al., 2017), latency to the DNS and CDN services significantly
contributes to the application/user-perceived latency and usually has a
multiplicative effect on user-perceived delays, due to protocol overheads.

Fig. LABEL:fig:dns_cloud_lat_cdf plots the CDFs of ping latencies to Google’s
anycast DNS servers ($8$.$8$.$8$.$8$) (Google, 2023). The minimum and maximum
median latencies to Google DNS are $30.8$ ms (BCN-2) and $35.7$ ms (EDH)
respectively. Interestingly, at all of these locations, the latencies have
long tails with the $99^{th}$ percentiles being $>$$2.4\times$ the median
(maximum latency is $11$-$16\times$ the median). We observe similar trends
with hping and with Cloudflare’s DNS ($1$.$1$.$1$.$1$) (Cloudflare, 2023). In
the latency time-series analysis below, we deep-dive into the long-tail
latencies observed over Starlink’s LEO network.

For CDN services, we get the top Web domains from the Tranco list 333Available
at https://tranco-list.eu/list/LY344 (generated on $04$ Dec, $2022$) (tra,
2023), get the final redirected domains, filter domains (TLDs: com, net, edu,
org, tv, io, site, gov), get corresponding IP addresses, and get the OrgName
and NetName from whois (who, 2023) registry to identify top $100$ sites fully
served by (IP addresses belong to the CDN) $3$ popular CDN services (clo,
2020) — Akamai, Cloudflare, and Fastly. Note that these CDN services are the
ones that fill up these $100$ full-site delivery slots fastest for the ordered
Tranco list. Fig. LABEL:fig:bcn-1_cdn_cdf plots the CDFs to these $3$ CDNs
from BCN-$1$. All three CDN services have comparable performance for BCN-$1$
with the median latency being $37.3$ ms (Akamai), $35.2$ ms (Cloudflare), and
$36.5$ ms (Fastly). However, latency to Akamai-hosted servers is $60\%$ higher
at the $95^{th}$ percentile. Fig. LABEL:fig:akamai_cdf shows the latencies to
Akamai domains across all locations. Interestingly, we observed a clear
separation in the latency performance across different measurement clients –
$95^{th}$ percentiles for EDH, CMG and LDN nodes in the UK are $64.6$ ms,
$65.1$ ms, $68.3$ ms respectively while the same for BCN-$1$ and BCN-$2$ in
Spain are $136$ ms and $129$ ms respectively. This translates to more than
$2\times$ latency inflation for the latter nodes at the $95^{th}$ percentile.
This could be a reflection of a mix of multiple factors including physical
locations of gateway and CDN deployments.

The clear trend that we observe across all $3$ plots in Fig. LABEL:fig:latency
is that, while median latencies across all locations and services are low
($30$-$50$ ms), tail latencies are significantly higher, sometimes by an order
of magnitude. We investigate the potential causes next.

###  3.3. Investigating latency fluctuations

In order to understand the long-tail latencies better, in Fig.
LABEL:fig:latency_time_series, we plot the latency time-series for the ping
latency to Google’s $8$.$8$.$8$.$8$ from the BCN-$1$ node for a period of $15$
min picked at random from the measurement windows in Fig.
LABEL:fig:dns_cloud_lat_cdf. While the observed median latency is $35.2$ ms
for the larger window, we can see in this $15$ min granularity a few spikes
with reported latencies larger than $70$ ms ($2\times$ the median) each
persisting for roughly $15$ seconds or more. The first large spike starting at
the $523^{rd}$ second of the window lasts for $30$ seconds. We echo the
popular speculation (Nelson’s log, 2021; Tysonpower, 2021) that such high
latencies occur due to sub-optimal hand-offs that last for multiples of $15$
seconds. Interestingly, there are many smaller spikes throughout with peak
latencies around $50$ ms, and the ping latencies change continually reflecting
the LEO dynamics. Such highly dynamic behavior might translate to transport
and application performance bottlenecks. We also observe transient packet loss
happening often when the spikes start/end.

The latency time series analysis sheds light on the unique implications of LEO
dynamics: frequent hand-offs could lead to a user terminal being mapped to a
satellite such that the latencies are inflated persistently for the entire
duration that the terminal is connected to that satellite. Given satellite
trajectories are publicly available (cel, 2023), can we predict LEO’s
dynamicity? To put it in a different way, can we build LEO performance
predictors that could provide useful insights to applications to tackle such
predictable dynamicity? We explore this in §4.

###  3.4. Dissecting LEO’s bent-pipe latency

Before building a LEO performance predictor based on the known orbital
dynamics of Starlink, it is imperative to ascertain that the performance is
predominantly influenced by the LEO component which is the satellite bent-pipe
connection, rather than the terrestrial wide area Internet element, within the
end-to-end path. The end-to-end path between Starlink terminals and the
Internet services can be broken down into distinct segments which we designate
as S$1$ through S$6$ as in Fig. LABEL:fig:leo_path. S$1$ is the host-to-
Starlink terminal component, S$2$ is the satellite bent-pipe, S$3$ – S$5$ are
Starlink’s internal terrestrial components (verified by mapping IP addresses
to ASNs), and S$6$ is the wide area Internet part of the path.

To gain insights into how each of these segments contributes to the end-to-end
latency, we run traceroute experiments from all measurement clients to
Google’s anycast DNS server ($8$.$8$.$8$.$8$). These traceroutes run at a
cadence of $1$ s for $2$ hours, $3$ times a day, over a span of $7$ days. We
plot the segment-wise one-way median latency in Fig.
LABEL:fig:latency_bar_plot. The LEO bent-pipe component of the path (S$2$)
contributes the largest component to the end-to-end path latency – $54\%$ for
CMG, $65\%$ for EDH, $63\%$ for LDN, $52\%$ for BCN-$1$, and $59\%$ for
BCN-$2$ at the median.

###  3.5. Some terminal orientations could be worse!

The Starlink terminals at the measurement clients have an inbuilt telemetry
system that continually monitors the physical orientation of the terminal
(azimuth and elevation) along with packet drop rate to PoP (point of presence
on the other side of the bent-pipe last mile), ping latency to PoP, etc. To
understand how/if user terminal orientation impacts observed latency, we
collect terminal telemetry data for $320+$ hours from the BCN-$1$ node at a
granularity of $\sim$$1$ second.

The physical orientation of the user terminal (antenna) seldom changes –
$90\%$ of the azimuth and elevation values lie within narrow bands of [$0.2$,
$1.8$] (possibly to avoid GEO-arcs) and [$64.5$, $65.4$] degrees respectively.
Fig. LABEL:fig:grpc_azimuth_elevation shows a few clusters of popular azimuth
and elevation combinations with a long tail of other values. Note that the
antenna changes its azimuth and elevation gradually, albeit at a slow median
pace of $0.0001\degree$ per second for both axes. We do not have visibility
into the phased-array beam (Nardi, 2020) orientation on top of the physical
orientation of the antenna, but as we see next, the physical orientation could
have important network performance implications.

Fig. LABEL:fig:grpc_orientation_latency plots the heatmap of $95^{th}$
percentile latencies for different combinations of terminal azimuth and
elevation. The deepest hues (largest values of latency; still less than $100$
ms though) correspond to less popular orientations (see Fig.
LABEL:fig:grpc_azimuth_elevation). We observe similar trends for median
(instead of $95^{th}$ percentile) latencies too. This interesting observation,
alongside the persistent latency spikes that we see in Fig.
LABEL:fig:latency_time_series, and the observation in Fig.
LABEL:fig:latency_bar_plot that the bent-pipe adds the largest component to
the end-to-end latency, entice us to believe LEO network performance could be
predicted to a reasonable degree. Satellite trajectories are predictable over
time: publicly available satellite TLEs (two-line elements) (cel, 2023) define
these trajectories for the next tens of hours and are updated regularly.
Hence, the delay component driven by the satellite trajectories and gateway
locations could be predictable.

Note that the most frequent antenna orientations in Fig.
LABEL:fig:grpc_azimuth_elevation do not offer the lowest latencies in Fig.
LABEL:fig:grpc_orientation_latency though. We do not claim that the antenna
orientations are geared toward lower latency, rather we claim some of the
infrequent orientations could lead to latency inflation.

§3 Takeaway: Starlink-like LEO networks could offer low ($30$-$50$ ms) latency
at the median but are subject to very high tail latencies. Such high latencies
are a result of a complex mix of constellation geometry, orientation and
location of the user terminal, deployment of gateways, satellite hand-offs, as
well as other unforeseen variable factors in the terrestrial path segments.

##  4\. Predictors: Toward infusing applications with LEO awareness

As we observe in the previous section, while Starlink LEO offers reasonable
median latencies of few tens of milliseconds, the tail values could be an
order of magnitude higher. Such high latencies are due to persistent inflated
latency periods (probably due to sub-optimal hand-offs), sub-optimal antenna
positioning, etc. This, accompanied by high ($\sim$$2\%$-$5\%$) packet loss
(Kassem et al., 2022; Stevenson, 2023) in LEO, makes it difficult for upper
layers in the stack to adapt to changing network conditions over LEO. For
example, both loss and delay-based congestion control schemes suffer as these
signals become noisier due to non-congestive latency inflation and loss.
Applications like video streaming, which rely on throughput predictions, could
suffer due to such high dynamicity that happens naturally in these networks.
Given, satellite position information is publicly available (cel, 2023) and
the Starlink terminals expose a rich interface to access terminal orientation
data, continuous telemetry, etc., we explore the opportunity to predict LEO
performance (latency and throughput) in this section. The predictors, as part
of the T$3$P stack, could offer useful signals to the upper layer to adapt on.
We demonstrate how our custom-built predictor improves video streaming quality
of experience.

###  4.1. Internals of the LEO predictors

Algorithms: For the latency and throughput predictor models, we tried $3$
algorithms – XGBoost (Chen and Guestrin, 2016) (Extreme Gradient Boosting),
LSTM (Sherstinsky, 2020) (Long Short-Term Memory), and ARIMA (Wikipedia,
2023a) (AutoRegressive Integrated Moving Average). XGBoost is a gradient-
boosted decision tree-based algorithm widely used for classification and
regression. We use XGBoost for this regression problem as it effectively
handles non-linear relationships between the features (we describe below) and
the target variable (latency/throughput). Moreover, it is an ensemble learning
method that combines the predictions of multiple decision trees, leading to
improved accuracy and stability. LSTM is a specialized recurrent neural
network that is adept at learning and remembering dependencies in sequential
data, making it highly suitable for historical data (like latency) and its
correlation with the features. ARIMA, on the other hand, is a simple time-
series analysis technique that is particularly useful for handling stationary
data and data with seasonality. In our experiments, ARIMA offered
significantly lower accuracies than both XGBoost and LSTM. Hence, we refrain
from discussing ARIMA further.

Features: We use the following features to train the predictors – $1$. the
terminal location coordinates (latitude, longitude, altitude), $2$. publicly
available satellite trajectory information that gives the instantaneous
locations (azimuth, elevation, and distance from the user terminal) of all
satellites within the field of view of the user terminal, $3$. the user
terminal orientation data (azimuth & elevation), $4$. the historic (last $5$
seconds) performance (latency/throughput) data recorded by the user terminal,
and $5$. the time (timestamp in second).

Experimental setup: Across $2$ measurement client locations (BCN-1 in Spain
and EDH in the UK), we train the models on $\sim$$19$ hours of data each at a
granularity of $1$ second and test them with $\sim$$5$ hours of data at the
same granularity. We predict end-to-end latencies to servers hosted on
geographically closest Azure locations.

Results - Latency: Across both locations, LSTM demonstrates superior accuracy,
as seen in Table. 1, compared to the XGBoost model. It achieves a mean
accuracy of $\sim$$96\%$ and performs consistently better than XGBoost at the
median, $95^{th}$, and $99^{th}$ percentiles. When comparing the Relative Root
Mean Square Error (RMSE) between XGBoost and LSTM models, LSTM’s superiority
is evident. Its RMSE remains below $2.27$ ms across all locations, whereas the
XGBoost model reaches a maximum RMSE value of $5.6$ ms for a link with a mean
latency of $32.46$ ms.

Results - Throughput: We trained similar models for predicting throughput,
though the accuracies of these models are relatively lower (throughput is hard
to predict (Schmid et al., 2021), especially given the LEO dynamics). We were
able to achieve maximum accuracy of $73.92\%$ for LSTM and $80.80\%$ for the
XGBoost model. In contrast to the latency predictors, in the case of the
throughput predictors, we observe XGBoost to be performing better than the
LSTM model, as seen in Table. 1. This is because XGBoost is based on decision
trees and is effective in capturing complex interactions between features,
which is useful for predicting throughput that usually depends more on a
complex calculus involving various factors. Latency movements on the other
hand are more continuous, in general, and are predicted well by LSTM which is
particularly effective in capturing the temporal dependencies in sequential
data.

| Throughput Predictors | Latency Predictors
---|---|---
Testbed | XGBoost | LSTM | XGBoost | LSTM
Location | MAPE | RMSE | $5\%$ | $10\%$ | MAPE | RMSE | $5\%$ | $10\%$ | MAPE | RMSE | $5\%$ | $10\%$ | MAPE | RMSE | $5\%$ | $10\%$
BCN-1 | 19.2% | 190 kbps | 84.99% | 92.92% | 26.08% | 153 kbps | 40.47% | 61.33% | 10.68% | 5.53 ms | 36.52% | 61.91% | 3.85% | 2.03 ms | 75.10% | 93.55%
EDH | 41.67% | 291 kbps | 9.02% % | 17.20% | 87.34% | 456 kbps | 1.13% | 2.32% | 9.95% | 5.60 ms | 37.55 % | 65.00% | 3.65% | 2.27 ms | 76.38% | 94.51%

Table 1. Comparison of the accuracy of different prediction models across two
locations in Spain and the UK. MAPE stands for Mean Average Percentage Error,
RMSE represents the Root Mean Squared Error, and the $5\%$ and $10\%$ column
headers represent the percentage of predicted values that lie within $\pm 5\%$
and $\pm 10\%$ range of the actual values, respectively.

###  4.2. Augmenting applications with LEO predictors

Can we use these predictors to improve the performance of applications running
over LEO networks? Toward understanding this broad opportunity, we run a
simple video streaming experiment in $2$ of our deployment sites – BCN-$1$ in
Spain, and EDH in the UK. We generate network traces using iperf and the
terminal-generated data for $10$ hours at a granularity of $1$ second and use
a similar setup as in (Mao et al., 2017) to run our video streaming tests – an
emulated LEO network over Mahimahi (Netravali et al., 2015), and a custom
encoded video from Pensieve of $3$ min duration with $6$ qualities. The
encoded bitrate ratio is $\sim$$1.5$ between adjacent qualities and tracks. We
use RobustMPC (Yin et al., 2015) (control-theoretic) as the ABR (adaptive
bitrate) algorithm and measure the general video streaming QoE metric (Mao et
al., 2017) (combines high bitrate, minimal rebuffering, and smoothness) across
$100$ runs of each algorithm as described below.

We use $3$ variants of the MPC algorithm – $1$. Default MPC (MPC-D): the
default implementation that uses its own custom throughput predictor, $2$.
LEO-aware MPC (MPC-L): uses the LEO throughput predictor (XGBoost) we trained
with client-local data, and $3$. MPC with Oracle (MPC-O): knows the throughput
trace in advance. Across both locations and multiple runs, we observe MPC-O (a
hypothetical upper-bound) outperforms the other schemes (improves QoE by
$30\%$ at the median over the baseline MPC-D) consistently, which is along the
lines of expectation. Interestingly though, MPC-L, backed by our custom-built
LEO throughput predictor, also consistently outperforms MPC-D by $25\%$ at the
median and $23\%$ at the $95^{th}$ percentile.

§4 Takeaway: LEO predictors could offer useful signals to upper layers.
Applications, like video streaming, that rely on predicting network
performance could significantly improve QoE by adopting the T$3$P stack. Since
these adaptations are application-specific, we are able to optimize for
application-specific requirements, such as QoE for video streaming.

##  5\. Transport: Fine-tuning congestion control over LEO

The Internet transport is evolving from an era of loss-based congestion
control (CC) schemes (Ha et al., 2008; Floyd et al., 2004) toward delay-based
ones (Cardwell et al., 2017, 2019; Arun and Balakrishnan, 2018; Dong et al.,
2018), most of which rely on micro-experiments to identify the right sending
rates. How do such CC schemes fare over the SpaceX Starlink LEO network, given
network latencies change continually due to satellite motion and occasional
hand-offs? In this section, we quantify the performance (and shortcomings) of
CC schemes over Starlink LEO, explore the joint-optimization of $2$ intuitive
parameters of BBRv$2$ which clearly falls short of expectations otherwise, and
briefly comment on the fairness of such optimized variants to Cubic flows.

Recent work (Michel et al., 2022) on measuring Starlink performance assumes a
fixed CC scheme (Cubic) for all the experiments. Given the low latency promise
(Handley, 2018b) of these networks, Cubic might not be the most efficient
choice – it is known to fill up buffers thus inflating end-to-end latencies.
Another recent study (Kassem et al., 2022) briefly compares different CC
performances over LEO, finding that BBRv$1$ offers higher throughput than
others. We rather focus on BBRv$2$, a more recent BBR version, which struggles
to perform over the Starlink network.

While the previous section leveraged raw signals of predictable LEO dynamicity
to generate more sophisticated network performance signals that could improve
application performance, in this section, we take a somewhat complementary
stance – augmenting existing end-to-end transport with LEO intuition. Note
that these are different choices for application providers available in the
T$3$P stack (check §7.2) toward building LEO-aware applications.

###  5.1. Battle of the CC schemes

At the BCN-$1$ node, we ran iperf (ipe, 2023) upload tests to an Azure-hosted
measurement server in Netherlands444This cloud server is physically close to
the BCN-$1$ node to make sure that the LEO bent-pipe (terminal-satellite-
ground station) constitutes a significant component of the end-to-end path. to
compare the performance (throughput) of loss-based (Cubic (Ha et al., 2008),
NewReno (Floyd et al., 2004)) and delay-based (BBRv$1$ (Cardwell et al.,
2017), BBRv$2$ (Cardwell et al., 2019)) transport over Starlink and the
terrestrial Internet. Note that BBRv$1$ is agnostic to packet loss while the
more recent BBRv$2$ considers loss and ECN markings (Ramakrishnan et al.,
2001) as signals of congestion alongside delay. We ran each CC scheme for $4$
min each time, $3$ times a day, for a total of $100$ min (per CC scheme) over
both LEO and the terrestrial Internet. All plots are at $1$ s granularity,
thus translating to $6$,$000$ data points for each. Note that each $1$ s data
point, in turn, translates to $\sim$$30$ round-trips in our setup.

In Fig. LABEL:fig:cc_comparison, we plot the CDFs of throughput for CC schemes
normalized with the maximum observed throughput for the terrestrial and LEO
settings separately for the BCN-$1$ node. For the terrestrial setting, we only
report BBR, as the loss-based schemes are known to operate close to the
capacity. We observe that CC performance over LEO is significantly worse than
over the terrestrial Internet. This is in line with observations in prior
simulation-based work (Kassing et al., 2020) and a browser-plugin-based study
(Kassem et al., 2022). While terrestrially both BBRv$1$ and BBRv$2$ could
achieve median throughput of $82\%$ the maximum capacity (maximum observed
throughput), over Starlink LEO they could only achieve $32\%$ and $8\%$,
respectively. Also, the loss-based schemes, Cubic and NewReno, perform
significantly worse, with median throughput reaching only $8\%$ and $7\%$ of
the capacity respectively. Note that this is contrary to results from previous
simulations (Kassing et al., 2020), and is likely due to the high loss rates
(Kassem et al., 2022) alongside latency variations over Starlink LEO. We
perform this experiment for all measurement clients (and similar setups with
nearby servers) and observe similar trends. The interesting and somewhat
counter-intuitive observation that BBRv$2$ performs significantly worse than
the older BBRv$1$ (due to its greater sensitivity to loss) pushes us to
jointly tune $2$ relevant configuration parameters of BBRv$2$ toward improving
performance over these highly dynamic networks.

###  5.2. Pushing BBRv$2$ beyond the default

BBR and other rate-based CC schemes regularly probe the network (bandwidth,
minimum round trip time or RTT) and adapt the sending rate and congestion
window accordingly. BBRv$2$ has $2$ distinct modes in its steady state – $1$.
ProbeBW: probes the link bandwidth at regular intervals by sending at a higher
rate, and $2$. ProbeRTT: probes the RTT, if it is not updated recently, by
sending at a lower rate thus draining the queue. Both these distinct modes are
governed by multiple configurable parameters that decide the frequency and
intensity of the probes. We intuitively pick $2$ of these parameters, given
the LEO setting, and jointly tune both by performing a coarse-grained
$2$-dimensional grid search for this demonstration. These parameters are
probe_rtt_win_ms ($\alpha$), and loss_thresh ($\beta$). $\alpha$ denotes the
time interval between consecutive ProbeRTT mode entries. As Starlink latencies
change continually, this is an intuitive knob to tune. $\beta$ denotes the
loss rate threshold to stop probing for more bandwidth. Given Starlink is
lossy (Stevenson, 2023; Kassem et al., 2022), intuitively this is another
interesting parameter to tune.

We perform a coarse-grained $2$ dimensional grid search on the ($\alpha$,
$\beta$) parameter space, trying out $\alpha$ values that lead to more
frequent RTT probes and $\beta$ values more robust to loss. We run the
different combinations of $\alpha$ and $\beta$ for the BCN-$1$ uplink for $4$
min each, $3$ times a day, for a total of $100$ min. At $1$ s granularity,
this translates to $6$,$000$ data points per CC scheme. Each data point
constitutes of $\sim$$30$ round trips. Across all measurements ($1$ s data
points) for a CC scheme, we compute the mean throughput and the $95^{th}$
percentile latency, and compare them with that of the default setting
(reported in Fig. LABEL:fig:bbrv2_1 and LABEL:fig:bbrv2_2 heatmaps). We
observe that for lower values of $\alpha$ and higher values of $\beta$ we get
better throughput than the default, along the lines of our intuition, but
often at the cost of increased latency (or RTT). For both the heatmaps, the
bottom-right has warmer shades (higher values), signifying improved throughput
at the cost of latency inflation. A close manual examination reveals that this
is due to the costs and side-effects of probing. For example, probing RTT more
frequently might also mean BBRv$2$ goes into the exponential startup phase
more often when exiting the ProbeRTT mode. As a result, BBRv$2$ fills up the
buffer, leading to increased queuing delay and thus an overall increased
latency. Nevertheless, we find out interesting combinations of $\alpha$ and
$\beta$ for which throughput is increased significantly with minimal inflation
in latency. We define the best performing configuration as the maximum
throughput improvement with minimal increase in the RTT (i.e., $<10\%$). For
BCN-$1$, the best configuration ($\alpha$=$4000$ ms and $\beta$=$8\%$) offers
$115\%$ increase in the throughput over the default with only $5\%$ increase
in RTT.

Toward BBRv$2$ spatial tuning While the general trends across all locations
are similar (warmer shades toward the bottom-right of the heatmaps), the best
parameter combinations (defined as the parameters which obtain the highest
throughput improvements with $<10\%$ latency inflation over the default
parameter values) for $\alpha$, $\beta$ vary across locations, with
$\alpha=5$,$000/4$,$000/2$,$000$ ms and $\beta=8/16\%$ offering better
operating points than the default. This observation opens up the possibility
to tune BBRv$2$ differently across different locations based on the
connectivity profile.

Fairness to Cubic While a thorough analysis of the fairness of such parameter-
tuned BBRv$2$ flows is left to future work, one significant question is
whether the parameter modifications we advocate might affect fairness to
legacy CC algorithms such as Cubic. For this purpose, we pick the default as
well as three other $\alpha$, $\beta$ combinations (marked in red in Fig.
LABEL:fig:bbrv2_1 and LABEL:fig:bbrv2_2) along the diagonal of the heatmaps,
and the best choice for BCN-$1$ (marked in white). For each combination, we
run $8$ Cubic and $8$ tuned BBRv$2$ flows in parallel for $100$ min. Fig.
LABEL:fig:fair plots the CDFs of the ratio of aggregate Cubic throughput and
the aggregate BBRv$2$ throughput at $1$ s granularity. We observe that as we
move toward the bottom-right along the diagonal of the heatmaps, the ratios
are lower in Fig. LABEL:fig:fair, signifying BBRv$2$ becoming more aggressive.
Nevertheless, the ratios with the best choice $\alpha$, $\beta$ combinations
follow the default CDF plot closely with only a nominal reduction in values.
This demonstrates the opportunity to find good $\alpha$, $\beta$ combinations
that do not significantly penalize legacy Cubic flows.

§5 Takeaway: Starlink LEO creates unique performance bottlenecks for both
loss-based and delay-based CC schemes. BBRv$2$ could be potentially tuned to
offer significantly higher throughput at the cost of minimal latency
inflation.

##  6\. LEOScope Testbed

We built a testbed, called LEOScope, to run all the experiments over Starlink
LEO network. The testbed:

  * •

Allows for orchestration and scheduling of experiments across measurement
clients and servers.

  * •

Gathers telemetry data that could be used to build a continuous LEO
performance dashboard.

  * •

Generates LEO link profiles (LEOScope Team, 2023)that could be used as inputs
by various LEO simulators (like Hypatia (Kassing et al., 2020)) and emulators
(like StarryNet (Lai et al., 2023)).

  * •

Allows Starlink users to volunteer/donate resources to the testbed without
being affected while running personal applications. The LDN client in our
experiments is one such node.

We also enabled useful triggers on LEOScope based on various networking
(latency, loss, etc.) and non-networking (weather conditions, satellite
positions, etc.) conditions that could initiate experiments. Given the high
dynamicity of Starlink LEO, this setup is useful to zoom in on interesting
LEO-specific scenarios. As part of this work, we have released the LEOScope
code (GitHub, 2023). The rest of this section touches upon LEOScope design.

###  6.1. Broad design

Fig. LABEL:fig:leopard_design explains LEOScope in a nutshell. We touch upon
the individual key components below.

Users can submit network/application-level experiments to LEOScope, request a
schedule and a set of measurement clients and servers to run these experiments
on, and view the results dashboard.

Cloud orchestrator is responsible for scheduling experiments across
measurement clients and servers, managing schedule conflicts, and monitoring
measurement node health.

Measurement servers are deployed on public cloud. For client-server
measurement experiments like iperf (ipe, 2023), measurement clients could
reach out to these servers. The server daemon receives an experiment schedule
from the cloud orchestrator, downloads the necessary artifacts, and sets up
the local environment (if needed) before the client requests arrive.

Measurement clients consist of Starlink LEO user terminals with a reasonably
clear view of the sky (see Fig. LABEL:fig:leopard_terminal), and hosts (could
be standard workstations, Raspberry Pi (ras, 2023), or more capable edge
devices like Azure Stack Edge (Microsoft Azure, 2023) or Intel NUC (Intel,
2023)) behind the terminals that could access the Internet via the Starlink
network.

Cloud storage receives measurement data from both measurement clients and
servers. The storage is hierarchically structured into directories such that
it is straightforward to navigate to a specific instance of an experiment on a
measurement node.

Some engineering design choices we have made toward easy operations are:

Lightweight isolation LEOScope runs experiments on lightweight containers
(con, 2023) that help automate experiment setup on measurement clients and
servers and faster experiment startup.

Kernel Service LEOScope exposes a firewalled kernel service for experiments
that require low-level kernel-space configuration/modification (for example,
testing kernel-space congestion control schemes and modifying interface-level
queuing disciplines).

###  6.2. LEOScope’s unique features

Here we touch upon the unique LEOScope features that help us dissect LEO
network subtleties.

Ingesting user terminal data The Starlink user terminals run continuous
measurements in the background and collect bent-pipe latency, traffic volume,
packet loss, antenna directional data, etc (as touched upon briefly in §3.5).
These statistics are made available to the private network behind the user
terminal via gRPC APIs (sta, 2022). LEOScope clients continually gather this
data to generate insights that enable the two following unique features of the
testbed, discussed next.

Scavenger mode This unique feature allows LEOScope to locally (at the
measurement client) preempt and reschedule experiments that result in
upload/download traffic overhead when (higher-priority) user traffic is sensed
leveraging the terminal-provided insights. This makes LEOScope scalable –
volunteers with access to user terminals could join the testbed and contribute
compute/connectivity free-cycles to the testbed. The LDN measurement client in
the UK is run as a volunteer node on which we successfully tested scavenger
mode.

Figure 7. CDFs of BBRv$2$ throughput for BCN-$1$ uplink when triggers fire
randomly versus on sensing inflated latency. The difference in median
throughput is $43\%$, underscoring the utility of trigger-based scheduling in
capturing LEO-specific events that impact the end-to-end performance of
congestion-control schemes.

Trigger-based scheduling LEOScope allows dynamic scheduling of experiments by
specifying triggers that execute experiments on the occurrence of diverse
events of interest in the LEO context. For example, a trigger can be defined
as follows: “if current network latency $>$ $10\times$ the moving average (of
network latency) of last ten windows, trigger traceroute probes to Google’s
$8$.$8$.$8$.$8$ service”. Such trigger-based dynamic scheduling has $2$ key
advantages: ($1$) Triggers allow the experimenter to pivot their measurements
around specific network/physical events to capture interesting phenomena, more
common in this dynamic LEO setting. Fig. 7 shows how latency-sensitive
triggers can be used to evaluate the performance of congestion-control schemes
(in this case, BBRv$2$) when the LEO link latency is high. For the BCN-$1$
node, we run trigger-based BBRv$2$ (default config) upload experiments to an
Azure measurement server in the Netherlands for a period of $7$ days, with the
triggers firing randomly, versus when the latency jumps to a value at least
twice the mean observed latency in the last $5$ s. The difference in median
throughput obtained from the high-latency trigger schedule versus a random
schedule is $43\%$ (with random runs offering a higher median and a longer
tail), showing how triggers can help capture LEO-related events that impact
congestion-control performance. Triggers can also be defined on publicly
available information that can impact LEO link properties such as satellite
trajectory information (cel, 2023), local weather, etc. ($2$) Trigger-based
schedules save data-transfer costs and storage space by confining the
measurements to events of interest. For example, scheduling an experiment that
runs a TCP flow for $24$ hours with an average throughput of $40$ Mbps incurs
$\sim$$3.5$ Tb of data-transfer costs and $\sim$$138$Gb of storage space (only
packet header captures) on a single measurement client. Defining a trigger-
based schedule that restricts the run-time to $10\%$ of the total time can
save $\sim$$3$ Tb of data-transfer costs and $124$ Gb of storage. Triggers,
therefore, contribute significantly to reducing operational costs.

While we have used triggers to reduce the experiment footprint, they could be
used to inform upper layers of interesting networking and physical events. The
same functionality could be used to inform an application or transport about
network latency elevation thus allowing them to take proactive actions toward
optimizing the users’ QoE. These triggers, due to their potential diverse
usage, are offered as part of the T$3$P stack on LEOScope.

##  7\. Discussions and Future Work

This work sheds light on not only the LEO dynamics but also on the performance
optimization opportunities that lie ahead. T$3$P or LEO awareness could be
plugged into the network stack toward such optimizations that could impact the
experience of the next billion users connecting to the Internet, cloud
applications, and cloud services over the LEO broadband.

###  7.1. Network stack optimisations

Clean-slate transport While we observe reasonable performance improvement by
tuning BBRv$2$ over Starlink, we plan to explore clean-slate congestion
control with LEO awareness baked into the design. Both loss and delay-based
schemes run with the basic assumption that the minimum (propagation) delay in
a network is fixed which does not hold true over LEO. An LEO-aware end-to-end
transport should ideally be able to detect an LEO component on the path,
accommodate dynamic minimum latencies, and consume predictions and/or triggers
to augment its rate-finding algorithm.

Stochastic model for hand-off predictions Although the latency and throughput
predictors could offer useful LEO signals to application and transport, it
could be useful to have stochastic models for hand-offs. Hand-offs could often
be sub-optimal, resulting in persistent inflated latencies and/or transient
non-congestive loss, thus necessitating applications to proactively react
beforehand. For example, users of video conferencing applications report
(Starlink SubReddit, 2022a, b) occasional stutters over Starlink LEO.
Application-level mitigation could be to consume the stochastic hand-off
signals and reduce the video stream quality when the probability of a hand-off
is high.

Latency and Throughput Prediction We are committed to improving the T$3$P
stack by leveraging lightweight ML predictors or an ensemble of multiple
predictors. We also plan to explore an online learning framework that quickly
re-trains the models as LEO network conditions change over time even at a
given location (e.g., due to new satellite deployments, increase in number of
users, etc.)

Multi-access connectivity Several of our measurement clients have access to
both LEO and terrestrial Internet. This could be more common in the future
with Starlink adopters in the developed world using it as a secondary means of
Internet connectivity. Note that the LEO and terrestrial fiber channels could
offer drastically different offerings with the advent of inter-satellite
lasers – over longer distances, LEO offers lower latencies than terrestrial
fiber. How could one design a transport that could take into account the
diverse properties of the various access networks? Clearly, MPTCP-like
(Wischik et al., 2011) solutions fall short as they assume access links to
have similar offerings.

Understanding the physical layer Starlink does not provide low-layer
information, such as signal structure, Received Signal Strength (RSS), the
satellite that the terminal is connected to, and the used channel. (Humphreys
et al., [n. d.]) proposes a blind signal identification approach to infer the
synchronization sequences in the Starlink downlink signal. One can potentially
develop approaches to estimate RSS based on the inferred synchronization
sequences (e.g., using correlation).

Figure 8. The LEO-aware T$3$P stack could offer rich signals to applications.

###  7.2. Toward an LEO awareness library

We anticipate that all the above performance optimization opportunities could
build on one or more of T$3$P components. Hence, we envision the T$3$P stack,
as shown in Fig. 8, as an inherent building block of LEOScope that could not
only help run various network and application-level experiments on LEOScope
but also offer rich LEO-awareness signals to the upper layers (transport and
application). Applications will have the flexibility to consume raw telemetry
and/or triggers or more sophisticated predictions built on top of the raw
signals.

###  7.3. Future of LEOScope

As we have effectively used LEOScope to run our experiments over Starlink LEO
across multiple locations with ease, we plan to release LEOScope. The T$3$P
offerings are inherently baked into LEOScope – adopters could build
experiments on top of these LEO-awareness signals or leverage the same to
augment application performance. Also, LEOScope runs continuous measurements
across multiple locations, generating real LEO link profiles (LEOScope Team,
2023) in the process, which can act as important inputs for a large body of
simulation (Kassing et al., 2020) and emulation work (Lai et al., 2023) in
this space.

##  8\. Related Work

A large body of work from the nineties (Wood, 2001; Bau, 2002; Chan, 2000;
Kwok, 2001; Evans, 1998; Werner et al., 1995; Chan, 1999; Wu et al., 1994;
Kiesling, 1990; Comparetto and Hulkower, 1994; Sterling and Hatlelid, 1991;
WIEDEMAN et al., 1992; Akturan and Vogel, 1997; Maral, 1994) explores various
networking aspects of the previous generation satellite networks. A lot has
changed since then – the scale of deployments, the LEO broadband goals,
coverage, performance expectations, and the way today’s Internet protocols
work. With this new wave of LEO satellite mega-constellations, the networking
community has quickly realized that most of these past works need to be re-
evaluated and re-calibrated in this new context. Three papers (Bhattacherjee
et al., 2018; Handley, 2018a; Klenze et al., 2018) in HotNets’$18$ set the
path for such explorations in this ‘new space’ context and have led to a
flurry of interesting work in the last few years. While such work includes LEO
network design (Bhattacherjee and Singla, 2019), inter-domain routing
(Giuliari et al., 2020), satellite downlink scheduling (Vasisht et al., 2021),
etc., Hypatia (Kassing et al., 2020) was an early effort to deeply understand
these networks by leveraging packet-level simulations. While it effectively
captures the LEO dynamicity in its simulations, it lacks the realism of the
actual LEO networks that are impacted by various factors like weather, ground
station locations, link modulation, etc. This work was also the first to shed
light on the CC performance bottlenecks over these dynamic LEO networks.
Another work (Kassem et al., 2022) leverages a browser plugin to collect
insights on Starlink performance and passively analyze collected traces. In
our work, we rather deploy a real Starlink testbed (unlike Hypatia’s
simulations) and run active end-to-end experiments. A more recent work,
StarryNet (Lai et al., 2023), emulates integrated space-terrestrial networks.
LEOScope generates link profiles (LEOScope Team, 2023) from real telemetry
that could be treated as inputs to this and similar emulators.

Past works on satellite CC (Caini and Firrincieli, 2004; Casetti et al., 2002;
Akyildiz et al., 2001) primarily focus on the GEO (Geostationary Orbit;
$35$,$786$ km) satellites which experience high latencies due to the long
bent-pipes. TCP Hybla (Caini and Firrincieli, 2004), although a good fit for
such large RTTs, is not relevant in the context of these LEO networks that are
characterized by reasonable RTTs but with large fluctuations. Our work,
instead, explores CC schemes that dominate today’s Internet in this new
context.

We built LEOScope by adopting learnings from existing testbeds like PlanetLab
(pla, 2023), M-Lab (Dovrolis et al., 2010), RIPE Atlas (atl, 2023), BISmark
(Sundaresan et al., 2014), SamKnows (sam, 2023), ORBIT (Orbit testbed, [n.
d.]), PhoneLab (Nandugudi et al., 2013), and Monroe (Alay et al., 2017).
LEOScope offers unique features useful for the LEO setting and integrates the
T$3$P offerings that leverage LEO network signals.

##  9\. Conclusion

In this paper, we deep dive into the performance of Starlink tail latencies
via a distributed testbed spanning multiple latitudes from Edinburgh in North
Europe to Barcelona in the South. We find that although Starlink provides high
throughput at low latencies, the tail of the latency distribution can be a
problem, with the worst-case latency being $11$-$16\times$ the median. To help
applications manage such huge network performance changes, we build predictors
and show how applications such as video streaming can consume such predictions
to obtain better performance (up to $25\%$). We then show that transport
protocols such as BBRv$2$ can be fine-tuned to work better on LEO networks,
obtaining an impressive $115\%$ improvement over the default parameters. These
points to directions for future work on making networked applications better
on LEO networks.

Ethics.  We run active experiments on LEOScope, with due permissions from the
measurement client hosts, to collect all data. All measurement servers are
either deployed by us on the public cloud or host publicly known Internet
services (DNS and CDN). Hence, none of the IP addresses logged in our
experiments raise PII concerns. For the motivation plots in §2, we only use
publicly available data and screenshots gathered using standard APIs. Hence
this work does not raise any ethical concerns.

## References

  * (1)
  * kui (2018) 2018\.  Kuiper USASAT-NGSO-8A ITU filing.  https://www.itu.int/ITU-R/space/asreceived/Publication/DisplayPublication/8716.
  * clo (2020) 2020\.  CDN Industry: Trends, Size, And Market Share.  https://blog.intricately.com/cdn-industry-trends-market-share-customer-size
  * sta (2022) 2022\.  starlink-grpc-tools.  https://github.com/sparky8512/starlink-grpc-tools.
  * cel (2023) 2023\.  CelesTrak.  https://celestrak.org/.
  * ipe (2023) 2023\.  iPerf.  https://iperf.fr/.
  * one (2023) 2023\.  OneWeb.  https://oneweb.net/.
  * pla (2023) 2023\.  PlanetLab.  https://planetlab.cs.princeton.edu/
  * ras (2023) 2023\.  Raspberry Pi.  https://www.raspberrypi.com/.
  * atl (2023) 2023\.  RIPE Atlas.  https://atlas.ripe.net/
  * sam (2023) 2023\.  SamKnows.  https://samknows.com/
  * lau (2023) 2023\.  satellitemap.space.  https://satellitemap.space/starlink/launches.html.
  * tel (2023) 2023\.  Telesat.  https://www.telesat.com/.
  * tra (2023) 2023\.  Tranco.  https://tranco-list.eu/.
  * con (2023) 2023\.  What is a Container? - Docker.  https://www.docker.com/resources/what-container/.
  * who (2023) 2023\.  Whois Domain Lookup.  https://www.whois.com/whois.
  * Akturan and Vogel (1997) Riza Akturan and Wolfhard J Vogel. 1997.  Path diversity for LEO satellite-PCS in the urban environment. In _IEEE Transactions on Antennas and Propagation_.
  * Akyildiz et al. (2001) Ian F Akyildiz, Giacomo Morabito, and Sergio Palazzo. 2001.  TCP-Peach: a new congestion control scheme for satellite IP networks.  _IEEE/ACM TON_ (2001).
  * Alay et al. (2017) Özgü Alay, Andra Lutu, Miguel Peón-Quirós, Vincenzo Mancuso, Thomas Hirsch, Kristian Evensen, Audun Hansen, Stefan Alfredsson, Jonas Karlsson, Anna Brunstrom, Ali Safari Khatouni, Marco Mellia, and Marco Ajmone Marsan. 2017.  Experience: An Open Platform for Experimentation with Commercial Mobile Broadband Networks. In _ACM MobiCom_.
  * Amazon (2023) Amazon. 2023.  Project Kuiper Uses NICE DCV to Securely, Remotely Access EDA Software for Satellite Development.  https://aws.amazon.com/solutions/case-studies/project-kuiper-case-study/.
  * Arun and Balakrishnan (2018) Venkat Arun and Hari Balakrishnan. 2018.  Copa: Practical Delay-Based Congestion Control for the Internet. In _USENIX NSDI_.
  * Bau (2002) Jason H Bau. 2002.  _Topologies for satellite constellations in a cross-linked space backbone network_.  Ph. D. Dissertation. Massachusetts Institute of Technology.
  * Bhattacherjee et al. (2018) Debopam Bhattacherjee, Waqar Aqeel, Ilker Nadi Bozkurt, Anthony Aguirre, Balakrishnan Chandrasekaran, P Brighten Godfrey, Gregory Laughlin, Bruce Maggs, and Ankit Singla. 2018.  Gearing up for the 21st century space race. In _ACM HotNets_.
  * Bhattacherjee and Singla (2019) Debopam Bhattacherjee and Ankit Singla. 2019.  Network topology design at 27,000 km/hour. In _ACM CoNEXT_.
  * Boe (2023) Bryce Boe. 2023.  PRAW: The Python Reddit API Wrapper.  https://praw.readthedocs.io/en/latest/.
  * Bozkurt et al. (2017) Ilker Nadi Bozkurt, Anthony Aguirre, Balakrishnan Chandrasekaran, P Brighten Godfrey, Gregory Laughlin, Bruce Maggs, and Ankit Singla. 2017.  Why is the Internet so slow?!. In _PAM_.
  * Brodkin (2020) Jon Brodkin. 2020.  Elon Musk: Starlink latency will be good enough for competitive gaming.  https://arstechnica.com/information-technology/2020/03/musk-says-starlink-isnt-for-big-cities-wont-be-huge-threat-to-telcos/.
  * Caini and Firrincieli (2004) Carlo Caini and Rosario Firrincieli. 2004.  TCP Hybla: a TCP enhancement for heterogeneous networks.  _International journal of satellite communications and networking_ 22, 5 (2004), 547–566.
  * Cardwell et al. (2017) Neal Cardwell, Yuchung Cheng, C Stephen Gunn, Soheil Hassas Yeganeh, and Van Jacobson. 2017.  BBR: congestion-based congestion control. In _Communications of the ACM_.
  * Cardwell et al. (2019) Neal Cardwell, Yuchung Cheng, Soheil Hassas Yeganeh, Priyaranjan Jha, Yousuk Seung, Kevin Yang, Ian Swett, Victor Vasiliev, Bin Wu, Luke Hsiao, et al. 2019\.  BBRv2: A model-based congestion control performance optimization. In _Proc. IETF 106th Meeting_. 1–32.
  * Casetti et al. (2002) Claudio Casetti, Mario Gerla, Saverio Mascolo, Medy Y Sanadidi, and Ren Wang. 2002.  TCP Westwood: end-to-end congestion control for wired/wireless networks.  _Wireless Networks_ 8, 5 (2002), 467–479.
  * Chan (1999) Vincent WS Chan. 1999.  Optical space communications: a key building block for wide area space networks. In _IEEE LEOS Annual Meeting Conference Proceedings_.
  * Chan (2000) Vincent WS Chan. 2000.  Optical space communications. In _IEEE Journal of Selected Topics in Quantum Electronics_.
  * Chen and Guestrin (2016) Tianqi Chen and Carlos Guestrin. 2016.  Xgboost: A scalable tree boosting system. In _Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining_.
  * Cloudflare (2023) Cloudflare. 2023.  Cloudflare DNS.  https://www.cloudflare.com/dns/.
  * Comparetto and Hulkower (1994) Gary Comparetto and Neal Hulkower. 1994.  Global mobile satellite communications-A review of three contenders. In _15th International Communications Satellite Systems Conference and Exhibit_. AIAA.
  * Dong et al. (2018) Mo Dong, Tong Meng, Doron Zarchy, Engin Arslan, Yossi Gilad, Brighten Godfrey, and Michael Schapira. 2018.  $\\{$PCC$\\}$ vivace: Online-learning congestion control. In _USENIX NSDI_.
  * Dovrolis et al. (2010) Constantine Dovrolis, Krishna Gummadi, Aleksandar Kuzmanovic, and Sascha D. Meinrath. 2010.  Measurement Lab: Overview and an Invitation to the Research Community.  _ACM SIGCOMM CCR_ (2010).  https://doi.org/10.1145/1823844.1823853
  * Elon Musk (2021) Elon Musk. 2021.  Ping should improve dramatically in coming months.  https://twitter.com/elonmusk/status/1415480145830465539.
  * Evans (1998) John V Evans. 1998.  Satellite systems for personal communications. In _Proceedings of the IEEE_.
  * Floyd et al. (2004) Sally Floyd, Tom Henderson, and Andrei Gurtov. 2004.  _The NewReno modification to TCP’s fast recovery algorithm_.  Technical Report.
  * Forrester (2022) Chris Forrester. 2022.  Starlink has 700,000 subs.  https://advanced-television.com/2022/09/19/starlink-has-700000-subs/.
  * GitHub (2023) GitHub. 2023.  LEOScope.  https://github.com/leoscope-testbed.
  * Giuliari et al. (2020) Giacomo Giuliari, Tobias Klenze, Markus Legner, David Basin, Adrian Perrig, and Ankit Singla. 2020.  Internet Backbones in Space. In _ACM SIGCOMM CCR_.
  * Google (2023) Google. 2023.  Google Public DNS.  https://developers.google.com/speed/public-dns/.
  * Ha et al. (2008) Sangtae Ha, Injong Rhee, and Lisong Xu. 2008.  CUBIC: a new TCP-friendly high-speed TCP variant.  _ACM SIGOPS operating systems review_ (2008).
  * Handley (2018a) Mark Handley. 2018a.  Delay is not an option: Low latency routing in space. In _ACM HotNets_.
  * Handley (2018b) Mark Handley. 2018b.  Delay is Not an Option: Low Latency Routing in Space. In _ACM HotNets_.
  * Humphreys et al. ([n. d.]) Todd E. Humphreys, Peter A. Iannucci, Zacharias M. Komodromos, and Andrew M. Graff. [n. d.].  Signal Structure of the Starlink Ku-Band Downlink.  ([n. d.]).
  * Intel (2023) Intel. 2023.  Intel NUC Mini PCs, Elements and Laptops.  https://www.intel.in/content/www/in/en/products/details/nuc.html.
  * Iridium Communications Inc. (2023) Iridium Communications Inc. 2023.  Iridium Satellite Communications.  https://www.iridium.com/.
  * Jonathan’s Space Pages (2023) Jonathan’s Space Pages. 2023.  Starlink Statistics.  https://planet4589.org/space/con/star/stats.html.
  * Kan (2022) Michael Kan. 2022.  SpaceX Beats Annual Launch Record As It Preps More Starlink Satellites.  https://www.pcmag.com/news/spacex-beats-annual-launch-record-as-it-preps-more-starlink-satellites.
  * Kassem et al. (2022) Mohamed M Kassem, Aravindh Raman, Diego Perino, and Nishanth Sastry. 2022.  A browser-side view of starlink connectivity. In _ACM IMC_.
  * Kassing et al. (2020) Simon Kassing, Debopam Bhattacherjee, André Baptista Águas, Jens Eirik Saethre, and Ankit Singla. 2020.  Exploring the ”Internet from space” with Hypatia. In _ACM IMC_.
  * Kiesling (1990) John D Kiesling. 1990.  Land mobile satellite systems. In _Proceedings of the IEEE_.
  * Klenze et al. (2018) Tobias Klenze, Giacomo Giuliari, Christos Pappas, Adrian Perrig, and David Basin. 2018.  Networking in Heaven as on Earth. In _ACM HotNets_.
  * Kwok (2001) Kenneth Chun Hei Kwok. 2001.  _Cost optimization and routing for satellite network constellations_.  Ph. D. Dissertation. Massachusetts Institute of Technology.
  * Lai et al. (2023) Zeqi Lai, Hewu Li, Yangtao Deng, Qian Wu, Jun Liu, Yuanjie Li, Jihao Li, Lixin Liu, Weisen Liu, and Jianping Wu. 2023.  $\\{$StarryNet$\\}$: Empowering Researchers to Evaluate Futuristic Integrated Space and Terrestrial Networks. In _USENIX NSDI_.
  * LEOScope Team (2023) LEOScope Team. 2023.  LEOScope Simulation Profiles.  https://anonymous.4open.science/r/leoscope-simulation-profiles/README.md.
  * Mao et al. (2017) Hongzi Mao, Ravi Netravali, and Mohammad Alizadeh. 2017.  Neural adaptive video streaming with pensieve. In _ACM SIGCOMM_.
  * Maral (1994) Gérard Maral. 1994.  The ways to personal communications via satellite. In _International journal of satellite communications_. Wiley Online Library.
  * Mauldin (2019) Alan Mauldin. 2019.  Will New Satellites End the Dominance of Submarine Cables?  https://blog.telegeography.com/will-new-satellites-end-the-dominance-of-submarine-cables.
  * Mellow (2004) Craig Mellow. 2004.  The Rise and Fall and Rise of Iridium.  https://www.smithsonianmag.com/air-space-magazine/the-rise-and-fall-and-rise-of-iridium-5615034/.
  * Michel et al. (2022) François Michel, Martino Trevisan, Danilo Giordano, and Olivier Bonaventure. 2022.  A first look at starlink performance. In _ACM IMC_.
  * Microsoft (2023) Microsoft. 2023.  Azure Cognitive Services.  https://azure.microsoft.com/en-in/services/cognitive-services/.
  * Microsoft Azure (2023) Microsoft Azure. 2023.  Azure Stack Edge.  https://azure.microsoft.com/en-us/products/azure-stack/edge/.
  * Musk (2021) Elon Musk. 2021.  Starlink simultaneously active users just exceeded the strategically important threshold of 69,420 last night!  https://twitter.com/elonmusk/status/1408558492009566214.
  * Musk (2022) Elon Musk. 2022.  Over 250k Starlink user terminals.  https://twitter.com/elonmusk/status/1493358044989767683.
  * Nandugudi et al. (2013) Anandatirtha Nandugudi, Anudipa Maiti, Taeyeon Ki, Fatih Bulut, Murat Demirbas, Tevfik Kosar, Chunming Qiao, Steven Y. Ko, and Geoffrey Challen. 2013.  PhoneLab: A Large Programmable Smartphone Testbed. In _Proc. of SENSEMINE_.
  * Nardi (2020) Tom Nardi. 2020.  Literally Tearing Apart A SpaceX Starlink Antenna - Hackaday.  https://hackaday.com/2020/11/25/literally-tearing-apart-a-spacex-starlink-antenna/.
  * Nelson’s log (2021) Nelson’s log. 2021.  Starlink Dishy: getting some deeper status.  https://nelsonslog.wordpress.com/2021/06/27/starlink-dishy-getting-some-deeper-status/.
  * Netravali et al. (2015) Ravi Netravali, Anirudh Sivaraman, Somak Das, Ameesh Goyal, Keith Winstein, James Mickens, and Hari Balakrishnan. 2015.  Mahimahi: Accurate Record-and-Replay for HTTP.. In _Usenix annual technical conference_. 417–429.
  * NLTK Project (2023) NLTK Project. 2023.  Natural Language Toolkit.  https://www.nltk.org/.
  * OneWeb (2023) OneWeb. 2023.  Our Story.  https://oneweb.net/about-us/our-story.
  * Orbit testbed ([n. d.]) Orbit testbed [n. d.].  Orbit Testbed.  https://winlab.rutgers.edu/research-projects/orbit-testbed/.
  * Ramakrishnan et al. (2001) K Ramakrishnan, Sally Floyd, and D Black. 2001.  Rfc3168: The addition of explicit congestion notification (ecn) to ip.
  * Reddit (2023) Reddit. 2023.  Starlink - SpaceX Satellite Internet Constellation.  https://www.reddit.com/r/Starlink/.
  * Roberts (2022) Thomas G. Roberts. 2022.  Space Launch to Low Earth Orbit: How Much Does It Cost?  https://aerospace.csis.org/data/space-launch-to-low-earth-orbit-how-much-does-it-cost/.
  * Schmid et al. (2021) Josef Schmid, Alfred Höss, and Björn W. Schuller. 2021.  A Survey on Client Throughput Prediction Algorithms in Wired and Wireless Networks.  _ACM Comput. Surv._ 54, 9, Article 194 (oct 2021), 33 pages.  https://doi.org/10.1145/3477204
  * Sheetz (2021a) Michael Sheetz. 2021a.  SpaceX says its Starlink satellite internet service now has over 10,000 users.  https://www.cnbc.com/2021/02/04/spacex-starlink-satellite-internet-service-has-over-10000-users.html.
  * Sheetz (2021b) Michael Sheetz. 2021b.  SpaceX says Starlink has about 90,000 users as the internet service gains subscribers.  https://www.cnbc.com/2021/08/03/spacex-starlink-satellite-internet-has-about-90000-users.html.
  * Sheetz (2022a) Michael Sheetz. 2022a.  SpaceX’s Starlink internet service has more than 145,000 users so far.  https://tinyurl.com/33p7eexp.
  * Sheetz (2022b) Michael Sheetz. 2022b.  SpaceX’s Starlink satellite internet surpasses 400,000 subscribers globally.  https://tinyurl.com/yc5rhh3n.
  * Sherstinsky (2020) Alex Sherstinsky. 2020.  Fundamentals of recurrent neural network (RNN) and long short-term memory (LSTM) network.  _Physica D: Nonlinear Phenomena_ 404 (2020), 132306.
  * SpaceX (2022) SpaceX. 2022.  Starlink now has more than 1,000,000 active subscribers.  https://twitter.com/SpaceX/status/1604872936976154624.
  * SpaceX (2023a) SpaceX. 2023a.  Starlink.  https://www.starlink.com/.
  * SpaceX (2023b) SpaceX. 2023b.  Thank you to our 1.5M+ customers around the world!  https://twitter.com/Starlink/status/1654673695007457280.
  * SpaceX FCC update (2018) SpaceX FCC update. 2018.  SPACEX NON-GEOSTATIONARY SATELLITE SYSTEM.  https://licensing.fcc.gov/myibfs/download.do?attachment_key=1569860.
  * Starlink SubReddit (2022a) Starlink SubReddit. 2022a.  How is Starlink for video calls?  https://www.reddit.com/r/Starlink/comments/zaes5u/how_is_starlink_for_video_calls/.
  * Starlink SubReddit (2022b) Starlink SubReddit. 2022b.  Video Conferencing.  https://www.reddit.com/r/Starlink/comments/zhtj1w/video_conferencing/.
  * Sterling and Hatlelid (1991) David E Sterling and John E Hatlelid. 1991.  The IRIDIUM system-a revolutionary satellite communications system developed with innovative applications of technology. In _MILCOM 91-Conference record_. IEEE.
  * Stevenson (2023) Doug Stevenson. 2023.  Is Starlink Good for Gaming? The Truth [2023].  https://blinqblinq.com/starlink-for-gaming/.
  * Sundaresan et al. (2014) Srikanth Sundaresan, Sam Burnett, Nick Feamster, and Walter De Donato. 2014.  Bismark: A testbed for deploying measurements and applications in broadband access networks. In _2014 USENIX Annual Technical Conference (ATC)_. 383–394.
  * Telesat (2020) Telesat. 2020.  APPLICATION FOR MODIFICATION OF MARKET ACCESS AUTHORIZATION.  https://fcc.report/IBFS/SAT-MPL-20200526-00053/2378318.pdf.
  * Tysonpower (2021) Tysonpower. 2021.  SpaceX shutdown part of Dishys API because of me (and others).  https://tysonpower.de/blog/spacex-shutdown-part-of-dishys-api-because-of-me-and-others.
  * u/KeyserSosa (2023) u/KeyserSosa. 2023.  An Update Regarding Reddit’s API.  https://www.reddit.com/r/reddit/comments/12qwagm/an_update_regarding_reddits_api/.
  * Vasisht et al. (2021) Deepak Vasisht, Jayanth Shenoy, and Ranveer Chandra. 2021.  L2D2: Low latency distributed downlink for LEO satellites. In _ACM SIGCOMM_.
  * Werner et al. (1995) Markus Werner, Axel Jahn, Erich Lutz, and Axel Bottcher. 1995.  Analysis of system parameters for LEO/ICO-satellite communication networks. In _IEEE Journal on Selected areas in Communications_.
  * WIEDEMAN et al. (1992) ROBERT WIEDEMAN, ALLEN SALMASI, and Dennis Rouffet. 1992.  Globalstar-Mobile communications where ever you are. In _14th International Communication Satellite Systems Conference and Exhibit_. AIAA.
  * Wikipedia (2023a) Wikipedia. 2023a.  Autoregressive integrated moving average.  https://en.wikipedia.org/wiki/Autoregressive_integrated_moving_average.
  * Wikipedia (2023b) Wikipedia. 2023b.  CubeSat.  https://en.wikipedia.org/wiki/CubeSat.
  * Wikipedia (2023c) Wikipedia. 2023c.  Geosynchronous orbit.  https://en.wikipedia.org/wiki/Geosynchronous_orbit.
  * Wikipedia (2023d) Wikipedia. 2023d.  Laser communication in space.  https://en.wikipedia.org/wiki/Laser_communication_in_space.
  * Wikipedia (2023e) Wikipedia. 2023e.  List of Starlink and Starshield launches.  https://en.wikipedia.org/wiki/List_of_Starlink_and_Starshield_launches#Starlink_Launches.
  * Wikipedia (2023f) Wikipedia. 2023f.  Low Earth orbit.  https://en.wikipedia.org/wiki/Low_Earth_orbit.
  * Wikipedia (2023g) Wikipedia. 2023g.  SpaceX reusable launch system development program.  https://en.wikipedia.org/wiki/SpaceX_reusable_launch_system_development_program.
  * Wikipedia (2023h) Wikipedia. 2023h.  Starlink.  https://en.wikipedia.org/wiki/Starlink.
  * Wischik et al. (2011) Damon Wischik, Costin Raiciu, Adam Greenhalgh, and Mark Handley. 2011.  Design, Implementation and Evaluation of Congestion Control for Multipath TCP.. In _NSDI_.
  * Wood (2001) Lloyd Wood. 2001.  _Internetworking with satellite constellations_.  Ph. D. Dissertation. University of Surrey.
  * Wu et al. (1994) William W Wu, Edward F Miller, Wilbur L Pritchard, and Raymond L Pickholtz. 1994.  Mobile satellite communications. In _Proceedings of the IEEE_.
  * Yin et al. (2015) Xiaoqi Yin, Abhishek Jindal, Vyas Sekar, and Bruno Sinopoli. 2015.  A control-theoretic approach for dynamic adaptive video streaming over HTTP. In _ACM SIGCOMM_.