# Making Sense of Constellations

Methodologies for Understanding Starlink’s Scheduling Algorithms

Hammas Bin Tanveer  University of Iowa ,  Mike Puchol  Google X ,  Rachee
Singh  Cornell University ,  Antonio Bianchi  Purdue University and  Rishab
Nithyanand  University of Iowa

††copyright: none††doi: ††isbn: ††price:

## Abstract

Starlink constellations are currently the largest LEO WAN and have seen
considerable interest from the research community. In this paper, we use high-
frequency and high-fidelity measurements to uncover evidence of hierarchical
traffic controllers in Starlink — a global controller which allocates
satellites to terminals and an on-satellite controller that schedules
transmission of user flows. We then devise a novel approach for identifying
how satellites are allocated to user terminals. Using data gathered with this
approach, we measure the characteristics of the global controller and identify
the factors that influence the allocation of satellites to terminals. Finally,
we use this data to build a model which approximates Starlink’s global
scheduler. Our model is able to predict the characteristics of the satellite
allocated to a terminal at a specific location and time with reasonably high
accuracy and at a rate significantly higher than baseline.

##  1\. Introduction

Low-earth orbit (LEO) satellite networks are expected to play an important
role in achieving global broadband-like Internet connectivity since they
enable low-latency, last-mile connectivity without heavy infrastructure costs
(_e.g.,_ cell towers and fiber deployments), Unfortunately, prior work has
shown lackluster network performance from Starlink-connected end-hosts, both
via measurements (Michel et al., 2022) as well as simulations (Bhosale et al.,
2023; Kassing et al., 2020). Despite these findings, researchers have been
unable to propose and validate methods for improving the performance of the
Starlink network. This is, in large part, due to the opacity of the network
and its scheduling algorithms. Knowledge of algorithms responsible for
determining which satellites route traffic from specific user terminal
locations is key to engineering performance improvements for the network. In
this paper, we address this gap in knowledge. In particular, we empirically
uncover the scheduling algorithms used by the Starlink network by analyzing
data from high-frequency measurements from four Starlink terminals (deployed
both in the US and the EU) to servers co-located at their corresponding Point-
of-Presence.

From our longitudinal and high-frequency measurements, we find that Starlink
routes traffic from user terminals to their ground stations in a two-step
process. First, a global network controller allocates a satellite to each user
terminal based on a variety of factors including load, geospatial conditions,
satellite charge, etc. Our experiments show that these allocations are made
every 15-seconds, globally. Second, a local on-satellite controller schedules
flows from the user terminals assigned to it. Taken together, these findings
suggest that the hierarchical traffic engineering mechanisms, commonly
deployed in terrestrial WANs (Hong et al., 2018), are also deployed by
Starlink. We were able to confirm the validity of these major findings using
recent FCC filings by SpaceX (Sta, [n. d.]).

This work is the first to uncover hierarchical controllers and the
characteristics of traffic engineering in Starlink, due to the following three
key reasons. First, our high-frequency (millisecond granularity) measurements
allow us to observe signatures of traffic engineering (_e.g.,_ abrupt latency
changes) that cannot be observed with the coarse-grained measurements of prior
work (Michel et al., 2022). Second, by co-locating our destination server at
the Starlink PoP, we minimize the influence of terrestrial latency on our
measurements. Finally, our novel methodology for identifying the satellite
currently serving a given terminal allows us to obtain ground-truth regarding
the traffic engineering decisions made by Starlink.

Contributions. This paper makes four main contributions.

  * $\bullet$

We use high-frequency measurements to show evidence of hierarchical traffic
engineering on Starlink (§3).

  * $\bullet$

We develop a novel technique for identifying the satellite that serves a user
terminal (§4).

  * $\bullet$

We uncover the characteristics and preferences of Starlink’s global scheduler
— _i.e.,_ the algorithm responsible for allocating satellites to specific user
terminals (§5).

  * $\bullet$

We develop an approximation of the Starlink global scheduler that can predict
the satellites allocated to user terminals with reasonably high accuracy (§6).

##  2\. Background

Starlink is a low-earth orbit satellite constellation consisting of nearly
4000 satellites in the low Earth orbit (LEO). The Starlink ecosystem has four
key components: (1) In-orbit satellites, (2) User terminals or dishes (3)
Ground stations and (4) Points of Presence (PoPs). Figure 1 shows how these
components interact with each other to provide Internet connectivity to
Satrlink’s end users.

User terminals. Terminals are deployed on user premises to connect to in-orbit
satellites. Starlink user terminals are sophisticated phased-array antennas
equipped with a motor that can physically reposition the angle of the dish to
track fast moving satellites in the sky. User terminals can connect to any
satellite at an angle of elevation higher than 25∘s (see Figure 1). While tens
of satellites satisfy the angle of elevation constraints, a terminal can
connect to only one satellite at a time. Terminals forward user traffic to the
satellite assigned to them. Internals of the algorithm that maps user
terminals to satellites are currently known only to the operators of the
Starlink network. In this work, we empirically demonstrate characteristics of
this algorithm and build its approximation.

Satellites. A Starlink satellite connects to multiple user terminals at a
time. They allocate radio frames to user terminals mapped to them to exchange
data. In our work, we find evidence that this allocation is determined by a
local on-satellite controller. In fact, we also find the description of this
controller, referred to as the medium access control scheduler (Iyer et al.,
2022), in recent FCC filings from SpaceX. This controller considers factors
such as user priority, current load, and per-terminal flow characteristics
when forwarding the traffic from user terminals to ground stations.

Ground stations and PoPs. Ground stations consist of a set of phased-array
antennas that receive traffic from satellites and send it through wired links
to Starlink’s PoPs. Like user terminals, ground stations can communicate with
satellites at an angle of elevation higher than 25∘ above the horizon,
relative to the ground station. A PoP is a terrestrial server with wired
connectivity to a ground station. PoPs are connected to the Internet backbone.
From the PoP, the traffic is routed to destination on the Internet.

##  3\. Evidence of Traffic Engineering

Experiment setup: Vantage points. We perform our measurement using four
Starlink terminals — one each in Western Europe, Northeast US, Midwest US, and
Northwest US. To improve the precision of our measurements, we configured the
Starlink router to operate in bridge mode and connected them to a dedicated
Raspberry Pi via Ethernet. These Pi’s were the source of our measurements.
This approach prevents the complexities which arise from using wireless
routers in the measurement infrastructure. The destination of our measurements
were servers co-located at the Starlink PoPs assigned to the regions of our
user terminals. This choice of destination allows our measurements to be
relatively unimpacted by the vagaries of terrestrial networking.

Figure 1. Starlink ecosystem.

Experiment setup: Measurements. We conduct high- frequency measurements of the
round-trip times and packet loss rates between our sources and destination
servers. Packets were sent using iRTT (irt, [n. d.]) at the rate of 1
packet/20 ms and iPerf3 at a bandwidth of 50% of the upstream connection.
These parameters were chosen because they allowed stable and reliable
measurements of the Starlink network. At higher frequencies and bandwidths,
the packet loss rates and measured round-trip times were highly variable even
within the same measurement period. From these measurements, we recorded high-
resolution round-trip times and packet loss rates. To facilitate accurate
measurements of the round-trip times, the clocks of our vantage points and
servers were routinely synchronized using NTP.

Figure 2. Measured RTT from EU terminal.

Observation: Starlink relies on a global controller for satellite-to-terminal
scheduling. Figure 2 shows the changes in measured latency during a brief
measurement window of two minutes for our Midwest US terminal. It is
immediately obvious that major changes in latency characteristics occur every
15 seconds — specifically, at the 12th, 27th, 42nd, and 57th second past every
minute. Notably, these changes are observed from all our measured locations
for all periods of time. In addition to visual tests, we are also able to
confirm that the latency characteristics observed during these consecutive
15-second windows are statistically different (Mann-Whitney U test; $p$ ¡ .05)
from each other for all locations and over the entire period of our
measurements. These drastic changes in latency are suggestive of global
changes in the satellites allocated to user terminals for several reasons,
including: (1) our measurements effectively nullify the impact of terrestrial
networks; (2) these effects were observed, simultaneously, from all our
vantage points; and (3) these effects were noticed even when our terminals
were running well under capacity. Upon further investigation, we discovered an
FCC filing from SpaceX which describes a global scheduler for periodically
allocating terminals to satellites (Sta, [n. d.]). We conclude that our
measurements have uncovered evidence of this scheduler and the periodicity of
reallocations. It is important to note that this finding renders impossible
the hypothesis that performance characteristics are associated with the
movement of the satellite assigned to serve a terminal (_e.g.,_ Figure 7 in
(Kassem et al., 2022)). This is because changes in satellite allocation occur
every 15 seconds which is insufficient time to meaningfully cause impacts on
performance due to change in satellite positions/distances. In the remainder
of this paper, we will uncover the characteristics of this scheduler and
develop an offline approximation for it.

Observation: Starlink uses an on-satellite controller for scheduling terminal
flows. The second peculiar characteristic of the latency measurements from our
user terminals is that within the fifteen-second time interval, latency
measurements the user terminal frequently form parallel bands that are a few
milliseconds apart. These bands reflect evidence that radio frames are
allocated to user terminals by an on-satellite controller in a somewhat round-
robin fashion. Further investigation is required to exactly identify the
characteristics of this controller, which we believe to be the on-satellite
Medium Access Control scheduler described in a SpaceX FCC filing (Iyer et al.,
2022).

##  4\. Obtaining Satellite Allocations

Our results show that the Starlink network uses a global scheduler to assign
satellites to user terminals every 15 seconds (Section 2). Unfortunately, the
Starlink mobile app no longer identifies the satellite that a user terminal is
connected to. Not having this knowledge limits our ability to reverse-engineer
the mechanics of Starlink’s scheduling algorithms. In this section, we develop
a novel technique that leverages Starlink’s obstruction maps to identify the
satellite allocated to a specific user terminal. At a high-level, our approach
involves correlating the publicly known positions of the Starlink satellites
with observations of connected satellites recorded by in the obstruction maps
of each terminal.

Data: Obstruction maps. Obstruction maps are 123px x 123px, 2-dimensional
images which mark the trajectory of satellites that recently served the user
terminal. These images are used to create a 3-dimensional map made available
to users via the Starlink mobile app. The 3-d map is meant to help users
identify the quality of the location of their terminal, highlighting any
physical obstructions between their terminal and any satellites meant to serve
the terminal. Figure 3(a) shows an example of this map from the Starlink app.
We used starlink-grpc-tools (spa, [n. d.]) to extract the 2-d obstruction maps
every 15 seconds from each terminal. The 3-d maps are not possible to obtain
programmatically. Figures 3(b) and 3(c) shows examples of these maps from two
consecutive 15-second time slots.

(a) Starlink app.

(b) gRPC($x-1$)

(c) gRPC($x$)

(d) gRPC($x-1$) $\oplus$ gRPC($x$)

(e) gRPC map after 2 days.

Figure 3. Obstruction maps (a) obtained from the Starlink app, (b, c) obtained
from gRPC for two consecutive 15-second slots $x-1$ and $x$, (d) their XOR,
and (e) the gRPC map after two days without a terminal reset.

Data: Satellite positions. Positions of Starlink satellites are available
publicly from a variety of sources in a two-line element(TLE) format. We use
CelesTrak (Cel, [n. d.]) to get the TLEs for Starlink satellites. Since these
files only indicate satellite positions every six hours, we use the SGP4
satellite propagation algorithm (Rhodes, 2019) to calculate satellite
positions, relative to a terminal location, for a specific point in time.

Method: Uncovering gRPC obstruction map parameters. As seen in Figures 3(b)
and 3(c), the 2-d obstruction maps are plain — only containing white pixels
which indicate the trajectory of recently connected satellites without any
further context (_e.g.,_ the angle of elevation and azimuth). Identifying
these parameters from this map is crucial for identifying the connected
satellite. To do this, we align the recorded 2-d maps with the 3-d maps
observed on the Starlink app. From this alignment, we find that: (1) the 2-d
obstruction map is a polar plot centered at 62x62; (2) the radius of the polar
plot represents the angle of elevation and ranges from 25 to 90; and (3) the
$\theta$ of the polar plot represents the azimuth, where $\theta=0$ represents
the North. Finally, since the obstruction map is a square which contains a
polar plot, we also need to get the boundaries of the polar plot within this
square. We accomplish this by keeping the terminal online for two consecutive
days. In fact, over a 2-day period the terminal will establish connections
with satellites from practically all the regions of the sky that are within
the field of view. In turn, this will result in essentially fully coloring the
polar plot region in the gRPC map, since the gRPC map does not reset (unless
the terminal goes offline). Using this fully colored map (an example is shown
in Figure 3(e)), we find that the radius of the contained polar plot is 45
pixels.

Method: Isolating satellite trajectory. After recovering the parameters of the
gRPC obstruction map, we use them to identify the trajectory of the satellite
that is allocated to the terminal for a specific 15-second slot (denoted by
$x$). To obtain this trajectory, we perform an XOR operation on the
obstruction map from $x$ and $x-1$ (_i.e.,_ the prior 15-second slot). This
will result in the erasure of all satellite trajectories which were common to
the two figures — leaving visible only the trajectory associated with the
satellite connected to the terminal during slot $x$. We note that for this
method to work, we require that satellite trajectories are not overlapping
with the trajectories of previously connected satellites (since an XOR would
erase the overlaps). To ensure that this condition is met, we perform a
terminal reset every 10 minutes (since resetting the terminal starts a fresh
gRPC map).

Method: Identifying serving satellite. From the prior step, we have the angle
of elevation and azimuth trajectories of the satellite that served the
terminal for a specific 15-second slot. To identify the specific ID of the
serving satellite, we use the computed relative locations of the satellites in
the area (as previously explained) to obtain the IDs of all satellites which
are visible to our terminal during the specific time slot. Next, we compute
the angle of elevations and azimuths for each of these satellites during the
given slot. Finally, we convert the trajectories to cartesian coordinates and
then use the Dynamic Time Warping (DTW) (Müller, 2007) distance measure to
compute the similarity of trajectories. We select the satellite whose angle of
elevation and azimuth trajectories are the most similar to those recorded from
the gRPC maps. We validate our similarity matching via a manual (visual) pilot
test study, in which the authors manually identified the best match between
500 sets of gRPC and TLE trajectories. The DTW similarity method and our
manual tests overlapped on over 99% of all outcomes.

##  5\. Starlink’s global scheduler

The Starlink network uses a global scheduler to allocate user terminals to
individual satellites every 15 seconds (Section 2). However, the scheduling
algorithm is not publicly known. In this section, we analyze characteristics
of satellites that were allocated to our terminals with the goal of reverse
engineering the algorithm of the global scheduler. On average, there are 35–44
satellites in the field of view of a user terminal in any 15 second slot.
Having identified the satellite that a terminal is connected to during a slot
(Section 4), we compare properties of the satellites that are selected by the
scheduler with those that were available but not chosen.

###  5.1. Impact of satellite position

Angle of elevation. First, we compare the positions in the sky of satellites
that were available but not selected to the positions of satellites that were
selected by the global scheduler. The position of a satellite in the sky is
defined by its AOE and azimuth with respect to the user terminal. Figure 4
shows that the median angle of elevation of selected satellites (solid lines)
is 22.9∘s higher than that of the available but unselected satellites (dotted
lines). Although only 30% of all available satellites had their AOEs in 45∘ to
90∘ range, the global scheduler picked 80% of satellites from the range
(averaged over all locations).

Direction. Figure 5 shows the distribution of azimuths of the two sets of
satellites. The plot is divided into 4 quadrants which represent the direction
– stated at the top of each quadrant – of the satellites relative to the face
of the user terminal. Although the azimuths of available satellites (dotted
lines) are evenly distributed throughout the 4 quadrants, azimuths of selected
satellites (solid lines) are skewed towards the north of the dish, except for
the dish in Ithaca, NY. We investigated this difference and found that our
user terminal in Ithaca was severely obstructed by trees towards the north
west direction causing it to pick fewer satellites from the north west. The
terminal in Ithaca was assigned only 9.7% of the satellites from the region
compared to 55.4% on average by user terminals in other locations. In other
locations, 58% of satellites on average were available towards the north of
the user terminals, however, the user terminal was mapped to satellites from
the north 82% of the times.

Figure 4. AOEs of available (dotted) vs. selected (solid) satellites. Figure
5. CDF of AZs of satellites available (dotted) vs. AZs of satellites chosen
(solid). The x-axis is divided into 4 quadrants of 90 degrees each; the
direction of the quadrant relative to the user terminal’s face is listed on
top of each quadrant.

Rationale: The International Telecommunication Union has imposed a mandatory
geo-stationary orbit exclusion zone, which prohibits LEO satellites from
transmitting to or receiving from a ground station while being in the
protected part of the sky (itu, [n. d.]). This mandate forces terminals at
latitudes more than 40∘N, the approximate latitude of our terminals, to point
much higher than required by the minimum angle of elevation constraint. This
is why the global scheduler assigns satellites higher up in the fields-of-view
of our terminals. Moreover, satellites with a higher AOE can communicate with
terminals in a more energy efficient way. The distance between the user
terminal and satellite increases inversely with AOE. As radio frequency (RF)
power decreases inversely with distance, satellites farther away need to use
significantly more power to communicate with user terminals.

###  5.2. Impact of satellite launch dates

The Starlink constellation consists of more than 4,000 satellites which were
released in batches since 2018. We analyze whether the global scheduler
prefers satellites from certain batches over others. To achieve this, we bin
satellites by the year and month of their launch batch. We then compute %(No.
of slots a satellite from a launch was picked/No. of slots a satellite from a
launch was available) for all 15 second slots in our observation. Figure 6
shows the distributions of these probabilities with the launch dates of
satellites binned by year and month. Averaged over all locations, the
probability of picking a satellite increases with an increase of one month in
the satellite’s launch date.

Rationale: As Starlink satellites are launched in batches, the difference in
service time between the latest and oldest satellites can differ by years. As
a Starlink satellite has a service lifetime of about 5 years, some parts of
the constellation would be out of service years before others. This would
require consistent replacement efforts to maintain the constellation’s
coverage. Hence, using satellites launched later in time can help offset the
discrepancy in service times of satellites allowing the constellation to stay
stable for longer.

Figure 6. Correlation of probability of picking a satellite from a launch vs.
launch date

###  5.3. Impact of being sunlit

Starlink satellites are equipped with solar panels to provide power for their
operations. However, as satellites orbit the earth, they periodically go in
and out of sunlight. To serve user terminals during times when a satellite is
not under sunlight, it has to conserve energy to stay functional until it gets
sunlight again. For this reason, we analyze whether Starlink’s global
scheduler has preference for sunlit satellites.

For every 15 second slot, we calculate positions of all available satellites
relative to the sun using the SkyField library. We then extract 15 second
slots during which at least one sunlit and at least one dark satellite was
available. During such slots, the global scheduler opts for the sunlit
satellites $72.3\%$ of the time averaged over all locations. We also find that
the global scheduler only picks dark satellites during 15 second slots where
the %(dark/available) satellites is $>=35\%$ (averaged over all locations).
Next, we compare the positions of the dark satellites that were picked by the
global scheduler with the positions of their sunlit counterparts. We find that
the AOE of dark satellites picked by the scheduler was 25 degrees higher than
their sunlit counterparts.

Rationale:  As mentioned earlier, the RF power decreases inversely with
distance causing satellites to expend more energy in communicating with the
user terminal. As dark satellites have limited battery to stay functional, the
global scheduler assigns satellites to user terminals that are higher up in
the field-of-view to conserve energy.

##  6\. Modeling the Global Scheduler

Our analysis has shown that the Starlink global scheduler has specific
preferences — _i.e.,_ it selects newer satellites that are sunlit, located
towards the North West, and at a high angle of elevation. We now use these
preferences and our data to build an offline model of the scheduler. The goal
of this model is to predict the characteristics of the satellite allocated to
serve a terminal in a given location and time.

Feature selection. Given a specific location and time, we first identify the
satellites that are available to serve the user terminal using our TLE dataset
and the SGP4 algorithm. Next, we cluster the available satellites based on
their azimuth ($\theta$), angle of elevation ($\phi$), age ($a$), and sunlit
status ($\epsilon$) as follows: Given a set of satellites ($S$) available at
time $t$ for location $l$, the satellite $s\in S$ with parameters
($\theta_{s}$, $\phi_{s}$, $a_{s}$, $\epsilon_{s}$) is placed in the cluster:
($\frac{\theta_{s}-\mu(\theta)}{\sigma(\theta)}$,
$\frac{\phi_{s}-\mu(\phi)}{\sigma(\phi)}$, $\frac{a_{s}-\mu(a)}{\sigma(a)}$,
$\epsilon$). Here $\mu(x)$ and $\sigma(x)$ denote the mean and standard
deviation of the feature $x$ (computed from $S$). Effectively, this approach
clusters satellites by how many standard deviations away from the group mean
each of their features are. For example, a satellite in the cluster (1, 0, 2,
1) is further than 1 standard deviation away from the mean azimuth and 2
standard deviations away from the mean age of all the satellites in $S$. As
features for our scheduler, we present the local time ($t_{l}$) and the count
of satellites available in each cluster at the start of the nearest 15-second
interval.

Training, testing, and validating our model. Our goal is to construct a model
which takes the above features as input and returns the cluster to which the
allocated satellite belongs. We train a random forest model because of its
robustness to overfitting and the explainability of its predictions. We got
the parameters of this model using grid-search and five-fold cross-validation.
We use the data gathered from each location and measured satellite allocations
(Section 4) to construct our model — 80% of the data is used to create a
training/testing dataset for a five-fold cross-validation evaluation and the
remaining 20% is used to create a holdout dataset to validate our model’s
robustness to overfitting.

Evaluating the model. To measure the accuracy of our model, we use a top-k-
accuracy metric — _i.e.,_ we analyze the accuracy of the $k$ most likely
predictions made by the model. To put this accuracy in context, we compare it
to the baseline model which simply returns the (top-$k$) cluster(s) with the
most number of available satellites as its prediction. Our results obtained
over our holdout set are shown in Figure 7. The proposed model significantly
outperforms the baseline model and predicts the correct allocated satellite
characteristics 65% of the time ($k$=5 guesses), in comparison to the baseline
of 22%.

Figure 7. Accuracy of our model compared to baseline using the top-$k$
accuracy metric.

Limitations. Our model is constructed using only publicly measurable data
related to Starlink’s satellites. However, based on disclosures in SpaceX’s
FCC filings (Sta, [n. d.]; Iyer et al., 2022), we expect that other publicly-
unavailable features such as terminal density in a region and satellite load
characteristics will also impact the global scheduler. Therefore, the
performance of our model is constrained by the unavailability of data. Despite
this, our model is demonstrably robust to over-fitting and is able to make
reasonably accurate and explainable predictions at a rate far higher than
baseline.

Model release. In order to facilitate future simulations and evaluations of
the Starlink network, our model will be publicly available on paper
acceptance.

##  7\. Related work

Previous work has studied Starlink’s performance along the axes of latency and
throughput with coarse-grained measurements (Uran et al., 2021; Michel et al.,
2022). Researchers have found geographic variability (Kassem et al., 2022) in
Starlink performance. Others have developed simulations of LEO constellations
(Valentine and Parisis, 2021; Kassing et al., 2020) to suggest improvements to
the network (Cakaj, 2021). Researchers have proposed to improve end host
performance in Starlink using better routing protocols (Handley, 2019, 2018),
better beamforming techniques (Kumar and Arnon, 2022), satellite network
topology reconfiguration (Bhattacherjee and Singla, 2019), clean-slate
protocol design (Giuliari et al., 2020) and efficient satellite handoff
techniques (Juan et al., 2022; Liu et al., 2022). Security researchers have
also found potential attack surfaces in Starlink (Smailes et al., 2023;
Giuliari et al., 2021).

##  8\. Conclusions

In this paper, we leveraged high-frequency and high-fidelity measurements from
four globally distributed vantage points to uncover the presence of a
hierarchical traffic controller — a global controller for allocating
satellites to user terminals and an on-satellite local controller for
scheduling user flows. Using a novel technique for identifying the satellites
allocated to terminals, we identified the characteristics and preferences of
the global scheduler and developed an offline approximation. Taken all
together, our work is a first step towards understanding Starlink’s traffic
engineering decisions and supporting future efforts to model and evaluate
scheduling algorithms for the Starlink network.

## References

  * (1)
  * itu ([n. d.]) [n. d.].  47 CFR § 25.289 - Protection of GSO networks by NGSO systems.  https://www.law.cornell.edu/cfr/text/47/25.289. ([n. d.]).  (Accessed on 06/25/2023).
  * Cel ([n. d.]) [n. d.].  CelesTrak.  https://celestrak.org/. ([n. d.]).  (Accessed on 06/25/2023).
  * irt ([n. d.]) [n. d.].  IRTT.  https://iperf.fr/. ([n. d.]).  (Accessed on 06/25/2023).
  * spa ([n. d.]) [n. d.].  sparky8512/starlink-grpc-tools: Random scripts and other bits for interacting with the SpaceX Starlink user terminal hardware.  https://github.com/sparky8512/starlink-grpc-tools. ([n. d.]).  (Accessed on 06/25/2023).
  * Sta ([n. d.]) [n. d.].  Starlink Services LLC Application for ETC Designation — PDF Host.  https://pdfhost.io/v/BnYWSR~wq_Starlink_Services_LLC_Application_for_ETC_Designation. ([n. d.]).  (Accessed on 06/27/2023).
  * Bhattacherjee and Singla (2019) Debopam Bhattacherjee and Ankit Singla. 2019.  Network topology design at 27,000 km/hour. In Proceedings of the 15th International Conference on Emerging Networking Experiments And Technologies. 341–354.
  * Bhosale et al. (2023) Vaibhav Bhosale, Ahmed Saeed, Ketan Bhardwaj, and Ada Gavrilovska. 2023.  A Characterization of Route Variability in LEO Satellite Networks. In International Conference on Passive and Active Network Measurement. Springer, 313–342.
  * Cakaj (2021) Shkelzen Cakaj. 2021\.  The parameters comparison of the “Starlink” LEO satellites constellation for different orbital shells.  Frontiers in Communications and Networks 2 (2021), 643095.
  * Giuliari et al. (2021) Giacomo Giuliari, Tommaso Ciussani, Adrian Perrig, Ankit Singla, and E Zurich. 2021.  ICARUS: Attacking low Earth orbit satellite networks.. In USENIX Annual Technical Conference. 317–331.
  * Giuliari et al. (2020) Giacomo Giuliari, Tobias Klenze, Markus Legner, David Basin, Adrian Perrig, and Ankit Singla. 2020\.  Internet backbones in space.  ACM SIGCOMM Computer Communication Review 50, 1 (2020), 25–37.
  * Handley (2018) Mark Handley. 2018\.  Delay is not an option: Low latency routing in space. In Proceedings of the 17th ACM Workshop on Hot Topics in Networks. 85–91.
  * Handley (2019) Mark Handley. 2019\.  Using ground relays for low-latency wide-area routing in megaconstellations. In Proceedings of the 18th ACM Workshop on Hot Topics in Networks. 125–132.
  * Hong et al. (2018) Chi-Yao Hong, Subhasree Mandal, Mohammad Al-Fares, Min Zhu, Richard Alimi, Kondapa Naidu B., Chandan Bhagat, Sourabh Jain, Jay Kaimal, Shiyu Liang, Kirill Mendelev, Steve Padgett, Faro Rabe, Saikat Ray, Malveeka Tewari, Matt Tierney, Monika Zahn, Jonathan Zolla, Joon Ong, and Amin Vahdat. 2018\.  B4 and after: Managing Hierarchy, Partitioning, and Asymmetry for Availability and Scale in Google’s Software-Defined WAN. In Proceedings of the 2018 Conference of the ACM Special Interest Group on Data Communication (SIGCOMM ’18). Association for Computing Machinery, New York, NY, USA, 74–87.  https://doi.org/10.1145/3230543.3230545
  * Iyer et al. (2022) Jayasuryan V Iyer, Khasim Shaheed Shaik Mahammad, Yashodhan Dandekar, Ramakrishna Akella, Chen Chen, Phillip E Barber, and Peter J Worters. 2022\.  System and method of providing a medium access control scheduler.  (Dec. 27 2022).  US Patent 11,540,301.
  * Juan et al. (2022) Enric Juan, Mads Lauridsen, Jeroen Wigard, and Preben Mogensen. 2022.  Performance evaluation of the 5G NR conditional handover in LEO-based non-terrestrial networks. In 2022 IEEE Wireless Communications and Networking Conference (WCNC). IEEE, 2488–2493.
  * Kassem et al. (2022) Mohamed M Kassem, Aravindh Raman, Diego Perino, and Nishanth Sastry. 2022.  A browser-side view of starlink connectivity. In Proceedings of the 22nd ACM Internet Measurement Conference. 151–158.
  * Kassing et al. (2020) Simon Kassing, Debopam Bhattacherjee, André Baptista Águas, Jens Eirik Saethre, and Ankit Singla. 2020\.  Exploring the” Internet from space” with Hypatia. In Proceedings of the ACM Internet Measurement conference. 214–229.
  * Kumar and Arnon (2022) Rajnish Kumar and Shlomi Arnon. 2022.  DNN Beamforming for LEO Satellite Communication at Sub-THz Bands.  Electronics 11, 23 (2022), 3937.
  * Liu et al. (2022) Yaoqi Liu, Xiaogang Tang, Yiqing Zhou, Jinglin Shi, Manli Qian, and Shaoyang Li. 2022\.  Channel Reservation based Load Aware Handover for LEO Satellite Communications. In 2022 IEEE 95th Vehicular Technology Conference:(VTC2022-Spring). IEEE, 1–5.
  * Michel et al. (2022) François Michel, Martino Trevisan, Danilo Giordano, and Olivier Bonaventure. 2022.  A first look at starlink performance. In Proceedings of the 22nd ACM Internet Measurement Conference. 130–136.
  * Müller (2007) Meinard Müller. 2007\.  Dynamic time warping.  Information retrieval for music and motion (2007), 69–84.
  * Rhodes (2019) Brandon Rhodes. 2019\.  Skyfield: High precision research-grade positions for planets and Earth satellites generator.  Astrophysics Source Code Library (2019), ascl–1907.
  * Smailes et al. (2023) Joshua Smailes, Edd Salkield, Simon Birnbach, Martin Strohmeier, and Ivan Martinovic. 2023\.  Dishing out DoS: How to Disable and Secure the Starlink User Terminal.  arXiv preprint arXiv:2303.00582 (2023).
  * Uran et al. (2021) C Uran, K Horvath, and H Wöllik. 2021.  Analysis of a Starlink-based Internet connection.  Technical Report. tech. rep., Carinthia University of Applied Science, Research Group ROADMAP-5G.
  * Valentine and Parisis (2021) Aiden Valentine and George Parisis. 2021.  Developing and experimenting with leo satellite constellations in omnet++.  arXiv preprint arXiv:2109.12046 (2021).


# Exploring Explainability in Video Action Recognition

Avinab Saha∗, Shashank Gupta∗, Sravan Kumar Ankireddy∗, Karl Chahine, Joydeep
Ghosh
The University of Texas at Austin

###### Abstract

Image Classification and Video Action Recognition are perhaps the two most
foundational tasks in computer vision. Consequently, explaining the inner
workings of trained deep neural networks is of prime importance. While
numerous efforts focus on explaining the decisions of trained deep neural
networks in image classification, exploration in the domain of its temporal
version, video action recognition, has been scant. In this work, we take a
deeper look at this problem. We begin by revisiting Grad-CAM, one of the
popular feature attribution methods for Image Classification, and its
extension to Video Action Recognition tasks and examine the method’s
limitations. To address these, we introduce Video-TCAV, by building on TCAV
for Image Classification tasks, which aims to quantify the importance of
specific concepts in the decision-making process of Video Action Recognition
models. As the scalable generation of concepts is still an open problem, we
propose a machine-assisted approach to generate spatial and spatiotemporal
concepts relevant to Video Action Recognition for testing Video-TCAV. We then
establish the importance of temporally-varying concepts by demonstrating the
superiority of dynamic spatiotemporal concepts over trivial spatial concepts.
In conclusion, we introduce a framework for investigating hypotheses in action
recognition and quantitatively testing them, thus advancing research in the
explainability of deep neural networks used in video action recognition.

**footnotetext: Equal Technical Contribution. Correspondence to Avinab Saha,
Shashank Gupta and Sravan Kumar Ankireddy. Email:
<EMAIL_ADDRESS>
##  1 Introduction

Understanding human actions in videos is crucial for various applications like
behavior analysis, video retrieval, and human-robot interaction. Human action
understanding involves recognizing, localizing, and predicting human
behaviors. The task to recognize human actions in a video is termed as _Video
Action Recognition_. In recent years, significant research efforts have
focused on developing effective models for this task, including I3D [6],
SlowFast [8], and Video Swin Transformer [13]. Consequently, an intriguing
avenue for further exploration lies in understanding the decision-making
processes of these networks.
Recently, multiple works have focused on analyzing decisions made by neural
networks in the context of image classification tasks by developing feature
attribution methods: Integrated gradients [15], Class Activation Mapping (CAM)
[18], and Grad-CAM [14]. A widely embraced alternative for studying the
explainability of deep neural networks in image classification tasks over
traditional feature attribution methods is TCAV [11]. TCAV is a global
explanation method focusing on more abstract details instead of granular,
pixel-level changes typically associated with feature attribution methods.
Although there has been significant progress in post-training explainability
methods for deep learning-based image classification, very little research has
been done on the applicability of these methods in the context of video action
recognition. In this work, we aim to investigate this direction by exploring a
feature attribution method for video action recognition, discussing its
limitations, and further introducing a video counterpart to TCAV, which we
refer to as Video-TCAV. Specifically, we opt for _playing tennis_ class from
the Kinetics-400 dataset [10] to visualize the outcomes of the Grad-CAM as
well as to formulate the proposed Video-TCAV framework.
The rest of the paper is organized as follows. Section 2 discusses the
performance of Grad-CAM when employed in the context of video action
recognition to set up a baseline representing popular feature attribution
methods. In Section 3, we introduce Video-TCAV, which includes an automated
pipeline for generating high-level concepts and are evaluated in the proposed
Video-TCAV framework. Section 4 concludes the paper by summarizing the ideas
explored in this paper and discussing future research directions.

##  2 Grad-CAM revisited

(a)

(b)

Figure 1:  Grad-CAM outputs with respect to the class playing tennis for Video
Swin Transformer model. (a): Grad-CAM correctly highlights the regions of
movement for tennis rackets. (b): Grad-CAM focuses on the tennis court in the
background and ignores the players in the frame.

We first revisit Gradient-weighted Class Activation Mapping (Grad-CAM)[14], a
popular feature attribution method renowned for its effectiveness in
understanding image recognition models. Class Activation Maps (CAMs) visually
represent the focus areas for an image recognition model during prediction by
utilizing the weights of the model’s final convolutional layer. Building on
this, Grad-CAM has emerged as a popular technique for generating visual
explanations for various image recognition models by leveraging the gradient
of the predicted class concerning the activations of the final neural network
layer to produce the heatmap, thereby offering class-specific visualizations.

###  2.1 Extending Grad-CAM to Videos

A straightforward approach to adapting feature attribution methods from images
to videos is to consider each frame as an individual image. However, this
bears a significant limitation: it neglects the temporal interplay among
frames, a crucial aspect for understanding the actions in videos.
This limitation can be overcome by utilizing a collection of frames as input,
thus preserving the temporal connections. These frames are input to the Video
Swin Transformer model [13], which was pre-trained on the Kinetics-400
dataset. Subsequently, gradients are computed with respect to the
corresponding activation maps. Illustrative instances displaying Grad-CAM
outputs for inputs related to the class _playing tennis_ for the Swin
Transformer model are presented in Figure 1(a). The overlaid heatmap
demonstrates that the outputs of the Grad-CAM are concentrated in the moving
regions associated with the tennis racket.

###  2.2 Limitations in Feature Attribution Methods

Extending feature attribution methods designed for image recognition models to
work well with videos is a non-trivial task and raises concerns regarding
robustness. A significant bottleneck lies in processing temporal data in the
correct semantic order. For example, the same set of frames played in reverse
order should mean the output should be reversed for video action recognition
i.e., a person picking up an object and placing down an object would exhibit
identical spatial data, but opposite temporal directionality. While coarse
details suffice for overall action labeling in videos, they may not meet the
requirements of explaining complex models such as the Video Swin Transformer.
To ensure reliable and smooth label explanations across frames, a finer time
domain analysis becomes necessary. Background elements further complicate
Grad-CAM decisions, as seen in Figure 1(b), where the focus shifts to the
tennis court rather than the players. Further, Grad-CAM remains a local
explanation method, implying that the analysis needs to be done individually
for each attribution to draw any class-level conclusions. While this process
is feasible for images, which can be viewed in batches, it becomes impractical
for videos.

Beyond these domain-specific issues, attribution methods exhibit several other
weaknesses. In [1], the authors demonstrated that networks with random weights
generate similar attribution maps as trained networks. Additionally,
attribution methods are vulnerable to adversarial attacks [9], changes in data
preprocessing [12], and even the introduction of random noise [2]. Their
widespread use poses particular concerns in high-risk applications such as
medical imaging [3]. Due to these limitations in feature attribution methods,
interest has grown in more meaningful methods tailored for human
understanding, such as concept attribution, introduced in Testing with Concept
Activation Vectors (TCAV) [11], that we extend to videos in Section 3.

##  3 Video TCAV

###  3.1 TCAV Basics

TCAV offers a popular alternative to feature attribution methods like Grad-
CAM. Unlike granular, pixel-level changes, TCAV focuses on abstract details.
It is widely accepted that the human brain converts raw pixel values into
high-level concepts like texture, specific objects, and their interactions
[7]. Similarly, neural networks encode analogous high-level concepts in the
output of embedding layers. TCAV attempts to derive explanations by analyzing
properties within these embedding spaces.
In TCAV, a concept is delineated by a collection of inputs that characterize
it. For instance in image classification, the concept of _stripedness_ can be
delineated by a set of images portraying striped objects (a in Figure 2). When
mapped to the corresponding feature space, the representations of this
collection would ideally be aligned with a specific _stripedness_ axis (c and
d in Figure 2). On the other hand, a collection of random images would not
display any such alignment. Thus, if we were to find a hyperplane that
separates the embeddings of a set of concept-specific images and a set of
random (or control) images, the normal to that hyperplane should give us a
representation of the concept. This normal may also be considered a basis
vector of the feature space associated with that concept (e.g., a degree of
_stripedness_), and is called the Concept Activation Vector (CAV). Now, given
any input for our specific task, say a picture of a zebra (b in Figure 2), the
directional derivative of its embedding along the CAV would measure the
sensitivity of the embedding to the concept. Figure 2 taken from [11] shows a
graphical representation of the entire process discussed above.

Figure 2: TCAV process in Image Classification. Image taken from [11]. Best
viewed zoomed.

This scalar quantity may be aggregated differently for multiple inputs to get
a more robust value. Moreover, since any learned hyperplane will return a CAV,
even if it does not separate the data well, this process is usually repeated
for multiple control sets to ensure the sensitivity is statistically
significant. Thus, TCAV allows us to quantify the importance of any given
concept to a specific step in the neural networks. It is especially useful for
comparing concepts, which is hard with attribution maps.

###  3.2 Video-TCAV Components

This section discusses the various components of the proposed Video-TCAV
framework.

Video Action Recognition Model: Similar to TCAV, Video-TCAV is a post-training
explanation method, requiring a pre-trained Video Action Recognition model. We
opt for the state-of-the-art Video Swin Transformer model trained on the
Kinetics-400 dataset for quantitative evaluation of post-training
explanations. Figure 3 illustrates the schematic diagram of the Video Swin
Transformer, including the three layers whose activations we utilize in
testing CAVs in Video-TCAV.

Concepts: The most important component of Video-TCAV, similar to TCAV, is the
concepts we want to test. In contrast to image classification, where selecting
concepts is relatively simple, generating concepts for video classification
poses a more intricate challenge. This complexity arises because choosing
clips to represent concepts for Video-TCAV, similar to image crops in the case
of TCAV, is not trivial. We generate two categories of concepts:

  * •

_Spatial Concepts_ : These are simply images of objects repeated temporally to
form videos. This is done as the Video Swin Transformer can only take video
input.

  * •

_Spatiotemporal Concepts_ : These are objects or humans in motion. These are
obtained using YOLO-v7 [16] object detector frame-wise to track a concept of
our choice.

Figure 3: Video Swin Transformer block diagram. The 3 layers with red arrows
whose activations we study while testing CAVs are marked. Best viewed zoomed.

###  3.3 Generating Video-TCAV Concepts

We adopt a machine-assisted method to generate spatial and spatiotemporal
concepts for Video-TCAV. Utilizing YOLO-v7, we detect objects in videos and
produce spatial and spatiotemporal crops for concept testing in Video-TCAV.
Given that the object detector is not flawless, we manually verify all concept
video crops to ensure accuracy in our experiments. Following [11], we present
our visualization and results for a single action class _playing tennis_ for
visualization and results demonstration. Figure 4 illustrates sample
detections generated by YOLO-v7 on a video frame from the _playing tennis
class_. Notably, YOLO-v7 provides accurate detections of a tennis racket,
sports ball, and individuals, which we consider potential concepts aiding in
explainability of videos belonging to _playing tennis_ class.

Generating Spatial Concepts: We execute the YOLO-v7 object detector on all
videos from concept generating split of the test set videos of the
Kinetics-400 dataset labeled as _playing tennis’_. Since YOLO-v7 is trained on
object detection with 80 different object classes, it leads to multiple object
detections in each video frame. For our experiments, we considered all the
per-frame detections from the three classes, i.e., person, tennis racket, and
sports ball, as concepts that we want to test, and all the other detections
are used to generate random concept sets. Figure 5 illustrates our generated
spatial concepts. The generated spatial concepts are temporally repeated to
generate a static video of 3-minute duration.

Generating Spatiotemporal Concepts: Contrary to spatial concepts, generating
spatiotemporal concepts is non-trivial. The basic idea of generating
spatiotemporal concepts is tracking a particular instance of an object across
frames, like the person playing tennis, a tennis racket, or a sports ball
across video frames. This is non-trivial as there might be multiple instances
of the same object class in the frame, and YOLO-v7 results in multiple
detections and it is necessary to choose the same instance of the object in
consecutive frames to prevent the concept video from changing in terms of
content. We ensured only one instance of the object was tracked across frames,
and we took a spatiotemporal crop based on the detections returned by YOLO-v7.
Since the size of the detections of a particular object changes across frames,
we generate the concept video with the largest-sized object detected in
consecutive frames and pad the other smaller-sized frames, positioning them at
the center. Figure 6 illustrates our generated spatiotemporal concepts.

Figure 4: YOLO-v7 detections on a frame of a video from the _playing tennis_
class. Best viewed zoomed. Figure 5: Exemplars of Spatial Concepts: person
playing tennis, tennis racket, and sports ball generated from Kinetics-400
dataset.  Figure 6: Exemplars of Spatiotemporal Concepts: movement of tennis
racket in a match and person playing tennis. We show some sampled frames
across each concept video.

###  3.4 Video-TCAV Experiments

Experimental Setup: We replicate the experiments of [11] to demonstrate the
advantages of our pipeline. We collect 30 videos that have the ground truth
label as _playing tennis_ and generate 25-30 videos per concept. We use these
videos and concepts in our Video-TCAV framework. For random concepts, we
randomly select 30 videos from our entire corpus of concepts, including those
not related to playing tennis e.g. dining and dancing. All experiments use the
relative sign-count variant of TCAV. In this, we consider a group of multiple
concepts at a time, and learn a one-vs-rest classifier for each concept, which
gives us the CAV for that concept. The fraction of data points positively
affected by the CAV is the relative TCAV for that concept. Using relative
TCAVs helps disentangle the effect of correlation between different concepts.
A more detailed discussion may be found in [11].

###  3.5 Results & Discussion

Spatial Concepts: The generated spatial concepts of static frames with no
movement information serve as our baseline. Figure 7 shows the relative TCAVs
of the tested static concepts in different layers. While all concepts show
some relative importance, it is hard to distinguish their importance from a
random concept, especially in the initial layers. This is in line with our
expectations, as static concepts should provide some information the context
of the video (e.g. the presence of a tennis racket indicates tennis being
played) but should not be able to pinpoint the exact action performed. Another
aspect to note is that the importance of the random concept persists until the
last layer, which validates the static features are not fully discriminative
for the action recognition task.

Figure 7: Relative TCAVs for static concepts.

Spatio-Temporal Concepts: The dynamic spatio-temporal concepts, on the other
hand, have more predictive power than the static concepts. Figure 8 shows the
relative TCAVs of the tested dynamic concepts in different layers. In the
initial layers, the importance of the dynamic concepts is more distinguishable
from random, but is overall similar to static case, which is in line with the
understanding that initial layers extract general-purpose features. However,
in later layers, the importance of temporal concepts is dramatically more
pronounced. In contrast with the static concepts, the random concept holds
negligible importance in the last layer. Interestingly, this increase of
importance of temporal information with depth correlates well with how the
human brain perceives motion, with visual features first processed by the V1
area, then motion being handled later in the V5/MT area [17].

Figure 8: Relative TCAVs for dynamic concepts

Spatial vs Spatio-Temporal Concepts: Next, we directly compare the spatial and
spatio-temporal versions of a specific concept. Figure 9 shows their relative
TCAVs in different layers. Evidently, when compared side-by-side, the temporal
concepts dominate over static concepts in most layers of the network, with
this effect becoming more pronounced with depth. It is particularly
interesting to note how the final layer exclusively prefers the temporal
concept for prediction compared to the static version of the concept.

Figure 9: Relative TCAVs for static vs dynamic concepts

Statistical Testing: We tested the validity of the scores with 10 other random
sets and applied a two-sided t-test with Bonferroni correction, as in [11].
The results are in the Figure 10. The $p<0.05$ for our concepts implies that
the CAVs we found are robust and statistically significant, where as the CAVs
learnt by separating random concepts are not meaningful ($p>0.05$).

Figure 10: Results of hypothesis testing with spatiotemporal concepts. Best
viewed zoomed.

##  4 Conclusion and Future Work

In this work, we proposed Video-TCAV, a new approach for generating human-
friendly concepts and quantitatively measuring their impact on the decision
process of deep networks used for action recognition. Further, we provide
preliminary results on the Video Swin Transformer to demonstrate the success
and pitfalls of our method. However, more experiments using other video action
recognition models would strengthen the effectiveness of Video-TCAV.
Additionally, Video-TCAVs may also share some of the weaknesses of attribution
methods, such as [5], which demonstrates adversarial attacks on TCAV that can
artificially raise or lower the importance of a particular concept, leading to
bizarre conclusions. Generating concepts for Video-TCAV is challenging. In
this work, we proposed a method using YOLO-v7 object detector. An alternative
research direction could involve utilizing text-to-video diffusion models, as
in [4], to generate concept clips based on human-interpretable text prompts.
We hope that this work serves as a starting direction for developing scalable
explainable methods for Video Action Recognition.

## References

  * Adebayo et al. [2018] Julius Adebayo, Justin Gilmer, Ian Goodfellow, and Been Kim.  Local explanation methods for deep neural networks lack sensitivity to parameter values.  _arXiv preprint arXiv:1810.03307_ , 2018.
  * Alvarez-Melis and Jaakkola [2018] David Alvarez-Melis and Tommi S. Jaakkola.  On the robustness of interpretability methods.  _CoRR_ , abs/1806.08049, 2018.
  * Arun et al. [2021] Nishanth Arun, Nathan Gaw, Praveer Singh, Ken Chang, Mehak Aggarwal, Bryan Chen, Katharina Hoebel, Sharut Gupta, Jay Patel, Mishka Gidwani, et al.  Assessing the trustworthiness of saliency maps for localizing abnormalities in medical imaging.  _Radiology: Artificial Intelligence_ , 3(6), 2021.
  * Blattmann et al. [2023] Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, Varun Jampani, and Robin Rombach.  Stable video diffusion: Scaling latent video diffusion models to large datasets, 2023.
  * Brown and Kvinge [2021] Davis Brown and Henry Kvinge.  Brittle interpretations: The vulnerability of TCAV and other concept-based explainability tools to adversarial attack.  _CoRR_ , abs/2110.07120, 2021.
  * Carreira and Zisserman [2017] Joao Carreira and Andrew Zisserman.  Quo vadis, action recognition? a new model and the kinetics dataset.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2017.
  * Epstein and Baker [2019] Russell A. Epstein and Chris I. Baker.  Scene perception in the human brain.  _Annual Review of Vision Science_ , 5(1):373–397, 2019.
  * Feichtenhofer et al. [2019] Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He.  Slowfast networks for video recognition.  In _Proceedings of the IEEE/CVF international conference on computer vision_ , pages 6202–6211, 2019.
  * Ghorbani et al. [2019] Amirata Ghorbani, Abubakar Abid, and James Zou.  Interpretation of neural networks is fragile.  In _Proceedings of the AAAI conference on artificial intelligence_ , pages 3681–3688, 2019.
  * Kay et al. [2017] Will Kay, João Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijayanarasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, Mustafa Suleyman, and Andrew Zisserman.  The kinetics human action video dataset.  _CoRR_ , abs/1705.06950, 2017.
  * Kim et al. [2018] Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, et al.  Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav).  In _International conference on machine learning_ , pages 2668–2677. PMLR, 2018.
  * Kindermans et al. [2019] Pieter-Jan Kindermans, Sara Hooker, Julius Adebayo, Maximilian Alber, Kristof T Schütt, Sven Dähne, Dumitru Erhan, and Been Kim.  The (un) reliability of saliency methods.  In _Explainable AI: Interpreting, Explaining and Visualizing Deep Learning_ , pages 267–280. Springer, 2019.
  * Liu et al. [2021] Ze Liu, Jia Ning, Yue Cao, Yixuan Wei, Zheng Zhang, Stephen Lin, and Han Hu.  Video swin transformer, 2021.
  * Selvaraju et al. [2017] Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra.  Grad-cam: Visual explanations from deep networks via gradient-based localization.  In _Proceedings of the IEEE international conference on computer vision_ , pages 618–626, 2017.
  * Sundararajan et al. [2017] Mukund Sundararajan, Ankur Taly, and Qiqi Yan.  Axiomatic attribution for deep networks.  In _International conference on machine learning_ , pages 3319–3328. PMLR, 2017.
  * Wang et al. [2022] Chien-Yao Wang, Alexey Bochkovskiy, and Hong-Yuan Mark Liao.  Yolov7: Trainable bag-of-freebies sets new state-of-the-art for real-time object detectors, 2022.
  * Watson et al. [1993] J. D. G. Watson, R. Myers, R. S. J. Frackowiak, J. V. Hajnal, R. P. Woods, J. C. Mazziotta, S. Shipp, and S. Zeki.  Area V5 of the Human Brain: Evidence from a Combined Study Using Positron Emission Tomography and Magnetic Resonance Imaging.  _Cerebral Cortex_ , 3(2):79–94, 1993.
  * Zhou et al. [2016] Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba.  Learning deep features for discriminative localization.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 2921–2929, 2016.