#  SoccerNet Game State Reconstruction:
End-to-End Athlete Tracking and Identification on a Minimap

Vladimir Somers1,3,8* Victor Joos1* Anthony Cioppa2,4* Silvio Giancola4 Seyed
Abolfazl Ghasemzadeh1 Floriane Magera2,7 Baptiste Standaert1 Amir M.
Mansourian5 Xin Zhou6 Shohreh Kasaei5 Bernard Ghanem4 Alexandre Alahi3 Marc
Van Droogenbroeck2 Christophe De Vleeschouwer1 1 UCLouvain 2 University of
Liège 3 EPFL 4 KAUST 5 SUT 6 Baidu Research 7 EVS 8 Sportradar

###### Abstract

Tracking and identifying athletes on the pitch holds a central role in
collecting essential insights from the game, such as estimating the total
distance covered by players or understanding team tactics. This tracking and
identification process is crucial for reconstructing the _game state_ ,
defined by the athletes’ positions and identities on a 2D top-view of the
pitch, (_i.e_. a _minimap_). However, reconstructing the game state from
videos captured by a single camera is challenging. It requires understanding
the position of the athletes and the viewpoint of the camera to localize and
identify players within the field. In this work, we formalize the task of Game
State Reconstruction and introduce _SoccerNet-GSR_ , a novel Game State
Reconstruction dataset focusing on football videos. _SoccerNet-GSR_ is
composed of $200$ video sequences of $30$ seconds, annotated with $9.37$
million line points for pitch localization and camera calibration, as well as
over $2.36$ million athlete positions on the pitch with their respective role,
team, and jersey number. Furthermore, we introduce _GS-HOTA_ , a novel metric
to evaluate game state reconstruction methods. Finally, we propose and release
an end-to-end baseline for game state reconstruction, bootstrapping the
research on this task. Our experiments show that GSR is a challenging novel
task, which opens the field for future research. Our dataset and codebase are
publicly available at https://github.com/SoccerNet/sn-gamestate. ††(*) Equal
contributions. Data/code available at www.soccer-net.org.

##  1 Introduction

Recently, sports companies and teams have shown a growing interest in
collecting athlete-centric data. One key focus area lies in tracking and
identifying athletes on the sports field throughout the entire game, using
available video footage. These analytics hold immense value for a broad
spectrum of sports applications, ranging from (i) supporting team coaching and
athlete training, (ii) assisting scouters in discovering new talents, (iii)
offering valuable insights for medical staff, and (iv) boosting fan engagement
through personalized content creation [12, 13, 28].

However, the manual generation of such data by human annotators is time-
consuming and costly. Sensor-based solutions offer a time-efficient
alternative, but require athletes to wear special, sometimes expensive,
equipment. Recently, automatic solutions based on optical tracking systems
have gained prominence. These systems necessitate the installation of
sophisticated, well-calibrated static multi-camera setups in stadiums. Hence,
they come with significant drawbacks in terms of cost and scalability, which
restricts their use to elite competitions, exemplified by their deployment at
events like the 2022 Qatar World Cup.

Figure 1:  SoccerNet-GSR. We introduce a novel Game State Reconstruction task,
dataset, evaluation metric and baseline. Our SoccerNet-GSR dataset contains
unique identifications for players along with their localization on the pitch,
for $200$ video sequences.

Meanwhile, recent progress in computer vision opened up a growing potential to
automatically and reliably extract athlete localization and identification
data solely from broadcast camera feeds. In line with this objective, Multi-
Object Tracking (MOT) methods have long been popular for sports video
analysis. However, they offer only a partial solution to the aforementioned
requirements. Indeed, the bounding-box-based tracking data produced by MOT (1)
lacks critical identification information necessary to analyze specific
athletes and (2) lacks interpretability due to the absence of grounding in a
real-world coordinate system. These significant limitations hinder the
usability of such tracking data for many downstream sports applications.

To address the above limitations, we introduce the concept of Game State
Reconstruction (GSR), a novel computer vision task tailored for sports
analytics. GSR aims to recognize the state of a sports game by identifying and
tracking all athletes on the pitch based on input videos captured by a single
camera. Moreover, game state data can be visualized in a minimap of the game,
as depicted in Fig. 1, offering a concise representation of the ongoing
gameplay dynamics. To support research on this task, we publicly release
_SoccerNet-GSR_ , the first dataset for Game State Reconstruction, consisting
of 200 30-second fully annotated clips. Our proposed GSR annotations include
over 9.37 million line points for football pitch registration, and over 2.36
million athlete positions on the pitch with unique identification information,
including their role, team, and jersey number. Since existing metrics for
Multi-Object Tracking [5, 57] are not suited for our proposed task, we
introduce the _GS-HOTA_ , a new evaluation metric to benchmark GSR methods.
Finally, we propose _GSR-Baseline_ , the first end-to-end and open-source
pipeline for game state reconstruction, built upon state-of-the-art tracking,
re-identification, team affiliation, jersey number recognition, pitch
localization, and camera calibration methods. Our analysis underscores the
complexity of Game State Reconstruction and highlights the importance of
introducing this new benchmark. This initiative establishes an ideal platform
for future research in the field, aiming to democratize access to this
valuable game state data for all leagues.

Contributions. We summarize our contributions as follows. (i) We introduce and
concretely define the concept of _Game State Reconstruction_ , a task aiming
to track and identify all athletes on a minimap of the pitch. (ii) We publicly
release SoccerNet-GSR, the first open-source sports video dataset for Game
State Reconstruction. (iii) We introduce GS-HOTA, a new metric to evaluate
game state reconstruction methods. (iv) We propose GSR-Baseline, the first
end-to-end GSR pipeline for football videos.

##  2 Related Work

Game State Reconstruction relates to the general topic of sports video
understanding and, more particularly, to tracking, identification, and sports
field registration.

Sports Video Understanding. Sports video understanding has emerged as a
prominent research topic over the past decade [64, 86, 66, 37]. Some works
focused on low-level semantics, aiming to build a bottom-up understanding of
the game [15], such as segmenting [16] or detecting [88, 77, 73, 69] players
and keypoints [56, 31]. Recent advances in computer vision allowed for a
higher semantic understanding of the game, focusing for instance on the action
spotting task, aiming to spot a series of events during the game [17, 33, 100,
101, 80, 10, 35, 91, 38, 51, 59]. Fortunately, these works can rely on the
availability of large-scale datasets [68, 95, 76, 43, 24, 20, 21, 63, 36] and
challenges [22, 34, 42, 87, 62, 47]. Our novel Game State Reconstruction task
stands in between low- and high-level semantics, providing both local
information about the players but also global information about the whole
state of the game through time. This information can later be used to better
understand player actions [19, 9], enhance the generation of engaging captions
[11, 63], improve visualizations [74, 29, 102, 8], or derive high-level
analytics [70, 23, 67, 3, 50, 1]. In this work, we complement the literature
in sports video understanding by proposing a novel task of Game State
Reconstruction that aggregates several tasks ranging from field to player
understanding.

Player Tracking and (Re-)Identification. Multiple Object Tracking (MOT) has
often been approached through the tracking-by-detection paradigm [6, 90, 7, 4,
83, 99, 98, 85]. However, applying the tracking-by-detection paradigm to
sports introduces unique challenges compared to generic scenarios. Previous
works [18, 88, 21, 93, 79, 39, 72] tackled the challenges of similar
appearances and fast motion of people and object in sports. Furthermore,
unlike generic MOT scenarios, athletes come in and out of the camera view,
requiring long-term Re-Identification (ReID) [97, 32, 48, 96, 61]. Finally,
uniquely identifying actors in a sports scene has been widely investigated in
the literature. Some approaches focused on athletes’ role (_e.g_., player,
referee, coach, _etc_.) [20, 88, 61], their team [41, 61], or jersey numbers
[94, 30, 54, 65, 89, 2, 55]. Different from previous works, our new game state
reconstruction task combines athlete tracking and identification, including
the role, team, and jersey number under a single task.

Sports Field Registration. Mapping the video tracking data into a real-world
coordinate system requires camera calibration. Sports games come naturally
with a coordinate system based on the sports pitch. Hence, combining the
location of the field [93, 75] with video camera calibration [27, 71, 78, 84],
one can reconstruct a game state as illustrated in Fig. 1. Unifying tracking
and camera calibration as proposed in this paper has been investigated in
previous works.[82, 76, 46, 19] Scott  _et al_. [76] collected data from fish-
eye camera, drone, and GNSS, while Karungaru  _et al_. [46] focused on the
mapping of players onto the field in one video frame. Cioppa  _et al_. [19]
leveraged tracking and camera calibration to reproject players’ positions on
the pitch for the task of action spotting. Maglo  _et al_. [60] introduced a
robust player tracking method, incorporating test-time fine-tuning and a novel
football field registration technique, which were combined to explore player
localization on a minimap. However, due to the lack of annotations, they did
not perform either player identification or quantitative evaluations of their
localization results. Finally, Theiner  _et al_. [85] introduced a pipeline to
localize players on a pitch minimap from broadcast videos but omitted player
identification and tracking. Different from previous work, our proposed GSR
benchmark addresses the combined athlete pitch localization and identification
task.

##  3 Game State Reconstruction Task

Game State Reconstruction (GSR) is a form of video compression task aiming to
extract high-level information about the dynamics of a sports game from an
input video. It includes (1) the 2D position of all athletes on the sports
pitch, (2) their roles in the game (e.g., “player”, “goalkeeper”, or
“referee”), and (3), for players, their jersey number and team affiliation.
This information can be visualized on a 2D top-view of the pitch, or minimap,
as illustrated in Fig. 1. In the following, we refer to all individuals to be
identified and localized, irrespective of their specific roles, as “athletes”.
GSR is a multifaceted task that requires addressing various intricate sub-
tasks, including: (a) pitch localization and camera calibration, (b) athlete
detection, re-identification, and tracking, and (c) role classification, team
affiliation, and jersey number recognition.

We formalize the Game State Reconstruction task as follows. Given a team
sports video composed of $T$ frames, the objective is to predict a set of
detections $d_{i}^{t}$ for each frame $t$, where $i$ indexes the detections
within frame $t$. A detection encapsulates each athlete’s location on the
pitch ($pitch\\_x$, $pitch\\_y$) in a real-world coordinate system, and their
role, team, and jersey number. A detection is therefore represented as
follows:

$d_{i}^{t}=\\{\underbrace{\textit{pitch\\_x},\textit{pitch\\_y}}_{\text{localization}},\underbrace{\textit{role},\textit{team},\textit{jersey\\_number}}_{\text{identification}}\\}.$
(1)

While our main focus is football, the definition of the GSR task can extend to
other team sports.

##  4 SoccerNet-GSR Dataset

Our dataset expands upon SoccerNet-Tracking [21], which consists of $200$
$30$-second clips split into train, validation, test, and a segregated
challenge set. In the original dataset, each frame includes bounding box
annotations for the localization of players, referees, and balls tracked over
time with extra role, team, and jersey number attributes. Despite the
comprehensive annotations, SoccerNet-Tracking lacks information like pitch
localization, camera calibration111A camera calibration and pitch localization
dataset was already introduced for the SoccerNet Camera Calibration challenge,
but the corresponding annotations were provided on a separate set of data.,
and athlete positions on the pitch, critical for the Game State Reconstruction
task. In subsequent sections, we detail how we augmented the SoccerNet-
Tracking annotations to create our proposed SoccerNet-GSR dataset. The new
annotations now include over $9.37$ million line points for pitch localization
and camera calibration, as well as over $2.36$ million athlete positions on
the pitch with their respective role, team, and jersey number. Since the
SoccerNet-GSR videos are uncut broadcast sequences captured by a single moving
camera, only a portion of the football pitch is visible at any given time. As
a result, the GSR task is limited to players within the camera’s field of
view.

###  4.1 Athlete Localization on the Pitch

Expressing the 2D image location of athletes in the real-world pitch
coordinates requires pitch localization and camera calibration. Together,
these information enable precise mapping of player positions from the image
onto the pitch. Our new annotations described in this section therefore
include: (1) pitch 2D positions, (2) camera parameters, and (3) positions on
the pitch.

Pitch localization. Following the same procedure as SoccerNet-v3 [20], we
manually annotate every line on the football pitch by placing a series of
points along its length to accurately define its shape, including curves such
as the circles or the ones due to camera distortions. We categorize each line
(_e.g_., side line left, side line top, _etc_.) and part of the goals, (left
and right posts and the crossbar), totaling $26$ classes. Next, we
continuously track all these annotations over time using key frame annotations
and interpolations in-between when it is appropriate, mirroring the player
tracking data as described in [21], resulting in a densely marked pitch
annotation throughout the entire video. This annotation process is core for
calibrating the camera through time.

Camera calibration. Camera calibration is the process of determining the
camera parameters for each frame, allowing to link the image-plane to the 3D
world. It is required to compensate for the lack of a pre-calibrated camera.
Usually, this process requires correspondences between a known 3D object and
its image. In the context of football, the pitch is a convenient object [40]
to obtain correspondences from. In this work, we assume that the pitch has a
conventional size of $105$ by $68$ meters. However, as the pitch is only
partially visible in the images, the calibration of broadcast cameras is a
challenging task. Hence, due to the lack of visible lines, some frames may not
be calibrated correctly and are discarded in the evaluation. For the frames
presenting a sufficient amount of pitch line annotations, the camera
parameters are obtained from the best of several open-source techniques [14,
58] or an industrial tool [26]. The complete process is described in the
supplementary materials.

Position on the pitch. The point of calibrating the cameras is to derive
positions in the real-world. Our 3D world reference axis system is centered on
the pitch center mark, the X-axis points to the right goal, the Y-axis follows
the middle line towards the camera and the Z-axis is perpendicular to the XY –
or the pitch – plane, pointing towards earth’s center. Once the camera
parameters are known, the inverse of the camera projection function applied to
a point gives a 3D ray that can be intersected with the pitch plane to derive
the 3D position. We assume that the athlete’s feet, and more specifically the
center of the bottom part of their detection bounding boxes lies on the pitch.
Unfortunately, this approximation limits the precision of the estimated
locations in the case of jumps. Hence, we remove the ball as it spends
significant time in the air. A precise 3D localization of all elements would
require tracking hardware, which is unavailable for open-science research at
the moment.

###  4.2 Athlete Identification

To identify athletes during a game, we leverage three distinct manual
annotations provided in the SoccerNet-tracking dataset that have been
previously overlooked in standard multi-object tracking: role, team, and
jersey number. The following paragraphs detail each annotation and the
utilization of an additional track id for cases where targets cannot be
uniquely identified by their attributes.

Role. In the SoccerNet-GSR dataset, athletes are categorized into four
distinct roles during the game: ’player’, ’goalkeeper’, ’referee’, or ’other’.
The ’other’ role encompasses individuals entering the pitch, such as coaches,
medical staff, and any additional person not falling into the previous three
categories. For the first version of the SoccerNet-GSR benchmark, both referee
responsibilities (i.e. main, bottom/top assistants) and ball detections are
ignored.

Team. Detections with the ’player’ and ’goalkeeper’ roles are annotated with a
’team’ attribute, which can be assigned one of two values: ’left’ or ’right’.
Since the dataset consists of $30$-second sequences captured from a single
camera, we determine the ’left’ and ’right’ teams based on their goal’s
position relative to the camera viewpoint.

Jersey Number. Players and goalkeepers in the SoccerNet-GSR dataset are
annotated with an additional ’jersey number’ attribute. However, unlike the
role and team attributes, which are always available, a jersey number may not
be visible at any point during the entire $30$-second video sequence. In such
cases, players with invisible shirt numbers are assigned a ’null’ value for
this attribute. If a player’s jersey number is visible in at least one frame
of the sequence, then the entire tracklet is annotated with that jersey
number. Therefore, a jersey number assigned to a detection does not
necessarily mean that it is visible in that particular frame.

Track Id. We utilize the combination of role, team, and jersey number
attributes to identify each athlete during a game. However, athletes cannot
always be uniquely identified by their attributes. This occurs, for example,
when two players from the same team do not have visible jersey numbers or when
multiple individuals with the role of ’referee’ or ’other’ appear
simultaneously. Although these cases represent a small proportion of all
annotated athletes, they prevent unique identification using attributes alone.
To address this, we also include the standard ’track id’ annotation from
standard MOT. This requires methods for the SoccerNet-GSR task to output four
values per detection: role, team, jersey number, and track id. The impact of
non-uniquely identifiable targets is further discussed in Sec. 5.

##  5 GS-HOTA Evaluation Metric

Game State Reconstruction (GSR) is a novel computer vision task closely
related to multi-object tracking (MOT). Yet, standard evaluation metrics for
MOT, such as MOTA [5] and HOTA [57], cannot be used to evaluate GSR for two
main reasons. First, these metrics do not account for additional attributes
predicted on the tracked targets, such as team, role, and jersey numbers.
Second, these metrics rely on an IoU score to match predicted and ground truth
bounding boxes in the image space, while GSR operates on 2D points within the
pitch coordinate system.

To address these issues, we introduce _GS-HOTA_ , a novel evaluation metric to
measure the ability of a GSR method to correctly track and identify all
athletes on the sports pitch. GS-HOTA is derived from the HOTA [57] metric,
which is formulated as follows:

$\text{HOTA}=\int\limits_{\begin{subarray}{c}0<\alpha\leq
1\end{subarray}}\sqrt{\text{DetA}_{\alpha}\cdot\text{AssA}_{\alpha}}$ (2)

DetA/AssA are the detection/association accuracy respectively, and $\alpha$ is
a similarity threshold. To compute these two underlying accuracy metrics,
ground truth and predicted detections must first be matched according to a
similarity score. Pairs with a similarity score below the $\alpha$ threshold
are not matched. For predictions (P) and ground truth (G) represented as
bounding boxes in the image space, the Intersection-over-Union (IoU) is
employed as the similarity score for the HOTA metric. The key distinction
setting GS-HOTA apart from HOTA is the use of a new similarity score, that
accounts for the specificities of the GSR task, i.e. the additional target
attributes (jersey number, role, team) and the detections provided as 2D
points instead of bounding boxes. This new similarity score, denoted $Sim_{GS-
HOTA}(P,G)$, is formulated as follows:

$Sim_{\text{GS-HOTA}}(P,G)=\text{LocSim}(P,G)\times\text{IdSim}(P,G),\ $ (3)

$\text{with LocSim}(P,G)=e^{\ln(0.05)\frac{\|P-G\|_{2}^{2}}{\tau^{2}}}\ ,$ (4)

$\text{and IdSim}(P,G)=\begin{cases}1&\text{if all attributes match,}\\\
0&\text{otherwise.}\end{cases}$ (5)

$Sim_{\text{GS-HOTA}}$, is therefore a combination of two similarity metrics.
The first metric, the localization similarity $\text{LocSim}(P,G)$, computes
the Euclidean distance $\|P-G\|_{2}$ between prediction P and ground truth G
in the pitch coordinate system. This distance is subsequently processed using
a Gaussian kernel with a special distance tolerance parameter $\tau$,
resulting in a final score falling within the $[0,1]$ range. The second
metric, the identification similarity $\text{IdSim}(P,G)$, is set to one only
if all attributes match, i.e. role, team, and jersey numbers. Attributes not
provided in G are ignored, e.g. jersey numbers for referees222GSR methods must
ignore the team and jersey number for non-player roles, as well as the jersey
number when it is not visible in the video.. Finally, once P and G are
matched, DetA and AssA are computed and integrated into a final GS-HOTA score,
following the original formulation of the HOTA metric in Eq. 2.

###  5.1 GS-HOTA Distance Tolerance Parameter

Our GS-HOTA metric relies on a single $\tau$ parameter introduced in Eq. 4. In
practice, the continuous integral in Eq. 1 is computed over a discrete
interval $\alpha\in[0.05,0.95]$ with $0.05$ steps. This means that (P, G)
pairs with a similarity below or equal to $0.05$ are never matched. Hence, our
distance tolerance parameter $\tau$ defines the maximum distance in meters for
a prediction P and a ground truth G to be matched, as illustrated in Fig. 2.
Furthermore, since all similarity thresholds in the range $[0.05,0.95]$ are
considered, a distance smaller than $\tau$ meters between P and G still
results in a higher GS-HOTA. This way, methods are still incentivized to
produce athlete localization closer to the ground truth. In this work, we
define $\tau$ as $5$ meters, considering it a reasonable distance tolerance
given the average dimensions of a soccer pitch ($68\times 105$ meters) and the
substantial distance between the camera and the athletes.

Figure 2:  The localization similarity function for $\tau=5$ meters.

###  5.2 Motivation and Discussion

As introduced in Sec. 4.2, we consider the combination of attributes (role,
team, jersey) as a way to identify athletes. If each person was uniquely
identifiable by the combination of its attributes, association would become
trivial, and as a consequence, a simple average of the Detection Accuracy
across all identities would suffice as a robust performance metric. However,
as explained in Sec. 4.2, not all identities in our SoccerNet GSR dataset can
be uniquely identified by their attributes. Therefore, the Association
Accuracy must also be taken into account to account for identity switches
among athletes sharing the same attributes (e.g. players from the same team
with no visible jersey number).

Finally, a key difference that sets GSR apart from MOT — and by extension, GS-
HOTA from HOTA — is the necessity to identify athletes by their attributes.
This requirement is specified by Eq. 5, according to which failing to
correctly predict at least one attribute turns the corresponding detection
into a False Positive. Requiring the correct prediction of all attributes
simultaneously is a strict constraint, which we justify based on the severe
impact that incorrectly assigning localization data to a nonexistent or
incorrect identity can have on downstream applications.

Figure 3:  Architecture overview of our proposed baseline. GSR-Baseline takes
a video as input and outputs the complete game state. Two modules are first
applied on the input images: an object detector and a pitch localization
model. Then, PRTreID [61] produces a ReID embedding for each detection, that
is identity, team, and role aware. These embeddings are then forwarded to
subsequent modules to perform role classification, team affiliation, and
multi-object tracking. Finally, the pitch localization output is used for
camera calibration, which enables the tracked bounding boxes to be transformed
into 2D positions on the pitch coordinate system.

##  6 GSR Baseline

In this section, we introduce the GSR-Baseline, a pipeline designed to
reconstruct the game state of any broadcast football video. Our baseline
splits the Game State Reconstruction task into several sub-tasks, selecting
popular and open-source state-of-the-art methods for each sub-task. To
facilitate the development of such a complex video processing pipeline, we
leverage TrackLab [45], a research-oriented PyTorch-based framework for multi-
object tracking. The overall architecture of the GSR-Baseline is depicted in
Fig. 3, and a detailed description of each of the pipeline modules is provided
hereafter.

###  6.1 Athlete Detection and Tracking

We employ a pre-trained YOLOv8 [44] model as our athlete detector, without
fine-tuning it on the SoccerNet dataset, since it already provides decent
performance on football videos. We filter the model’s output to retain only
the ”person” class detections. To leverage existing strong multi-object
trackers, our GSR-Baseline performs tracking in the image space based on
bounding boxes. As illustrated in Fig. 3, these bounding boxes are converted
into 2D pitch positions later within the pipeline. Next, we employ StrongSORT
[25] as our multi-object tracker, for its SOTA performance and its ability to
leverage both spatio-temporal and appearance cues, the latter being provided
by the re-identification model PRTreID [61] described in Sec. 6.3.

###  6.2 Pitch Localization and Camera Calibration

Camera calibration is performed using TVCalib [84], which is composed of two
modules. The first module performs pitch localization through semantic
segmentation. The second estimates the camera calibration parameters by
iteratively minimizing the pitch segments reprojection errors. Once the camera
has been calibrated, its corresponding homography is used to transform image
bounding boxes into 2D positions on the pitch. For this purpose, we assume
that the bottom of the bounding box lies on the ground field.

###  6.3 Athlete Identification

Athlete identification is performed by two key models: PRTreID [61] to produce
team and role-aware ReID embeddings, and MMOCR for jersey number recognition.
The output of these two models is further processed for tracklet consistency,
team affiliation, and role classification, to produce the final game state
identification data.

Re-Identification. The sportsperson representation model PRTreID [61] is
designed to jointly solve person re-identification, role classification, and
team affiliation with a single backbone. Therefore, it produces an embedding
that is team, role, and identity discriminative, thanks to a multi-task
learning setup with three learning objectives. PRTreID builds upon the SOTA
part-based ReID method BPBreID [81]. During the PRTreID training procedure,
re-identification and team affiliation are formulated as deep metric learning
tasks, where persons with the same identity/team are pulled close to each
other in the embedding space with a triplet loss. Role prediction is framed as
a classification task with four target classes, employing a focal loss to
address class imbalance. At inference in the GSR-Baseline pipeline, PRTreID
produces an embedding for each input detection, that is forwarded to
subsequent modules to perform tracking, team clustering with left/right
labeling, and role classification.

Role Classification. The embeddings described above are processed by the
PRTreID classification layer to output the target’s role: player, goalkeeper,
referee, or other.

Jersey Number Recognition. Jersey numbers recognition is performed in two
separate steps with the open-source optical character recognition library
MMOCR [49]. First, the YOLOv8 detections are fed to the DBNet [53] text
detection model. Subsequently, the detected texts are forwarded to the SAR
[52] text recognition model. Finally, the highest-scored detected text
containing a number is considered as the player’s jersey number.

Tracklet Consistency. As described, jersey numbers and roles are predicted
independently for each detection, potentially leading to inconsistencies
within tracklets. We adopt a majority voting approach within each tracklet to
select the most common role and jersey number, ensuring uniformity.

Team Affiliation. Team affiliation is performed in three steps for tracklets
having the ”player” role assigned. First, the PRTReID embeddings of all
detections within each tracklet are averaged to create a single tracklet-level
representation of the player. Next, these tracklet-level embeddings are
separated by a K-means clustering algorithm into two clusters representing two
teams. Finally, the average 2D positions of each team on the pitch are
compared to determine which team is positioned more to the left or right.

##  7 Experiments

###  7.1 Implementation details

To provide a baseline that is generic, we employ mostly pretrained networks
that were not finetuned on SoccerNet. The only exceptions are PRTReid [61] and
TVCalib [84]. We use the standard weights provided by TVCalib’s authors in our
baseline. Finally, PRTReid [61] is trained on the SoccerNet-GSR train set
using parameters from the original paper. For more implementation details, we
invite readers to visit our project’s GitHub repository and
Tracklab333https://github.com/TrackingLaboratory/tracklab.

###  7.2 Evaluation

To evaluate the performance of our proposed method on the Game State
Reconstruction task, we employ the GS-HOTA metric introduced in Sec. 5. In the
supplementary materials, we evaluate the performance of our GSR-Baseline in
the image plane on the standard Multi-Object Tracking (MOT) task. Unless
specified otherwise, all experiments are performed on the SoccerNet-GSR test
set.

###  7.3 Results and Analysis

Main Results and GS-HOTA Analysis. We report the performances of our GSR-
Baseline in Tab. 1, which achieves $22.26\%$ in GS-HOTA on the test set. All
experiments in this table correspond to slight variations of the
$Sim_{\text{GS-HOTA}}(P,G)$ introduced in Eq. 3. First, when “Pitch” is
disabled, the LocSim function in Eq. 4 is replaced with the bounding boxes IoU
in the image space: pitch localization and camera calibration have therefore
no impact. Second, we ablate each attribute of the identification component in
Eq. 5 (IdSim is set to 1 when all attributes are disabled). The first
experiment in Tab. 1 falls back to the standard HOTA, _i.e_. with the IOU in
image space as a similarity function. The remaining experiments illustrate how
enabling attributes in Eq. 5 induces successive drops in performance, since it
introduces additional predictions in the evaluation and therefore potential
errors. Tab. 1 also highlights the key challenges of this task, showing that
our GSR-Baseline struggles mostly with jersey number recognition, followed by
pitch localization, team affiliation, and finally role classification.
Finally, the influence of the GS-HOTA distance tolerance parameter $\tau$
introduced in Sec. 5 is illustrated in Fig. 4. According to this plot, picking
$\tau=5$ meters is a reasonable choice since performance quickly drops with a
stricter tolerance.

Table 1:  Main Results and GS-HOTA Analysis. Attributes (Role, Team, Jersey)
are ignored in the GS-HOTA computation when disabled. IoU in image space is
used when Pitch is disabled.

Split | GS-HOTA components | GS-HOTA $\uparrow$
---|---|---
Pitch | Role | Team | Jersey
Test | ✗ | ✗ | ✗ | ✗ | 57.64
✓ | ✗ | ✗ | ✗ | 42.65
✓ | ✓ | ✗ | ✗ | 40.76
✓ | ✗ | ✓ | ✗ | 37.03
✓ | ✗ | ✗ | ✓ | 25.65
✗ | ✓ | ✓ | ✓ | 29.50
✓ | ✓ | ✓ | ✓ | 22.26
Valid | ✓ | ✓ | ✓ | ✓ | 18.05
Challenge | ✓ | ✓ | ✓ | ✓ | 23.36

Figure 4:  Distance Tolerance Parameter $\tau$: its influence on the GS-HOTA
score. We pick $\tau$=5, illustrated by the orange line.

Ablation Study of GSR-Baseline Modules. Table 2 illustrates the impact of each
module on the overall performance. This study employs the ground truth as an
oracle for all modules except the module of interest and its downstream
modules in the pipeline. For instance, when examining the ReID module, the
tracking, role classification, and team clustering modules are also activated.
Dependencies between modules are depicted as a flowchart in Fig. 3. The first
experiment (Exp. 1) shows that the heuristic chosen for team ’left’/’right’
affiliation is highly effective, especially considering the significant impact
that swapping two teams can have on GS-HOTA. Similarly, Exp. 2 demonstrates
the solid performance of all modules depending on the ReID embeddings (_i.e_.
tracking, role cls, and team aff.). Furthermore, Exp. 3 and 4 show the severe
performance impact of enabling calibration and pitch localization, suggesting
ample opportunities for improvements with these two modules. Similarly, Exp. 5
with the jersey number recognition module exposes it as another key weakness
of the pipeline. Finally, performance in Exp. 6 is close to the complete
baseline, since the object detector is the starting point for most of the
pipeline, and ground truth data is therefore employed here only for pitch
localization and camera calibration.

Our ablation study shows that while localization and identification are
challenging alone, their intricate combination in GSR proves even more
challenging.

(a) Predictions on input image

(b) Ground Truth minimap

(c) Predicted minimap

Figure 5: Qualitative results. Output predictions of two frames from videos
with different GS-HOTA values. (Top) High GS-HOTA (49.69%), with robust pitch
localization and accurate athlete identification. (Bottom) Calibration failure
(_e.g_. due to insufficient pitch elements) leads to completely erroneous
athlete localization and poor GS-HOTA (0.23%).  Table 2:  GSR-Baseline
Ablation Study. We report the GS-HOTA for each GSR-Baseline module and its
corresponding downstream modules by replacing other modules by a ground truth
oracle. We also report their speed in FPS and their input batch sizes.

| Module | GS-HOTA $\uparrow$ | Batch S. | FPS
---|---|---|---|---
(1) | Team Side | 92.00 | Video | 1.5K
(2) | ReID (PRTReID) | 87.42 | 16 | 14.5
(3) | Calibration (TVCalib) | 51.39 | 512 | 7.6
(4) | Pitch (TVCalib) | 49.99 | 16 | 2.9
(5) | Jersey N° (MMOCR) | 56.75 | 32 | 3.8
(6) | BBox Det. (YOLOv8) | 35.28 | 32 | 16.5
| Full Baseline | 22.26 | N/A | 1.1

Inference Time. Since the GSR-Baseline is an offline pipeline, each module
processes its input in batches, where a single batch can span multiple images.
The batch size and average frame rate of each module are reported in Tab. 2.
All inference speed tests are performed with an NVIDIA A100 32GB GPU. As
illustrated, pitch localization, camera calibration, and jersey-number
recognition emerge as the most time-consuming modules. It takes on average
$11$ minutes to process one $30$s sequence from our dataset.

Qualitative Results. Fig. 5 illustrates two game state minimaps predicted by
our GSR-Baseline and their respective ground truths. Our GSR-Baseline achieves
a high GS-HOTA score of $49.69\%$ on the video illustrated in the first row,
accurately predicting most athletes’ pitch positions and attributes. The
bottom example, from a video with a GS-HOTA of $0.23\%$, exemplifies common
failure cases, where even minor calibration inaccuracies can cause major pitch
registration errors. In this frame, poor calibration is caused by the small
number of visible salient points on the pitch.

##  8 Conclusion

Our work introduces the first Game State Reconstruction (GSR) benchmark for
athlete identification and tracking on a minimap, comprising a new dataset,
evaluation metric, and open-source baseline. Unlike previous efforts in sports
video understanding that focused on specific subtasks, our approach stands out
by benchmarking a complete pipeline, whose high-level game semantics outputs
are directly relevant to a broad spectrum of downstream applications.
Moreover, experiments with our proposed baseline reveal the inherent
complexity of the GSR task and the significant interdependencies among its
various subtasks. We hope that our introduced benchmark will pave the way for
a new line of exciting research on specialized GSR methods. We anticipate
future efforts to focus on (1) enhancing specific modules to increase
performance, (2) implementing real-time pipelines, or even (3) developing end-
to-end differentiable methods for tackling the task in one step.

#### Acknowledgments.

This work was supported by SportRadar, the Service Public de Wallonie (SPW)
Recherche, under the ReconnAIssance project and Grant
$\text{N}^{\text{o}}$8573, the F.R.S-FNRS, FRIA/FNRS, the King Abdullah
University of Science and Technology (KAUST) Office of Sponsored Research
through the Visual Computing Center (VCC) funding and the SDAIA-KAUST Center
of Excellence in Data Science and Artificial Intelligence (SDAIA-KAUST AI).
Computational resources have been provided by the supercomputing facilities of
the Université catholique de Louvain (CISM/UCLouvain).

## References

  * Arbués Sangüesa et al. [2020] Adrià Arbués Sangüesa, Adriàn Martín, Javier Fernández, Coloma Ballester, and Gloria Haro.  Using player’s body-orientation to model pass feasibility in soccer.  In _IEEE/CVF Conf. Comput. Vis. Pattern Recognit. Work. (CVPRW)_ , pages 3875–3884, Seattle, WA, USA, 2020. Inst. Electr. Electron. Eng. (IEEE).
  * Balaji et al. [2023] Bavesh Balaji, Jerrin Bright, Harish Prakash, Yuhao Chen, David A. Clausi, and John Zelek.  Jersey number recognition using keyframe identification from low-resolution broadcast videos.  In _Proceedings of the 6th International Workshop on Multimedia Content Analysis in Sports_ , page 123–130, Ottawa, Ontario, Can., 2023. ACM.
  * Beal et al. [2020] Ryan Beal, Georgios Chalkiadakis, Timothy J. Norman, and Sarvapali D. Ramchurn.  Optimising game tactics for football.  _arXiv_ , abs/2003.10294, 2020.
  * Bergmann et al. [2019] Philipp Bergmann, Tim Meinhardt, and Laura Leal-Taixe.  Tracking without bells and whistles.  In _IEEE/CVF Int. Conf. Comput. Vis. (ICCV)_ , pages 941–951, Seoul, North Korea, 2019. Inst. Electr. Electron. Eng. (IEEE).
  * Bernardin and Stiefelhagen [2008] Keni Bernardin and Rainer Stiefelhagen.  Evaluating multiple object tracking performance: The CLEAR MOT metrics.  _EURASIP J. Image Video Process._ , 2008:1–10, 2008.
  * Bewley et al. [2016] Alex Bewley, Zongyuan Ge, Lionel Ott, Fabio Ramos, and Ben Upcroft.  Simple online and realtime tracking.  In _IEEE Int. Conf. Image Process. (ICIP)_ , pages 3464–3468, Phoenix, AZ, USA, 2016. Inst. Electr. Electron. Eng. (IEEE).
  * Bochinski et al. [2018] Erik Bochinski, Tobias Senst, and Thomas Sikora.  Extending IOU based multi-object tracking by visual information.  In _IEEE Int. Conf. Adv. Video Signal Based Surveill. (AVSS)_ , pages 1–6, Auckland, New Zealand, 2018. Inst. Electr. Electron. Eng. (IEEE).
  * Boeker and Midoglu [2023] Matthias Boeker and Cise Midoglu.  Soccer athlete data visualization and analysis with an interactive dashboard.  In _Int. Conf. Multimedia Retr._ , pages 565–576. Springer Int. Publ., 2023.
  * Cabado et al. [2024] Bruno Cabado, Anthony Cioppa, Silvio Giancola, Andrés Villa, Bertha Guijarro-Berdiñas, Emilio Padrón, Bernard Ghanem, and Marc Van Droogenbroeck.  Beyond the Premier: Assessing action spotting transfer capability across diverse domains.  In _IEEE Int. Conf. Comput. Vis. Pattern Recognit. Work. (CVPRW), CVsports_ , Seattle, WA, USA, 2024.
  * Cao et al. [2022] Mengqi Cao, Min Yang, Guozhen Zhang, Xiaotian Li, Yilu Wu, Gangshan Wu, and Limin Wang.  SpotFormer: A transformer-based framework for precise soccer action spotting.  In _Int. Work. Multimedia Signal Process. (MMSP)_ , pages 1–6, Shanghai, China, 2022. Inst. Electr. Electron. Eng. (IEEE).
  * Chan et al. [2022] Cheuk-Yiu Chan, Chun-Chuen Hui, Wan-Chi Siu, Sin-wai Chan, and H. Anthony Chan.  To start automatic commentary of soccer game with mixed spatial and temporal attention.  In _IEEE Region 10 Conference (TENCON)_ , pages 1–6, Hong Kong, China, 2022. Inst. Electr. Electron. Eng. (IEEE).
  * Chen and De Vleeschouwer [2010] Fan Chen and Christophe De Vleeschouwer.  Personalized production of basketball videos from multi-sensored data under limited display resolution.  _Comput. Vis. Image Underst._ , 114(6):667–680, 2010.
  * Chen and De Vleeschouwer [2011] Fan Chen and Christophe De Vleeschouwer.  Automatic summarization of broadcasted soccer videos with adaptive fast-forwarding.  In _IEEE Int. Conf. Multimedia Expo (ICME)_ , pages 1–6, Barcelona, Spain, 2011. Inst. Electr. Electron. Eng. (IEEE).
  * Chen et al. [2018] Jianhui Chen, Fangrui Zhu, and James J. Little.  A two-point method for PTZ camera calibration in sports.  In _IEEE Winter Conf. Appl. Comput. Vis. (WACV)_ , pages 287–295, Lake Tahoe, NV, USA, 2018. Inst. Electr. Electron. Eng. (IEEE).
  * Cioppa et al. [2018] Anthony Cioppa, Adrien Deliège, and Marc Van Droogenbroeck.  A bottom-up approach based on semantics for the interpretation of the main camera stream in soccer games.  In _IEEE Int. Conf. Comput. Vis. Pattern Recognit. Work. (CVPRW), CVsports_ , pages 1846–1855, Salt Lake City, UT, USA, 2018.
  * Cioppa et al. [2019] Anthony Cioppa, Adrien Deliege, Maxime Istasse, Christophe De Vleeschouwer, and Marc Van Droogenbroeck.  ARTHuS: Adaptive real-time human segmentation in sports through online distillation.  In _IEEE Int. Conf. Comput. Vis. Pattern Recognit. Work. (CVPRW), CVsports_ , pages 2505–2514, Long Beach, CA, USA, 2019. Inst. Electr. Electron. Eng. (IEEE).
  * Cioppa et al. [2020a] Anthony Cioppa, Adrien Deliège, Silvio Giancola, Bernard Ghanem, Marc Van Droogenbroeck, Rikke Gade, and Thomas B. Moeslund.  A context-aware loss function for action spotting in soccer videos.  In _IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR)_ , pages 13123–13133, Seattle, WA, USA, 2020a. Inst. Electr. Electron. Eng. (IEEE).
  * Cioppa et al. [2020b] Anthony Cioppa, Adrien Deliège, Noor Ul Huda, Rikke Gade, Marc Van Droogenbroeck, and Thomas B. Moeslund.  Multimodal and multiview distillation for real-time player detection on a football field.  In _IEEE Int. Conf. Comput. Vis. Pattern Recognit. Work. (CVPRW), CVsports_ , pages 3846–3855, Seattle, WA, USA, 2020b.
  * Cioppa et al. [2021] Anthony Cioppa, Adrien Deliège, Silvio Giancola, Floriane Magera, Olivier Barnich, Bernard Ghanem, and Marc Van Droogenbroeck.  Camera calibration and player localization in SoccerNet-v2 and investigation of their representations for action spotting.  In _IEEE Int. Conf. Comput. Vis. Pattern Recognit. Work. (CVPRW), CVsports_ , pages 4532–4541, Nashville, TN, USA, 2021.
  * Cioppa et al. [2022a] Anthony Cioppa, Adrien Deliège, Silvio Giancola, Bernard Ghanem, and Marc Van Droogenbroeck.  Scaling up SoccerNet with multi-view spatial localization and re-identification.  _Sci. Data_ , 9(1):1–9, 2022a.
  * Cioppa et al. [2022b] Anthony Cioppa, Silvio Giancola, Adrien Deliege, Le Kang, Xin Zhou, Zhiyu Cheng, Bernard Ghanem, and Marc Van Droogenbroeck.  SoccerNet-tracking: Multiple object tracking dataset and benchmark in soccer videos.  In _IEEE Int. Conf. Comput. Vis. Pattern Recognit. Work. (CVPRW), CVsports_ , pages 3490–3501, New Orleans, LA, USA, 2022b. Inst. Electr. Electron. Eng. (IEEE).
  * Cioppa et al. [2023] Anthony Cioppa, Silvio Giancola, Vladimir Somers, Floriane Magera, Xin Zhou, Hassan Mkhallati, Adrien Deliège, Jan Held, Carlos Hinojosa, Amir M. Mansourian, Pierre Miralles, Olivier Barnich, Christophe De Vleeschouwer, Alexandre Alahi, Bernard Ghanem, Marc Van Droogenbroeck, Abdullah Kamal, Adrien Maglo, Albert Clapés, Amr Abdelaziz, Artur Xarles, Astrid Orcesi, Atom Scott, Bin Liu, Byoungkwon Lim, Chen Chen, Fabian Deuser, Feng Yan, Fufu Yu, Gal Shitrit, Guanshuo Wang, Gyusik Choi, Hankyul Kim, Hao Guo, Hasby Fahrudin, Hidenari Koguchi, Håkan Ardö, Ibrahim Salah, Ido Yerushalmy, Iftikar Muhammad, Ikuma Uchida, Ishay Be’ery, Jaonary Rabarisoa, Jeongae Lee, Jiajun Fu, Jianqin Yin, Jinghang Xu, Jongho Nang, Julien Denize, Junjie Li, Junpei Zhang, Juntae Kim, Kamil Synowiec, Kenji Kobayashi, Kexin Zhang, Konrad Habel, Kota Nakajima, Licheng Jiao, Lin Ma, Lizhi Wang, Luping Wang, Menglong Li, Mengying Zhou, Mohamed Nasr, Mohamed Abdelwahed, Mykola Liashuha, Nikolay Falaleev, Norbert Oswald, Qiong Jia, Quoc-Cuong Pham, Ran Song, Romain Hérault, Rui Peng, Ruilong Chen, Ruixuan Liu, Ruslan Baikulov, Ryuto Fukushima, Sergio Escalera, Seungcheon Lee, Shimin Chen, Shouhong Ding, Taiga Someya, Thomas B. Moeslund, Tianjiao Li, Wei Shen, Wei Zhang, Wei Li, Wei Dai, Weixin Luo, Wending Zhao, Wenjie Zhang, Xinquan Yang, Yanbiao Ma, Yeeun Joo, Yingsen Zeng, Yiyang Gan, Yongqiang Zhu, Yujie Zhong, Zheng Ruan, Zhiheng Li, Zhijian Huangi, and Ziyu Meng.  SoccerNet 2023 challenges results.  _arXiv_ , abs/2309.06006, 2023.
  * Decroos et al. [2018] Tom Decroos, Jan Van Haaren, and Jesse Davis.  Automatic discovery of tactics in spatio-temporal soccer match data.  In _ACM SIGKDD Int. Conf. Knowl. Discov. Data Min. (KDD)_ , page 223–232. ACM, 2018.
  * Deliège et al. [2021] Adrien Deliège, Anthony Cioppa, Silvio Giancola, Meisam J. Seikavandi, Jacob V. Dueholm, Kamal Nasrollahi, Bernard Ghanem, Thomas B. Moeslund, and Marc Van Droogenbroeck.  SoccerNet-v2: A dataset and benchmarks for holistic understanding of broadcast soccer videos.  In _IEEE Int. Conf. Comput. Vis. Pattern Recognit. Work. (CVPRW), CVsports_ , pages 4508–4519, Nashville, TN, USA, 2021.
  * Du et al. [2023] Yunhao Du, Zhicheng Zhao, Yang Song, Yanyun Zhao, Fei Su, Tao Gong, and Hongying Meng.  StrongSORT: Make DeepSORT great again.  _IEEE Trans. Multimedia_ , 25:8725–8737, 2023.
  * EVS Broadcast Equipment [2022] EVS Broadcast Equipment.  Multi-camera review system - Xeebra.  https://evs.com/products/video-assistance/xeebra, 2022.
  * Farin et al. [2003] D. Farin, S. Krabbe, and W. Effelsberg et.al.  Robust camera calibration for sport videos using court models.  In _Storage and Retrieval Methods and Applications for Multimedia_ , pages 80–92, San Jose, California, USA, 2003.
  * Fernandez et al. [2010] Ivan Alen Fernandez, Fan Chen, Fabien Lavigne, Xavier Desurmont, and Christophe De Vleeschouwer.  Browsing sport content through an interactive H.264 streaming session.  In _International Conferences on Advances in Multimedia_ , pages 155–161, Athens, Greece, 2010. Inst. Electr. Electron. Eng. (IEEE).
  * Fischer et al. [2019] Maximilian T. Fischer, Daniel A. Keim, and Manuel Stein.  Video-based analysis of soccer matches.  In _Int. ACM Work. Multimedia Content Anal. Sports (MMSports)_ , page 1–9, Nice, France, 2019. ACM.
  * Gerke et al. [2015] Sebastian Gerke, Karsten Muller, and Ralf Schafer.  Soccer jersey number recognition using convolutional neural networks.  In _IEEE Int. Conf. Comput. Vis. Work. (ICCV Work.)_ , pages 734–741, Santiago, Chile, 2015. Inst. Electr. Electron. Eng. (IEEE).
  * Ghasemzadeh et al. [2021] Seyed Abolfazl Ghasemzadeh, Gabriel Van Zandycke, Maxime Istasse, Niels Sayez, Amirafshar Moshtaghpour, and Christophe De Vleeschouwer.  DeepSportLab: a unified framework for ball detection, player instance segmentation and pose estimation in team sports scenes.  _arXiv_ , abs/2112.00627, 2021.
  * Ghosh et al. [2023] Adhiraj Ghosh, Kuruparan Shanmugalingam, and Wen-Yan Lin.  Relation preserving triplet mining for stabilising the triplet loss in re-identification systems.  In _IEEE/CVF Winter Conf. Appl. Comput. Vis. (WACV)_ , pages 4829–4838, Waikoloa, HI, USA, 2023. Inst. Electr. Electron. Eng. (IEEE).
  * Giancola and Ghanem [2021] Silvio Giancola and Bernard Ghanem.  Temporally-aware feature pooling for action spotting in soccer broadcasts.  In _IEEE Int. Conf. Comput. Vis. Pattern Recognit. (CVPR)_ , pages 4490–4499, Nashville, TN, USA, 2021.
  * Giancola et al. [2022] Silvio Giancola, Anthony Cioppa, Adrien Deliège, Floriane Magera, Vladimir Somers, Le Kang, Xin Zhou, Olivier Barnich, Christophe De Vleeschouwer, Alexandre Alahi, Bernard Ghanem, Marc Van Droogenbroeck, Abdulrahman Darwish, Adrien Maglo, Albert Clapés, Andreas Luyts, Andrei Boiarov, Artur Xarles, Astrid Orcesi, Avijit Shah, Baoyu Fan, Bharath Comandur, Chen Chen, Chen Zhang, Chen Zhao, Chengzhi Lin, Cheuk-Yiu Chan, Chun Chuen Hui, Dengjie Li, Fan Yang, Fan Liang, Fang Da, Feng Yan, Fufu Yu, Guanshuo Wang, H. Anthony Chan, He Zhu, Hongwei Kan, Jiaming Chu, Jianming Hu, Jianyang Gu, Jin Chen, João V. B. Soares, Jonas Theiner, Jorge De Corte, José Henrique Brito, Jun Zhang, Junjie Li, Junwei Liang, Leqi Shen, Lin Ma, Lingchi Chen, Miguel Santos Marques, Mike Azatov, Nikita Kasatkin, Ning Wang, Qiong Jia, Quoc Cuong Pham, Ralph Ewerth, Ran Song, Rengang Li, Rikke Gade, Ruben Debien, Runze Zhang, Sangrok Lee, Sergio Escalera, Shan Jiang, Shigeyuki Odashima, Shimin Chen, Shoichi Masui, Shouhong Ding, Sin-wai Chan, Siyu Chen, Tallal El-Shabrawy, Tao He, Thomas B. Moeslund, Wan-Chi Siu, Wei Zhang, Wei Li, Xiangwei Wang, Xiao Tan, Xiaochuan Li, Xiaolin Wei, Xiaoqing Ye, Xing Liu, Xinying Wang, Yandong Guo, Yaqian Zhao, Yi Yu, Yingying Li, Yue He, Yujie Zhong, Zhenhua Guo, and Zhiheng Li.  SoccerNet 2022 challenges results.  In _Int. ACM Work. Multimedia Content Anal. Sports (MMSports)_ , pages 75–86, Lisbon, Port., 2022. ACM.
  * Giancola et al. [2023] Silvio Giancola, Anthony Cioppa, Julia Georgieva, Johsan Billingham, Andreas Serner, Kerry Peek, Bernard Ghanem, and Marc Van Droogenbroeck.  Towards active learning for action spotting in association football videos.  In _IEEE/CVF Conf. Comput. Vis. Pattern Recognit. Work. (CVPRW)_ , pages 5098–5108, Vancouver, Can., 2023. Inst. Electr. Electron. Eng. (IEEE).
  * Held et al. [2023] Jan Held, Anthony Cioppa, Silvio Giancola, Abdullah Hamdi, Bernard Ghanem, and Marc Van Droogenbroeck.  VARS: Video assistant referee system for automated soccer decision making from multiple views.  In _IEEE/CVF Conf. Comput. Vis. Pattern Recognit. Work. (CVPRW)_ , pages 5086–5097, Vancouver, Can., 2023. Inst. Electr. Electron. Eng. (IEEE).
  * Held et al. [2024] Jan Held, Hani Itani, Anthony Cioppa, Silvio Giancola, Bernard Ghanem, and Marc Van Droogenbroeck.  X-vars: Introducing explainability in football refereeing with multi-modal large language models.  In _IEEE Int. Conf. Comput. Vis. Pattern Recognit. Work. (CVPRW), CVsports_ , Seattle, WA, USA, 2024.
  * Hong et al. [2022] James Hong, Haotian Zhang, Michaël Gharbi, Matthew Fisher, and Kayvon Fatahalian.  Spotting temporally precise, fine-grained events in video.  In _Eur. Conf. Comput. Vis. (ECCV)_ , pages 33–51, Tel Aviv, Israël, 2022. Springer Nat. Switz.
  * Huang et al. [2024] Hsiang-Wei Huang, Cheng-Yen Yang, Jiacheng Sun, Pyong-Kun Kim, Kwang-Ju Kim, Kyoungoh Lee, Chung-I Huang, and Jenq-Neng Hwang.  Iterative scale-up ExpansionIoU and deep features association for multi-object tracking in sports.  In _IEEE/CVF Winter Conf. Appl. Comput. Vis. Work. (WACVW)_ , pages 163–172, Waikoloa, HI, USA, 2024.
  * IFAB [2022] IFAB.  Laws of the game.  Technical report, The International Football Association Board, Zurich, Switzerland, 2022.
  * Istasse et al. [2019] Maxime Istasse, Julien Moreau, and Christophe De Vleeschouwer.  Associative embedding for team discrimination.  In _IEEE Int. Conf. Comput. Vis. Pattern Recognit. Work. (CVPRW), CVsports_ , pages 2477–2486, Long Beach, CA, USA, 2019.
  * Istasse et al. [2023] Maxime Istasse, Vladimir Somers, Pratheeban Elancheliyan, Jaydeep De, and Davide Zambrano.  DeepSportradar-v2: A multi-sport computer vision dataset for sport understandings.  In _Int. ACM Work. Multimedia Content Anal. Sports (MMSports)_ , pages 23–29, Ottawa, Ontario, Can., 2023. ACM.
  * Jiang et al. [2020] Yudong Jiang, Kaixu Cui, Leilei Chen, Canjin Wang, and Changliang Xu.  SoccerDB: A large-scale database for comprehensive video understanding.  In _Int. ACM Work. Multimedia Content Anal. Sports (MMSports)_ , page 1–8, Seattle, WA, USA, 2020. ACM.
  * Jocher et al. [2023] Glenn Jocher, Ayush Chaurasia, and Jing Qiu.  Ultralytics YOLOv8.  https://github.com/ultralytics/ultralytics, 2023.
  * Joos et al. [2024] Victor Joos, Vladimir Somers, and Baptiste Standaert.  TrackLab.  https://github.com/TrackingLaboratory/tracklab, 2024.
  * Karungaru et al. [2022] Stephen Karungaru, Hiroki Tanioka, and Kenji Matsuura.  Soccer players real location determination using perspective transformation.  In _Int. Conf. Soft Comput. Intell. Syst., Int. Symp. Adv. Intell. Syst. (SCIS &ISIS)_, pages 1–4, Ise, Japan, 2022. Inst. Electr. Electron. Eng. (IEEE).
  * Kiefer et al. [2023] Benjamin Kiefer, Matej Kristan, Janez Perš, Lojze Žust, Fabio Poiesi, Fabio Andrade, Alexandre Bernardino, Matthew Dawkins, Jenni Raitoharju, Yitong Quan, Adem Atmaca, Timon Höfer, Qiming Zhang, Yufei Xu, Jing Zhang, Dacheng Tao, Lars Sommer, Raphael Spraul, Hangyue Zhao, Hongpu Zhang, Yanyun Zhao, Jan Lukas Augustin, Eui-ik Jeon, Impyeong Lee, Luca Zedda, Andrea Loddo, Cecilia Di Ruberto, Sagar Verma, Siddharth Gupta, Shishir Muralidhara, Niharika Hegde, Daitao Xing, Nikolaos Evangeliou, Anthony Tzes, Vojtěch Bartl, Jakub Špaňhel, Adam Herout, Neelanjan Bhowmik, Toby P. Breckon, Shivanand Kundargi, Tejas Anvekar, Ramesh Ashok Tabib, Uma Mudenagudi, Arpita Vats, Yang Song, Delong Liu, Yonglin Li, Shuman Li, Chenhao Tan, Long Lan, Vladimir Somers, Christophe De Vleeschouwer, Alexandre Alahi, Hsiang-Wei Huang, Cheng-Yen Yang, Jenq-Neng Hwang, Pyong-Kun Kim, Kwangju Kim, Kyoungoh Lee, Shuai Jiang, Haiwen Li, Zheng Ziqiang, Tuan-Anh Vu, Hai Nguyen-Truong, Sai-Kit Yeung, Zhuang Jia, Sophia Yang, Chih-Chung Hsu, Xiu-Yu Hou, Yu-An Jhang, Simon Yang, and Mau-Tsuen Yang.  1st workshop on maritime computer vision (macvi) 2023: Challenge results.  In _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) Workshops_ , pages 265–302, 2023.
  * Kim et al. [2023] Minjung Kim, MyeongAh Cho, and Sangyoun Lee.  Feature disentanglement learning with switching and aggregation for video-based person re-identification.  In _IEEE/CVF Winter Conf. Appl. Comput. Vis. (WACV)_ , pages 1603–1612, Waikoloa, HI, USA, 2023. Inst. Electr. Electron. Eng. (IEEE).
  * Kuang et al. [2021] Zhanghui Kuang, Hongbin Sun, Zhizhong Li, Xiaoyu Yue, Tsui Hin Lin, Jianyong Chen, Huaqiang Wei, Yiqin Zhu, Tong Gao, Wenwei Zhang, Kai Chen, Wayne Zhang, and Dahua Lin.  MMOCR: A comprehensive toolbox for text detection, recognition and understanding.  In _Proceedings of the 29th ACM International Conference on Multimedia_ , page 3791–3794. ACM, 2021.
  * Kurach et al. [2020] Karol Kurach, Anton Raichuk, Piotr Stanczyk, Michał Zajac, Olivier Bachem, Lasse Espeholt, Carlos Riquelme, Damien Vincent, Marcin Michalski, Olivier Bousquet, and Sylvain Gelly.  Google research football: A novel reinforcement learning environment.  In _AAAI Conf. Artif. Intell._ , pages 4501–4510. Association for the Advancement of Artificial Intelligence (AAAI), 2020.
  * Leduc et al. [2024] Arnaud Leduc, Anthony Cioppa, Silvio Giancola, Bernard Ghanem, and Marc Van Droogenbroeck.  SoccerNet-Depth: a scalable dataset for monocular depth estimation in sports videos.  In _IEEE Int. Conf. Comput. Vis. Pattern Recognit. Work. (CVPRW), CVsports_ , Seattle, WA, USA, 2024.
  * Li et al. [2019] Hui Li, Peng Wang, Chunhua Shen, and Guyu Zhang.  Show, attend and read: A simple and strong baseline for irregular text recognition.  In _AAAI Conf. Artif. Intell._ , pages 8610–8617. Association for the Advancement of Artificial Intelligence (AAAI), 2019.
  * Liao et al. [2020] Minghui Liao, Zhaoyi Wan, Cong Yao, Kai Chen, and Xiang Bai.  Real-time scene text detection with differentiable binarization.  In _AAAI Conf. Artif. Intell._ , pages 11474–11481. Association for the Advancement of Artificial Intelligence (AAAI), 2020.
  * Liu and Bhanu [2019] Hengyue Liu and Bir Bhanu.  Pose-guided R-CNN for jersey number recognition in sports.  In _IEEE Int. Conf. Comput. Vis. Pattern Recognit. Work. (CVPRW)_ , pages 2457–2466, Long Beach, CA, USA, 2019.
  * Liu et al. [2023] Hongshan Liu, Colin Adreon, Noah Wagnon, Abdul Latif Bamba, Xueshen Li, Huapu Liu, Steven MacCall, and Yu Gan.  Automated player identification and indexing using two-stage deep learning network.  _Sci. Reports_ , 13(1), 2023.
  * Ludwig et al. [2023] Katja Ludwig, Julian Lorenz, Robin Schön, and Rainer Lienhart.  All keypoints you need: Detecting arbitrary keypoints on the body of triple, high, and long jump athletes.  In _IEEE/CVF Conf. Comput. Vis. Pattern Recognit. Work. (CVPRW)_ , pages 5179–5187, Vancouver, Can., 2023. Inst. Electr. Electron. Eng. (IEEE).
  * Luiten et al. [2020] Jonathon Luiten, Aljosa Osep, Patrick Dendorfer, Philip Torr, Andreas Geiger, Laura Leal-Taixé, and Bastian Leibe.  HOTA: A higher order metric for evaluating multi-object tracking.  _Int. J. Comput. Vis._ , 129(2):548–578, 2020.
  * Magera [2022] Floriane Magera.  SoccerNet camera calibration challenge.  https://github.com/SoccerNet/sn-calibration, 2022.
  * Magera et al. [2024] Floriane Magera, Thomas Hoyoux, Olivier Barnich, and Marc Van Droogenbroeck.  A universal protocol to benchmark camera calibration for sports.  In _IEEE Int. Conf. Comput. Vis. Pattern Recognit. Work. (CVPRW), CVsports_ , Seattle, WA, USA, 2024.
  * Maglo et al. [2023] Adrien Maglo, Astrid Orcesi, Julien Denize, and Quoc Cuong Pham.  Individual locating of soccer players from a single moving view.  _Sensors_ , 23(18):1–28, 2023.
  * Mansourian et al. [2023] Amir M. Mansourian, Vladimir Somers, Christophe De Vleeschouwer, and Shohreh Kasaei.  Multi-task learning for joint re-identification, team affiliation, and role classification for sports visual tracking.  In _Int. ACM Work. Multimedia Content Anal. Sports (MMSports)_ , page 103–112, Ottawa, Ontario, Can., 2023. ACM.
  * Midoglu et al. [2022] Cise Midoglu, Steven Hicks, Vajira Thambawita, Tomas Kupka, and Pål Halvorsen.  MMSys’22 grand challenge on AI-based video production for soccer.  In _ACM Multimedia Systems Conference (MMSys)_ , pages 1–6, Athlone, Ireland, 2022.
  * Mkhallati et al. [2023] Hassan Mkhallati, Anthony Cioppa, Silvio Giancola, Bernard Ghanem, and Marc Van Droogenbroeck.  SoccerNet-caption: Dense video captioning for soccer broadcasts commentaries.  In _IEEE/CVF Conf. Comput. Vis. Pattern Recognit. Work. (CVPRW)_ , pages 5074–5085, Vancouver, Can., 2023. Inst. Electr. Electron. Eng. (IEEE).
  * Moeslund et al. [2014] Thomas B. Moeslund, Graham Thomas, and Adrian Hilton.  _Computer vision in sports_.  Springer, 2014.
  * Nady and Hemayed [2021] Ahmed Nady and Elsayed Hemayed.  Player identification in different sports.  In _Comput. Vis. Imaging Comput. Graph. Theory Appl. (VISIGRAPP)_ , pages 1–8, Vienna, Austria, 2021. SCITEPRESS - Science and Technology Publications.
  * Naik et al. [2022] Banoth Thulasya Naik, Mohammad Farukh Hashmi, Neeraj Dhanraj Bokde, and Zaher Mundher Yaseen.  A comprehensive review of computer vision in sports: Open issues, future trends and research directions.  _Appl. Sci._ , 12(9):1–49, 2022.
  * Pappalardo et al. [2019a] Luca Pappalardo, Paolo Cintia, Paolo Ferragina, Emanuele Massucco, Dino Pedreschi, and Fosca Giannotti.  PlayeRank: Data-driven performance evaluation and player ranking in soccer via a machine learning approach.  _ACM Trans. Intell. Syst. Technol._ , 10(5):1–27, 2019a.
  * Pappalardo et al. [2019b] Luca Pappalardo, Paolo Cintia, Alessio Rossi, Emanuele Massucco, Paolo Ferragina, Dino Pedreschi, and Fosca Giannotti.  A public data set of spatio-temporal match events in soccer competitions.  _Sci. Data_ , 6(1):1–15, 2019b.
  * Parisot and De Vleeschouwer [2017] Pascaline Parisot and Christophe De Vleeschouwer.  Scene-specific classifier for effective and efficient team sport players detection from a single calibrated camera.  _Comput. Vis. Image Underst._ , 159:74–88, 2017.
  * Perin et al. [2013] Charles Perin, Romain Vuillemot, and Jean-Daniel Fekete.  SoccerStories: A kick-off for visual soccer analysis.  _IEEE Trans. Vis. Comput. Graph._ , 19(12):2506–2515, 2013.
  * Pourreza et al. [2008] Reza Pourreza, Morteza Khademi, Hamidreza Pourreza, and Habib Rajabi Mashhadi.  Robust camera calibration of soccer video using genetic algorithm.  In _IEEE Int. Conf. Intell. Comput. Commun. Process. (ICCP)_ , pages 123–127, Cluj-Napoca, Romania, 2008. Inst. Electr. Electron. Eng. (IEEE).
  * Raja et al. [2024] S. Kanaga Suba Raja, K. Kausalya, B. Sandhiya, K. Abdul Waseem Nihaal W., A. Abiya Feba Mary, and J. Afra Thahseen.  Tracking of multi athlete and action recognition in soccer sports video using deep learning techniques.  _AIP Conference Proceedings_ , 2802(1), 2024.
  * Rao and Pati [2015] Upendra M. Rao and Umesh C. Pati.  A novel algorithm for detection of soccer ball and player.  In _Int. Conf. Commun. Signal Process. (ICCSP)_ , pages 344–348, Melmaruvathur, India, 2015.
  * Sacha et al. [2017] D. Sacha, F. Al‐Masoudi, M. Stein, T. Schreck, D. A. Keim, G. Andrienko, and H. Janetzko.  Dynamic visual abstraction of soccer movement.  _Computer Graphics Forum_ , 36(3):305–315, 2017.
  * Santos Marques et al. [2023] Miguel Santos Marques, Ricardo Gomes Faria, and José Henrique Brito.  Hierarchical line extremity segmentation U-Net for the SoccerNet 2022 calibration challenge - pitch localization.  In _Iberian Conference on Pattern Recognition and Image Analysis_ , pages 442–453. Springer Nat. Switz., 2023.
  * Scott et al. [2022] Atom Scott, Ikuma Uchida, Masaki Onishi, Yoshinari Kameda, Kazuhiro Fukui, and Keisuke Fujii.  SoccerTrack: A dataset and tracking algorithm for soccer with fish-eye and drone videos.  In _IEEE/CVF Conf. Comput. Vis. Pattern Recognit. Work. (CVPRW)_ , pages 3568–3578, New Orleans, LA, USA, 2022. Inst. Electr. Electron. Eng. (IEEE).
  * Seweryn et al. [2024] Karolina Seweryn, Gabriel Cheć, Szymon Łukasik, and Anna Wróblewska.  Improving object detection quality in football through super-resolution techniques.  _arXiv_ , abs/2402.00163, 2024.
  * Sha et al. [2020] Long Sha, Jennifier Hobbs, Panna Felsen, Winyu Wei, Patrick Lucey, and Sujoy Ganguly.  End-to-end camera calibration for broadcast videos.  In _IEEE Int. Conf. Comput. Vis. Pattern Recognit. (CVPR)_ , pages 13627–13636, Seattle, WA, USA, 2020. Inst. Electr. Electron. Eng. (IEEE).
  * Shitrit et al. [2023] Gal Shitrit, Ishay Be’ery, and Ido Yerhushalmy.  SoccerNet 2023 tracking challenge – 3rd place MOT4MOT team technical report.  _arXiv_ , abs/2308.16651, 2023.
  * Soares et al. [2022] João V. B. Soares, Avijit Shah, and Topojoy Biswas.  Temporally precise action spotting in soccer videos using dense detection anchors.  In _IEEE Int. Conf. Image Process. (ICIP)_ , pages 2796–2800, Bordeaux, France, 2022. Inst. Electr. Electron. Eng. (IEEE).
  * Somers et al. [2023] Vladimir Somers, Christophe De Vleeschouwer, and Alexandre Alahi.  Body part-based representation learning for occluded person Re-Identification.  In _IEEE/CVF Winter Conf. Appl. Comput. Vis. (WACV)_ , pages 1613–1623, Waikoloa, HI, USA, 2023. Inst. Electr. Electron. Eng. (IEEE).
  * Stein et al. [2018] Manuel Stein, Halldor Janetzko, Andreas Lamprecht, Thorsten Breitkreutz, Philipp Zimmermann, Bastian Goldlucke, Tobias Schreck, Gennady Andrienko, Michael Grossniklaus, and Daniel A. Keim.  Bring it to the pitch: Combining video and movement data to enhance team sport analysis.  _IEEE Trans. Vis. Comput. Graph._ , 24(1):13–22, 2018.
  * Sun et al. [2019] ShiJie Sun, Naveed Akhtar, HuanSheng Song, Ajmal S. Mian, and Mubarak Shah.  Deep affinity network for multiple object tracking.  _IEEE Trans. Pattern Anal. Mach. Intell._ , 43(1):104–119, 2019.
  * Theiner and Ewerth [2023] Jonas Theiner and Ralph Ewerth.  TVCalib: Camera calibration for sports field registration in soccer.  In _IEEE/CVF Winter Conf. Appl. Comput. Vis. (WACV)_ , pages 1166–1175, Waikoloa, HI, USA, 2023. Inst. Electr. Electron. Eng. (IEEE).
  * Theiner et al. [2021] Jonas Theiner, Wolfgang Gritz, Eric Müller-Budack, Robert Rein, Daniel Memmert, and Ralph Ewerth.  Extraction of positional player data from broadcast soccer videos.  _arXiv_ , abs/2110.11107, 2021.
  * Thomas et al. [2017] Graham Thomas, Rikke Gade, Thomas B. Moeslund, Peter Carr, and Adrian Hilton.  Computer vision for sports: current applications and research topics.  _Comput. Vis. Image Underst._ , 159:3–18, 2017.
  * Van Zandycke et al. [2022] Gabriel Van Zandycke, Vladimir Somers, Maxime Istasse, Carlo Del Don, and Davide Zambrano.  DeepSportradar-v1: Computer vision dataset for sports understanding with high quality annotations.  In _Int. ACM Work. Multimedia Content Anal. Sports (MMSports)_ , pages 1–8, Lisbon, Port., 2022. ACM.
  * Vandeghen et al. [2022] Renaud Vandeghen, Anthony Cioppa, and Marc Van Droogenbroeck.  Semi-supervised training to improve player and ball detection in soccer.  In _IEEE Int. Conf. Comput. Vis. Pattern Recognit. Work. (CVPRW), CVsports_ , pages 3480–3489, New Orleans, LA, USA, 2022. Inst. Electr. Electron. Eng. (IEEE).
  * Vats et al. [2021] Kanav Vats, Mehrnaz Fani, David A. Clausi, and John Zelek.  Multi-task learning for jersey number recognition in ice hockey.  In _Int. ACM Work. Multimedia Content Anal. Sports (MMSports)_ , page 11–15. ACM, 2021.
  * Veeramani et al. [2018] Balaji Veeramani, John W. Raymond, and Pritam Chanda.  DeepSort: deep convolutional networks for sorting haploid maize seeds.  _BMC Bioinformatics_ , 19(S9), 2018.
  * Wang et al. [2023] Luping Wang, Hao Guo, and Bin Liu.  A boosted model ensembling approach to ball action spotting in videos: The runner-up solution to CVPR’23 SoccerNet challenge.  _arXiv_ , abs/2306.05772, 2023.
  * Wojke et al. [2017] Nicolai Wojke, Alex Bewley, and Dietrich Paulus.  Simple online and realtime tracking with a deep association metric.  In _IEEE Int. Conf. Image Process. (ICIP)_ , pages 3645–3649, Beijing, China, 2017. Inst. Electr. Electron. Eng. (IEEE).
  * Yang et al. [2022] Fan Yang, Shigeyuki Odashima, Shoichi Masui, and Shan Jiang.  The second-place solution for CVPR 2022 SoccerNet tracking challenge.  _arXiv_ , abs/2211.13481, 2022.
  * Ye et al. [2005] Qixiang Ye, Qingming Huang, Shuqiang Jiang, Yang Liu, and Wen Gao.  Jersey number detection in sports video for athlete identification.  In _Visual Communications and Image Processing_ , Beijing, China, 2005. SPIE.
  * Yu et al. [2018] Junqing Yu, Aiping Lei, Zikai Song, Tingting Wang, Hengyou Cai, and Na Feng.  Comprehensive dataset of broadcast soccer videos.  In _IEEE Conf. Multimedia Inf. Process. Retr. (MIPR)_ , pages 418–423, Miami, FL, USA, 2018. Inst. Electr. Electron. Eng. (IEEE).
  * Zhang et al. [2023] Guiwei Zhang, Yongfei Zhang, Tianyu Zhang, Bo Li, and Shiliang Pu.  PHA: Patch-wise high-frequency augmentation for transformer-based person re-identification.  In _IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR)_ , pages 14133–14142, Vancouver, Can., 2023. Inst. Electr. Electron. Eng. (IEEE).
  * Zhang et al. [2022a] Pengyi Zhang, Huanzhang Dou, Yunlong Yu, and Xi Li.  Adaptive cross-domain learning for generalizable person re-identification.  In _Eur. Conf. Comput. Vis. (ECCV)_ , pages 215–232. Springer Nat. Switz., 2022a.
  * Zhang et al. [2021] Yifu Zhang, Chunyu Wang, Xinggang Wang, Wenjun Zeng, and Wenyu Liu.  FairMOT: On the fairness of detection and re-identification in multiple object tracking.  _Int. J. Comput. Vis._ , 129(11):3069–3087, 2021.
  * Zhang et al. [2022b] Yifu Zhang, Peize Sun, Yi Jiang, Dongdong Yu, Fucheng Weng, Zehuan Yuan, Ping Luo, Wenyu Liu, and Xinggang Wang.  ByteTrack: Multi-object tracking by associating every detection box.  In _Eur. Conf. Comput. Vis. (ECCV)_ , pages 1–21. Springer Nat. Switz., 2022b.
  * Zhou et al. [2021] Xin Zhou, Le Kang, Zhiyu Cheng, Bo He, and Jingyu Xin.  Feature combination meets attention: Baidu soccer embeddings and transformer based temporal detection.  _arXiv_ , abs/2106.14447, 2021.
  * Zhu et al. [2022] He Zhu, Junwei Liang, Chengzhi Lin, Jun Zhang, and Jianming Hu.  A transformer-based system for action spotting in soccer videos.  In _Int. ACM Work. Multimedia Content Anal. Sports (MMSports)_ , pages 103–109, Lisbon, Port., 2022. ACM.
  * Zhu-Tian et al. [2023] Chen Zhu-Tian, Qisen Yang, Xiao Xie, Johanna Beyer, Haijun Xia, Yingcai Wu, and Hanspeter Pfister.  Sporthesia: Augmenting sports videos using natural language.  _IEEE Trans. Vis. Comput. Graph._ , 29(1):918–928, 2023.

Supplementary Material

##  A Camera calibration

The estimated camera parameters follow the pinhole camera model augmented with
one radial distortion coefficient that may be needed in wide camera shots. The
camera parameters are estimated in four steps. First, as a global pre-
processing step, the intersections of the pitch markings are computed to
obtain both line-to-line and point-to-point correspondences between the image
and the soccer pitch model. Then, depending on the visible parts of the soccer
pitch in the image, different strategies are used to retrieve camera
parameters. When there is a sufficient amount of pitch markings in an image, a
homography mapping the image plane to the soccer pitch plane is estimated,
then converted into pinhole camera parameters. Moreover, an optimization is
conducted to determine one radial distortion coefficient given the curvature
of the annotated polylines. For the frames that do not display enough pitch
markings, the knowledge that each sequence is shot by a single camera is
leveraged. As broadcast cameras are both zooming and rotating, only the camera
position can be considered fixed. It is estimated as the median 3D position of
the camera parameters estimated in the previous step. A similar version of the
Two-Point PTZ algorithm is used to compute the focal length, pan and tilt
parameters. Finally, as there are still some frames that can not be calibrated
with sufficient accuracy, an industrial tool is used to compute the camera
parameters of the missing frames.

Some examples of the pitch annotations used for the camera calibration can be
found in Fig. S2.

Figure S1: Distribution of the acc@5 metric for the different sets

Figure S2: Pitch annotations. Examples of pitch annotations.

##  B GS-HOTA additional discussion

The HOTA authors introduced a “Classification-Aware HOTA” that shares
similarities with our proposed GS-HOTA. However, the Classification-Aware HOTA
is not suitable for evaluating game state reconstruction for several reasons:
first, it imposes a less rigid constraint on class predictions than Eq. 5,
second, it is tailored for a single classification objective, and third, it
necessitates to output one classification score for each potential class, all
summing up to one, an unsuitable requirement for team affiliation and jersey
number recognition.

##  C Comparison with Standard MOT Methods.

We present the performance of the ‘image tracking only’ component of our
baseline, which includes the detector, ReID, and tracking modules, to compare
with existing SOTA Multi-Object Tracking (MOT) methods. To this end, we employ
two well-established metrics: MOTA [5] and HOTA [57]. Results in Tab. S1
reveal our superior performance over methods using non-fine-tuned object
detectors. Finally, a specialized soccer tracking method such as [60]
highlights the potential for improvement in image-based tracking. This method
relies on a strong object player detector fine-tuned on soccer data, and a
heavy test-time fine-tuning of a ReID model to associate short tracklets into
long tracks and achieve long-term tracking.

Table S1:  GSR-Baseline on SoccerNet Tracking [21]. FT Det indicate an object
detector fine-tuned on SoccerNet. GSR-B uses an out-of-the-box YOLOv8, not
fine-tuned on soccer data.

Algorithm | FT Det | HOTA | DetA | AssA | MOTA
---|---|---|---|---|---
DeepSORT [92] |  | 36.66 | 40.02 | 33.76 | 33.91
FairMOT [98] |  | 43.91 | 46.32 | 41.78 | 50.70
ByteTrack [99] |  | 47.23 | 44.49 | 50.26 | 31.74
GSR-B (ours) |  | 57.64 | 67.42 | 49.42 | 80.79
FairMOT-ft [98] | ✓ | 57.88 | 66.56 | 50.49 | 83.56
SNT23-Winners [60] | ✓ | 73.29 | 73.26 | 73.42 | 87.74

##  D Annotation sample

An annotation sample in JSON format is illustrated in Fig. S3 for a single
video.

⬇

1{

2 "info":{

3 "version":"1.1",

4 "game_id":"11",

5 "id":"200",

6 "num_tracklets":"20",

7 "action_position":"956196",

8 "action_class":"Shots on target",

9 "visibility":"visible",

10 "game_time_start":"2 - 15:41",

11 "game_time_stop":"2 - 16:11",

12 "clip_start":"941000",

13 "clip_stop":"971000",

14 "name":"SNGS-200",

15 "im_dir":"img1",

16 "frame_rate":25,

17 "seq_length":750,

18 "im_ext":".jpg"

19 },

20 "images": [

21 {

22 "is_labeled":true,

23 "image_id":"3200000001",

24 "file_name":"000001.jpg",

25 "height":1080,

26 "width":1920,

27 "has_labeled_person":true,

28 "has_labeled_pitch":true,

29 "has_labeled_camera":true,

30 "ignore_regions_y":[],

31 "ignore_regions_x":[]

32 },

33 // Additional images annotations...

34 ],

35 "annotations":[

36 {

37 "id":"3200000001",

38 "image_id":"3200000001",

39 "track_id":1,

40 "supercategory":"object",

41 "category_id":1,

42 "attributes":{

43 "role":"player",

44 "jersey":"14",

45 "team":"left"

46 },

47 "bbox_image":{

48 "x":1020,

49 "y":508,

50 "x_center":1043.0,

51 "y_center":557.5,

52 "w":46,

53 "h":99

54 },

55 "bbox_pitch":{

56 "x_bottom_left":-29.17307773076183,

57 "y_bottom_left":-13.960906317008366,

58 "x_bottom_right":-28.399824812615115,

59 "y_bottom_right":-14.278786952621587,

60 "x_bottom_middle":-28.786446826184775,

61 "y_bottom_middle":-14.119801608871501

62 }

63 },

64 // Additional athletes annotations...

65 ...

⬇

66 ...

67 {

68 "id":"3200000019",

69 "image_id":"3200000001",

70 "supercategory":"pitch",

71 "category_id":5,

72 "lines": {

73 "Side line top":[{"x":0.21, "y":0.34}, {"x":0.61, "y":0.39}, {"x":1.0,
"y":0.43}],

74 "Side line left":[{"x":0.0, "y":0.45}, {"x":0.10, "y":0.39}, {"x":0.21,
"y":0.34}],

75 "Small rect. left top":[{"x":0.07, "y":0.53}, {"x":0.01, "y":0.53},
{"x":0.01, "y":0.53}],

76 "Small rect. left main":[{"x":0.0, "y":0.54}, {"x":0.01, "y":0.54},
{"x":0.01, "y":0.53}],

77 "Big rect. left top":[{"x":0.04, "y":0.42}, {"x":0.23, "y":0.45},
{"x":0.41, "y":0.48}],

78 "Big rect. left main":[{"x":0.0, "y":0.81}, {"x":0.20, "y":0.65},
{"x":0.41, "y":0.48}],

79 "Circle left":[{"x":0.02, "y":0.79}, {"x":0.04, "y":0.79}, ...],

80 }

81 }

82 // Additional pitch annotations...

83 ]

84}

Figure S3: Sample JSON annotation for one video of the SoccerNet-GSR dataset