# PlayerTV: Advanced Player Tracking and Identification for Automatic Soccer
Highlight Clips

Håkon Maric Solberg1, Mehdi Houshmand Sarkhoosh24, Sushant Gautam23,
Saeed Shafiee Sabet4, Pål Halvorsen1234, Cise Midoglu34  1University of Oslo,
Norway  2Oslo Metropolitan University, Norway  3SimulaMet, Norway  4Forzasys
AS, Norway

###### Abstract

In the rapidly evolving field of sports analytics, the automation of targeted
video processing is a pivotal advancement. We propose PlayerTV, an innovative
framework which harnesses state-of-the-art AI technologies for automatic
player tracking and identification in soccer videos. By integrating object
detection and tracking, Optical Character Recognition (OCR), and color
analysis, PlayerTV facilitates the generation of player-specific highlight
clips from extensive game footage, significantly reducing the manual labor
traditionally associated with such tasks. Preliminary results from the
evaluation of our core pipeline, tested on a dataset from the Norwegian
Eliteserien league, indicate that PlayerTV can accurately and efficiently
identify teams and players, and our interactive Graphical User Interface (GUI)
serves as a user-friendly application wrapping this functionality for
streamlined use.

###### Index Terms:

association football, player tracking, optical character recognition, object
detection, automatic highlight generation

##  I Introduction

Sport video content is consumed in various ways, both for entertainment and
for analysis [1]. In this respect, a lot of manual tedious work is required to
prepare and compile compelling and usable summaries of the potentially long
video assets. Modern technologies enable automatic computer vision and
artificial intelligence (AI) based solutions to make more efficient production
pipelines [2], e.g., using automatic event detection [3], clipping [4], video
aspect ratio retargeting (cropping) [5], and using the available multimodal
data to create text summaries [6].

Using soccer as an example, we aim to push the automation one step further and
allow the creation of video clips which focus on individual athletes. In this
context, the development and use of advanced player tracking technologies is
emerging as a cornerstone to enhance tournament strategies, player
performance, game analytics and fan engagement. With our proposed framework
PlayerTV, we deploy state-of-the-art AI models to track the players in video
frames, perform player identification through team detection and kit number
identification (relying on color analysis and OCR, resolving occlusion and
collision situations in the process), and clip parts of the input video, to
extract frames containing the target player, i.e., generating a sophisticated
video processing pipeline to enable player specific soccer highlight clips. We
hope that our proof-of-concept demonstration of the technological advantages
of such a framework can open up new avenues for applied research, especially
in the areas of computer vision for player performance analysis, automatic
multimedia content for fan experiences and highlights. Our contributions are
as follows:

  * •

We present a framework called PlayerTV for automatically creating per player
highlight clips from input videos using object detection and tracking, OCR,
unsupervised clustering, and database lookup. The core pipeline comprises 5
modules (player tracking, RGB, OCR, team mapping, player mapping) and is
highly configurable.

  * •

An interactive GUI serves as a wrapper around the core pipeline, making
PlayerTV a user-friendly application where users can specify an input video
and configuration parameters, and retrieve a clipped video including only the
scenes of a selected player.

  * •

We provide an open source implementation for the entire framework, along with
additional artifacts.

  * •

We present initial experiment results serving as a preliminary evaluation of
the performance of the pipeline under different configurations.

##  II Background and Related Work

###  II-A Sports Analytics

The domain of sports analytics has witnessed substantial growth, focusing
primarily on enhancing team performance, refining strategic approaches, and
improving player evaluation [7, 8]. Professional sports teams invest
significantly in analytical tools to gain competitive advantages [9, 10].
Sports analytics platforms [11, 12] are extensively utilized in professional
soccer for detailed player and game analysis. Currently, much of the data
annotation involved in sports analytics remains manual, a labour-intensive
process that limits the speed and scalability of data utilization [13, 14].
There is a compelling case for automation in this field, which could not only
streamline existing processes but also enable advanced, real-time analytical
capabilities, thus transforming the landscape of sports analytics [15]. Other
interesting research includes spotting key game events such as goals or fouls
[16, 17], summarizing games automatically [6], multimodal game understanding
[18, 19], creating exciting highlights [2], and generating captions for game
footage [20, 21].

###  II-B Object Detection and Tracking

In the realm of sports analytics, object detection and tracking algorithms are
indispensable for the automatic localization and movement analysis of players
and balls across video frames [22]. YOLO [23] excels in high-speed detection
through its unique approach to processing images in a single evaluation,
making it especially suitable for dynamic sporting events. SSD [24], on the
other hand, achieves a commendable balance between speed and accuracy by
employing a deep neural network to detect multiple objects simultaneously. For
object tracking, SORT [25] and DeepSORT [26] offer robust solutions, enhancing
tracking consistency by integrating deep learning to manage the dynamic nature
of sports environments. The Deep-EIoU [27] algorithm improves tracking
precision through iterative alignment using an advanced Intersection-over-
Union (IoU) operation and deep features, essential for handling fast-moving
and overlapping objects. These technological advancements are pivotal in
enhancing the automation of data annotation and enabling sophisticated real-
time analysis [28], thereby aiding in the formulation of immediate and
strategic decisions in sports analytics.

###  II-C Optical Character Recognition (OCR)

OCR converts different types of documents into editable and searchable data
and has significantly advanced due to improvements in machine learning and
image processing [29]. In sports analytics, OCR is utilized to analyze game
footage by extracting jersey (kit) numbers and statistics from scoreboards
[30, 31, 32, 33]. This technology automates real-time decision-making and
historical data analysis, proving vital for coaching strategies and
performance assessment [34]. Future developments in OCR may focus on
increasing accuracy in dynamic environments [35], offering profound impacts on
sports analytics and media broadcasting.

###  II-D Player Tracking in Sports

The evolution of player tracking technologies in sports, particularly in
soccer, illustrates a significant technological advancement in the realm of
sports analytics [27]. These technologies have transitioned from basic manual
methods to highly sophisticated systems, encompassing both optical tracking
and sensor-based systems [36]. Optical tracking systems utilize high-
definition cameras positioned around the playing field to capture
comprehensive visual data [13], which is then processed to track player and
ball movements across the pitch. Sensor-based systems, on the other hand,
involve the use of wearable technologies embedded in player gear, providing
real-time data on player positions, movements, and even biometrics [37].

Importance of tracking information: Accurate game data, such as player
positions and ball location, is indispensable in modern sports science [8].
This data underpins not only tactical and strategic decisions made by coaching
staff but also enhances sports science research focused on performance
enhancement and injury prevention [8, 38]. Furthermore, tracking information
enriches fan engagement by offering detailed game analytics, contributing to
more immersive viewing experiences and interactive fan involvement [39].
Through the use of comprehensive tracking data, teams can optimize their
strategies, improve player conditioning programs, and enhance the overall
spectator experience by providing insights that were previously inaccessible.

Challenges in player tracking: Accurately tracking players in sports settings
presents several distinct challenges [10, 22]. First, occlusion issues arise
frequently, where players or the ball are obscured by other players, leading
to temporary loss of tracking data [13]. Secondly, varying lighting
conditions, such as shadows or stadium lights, can significantly alter the
visibility and appearance of players [40, 22], complicating the tracking
process. Additionally, the fast-paced and unpredictable movements typical of
sports scenarios challenge the tracking algorithms’ ability to maintain
consistent tracking. These movements often result in motion blur, which can
degrade the quality of the visual data used for tracking. State-of-the-art
tracking algorithms, such as those employing Kalman filters, are designed for
objects moving in smooth, predictable trajectories and thus struggle with the
erratic movement patterns found in sports [13, 26, 25, 41, 42]. Research in
player Re-identification (Re-ID) techniques focuses on overcoming the
challenges posed by changes in appearance or occlusions in sports
environments, aiming to accurately track and identify players across different
situations [43]. These studies delve into innovative methods to enhance
recognition accuracy, crucial for various applications including player
performance analysis and sports event security.

Data integration and visualization: Previous efforts have centered on
seamlessly integrating tracking data with game metadata, enabling
comprehensive analysis and enhancing broadcast experiences [43, 32]. By
visualizing this integrated information, stakeholders can gain deeper insights
into player movements, team strategies, and overall game dynamics, fostering a
richer understanding of sporting events [5, 19].

Improvements in all the underlying technologies point towards the
transformative potential of advanced player tracking and AI-driven analysis in
the realm of sports, especially soccer, by refining tactics and enriching fan
experiences. However, challenges like occlusion, varying lighting, and
unpredictable player movements still hinder the accuracy of these
technologies. Our aim is to overcome these limitations by developing
innovative methods that enhance the precision and functionality of player
tracking, paving the way for advanced sports analytics solutions.

Figure 1: PlayerTV framework overview.

###  II-E Motivation and Potential Use Cases for PlayerTV

There are various use cases for the PlayerTV framework:

  * •

Tactical analysis: coaches and analysts can use PlayerTV for in-depth tactical
analysis, identifying patterns in player movements, and optimizing team
strategies.

  * •

Performance analysis and player development: tracking data can help in
evaluating player performances, designing personalized training programs, and
monitoring progress over time.

  * •

Broadcast enhancements: tracking data can be incorporated to provide viewers
with enriched content, such as live player statistics, heat maps, and tactical
overlays.

  * •

Fan engagement and interactive experiences: tracking data can be used to
create immersive fan experiences, such as augmented reality (AR) applications
that overlay player stats and movements in real-time during a match.

  * •

Betting and fantasy sports: tracking data can provide insights for betting
odds and fantasy sports, offering detailed analytics to users for making
informed decisions.

  * •

Injury prevention and management: analyzing player movements and workload can
aid in identifying potential injury risks and managing player fitness levels.

##  III Proposed Framework

The PlayerTV framework consists of a core pipeline and an interactive GUI that
serves as a wrapper around the core pipeline. Figure 1 presents an overview of
the framework, where the dashed line indicates the core pipeline and the solid
line indicates the wrapper.

###  III-A Core Pipeline

The PlayerTV core pipeline comprises 5 modules, takes a JSON configuration
file as input, and outputs a JSON file.

####  III-A1 Player Tracking Module

For tracking different players across between frames, we have taken the Deep-
EIoU tracker [27] as a starting point. This tracker has been shown to
outperform DeepSORT [26], BoT-SORT [41] and OC-SORT [42] for Multi-Object-
Tracking (MOT) in sports. The tracking module supports using a YOLOX [44]
model that has been trained on the SoccerNet tracking dataset [43], as well as
SportMOT [45]. Figure 2 presents sample "tracklet"s (crops of detected
players), which are still challenging to use for direct player identification
due to noise and occlusions. We have modified the tracker to also extract IoU
and BRISQUE-score [46] from the tracklets. In addition to this, we also keep
an average color value from a region of interest within each crop. These are
later used for mapping players to teams.

Figure 2: Sample tracklets. Both noise from other players as well as occlusion
makes it difficult to correctly detect kit numbers.

####  III-A2 RGB Module

The RGB information retrieved from the region of interest within the tracklets
are kept for later clustering into the 2 different teams. As there is a lot of
grass (green) surrounding each player, this would blend out the colors, as we
are trying to capture values closest to the players kit color. By using an
offset where we only look at the top half, while squeezing in the sides and
top of the detection, we retrieve a more distinct color, creating more
distance between samples of different teams. Figure 3 presents examples of
these regions of interest. The offset proved to be most effective for
clustering, is a clipping removing $\frac{1}{3}$ of the crop on both sides,
$\frac{1}{4}$ of the top and $\frac{1}{2}$ of the bottom.

Figure 3: Sample regions of interest within tracklets.

All RGB values for detections with no intersections to other detections are
used as a sample base. After normalizing, then clustering, we use the distance
from cluster centers to map tracked player to the correct team, unless their
RGB were used for clustering itself. Team home color is retrieved from the
PLAYER DB, and the cluster center closest to it is mapped accordingly.

When extracting the team colors we provide 2 options for the color space:
CIELAB111https://en.wikipedia.org/wiki/CIELAB_color_space and weighted RGB.
CIELAB is a color space consisting of L* for lightness, a* from green to red,
and b* from blue to yellow. This color space is designed to better mimic human
perception. A rough estimation to this is using the weighted RGB, where the
rgb channels are weighted 0.3, 0.59, and 0.11, respectively. In this stage, an
IoU threshold can also be introduced, where only the tracklets with low IoU
(for which the sum of all overlaps with other tracklets are below the
specified threshold) are considered, instead of all tracklets. Then, we run
k-means clustering with k=2 (Figure 4).

Figure 4: Sample clustering. Illustrated in RGB space.

####  III-A3 OCR Module

An additional step for identifying the player is to extract the kit number.
This module supports using two alternative methods on detected crops:
PaddleOCR [47] or EasyOCR [48]. To increase the probability of getting good
results from the OCR, we run through the best frames for each tracked player.
The scoring function for a detection $d$ is:

$score_{d}=(d.BRISQUE*d.IoU)+d.BRISQUE$ (1)

A lower BRISQUE score is associated with a better image quality, we therefore
try minimize this value. When there is no overlap with another detection in
the frame, only BRISQUE accounts for the score. The use of OCR could
potentially confuse two detections that overlap, giving another detection’s
kit number as output. We therefore value a small overlapping area greatly.

When using this score function for deciding which crops to run OCR detecion
on, crops close to each other tend to be high-scoring, while very similar. To
increase the probability of correctly running character recognition for crops
with kit number present, we divide the player’s frames into different time
zones, with each time zone containing the top scoring crops in descending
order. This to spread out the player kit number detection in time.

By splitting a player’s frames into 5 different time segments, each containing
a maximum 10 proposed frames. This is to make sure we use samples spread
across different time periods. When a detection is above a certain threshold,
we skip the rest of the samples within that same time window. After this
process we are set with a set of proposed kit numbers associated with each
track ID.

In addition to splitting each player tracklet into time zones, we also provide
different paradigms for running either of the OCR model on crops:

  * •

All tracklets, keep top 20: This is an offline OCR extraction method where we
keep a stack of the top n scoring crops, running OCR after the tracker is
finished.

  * •

OCR output confidence threshold: Running OCR on all crops with a score above a
threshold.

Also a "Stop at detection" parameter decides whether or not we want to run
further kit number detection for tracklets where a detection has been found.
When split into time zones, this resets for each time zone.

####  III-A4 Team Mapping Module

In this module, home and away team information (where applicable222If the
pipeline is run with an input video from our partner company Forzasys
(https://forzasys.com/), the pipeline can use the corresponding game id in the
M3U8 playlist URL for fetching home and away team information from the
Forzasys API.) are used for a kit color lookup. For this, we use a TEAM DB
where we have mapped each Eliteserien team to two RGB values, corresponding to
home and away kit colors, as follows:

<team_name><home_kit><R,G,B><away_kit><R,G,B>

The cluster closest to the home team color is then mapped accordingly, giving
a tracklet its associated team name, depending on its cluster id.

####  III-A5 Player Mapping Module

Based on the OCR and team mapping results, we identify, per tracklet, a player
entity. For this, we use a PLAYER DB where we have mapped each player in each
Eliteserien team to their corresponding position and kit number within the
team.

<team_name><player_name><position><kit_number>

As there is only one kit number per player per team, collisions are handled
according to team vs kit number confidence. If a kit number has higher
confidence than the team affiliation, we override the team mapping if the kit
number detected is unused for the other team. Vice versa, we decide kit number
using a less confident kit detection if present, and the colliding tracklet is
more confident. This module provides the final online team/kit affiliation per
tracklet (output.json in Figure 1).

<frame_number><tracker_id><x><y><w><h>
<conf>[<r><g><b>] <brisque_score><IoU><team_id>
<kit_number>

.1  output.json . .2 "metadata" . .3 "id": <id> . .3 "game_id": <game_id> . .3
"clip_id": <clip_id> . .3 "width": <width> . .3 "height": <height> . .3 "fps":
<fps> . .3 "video_url": <video_url> . .3 "api_info" . .4 "home_team":
<home_team> . .4 "home_color": <home_color> . .4 "away_team": <away_team> . .4
"away_color": <away_color> . .4 "home_team_idx": <home_team_idx> . .4
"away_team_idx": <away_team_idx> . .4 "home_team_player" . .5  . .6 "name":
<player_name> . .6 "shirt_number": <shirt_number> . .5 … . .4
"away_team_player" . .5  . .5 … . .2 "frame" . .3 <frame_number> . .4
<track_id> . .5 "frame": <frame_number> . .5 "track_id": <track_id> . .5
"xywh": . .6 <x> . .6 <y> . .6 <width> . .6 <height> . .5 "conf": <conf> . .5
"avg_rgb": <avg_rgb> . .5 "brisque_score": <brisque_score> . .5 "iou": <iou> .
.5 "team_id": <team_id> . .5 "kit_number": <kit_number> . .4 … . .3 … .

Figure 5: Structure of output.json.

Figure 5 presents the structure of the output.json file.

###  III-B Interactive GUI

The PlayerTV GUI wraps around the core pipeline. Built upon the Flask web
framework, this GUI provides an interactive platform for visual applications
of the core pipeline. Central to its functionality, Flask plays a pivotal role
in seamlessly integrating front-end user interactions with the back-end
processing. This integration is key to delivering a smooth and responsive user
experience. The GUI’s front-end design, created using HTML and CSS, offers an
easy-to-navigate interface, thereby significantly enhancing user engagement
and operational efficiency. Overall, the GUI not only enables users to run the
core pipeline and clip soccer videos based on selected players but also
displays an overlay indicating the location within frames where the target
player has been identified.

####  III-B1 Video Input

In the first step, PlayerTV is configured to process HTTP Live Streaming (HLS)
playlists, specifically from live servers. This functionality is distinct in
its capacity to handle the M3U8 format. While the overall architecture of the
system, encompassing both backend and frontend, has the capability to manage
local MP4 video paths, this module focuses exclusively on M3U8. The choice of
this format is integral to the application’s functioning, as it facilitates
internal API calls. These calls play a crucial role in extracting detailed
information about the soccer teams in the live stream, such as identifying
home and away teams in a given match. This specific information is accessible
through public APIs provided by Forzasys, which our system leverages to
enhance the user experience and provide relevant context within the
application (see Figure 6).

Figure 6: PlayerTV GUI step 1.

####  III-B2 Configuration

The second step is dedicated to the configuration of the core pipeline. This
section works closely with the backend to provide the user with options to
select different methods and metrics for various modules. Minimal (Figures 7)
or advanced (8) configuration options can be used, depending on whether the
user wants to retain or update system defaults. Table I presents a list of
selected core pipeline configuration parameters. The parameters in the GUI
serve as a user-friendly interface to set these core parameters.

  * •

RGB Metric: The user has the option to choose between two metrics for
clustering and identifying teams: Cielab and a weighted RGB metric. Each
method focuses on clustering based on the average RGB color of the bounding
boxes around detected players, helping to distinguish the home team from the
away team.

  * •

OCR Method: Users can choose between PaddleOCR and EasyOCR for the
identification of players’ kit numbers. These options facilitate the
recognition of specific player details.

  * •

Detector: The application offers a choice between YOLOv8 and YOLOvX models for
object detection. Both models have been fine-tuned separately on a soccer
dataset, encompassing various classes, with a focus on player detection.

Metadata: In this section, users can select from teams and players that
PlayerTV’s backend has identified. This feature is based on our algorithms for
detection, re-identification, and tracking. PlayerTV maintains a database of
the Eliteserien 2023 league’s teams and players. Selecting a team from the API
call presents a list of players matching our OCR detection. Users can then
choose their preferred player for video clipping. Additionally, by selecting a
team, users can view the home and away jerseys on the left sidebar, and by
selecting a player, the player’s photo is displayed. This visualize the player
which we are going to track and clip videos.

After selecting the desired options, pressing the "Clip Video" runs the
PlayerTV core pipeline, and ultimately clips the input video based on the
selected player, showcasing only the relevant frames.

Figure 7: PlayerTV GUI step 2.

Figure 8: PlayerTV GUI step 2 (advanced settings). Module | Configurable Parameters | Options
---|---|---
Tracking | Object detection model | YOLOX, YOLOv8
DeepEIoU parameters | (mult.)
Appearance threshold |  Default: $0.25$
Obj. det. confidence threshold |  Default: $0.25$
RGB | Color space | RGB, CIELAB, Weighted RGB
Clustering method | KMeans
Offset type | trim, center, none
Normalization | TRUE/FALSE
Scaling | TRUE/FALSE
IoU threshold | [0, 1]
OCR |  OCR method | PaddleOCR, EasyOCR
Crop score function | iou_score, iou_area, iou_random
Get OCR function |  top n, split into time zones
Stop at detection | TRUE/FALSE
TABLE I: PlayerTV core pipeline selected parameters.

####  III-B3 Output

The output of this step is an MP4 file, which retains the same frame rate
(FPS) as the original video. This file exclusively displays the frames where
the selected player is detected and tracked. In these frames, the player is
highlighted with a bounding box. Additionally, the player’s name and jersey
number are displayed above this bounding box for easy identification (see
Figure 9).

Figure 9: PlayerTV GUI step 3.

###  III-C Open Artifacts

The entire codebase for the PlayerTV framework, including the core pipeline
and the interactive GUI is publicly available as open source software under
[49]. The repository also includes additional assets (team and player lookup
tables), as well as a README which provides an overview of the software and
hardware requirements, technical implementation details, and instructions on
how to replicate the complete framework in an end-to-end manner. A video
demonstration of the interactive GUI is available on YouTube [50].

##  IV Evaluation

| Team Mapping Performance (%) | OCR Performance (%) | Combined (%)
---|---|---|---
| W. RGB | CIELAB | Paddle | Easy | Paddle | Easy
Video 1 | 55.6 / 66.7 | 77.8 / 85.2 | 23.8 (33.3) | 14.3 (14.3) | 23.8 | 14.3
Video 2 | 56.8 / 70.3 | 83.8 / 86.5 | 39.1 (43.5) | 17.4 (17.4) | 39.1 | 17.4
Video 3 | 96.7 / 100.0 | 96.7 / 100.0 | 20.0 (20.0) | 0 (0) | 20.0 | 0.0
Video 4 | 95.0 / 100.0 | 95.0 / 100.0 | 41.2 (50.0) | 5.9 (5.9) | 41.2 | 5.9
Video 5 | 94.9 / 100.0 | 92.3 / 100.0 | 40.0 (40.0) | 10.0 (10.0) | 40.0 | 10.0
Video 6 | 93.8 / 93.8 | 100.0 / 100.0 | 46.2 (53.8) | 0.0 (7.7) | 46.2 | 0.0
Video 7 | 98.0 / 100.0 | 98.0 / 98.0 | 29.2 (29.2) | 4.2 (4.2) | 29.2 | 4.2
Video 8 | 94.6 / 94.6 | 94.6 / 94.6 | 22.7 (22.7) | 4.5 (4.5) | 22.7 | 4.5
Video 9 | 76.2 / 71.4 | 100.0 / 95.2 | 0 (0) | 0 (0) | 0 | 0
Video 10 | 97.7 / 100.0 | 90.7 / 95.3 | 12.5 (12.5) | 6.2 (6.2) | 12.5 | 6.2
Video 11 | 100.0 / 100.0 | 100.0 / 100.0 | 38.5 (38.5) | 7.7 (7.7) | 38.5 | 7.7
Video 12 | 100.0 / 100.0 | 100.0 / 100.0 | 42.9 (50.0) | 17.9 (17.9) | 42.9 | 17.9
Video 13 | 100.0 / 96.2 | 96.2 / 96.2 | 27.8 (27.8) | 5.6 (11.1) | 27.8 | 5.6
Video 14 | 92.9 / 100.0 | 100.0 / 100.0 | 62.5 (75.0) | 62.5 (62.5) | 62.5 | 62.5
Video 15 | 85.0 / 90.0 | 100.0 / 90.0 | 33.3 (33.3) | 11.1 (22.2) | 33.3 | 11.1
Video 16 | 87.5 / 87.5 | 100.0 / 93.8 | 33.3 (33.3) | 33.3 (33.3) | 33.3 | 33.3
Overall | 87.6 / 89.8 | 91.5 / 93.7 | 30.6 (35.7) | 11.3 (12.3) | 29.8 | 11
TABLE II: PlayerTV evaluation. Figure 10: Sample frame from Video 9 -
Challenging light conditions make weighted RGB perform worse than CIELAB.

For evaluating the PlayerTV core pipeline, we have taken the tracker output as
baseline for tracklet ground truths. Running through all tracklets in a
900-frame sequence video, before manually labeling each with team name as well
as kit number. To ensure the integrity of the data annotated, filtering was
applied. We retained only those player tracklets that consistently maintained
focus on a single player across frames. Additionally, not all tracklets have a
visible kit number throughout the clip. Such tracklets have been labeled "-1"
and are excluded from the OCR performance metrics. A total of 16 videos from
the Norwegian top division "Eliteserien" have been manually annotated. This
resulted in a sample base consisting of 511 instances of player tracklets,
each mapped to a specific team. A subset of these 511 tracklets have visible
kit numbers throughout the clip, resulting in 300 tracklets annotated with
both team and kit number.

Dataset creation and evaluation process is as follows:

  1. 1.

Default Deep-EIoU tracker is run for clips, providing a file with tracklet
information.

  2. 2.

Each individual tracklet with tracking id is mapped to correct team and
number. Inconsistent tracklets are discarded. This results in sequences of
crops of varying lengths, each consistently featuring the same player.

  3. 3.

By combining these mappings, we evaluate the RGB module and OCR module
individually, as well as their combination.
RGB Module: Accuracy measurement for correctly mapped team from player
tracklet. Both for CIELAB and weighted RGB color space.
OCR Module: Accuracy as well as top-5 accuracy (in parentheses) for each
player tracklet.
Combined: Accuracy where both team and kit number are correct for a tracklet.

  4. 4.

We evaluate the same metrics for post processed results as well as results
achieved in online mode. Table II presents the results of our evaluation,
combined columns indicate PaddleOCR and EasyOCR with CIELAB.

###  IV-A RGB Module Performance

This module is the most accurate. Not surprising considering its task. Using
the CIELAB color space also improved correct team labeling by 3.9% compared to
using the weighted RGB metric. Post processing the results, increased
performance by an additional 2%. These findings suggests that relatively
straightforward measures, as mean color value, could be effective. This
approach could serve as a robust alternative to employing more complex machine
learning modules for processing player crops. Figure 10 shows an example frame
from Video 9 in Table II, revealing the limitations of the weighted RGB
approach. In the challenging light conditions, the CIELAB approach is perfect
for the online clustering, while the weighted RGB struggles having an accuracy
of only 76.2.

###  IV-B OCR Module Performance

The main constraint within this pipeline is the kit association, which peaks
at just over 30%. The the gap in performance between the EasyOCR vs PaddleOCR
models is notable, with PaddleOCR outperforming by a factor of three.
Specifically, within our dataset of 300 tracklets, the PaddleOCR model, ranks
the correct kit number within the top 5 detections for 107 instances. In 92 of
these 107 instances, the correct number appears as the top-ranked detection.
To further refine our results, reducing the disparity between these measures
is crucial. A possible improvement could involve assigning additional weight
to the detection’s position within the crop itself.

Given that both the EasyOCR and PaddleOCR models were initially developed for
purposes slightly different from their current usage in this pipeline,
training a custom detector in the future is desirable. Another factor to
consider when reading the OCR performance is also the length of the clips
provided. If clips span over a longer period of time, the OCR module will have
more chances of detecting a kit number for a tracklet, improving performance.

###  IV-C Team and Player Mapping Performance

As the OCR detection module is the limiting factor, we see that the combined
performance is marginally worse than the OCR module independently. Given this
situation, it is reasonable to assume that enhancements in OCR accuracy would
lead to proportional improvements in the pipeline’s overall performance.
Improvements for the OCR component could therefore significantly improve
overall performance.

###  IV-D System Performance

In order to evaluate system performance, we consider the end-to-end pipeline
processing speed. The PlayerTV pipeline was run on a computational cluster
known as ex3333https://www.ex3.simula.no/. Performance measures was carried
out on a node equipped with an NVIDIA Tesla V100-SXM3 GPU. It is worth noting
that the traffic on this cluster varies, influencing performance slightly. Our
pipeline, using a greedy OCR approach that involves running EasyOCR across all
crops in all frames, produced results at a rate of approximately 1 frame per
second (fps).

##  V Conclusion

We have introduced PlayerTV, a framework which combines technologies such as
object detection and tracking, text recognition, and color analysis to
automate the creation of player-specific video highlights. Our evaluations
show promising accuracy in player and team identification, which substantiates
the framework’s utility in professional sports analytics. As we continue to
refine the technology and expand its applications, PlayerTV is poised to
transform how sports footage is processed and utilized, enhancing both the
strategic elements of sports team management and the overall fan experience.

## Acknowledgment

This research was partly funded by the Research Council of Norway, project
number 346671 (AI-Storyteller), and has benefited from the Experimental
Infrastructure for Exploration of Exascale Computing (eX3), which is
financially supported by the Research Council of Norway under contract 270053.
The authors would like to thank the Norwegian Professional Football League
("Norsk Toppfotball") for making videos available for research.

## References

  * [1] M. H. Sarkhoosh _et al._ , “Soccer on Social Media,” 2023. [Online]. Available: https://arxiv.org/abs/2310.12328
  * [2] C. Midoglu _et al._ , “AI-Based Sports Highlight Generation for Social Media,” in _MHV ’24: Proceedings of the 3rd Mile-High Video Conference_. New York, NY, USA: Association for Computing Machinery, Feb. 2024, pp. 7–13.
  * [3] O. A. Nergård Rongved _et al._ , “Automated Event Detection and Classification in Soccer: The Potential of Using Multiple Modalities,” _Mach. Learn. Knowl. Extr._ , vol. 3, no. 4, pp. 1030–1054, Dec. 2021.
  * [4] J. O. Valand _et al._ , “Automated Clipping of Soccer Events using Machine Learning,” in _2021 IEEE International Symposium on Multimedia (ISM)_. IEEE, 2021, pp. 2021–01.
  * [5] S. M. M. Dorcheh _et al._ , “SmartCrop: AI-Based Cropping of Soccer Videos,” in _2023 IEEE International Symposium on Multimedia (ISM)_ , 2023, pp. 20–27.
  * [6] S. Gautam _et al._ , “Soccer Game Summarization using Audio Commentary, Metadata, and Captions,” in _NarSUM ’22: Proceedings of the 1st Workshop on User-centric Narrative Summarization of Long Videos_. New York, NY, USA: Association for Computing Machinery, Oct. 2022, pp. 13–22.
  * [7] J. Garganta, “Trends of tactical performance analysis in team sports: bridging the gap between research, training and competition.” _Revista Portuguesa de Ciências do desporto_ , vol. 9, no. 1, 2009.
  * [8] R. Rein and D. Memmert, “Big data and tactical analysis in elite soccer: future challenges and opportunities for sports science,” _SpringerPlus_ , vol. 5, no. 1, pp. 1410–13, Aug. 2016.
  * [9] A. D. Pizzo _et al._ , “The strategic advantage of mature-stage firms: Digitalization and the diversification of professional sport into esports,” _Journal of Business Research_ , vol. 139, pp. 257–266, Feb. 2022.
  * [10] L. Torres-Ronda _et al._ , “Tracking Systems in Team Sports: A Narrative Review of Applications of the Data and Sport Specific Analysis,” _Sports Med. - Open_ , vol. 8, no. 1, pp. 15–22, Jan. 2022.
  * [11] “Wyscout - The world’s biggest library of football, Hudl,” Apr. 2024. [Online]. Available: https://www.hudl.com/en\\_gb/products/wyscout
  * [12] “SportsBase,” Feb. 2024. [Online]. Available: https://sportsbase.world
  * [13] J. Liu _et al._ , “A survey on location and motion tracking technologies, methodologies and applications in precision sports,” _Expert Syst. Appl._ , vol. 229, p. 120492, Nov. 2023.
  * [14] C. Midoglu _et al._ , “MMSys’22 Grand Challenge on AI-based Video Production for Soccer,” 2022. [Online]. Available: https://arxiv.org/abs/2202.01031
  * [15] D. Patel, D. Shah, and M. Shah, “The Intertwine of Brain and Body: A Quantitative Analysis on How Big Data Influences the System of Sports,” _Ann. Data. Sci._ , vol. 7, no. 1, pp. 1–16, Mar. 2020.
  * [16] L. Morra _et al._ , “Slicing and Dicing Soccer: Automatic Detection of Complex Events from Spatio-Temporal Data,” in _Image Analysis and Recognition_. Cham, Switzerland: Springer, Jun. 2020, pp. 107–121.
  * [17] O. A. N. Rongved _et al._ , “Real-Time Detection of Events in Soccer Videos using 3D Convolutional Neural Networks,” in _IEEE International Symposium on Multimedia (ISM)_. IEEE, 2020, pp. 02–04.
  * [18] S. Gautam _et al._ , “Assisting Soccer Game Summarization via Audio Intensity Analysis of Game Highlights,” in _Proc. 12th IOE Graduate Conference_. Inst. of Eng. Tribhuvan University, Nepal, Oct. 2022, pp. 25–32. [Online]. Available: http://conference.ioe.edu.np/publications/ioegc12/IOEGC-12-004-12009.pdf
  * [19] S. Gautam, “Bridging Multimedia Modalities: Enhanced Multimodal AI Understanding and Intelligent Agents,” in _Proceedings of the 25th International Conference on Multimodal Interaction_ , ser. ICMI ’23. New York, NY, USA: Association for Computing Machinery, 2023, p. 695–699.
  * [20] H. Mkhallati _et al._ , “SoccerNet-Caption: Dense Video Captioning for Soccer Broadcasts Commentaries,” in _2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)_. IEEE, 2023, pp. 17–24.
  * [21] J. Qi _et al._ , “GOAL: A Challenging Knowledge-grounded Video Captioning Benchmark for Real-time Soccer Commentary Generation,” in _CIKM ’23: Proceedings of the 32nd ACM International Conference on Information and Knowledge Management_. New York, NY, USA: Association for Computing Machinery, Oct. 2023, pp. 5391–5395.
  * [22] B. T. Naik, M. F. Hashmi, and N. D. Bokde, “A Comprehensive Review of Computer Vision in Sports: Open Issues, Future Trends and Research Directions,” _Appl. Sci._ , vol. 12, no. 9, p. 4429, Apr. 2022.
  * [23] J. Redmon _et al._ , “You Only Look Once: Unified, Real-Time Object Detection,” in _2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_. IEEE, 2016, pp. 27–30.
  * [24] W. Liu _et al._ , “SSD: Single Shot MultiBox Detector,” in _Computer Vision – ECCV 2016_. Cham, Switzerland: Springer, Sep. 2016, pp. 21–37.
  * [25] N. Wojke, A. Bewley, and D. Paulus, “Simple online and realtime tracking with a deep association metric,” in _2017 IEEE International Conference on Image Processing (ICIP)_. IEEE, 2017, pp. 17–20.
  * [26] B. Veeramani, J. W. Raymond, and P. Chanda, “DeepSort: deep convolutional networks for sorting haploid maize seeds,” _BMC Bioinf._ , vol. 19, no. 9, pp. 289–9, Aug. 2018.
  * [27] H.-W. Huang _et al._ , “Iterative Scale-Up ExpansionIoU and Deep Features Association for Multi-Object Tracking in Sports,” in _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_ , 2024, pp. 163–172.
  * [28] J. Pavitt, D. Braines, and R. Tomsett, “Cognitive analysis in sports: Supporting match analysis and scouting through artificial intelligence,” _Applied AI Letters_ , vol. 2, no. 1, p. e21, Mar. 2021.
  * [29] J. Memon _et al._ , “Handwritten Optical Character Recognition (OCR): A Comprehensive Systematic Literature Review (SLR),” _IEEE Access_ , vol. 8, pp. 142 642–142 668, Jul. 2020.
  * [30] R. Alhejaily _et al._ , “Automatic Team Assignment and Jersey Number Recognition in Football Videos,” _IASC_ , vol. 36, no. 3, pp. 2669–2684, Mar. 2023.
  * [31] R. Sharma, “Analyzing Racket Sports From Broadcast Videos,” Master’s thesis, International Institute of Information Technology, Hyderabad, April 2024. [Online]. Available: https://cvit.iiit.ac.in/research/thesis/thesis-students/analyzing-racket-sports-from-broadcast-videos
  * [32] A. Cioppa _et al._ , “SoccerNet-Tracking: Multiple Object Tracking Dataset and Benchmark in Soccer Videos,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops_ , June 2022, pp. 3491–3502.
  * [33] H.-C. Shih, “A Survey of Content-Aware Video Analysis for Sports,” _IEEE Trans. Circuits Syst. Video Technol._ , vol. 28, no. 5, pp. 1212–1231, Jan. 2017.
  * [34] K. Seweryn, A. Wróblewska, and S. Łukasik, “Survey of Action Recognition, Spotting and Spatio-Temporal Localization in Soccer – Current Trends and Research Perspectives,” 2023. [Online]. Available: https://arxiv.org/abs/2309.12067
  * [35] A. Hamdi _et al._ , “In-depth analysis of the impact of OCR errors on named entity recognition and linking,” _Nat. Lang. Eng._ , vol. 29, no. 2, pp. 425–448, Mar. 2023.
  * [36] A. Ç. Seçkin, B. Ateş, and M. Seçkin, “Review on Wearable Technology in Sports: Concepts, Challenges and Opportunities,” _Appl. Sci._ , vol. 13, no. 18, p. 10399, Sep. 2023.
  * [37] R. De Fazio _et al._ , “Wearable Sensors and Smart Devices to Monitor Rehabilitation Parameters and Sports Performance: An Overview,” _Sensors_ , vol. 23, no. 4, p. 1856, Feb. 2023.
  * [38] F. Lord _et al._ , “Methods of performance analysis in team invasion sports: A systematic review,” _J. Sports Sci._ , Oct. 2020.
  * [39] C.-W. Wu _et al._ , “Enhancing Fan Engagement in a 5G Stadium With AI-Based Technologies and Live Streaming,” _IEEE Syst. J._ , vol. 16, no. 4, pp. 6590–6601, May 2022.
  * [40] P. Rahimian and L. Toka, “A survey on player and ball tracking methods in soccer and other team sports,” _Journal of Quantitative Analysis in Sports_ , vol. 18, no. 1, pp. 35–57, Mar. 2022.
  * [41] N. Aharon, R. Orfaig, and B.-Z. Bobrovsky, “BoT-SORT: Robust Associations Multi-Pedestrian Tracking,” 2022. [Online]. Available: https://arxiv.org/abs/2206.14651
  * [42] J. Cao _et al._ , “Observation-Centric SORT: Rethinking SORT for Robust Multi-Object Tracking,” 2023. [Online]. Available: https://arxiv.org/abs/2203.14360
  * [43] A. Cioppa _et al._ , “SoccerNet 2023 Challenges Results,” 2023. [Online]. Available: https://arxiv.org/abs/2309.06006
  * [44] Z. Ge _et al._ , “YOLOX: Exceeding YOLO Series in 2021,” 2021. [Online]. Available: https://arxiv.org/abs/2107.08430
  * [45] Y. Cui _et al._ , “SportsMOT: A Large Multi-Object Tracking Dataset in Multiple Sports Scenes,” 2023. [Online]. Available: https://arxiv.org/abs/2304.05170
  * [46] A. Mittal, A. K. Moorthy, and A. C. Bovik, “No-Reference Image Quality Assessment in the Spatial Domain,” _IEEE Transactions on Image Processing_ , vol. 21, no. 12, pp. 4695–4708, 2012.
  * [47] Y. Du _et al._ , “PP-OCRv2: Bag of Tricks for Ultra Lightweight OCR System,” 2021. [Online]. Available: https://arxiv.org/abs/2109.03144
  * [48] “EasyOCR,” Apr. 2024. [Online]. Available: https://github.com/JaidedAI/EasyOCR
  * [49] H. Maric Solberg _et al._ , “PlayerTV: Advanced Player Tracking and Identification for Automatic Soccer Highlight Clips,” 2024. [Online]. Available: https://github.com/simula/PlayerTV
  * [50] ——, “PlayerTV GUI v2.0,” 2024. [Online]. Available: https://youtu.be/ol8kSMyOy18