# Visualizing Skiers’ Trajectories in Monocular Videos

Matteo Dunnhofer  Luca Sordi  Christian Micheloni  Machine Learning and
Perception Lab, University of Udine, Udine, Italy

###### Abstract

Trajectories are fundamental to winning in alpine skiing. Tools enabling the
analysis of such curves can enhance the training activity and enrich
broadcasting content. In this paper, we propose SkiTraVis, an algorithm to
visualize the sequence of points traversed by a skier during its performance.
SkiTraVis works on monocular videos and constitutes a pipeline of a visual
tracker to model the skier’s motion and of a frame correspondence module to
estimate the camera’s motion. The separation of the two motions enables the
visualization of the trajectory according to the moving camera’s perspective.
We performed experiments on videos of real-world professional competitions to
quantify the visualization error, the computational efficiency, as well as the
applicability. Overall, the results achieved demonstrate the potential of our
solution for broadcasting media enhancement and coach assistance.

Figure 1: Visualization of skiers’ trajectories in monocular videos. We
present SkiTraVis, an algorithm to compute and visualize the trajectory
performed by an alpine skier in videos acquired by a single monocular camera.
SkiTraVis takes in input a streaming video capturing a skier and visualizes
its past trajectory in each frame. In this picture, the first and third rows
show the input frames, while the second and fourth the trajectory computed by
SkiTraVis. The interval in seconds from the start of the video is also
reported.

##  1 Introduction

Due to the popularity of alpine skiing [60], the competitive form of such a
sport discipline is one of the most important markets of the winter sports
industry [12]. The international regulations of alpine skiing competitions
[54] require the definition of a path of so-called turning gates down a hill.
Differing in the overall length of the course and in the distance between the
gates, four popular alpine skiing sub-disciplines are present nowadays: Slalom
(SL), Giant Slalom (GS), Super-G (SG), and Downhill (DH). In each of these,
the winning goal is the same: ski down through the gates of the designed
course in the shortest time possible.

To do so, an athlete should optimize its skiing trajectories along the track.
Indeed, in alpine skiing the turns performed by an athlete form a trajectory
of points on the slope surface that if optimized to reduce the distance with
the turning gates can lead to saved time during the descent [6, 7]. Toward
such an objective, it becomes important to measure and analyze a trajectory in
order to determine the specific points of the performance affecting the
overall time taken. Machines able to perceive such evidence are valuable tools
to obtain increased knowledge of the skiing performance. The currently
available solutions to measure a trajectory of an alpine skier make use of
wearable devices (e.g. GNSS trackers, IMUs) put on the body or skis, and of 3D
terrain models to position the data coming from the sensors at different time
steps [24, 35]. The drawback of such an approach is the repeated time-
consuming installation and calibration of the sensor device on the athlete’s
body, and the acquisition of a precise terrain model. Moreover, the contextual
information influencing the skiing performance (e.g. positions of the turning
gates, conditions of slope surface, weather, and visibility) is completely
lost by such kinds of system. Research on training process improvement for
skiing pointed out the opportunity for a post-performance video review to
overcome such limitations [58, 44].

The importance of video-based performance analysis paves the way for training-
assistive tools based on computer vision. Indeed, computer vision applied to
videos capturing the athlete’s performance is a valid option to measure skiing
performance without the need for GNSS sensors or ground surface models [55,
44]. Previous work on this topic focused on 3D skier pose estimation [1] on a
motion capture system comprising six manually-operated mutually-calibrated
pan-tilt-zoom cameras capturing a Giant Slalom track section of three gates.
By means of multi-view human pose estimation algorithms [47, 64, 63], it was
shown that the 3D pose of athletes can be reconstructed at different time
steps, and from that time series, the skiing trajectory could be computed
inside a virtual 3D environment. Despite proving the suitability of computer
vision for trajectory analysis, such work has limited applicability in
practice because of the complex multi-camera setup.

In this paper, we aim for a different goal: to visualize, directly inside the
frames of a capturing monocular video, the trajectory of a skier. We describe
the attempt to design and implement a computer vision algorithm to compute and
visualize the trajectory in real competition conditions. Our approach, we
refer to as SkiTraVis, enables the analysis of a trajectory in a qualitative
way, meaning that the trajectory can be visualized in relation to the visual
appearance of the context appearing in the video. SkiTraVis is inputted with
just a single video with sufficient contextual details and works in a
streaming fashion, i.e. in each video frame it outputs the perspective-correct
trajectory exploiting the pixel information of the frame and the one occurring
in the previous frames. Our solution decouples the skier’s and the camera’s
motions between consecutive frames through a visual object tracker [67] and an
image key-point detection [53] and tracking algorithm [39]. The corresponding
key-points are used by RANSAC [23] to estimate the perspective transformation
between the frames, which in turn is used to map the previous motion of the
skier to the camera’s changed perspective. We evaluated SkiTraVis on multiple
clips taken from broadcasting videos capturing skier performance in real-world
conditions. The qualitative and quantitative results achieved suggest the
applicability of the method for augmented performance review and analysis
during training sessions, and also for enriched broadcasting content.
SkiTraVis works not only for alpine skiing but even for snowboarding and for
flight trajectory visualization in ski jumping and freestyle skiing.

##  2 Related Work

###  2.1 Computer Vision Applications in Skiing

Thanks to the advancements in computer vision methodologies [34, 26, 46, 9],
recently several applications have been possible for vision-based performance
analysis in skiing. In [56], a dataset was proposed to evaluate object
detection and tracking algorithms to localize recreational alpine skiers in
images and videos. The work described in [68] evaluated object detection and
human pose estimation algorithms to recognize falls of alpine skiers. On a
similar topic, it has been discussed about the combination of 2D pose
detectors with kinematics models to distinguish normal skiing situations from
out-of-balance and fall ones in pictures capturing competitive alpine skiers
[69]. Štepec et al. designed an algorithm based on 2D human pose trajectories
to score the style of jumps in ski jumping from a monocular video [62].
Concerning the same skiing discipline, Ludwig et al. designed improved vision
transformer architectures [37] to detect arbitrary key points on the human
body [40] and on the skis [41] in still frames of the jumps. For freestyle
skiing and snowboarding, Wang et al. developed a systematic approach composed
of a visual tracker, a human pose estimator, and a pose classifier, to
evaluate the quality of aerial jumps in monocular videos [65]. In [43],
research focused on the design of an algorithm to synchronize, spatially and
temporally, two videos capturing snowboarders to compare the timing and
spatial extent of their maneuvers.

The proposed SkiTraVis shares some similarities with [62, 65] in the
systematic design based on a visual object tracker [67, 17, 19, 16] to model
the motion of a skier. However, differently from all the other works, here we
tackle a different task that, to the best of our knowledge, no other solution
addressed before.

###  2.2 Vision-based Trajectory Analysis in Sports

Computer vision-based techniques have been successfully used in different
sports disciplines to reconstruct the trajectory of various elements [10, 42,
30, 8, 11]. Chen et al. developed an algorithm to compute the 3D trajectory of
a volleyball ball in a static-camera video by transforming the output of the
ball’s visual tracker with prior knowledge of the volleyball court [10]. In
similar capturing settings, Calandre et al. reconstructed the trajectory of
table-tennis balls with a visual detection and tracking module, a registration
between image and table coordinates, and a kinematic ball motion model [8]. In
[30], the tracking-by-deblatting paradigm was introduced for better localize
of – hence compute better trajectories – fast-moving objects such as table-
tennis or tennis balls in static camera videos. In [42], Mixed Integer
Programming was used to track, under motion blur conditions, the position of
volleyball, basketball, and soccer balls, with the goal of computing their
motion trajectories.

In basketball analytics, different solutions have been proposed to reconstruct
the trajectories of moving players [45, 28, 38, 11]. All these research
endeavors presented a pipeline composed of a visual tracking algorithm to get
player-specific localizations across video frames, and the application of a
homography transformation between frame and court coordinates. Their
differences lie in the tracking principles used, such as histogram-based
condensation [45], Cam-Shift [28], deformable part models and kalman filters
[38], or deep learning methods [11], and in the homography estimation, based
either on manual annotation [45] or on the matching of court lines and
boundaries [28, 38, 11]. Similar approaches have been also exploited in
computer vision-based applications for soccer [27] and ice-hockey [61].

The methodology described in this paper is similar to [45, 28, 38, 11] in the
design of a system to track an athlete (in our case a skier), and to compute
key-point correspondence from which to estimate a homography. Differently from
such solutions, we use a more recent transformer-based approach [67] to
implement the athlete tracking module, and we do not compute key-point matches
based on some form of prior knowledge of the playing field. This is because
alpine skiing is performed on a course that not only varies across competition
locations but even across the different sections of the same mountain slope
(e.g. in the width, steepness, and snow appearance). Hence, it is difficult to
define a model of the playing field as in soccer, basketball, or ice-hockey.
For this motivation, in this work, we explore the usage of generic key-point
correspondence algorithms [51, 57, 53, 39] to compute the homography that
enables the computation of the skier trajectory.

Figure 2: Pipeline of SkiTraVis. The algorithm processes each frame $F_{t}$ in
an online fashion. A visual tracker is run to model the skier’s motion between
the previous frame $F_{t-1}$ and $F_{t}$. Detection and tracking of static
key-points in the frames are run to estimate the camera’s motion. Then, the
homography between the two frames is computed and used to transform the
previous trajectory $\tau_{t-1}$ into $\tau_{t}$ according to the perspective
change.

##  3 Methodology

The goal of this paper is to develop an algorithm for augmenting a video
capturing the performance of an alpine skier with the visualization of the
trajectory he/she skied from his/her position in the first frame up to his/her
position in each other frame. Qualitative examples of our intention are
presented in Figure 1.

###  3.1 Preliminaries

The videos given as input to our SkiTraVis are considered to capture the
performance of an individual skier while he/she is constantly visible (at
least partially) in the scene. We do not put any constraints on the
configuration (intrinsic and extrinsic parameters) of the camera that captured
the videos, if not the presence of a sufficient amount of visual features in
the scene (more details about this will be given in Section 4). Formally, we
consider a video $\mathcal{V}=\big{\\{}F_{t}\in\mathcal{I}\big{\\}}_{t=0}^{T}$
as a sequence of $T\in\mathbb{N}$ frames $F_{t}$, where
$\mathcal{I}=\\{0,\cdots,255\\}^{w\times h\times 3}$ is the space of RGB
images. We use $p^{(t)}_{t}=(x^{(t)}_{t},y^{(t)}_{t})$ to denote the
coordinates of the point that summarizes the position of the athlete in the
image coordinate system of $F_{t}$ (e.g. the point of contact between the
skier and the snow surface).

#### Problem Formulation.

The goal is to compute, at each frame $F_{t}$, a trajectory

$\displaystyle\tau_{t}=\big{\\{}p^{(t)}_{i}\big{\\}}_{i=0}^{t},$ (1)

which is the sequence of points $p^{(t)}_{i}$ traversed by the athlete in the
2D space of previous frames, from $F_{0}$ up to $F_{t}$. To respect an online
processing modality, the algorithm is not allowed to process
$F_{\widehat{t}},\widehat{t}>t$ but just the pixel information available in
$F_{t}$ and eventual computations performed in preceding frames. After its
computation, the points in $\tau_{t}$ are used to highlight the pixels of
$F_{t}$, hence showing the trajectory performed by the skier. The online
setting aligns with the real-time requirements of different applications (e.g.
broadcasting and coach assistance) because it does not require waiting for the
athlete’s exercise to be terminated for the trajectory to be produced.

###  3.2 Pipeline of SkiTraVis

Figure 2 depicts the pipeline of the proposed SkiTraVis. The algorithm
processes each $F_{t}$ online according to their temporal order. $F_{t}$ is
first given to a visual object tracking algorithm designed to model the motion
of the skier and to provide its position $p^{(t)}_{t}$. Then, the pipeline
estimates the camera’s motion between the consecutive frames $F_{t-1},F_{t}$
by computing the homography transformation
$\mathbf{H}_{t}\in\mathbb{R}^{3\times 3}$. This is found by detecting and
tracking static key-points present in the two frames and by using RANSAC on
the corresponding key-points. $\mathbf{H}_{t}$ is used to map the points of
the trajectory available in the previous frame, $\tau_{t-1}$, into the
coordinate system of $F_{t}$. After that, $p^{(t)}_{t}$ is appended to such a
transformed trajectory to obtain $\tau_{t}$, the list of all the points
traversed by the athlete with respect to $F_{t}$’s perspective. We now
describe the different components of SkiTraVis in more detail.

Figure 3: The points forming a trajectory. The points $p^{(t)}_{t}$ that
compose the trajectory $\tau_{t}$ (represented here as red dots) are extracted
from the bounding-box $b_{t}$ (represented by the blue rectangles) outputted
by the skier tracking algorithm. The points roughly denote the skiers’ feet’
position.

#### Skier Motion Tracking.

The first step of the pipeline is to exploit a visual object tracker [31, 18,
16] to track the motion of the athlete across all the frames up to $F_{t}$.
For this, we used a tracker outputting a bounding-box
$b_{t}=(x^{(b)}_{t},y^{(b)}_{t},w^{(b)}_{t},h^{(b)}_{t})\in\mathbb{R}^{4}$
enclosing the athlete’s appearance at every $F_{t}$. $x^{(b)}_{t},y^{(b)}_{t}$
represent the coordinates of the top-left corner of the box while
$w^{(b)}_{t},h^{(b)}_{t}$ estimate the width and height. We consider the
position of the skier $p^{(t)}_{t}=(x^{(t)}_{t},y^{(t)}_{t})$ in $F_{t}$ as

$\displaystyle x^{(t)}_{t}$ $\displaystyle=x^{(b)}_{t}+w^{(b)}_{t}\cdot 0.5$
(2) $\displaystyle y^{(t)}_{t}$ $\displaystyle=y^{(b)}_{t}+h^{(b)}_{t}\cdot k$
(3) $\displaystyle k$ $\displaystyle\in[0,1].$ (4)

Given an accurate bounding-box and $k=0.9$, we assume $p^{(t)}_{t}$ to
approximate the closest point of contact between the skier’s feet and the
snow, as shown by Figure 3.111More information is given in Appendix A.1 of the
supplementary document. The tracker is initialized in the first frame $F_{0}$
of the video with the bounding-box $b_{0}$ that outlines the appearance of the
target skier. Such a piece of information can be obtained by asking a human
operator to provide the bounding-box for the athlete of interest via some
user-friendly annotation system or by a skier-specific detection algorithm.
Based on experiments, we found the state-of-the-art transformer-based single
object tracker STARK [67] to provide a great balance between accuracy and
efficiency. We fine-tuned the pre-trained publicly-available model executing
the original training methodology on video data specifically designed for
alpine skier tracking.

#### Camera Motion Tracking.

To render the trajectory of the skier $\tau_{t-1}$ into $F_{t}$, we need to
compute the perspective transformation occurring between $F_{t-1}$ and
$F_{t}$. To implement this step we make use of the homography matrix
$\mathbf{H}_{t}$ mapping the pixel coordinates of $F_{t-1}$ into $F_{t}$. To
compute $\mathbf{H}_{t}$, we match static key-points in the fields of view of
$F_{t-1}$ and $F_{t}$ through an off-the-shelf generic key-point detector and
tracker. Key-points are first detected in $F_{t-1}$, and then the
correspondences of such points are found in $F_{t}$ by local tracking. This
procedure computes an alignment between the same semantic points of the images
that express how static objects in the scene have apparently moved between the
frames. Since the objects are static and do not actually move, the camera’s
motion is quantified [25, 5]. Based on experiments, we have found traditional
methods [53, 39] to work the best. The method of Shi-Tomasi [53] was used to
detect visual features in $F_{t-1}$, and the Lucas-Kanade’s optical flow
method [39] to track such visual descriptors in $F_{t}$. The usage of such
off-the-shelf instances is motivated by the fact that alpine skiing is
performed on courses that change a lot, across competition locations and even
across the different sections of the same mountain slope. Hence, it is not
trivial to define a generally applicable model of the playing field, which we
leave for future investigations.

#### Trajectory Computation.

With the alignments of corresponding key-points, we compute the homography
matrix $\mathbf{H}_{t}$ through the RANSAC algorithm [23], which applies an
iterative optimization procedure on the correspondences in order to find the
best homography matrix that explains them. We excluded the key-points in
$F_{t}$ within the bounding-box $b_{t}$ because they belong to the skier which
is a non-static object. If present, we also discarded all the key-points lying
on the superimposed virtual graphics showing the characteristics of the
athlete’s performance (e.g. the running time). After $\mathbf{H}_{t}$ is
computed, it is used to map the points $p^{(t-1)}_{i}$ of $\tau_{t-1}$ in the
new frame $F_{t}$. In more detail, at each $F_{t}$, $\tau_{t-1}$ consists of
all the skier positions tracked in the preceding $t-1$ frames and transformed
according to the perspective of $F_{t-1}$. The trajectory $\tau_{t}$ for the
latest $F_{t}$ is obtained by the multiplication of the homogeneous
coordinates of each $p^{(t-1)}_{i}\in\tau_{t-1}$ with the homography matrix,
such that

$\displaystyle\tau_{t}=\big{\\{}p^{(t)}_{i}\big{\\}}_{i=0}^{t},$ (5)
$\displaystyle p^{(t)}_{i}=[x^{(t)}_{i},y^{(t)}_{i}]:0\leq x^{(t)}_{i}<w,0\leq
y^{(t)}_{i}<h,$ (6)
$\displaystyle[x^{(t)}_{i},y^{(t)}_{i},1]=[x^{(t-1)}_{i},y^{(t-1)}_{i},1]^{\mathbf{T}}\cdot\mathbf{H}_{t},$
(7) $\displaystyle[x^{(t-1)}_{i},y^{(t-1)}_{i}]=p^{(t-1)}_{i}.$ (8)

As can be noticed by Eq. (5), $\tau_{t}$ also includes the point $p^{(t)}_{t}$
extracted by the tracker’s bounding-box for $F_{t}$. This is achieved by
appending such a point to $\tau_{t}$ after the application of $\mathbf{H}_{t}$
to $\tau_{t-1}$. Eq. (6) specifies that only the transformed points falling
inside the $F_{t}$’s image coordinate system are retained in $\tau_{t}$.

More concretely, at the first time step in which the frame matching procedure
is executed, i.e. $t=1$, $\tau_{0}=\\{p^{(0)}_{0}\\}$ consists only of the
point extracted by the bounding-box (detected or manually labeled)
highlighting the target skier in $F_{0}$. To obtain the transformation of
$p^{(0)}_{0}$ in $F_{1}$, its homogeneous representation is multiplied by
$\mathbf{H}_{1}$ to compute the homogeneous vector
$[x^{(1)}_{0},y^{(1)}_{0},1]$, from which $p^{(1)}_{0}$ is extracted.
$\tau_{1}=\\{p^{(1)}_{0},p^{(1)}_{1}\\}$ is obtained by appending
$p^{(1)}_{1}$. The process is repeated until the end of the video.

##  4 Experimental Settings222Detailed information is provided in the Appendix
B.

###  4.1 Data and Annotations

To determine the quality of SkiTraVis’s trajectories, we make use of a
dedicated evaluation dataset representing real-world application scenarios.
The videos belonging to this dataset appear in the test-set of SkiTB, a video
dataset designed for the development of athlete tracking methods in different
skiing disciplines.444SkiTB is developed by our research team and it will be
released soon with a separate publication. Such a dataset features 100 multi-
camera broadcasting videos (60 videos for training and 40 for testing) that
capture completely, from the start to the finish line, the performance of
professional alpine skiers. Each frame of each video is manually annotated
with a bounding-box containing the appearance of the athlete’s body and
equipment. Manually annotating the trajectories performed by skiers in each
frame of the video clips is a challenging task other than very time-consuming.
Indeed, it is hard to visually identify the specific points at which the skier
passed since the whitish appearance of the snow’s surface does not show easily
identifiable reference points to localize them. We hence exploited an
automatic annotation procedure followed by manual verification. We executed
the state-of-the-art LOFTR image matching algorithm [57] to detect and match
key-points of consecutive frames of SkiTB’s monocular test videos. RANSAC [23]
was then used to compute homographies on the matches. The homographies were
used to map the skier localization points extracted from the manual bounding-
box annotations into trajectories for each frame, in a similar fashion as
described for SkiTraVis. The videos with the generated trajectories have been
visually assessed and selected by our team, after being instructed by two
professional alpine skiing coaches on meaningful trajectory visualizations.
Overall, this procedure allowed us to obtain 233 monocular clips whose
characteristics are reported in Table 1.

Table 1: Statistics of the dataset used to evaluate  SkiTraVis. The first block of rows gives details about the characteristics of the video clips, while the second provides information about the featured real-world application scenarios. # monocular video clips | 233
---|---
# frames | 25783
Video clip length in # frames (min, avg, max) | 30, 111, 354
Video clip length in seconds (min, avg, max) | 1.0, 3.7, 11.8
Resolution, frame-rate | 720p, 30 FPS
$\tau_{t}$ length in # points (min, avg, max) | 1, 25, 229
# clips for alpine skiing sub-discipline (SL, GS, SG, DH) | 35, 55, 71, 72
# athletes (men, women) | 23 (13, 10)
# locations | 29
# weather conditions (clear, overcast, fog, snowing) | 200, 30, 2, 1
Figure 4: RGB color distributions for the applicability of SkiTraVis. The top
plot shows the average RGB values distribution and an example (with the
generated reference trajectory) of the frames belonging to the evaluation set
used for SkiTraVis, while the bottom one shows the RGB values distributions
and an example (with a wrongly generated trajectory) of unsuitable frames. As
can be noticed, suitable clips include frames that have a larger image
contrast.

Leveraging the proposed semi-automatic annotation protocol limits the validity
of the proposed SkiTraVis to the videos whose frames are successfully
processed by LOFTR [57]. We tried to quantify the conditions in which such an
algorithm works by analyzing the difference between used and discarded video
clips. As can be noticed by the width of the RGB histograms in Figure 4, the
suitable videos include frames with a generally larger image contrast. We also
evaluated the number of features per-frame discovered by the Shi-Tomasi
algorithm [53], resulting in an average number of 1430 (median 1219) for
suitable clips and 1126 (median 811) for unsuitable ones. These differences
are mostly due to the appearance of winter scenarios in images, which provide
whitish textures containing few visual anchors that feature detection
algorithms can exploit.

###  4.2 Evaluation Protocol

The protocol used to execute SkiTraVis for evaluation is similar to the One-
Pass Evaluation for visual tracking [66]. The algorithm is run on each video
separately, and it is given the first frame and the respective ground-truth
bounding-box to initialize the visual tracker on the skier. Then, SkiTraVis is
inputted with each of the remaining frames $F_{t}$ iteratively, it processes
the frames according to the described methodology and returns each $\tau_{t}$.
To compute quantitative results, each $\tau_{t}$ is compared with the
respective reference trajectory $\sigma_{t}$, created as described in the
previous sub-section. In addition, we evaluate the bounding-boxes $b_{t}$
predicted by the visual tracker and the homographies $\mathbf{H}_{t}$
generated after the frame matching step, in order to understand the impact of
different implementations.

###  4.3 Evaluation Measures

To quantify the distance between $\tau_{t}$ and $\sigma_{t}$, we compute the
average Euclidean pixel distance between the points of the reference
$\sigma_{t}$ with the respective, ordered based on the insertion time step, of
$\tau_{t}$. We refer to this measure as Mean Per Point Trajectory Error (MPPTE
$\downarrow$). In addition, we report the pixel values of Dynamic Time Warping
(DTW $\downarrow$) [50], a popular measure in trajectory analysis. To evaluate
the skier tracking, we employ the Area Under the Curve (AUC $\uparrow$) of the
success plot which is equivalent to the average bounding-box overlap [66]. For
the frame matching module, we evaluate the mean squared error (MSE
$\downarrow$) between the estimated homographies and the respective LOFTR-
validated ones.

Besides the error, we also rated the efficiency of SkiTraVis under different
configurations of visual trackers and frame matching methods. For both
categories, we report the average module-specific running speed in
milliseconds (ms $\downarrow$) and in Frames-Per-Second (FPS $\uparrow$), and
the average delay in seconds (Delay $\downarrow$) between the time instant of
the video clip’s end and the end instant of the whole SkiTraVis pipeline’s
processing time. This value measures the average overall delay in visualizing
the trajectory for the entire video clip, and it can be used to understand how
much time has to be waited to obtain the trajectory after the observation of
the skier’s performance.

###  4.4 Evaluated Implementations

To the best of our knowledge, SkiTraVis is the first algorithm for skier
trajectory visualization in monocular videos, and there is no comparable
solution present in the literature. Thus, to provide useful insights, we
evaluate the performance of our algorithm under different configurations of
state-of-the-art visual trackers and frame correspondence algorithms. For the
visual tracker, we evaluated: the deep learning-based methodologies STARK
[67], SuperDiMP [2, 32], SiamRPN++ [36], trained for generic object tracking;
and the traditional method MOSSE [3]. We also fine-tuned STARK with the
aforementioned SkiTB training data. For key-point detection and matching, we
evaluated: the learning-based method of SuperPoint (SP) [13] and SuperGlue
(SG) [51], and the Recurrent All-pairs Field Transform (RAFT) dense optical
flow method [59]; and the traditional methods as the Shi-Tomasi (ST) [53]
detector with the optical flow algorithm Lucas-Kanade [39], and the Oriented
FAST and Rotated BRIEF (ORB) descriptor [49] with the brute force (BF)
matcher. Code was implemented in Python and run on a machine with an Intel
Xeon E5-2690 v4 @ 2.60GHz CPU, 320 GB of RAM, and an NVIDIA TITAN V GPU.

Table 2: SkiTraVis’s performance with different visual trackers for skier motion modeling. The STARK tracker [67] fine-tuned for skier tracking presents great accuracy coupled with a good running time. In all of these experiments, the ST + LK key-point detector and tracker were used. Tracker | MPPTE $\downarrow$ | DTW $\downarrow$ | AUC $\uparrow$ | ms $\downarrow$ | FPS $\uparrow$ | Delay $\downarrow$
---|---|---|---|---|---|---
Oracle | 9.8 | 278 | 100 | - | - | -
STARK-ft | 12.1 | 313 | 90.1 | 29 | 34 | 6
STARK | 19.3 | 484 | 73.2 | 29 | 34 | 6
SuperDiMP | 21.2 | 479 | 67.6 | 89 | 12 | 11
SiamRPN++ | 31.6 | 791 | 62.8 | 20 | 49 | 5
MOSSE | 76.1 | 1075 | 37.0 | 9 | 117 | 3

##  5 Discussion

In this section, we discuss the results achieved by SkiTraVis. Overall, by
looking at the numbers in Table 2 and 3, we observe that the configuration
with the fine-tuned STARK-ft tracker and the ST + LK key-point detector and
tracker has the best balance between trajectory error and processing speed. A
visualization of the MPPTE $\downarrow$ and DTW $\downarrow$ errors, as
presented in Figure 5 demonstrate that MPPTE $\downarrow$ and DTW $\downarrow$
values lower than 15 and 400 pixels lead to a consistent perception of the
trajectory. Some demonstrative videos are available at tinyurl.com/2raemvf5.
The MPPTE $\downarrow$ score is less than the 10% of the average skier
appearance’s height which is 123 pixels (computed from the manually annotated
bounding-boxes). The running time of the STARK-ft ST + LK configuration has an
average Delay $\downarrow$ of 6 seconds which is promising for usage in
broadcasting applications (e.g. for replays) and during training activities
(as the real-time requirement is not so strict).

#### Skier Motion Tracking.

Table 2 reports the accuracy of SkiTraVis when different visual trackers are
employed. Of the models trained for generic object tracking (shown in rows
3-6), STARK results in the best performing under the MPPTE $\downarrow$ and
AUC $\uparrow$ measures, while SuperDiMP achieves the lowest DTW $\downarrow$
score. SiamRPN++ and MOSSE fall behind in all the scores, apart from the
processing speed where they result better. Considering this aspect, it is
worth noticing that STARK presents a great balance between accuracy and
efficiency. For this motivation, we fine-tuned it (as STARK-ft, second row of
Table 2) with the alpine skier-specific training-set of SkiTB. Such a version
reduces the MPPTE $\downarrow$ and DTW $\downarrow$ of 37% and 35%,
respectively, and presents an MPPTE $\downarrow$ error higher only of 2.3
pixels with respect to the trajectories extracted by the manually labeled
bounding-boxes (denoted as Oracle, first row).

Table 3: SkiTraVis’s performance with different frame matching algorithms. The ST + LK [53, 39] combination shows the greatest balance between accuracy and processing speed. In all of these experiments, the STARK-ft visual tracker was used. Detection & Matching | MPPTE $\downarrow$ | DTW $\downarrow$ | MSE $\downarrow$ | ms $\downarrow$ | FPS $\uparrow$ | Delay $\downarrow$
---|---|---|---|---|---|---
LOFTR | 4.8 | 101 | 0 | 801 | 1 | 90
ST + LK | 12.1 | 313 | 9952 | 42 | 24 | 6
SP + SG | 15.1 | 352 | 25.7 | 135 | 7 | 15
ORB + BF | 28.7 | 836 | 111732 | 783 | 1 | 84
RAFT | 39.8 | 807 | 22 | 641 | 2 | 73
Figure 5: Visualizations of trajectory errors. Here we provide visual
demonstrations of how the MPPTE $\downarrow$ and DTW $\downarrow$ values
represent the quality of the trajectory visualizations. The first two rows
show frames where the reconstruction is visually consistent, while the last
row presents video frames with a large error.

#### Camera Motion Tracking.

Table 3 presents the performance of SkiTraVis under different frame matching
strategies. The ST detector and the LK optical-flow method (ST + LK, row two)
results in the closest to LOFTR in the quality of the trajectories, and the
best in the computational efficiency. The SG strategy (SP + SG, row three)
follows shortly in MPPTE $\downarrow$ and DTW $\downarrow$, while it is much
slower in the computation. We observe a good trajectory computation by ST + LK
even though its computed homographies present a larger MSE $\downarrow$ to
those of the latter strategy. The remaining tested methods, ORB + BF and RAFT,
show a much larger error in all the considered measures. Table 3 also reports
the results achieved with the LOFTR method (row one) used to generate the
reference trajectories. When run in this configuration with STARK-ft as the
skier tracker, the MPPTE $\downarrow$ and DTW $\downarrow$ decrease by 60% and
68% respectively. By comparing these improvements with those given by the
Oracle tracker, we can conclude that most of the SkiTraVis’s error is due to
the frame matching module. Due to its good MPPTE $\downarrow$ and DTW
$\downarrow$ values, as well as considering the discussion in Section 4.1, the
results of the STARK-ft and LOFTR configuration demonstrate its applicability
in practice, but when the processing time is not an issue since LOFTR is the
slowest frame matching strategy.

Figure 6: Generalization to other skiing disciplines. We found SkiTraVis to
work well for the generation of trajectories in other skiing disciplines such
as snowboarding, and for the visualization of the flight trajectory in ski
jumping and freestyle skiing.

#### Generalization.

Our algorithm generalizes to other skiing disciplines. Figure 6 shows
qualitative examples of SkiTraVis applied to the generation of the trajectory
performed by alpine snowboarders, and to the visualization of the flight
trajectory in ski jumping and freestyle skiing.

##  6 Conclusions

In this paper, we present SkiTraVis, an algorithm to visualize the
trajectories performed on an alpine skier in a monocular video. The solution
uses a skier-specific visual tracking algorithm to model its motion, and an
image-generic key-point-based frame matching algorithm to estimate the camera
motion and the consequent transformation between the video frames. Thanks to
the separation of the two motions, it is possible to relocate the previous
skier positions according to the captured perspective, hence obtaining a
visualization of the trajectory performed by the athlete. The experimental
campaign focused on analyzing the algorithm’s accuracy and efficiency, as well
as the conditions of applicability. Under some circumstances, we believe that
SkiTraVis can be used in broadcasting and coach assistive applications, and
more generally, it will serve as a baseline for future research on trajectory
visualization for computer vision-based skiing analytics.

#### Acknowledgments.

Research supported by the project between the University of Udine and the
organizing committee of EYOF 2023 Friuli-Venezia Giulia.

## References

  * [1] Roman Bachmann, Jörg Spörri, Pascal Fua, and Helge Rhodin.  Motion capture from pan-tilt cameras with unknown orientation.  In 2019 International Conference on 3D Vision (3DV), pages 308–317. IEEE, 2019.
  * [2] Goutam Bhat, Martin Danelljan, Luc Van Gool, and Radu Timofte.  Learning discriminative model prediction for tracking.  In Proceedings of the IEEE/CVF international conference on computer vision, pages 6182–6191, 2019.
  * [3] David S Bolme, J Ross Beveridge, Bruce A Draper, and Yui Man Lui.  Visual object tracking using adaptive correlation filters.  In 2010 IEEE computer society conference on computer vision and pattern recognition, pages 2544–2550. IEEE, 2010.
  * [4] G. Bradski.  The OpenCV Library.  Dr. Dobb’s Journal of Software Tools, 2000.
  * [5] Matthew Brown and David G Lowe.  Automatic panoramic image stitching using invariant features.  International journal of computer vision, 74:59–73, 2007.
  * [6] Congying Cai and Xiaolan Yao.  Dynamic analysis and trajectory optimization for the nonlinear ski-skier system.  Control Engineering Practice, 114:104868, 2021.
  * [7] Cong-ying Cai and Xiao-lan Yao.  Trajectory optimization with constraints for alpine skiers based on multi-phase nonlinear optimal control.  Frontiers of Information Technology & Electronic Engineering, 21(10):1521–1534, 2020.
  * [8] Jordan Calandre, Renaud Péteri, Laurent Mascarilla, and Benoit Tremblais.  Table tennis ball kinematic parameters estimation from non-intrusive single-view videos.  In 2021 International Conference on Content-Based Multimedia Indexing (CBMI), pages 1–6, 2021.
  * [9] Zhe Cao, Tomas Simon, Shih-En Wei, and Yaser Sheikh.  Realtime multi-person 2d pose estimation using part affinity fields.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7291–7299, 2017.
  * [10] Hua-Tsung Chen, Chien-Li Chou, Wen-Jiin Tsai, and Suh-Yin Lee.  3d ball trajectory reconstruction from single-camera sports video for free viewpoint virtual replay.  In 2011 Visual Communications and Image Processing (VCIP), pages 1–4, 2011.
  * [11] Liang-Hua Chen, Chih-Wen Su, and Hsiang-An Hsiao.  Player trajectory reconstruction for tactical analysis.  Multimedia Tools and Applications, 77(23):30475–30486, 2018.
  * [12] The Nielsen Company.  Audi FIS Ski World Cup 2021/22 - TV Media Evaluation Event Summary, 2022\.
  * [13] Daniel DeTone, Tomasz Malisiewicz, and Andrew Rabinovich.  Superpoint: Self-supervised interest point detection and description.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pages 224–236, 2018.
  * [14] Matteo Dunnhofer, Antonino Furnari, Giovanni Maria Farinella, and Christian Micheloni.  Is first person vision challenging for object tracking?  In Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops, pages 2698–2710, 2021.
  * [15] Matteo Dunnhofer, Antonino Furnari, Giovanni Maria Farinella, and Christian Micheloni.  Visual object tracking in first person vision.  International Journal of Computer Vision, 131(1):259–283, 2023\.
  * [16] Matteo Dunnhofer and Christian Micheloni Machine.  Cocolot: Combining complementary trackers in long-term visual tracking.  In 2022 26th International Conference on Pattern Recognition (ICPR), pages 5132–5139. IEEE, 2022.
  * [17] Matteo Dunnhofer, Niki Martinel, Gian Luca Foresti, and Christian Micheloni.  Visual Tracking by means of Deep Reinforcement Learning and an Expert Demonstrator.  In Proceedings of The IEEE/CVF International Conference on Computer Vision Workshops, 2019.
  * [18] Matteo Dunnhofer, Niki Martinel, and Christian Micheloni.  Tracking-by-Trackers with a Distilled and Reinforced Model.  In Asian Conference on Computer Vision, 2020.
  * [19] Matteo Dunnhofer, Niki Martinel, and Christian Micheloni.  Weakly-supervised domain adaptation of deep regression trackers via reinforced knowledge distillation.  IEEE Robotics and Automation Letters, 6(3):5016–5023, 2021.
  * [20] Heng Fan, Liting Lin, Fan Yang, Peng Chu, Ge Deng, Sijia Yu, Hexin Bai, Yong Xu, Chunyuan Liao, and Haibin Ling.  Lasot: A high-quality benchmark for large-scale single object tracking.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5374–5383, 2019.
  * [21] Hao-Shu Fang, Jiefeng Li, Hongyang Tang, Chao Xu, Haoyi Zhu, Yuliang Xiu, Yong-Lu Li, and Cewu Lu.  Alphapose: Whole-body regional multi-person pose estimation and tracking in real-time.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022\.
  * [22] fengyang95.  pyCFTrackers, 2019.
  * [23] Martin A Fischler and Robert C Bolles.  Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography.  Communications of the ACM, 24(6):381–395, 1981.
  * [24] Matthias Gilgien, Jörg Spörri, Julien Chardonnens, Josef Kröll, and Erich Müller.  Determination of external forces in alpine skiing using a differential global navigation satellite system.  Sensors, 13(8):9821–9835, 2013.
  * [25] Richard Hartley and Andrew Zisserman.  Multiple View Geometry in Computer Vision.  Cambridge University Press, New York, NY, USA, 2 edition, 2003.
  * [26] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  * [27] Yutaro Honda, Rei Kawakami, Ryota Yoshihashi, Kenta Kato, and Takeshi Naemura.  Pass receiver prediction in soccer using video and players’ trajectories.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, pages 3503–3512, June 2022.
  * [28] Min-Chun Hu, Ming-Hsiu Chang, Ja-Ling Wu, and Lin Chi.  Robust camera calibration and player tracking in broadcast basketball video.  IEEE Transactions on Multimedia, 13(2):266–279, 2011.
  * [29] Glenn Jocher, Ayush Chaurasia, Alex Stoken, Jirka Borovec, NanoCode012, Yonghye Kwon, Kalen Michael, TaoXie, Jiacong Fang, imyhxy, Lorna, Zeng Yifu, Colin Wong, Abhiram V, Diego Montes, Zhiqiang Wang, Cristi Fati, Jebastin Nadar, Laughing, UnglvKitDe, Victor Sonck, tkianai, yxNONG, Piotr Skalski, Adam Hogan, Dhruv Nair, Max Strobel, and Mrinal Jain.  ultralytics/yolov5: v7.0 - YOLOv5 SOTA Realtime Instance Segmentation, Nov. 2022.
  * [30] Jan Kotera, Denys Rozumnyi, Filip Sroubek, and Jiri Matas.  Intra-frame object tracking by deblatting.  In Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops, pages 0–0, 2019.
  * [31] Matej Kristan, Aleš Leonardis, Jiří Matas, Michael Felsberg, Roman Pflugfelder, Joni-Kristian Kämäräinen, Hyung Jin Chang, Martin Danelljan, Luka Čehovin Zajc, Alan Lukežič, Ondrej Drbohlav, et al.  The tenth visual object tracking vot2022 challenge results.  In Computer Vision – ECCV 2022 Workshops, pages 431–460, Cham, 2023. Springer Nature Switzerland.
  * [32] Matej Kristan, Aleš Leonardis, Jiří Matas, Michael Felsberg, Roman Pflugfelder, Joni-Kristian Kämäräinen, Martin Danelljan, Luka Čehovin Zajc, Alan Lukežič, Ondrej Drbohlav, et al.  The eighth visual object tracking vot2020 challenge results.  In Computer Vision–ECCV 2020 Workshops: Glasgow, UK, August 23–28, 2020, Proceedings, Part V 16, pages 547–601. Springer, 2020.
  * [33] Matej Kristan, Jiří Matas, Aleš Leonardis, Michael Felsberg, Roman Pflugfelder, Joni-Kristian Kämäräinen, Hyung Jin Chang, Martin Danelljan, Luka Cehovin, Alan Lukežič, Ondrej Drbohlav, Jani Käpylä, Gustav Häger, Song Yan, Jinyu Yang, Zhongqun Zhang, and Gustavo Fernández.  The ninth visual object tracking vot2021 challenge results.  In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) Workshops, pages 2711–2738, October 2021.
  * [34] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton.  Imagenet classification with deep convolutional neural networks.  Communications of the ACM, 60(6):84–90, 2017.
  * [35] Andreas Krüger and Jürgen Edelmann-Nusser.  Application of a full body inertial measurement system in alpine skiing: A comparison with an optical video based system.  Journal of applied biomechanics, 26(4):516–521, 2010.
  * [36] Bo Li, Wei Wu, Qiang Wang, Fangyi Zhang, Junliang Xing, and Junjie Yan.  Siamrpn++: Evolution of siamese visual tracking with very deep networks.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4282–4291, 2019.
  * [37] Yanjie Li, Shoukui Zhang, Zhicheng Wang, Sen Yang, Wankou Yang, Shu-Tao Xia, and Erjin Zhou.  Tokenpose: Learning keypoint tokens for human pose estimation.  In Proceedings of the IEEE/CVF International conference on computer vision, pages 11313–11322, 2021.
  * [38] Wei-Lwun Lu, Jo-Anne Ting, James J. Little, and Kevin P. Murphy.  Learning to track and identify players from broadcast sports videos.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 35(7):1704–1716, 2013.
  * [39] Bruce D Lucas and Takeo Kanade.  An iterative image registration technique with an application to stereo vision.  In IJCAI’81: 7th international joint conference on Artificial intelligence, volume 2, pages 674–679, 1981.
  * [40] Katja Ludwig, Philipp Harzig, and Rainer Lienhart.  Detecting arbitrary intermediate keypoints for human pose estimation with vision transformers.  In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) Workshops, pages 663–671, January 2022.
  * [41] Katja Ludwig, Daniel Kienzle, Julian Lorenz, and Rainer Lienhart.  Detecting arbitrary keypoints on limbs and skis with sparse partly correct segmentation masks.  In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) Workshops, pages 461–470, January 2023.
  * [42] Andrii Maksai, Xinchao Wang, and Pascal Fua.  What players do with the ball: A physically constrained interaction modeling.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016.
  * [43] Seiji Matsumura, Dan Mikami, Naoki Saijo, and Makio Kashino.  Spatiotemporal motion synchronization for snowboard big air, 2021.
  * [44] Mirela Ostrek, Helge Rhodin, Pascal Fua, Erich Müller, and Jörg Spörri.  Are existing monocular computer vision-based 3d motion capture approaches ready for deployment? a methodological study on the example of alpine skiing.  Sensors, 19(19):4323, 2019.
  * [45] Matej Perše, Matej Kristan, Stanislav Kovačič, Goran Vučkovič, and Janez Perš.  A trajectory-based analysis of coordinated team activity in a basketball game.  Computer Vision and Image Understanding, 113(5):612–621, 2009.  Computer Vision Based Analysis in Sport Environments.
  * [46] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun.  Faster r-cnn: Towards real-time object detection with region proposal networks.  Advances in neural information processing systems, 28, 2015.
  * [47] Helge Rhodin, Jörg Spörri, Isinsu Katircioglu, Victor Constantin, Frédéric Meyer, Erich Müller, Mathieu Salzmann, and Pascal Fua.  Learning monocular 3d human pose estimation from multi-view images.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 8437–8446, 2018.
  * [48] E. Riba, D. Mishkin, D. Ponsa, E. Rublee, and G. Bradski.  Kornia: an open source differentiable computer vision library for pytorch.  In Winter Conference on Applications of Computer Vision, 2020.
  * [49] Ethan Rublee, Vincent Rabaud, Kurt Konolige, and Gary Bradski.  Orb: An efficient alternative to sift or surf.  In 2011 International conference on computer vision, pages 2564–2571. Ieee, 2011.
  * [50] H. Sakoe and S. Chiba.  Dynamic programming algorithm optimization for spoken word recognition.  IEEE Transactions on Acoustics, Speech, and Signal Processing, 26(1):43–49, 1978.
  * [51] Paul-Edouard Sarlin, Daniel DeTone, Tomasz Malisiewicz, and Andrew Rabinovich.  Superglue: Learning feature matching with graph neural networks.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4938–4947, 2020.
  * [52] Abraham Savitzky and Marcel JE Golay.  Smoothing and differentiation of data by simplified least squares procedures.  Analytical chemistry, 36(8):1627–1639, 1964.
  * [53] Jianbo Shi and Tomasi.  Good features to track.  In 1994 Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, pages 593–600, 1994.
  * [54] International Ski and Snowboard Federation (FIS).  The International Ski Competition Rules (ICR) Book IV Joint Regulations For Alpine Skiing, 2022.
  * [55] Jörg Spörri.  Reasearch dedicated to sports injury prevention-the’sequence of prevention’on the example of alpine ski racing.  Habilitation with Venia Docendi in Biomechanics, 1(2):7, 2016.
  * [56] Philip Steinkellner and Klaus Schöffmann.  Evaluation of object detection systems and video tracking in skiing videos.  In 2021 International Conference on Content-Based Multimedia Indexing (CBMI), pages 1–6, 2021.
  * [57] Jiaming Sun, Zehong Shen, Yuang Wang, Hujun Bao, and Xiaowei Zhou.  LoFTR: Detector-free local feature matching with transformers.  CVPR, 2021.
  * [58] Matej Supej, Jörg Spörri, and Hans-Christer Holmberg.  Methodological and practical considerations associated with assessment of alpine skiing performance using global navigation satellite systems.  Frontiers in Sports and Active Living, 1:74, 2020.
  * [59] Zachary Teed and Jia Deng.  Raft: Recurrent all-pairs field transforms for optical flow.  In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part II 16, pages 402–419. Springer, 2020.
  * [60] Laurent Vanat.  2022 International Report on Snow & Mountain Tourism, 2022.
  * [61] Kanav Vats, Pascale Walters, Mehrnaz Fani, David A. Clausi, and John S. Zelek.  Player tracking and identification in ice hockey.  Expert Systems with Applications, 213:119250, 2023.
  * [62] Dejan Štepec and Danijel Skočaj.  Video-based ski jump style scoring from pose trajectory.  In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) Workshops, pages 682–690, January 2022.
  * [63] Bastian Wandt, James J Little, and Helge Rhodin.  Elepose: Unsupervised 3d human pose estimation by predicting camera elevation and learning normalizing flows on 2d poses.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6635–6645, 2022.
  * [64] Bastian Wandt, Marco Rudolph, Petrissa Zell, Helge Rhodin, and Bodo Rosenhahn.  Canonpose: Self-supervised monocular 3d human pose estimation in the wild.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13294–13304, 2021.
  * [65] Jianbo Wang, Kai Qiu, Houwen Peng, Jianlong Fu, and Jianke Zhu.  Ai coach: Deep human pose estimation and analysis for personalized athletic training assistance.  In Proceedings of the 27th ACM international conference on multimedia, pages 374–382, 2019.
  * [66] Yi Wu, Jongwoo Lim, and Ming-Hsuan Yang.  Object tracking benchmark.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 37(9):1834–1848, 2015.
  * [67] Bin Yan, Houwen Peng, Jianlong Fu, Dong Wang, and Huchuan Lu.  Learning spatio-temporal transformer for visual tracking.  In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 10448–10457, October 2021.
  * [68] Yulin Zhu and Wei Qi Yan.  Ski fall detection from digital images using deep learning.  In Proceedings of the 5th International Conference on Control and Computer Vision, ICCCV ’22, page 70–78, New York, NY, USA, 2022. Association for Computing Machinery.
  * [69] Michael Zwölfer, Dieter Heinrich, Kurt Schindelwig, Bastian Wandt, Helge Rhodin, Joerg Spoerri, and Werner Nachbauer.  Improved 2d keypoint detection in out-of-balance and fall situations – combining input rotations and a kinematic model, 2021.

Visualizing Skiers’ Trajectories in Monocular Videos
Supplementary Document
Matteo Dunnhofer, Luca Sordi, Christian Micheloni
Corresponding author e-mail<EMAIL_ADDRESS>
##  Appendix A Further Motivations and Details about SkiTraVis

###  A.1 Trajectory Points $p^{(t)}_{t}$

To represent the skier localization point $p^{(t)}_{t}$, we did not use a
human skeleton parsing algorithm, such as a 2D human pose detector [9, 21], to
not increase the complexity and the efficiency of the SkiTraVis pipeline. In
fact, such a procedure would have required the execution of an additional deep
neural network at the cost of incrementing the overall time to visualize the
trajectory. Moreover, even though alpine skier-specific pose detectors have
been studied [1], we found them committing displacement errors in the
prediction of the human and equipment key-points close to the ground surface
(e.g. for prediction feet’ or skis’ positions). For instance, AlphaPose [21]
fine-tuned and tested, respectively, on the SkiPose2D training and test
datasets [1] achieves a Mean Per Joint Prediction Error (MPJPE) of 13.2 pixels
at the skier’s feet and of 9.4 pixels in the upper body parts. By calculating
the average pixel distance between $p^{(t)}_{t}$ and its corresponding
reference point (i.e. the point computed following Eq. (2,3,4) on the manually
annotated bounding-box) we obtain a value of 5.7 pixels, which is much lower
than the errors committed by the skier pose detector. These results
demonstrate that the bounding-box tracker is more robust in localizing the
point $p^{(t)}_{t}$ in practice. Nevertheless, we believe that SkiTraVis does
not require substantial changes in the pipeline to compute more semantically
meaningful $p^{(t)}_{t}$, since 2D human pose detector could be just run on
the image patches extracted from the bounding-boxes predicted by the visual
tracker. We hence leave the integration of 2D pose detectors for future work.
Moreover, the selection of values $k<0.9$ (e.g. $k=0.5$) could be used to move
the trajectory toward representing the athlete’s center of mass, which is of
interest for biomechanical analysis [58], but it should be noted that in 2D
images such a point does not fall at the same depth level in which the
homography is computed. Indeed, in nearby locations to the center of mass, the
homography is computed by exploiting static key-points in the background that
lie on the snow’s surface behind the skier. Hence, transforming the eventual
$p^{(t)}_{t}$ with such a homography would shift the point by a pixel amount
that does not respect the camera motion at the depth level at which the skier
is actually localized.

##  Appendix B Experimental Details

Figure 7: Suitable and unsuitable clips for trajectory visualization. This
figure extends the plots of Figure 4 of the main paper by giving other
examples of the frames where was possible to obtain the reference trajectory.
As can be noticed, suitable video clips include frames where different visual
features contrast with the whitish appearance of the snow and are evenly
distributed over the whole frame.

###  B.1 Data and Annotations

As stated in the main paper, to determine the quality of SkiTraVis’s
trajectories, we make use of a dedicated evaluation dataset representing real-
world application scenarios. The videos belonging to this dataset appear in
the test-set of SkiTB. SkiTB (“Skiers from the Top to the Bottom”) is a video
dataset we collected to implement deep learning-based athlete-tracking methods
for different skiing disciplines. It contains 300 broadcasting videos (for a
total of 392K frames) of 196 professional skiers (alpine skiers, ski jumpers,
and freestyle skiers) performing their gestures from the start to the finish.
Due to the high spatial extent of skiing courses, each performance is captured
across several different cameras put in sequence. Each frame of the videos was
manually annotated by our research team with a bounding-box containing the
appearance of the athletes’ bodies and equipment, following the instructions
commonly used for the creation of visual tracking datasets [66, 33, 31, 20,
14, 15]. All videos are labeled with the skiing sub-discipline performed, the
course location, as well as the weather conditions. 100 of such 300 multi-
camera videos feature the performance of alpine skiers, where the first 60 are
set as training set and the remaining 40 as test-set, considering an ordering
based on the date of the competition. This setting respects the real-world
condition where the learning-based algorithms are applied to test data
acquired at a later time than the training data. Such a training set, which
comprises 662 monocular videos and 130934 frames, has been used to fine-tune
the STARK [67] tracker, while the 439 monocular videos present in the test set
have been exploited as candidates for the generation of the reference
trajectories as described in Section 4 of the main paper. Figure 7 gives
additional examples of the situations in which was possible to obtain a
reference trajectory, and hence evaluate the error committed by SkiTraVis. The
virtual graphics present in the video frames have been manually labeled with
bounding-boxes in order to avoid the influence of their static visual features
in the execution of the camera motion estimation step. All the video frames
presented in the main paper and in this supplementary document (Figures 1, 2,
3, 4, 5, 7) are taken from video clips appearing in the dataset used for
validating SkiTraVis.

Other already-published datasets [1, 55] were not used for their unsuitability
to the task of interest. SkiPose2D [1] provides just 2D images annotated with
sparse pose annotations, thus it is not suitable for dense per-frame tracking
as it is required by continuous trajectory visualization in each frame.
SkiPosePTZ [55] was not employed because of its limited representation of
real-world scenarios. Such a dataset was collected by capturing six alpine
skiers on a single course composed of just three turning gates. Moreover,
poles with markers have been densely placed on the side of the track in order
to register the video frames to a geo-spatial reference system. Such marking
poles, which are clearly visible in the frames, are not present in normal
conditions of alpine skiing training or competition. Thus, in SkiPosePTZ, the
frame matching methods for camera motion estimation would have had the
opportunity to exploit visual features not present in real conditions,
ultimately leading to a wrong assessment of the error committed by SkiTraVis.

###  B.2 Performance Measures

In the following, we give more information about the employed performance
measures. The proposed Mean Per Point Trajectory Error (MPPTE $\downarrow$) is
defined, for a video clip $\mathcal{V}$, as follows:

$\displaystyle\text{MPPTE}\downarrow_{\mathcal{V}}\>=\frac{1}{T}\sum_{t=0}^{T-1}\Big{(}\frac{1}{|\sigma_{t}|}\sum_{i\in\mathcal{I}_{t}}||p_{i}^{(t)}-q_{i}^{(t)}||_{2}^{2}\Big{)},$
(9)

where $q_{i}^{(t)}\in\sigma_{t}$ and
$\mathcal{I}_{t}=\\{i\\}\subseteq\\{0,\cdots,t\\}:q_{i}^{(t)}=[x_{i}^{(t)},y_{i}^{(t)}],0\leq
x_{i}^{(t)}<w,0\leq y_{i}^{(t)}<h$. $\mathcal{I}_{t}$ is the set of the
indices of points $q_{i}^{(t)}$ that remain visible in $F_{t}$. To obtain a
single score that summarizes the performance of SkiTraVis, the average
$\text{MPPTE}\downarrow_{\mathcal{V}}$ value across all evaluation video clips
is used. In simple words, in each frame,
$\text{MPPTE}\downarrow_{\mathcal{V}}$ computes the Euclidean distance between
the points of the reference trajectory $\sigma_{t}$ and the corresponding
ones, according to the temporal index of insertion, of the predicted
trajectory $\tau_{t}$. Overall, this measure gives information on the spatial
distance in pixels occurring between the reference trajectory’s points that
are visible and the respective but predicted by SkiTraVis.

The MPPTE $\downarrow$ does not take into account the fact that the
$p_{i}^{(t)}$ could be removed from $\tau_{t}$ because of Eq. (6). Due to such
a condition, $\tau_{t}$ and $\sigma_{t}$ could have different lengths since
points that are discarded are not inserted back later on. Thus, to compare
trajectories of different lengths we employed Dynamic Time Warping (DTW
$\downarrow$) which measures the alignment of two trajectories in such
conditions [50]. For a video clip $\mathcal{V}$, the score is defined as

$\displaystyle\text{DTW}\downarrow_{\mathcal{V}}\>=\frac{1}{T}\sum_{t=0}^{T-1}dtw(\tau_{t},\sigma_{t})$
(10)

where $dtw(\tau_{t},\sigma_{t})$ is the function that computes the Dynamic
Time Warping distance in pixels between the predicted and reference
trajectories. The overall DTW $\downarrow$ measure is obtained by averaging
across all the evaluation videos.

The MSE $\downarrow$ measure used to compare $\mathbf{H}_{t}$ with the
respective reference $\mathbf{H}^{(\text{LOFTR})}_{t}$, was implemented for a
video clip $\mathcal{V}$ as:

$\displaystyle\text{MSE}\downarrow_{\mathcal{V}}\>=\frac{1}{T}\sum_{t=0}^{T-1}\Big{(}\frac{1}{9}||\mathbf{H}_{t}-\mathbf{H}^{(\text{LOFTR})}_{t}||_{2}^{2}\Big{)}.$
(11)

As for the MPPTE $\downarrow$ and DTW $\downarrow$ measures, the average
across all the video clips was employed to obtain a single score.

For further details about the AUC $\uparrow$ measure, please see [66].

###  B.3 Implementation Details

To obtain the fine-tuned version of STARK [67] (STARK-ft), we exploited the
frames of the aforementioned training set. We used the original code provided
by the authors to adapt the STARK-ST50 model pre-trained for generic object
tracking. Except for the number of epochs in the stage-one training (which was
set to 200), default values have been used for the hyper-parameters. Also for
the SiamRPN++ [36] and SuperDiMP [2, 32] visual trackers we used the code
provided by the authors along with their pre-trained models. MOSSE [3] instead
was implemented by the pyCFtracking library [22]. Regarding the camera motion
tracking estimation methods, the code provided by the authors was used to
implement SuperPoint [13] and SuperGlue [51] with the pre-trained weights for
outdoor conditions. The Kornia library [48] was instead used to implement
LOFTR [57] (the instance for outdoor environments has been exploited). For the
ST + LK [53, 39] and ORB + BF [49] methods as well as for RANSAC [23] the
OpenCV implementations [4] were employed.

##  Appendix C Additional Results

Table 4: SkiTraVis’s performance when sharpening filters are applied. A small filter of size $3\times 3$ slightly improves the MPPTE $\downarrow$ results, while the Unsharp mask slightly decreases the quality of trajectories. Sharpening | w/o | filter $S$ | Unsharp mask
---|---|---|---
MPPTE $\downarrow$ | 12.1 | 11.6 | 12.2
DTW $\downarrow$ | 313 | 314 | 343

#### Sharpening Frames.

Considering the limited availability of visual features in video frames
capturing winter scenarios, we explored the impact of the following
convolutional filter

$\displaystyle S=\begin{bmatrix}-1&-1&-1\\\ -1&9&-1\\\ -1&-1&-1\end{bmatrix}$
(12)

and of the Unsharp mask to sharpen each video frame before the key-point ST +
LK detection and tracking step. The results are reported in Table 4 and show
that a $3\times 3$ filter slightly improves the trajectories in terms of MPPTE
$\downarrow$. The other filter instead reduces the performance.

#### Initialization with a Skier Detector.

We performed experiments to understand the impact on SkiTraVis of an
initialization bounding-box given by a detector rather than a human annotator.
In such initialization conditions, SkiTraVis becomes a fully automatic system
that does not require any human intervention. To implement the detector, we
used a YOLOv5x instance [29] fine-tuned (with default hyper-parameters) on the
same SkiTB training-set used to fine-tune STARK-ft. With the initialization
given by the detector, the SkiTraVis’s configuration with STARK-ft as skier
tracker and ST + LK as camera motion estimation method achieves MPPTE
$\downarrow$ and DTW $\downarrow$ errors of 15.8 and 344 pixels, respectively.
The quality of the initial annotation hence influences negatively the
trajectory visualization, since the two measures degrade by 3.7 and 31 pixels,
respectively. We hypothesize that this behavior is due to the visual object
tracker when initialized with the detector’s localization. Such a noisy
bounding-box prejudices the initialization step that builds models of the
target skier, and in turn, such noisy models affect the localization ability
of the tracker.

Figure 8: Aesthetically pleasant trajectory visualizations. This figure shows
examples of enhanced visualizations based on the trajectory generated by
SkiTraVis for alpine skiers, snowboarders, and ski jumpers. In these cases, in
each frame of the video clips, the trajectories were smoothed using the
Savitzky–Golay filter [52] and the trajectory points falling inside the area
defined by the bounding-box $b_{t}$ were filtered out.

#### More Qualitative Examples.

Figure 8 shows enhanced trajectory visualizations for different disciplines
and application scenarios. Additional qualitative videos showing the
trajectories generated by SkiTraVis can be reached through this link
tinyurl.com/2raemvf5.

##  Appendix D Limitations and Future Work

We think SkiTraVis can serve as a baseline for future research on trajectory
visualization and reconstruction in monocular video-based skiing performance
analytics. We hypothesize that the capabilities of SkiTraVis could be improved
by better integrating the different modules of the pipeline, and potentially
through an end-to-end optimization stage of the learning modules and backbone
networks involved. Datasets to train such an approach should be also
investigated. The system could be also enhanced by exploiting human pose
trackers instead of bounding-box ones. A human pose representation could be
exploited to compute a more consistent point of contact between the athlete
and the snow surface. Furthermore, the motion modeling of the different human
body parts could enable the development of solutions able to simultaneously
reconstruct the trajectory of disparate parts of the athlete (e.g. hands or
feet) or of its equipment (e.g. skis). This direction should be investigated
in parallel with research for more accurate human pose detection and tracking
in skiing. Finally, we think that the better exploitation of the specific cues
appearing on the slope and in training/competition scenarios could lead to an
enhanced trajectory visualization. Methods that extend the applicability
conditions of the proposed SkiTraVis to situations in which the employed
LOFTR-based validation process failed, should be also studied in the future.
In some contexts (e.g. for broadcasting applications), such a process could
also exploit the repetitive movements of camera operators that follow
different skiers during competitions on the same track to develop more
accurate camera motion tracking models.