# ShuttleSet: A Human-Annotated Stroke-Level Singles Dataset for Badminton
Tactical Analysis

Wei-Yao Wang  0000-0002-6551-1720 National Yang Ming Chiao Tung
UniversityHsinchuTaiwan<EMAIL_ADDRESS>,  Yung-Chang Huang
0009-0006-6648-0598 National Yang Ming Chiao Tung UniversityHsinchuTaiwan
<EMAIL_ADDRESS>,  Tsi-Ui Ik  0000-0001-6432-9161 National Yang
Ming Chiao Tung UniversityHsinchuTaiwan<EMAIL_ADDRESS>and  Wen-Chih Peng
0000-0002-0172-7311 National Yang Ming Chiao Tung UniversityHsinchuTaiwan
<EMAIL_ADDRESS>
(2023)

###### Abstract.

With the recent progress in sports analytics, deep learning approaches have
demonstrated the effectiveness of mining insights into players’ tactics for
improving performance quality and fan engagement. This is attributed to the
availability of public ground-truth datasets. While there are a few available
datasets for turn-based sports for action detection, these datasets severely
lack structured source data and stroke-level records since these require high-
cost labeling efforts from domain experts and are hard to detect using
automatic techniques. Consequently, the development of artificial intelligence
approaches is significantly hindered when existing models are applied to more
challenging structured turn-based sequences. In this paper, we present
ShuttleSet, the largest publicly-available badminton singles dataset with
annotated stroke-level records. It contains 104 sets, 3,685 rallies, and
36,492 strokes in 44 matches between 2018 and 2021 with 27 top-ranking men’s
singles and women’s singles players. ShuttleSet is manually annotated with a
computer-aided labeling tool to increase the labeling efficiency and
effectiveness of selecting the shot type with a choice of 18 distinct classes,
the corresponding hitting locations, and the locations of both players at each
stroke. In the experiments, we provide multiple benchmarks (i.e., stroke
influence, stroke forecasting, and movement forecasting) with baselines to
illustrate the practicability of using ShuttleSet for turn-based analytics,
which is expected to stimulate both academic and sports communities. Over the
past two years, a visualization platform has been deployed to illustrate the
variability of analysis cases from ShuttleSet for coaches to delve into
players’ tactical preferences with human-interactive interfaces, which was
also used by national badminton teams during multiple international high-
ranking matches.

badminton dataset; stroke-level records; sports analytics; machine learning

††journalyear: 2023††copyright: acmlicensed††conference: Proceedings of the
29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining; August
6–10, 2023; Long Beach, CA, USA††booktitle: Proceedings of the 29th ACM SIGKDD
Conference on Knowledge Discovery and Data Mining (KDD ’23), August 6–10,
2023, Long Beach, CA, USA††price: 15.00††doi: 10.1145/3580305.3599906††isbn:
979-8-4007-0103-0/23/08††ccs: Computing methodologies Machine learning††ccs:
Information systems Data mining††ccs: Information systems Data warehouses

##  1\. Introduction

With the advancement of artificial intelligence in recent years, sports
analytics has significantly changed the sports environment with video
understanding and match analysis using machine learning (ML) and deep learning
(DL) approaches. For example, a video assistant referee system has been
deployed in football over 100 competitions including the 2022 FIFA World Cup
to support the referee team to reduce
misjudgments111https://www.fifa.com/technical/football-technology/football-
technologies-and-innovations-at-the-fifa-world-cup-2022/video-assistant-
referee-var. On the other hand, player evaluation and tactic investigation
have improved the effectiveness in multiple sports, such as football (Decroos
et al., 2019; Azad et al., 2022; Kim et al., 2022; Simpson et al., 2022),
baseball (Kincaid et al., 2021; Han et al., 2022), basketball (Chen et al.,
2022; Yanai et al., 2022), and badminton (Wang et al., 2022b; Chang et al.,
2022). To train models for these applications, the datasets are mainly
collected from either videos (Deliège et al., 2021; Zhu et al., 2022) or
sensors (Ito and Gotoh, 2018; Haladjian et al., 2020; Pillitteri et al., 2021)
to extract the enormous variable metadata in the matches. Therefore, it is
beneficial for researchers to conduct analysis and introduce fruitful findings
from the collected dataset that are able to foster not only coaches and
players but also sports media and spectators.

Figure 1. Two examples with potential returning strokes of men’s and women’s
singles.

Generally, there are several available datasets with frame-based or event-
based records for various sports; for instance, Deliège et al. (2021) proposed
SoccerNet-v2 for soccer understanding and broadcast production via annotated
soccer match videos, and Yue et al. (2014) released SportsVU, which is a
tracking dataset for games recorded from several seasons of NBA. However, in
turn-based sports, there are only a few public datasets with tournament-level
records (Breno, 2018; Rahman, 2022), but none of them are labeled with the
stroke-level records due to the human cost and domain gap for labeling (e.g.,
what shot type a player performs). These issues hinder researchers’ efforts to
design techniques to analyze demanding applications such as hitting patterns,
tactic investigations, and action recognition from such turn-based sequences
(Wang, 2022). As illustrated in Figure 1, players have multiple choices to
organize their tactics so as not to be seen through by their opponents in
badminton matches, and thus pre-match data collection and pattern
investigation are needed to forecast the next stroke on the court. Therefore,
we aim to create microscopic metadata from badminton matches to accurately
describe the process from strokes to matches.

Although existing automatic detecting tools, e.g., (Chu and Situmeang, 2017),
can be adopted to extract spatial-temporal stroke features from badminton
videos, the automation introduces a constraint on the variability in the
dataset since corresponding structured source data (i.e., the compositions of
stroke, rally, set, and match) must be available to analyze the similarities
and discrepancies between players within a match as well as across matches.
Furthermore, providing high-quality annotation requires serious efforts from
domain experts since not only the number of videos is rapidly increasing but
it is also hard for non-domain experts to distinguish the tactical metadata of
a stroke. Therefore, one of the key challenges is to implement a labeling tool
that can provide a straightforward process for labeling from each stroke to
each match to reduce the difficulty of annotators reviewing videos repeatedly.

In this paper, we present the ShuttleSet dataset annotated by domain experts
with the efficient and precise S2-labeling tool (Huang et al., 2022) using the
BLSR format (Wang et al., 2022a). Based on the labeling tool, ShuttleSet
provides stroke-by-stroke human annotation ground truth records for 44 matches
from 2018 to 2021 with 27 high-ranking players (16 men’s singles and 11
women’s singles), which consist of 104 sets, 3,685 rallies, and 36,492
strokes. Following the current public sports dataset, ShuttleSet will likewise
be made available to the public in order to stimulate both the research and
sports analysis communities for more advanced applications.

Our ShuttleSet dataset distinguishes itself in the following aspects:

  1. 1)

The Largest Public Stroke-Level Dataset: To the best of our knowledge, this is
the largest public turn-based sports dataset with stroke-level records which
attempts to mitigate the gap between turn-based sports and research
communities.

  2. 2)

Efficient Expert Annotation: In contrast to automation approaches to generate
the data set, we rely on annotations from domain experts to accurately
describe the detailed and structured process during a badminton match.
Moreover, we adopt a computer-aided labeling tool with user-friendly designs
and computer vision techniques to reduce the burden of watching the video
repeatedly.

  3. 3)

Detailed Fields: The fields in the dataset include temporal, spatial, posture,
and skill categories, which can be used for spatial-temporal mining, action
recognition, and tactic investigation.

  4. 4)

Customized Train, Validation, and Test Sets: As we do not intend to limit the
scope of the application usage (i.e., no fixed labels), ShuttleSet provides
the customized availability to split the datasets instead of the fixed train-,
validation, and test-sets.

  5. 5)

Paradigms of Practical Usage: We illustrate multiple usages in Sections 5 and
6 and provide a Python implementation of the ShuttleSet
dataset222https://github.com/wywyWang/CoachAI-Projects. to advance the
research of multifaceted applications.

##  2\. Related Work

Nowadays, with the development of video understanding technologies, sports
analytics has improved player and team performance and raised social
engagement by mining effective insights to address several challenges between
sports analytics and artificial intelligence (Decroos et al., 2020). To
collect previous match records for analyzing player behaviors, researchers
have designed applications for detecting actions from videos automatically (Wu
et al., 2022), e.g., Sports-1M (Karpathy et al., 2014), UCF Sports (Rodriguez
et al., 2008), and J-HMDB (Jhuang et al., 2013). Recently, Li et al. (2021)
presented a multi-person dataset consisting of spatio-temporal localized
sports actions, which includes faster movement and multiple concurrent actions
by multiple players. SoccerNet-v2 is collected to encourage future research to
advance soccer understanding and broadcast production with action recognition
and camera shot segmentation (Deliège et al., 2021). These public datasets
motivate the research community to develop the research of these sports and
understand teams’ and individuals’ behaviors in fulfilling the coaches’
strategies.

Figure 2. The matchup matrix of men’s singles.

In the field of badminton, existing approaches mainly focus on action and
stroke detection from videos as well. Chu and Situmeang (2017) developed an
interface illustrating court detection, shot type classification, and
offensive/defensive classification results from the given video. Yoshikawa et
al. (2021) incorporates skeleton information to detect the timing of the
overhead stroke by their collected videos with specialized cameras. To broaden
the availability of videos to the general matches, an automatic annotation
framework was designed to detect players from the official broadcast videos
without special cameras or additional sensors (Ghosh et al., 2018). Although
these techniques are useful for tracking the motion recognition of players to
construct datasets automatically (Fang and Sun, 2021), it remains a constraint
of the opportunity of structured data from source videos, which obstructs
researchers from conducting various analyses for delving into advanced tactic
investigations. In addition, there is no existing public turn-based dataset
with sufficient stroke-level records, which requires high-quality annotation
from domain experts for labeling. Therefore, we propose ShuttleSet as the
largest stroke-level badminton dataset for turn-based sports analytics. The
aim is to not only foster the academic community’s research on turn-based
sequences but also the sports community’s development of applications to
enhance their tactical strategies.

##  3\. The ShuttleSet Dataset

###  3.1. Overview

ShuttleSet contains 36,492 strokes, 3,685 rallies, and 104 sets from 44
matches between 2018 and 2021 played by 27 high-ranking men’s and women’s
players. The details of annotation matches and the corresponding players are
listed in Appendix A.1. Each match consists of 2 or 3 sets (best of 3), each
of which is comprised of the metadata of strokes to describe the microscopic
process from stroke to rally. The format of each stroke can be divided into
four categories, which mainly expands from the BLSR design format (Wang et
al., 2021):

  * •

Rally category: 1) The current scores of both players in the set. 2) The
player who won the rally, which only appears in the last stroke of each rally.
3) Lose reason is classified from out, touched the net, not pass over the net,
opponent’s ball landed, and misjudged.

  * •

Temporal category: 1) The hitting time of the stroke. 2) The frame of the
hitting stroke. We note that the frame number can be converted to the hitting
time based on the frame rate of the video.

  * •

Spatial category: 1) The locations of both players on the court at the hitting
time. 2) The hitting locations of the shuttle at the hitting time. These
locations are recorded by both accurate x and y coordinates and overall grids.
3) The height at the hitting time, which is a binary variable indicates
whether the hitting height is below or above the net. This field can be used
to determine if the player returns a stroke passively or aggressively.

  * •

Hitting category: 1) The status of the hitting stroke, which indicates if the
stroke belongs to the serve, return, or dead bird type. 2) Hit the shuttle
with the backhand or not. 3) Hit the shuttle around the head or not. 4) The
shot type of the hitting stroke, where the shot type is selected from one of
the 18 types: net shot, return net, smash, wrist smash, lob, defensive return
lob, clear, drive, driven flight, back-court drive, drop, passive drop, push,
rush, defensive return drive, cross-court net shot, short service, and long
service. The detailed explanations of each type are introduced in (Wang et
al., 2022a).

Figure 3. The matchup matrix of women’s singles.

In addition to the definition of stroke-level records, we require that the
labeled matches in ShuttleSet adhere to a few conditions. First, the matches
are selected from quarter-finals, semi-finals, and finals333There are a few
matches are from the group stage of BWF World Tour Finals. of top-selective
tournaments (e.g., Super 1000 series, BWF World Tour Finals) to provide state-
of-the-art and highly-organized tactics from top players, which can be used to
investigate players’ tactics and improve the hitting quality of low-ranking
and junior players. As a second condition, videos are crawled from public
sources444http://bwf.tv/, which not only broadens the available usage of
videos to general matches but also does not require specialized cameras to
collect videos. Third, ShuttleSet consists of both men’s and women’s singles
matches to obtain maximum variability, which can be analyzed for the different
hitting patterns from different matchups. Moreover, the number of players and
the number of the played count of each player are various in ShuttleSet to
enable researchers to investigate the hitting patterns from the same player
and between different players, which also points out the challenging
applications due to the rare-appearance players. In Figures 2 and 3, we
demonstrate the matching matrix of each player contained in ShuttleSet with
the corresponding appearances.

The BLSR format only adopts the grids on the badminton court to record
location systems for traditional analysis (e.g., (Valldecabres et al., 2020;
Wang et al., 2020)), which fails to record the fine-grained coordinates of
both players and shuttles. To integrate the traditional tactical analysis, the
conversion from 2D coordinates of players and shuttles is designed to map them
to the corresponding grids as illustrated in Figure 4. In general, the grid
system is introduced by domain experts with nine grids for locations in the
court and an additional seven grids for locations outside the court. As the
court is divided into two symmetric sub-courts, the grids of each sub-court
are identical to provide a consistent format for location analysis. Formally,
the conversion can be achieved by first converting the camera coordinate
system to the real-world coordinate system:

(1) $p^{\prime}=Hp,$

where $p^{\prime}$ is the 2D real-world coordinate system, $p$ is the 2D
camera coordinate system, and $H\in\mathbb{R}^{3\times 3}$ is a homography
matrix between these systems, which can be solved by providing at least four
pairs between two coordinate systems. The corners of the court (i.e., (0, 0),
(0, 6.1), (13.4, 0), (13.4, 6.1)) are consistent since the court in the real
world is 13.4 meters in length and 6.1 meters in width; therefore, the
corners’ coordinates in the camera coordinate system can be acquired by only
labeling them in the camera coordinate system.

Figure 4. The grid system layout on the badminton court. Numbers denote the
corresponding locations. The red color is outside the singles court.

As ShuttleSet is annotated by domain experts with a computer-aided labeling
tool, we sample three matches for labeling again to ensure data consistency.
For the temporal category, the time differences of serve, return, and dead
bird are 0.24, 0.02, and 0 frames in terms of the mean absolute error (MAE),
respectively. For the spatial category, the MAE of the shuttle’s position is
0.18 on average and the MAE of the player’s position is 0.86 on average. For
the labels of shot type, the quality difference of serve type, return type,
and dead bird type is 0, 0.05, and 0.11 in terms of accuracy.

Table 1 summarizes the distribution of each shot type of both men’s and
women’s singles in ShuttleSet. The two most common shot types of men’s and
women’s singles are net shot and lob. The third most common type of men’s
singles is return net and that of women’s singles is clear. This also boosts
the variability of the dataset by mixing stroke-level records of both men’s
and women’s singles. In addition, ShuttleSet contains different proportions of
shot types, which might be a challenge for designing turn-based methods to
take imbalance types into account.

Since the goal of ShuttleSet is to provide turn-based strokes to stimulate
advanced research on turn-based sports, we do not split up our dataset into
pre-defined train-, validation-, and test-sets. In contrast, we provide
multiple real-world but challenging analysis paradigms in the experiments to
showcase the usage of ShuttleSet, which can also be used as the benchmarks for
the research community for developing new algorithms.

Table 1. Distribution of the shot type in ShuttleSet. Along with the frequency of each class label, we present the relative occurrence based on men’s and women’s singles. Shot type | Singles | Count | Proportion (%) | Total (%)
---|---|---|---|---
net shot | Men | 3,781 | 15.45 | 17.60
Women | 527 | 2.15
return net | Men | 2,130 | 8.70 | 10.13
Women | 349 | 1.43
smash | Men | 1,430 | 5.85 | 7.13
Women | 314 | 1.28
wrist smash | Men | 950 | 3.88 | 4.61
Women | 178 | 0.73
lob | Men | 3,124 | 12.76 | 14.68
Women | 470 | 1.92
defensive return lob | Men | 149 | 0.61 | 0.76
Women | 36 | 0.15
clear | Men | 1,417 | 5.79 | 7.51
Women | 421 | 1.72
drive | Men | 451 | 1.84 | 2.06
Women | 52 | 0.22
driven flight | Men | 43 | 0.18 | 0.18
Women | 0 | 0.00
back-court drive | Men | 274 | 1.12 | 1.36
Women | 59 | 0.24
drop | Men | 1,112 | 4.54 | 5.68
Women | 279 | 1.14
passive drop | Men | 811 | 3.31 | 3.73
Women | 101 | 0.42
push | Men | 1,736 | 7.09 | 8.51
Women | 347 | 1.42
rush | Men | 296 | 1.21 | 1.36
Women | 37 | 0.15
defensive return drive | Men | 212 | 0.87 | 1.10
Women | 56 | 0.23
cross-court net shot | Men | 843 | 3.44 | 3.99
Women | 135 | 0.55
short service | Men | 1,728 | 7.06 | 8.16
Women | 271 | 1.10
long service | Men | 183 | 0.75 | 1.45
Women | 172 | 0.70

Despite being cost-intense and far less scalable than automatic labeling
tools, the human annotation has several benefits over automated ground-truth
generation. The first and most obvious reason to adopt human annotations is
the freedom to annotate any type of match without customizing a programmatic
source. For most matches, the camera angles are different due to the stadium’s
condition, and the spectators, referees, and sometimes matches from other
courts are also filmed in the video. These issues are not a hard constraint
with human annotation since it is simple to capture the labeling target (e.g.,
players and shuttles) from the video, while these issues introduce noise for
automatic methods. A second reason to leverage human annotation is that it
usually supports a more natural and structural interpretation of describing
the process from a match to a rally, which requires severe efforts for both.
Automatic techniques on turn-based sports need not only detection accuracy but
also complex processing to label end-to-end (e.g., detecting if the current
set is an end). This might be challenging even for human annotation; we,
therefore, incorporate a computer-aided labeling tool to simplify the
annotation process as outlined in Section 4. Third, labeling ground truth
through human annotation is usually considered a natural upper bound on the
dataset quality, which is able to guarantee the performance expectation in
training deep neural network models and avoiding overfitting. Automatic
methods suffer from the degradation of the detection if the previous stage
fails to detect the target. For instance, the automatic techniques need to
segment players on the court to record their current position. The program
requires customization to ensure that it can only detect two players but not
the referees and fans. On the flip side, our labeling process reduces labeling
errors and maintains annotation accuracy based on the detection process.

##  4\. Annotation Process

The annotation process was carried out mainly with the badminton experts using
the shot-by-shot (S2) labeling tool (Huang et al., 2022). It is a microscopic
labeling system for badminton and has been demonstrated as having efficient
and effective labeling quality by badminton experts. For example, the S2
labeling tool has proven to meet the requirement of completing the labeling
process on match day for tactic investigation for the next match. The
collection process of stroke-level annotation of a match using the S2 labeling
tool can be mainly divided into four stages:

  * •

Match preparation: The collected matches are described in Section 3. To
provide both practical and efficient requirements, the frame rate of the video
is 30 frames per second and the resolution is 1280x720 (720p). Afterwards, the
metadata of the match is input, e.g., the tournament name, the tournament
level, round in the tournament, date, player names, played sets, duration (in
minutes), and the four corners of the court boundary lines are marked for
computing the corresponding homography matrix.

  * •

Rally Segmentation: Annotators are required to segment the start and end of
each rally and the scoring result by recording the time of the corresponding
event, which is supported by the shuttle trajectory to reduce the switching
time of annotators.

  * •

Video preprocessing: S2 labeling tool splits the match video into multiple
rally videos to avoid heavily wasting computing resources and to eliminate
execution time due to the break time. Then, the video is preprocessed with
detection techniques such as trajectory detection, player detection, and
skeleton detection. For the use of labeling, only trajectory detection is
employed.

  * •

Shot-by-shot labeling: Initiate the labeling process. Annotators need to click
on the screen and select the corresponding shot types with a one-pass labeling
operation. The location information can be labeled by directly clicking on the
video screen to avoid manual alignment errors, and the corresponding clicking
point is displayed simultaneously on the video screen for instant
confirmation. The details will be introduced next.

Figure 5. An interface of the labeling tool with seven steps. Figure 6. The
labeling process of the fourth stage. The corresponding mark turns to red to
remind the current step.

A group of six dedicated annotators with badminton knowledge was assembled and
supervised for ShuttleSet annotation with the S2 labeling tool. Figure 5 shows
an interface of the labeling process of stage four and Figure 6 illustrates
the labeling process consisting of seven steps, which is optimized to achieve
a one-pass labeling operation. The labeling of each rally starts from the time
of the serve type to the end of the dead bird type, which requires labeling in
order: 1) the hitting time, 2) the location of the player returning the
stroke, 3) the location of the opponent, 4) if the shuttle is hit with a
backhand, around the head, and above or below the net, 5) the returning time
of the opponent (the landing time of a shuttle if the rally ends), 6) the
landing location of a shuttle, and 7) the corresponding shot type. The
labeling cycle is then repeated for each shot in a rally, where the next shot
starts from the opponent’s returning time of the previous returning time and
is treated as the hitting time of the current shot until the dead bird happens
as the last stroke.

The labeled records are illustrated below in the labeling interface following
the format in Section 3, and it can be instantaneously saved by clicking the
uploaded button. To reduce the labeling error, the S2 labeling tool also
incorporates a fool-proof design to avoid unintended manual clicking errors or
misjudgment. In addition, rallies in a match will be randomly sampled for
quality checking to verify the correctness of the stroke-level labels.

##  5\. Benchmarks

The primary goal of ShuttleSet is to obtain high-quality ML and DL models
capable of analyzing players’ patterns on a wide variety of tactical strokes.
Since ShuttleSet is not limited to the specific use as most of the standard
public datasets, the train, validation, and test sets are able to customize
based on the corresponding tasks instead of being pre-defined. As such, we
will relate to present various application tasks in this paper and apply
state-of-the-art methods to address these problems. They can be used as
benchmarks for researchers who aim to advance their approaches on turn-based
sequences following the corresponding train, validation, and test sets.

In this section, we present 3 application scenarios related to the performance
on ShuttleSet, including measuring the win probability of each stroke in a
rally (shot influence (Wang et al., 2021, 2022a), Section 5.1), forecasting
the future strokes (stroke forecasting (Wang et al., 2022b), Section 5.2) and
the movements of both players (movement forecasting (Chang et al., 2022),
Section 5.3) given the previous strokes. Following their evaluation settings
for data separation and evaluation metrics, the shot influence task formulates
the task as predicting the final outcome of a rally and adopts the latest 10
matches as the test set and the remaining for the train set to evaluate model
effectiveness for inferring the win probability of each stroke in the unseen
matches. The quality of a predicted probability is measured with the area
under the receiver operator curve (AUC), accuracy (ACC), and brier score (BR).
The coordinates of both shuttle and players’ movements are preprocessed with
standardization.

Both stroke forecasting and movement forecasting utilize the first 80% of
rallies of each match as the train set and the remaining 20% of rallies as the
test set to enforce the model equipped with the information of previous
matches of each player. As these forecasting tasks exist stochasticity, both
models are required to generate 10 sequences for each rally, and the sequence
closest to the ground truth is the final prediction for evaluation. In
addition, three different lengths of given strokes (i.e., given 8, 4, and 2 of
each rally) are used to examine the robustness of models in terms of the
variable encoding strokes. The evaluation metrics include cross entropy (CE)
for shot type evaluation, mean absolute error (MAE), and mean square error
(MSE) for location evaluation of both shuttles and two players.

###  5.1. Benchmark 1: Shot Influence

Wang et al. (2021, 2022a) introduces the shot influence task by framing as
predicting which player wins the rally by giving the stroke-level information
(e.g., shot type, player location, if returning around the head), which can be
used to measure the quality of each shot in a rally for enhancing players’
decision-making confidence. Generally, the state-of-the-art model is
ShuttleScorer (Wang et al., 2022a), which consists of a shot encoder for
encoding stroke-level information with various timestamps, a pattern extractor
for considering short-term patterns alternatively, and a rally encoder with a
Transformer encoder (Vaswani et al., 2017). We also implemented a bi-
directional gated recurrent unit (Bi-GRU) (Cho et al., 2014), Prototype
Sequence Network (ProSeNet) (Ming et al., 2019), DeepMoji (Felbo et al.,
2017), Ordered neurons LSTM (ON-LSTM) (Shen et al., 2019), Transformer
(Vaswani et al., 2017), Reformer (Kitaev et al., 2020), and TokenLearner (Ryoo
et al., 2021) as the baselines for comparison.

Table 2. Performance of the shot influence in terms of AUC, accuracy (ACC), and brier score (BR). Task | Model |  AUC ($\uparrow$) |  ACC ($\uparrow$) |  BR ($\downarrow$)
---|---|---|---|---
Shot Influence | ProSeNet | 0.5689 | 0.5000 | 0.2500
Bi-GRU | 0.6603 | 0.7054 | 0.2158
ON-LSTM | 0.6766 | 0.6808 | 0.2244
DeepMoji | 0.6786 | 0.6802 | 0.2238
Transformer | 0.6846 | 0.7570 | 0.1986
Reformer | 0.7046 | 0.7454 | 0.1994
TokenLearner | 0.7166 | 0.7698 | 0.1933
ShuttleScorer | 0.8371 | 0.7869 | 0.1574

Table 2 shows the baseline experiments of ShuttleScorer in the average of 5
random seeds. The hyper-parameters are identical to the implementation details
in the paper. As one can observe, Bi-GRU only reaches 0.6603, 0.7054, and
0.2158 in terms of AUC, ACC, and BR, respectively. In addition, the
performance of AUC, accuracy, and brier score of ShuttleScorer significantly
outperform Bi-GRU for 26.8%, 11.6%, and 27.1%, respectively, but there still
remains a gap to be improved since the current AUC score reaches 0.8371. This
gives a good indication that ShuttleSet provides more diverse matches and
poses a worthwhile challenge for the research community to design turn-based
approaches. Moreover, this also provides an intuition into the technique of
incorporating players with only a few matches as shown in Figures 2 and 3. It
is interesting to see how future work tackles these imbalanced and limited
players.

###  5.2. Benchmark 2: Stroke Forecasting

Table 3. Performance of the stroke forecasting for different observed stroke sequence lengths ($\tau$). |  | $\tau=8$ | $\tau=4$ | $\tau=2$
---|---|---|---|---
Task | Model | CE | MSE | MAE | CE | MSE | MAE | CE | MSE | MAE
Stroke Forecasting | Seq2Seq | 3.0588 | 1.8414 | 1.4840 | 3.2044 | 1.7839 | 1.4534 | 3.1116 | 1.7348 | 1.4404
CFLSTM | 2.8427 | 2.3835 | 1.7551 | 2.8979 | 2.4119 | 2.7676 | 2.8914 | 2.3348 | 1.7173
TF | 2.9555 | 1.9530 | 1.5289 | 3.0254 | 1.7790 | 1.4623 | 2.9477 | 1.8207 | 1.4667
dNRI | 2.9736 | 2.8238 | 1.9245 | 2.9689 | 2.6518 | 1.8691 | 2.9819 | 2.9194 | 1.9540
DMA-Nets | 3.1041 | 2.2182 | 1.6494 | 2.9932 | 2.1020 | 1.5940 | 2.9600 | 2.0101 | 1.5565
| ShuttleNet | 2.4125 | 1.8121 | 1.3582 | 2.4065 | 1.5673 | 1.3777 | 2.3901 | 1.5034 | 1.3615
Table 4. Performance of the movement forecasting for different observed stroke sequence lengths ($\tau$). |  | $\tau=8$ | $\tau=4$ | $\tau=2$
---|---|---|---|---
Task | Model | CE | MSE | MAE | CE | MSE | MAE | CE | MSE | MAE
Movement Forecasting | Seq2Seq | 2.3819 | 1.1575 | 1.6706 | 2.3971 | 1.1279 | 1.6556 | 2.4280 | 1.1463 | 1.6588
TF | 2.3969 | 1.1716 | 1.6772 | 2.4102 | 1.1580 | 1.6681 | 2.4551 | 1.1400 | 1.6559
ShuttleNet | 2.3801 | 1.1926 | 1.6727 | 2.3810 | 1.1312 | 1.6255 | 2.3655 | 1.0832 | 1.5852
dNRI | 2.9587 | 1.5351 | 1.6758 | 2.9626 | 1.6084 | 1.7219 | 2.9526 | 1.5313 | 1.6614
GCNPM | 2.3910 | 1.3145 | 1.7969 | 2.4027 | 1.2954 | 1.7703 | 2.3641 | 1.2535 | 1.7372
R-GCNPM | 2.3888 | 1.1128 | 1.6165 | 2.5110 | 1.1971 | 1.6571 | 2.4370 | 1.2636 | 1.6873
DyMF | 2.3566 | 1.0963 | 1.5875 | 2.3641 | 1.1107 | 1.6107 | 2.3146 | 1.1371 | 1.6142

In addition to the classification task, we also implement forecasting tasks
using ShuttleSet, which is generally more challenging than the classification
task. Wang et al. (2022b) proposes the stroke forecasting task, which is
beneficial for not only simulating players’ tactics but also assessing the
returning probability of future strokes for storytelling. This task is defined
as predicting future strokes including shot types and the corresponding
destination locations based on the previous strokes in a rally. We reproduced
the state-of-the-art model, ShuttleNet (Wang et al., 2022b), and the baselines
Seq2Seq (Sutskever et al., 2014), CF-LSTM (Xu et al., 2020), TF (Giuliari et
al., 2020), dNRI (Graber and Schwing, 2020), and DMA-Nets (Ji et al., 2021)
for illustrating the performance of shot types and landing locations. Seq2Seq
uses two long short-term memory (LSTM) (Sak et al., 2014) as an encoder and a
decoder. ShuttleNet contains a Transformer-based player extractor for modeling
styles of each player separately, a Transformer-based rally extractor for
capturing the current progress of a rally, and a position-aware gated fusion
network for fusing contexts of two players and rally based on the positions of
the rally. All hyper-parameters are the same as the original paper.

Table 3 reports the performance in different given stroke sequences, which can
be seen that ShuttleNet consistently outperforms the Seq2Seq model in terms of
both shot type prediction (CE) and location prediction (MSE and MAE).
Furthermore, the shot type performance of ShuttleNet does not change
significantly across various encoding strokes, while the performance of
Seq2Seq is more unstable. It is worth noting that both models degrade the shot
type performance compared with those in the original paper, which is because
there are 18 shot types while the original paper only used 10 shot types for
simplification. This result points out that the fine-grained definitions of
shot types in ShuttleSet increase the uncertainty of current models, which is
expected to close the gap from the research community.

###  5.3. Benchmark 3: Movement Forecasting

As stroke forecasting only tackles the future shot types and the corresponding
locations, Chang et al. (2022) introduces a more challenging task: movement
forecasting, where the goal is to predict the same targets of stroke
forecasting as well as the movements of players. This task makes the analyzing
scenarios more complete since coaches and players can take advantage of it for
investigating the physical ability and the habit of defensive positions of
players. DyMF (Chang et al., 2022) is the state-of-the-art graph-based model
for the movement forecasting task, which represents player interactions in a
rally as a strategic graph. DyMF consists of the interaction style extractor
for dynamically capturing the behaviors of mutual interactions and players’
tactics based on previous patterns, and the hierarchical fusion for fusing the
style of both players with their opponents’ styles and rally interactions. We
reproduce DyMF, Seq2Seq, TF (Giuliari et al., 2020), ShuttleNet (Wang et al.,
2022b), dNRI (Graber and Schwing, 2020), GCNPM (Kipf and Welling, 2017), and
R-GCNPM (Schlichtkrull et al., 2018) as the baselines and all hyper-parameters
are the same as in the original paper.

Table 4 illustrates the performance of Seq2Seq and DyMF in different observed
stroke sequences. DyMF consistently outperforms Seq2Seq for shot type and
movement performance, which further reaches lower uncertainty even for a more
challenging task compared with ShuttleNet in the stroke forecasting task. It
is interesting that Seq2Seq performs stably for shot type performance in this
task, while DyMF does not have a pattern of different lengths of given
strokes. This benchmark provides intuitions and raises an opportunity for the
graph-based models that are able to be utilized in the turn-based matches
instead of only using sequence-based models due to the constraint of
heterogeneous relations of players themselves and between players.

##  6\. Deployment: Visualization Platform

###  6.1. Deployment Setup

As the records in ShuttleSet are labeled from real-world top-ranking matches,
in addition to providing benchmarks for advancing turn-based methods, we also
construct a visualization platform to deploy these records for analysis. This
benefits the direct usage and reduces the technical gaps for the sports
community, e.g., coaching players and tactic investigations in data-driven
methods.

Specifically, we have built an analysis
platform555https://coachai.cs.nctu.edu.tw:55000 to visualize statistics from
ShuttleSet with respect to various aspects for coaches and players to review
instateneously without repeating match videos. Records in ShuttleSet are
converted to the JSON format for the Flask
engine666https://flask.palletsprojects.com/en/2.2.x/ to consume. The platform
consists of two major components, analysis for each match and summarized
comparisons from all matches of the same matchup. Here we illustrate the
analysis for each match as an example, which provides position analysis,
win/lose analysis (Wang et al., 2020), and shot types analysis.

###  6.2. Case Studies

Figure 7 illustrates three analysis techniques with Finals (Kento Momota vs.
Viktor Axelsen) and Semi-finals (Viktor Axelsen vs. Ng Ka Long Angus) in the
Men’s Singles of the Malaysia Masters 2020. The first stage shows the overall
losing distribution of each player separated by the grid system according to
Figure 4, where the area with a higher proportion is colored with lower
transparency and vice versa. The second stage demonstrates the players’
movements when each player returns the net shot. The blue and yellow points on
the upper court represent the locations of the player when the opponent
returns the stroke (i.e., the last stroke of the net shot), and the yellow and
blue points on the lower court represent the locations of the opponent when
the opponent returns the stroke. The green points represent the locations of
the player when returning the net shot. The third stage illustrates the
analysis of the hitting patterns from the selected location (grid 1 in the
figure), which demonstrates the five most common returning locations.

Figure 7. Three analysis techniques are presented in the Semi-finals (Viktor
Axelsen vs. Ng Ka Long Angus) and Finals (Kento Momota vs. Viktor Axelsen) of
the Malaysia Masters 2020. For each match, the winning player is displayed on
the left side and the losing player is on the right side.

In the first stage, we can observe that the right side of Ng Ka Long Angus is
the main weak area, which is also true for Viktor Axelsen, while during the
Finals, both Kento Momota and Viktor Axelsen have more uniform losing
distributions. This shows that it was hard for Ng Ka Long Angus to get the
winning point if he attacked to the middle of the court, while Kento Momota
was able to win the rally by returning to the middle court. In addition, the
outside distributions of Viktor Axelsen with Kento Momota are higher than
those with Ng Ka Long Angus (0.25 vs 0.37), which indicates that Kento Momota
is able to return strokes that cause Viktor Axelsen to make unforced errors
(i.e., return to outside by himself). As Kento Momota is a left-handed player,
we can see that the front court with a backhand (grid 1) is the main area
where he will lose a point in the front court, which suggests the opponent has
an opportunity to consider the strategy when returning to the front court.

To delve into the strategies of the net shot in the front court, the second
stage illustrates the hitting movements of both players for dealing with the
net shot. From the Semi-finals, it is obvious that the positions of Viktor
Axelsen mainly stand in the middle of the court when preparing to return the
net shot, while the positions of Ng Ka Long Angus show some patterns in the
back court. This analysis indicates that Ng Ka Long Angus is often mobilized
by Viktor Axelsen since he needs to move a greater distance from the back
court to the front court, which consumes more physical energy; however, this
issue does not occur for Viktor Axelsen as he is able to move to the center of
the court rapidly after he returns the stroke. Nonetheless, this situation
does not hold during the Finals, as Kento Momota and Viktor Axelsen fail to
stand in the middle of the court consistently to prepare for the net shot,
while Kento Momota stands nearer (yellow points in the right subfigure) the
net when Viktor Axelsen returns the net shot. Since Kento Momota is proficient
in performing net shots (Kevin, 2019), his opponents often find it difficult
to get the point from returning to the front court, which might be one of the
reasons why Kento Momota won the championship in the Malaysia Masters 2020.

In the third stage, we aim to further investigate the common returning
patterns from grid 1. We can see that Viktor Axelsen has the same common
returning areas when in the semi-finals and finals, while Ng Ka Long Angus is
more likely to return to the right side of his opponent. It is interesting
that Kento Momota is likely to return to the front court, which also verifies
the confident skills of net shots as the aforementioned advantage.

##  7\. Discussion

Throughout this paper, we claim that ShuttleSet’s wider variety of stroke-
level records is able to foster more advanced ML and DL models for mitigating
the gap between the research and sports communities. Although ShuttleSet is
the largest public turn-based sports dataset with microscopic information, we
illustrate several benchmarks with both classification tasks (Benchmark 1) and
forecasting tasks (Benchmarks 2 and 3) and provide some intuitions and
insights into the current state-of-the-art approaches. From the benchmarks,
the performance looks decent but still requires improvements to produce more
robust and better results for these tasks, which also provide opportunities
for both sequence- and graph-based approaches to tackle the turn-based
sequences. It is also expected that there will be other beneficial
applications that are proposed and investigated for turn-based sports
analytics, e.g., style comparison and limited records of players. Moreover, we
demonstrate the practical deployment usage of ShuttleSet with a visualization
platform using two matches as paradigms, which enables the sports community to
investigate players’ tactics from previous matches without repeating the
videos. The platform is designed with a friendly user interface and does not
require any technical background, which can be used directly.

##  8\. Conclusion

In this paper, we present the ShuttleSet dataset, which provides both sports
analysis and academic research communities with the largest and challenging
badminton dataset with stroke-level records to stimulate and improve novel ML
and DL methods on turn-based sequences. Instead of detecting automatically,
ShuttleSet was created by human annotation to obtain not only reliable and
structural ground truth from matches to strokes but also a wide variety of
general match videos of high-ranking men’s and women’s singles. In addition,
several statistics illustrate the compositions and intuitions of ShuttleSet to
mitigate the gaps for researchers not familiar with turn-based sports. Based
on the dataset, multiple paradigms from the experiments have showcased the
practical usage of turn-based analytics including an online visualization
platform for badminton coaches to investigate players’ tactics and patterns.
These applications represent the benchmarks for the research community to
advance their methods as well as the inspirations for the sports community to
foster multifaceted analysis applications. To date, there is still a
significant gap between the sports community and the research community on
turn-based research applications, and so we hope this work will inspire both
communities to advance techniques for closing that gap.

## Acknowledgments

This work was supported by the Ministry of Science and Technology of Taiwan
under Grants 110-2221-E-A49-063-MY3, 111-2622-E-A49-009, 111-3114-H-A49-001,
and 112-2425-H-A49-001.

## References

  * (1)
  * Azad et al. (2022) Abdus Salam Azad, Edward Kim, Qiancheng Wu, Kimin Lee, Ion Stoica, Pieter Abbeel, Alberto L. Sangiovanni-Vincentelli, and Sanjit A. Seshia. 2022.  Programmatic Modeling and Generation of Real-Time Strategic Soccer Environments for Reinforcement Learning. In _AAAI_. AAAI Press, 6028–6036.
  * Breno (2018) Frederico Breno. 2018\.  Simple dataset with decisions about playing tennis.  https://www.kaggle.com/datasets/fredericobreno/play-tennis.
  * Chang et al. (2022) Kai-Shiang Chang, Wei-Yao Wang, and Wen-Chih Peng. 2022\.  Where Will Players Move Next? Dynamic Graphs and Hierarchical Fusion for Movement Forecasting in Badminton.  _CoRR_ abs/2211.12217 (2022).
  * Chen et al. (2022) Xiusi Chen, Jyun-Yu Jiang, Kun Jin, Yichao Zhou, Mingyan Liu, P. Jeffrey Brantingham, and Wei Wang. 2022.  ReLiable: Offline Reinforcement Learning for Tactical Strategies in Professional Basketball Games. In _CIKM_. ACM, 3023–3032.
  * Cho et al. (2014) Kyunghyun Cho, Bart van Merrienboer, Dzmitry Bahdanau, and Yoshua Bengio. 2014.  On the Properties of Neural Machine Translation: Encoder-Decoder Approaches. In _SSST@EMNLP_. Association for Computational Linguistics, 103–111.
  * Chu and Situmeang (2017) Wei-Ta Chu and Samuel Situmeang. 2017.  Badminton Video Analysis based on Spatiotemporal and Stroke Features. In _ICMR_. ACM, 448–451.
  * Decroos et al. (2019) Tom Decroos, Lotte Bransen, Jan Van Haaren, and Jesse Davis. 2019.  Actions Speak Louder than Goals: Valuing Player Actions in Soccer. In _KDD_. ACM, 1851–1861.
  * Decroos et al. (2020) Tom Decroos, Lotte Bransen, Jan Van Haaren, and Jesse Davis. 2020.  VAEP: An Objective Approach to Valuing On-the-Ball Actions in Soccer (Extended Abstract). In _IJCAI_. ijcai.org, 4696–4700.
  * Deliège et al. (2021) Adrien Deliège, Anthony Cioppa, Silvio Giancola, Meisam Jamshidi Seikavandi, Jacob V. Dueholm, Kamal Nasrollahi, Bernard Ghanem, Thomas B. Moeslund, and Marc Van Droogenbroeck. 2021.  SoccerNet-v2: A Dataset and Benchmarks for Holistic Understanding of Broadcast Soccer Videos. In _CVPR Workshops_. Computer Vision Foundation / IEEE, 4508–4519.
  * Fang and Sun (2021) Lei Fang and Mingsi Sun. 2021.  Motion recognition technology of badminton players in sports video images.  _Future Gener. Comput. Syst._ 124 (2021), 381–389.
  * Felbo et al. (2017) Bjarke Felbo, Alan Mislove, Anders Søgaard, Iyad Rahwan, and Sune Lehmann. 2017.  Using millions of emoji occurrences to learn any-domain representations for detecting sentiment, emotion and sarcasm. In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (EMNLP’17))_. ACL, Copenhagen, Denmark, 1615–1625.
  * Ghosh et al. (2018) Anurag Ghosh, Suriya Singh, and C. V. Jawahar. 2018\.  Towards Structured Analysis of Broadcast Badminton Videos. In _WACV_. IEEE Computer Society, 296–304.
  * Giuliari et al. (2020) Francesco Giuliari, Irtiza Hasan, Marco Cristani, and Fabio Galasso. 2020.  Transformer Networks for Trajectory Forecasting. In _25th International Conference on Pattern Recognition_. IEEE, 10335–10342.
  * Graber and Schwing (2020) Colin Graber and Alexander G. Schwing. 2020.  Dynamic Neural Relational Inference. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_.
  * Haladjian et al. (2020) Juan Haladjian, Daniel Schlabbers, Sajjad Taheri, Max Tharr, and Bernd Bruegge. 2020.  Sensor-based Detection and Classification of Soccer Goalkeeper Training Exercises.  _ACM Trans. Internet Things_ 1, 2 (2020), 12:1–12:20.
  * Han et al. (2022) Yewon Han, Jaeho Kim, Hon Keung Tony Ng, and Seong W. Kim. 2022\.  Logistic Regression Model for a Bivariate Binomial Distribution with Applications in Baseball Data Analysis.  _Entropy_ 24, 8 (2022), 1138.
  * Huang et al. (2022) Yu-Hsien Huang, Yung-Chang Huang, Hao Syuan Lee, Tsì-Uí Ik, and Chih-Chuan Wang. 2022.  S${}^{\mbox{2}}$-Labeling: Shot-By-Shot Microscopic Badminton Singles Tactical Dataset. In _APNOMS_. IEEE, 1–6.
  * Ito and Gotoh (2018) Hiroaki Ito and Yusuke Gotoh. 2018.  Technical Support System for Baseball Beginners by Analyzing Batting Stance with Sensors. In _MoMM_. ACM, 128–132.
  * Jhuang et al. (2013) Hueihan Jhuang, Juergen Gall, Silvia Zuffi, Cordelia Schmid, and Michael J. Black. 2013.  Towards Understanding Action Recognition. In _ICCV_. IEEE Computer Society, 3192–3199.
  * Ji et al. (2021) Taoran Ji, Nathan Self, Kaiqun Fu, Zhiqian Chen, Naren Ramakrishnan, and Chang-Tien Lu. 2021.  Dynamic Multi-Context Attention Networks for Citation Forecasting of Scientific Publications. In _Thirty-Fifth AAAI Conference on Artificial Intelligence_. 7953–7960.
  * Karpathy et al. (2014) Andrej Karpathy, George Toderici, Sanketh Shetty, Thomas Leung, Rahul Sukthankar, and Li Fei-Fei. 2014\.  Large-Scale Video Classification with Convolutional Neural Networks. In _CVPR_. IEEE Computer Society, 1725–1732.
  * Kevin (2019) Kevin. 2019.  Kento Momota Badminton – A Player Study.  https://getgoodatbadminton.com/kento-momota-badminton-a-player-study.
  * Kim et al. (2022) Hyunsung Kim, Bit Kim, Dongwook Chung, Jinsung Yoon, and Sang-Ki Ko. 2022.  SoccerCPD: Formation and Role Change-Point Detection in Soccer Matches Using Spatiotemporal Tracking Data. In _KDD_. ACM, 3146–3156.
  * Kincaid et al. (2021) Jack A. Kincaid, Fengchen Gong, Tianjie Jia, Hong Z. Tan, Casey Kohr, and Gary R. Bertoline. 2021\.  A Study on VR Training of Baseball Athletes. In _HCI (20)_ _(Lecture Notes in Computer Science, Vol. 12781)_. Springer, 187–204.
  * Kipf and Welling (2017) Thomas N. Kipf and Max Welling. 2017.  Semi-Supervised Classification with Graph Convolutional Networks. In _ICLR (Poster)_. OpenReview.net.
  * Kitaev et al. (2020) Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. 2020\.  Reformer: The Efficient Transformer. In _8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020_. OpenReview.net.
  * Li et al. (2021) Yixuan Li, Lei Chen, Runyu He, Zhenzhi Wang, Gangshan Wu, and Limin Wang. 2021\.  MultiSports: A Multi-Person Video Dataset of Spatio-Temporally Localized Sports Actions. In _ICCV_. IEEE, 13516–13525.
  * Ming et al. (2019) Yao Ming, Panpan Xu, Huamin Qu, and Liu Ren. 2019\.  Interpretable and Steerable Sequence Learning via Prototypes. In _Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (KDD’19)_. ACM, AK, USA, 903–913.
  * Pillitteri et al. (2021) Guglielmo Pillitteri, Ewan Thomas, Giuseppe Battaglia, Giovanni Angelo Navarra, Antonino Scardina, Viviana Gammino, Dario Ricchiari, and Marianna Bellafiore. 2021\.  Validity and Reliability of an Inertial Sensor Device for Specific Running Patterns in Soccer.  _Sensors_ 21, 21 (2021), 7255.
  * Rahman (2022) Kalilur Rahman. 2022\.  ATP Tour Ranking - Decade-wise and year-wise.  https://www.kaggle.com/datasets/kalilurrahman/atp-tennis-player-ranking-dataset.
  * Rodriguez et al. (2008) Mikel D. Rodriguez, Javed Ahmed, and Mubarak Shah. 2008\.  Action MACH a spatio-temporal Maximum Average Correlation Height filter for action recognition. In _CVPR_. IEEE Computer Society.
  * Ryoo et al. (2021) Michael S. Ryoo, A. J. Piergiovanni, Anurag Arnab, Mostafa Dehghani, and Anelia Angelova. 2021\.  TokenLearner: Adaptive Space-Time Tokenization for Videos. In _Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual_. 12786–12797.
  * Sak et al. (2014) Hasim Sak, Andrew W. Senior, and Françoise Beaufays. 2014\.  Long Short-Term Memory Based Recurrent Neural Network Architectures for Large Vocabulary Speech Recognition.  _CoRR_ abs/1402.1128 (2014).
  * Schlichtkrull et al. (2018) Michael Sejr Schlichtkrull, Thomas N. Kipf, Peter Bloem, Rianne van den Berg, Ivan Titov, and Max Welling. 2018.  Modeling Relational Data with Graph Convolutional Networks. In _ESWC_ _(Lecture Notes in Computer Science, Vol. 10843)_. Springer, 593–607.
  * Shen et al. (2019) Yikang Shen, Shawn Tan, Alessandro Sordoni, and Aaron C. Courville. 2019.  Ordered Neurons: Integrating Tree Structures into Recurrent Neural Networks. In _7th International Conference on Learning Representations (ICLR’19)_. OpenReview.net, LA, USA.
  * Simpson et al. (2022) Ian Simpson, Ryan J. Beal, Duncan Locke, and Timothy J. Norman. 2022.  Seq2Event: Learning the Language of Soccer Using Transformer-based Match Event Prediction. In _KDD_. ACM, 3898–3908.
  * Sutskever et al. (2014) Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. 2014\.  Sequence to Sequence Learning with Neural Networks. In _NIPS_. 3104–3112.
  * Valldecabres et al. (2020) Raúl Valldecabres, Claudio A Casal, João Guilherme Cren Chiminazzo, and Ana María De Benito. 2020\.  Players’ on-court movements and contextual variables in badminton world championship.  _Frontiers in Psychology_ 11 (2020), 1567.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017\.  Attention is All you Need. In _Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems_. 5998–6008.
  * Wang (2022) Wei-Yao Wang. 2022\.  Modeling Turn-Based Sequences for Player Tactic Applications in Badminton Matches. In _CIKM_. ACM, 5128–5131.
  * Wang et al. (2021) Wei-Yao Wang, Teng-Fong Chan, Hui-Kuo Yang, Chih-Chuan Wang, Yao-Chung Fan, and Wen-Chih Peng. 2021.  Exploring the Long Short-Term Dependencies to Infer Shot Influence in Badminton Matches. In _ICDM_. IEEE, 1397–1402.
  * Wang et al. (2022a) Wei-Yao Wang, Teng-Fong Chan, Hui-Kuo Yang, Chih-Chuan Wang, Yao-Chung Fan, and Wen-Chih Peng. 2022a.  How Is the Stroke? Inferring Shot Influence in Badminton Matches via Long Short-Term Dependencies.  _ACM Trans. Intell. Syst. Technol._ 14, 1 (2022).
  * Wang et al. (2022b) Wei-Yao Wang, Hong-Han Shuai, Kai-Shiang Chang, and Wen-Chih Peng. 2022b.  ShuttleNet: Position-Aware Fusion of Rally Progress and Player Styles for Stroke Forecasting in Badminton. In _AAAI_. AAAI Press, 4219–4227.
  * Wang et al. (2020) Wei-Yao Wang, Kai-Shiang Chang, Teng-Fong Chen, Chih-Chuan Wang, Wen-Chih Peng, and Chih-Wei Yi. 2020\.  Badminton Coach AI: A Badminton Match Data Analysis Platform Based on Deep Learning.  _Physical Education Journal_ 53, 2 (2020), 201–213.
  * Wu et al. (2022) Fei Wu, Qingzhong Wang, Jian Bian, Haoyi Xiong, Ning Ding, Feixiang Lu, Jun Cheng, and Dejing Dou. 2022\.  A Survey on Video Action Recognition in Sports: Datasets, Methods and Applications.  _CoRR_ abs/2206.01038 (2022).
  * Xu et al. (2020) Yi Xu, Jing Yang, and Shaoyi Du. 2020.  CF-LSTM: Cascaded Feature-Based Long Short-Term Networks for Predicting Pedestrian Trajectory. In _The Thirty-Fourth AAAI Conference on Artificial Intelligence_. 12541–12548.
  * Yanai et al. (2022) Chen Yanai, Adir Solomon, Gilad Katz, Bracha Shapira, and Lior Rokach. 2022.  Q-Ball: Modeling Basketball Games Using Deep Reinforcement Learning. In _AAAI_. AAAI Press, 8806–8813.
  * Yoshikawa et al. (2021) Yui Yoshikawa, Hidehiko Shishido, Masashi Suita, Yoshinari Kameda, and Itaru Kitahara. 2021\.  Shot detection using skeleton position in badminton videos. In _International Workshop on Advanced Imaging Technology 2021_ , Vol. 11766. 312 – 317.
  * Yue et al. (2014) Yisong Yue, Patrick Lucey, Peter Carr, Alina Bialkowski, and Iain A. Matthews. 2014.  Learning Fine-Grained Spatial Models for Dynamic Sports Play Prediction. In _ICDM_. IEEE Computer Society, 670–679.
  * Zhu et al. (2022) Kevin Zhu, Alexander Wong, and John McPhee. 2022.  FenceNet: Fine-grained Footwork Recognition in Fencing. In _CVPR Workshops_. IEEE, 3588–3597.

##  Appendix A Appendix

Table 5. Detailed information of each labeled match. Tournament | Round | Year | Set | Duration (m) | Winner | Loser
---|---|---|---|---|---|---
Fuzhou Open 2018 | Finals | 2018 | 3 | 67 | Kento Momota | Chou Tien Chen
Denmark Open 2018 | Finals | 2018 | 3 | 77 | Kento Momota | Chou Tien Chen
Malaysia Open 2018 | Quarter-Finals | 2018 | 2 | 53 | Kento Momota | Chou Tien Chen
Fuzhou Open 2019 | Finals | 2019 | 3 | 83 | Kento Momota | Chou Tien Chen
World Tour Finals | Group Stage | 2019 | 2 | 52 | Chen Long | Chou Tien Chen
KOREA OPEN 2019 | Finals | 2019 | 2 | 53 | Kento Momota | Chou Tien Chen
Denmark Open 2019 | Quarter-Finals | 2019 | 2 | 54 | Chen Long | Chou Tien Chen
Fuzhou Open 2019 | Semi-Finals | 2019 | 2 | 45 | Chou Tien Chen | Anders Antonsen
Sudirman Cup 2019 | Quarter-Finals | 2019 | 2 | 36 | Chou Tien Chen | Jonatan Christie
Sudirman Cup 2019 | Group Stage | 2019 | 2 | 41 | Chou Tien Chen | Ng Ka Long Angus
Indonesia Open 2019 | Quarter-Finals | 2019 | 3 | 76 | Chou Tien Chen | Jonatan Christie
Thailand Masters 2020 | Semi-Finals | 2020 | 2 | 37 | Ng Ka Long Angus | Shi Yu Qi
Malaysia Masters 2020 | Semi-Finals | 2020 | 2 | 44 | Viktor Axelsen | Ng Ka Long Angus
Malaysia Masters 2020 | Quarter-Finals | 2020 | 3 | 65 | Ng Ka Long Angus | Jonatan Christie
Malaysia Masters 2020 | Quarter-Finals | 2020 | 3 | 68 | Viktor Axelsen | Chen Long
Malaysia Masters 2020 | Finals | 2020 | 2 | 54 | Kento Momota | Viktor Axelsen
Indonesia Masters 2020 | Quarter-Finals | 2020 | 3 | 68 | Anders Antonsen | Jonatan Christie
Indonesia Masters 2020 | Finals | 2020 | 3 | 71 | Anthony Sinisuka Ginting | Anders Antonsen
Indonesia Masters 2020 | Semi-Finals | 2020 | 2 | 43 | Anthony Sinisuka Ginting | Viktor Axelsen
All England Open 2020 | Quarter-Finals | 2020 | 2 | 40 | Viktor Axelsen | Shi Yu Qi
HSBC BWF World Tour Finals 2020 | Quarter-Finals | 2020 | 3 | 55 | An Se Young | Carolina Marin
HSBC BWF World Tour Finals 2020 | Quarter-Finals | 2020 | 3 | 50 | Anthony Sinisuka Ginting | Lee Zii Jia
HSBC BWF World Tour Finals 2020 | Quarter-Finals | 2020 | 2 | 34 | Evgeniya Kosetskaya | Michelle Li
HSBC BWF World Tour Finals 2020 | Quarter-Finals | 2020 | 3 | 65 | Ng Ka Long Angus | Kidambi Srikanth
HSBC BWF World Tour Finals 2020 | Quarter-Finals | 2020 | 2 | 42 | Pusarla V. Sindhu | Pornpawee Chochuwong
HSBC BWF World Tour Finals 2020 | Semi-Finals | 2020 | 2 | 40 | Carolina Marin | Pornpawee Chochuwong
HSBC BWF World Tour Finals 2020 | Semi-Finals | 2020 | 3 | 60 | Anders Antonsen | Viktor Axelsen
Yonex Thailand Open 2021 | Quarter-Finals | 2021 | 2 | 44 | An Se Young | Ratchanok Intanon
Yonex Thailand Open 2021 | Quarter-Finals | 2021 | 2 | 45 | Mia Blichfeldt | Busanan Ongbamrungphan
Yonex Thailand Open 2021 | Quarter-Finals | 2021 | 2 | 44 | Ng Ka Long Angus | Lee Cheuk Yiu
Yonex Thailand Open 2021 | Quarter-Finals | 2021 | 3 | 66 | Anthony Sinisuka Ginting | Rasmus Gemke
Yonex Thailand Open 2021 | Quarter-FInals | 2021 | 2 | 44 | Carolina Marin | Supanida Katethong
Yonex Thailand Open 2021 | Quarter-Finals | 2021 | 2 | 40 | Viktor Axelsen | Jonatan Christie
Yonex Thailand Open 2021 | Semi-Finals | 2021 | 3 | 63 | Viktor Axelsen | Anthony Sinisuka Ginting
Yonex Thailand Open 2021 | Finals | 2021 | 2 | 44 | Viktor Axelsen | Ng Ka Long Angus
Toyota Thailand Open 2021 | Quarter-Finals | 2021 | 2 | 44 | An Se Young | Pornpawee Chochuwong
Toyota Thailand Open 2021 | Quarter-Finals | 2021 | 3 | 81 | Anders Antonsen | Sameer Verma
Toyota Thailand Open 2021 | Quarter-Finals | 2021 | 2 | 34 | Carolina Marin | Neslihan Yigit
Toyota Thailand Open 2021 | Quarter-Finals | 2021 | 3 | 77 | Hans-Kristian Solberg Vittinghus | Lee Cheuk Yiu
Toyota Thailand Open 2021 | Quarter-Finals | 2021 | 2 | 41 | Viktor Axelsen | Luiw Daren
Toyota Thailand Open 2021 | Quarter-Finals | 2021 | 2 | 38 | Ratchanok Intanon | Pusarla V. Sindhu
Toyota Thailand Open 2021 | Semi-Finals | 2021 | 2 | 51 | Carolina Marin | An Se Young
Toyota Thailand Open 2021 | Semi-Finals | 2021 | 2 | 45 | Hans-Kristian Solberg Vittinghus | Anders Antonsen
Toyota Thailand Open 2021 | Finals | 2021 | 2 | 40 | Viktor Axelsen | Hans-Kristian Solberg Vittinghus

###  A.1. Details of labeled matches.

Table 5 presents the detailed information of each labeled match, including the
tournament name, round, year, played set, duration, winner, and loser.
Generally, the matches were collected from 2018 to 2021 with high-ranking
players.


pnasresearcharticle Handling high sparsity presents a significant challenge in
big data analysis. Currently, there are many strategies, e.g., imputation,
regularization, sampling, dimensionality reduction. Yet, a fundamental
question remains largely unanswered, particularly in the field of biology: to
what degree does sparsity reflect meaningful disparities between groups rather
than being largely a result of technical errors in measurement? In this study,
we conducted a systematic reanalysis of datasets encompassing the
transcriptome, proteome, metabolome, immune profiling, microbiome, and social
sciences. We demonstrate that sparsity differences are highly meaningful and
carry as much significance as the more commonly used measures of dissimilarity
based on mean differences.

# Matrix dissimilarities based on differences in moments and sparsity

Tuobang Li  Guangdong Technion-Israel Institute of Technology, Shantou 515063,
China  Technion-Israel Institute of Technology, Haifa 32000, Israel
University of California, Berkeley, CA 94720

###### Abstract

Generating a dissimilarity matrix is typically the first step in big data
analysis. Although numerous methods exist, such as Euclidean distance,
Minkowski distance, Manhattan distance, Bray–Curtis dissimilarity, Jaccard
similarity and Dice dissimilarity, it remains unclear which factors drive
dissimilarity between groups. In this paper, we introduce an approach based on
differences in moments and sparsity. We show that this method can delineate
the key factors underlying group differences. For example, in biology, mean
dissimilarity indicates differences driven by up/down-regulated gene
expressions, standard deviation dissimilarity reflects the heterogeneity of
response to treatment, and sparsity dissimilarity corresponds to differences
prompted by the activation/silence of genes. Through extensive reanalysis of
genome, transcriptome, proteome, metabolome, immune profiling, microbiome, and
social science datasets, we demonstrate insights not captured in previous
studies. For instance, it shows that the sparsity dissimilarity is as
effective as the mean dissimilarity in predicting the alleviation effects of a
COVID-19 drug, suggesting that sparsity dissimilarity is highly meaningful.

###### keywords:

Metabolism $|$ Moments $|$ Mass spectra

This manuscript was compiled on

Dissimilarity measures are critical in big data analysis. They quantify how
different or similar two data points are. Different measures can significantly
affect the performance of clustering algorithms and many other machine
learning models. This is because different measures indeed reflect distinct
facets of the differences. For example, the Euclidean distance, the most well-
known distance, is the square root of the sum of the squares of the
differences in each dimension. This effectively captures the average shift in
each dimension, aptly termed a measure of mean changes. Other measures of mean
changes include Minkowski distance, Manhattan distance, and Average distance.
A notable limitation of these measures of mean changes is that, the largest
feature would dominate the others. Instead of normalizing the dataset, a
different solution proposed in this report is introduced as: let
$P=(p_{1},p_{2},\ldots,p_{n})$ and $Q=(q_{1},q_{2},\ldots,q_{n})$ be two
points in $\mathbb{R}^{n}$. The mean dissimilarity $\mu\Delta_{\hat{L}_{n}}$
between $P$ and $Q$ is defined as:

$\mu\Delta_{\hat{L}_{n}}(P,Q)=\hat{L}_{n}\\{|p_{1}-q_{1}|,|p_{2}-q_{2}|,\ldots,|p_{n}-q_{n}|\\},$

where $|p_{i}-q_{i}|$ represents the absolute difference between the $i$-th
coordinates of $P$ and $Q$, and $\hat{L}_{n}\\{\cdot\\}$ denotes a location
estimate of a set of values. In this report, Hodges-Lehmann estimator (H-L)
(1) is used. When the objective is to compare dissimilarities between groups
rather than individual samples, we extend the above definition for points to
matrices. Suppose we have two matrices $A$ and $B$ in $\mathbb{R}^{m\times
n_{1}}$ and $\mathbb{R}^{m\times n_{2}}$, represented as:
$A=\begin{bmatrix}a_{11}&a_{12}&\cdots&a_{1n_{1}}\\\
a_{21}&a_{22}&\cdots&a_{2n_{1}}\\\ \vdots&\vdots&\ddots&\vdots\\\
a_{m1}&a_{m2}&\cdots&a_{mn_{1}}\end{bmatrix}$ and
$B=\begin{bmatrix}b_{11}&b_{12}&\cdots&b_{1n_{2}}\\\
b_{21}&b_{22}&\cdots&b_{2n_{2}}\\\ \vdots&\vdots&\ddots&\vdots\\\
b_{m1}&b_{m2}&\cdots&b_{mn_{2}}\end{bmatrix}$. For each row $i$, compute the
mean of the row in matrix $A$ as
$\bar{a}_{i}=\frac{1}{n_{1}}\sum_{j=1}^{n_{1}}a_{ij}$ and the mean of the row
in matrix $B$ as $\bar{b}_{i}=\frac{1}{n_{2}}\sum_{j=1}^{n_{2}}b_{ij}$. We
then define the set of absolute differences between the means of corresponding
rows of $A$ and $B$ as
$\Delta_{\mu}=\\{|\bar{a}_{1}-\bar{b}_{1}|,|\bar{a}_{2}-\bar{b}_{2}|,\ldots,|\bar{a}_{m}-\bar{b}_{m}|\\}$.
The mean dissimilarity $\mu\Delta_{\hat{L}_{m}}$ between the two matrices $A$
and $B$ is then defined as:

$\mu\Delta_{\hat{L}_{m}}(A,B)=\hat{L}_{m}(\Delta_{m}).$

Then, analogously, for each row $i$, compute the standard deviation
$\sigma_{a_{i}}$ for the row in matrix $A$ and $\sigma_{b_{i}}$ for the row in
matrix $B$. Define the sets of absolute differences for standard deviation as
$\Delta_{\sigma}=\\{|\sigma_{a_{1}}-\sigma_{b_{1}}|,|\sigma_{a_{2}}-\sigma_{b_{2}}|,\ldots,|\sigma_{a_{m}}-\sigma_{b_{m}}|\\}$.
The standard deviation dissimilarity between the two matrices $A$ and $B$ are
then defined as:

$\sigma\Delta_{\hat{L}_{m}}(A,B)=\hat{L}_{m}(\Delta_{\sigma}).$

Dissimilarities based on higher-order standardized moments can also be
defined, although their practical relevance may be less pronounced.

Another kind of dissimilarity focuses on the commonality of attribute values,
exemplified by metrics such as the Bray–Curtis dissimilarity, Jaccard
similarity and Dice dissimilarity. These metrics can be adept at capturing the
disparities in data sparsity between two points, hence we refer to them as
measures of sparsity changes. In this report, we introduce sparsity
dissimilarity defined as follows: for each row $i$, let $z_{a_{i}}$ be the
percentage of zeros in the row in matrix $A$ and $z_{b_{i}}$ be the percentage
of zeros in the row in matrix $B$. Also, let $\bar{a}_{i}$ and $\bar{b}_{i}$
represent the mean of the rows in matrices $A$ and $B$, respectively. The
sparsity dissimilarity $s\Delta(A,B)$ is defined as:

$s\Delta(A,B)=\sum_{i=1}^{m}s_{i}w_{i}$

where $s_{i}=|\bar{a}_{i}-\bar{b}_{i}|$, $w_{i}=|z_{a_{i}}-z_{b_{i}}|$.

By substituting absolute differences with relative differences, we obtain the
relative mean, standard deviation, and sparsity dissimilarities, denoted as
$\mu\delta_{\hat{L}_{m}}(A,B)$, $\sigma\delta_{\hat{L}_{m}}(A,B)$, and
$s\delta_{\hat{L}_{m}}(A,B)$, respectively.

Finally, the mean, standard deviation, and sparsity dissimilarity and their
relative dissimilarity can be standardized by the mean of {$\bar{a}_{1}$,
$\cdots$, $\bar{a}_{i}$, $\cdots$, $\bar{a}_{m}$, $\bar{b}_{1}$, $\cdots$,
$\bar{b}_{i}$, $\cdots$, $\bar{b}_{m}$}, {$\sigma_{a_{1}}$, $\cdots$,
$\sigma_{a_{i}}$, $\cdots$, $\sigma_{a_{m}}$, $\sigma_{b_{1}}$, $\cdots$,
$\sigma_{b_{i}}$, $\cdots$, $\sigma_{b_{m}}$}, and the half sum of
{$\bar{a}_{1}$, $\cdots$, $\bar{a}_{i}$, $\cdots$, $\bar{a}_{m}$,
$\bar{b}_{1}$, $\cdots$, $\bar{b}_{i}$, $\cdots$, $\bar{b}_{m}$}. The
standardized versions are denoted as $\mu\Delta_{\hat{L}_{m}}^{*}(A,B)$,
$\sigma\Delta_{\hat{L}_{m}}^{*}(A,B)$, $s\Delta_{\hat{L}_{m}}^{*}(A,B)$,
$\mu\delta_{\hat{L}_{m}}^{*}(A,B)$, $\sigma\delta_{\hat{L}_{m}}^{*}(A,B)$, and
$s\delta_{\hat{L}_{m}}^{*}(A,B)$, respectively.

Often, the variables in the matrixes are interelated, and these relations, in
most cases, can be demonstrated by a phylogenetic tree. Converting the
phylogenetic tree into corresponding weights by their length from leave to
root (Algorithm 1), the mean, standard deviation, and sparsity dissimilarity
can be further weighted by the phylogenetic tree.

## Results

In this study, we revisited the findings reported by Wyler et al.(2) regarding
the protective effects of HSP90 inhibition (17-AAG) on alveolar epithelial
cells (AECs) in the context of SARS-CoV-2 infection. We applied our
dissimilarity measures to RNA-sequencing data obtained from these cells,
whether exposed to SARS-CoV-2 or not. Our results showed that treatment with
the HSP90 inhibitor, 17-AAG, at a concentration of 200 nM significantly
reduced mean and sparsity dissimilarities (from infection to non-infection)
compared to the solvent control, besides the 24 hour group, suggesting a
weakened response to infection. Furthermore, the treatment has little impact
on the standard deviation dissimilarity, indicating that the cellular response
to this inhibitor is generally homogeneous. These findings underscore the
potential of our dissimilarity measure as a tool for quantitatively assessing
the overall impact of therapeutic agents on cellular dynamics.

Table 1: Standardized mean, standard deviation, and sparsity dissimilarities
of Wyler et al.’s RNA sequencing dataset

Treatment | Time | Comparison | $\mu\Delta_{\text{H-L}}^{*}$ | CI | $\sigma\Delta_{\text{H-L}}^{*}$ | CI | $s\Delta_{\text{H-L}}^{*}$ | CI
---|---|---|---|---|---|---|---|---
AAG | 24h | SARS-CoV-2-Mock | 0.032 | (0.030,0.033) | 0.295 | (0.282,0.307) | 0.028 | (0.026,0.030)
AAG | 48h | SARS-CoV-2-Mock | 0.037 | (0.035,0.039) | 0.282 | (0.274,0.294) | 0.031 | (0.029,0.033)
AAG | 72h | SARS-CoV-2-Mock | 0.064 | (0.062,0.066) | 0.208 | (0.199,0.214) | 0.029 | (0.028,0.032)
DMSO | 24h | SARS-CoV-2-Mock | 0.024 | (0.023,0.025) | 0.250 | (0.236,0.264) | 0.030 | (0.027,0.031)
DMSO | 48h | SARS-CoV-2-Mock | 0.148 | (0.142,0.153) | 0.234 | (0.223,0.244) | 0.110 | (0.074,0.158)
DMSO | 72h | SARS-CoV-2-Mock | 0.091 | (0.089,0.095) | 0.277 | (0.267,0.288) | 0.062 | (0.056,0.071)

AAG: 17-AAG, a kind of HSP90 inhibitor; DMSO: Solvent control. The sparsity
dissimilarity is in units of $10^{-3}$. The relative dissimilarities can be
found in the SI Dataset S1.

## Data and Software Availability

All data are included in the brief report and SI Dataset S1. All codes have
been deposited in GitHub. Please include your acknowledgments here, set in a
single paragraph. Please do not include any acknowledgments in the Supporting
Information, or anywhere else in the manuscript.

## References

  * (1) J Hodges Jr, E Lehmann, Estimates of location based on rank tests.  The Annals of Mathematical Statistics 34, 598–611 (1963).
  * (2) E Wyler, et al., Transcriptomic profiling of sars-cov-2 infected human cell lines identifies hsp90 as target for covid-19 therapy.  IScience 24, 102151 (2021).