plus1sp

Nakahara et al.

[3]Keisuke Fujii

# Pitching strategy evaluation via stratified analysis using propensity score

Hiroshi Nakahara  Graduate School of Informatics, Nagoya University, Email:
<EMAIL_ADDRESS>Kazuya Takeda  Graduate School of
Informatics, Nagoya University, Email<EMAIL_ADDRESS>
###### Abstract

Recent measurement technologies enable us to analyze baseball at higher
levels. There are, however, still many unclear points around the pitching
strategy. The two elements make it difficult to measure the effect of pitching
strategy. First, most public datasets do not include location data where the
catcher demands a ball, which is essential information to obtain the battery's
intent. Second, there are many confounders associated with pitching/batting
results when evaluating pitching strategy. We here clarify the effect of
pitching attempts to a specific location, e.g., inside or outside. We employ a
causal inference framework called stratified analysis using a propensity score
to evaluate the effects while removing the effect of disturbing factors. We
used a pitch-by-pitch dataset of Japanese professional baseball games held in
2014-2019, which includes location data where the catcher demands a ball. The
results reveal that an outside pitching attempt is more effective than an
inside one to minimize allowed run on average. Besides, the stratified
analysis shows that the outside pitching attempt was always effective despite
the magnitude of the estimated batter's ability, and the ratio of pitched
inside for pitcher/batter. Our analysis would provide practical insights into
selecting a pitching strategy to minimize allowed runs.

##  1 Introduction

Baseball is a good friend of statistics due to its features. Each scene of
baseball is discrete, making it easy to allocate responsibilities of each play
to players. Many formulas and statistics (stats) were developed and employed
to evaluate players' performance from a long time ago (James, 2010, Lewis,
2004, Click and Keri, 2006, Tango etal., 2007, Beneventano etal., 2012, Costa
etal., 2012). Most stats are computed using batting, pitching, and fielding
results (for detail, see Section LABEL:sec:related). Currently, measurement
technologies and data platforms are getting developed, and all sensing data
measured in Major League Baseball (MLB) games are provided and accessible. It
enables us to develop stats to evaluate player's performance using speed and
angle of batted balls, such as expected weighted On Base Average (xwOBA) and
expected Earned Run Average (xERA) (MLB.com, 2015b, a), which can acquire more
important information and findings to win. ``Fly ball revolution'', which
specifies the effective speed and angle of batted balls to hit a long hit, is
a symbol of that.

While a lot of stats are widely used to evaluate a player's performance, there
are many unclear points around the pitching strategy. The pitching strategy
here means a strategy of battery (pitcher and catcher) to defeat the opponent
batter. Battery combines various pitch types, courses, and pitch combinations
(e.g., inside fastball after outside breaking ball). During a game, a catcher
often suggests where/which to pitch the next ball to a pitcher (especially in
Japanese Professional Baseball (NPB)). The catcher then sends a signal to the
pitcher with their mitt and finger. The pitcher receives information about the
next pitch type/course and accepts or rejects it. Sometimes, their coach in
their dugout sends the signal to the catcher in advance.

It is unknown whether the ability for selecting an appropriate pitching
strategy exists. Woolner (2002) concluded the ability might exist, but cannot
detect from batting results due to noise. In other words, the ability does not
include noises or lucks and an unmeasured variable. There are some
quantitative studies on the effects of pitching sequences. Gray (2002)
revealed that ``fastball after a series of late balls is effective'' by
virtual experiments. However, there are no statistics to evaluate the overall
effect to select a pitching strategy. We consider that two elements make it
difficult to measure the effect of pitching strategy: 1) most public datasets
do not include location data where the catcher demands a ball, 2) there are a
lot of confounders associated with pitching/batting results.

First, we explain why we need the location data where the catcher demands. It
is known that a pitcher cannot always pitch to the location of the catcher's
demand perfectly. For example, Shinya etal. (2017) revealed that pitched
location follows a two-dimensional normal distribution. Figure
LABEL:fig:_pitch_distribution shows the pitch distribution where the catcher
demands a lower 1st-base/3rd-base side (unless otherwise noted, figures are
from the pitcher's perspective). There is a gap between demanded location and
the actually pitched one. Thus, a catcher needs to suggest the next pitch
location considering the gap between the required course and the pitched one.
For the above reason, in the fielding team's position, we should measure the
effects of pitching ``attempts'' to a specific course, not pitched.

Fig. 1: All-pitched location of four-seam where the catcher demands lower 1st-
base/3rd-base side (regardless of batter's hand) from pitcher's perspective
pitched by Taisuke Yamaoka, right-handed pitcher belongs to Orix Buffaloes in
2019 ($N=935$ in total). Units are centimeters.fig: pitch_distribution

In addition to the difficulty of observing the catcher's demand, many
confounders associated with pitching/batting results make it difficult to
evaluate pitching strategy. For example, it is known that a good batter is
more frequently pitched inside (especially in NPB). The correlation between
weighted On Base Average (wOBA; high wOBA means good batter), and the ratio of
inside pitching attempts was $r=0.31$ in our computation111We computed the
correlation (Pearson’s $r$-value) for the batters which have more than $300$
plate appearances in a season from 2014 to 2019 in NPB games. Please refer to
Section LABEL:ssec:dataset for the definition of inside pitching attempts, and
the detail.. If simply comparing the results where demanding inside with the
outside, the effect of outside pitching attempts can be overestimated since
poor hitters are attempted outside more frequently than hard hitters. Thus, we
cannot compare them directly.

For the above reason, we propose a method of estimating the effect of
demanding inside/outside with causal inference called stratified analysis
using propensity score (e.g., Rubin (1997), Damluji etal. (2019)), which can
statistically reduce confounders and estimate the effects. We define the
effect of each demand as a variation of run expectancy, computed by
pitching/batting results. Thus, we develop the model to compute the propensity
score (i.e., the probability of demanding inside) with confounding variables
(e.g., the ratio of pitched inside for the batter, the pitcher in the season),
and make up for unobserved data with it. We then estimate the expected effect
of demanding inside and outside. Furthermore, we investigate specific
questions about unknown problems in baseball. We group the data into some
groups based on the estimated batter's batting ability, and the ratio of
pitched inside for pitcher/batter, and estimated the effect for each group
(i.e., stratified analysis). For the estimated batter's batting ability, we
examine the effectiveness of the inside pitch attempt against a good batter,
because it is considered that a good batter is more frequently pitched inside
(especially in NPB). In the same way, we investigate the effect of the ratio
of pitched inside for pitcher/batter because the pitcher would think their
inside pitch attempts are effective and pitch more frequently.

In summary, the main contributions of the study are as follows: (1) We
analyzed the effects of pitching attempts to a specific location
(inside/outside) for the first time; (2) We used the stratified analysis using
the propensity score to estimate the effect by removing confounders in
complicated scenes of baseball; (3) We revealed the effect of pitching
attempts on a specific course exists. The remainder of the paper is organized
as follows. We review the related work in Section LABEL:sec:related. In
Section LABEL:sec:method, we describe the proposed method. Section
LABEL:sec:results presents the experimental results. Finally, we discuss the
results and present the conclusions in Sections LABEL:sec:discussion and
LABEL:sec:conclusion, respectively.

##  2 Related work

sec:related A lot of formulas and stats were developed, and employed to
evaluate past or predict future player's performance from a long time ago
(Silver, 2003, James, 2010, Lewis, 2004, Click and Keri, 2006, Tango etal.,
2007, Beneventano etal., 2012, Costa etal., 2012). These stats are computed by
batting results. One of the most essential perspectives when evaluating the
player's performance is to consider uncertainty. Even if two players have the
same ability, the two results can be different due to uncertainty. To deal
with the problem, some stats focus on the ``true ability'': strikeout, home
run, walk (Davies and Basco, 2010). These stats are considered to be stable
for each player for evaluating the player's performance.

Recently, measurement technology and data platform have been developed, and
all sensing data measured in MLB games are provided and accessible. It enables
us to develop stats to evaluate player's performance using speed and angle of
batted balls, such as expected weighted On Base Average (xwOBA) and expected
Earned Run Average (xERA) (MLB.com, 2015b, a). The speed and angle of batted
balls are relatively stable stats for an individual player (i.e., the value
does not fluctuate significantly across seasons.). These new stats can extract
a player's potential more validly. Comparing with evaluating batting/pitching
performance, evaluating pitching strategy is a challenging field. Some
researches predicted the next pitch type using various machine learning
methods (Hoang etal., 2015, Bock, 2015). Besides, the methods for predicting
the batting result are suggested (Martin, 2019, Harrison and Salmon, 2019,
Healey, 2015). When estimating the ability to select a pitching strategy for
the catcher, the simplest idea is computing ERA where the catcher is in the
games, called catcher's ERA (Herrlin, 2015). Woolner (2002) showed that the
catcher's ERA computed using batting results cannot detect the difference in
abilities. It seems difficult to estimate the catcher's ability to select a
pitching strategy with batting results. Besides, there is no study with the
data of catcher's demanding as far as we know.

In team sports, propensity score matching was used to investigate the causal
effect of some plays or timeouts in many sports such as going for the
touchdown in American football (Yam and Lopez, 2019), clearing the puck in ice
hockey (Toumi and Lopez, 2019), the effectiveness of timeouts in basketball
(Gibbs etal., 2020), and crossing the ball in soccer (Wu etal., 2021). In the
dynamic setting, Vock and Vock (2018) applied a g-computation method to
examine the effect of taking a pitch during a 3-0 count in MLB and Fujii etal.
(2022) proposed a framework for estimating individual treatment effect (ITE)
in basketball motions. Among approaches without causal inference methods, in
baseball, counterfactual predictions could be performed such as using
simulation and supervised learning in team hitting strategies (Nakahara etal.,
2022a) and using game theory and semi-supervised learning in a third base
coach's decision making (Nakahara etal., 2022b). Compared with these
approaches, since the outcome prediction in our task was very difficult (in a
more general pitching case than the study of (Vock and Vock, 2018)), we used
stratified analysis (not to predict outcomes) rather than regression analysis
to predict outcomes.

##  3 Methods

sec:method

In this section, we describe the proposed method to clarify the effect of
pitching attempts to specific locations. First, we describe the stratified
analysis using the propensity score to estimate the effect of demanding
inside/outside pitch. Second, we introduce the new dataset, including the
location where the catcher demands. Finally, we describe the processing method
to compute the propensity score.

###  3.1 Stratified analysis using propensity score

Our motivation is to reveal the effect of demanding inside/outside, and we
need to reduce confounders for valid estimations. To deal with confounding, we
employed methods introduced by Rosenbaum and Rubin (1985), defined as follows.
$X$ is a confounding variable. $Z\in(0,1)$ is a treatment allocation. $Z=0$
corresponds to nontreatment, and $Z=1$ corresponds to given treatment. In this
study, $Z=0$ ($Z=1$) means catcher demands outside (inside) pitch. $Y$ is a
potential outcome. $Y^{(1)}$ ($Y^{(0)}$) denotes the outcome if (not)
treatments are applied, in this case, the outcome if a catcher demands inside
(outside). The outcome is defined as the variation of run expectancy before
and after the event (i.e., pitch result or batting result) ($\Delta RE$). For
details, see Section LABEL:ssec:preprocess. For simplicity, denote $X$ in the
$i$-th unit (sample or pitch) as $X_{i}$. The same is true for the other
variables. $\tau$ denotes the average treatment effect (ATE), and is described
as follows:

$\displaystyle\tau=E[Y^{(1)}]-{E[Y^{(0)}]}.$ (1)

The main problem in causal inference is that we cannot observe $Y_{i}^{1}$ and
$Y_{i}^{0}$ simultaneously, known as ``the fundamental problem of causal
inference'' (Holland, 1986). To overcome the problem, various methods are
proposed, and one of the popular fundamental ideas is comparing groups (i.e.,
sets of units), which have similar confounding variables. It should be noted
that we used the stratified analysis (not to predict outcomes) rather than
regression analysis to predict outcomes. This is because the outcome
prediction in our task was very difficult in our preliminary experiment. The
propensity score is considered as the criteria for measuring the similarity
between each unit. The propensity score (here denoted by $P(X)$) is the
probability of being assigned treatment computed using confounding variables.
We can consider the distributions of confounding variables are the same
between treatment and nontreatment groups when comparing data with a similar
propensity score, according to Rosenbaum and Rubin (1985). That is, comparing
data with similar propensity scores reduces confounding. Note that here we
assume three popular assumptions in causal inference: consistency,
exchangeability, and positivity (Cole and Hernán, 2008, Austin and Stuart,
2015). Exchangeability (or ignorable treatment assignment) is the assumption
of no unmeasured confounders that affect treatment selection and outcomes.
However in the actual baseball data, we cannot say that there is no
possibility of that. We also discuss the limitation of this study from this
viewpoint in Section LABEL:sec:discussion. Consistency means that a pitch's
potential outcome under the treatment actually received is equal to the
pitch's observed outcome. Positivity is the assumption that all pitches have a
non-zero probability of receiving each treatment.

We use the Inverse Probability Weighting (IPW), one of the practical causal
inference methods using propensity scores (Robins etal., 2000). In IPW, the
propensity score is used to make up for unobserved outcomes. It is the same
with comparing units which have similar propensity scores, mathematically. IPW
estimates the treatment effect ($\tau$) as follows.

$\displaystyle\hat{\tau}$
$\displaystyle={E[\hat{Y}^{(1)}]}-{E[\hat{Y}^{(0)}]},{eq:ATE}$ (2)
$\displaystyle{E[\hat{Y}^{(1)}]}$
$\displaystyle=\left.\sum_{i=1}^{N}\frac{Y_{i}^{(1)}Z_{i}}{P(X_{i})}\right/\sum_{i=1}^{N}\frac{Z_{i}}{P(X_{i})},{eq:ATE2}$
(3) $\displaystyle{E[\hat{Y}^{(0)}]}$
$\displaystyle=\left.\sum_{i=1}^{N}\frac{Y_{i}^{(0)}(1-Z_{i})}{1-P(X_{i})}\right/\sum_{i=1}^{N}\frac{1-Z_{i}}{1-P(X_{i})},{eq:ATE3}$
(4)

where $\hat{\tau},\hat{Y}^{(0)}$, and $\hat{Y}^{(1)}$ are expected values of
$\tau,Y^{(0)}$, and $Y^{(1)}$, respectively. As an example, we consider the
units that satisfy $X=x,Z=1,P(Z=1|X=x)=0.3$. Then, $P(Z=0|X=x)=0.7$, and
$70\%$ of units with $X=x$ do not be assigned treatment. We cannot observe
these outcomes with treatment. To obtain the unobserved outcomes
counterfactually, each observed outcome is multiplied by $1/P(Z=1|X=x)$ (i.e.,
$1/0.3$). Since we cannot observe the true propensity score, we need to
develop a prediction model for its estimation. Using random forest, propensity
scores were estimated by modeling the associations of covariates with
treatment (i.e., the random forest classifies whether the treatment exists or
not). The number of trees was $130$, and the maximum depth of the tree was
$9$.

We compute the expected effect and confidence interval to examine the
treatment effect to be statistically significant. We compute this on average
for ATE, and this on each interested group (e.g., the estimated batter's
batting ability and the ratio of pitched inside for pitcher/batter) as a
stratified analysis. We employed the bootstrap method, which is one of the
Monte Carlo methods, resampling data randomly to estimate the confidence
interval (such as used in Bock etal. (2012)). The bootstrap method repeats the
following operation multiple times: extracting the same number of data as the
original from the original dataset randomly with replacement, and estimating
the effect using IPW. Among various methods to estimate confidence interval of
the treatment effect (e.g., Haukoos and Lewis (2005)), we used the normal
approximation method in this paper. It approximates the resampling
distribution as the normal distribution. Then, the confidence interval is
calculated using the sample average and its standard deviation.

###  3.2 Dataset

ssec:dataset We used the dataset of NPB games held in 2014-2019 provided by
Delta Inc. since it contains location data where the catcher demands. Although
most baseball studies used MLB data (Koseler and Stephan, 2017), no MLB
dataset includes the demanding location in public. We used the pitch-by-pitch
data (e.g., game situation, the location of a pitch, the location of catcher's
demanding by catcher just before pitching, pitch type, pitch speed, and
pitching/batting result) of $1{,}591{,}586$ pitches. We assume that demanding
location indicates the catcher's intent since the pitch distribution of inside
and outside are different and scattered around the edge of the inside and
outside, respectively (Figure LABEL:fig:_pitch_distribution). Note that the
location data is inputted manually (most teams collect sensing data
themselves, but not in public in Japanese professional baseball). We show the
detail of features in Section LABEL:ssec:preprocess.

Fig. 2: The coordinate system of pitching location and the definition of
inside/outside. Units are centimeters.fig:in_out

Figure LABEL:fig:in_out shows the coordinate system of pitching location and
the definition of inside/outside pitch. Demanding and pitched locations are
plotted on $160$ $\times$ $200$ px ($100\times 125$ cm) campus. The coordinate
system has the $X$-axis horizontal and the $Y$-axis vertical with the origin
$(0,0)$ at the lower 1st-base side corner. Note that we use a case when a
pitcher faces a right-handed batter for a simple explanation (in the case of
left-handed batter, reverse right-handed case). We define ``the catcher
demands inside (outside)'' where the plot is located in 1) more right (left)
than one-third of a home base's right side (left side), and 2) lower than the
top-third of the strike zone. The data with the high demanded ball was
excluded due to the small sample size, and the data where the balls touched
the ground was included. We group into ``the catcher demands inside'' where
$D_{x}\geq 96$ and $D_{y}\leq 80$, ``the catcher demands outside'' where
$D_{x}\leq 64$ and $D_{y}\leq 80$. Here, $D_{x},D_{y}$ denotes the coordinates
of the location of demand.

###  3.3 Data processing

ssec:preprocess

In this study, we used the data that games were held from March to June each
season to compute feature values (e.g., batter's batting stats, the ratio of
pitching inside for the pitcher, and so on). Thus, we predicted them for the
games held from July to September in each season. We used only data pitched
four-seam fastball since the location of each breaking ball's demanding is
fixed (e.g., most right-handed pitchers try pitching their slider into the
left side of the home base from the pitcher's perspective). The data on the
batter's plate appearance or the pitcher's total faced batters less than $100$
through a season were excluded due to the small samples. In addition, the data
grouped into neither inside nor outside were also excluded. We used
$286{,}430$ pitches to execute causal inference. The number of inside pitches
was $91{,}186$, while that of outside pitches was $195{,}224$.

Here we describe the computation of the input feature for propensity score and
the difference in run expectancy as the outcome. The original dataset in
baseball includes pitch results (called strike, swinging strike, foul, ball)
and batting results (e.g., hit, strikeout, and walk). To evaluate pitch
results and batting results with unified criteria, we converted
pitching/batting results into the run scale. It is because there is a strong
correlation between a win and earned runs, and the pitching strategy should be
evaluated using runs. The linear weights (LWTS) is one of the most popular
ideas for converting each event (pitching/batting results) into run scale and
ideas for evaluating each result in sabermetrics (Thorn etal., 2015). In LWTS,
the run expectancy is computed for each game situation, considering out count,
and runners on base regularly (e.g., the run expectancy of $0$ out without
runner is $0.44$). With the run expectancy for each game situation, we can
compute the variation of run expectancy as follows.

$\Delta RE=RE_{after}-RE_{before},$

where $RE_{after}$ and $RE_{before}$ are the run expectancy of after/before
event. In other words, the value of each event is regarded as the variation of
run expectancy. To compute the variation of run expectancy, we compute the
expected variation of run expectancy for each event as follows:

$\Delta RE(event)=\frac{1}{N}\sum_{i\in event}^{N}{\Delta RE_{i}},$

where $\Delta RE(event)$ is the expected variation of run expectancy for the
specific event, observed $N$ times. Table LABEL:tab:re shows the specific
$\Delta RE$ of each event.

Tab. 1: Variation of run expectancy ($\Delta RE$) for each event.

Result | $\Delta RE$ | Result | $\Delta RE$
---|---|---|---
strike | -0.038 | hit by pitch | 0.311
ball | 0.032 | intended walk | 0.155
single | 0.437 | bunt | -0.133
double | 0.786 | bunt and error | 0.720
triple | 1.117 | bunt strike out | -0.249
home run | 1.408 | bunt and fielder's choice | 0.700
field out | -0.235 | sacrifice fly | 0.007
double play | -0.746 | sacrifice fly and error | 0.713
foul fly | -0.266 | error | 0.664
swinging strike out | -0.255 | fielding interference | -0.337
called strike out | -0.238 | batting interference | 0.516
uncaught third strike | 0.294 | foul liner | -0.426
walk | 0.292 |  |

tab:re

For valid computation of the propensity score, we propose another feature
value called pitch confidence $C(n)$. It indicates how pitcher and catcher are
confident about inside/four-seam pitch in $n$-th pitch. We only illustrate the
inside confidence model. The four-seam confidence function is the same as the
inside one. The formula is computed as follows:

$\displaystyle{eq:Cin}C_{in}(n)=\begin{cases}\alpha\Delta
RE(n-1)+(1-\alpha)C_{in}(n-2)\\\ \text{($(n-1)$th pitch was inside)}\\\
C_{in}(n-1)\text{(otherwise)},\end{cases}$ (5)
$\displaystyle{eq:Cout}C_{out}(n)=\begin{cases}\alpha\Delta
RE(n-1)+(1-\alpha)C_{out}(n-2)\\\ \text{($(n-1)$th pitch was outside)}\\\
C_{out}(n-1)\text{(otherwise)},\end{cases}$ (6)
$\displaystyle{eq:Cn}C(n)=C_{in}(n)-C_{out}(n),$ (7)

where $\alpha$ denotes a constant value ($0<\alpha<1$). It expresses how much
``confidence'' remembers recent results. The larger, the more ``confidence''
forgets old pitch results and weights recent pitch results. The smaller, the
more ``confidence'' remembers old results well and does not weight recent
results relatively. In this study, we added two confidence models. One is
$\alpha$ = 0.6 and the other is $\alpha=0.001$, determined empirically.

Finally, we used $18$ feature values for prediction, as follows: game
condition (ball count, out count, runner on base, run difference, whether the
batter's hand is the same as the pitcher's hand or not), pitcher (total pitch
in the game, pitch result one pitch ago, pitch result two-pitch ago, pitch
speed one pitch ago, pitch speed two-pitch ago, the confidence of pitching
into the inside ($\alpha=0.6,0.01$), the confidence of pitching four-seam
($\alpha=0.6,0.01$), previous batting result, the ratio of pitched into inside
in the season), batter (the ratio of pitched into inside in the season, wOBA).

##  4 Results

sec:results In this section, we first verified the propensity scores, which is
the basis of the causal inference method. Second, we show the contribution of
each variable to propensity score computation. Third, we show the estimated
total treatment effect (i.e., the effect of demanding inside/outside).
Finally, we show the treatment effect within the groups based on a) the
magnitude of the estimated batter's batting ability (wOBA in this case), b)
that of the ratio of the inside pitch for the batter, c) that of the ratio of
the inside pitch for the pitcher (i.e., stratified analysis).

###  4.1 Verification of propensity scores

ssec:verifyPS

Fig. 3: Overlap plot of the density of the propensity scores in (left) raw
data and (right) weighted data via Inverse Probability Weighting (IPW).
Although there was a large difference in the distribution of demanding outside
and inside (left), the weighted results in outside and inside were similar
(right).fig:psdis Fig. 4: Absolute Standardized Absolute Mean distance (ASAM).
Red and blue markers indicate before/after adjusted ASAM, respectively. Each
ASAM should be lower than $0.1$.fig:asam

As a verification of the causal inference, propensity score distributions must
be similar between treatment and nontreatment groups. Figure LABEL:fig:psdis
shows the plots of the density of the propensity scores in (left) raw data and
(right) weighted data via IPW. We confirmed that the distribution was adjusted
by the IPW method using the propensity score adequately (for details, see
Figure LABEL:fig:psdis and the caption).

Next, each covariate must be balanced between the treatment and nontreatment
groups. We employed Average Standardized Absolute Mean distance (ASAM) to
balance them, and it is computed as follows.

$\displaystyle ASAM$ $\displaystyle=$
$\displaystyle\frac{|\overline{x_{in}}-\overline{x_{out}}|}{s},$ (8)
$\displaystyle s$ $\displaystyle=$
$\displaystyle\sqrt{\frac{n_{in}s_{in}^{2}+n_{out}s_{out}^{2}}{n_{in}+n_{out}}},$
(9)

where $x_{in}$ and $x_{out}$ are the confounders with pitched inside and
outside, respectively. $\overline{x_{in}}$ and $\overline{x_{out}}$ are the
arithmetical means of $x_{in}$ and $x_{out}$. $n_{in}$ and $n_{out}$ are the
number of data with pitched inside and outside, respectively. $s_{in}$ and
$s_{out}$ are the standard deviation of $x_{in}$ and $x_{out}$, respectively.
According to the previous work (Cannas and Arpino, 2019), researchers should
aim at obtaining an ASAM lower than 0.1 for as many variables as possible.
Figure LABEL:fig:asam shows the adjusted and nonadjusted ASAM. Although there
was only one covariate (i.e., the ratio of pitched inside for pitcher) over
$0.1$, the value $0.116$ was close to $0.1$. We consider each covariate was
adjusted adequately using a propensity score since almost all of ASAM of
confounding variables were lower than $0.1$. These results show that control
for confounding would be possible across the entire range of the propensity
score.

###  4.2 Contribution of each variable

ssec:shap

The contribution of the input variables to the propensity score model
prediction was computed by SHAP (SHapley Additive exPlanations) (Lundberg and
Lee, 2017), which utilizes an interpretable approximate model of the original
nonlinear prediction model and also used for sports analysis (Toda etal.,
2021). Figure LABEL:fig:shap shows the contribution of the input variables
(SHAP). Note that variables including ``result'' are converted into run scale,
and a negative value means a pitcher is in advantage. The ratios of pitched
inside in the season for pitcher/batter were important for prediction.
Following them, previous pitch result, runner, whether pitcher's hand is the
same as the batter's one were also important.

Fig. 5: Contribution of the input variables to the propensity score model
prediction. The variables related to the prediction are shown in the order of
their contribution. Each dot (but merged) represents each event. The color
represents the value of the feature (blue and red indicate low and high,
respectively). The horizontal axis shows the impact on the prediction
(strongly positive and negative impacts are plotted to the right and left,
respectively). fig:shap

###  4.3 Total ATE

ssec:total

We then analyzed the total ATE computed in Equation (LABEL:eq:ATE). The
positive effect means the outside pitching attempt is effective, in contrast,
the negative effect means the inside pitching attempt is effective. The ATE
was $6.29\times 10^{-3}$, and the confidence interval ($99\%$) was $6.21\times
10^{-3}\leq ATE\leq 6.36\times 10^{-3}$. It means that demanding an outside
pitch is more effective than demanding an inside pitch. Since the effect of
getting a strike is about $-3.8\times 10^{-2}$, it did not seem a large
effect.

Next, for comparison, we just compared the variation of run expectancy between
inside attempts and outside ones (without IPW). Results show that that of an
outside pitching attempt was $7.71\times 10^{-3}$ runs/pitch smaller than that
of an inside one. The results suggest that the outside pitching attempt was
more effective than the inside pitching attempt compared with the analysis
with IPW, which overestimated the value of outside pitching attempts.

###  4.4 Stratified analysis

ssec:stratified Next, we show the treatment effect within the groups based on
a) the magnitude of the estimated batter's batting ability (wOBA in this
case), b) that of the ratio of the inside pitch for the batter, c) that of the
ratio of the inside pitch for the pitcher (Figure LABEL:fig:effect). Each data
was divided into some groups based on the magnitude of each confounding
variable. We eliminated the data with the confounding variable greater than
the $10,000$ samples. Each point represents a mean value, and the extended
line represents a ($99\%$) confidence interval computed using the bootstrap
method. The result indicates that the pitching attempt to the outside is
effective, regardless of the magnitude of each confounding variable.

For the wOBA levels in Fig. 6(a), comparing groups with wOBA less than 0.3 and
0.3 or more, the effectiveness of pitching to the outside decreased when wOBA
was large. For the ratio of the inside pitch for the batter in Fig. 6(b), an
outside pitch attempt was effective even against batters frequently pitched
inside ($0.4$ or more), but these effects were smaller than the rest of the
groups. For the ratio of the inside pitch for the pitcher in Fig. 6(c), an
outside pitch attempt was also effective even for pitchers frequently pitched
inside ($0.6$ or more), but these effects were smaller than the rest of the
groups.

Fig. 6: The estimated treatment effects for each group based on (a) estimated
batting ability (wOBA), (b) ratio of inside pitch for batter, (c) ratio of
inside pitch for pitcher. The vertical axis is the estimated treatment effects
via IPW, in which the scale is run/pitch. We eliminated the data where the
data with the confounding variable was greater than the criterion, due to the
small sample size.fig:effect

##  5 Discussion

sec:discussion In this study, we clarified the effect of pitching attempts to
a specific location (e.g., inside or outside) in baseball. In this section, we
discuss the results of ATE, stratified analysis, propensity scores, and
limitations and future perspectives of this study. As shown in Section
LABEL:ssec:total, the pitching attempt to the outside was found to be more
effective than to the inside when pitching the four-seam fastball. In NPB, the
pitching strategy is based on pitching to the outside with an occasional mix
of pitching to the inside, generally. An inside pitching attempt is considered
risky and requires more courage to pitch compared to an outside one. This
common belief is consistent with the results.

The results revealed that the total ATE of selecting outside pitch was about
$6.29\times 10^{-3}$ runs. We assume that every straight pitch has a right
pitching strategy and the effect is about $6.29\times 10^{-3}$ (i.e., a right
pitching strategy reduces $6.29\times 10^{-3}$ runs/pitch than the wrong one).
Then, we also assume that $150$ pitches are thrown in a game and $50\%$ of
that are four-seam fastball ($75$ four-seam fastballs are thrown in a game).
Under these assumptions, the maximum contribution of saving runs by selecting
an appropriate pitching course was estimated at $0.47$ runs/game. Although the
difference of pitching strategy among catchers needs to be considered, this
implies that we can measure the effect of selecting pitch course to save runs
(and type) may exist.

As shown in Section LABEL:ssec:stratified, the outside pitching attempt was
found to be always more effective in each situation. For the estmiated batting
ability, it is known that a hard hitter is more frequently pitched inside
(especially in NPB), which means they think an inside pitch attempt is
effective against a good batter.  For the wOBA levels in Fig.
LABEL:fig:effect(a), the effectiveness of pitching to the outside decreased
when wOBA was large. For the ratio of the inside pitch for the batter/pitcher
in Figs. LABEL:fig:effect(b) and (c), an outside pitch attempt was effective
even against batters and for pitchers frequently pitched inside, but these
effects were smaller than the rest of the groups. Although the pitch attempt
to the inside was less effective than the pitch attempt to the outside from
the perspective of the outside pitch attempt, overall, the pitch to the
outside was more effective than those to the inside.  Considering the
difficulty of controlling the location of the pitch, an inside pitching
attempt might be overrated. The scene where an inside pitching attempt was
effective might be limited than expected.

We also computed the contribution of each variable to the propensity score
model prediction (whether the catcher demands an inside pitch or not). We
clarified that the ratios of pitched inside for pitcher/batter were important,
which seems to be obvious. On the other hand, we also clarified that some
variables contributed to predicting the propensity score model unexpectedly.
Previous pitch results one/two pitch ago were the third and fifth most
important variables, respectively. For both of them, the high SHAP value was
associated with a low value, which means they tend to try to pitch inside more
frequently where they are in advantage. It might be also an important insight
since there is no study about predicting the course of catcher's demand ever.

If you are a baseball fan, you can easily understand that it is inappropriate
to conclude that the pitcher should try to always pitch to the outside from
these results. It is because the ratio of pitching inside and outside needs to
be balanced to keep their effects. If all pitches were thrown to the outside,
the batter could prepare for hitting the outside pitch (e.g., standing closer
to home base and stepping outside). The batter only needs to focus on hitting
the same course, improving the reproducibility of the accurate swing to hit a
ball. Thus, the effect of the scattering pitching course should be considered
to obtain a more practical result.

From the perspective of causal inference, this study has mainly two
limitations. First, there might be unmeasured confounders because we can
measure and use a limited number of variables in baseball games. For example,
batter's intentions and pitcher/batter's condition are possible unobserved
confounders. Second, datasets of baseball games are longitudinal in each game
and in all games for each team. Although we added the history of performances
for batters and pitchers to the input features, we need to consider the models
including time-varying confounders and treatment for future work.

In this study, we employed the IPW method that estimates ATE, adjusting the
distribution, and comparing by groups. Estimating the effect by comparing
groups, such as IPW, is a common and traditional idea in causal inference. On
the other hand, various causal inference methods have been proposed, which can
estimate Individual Treatment Effect (ITE) (e.g., Künzel etal. (2019)). These
methods estimate the potential outcome, which is not observed in real,
counterfactually. While the IPW method reduces a causal problem to predict the
probability of allocated treatment for each data, the method estimating ITE
reduces it to predict counterfactual pitching/batting results. It is
equivalent to predicting results with game information. Predicting
pitching/batting result is more difficult compared to predicting treatment
probability. Although we did not employ a method for estimating ITE due to the
difficulty in the prediction, we are sure that estimating ITE will extend or
improve this study.

##  6 Conclusions

sec:conclusion

In this study, a causal inference method using the location of the catcher
mitt right before pitching revealed that an outside pitching attempt is more
effective than an inside pitching attempt. We also found that the outside
pitching attempt was always effective despite the magnitude of the ratio of
pitched inside for pitcher/batter. Besides, we revealed that the previous
pitch result is also important to predict if the catcher demands an inside
pitch.

For future perspectives, the application of this study is 1) evaluating the
catcher's selection of pitching strategy and 2) seeking a better pitching
strategy. First, our approach can reveal where courses and which types to
pitch is effective in each game situation. This enables us to evaluate each
pitching strategy. If we can compute ITE in our problem, we can evaluate each
catcher's selections of pitching strategy.

Second, estimating the effect of each pitch makes it possible to seek a better
pitching strategy, if we can develop a pitching strategy model. Some Japanese
companies are developing an automatic selecting pitching strategy system
called ``AI catcher''. Systems for predicting better strategy can entertain
enthusiastic baseball fans as well as support decision-making.

## Acknowledgments

The data was provided by Delta Inc.  This work was supported by JSPS KAKENHI
(Grant Numbers 19H04941 and 20H04075) and JST PRESTO (JPMJPR20CA).

## Bibliography

  * Austin and Stuart (2015) P.C. Austin and E.A. Stuart.  Moving towards best practice when using inverse probability of treatment weighting (iptw) using the propensity score to estimate causal treatment effects in observational studies.  _Statistics in Medicine_ , 34(28):3661–3679, 2015\.
  * Beneventano etal. (2012) P.Beneventano, P.D. Berger, and B.D. Weinberg.  Predicting run production and run prevention in baseball: the impact of sabermetrics.  _Int J Bus Humanit Technol_ , 2(4):67–75, 2012\.
  * Bock (2015) J.R. Bock.  Pitch sequence complexity and long-term pitcher performance.  _Sports_ , 3(1):40–55, 2015.
  * Bock etal. (2012) J.R. Bock, A.Maewal, and D.A. Gough.  Hitting is contagious in baseball: evidence from long hitting streaks.  _PloS One_ , 7(12):e51367, 2012.
  * Cannas and Arpino (2019) M.Cannas and B.Arpino.  A comparison of machine learning algorithms and covariate balance measures for propensity score matching and weighting.  _Biometrical Journal_ , 61(4):1049–1072, 2019\.
  * Click and Keri (2006) J.Click and J.Keri.  _Baseball between the numbers: Why everything you know about the game is wrong_.  Perseus Books Group, 2006.
  * Cole and Hernán (2008) S.R. Cole and M.A. Hernán.  Constructing inverse probability weights for marginal structural models.  _American Journal of Epidemiology_ , 168(6):656–664, 2008.
  * Costa etal. (2012) G.B. Costa, M.R. Huber, and J.T. Saccoman.  _Reasoning with Sabermetrics: Applying Statistical Science to Baseball's Tough Questions_.  McFarland, 2012.
  * Damluji etal. (2019) A.A. Damluji, K.Bandeen-Roche, C.Berkower, C.M. Boyd, M.S. Al-Damluji, M.G. Cohen, D.E. Forman, R.Chaudhary, G.Gerstenblith, J.D. Walston, etal.  Percutaneous coronary intervention in older patients with st-segment elevation myocardial infarction and cardiogenic shock.  _Journal of the American College of Cardiology_ , 73(15):1890–1900, 2019.
  * Davies and Basco (2010) M.A. Davies and D.Basco.  The many flavors of dips: A history and an overview.  _Baseball Research Journal_ , 39(2):41–50, 2010\.
  * Fujii etal. (2022) K.Fujii, K.Takeuchi, A.Kuribayashi, N.Takeishi, Y.Kawahara, and K.Takeda.  Estimating counterfactual treatment outcomes over time in complex multi-agent scenarios.  _arXiv preprint arXiv:2206.01900_ , 2022.
  * Gibbs etal. (2020) C.Gibbs, R.Elmore, and B.Fosdick.  The causal effect of a timeout at stopping an opposing run in the nba.  _arXiv preprint arXiv:2011.11691_ , 2020.
  * Gray (2002) R.Gray.  Behavior of college baseball players in a virtual batting task.  _Journal of Experimental Psychology: Human Perception and Performance_ , 28(5):1131, 2002.
  * Harrison and Salmon (2019) W.K. Harrison and J.L. Salmon.  Leveraging pitcher-batter matchups for optimal game strategy.  _MIT Sloan Sports Analytics Conference_ , 2019.
  * Haukoos and Lewis (2005) J.S. Haukoos and R.J. Lewis.  Advanced statistics: bootstrapping confidence intervals for statistics with “difficult” distributions.  _Academic Emergency Medicine_ , 12(4):360–365, 2005.
  * Healey (2015) G.Healey.  Modeling the probability of a strikeout for a batter/pitcher matchup.  _IEEE Transactions on Knowledge and Data Engineering_ , 27(9):2415–2423, 2015.
  * Herrlin (2015) D.L. Herrlin.  Forecasting mlb performane utilizing a bayesian approach in order to optimize a fantasy baseball draft.  2015\.
  * Hoang etal. (2015) P.Hoang, M.Hamilton, J.Murray, C.Stafford, and H.Tran.  A dynamic feature selection based lda approach to baseball pitch prediction.  In _Trends and applications in knowledge discovery and data mining_ , pages 125–137. Springer, 2015.
  * Holland (1986) P.W. Holland.  Statistics and causal inference.  _Journal of the American statistical Association_ , 81(396):945–960, 1986.
  * James (2010) B.James.  _The new Bill James historical baseball abstract_.  Simon and Schuster, 2010.
  * Koseler and Stephan (2017) K.Koseler and M.Stephan.  Machine learning applications in baseball: A systematic literature review.  _Applied Artificial Intelligence_ , 31(9-10):745–763, 2017.
  * Künzel etal. (2019) S.R. Künzel, J.S. Sekhon, P.J. Bickel, and B.Yu.  Metalearners for estimating heterogeneous treatment effects using machine learning.  _Proceedings of the National Academy of Sciences_ , 116(10):4156–4165, 2019.  ISSN 0027-8424.  10.1073/pnas.1804597116.
  * Lewis (2004) M.Lewis.  _Moneyball: The art of winning an unfair game_.  WW Norton & Company, 2004.
  * Lundberg and Lee (2017) S.M. Lundberg and S.-I. Lee.  _A Unified Approach to Interpreting Model Predictions_.  2017\.
  * Martin (2019) E.P. Martin.  Predicting major league baseball strikeout rates from differences in velocity and movement among player pitch types.  _MIT Sloan Sports Analytics Conference_ , 2019.
  * MLB.com (2015a) MLB.com.  Expected era (xera).  https://www.mlb.com/glossary/statcast/expected-era, 2015a.  Accessed: 2021-07-05.
  * MLB.com (2015b) MLB.com.  Expected weighted on-base average (xwoba).  https://www.mlb.com/glossary/statcast/expected-woba, 2015b.  Accessed: 2021-07-05.
  * Nakahara etal. (2022a) H.Nakahara, K.Takeda, and K.Fujii.  Estimating the effect of team hitting strategies using counterfactual virtual simulation in baseball.  _arXiv preprint arXiv:2206.01871_ , 2022a.
  * Nakahara etal. (2022b) H.Nakahara, K.Takeda, and K.Fujii.  Evaluating a third base coach's decision making via game theory and machine learning.  _MathSport International_ , 2022b.
  * Robins etal. (2000) J.M. Robins, M.A. Hernan, and B.Brumback.  Marginal structural models and causal inference in epidemiology, 2000\.
  * Rosenbaum and Rubin (1985) P.R. Rosenbaum and D.B. Rubin.  Constructing a control group using multivariate matched sampling methods that incorporate the propensity score.  _The American Statistician_ , 39(1):33–38, 1985\.
  * Rubin (1997) D.B. Rubin.  Estimating causal effects from large data sets using propensity scores.  _Annals of internal medicine_ , 127(8_Part_2):757–763, 1997.
  * Shinya etal. (2017) M.Shinya, S.Tsuchiya, Y.Yamada, K.Nakazawa, K.Kudo, and S.Oda.  Pitching form determines probabilistic structure of errors in pitch location.  _Journal of sports sciences_ , 35(21):2142–2147, 2017.
  * Silver (2003) Silver.  Introducing pecota, 2003.
  * Tango etal. (2007) T.M. Tango, M.G. Lichtman, and A.E. Dolphin.  _The book: Playing the percentages in baseball_.  Potomac Books, Inc., 2007.
  * Thorn etal. (2015) J.Thorn, P.Palmer, and D.Reuther.  _The hidden game of baseball: A revolutionary approach to baseball and its statistics_.  University of Chicago Press, 2015.
  * Toda etal. (2021) K.Toda, M.Teranishi, K.Kushiro, and K.Fujii.  Evaluation of soccer team defense based on prediction models of ball recovery and being attacked.  _arXiv preprint arXiv:2103.09627_ , 2021.
  * Toumi and Lopez (2019) A.Toumi and M.Lopez.  From grapes and prunes to apples and apples: Using matched methods to estimate optimal zone entry decision-making in the national hockey league.  In _Carnegie Mellon Sports Analytics Conference 2019_ , 2019.
  * Vock and Vock (2018) D.M. Vock and L.F.B. Vock.  Estimating the effect of plate discipline using a causal inference framework: an application of the g-computation algorithm.  _Journal of Quantitative Analysis in Sports_ , 14(2):37–56, 2018.
  * Woolner (2002) K.Woolner.  Aim for the head: Simulating catcher’s era, 2002.  http://www.soumu.go.jp/menu_news/s-news/01tsushin02_02000072.html.
  * Wu etal. (2021) L.Y. Wu, A.J. Danielson, X.J. Hu, and T.B. Swartz.  A contextual analysis of crossing the ball in soccer.  _Journal of Quantitative Analysis in Sports_ , 17(1):57–66, 2021.
  * Yam and Lopez (2019) D.R. Yam and M.J. Lopez.  What was lost? a causal estimate of fourth down behavior in the national football league.  _Journal of Sports Analytics_ , 5(3):153–167, 2019.

lastpage


# Unveiling Dynamics and Patterns: A Comprehensive Analysis of Spreading
Patterns and Similarities in Low-Labelled Ransomware Families

Francesco Zola1, Mikel Gorricho1, Jon Ander Medina1, Lander Segurola1, Raul
Orduna Urrutia1  0000-0002-1733-5515 0009-0008-1107-0617 0000-0003-4278-9081
0000-0002-5932-0987 1 Vicomtech, Basque Research and Technology Alliance
(BRTA)
Paseo Mikeletegi 57, 20009 Donostia/San Sebastian, Spain
{fzola, mgorricho, jamedina, lsegurola<EMAIL_ADDRESS>
###### Abstract

Ransomware has become one of the most widespread threats, primarily due to its
easy deployment and the accessibility to services that enable attackers to
raise and obfuscate funds. This latter aspect has been significantly enhanced
with the advent of cryptocurrencies, which, by fostering decentralisation and
anonymity, have transformed this threat into a large-scale outbreak. However,
recent reports indicate that a small group of individuals dominate the
ransomware ecosystem and try to obfuscate their activity using multiple
strains characterised by a short time to live. This scenario suggests that
different strains could share mechanisms in ransom collection, fund movement,
and money laundering operations. For this reason, this study aims to analyse
the address-transaction graphs generated in the Bitcoin network by low-
labelled ransomware families. Our goals are to identify payment spreading
patterns for evaluating the evolution of ransomware families and to detect
similarities among different strains that potentially can be controlled by the
same attacker. Specifically, this latter task assigns an address behaviour to
each node in the address-transaction graphs according to its dynamics. The
distribution of the behaviours in each strain is finally used to evaluate the
closeness among different ransomware families. Our findings show that although
ransomware families can quickly establish connections with millions of
addresses, numerous families require multiple-step analysis. Furthermore, the
study demonstrates that the introduced behaviours can effectively be used to
highlight similarities among different ransomware strains. The outcome shows
that families are similar primarily due to behaviours usually associated with
ransom collection and money laundering operations. To the best of our
knowledge, this work contributes to dissecting the evolution of ransomware
strains and detecting distinctive markers they show within the Bitcoin
network.

###### Index Terms:

Ransomware, Spread Evolution, Behaviour Distribution, Similarities, Bitcoin
Analysis

##  I Introduction

In the last two decades, ransomware has emerged as one of the most widespread
threats, locking or encrypting computers and data using symmetric and
asymmetric encrypting algorithms while asking for the payment of a ransom [18,
22]. It operated primarily as a form of common vandalism among cybercriminals
[23] until it became a sophisticated criminal practice, for example, utilising
Ransomware as a Service (RaaS) models, as described by Europol in The 2023
Internet Organised Crime Threat Assessment (IOCTA) [8]. In fact, this report
shows that the presence of cybercrime services within the network brought
together and increased the number of criminal actors.

Over the years, ransomware attacks have also changed their primary targets,
starting from ordinary end-users; they pass to compromise governments, banks,
schools, hospitals, and business organisations in almost any sector [19]. In
the beginning, victims of ransom were directed to make payments through
methods such as SMS text messages and by sending pre-paid cards via mail,
calling a premium-rate telephone number, etc. [22]. However, with the advent
of cryptocurrencies, we witnessed the first large-scale outbreak of ransomware
[18]. In fact, by promoting decentralisation and anonymity (or pseudo-
anonymity in some cases), these digital assets have created an ideal
environment for the ransomware business. In particular, Bitcoin and Monero are
the top digital currencies used in ransomware operations [9]. On the one hand,
Bitcoin represents the most known cryptocurrency, with a very high market
value, and especially the easiest to acquire for victims without a tech
background. On the other hand, Monero is a privacy coin that increases user
and transaction anonymity, ideal for laundering activities [6]. Furthermore,
the relationship between the ransomware impact, their variants, and the
crypto-ecosystem has been studied and demonstrated in many researches [29, 5].

Despite the number of ransomware variants rising to over 10,000 strains (or
families) in the first half of 2022, the average lifespan of each strain
dropped to 70 days, indicating hidden tactics and the use of multiple strains
by attackers, as suggested in [3]. Furthermore, the document indicates that a
small group of individuals dominate the ransomware ecosystem. The 2024 Crypto
Crime Report [4] reports that ransomware actors linked to specific strains
continuously launch new variants to make victims more willing to pay. Indeed,
this ”rebranding” strategy enables ransomware attackers to distance themselves
from strains that are publicly associated with sanctions or have attracted
excessive scrutiny. This scenario would lead to the hypothesis that different
strains controlled by a concrete attacker could share similar mechanisms for
their activities, such as ransom collection, fund movement, and money
laundering. If verified, this could allow Law Enforcement Officers (LEOs) to
anticipate attackers’ strategies, helping them track different strains towards
known entities that implement Know Your Customer (KYC) policies useful for
extracting real-world information, and, at the same time, could allow them to
optimise their approaches by using investigation techniques followed in
previous successful and similar cases.

For this reason, in this paper, we propose to study the payment spreading
patterns that ransomware families generate within the Bitcoin blockchain. Our
idea is to analyse and characterise the different families by evaluating their
temporal evolution and, simultaneously, the dynamics generated by each address
involved in the transactions. In this way, we aim to detect similarities (and
differences) between ransomware variants and, so, verify if they effectively
share mechanisms for their activities. Specifically, we first propose
delineating distinct spreading patterns based on the population reached by
each family. To achieve this, we compute the address-transaction graph [10] to
represent the dynamics of the ransomware. Thus, we study its temporality,
evaluating the time required for the strain to spread throughout the
population. Our analysis is designed to leverage the identified spreading
patterns as a lens through which we can observe and understand the
evolutionary trajectories of these ransomware families. Moreover, address-
transaction graphs are also used to introduce address behaviours, i.e., labels
used to identify common dynamics of the addresses within the network. The
analysis demonstrates that these topological dynamics can effectively
characterise each strain. Therefore, these behaviours and their distributions
in each ransomware become instrumental in verifying the similarities and
distinctions among the diverse families. In this work, low-labelled ransomware
families, i.e., families characterised by a scant number of labelled/tagged
addresses (or seeds), are used during the analyses. The motivation for this
decision and the possible limitations are discussed in Section III-A.

##  II Background

Traditional ransomware analysis aims to study how they are
developed/programmed [17], [14]. However, these attacks are continuously
evolving, and hackers always try new mechanisms to deceive their victims and
spread the attacks, such as leveraging new attack vectors, incorporating
advanced encryption algorithms, expanding the number of file types it targets,
using anonymous networks for moving the ransom, etc. [15]. In this sense, the
success of cryptocurrencies, in part related to the pseudo-anonymity they
guarantee, has converted them into the principal way to demand ransomware
payments [27]. This scenario has led researchers to follow the ransom within
the crypto-ecosystem and study how the funds are moved, with the main goal of
implementing new ransomware detection systems. This is the case of [1], in
which an efficient data analytics framework is designed to detect new
malicious addresses automatically. A ransomware detection system is presented
in [11], employing the backpropagation artificial neural network method with
Weka software. In a similar way, in [2], a Bitcoin transaction predictive
system for classifying ransomware payments has been presented, while in [21],
a study on detecting suspicious users and transactions using machine learning
models is presented. Huang et al. [13] introduced a measurement framework for
ransomware payments, victims, and operators. In the research, they collected a
variety of data sources, including ransomware binaries, seed ransom payments,
victim telemetry from infections, and an extensive database of Bitcoin
addresses. In [26], graph algorithms are used to analyse Bitcoin ransomware
graphs for preventing attacks.

On the other hand, other studies are more directed at studying ransomware
evolution and how they are spread within the network. This is the case of
[28], where authors aim to track ransom payments and the trajectory of the
funds across different industries. In [25], authors investigate how crypto
exchanges and mixers are used (and combined) to launder ransomware illegal
funds, while transactions associated with 35 ransomware families are further
analysed in [20], using GraphSense tool for revealing the impact from an
economic point of view.

Inspired by these last works, in this paper, we aim to evaluate if different
strains follow similar evolution trajectories and apply similar strategies for
their activities. In particular, we analyse low-labelled ransomware families
with the aim to a) identify spreading patterns based on the family’s capacity
to reach a large population within a fixed timeframe, b) study the temporal
evolution of each family, c) define address behaviours based on the
topological structures families generated in the address-transaction graph, d)
demonstrate that these behaviours are enough for characterising each family,
and e) finally, exploit behaviour distributions to find similarity among the
different families.

##  III Experimental Framework

###  III-A Dataset

In this paper, the entire Bitcoin blockchain data until the block 810,000 are
downloaded, i.e., all the transactions until September 30th, 2023 (about 900M
transactions). On the other hand, Graphsense111https://graphsense.info/ [12],
Ransomware Payments dataset [20], and BitcoinHeist dataset [1] have been used
to collect addresses labelled as ransomware (seeds). These sources represent
valid solutions used in many researches [20, 24], and allowed us to gather
about 26,300 Bitcoin addresses tagged as ransomware of 89 different families,
as shown in Figure 1.

As mentioned, this work is focused only on low-labelled families, i.e., those
strains with less than 10 labelled addresses. This decision was made after
considering several factors:

  1. 1.

families with a large number of seeds have been extensively studied in
existing literature [20, 24];

  2. 2.

given that RaaS involves mass production, we can infer that families
containing numerous labelled addresses (seeds) can likely be interconnected
due to extensive RaaS operations behind the scenes. Therefore, the results of
this analysis could be skewed;

  3. 3.

as depicted in Figure 1, families with more than 10 seeds are not the most
common ones;

  4. 4.

low-labelled families may have a limited number of seeds for several reasons.
They could belong to smaller ransomware operations that are not prominent
within the industry, or they might not have been thoroughly discovered.
Additionally, they could employ sophisticated mechanisms that make them
difficult to detect, thereby increasing their value.

The constraint left us with a dataset of 137 addresses of 65 different
families (Table I). Although the small number of addresses could seem a
limitation in the validation of the results, it is to be noted that the
dataset still contains $\sim$72% of the original ransomware families. This
aspect is the most relevant one since the main goal of this study is to check
and evaluate similarities among the spreading patterns and the behaviour
distribution of different ransomware families. similarities among different
families.

Once the dataset is reduced, all the seeds are used as a starting point for
extracting each ransomware’s address-transaction graph (Section III-B). In
particular, one single graph is extracted for each family, so in the cases of
multiple seeds, the generated graphs are merged into a single one.

Figure 1: Seed distribution in the gathered dataset.

Year | # Seeds |  Ransomware family name
---|---|---
2015 | 1 |  chimera, teslacrypt
2016 | 1 |  7ev3n, bucbi, comradecircle, cryptohost, ctb-locker, ecovector, exotic, nullbyte, phoenix, popcorntime, sam
4 |  cryptohitman
5 |  apt, xtplocker
7 |  towerweb
8 |  cryptconsole,venuslocker
2017 | 1 |  black mamba, globeimposter, lamdalocker, lockon, ransomnix, spora, storagecrypter, vevolocker, wannasmile, xlocker
3 |  xorist
4 |  xlockerv5.0
6 |  hc6/hc7
2018 | 1 |  black ruby, blackrouter, gula, qweuirtksd
2019 | 1 |  encrpt3d, git, tejodes
2 |  decryptiomega
2020 | 1 |  ako, bitpaymer, kelly, mountlocker, wannaren
2 |  black kingdom
3 |  lockbit
4 |  ragnarlocker
7 |  rEvil
9 |  egregor
2021 | 1 |  albdecryptor, avaddon, avoslocker, bagli, blackmatter, chupacabra, hellokitty, ranzy locker, suncrypt, vega
3 |  darkside
2022 | 1 |  darkangels, hive, quantum
9 |  deadbolt
| 137 |  65 families

TABLE I: Ransomware families used in the analysis separated per year and
number of seeds.

###  III-B Address-Transaction graph

Cryptocurrencies have an inherent structure of information that allows it to
be represented in a directed graph. In fact, since transactions are used for
connecting two or more crypto addresses, an address-transaction graph can be
implied [10]. This graph approximates the cryptocurrency flow by connecting
public key addresses over time. The vertices within the graph represent
addresses and transactions. Thus, directed edges (arrows) between addresses
and transactions represent incoming relations, while directed edges between
transactions and addresses denote outgoing relations. Each edge may
incorporate additional features like amount, fee, timestamps, etc.

In this research, we create specific address-transaction graphs starting from
the seeds and exploring their neighbours in $n$ steps (or hops). Let $X1$ be a
seed to be used as a starting point; its $n$-step address-transaction graph is
a graph in which all the paths from $X1$ involve maximum $n$ transactions;
therefore, they have a maximum length of $2n$. These paths must include $X1$
as a starting/ending point, and consequently, they should end/start with an
address node.

##  IV Ransomware Payment Spreading Pattern

To investigate how ransomware evolves and is spread within the Bitcoin
network, 2-step address-transaction graphs have been analysed. Through this
task, we have identified and utilised four spreading patterns: slow, moderate,
fast and extremely fast (or exFast). The first category includes families that
reach less than 500 distinct addresses in the 2-step graphs; the second
includes families that link from 500 to 50,000 addresses; the third one
connects 50,000 to 500,000 addresses; and the last one reaches more than
500,000 addresses.

Figure 2(a) shows that the majority of the families - 26 families out of the
65 considered ($40$%) - follow the slow pattern in the 2-step graphs, while 12
families ($\sim 19$%) use a moderate spreading pattern (Figure 2(b)). A
similar population ($\sim 19$%) is shown in Figure 2(c), where families with a
fast spreading pattern are reported. Finally, Figure 2(d) shows that $23$% of
the considered ransomware families (15 out of 65) have an exFast spreading
pattern. These strains involve more than 500,000 addresses with peaks of
$\sim$3M, as is the case of albdecryptor, git and vega families.

(a) Slow pattern ($<$500)

(b) Moderate pattern (500-50,000)

(c) Fast pattern (50,000-500,000)

(d) Extremely Fast pattern ($>$500,000)

Figure 2: Ransomware payment spreading patterns.

Figure 3 reports the block heights of the transactions discovered in the
2-step address-transaction graph of each family. This analysis allows us to
highlight the temporality of the transactions related to seed addresses with
the spreading pattern of the family. In general, the slow and moderate
spreading patterns are composed mainly of transactions achieved in close
proximity to the seed transaction, i.e., the transaction in which the seed
appears for the first time (Figure 3(a) and Figure 3(b)). On the other hand,
as shown in Figure 3(c), fast-families show two different situations: 4
strains still achieve transactions just near to the seed transaction, while
the other 8 are characterised by transactions achieved over 5/7 years, i.e.,
their activities are spread over 200,000/300,000 blocks homogenously. Finally,
Figure 3(d) shows that all the exFast-families involve transactions
continuously in time. The clearest instance is presented by the git strain,
which is able to link transactions from about block 100,000 (2010) until block
800,000 (2024), considering just the 2-step graph. Furthermore, it is
interesting to highlight that 2-step graphs are enough for determining that
some families, such as ecovector, ransomnix, gula, spora, towerweb,
blackrouter, mainly comprise transactions achieved after the seed transaction,
suggesting that the seed is directly connected to output operations, like the
ones related to moving the funds and money laundering. On the other hand, some
families also show multiple transactions in the 2-step graphs previously to
the seed transaction, hinting at the presence of strong mechanisms for raising
funds and collecting ransom.

(a) Families with slow pattern

(b) Families with moderate pattern

(c) Families with fast pattern

(d) Families with exFast pattern

Figure 3: Block height of the transactions within the 2-step graphs for each
ransomware family. Highlighted the transactions where the seeds are involved.

##  V Graph Behavioural Analysis

###  V-A Behaviour Similarity

As mentioned in Section III-B, address-transaction graphs capture all the
dynamics a seed generates within the network, linking address and transaction
nodes. For this analysis, the topological structure (or subgraph) of each
address node is evaluated to assign a behaviour to it (transaction nodes
remain unlabelled). Before introducing the available behaviours, let $X1$ be
the address to be analysed, $N$ its input degree (i.e., the number of input
edges coming from transaction nodes) and $M$ the output degree (i.e., the
output edges to transaction nodes), depicted in Figure 4. In the same way, it
is possible to define $Np$ as the input degree of a predecessor node
(transaction node) of $X1$, while $Mp$ is the output degree of the predecessor
node. Finally, $Ns$ and $Ms$ are the input and output degrees of the successor
node (transaction node), respectively.

Figure 4: Schema of a $n$-step address-transaction graph.

In this paper, 7 behaviours have been defined, based on the most common
topological structures. The first four (A.1, A.2, A.3, and A.4) are defined by
evaluating the number of input and output addresses of the transaction nodes
directly linked to the starting point ($Ns$, $Ms$, $Np$, $Mp$), while the
other three behaviours (B.1, B.2, and B.3) are defined by checking only the
ratio between input and output transaction nodes linked ($N$, $M$). More
specifically:

######  Definition A. 1 (Collector)

Let $X1$ be a starting address node; $X1$ is said to be a collector node if it
receives money from at least one node transaction (i.e., $N>0$) that in turn
receives money from multiple addresses ($Np>3$), while $X1$ sends money to no
more than one transaction ($M<2$) that in turn sends money to no more than 2
addresses ($Ms<3$). As shown in Figure 5-A1, this behaviour is associated with
topological structures that aggregate address flows usually generated (but not
only) by ransom collecting operations.

######  Definition A. 2 (Expansion or EXP)

Let $X1$ be a starting address node; it represents a EXP if it sends money at
least to one transaction ($M>0$), which in turn sends money to multiple
addresses ($Ms>3$), and at the same time, $X1$ receives money from at least
one node transaction ($N>0$) which in turn receive money from multiple
addresses ($Np>3$). This behaviour is related to topological structures where
the central address ($X1$) serves as an intermediary between multiple
addresses (Figure 5-A2), revealing potential ”money mule” situations.

######  Definition A. 3 (Mixed Address or MA)

Let $X1$ be a starting address node; it represents a MA if it sends money to
no more than one transaction ($M<2$), and it receives money from at least one
transaction nodes that in turn receive money from multiple addresses ($Np>3$),
and, at the same time, this transaction nodes sends money not only to $X1$ but
to multiple addresses ($Mp>3$). As shown in Figure 5-A3, this behaviour is
associated with topological structures that blend address flows, potentially
to obfuscate the origin of the funds. Such structures can suggest, among other
possibilities, involvement in money laundering activities.

######  Definition A. 4 (Branching or BRANCH)

Let $X1$ be a starting address node; it represents a BRANCH if it sends money
at least to one transaction ($M>0$), which in turn sends money to no more than
2 addresses ($Ms<3$), and at the same time, $X1$ receives money from at least
one transaction node ($N>0$) which in turn receive money from multiple
addresses ($Np>3$). This behaviour is related to topological structures where
the central address ($X1$) serves as an intermediary to sort and accumulate
funds (Figure 5-A4). These structures can indicate potential money laundering
operations.

######  Definition B. 1 (Suspicious Address or SA)

Let $X1$ be a starting address node; it represents a SA if it receives money
from at least 1 transaction ($N>0$) and sends money to few transactions,
keeping a ratio of $\frac{M}{N}<0.5$. This condition generates topological
structures, such as the one depicted in Figure 5-B1, where the central address
($X1$) may function as an undetected seed or as an address involved in ransom
collection operations, potentially colluding with the attacker.

######  Definition B. 2 (Symmetric Hub or HUB)

Let $X1$ be a starting address node; it represents a HUB if it receives money
from at least 1 transaction ($N>0$) and sends money to a number of
transactions, keeping a ratio of $0.5<\frac{M}{N}<1.5$. As shown in Figure
5-B2, this behaviour is associated with topological structures that blend
multiple transaction flows, no addresses as happen for MA. Yet, these
structures can be a hint of operations aimed at obfuscating and transferring
ransom funds.

######  Definition B. 3 (Diversification)

Let $X1$ be a starting address node; it represents a Diversification if it
receives money from at least 1 transaction ($N>0$) and sends money to a large
number of transactions, keeping a ratio of $\frac{M}{N}>1.5$. As shown in
Figure 5-B3, this behaviour is associated with topological structures that
spread funds through multiple transactions. These structures may resemble,
among others, cash-back strategies used to pay various involved attackers or
to increase the complexity of fund traceability.

All the address nodes that do not fit any previously listed behaviours are not
classified, i.e., labelled as None. It is to be noted that each address node
can show none, one or two aforementioned behaviours. In this latter case, one
will be from the A-definition and the other from the B-definition.

Figure 5: Euclidean distance between ransomware families distributions.

The analysis starts by identifying the composition of each ransomware family
and counting how many address nodes it has labelled for each of the previously
defined behaviours. Therefore, these values are rescaled by the total number
of the address nodes in the graph, including those not classified but
excluding the transaction nodes. In this way, each ransomware family has been
characterised by a 7-dimensional vector, where each element represents the
probability that a node in the graph belongs to each behaviour. These vectors
are firstly analysed in a 2-dimensional space using the Principal Component
Analysis (or PCA [16]). Then, in order to evaluate the proximity and,
therefore, the similarity among the different families, their distance is
computed using the Euclidean distance [7]. In particular, let
$\mathbf{p}=(p_{1},p_{2},\ldots,p_{7})$ and
$\mathbf{q}=(q_{1},q_{2},\ldots,q_{7})$ the behaviour distribution of two
families, their distance is computed applying the Equation 1.

Figure 6 shows the PCA of the vectors. This figure highlights that slow
elements are rare (almost linear) distribution in the space (black circle),
while all the other samples are concentrated in a specific area of the space
(red square). The closeness among the family distributions can be further
appreciated in Figure 7, where a heatmap with the computed Euclidean distances
is reported. The figure reveals that different families have a very close
distribution (dark points), such as 7eve3n with bucbu, comradecircle, and
other families. On the other hand, some strains possess a unique distribution
that results in significant distances (white points), as is the case of
darkangles or storagecrypter.

$d=\sqrt{\sum_{i=1}^{7}(q_{i}-p_{i})^{2}}$ (1)

Figure 6: Visualization of all families using 2D PCA. Figure 7: Euclidean
distance between ransomware families distributions.

The distance heatmap reported in Figure 7 can be used to assess similar
distributions by establishing a predefined threshold ($\lambda$) and
subsequently determining the number of distances that are less than/equal to
$\lambda$. Specifically, the families with distances $\leq\lambda$ can be
grouped in clusters, while the others represent isolated strains. In this
analysis, values of $\lambda$ ranging from 1% to 10% of the maximum distance
reported in the heatmap ($d_{\text{max}}$, which is 76.12) are used, as shown
in Table II. The results show that the choice of $\lambda$ affects the
clusters as expected. Increasing the $\lambda$ values results in more clusters
being created, with higher average populations and fewer isolated strains.
This analysis confirms the closeness in the family behavioural distribution,
indeed, considering only 10% of the maximum distance, 57 families are grouped
in 51 different clusters, leaving only 8 strains isolated. Yet, a validation
of the closeness between ransomware behavioural distributions is reported in
Figure 8, where three behaviour distributions of random strains are chosen as
baseline (avaddon, deadbolt, and spora), and for each of them samples with
different distances are depicted.

$\lambda_{\%}$ | $\lambda=\frac{\lambda_{\%}*d_{max}}{100}$ |  | Clusters
---
number
| Clusters
---
avg. population
| Clusters
---
max. population
| Clusters
---
min. population
| Clustered
---
strains
| Isolated
---
strains
1 | 0.76 | 3 | 2.00 | 2 | 2 | 6 | 59
2 | 1.52 | 10 | 2.20 | 3 | 2 | 18 | 47
3 | 2.28 | 24 | 3.58 | 6 | 2 | 21 | 34
4 | 3.05 | 30 | 5.50 | 9 | 2 | 38 | 27
5 | 3.81 | 39 | 6.28 | 12 | 2 | 49 | 16
10 | 7.61 | 51 | 15.24 | 29 | 2 | 57 | 8
TABLE II: Distance threshold $\lambda$ with $d_{\text{max}}$=76.12.

Figures 8(b) and 8(c) confirm that strains with close distance (small
$\lambda$) indeed exhibit similar distributions, while Figure 8(a) shows that,
in this case, families display varying but limited variations in the
probabilities of behaviours such as Collector, MA, and HUB. On the other hand,
Figures 8(d), 8(e) and 8(f) demonstrate that significant variations in these
three behaviours, and in one case, also in SA, leading to increased distance
between the strains and promoting their differentiation. These results confirm
once again the outcome of the previous temporal analysis since the topological
structure associated with MA behaviours is typical of mixer transactions that
usually are used for money laundering operations, while SA and Collector
structures recall mechanisms for gathering ransom from multiple victims.

(a) Avaddon and random samples from close related families ($\lambda\leq$3.81)

(b) Deadbolt and random samples from close related families
($\lambda\leq$3.81)

(c) Spora and random samples from close related families ($\lambda\leq$3.81)

(d) Avaddon and random samples from the distant related families
($\lambda\geq$3.81)

(e) Deadbolt and random samples from the distant related families
($\lambda\geq$3.81)

(f) Spora and random samples from the distant related families
($\lambda\geq$3.81)

Figure 8: Three ransomware behaviour distributions (avaddon, deadbolt, and
spora), and their close and distant strains (sampled).

##  VI Conclusion

This study aims to evaluate if different ransomware strains follow similar
evolution trajectories and share similar strategies for their activities
(gathering the ransom, money laundering, moving the funds, etc.). To do that,
we first study the payment spreading pattern that low-labelled ransomware
families generate within the Bitcoin network. The temporal analysis revealed
that several families are able to quickly link millions of Bitcoin addresses,
analysing just their connections in 2 steps. Indeed, they have continuously
generated payment operations and activities in the Bitcoin network for years.
Furthermore, we introduce $7$ specific address behaviours related to graph
topologies that are used to validate commonalities that may exist among
families. The study demonstrates that the introduced behaviours enable not
only the characterisation of ransomware families that show quick spreading
patterns but also the slower families (the ones that require more-step
analyses). The study aids in detecting close families and confirms that
different strains employ comparable approaches in their operations, revealing
that a crucial factor lies in the topological structure created by specific
behaviours usually related to money laundering and ransom collections.

This work represents a preliminary step, offering an intriguing yet partial
understanding of the ransomware ecosystem. In fact, there is no definitive
evidence to suggest that the same attacker or group is responsible for
multiple low-labeled ransomware families. Additionally, high-represented
families should be incorporated into the analysis to refine the approach
further. Nonetheless, the results are promising, providing valuable insights
into the distinctive traits that differentiate each strain in terms of
temporal evolution and behaviour distribution.

As future work, it will be useful to involve topological structures created
starting from other known (no-ransomware-related) entities, such as Exchanges
and Mixers, in order to a) underline the validity of the behaviours defined in
this work and b) study how these entities act in the ransomware propagation.
On the other hand, ML-based clustering algorithms can be used over the
extracted characterisation of strains to gather deeper (dis)similarity
insights. These extensions will allow LEOs to develop effective ransomware
tracking mechanisms and optimise their modus operandi.

## ACKNOWLEDGMENT

This work has been partially supported by the European Union’s Horizon 2020
Research and Innovation Program under the project FALCON (Grant Agreement No.
101121281)

## References

  * [1] Akcora, C.G., Li, Y., Gel, Y.R., Kantarcioglu, M.: Bitcoinheist: Topological data analysis for ransomware detection on the bitcoin blockchain. CoRR abs/1906.07852 (2019), http://arxiv.org/abs/1906.07852
  * [2] Al-Haija, Q.A., Alsulami, A.A.: High performance classification model to identify ransomware payments for heterogeneous bitcoin networks. Electronics 10(17) (2021). https://doi.org/10.3390/electronics10172113
  * [3] Chainalysis Inc.: The 2023 Crypto Crime Report. https://go.chainalysis.com/2023-crypto-crime-report.html (2023)
  * [4] Chainalysis Inc.: The 2023 Crypto Crime Report. https://go.chainalysis.com/crypto-crime-2024.html (2024)
  * [5] Conti, M., Gangwal, A., Ruj, S.: On the economic significance of ransomware campaigns: A bitcoin transactions perspective. Computers & Security 79, 162–189 (2018). https://doi.org/https://doi.org/10.1016/j.cose.2018.08.008
  * [6] Custers, B., Oerlemans, J.J., Pool, R.: Laundering the profits of ransomware: Money laundering methods for vouchers and cryptocurrencies. European Journal of Crime, Criminal Law and Criminal Justice 28, 121–152 (07 2020). https://doi.org/10.1163/15718174-02802002
  * [7] Elmore, K.L., Richman, M.B.: Euclidean distance as a similarity metric for principal component analysis. Monthly weather review 129(3), 540–549 (2001)
  * [8] Europol: Iocta 2023. Internet Organised Crime Threat Assessment 2023, Secciones relevantes sobre ransomware (2023)
  * [9] Evenepoel, M.: Tracing ransomware transactions on the blockchain (2022)
  * [10] Fleder, M., Kester, M.S., Pillai, S.: Bitcoin transaction graph analysis. arXiv preprint arXiv:1502.01657 (2015)
  * [11] Hairil, Cahyani, N.D.W., Nuha, H.H.: Ransomware detection on bitcoin transactions using artificial neural network methods. In: 2021 9th International Conference on Information and Communication Technology (ICoICT). pp. 1–5 (2021). https://doi.org/10.1109/ICoICT52021.2021.9527414
  * [12] Haslhofer, B., Stütz, R., Romiti, M., King, R.: Graphsense: A general-purpose cryptoasset analytics platform. Arxiv pre-print (2021), https://arxiv.org/abs/2102.13613
  * [13] Huang, D.Y., Aliapoulios, M.M., Li, V.G., Invernizzi, L., Bursztein, E., McRoberts, K., Levin, J., Levchenko, K., Snoeren, A.C., McCoy, D.: Tracking ransomware end-to-end. In: 2018 IEEE Symposium on Security and Privacy (SP). pp. 618–631 (2018). https://doi.org/10.1109/SP.2018.00047
  * [14] Jung, S., Won, Y.: Ransomware detection method based on context-aware entropy analysis. Soft Computing 22, 6731–6740 (2018)
  * [15] Kotov, V., Rajpal, M.: Understanding crypto-ransomware (2023)
  * [16] Maćkiewicz, A., Ratajczak, W.: Principal components analysis (pca). Computers & Geosciences 19(3), 303–342 (1993)
  * [17] Morato, D., Berrueta, E., Magaña, E., Izal, M.: Ransomware early detection by the analysis of file sharing traffic. Journal of Network and Computer Applications 124, 14–32 (2018). https://doi.org/https://doi.org/10.1016/j.jnca.2018.09.013
  * [18] O’Kane, P., Sezer, S., Carlin, D.: Evolution of ransomware. IET Networks 7(5), 321–327 (2018). https://doi.org/https://doi.org/10.1049/iet-net.2017.0207
  * [19] Oz, H., Aris, A., Levi, A., Uluagac, A.S.: A survey on ransomware: Evolution, taxonomy, and defense solutions. CoRR abs/2102.06249 (2021), https://arxiv.org/abs/2102.06249
  * [20] Paquet-Clouston, M., Haslhofer, B., Dupont, B.: Ransomware payments in the bitcoin ecosystem. Journal of Cybersecurity 5(1), tyz003 (2019)
  * [21] Pham, T., Lee, S.: Anomaly detection in bitcoin network using unsupervised learning methods (2017)
  * [22] Richardson, R., North, M.M.: Ransomware: Evolution, mitigation and prevention. International Management Review 13(1), 10 (2017)
  * [23] Srinivasan, C.: Hobby hackers to billion-dollar industry: the evolution of ransomware. Computer Fraud & Security 2017(11), 7–9 (2017). https://doi.org/https://doi.org/10.1016/S1361-3723(17)30081-7
  * [24] Sun, Y., Xiong, H., Yiu, S.M., Lam, K.Y.: Bitanalysis: A visualization system for bitcoin wallet investigation. IEEE Transactions on Big Data 9(2), 621–636 (2022)
  * [25] Turner, A., McCombie, S., Uhlmann, A.: Follow the money: Revealing risky nodes in a ransomware-bitcoin network (2021)
  * [26] Turner, A.B., McCombie, S., Uhlmann, A.J.: Discerning payment patterns in bitcoin from ransomware attacks. Journal of Money Laundering Control 23(3), 545–589 (2020)
  * [27] Turner, A.B.: Addressing The Intelligence Applications of Bitcoin Payments Related to Ransomware. Ph.D. thesis, Macquarie University (2023). https://doi.org/10.25949/23902488.v1
  * [28] Wang, K., Pang, J., Chen, D., Zhao, Y., Huang, D., Chen, C., Han, W.: A large-scale empirical analysis of ransomware activities in bitcoin. ACM Trans. Web 16(2) (dec 2021). https://doi.org/10.1145/3494557
  * [29] Zimba, A., Chishimba, M.: On the economic impact of crypto-ransomware attacks: The state of the art on enterprise systems. European Journal for Security Research 4(1), 3–31 (2019)