# XPose: eXplainable Human Pose Estimation

Luyu Qiu1 Jianing Li2 Lei Wen1 Chi Su2 Fei Hao1 Chen Jason Zhang2&Lei Chen1,2
1Hong Kong University of Science and Technology
2Hong Kong Polytechnic University
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>chisu,
ffaye.hao<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Current approaches in pose estimation primarily concentrate on enhancing model
architectures, often overlooking the importance of comprehensively
understanding the rationale behind model decisions. In this paper, we propose
XPose, a novel framework that incorporates Explainable AI (XAI) principles
into pose estimation. This integration aims to elucidate the individual
contribution of each keypoint to final prediction, thereby elevating the
model’s transparency and interpretability. Conventional XAI techniques have
predominantly addressed tasks with single-target tasks like classification.
Additionally, the application of Shapley value, a common measure in XAI, to
pose estimation has been hindered by prohibitive computational demands.

To address these challenges, this work introduces an innovative concept called
Group Shapley Value (GSV). This approach strategically organizes keypoints
into clusters based on their interdependencies. Within these clusters, GSV
meticulously calculates Shapley value for keypoints, while for inter-cluster
keypoints, it opts for a more holistic group-level valuation. This dual-level
computation framework meticulously assesses keypoint contributions to the
final outcome, optimizing computational efficiency. Building on the insights
into keypoint interactions, we devise a novel data augmentation technique
known as Group-based Keypoint Removal (GKR). This method ingeniously removes
individual keypoints during training phases, deliberately preserving those
with strong mutual connections, thereby refining the model’s predictive
prowess for non-visible keypoints. The empirical validation of GKR across a
spectrum of standard approaches attests to its efficacy. GKR’s success
demonstrates how using Explainable AI (XAI) can directly enhance pose
estimation models.

##  1 Introduction

Recent years witness the rapid develop of deep learning technology on various
of field include computer vision Krizhevsky et al. (2012); He et al. (2016)
and neural language process Vaswani and others (2017); Devlin et al. (2018).
However, the increasing complexity makes model challenging to design through
manual intuition. Therefore, it’s crucial to understand model’s decision-
making process.

The emergence of eXplainable AI (XAI) has addressed the opacity and lack of
interpretability associated with advanced machine learning models Ignatiev
(2020), enabling researchers and practitioners to comprehend the rationale
behind their predictions Zeiler and Fergus (2014). Existing XAI methods have
primarily concentrated on single-target tasks like classification Petsiuk et
al. (2018) and detection Sarp et al. (2023), offering valuable insights into
the decision-making processes of these models. However, as AI tasks become
more intricate, the inherent correlations in multi-target tasks are often
neglected, limiting the applicability of existing XAI techniques.

Figure 1: Illustrations of (a) human image, and corresponding (b) human body
keypoints and (c) groupped keypoints.

Multi-Person Pose Estimation (MPPE) aims to identify and localize human body
keypoints. Fig.1(a-b) illustrates the MPPE task. As a fundamental yet
challenging task in human behavior understanding, pose estimation has
attracted increasing attention in recent years. In MPPE task, the prediction
of individual body keypoint will be influenced by each other keypoints.
However, existing methods regard body keypoints as separate ones, and mainly
concentrate on design superior models Wei et al. (2016); He et al. (2017);
Newell et al. (2017); Cheng et al. (2020) or unbiased keypoint formulation
Zhou et al. (2019); Zhang et al. (2020), overlooking the correlation between
keypoints, which could be beneficial for keypoint predictions. On the other
hand, current XAI methods are mainly designed for single-target tasks, may
fall short in capturing the intricate relationships. Hence, these methods are
challenging to directly apply for MPPE models.

This paper focuses on revealing the decision-making process of MPPE models
through elucidating the individual contribution of each keypoint to final
predictions, offering insights for future model design. In MPPE task, the
black-box model aims to predicts biological body keypoints of each person
presents in input image. The shapley value Shapley (1951) is a common measure
in XAI which used to quantify the contribution of each feature by calculating
the average marginal impact across all possible feature combinations. However,
the abundance of keypoints poses a challenge as computing shapley value for
each individual keypoint requires evaluating all possible combinations of
keypoints, which will cause unaffordable computation cost. To handle this
difficulty, we design a Group Shapley Value (GSV), to compute the
contributation of each keypoint in a coarse-to-fine manner. As shown in Fig.
1(b), a body keypoint only biologically connects to a limited number of
keypoints, hence we compute fine-grained shapley value for only closely
related keypoints, through which avoid redundant computations. While for
loosely related keypoints, GSV calculates coarse-grained group-level shapley
value to reduce the computation cost. Both fine-grained and coarse-grained
constitute to overall contribution of each keypoint to final contribution. To
group keypoints, we first estimate the interdependencies of each keypoint
pair. Following RISE Petsiuk et al. (2018), we assess the interdependencies of
each two keypoints through removing pixels around keypoints in the image.
Since the prior information from physiological structure of human body is also
important, we introduce connectivity of keypoints when evaluating
interdependencies. Fig. 1(c) illustrates the group result.

Building on the insights into keypoint interactions, we design Group-based
Kepoint Removal (GKR) data augmentation to enhence model’s ability of
inferring invisible keypoint. During training, GKR randomly remove individual
keypoint while keep its related keypoints unchanged, the loss of single
keypoint compels model to infer through other closely related ones.
Consequently, the model learns to better deduce invisible keypoints during
inference.

The contribution of this work can be summarized into following aspects: 1) We
propose XPose, a XAI method to reveal the decision make process of MPPE task.
To best of our knowledge, this is an original work introducing XAI into MPPE.
2) A group shapley value is designed to reduce the computation complexity in
shapley value, makes it better fit pose estimation task. 3) Based on
interpretation of XPose, we further design a Group-based Kepoint Removal to
enhence the model’s ability of inferring invisible keypoints, and
significantly boosts the performance of several typical methods.

##  2 Related work

This work is closely related to eXplainable AI(XAI) and human pose estimation.
This section briefly summarizes those two categories of works.

###  2.1 eXplainable AI

Gradient-based XAI methods offer detailed insights into model decision-making
by requiring the model to be a white-box and its processes differentiable,
which limits their real-world applicability. In contrast, perturbation-based
XAI methods, which do not assume the model to be transparent or
differentiable, have broader applications due to their versatile techniques
like addition, removal, and replacement, with removal being the most utilized
Qiu et al. (2022).

A recent survey Covert et al. (2021) provided a comprehensive overview of
removal-based XAI methods, proposing a framework that dissects these methods
through feature removal methods, model behavior, and summary technique.
Feature removal techniques are characterized by diverse approaches to
substituting altered segments, each with distinct methodologies for
recalculating or replacing feature values. Simple methods such as zero Petsiuk
et al. (2018) or default value Ribeiro et al. (2016) directly modify feature
values with zero or a default value. In contrast, missingness during training
Yoon et al. (2019) conditions the model to interpret zeros or alternative
replacement values as indicators of missing data, rather than features
naturally equating to zero. Techniques like blurring with a Gaussian kernel
Fong et al. (2019) or extending pixel values from neighboring pixels Zhou et
al. (2015) offer more nuanced alterations. Generative model Chang et al.
(2019) approaches calculate the fill values through generative techniques
whereas separate models Williamson and Feng (2020) replace the single model
with separate models for every feature.

Model behavior techniques can be categorized into three types. Prediction
methods evaluate the impact of excluding specific features on the individual
prediction Zeiler and Fergus (2014). Prediction loss methods measure the
prediction loss Lundberg et al. (2020) or prediction mean loss Yoon et al.
(2019) calculated by true labels and prediction labels. Dataset loss methods
evaluate the overall performance alteration of a model when every features
removed systematically Williamson and Feng (2020).

Summary techniques encompass several approaches. Shapley valuequantifies an
individual feature’s contribution by considering all possible permutations
Winter (2002). This paper also involves the concept of Shapley value.
Techniques for removing Zeiler and Fergus (2014) or including Guyon and
Elisseeff (2003) individual features assess the impact of these actions on
model performance. Additive models employ regularized additive models to
analyze the dataset comprised of perturbed examples Ribeiro et al. (2016).
High-value subset and low-value subset formulate the summarization as an
optimization problem. The former Zhou et al. (2015) seeks a feature subset
that maximizes the set function’s value, whereas the latter Fong et al. (2019)
selects features whose exclusion minimizes the set function’s value.

Figure 2: Overview of the proposed XPose for explainable pose estimation.
XPose takes masked person image as input and evaluate the influence of each
keypoint on all other keypoints.

###  2.2 Pose estimation

As a fundamental yet challenging task in human behavior understanding, pose
estimation has attracted increasing attention in recent years. Based on the
keypoint formulation, existing works can be summarized into two categories,
_i.e._ , heatmap-based methods and regression based methods.

Heatmap based pose estimation predicts a probability for each person and
locate the keypoint through taking the maximum of heatmap. Based on the order
of keypoints localization and grouping, heatmap based pose estimation works
can be summarised into top-down and bottom-up methods Top-down methods He et
al. (2017); Fang et al. (2017); Sun et al. (2018, 2019) first detect and crop
bounding box for each person in image, and resize cropped images to the same
size before performing single person pose estimation. He _et al._ He et al.
(2017) add a pose estimation branch based on Faster R-CNN Ren et al. (2015)
and reuse the features for multi-task learning. Fang _et al._ Fang et al.
(2017) propose a symmetric spatial transformer network to refine the
inaccurate bounding boxes. The top-down models achieve promising performance
on various MPPE datasets. However, the top-down paradigms need to crop, resize
and estimate pose for every person instance individually, which significantly
increases the computation and inference time.

In contrast, bottom-up methods Cao et al. (2017); Newell et al. (2017); Cheng
et al. (2020) first localize all keypoints in the image, then group them into
different persons. OpenPose Cao et al. (2017) introduces a two-branch network
to predict the heatmap for body joints and affinity field, respectively. The
grouping is implemented through integrating two keypoints on affinity fields
and grouping the keypoint pairs with the largest integral. Cheng _et al._
Cheng et al. (2020) design a HigherHRNet to predict high resolution heatmap
for keypoint estimation to solve the scale variation challenge in bottom-up
methods. Bottom-up method does not rely on extra human detectors, thus is more
robust to occlusions. The grouping process could be complicated and time-
consuming because it is hard to be processed in parallel.

Regression based pose estimation Zhou et al. (2019); Nie et al. (2019); Wei et
al. (2020) formulates MPPE as a regression task, where human keypoints are
defined as person center point and a set of offsets. Zhou _et al._ Zhou et al.
(2019) first regress the offset between body keypoints and person center, then
match the regression result to the closest joint detected from heatmap to
refine the result. Instead of directly regressing from the center point, Wei
_et al._ Wei et al. (2020) define a set of anchor points and regress the
keypoints from anchor points to avoid long-range regression. Li _et al._ Li et
al. (2023) try the factorize the 2D regression into angle and length in polar
coordinate to simplify the regression through classification. Compared with
the two-stage heatmap based methods, regression based methods show better
efficiency.

However, all above methods treat body keypoints as isolated entity,
overlooking the inherent correlation, which is crucial for MPPE task. In this
paper, we focus on XAI and reveal the intrinsic connections between body
keypoints in decision making process. We believe this will provide guidance
for the design of future pose estimation methods.

##  3 Method

###  3.1 Formulation

Given an image $I$, Multi-Person Pose Estimation (MPPE) aims to estimate the
pose for each person via locating coordinates of body keypoints $\mathcal{P}$,
$\mathcal{P}$ can be represented as,

$\displaystyle\mathcal{P}=\\{k_{1},k_{2},...k_{n}\\},$ (1)

where $n$ is the number of keypoints. $k_{i}=(x_{i},y_{i},v_{i})$ is the
$i$-th keypoint, where $(x_{i},y_{i})$ and $v_{i}$ are coordinate and
visibility of $i$-th keypoint, respectively.

For each keypoint $k_{i}$, XPose aims to evaluate the influence and
significance of other keypoints on $k_{i}$.

$\displaystyle
X(k_{i})=\\{\sigma(k_{i},k_{1}),\sigma(k_{i},k_{2}),...,\sigma(k_{i},k_{n})\\},$
(2)

where $\sigma(k_{i},k_{j})$ denotes the contributation of keypoint $k_{j}$
when predicting keypoint $k_{j}$, and the contribution $\sigma$ satisfy

$\displaystyle\sigma(k_{i},k_{j})$ $\displaystyle\geq 0$ (3)
$\displaystyle\sum_{j=1}^{n}(\sigma(k_{i},\cdot))$ $\displaystyle=1.$

The shapley value is a common measure in XAI to measure the contribution of
each input feature. However, shapley value need to compute all combination of
input features, which is unaffordable for high input dimensions. To simplify
the computation, we propose to compute shapley value in a coarse to fine
manner. We first cluster the keypoints into several groups based on their
interdependencies, where 1) keypoints inside same group are closely-related
with each other; 2) keypoints in different groups are loosely-related. Based
on the group, we compute fine-grained shapley value in each group and group-
level coarse-grained shapley value.

Beside, with the guidance of keypoint interaction from XPose, we design a
Group-based Keypoint Removal data augmentation to enhance model’s inferential
capability for invisible keypoints. Fig. 2 illustrates the overall framework
of XPose. The following parts proceed to introduce details of XPose, including
grouping process and group shapley value.

###  3.2 Grouping process

To cluster keypoints, we evaluate interdependencies between each pair of
keypoints One way to measure the interdependencies of a keypoint is to perturb
it and observe how much this affects the model’s decision of other keypoint.
RISE Petsiuk et al. (2018) try to explore the importance of image areas
through applying thousands of randomly generated masks on image and observe
the output of black-box models.

Randomly perturb input image and observe output is intuitionistic but
ineffective, especially for MPPE task, as majority of regions in image are
unrelated background. Considering keypoint’s location is highly related with
specific local region in image, we guide perturbation using the coordinates of
keypoints. Specially, to evaluate the influence of keypoint $k_{j}$ to
keypoint $k_{i}$, we randomly generate $m$ masks with random aspect ratio and
area, and average the performance of keypoint $k_{i}$ as the perturbation
$\Delta p(k_{i},k_{j})$. We normalize $\Delta p(k_{i},k_{j})$ as perturbation
influence of each keypoint pair,

$\displaystyle\operatorname{PI}(k_{1},k_{2})=\frac{1}{2}({\frac{\Delta
p(k_{i},k_{j})}{\sum_{a=1}^{n}\Delta p(k_{i},k_{a})}+\frac{\Delta
p(k_{j},k_{i})}{\sum_{a=1}^{n}\Delta p(k_{j},k_{a})}})$ (4)

where $\Delta p(k_{i},k_{j})$ denotes the performance drop of keypoint $k_{i}$
when perturbing $k_{j}$. A higher $PI$ denotes the two keypoints are highly
related.

Besides, we consider the structural information of the human body.
Physiologically connected keypoints exhibit stronger interdependencies,
whereas is weaker, _e.g._ , the elbow keypoint has closer connection with
shoulder and wrist keypoints, but weaker connection with knees. We then
incorporate keypoint connectivity into interdependencies. For keypoint pair
$(k_{1},k_{2})$, we define the keypoint connectivity as,

$\displaystyle\operatorname{KC}(k_{i},k_{j})=\frac{1}{2}({\frac{conn(k_{i},k_{j})}{\sum_{a=1}^{n}conn(k_{i},k_{a})}+\frac{conn(k_{j},k_{i})}{\sum_{a=1}^{n}conn(k_{j},k_{a})}})$
(5)

where $conn(k_{i},k_{j})$ is connectivity of two keypoints, $conn=1$ if 2
keypoints are biologically connected else $0$. Similarly, we normalize the
connectivity to keypoint connectivity.

The interdependencies $s_{inter}$ of keypoint pair jointly considers
perturbation influence and keypoint connectivity,

$\displaystyle s_{inter}(k_{1},k_{2})=\operatorname{PI}+\operatorname{KC}$ (6)

Based on $s_{inter}$, we group kypoints and the grouping results are
summarised in Table 1. In Table 1, keypoint are clustered into 5 groups, which
aligns well with human body structure, _e.g._ , head, arms and legs. The
grouped keypoints are subsequently used to compute group shapley value.

Table 1: The group result of keypoints. Group id |  keypoint name
---|---
Group 1 |  l-eye, r-eye, nose, l-ear, r-ear
Group 2 |  l-shoulder, l-elbow, l-wrist
Group 3 |  r-shoulder, r-elbow, r-wrist
Group 4 |  l-hip, l-knee, l-foot
Group 5 |  r-hip, r-knee, r-knee

###  3.3 Group Shapley Value

Directly compute shapley value on all keypoints will suffer unaffordable
computation cost. Considering each keypoint is related with only by a subset
of keypoints, we propose Group Shapley Value (GSV) to compute shapley value in
a coarse-to-fine manner.

GSV first cluster all keypoints $\mathcal{P}$ into $g$ groups based on
interdependencies $s_{inter}$ in Sec. 3.2,

$\displaystyle\operatorname{Cluster}(\mathcal{P})=\\{\mathcal{G}_{1},\mathcal{G}_{2},...,\mathcal{G}_{g}\\}.$
(7)

GSV then computes fine-grained shapley value for keypoints within same group
$\mathcal{G}$, in which keypoints are highly interdependent. And course-
grained group-level shapley value for keypoints cross groups which are weakly
interdependent.

For keypoint $k_{i}$, we evaluate the contribution of keypoint $k_{j}$ to
predict $k_{i}$ through computing fine-grained shapley value within group,

$\displaystyle\phi_{ij}=\sum_{g\in\mathcal{G}}\omega(|g|)[v_{i}(g)-v_{i}(g\setminus\\{j\\})]$
(8)

where $g$ is a keypoint subset of group $\mathcal{G}$, $v_{i}(g)$ denotes the
performance of keypoint $k_{i}$ with the keypoints in set $g$ visible, and
$v_{i}(s\setminus\\{j\\})$ denotes the performance with keypoint $k_{j}$
perturbed, $\omega(|g|)$ is the weight of subset $g$.

And the group level shapley value $\varphi_{ij}$ treat each group as
individual input, and compute the contribution of other groups to group
$\mathcal{G}_{i}$,

$\displaystyle\varphi_{ij}=\sum_{\mathcal{G}\in\operatorname{Cluster(\mathcal{P})}}\omega(|g|)[v_{i}(g)-v_{i}(g\setminus\\{j\\})],$
(9)

where $v_{i}(\mathcal{G}$ denotes the average performance of keypoints in
group $\mathcal{G}_{i}$ with subset $g$ visible. Both keypoint level $\phi$
and group shapley $\varphi$ value are normalized to ensure its sum is equal to
1.

###  3.4 Group-based Keypoint Removal

The result of group shapley value indicates that each keypoint only highly
related to a subset of keypoints rather than all keypoints. Hence, we can
conclude that when a keypoint is occluded, deep model can infer invisible
keypoint based on the information from related keypoints, _e.g._ , the elbow
keypoint can be inferred by shoulder and wrist. Based on this observation, we
design Group-based Keypoint Removal (GKR) data augmentation to enhance model’s
inferential capability of invisible keypoints.

GKR applies random perturbations on individual keypoint within each group
while keeping the other keypoints unchanged during training. The perturbation
is implemented through replace a randomly selected rectangular region centered
at keypoints with random noise. The detail process of GKR augmentation as
follow,

Input : Input image $I$;

Image size $W$ and $H$;

Perturb probability $p$;

Group keypoint
$\mathcal{P}=\\{\mathcal{G}_{1},\mathcal{G}_{2},...,\mathcal{G}_{c}\\}$;

Perturb area scale $S=\\{s_{1},s_{2},...,s_{n}\\}$.

Output : Erased image $I^{\ast}$.

Initialization: $p_{1}\leftarrow$ Rand (0, 1).

1 for  _$i=1$ to c_ do

2 $p_{i}\leftarrow$ Rand (0, 1)

3 if  _$p_{i} >p$_ then

4 $j\leftarrow$ Randint (1, $|\mathcal{P}_{i}|$)

5 $x,y\leftarrow\mathcal{P}_{i}[j][0]$, $\mathcal{P}_{i}[j][1]$

6 $s\leftarrow S[\mathcal{P}_{i}]$

7 $h\leftarrow H\times s,w=W\times s$

8 $I_{p}\leftarrow(x-\frac{w}{2},y-\frac{h}{2},x+\frac{w}{2},y+\frac{h}{2})$,

9 $I(I_{P})\leftarrow$Rand(0,255)

10 $I^{\ast}\leftarrow I(I_{P})$

11 end if

12

13 end for

Algorithm 1 GKR augmentation Procedure Table 2: Performance drop with when perturbing individual keypoint. $\Delta AP$ | baseline | nose | l-eye | r-eye | l-ear | r-ear | l-shd | r-shd | l-elbow | r-elbow | l-wrist | r-wrist | l-hip | r-hip | l-knee | r-knee | l-foot | r-foot
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
nose | 76.1 | 20.6 | 5.3 | 4.1 | 1.7 | 1.3 | 0.7 | 0.9 | 0.0 | 0.1 | 0.1 | 0.0 | 0.1 | 0.2 | 0.2 | 0.1 | 0.1 | 0.0
l-eye | 77.5 | 9.3 | 7.1 | 2.3 | 2.7 | 2.3 | 0.6 | 0.7 | 0.1 | 0.4 | 0.1 | 0.1 | 0.2 | 0.0 | 0.1 | 0.1 | 0 | 0.0
r-eye | 77.5 | 10.2 | 3.3 | 7.5 | 1.2 | 1.6 | 0.6 | 0.4 | 0.4 | 0.2 | 0.0 | 0.1 | 0.2 | 0.2 | 0.1 | 0.1 | 0.1 | 0.1
l-ear | 79.4 | 2.4 | 2.3 | 1.8 | 8.9 | 1.7 | 0.5 | 0.2 | 0.3 | 0.6 | 0.5 | 0.6 | 0.1 | 0.1 | 0.1 | 0.2 | 0.2 | 0.1
r-ear | 78.6 | 2.4 | 2.8 | 2.6 | 1.9 | 9.3 | 0.4 | 0.6 | 0.1 | 0.2 | 0.1 | 0.0 | 0.1 | 0.0 | 0.1 | 0.1 | 0.0 | 0.0
l-shoulder | 81.5 | 0.5 | 0.5 | 0.3 | 0.4 | 0.2 | 14.0 | 0.2 | 3.2 | 0.4 | 1.9 | 0.3 | 0.5 | 0.2 | 0.1 | 0.0 | 0.1 | 0.0
r-shoulder | 81.6 | 0.8 | 0.5 | 0.4 | 0.2 | 0.2 | 3.2 | 12.5 | 0.5 | 2.5 | 0.3 | 1.9 | 0.2 | 0.3 | 0.2 | 0.3 | 0.0 | 0.2
l-elbow | 76.4 | 0.2 | 0.3 | 0.0 | 0.2 | 0.1 | 4.8 | 0.8 | 15.1 | 0.5 | 6.0 | 0.7 | 0.9 | 0.5 | 0.4 | 0.0 | 0.0 | 0.1
r-elbow | 75.7 | 0.1 | 0.1 | 0.0 | 0.2 | 0.0 | 0.9 | 4.9 | 0.5 | 14.5 | 0.6 | 6.3 | 0.2 | 0.7 | 0.0 | 0.2 | 0.2 | 0.0
l-wrist | 73.2 | 0.2 | 0.3 | 0.1 | 0.0 | 0.1 | 2.9 | 0.8 | 9.4 | 1.0 | 27.3 | 1.9 | 3.0 | 1.2 | 0.6 | 0.1 | 0.1 | 0.0
r-wrist | 73.1 | 0.2 | 0.2 | 0.0 | 0.1 | 0.1 | 0.6 | 3.0 | 1.2 | 8.2 | 0.9 | 27.4 | 1.0 | 3.1 | 0.6 | 0.0 | 0.1 | 0.1
l-hip | 76.0 | 0.6 | 0.3 | 0.3 | 0.2 | 0.1 | 1.6 | 1.1 | 0.8 | 0.6 | 0.6 | 0.7 | 4.3 | 1.3 | 2.2 | 0.5 | 1.4 | 0.4
r-hip | 76.0 | 0.2 | 0.2 | 0.1 | 0.1 | 0.0 | 1.2 | 1.4 | 0.4 | 0.6 | 0.2 | 0.6 | 1.4 | 4.5 | 0.6 | 1.9 | 0.2 | 1.2
l-knee | 78.1 | 0.1 | 0.1 | 0.0 | 0.0 | 0.1 | 0.2 | 0.1 | 0.0 | 0.0 | 0.3 | 0.0 | 2.3 | 0.3 | 8.9 | 0.8 | 1.5 | 0.2
r-knee | 78.5 | 0 | 0.2 | 0 | 0.0 | 0.2 | 0.1 | 0.3 | 0.0 | 0.1 | 0.3 | 0.0 | 0.8 | 2.7 | 1.6 | 10.2 | 0.3 | 2.8
l-foot | 79.2 | 0.1 | 0.0 | 0.0 | 0.0 | 0.1 | 0.2 | 0.1 | 0.3 | 0.2 | 0.3 | 0.0 | 1.6 | 0.5 | 2.1 | 0.7 | 10.6 | 0.9
r-foot | 79.9 | 0 | 0.0 | 0.1 | 0.1 | 0.1 | 0.4 | 0.2 | 0.0 | 0.0 | 0.0 | 0.2 | 0.3 | 1.9 | 0.8 | 2.1 | 1.0 | 10.9

##  4 Experiment

###  4.1 Dataset

We explore XPose on widely used multi-person pose estimation datasets MSCOCO
Lin et al. (2014). MSCOCO dataset contains about 80k images and over 200k
person instances with 17 annotated body keypoints. COCO dataset is divided
into train, val, and test-dev sets with 57k, 5k and 20k images. The
performance is evaluated by Average Precision (AP). Our models are trained on
COCO train2017 with about 57k images. All ablation study are conducted on
val2017 set.

###  4.2 Implementation Details

Training: We select 3 typical method to implement XPose, include top-down
method HRNet Sun et al. (2019), bottom-up method HigherHRNet Cheng et al.
(2020) and regression method PolarPose Li et al. (2023).

For training, We employed the data augmentation mentioned in corresponding
paper Sun et al. (2019); Cheng et al. (2020); Li et al. (2023), including
random rotation, random scale, random translation and random flipping. The
input images are resized to $256\times 192$ for HRNet Sun et al. (2019) and
$512\times 512$ for HigherHRNet Cheng et al. (2020) and PolarPose Li et al.
(2023). For perturb scale $s$ in GKR data augmentation, we set $s=0.05$ for
head keypoints and $s=0.15$ for other keypoints.

Test: Following original paper Sun et al. (2019); Cheng et al. (2020); Li et
al. (2023), flip testing is used for all experiments. All reported performance
is obtained with single model without ensembling.

Table 3: Normalized shapely value of group. value(%) | head | l-arm | r-arm | l-leg | r-leg
---|---|---|---|---|---
head | 96.2 | 1.4 | 1.9 | 0.4 | 0.1
l-arm | 6.8 | 83.1 | 5.8 | 2.7 | 1.7
r-arm | 6.0 | 6.6 | 83.5 | 1.2 | 2.7
l-leg | 4.0 | 6.3 | 4.1 | 69.5 | 16.1
r-leg | 4.2 | 5.3 | 5.9 | 18.2 | 66.4

Table 4: Normalized shapely value of head keypoint. value(%) | nose | l-eye | r-eye | l-ear | r-ear
---|---|---|---|---|---
nose | 52.4 | 13.3 | 26.7 | 4.4 | 3.4
l-eye | 41.6 | 31.8 | 8.8 | 13.0 | 4.9
r-eye | 44.8 | 12.4 | 30.3 | 4.4 | 8.1
l-ear | 10.7 | 10.8 | 3.8 | 65.6 | 9.1
r-ear | 15.0 | 3.6 | 12.3 | 3.4 | 65.7

Table 5: Left arm shapely value. value | l-shd | l-elbow | l-wrist
---|---|---|---
l-shoulder | 74.2 | 18.6 | 7.2
l-elbow | 25.3 | 50.3 | 24.4
r-wrist | 8.7 | 23.7 | 67.7

Table 6: right arm shapely value. value | r-shd | r-elbow | r-wrist
---|---|---|---
r-shoulder | 73.5 | 17.9 | 8.6
r-elbow | 24.6 | 49.6 | 25.8
r-wrist | 9.0 | 22.5 | 68.5

Table 7: Left leg shapely value. value | l-hip | l-knee | l-ankle
---|---|---|---
l-hip | 68.8 | 23.8 | 7.3
l-knee | 14.9 | 65.0 | 20.1
l-ankle | 6.7 | 25.5 | 67.9

Table 8: Right leg shapely value. value | r-hip | r-knee | r-ankle
---|---|---|---
r-hip | 72.4 | 20.3 | 7.3
r-knee | 14.3 | 66.1 | 19.6
r-ankle | 6.7 | 23.4 | 69.1

Figure 3: The correlation of predicted keypoint confidence on whole dataset.
The high correlation value denotes the 2 keypoint concurrently receive high or
low confidence scores.

###  4.3 Experimental result and analysis

This section exhibites the experimental result and provide detail explaination
and analysis.

The Interdependencies of each keypoint: This part investigates the
interdependencies of each keypoint pair. Experimental results are summarized
in Table 2. Each line in Table 2 shows the performance of individual keypoint,
and each column indicates perturbing corresponding keypoint.

As shown Table 2 that, the largest peformance drop of each keypoint comes from
removing itself, _e.g._ , nose keypoint suffer 20.6% performance drop when
remove itself. While perturbing a single keypoint does not decrease its
performance to near zero, this shows deep model could infer invisible
keypoints based on contextual cues from other visible keypoints. Table 2 also
shows that, besides itself, keypoints are most influenced by physiologically
connected points, _e.g._ , the elbow keypoint is mostly influenced by shoulder
and wrist.

Group Shapley Value: Tthis section investigates the GSV. Table 8 shows the
group-level shapley value and Table 8\- 8 summarizes the intra-group shapley
inside each group. To better observe the contribution of keypoints, we
normalize shapley values to make their sum equal to 1.

It can be observed from Table 8 that, there is little correlation between
groups, _e.g._ , left arm information only contribute 1.4% in head keypoints
prediction. It can also observed from table, there is non-negligible
correlation between the keypoints of the left and right leg. The reason is
because that, the keypoints in left and right leg are usually overlap by each
other, the removal operation on individual keypoint may also remove other
keypoints, hence make left and right legs have relatively high correlations.

Table 8 shows the normalize the shapley value of head keypoints. It can be
observed from table that, nose and eyes have high correlation, the reason is
that the eyes and nose keypoints are close in proximity, making it arbitrary
to predict one keypoint based on the other. While the ear keypoints are
relatively far from other keypoints, and predicting their position based on
other keypoints can only be done vaguely.

Table 8 and Table 8 summarised the shapley value on left and right arms
keypoints. Compared with head keypoints, the arm keypoints have a closer
correlation with itself, _e.g._ , left shoulder keypoint contribute 74.2% to
left shoulder accuracy. It’s interesting to observe that, the correlation
between keypoints is consistent with the anatomical structure of the human
body, _e.g._ , the shoulder keypoint have high correlation with elbow
keypoint, while low correlation with wrist keypoint, the elbow keypoint has
high correlation with both shoulder and wrist keypoints. The reason is that
the elbow keypoint is directly connected to both shoulder and wrist keypoints,
allowing the inference of these two keypoints based on the human body
structure, whereas the shoulder and elbow are not directly connected, making
it difficult to infer.

Figure 4: Illustration of pose estimation results achieved by HRNet (1st line)
and HRNet+GKR (2nd line), keypoins of different persons are annotated with
different colors.

Table 8 and Table 8 summarised the shapley value on left and right legs
keypoints. Similar with arms keypoints, the leg keypoints is also consistent
with the anatomical structure of human body. The contribution of knee to hip
keypoint is relatively low, mainly because the hip keypoint area is larger,
leading to significant errors when inferring based on the knee.

Based on inter and intra-group shapley value, we elucidate the comprehensive
contributions of each keypoint to final prediction. Specifically, within the
same group, we leverage fine-grained intra-group Shapley values to elucidate
contributions of individual keypoint. Meanwhile, for keypoints belonging to
different groups, we utilize the inter-group Shapley values associated with
their respective groups. The overall contribution is We present the overall
Shapley value and Shapley value based on individual image in appendix.

Correlation of Confidence score: This part exhibits the correlation of
predicted keypoint confidence score on overall dataset, the result is
illustrated in Fig. 3. The correlation between keypoints is computed based on
all instances on COCO training set. A high correlation indicates that these
two keypoints concurrently receive either high or low confidence scores. As
shown in Fig. 3, the correlation of confidence score exhibits similar
conclusion with grouping result in Table 1, _e.g._ , the confidence of
keypoints in same group is relatively high. The reason is that deep model can
infer a keypoint based on its related keypoints, resulting in simultaneous
high/low confidence scores for multiple keypoints.

Table 9: The effectiveness of GKR on COCO dataset.  Method | data aug | occlusion ratio
---|---|---
0 | 0.25 | 0.5 | 0.75
| None | 74.5 | 68.2 | 54.6 | 14.9
HRNet | RE | 74.9 | 68.5 | 55.0 | 15.3
| GKR | 75.5 | 68.9 | 55.4 | 15.5
| None | 66.4 | 60.0 | 43.8 | 9.2
HrHRNet | RE | 66.7 | 60.5 | 44.5 | 9.6
| GKR | 67.0 | 61.0 | 44.7 | 9.9
| None | 68.5 | 62.1 | 47.3 | 10.3
PolarPose | RE | 68.9 | 62.7 | 47.7 | 10.6
| GKR | 69.3 | 63.3 | 47.9 | 11.0

###  4.4 Evaluation of GKR data augmentation

This part verifies the effectiveness of GKR. We divide COCO dataset into
subsets with different occlusion ratio. The occlusion ratio is defined as the
ratio of invisible keypoint to all keypoint in an image. We implement 3
typical methods to verify GKR, include HRNet Sun et al. (2019), HigherHRNet
Cheng et al. (2020) and PolarPose Li et al. (2023). The experimental results
are summarized in Table 9. In Table 9, “None” denotes the baseline method
which directly trains deep model without any data augmentation, “RE” applies
random erasing data agumentation Zhong et al. (2020), and “GKR” applies GKR
data augmentation for training.

In Table 9, based on baseline “None”, both RE and GKR promote performance on
non-occlusion data, _e.g._ , RE promotes baseline by 0.4% on HRNet Sun et al.
(2019). GKB promotes the perofrmance by 1.0%, which is more than twice of RE.
The reason is because that, without the guidance of keypoints, random erasing
data augmentation may simultaneously remove multiple related keypoints, which
confusing the model training. In contrast, our approach removes single
keypoint from each group, allowing model infer deleted keypoint based on
remaining keypoints, hence strength the inferential capability. We further
verify GKR on samples with occlusion. As shown in Table 9, under the occlusion
scenarios, GKR demonstrates superior accuracy.This shows the effectiveness of
GKR to enhence the inferential capability for invisible keypoints.

Finally, we illustrate some visualization results in Fig. 4 achieved by HRNet
Sun et al. (2019) and HRNet+GKR. We emphasize the difference between
predictions in the figure. As shown in the figure, the result achieved by GKR
can better predict invisible body keypoints.

##  5 Conclusion

This paper introduces XPose, a framework that integrates XAI into pose
estimation task to elucidate the individual contribution of each keypoint to
the final predictions. XPose assesses the contribution of each keypoint
through a novel group Shapley value, designed to circumvent the complexities
associated with the original Shapley value computation. Building on XPose, we
also propose a group-based keypoint removal data augmentation method to
enhance the model’s inferential capability for invisible keypoints.
Comprehensive experimental results across three typical methods showcase the
effectiveness of the proposed data augmentation approach.

## References

  * Cao et al. [2017] Zhe Cao, Tomas Simon, Shih-En Wei, and Yaser Sheikh.  Realtime multi-person 2d pose estimation using part affinity fields.  In CVPR, pages 7291–7299, 2017.
  * Chang et al. [2019] Chun-Hao Chang, Elliot Creager, Anna Goldenberg, and David Duvenaud.  Explaining image classifiers by counterfactual generation, 2019.
  * Cheng et al. [2020] Bowen Cheng, Bin Xiao, Jingdong Wang, Honghui Shi, Thomas S Huang, and Lei Zhang.  Higherhrnet: Scale-aware representation learning for bottom-up human pose estimation.  In CVPR, pages 5386–5395, 2020.
  * Covert et al. [2021] Ian C. Covert, Scott Lundberg, and Su-In Lee.  Explaining by removing: a unified framework for model explanation.  J. Mach. Learn. Res., 22(1), jan 2021.
  * Devlin et al. [2018] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding.  arXiv preprint arXiv:1810.04805, 2018.
  * Fang et al. [2017] Hao-Shu Fang, Shuqin Xie, Yu-Wing Tai, and Cewu Lu.  Rmpe: Regional multi-person pose estimation.  In ICCV, pages 2334–2343, 2017.
  * Fong et al. [2019] Ruth Fong, Mandela Patrick, and Andrea Vedaldi.  Understanding deep networks via extremal perturbations and smooth masks.  In 2019 IEEE/CVF International Conference on Computer Vision, ICCV 2019, Seoul, Korea (South), October 27 - November 2, 2019, pages 2950–2958. IEEE, 2019.
  * Guyon and Elisseeff [2003] Isabelle Guyon and André Elisseeff.  An introduction to variable and feature selection.  J. Mach. Learn. Res., 3(null):1157–1182, mar 2003.
  * He et al. [2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  * He et al. [2017] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick.  Mask r-cnn.  In CVPR, pages 2961–2969, 2017.
  * Ignatiev [2020] Alexey Ignatiev.  Towards trustable explainable ai.  In Christian Bessiere, editor, Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI-20, pages 5154–5158. International Joint Conferences on Artificial Intelligence Organization, 7 2020.  Early Career.
  * Krizhevsky et al. [2012] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton.  Imagenet classification with deep convolutional neural networks.  Advances in neural information processing systems, 25, 2012.
  * Li et al. [2023] Jianing Li, Yaowei Wang, and Shiliang Zhang.  Polarpose: Single-stage multi-person pose estimation in polar coordinates.  IEEE Transactions on Image Processing, 32:1108–1119, 2023.
  * Lin et al. [2014] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick.  Microsoft coco: Common objects in context.  In ECCV, pages 740–755. Springer, 2014.
  * Lundberg et al. [2020] Scott M. Lundberg, Gabriel Erion, and Hugh et al. Chen.  From local explanations to global understanding with explainable ai for trees.  Nature Machine Intelligence, 2:56–67, 2020.
  * Newell et al. [2017] Alejandro Newell, Zhiao Huang, and Jia Deng.  Associative embedding: End-to-end learning for joint detection and grouping.  In NeurIPS, pages 2277–2287, 2017.
  * Nie et al. [2019] Xuecheng Nie, Jiashi Feng, Jianfeng Zhang, and Shuicheng Yan.  Single-stage multi-person pose machines.  In ICCV, pages 6951–6960, 2019.
  * Petsiuk et al. [2018] Vitali Petsiuk, Abir Das, and Kate Saenko.  Rise: Randomized input sampling for explanation of black-box models.  arXiv preprint arXiv:1806.07421, 2018.
  * Qiu et al. [2022] Luyu Qiu, Yi Yang, Caleb Chen Cao, Yueyuan Zheng, Hilary Ngai, Janet Hsiao, and Lei Chen.  Generating perturbation-based explanations with robustness to out-of-distribution data.  In Proceedings of the ACM Web Conference 2022, WWW ’22, page 3594–3605, New York, NY, USA, 2022. Association for Computing Machinery.
  * Ren et al. [2015] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun.  Faster r-cnn: Towards real-time object detection with region proposal networks.  In NeurIPS, pages 91–99, 2015.
  * Ribeiro et al. [2016] Marco Túlio Ribeiro, Sameer Singh, and Carlos Guestrin.  ”why should I trust you?”: Explaining the predictions of any classifier.  In Balaji Krishnapuram, Mohak Shah, Alexander J. Smola, Charu C. Aggarwal, Dou Shen, and Rajeev Rastogi, editors, Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, San Francisco, CA, USA, August 13-17, 2016, pages 1135–1144. ACM, 2016\.
  * Sarp et al. [2023] Salih Sarp, Ferhat Ozgur Catak, Murat Kuzlu, Umit Cali, Huseyin Kusetogullari, Yanxiao Zhao, Gungor Ates, and Ozgur Guler.  An xai approach for covid-19 detection using transfer learning with x-ray images.  Heliyon, 9(4):e15137, 2023.
  * Shapley [1951] Lloyd S Shapley.  Notes on the n-person game—ii: The value of an n-person game.  1951\.
  * Sun et al. [2018] Xiao Sun, Bin Xiao, Fangyin Wei, Shuang Liang, and Yichen Wei.  Integral human pose regression.  In ECCV, pages 529–545, 2018.
  * Sun et al. [2019] Ke Sun, Bin Xiao, Dong Liu, and Jingdong Wang.  Deep high-resolution representation learning for human pose estimation.  In CVPR, pages 5693–5703, 2019.
  * Vaswani and others [2017] Ashish Vaswani et al.  Attention is all you need.  Advances in neural information processing systems, 30, 2017.
  * Wei et al. [2016] Shih-En Wei, Varun Ramakrishna, Takeo Kanade, and Yaser Sheikh.  Convolutional pose machines.  In CVPR, pages 4724–4732, 2016.
  * Wei et al. [2020] Fangyun Wei, Xiao Sun, Hongyang Li, Jingdong Wang, and Stephen Lin.  Point-set anchors for object detection, instance segmentation and pose estimation.  arXiv preprint arXiv:2007.02846, 2020.
  * Williamson and Feng [2020] Brian Williamson and Jean Feng.  Efficient nonparametric statistical inference on population feature importance using shapley values.  In Hal Daumé III and Aarti Singh, editors, Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 10282–10291. PMLR, 13–18 Jul 2020.
  * Winter [2002] Eyal Winter.  Chapter 53 the shapley value.  volume 3 of Handbook of Game Theory with Economic Applications, pages 2025–2054. Elsevier, 2002.
  * Yoon et al. [2019] Jinsung Yoon, James Jordon, and Mihaela van der Schaar.  INVASE: Instance-wise variable selection using neural networks.  In International Conference on Learning Representations, 2019.
  * Zeiler and Fergus [2014] Matthew D. Zeiler and Rob Fergus.  Visualizing and understanding convolutional networks.  In David Fleet, Tomas Pajdla, Bernt Schiele, and Tinne Tuytelaars, editors, Computer Vision – ECCV 2014, pages 818–833, Cham, 2014. Springer International Publishing.
  * Zhang et al. [2020] Feng Zhang, Xiatian Zhu, Hanbin Dai, Mao Ye, and Ce Zhu.  Distribution-aware coordinate representation for human pose estimation.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 7093–7102, 2020.
  * Zhong et al. [2020] Zhun Zhong, Liang Zheng, Guoliang Kang, Shaozi Li, and Yi Yang.  Random erasing data augmentation.  In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 13001–13008, 2020.
  * Zhou et al. [2015] Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba.  Object detectors emerge in deep scene cnns, 2015.
  * Zhou et al. [2019] Xingyi Zhou, Dequan Wang, and Philipp Krähenbühl.  Objects as points.  arXiv preprint arXiv:1904.07850, 2019.


2*MRCE($K=3$)     $\btheta$    4.823(0.118)	   0.784(0.024)	   0.322(0.052)    2*0.029(0.023)    2*3(0)

$\bgamma$    8.804(0.619)	   0.617(0.167)    0.070(0.028)

2*MCGGM($K=3$)     $\btheta$
4.459(0.578)	   0.703(0.193)	   0.143(0.112)
2*0.473(0.141)    2*3(0)

$\bgamma$    -    -    -

18*(150,200,250)    2*Proposed     $\btheta$    2.558(0.338)	   0.879(0.028)	   0.113(0.053)    2*0.934(0.136)     2*2.80(0.41)

$\bgamma$    3.482(0.751)	   0.844(0.079)    0.003(0.003)

2*HeteroGGM     $\btheta$    4.820(0.071)	   0.971(0.007)	   0.919(0.011)    2*0.144(0.243)    2*5.20(1.54)

$\bgamma$    -    -    -

2*CGLasso($K=6$)     $\btheta$    4.301(0.045)	   0.702(0.022)	   0.314(0.007)    2*0.029(0.012)    2*6(0)

$\bgamma$    8.350(0.235)	   0.879(0.029)    0.137(0.004)

2*CGLasso($K=4$)     $\btheta$    4.543(0.069)	   0.740(0.020)	   0.286(0.010)    2*0.040(0.029)    2*4(0)

$\bgamma$    7.824(0.411)	   0.937(0.025)    0.109(0.006)

2*CGLasso($K=3$)     $\btheta$    4.663(0.101)	   0.765(0.028)    0.257(0.013)    2*0.047(0.063)    2*3(0)

$\bgamma$    7.974(0.796)	   0.914(0.042)    0.092(0.007)

2*MRCE($K=6$)     $\btheta$    4.723(0.591)	   0.708(0.142)	   0.381(0.081)    2*0.029(0.012)    2*6(0)

$\bgamma$    9.107(0.228)	   0.516(0.117)    0.067(0.019)

2*MRCE($K=4$)     $\btheta$    4.654(0.132)	   0.771(0.038)	   0.363(0.057)    2*0.040(0.029)    2*4(0)

$\bgamma$    8.436(0.515)	   0.655(0.093)    0.082(0.023)

2*MRCE($K=3$)     $\btheta$    4.693(0.176)	   0.792(0.030)	   0.333(0.058)    2*0.047(0.063)    2*3(0)

$\bgamma$    8.352(0.898)	   0.666(0.132)    0.088(0.023)

2*MCGGM($K=3$)     $\btheta$
4.612(0.555)	   0.604(0.199)	   0.098(0.109)	   2*0.473(0.086)    2*3(0)

$\bgamma$    -    -    -

18*(500,500,500)    2*Proposed     $\btheta$    0.930(0.040)	   0.980(0.006)	   0.038(0.003)    2*1.000(0.000)    2*3.00(0.00)

$\bgamma$    0.369(0.062)	   1.000(0.001)    0.001(0.001)

2*HeteroGGM     $\btheta$    4.660(0.021)	   0.980(0.004)    0.895(0.009)    2*0.642(0.047)    2*5.90(0.45)

$\bgamma$    -    -    -

2*CGLasso($K=6$)     $\btheta$    3.894(0.184)	   0.800(0.018)	   0.143(0.018)    2*0.298(0.049)    2*6(0)

$\bgamma$    4.972(0.603)	   0.993(0.008)    0.057(0.019)

2*CGLasso($K=4$)     $\btheta$    3.679(0.303)	   0.822(0.022)	   0.090(0.022)    2*0.476(0.120)    2*4(0)

$\bgamma$    3.890(0.901)	   0.999(0.002)    0.054(0.014)

2*CGLasso($K=3$)     $\btheta$    3.954(0.682)	   0.827(0.033)	   0.089(0.051)    2*0.400(0.303)    2*3(0)

$\bgamma$    4.610(2.453)	   0.999(0.002)    0.052(0.026)

2*MRCE($K=6$)     $\btheta$    3.586(0.257)	   0.872(0.036)	   0.283(0.037)    2*0.298(0.049)    2*6(0)

$\bgamma$    4.854(0.698)	   0.960(0.052)    0.126(0.013)

2*MRCE($K=4$)     $\btheta$    3.387(0.401)	   0.907(0.032)    0.235(0.031)    2*0.476(0.120)    2*4(0)

$\bgamma$    3.910(1.060)	   0.985(0.034)    0.081(0.006)

2*MRCE($K=3$)     $\btheta$    3.766(0.905)	   0.899(0.067)	   0.215(0.044)    2*0.400(0.303)    2*3(0)

$\bgamma$    4.825(2.778)	   0.939(0.108)    0.057(0.004)

2*MCGGM($K=3$)     $\btheta$
4.493(0.544)	   0.665(0.236)	   0.064(0.052)
2*0.493(0.188)    2*3(0)

$\bgamma$    -    -    -

Simulation results under S2 with $p=q=100$. In each cell, mean (sd).

$n$     Method         RMSE    TPR    FRP    RI    $\hat{K}_0$

18*(200,200,200)    2*Proposed     $\btheta$    4.313(0.707)	   0.869(0.021)	   0.092(0.028)    2*0.678(0.191)    2*2.25(0.44)

$\bgamma$    6.810(1.586)	   0.846(0.059)    0.003(0.002)

2*HeteroGGM     $\btheta$    6.661(0.181)	   0.966(0.007)	   0.893(0.012)    2*0.019(0.013)    2*5.15(1.53)

$\bgamma$    -    -    -

2*CGLasso($K=6$)     $\btheta$    5.813(0.045)	   0.723(0.014)	   0.277(0.005)    2*0.025(0.012)    2*6(0)

$\bgamma$    11.951(0.317)	   0.823(0.044)	   0.089(0.002)

2*CGLasso($K=4$)     $\btheta$    6.210(0.038)	   0.771(0.011)	   0.254(0.004)    2*0.030(0.015)    2*4(0)

$\bgamma$    11.419(0.416)	   0.901(0.032)    0.073(0.002)

2*CGLasso($K=3$)     $\btheta$    6.285(0.287)	   0.799(0.011)	   0.218(0.020)    2*0.095(0.127)    2*3(0)

$\bgamma$    10.535(1.519)	   0.941(0.037)    0.056(0.008)

2*MRCE($K=6$)     $\btheta$    6.613(0.076)	   0.756(0.018)    0.363(0.013)    2*0.025(0.012)    2*6(0)

$\bgamma$    13.084(0.098)	   0.333(0.046)    0.025(0.002)

2*MRCE($K=4$)     $\btheta$    6.902(0.836)	   0.747(0.157)    0.321(0.071)    2*0.030(0.015)    2*4(0)

$\bgamma$    12.713(0.296)	   0.357(0.115)    0.023(0.009)

2*MRCE($K=3$)     $\btheta$    6.458(0.518)	   0.815(0.024)    0.289(0.056)    2*0.095(0.127)    2*3(0)

$\bgamma$    11.535(1.958)	   0.583(0.254)    0.041(0.030)

2*MCGGM($K=3$)     $\btheta$
6.746(0.510)	   0.539(0.199)	   0.023(0.027)	   2*0.729(0.362)    2*3(0)

$\bgamma$    -    -    -

18*(150,200,250)    2*Proposed     $\btheta$    4.234(0.401)	   0.865(0.020)	   0.090(0.019)    2*0.723(0.168)    2*2.25(0.44)

$\bgamma$    5.675(0.855)	   0.842(0.091)    0.002(0.001)

2*HeteroGGM     $\btheta$    6.539(0.094)	   0.965(0.006)	   0.898(0.006)    2*0.028(0.013)    2*5.70(0.66)

$\bgamma$    -    -    -

2*CGLasso($K=6$)     $\btheta$    5.781(0.038)	   0.722(0.012)	   0.274(0.003)    2*0.023(0.011)    2*6(0)

$\bgamma$    11.656(0.250)	   0.833(0.026)    0.088(0.002)

2*CGLasso($K=4$)     $\btheta$    6.154(0.122)	   0.772(0.009)	   0.250(0.008)    2*0.036(0.044)    2*4(0)

$\bgamma$    10.815(0.722)	   0.919(0.027)    0.071(0.004)

2*CGLasso($K=3$)     $\btheta$    6.058(0.566)	   0.805(0.021)	   0.204(0.033)    2*0.161(0.210)    2*3(0)

$\bgamma$    10.017(2.068)	   0.903(0.056)    0.051(0.011)

2*MRCE($K=6$)     $\btheta$    6.800(0.963)	   0.717(0.169)    0.337(0.081)    2*0.023(0.011)    2*6(0)

$\bgamma$    12.824(0.125)	   0.366(0.095)    0.024(0.006)

2*MRCE($K=4$)     $\btheta$    6.640(0.508)	   0.772(0.085)    0.311(0.049)    2*0.036(0.044)    2*4(0)

$\bgamma$    12.111(0.854)	   0.495(0.120)    0.030(0.016)

2*MRCE($K=3$)     $\btheta$    6.085(0.823)	   0.825(0.036)    0.269(0.037)    2*0.161(0.210)    2*3(0)

$\bgamma$    10.503(2.385)	   0.676(0.204)    0.057(0.028)

2*MCGGM($K=3$)     $\btheta$    6.528(0.628)	   0.700(0.144)	   0.088(0.075)
2*0.633(0.132)    2*3(0)

$\bgamma$    -    -    -

18*(500,500,500)    2*Proposed     $\btheta$    1.432(0.048)	   0.958(0.011)	   0.024(0.011)    2*1.000(0.000)    2*3.00(0.00)

$\bgamma$    0.575(0.178)	   0.999(0.002)    0.001(0.001)

2*HeteroGGM     $\btheta$    6.393(0.046)	   0.981(0.006)	   0.859(0.009)    2*0.707(0.120)    2*5.25(1.02)

$\bgamma$    -    -    -

2*CGLasso($K=6$)     $\btheta$    4.766(0.654)	   0.836(0.012)	   0.117(0.029)    2*0.440(0.152)    2*6(0)

$\bgamma$    6.190(1.667)	   0.992(0.012)    0.045(0.018)

2*CGLasso($K=4$)     $\btheta$    3.541(0.279)	   0.892(0.026)	   0.055(0.021)    2*0.794(0.018)    2*4(0)

$\bgamma$    3.156(0.556)	   0.999(0.009)    0.023(0.008)

2*CGLasso($K=3$)     $\btheta$    2.799(0.130)	   0.940(0.013)    0.025(0.005)    2*0.956(0.007)    2*3(0)

$\bgamma$    1.916(0.042)	   1.000(0.000)    0.011(0.001)

2*MRCE($K=6$)     $\btheta$
4.424(0.766)	   0.884(0.025)	   0.222(0.031)    2*0.440(0.152)    2*6(0)

$\bgamma$    6.336(1.803)	   0.932(0.082)    0.070(0.011)

2*MRCE($K=4$)     $\btheta$    2.908(0.169)	   0.952(0.014)	   0.162(0.019)    2*0.794(0.018)    2*4(0)

$\bgamma$    2.850(0.542)	   0.999(0.004)    0.058(0.008)

2*MRCE($K=3$)     $\btheta$    2.242(0.102)	   0.971(0.008)    0.089(0.015)    2*0.956(0.007)    2*3(0)

$\bgamma$    1.520(0.043)	   1.000(0.000)    0.053(0.006)

2*MCGGM($K=3$)     $\btheta$    6.303(0.641)	   0.676(0.230)	   0.042(0.039)	   2*0.787(0.125)    2*3(0)

$\bgamma$    -    -    -

Simulation results under S3 with $p=q=50$. In each cell, mean (sd).

$n$     Method         RMSE    TPR    FRP    RI    $\hat{K}_0$

18*(200,200,200)    2*Proposed     $\btheta$    1.728(1.083)	   0.938(0.034)	   0.048(0.021)    2*0.994(0.029)    2*3.05(0.22)

$\bgamma$    1.241(1.336)	   0.983(0.027)	   0.009(0.017)

2*HeteroGGM     $\btheta$    4.131(0.171)	   0.983(0.010)	   0.900(0.012)    2*0.473(0.244)    2*5.15(1.38)

$\bgamma$    -    -    -

2*CGLasso($K=6$)     $\btheta$    3.496(0.181)	   0.869(0.018)	   0.249(0.018)    2*0.387(0.089)    2*6(0)

$\bgamma$    5.589(0.541)	   0.964(0.034)	   0.102(0.006)

2*CGLasso($K=4$)     $\btheta$    3.284(0.292)	   0.908(0.018)	   0.184(0.025)    2*0.572(0.076)    2*4(0)

$\bgamma$    4.013(0.676)	   0.992(0.007)	   0.064(0.009)

2*CGLasso($K=3$)     $\btheta$    2.982(0.429)	   0.919(0.021)	   0.112(0.038)    2*0.738(0.145)    2*3(0)

$\bgamma$    2.853(1.138)	   0.998(0.004)	   0.037(0.012)

2*MRCE($K=6$)     $\btheta$
3.929(0.640)	   0.867(0.149)	   0.353(0.063)    2*0.387(0.089)    2*6(0)

$\bgamma$    6.463(0.697)	   0.784(0.153)	   0.116(0.030)

2*MRCE($K=4$)     $\btheta$    3.035(0.320)	   0.959(0.013)	   0.293(0.044)    2*0.572(0.076)    2*4(0)

$\bgamma$    4.066(0.717)	   0.977(0.028)	   0.148(0.022)

2*MRCE($K=3$)     $\btheta$    2.625(0.519)	   0.979(0.013)	   0.238(0.033)    2*0.738(0.145)    2*3(0)

$\bgamma$    2.640(1.222)	   0.997(0.005)	   0.120(0.013)

2*MCGGM($K=3$)     $\btheta$     4.654(1.275)	   0.742(0.250)	   0.141(0.115)	   2*0.528(0.219)    2*3(0)

$\bgamma$    -    -    -

18*(150,200,250)    2*Proposed     $\btheta$    1.399(0.054)	   0.949(0.018)	   0.053(0.004)    2*1.000(0.000)     2*3.00(0.00)

$\bgamma$    0.636(0.173)	   0.998(0.005)	   0.005(0.002)

2*HeteroGGM     $\btheta$    4.142(0.143)	   0.988(0.007)	   0.905(0.013)    2*0.467(0.270)    2*4.55(1.64)

$\bgamma$    -    -    -

2*CGLasso($K=6$)     $\btheta$    3.432(0.147)	   0.899(0.020)	   0.237(0.016)    2*0.337(0.067)    2*6(0)

$\bgamma$    5.204(0.500)	   0.972(0.024)	   0.094(0.007)

2*CGLasso($K=4$)     $\btheta$    3.196(0.233)	   0.922(0.018)	   0.173(0.026)    2*0.541(0.089)    2*4(0)

$\bgamma$    3.741(0.784)	   0.995(0.008)	   0.058(0.009)

2*CGLasso($K=3$)     $\btheta$    3.164(0.542)	   0.927(0.028)	   0.134(0.050)    2*0.638(0.226)    2*3(0)

$\bgamma$    3.440(1.657)	   0.996(0.008)	   0.044(0.016)

2*MRCE($K=6$)     $\btheta$
3.814(0.686)	   0.927(0.019)	   0.342(0.035)    2*0.337(0.067)    2*6(0)

$\bgamma$    5.767(0.563)	   0.884(0.054)	   0.146(0.029)

2*MRCE($K=4$)     $\btheta$    2.921(0.284)	   0.969(0.011)	   0.287(0.029)    2*0.541(0.089)    2*4(0)

$\bgamma$    3.737(0.860)	   0.977(0.049)	   0.150(0.015)

2*MRCE($K=3$)     $\btheta$    2.807(0.653)	   0.970(0.013)	   0.277(0.035)    2*0.638(0.226)    2*3(0)

$\bgamma$    3.278(1.770)	   0.996(0.009)	   0.133(0.017)

2*MCGGM($K=3$)     $\btheta$    4.515(0.381)	   0.714(0.150)	   0.064(0.061)
2*0.589(0.112)    2*3(0)

$\bgamma$    -    -    -

18*(500,500,500)    2*Proposed     $\btheta$    0.755(0.035)	   0.997(0.005)	   0.032(0.007)    2*1.000(0.000)     2*3.00(0.00)

$\bgamma$    0.324(0.021)	   1.000(0.000)	   0.001(0.000)

2*HeteroGGM     $\btheta$    4.055(0.096)	   0.990(0.005)	   0.881(0.008)    2*0.669(0.085)    2*5.80(0.70)

$\bgamma$    -    -    -

2*CGLasso($K=6$)     $\btheta$    2.802(0.175)	   0.922(0.019)	   0.080(0.021)    2*0.517(0.041)    2*6(0)

$\bgamma$    2.951(0.404)	   0.999(0.003)	   0.043(0.016)

2*CGLasso($K=4$)     $\btheta$    2.581(0.173)	   0.958(0.024)	   0.065(0.026)    2*0.751(0.014)    2*4(0)

$\bgamma$    2.243(0.236)	   1.000(0.000)	   0.037(0.018)

2*CGLasso($K=3$)     $\btheta$    2.315(0.153)	   0.970(0.026)	   0.039(0.015)    2*0.877(0.015)    2*3(0)

$\bgamma$    1.555(0.052)	   1.000(0.000)	   0.022(0.001)

2*MRCE($K=6$)     $\btheta$
2.331(0.162)	   0.982(0.009)	   0.226(0.031)    2*0.517(0.041)    2*6(0)

$\bgamma$    2.726(0.382)	   0.991(0.029)	   0.113(0.012)

2*MRCE($K=4$)     $\btheta$    2.080(0.083)	   0.997(0.004)	   0.201(0.025)    2*0.751(0.014)    2*4(0)

$\bgamma$    1.950(0.117)	   1.000(0.000)	   0.093(0.012)

2*MRCE($K=3$)     $\btheta$    1.884(0.099)	   0.996(0.007)	   0.136(0.009)    2*0.877(0.015)    2*3(0)

$\bgamma$    1.299(0.067)	   1.000(0.000)	   0.060(0.003)

2*MCGGM($K=3$)     $\btheta$    4.220(0.432)	   0.825(0.206)	   0.070(0.046)	   2*0.487(0.180)    2*3(0)

$\bgamma$    -    -    -

Simulation results under S3 with $p=q=100$. In each cell, mean (sd).

$n$     Method         RMSE    TPR    FRP    RI    $\hat{K}_0$

18*(200,200,200)    2*Proposed     $\btheta$    3.901(0.844)	   0.952(0.024)	   0.081(0.014)    2*0.721(0.210)    2*2.35(0.49)

$\bgamma$    5.865(1.936)	   0.885(0.070)	   0.003(0.002)

2*HeteroGGM     $\btheta$    6.175(0.570)	   0.979(0.007)	   0.895(0.013)    2*0.024(0.016)    2*5.30(1.34)

$\bgamma$    -    -    -

2*CGLasso($K=6$)     $\btheta$    5.412(0.178)	   0.827(0.016)	   0.272(0.019)    2*0.126(0.159)    2*6(0)

$\bgamma$    11.072(0.954)	   0.826(0.090)	   0.083(0.008)

2*CGLasso($K=4$)     $\btheta$    4.636(0.984)	   0.917(0.024)	   0.176(0.066)    2*0.490(0.334)    2*4(0)

$\bgamma$    6.655(3.210)	   0.976(0.043)	   0.049(0.019)

2*CGLasso($K=3$)     $\btheta$    3.845(1.242)	   0.926(0.018)	   0.088(0.070)    2*0.775(0.355)    2*3(0)

$\bgamma$    4.270(3.336)	   0.983(0.044)	   0.025(0.017)

2*MRCE($K=6$)     $\btheta$
5.933(0.538)	   0.838(0.099)	   0.359(0.036)    2*0.126(0.159)    2*6(0)

$\bgamma$    12.153(0.608)	   0.267(0.068)	   0.026(0.006)

2*MRCE($K=4$)     $\btheta$    4.806(0.981)	   0.947(0.034)	   0.278(0.081)    2*0.490(0.334)    2*4(0)

$\bgamma$    7.751(3.422)	   0.716(0.285)	   0.042(0.022)

2*MRCE($K=3$)     $\btheta$    3.635(1.485)	   0.962(0.026)	   0.182(0.092)    2*0.775(0.355)    2*3(0)

$\bgamma$    4.597(3.837)	   0.903(0.196)	   0.040(0.018)

2*MCGGM($K=3$)     $\btheta$    6.464(0.863)	   0.633(0.269)	   0.067(0.094)	   2*0.869(0.128)    2*3(0)

$\bgamma$    -    -    -

18*(150,200,250)    2*Proposed     $\btheta$    3.515(1.097)	   0.948(0.017)	   0.076(0.017)    2*0.809(0.203)     2*2.50(0.51)

$\bgamma$    4.434(2.427)	   0.905(0.115)

2*HeteroGGM     $\btheta$    5.977(0.172)	   0.985(0.007)	   0.897(0.011)    2*0.020(0.015)    2*5.40(1.27)

$\bgamma$    -    -    -

2*CGLasso($K=6$)     $\btheta$    5.782(0.217)	   0.856(0.012)	   0.283(0.007)    2*0.027(0.024)    2*6(0)

$\bgamma$    11.572(0.570)	   0.822(0.064)	   0.088(0.004)

2*CGLasso($K=4$)     $\btheta$    4.759(1.327)	   0.936(0.015)	   0.173(0.072)    2*0.441(0.323)    2*4(0)

$\bgamma$    6.592(3.503)	   0.974(0.041)	   0.047(0.021)

2*CGLasso($K=3$)     $\btheta$    3.760(0.918)	   0.935(0.024)	   0.087(0.053)    2*0.790(0.276)    2*3(0)

$\bgamma$    4.368(2.994)	   0.959(0.100)	   0.024(0.011)

2*MRCE($K=6$)     $\btheta$
6.036(0.160)	   0.875(0.010)	   0.371(0.022)    2*0.027(0.024)    2*6(0)

$\bgamma$    12.669(0.412)	   0.217(0.108)	   0.025(0.007)

2*MRCE($K=4$)     $\btheta$    4.983(1.724)	   0.910(0.215)	   0.252(0.096)    2*0.441(0.323)    2*4(0)

$\bgamma$    7.689(3.910)	   0.675(0.364)	   0.039(0.021)

2*MRCE($K=3$)     $\btheta$    3.620(0.991)	   0.972(0.011)	   0.186(0.063)    2*0.790(0.276)    2*3(0)

$\bgamma$    4.493(3.227)	   0.926(0.152)	   0.050(0.019)

2*MCGGM($K=3$)     $\btheta$    6.876(0.609)	   0.512(0.177)	   0.023(0.033)	   2*0.787(0.087)    2*3(0)

$\bgamma$    -    -    -

18*(500,500,500)    2*Proposed     $\btheta$    1.177(0.037)	   0.995(0.003)	   0.017(0.008)    2*1.000(0.000)     2*3.00(0.00)

$\bgamma$    0.460(0.015)	   1.000(0.000)	   0.000(0.001)

2*HeteroGGM     $\btheta$    5.851(0.230)	   0.991(0.006)	   0.855(0.009)    2*0.728(0.134)    2*5.35(1.18)

$\bgamma$    -    -    -

2*CGLasso($K=6$)     $\btheta$    3.618(0.330)	   0.944(0.029)	   0.095(0.031)    2*0.651(0.038)    2*6(0)

$\bgamma$    3.900(0.628)	   0.999(0.003)	   0.038(0.017)

2*CGLasso($K=4$)     $\btheta$    2.962(0.390)	   0.961(0.025)	   0.049(0.019)    2*0.859(0.016)    2*4(0)

$\bgamma$    2.546(0.623)	   1.000(0.001)	   0.021(0.005)

2*CGLasso($K=3$)     $\btheta$    2.755(0.416)	   0.964(0.026)	   0.017(0.014)    2*0.956(0.125)    2*3(0)

$\bgamma$    2.190(1.781)	   0.984(0.074)	   0.012(0.002)

2*MRCE($K=6$)     $\btheta$
3.376(0.473)	   0.974(0.039)	   0.157(0.032)    2*0.651(0.038)    2*6(0)

$\bgamma$    4.122(0.865)	   0.957(0.070)	   0.063(0.018)

2*MRCE($K=4$)     $\btheta$    2.487(0.224)	   0.993(0.017)	   0.083(0.026)    2*0.859(0.016)    2*4(0)

$\bgamma$    2.270(0.563)	   0.993(0.024)	   0.054(0.010)

2*MRCE($K=3$)     $\btheta$    2.147(0.428)	   0.998(0.002)    0.050(0.017)    2*0.956(0.125)    2*3(0)

$\bgamma$    1.810(1.871)	   0.984(0.070)	   0.044(0.004)

2*MCGGM($K=3$)     $\btheta$    5.771(0.457)	   0.924(0.092)	   0.068(0.034)	   2*0.716(0.132)    2*3(0)

$\bgamma$    -    -    -


# Tidal capture of stars by supermassive black holes: implications for
periodic nuclear transients and quasi-periodic eruptions

M. Cufari1, C. J. Nixon2,3, and Eric R. Coughlin1
1 Department of Physics, Syracuse University, Syracuse, NY 13244, USA
2 School of Physics and Astronomy, University of Leicester, Leicester LE1 7RH,
UK
3 School of Physics and Astronomy, University of Leeds, Leeds LS2 9JT, UK
<EMAIL_ADDRESS>
(Accepted XXX. Received YYY; in original form ZZZ)

###### Abstract

Stars that plunge into the center of a galaxy are tidally perturbed by a
supermassive black hole (SMBH), with closer encounters resulting in larger
perturbations. Exciting these tides comes at the expense of the star’s orbital
energy, which leads to the naive conclusion that a smaller pericenter (i.e., a
closer encounter between the star and SMBH) always yields a more tightly bound
star to the SMBH. However, once the pericenter distance is small enough that
the star is partially disrupted, morphological asymmetries in the mass lost by
the star can yield an _increase_ in the orbital energy of the surviving core,
resulting in its ejection – not capture – by the SMBH. Using smoothed-particle
hydrodynamics simulations, we show that the combination of these two effects –
tidal excitation and asymmetric mass loss – result in a maximum amount of
energy lost through tides of $\sim 2.5\%$ of the binding energy of the star,
which is significantly smaller than the theoretical maximum of the total
stellar binding energy. This result implies that stars that are repeatedly
partially disrupted by SMBHs many ($\gtrsim 10$) times on short-period orbits
($\lesssim$ few years), as has been invoked to explain the periodic nuclear
transient ASASSN-14ko and quasi-periodic eruptions, must be bound to the SMBH
through a mechanism other than tidal capture, such as a dynamical exchange
(i.e., Hills capture).

###### keywords:

hydrodynamics — black holes — galaxies: nuclei

††pubyear: 2022††pagerange: Tidal capture of stars by supermassive black
holes: implications for periodic nuclear transients and quasi-periodic
eruptions–References

##  1 Introduction

Binary stars may form when two field (i.e., individual) stars pass close to
one another on nearly parabolic trajectories. As the stars pass, tidal
oscillatory modes are excited in the stars at the expense of the stars’
orbital energy. If the encounter is sufficiently close, the tides dissipate
enough orbital energy to bind the stars to one another (Fabian et al., 1975;
Press & Teukolsky, 1977).

A star may also be scattered into the region of parameter space, known as the
“loss-cone,” that brings the star’s distance of closest approach very near a
supermassive black hole in the nucleus of a galaxy (SMBH; Frank & Rees, 1976;
Lightman & Shapiro, 1977; Cohn & Kulsrud, 1978). Reasoning analogously to the
binary formation scenario, one is tempted to conclude that stellar orbits with
distances of closest approach nearer the black hole are more strongly tidally
perturbed by, and thus more tightly bound to, the SMBH. However, there is a
fundamental upper limit as to how efficient this process can be, because tides
cannot transfer more energy into oscillations than the binding energy of the
star itself. Additionally, as the pericenter distance of the star continues to
decrease the perturbative tidal limit breaks down, and the star starts to lose
a fraction of its outer envelope (known as a partial Tidal Disruption Event;
TDE). Recent simulations have demonstrated that the asymmetric ejection of
tidal debris in a partial TDE results in the _unbinding_ of the surviving
stellar core from the SMBH (Manukian et al., 2013; Gafton et al., 2015).
Therefore, the maximum binding energy a star can achieve through tidal
interactions with an SMBH could be substantially smaller than the theoretical
limit of the star’s own binding energy. This maximum binding energy generated
through tidal dissipation sets a fundamental timescale over which one would
expect repeating partial TDEs – a star that is bound to a SMBH that is
partially destroyed on each pericenter passage (e.g., Zalamea et al. 2010;
Campana et al. 2015; Miniutti et al. 2019; Payne et al. 2021) – to recur if
the bound star is supplied through tidal dissipation.

In this paper we present the results of smoothed particle hydrodynamics (SPH)
simulations of partial TDEs from which we obtain the maximum binding energy.
In Section 2.1 we describe the setup of the simulations, and in Section 2.2 we
present the results. In Section 3 we discuss the implications of our findings
in the context of periodic nuclear transients (PNTs) and quasi-periodic
eruptions (QPEs). We summarize and conclude in Section 4.

##  2 Simulations

###  2.1 Methodology

We performed simulations of partial TDEs with the SPH code phantom (Price et
al., 2018). The star was modeled as a polytrope with a $\gamma=5/3$, adiabatic
equation of state, with mass $M_{\star}=1M_{\odot}$ and radius
$R_{\star}=1R_{\odot}$. In each simulation the star was injected from a
distance of $12~{}r_{t}$, where $r_{\rm
t}=R_{\star}\left(M_{\bullet}/M_{\star}\right)^{1/3}$, from the SMBH of mass
$M_{\bullet}=10^{6}M_{\odot}$ with the center of mass on a parabolic
trajectory. The impact parameter $\beta\equiv r_{\rm{t}}/r_{\rm{p}}$, where
$r_{\rm p}$ is the pericenter distance of the star, was varied between
$\beta=0.4$ and $\beta=0.8$ in steps of $\Delta\beta=0.02$. Additional details
of the simulation setup can be found in Coughlin & Nixon (2015), and the
algorithms for, e.g., self-gravity can be found in Price et al. (2018).

We performed simulations with $1.25\times 10^{5}$, $10^{6}$ particles and
$8\times 10^{6}$ particles, and found very little deviation in the outcome.
All results presented here are from the $8\times 10^{6}$ particle runs. For
the purposes of determining the properties of the surviving star, we calculate
averages using only those particles that have a density greater than 1% of the
maximum global density within the simulation, $\rho_{\rm max}$; we define this
subset of particles as the core. For example, the location of the core is
calculated as the the average position of the subset of particles that satisfy
this criterion, and the distance from the SMBH, $r_{\rm c}$, is determined
from the Euclidean norm of the position. The analogous statement applies to
the core velocity and speed, $v_{\rm c}$. Over the range of $\beta$ we
simulated this criterion excludes the tidally stripped tails, and reducing the
fraction to $0.1$% does not impact the results because the tidal tails have a
substantially lower density than $\sim 1$% of $\rho_{\rm max}$. Similarly,
increasing the fraction to $10$% has a negligible effect on the results.
However, values $>10\%$ can lead to significant noise in the results because
in this case there are too few particles that contribute to the average.

We run each simulation until the core energy has settled to a constant value
that we measure for our results. For disruptions with $\beta<0.6$, this time
is $\lesssim 1$ day after pericenter passage. Simulations with $\beta>0.6$ are
run up to $\sim 5$ days post-pericenter to ensure that the energy has
converged to a constant value. Running our simulations to later times has no
affect on our results.

###  2.2 Results

The left panel of Figure 1 shows the specific orbital energy of the core,
calculated according to

$\epsilon_{\rm
c}=\frac{1}{2}{v_{\rm{c}}^{2}}-\frac{GM_{\bullet}}{r_{\rm{c}}}.$ (1)

For $0.4\lesssim\beta\lesssim 0.62$, the orbital energy is negative, implying
that the surviving core is bound to the SMBH. The global minimum energy is
$\sim 2-3\%$ of the binding energy of the star, i.e.,

$\epsilon_{\rm c,min}\simeq-0.025\frac{GM_{\star}}{R_{\star}}.$ (2)

The fact that the star becomes bound to the SMBH following its tidal
interaction is in agreement with the classical calculations of tidal
dissipation by Fabian et al. (1975); Press & Teukolsky (1977).

Conversely, all disruptions with $\beta\gtrsim 0.62$ result in a core with a
net positive energy following the interaction with the SMBH, indicating that
the star is on an unbound trajectory. This result agrees with those of
Manukian et al. (2013); Gafton et al. (2015), who found that the “kick” to the
star could result in a velocity that is $\sim$ the escape speed of the star
for $\beta$ very close to the value at which complete disruption occurs
($\beta\simeq 0.9$ for a 5/3 polytrope; Guillochon & Ramirez-Ruiz 2013;
Mainetti et al. 2017). Therefore, if a star reaches a pericenter distance with
$0.62\lesssim\beta\lesssim 0.9$, the star is not tidally captured but is
instead tidally ejected, never to return to pericenter.

Figure 1: Left: The energy imparted to the star as a function of the impact
parameter $\beta$. Stars with $\epsilon_{\rm c}<0$ are bound to the black
hole. There is a small range of $\beta$ for which mass loss occurs on first
pericenter passage (i.e., $\beta>0.5$) and for which $\epsilon_{\rm c}<0$, the
necessary condition for a repeating partial disruption to occur. Right: the
period of the orbit of the star for encounters that yield a captured star as a
function of the impact parameter $\beta$. The minimum period corresponds
minimum orbital energy (i.e. the maximum binding energy), and for these
parameters is $>10^{4}$ yrs.

With the energy-semimajor axis and energy-period relationships for a Keplerian
orbit, we find that the semimajor axis of the captured star (for
$\beta\lesssim 0.62$, i.e., for stars that are not ejected) is

$a_{\rm c}=-\frac{GM_{\bullet}}{2\epsilon_{\rm c}}=\frac{1}{\eta_{\rm
c}}\frac{R_{\star}}{2}\left(\frac{M_{\bullet}}{M_{\star}}\right),$ (3)

while the orbital period is

$T_{\rm c}=\frac{2\pi GM_{\bullet}}{\left(-2\epsilon_{\rm
c}\right)^{3/2}}=\frac{1}{\eta_{\rm
c}^{3/2}}\left(\frac{R_{\star}}{2}\right)^{3/2}\frac{2\pi}{\sqrt{GM_{\star}}}\left(\frac{M_{\bullet}}{M_{\star}}\right).$
(4)

Here we defined the specific orbital energy of the captured star as
$\epsilon_{\rm c}=-\eta_{\rm c}GM_{\star}/R_{\star}$, where $\eta_{\rm
c}\lesssim 0.025$ (from Figure 1). Using $M_{\bullet}/M_{\star}=10^{6}$ in
Equation (3), the minimum apocenter distance of the star (with $\eta_{\rm
c}=0.025$) is $\sim 1$ pc for $R_{\star}=1R_{\odot}$.

The right panel of Figure 1 shows the orbital period of the captured star as a
function of $\beta$ from the simulations and using Equation (4). We see that
the minimum orbital period of the captured star is $\sim 3\times 10^{4}$ yr.
However, because of the strong dependence on $\eta_{\rm c}$ in Equation (4),
the period can be much larger for only small changes in $\beta$.

##  3 discussion

Some current models for PNTs and QPEs suggest that they can be produced by
stars on bound orbits about SMBHs with periods from hours to $\sim few$ years,
with the star being partially disrupted and feeding an accretion flare every
pericenter passage (Miniutti et al., 2019; King, 2020; Payne et al., 2021;
King, 2022; Wevers et al., 2022). PNTs have been suggested to have system
parameters that are comparable to $M_{\bullet}=10^{7}M_{\odot}$,
$R_{\star}=1R_{\odot}$, $M_{\star}=1M_{\odot}$, which gives, from Equation (4)
with $\eta_{\rm c}=0.025$,

$T_{\rm PNT}\simeq 3\times 10^{5}\textrm{ yr},$ (5)

which is obviously too long to explain the observed periods $\lesssim 1$ year
(Payne et al., 2021; Wevers et al., 2022). Even if one could inject the
theoretical maximum energy into the star, and thus set $\eta_{\rm c}=1$ in
Equation (4), we would obtain $T_{\rm PNT}\simeq 10^{3}$ yr. As argued by
Cufari et al. (2022), even in this overly optimistic scenario, PNTs need an
additional mechanism to bind the star sufficiently tightly to the SMBH to
produce their observed periods in situ.

On the other hand, it is possible that a PNT could _initially_ be generated
with an orbital period given by Equation (5), but the period then decays
through gravitational-wave emission (over millions of years) to produce a
period as short as $\sim 1$ year by the time that we observe it111As pointed
out in Payne et al. (2021), the rate of change of the orbital period due to
gravitational waves is actually too small to explain the observed value for
ASASSN-14ko, but it should be possible for gravitational waves to shrink the
orbit in general and for other systems.. However, a problem with this
interpretation is that even for the maximum value of $\eta_{\rm c}=0.025$,
Equation (3) for the semimajor axis of the orbit yields an apocenter distance
of $\sim 2a_{\rm c}\sim 10$ pc for $M_{\bullet}/M_{\star}=10^{7}$. Thus, we
would expect the star to interact with, and be once again perturbed by, the
nuclear star cluster, and it seems very unlikely that the star will repeatedly
return to the same pericenter over multiple passages.

Additionally, while we have assumed that the star is on a parabolic orbit, it
could have a velocity at infinity that is $v_{\infty}\simeq\sigma$, with
$\sigma$ the galactic velocity dispersion (Miller et al., 2005). From Figure
1, the maximum amount of energy able to be dissipated through tides is

$\epsilon_{\rm c,max}\simeq
0.025\frac{GM_{\star}}{R_{\star}}\simeq\frac{1}{2}\left(\frac{M_{\star}}{M_{\odot}}\right)\left(\frac{R_{\star}}{R_{\odot}}\right)^{-1}\left(100\textrm{
km s}^{-1}\right)^{2}.$ (6)

For a $10^{7}M_{\odot}$ SMBH, the velocity dispersion from the $M-\sigma$
relation is $\sigma\sim 110$ km s-1(Marsden et al., 2020). Thus, tides may not
actually be capable of dissipating the true (positive) energy of the orbit,
and hence binding the star through this mechanism may be impossible in the
first place.

QPEs – with periods of the order hours – are typically modeled with a $\sim
10^{5}M_{\odot}$ SMBH partially disrupting a white dwarf star. For such a
system, Equation (4) with $M_{\star}=0.6M_{\odot}$, $R_{\star}=0.011R_{\odot}$
(Nauenberg, 1972), $M_{\bullet}=4\times 10^{5}M_{\odot}$, and $\eta_{\rm
c}=0.025$ gives

$T_{\rm QPE}\simeq 28\textrm{ yr},$ (7)

which shows that it is not possible to bind a star to a SMBH through tidal
dissipation and immediately reproduce the observed orbital periods of QPEs. As
for PNTs, it is nonetheless possible that gravitational-wave emission shrinks
the orbit222The orbital period may also be reduced through the interaction
with a pre-existing AGN disc (Syer et al., 1991; Cufari et al., 2022; Lu &
Quataert, 2022). to $\sim$ hours before we detect them. Because the binding
energy of a white dwarf is substantially larger than that of a main sequence
star, the minimum apocenter distance is correspondingly smaller (from Equation
3), and the maximum imparted energy through tides is substantially larger than
the energy at infinity. Thus, without further investigation that is outside
the scope of the present work, we cannot conclusively state whether or not an
additional mechanism is required for producing the observed orbital periods in
QPEs (under the paradigm that they are powered by repeatedly partially
disrupted white dwarfs).

On the other hand, the tidal breakup of a binary star system (i.e., Hills
capture; Hills 1975) can bind the star with a substantially shorter period
(compared to just tidal dissipation) if the binary is sufficiently tight. As
demonstrated by Cufari et al. (2022), this is a plausible explanation for the
origin of the $\sim 114$ day period of ASASSN-14ko. An additional argument
against the gravitational-wave inspiral scenario for explaining ASASSN-14ko is
that the mass lost by the star must be $\sim 1$% of the mass of the star to
power the emission (Cufari et al., 2022), and hence it cannot have survived
many ($\gtrsim 100$ s) interactions prior to the $\sim 10$ that have been
observed since the initial detection. Similarly, if the white dwarf binary
separation is not much larger than the radius of the white dwarf itself, then
Equation (5) from Cufari et al. (2022) with $a_{\star}=0.04R_{\odot}$,
$M_{\bullet}=4\times 10^{5}M_{\odot}$, and $M_{\star}=0.6M_{\odot}$ yields an
orbital period of $\sim 8.3$ hours for the period of the bound star following
a Hills-capture event. As also suggested in Cufari et al. (2022), it therefore
seems plausible that the periods of QPEs can be generated with a dynamical
exchange process without the need for additional dissipation through other
means.

##  4 Summary and Conclusions

We presented SPH simulations of the interaction between a star and an SMBH to
determine the maximum degree by which a star can be bound to an SMBH through
tidal dissipation. Two competing mechanisms prevent the star from becoming
arbitrarily bound to the SMBH. As the distance of closest approach between the
star and SMBH shrinks, energy from the star’s orbit is expended in exciting
dynamical tides in the star. However, once the star reaches a distance of
closest approach comparable to its tidal radius, the star is kicked to
positive energies as a result of asymmetry in the tidal tails that are
liberated from the star. The competition between these two physical effects
results in a minimum possible orbital energy of the star following the tidal
encounter. For the parameters we simulated here, the location of this minimum
is at $\beta\sim 0.55$ (pericenter distance of $r_{\rm t}/0.55$ with $r_{\rm
t}$ the usual tidal radius) and the binding energy of the orbit is $\sim
2.5\%$ of the star’s binding energy; see Equation (2) specifically. This
minimum energy is significantly smaller than the theoretical maximum, being
the entirety of the stellar binding energy.

In order to produce a repeating partial tidal disruption via a tidal
dissipation mechanism, the energy kick imparted to the star must be negative,
otherwise the star will be ejected on a hyperbolic trajectory. Hence, it would
appear that there is a relatively small region of parameter space within which
the star is only partially destroyed, not ejected, and survives for many
($\gtrsim 10$) pericenter passages, specifically $0.4\lesssim\beta\lesssim
0.5$ for the type of star considered here. Initially non-rotating stars in
this range of $\beta$ will be rotating at a nontrivial fraction of breakup
following the initial interaction, which will move the effective tidal radius
out (Golightly et al., 2019), but provided that the pericenter (effectively
unaltered because of the small ratio of the maximal angular momentum of the
star to the angular momentum of the orbit itself) is still within this tidal
radius, the star may transfer a small amount of mass during each additional
pericenter passage. In this manner, a star may undergo many cycles of partial
disruptions before being destroyed or ejected.

In our simulations we modeled the star as a 5/3 polytrope, which is most
applicable to low-mass main sequence stars and and low-mass white dwarfs. By
number, most stars are thought to fall into this regime. However, more massive
(radiative) stars are considerably more centrally concentrated than predicted
by a 5/3-polytropic model. Here we have shown that the minimum energy for the
captured star – modeled as a 5/3 polytrope – occurs at $\beta\approx 0.55$.
This result will depend somewhat on the type of star being considered. For
example, Figure 3 of Manukian et al. (2013) shows that it occurs for $\beta<1$
when the star is modeled as a 4/3-polytrope and that there is very little
dependence of the result on the black hole mass. Faber et al. (2005)
considered the tidal capture of a planet by a star in which the mass ratio was
$q=0.001$, and found a minimum binding energy of $\sim 14\%$ of the binding
energy of the planet at $\beta=10/19\simeq 0.523$ (see their Table 1). Kremer
et al. (2022) also recently considered black hole-star systems with mass
ratios closer to unity, and found a similar effect to the one described here
if the mass ratio was $0.02$ or $0.05$ if the star was modeled as a
$\gamma=5/3$ polytrope, but that the star was able to go from bound to
completely disrupted – without being ejected – as the mass ratio increased
beyond 0.05 and the star was modeled with the Eddington standard model (see
their Figure 1). We defer an analysis of the minimum orbital energy – and the
$\beta$ at which the minimum energy occurs – as a function of the mass ratio
and the type of star to future work.

In the context of extreme mass-ratio inspirals, Zalamea et al. (2010)
demonstrated that a white dwarf (or other compact object) completes thousands
of large-eccentricity orbits before reaching the direct capture radius of the
SMBH. However, their model accounts only for the orbital decay due to
gravitational wave emission and omits tidal dissipation and mass loss
asymmetry effects. Likewise, our simulations omit the effects of orbital decay
due to gravitational wave emission. The pericenter distance of our simulated
disruptions is $>50~{}r_{\rm{g}}$, so orbital decay due to general
relativistic effects (at least on the first pericenter passage) is negligible.
For more compact stars with smaller tidal radii nearer the event horizon,
orbital decay due to general relativistic effects will be more significant
over fewer orbital periods (though the change in the pericenter will still be
extremely small, as all of the dissipation occurs near pericenter for these
high-eccentricity systems; thus the tidal interaction itself may be relatively
unaltered, aside from the stronger tidal field of the SMBH due to relativistic
gravity). Future work on partial TDEs nearer the horizon of the SMBH should
incorporate the change in orbital energy due to tidal interaction and mass
loss asymmetry alongside gravitational wave emission.

Finally, here we focused on orbits that produce partial TDEs in the
traditional sense, i.e., the tidal force is not sufficiently strong to destroy
the star completely. However, Nixon & Coughlin (2022) found that at very high
$\beta$ (in their case $\beta=16$), the compression experienced by the star
near pericenter could revive self-gravity to the point that a core _reformed_
with a binding energy (to the SMBH) that was much larger than the value
predicted by Equation (2) (though we caution that while the mass contained in
the core was converged, the orbital period – and thus the binding energy – was
resolution-dependent in their simulations). While encounters with high-$\beta$
are rare333For example, Equation 16 of Coughlin & Nixon (2022) shows that the
fraction of TDEs with $\beta>10$ for a $10^{6}M_{\odot}$ Schwarzschild SMBH –
including general relativistic effects – is 0.0046; note that this is a factor
of $\sim 4$ smaller than the value derived by ignoring general relativistic
effects, given by their Equation 17., it may be possible for tidal capture in
this considerably more exotic scenario to produce shorter-period orbits than
through the traditional means.

## Data Availability Statement

Code to reproduce the results in this paper is available upon reasonable
request to the corresponding author.

## acknowledgements

M.C. acknowledges support from the Syracuse Office of Undergraduate Research
and Creative Engagement (SOURCE). CJN acknowledges support from the Science
and Technology Facilities Council (grant no. ST/W000857/1), and the Leverhulme
Trust (grant no. RPG-2021-380). E.R.C. acknowledges support from the National
Science Foundation through grant no. AST-2006684 and the Oakridge Associated
Universities through a Ralph E. Powe Junior Faculty Enhancement Award. This
work was performed using the DiRAC Data Intensive service at Leicester,
operated by the University of Leicester IT Services, which forms part of the
STFC DiRAC HPC Facility (www.dirac.ac.uk). The equipment was funded by BEIS
capital funding via STFC capital grants ST/K000373/1 and ST/R002363/1 and STFC
DiRAC Operations grant ST/R001014/1. DiRAC is part of the National
e-Infrastructure.

## References

  * Campana et al. (2015) Campana S., Mainetti D., Colpi M., Lodato G., D’Avanzo P., Evans P. A., Moretti A., 2015, A&A, 581, A17
  * Cohn & Kulsrud (1978) Cohn H., Kulsrud R. M., 1978, ApJ, 226, 1087
  * Coughlin & Nixon (2015) Coughlin E. R., Nixon C., 2015, ApJ, 808, L11
  * Coughlin & Nixon (2022) Coughlin E. R., Nixon C. J., 2022, ApJ, 936, 70
  * Cufari et al. (2022) Cufari M., Coughlin E. R., Nixon C. J., 2022, ApJ, 929, L20
  * Faber et al. (2005) Faber J. A., Rasio F. A., Willems B., 2005, Icarus, 175, 248
  * Fabian et al. (1975) Fabian A. C., Pringle J. E., Rees M. J., 1975, MNRAS, 172, 15
  * Frank & Rees (1976) Frank J., Rees M. J., 1976, MNRAS, 176, 633
  * Gafton et al. (2015) Gafton E., Tejeda E., Guillochon J., Korobkin O., Rosswog S., 2015, MNRAS, 449, 771
  * Golightly et al. (2019) Golightly E. C. A., Coughlin E. R., Nixon C. J., 2019, ApJ, 872, 163
  * Guillochon & Ramirez-Ruiz (2013) Guillochon J., Ramirez-Ruiz E., 2013, ApJ, 767, 25
  * Hills (1975) Hills J. G., 1975, Nature, 254, 295
  * King (2020) King A., 2020, MNRAS, 493, L120
  * King (2022) King A., 2022, MNRAS, 515, 4344
  * Kremer et al. (2022) Kremer K., Lombardi J. C., Lu W., Piro A. L., Rasio F. A., 2022, ApJ, 933, 203
  * Lightman & Shapiro (1977) Lightman A. P., Shapiro S. L., 1977, ApJ, 211, 244
  * Lu & Quataert (2022) Lu W., Quataert E., 2022, arXiv e-prints, p. arXiv:2210.08023
  * Mainetti et al. (2017) Mainetti D., Lupi A., Campana S., Colpi M., Coughlin E. R., Guillochon J., Ramirez-Ruiz E., 2017, A&A, 600, A124
  * Manukian et al. (2013) Manukian H., Guillochon J., Ramirez-Ruiz E., O’Leary R. M., 2013, ApJ, 771, L28
  * Marsden et al. (2020) Marsden C., Shankar F., Ginolfi M., Zubovas K., 2020, Frontiers in Physics, 8, 61
  * Miller et al. (2005) Miller M. C., Freitag M., Hamilton D. P., Lauburg V. M., 2005, ApJ, 631, L117
  * Miniutti et al. (2019) Miniutti G., et al., 2019, Nature, 573, 381
  * Nauenberg (1972) Nauenberg M., 1972, ApJ, 175, 417
  * Nixon & Coughlin (2022) Nixon C. J., Coughlin E. R., 2022, ApJ, 927, L25
  * Payne et al. (2021) Payne A. V., et al., 2021, ApJ, 910, 125
  * Press & Teukolsky (1977) Press W. H., Teukolsky S. A., 1977, ApJ, 213, 183
  * Price et al. (2018) Price D. J., et al., 2018, Publ. Astron. Soc. Australia, 35, e031
  * Syer et al. (1991) Syer D., Clarke C. J., Rees M. J., 1991, MNRAS, 250, 505
  * Wevers et al. (2022) Wevers T., et al., 2022, arXiv e-prints, p. arXiv:2209.07538
  * Zalamea et al. (2010) Zalamea I., Menou K., Beloborodov A. M., 2010, MNRAS, 409, L25