11institutetext:  1UC Merced 2Adobe 3Google 4Yonsei University

# Learning Visibility for Robust Dense Human Body Estimation

Chun-Han Yao1 Jimei Yang2 Duygu Ceylan2 Yi Zhou2
Yang Zhou2 Ming-Hsuan Yang134

###### Abstract

Estimating 3D human pose and shape from 2D images is a crucial yet challenging
task. While prior methods with model-based representations can perform
reasonably well on whole-body images, they often fail when parts of the body
are occluded or outside the frame. Moreover, these results usually do not
faithfully capture the human silhouettes due to their limited representation
power of deformable models (e.g., representing only the naked body). An
alternative approach is to estimate dense vertices of a predefined template
body in the image space. Such representations are effective in localizing
vertices within an image but cannot handle out-of-frame body parts. In this
work, we learn dense human body estimation that is robust to partial
observations. We explicitly model the visibility of human joints and vertices
in the x, y, and z axes separately. The visibility in x and y axes help
distinguishing out-of-frame cases, and the visibility in depth axis
corresponds to occlusions (either self-occlusions or occlusions by other
objects). We obtain pseudo ground-truths of visibility labels from dense UV
correspondences and train a neural network to predict visibility along with 3D
coordinates. We show that visibility can serve as 1) an additional signal to
resolve depth ordering ambiguities of self-occluded vertices and 2) a
regularization term when fitting a human body model to the predictions.
Extensive experiments on multiple 3D human datasets demonstrate that
visibility modeling significantly improves the accuracy of human body
estimation, especially for partial-body cases. Our project page with code is
at: https://github.com/chhankyao/visdb.

##  1 Introduction

Figure 1: Dense human body estimation with/without visibility modeling. We
propose to learn dense visibility to improve human body estimation in terms of
faithfulness to the input image and robustness to truncation (top) or
occlusions (bottom). We show the estimated meshes without/with visibility
modeling in columns 2-3 and the vertex visibility labels in columns 4-5
(purple:visible, orange:invisible).

Estimating 3D human pose and shape from monocular images is a crucial task for
various applications such as performance retargeting, virtual avatars, and
human action recognition. It is a fundamentally challenging problem due to the
depth ambiguity and the complex nature of human appearances that vary with
articulation, clothing, lighting, viewpoint, and occlusions. To represent the
complicated 3D human bodies via compact parameters, model-based methods like
SMPL [24] have been widely used in the community. However, SMPL parameters
represent human bodies in a holistic manner, causing their limited flexibility
to fit real-world images faithfully via direct regression. More importantly,
the regression-based methods tend to fail when a human body is not fully
visible in the image, _e.g._ , occluded or out of frame [16]. In this work, we
aim to learn human body estimation that is faithful to the input images and
robust to partial-body cases.

Instead of directly regressing SMPL parameters, we train a neural network to
predict the coordinate heatmaps in three dimensions for each human joint and
mesh vertex. The dense heatmap-based representation can preserve the spatial
relationship in the image domain and model the uncertainty of predictions. It
is shown to be effective in localizing visible joints/vertices and flexible to
fit an input image faithfully [39, 27, 28, 29]. Nonetheless, the x and y-axis
heatmaps are defined in the image coordinates, which cannot represent the out-
of-frame (_i.e._ , truncated by image boundaries) body parts. In addition,
occlusions by objects or the human body itself could cause ambiguity for
depth-axis predictions. Without knowing which joints/vertices are visible, the
network tends to produce erroneous outputs on partial-body images. To address
this, we propose _Visibility-aware Dense Body (VisDB)_ , a heatmap-based dense
representation augmented by visibility. Specifically, we train a network to
predict binary truncation and occlusion labels along with the heatmaps for
each human joint and vertex. With the visibility modeling, the proposed
network can learn to make more accurate predictions based on the observable
cues. In addition, the vertex-level occlusion predictions can serve as a depth
ordering signal to constrain depth predictions. Finally, by using visibility
as the confidence of 3D mesh prediction, we demonstrate that VisDB is a
powerful intermediate representation which allows us to regress and/or
optimize SMPL parameters more effectively. In Figure 1, we show examples of
truncation and occlusions as well as the dense human body estimations with and
without visibility modeling.

Considering that most existing 3D human datasets lack dense visibility
annotations, we obtain pseudo ground-truths from dense UV estimations [8].
Given the estimated UV map of an image, we calculate the pixel-to-vertex
correspondence by minimizing the distance of their UV coordinates. Each vertex
mapped to a human pixel is considered visible, and vice versa. Note that this
covers the cases of truncation, self-occlusions, and occlusions by other
objects. We further show that the dense vertex-to-pixel correspondence
provides a good supervisory signal to localize vertices in the image space.
Since dense UV estimations are based on part-wise segmentation masks which are
robust to partial-body images, the dense correspondence loss can mitigate the
inaccurate pseudo ground-truth meshes and better align the outputs with human
silhouettes. To demonstrate the effectiveness of our method, we conduct
extensive experiments on multiple human datasets used by prior arts. Both
qualitative and quantitative results on the Human3.6M [12], 3DPW [25], 3DPW-
OCC [25, 45], and 3DOH [45] datasets show that learning visibility
significantly improves the accuracy of dense human body estimation, especially
on images with truncated or occluded human bodies.

The main contributions of our work are:

  * •

We propose VisDB, a heatmap-based human body representation augmented with
dense visibility. We train a neural network to predict the 3D coordinates of
human joints and vertices as well as their truncation and occlusion labels. We
obtain pseudo ground-truths of visibility labels from image-based dense UV
estimates, which are also used as additional supervision signal to better
align our predictions with the input image.

  * •

We show how the dense visibility predictions can be used for robust human body
estimation. First, we exploit occlusion labels to supervise vertex depth
predictions. Second, we regress and optimize SMPL parameters to fit VisDB
(partial-body) outputs by using visibility as confidence weighting.

##  2 Related Work

Model-based human body estimation. Most existing methods on human body
estimation adopt a model-based representation. For instance, SMPL [24] is a
widely-used statistical human body model that maps a set of pose
$\theta\in\mathbb{R}^{72}$ and shape $\beta\in\mathbb{R}^{10}$ parameters to a
3D human mesh $V\in\mathbb{R}^{6890\times 3}$. In SMPL, $\theta$ represents
the axis-angle 3D rotations of 24 joints, and $\beta$ is the top-10 PCA
coefficients of a statistical human shape space. Early methods iteratively
optimize the SMPL parameters to fit the estimated 2D keypoints [2] or
silhouettes [20]. Several recent works [13, 35, 31, 34, 17, 19] train a deep
neural network to directly regress SMPL parameters from an input image.
However, the SMPL representation is not always informative enough for a
network to learn as it embeds the articulated body shapes in a low dimensional
space. The regression-based methods often fail on truncation and occlusion
cases since the networks tend to make holistic predictions based on certain
body parts only [16]. Instead, we show that localizing 3D vertices is a more
suitable task to learn for such scenarios. The network needs to learn the
relationship between the parameters and the shape as well in order to estimate
accurate SMPL parameters.

Dense human body representations. To fit the complicated shapes more
faithfully, dense human body representations have been proposed, including
volumetric space [41], occupancy field [37, 38], dense UV correspondence [1,
44], and 3D mesh [18, 4, 29, 21, 22]. Among these methods, I2L-MeshNet [29]
proposes an efficient heatmap representation to estimate human joints and
vertices in the image space and root-relative depth axis. It can fit the input
images accurately since heatmaps preserve the spatial relationship in image
features extracted by a convolutional neural network (CNN). Nonetheless, even
when certain body parts are not visible in the image, this model is designed
to localize all the joints and vertices within the image frame. We show that
it can negatively affect the model performance and emphasize the importance of
additional visibility information.

Occlusion-aware methods. Several methods have been proposed to deal with the
challenging scenarios where human bodies are partially truncated or occluded.
Muller  _et al._ [30] and Hassan  _et al._ [9] introduce explicit modeling of
human body self-contact and human-scene interactions, respectively. These
methods require ground-truth annotations which are hard to obtain. Other
methods leverage human-centric heatmaps, part segmentation masks, or dense UV
estimations [8], to increase the model robustness on truncated images [36],
crowded scenes (occluded by other people) [40] or general occlusions [43, 7,
45, 16]. Although effective in particular scenarios, most of them directly
regress SMPL parameters which still suffer from the limited representation
strength. To the best our our knowledge, the proposed VisDB representation is
the first to explicitly model dense human body visibility (including
truncation and all occlusion scenarios), which is trained with pseudo ground-
truth visibility labels from dense UV estimates.

Figure 2: VisDB framework overview (best viewed in color). Given an input
image, the network extracts features in the image and depth coordinates, from
where we predict the x, y, z heatmaps for each human joint and vertex. In
addition, we predict a binary visibility label (purple:visible,
orange:invisible) of each axis, _i.e._ , x-truncation, y-truncation, To obtain
a more regularized and complete human body, we train a regression network to
estimate SMPL parameters based on the dense 3D coordinates and visibility. At
test time, we can further optimize the regressed SMPL parameters to fit the
partial-body predictions from heatmaps.

##  3 Approach

We illustrate our overall framework in Figure 2. In Section 3.1, we describe a
heatmap-based representation which we build our method upon. Then, we
introduce the proposed Visibility-aware Dense Body (VisDB) in Section 3.2.
Each human joint and mesh vertex is represented by 1) three 1D heatmaps (x, y,
z dimensions) which define its 3D coordinate and 2) three binary labels
indicating its visibility in three dimensions. We train a network model to
predict the dense heatmaps and visibility, which represents a partial body
faithful to the input image. The visibility estimations can be interpreted as
depth ordering signals or prediction confidence. In Section 3.3, we design a
visibility-guided depth ordering loss to self-supervise depth estimation. In
Section 3.4, we show that VisDB outputs can be used to fit SMPL models
accurately and efficiently. We train a regression network to estimate SMPL
parameters based on the joint and vertex coordinates as well as their
visibility labels. During inference, we initialize the SMPL parameters by the
regressor and further optimize them to align with the VisDB predictions.
Finally, in Section 3.5, we exploit dense UV correspondence to obtain robust
pseudo labels of visibility and weakly supervise vertex localization in the
image space.

###  3.1 Preliminaries: Heatmap-based Representation

Given an input image, a prior heatmap-based method [29] estimates three 1D
heatmaps $H=\\{H^{x},H^{y},H^{z}\\}$ for each human joint and mesh vertex. The
x and y-axis heatmaps $H^{x},H^{y}$ are defined in the image space, and the
z-axis heatmaps $H^{z}$ are defined in the depth space relative to root joint.
We denote the joint heatmaps as $H_{J}\in\mathbb{R}^{N_{J}\times D\times 3}$
and vertex heatmaps as $H_{V}\in\mathbb{R}^{N_{V}\times D\times 3}$, where
$N_{J}$ is the number of joints, $N_{V}$ is the number of vertices, and $D$ is
the heatmap resolution. The heatmaps are predicted based on image features
$F\in\mathbb{R}^{c\times h\times w}$ extracted by a backbone network as
follows:

$\begin{split}H^{x}&=f^{\text{ 1D},x}(\text{avg}^{y}(f^{\text{ up}}(F))),\\\
H^{y}&=f^{\text{ 1D},y}(\text{avg}^{x}(f^{\text{ up}}(F))),\\\
H^{z}&=f^{\text{ 1D},z}(\psi(\text{avg}^{x,y}(F)))),\end{split}$ (1)

where $f^{\text{1D},i}$ is 1-by-1 1D convolution for the $i$-th axis heatmaps,
$\text{avg}^{i}$ is i-axis marginalization by averaging, $f^{\text{ up}}$
denotes up-sampling by deconvolution, and $\psi$ is a 1D convolution layer
followed by reshaping operation. Finally, the continuous 3D coordinates of
joints $J\in\mathbb{R}^{N_{J}\times 3}$ and vertices
$V\in\mathbb{R}^{N_{V}\times 3}$ can be obtained by applying soft-argmax on
the discrete heatmaps $H_{J}$ and $H_{V}$, respectively. More details can be
found in [29] and supplementary material.

###  3.2 Visibility-aware Dense Body

Heatmap-based representations are shown effective in estimating human bodies
in the image space. However, they often fail when the human bodies are
occluded or truncated since the predictions are based on spatial image
features and limited by the image boundaries. Without knowing which
joints/vertices are invisible, fitting a SMPL model on the entire body tends
to generate erroneous outputs. To deal with more practical scenarios where
only partial bodies are visible, we make the following adaptations to a
heatmap-based representation: 1) To augment the x and y-axis heatmaps, we
predict binary truncation labels $S^{x},S^{y}$, indicating whether a joint or
vertex is within the image frame, 2) For the z-axis heatmaps, we predict a
binary occlusion label $S^{z}$ which specifies the depth-wise visibility. The
visibility labels are predicted in a similar fashion as the heatmaps in Eq.
(1):

$\begin{split}S^{x}&=\sigma(\text{avg}^{x}(g^{\text{1D},x}(\text{avg}^{y}(f^{\text{
up}}(F))))),\\\
S^{y}&=\sigma(\text{avg}^{y}(g^{\text{1D},y}(\text{avg}^{x}(f^{\text{
up}}(F))))),\\\
S^{z}&=\sigma(\text{avg}^{z}(g^{\text{1D},z}(\psi(\text{avg}^{x,y}(F))))),\end{split}$
(2)

where $g^{\text{1D}}$ is a 1-by-1 1D convolutional layer similar to
$f^{\text{1D}}$ and $\sigma$ is a sigmoid operator. We then concatenate the
$\\{S^{x},S^{y},S^{z}\\}$ predictions to obtain joint visibility
$S_{J}\in\mathbb{R}^{N_{J}\times 3}$ and vertex visibility
$S_{V}\in\mathbb{R}^{N_{V}\times 3}$. By applying the soft-argmax operators to
the predicted 1D heatmaps, the final output of our network becomes
$\\{J,V,S_{J},S_{V}\\}$, referred to as Visibility-aware Dense Body (VisDB).
With the visibility information, the network model can learn to focus on the
visible body parts and push the invisible parts towards the image boundaries.
In our experiments (Table 3), we demonstrate that visibility modeling
significantly reduces the errors of visible vertices. Moreover, the visibility
labels can be seen as the confidence of coordinate predictions, which are
essential to mesh regularization and completion via SMPL model fitting as
described in Section 3.4.

We denote the ground-truth VisDB as
$\\{J^{*},V^{*},{S_{J}}^{*},{S_{V}}^{*}\\}$ and train the network by using the
following losses. The joint coordinate loss $\mathcal{L}_{joint}$ is defined
as:

$\mathcal{L}_{joint}=\|J-J^{*}\|_{1}.$ (3)

The vertex coordinate loss $\mathcal{L}_{vert}$ is defined as:

$\mathcal{L}_{vert}=\|V-V^{*}\|_{1}.$ (4)

We also regress the joints from vertices using a pre-defined regressor
$W\in\mathbb{R}^{N_{V}\times N_{J}}$ and calculate a regressed-joint loss
$\mathcal{L}_{r-joint}$:

$\mathcal{L}_{r-joint}=\|WV-J^{*}\|_{1}.$ (5)

Similar to [29], we apply losses on the mesh surface normal and edge length as
shape regularization. The normal loss $\mathcal{L}_{norm}$ and edge loss
$\mathcal{L}_{edge}$ are:

$\displaystyle\mathcal{L}_{norm}$
$\displaystyle=\sum_{f}\sum_{\\{v_{i},v_{j}\\}\subset
f}\Big{|}\Big{\langle}\frac{v_{i}-v_{j}}{\|v_{i}-v_{j}\|_{2}},{n_{f}}^{*}\Big{\rangle}\Big{|},$
(6) $\displaystyle\mathcal{L}_{edge}$
$\displaystyle=\sum_{f}\sum_{\\{v_{i},v_{j}\\}\subset
f}\Big{|}\|v_{i}-v_{j}\|_{2}-\|{v_{i}}^{*}-{v_{j}}^{*}\|_{2}\Big{|},$ (7)

where $f$ is a mesh surface, $n_{f}$ is the unit normal vector of f, and
$v_{i},v_{j}$ are the coordinates of vertex $i$ and $j$, respectively.
Finally, we define the joint and vertex visibility loss $\mathcal{L}_{vis}$
with binary cross entropy (BCE):

$\mathcal{L}_{vis}=\text{BCE}(S_{J},{S_{J}}^{*})+\text{BCE}(S_{V},{S_{V}}^{*}).$
(8)

The VisDB prediction is illustrated in Figure 2 (left).

###  3.3 Resolving Depth Ambiguity via Visibility

Vertex-level visibility can not only be seen as model confidence for SMPL
fitting but also provide depth ordering information. Intuitively, visible
vertices should have lower depth value compared to the invisible vertices
projected to the same pixel. We observe that VisDB network generally predicts
accurate 2D coordinates and visibility, but sometimes fails at depth
predictions when the human body occludes itself and the pose is less common in
the training datasets. To resolve the depth ambiguity in self-occlusion cases,
we propose a depth ordering loss $\mathcal{L}_{depth}$ based on vertex
visibility as follows:

$\mathcal{L}_{depth}=\sum_{x}\sum_{y}\text{ReLU}\Big{(}\max_{v\in
Q(x,y)}v^{z}-\min_{\overline{v}\in\overline{Q}(x,y)}\overline{v}^{z}\Big{)},$
(9)

where $Q(x,y)$ is the set of vertices projected to a discretized image
coordinate $(x,y)$ which belong to the front (occluding) part, and
$\overline{Q}$ contains the vertices of the back (occluded) part(s). The
definition can be written as:

$\begin{split}Q(x,y)&=\Big{\\{}v\big{|}v\mapsto(x,y)\wedge
P(v)=p^{*}(x,y)\Big{\\}}\\\
\overline{Q}(x,y)&=\Big{\\{}v\big{|}v\mapsto(x,y)\wedge P(v)\neq
p^{*}(x,y)\Big{\\}},\end{split}$ (10)

where $\mapsto$ denotes the discrete projection and $P(v)$ is the part label
of vertex $v$ defined in DensePose [8]. We define the front part $p^{*}(x,y)$
by finding the vertex with highest z-axis visibility score $s^{z}$ as:

$p^{*}(x,y)=P\Big{(}\operatorname*{arg\,max}_{v\mapsto(x,y)}{s_{v}}^{z}\Big{)}.$
(11)

$\mathcal{L}_{depth}$ is designed to push the self-occluded part(s)
$\overline{Q}$ to the back and non-occluded part $Q$ to the front, where the
occlusion information is given by the z-axis visibility. Note that we compare
the maximum depth (back side) of $Q$ and the minimum depth (front side) of
$\overline{Q}$, and thus $\mathcal{L}_{depth}$ will be nonzero if the depth
ordering disagrees with occlusion prediction and zero if the parts do not
overlap anymore. Since this loss depends on accurate visibility estimations,
we only apply it during the fine-tuning stage.

###  3.4 SMPL Fitting from Visible Dense Body

From the VisDB predictions, we can obtain the 3D coordinates and visibility of
human joints and vertices. While the partial-body outputs are faithful to the
input image from the front view, they sometimes look abnormal from a side view
or contain rough surfaces. To regularize the body shape and complete the
truncated parts, we perform model fitting on the visible dense body
predictions. Given the coordinates and visibility of joints and vertices, we
train a regression network to estimate SMPL pose $\theta\in\mathbb{R}^{72}$
and shape $\beta\in\mathbb{R}^{10}$ parameters. The regressed parameters are
then forwarded to the SMPL model to generate the mesh coordinates denoted as
$\text{SMPL}(\theta,\beta)\in\mathbb{R}^{N_{V}\times 3}$. Unlike prior art
[29] which regresses a SMPL model from all the joints regardless of their
visibility, our VisDB representation allows us to fit the visible partial body
only. The training objectives of the SMPL regressor include SMPL parameter
error, vertex error, joint error, and the negative log-likelihood of a pose
prior distribution. The SMPL parameter loss $\mathcal{L}_{smpl}$ is defined
as:

$\mathcal{L}_{smpl}=\|\theta-\theta^{*}\|_{1}+\|\beta-\beta^{*}\|_{1},$ (12)

where $\theta^{*}$ and $\beta^{*}$ are the ground-truth pose and shape
parameters. The SMPL vertex loss $\mathcal{L}_{smpl-vert}$ and joint loss
$\mathcal{L}_{smpl-joint}$ are defined similarly as in Eq. (4) and (5) but
weighted by visibility $S_{V},S_{J}$ as:

$\mathcal{L}_{smpl-
vert}=S_{V}\odot\|\text{SMPL}(\theta,\beta)-{V_{c}}^{*}\|_{1},$ (13)
$\mathcal{L}_{smpl-
joint}=S_{J}\odot\|W\text{SMPL}(\theta,\beta)-{J_{c}}^{*}\|_{1},$ (14)

where $\odot$ denotes element-wise multiplication, and
$({V_{c}}^{*},{J_{c}}^{*})$ are the ground-truth root-relative coordinates of
vertices and joints in the camera space. Ideally, the VisDB network makes more
confident predictions on the clearly visible joints and vertices. Hence, we
see the visibility labels as prediction confidence and use them to weight the
coordinate losses. In addition, we apply a pose prior loss
$\mathcal{L}_{prior}$ using a fitted Gaussian Mixture Model (GMM) provided by
[33]:

$\mathcal{L}_{prior}=-\text{log}\Big{(}\sum_{i}G_{i}(\theta)\Big{)},$ (15)

where $G_{i}$ is the $i$-th component of GMM.

We observe that the regressed SMPL meshes roughly capture the human pose and
shape but do not always align with the VisDB predictions in details.
Therefore, we use the regressed parameters as initialization and propose
efficient test-time optimization to further optimize the SMPL parameters
against VisDB predictions. For this optimization, we apply similar losses as
in Eq. (13)-(15), except that the ground-truths
$\\{{V_{c}}^{*},{J_{c}}^{*}\\}$ are replaced by the VisDB predictions
converted into root-relative coordinates in the camera space. Please refer to
the supplemental material for details on estimating the root joint coordinate
in the camera space. Since we initialize the SMPL parameters by the regression
network and the use strong supervisory signal, _i.e._ , 3D joint and vertex
coordinates, the test-time optimization only takes around 100 iterations to
converge using an Adam optimizer [14]. We illustrate the process of SMPL
regression and optimization in Figure 2 (right).

Figure 3: Dense UV correspondence and visibility labels. Given an input image,
we obtain a fitted SMPL mesh and dense UV estimation from off-the-shelf
algorithms. To acquire the dense visibility labels for training, we identify
the truncated vertices from the fitted mesh. From the dense UV map, we
calculate the pixel-to-vertex correspondence to obtain pseudo ground-truths of
vertex occlusions as well as image-space coordinates for weak supervision.

###  3.5 Exploiting Dense UV Correspondence

Most existing 3D human datasets do not provide joint visibility labels, and
none annotates vertex visibility. To train our VisDB network, we obtain pseudo
ground-truths from the fitted SMPL meshes and dense UV estimations. For x and
y-axis truncation, we can simply identify the truncated joints/vertices by
projecting the fitted mesh onto the image plane. Occlusion, however, cannot be
easily inferred from the input image or fitted mesh alone. One can estimate
self-occlusion by rendering a fitted mesh, but this does not capture
occlusions by other objects. More importantly, the fitting algorithm used to
get the pseudo ground-truth meshes is not robust to partial-body cases. To
address this, we propose to exploit dense UV correspondence between the input
image and a SMPL mesh. Dense UV estimation provides the part-based
segmentation mask of a human body as well as continuous UV coordinates of each
human pixel, which are robust to truncation and occlusions. We calculate the
UV coordinate of each pixel by applying an off-the-shelf dense UV estimation
method [8]. For each human pixel $p$, we then find the corresponding mesh
vertex $v$ whose UV coordinate is closest to the pixel. The pixel-to-vertex
$M_{P}$ and vertex-to-pixel $M_{V}$ mappings can be expressed as:

$\begin{split}M_{P}&=\big{\\{}p\rightarrow
v\big{|}v=\text{argmin}_{v^{\prime}}\big{\|}\text{UV}(v^{\prime})-\text{UV}(p)\big{\|}_{2}\forall
p\big{\\}}\\\
M_{V}&=\big{\\{}v\rightarrow\\{p^{\prime}\\}\big{|}M_{P}(p^{\prime})=v\forall
v\big{\\}}.\end{split}$ (16)

A vertex mapped to at least one pixel is labeled as visible or occluded
otherwise.

Similar to [7, 43, 44], we also utilize the dense vertex-pixel correspondence
as weak supervision for better alignment with the human silhouettes. For each
vertex $v$, we calculate the center of its corresponding pixels $M_{V}(v)$ and
define a UV correspondence loss $\mathcal{L}_{uv}$ as:

$\mathcal{L}_{uv}=\sum_{v}{s_{v}}^{z}\Big{\|}v^{x,y}-\sum_{p\in
M_{V}(v)}\frac{p}{|M_{V}(v)|}\Big{\|}_{1},$ (17)

where $v^{x,y}$ is the 2D projection of vertex $v$ and ${s_{v}}^{z}$ is the
binary occlusion label with ${s_{v}}^{z}=1$ indicating that the vertex $v$ is
visible. The UV correspondence loss can not only mitigate the inaccurate
pseudo ground-truth meshes, but improve the faithfulness to human silhouettes
since it is based on segmentation mask predictions. We empirically discover
that this direct vertex-level supervision is more efficient and effective for
VisDB training compared to rendering-based losses [43, 6]. The proposed
vertex-pixel correspondence and visibility labeling are illustrated in Figure
3.

###  3.6 Model Training and Inference

We first train the VisDB network on 3D data with mesh annotations, then fine-
tune it on all training data by adding the depth ordering and UV
correspondence losses. The regressor network is trained to estimate the SMPL
parameters based on the estimated coordinates and visibility of joints and
vertices. During inference, we apply optional optimization on the regressed
SMPL parameters to best align with the VisDB predicted mesh. For the VisDB
network backbone, we use a ResNet50 [11] model pre-trained on the ImageNet
dataset [5]. The weights are updated by the Adam optimizer [14] with a mini-
batch size of 64. We represent a human body by $N_{J}=30$ joints and
$N_{V}=6890$ vertices, and the heatmap resolution $D=64$. In addition, we use
the ground-truth bounding boxes to crop the human region from an input image
and resize it to 256$\times$256\. The bounding boxes of testing data are
estimated by a pre-trained Mask R-CNN [10] model if not available in the
dataset. We apply common data augmentations such as random scaling (±25%),
rotation (±45°), horizontal flip, and color jittering (±20%) during training.
Considering that truncation and occlusion examples are rare in most 3D human
datasets, we include random occlusion masks and bounding box shifting (±25%)
as additional augmentations to increase the partial-body/whole-body ratio. Our
models are implemented with PyTorch [32] and trained with NVIDIA Tesla V100
GPUs. More implementation details are presented in the supplemental material.

##  4 Experiments

###  4.1 Datasets and Metrics

Following most prior arts, we adopt mixed 2D-3D training on the MSCOCO [23],
Human3.6M [12], MuCo-3DHP [26], and 3DPW [25] datasets. The pseudo ground-
truth meshes of Human3.6M and MSCOCO are obtained by applying SMPLify-X [33]
to fit the joint annotations. We evaluate our models on the Human3.6M, 3DPW,
3DPW-OCC [25, 45], and 3DOH [45] testing sets. Note that 3DOH is composed of
images with object occlusions and 3DPW-OCC contains a subset of 3DPW sequences
where the human bodies are partially occluded. For quantitative evaluation, we
calculate the common joint and vertex error metrics in the camera space and
report them in millimeters (mm), including MPJPE (mean per-joint position
error) [12], PA-MPJPE (Procrustes-aligned mean per-joint position error) [46],
and MPVE (mean per-vertex error) [35].

Table 1: Quantitative evaluations on Human3.6M [12] and 3DPW [25]. To align the settings, we train our baseline, I2L-MeshNet [29], on the same datasets, and denote it by I2L-MeshNet†. Both our mesh and SMPL parameter outputs perform favorably against the prior state-of-the-arts. |  | Human3.6M | 3DPW
---|---|---|---
Method | Output |  MPJPE$\downarrow$ |  PA-MPJPE$\downarrow$ |  MPJPE$\downarrow$ |  PA-MPJPE$\downarrow$ |  MPVE$\downarrow$
GraphCMR [18] | Mesh | - | 50.1 | - | 70.2 | -
Pose2Mesh [4] | Mesh | 64.9 | 47.0 | 89.2 | 58.9 | 109.3
I2L-MeshNet [29] | Mesh | 55.7 | 41.1 | 93.2 | 57.7 | 109.2
I2L-MeshNet† [29] | Mesh | - | - | 84.5 | 51.1 | 98.2
METRO [21] | Mesh | 54.0 | 36.7 | 77.1 | 47.9 | 88.2
Mesh Graphormer [22] | Mesh | 51.2 | 34.5 | 74.7 | 45.6 | 87.7
VisDB (mesh) | Mesh | 51.0 | 34.5 | 73.5 | 44.9 | 85.5
NBF [31] | Param | - | 59.9 | - | - | -
HMR [13] | Param | 88.0 | 56.8 | - | 81.3 | -
DenseRaC [43] | Param | 76.8 | 48.0 | - | - | -
I2L-MeshNet [29] | Param | - | - | 100.0 | 60.0 | 121.5
OOH [45] | Param | - | 41.7 | - | - | -
SPIN [17] | Param | - | 41.1 | - | 59.2 | 116.4
I2L-MeshNet† [29] | Param | - | - | 88.0 | 55.5 | 102.3
DSR [6] | Param | 60.9 | 40.3 | 85.7 | 51.7 | 99.5
VIBE [15] | Param | 65.6 | 41.4 | 82.0 | 51.9 | 99.1
TCMR [3] | Param | 62.3 | 41.1 | - | - | -
DecoMR [44] | Param | 60.6 | 39.3 | - | - | -
PARE [16] | Param | - | - | 79.1 | 46.4 | 94.2
VisDB (param) | Param | 50.0 | 33.8 | 72.1 | 44.1 | 83.5

###  4.2 Quantitative Comparisons

Human3.6M and 3DPW. In Table 1, we compare the performance of our method and
prior arts on the Human3.6M [12] and 3DPW [25] datasets. For VisDB and
I2L-MeshNet [29], we report the results of both heatmap-based mesh outputs
(mesh) and SMPL parameters (param). Our SMPL parameters are obtained from
regression and test-time optimization. Note that each method uses different
network backbone, human body representation, training datasets, and inference
strategy. For instance, METRO [21] and Mesh Graphormer [22] adopt a
transformer-based [42] network while the others use CNN backbones. VIBE [15]
and TCMR [3] are video-based approaches whereas the others only take images as
input. Despite these differences, VisDB performs favorably against prior
methods in term of most evaluation metrics. Particularly, our method achieves
larger performance gains on the 3DPW dataset since it contains more truncation
and occlusion cases. The VisDB performance is most directly comparable with
I2L-MeshNet [29] as we adopt similar training settings. For fair comparisons,
we re-train its model on the same datasets and denote it as I2L-MeshNet†. The
results demonstrate that our visibility learning improves both the mesh and
SMPL outputs significantly. In prior literature, SMPL parameters generally
lead to higher errors compared to dense mesh outputs, which we conjecture is
caused by the difficulty to directly regress low-dimensional parameters. On
the contrary, VisDB is a powerful intermediate representation that provides
dense 3D information of visible partial body, allowing us to regress and
optimize SMPL parameters more accurately. In our experiments, we observe that
VisDB (mesh) captures the human silhouettes better but VisDB (param) produces
lower errors since the ground-truth meshes are also regularized by SMPL
representation.

Table 2: Quantitative evaluations on 3DOH [45] and 3DPW-OCC [25, 45]. We compare VisDB with prior occlusion-aware methods to demonstrate its robustness on partial-body cases. For VisDB and I2L-MeshNet† [29], We report both the mesh and SMPL parameter (mesh/param) results. | 3DOH | 3DPW-OCC
---|---|---
Method |  MPJPE$\downarrow$ |  PA-MPJPE$\downarrow$ |  MPJPE$\downarrow$ |  PA-MPJPE$\downarrow$ |  MPVE$\downarrow$
OOH [45] | - | 58.5 | - | 72.2 | -
I2L-MeshNet† [29] | 67.0/69.3 | 46.3/47.9 | 96.5/98.0 | 61.0/62.6 | 120.2/127.0
PARE [16] | 63.3 | 44.3 | 91.4 | 57.4 | 115.3
VisDB | 62.1/60.9 | 43.2/42.7 | 90.3/87.3 | 57.1/56.0 | 114.0/110.5

3DPW-OCC and 3DOH. To emphasize the robustness on partial-body images, we
further evaluate on two occlusion datasets: 3DPW-OCC [25, 45] and 3DOH [45].
As shown in Table 2, VisDB produces lower errors on both datasets compared to
prior occlusion-aware methods. While I2L-MeshNet† performs considerably worse
on these images, the errors by our model remain relatively low.

###  4.3 Ablation Studies

Table 3: Ablation studies of VisDB. We compare the joint/vertex errors of VisDB mesh outputs on 3DPW [25] with/without individual components. The results show that truncation modeling (${\mathcal{L}_{vis}}^{x,y}$), occlusion modeling (${\mathcal{L}_{vis}}^{z}$), depth ordering loss $\mathcal{L}_{depth}$, and UV correspondence loss $\mathcal{L}_{uv}$ each reduces the errors by a clear margin. ${\mathcal{L}_{vis}}^{x,y}$ | ${\mathcal{L}_{vis}}^{z}$ | $\mathcal{L}_{depth}$ | $\mathcal{L}_{uv}$ | MPJPE | PA-MPJPE | MPVE
---|---|---|---|---|---|---
|  |  |  | 84.5 | 51.1 | 98.2
| ✓ | ✓ | ✓ | 79.4 | 47.8 | 91.1
✓ |  | ✓ | ✓ | 75.8 | 45.5 | 88.0
✓ | ✓ |  | ✓ | 77.3 | 46.3 | 88.9
✓ | ✓ | ✓ |  | 74.9 | 45.6 | 87.1
✓ | ✓ | ✓ | ✓ | 73.5 | 44.9 | 85.5

VisDB network training. To evaluate the contribution of individual components
in our method, we perform ablation studies on the 3DPW dataset [25]. Table 3
shows the performance of VisDB mesh outputs with/without truncation modeling
${\mathcal{L}_{vis}}^{x,y}$, occlusion modeling ${\mathcal{L}_{vis}}^{z}$,
depth ordering loss $\mathcal{L}_{depth}$, and dense UV correspondence loss
$\mathcal{L}_{uv}$. Without ${\mathcal{L}_{vis}}^{x,y}$,
${\mathcal{L}_{vis}}^{z}$, $\mathcal{L}_{depth}$, and $\mathcal{L}_{uv}$, the
vertex error increases by 6.3mm, 3.1mm, 3.9mm, and 1.9mm, respectively. These
results show that both visibility modeling and depth ordering loss play a
crucial role in VisDB training.

SMPL parameter fitting. In Table 4, we quantitatively compare the SMPL models
obtained from different methods. Given an estimated VisDB mesh, we can regress
the SMPL parameters and/or optimize them during inference, and each process
can be done with/without dense visibility weighting (Eq. (13) and (14)). By
using visibility, the mean vertex error of regressed SMPL models drops by
8.7mm. With the proposed test-time optimization, we can further reduce the
error by 3.8mm.

Table 4: Ablation studies of SMPL models. We report the performance of SMPL outputs on the 3DPW dataset [25], which shows the effectiveness of our optimization and the importance of visibility in both regression and optimization work flows. Regression | Optimization | MPJPE | PA-MPJPE | MPVE
---|---|---|---|---
- | - | 73.5 | 44.9 | 85.5
w/o vis | - | 79.0 | 48.8 | 96.2
w/o vis | w/o vis | 77.6 | 47.0 | 93.9
w/ vis | - | 74.9 | 45.3 | 87.3
w/ vis | w/ vis | 72.1 | 44.1 | 83.5
|  |  |  |  |
---|---|---|---|---|---
|  |  |  |  |
|  |  |  |  |
|  |  |  |  |
Input |  I2L-MeshNet [29] | VisDB | VisDB | Visibility | Visibility
| (param) | (mesh) | (param) | (front view) | (side view)
Figure 4: Qualitative results on the 3DPW dataset [25]. For each example, we
show the results of I2L-MeshNet [29] SMPL model, our VisDB mesh, our optimized
SMPL model, as well as visibility predictions in the front and side views
(purple:visible, orange:invisible). When the human body is occluded (top two
rows) or truncated (bottom two rows), both our VisDB output and optimized SMPL
mesh capture the human silhouettes faithfully (_e.g._ , the left hand in row 1
and the head region in rows 2,3,4).

###  4.4 Qualitative Results

Figure 4 shows sample results by VisDB and I2L-MeshNet [29] on the 3DPW
dataset [25]. I2L-MeshNet [29] regresses SMPL parameters from the entire
heatmap-based mesh output, which leads to erroneous output meshes on truncated
or occluded examples. VisDB predicts accurate vertex visibility labels,
improving both the image-space dense body estimation and SMPL parameter
optimization. The results show that VisDB (mesh) outputs can fit the human
silhouettes faithfully, and VisDB (params) further regularizes and smooths the
mesh surfaces. More qualitative results are shown in the supplemental
material.

##  5 Conclusions

In this work, we address the problem of dense human body estimation from
monocular images. Particularly, we identify the limitations of existing model-
based and heatmap-based representations on truncated or occluded bodies. As
such, we propose a visibility-aware dense body representation, VisDB. We
obtain visibility pseudo ground-truths from dense UV correspondences and train
a network to predict 3D coordinates as well as truncation and occlusion labels
for each human joint and vertex. Extensive experimental results show that
visibility modeling can facilitate human body estimation and allow accurate
SMPL fitting from partial-body predictions.

## References

  * [1] Alldieck, T., Pons-Moll, G., Theobalt, C., Magnor, M.: Tex2shape: Detailed full human body geometry from a single image. In: ICCV (2019)
  * [2] Bogo, F., Kanazawa, A., Lassner, C., Gehler, P., Romero, J., Black, M.J.: Keep it smpl: Automatic estimation of 3d human pose and shape from a single image. In: ECCV. pp. 561–578 (2016)
  * [3] Choi, H., Moon, G., Chang, J.Y., Lee, K.M.: Beyond static features for temporally consistent 3d human pose and shape from a video. In: CVPR. pp. 1964–1973 (2021)
  * [4] Choi, H., Moon, G., Lee, K.M.: Pose2mesh: Graph convolutional network for 3d human pose and mesh recovery from a 2d human pose. In: ECCV. pp. 769–787 (2020)
  * [5] Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: Imagenet: A large-scale hierarchical image database. In: CVPR. pp. 248–255 (2009)
  * [6] Dwivedi, S.K., Athanasiou, N., Kocabas, M., Black, M.J.: Learning to regress bodies from images using differentiable semantic rendering. In: ICCV. pp. 11250–11259 (2021)
  * [7] Guler, R.A., Kokkinos, I.: Holopose: Holistic 3d human reconstruction in-the-wild. In: CVPR. pp. 10884–10894 (2019)
  * [8] Güler, R.A., Neverova, N., Kokkinos, I.: Densepose: Dense human pose estimation in the wild. In: CVPR. pp. 7297–7306 (2018)
  * [9] Hassan, M., Choutas, V., Tzionas, D., Black, M.J.: Resolving 3d human pose ambiguities with 3d scene constraints. In: ICCV. pp. 2282–2292 (2019)
  * [10] He, K., Gkioxari, G., Dollár, P., Girshick, R.: Mask r-cnn. In: ICCV. pp. 2961–2969 (2017)
  * [11] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: CVPR. pp. 770–778 (2016)
  * [12] Ionescu, C., Papava, D., Olaru, V., Sminchisescu, C.: Human3. 6m: Large scale datasets and predictive methods for 3d human sensing in natural environments. PAMI 36(7), 1325–1339 (2013)
  * [13] Kanazawa, A., Black, M.J., Jacobs, D.W., Malik, J.: End-to-end recovery of human shape and pose. In: CVPR. pp. 7122–7131 (2018)
  * [14] Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)
  * [15] Kocabas, M., Athanasiou, N., Black, M.J.: Vibe: Video inference for human body pose and shape estimation. In: CVPR. pp. 5253–5263 (2020)
  * [16] Kocabas, M., Huang, C.H.P., Hilliges, O., Black, M.J.: PARE: Part attention regressor for 3D human body estimation. In: ICCV. pp. 11127–11137 (2021)
  * [17] Kolotouros, N., Pavlakos, G., Black, M.J., Daniilidis, K.: Learning to reconstruct 3d human pose and shape via model-fitting in the loop. In: ICCV. pp. 2252–2261 (2019)
  * [18] Kolotouros, N., Pavlakos, G., Daniilidis, K.: Convolutional mesh regression for single-image human shape reconstruction. In: CVPR. pp. 4501–4510 (2019)
  * [19] Kolotouros, N., Pavlakos, G., Jayaraman, D., Daniilidis, K.: Probabilistic modeling for human mesh recovery. In: ICCV (2021)
  * [20] Lassner, C., Romero, J., Kiefel, M., Bogo, F., Black, M.J., Gehler, P.V.: Unite the people: Closing the loop between 3d and 2d human representations. In: CVPR. pp. 6050–6059 (2017)
  * [21] Lin, K., Wang, L., Liu, Z.: End-to-end human pose and mesh reconstruction with transformers. In: CVPR. pp. 1954–1963 (2021)
  * [22] Lin, K., Wang, L., Liu, Z.: Mesh graphormer. In: ICCV (2021)
  * [23] Lin, T.Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., Zitnick, C.L.: Microsoft coco: Common objects in context. In: ECCV. pp. 740–755 (2014)
  * [24] Loper, M., Mahmood, N., Romero, J., Pons-Moll, G., Black, M.J.: Smpl: A skinned multi-person linear model. TOG 34(6), 1–16 (2015)
  * [25] von Marcard, T., Henschel, R., Black, M.J., Rosenhahn, B., Pons-Moll, G.: Recovering accurate 3d human pose in the wild using imus and a moving camera. In: ECCV. pp. 601–617 (2018)
  * [26] Mehta, D., Sotnychenko, O., Mueller, F., Xu, W., Sridhar, S., Pons-Moll, G., Theobalt, C.: Single-shot multi-person 3d pose estimation from monocular rgb. In: 3DV. pp. 120–130 (2018)
  * [27] Moon, G., Chang, J.Y., Lee, K.M.: V2v-posenet: Voxel-to-voxel prediction network for accurate 3d hand and human pose estimation from a single depth map. In: CVPR. pp. 5079–5088 (2018)
  * [28] Moon, G., Chang, J.Y., Lee, K.M.: Camera distance-aware top-down approach for 3d multi-person pose estimation from a single rgb image. In: ICCV. pp. 10133–10142 (2019)
  * [29] Moon, G., Lee, K.M.: I2l-meshnet: Image-to-lixel prediction network for accurate 3d human pose and mesh estimation from a single rgb image. In: ECCV. pp. 752–768 (2020)
  * [30] Muller, L., Osman, A.A., Tang, S., Huang, C.H.P., Black, M.J.: On self-contact and human pose. In: CVPR. pp. 9990–9999 (2021)
  * [31] Omran, M., Lassner, C., Pons-Moll, G., Gehler, P., Schiele, B.: Neural body fitting: Unifying deep learning and model based human pose and shape estimation. In: 3DV. pp. 484–494 (2018)
  * [32] Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al.: Pytorch: An imperative style, high-performance deep learning library. NeurIPS 32, 8026–8037 (2019)
  * [33] Pavlakos, G., Choutas, V., Ghorbani, N., Bolkart, T., Osman, A.A., Tzionas, D., Black, M.J.: Expressive body capture: 3d hands, face, and body from a single image. In: CVPR. pp. 10975–10985 (2019)
  * [34] Pavlakos, G., Kolotouros, N., Daniilidis, K.: Texturepose: Supervising human mesh estimation with texture consistency. In: ICCV. pp. 803–812 (2019)
  * [35] Pavlakos, G., Zhu, L., Zhou, X., Daniilidis, K.: Learning to estimate 3d human pose and shape from a single color image. In: CVPR. pp. 459–468 (2018)
  * [36] Rockwell, C., Fouhey, D.F.: Full-body awareness from partial observations. In: ECCV. pp. 522–539 (2020)
  * [37] Saito, S., Huang, Z., Natsume, R., Morishima, S., Kanazawa, A., Li, H.: Pifu: Pixel-aligned implicit function for high-resolution clothed human digitization. In: ICCV. pp. 2304–2314 (2019)
  * [38] Saito, S., Simon, T., Saragih, J., Joo, H.: Pifuhd: Multi-level pixel-aligned implicit function for high-resolution 3d human digitization. In: CVPR. pp. 84–93 (2020)
  * [39] Sun, X., Xiao, B., Wei, F., Liang, S., Wei, Y.: Integral human pose regression. In: ECCV. pp. 529–545 (2018)
  * [40] Sun, Y., Bao, Q., Liu, W., Fu, Y., Black, M.J., Mei, T.: Monocular, one-stage, regression of multiple 3d people. In: ICCV. pp. 11179–11188 (2021)
  * [41] Varol, G., Ceylan, D., Russell, B., Yang, J., Yumer, E., Laptev, I., Schmid, C.: Bodynet: Volumetric inference of 3d human body shapes. In: ECCV. pp. 20–36 (2018)
  * [42] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł., Polosukhin, I.: Attention is all you need. In: NeurIPS. pp. 5998–6008 (2017)
  * [43] Xu, Y., Zhu, S.C., Tung, T.: Denserac: Joint 3d pose and shape estimation by dense render-and-compare. In: ICCV. pp. 7760–7770 (2019)
  * [44] Zeng, W., Ouyang, W., Luo, P., Liu, W., Wang, X.: 3d human mesh regression with dense correspondence. In: CVPR. pp. 7054–7063 (2020)
  * [45] Zhang, T., Huang, B., Wang, Y.: Object-occluded human shape and pose estimation from a single color image. In: CVPR. pp. 7376–7385 (2020)
  * [46] Zhou, X., Zhu, M., Pavlakos, G., Leonardos, S., Derpanis, K.G., Daniilidis, K.: Monocap: Monocular human motion capture using a cnn coupled with a geometric prior. PAMI 41(4), 901–914 (2018)


which is clearly antisymmetric under  $a\leftrightarrow b$ together with $c\leftrightarrow d$. However under the corresponding simultaneous exchange of $(1,2)(3,4)$ the $\mathrm{SO}(6)$ part is  symmetric, so finally their product vanishes.

§ USEFUL INTEGRALS FOR COMPUTING THE INVERSION FORMULA

In this appendix we collect all the relevant computations necessary to obtain the OPE coefficients and anomalous dimensions in section <ref>.

Let us start from the $\zb$ integral in (<ref>). Since in the studied examples we have at most $\log (1-\zb)$, which has vanishing double discontinuity,  we will only focus on

ℐ^, (λ)=∫_0^1 d/^2 (1-)^-/2κ_2h+2ℓ^,  () ^-/2 dDisc[(1-/)^λ]   ,
where we recall that we have defined $h$ as half the twist $2h=\tau=\Delta-\ell$.
This integral can be performed by replacing the hypergeometric function in $\kappa_{2h+2\ell}^{\Do, \Dt}(\zb)$ with its integral transform by introducing a fictitious integration variable $V$ and then by performing the change of variables $\zb \to\frac{T}{(T-1) V+1}$. The result is

\begin{align} \nonumber
\overline{\mathcal{I}}_\lambda^{\Do, \Dt}&=\frac{2 \Gamma (2 (h+\ell )) \Gamma \mleft(h+\ell -\frac{\Dt}{2}-\lambda -1\mright)}{\Gamma
\mleft(h+\ell -\frac{\Do}{2}\mright) \Gamma \mleft(h+\ell -\frac{\Dt}{2}\mright) \Gamma \mleft(h+\ell +\frac{\Dt}{2}+\lambda +1\mright)} \times
\\  \label{eq:intDiscRational}& \;\quad\times \sin (\pi  \lambda ) \Gamma (\lambda +1) \sin \mleft(\pi  \frac{\Dt-\Do}{2}+\pi\lambda \mright)\, \Gamma \mleft(-\frac{\Do}{2}+\frac{\Dt}{2}+\lambda +1\mright)\\ \nonumber
&=\frac{2 \pi ^2 \Gamma (2 h+2 \ell ) \Gamma \mleft(h+\ell -\frac{\Dt}{2}-\lambda
-1\mright)}{\Gamma (-\lambda ) \Gamma \mleft(\frac{\Do}{2}-\frac{\Dt}{2}-\lambda \mright) \Gamma \mleft(h+\ell -\frac{\Do}{2}\mright) \Gamma \mleft(h+\ell
-\frac{\Dt}{2}\mright) \Gamma \mleft(h+\ell +\frac{\Dt}{2}+\lambda
+1\mright)} \, ,
\end{align}

where in the second line we have assumed $\lambda<0$ and we have used Euler's reflection formula $\Gamma(1-x)\Gamma(x)=\pi/\sin(\pi x)$. Notice that the $\Gamma$ functions that appear are regular for all the relevant values of $\lambda$ and $\Delta_i$. Translated to $c_k(\Delta, \ell)$ this means that the $\zb$ integral just provides the spin dependence of the OPE data that we want to extract and does not give us any information about the twist of the possible exchange operators.

The information we are after is instead encoded in the $z$ integrals.  These can be distinguished in three different types

ℐ_1^, (λ)   =∫_0^1 d z/z^2(1-z)^-/2κ_4-2h^, (z)[(z/1-z)^λz^-/2]   ,
ℐ_2^, (λ)   =∫_0^1 d z/z^2(1-z)^-/2κ_4-2h^, (z)[ z^λz^-/2 ]  ,
ℐ_3^, (λ)   =∫_0^1 d z/z^2(1-z)^-/2κ_4-2h^, (z)[logz (z/1-z)^λz^-/2]   .
In all these cases $\lambda$ has to be considered positive and we are assuming $h \in \mathbb{N},\, h \geq 1$.[When solving these integrals spurious poles can appear at half integer values of $h$. We will ignore them having in mind that they can be cancelled by adding a reflected block with $h \to 1-h$ <cit.>.] The first integral can be computed similarly to the $\zb$ one and gives

ℐ_1^, (λ)=-πr^, _h, λ  sin(π(/2+h)) sin(π(/2+h)) sin(π(-/2+h+λ))/sin(2 πh) sin(πλ) sin( π(-/2+ λ)) sin(π(/2+h-λ))  ,
where we have collected ratios of $\Gamma$ functions  appearing throughout  these computations in a single function

r^, _h, λ =Γ(h+/2-1) Γ(h+/2-1) Γ(h-/2+λ-2)/Γ(2 h-3) Γ(λ) Γ(/2-/2+λ)
Γ(h+/2-λ)   .
We are now going to analyze in detail the poles that $\mathcal{I}_1^{\Do, \Dt}(\lambda)$ develops and the corresponding residues for all the values of the external dimensions appearing in the main text.

* $\mathbf{\Do=\Dt=0}$: we have simple poles for $h=\lambda+1+n$, $n \in \mathbb{N}$ with residues

Res_h=λ+1+nℐ_1^0,0(λ)=-r^0,0_λ+n+1, λ .

* $\mathbf{\Do=-\Dt=-2}$: we have simple poles for $h=\lambda+n$, $n \in \mathbb{N}$ with residues

Res_h=λ+n ℐ_1^-2,2(λ)=-r^-2,2_λ+n, λ .

* $\mathbf{\Do=\Dt=2}$: we have simple poles for $h=\lambda+1+n$, $n \in \mathbb{N}$ with residues

Res_h=λ+nℐ_1^2,2(λ)=-r^2,2_λ+n, λ .
There is a small caveat in this case: the function $r_{h, \lambda}^{2,2}$ is not well defined for $\lambda=1$ at $h=1,\,2$.  More generally it is not possible to properly define the residue of the corresponding integral for these specific values. To overcome this problem,  in the main text, when computing the contributions from  $\langle\qBPS \hBPS \qBPS \hBPS\rangle$ to $\langle a^{(0)}_{k, \Delta, \ell}\rangle$, we have used the usual expansion in conformal blocks. These results seem to suggest that  $\mathrm{Res}_{h=1,2} \mathcal{I}_1^{2,2}(1)=0$, as well as $\mathrm{Res}_{h=2} \mathcal{I}_1^{2,2}(2)=0$.

Moving to the second kind of integral,  the integration can now  be performed by replacing the hypergeoemetric function contained in $\kappa_{4-2h}^{\Do, \Dt}$ with its series representation. By resumming the results after integration we get

ℐ_2^, (λ)   =Γ(1/2 (-++2)) Γ(-/2-h+λ+1)/Γ(-/2-h+λ+2) ×
× _3F_2(
-/2-h+2 ,   /2-h+2  ,   -/2-h+λ+1

4-2 h  ,   -/2-h+λ+2
; 1)   .
By using some identities for the generalized hypergeometric function it is easy to study the residues associated to this integral for the interesting values of the external dimensions.

* $\boldsymbol{\Do=\Dt=0}$: we have simple poles for $h=\lambda+n+1, \, n \in \mathbb{N}$  with residues

Res_h=λ+n+1 ℐ_2^0,0(λ)=(1)^n+1r_λ+n+1, λ^0,0 .

* $\boldsymbol{\Do=\Dt=2}$: we have simple poles for $h=\lambda+n \, n \in \mathbb{N}$  with residues

Res_h=λ+n ℐ_2^2,2(λ) = (λ-2) (λ-1) (-1)^n+1/(λ+n-2) (λ+n-1)r_λ+n, λ^2,2 .
Notice that the residue vanishes for $\lambda=1, \, 2$ for any twist.

The last integral, the one containing $\log z$, is the hardest one and it was not possible to find a closed formula valid for any value of $\Do$ and $\Dt$. Thus we will report only the results necessary to reproduce the computations of the main discussion.

* $\boldsymbol{\Do=\Dt=0}$: for $\lambda>0$ and $h\geq 1$

ℐ^0,0_3(λ)   = -π^2 r_h,λ^0,0 /sin^2(π(λ-h))+ πtan(πh) sin(π(h+λ)) ) r_h,λ/2  sin^2(πλ)sin(π(λ-h)(H_h-λ  +H_λ+h
-2 H_h -1/λ+h -2-1/λ+h -1-1/λ+h+1/λ-h
+2/h-1+2/h  ) ,
where $H_n = \sum_{k=1}^n 1/k$ is the $n$-th harmonic number, or rather its analytic continuation to the complex plane.

* $\boldsymbol{-\Do=\Dt \in \mathbb{N}>0}$: for $\lambda>0$ and $h\geq 1$

ℐ^-,_3(λ)   =-π^2 r_h, λ^-, /sin^2(π(/2-λ+h ))+Π^()_h, λ r_h, λ^-,  ( -2/-2 λ+2 h
∑_α=±1( 2/2 h+ α-2+2/2 h -α)+ ∑_k=0^2 2/-2 (-k+λ+h )

+H_/2-λ+h +H_-/2+λ+h -H_h -/2-H_/2+h  )  ,
where we have defined

Π^()_h, λ= πsin(1/2 π(-2 h )) sin(1/2 π(+2 h ))
sin(1/2 π(-2 (λ+h )))/sin(πλ) sin(2 πh ) sin(π(-λ)) sin(π(/2-λ+h ))   .

In both cases notice the appearance of double poles: these are the signs of anomalous dimensions.

§ USEFUL TOOLS FOR RESOLVING THE AMBIGUITY

§.§ Recursion relations for conformal blocks

In subsection <ref> we explained how to fix part of the ambiguity by imposing that some twist-two operators vanish. The way in which this happens in practice is that a twist-four addition to the ambiguity functions $\CA_m(z,\zb)$ can result into twist-two and higher contributions to the correlator. This is a consequence of the entries of $v_k^{(m)}(z,\zb)$ which can multiply the blocks and shift their dimension. An explicit example that shows how this comes about was given in equation (<ref>).

All these contributions with shifted twist can be obtained thanks to the recursion relations satisfied by the conformal blocks. The special case with equal external dimensions was given in <cit.>. For a general scalar conformal block in four dimensions one has the following recursion relations

1+v/u g_Δ,ℓ^(a,b)    = [t]
-1/2 g_Δ-1,ℓ-1^(a,b)-2g_Δ-1,ℓ+1^(a,b) -
(J^2-a^2)(J^2-b^2)/8 (J -1) J^2 (J +1) g_Δ+1,ℓ+1^(a,b)

((τ-2)^2-a^2) ((τ-2)^2-b^2)/32 (τ-1) (τ-2)^2 (τ-3) g_Δ+1,ℓ-1^(a,b)
a b ((Δ-2)^2+ℓ(ℓ+2)) /(τ-2) (τ-4) (J -2) J g_Δ,ℓ^(a,b)

1-v/u g_Δ,ℓ^(a,b)    =  [t]
2 g_Δ-2,ℓ^(a,b)-a b (a^2-J^2) (J^2-b^2)/8 (J-1) J^2 (J+1) (2-τ) (4-τ)  g_Δ+1,ℓ+1^(a,b)
(J^2-a^2) (J^2-b^2)/2 (J-1) J^2 (J+1) g_Δ,ℓ+2^(a,b)
1/2 (a^2 b^2/(J-2) J (τ-2) (τ-4)+1) g_Δ,ℓ^(a,b)
a b ((τ-2)^2-a^2) ((τ-2)^2-b^2)/32 (J-2) J (τ-1) (τ-2)^2 (τ-3) g_Δ+1,ℓ-1^(a,b)
((τ-2)^2-a^2) ((τ-2)^2-b^2) /32 (τ-1) (τ-2)^2 (τ-3) g_Δ,ℓ-2^(a,b)
((τ-2)^2-a^2) ((τ-2)^2-b^2)  (J^2-b^2)(J^2-a^2) /128 (J-1) J^2 (J+1) (τ-1) (τ-2)^2 (τ-3) g_Δ+2,ℓ^(a,b)
2 a b /(J-2) J g_Δ-1,ℓ+1^(a,b)-a b /2 (τ-2) (τ-4) g_Δ-1,ℓ-1^(a,b)

where $a=\Delta_{12}$, $b=\Delta_{34}$, $\tau=\Delta-\ell$, $J=\Delta+\ell$ and $u=z\zb$, $v=(1-z)(1-\zb)$. These two relations are sufficient for all cases considered here.

§.§ Resumming blocks with coefficients

The ambiguity resolution for the correlators $\langle\qBPS\qBPS\hBPS\hBPS\rangle$ and $\langle\qBPS\qBPS\qBPS\qBPS\rangle$ involves an infinite family of conformal blocks of fixed twist

_m(z,) = ∑_ℓ=ℓ_0^∞ a_τ+ℓ,ℓ g_τ+ℓ,ℓ(z,) ,
with some given coefficients $a_{\tau+\ell,\ell}$ and fixed integer $\tau$. The goal of this appendix is to show how to perform this type of sums. First we separate the $z$ and the $\zb$ dependent parts by multiplying the sum by $z-\zb$

(z-)_m(z,) = z κ_τ-2()∑_ℓ=ℓ_0^∞a_τ+ℓ,ℓ/(-2)^ℓκ_2ℓ+τ(z) - (z↔) ,
Next we use the integral representation for the hypergeometric function inside $\kappa_{2\ell+\tau}(z)$. Under the assumption that the coefficients are suppressed enough to make the series convergent, we can swap the sum and the integral signs to obtain

S(z,t) = t^τ+b-1/2(1-t)^τ-b-1/2/(1-tz)^τ-a/2 ∑_ℓ=ℓ_0^∞a_τ+ℓ,ℓ/(-2)^ℓΓ(2ℓ+τ)/Γ(ℓ+τ+b/2)Γ(ℓ+τ-b/2) (t(1-t)/1-tz)^ℓ ,
with $a=\Delta_{12}$ and $b=\Delta_{34}$.  If the coefficients $a_{\tau+\ell,\ell}$ come from a free theory OPE, we expect their expressions to involve ratios of $\Gamma$ functions and polynomials in $\ell$, possibly with some $(-1)^\ell$. If that is the case, the sum $S(z,t)$ can be performed in terms of hypergeometric functions. When $\tau$ is an integer, such hypergeometric functions will reduce to rational functions. Then we only need to perform the final integration in $\di t$ over the interval $[0,1]$.

_m(z,) = z/z- κ_τ-2() ∫_0^1t  S(z,t) + (z↔) .
For this purpose, it is convenient to make the change of variables $t = \frac{s-z}{(s-1)z}$ which makes the integral somewhat easier to perform.

§ D-FUNCTIONS

It is possible to strip a kinematic prefactor from the $D$-functions defined in (<ref>) so as to obtain a function of the cross ratios $z,\zb$ only. Furthermore we can also remove the pole in $\Sigma=2$ that renders $D_{1111}$ ill defined. This leads to a “reduced” $D$-function denoted as $\Db$. The standard definition is the following

_Δ_1Δ_2Δ_3Δ_4(z,) = (x_13^2)^Σ-Δ_4(x_24^2)^Δ_2/(x_14^2)^Σ-Δ_1-Δ_4(x_34^2)^Σ-Δ_3-Δ_42∏_i=1^4Γ(Δ_i)/π^2Γ(Σ-2) D_Δ_1Δ_2Δ_3Δ_4(x_1,…,x_4) .
The recursion relations (<ref>) can be written as derivatives with respect to $z$ and $\zb$ of this reduced function as follows <cit.>

_Δ_1+1 Δ_2+1 Δ_3 Δ_4    = (1-z)∂_z-(1-)∂_/z-_Δ_1Δ_2Δ_3Δ_4 ,

_Δ_1 Δ_2 Δ_3+1 Δ_4+1    = (z (1-z)∂_z-(1-)∂_/z-+Σ-Δ_1-Δ_2)_Δ_1Δ_2Δ_3Δ_4 ,

_Δ_1 Δ_2+1 Δ_3+1 Δ_4    = z∂_z-∂_/z-_Δ_1Δ_2Δ_3Δ_4 ,

_Δ_1+1 Δ_2 Δ_3 Δ_4+1    = ((1-z)(1-) z∂_z-∂_/z-+Σ-Δ_2-Δ_3)_Δ_1Δ_2Δ_3Δ_4 ,

_Δ_1 Δ_2+1 Δ_3 Δ_4+1    = (z(z-1)∂_z-(-1)∂_/z-+Δ_2)_Δ_1Δ_2Δ_3Δ_4 ,

_Δ_1+1 Δ_2 Δ_3+1 Δ_4    = (z(z-1)∂_z-(-1)∂_/z-+Σ-Δ_4)_Δ_1Δ_2Δ_3Δ_4 .
with $\Sigma = \frac12(\Delta_1+\Delta_2+\Delta_3+\Delta_4)$.
The seed $\Db_{1111}$ is known in closed form and it reads

_1111(z,) = 1/z-(2Li_2(z) -2Li_2() + log(z)  log(1-z/1-)) .
The $\Db$-functions satisfy the same permutation identities as conformal four-point functions. Namely

_Δ_1Δ_2Δ_3Δ_4(z,)    =
((1-z)(1-))^-Δ_2_Δ_1Δ_2Δ_4Δ_3(z/z-1,/-1) ,
((1-z)(1-))^Δ_4-Σ_Δ_2Δ_1Δ_3Δ_4(z/z-1,/-1) ,
((1-z)(1-))^Δ_1+Δ_4-Σ_Δ_2Δ_1Δ_4Δ_3(z,) ,
_Δ_3Δ_2Δ_1Δ_4(1-z,1-) ,
They also satisfy the following identity <cit.>

_Δ_1Δ_2Δ_3Δ_4(z) = _Σ-Δ_3 Σ-Δ_4 Σ-Δ_1 Σ-Δ_2(z) .


11institutetext: Information Science, Department of Mathematics, Drexel
University
{rw37, pg427, ss4588<EMAIL_ADDRESS>
# Longitudinal Distance:
Towards Accountable Instance Attribution

Rosina O. Weber1 Prateek Goel1 Shideh Amiri3 and Gideon Simpson4

###### Abstract

Previous research in interpretable machine learning (IML) and explainable
artificial intelligence (XAI) can be broadly categorized as either focusing on
seeking interpretability in the agent’s model (_i.e_., IML) or focusing on the
context of the user in addition to the model (_i.e_., XAI). The former can be
categorized as feature or instance attribution. Example- or sample-based
methods such as those using or inspired by case-based reasoning (CBR) rely on
various approaches to select instances that are not necessarily attributing
instances responsible for an agent’s decision. Furthermore, existing
approaches have focused on interpretability and explainability but fall short
when it comes to accountability. Inspired in case-based reasoning principles,
this paper introduces a pseudo-metric we call Longitudinal distance and its
use to attribute instances to a neural network agent’s decision that can be
potentially used to build accountable CBR agents.

###### Keywords:

explainable artificial intelligence, interpretable machine learning, case-
based reasoning, pseudo-metric, accountability

##  1 Introduction and Background

The literature in interpretable machine learning (IML) and explainable
artificial intelligence (XAI) have proposed various categorization schemes and
taxonomies to describe their methods. One of those is through attribution –
attribution methods attempt to explain model behavior by associating a
classified (or predicted) test instance to elements of the model (_e.g_., [28,
32]).

There are two main categories of attribution methods. Instance attribution
methods select instances as integral elements to associate with a
classification or prediction (_e.g_., [8, 15, 17, 24, 34, 31]). Instance
attribution methods have been used for debugging models, detecting data set
errors, and creating visually-indistinguishable adversarial training examples
[8, 17, 24, 34]. Feature attribution methods select features to attribute to a
classification or prediction indicating which features play a more significant
role than others [5, 9, 20, 29, 30, 32].

When using case-based reasoning (CBR), a third category is often mentioned:
example- or sample-based. CBR can offer instances as examples but it does not
make attributions. The most successful use of CBR for explainability or
interpretability is in the ANN-CBR twin by [14]. In their work, Kenny and
Keane [14] were able to transfer the accuracy of a CNN into CBR for full
transparency. This was done by applying the feature attribution approach
DeepLift [30].

Both instance and feature attribution methods suffer strong criticisms.
DeepLift [30] demonstrated good performance when used in the investigation
reported in [14]. However, feature attribution methods have been shown to be
insensitive to changes in models and in data [1]. In addition, they do not
work in deep learning (DL) models that use a memory [18]. One important
limitation of feature attribution is how to evaluate them because of how the
data distribution changes when a feature is removed [13, 2].

If we extend the analysis beyond interpretability and explainability and
consider accountability, then feature attribution methods pose more
challenges. An accountable decision-making process should demonstrate its
processes align with legal and policy standards [19, 25]. This notion is
expanded in the literature around two goals. The first is to generate a fully
reproducible [19] audit of the system generating the decision (_e.g_.,[2, 6,
26, 27]); the second is to change the system when its decision is unwarranted
(_e.g_., [11, 25]). The additional challenge to feature attribution lies
therefore on the difficulty to change the behavior of a system that produces
unwanted decisions based on simply knowing the role of features in each
decision. Notwithstanding, it seems plausible to make changes when the
instances responsible for each decision are known.

Instance attribution methods are also target of criticisms. One criticism has
been directed to instances attributed when using influence functions [17] by
Brashan et al. [8]. It refers to the fact that the selected instances are
mostly outliers, a problem they overcome with relative influence [8]. Another
criticism is how time and processing consuming it is to compute influence
functions [15]. A third criticism is that selected sets of instances have
substantial overlap across multiple test instances to be classified.
Consequently, there seems there is still much to gain from further examining
instance attribution.

This paper proposes a new instance attribution method. It is intuitive and
thus simple to explain to users. It is also simple to compute. It has
limitations as well. It shares with influence functions the overlap between
sets of instances attributed to multiple test instance. It shares with ROAR
[13] and HYDRA [10] the requirement to retrain the model. It is a step into a
different direction that may spur further research. The proposed method is
based on a new pseudo-metric we call Longitudinal Distance (LD). This paper
focuses exclusively on classifications. For simplicity, we henceforth refer to
solutions as classifications and the NN structures as classifiers. In the next
sections, we describe the pseudo-metric LD and a non-pseudo-metric variant and
how they can be used for explanation. We then present some preliminary
studies. We conclude with a discussion and future steps.

##  2 Introducing Longitudinal Distances

Longitudinal distances are a class of distances based on a heuristic that an
iterative learning model such as NN can be used as an oracle for instance
attribution. Longitudinal distances operate in the metric space of instance
elements that are used to train and to be solved by NN methods through a
series of iterations (or epochs).

Given a classification model trained with neural networks (NN) on a space
where $x\in X$ are instances mapped by features $f\in F$, $X_{\rm train}\in X$
are training instances and hence include labels $y\in Y$ to indicate the class
they belong, and $X_{\rm test}$ are testing instances. The classifier
$C_{e}(x_{i})$ is a NN classifier that learns to assign labels $y$ using a set
of $n$ training instances $x_{i}$ through $e$ epochs, $e=1,\ldots,k$,
$x_{i}=1,\ldots,n$. We assume that the ability of a classifier to solve a
previously unseen instance is represented in the weights learned through a
sequence of learning iterations from a set of training instances. We therefore
hypothesize that there is at least one (or more) training instance(s)
responsible for the classification of an unseen instance, be it correct or
incorrect, and that the relation between the training and unseen instances are
somehow represented throughout the sequence of learning iterations.

We justify the proposed hypothesis based on the fact that when a previously
known solution, i.e., a known class, is selected to classify an unseen
problem, that both the training instance and the unseen instance, now both
labeled as members of the same class, meet some condition that causes their
solutions to be interchangeable, that is, their labels are the same. This
condition is an abstract concept that we wish to represent, let us call it the
oracle condition. We know that two instances together met this oracle
condition when they are members of the same class. Consequently, a trained
classifier can be perceived as a function that indicates that two instances
meet said oracle condition. However a trained classifier may not be ideal to
distinguish, among all instances that meet the oracle condition, which ones
are responsible for the classification of an unknown instance. We consider,
for example, the problem known as _catastrophic interference_ [23] to possibly
degrade the quality of a classifier with respect to some classes as it
continues to learn new classes. Because some abilities may be forgotten as an
NN method moves through new epochs, we contend that the complete experience
learned by a classifier is better represented by every weight matrix learned
in all epochs of its learning cycle. The last weight matrix represents the
learned classifier that has refined its ability to classify all instances
through generalization. Therefore, to successfully generalize, it is
reasonable to assume that particular abilities with respect to some instances
or labels were sacrificed. Hence, we propose to use the sequence of all
intermediary classifiers, the weight matrices resultant at the end of each
learning iteration, to measure the relationship between an unknown instance
and all the training instances and identify the one(s) responsible for the
classification.

We first propose the pseudometric Longitudinal Distance, ${d_{L}}$ to produce
the ratio of incorrect classifications to the total number of epochs.

$d_{L}(x_{i},x)=1-(\frac{1}{k}\sum_{e=1}^{e=k}\delta_{e}(x_{i},x))$ (1)

Now we propose the non-pseudometric Strict Longitudinal Distance,

$d_{SL}(x_{i},x)=1-\left(\frac{\sum_{e=1}^{e=k}w_{e}(x_{i})\delta_{e}(x_{i},x)}{\sum_{e=1}^{e=k}w_{e}(x_{i})}\right)$
(2)

We define the distance space as $(d,X)$ for the distance pseudo-metric
$d_{L}(x_{i},x)$, along with $d_{SL}(x_{i},x)$ where $x_{i},x\in X$ are
instances mapped by features $f\in F$ that are classified by a classifier
$C_{e}$ at epoch $e$ and receive a label $y\in Y$ to indicate their outcome
class.

In the above expressions,

$\delta_{e}(x_{i},x)=1_{C_{e}(x_{i})=C_{e}(x)}$ (3)

the $1_{C_{e}(xi)=C_{e}(x)}$ in Equation 3 is the indicator function taking
the value of 1 when they are equal and 0 otherwise.

For assigning relevance, we incorporate $w_{e}(x_{i})$ as a binary weight to
Equation 2 that indicates whether the classifier $C_{e}(x_{i})$ is correct or
wrong. When $w_{e}(x_{i})=1$, this means the label $y$ predicted by the
classifier $C_{e}(x_{i})$ for $x_{i}$ is equal to the label of the instance
and thus it is correct, otherwise $w_{e}(x_{i})=0$. We observe that Equation 1
assumes $x_{i}$ as a training instance and therefore there is a label $y$
designated as its class and thus $w_{e}(x_{i})$ can be computed. Note that
only the correctness of the classification of the training instance is
verified as the correct label of unseen instances is unknown. Consequently
Equation 2 is suited for computing the distances between a training instance
$x_{i}\in X_{\rm train}$ and a testing instance $x\in X_{\rm test}$, which is
the goal of this paper.

Note that we can rewrite

$d_{L}(x_{i},x)=\frac{1}{k}\sum_{e=1}^{e=k}(1-\delta_{e}(x_{i},x))=\frac{1}{k}\sum_{e=1}^{e=k}1_{C_{e}(x_{i})\neq
C_{e}(x)}$ (4)

which is the mean number of classifier mismatches over all the epochs.

Next we demonstrate that $d_{L}$ a pseudo-metric. Recall the pseudo-metric
properties:

  1. 1.

$d(x,x)=0$

  2. 2.

$d(x,y)=d(y,x)$

  3. 3.

$d(x,y)\leq d(x,z)+d(z,y)$

What makes $d$ a pseudo-metric and not necessarily a metric is that in order
to be a metric, if $d(x,y)=0$, then $x=y$. Because of mapping into feature
space to evaluate the classifier, there may exist predictors, $x_{i}$ and $x$,
for which $x_{i}\neq x$, but $d(x_{i},x)=0$.

For $d_{L}(x_{i},x)$,

$d_{L}(x_{i},x)=\frac{1}{k}\sum_{e=1}^{e=k}1_{C_{e}(x_{i})\neq
C_{e}(x)}=\frac{1}{k}\sum_{e=1}^{e=k}0=0.$ (5)

The symmetry property is also obvious. For the triangle property, given $x$,
$y$, and $z$, suppose $C_{e}(x)=C_{e}(y)$. Then,

$1_{C_{e}(x)\neq C_{e}(y)}=0\leq 1_{C_{e}(x)\neq C_{e}(z)}+1_{C_{e}(z)\neq
C_{e}(y)},$ (6)

regardless of what the two terms on the right are. If, instead, $C_{e}(x)\neq
C_{e}(y)$, then

$1_{C_{e}(x)\neq C_{e}(y)}=1$ (7)

Now we argue by contradiction. Suppose for some choice of $z$,

$1_{C_{e}(x)\neq C_{e}(z)}+1_{C_{e}(z)\neq C_{e}(y)}=0$ (8)

For this to be true, each of these two terms must be zero, so

$C_{e}(x)=C_{e}(z)=C_{e}(y)$ (9)

But this contradicts the assumption that $C_{e}(x)\neq C_{e}(y)$. Thus,
$1_{C_{e}(x)\neq C_{e}(z)}+1_{C_{e}(z)\neq C_{e}(y)}\geq 1$, and

$1_{C_{e}(x)\neq C_{e}(y)}\leq 1_{C_{e}(x)\neq C_{e}(z)}+1_{C_{e}(z)\neq
C_{e}(y)}$ (10)

Consequently the triangle equality holds for each epoch, and it must then hold
over any finite sum of epochs:

$d_{L}(x,y)=\frac{1}{k}\sum_{e=1}^{e=k}1_{C_{e}(x)\neq C_{e}(y)}\leq\\\
\frac{1}{k}\sum_{e=1}^{e=k}1_{C_{e}(x)\neq C_{e}(z)}+1_{C_{e}(z)\neq
C_{e}(y)}=\\\ d_{C}(x,z)+d_{C}(z,y)$ (11)

Consequently, this is a pseudo-metric.

###  2.1 Strict Longitudinal Distance

The variant Strict Longitudinal Distance considers a relevance weight based on
how correct the classifier $C_{e}(x)$ is throughout its life cycle.

If the $w_{e}$ were independent of the $x_{i}$ and $x$ arguments, then the
analogous computations would demonstrate $d_{SL}$, Equation 2, to also be a
pseudo-metric. However, because of $d_{SL}$’s dependence on the binary
weights, $w_{e}(x_{i})$, on the first argument, $x_{i}$, it fails to be a
pseudo-metric when $x\in X_{\rm test}$ is an element of the training set. It
does, however, satisfy the properties that $d_{SL}(x,y)\geq 0$ and
$d_{SL}(x,x)=0$, so it provides some weaker notion of distance between points.

##  3 Explaining with Longitudinal Distances

To use the proposed metrics $d_{L}$ and $d_{SL}$ for explaining decisions of
an NN, it is necessary that the weight matrices produced at every epoch are
preserved. If those have not been preserved, then it is necessary to retrain
the NN preserving its weight matrices. Due to its non-deterministic nature,
the instance to be explained has to be solved again by the trained classifier
for an explanation to be provided. To explain a given target instance, we
propose to compute the distance between the target instance and all training
instances using the longitudinal distance or the strict longitudinal distance.
With the results of distances, it is then possible to determine the minimum
(_i.e_., shortest) distance observed between the target and training
instances. Now note that multiple training instances can be equidistant to the
target, and this is exactly why longitudinal distances fail the axioms for
being metrics.

####  3.0.1 Definition 1

The shortest distance observed between a given target instance and all
training instances computed through longitudinal distances is defined as the
_explainer distance_.

####  3.0.2 Definition 2

The set of instances that are equidistant to the target instance at the
_explainer distance_ constitute the _positive explainer set_.

The _positive explainer set_ and the _negative explainer set_ are further
specified in that the _explainer set_ computed via longitudinal instances is,
by nature of equations (1) and (2), the _positive explainer set_. The premise
of the longitudinal distances may be reversed to compute the _negative
explainer set_ by modifying Equation (3) as follows:

$\delta_{e}(x_{i},x)=1_{C_{e}(x_{i})\neq C_{e}(x)}$ (12)

the $1_{C_{e}(x_{i})=C_{e}(x)}$ in Equation 12 is the indicator function
taking the value of 1 when they are not equal and 0 otherwise.

####  3.0.3 Definition 3

The _negative explainer set_ is the set of instances computed via longitudinal
distances that represent negative instances in classifying a given target
instance.

####  3.0.4 Definition 4

The _explainer set_ results from the combination of the _positive explainer
set_ and the _negative explainer set_.

The instances belonging to the _explainer set_ we theorize are responsible for
producing the solution to the target instance and consequently can explain its
solution. Once the explainer set is known, some considerations are needed.
First, the _explainer distance_ needs to be defined based on a precision value
$\mathcal{E}$. The value for $\mathcal{E}$ depends on the domain given that
explanations tend to be domain-dependent. Second, a training instance may
produce a direct or indirect explanation. A direct explanation would be one
that does not require any further processing as in all contents of the
training instance(s) suffices to explain the target instance. An indirect
explanation may require further processing such as comparing whether all
features of the target instance match the instances of the training instance
chosen to explain the target. Third, the explainer set may include one or more
training instances. When cardinality is greater than one, then a process to
select the most loyal explanation may be needed. As a result of this process,
it may be that no explanation is given. Fourth, if the explanation is meant to
foster human trust, the explainer set may need to be interpreted. When the set
is meant to produce accountability, then it has to be logical.

We note that the negative explainer set is needed to demonstrate positive and
negative instances that could be potentially used to train a classifier. When
used to select the best candidate for explanation, the positive explainer set
suffices.

##  4 Studies

Figure 1: Image of 9 on the left is the target instance, the set circled in
red is the positive explainer set with all members at $d_{L}$ = 0, the
explainer distance for the positive set. The set circled in blue is a sample
of a 3,569 (0.04%) image instances at the negative explainer distance $d_{L}$
= 1

###  4.1 Visual Inspection

We used the MNIST data set consisting of 55k training instances, and 10k
testing instances. We implemented a convolutional neural network design
consisting of 1 convolutional layer with 28 3x3 kernels, 1 max pool layer of
size 2x2 and stride of 1. The resultant layer is flattened and then connected
to 1 hidden layer with 128 nodes with ReLU activation function for every node.
A dropout of 20% is introduced following the hidden layer. The resultant
output is passed to the output layer with 10 nodes (each corresponding to the
label) with the Softmax activation function. This entire network is trained
with 55,000 28x28 MNIST images in batches of 64 over 10 epochs. This training
reached an accuracy of 83.9%111Code and results are available
https://github.com/Rosinaweber/LongitudinalDistances.

Fig. 1 shows the images of the entire positive explainer set for one of the
testing instances from MNIST labeled as a number nine. This positive explainer
set was selected for illustration because its cardinality is 12 and thus we
can show all the 12 images that are at distance zero from the target testing
instance, measured by $d_{L}$. The negative explainer set contains 3,569
images. Figure set shown circled in blue is a 0.4% sample of the complete set
of images at distance 1, which included 3,569 images.

Although not a validation of accuracy, images are often presented to
illustrate feature attribution methods. In Fig. 1 the negative explainer set
is a random selection of the large overall set. In that sample, the types of
nines are still nines but they are of a different category from those in the
positive explainer set and the target instance.

###  4.2 Fidelity of explanations with longitudinal distances

Fidelity of an explanation is a metric that aims to determine how well an
explanation generated by an XAI method aligns with ground-truth [4].This
concept has been discussed from multiple perspectives (_e.g_., [3, 7, 12, 16,
21, 22, 32, 33]), and also related to gold features [29]. In this study, we
use the data and method from [4]. Their approach is to create artificial data
sets by populating equation variables with values based on a given interval so
that each instance can be explained by the equation used to create it. This
way, an explanation is loyal to the data when it approximates the relationship
between the variables in the original equation. In this section, we use a data
set created from seven equations. The equations have three variables, which
were randomly populated. The result of these equations, the numeric value
produced when applying the values and executing the equation, is the fourth
feature of the data set. The label corresponds to the equation, hence there
are seven labels, $l=\\{0,1,2,3,4,5,6\\}$. The data set consists of 503,200
instances. The experiment is as follows.

####  4.2.1 Data set and deep neural network classifier.

We selected 403,200 instances for training and 100,000 for testing and
validation. We trained a deep learning architecture with 1 hidden layer and 8
nodes. The activation for the input and hidden layers was done using ReLU
functions and for the output layer using the Softmax activation function. The
loss calculated was categorical cross entropy. The 403,200 training instances
were trained in batches of 128 over 15 epochs and reached an accuracy of 95%
in the testing set. As required to implement the longitudinal distances, we
preserved the classifiers at each epoch.

####  4.2.2 Accuracy metric and hypothesis.

As earlier described, the data in this experiment was designed with equations
that are known and are used to designate the class of the instances. This way,
an XAI method produces a correct explanation if it selects to explain a target
instance with a training instance that implements the same equation with which
the instance being explained was built. The explanation will be given by an
existing training instance, which has a label. Consequently, correctness is
observed when the class label of the training instance selected to explain the
target instance has the same label as the target instance being explained.
With the definition of a correct explanation, we can define accuracy as the
ratio of correct explanations to the total explanations produced by the
proposed approach. Our hypothesis is that the proposed method using both
longitudinal distances will select instances to explain the classifications
produced by the deep NN architecture with average accuracy above 95%.

####  4.2.3 Methodology.

With the classifier trained, we randomly selected 1,000 from the 100,000
testing instances to measure the accuracy of the proposed approach. This
evaluation, as the metric describes, only seeks the selection of the class of
explanations, and thus it does not use the negative explainer set. For each
instance, we followed these steps:
1. Compute the positive explainer set using $d_{L}$ and $d_{SL}$.
2. Use the label of each instance to create subsets of instances for each label.
3. Determine the largest subset and use its label as explanation.
4. Assess whether label is correct.
5. Compute average accuracy.

##### Results.

Of the 1,000 testing instances, there were 978 and 980 correct explanations
for the approach when using $d_{L}$ and $d_{SL}$, respectively. Both average
accuracy levels are above 95%, which is consistent with our hypothesis. The
classifier is correct for 968 instances out of 1,000 (96.8%), being wrong in
32 instances. Both distances led to the selection of the wrong explanation
only when the classifier’s predictions were wrong. However, the distances do
not indicate the wrong explanations whenever the classifier is wrong.
Interestingly, both distances $d_{L}$ and $d_{SL}$ were correct, respectively,
in 10 and 12 instances for which the classifier was incorrect.

Table 1: Analysis of the predictions when $d_{L}$ selected the correct explanation when the classifier was wrong. The correlation between the explainer set (1st column) and the respective number of distinct predictions (4th column) and changes (5th column) is 0.60 and 0.74 Explainer set size |  Majority label set size |  15 Predictions |  Distinct predictions |  Changes
---|---|---|---|---
841 |  787 |  0, 0, 0, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4 |  2 |  1
527 |  480 |  4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 0, 4 |  2 |  2
527 |  480 |  4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 0, 4 |  2 |  2
321 |  321 |  4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 0, 4, 4, 4 |  2 |  2
321 |  321 |  4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 0, 4, 4, 4 |  2 |  2
120 |  120 |  5, 3, 3, 3, 3, 3, 3, 3, 3, 3, 1, 1, 1, 1, 1 |  2 |  2
32 |  32 |  6, 6, 6, 6, 6, 6, 0, 4, 0, 4, 0, 0, 4, 4, 4 |  3 |  6
29 |  29 |  6, 6, 6, 6, 6, 6, 0, 4, 4, 4, 4, 0, 4, 4, 4 |  3 |  4
15 |  15 |  6, 6, 6, 6, 6, 6, 0, 4, 4, 4, 0, 0, 4, 4, 4 |  3 |  4
5 |  5 |  3, 3, 3, 3, 3, 3, 3, 3, 0, 3, 3, 0, 0, 0, 0 |  2 |  3

##### Discussion.

It is curious that the distances led to the selection of the correct
explanation in instances when the classifier was wrong because the distances
are based on the results of the classifier. We examined whether there was
anything else unusual about those instances. We found that when $d_{L}$ was
able to select correct explanations while the classifier was wrong, the
cardinality of the explainer sets was much smaller than in other instances.
Considering when the classifier was wrong, the average size of the explainer
set was 274 instances when the correct explanation was selected in comparison
to 23,239 when the explanation was not correct. We then examined the
predictions of the classifier throughout the 15 epochs and computed two
values, namely, the number of distinct predictions and the number of times the
prediction changed. Table 1 shows the values for all instances where $d_{L}$
selected the correct explanation when the classifier was wrong. The
correlation between the explainer set and the respective number of distinct
predictions and changes is 0.60 and 0.74. We also computed the correlation
between the explainer set and the respective number of distinct predictions
and changes for when $d_{L}$ was not correct, and the results are 0.82 and
0.72. These correlations suggest that the more changes in labels the
classifier makes as it is learning, the more demanding the distances become
causing the explainer sets to be more efficient. There is a lot to investigate
further on this finding.

##  5 Conclusions and Future Work

In this paper, we introduced longitudinal distances to be computed between an
unseen target instance classified by an NN and all training instances used to
train it. The training instances at the shortest distance from the target
instance constitute its explainer set. The training instances that are members
of the explainer set are hypothesized to have contributed to classify the
target instance. As such, they can be potentially used to explain the target
instance.

Longitudinal instances are inspired by the similarity heuristics and the
principle that similar problems have similar solutions. Although not
demonstrated yet, instance attribution methods have the potential to bring to
example-based XAI, particularly when implemented with CBR, the facet of
attribution, currently missing in those approaches.

In this paper, the positive explainer set was used to select the best
candidate for explanation. We did not use the negative explainer set. It is
obvious that, if we are interested in accountability, as this paper described
in its motivation, then we first need to demonstrate that the explainer set
produces the explained decision; this is when we will use the negative
explainer set. Ultimately, we also need to determine how to modify the set to
change its decisions.

New XAI and IML approaches should demonstrate how they address any criticisms.
Demonstrating how they perform in presence of noise, the proportion of
outliers in the explainer set, and the proportion of overlap across explainer
sets for multiple instances are future work for longitudinal distances.

#### Acknowledgments.

Support for the preparation of this paper to Rosina O Weber and Prateek Goel
was provided by NCATS, through the Biomedical Data Translator program (NIH
award 3OT2TR003448-01S1). Gideon Simpson is supported by NSF Grant
no.DMS-1818716.

## References

  * [1] Adebayo, J., Gilmer, J., Muelly, M., Goodfellow, I., Hardt, M., Kim, B.: Sanity checks for saliency maps. In: 32nd NeurIPS. pp. 9525–9536 (2018)
  * [2] Adler, P., Falk, C., Friedler, S.A., Nix, T., Rybeck, G., Scheidegger, C., Smith, B., Venkatasubramanian, S.: Auditing black-box models for indirect influence. Knowledge and Information Systems 54(1), 95–122 (2018)
  * [3] Alvarez-Melis, D., Jaakkola, T.S.: Towards robust interpretability with self-explaining neural networks. arXiv preprint arXiv:1806.07538 (2018)
  * [4] Amiri, S.S., Weber, R.O., Goel, P., Brooks, O., Gandley, A., Kitchell, B., Zehm, A.: Data representing ground-truth explanations to evaluate xai methods. arXiv preprint arXiv:2011.09892 (2020)
  * [5] Bach, S., Binder, A., Montavon, G., Klauschen, F., Müller, K.R., Samek, W.: On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation. PloS one 10(7), e0130140 (2015)
  * [6] Baldoni, M., Baroglio, C., May, K.M., Micalizio, R., Tedeschi, S.: Computational accountability. In: Deep Understanding and Reasoning: A Challenge for Next-generation Intelligent Agents, URANIA 2016. vol. 1802, pp. 56–62. CEUR Workshop Proceedings (2016)
  * [7] Barr, B., Xu, K., Silva, C., Bertini, E., Reilly, R., Bruss, C.B., Wittenbach, J.D.: Towards ground truth explainability on tabular data. arXiv preprint arXiv:2007.10532 (2020)
  * [8] Barshan, E., Brunet, M.E., Dziugaite, G.K.: Relatif: Identifying explanatory training samples via relative influence. In: International Conference on Artificial Intelligence and Statistics. pp. 1899–1909. PMLR (2020)
  * [9] Bhatt, U., Ravikumar, P., et al.: Building human-machine trust via interpretability. In: AAAI Conference on Artificial Intelligence. vol. 33, pp. 9919–9920 (2019)
  * [10] Chen, Y., Li, B., Yu, H., Wu, P., Miao, C.: Hydra: Hypergradient data relevance analysis for interpreting deep neural networks. arXiv:2102.02515 (2021)
  * [11] Garfinkel, S., Matthews, J., Shapiro, S.S., Smith, J.M.: Toward algorithmic transparency and accountability. Communications of the ACM 60(9) (2017)
  * [12] Guidotti, R.: Evaluating local explanation methods on ground truth. Artificial Intelligence 291, 103428 (2021)
  * [13] Hooker, S., Erhan, D., Kindermans, P.J., Kim, B.: A benchmark for interpretability methods in deep neural networks. In: Wallach, H., Larochelle, H., Beygelzimer, A., d'Alché-Buc, F., Fox, E., Garnett, R. (eds.) Advances in Neural Information Processing Systems. vol. 32. Curran Associates, Inc. (2019), https://proceedings.neurips.cc/paper/2019/file/fe4b8556000d0f0cae99daa5c5c5a410-Paper.pdf
  * [14] Kenny, E.M., Keane, M.T.: Twin-systems to explain artificial neural networks using case-based reasoning: Comparative tests of feature-weighting methods in ann-cbr twins for xai. In: Twenty-Eighth International Joint Conferences on Artificial Intelligence (IJCAI), Macao, 10-16 August 2019. pp. 2708–2715 (2019)
  * [15] Khanna, R., Kim, B., Ghosh, J., Koyejo, S.: Interpreting black box predictions using fisher kernels. In: The 22nd International Conference on Artificial Intelligence and Statistics. pp. 3382–3390. PMLR (2019)
  * [16] Kim, B., Wattenberg, M., Gilmer, J., Cai, C., Wexler, J., Viegas, F., et al.: Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav). In: International conference on machine learning. pp. 2668–2677. PMLR (2018)
  * [17] Koh, P.W., Liang, P.: Understanding black-box predictions via influence functions. In: International Conference on Machine Learning. pp. 1885–1894. PMLR (2017)
  * [18] Koul, A., Greydanus, S., Fern, A.: Learning finite state representations of recurrent policy networks. arXiv preprint arXiv:1811.12530 (2018)
  * [19] Kroll, J.A., Huey, J., Barocas, S., Felten, E.W., Reidenberg, J.R., Robinson, D.G., Yu, H.: Accountable algorithms. UNIVERSITY of PENNSYLVANIA LAW REVIEW pp. 633–705 (2017)
  * [20] Lundberg, S., Lee, S.I.: A unified approach to interpreting model predictions. arXiv preprint arXiv:1705.07874 (2017)
  * [21] Mahajan, D., Tan, C., Sharma, A.: Preserving causal constraints in counterfactual explanations for machine learning classifiers. arXiv preprint arXiv:1912.03277 (2019)
  * [22] Man, X., Chan, E.P.: The best way to select features? comparing mda, lime, and shap. The Journal of Financial Data Science 3(1), 127–139 (2021)
  * [23] McCloskey, M., Cohen, N.: Catastrophic interference in connectionist networks: The sequential learning problem. Psychology of Learning and Motivation - Advances in Research and Theory 24(C), 109–165 (Jan 1989)
  * [24] Mercier, D., Siddiqui, S.A., Dengel, A., Ahmed, S.: Interpreting deep models through the lens of data. In: 2020 International Joint Conference on Neural Networks (IJCNN). pp. 1–8. IEEE (2020)
  * [25] Mittelstadt, B., Russell, C., Wachter, S.: Explaining explanations in ai. In: Proceedings of the conference on fairness, accountability, and transparency. pp. 279–288 (2019)
  * [26] Mittelstadt, B.D., Allo, P., Taddeo, M., Wachter, S., Floridi, L.: The ethics of algorithms: Mapping the debate. Big Data & Society 3(2), 2053951716679679 (2016)
  * [27] NOW, A.: Annual report–new york university (2018)
  * [28] Olah, C., Satyanarayan, A., Johnson, I., Carter, S., Schubert, L., Ye, K., Mordvintsev, A.: The building blocks of interpretability. Distill 3(3), e10 (2018)
  * [29] Ribeiro, M.T., Singh, S., Guestrin, C.: ” why should i trust you?” explaining the predictions of any classifier. In: 22nd ACM SIGKDD. pp. 1135–1144 (2016)
  * [30] Shrikumar, A., Greenside, P., Kundaje, A.: Learning important features through propagating activation differences. In: International Conference on Machine Learning. pp. 3145–3153. PMLR (2017)
  * [31] Sliwinski, J., Strobel, M., Zick, Y.: Axiomatic characterization of data-driven influence measures for classification. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 33, pp. 718–725 (2019)
  * [32] Sundararajan, M., Taly, A., Yan, Q.: Axiomatic attribution for deep networks. In: International Conference on Machine Learning. pp. 3319–3328. PMLR (2017)
  * [33] Yang, M., Kim, B.: Benchmarking attribution methods with relative feature importance. arXiv preprint arXiv:1907.09701 (2019)
  * [34] Yeh, C.K., Kim, J.S., Yen, I.E., Ravikumar, P.: Representer point selection for explaining deep neural networks. arXiv preprint arXiv:1811.09720 (2018)