# Test-time Adaptation vs. Training-time Generalization:
A Case Study in Human Instance Segmentation using Keypoints Estimation

Kambiz Azarian  Debasmit Das  Hyojin Park  Fatih Porikli  Qualcomm AI Research
{kambiza, debadas, hyojinp<EMAIL_ADDRESS>Qualcomm AI Research is
an initiative of Qualcomm Technologies, Inc.

###### Abstract

We consider the problem of improving the human instance segmentation mask
quality for a given test image using keypoints estimation. We compare two
alternative approaches. The first approach is a test-time adaptation (TTA)
method, where we allow test-time modification of the segmentation network’s
weights using a single unlabeled test image. In this approach, we do not
assume test-time access to the labeled source dataset. More specifically, our
TTA method consists of using the keypoint estimates as pseudo labels and
backpropagating them to adjust the backbone weights. The second approach is a
training-time generalization (TTG) method, where we permit offline access to
the labeled source dataset but not the test-time modification of weights.
Furthermore, we do not assume the availability of any images from or knowledge
about the target domain. Our TTG method consists of augmenting the backbone
features with those generated by the keypoints head and feeding the aggregate
vector to the mask head. Through a comprehensive set of ablations, we evaluate
both approaches and identify several factors limiting the TTA gains. In
particular, we show that in the absence of a significant domain shift, TTA may
hurt and TTG show only a small gain in performance, whereas for a large domain
shift, TTA gains are smaller and dependent on the heuristics used, while TTG
gains are larger and robust to architectural choices.

##  1 Introduction

Figure 1: From left to right: baseline, test-time adapted and training-time
generalized Mask-RCNN.

Human instance segmentation is an important task that requires finding the
image regions of different individuals in a scene. This task has multiple
applications ranging from video conferencing [15, 7], matting [35, 42, 18],
generation [28, 24], autonomous driving [38], robotics [32], extended reality
[39], pedestrian tracking [26, 25, 41], etc. For example, in extended reality
applications, human instance segmentation can be used to detect human
contours. Along with 3D information, the human contours can be used to render
virtual objects at arbitrary locations, which can then enhance augmented
reality experiences. For human-robot interaction, segmentation of individuals
is required for understanding spatial relationships that allows the robot to
plan movements and perform complex tasks. Since human instance segmentation
has multiple use cases, there has been considerable effort in recent years to
improve the performance of such models.

Most methods for human instance segmentation are built on top of general
instance segmentation frameworks. Such methods can be broadly divided into
single-stage and multi-stage methods. Single-stage methods [8, 33, 3] normally
use parallel branches for detection and segmentation. The detection branch is
used to localize each individual instance, while the segmentation branch
learns to annotate each pixel based on the feature information densely. The
output of the detection branch and the segmentation branch are employed to
obtain masks for all the instances in the scene. Since detection and
segmentation operate independently of each other, single-stage methods are
considerably faster than multi-stage counterparts. However, the segmentation
step does not utilize the localization information; thus, single-stage methods
perform poorer than multi-stage ones.

Two-stage methods [10, 23, 12] generally follow a sequential approach of
firstly detection and then segmentation. Multi-stage methods [4, 6] go a step
further and repeat this two-step method multiple times. The detection stage
normally crops out a region of interest that localizes a particular object.
Subsequently, segmentation is applied for different regions of interest to
produce masks for different instances in the scene. Since segmentation and
detection depend on each other during training or inference, segmentation
quality is much better compared to single-stage methods. Hence, in this paper,
we opt for the popular two-stage framework Mask-RCNN [12], which also carries
a pose estimation head to aid human instance segmentation.

Compared to general instance segmentation, human instance segmentation is
relatively more challenging for a couple of reasons. Firstly, there is large
intra-class variation within the human category due to various outfits, poses,
and deformability of the human body. Secondly, the number of humans and their
locations in a scene can be random, which increases the complexity of model
inference. Finally, humans tend to interact with each other and other objects
causing occlusions and obstructions, resulting in unnatural 2D shapes and
poses. To tackle such challenges, pose estimation [5, 9, 16] has been used to
enhance human instance segmentation.

There have been different approaches to using pose estimation for human
segmentation. The most popular method [36] follows a sequential approach,
where humans are firstly detected, and their corresponding poses are
estimated. The sparse keypoints are then grouped, and the generated heatmaps
are used to segment the humans. Although the proposed sequential approach is
seminal, it has the weakness of propagating the pose estimation errors to the
segmentation stage, which can negatively affect the performance, especially
for occluded humans. Alternatively, there are methods [27, 12] that carry out
a joint estimation of pose and instance segmentation, which do not suffer from
propagation issues. We build on the latter framework, i.e., Mask-RCNN [12] in
this paper. Specifically, we consider how human pose estimation can be used to
enhance the performance of a human instance-segmentation model (trained on a
source dataset) on test images from a target dataset. Our evaluations are done
on the OCHuman [36] and COCOPersons [20, 36] datasets from which we draw
conclusions about the feasibility and extent of such improvement using
generalization, adaptation, or both.

Our main contributions are summarized as follows:

  * •

We propose a _test-time_ method for adapting a human segmentation network to a
single unlabelled test image. It involves backpropagating keypoint pseudo-
labels to adjust the backbone weights. We devise three keypoints head variants
in addition to the Mask-RCNN’s keypoints head. Two of the variants are
transformer based, and all augment the pose estimates to include keypoint
visibility/occlusion indicators.

  * •

We propose a _training-time_ method for enhancing the performance of a human
segmentation network on new domains. It involves augmenting the backbone
features with those generated by the keypoints head and using the aggregate
feature vector for segmentation. We show our method achieves competitive
performance despite its simplicity.

  * •

We evaluate the performance of both methods and, through ablations, identify
factors that limit test-time adaptation (TTA) gains. We show that for a small
domain shift, TTA may hurt performance, and training-time generalization (TTG)
delivers only a small gain. For a large domain shift, TTA gains are small and
dependent on the heuristics used, while TTG gains are more prominent and
relatively insensitive to architectural details.

##  2 Related Work

Non-Adaptive Instance Segmentation: Non-adaptive Instance segmentation methods
generally have enhanced architectures either through single-stage approaches
[8, 33, 3] or multi-stage approaches [10, 23, 12]. Single-stage methods have a
distributed approach where they produce feature maps for the whole image and
then extract the feature maps for each instance to produce the corresponding
masks. For example, InstanceFCN [8] generates instance-specific scoring maps
and outputs instance masks using an assembly module that contains operations
like repooling and mask-voting. In similar spirit, CondInst [30] dynamically
generated convolutions conditioned on each instance to produce instance-
specific segmentation masks. On the other hand, YOLACT [3] is an efficient
method that has parallel branches for generating fixed number of prototype
masks and mask coefficients. Multi-stage methods generally have a two step
procedure where firstly objects are detected using bounding boxes. Then,
features are extracted from the region of interest to produce the desired
masks. Mask R-CNN [12] is a popular two-stage instance segmentation framework
that has an additional head for predicting segmentation masks for each region
extracted from the bounding box. In this paper, we focus specifically on the
Mask R-CNN architecture. QueryInst [10] is also a recent query-based multi-
stage framework that uses sequences of dynamic convolution and multi-head
self-attention blocks to produce more refined instance segmentation masks.
More comprehensive review of non-adaptive instance segmentation methods can be
found in [11].

Adaptive Instance Segmentation: There have been very few works on adaptive
instance segmentation. Most of these works target segmentation of biomedical
entities like nuclei, cells etc. For example, in [21], the authors propose a
multi-step procedure to tackle domain shift that includes inpainting of
images, producing domain-invariant features and a task reweighting scheme to
remove source bias. In [22], the authors extended the framework by adding a
mechanism of feature similarity maximization. Li et al. [17] also proposed
domain adaptation for nuclei segmentation by category-specific feature
alignment and self-training using pseudo-labels. In addition to augmented
pseudo-labelling, Hsu et al. [13] proposed to use a domain separation module
as well as a self-supervised consistency loss. Recently, in [19], the authors
propose domain adaptation for mitotic cells by aligning pixel-level feature
distributions and also additional supervision through a semantic head. For
adaptive human instance segmentation [29] is the only work that is known to
us. In this work, the authors propose to jointly estimate pose and instance
masks of clinicians by adapting from a source dataset. Adaptation is carried
out by a feature normalization strategy and self-training procedure where
pseudo-labels are refined using geometric consistency of augmentations.

Human Instance Segmentation: Human instance segmentation involves segmenting
humans with the aid of additional information like pose. Earlier works that
used pose to segment out human instances include Pose2Seg [36], PersonLab [27]
and Pose2Instance [31]. Pose2Seg is a two-stage framework where keypoint
heatmaps are initially generated, which are then transformed and aggregated to
pass through a segmentation decoder and produce human instance masks.
Similarly, PersonLab detected keypoints but then grouped into masks using a
geometric embedding descriptor. In Pose2Instance, the authors used the
distance transform of keypoints as priors for human masks. More recent works
on human instance segmentation include LSNet [37] and PosePlusSeg [1]. In
LSNet, the authors use pose attention module and keypoint sensitive
combination to aggregate information from multiple sampling points. In the
PosePlusSeg framework, the authors proposed a refinement network for improved
quality of poses and instances obtained from separate heads. In this paper, we
use a more simpler approach where we have multiple heads for pose estimation
and instance segmentation but use pose estimation task to select relevant
features for adaptation and generalization. Additional works on human instance
segmentation include ideas such as self-supervised consistency of human
structures across videos [14], iterative refinement using pose with shape
prior and part attention [40], deformable convolutions with geometric
transformation between keypoint offsets [2].

##  3 Method

We first describe the task of human segmentation using keypoints, and then we
detail our TTA and TTG approaches. Finally, we describe the heuristics and
keypoints head variants we devised to enhance TTA and TTG gains.

###  3.1 Human Segmentation using Keypoints

Segmentation networks may perform poorly especially if the test image exhibits
a large domain shift with respect to the training dataset, e.g., indoor vs.
outdoor or daylight vs. nighttime. For _human_ instance segmentation, such a
domain shift may be due to severe occlusion of human subjects. Figure 2
depicts a case where the baseline network’s mask for the man significantly
spills over to that of the woman. Interestingly, despite the network’s poor
segmentation performance, it estimates the human keypoints fairly well. This
observation motivated us to devise methods for enhancing human instance
segmentation using keypoints estimates. We consider two such methods, i.e.,
TTA and TTG (c.f., Figure 3). Since we need pose estimates, we consider (the
rather common) architecture (e.g., Mask-RCNN) where the network, in addition
to its instance-segmentation head, has a separate keypoints head, with a
common backbone.

Figure 2: High quality keypoints estimates can improve human segmentation
masks (left to right: baseline and TTA Mask-RCNN, where the man’s mask spill-
over has significantly been reduced). Figure 3: Test-time adaptation (top) and
training-time generalization (bottom) frameworks for enhancing human
segmentation using pose estimates.

###  3.2 Test-time Adaptation

Figure 3 (top) depicts our test-time-adaptation method. In this approach we
assume availability of a fully trained human instance segmentation/keypoints
estimation network, i.e.,
$\mathbf{m}=\\{\mathbf{m}_{b},\mathbf{m}_{m},\mathbf{m}_{k}\\}$, and a single
unlabelled test image. We also allow for _test-time_ adaptation of network
weights using backpropagation, but not access to the source dataset. These
assumptions make our TTA setup very realistic, but simultaneously very
challenging.

We give the steps of our TTA method in Algorithm 1. It includes multiple
rounds (e.g., $3$) of weight adaptation, each consisting of converting the
keypoints estimates to pseudo-labels, plugging them in the keypoints loss
together with the keypoint estimates, and backpropagating the resulting self-
supervised loss to adjust the backbone weights. While our method works for any
pseudo-label conversion method, $\mathbf{f}$, and keypoint loss,
$\mathcal{L}_{key}$, we use simple candidates, i.e., for person bounding boxes
scoring above $0.5$, we take keypoints with a minimum probability of $0.05$,
and declare the location in their $56\times 56$ heat-maps with the highest
value as pseudo-labels. We use the multi-category cross-entropy as our
keypoint loss. Finally, adapted segmentation masks are generated by running
the mask head on the adapted feature map.

Given: Model $\mathbf{m}=\\{\mathbf{m}_{b},\mathbf{m}_{m},\mathbf{m}_{k}\\}$,
test-image $x^{tgt}$

Step 1 (test-time): Adapt backbone to test image

Initialize $n$, $i\leftarrow 0$, $\mathbf{m}_{b}^{0}\leftarrow\mathbf{m}_{b}$

For $i<n$

Estimate keypoints, i.e.,
$y_{key}^{i}=\mathbf{m}_{k}\big{(}\mathbf{m}_{b}^{i}(x^{tgt})\big{)}$

Generate pseudo-labels, i.e.,
$\tilde{y}_{key}^{i}=\mathbf{f}\big{(}y_{key}^{i}\big{)}$

Update backbone, i.e., compute $\mathbf{m}_{b}^{i+1}$ by back-

propagating self-supervized
$\mathcal{L}_{key}(y_{key}^{i},\tilde{y}_{key}^{i})$

Save adapted model, i.e,
$\mathbf{m}_{b}^{\text{TTA}}\leftarrow\mathbf{m}_{b}^{n}$

Step 2 (test-time): Get TTA masks for test image

Infer TTA masks, i.e.,
$y_{mask}^{\text{TTA}}=\mathbf{m}_{m}\big{(}\mathbf{m}_{b}^{\text{TTA}}(x^{tgt})\big{)}$

Algorithm 1 TTA (test-time adaptation)

###  3.3 Training-time Generalization

Figure 3 (bottom) shows our training-time generalization method. We assume
availability of a human segmentation and pose estimation network, i.e.,
$\mathbf{m}=\\{\mathbf{m}_{b},\mathbf{m}_{m},\mathbf{m}_{k}\\}$ (which does
not need to be trained), and allow offline access to the labelled source
dataset. We do not, however, assume the availability of any test images or
knowledge about the target domain. We do not allow test-time adaptation of
network weights either. These assumptions make our TTG setup both realistic
and challenging.

We give the steps of our TTG method in Algorithm 2. Training-time
generalization consists of splitting the keypoints head into two subnets,
i.e.,
$\mathbf{m}_{k}=\mathbf{m}_{k}^{\text{reg}}\circ\mathbf{m}_{k}^{\text{fe}}$,
for feature extraction and regression, respectively (the split details are
given in the sequel). The mask head also needs to be modified to accommodate
the extra keypoints features. The TTG model
$\mathbf{m}^{\text{TTG}}=\\{\mathbf{m}_{b},\mathbf{m}_{m}^{\text{TTG}},\mathbf{m}_{k}\\}$
is then trained on the labeled source dataset using the original segmentation
and pose losses. At test-time, the segmentation masks are generated by running
the TTG mask head on the aggregated feature map.

Given: Model $\mathbf{m}=\\{\mathbf{m}_{b},\mathbf{m}_{m},\mathbf{m}_{k}\\}$,
source-dataset $\mathcal{X}^{src}=\\{(x^{src}_{i},t^{src}_{i})\\}$

Step 1 (training-time): Train generalized model on source dataset

Split keypoints head into feature-extractor & regressor

subnets, i.e.,
$\mathbf{m}_{k}=\mathbf{m}_{k}^{\text{reg}}\circ\mathbf{m}_{k}^{\text{fe}}$

Modify mask head, i.e., $\mathbf{m}_{m}^{\text{TTG}}$ to accommodate the

extra keypoints features, i.e.,
$\mathbf{m}_{k}^{\text{fe}}\circ\mathbf{m}_{b}(x^{tgt})$

Train TTG model, i.e.,
$\mathbf{m}^{\text{TTG}}=\\{\mathbf{m}_{b},\mathbf{m}_{m}^{\text{TTG}},\mathbf{m}_{k}\\}$
on

$\mathcal{X}^{src}=\\{(x^{src}_{i},t^{src}_{i})\\}$

Step 2 (test-time): Get TTG masks for test image $x^{tgt}$

Infer TTG masks using aggregated features, i.e.,

$y_{mask}^{\text{TTG}}=$
$\mathbf{m}_{m}^{\text{TTG}}\big{(}\mathbf{m}_{b}(x^{tgt}),\mathbf{m}_{k}^{\text{fe}}\circ\mathbf{m}_{b}(x^{tgt})\big{)}$

Algorithm 2 TTG (training-time generalization)

###  3.4 Heuristics and Keypoints Head Variants

The performance of the TTA and TTG methods greatly depends on the quality of
the pseudo-labels and features generated by the keypoints head, respectively.
This motivates us to devise three keypoints head variants besides the Mask-
RCNN’s original. In the following, we discuss each variant and the heuristics
behind it. Note that the input feature map from the backbone to all these
heads is of dimension $N\times 256\times 14\times 14$, where $N$ is the number
of person bounding boxes.

  * •

Mask-RCNN: $\mathbf{m}_{k}^{\text{fe}}$ consists of $8$ 2d-convolutional
layers, each with $512$ output channels, a $3\times 3$ kernel size and a
stride of $1$. Hence, there are $512$ extra keypoints features.
$\mathbf{m}_{k}^{\text{reg}}$ consists of a 2d-transposed-convolutional layer
with $17$ (i.e., number of keypoints) output channels, a $4\times 4$ kernel
and a stride of $2$, followed by a bilinear upsampler to increase the
keypoints heatmap resolution to $N\times 17\times 56\times 56$.

  * •

Variant1: While the Mask-RCNN keypoints head estimates the keypoint positions,
it does not predict whether they are visible or occluded. This may hurt the
TTA performance, e.g., in Figure 4, the position of the left person’s elbow
has correctly been estimated, however, not specifying it as occluded has
caused parts of the bouquet to be included in the TTA mask. To address this
all variants predict if a keypoint is visible or occluded, e.g., Variant 2 is
identical to Mask-RCNN except that the 2D-transposed-convolutional layer has
$51(=3\times 17)$ outputs to allow two additional outputs per location for
visible/occluded prediction.

  * •

Variant2: As Figure 5 shows, the keypoint estimates from a convolutional head
can be of very low quality, severely impacting the TTA/TTG gains. To enhance
keypoint estimation (through global attention) Variant2 uses transformers,
i.e., $\mathbf{m}_{k}^{\text{fe}}$ consists of a transformer decoder with $6$
layers, $8$ heads and $17$ queries of width $256$, operating on the backbone
feature map. The input queries are trainable parameters, while the output
queries are mapped to keypoints and decoded by a 3-layer MLP to $N\times
51\times 14\times 14$ keypoints heatmap (after reshaping). Hence, there are
$51$ extra keypoints features. $\mathbf{m}_{k}^{\text{reg}}$ consists of a
bilinear upsampler to increase the heatmap resolution to $56\times 56$.

  * •

Variant 3: As will be shown (c.f., Table 1), while Variant 2 shows slightly
better (i.e., $1\%$) $\textbf{AP}_{key}$ numbers on the target datasets, it
lags Mask-RCNN and Variant 1 on the source dataset by a large margin ($4\%$).
This is because in Variant 2, the transformer output queries are directly
decoded to $14\times 14$ heatmaps, without using convolutional layers. Variant
3 gets around this problem by using the last transformer layer’s value
projections, and attention weights to form a separate $15\times 14\times 14$
feature map for each keypoint. Hence there are $(255=17\times 15)$ extra
keypoints features. $\mathbf{m}_{k}^{\text{reg}}$ consists of two group
2D-convolutional and one group 2D-transposed-convolutional layers that further
process keypoint feature maps, independently from one another, followed by a
bilinear upsampler to get the $56\times 56$ heatmaps.

Figure 4: Backpropagating the left person’s elbow pseudo-label, without
specifying it as occluded, degrades TTA performance by inclusion of part of
the bouquet as that person’s mask (left to right: baseline and TTA for Mask-
RCNN). Figure 5: Keypoint estimates from a convolutional head can be of very
low quality.

##  4 Experiments

This section consists of experimental details, TTA vs. TTG comparison results
and ablation studies.

###  4.1 Experimental Details

We report evaluation results on the COCOPersons [36] and OCHuman [36]
datasets. The COCOPersons dataset, which consists of 60K images, is a refined
split obtained from MS-COCO [20]. In this split, non-person categories as well
as person categories with small annotations are removed since they do not
contain keypoint annotations. The OCHuman dataset is highly challenging and
mainly consists of occluded humans, e.g., the average MaxIOU for each person
in COCOPersons is 0.08 while that of OCHuman is 0.67. Both these datasets
contain 17 keypoints.

We use Mask-RCNN [12] for our TTA and TTG experiments, though our methods
apply to any architecture consistent with Figure 3. More specifically, we use
the detectron2 [34] codebase with ResNet-50-FPN and ResNet-101-FPN as
backbones and the mask and keypoints losses implemented therein. The three
variants in this work differ from the standard Mask-RCNN only in their
keypoints head as detailed in Section 3.

We train all models and variants offline on the COCOPersons _train_ set. For
baseline and TTG experiments, we keep model weights frozen throughout
evaluations on COCOPersons _val_ split and OCHuman _val_ and _test_ splits.
For TTA experiments, we reset the weights to their pre-adaptation state before
evaluating each test image. We then adapt the model to the image $n=3$ times
with a TTA learning rate of $1\mathrm{e}{-3}$, after which we segment the
image to get the TTA person masks (c.f. Algorithm 1). This process is repeated
for each test image in the dataset. We report $\textbf{AP}_{key}$ and
$\textbf{AP}_{mask}$ mean and standard deviation across multiple runs ($4$ and
$3$ seeds for ResNet50-FPN and ResNet-101-FPN, respectively).

###  4.2 TTA vs. TTG Results

Table 1 gives $\textbf{AP}_{key}$ numbers for Mask-RCNN and its variants. The
large drop, i.e., more than $30\%$, in Mask-RCNN’s $\textbf{AP}_{key}$ when
moving from COCOPersons _val_ to OCHuman _val_ and _test_ attests to the
significant domain shift between these datasets. Variant $1$ uses the same
keypoints head as Mask-RCNN, hence its numbers are similar. For the source
dataset, i.e., COCOPersons _val_ , Variant $2$ is lagging Mask-RCNN and
Variant $3$’s $\textbf{AP}_{key}$ by $4\%$ to $\%5$, however it shows around
$1\%$ advantage over the target datasets, i.e., OCHuman _val_ and _test_. This
is likely due to Variant $2$ not using convolutional layers and directly
decoding keypoint tokens into spatial heatmaps. Figure 6 depicts the keypoints
estimates from the three variants for a test image. As both Table 1 and Figure
6 suggest, use of transformers does not improve the quality of keypoint
pseudo-labels, which is a major factor limiting TTA gains.

Table 1: Keypoint head’s $\textbf{AP}_{key}$ (ResNet-50-FPN, 4 seeds). The
large difference ($>30\%$) between COCOPersons _val_ and OCHuman _val_ and
_test_ shows the large domain-shifts involved. Variant 2, despite a weaker
performance on source generalizes better ($1\%$) to the target domains, thanks
to using transformers.

Model |  COCO Persons _val_ |  OCHuman _val_ |  OCHuman _test_
---|---|---|---
Mask-RCNN |  64.85 (0.10) |  32.21 (0.32) |  31.91 (0.23)
Variant 1 |  64.87 (0.17) |  32.19 (0.17) |  31.83 (0.32)
Variant 2 |  59.90 (0.11) |  33.45 (0.54) |  32.91 (0.23)
Variant 3 |  63.94 (0.12) |  32.30 (0.31) |  31.67 (0.28)

Figure 6: Keypoints pseudo-label quality is a major limiting factor for TTA.
From left to right: Variant 1, 2 and 3.

Table 2 gives $\textbf{AP}_{mask}$ numbers for various models over the source
dataset, i.e., COCOPersons _val_. Note that while TTG improves
$\textbf{AP}_{mask}$ by around $1\%$ percent, TTA degrades it by about the
same amount. TTG improves $\textbf{AP}_{mask}$ by using extra, i.e., keypoint,
ground-truth labels for training the _segmentation_ head. TTA hurts
$\textbf{AP}_{mask}$ because there is not enough domain shift between
COCOPersons _train_ and _val_ to justify adjusting the model’s already
optimized weights based on a single unlabelled test image.

Table 2: $\textbf{AP}_{mask}$ on COCOPersons _val_ (ResNet-50-FPN, 4 seeds).
Without a large domain-shift TTA hurts performance and TTG shows only a small
gain ($1\%$).

Model |  Baseline |  TTA |  TTG
---|---|---|---
Mask-RCNN |  60.11 (0.12) |  58.59 (0.25) |  61.25 (0.12)
Variant 1 |  60.22 (0.25) |  58.62 (0.15) |  61.45 (0.10)
Variant 2 |  59.55 (0.04) |  58.55 (0.04) |  60.80 (0.15)
Variant 3 |  60.12 (0.15) |  59.02 (0.24) |  60.97 (0.03)

Tables 3 and 4 give $\textbf{AP}_{mask}$ numbers for the target datasets,
i.e., OCHuman _val_ and _test_. Here the domain shift is large enough (i.e.,
more than $40\%$ drop in $\textbf{AP}_{mask}$) to enable TTA to improve
various variants’ performances using a single unlabelled image. The largest
gain is around $1\%$ and corresponds to Variant $2$, due to this variant’s
slightly higher $\textbf{AP}_{key}$. It is noteworthy that TTA has little, if
any benefit, for Mask-RCNN as it does not distinguish between the visible and
occluded keypoints as discussed earlier. More importantly though, the TTG
gains are significantly higher than those offered by TTA, e.g., $3.77\%$ for
TTG compared to TTA’s $1.05\%$ for Variant $2$ on OCHuman _val_. Furthermore,
the TTG gains for all variants are similar, i.e., TTG $\textbf{AP}_{mask}$
numbers are within a standard deviation from one another. This is because TTG
does not use the keypoint _estimates_ directly; it instead uses the keypoint
_features_ that are richer in information, e.g., implicitly infer keypoints’
visibility/occlusion. It also reconfirms that TTG gain comes from using the
extra keypoint ground-truth labels for training the segmentation head and not
from the details of the keypoint head architecture.

Table 3: $\textbf{AP}_{mask}$ on OCHuman _val_ (ResNet-50-FPN, 4 seeds). With
a large domain-shift TTA improves all variants, however TTA gains are smaller
than TTG, and more sensitive to keypoint head’s architecture.

Model |  Baseline |  TTA |  TTG
---|---|---|---
Mask-RCNN |  17.74 (0.25) |  17.66 (0.15) |  21.48 (0.10)
Variant 1 |  17.93 (0.19) |  18.00 (0.19) |  21.56 (0.17)
Variant 2 |  18.15 (0.27) |  18.79 (0.28) |  21.51 (0.51)
Variant 3 |  18.41 (0.29) |  18.44 (0.29) |  21.20 (0.14)

Table 4: $\textbf{AP}_{mask}$ on OCHuman _test_ (ResNet-50-FPN, 4 seeds). Same
trends as Table 3, i.e., TTA gains are smaller and more heuristic dependent
than TTG.

Model |  Baseline |  TTA |  TTG
---|---|---|---
Mask-RCNN |  17.48 (0.15) |  17.51 (0.15) |  21.30 (0.18)
Variant 1 |  17.42 (0.33) |  17.69 (0.22) |  21.09 (0.07)
Variant 2 |  17.92 (0.21) |  18.50 (0.11) |  20.93 (0.08)
Variant 3 |  17.99 (0.06) |  18.04 (0.12) |  20.64 (0.21)

Tables 5, 6 and 7 give our results for ResNet-101-FPN backbone, and show very
similar trends, i.e., in the absence of a significant domain shift, TTA may
hurt and TTG show only a small gain (c.f., Table 5), whereas with a large
enough domain shift, TTA gains are smaller and more dependent on the
heuristics used, while TTG gains are larger and less sensitive to model
variations (c.f., Tables 6, 7).

Table 5: $\textbf{AP}_{mask}$ on COCOPersons _val_ (ResNet-101-FPN, 3 seeds).
Same trends as ResNet-50-FPN, i.e., without a large domain-shift, TTA hurts
and TTG gives a small gain ($1\%$).

Model |  Baseline |  TTA |  TTG
---|---|---|---
Mask-RCNN |  60.87 (0.10) |  59.22 (0.21) |  61.73 (0.09)
Variant 1 |  60.93 (0.10) |  59.14 (0.10) |  61.82 (0.02)
Variant 2 |  60.71 (0.15) |  59.49 (0.06) |  61.69 (0.12)
Variant 3 |  61.15 (0.10) |  59.70 (0.28) |  61.81 (0.19)

Table 6: $\textbf{AP}_{mask}$ on OCHuman _val_ (ResNet-101-FPN, 3 seeds). Same
trends as with ResNet-50-FPN, i.e., TTG gains are larger and less dependent on
pose head’s architectural nuances.

Model |  Baseline |  TTA |  TTG
---|---|---|---
Mask-RCNN |  19.21 (0.36) |  19.58 (0.23) |  22.67 (0.35)
Variant 1 |  19.10 (0.40) |  19.45 (0.40) |  22.60 (0.23)
Variant 2 |  19.32 (0.38) |  20.30 (0.17) |  22.51 (0.32)
Variant3 |  19.77 (0.18) |  20.46 (0.25) |  22.50 (0.15)

Table 7: $\textbf{AP}_{mask}$ on OCHuman _test_ (ResNet-101-FPN, 3 seeds).
Same trends as with ResNet-50-FPN, i.e., TTG gains are larger and more
consistent across various variants.

Model |  Baseline |  TTA |  TTG
---|---|---|---
Mask-RCNN |  18.76 (0.02) |  18.97 (0.28) |  21.95 (0.54)
Variant 1 |  18.71 (0.27) |  18.93 (0.23) |  22.42 (0.22)
Variant 2 |  18.92 (0.19) |  19.92 (0.11) |  22.09 (0.58)
Variant 3 |  19.68 (0.05) |  20.26 (0.28) |  22.27 (0.38)

Table 8 compares the TTG $\textbf{AP}_{mask}$ for Variant 2 against some of
the existing methods in the literature as described in Section 2 (numbers are
cited from the referenced papers). While an apple to apple comparison is
difficult due to the nuances in each paper setup, we observe that our simple
TTG method provides a competitive performance especially noting that we report
_average_ values across multiple seeds. This reconfirms our observation that
TTG gains are less dependent on the heuristics used and more robust to
architectural variations.

Table 8: Comparison of TTG $\textbf{AP}_{mask}$ for Variant 2 against methods
from the literature.

Model | Backbone |  COCO Persons _val_ |  OCHuman _val_ |  OCHuman _test_
---|---|---|---|---
Mask-RCNN [12] | ResNet-50-FPN |  53.2 |  16.3 |  16.9
Pose2Seg [36] | ResNet-50-FPN |  55.5 |  22.2 |  23.8
CondInst [30] | ResNet-50-FPN |  54.8 |  20.3 |  20.1
YOLACT [3] | ResNet-101-FPN |  50.2 |  13.2 |  13.5
Variant 2 (ours) | ResNet-50-FPN |  60.8 |  21.5 |  20.9
Variant 2 (ours) | ResNet-101-FPN |  61.7 |  22.5 |  22.1

###  4.3 Ablation Studies

In this section we report several ablation studies that shed light on some of
the factors limiting the performance of test-time adaptation. Tables 9, 10 and
11 give the TTA $\textbf{AP}_{mask}$ for Variant 2 on OCHuman _val_ when
sweeping the _adaptation_ learning-rate, min-person-score (i.e., threshold for
rejecting all keypoints for a detected person) and min-keypoint-prob (i.e.,
threshold for rejecting an individual keypoint), respectively. As the tables
show, our choices for these parameters (i.e., $1\mathrm{e}{-3}$, $0.5$ and
$0.05$, respectively) lie within the relevant sweet-spots, however, since we
do not assume any knowledge about the target domain, we cannot use the
_optimal_ values (as Table 9 indicates a learning-rate of $1\mathrm{e}{-3}$ is
not optimal) and have to rely on judgment calls which is an important limiting
factor. For example as Figure 7 shows, increasing the min-person-score from
$0.5$ to $0.8$ removes some of the false-positives from the volleyball scene,
but also causes some true-positives to be missed in the ice-skating scene.

Table 9: TTA $\textbf{AP}_{mask}$ on OCHuman _val_ with different learning
rates (ResNet-50-FPN, 4 seeds).

Model | Lr
---|---
0.5e-3 | 1e-3 | 2e-3 | 4e-3
Variant 2 |  18.49 (0.29) |  18.75 (0.29) |  18.92 (0.21) |  18.57 (0.38)

Table 10: TTA $\textbf{AP}_{mask}$ on OCHuman _val_ with different min-person-
score (ResNet-50-FPN, 4 seeds).

Model | Min. Person Score
---|---
0.5 | 0.6 | 0.7 | 0.8
Variant 2 |  18.78 (0.25) |  18.70 (0.27) |  18.74 (0.26) |  18.70 (0.31)

Table 11: TTA $\textbf{AP}_{mask}$ on OCHuman _val_ with different min-
keypoint-prob (ResNet-50-FPN, 4 seeds).

Model | Min. Keypoint Prob.
---|---
0.05 | 0.1 | 0.2 | 0.4
Variant 2 |  18.75 (0.30) |  18.54 (0.25) |  18.29 (0.25) |  18.08 (0.30)

Figure 7: Increasing the min-person-score from $0.5$ to $0.8$ removes some
false-positives from the volleyball scene (first and second from left,
respectively), but also causes some true-positives to be missed in the ice-
skating scene (third and fourth).

We note that dynamic selection of hyper-parameters such as TTA learning-rate,
depending on the test image at hand, may significantly improve the TTA
performance, e.g., for the particular image shown in Figure 8, a learning rate
of $0.5\mathrm{e}{-3}$ greatly cleans up the spill-over of the man’s mask.
Whereas a larger learning rate of $4\mathrm{e}{-3}$ degrades the performance
by shrinking the woman’s mask. We also note that such dynamic hyper-parameter
selection is impractical in the absence of prior knowledge about the target
domain (i.e., another TTA limiting factor).

Figure 8: Dynamic selection of hyper-parameters, based on the test image, may
improve TTA gain significantly, but is impractical in the absence of prior
knowledge about the target domain. From left to right: baseline and test-time-
adapted Mask-RCNN for TTA learning rates of $0.5\mathrm{e}{-3}$, and
$4\mathrm{e}{-3}$, respectively. For this test image a learning rate of
$0.5\mathrm{e}{-3}$ greatly cleans up the spill-over of the man’s mask. A
larger learning rate of $4\mathrm{e}{-3}$ degrades the performance by
shrinking the woman’s mask.

Table 12 reports numbers for the scenario where the backbone is only partially
adaptable (i.e., stage $5$ of ResNet-50-FPN). As the table shows, for smaller
learning rates the partially adapted backbone yields smaller TTA gains
compared to the fully-adaptable one. However for the large learning-rate of
$4\mathrm{e}{-3}$ it retains more of the gain which is consistent with being
partially adaptable, hence more regularized.

Table 12: TTA $\textbf{AP}_{mask}$ on OCHuman _val_ with backbone fully or
partially adapted (ResNet-50-FPN, 4 seeds).

Model | Lr
---|---
0.5e-3 | 1e-3 | 2e-3 | 4e-3
Variant 2 w. backbone fully adapted |  18.49 (0.29) |  18.75 (0.29) |  18.92 (0.21) |  18.57 (0.38)
Variant 2 w. backbone’s last-stage adapted |  18.34 (0.37) |  18.57 (0.39) |  18.85 (0.28) |  18.83 (0.35)

Finally, Table 13 gives $\textbf{AP}_{mask}$ numbers for the case where a TTG
model is test-time adapted. We note that TTA gains, if any, are even smaller
as TTG already takes advantage of the features generated by the keypoints
head.

Table 13: $\textbf{AP}_{mask}$ on OCHuman _val_ with both TTG and TTA
(ResNet-50-FPN, 4 seeds). Applying TTA on top of TTG has minimal gain, if any,
as TTG already uses keypoint features.

Model |  Baseline |  TTG |  TTG+TTA
---|---|---|---
Mask-RCNN |  17.74 (0.25) |  21.48 (0.10) |  21.37 (0.22)
Variant 1 |  17.93 (0.19) |  21.56 (0.17) |  21.77 (0.27)
Variant 2 |  18.15 (0.27) |  21.51 (0.51) |  21.70 (0.41)
Variant 3 |  18.41 (0.29) |  21.20 (0.14) |  20.96 (0.43)

##  5 Conclusion

We compared two approaches for enhancing human segmentation masks using
keypoints estimation. First was test-time adaptation (TTA), where we allowed
test-time adjustments of network weights based on the unlabeled test image
without access to the labeled source dataset. It worked by back-propagating
keypoints estimates, as pseudo-labels, to adjust the backbone weights. The
second approach was training-time generalization (TTG), where we allowed
offline access to the labeled source dataset but no test-time alteration of
weights. We further did not assume the availability of any test images. Our
TTG method worked by augmenting the backbone features with that of the
keypoints head and using it to infer masks. We also devised three additional
pose-head variants to improve keypoint pseudo-label quality, using keypoint
visibility/occlusion prediction, and use of global-attention (i.e.,
transformers).

We evaluated both approaches and, through ablations, identified factors
limiting the TTA gains, i.e., we showed that without a large domain shift, TTA
hurts performance, and TTG shows only a small gain. In contrast, for a large
domain shift, TTA gains are smaller and more heuristics dependent, while TTG’s
are larger and more robust.

## References

  * [1] Niaz Ahmad, Jawad Khan, Jeremy Yuhyun Kim, and Youngmoon Lee.  Joint human pose estimation and instance segmentation with poseplusseg.  Proceedings of the AAAI Conference on Artificial Intelligence, 36(1):69–76, Jun. 2022.
  * [2] Yang Bai and Weiqiang Wang.  Acpnet: anchor-center based person network for human pose estimation and instance segmentation.  In 2019 IEEE International Conference on Multimedia and Expo (ICME), pages 1072–1077. IEEE, 2019.
  * [3] Daniel Bolya, Chong Zhou, Fanyi Xiao, and Yong Jae Lee.  Yolact: Real-time instance segmentation.  In Proceedings of the IEEE/CVF international conference on computer vision, pages 9157–9166, 2019.
  * [4] Kai Chen, Jiangmiao Pang, Jiaqi Wang, Yu Xiong, Xiaoxiao Li, Shuyang Sun, Wansen Feng, Ziwei Liu, Jianping Shi, Wanli Ouyang, et al.  Hybrid task cascade for instance segmentation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4974–4983, 2019.
  * [5] Yilun Chen, Zhicheng Wang, Yuxiang Peng, Zhiqiang Zhang, Gang Yu, and Jian Sun.  Cascaded pyramid network for multi-person pose estimation.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7103–7112, 2018.
  * [6] Bowen Cheng, Ishan Misra, Alexander G Schwing, Alexander Kirillov, and Rohit Girdhar.  Masked-attention mask transformer for universal image segmentation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1290–1299, 2022.
  * [7] Lutao Chu, Yi Liu, Zewu Wu, Shiyu Tang, Guowei Chen, Yuying Hao, Juncai Peng, Zhiliang Yu, Zeyu Chen, Baohua Lai, et al.  Pp-humanseg: Connectivity-aware portrait segmentation with a large-scale teleconferencing video dataset.  In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 202–209, 2022.
  * [8] Jifeng Dai, Kaiming He, Yi Li, Shaoqing Ren, and Jian Sun.  Instance-sensitive fully convolutional networks.  In European conference on computer vision, pages 534–549. Springer, 2016.
  * [9] Hao-Shu Fang, Shuqin Xie, Yu-Wing Tai, and Cewu Lu.  Rmpe: Regional multi-person pose estimation.  In Proceedings of the IEEE international conference on computer vision, pages 2334–2343, 2017.
  * [10] Yuxin Fang, Shusheng Yang, Xinggang Wang, Yu Li, Chen Fang, Ying Shan, Bin Feng, and Wenyu Liu.  Instances as queries.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 6910–6919, 2021.
  * [11] Wenchao Gu, Shuang Bai, and Lingxing Kong.  A review on 2d instance segmentation based on deep neural networks.  Image and Vision Computing, page 104401, 2022.
  * [12] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick.  Mask r-cnn.  In Proceedings of the IEEE international conference on computer vision, pages 2961–2969, 2017.
  * [13] Joy Hsu, Wah Chiu, and Serena Yeung.  Darcnn: Domain adaptive region-based convolutional neural network for unsupervised instance segmentation in biomedical images.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1003–1012, 2021.
  * [14] Yalong Jiang, Wenrui Ding, Hongguang Li, Hua Yang, and Xu Wang.  A self-supervised framework for human instance segmentation.  In European Conference on Computer Vision, pages 479–495. Springer, 2020.
  * [15] Ziyu Jiang, Zhenhua He, Xueqin Huang, Zibin Yang, and Pearl Tan.  Ce-peopleseg: Real-time people segmentation with 10% cpu usage for video conference.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 914–922, 2021.
  * [16] Muhammed Kocabas, Salih Karagoz, and Emre Akbas.  Multiposenet: Fast multi-person pose estimation using pose residual network.  In Proceedings of the European conference on computer vision (ECCV), pages 417–433, 2018.
  * [17] Canran Li, Dongnan Liu, Haoran Li, Zheng Zhang, Guangming Lu, Xiaojun Chang, and Weidong Cai.  Domain adaptive nuclei instance segmentation and classification via category-aware feature alignment and pseudo-labelling.  arXiv preprint arXiv:2207.01233, 2022.
  * [18] Yaoyi Li and Hongtao Lu.  Natural image matting via guided contextual attention.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 11450–11457, 2020.
  * [19] Yonghui Li, Yao Xue, Liangfu Li, Xingjun Zhang, and Xueming Qian.  Domain adaptive box-supervised instance segmentation network for mitosis detection.  IEEE Transactions on Medical Imaging, 2022.
  * [20] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick.  Microsoft coco: Common objects in context.  In European conference on computer vision, pages 740–755. Springer, 2014.
  * [21] Dongnan Liu, Donghao Zhang, Yang Song, Fan Zhang, Lauren O’Donnell, Heng Huang, Mei Chen, and Weidong Cai.  Unsupervised instance segmentation in microscopy images via panoptic domain adaptation and task re-weighting.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4243–4252, 2020.
  * [22] Dongnan Liu, Donghao Zhang, Yang Song, Fan Zhang, Lauren O’Donnell, Heng Huang, Mei Chen, and Weidong Cai.  Pdam: A panoptic-level feature alignment framework for unsupervised domain adaptive instance segmentation in microscopy images.  IEEE Transactions on Medical Imaging, 40(1):154–165, 2020.
  * [23] Shu Liu, Lu Qi, Haifang Qin, Jianping Shi, and Jiaya Jia.  Path aggregation network for instance segmentation.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 8759–8768, 2018.
  * [24] Fengling Mao, Bingpeng Ma, Hong Chang, Shiguang Shan, and Xilin Chen.  Learning efficient text-to-image synthesis via interstage cross-sample similarity distillation.  Science China Information Sciences, 64(2):1–12, 2021.
  * [25] Jiayuan Mao, Tete Xiao, Yuning Jiang, and Zhimin Cao.  What can help pedestrian detection?  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3127–3136, 2017.
  * [26] Wanli Ouyang, Hui Zhou, Hongsheng Li, Quanquan Li, Junjie Yan, and Xiaogang Wang.  Jointly learning deep features, deformable parts, occlusion and classification for pedestrian detection.  IEEE transactions on pattern analysis and machine intelligence, 40(8):1874–1887, 2017.
  * [27] George Papandreou, Tyler Zhu, Liang-Chieh Chen, Spyros Gidaris, Jonathan Tompson, and Kevin Murphy.  Personlab: Person pose estimation and instance segmentation with a bottom-up, part-based, geometric embedding model.  In Proceedings of the European conference on computer vision (ECCV), pages 269–286, 2018.
  * [28] Tingting Qiao, Jing Zhang, Duanqing Xu, and Dacheng Tao.  Learn, imagine and create: Text-to-image generation from prior knowledge.  Advances in neural information processing systems, 32, 2019.
  * [29] Vinkle Srivastav, Afshin Gangi, and Nicolas Padoy.  Unsupervised domain adaptation for clinician pose estimation and instance segmentation in the operating room.  Medical Image Analysis, 80:102525, 2022.
  * [30] Zhi Tian, Chunhua Shen, and Hao Chen.  Conditional convolutions for instance segmentation.  In Computer Vision – ECCV 2020, pages 282–298, Cham, 2020.
  * [31] Subarna Tripathi, Maxwell Collins, Matthew Brown, and Serge Belongie.  Pose2instance: Harnessing keypoints for person instance segmentation.  arXiv preprint arXiv:1704.01152, 2017.
  * [32] Andre Ückermann, Robert Haschke, and Helge Ritter.  Realtime 3d segmentation for human-robot interaction.  In 2013 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 2136–2143. IEEE, 2013.
  * [33] Xinlong Wang, Tao Kong, Chunhua Shen, Yuning Jiang, and Lei Li.  Solo: Segmenting objects by locations.  In European Conference on Computer Vision, pages 649–665. Springer, 2020.
  * [34] Yuxin Wu, Alexander Kirillov, Francisco Massa, Wan-Yen Lo, and Ross Girshick.  Detectron2.  https://github.com/facebookresearch/detectron2, 2019.
  * [35] Jingwen Ye, Yongcheng Jing, Xinchao Wang, Kairi Ou, Dacheng Tao, and Mingli Song.  Edge-sensitive human cutout with hierarchical granularity and loopy matting guidance.  IEEE Transactions on Image Processing, 29:1177–1191, 2019.
  * [36] Song-Hai Zhang, Ruilong Li, Xin Dong, Paul Rosin, Zixi Cai, Xi Han, Dingcheng Yang, Haozhi Huang, and Shi-Min Hu.  Pose2seg: Detection free human instance segmentation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 889–898, 2019.
  * [37] Xiangzhou Zhang, Bingpeng Ma, Hong Chang, Shiguang Shan, and Xilin Chen.  Location sensitive network for human instance segmentation.  IEEE Transactions on Image Processing, 30:7649–7662, 2021.
  * [38] Ziyu Zhang, Sanja Fidler, and Raquel Urtasun.  Instance-level segmentation for autonomous driving with deep densely connected mrfs.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 669–677, 2016.
  * [39] Xiaomei Zhao, Fulin Tang, and Yihong Wu.  Real-time human segmentation by bowtienet and a slam-based human ar system.  Virtual Reality & Intelligent Hardware, 1(5):511–524, 2019.
  * [40] Desen Zhou and Qian He.  Poseg: Pose-aware refinement network for human instance segmentation.  IEEE Access, 8:15007–15016, 2020.
  * [41] Qinqin Zhou, Bineng Zhong, Yulun Zhang, Jun Li, and Yun Fu.  Deep alignment network based multi-person tracking with occlusion and motion reasoning.  IEEE Transactions on Multimedia, 21(5):1183–1194, 2018.
  * [42] Bingke Zhu, Yingying Chen, Jinqiao Wang, Si Liu, Bo Zhang, and Ming Tang.  Fast deep matting for portrait animation on mobile phone.  In Proceedings of the 25th ACM international conference on Multimedia, pages 297–305, 2017.


# IMLE-Net: An Interpretable Multi-level Multi-channel Model for ECG
Classification

Likith Reddy  Vivek Talwar  Shanmukh Alle  Raju. S. Bapi  U. Deva Priyakumar
Likith Reddy1, Vivek Talwar1, Shanmukh Alle2, Raju. S. Bapi3, U. Deva
Priyakumar2
1International Institute of Information Technology, Healthcare and AI Center,
Hyderabad
2International Institute of Information Technology, CCNSB Lab, Hyderabad
3International Institute of Information Technology, Cognitive Science Lab,
Hyderabad
Email<EMAIL_ADDRESS>
###### Abstract

Early detection of cardiovascular diseases is crucial for effective treatment
and an electrocardiogram (ECG) is pivotal for diagnosis. The accuracy of Deep
Learning based methods for ECG signal classification has progressed in recent
years to reach cardiologist-level performance. In clinical settings, a
cardiologist makes a diagnosis based on the standard 12-channel ECG recording.
Automatic analysis of ECG recordings from a multiple-channel perspective has
not been given enough attention, so it is essential to analyze an ECG
recording from a multiple-channel perspective. We propose a model that
leverages the multiple-channel information available in the standard
12-channel ECG recordings and learns patterns at the beat, rhythm, and channel
level. The experimental results show that our model achieved a macro-averaged
ROC-AUC score of 0.9216, mean accuracy of 88.85% and a maximum F1 score of
0.8057 on the PTB-XL dataset. The attention visualization results from the
interpretable model are compared against the cardiologist’s guidelines to
validate the correctness and usability.

##  I Introduction

Heart diseases are the leading cause of death globally. It is estimated that
17.9 million people died of heart diseases in the year 2016 which represents
about 31% of global human mortality according to the World Health
Organization. Early detection of heart disease is crucial for effective
treatment and reducing mortality. Electrocardiography is an inexpensive,
simple and non-invasive procedure that helps us to understand the functioning
of a heart which in turn helps us diagnose heart diseases. An
Electrocardiogram (ECG) records information about the electrical activity of
the heart based on which a cardiologist can identify abnormal functioning of
the heart to diagnose various conditions. But this process of analyzing an ECG
recording is time-consuming that requires a trained expert’s time and
attention. Moreover, it is prone to human error. Misdiagnosis of early signs
of heart diseases by cardiologists from an ECG recording is still a major
concern [1]. So work has been done to automate the ECG signal classification
to aid in human diagnosis.

Most existing approaches use traditional machine learning-based methods for
the detection of heart diseases from a patient’s ECG recording [2, 3]. Such
methods involve manually crafted feature extraction and the use of classifiers
on the extracted features. Asgari et al. [4] use the stationary wavelet
transform to extract features and a support vector machine (SVM) for the
detection of atrial fibrillation. Poddar et al. [5] employed the heart rate
variability (HRV) features computed in time-domain, frequency-domain and using
non-linear methods, the features are then classified using k-nearest neighbor
(KNN) and SVM classifiers. Unsupervised methods using time series clustering
were proposed in Annam et al. [6].

The advent of deep learning-based methods for ECG signal classification has
progressed in recent years to reach cardiologist-level performance [7]. Such
methods analyze ECG data without manual feature extraction in contrast to
traditional machine learning approaches. The work done by Rajpurkar et al. [7]
uses a time series ECG signal as an input to a 34-layer eural network (CNN) by
mapping a sequence of ECG samples to a sequence of rhythm classes. The model
exceeded the average cardiologist performance in both precision and recall.
Mousavi et al. [8] developed an automatic heartbeat classification method
using deep convolutional neural networks and sequence-to-sequence models.
Murugesan et al. [9] proposed a combination of CNNs and a long short-term
memory (LSTM) based feature extractor that can be directly trained without any
preprocessing. Recurrent neural network (RNN) along with an attention
mechanism is used by Schwab et al. [10] for the classification of a single-
channel ECG recording.

An interpretable model along with reliable performance is crucial to instil
confidence in medical practitioners to use computer-assisted
electrocardiography. A lot of work has been done to improve the
interpretability of deep learning models in computer vision tasks but the
progress made on interpreting ECG classification models has been limited and
has been gaining interest recently [11, 12]. Vijayarangan et al. [13] use a
gradient-weighted class activation map [14] for visualizing saliency on
single-channel ECG signals using their proposed model. Attention models have
been utilized by Hong et al. [15] to improve interpretability in the time and
frequency domains.

The majority of the interpretable deep learning work focuses on models which
deal with single-channel ECG recordings. A single-channel ECG recording is
used for basic heart monitoring whereas a multi-channel ECG recording provides
information about the 3-dimensional electrical activity of the heart. From a
clinical point of view, a cardiologist makes a diagnosis based on the standard
12-channel ECG recording [16]. When compared to a single-channel recording, a
12-channel recording accurately reproduces various features of an ECG such as
QRS complex, ST-segment and T waveforms which sometimes are poorly represented
in a single-channel recording [17]. For example, ST-segment elevation which is
an important characteristic for Myocardial Infarction (MI) is best identified
using a 12-channel recording and the localization of myocardial injury cannot
be identified without a 12-channel recording. Analysis of ECG recordings from
a multiple-channel perspective has not been given enough attention and it is
essential to analyze an ECG recording from a multi-channel perspective. To
address this gap, we propose a model that leverages the multi-channel
information available in a multi-channel ECG recording. Also, we make use of
the hierarchical structure of the ECG signal to learn patterns at the beat,
rhythm and channel level. The model is then evaluated on the PTB-XL dataset
[18] which comprises of a total 21837 clinical 12-channel ECG recordings of 10
seconds length from 18885 patients. The performance of the model is then
compared against the existing deep learning methods. The interpretability of
our model is evaluated by comparing the attention visualizations with the
cardiologist’s rules for identifying the subtypes of MI. We perform an
ablation study by reducing the number of available channels in an input ECG
recording to evaluate the robustness of the proposed model. The source code of
the proposed model and all the experiments performed is made publicly
available to ensure reproducibility for future
research111https://github.com/likith012/IMLE-Net.

##  II Methodology

###  II-A Preprocessing

In preprocessing, a multi-channel ECG signal is segmented into beat segments
of length $W$ for every individual channel. A popular approach to segment an
ECG signal into beat segments is based on first detecting the R-peaks in an
ECG signal using the Pan-Tompkins algorithm [19] and then using the adaptive
searching window for segmenting beats which is used in Mousavi et al. [20].
Other approaches for the detection of R-peaks include standard wavelet
transform [21] and Hamilton [22]. But these approaches perform poorly in the
presence of signal noise and HRV irregularities. Instead, we opted for a
sliding window approach with no overlap which was used by Hong et al. [15] to
segment beats of the ECG signal. To get the $k$th beat segment, the beat is
spanned from $(k-1)\times W$ to $k\times W$ over the ECG signal, where $W$ is
the length of the window.

###  II-B Architecture

The proposed model can be divided into two parts, the first part processes
each channel separately to generate a channel encoding for an input ECG
recording. The architecture of the first part is shown in Figure 1. The
segmented beats from a multi-channel ECG recording are passed through a beat
level block, the outputs from the beat level block are passed through a rhythm
level block to get channel-wise encodings. These channel-wise encodings are
passed into the second part of the model which pools these encodings from each
channel and generates predictions as shown in Figure 2.

Figure 1: Architecture of the model at each individual channel Figure 2:
Channel level architecture of the model

####  II-B1 Beat Level Block

The beat level block involves the processing of input beat segments. The beats
are first passed through a CNN and then an attention layer.

The architecture of the CNN used in the beat level block is shown in Figure 3.
Inspired by the ResNet architecture [23], the CNN uses a total of 6 residual
blocks. Skip connections in a residual block improve backpropagation in deep
neural networks which optimize model training. In a single residual block,
there are two convolutional layers and every convolutional layer is followed
by a batch normalization layer and a ReLU activation function. Between two
convolutional layers, we employ dropout to stabilise model training and to
avoid overfitting. No downsampling is done in the first two residual blocks.
In the remaining residual blocks, inputs are downsampled by a factor of two
after every alternate residual block. All the convolutional layers have a
filter length of 8 with 32 starting filters. In every residual block where
downsampling of inputs occurs, the number of filters is doubled. All the
weights in a CNN are shared across all the beat segments and across multiple
channels in the model.

Let $T$ be the length of an ECG signal for the $c$th channel and $W$ be the
beat length. The total number of beat segments are given by $N=\frac{T}{W}$.
Let $(b_{1}^{c},b_{2}^{c},b_{3}^{c},...b_{N}^{c})$ be the beat segments for
the $c$th channel of an ECG recording. The CNN takes in a time series of $k$th
beat segment $b_{k}^{c}$ of length $W$. The output from the CNN with input as
$b_{k}^{c}$ beat segment be $\bar{b}_{k}^{c}$ with an intermediate sequence
length of $\bar{W}$.

We know that all the areas within a heartbeat segment are not equally relevant
in identifying an abnormal heartbeat. The attention mechanism helps in
identifying more important areas within a beat segment by giving higher
attention scores to them. It is a two-layer neural network that takes inputs
from the CNN outputs and gets the attention scores by computing softmax over
the outputs from the two-layer neural network as given in eqs. 1 and 2. The
weights of the beat level attention layer are shared across all beat segments
similar to CNN.

${\centering\alpha_{k}^{c}=softmax(V_{b}\tanh({W_{b}\bar{b}_{k}^{c}+b_{b}}))\@add@centering}$
(1)
$\centering{B_{k}^{c}=\sum_{i=1}^{\bar{W}}\alpha_{k,i}^{c}\bar{b}_{k,i}^{c}}\@add@centering$
(2)

Where, $V_{b}$, $W_{b}$ and $b_{b}$ are beat level attention parameters that
are learned during training. $\alpha_{k}^{c}$ is the beat level attention
score for the $k$th beat segment in an ECG signal for the $c$th channel.
$B_{k}^{c}$ is the beat context vector for the $k$th beat segment in an ECG
signal for the $c$th channel. The beat context vector $B_{k}^{c}$ is a
weighted sum of $\bar{b}_{k,i}^{c}$ with weights given by attention scores
$\alpha_{k,i}^{c}$.

Figure 3: Architecture of the CNN used at the beat level.

####  II-B2 Rhythm Level Block

The rhythm level block deals with the input ECG rhythm signal which is
constructed from several beat segments. It consists of two layers, a Bi-
directional LSTM and a rhythm level attention layer. Bi-directional LSTM is
used as the current state can process both forward and backward directions of
the input sequence information. The input to the Bi-LSTM is a sequence of beat
context vectors $B_{k}^{c}$ for the $c$th ECG channel with $N$ beat sequences.
In the Bi-LSTM, the forward and the backward direction hidden states for the
current step $k$ are concatenated to obtain output $r_{k}^{c}$.

The output from the Bi-LSTM $r^{c}$ with a sequence length of $N$ for the
$c$th channel is passed through a rhythm attention layer. The rhythm attention
mechanism used here identifies the important beat segments within an ECG
rhythm by giving higher attention scores to them and is given by eqs. 3 and 4.

${\centering\beta^{c}=softmax(V_{r}\tanh({W_{r}r^{c}+b_{r}}))\@add@centering}$
(3) ${R^{c}=\sum_{i=1}^{N}\beta_{i}^{c}r_{i}^{c}}$ (4)

Where, $V_{r}$, $W_{r}$ and $b_{r}$ are rhythm level attention parameters that
are learned through training and $\beta^{c}$ is the rhythm level attention
score for the $c$th channel. $R^{c}$ is the rhythm context vector for the
$c$th channel which encodes entire ECG signal for a particular channel into a
vector (Channel Encoding).

####  II-B3 Channel Level Block

The channel level block processes the information present in several ECG
channels. The information present in individual ECG channels differ. The
characteristic information for diagnosing a particular condition might be
present only in few channels. So we need a mechanism to not treat each channel
equally always and give more importance to the channels that hold the
characteristics for a particular heart disease. So a channel-level attention
mechanism is used here which receives inputs $(R^{1},R^{2},...R^{c}...R^{M})$,
where $M$ is the total number of channels available in an ECG recording. The
channel level attention scores are calculated similar to beat and rhythm level
attention given by eqs. 5 and 6.

$\centering{\gamma=softmax(V_{c}\tanh(W_{c}R^{c}+b_{c}})\@add@centering$ (5)
$\centering{C=\sum_{i=1}^{M}\gamma_{i}R_{i}^{c}}\@add@centering$ (6)

Where, $V_{c}$, $W_{c}$ and $b_{c}$ are the channel level parameters that are
learned during training. $\gamma$ is the channel level attention score and $C$
is a context vector that encodes the entire ECG signal for all the available
channels. The context vector $C$ is passed through a linear neural network for
final predictions.

##  III Experiments

###  III-A Dataset

The dataset used is the PTB-XL dataset [18] which is the largest openly
available dataset that provides clinical 12 channel ECG waveforms. It
comprises 21837 ECG records from 18885 patients of 10 seconds length which
follow the standard set of channels (I, II, III, aVL, aVR, aVF, V1–V6). The
dataset is balanced concerning sex with 52% male and 48% female and covers age
ranging from 0 to 95 years. The dataset covers a wide range of pathologies
with many different co-occurring diseases. The ECG waveform records are
annotated by two certified cardiologists. Each ECG record has labels assigned
out of a set of 71 different statements conforming to the Standard
communications protocol for computer assisted electrocardiography (SCP-ECG)
standard. The ECG waveform was originally recorded at a sampling rate of 400
Hz and downsampled to 100 Hz. All the experiments in our work were performed
using the 100 Hz sampling rate.

###  III-B Evaluation

We select the samples from the PTB-XL dataset [18] in a way to cover a huge
cohort of diseases and use it to evaluate the model. Each SCP statement
annotated in the ECG recording belongs to one of the groups of form, rhythm,
and diagnostic statement groups, with a few SCP statement annotations
belonging to both rhythm and diagnostic statements. For evaluating the model
in our work only the diagnostic statements are considered, a total of 21430
ECG recordings are present in the dataset with at least one diagnostic
statement. A specific ECG recording can have multiple diagnostic statements
annotated against it. For the diagnostic statement labels, hierarchical groups
are provided in terms of 5 superclasses and 24 subclasses. In our experiments,
we have used the 5 superclass labels in which the ECG recordings in a
particular superclass belong to the same family of pathologies. The superclass
disease groups are Normal ECG (NORM), Conduction Disturbance (CD), Myocardial
Infarction (MI), Hypertrophy (HYP), and ST/T changes (STTC). Splitting of the
dataset for training and testing is based on the stratified folds provided by
the dataset. Each ECG recording is assigned to one of the ten folds, folds 1
to 8 are used for training the model, the 9th fold is used as the validation
set and the 10th fold is used as a test set. The train and test sets are
standardized before training.

The proposed model is compared against Resnet 101 model [23], Mousavi et al.
[8], ECGNet [9] and Rajpurkar et al. [7]. The performance of the model is
evaluated using metrics such as the mean accuracy, macro averaged Area under
Receiver Operating Characteristics (ROC-AUC) and maximum F1 score. Along with
these metrics, class-wise accuracy and class-wise ROC-AUC scores are also
reported.

###  III-C Interpretability

We used the beat attention scores, rhythm attention scores and channel
attention scores to gain insights into the model at beat, rhythm and channel
level.

As the training samples prepared in evaluating the model are grouped according
to the same family of pathologies, such training data is not ideal to validate
the channel level interpretations as the grouped ECG recordings do not hold
many variations in abnormal beat characteristics across different ECG
channels. For validating channel level interpretations, we chose two specific
sub-diagnostic diseases of MI, namely, Anteroseptal Myocardial Infarction
(ASMI) and Inferior Myocardial Infarction (IMI) along with normal (NORM) ECG
recordings and rejected the rest of the subtypes of MI mainly due to the
paucity of the number of ECG recordings for these subtypes. A total of 1344
(ASMI), 916 (IMI), and 1000 (NORM) samples are obtained. Normal ECG samples
are obtained by randomly selecting 100 samples from each fold. We have chosen
the stratified folds from 1 to 8 as training data and folds 9 and 10 as test
data.

###  III-D Ablation Study

We experimented our model with different combinations of ECG recording
channels (leads), by doing so we can estimate how well these combinations of
channels contribute to the classification of superclass labeled data which was
used in the evaluation of the model. The channel combinations selected are
limb leads (I, II, III, aVL, aVR and aVF), precordial leads (V1-V6), leads I,
II and II and all the available 12 leads in the ECG recording. The performance
of our proposed model is compared against different combinations of ECG
channels and class-wise ROC-AUC and class-wise accuracy scores are reported.

###  III-E Implementation Details

The model is trained for a maximum of 60 epochs in mini-batches of 32 with an
early stopping criterion. The Cross-Entropy loss function has been used for
training the model with Adam optimizer [24] with the first momentum value set
to 0.9 and the second momentum value set to 0.99. An adaptive learning rate is
used for minimizing the loss with an initial learning rate set to 0.001. The
learning rate is decreased by a factor of 10 after every 10 epochs if no
improvement in performance is observed. Training is done with window size
($W$) of 50 time-points and a dropout rate of 0.5. An L2 regularisation loss
with a coefficient of ($2\times 1e$-5) has been used to avoid overfitting of
the model. The model is developed using Tensorflow and all the experiments are
trained on an Nvidia RTX 2080Ti 11GB GPU workstation.

##  IV Results

A comparison of class-wise ROC-AUC and class-wise accuracy scores across the
models is shown in Tables I and II, respectively. Our model achieves the best
class-wise ROC-AUC score across all the classes except for the HYP class and
the class-wise accuracy score is highest for all classes except for the CD
class. Table III shows that our model achieved the best performance across the
metrics macro ROC-AUC, mean accuracy and maximum F1 score as compared to the
other models. This is due to the effective use of the multi-channel
information used by our model compared to other models by incorporating a
channel level attention. The processing of the hierarchical structure of an
ECG signal is efficiently used by using beat and rhythm level attention. The
model parameters are shared across multiple channels instead of having
separate model parameters for each channel resulting in fewer final model
parameters and better generalization.

Figure 4: Visualization of normalized attention scores with red having a
higher attention score and yellow having a lower attention score

Figure 5: Channel Importance scores TABLE I: Class-wise ROC-AUC comparison |  CD |  HYP |  MI |  NORM |  STTC
---|---|---|---|---|---
Resnet101 [23] |  0.8928 |  0.8636 |  0.8921 |  0.9098 |  0.9178
Mousavi et al. [8] |  0.8555 |  0.8093 |  0.8708 |  0.9096 |  0.8817
ECGNet [9] |  0.9102 |  0.8842 |  0.9021 |  0.9336 |  0.9205
Rajpurkar et al. [7] |  0.8910 |  0.9185 |  0.9066 |  0.9309 |  0.9306
IMLE-Net |  0.9196 |  0.8834 |  0.9244 |  0.9450 |  0.9354
TABLE II: Class-wise accuracy comparison |  CD |  HYP |  MI |  NORM |  STTC
---|---|---|---|---|---
Resnet101 [23] |  87.74 |  90.89 |  83.12 |  84.69 |  87.47
Mousavi et al. [8] |  86.26 |  87.19 |  82.47 |  81.78 |  83.26
ECGNet [9] |  88.71 |  90.66 |  84.60 |  85.89 |  86.87
Rajpurkar et al. [7] |  90.01 |  88.76 |  85.94 |  85.94 |  88.90
IMLE-Net |  88.53 |  91.58 |  87.97 |  87.10 |  89.08
TABLE III: Overall Performance comparison |  Macro ROC-AUC |  Mean Accuracy |  F1 score(Max)
---|---|---|---
Resnet101 [23] |  0.8952 |  86.78 |  0.7558
Mousavi et al. [8] |  0.8654 |  84.19 |  0.7315
ECGNet [9] |  0.9101 |  87.35 |  0.7712
Rajpurkar et al. [7] |  0.9155 |  87.91 |  0.7895
IMLE-Net |  0.9216 |  88.85 |  0.8057
TABLE IV: Accuracy and ROC-AUC metrics for two subtypes of Myocardial Infarction |  ASMI |  IMI |  NORM |  Overall
---|---|---|---|---
Accuracy |  89.47 |  87.24 |  94.09 |  90.27
ROC-AUC |  0.9594 |  0.9336 |  0.9906 |  0.9612

The performance metrics for the selected subtypes of MI are shown in Table IV.
The metrics shown in this table are class-wise accuracy and class-wise ROC-AUC
scores along with mean accuracy and macro ROC-AUC score. The best performance
is observed for the NORM class followed by the ASMI and IMI classes.

The beat level attention scores, the rhythm level attention scores and the
channel level attention scores are visualized for an ECG recording in Figure 5
with Figure 5 describing the importance of a specific channel responsible for
the classification. The patient in the ECG recording shown in Figure 5 has an
inferior myocardial infarction (IMI) present in leads II, III and aVF. Our
proposed model was able to identify the channels responsible for this
particular subtype of MI in Figure 5. These findings have been validated by an
independent clinician. The channel level attention scores assess the relevance
of specific channels in an ECG recording. This setting is crucial as the
abnormal beat characteristics might be spread only over a few of the channels
and may be completely absent in the rest of the channels. So along with the
beat level, rhythm level interpretations, it is also essential to capture such
channel level interpretations.

TABLE V: Accuracy score comparison on different ECG lead configurations (Ablation Study) |  CD |  HYP |  MI |  NORM |  STTC |  Mean Accuracy
---|---|---|---|---|---|---
Limb Leads |  87.97 |  89.59 |  83.63 |  84.48 |  85.85 |  86.38
Precordial Leads |  87.47 |  90.98 |  84.46 |  84.97 |  88.48 |  87.27
Leads I,II,III |  87.51 |  88.95 |  83.40 |  85.25 |  85.85 |  86.19
12 Leads |  88.53 |  91.58 |  87.97 |  87.10 |  89.08 |  88.85
TABLE VI: ROC-AUC score comparison on different ECG lead configurations (Ablation Study) |  CD |  HYP |  MI |  NORM |  STTC |  Macro ROC-AUC
---|---|---|---|---|---|---
Limb Leads |  0.8725 |  0.8407 |  0.8787 |  0.9292 |  0.9120 |  0.8866
Precordial Leads |  0.8972 |  0.8627 |  0.8683 |  0.9290 |  0.9261 |  0.8966
Leads I,II,III |  0.8755 |  0.8447 |  0.8609 |  0.9259 |  0.9106 |  0.8836
12 Leads |  0.9196 |  0.8834 |  0.9244 |  0.9450 |  0.9354 |  0.9216

Tables V and VI report the class-wise accuracy and class-wise ROC-AUC scores
for several different channel combinations as part of the ablation studies. It
can be seen that the performance metrics when utilizing all the available 12
channels in the ECG recording are the highest for each class and overall when
compared against different combinations of channels. The precordial leads gave
the second-best overall performance when compared against limb leads and leads
I, II and III. Further, disorder classes such as HYP and STTC were classified
with better metrics. Limb leads and leads I, II and III gave similar
performance metrics with marginally better performance on the limb leads. In
short, the standard 12-lead ECG setting gives the best performance and the
performance drops as the leads are removed due to loss of information in the
channels, again emphasizing the importance of multi-channel information in
diagnosing heart conditions.

##  V Conclusion

In this study, we proposed an interpretable multi-level model for multi-
channel ECG classification. The proposed model has been evaluated on the PTB-
XL dataset and it outperforms several of the existing models. An ablation
study was performed to understand the changes in the performance of the model
with different combinations of ECG channels. The attention scores generated
were visualized and compared with guidelines that cardiologists follow to
diagnose two subtypes of myocardial infarction. The interpretability aspect of
the proposed model enables a better understanding of how it can be useful in
clinical decision-making in a real-life scenario. In future, the model
interpretability can be evaluated against a wide range of heart pathologies.

## Acknowledgement

We thank IHub-Data, IIIT Hyderabad for financial support and Dr. Adithi Devi
for validating attention visualizations.

## References

  * [1] McCabe JM al.  “Prevalence and Factors Associated With False-Positive ST-Segment Elevation Myocardial Infarction Diagnoses at Primary Percutaneous Coronary Intervention–Capable Centers”  In _Arch Intern Med_ 172.11, 2012, pp. 864–871
  * [2] Mohamed Boussaa, Issam Atouf, Mohamed Atibi and Abdellatif Bennis  “ECG signals classification using MFCC coefficients and ANN classifier”  In _2016 International Conference on Electrical and Information Technologies (ICEIT)_ , 2016, pp. 480–484
  * [3] Ataollah Ebrahim Zadeh al.  “Classification of the electrocardiogram signals using supervised classifiers and efficient features”  In _Computer Methods and Programs in Biomedicine_ 99.2, 2010, pp. 179–194
  * [4] Shadnaz Asgari, Alireza Mehrnia and Maryam Moussavi  “Automatic detection of atrial fibrillation using stationary wavelet transform and support vector machine”  In _Computers in biology and medicine_ 60 Elsevier, 2015, pp. 132–142
  * [5] M.. Poddar, Vinod Kumar and Yash Paul Sharma  “Heart rate variability: Analysis and classification of healthy subjects for different age groups”  In _2015 2nd International Conference on Computing for Sustainable Global Development (INDIACom)_ , 2015, pp. 1908–1913
  * [6] Jagadeeswara Rao Annam and Bapi Raju Surampudi  “AAMI Based ECG Heart-Beat Time-Series Clustering Using Unsupervised ELM and Decision Rule”  In _ICIT_ IEEE, 2016, pp. 137–141
  * [7] Masoumeh Haghpanahi Awni Y Hannun  “Cardiologist-level arrhythmia detection and classification in ambulatory electrocardiograms using a deep neural network”  In _Nature Medicine_ 25, 2019, pp. 65–69
  * [8] Sajad Mousavi and Fatemeh Afghah  “Inter- and Intra- Patient ECG Heartbeat Classification for Arrhythmia Detection: A Sequence to Sequence Deep Learning Approach”  In _ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 2019, pp. 1308–1312
  * [9] Balamurali Murugesan et al.  “ECGNet: Deep Network for Arrhythmia Classification”  In _2018 IEEE International Symposium on Medical Measurements and Applications (MeMeA)_ , 2018, pp. 1–6
  * [10] Patrick Schwab et al.  “Beat by beat: Classifying cardiac arrhythmias with recurrent neural networks”  In _2017 Computing in Cardiology (CinC)_ , 2017, pp. 1–4
  * [11] Sajad Mousavi, Fatemeh Afghah, Abolfazl Razi and U. Acharya  “ECGNET: Learning Where to Attend for Detection of Atrial Fibrillation with Deep Visual Attention”  In _2019 IEEE EMBS International Conference on Biomedical Health Informatics (BHI)_ , 2019, pp. 1–4
  * [12] Yong-Yeon Jo al.  “Explainable artificial intelligence to detect atrial fibrillation using electrocardiogram”  In _International Journal of Cardiology_ 328 Elsevier, 2021, pp. 104–110
  * [13] Sricharan Vijayarangan et al.  “Interpreting Deep Neural Networks for Single-Lead ECG Arrhythmia Classification”  In _2020 42nd Annual International Conference of the IEEE Engineering in Medicine Biology Society (EMBC)_ , 2020, pp. 300–303
  * [14] Ramprasaath R. Selvaraju et al.  “Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization”  In _2017 IEEE International Conference on Computer Vision (ICCV)_ , 2017, pp. 618–626
  * [15] Shenda Hong et al.  “MINA: Multilevel Knowledge-Guided Attention for Modeling Electrocardiography Signals”  In _Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI-19_ International Joint Conferences on Artificial Intelligence Organization, 2019, pp. 5888–5894
  * [16] Drew BJ and Finlay DD  “Standardization of reduced and optimal lead sets for continuous electrocardiogram monitoring: where do we stand?”  In _Journal of Electrocardiology_ 41, 2008, pp. 458–465
  * [17] Paul Kligfield  “Value and limitations of 12-lead ECG monitoring”  In _Datex–Ohmeda Clin Window Web J_ 6, 2001, pp. 1–6
  * [18] Patrick Wagner et al.  “PTB-XL, a large publicly available electrocardiography dataset”  In _Scientific Data_ 7.1 Springer ScienceBusiness Media LLC, 2020, pp. 154
  * [19] Jiapu Pan and Willis J. Tompkins  “A Real-Time QRS Detection Algorithm”  In _IEEE Transactions on Biomedical Engineering_ BME-32.3, 1985, pp. 230–236
  * [20] Sajad Mousavi, Fatemeh Afghah and U Rajendra Acharya  “HAN-ECG: An interpretable atrial fibrillation detection model using hierarchical attention networks”  In _Computers in Biology and Medicine_ 127 Elsevier, 2020, pp. 104057
  * [21] T. M. and B. Larbi  “R-peaks detection based on stationary wavelet transform”  In _Computer methods and programs in biomedicine_ 121.3, 2015, pp. 149–160
  * [22] P. Hamilton  “Open source ECG analysis”  In _Computers in Cardiology_ , 2002, pp. 101–104
  * [23] Kaiming He, Xiangyu Zhang, Shaoqing Ren and Jian Sun  “Deep Residual Learning for Image Recognition”  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2016
  * [24] Diederik P. Kingma and Jimmy Ba  “Adam: A Method for Stochastic Optimization”, 2017  arXiv:1412.6980 [cs.LG]