Zhu and Ye

How a Small Amount of Data Sharing Benefits Distributed Optimization and Learning

How a Small Amount of Data Sharing Benefits Distributed Optimization and Learning

Mingxi Zhu
Graduate School of Business, Stanford University, Stanford, CA 94305<EMAIL_ADDRESS>Yinyu Ye
Management Science & Engineering, Stanford University, Stanford, CA 94305<EMAIL_ADDRESS>
Distributed optimization algorithms have been widely used in machine learning and statistical estimation, especially under the context where multiple decentralized data centers exist and the decision maker is required to perform collaborative learning across those centers. While distributed optimization algorithms have the merits in parallel processing and protecting local data security, they often suffer from slow convergence compared with centralized optimization algorithms. This paper focuses on how small amount of data sharing could benefit distributed optimization and learning for more advanced optimization algorithms. Specifically, we consider how data sharing could benefit distributed multi-block alternating direction method of multipliers (ADMM) and preconditioned conjugate gradient method (PCG) with application in machine learning tasks of linear and logistic regression. These algorithms are commonly known as algorithms between the first and the second order methods, and we show that data share could hugely boost the convergence speed for this class of the algorithms. Theoretically, we prove that a small amount of data share leads to improvements from near-worst to near-optimal convergence rate when applying ADMM and PCG methods to machine learning tasks. A side theory product is the tight upper bound of linear convergence rate for distributed ADMM applied in linear regression. We further propose a meta randomized data-sharing scheme and provide its tailored applications in multi-block ADMM and PCG methods in order to enjoy both the benefit from data-sharing and from the efficiency of distributed computing. From the numerical evidences, we are convinced that our algorithms provide good quality of estimators in both the least square and the logistic regressions within much fewer iterations by only sharing 5$\%$ of pre-fixed data, while purely distributed optimization algorithms may take hundreds more times of iterations to converge. We hope that the discovery resulted from this paper would encourage even small amount of data sharing among different regions to combat difficult global learning problems.

Distributed Learning; Distributed Optimization; Data Sharing

[Eckstein and Bertsekas, 1992]
Jonathan Eckstein and Dimitri P Bertsekas.
On the douglas—rachford splitting method and the proximal point
algorithm for maximal monotone operators.
Mathematical Programming, 550 (1-3):0
293–318, 1992a.

[Boyd et al., 2011]
Stephen Boyd, Neal Parikh, and Eric Chu.
Distributed optimization and statistical learning via the
alternating direction method of multipliers.
Now Publishers Inc, 2011.

[Nishihara et al., 2015]
Robert Nishihara, Laurent Lessard, Ben Recht, Andrew Packard, and Michael
A general analysis of the convergence of admm.
In International Conference on Machine Learning, pages
343–352. PMLR, 2015.

[Ghadimi et al., 2014]
Euhanna Ghadimi, André Teixeira, Iman Shames, and Mikael Johansson.
Optimal parameter selection for the alternating direction method of
multipliers (admm): quadratic problems.
IEEE Transactions on Automatic Control, 600
(3):0 644–658, 2014.

[Nilsson et al., 2018]
Adrian Nilsson, Simon Smith, Gregor Ulm, Emil Gustavsson, and Mats Jirstrand.
A performance evaluation of federated learning algorithms.
In Proceedings of the second workshop on distributed
infrastructures for deep learning, pages 1–8, 2018.

[Mohan et al., 2014]
Karthik Mohan, Palma London, Maryam Fazel, Daniela Witten, and Su-In Lee.
Node-based learning of multiple gaussian graphical models.
The Journal of Machine Learning Research, 150
(1):0 445–488, 2014.

[Huang and Sidiropoulos, 2016]
K. Huang and N. D. Sidiropoulos.
Consensus-admm for general quadratically constrained quadratic
IEEE Transactions on Signal Processing, 640
(20):0 5297–5310, Oct 2016.

[Taylor et al., 2016]
Gavin Taylor, Ryan Burmeister, Zheng Xu, Bharat Singh, Ankit Patel, and Tom
Training neural networks without gradients: A scalable admm approach.
In International conference on machine learning, pages
2722–2731, 2016.

[Gabay and Mercier, 1976]
Daniel Gabay and Bertrand Mercier.
A dual algorithm for the solution of nonlinear variational problems
via finite element approximation.
Computers & Mathematics with Applications, 20
(1):0 17–40, 1976.

[He et al., 2012]
Bingsheng He, Min Tao, and Xiaoming Yuan.
Alternating direction method with gaussian back substitution for
separable convex programming.
SIAM Journal on Optimization, 220 (2):0
313–340, 2012.

[Monteiro and Svaiter, 2013]
Renato DC Monteiro and Benar F Svaiter.
Iteration-complexity of block-decomposition algorithms and the
alternating direction method of multipliers.
SIAM Journal on Optimization, 230 (1):0
475–507, 2013.

[Glowinski, 2014]
Roland Glowinski.
On alternating direction methods of multipliers: a historical
In Modeling, simulation and optimization for science and
technology, pages 59–82. Springer, 2014.

[Deng and Yin, 2016]
Wei Deng and Wotao Yin.
On the global and linear convergence of the generalized alternating
direction method of multipliers.
Journal of Scientific Computing, 660 (3):0
889–916, 2016.

[Stellato et al., 2018]
Bartolomeo Stellato, Goran Banjac, Paul Goulart, Alberto Bemporad, and Stephen
Osqp: An operator splitting solver for quadratic programs.
In 2018 UKACC 12th International Conference on Control
(CONTROL), pages 339–339. IEEE, 2018.

[Zarepisheh et al., 2018]
Masoud Zarepisheh, Lei Xing, and Yinyu Ye.
A computation study on an integrated alternating direction method of
multipliers for large scale optimization.
Optimization Letters, 120 (1):0 3–15, 2018.

[Lin et al., 2017]
T. Lin, S. Ma, Y. Ye, and S. Zhang.
An ADMM-Based Interior-Point Method for Large-Scale Linear
ArXiv e-prints, 2017.

[Eckstein and Bertsekas, 1992]
Jonathan Eckstein and Dimitri P Bertsekas.
On the douglas—rachford splitting method and the proximal point
algorithm for maximal monotone operators.
Mathematical Programming, 550 (1-3):0
293–318, 1992b.

[He and Yuan, 2012]
Bingsheng He and Xiaoming Yuan.
On the o(1/n) convergence rate of the douglas–rachford alternating
direction method.
SIAM Journal on Numerical Analysis, 500 (2):0
700–709, 2012.

[Ouyang et al., 2013]
Hua Ouyang, Niao He, Long Tran, and Alexander Gray.
Stochastic alternating direction method of multipliers.
In International Conference on Machine Learning, pages 80–88,

[Hestenes, 1969]
Magnus R Hestenes.
Multiplier and gradient methods.
Journal of optimization theory and applications, 40
(5):0 303–320, 1969.

[Powell, 1978]
M. J. D. Powell.
Algorithms for nonlinear constraints that use lagrangian functions.
Mathematical Programming, 14, 1978.

[Sun et al., 2020]
Ruoyu Sun, Zhi-Quan Luo, and Yinyu Ye.
On the efficiency of random permutation for admm and coordinate
Mathematics of Operations Research, 450 (1):0
233–271, 2020.

[Mihić et al., 2020]
Krešimir Mihić, Mingxi Zhu, and Yinyu Ye.
Managing randomization in the multi-block alternating direction
method of multipliers for quadratic optimization.
Mathematical Programming Computation, pages 1–75, 2020.

[Iutzeler et al., 2015]
Franck Iutzeler, Pascal Bianchi, Philippe Ciblat, and Walid Hachem.
Explicit convergence rate of a distributed alternating direction
method of multipliers.
IEEE Transactions on Automatic Control, 610
(4):0 892–904, 2015.

[Chen et al., 2017]
Caihua Chen, Min Li, Xin Liu, and Yinyu Ye.
Extended admm and bcd for nonseparable convex minimization models
with quadratic coupling terms: convergence analysis and insights.
Mathematical Programming, Nov 2017.
ISSN 1436-4646.
URL <https://doi.org/10.1007/s10107-017-1205-9>.

[Luenberger et al., 1984]
David G Luenberger, Yinyu Ye, et al.
Linear and nonlinear programming, volume 2.
Springer, 1984.

[Shewchuk et al., 1994]
Jonathan Richard Shewchuk et al.
An introduction to the conjugate gradient method without the
agonizing pain, 1994.

[Le et al., 2011]
Quoc V Le, Jiquan Ngiam, Adam Coates, Ahbik Lahiri, Bobby Prochnow, and
Andrew Y Ng.
On optimization methods for deep learning.
In ICML, 2011.

[Helfenstein and Koko, 2012]
Rudi Helfenstein and Jonas Koko.
Parallel preconditioned conjugate gradient algorithm on gpu.
Journal of Computational and Applied Mathematics, 2360
(15):0 3584–3590, 2012.

[Hsia et al., 2018]
Chih-Yang Hsia, Wei-Lin Chiang, and Chih-Jen Lin.
Preconditioned conjugate gradient methods in truncated newton
frameworks for large-scale linear classification.
In Asian Conference on Machine Learning, pages 312–326. PMLR,

[Axelsson and Lindskog, 1986]
Owe Axelsson and Gunhild Lindskog.
On the rate of convergence of the preconditioned conjugate gradient
Numerische Mathematik, 480 (5):0 499–523,

[Kaasschieter, 1988]
Erik F Kaasschieter.
Preconditioned conjugate gradients for solving singular systems.
Journal of Computational and Applied mathematics, 240
(1-2):0 265–275, 1988.

[Kaporin, 1994]
Igor E Kaporin.
New convergence results and preconditioning strategies for the
conjugate gradient method.
Numerical linear algebra with applications, 10
(2):0 179–210, 1994.

[Herzog and Sachs, 2010]
Roland Herzog and Ekkehard Sachs.
Preconditioned conjugate gradient method for optimal control problems
with control and state constraints.
SIAM Journal on Matrix Analysis and Applications, 310
(5):0 2291–2317, 2010.

[Drineas and Mahoney, 2016]
Petros Drineas and Michael W Mahoney.
Randnla: randomized numerical linear algebra.
Communications of the ACM, 590 (6):0 80–90,

[Han et al., 2018]
Deren Han, Defeng Sun, and Liwei Zhang.
Linear rate convergence of the alternating direction method of
multipliers for convex composite programming.
Mathematics of Operations Research, 430 (2):0
622–637, 2018.

[Liu et al., 2018]
Yongchao Liu, Xiaoming Yuan, Shangzhi Zeng, and Jin Zhang.
Partial error bound conditions and the linear convergence rate of the
alternating direction method of multipliers.
SIAM Journal on Numerical Analysis, 560 (4):0
2095–2123, 2018.

[Konečnỳ et al., 2015]
Jakub Konečnỳ, Brendan McMahan, and Daniel Ramage.
Federated optimization: Distributed optimization beyond the
arXiv preprint arXiv:1511.03575, 2015.

[Bonawitz et al., 2019]
Keith Bonawitz, Hubert Eichner, Wolfgang Grieskamp, Dzmitry Huba, Alex
Ingerman, Vladimir Ivanov, Chloe Kiddon, Jakub Konečnỳ, Stefano
Mazzocchi, Brendan McMahan, et al.
Towards federated learning at scale: System design.
Proceedings of Machine Learning and Systems, 1:0
374–388, 2019.

[McMahan et al., 2017]
Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera
y Arcas.
Communication-efficient learning of deep networks from decentralized
In Artificial intelligence and statistics, pages 1273–1282.
PMLR, 2017.

[Stich, 2018]
Sebastian U Stich.
Local sgd converges fast and communicates little.
arXiv preprint arXiv:1805.09767, 2018.

[Khaled et al., 2020]
Ahmed Khaled, Konstantin Mishchenko, and Peter Richtárik.
Tighter theory for local sgd on identical and heterogeneous data.
In International Conference on Artificial Intelligence and
Statistics, pages 4519–4529. PMLR, 2020.

[Yuan and Ma, 2020]
Honglin Yuan and Tengyu Ma.
Federated accelerated stochastic gradient descent.
Advances in Neural Information Processing Systems,
33:0 5332–5344, 2020.

[Dwork et al., 2006]
Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam Smith.
Calibrating noise to sensitivity in private data analysis.
In Theory of cryptography conference, pages 265–284.
Springer, 2006.

[Dwork, 2008]
Cynthia Dwork.
Differential privacy: A survey of results.
In International conference on theory and applications of
models of computation, pages 1–19. Springer, 2008.

[Dwork and Lei, 2009]
Cynthia Dwork and Jing Lei.
Differential privacy and robust statistics.
In Proceedings of the forty-first annual ACM symposium on
Theory of computing, pages 371–380, 2009.

[Blocki et al., 2012]
Jeremiah Blocki, Avrim Blum, Anupam Datta, and Or Sheffet.
The johnson-lindenstrauss transform itself preserves differential
In 2012 IEEE 53rd Annual Symposium on Foundations of Computer
Science, pages 410–419. IEEE, 2012.

[Song et al., 2013]
Shuang Song, Kamalika Chaudhuri, and Anand D Sarwate.
Stochastic gradient descent with differentially private updates.
In 2013 IEEE Global Conference on Signal and Information
Processing, pages 245–248. IEEE, 2013.

[Sheffet, 2017]
Or Sheffet.
Differentially private ordinary least squares.
In International Conference on Machine Learning, pages
3105–3114. PMLR, 2017.

[Huang et al., 2019]
Zonghao Huang, Rui Hu, Yuanxiong Guo, Eric Chan-Tin, and Yanmin Gong.
Dp-admm: Admm-based distributed learning with differential privacy.
IEEE Transactions on Information Forensics and Security,
15:0 1002–1012, 2019.

[Dua and Graff, 2017]
Dheeru Dua and Casey Graff.
UCI machine learning repository, 2017.
URL <http://archive.ics.uci.edu/ml>.

[Makhdoumi and Ozdaglar, 2017]
Ali Makhdoumi and Asuman Ozdaglar.
Convergence rate of distributed admm over networks.
IEEE Transactions on Automatic Control, 620
(10):0 5082–5095, 2017.

[Sun et al., 2015]
Ruoyu Sun, Zhi-Quan Luo, and Yinyu Ye.
On the expected convergence of randomly permuted admm.
Optimization for Machine Learning, OPT2015, 2015.

[Floudas and Pardalos, 2008]
Christodoulos A Floudas and Panos M Pardalos.
Encyclopedia of optimization.
Springer Science & Business Media, 2008.

[Xiao et al., 2019]
Peijun Xiao, Zhisheng Xiao, and Ruoyu Sun.
Understanding limitation of two symmetrized orders by worst-case
arXiv preprint arXiv:1910.04366, 2019.

[Byrd et al., 2016]
Richard H Byrd, Samantha L Hansen, Jorge Nocedal, and Yoram Singer.
A stochastic quasi-newton method for large-scale optimization.
SIAM Journal on Optimization, 260 (2):0
1008–1031, 2016.

[Gopal and Yang, 2013]
Siddharth Gopal and Yiming Yang.
Distributed training of large-scale logistic models.
In International Conference on Machine Learning, pages
289–297. PMLR, 2013.

[Xu et al., 2017]
Zheng Xu, Gavin Taylor, Hao Li, Mario Figueiredo, Xiaoming Yuan, and Tom
Adaptive consensus admm for distributed optimization.
arXiv preprint arXiv:1706.02869, 2017.

[Plemmons, 1977]
Robert J Plemmons.
M-matrix characterizations. i—nonsingular m-matrices.
Linear Algebra and its Applications, 180 (2):0
175–188, 1977.

[Brinkhuis et al., 2005]
Jan Brinkhuis, Zhi-Quan Luo, and Shuzhong Zhang.
Matrix convex functions with applications to weighted centers for
semidefinite programming.
Technical report, Report / Econometric Institute, Erasmus University
Rotterdam., 2005.


# Convolution-Free Waveform Transformers for Multi-Lead ECG Classification

Annamalai Natarajan1 Gregory Boverman1 Yale Chang1 Corneliu Antonescu2
Jonathan Rubin1

1Philips Research North America  Cambridge MA  USA
2University of Arizona Banner Health  Phoenix AZ  USA

###### Abstract

We present our entry to the 2021 PhysioNet/CinC challenge — a waveform
transformer model to detect cardiac abnormalities from ECG recordings. We
compare the performance of the waveform transformer model on different ECG-
lead subsets using approximately 88,000 ECG recordings from six datasets. In
the official rankings, team prna ranked between 9 and 15 on 12, 6, 4, 3 and
2-lead sets respectively. Our waveform transformer model achieved an average
challenge metric of 0.47 on the held-out test set across all ECG-lead subsets.
Our combined performance across all leads placed us at rank 11 out of 39
officially ranking teams.

##  1 Introduction

Cardiovascular diseases (CVDs) are the leading cause of death globally with an
estimated 32% of deaths worldwide in 2019 [1]. It is important to detect
cardiovascular diseases early so treatments can be provided to mitigate
complications. The standard 12-lead ECG has been a popular choice in the
diagnosis of various cardiac abnormalities, but, more recently subsets of ECG
leads have been used due to their size, cost, performance and ease of use. In
this challenge, we utilize ECG recordings from subsets of standard 12-lead ECG
to evaluate the efficacy of detecting cardiac abnormalities. The subsets of
ECG leads include 12, 6, 4, 3, or 2 leads respectively. The challenge provides
$\sim$88K ECG recordings from five training data sources assigned to belong to
one or more of the 30 cardiac abnormalities. More details about this challenge
can be found in [2, 3].

Prior work has shown deep neural networks to be successful in detecting
cardiac abnormalities from 12-lead ECG signals [4]. In this work, we
experiment with a waveform transformer model which differentially weights
different parts of the inputs using an attention mechanism. This feature is a
natural fit to this problem since cardiac abnormalities tend to be transient
in ECG recordings. Our waveform transformer model is similar to our entry in
the 2020 PhysioNet challenge [4] with some modifications:

  1. 1.

We remove the embedding network that applied a series of convolution
operations to the input ECG recording and instead rely on a convolution-free
approach where the input signal is split into smaller segments and fed
directly to a transformer model.

  2. 2.

We seed the network with pretrained vision transformer network weights that
have been trained on two-dimensional still images. We tailor our architecture
to allow single-dimensional waveform inputs to make use of these pretrained
weights.

We continue to train wide and deep networks that include 22 static ECG and
demographic features, which are concatenated to the learned deep features of
the network. These 22 features are similar to the ones listed in Table 1 in
[4].

Figure 1: Architecture diagram illustrating the Waveform Transformer
(Waveformer) for multi-lead, multi-label ECG classification. Linear projection
is used to allow matching of pretrained weights. Wide (static) features can be
directly fed as input to the MLP head. Based upon an original image in [5].

##  2 Methods

###  2.1 Pre-processing

Recordings from all databases provided by the challenge organizers [3, 2],
including CPSC [6], INCART [7], PTB [8], PTB-XL [9], Chapman-Shaoxing [10] and
Ningbo [11], were used for model training. We first split the data into 10
folds utilizing multi-label stratification [12]. Each recording was
standardized to a sampling rate of 500Hz. We apply a finite impulse response
bandpass filter with bandwidth between 3 - 45 Hz. Each recording is also
normalized so that each channels’ signal lies within the range of -1 to +1. We
extract random fixed width windows from each recording across the subset of
leads. We set the fixed width to be 7680 samples ($T=15.36$ seconds), which
allows the signal to be split into divisible segments sizes. We apply zero-
padding to the recordings at the end when the sequence length is less than $T$
seconds.

Hyper-Parameter | Value
---|---
Global
ECG window size (secs), $T$ | 15.36
Sampling frequency (Hz) | 500
Batch size (train) | 128
Batch size (validation) | 64
Wide feature size, $d_{wide}$ | 22
Deep feature size, $d_{deep}$ | 64
Number of classes, $d_{class}$ | 26
Waveform Transformer
ECG patch size, $d_{patch}$ | 64
Number of encoding layers | 12
Embedding size, $d_{model}$ | 768
Number of heads | 12
Dimension of feed forward layer | 768
Dropout | 0.1
Fully connected layers
FC 1 size | 64
FC 2 size | 26
Dropout | 0.2
Table 1: A listing of hyper-parameters selected to train the wide and deep
neural network model for classification of cardiac abnormalities.

###  2.2 Waveform Transformer

An overview of the waveform transformer architecture is shown in Figure 1.
Input to the network is a multi-lead ECG recording (e.g. 12, 6, 4, 3, or 2
leads). The input ECG recording is first broken up into smaller contiguous
segments. Each segment undergoes a linear projection to embed it into a one
dimensional vector that captures information for that time point in the
overall recording. Linear projection of segments can be handled via a multi-
layer perceptron (MLP) or convolution111Making the approach _almost_
convolution-free.. A sequence of embedded segments are then fed to the
transformer encoder. Positional embedding is used to retain sequence order
information. In addition, an extra learnable class token is fed to the
transformer network that attends to all other tokens. The transformer model
consists of 12 layers, using 12 attention heads and an embedding dimension of
768. We rely on pretrained weights of a vision transformer [5] trained on
still image data, and as such, need to ensure the dimensions of the network
match. We chose a base vision transformer model trained on 16x16 image patches
with an embedding dimension of 768. To make a final prediction, the learnable
class embedding is sent as input into an MLP head consisting of two linear
layers. Static (wide) features are concatenated to the final linear layer of
the network and a sigmoid operation is applied to make binary predictions
about 26 classes222Equivalent classes are combined, reducing 30 classes to
26.. Table 1 provides details on model architecture, settings and hyper-
parameters used in our experiments. All models were trained using PyTorch
using base models and pretrained weights from PyTorch Image Models [13]

Leads | Training Set | Validation Set | Test Set | Official Ranking
---|---|---|---|---
12 | $0.675\pm 0.023$ | 0.58 | 0.49 | 9
6 | $0.658\pm 0.026$ | 0.55 | 0.49 | 9
4 | $0.669\pm 0.024$ | 0.55 | 0.46 | 14
3 | $0.663\pm 0.026$ | 0.54 | 0.47 | 11
2 | $0.661\pm 0.019$ | 0.53 | 0.44 | 15
All leads | – | 0.55 | 0.47 | 11
Table 2: Challenge scores for our final selected entry using 10-fold cross
validation on the public training set, repeated scoring on the hidden
validation set, and one-time scoring on the hidden test set as well as the
official ranking on the hidden test set.

##  3 Results and Conclusions

In this section, we present results from our waveform transformer model. Our
setup is a standard 10-fold nested cross validation. In each fold, we utilize
data from the validation fold to learn probability thresholds and other hyper-
parameters. In Table 2, we list the challenge metric on the test folds as well
as on the official test set for different ECG-lead subsets. We observe that
overall there is a monotonic improvement in the scores from the 2-lead to the
12-lead on both the test folds and the official held-out test set. Our scores
on the official held-out test set for the 12, 6, 4, 3 and 2 leads are 0.49,
0.49, 0.46, 0.47, 0.44 respectively, which shows a 0.05 units improvement in
the challenge metric between 2 and 12 leads. We also observe that there is a
significant gap in performance ($\sim 0.2$) between model performance on the
publicly available train dataset and the held-out dataset which hints at
potential over fitting to the train dataset.

We examined the AUROCs for each cardiac abnormality computed using the
probabilities as output by the transformer model on a single test fold. We
observed similar trends in AUROC scores across different subsets of ECG leads.
Our waveform transformer model achieved an AUROC of $0.85$ and $0.77$ on
detecting low qrs voltage in 12-lead and 2-lead models respectively. We
hypothesize that low qrs amplitude ECG recordings are very similar to normal
sinus rhythms but with low amplitudes which makes detecting them challenging.
We hypothesize that this poor performance on these cardiac abnormalities is
that they often co-occur with other abnormalities that can be detected with
high confidence. This low AUROC was followed by detection of T wave abnormal,
Q wave abnormal and T wave inversion with AUROC’s in the range of $0.88$ to
$0.90$ respectively. All ECG lead subsets did exceptionally well in detecting
pacing rhythm, left/right bundle block branch, tachycardia and bradycardia
with AUROC’s in the $0.98$ to $0.99$ range. This superior performance can be
attributed to the unique physiological signatures embedded in ECG recordings
and our inclusion of wide features such are heart rate. The AUROC for normal
sinus rhythm is $\sim 0.97$ across all ECG lead subsets.

Lastly, we present the attention maps from the waveform transformer model in
Figure 2. These attention maps illustrate which parts of the inputs are
critical to making a final prediction much like a clinical expert manually
scanning through ECG recordings. For each sample patient we show the ECG
recording, attention maps along with ground truth (GT) and predicted (Pred.)
labels. Both the x and y axis in the heatmaps represent time. Each cell
represents a block of 64 samples (in time). Each sample (time point) can
attend to all other time points. The high intensities (bright colors) along
vertical time slices indicate that there are common time points that all time
points are attending to. A potential shortcoming is highlighted for the bottom
right patient which incorrectly predicts sinus rhythm, here the attention of
all time points is on the length of the ECG recording which is non-
informative.

Figure 2: Six randomly selected attention maps that show where attention is
being paid in the corresponding ECG recordings (shown below). Recordings
consist of 7680 samples that are encoded via a waveform transformer network
into $N=120$ sequential embeddings. Attention heat maps are shown for lead II
in the 12-lead setup. Also shown are the ground truth (GT) and predicted
(Pred.) class labels.

## References

  * [1] Cardiovascular Diseases.  https://www.who.int/en/news-room/fact-sheets/detail/cardiovascular-diseases-(cvds).  Accessed: 2021-08-10.
  * [2] Perez Alday EA, Gu A, Shah A, Robichaux C, Wong AKI, Liu C, et al.  Classification of 12-lead ECGs: the PhysioNet/Computing in Cardiology Challenge 2020.  Physiological Measurement 2020;41.
  * [3] Reyna MA, Sadr N, Perez Alday EA, Gu A, Shah A, Robichaux C, et al.  Will Two Do? Varying Dimensions in Electrocardiography: the PhysioNet/Computing in Cardiology Challenge 2021.  Computing in Cardiology 2021;48:1–4.
  * [4] Natarajan A, Chang Y, Mariani S, Rahman A, Boverman G, Vij S, et al.  A wide and deep transformer neural network for 12-lead ecg classification.  In 2020 Computing in Cardiology. IEEE, 2020; 1–4.
  * [5] Dosovitskiy A, Beyer L, Kolesnikov A, Weissenborn D, Zhai X, Unterthiner T, et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  arXiv preprint arXiv201011929 2020;.
  * [6] Liu F, Liu C, Zhao L, Zhang X, Wu X, Xu X, et al.  An Open Access Database for Evaluating the Algorithms of Electrocardiogram Rhythm and Morphology Abnormality Detection.  Journal of Medical Imaging and Health Informatics 2018;8(7):1368––1373.
  * [7] Tihonenko V, Khaustov A, Ivanov S, Rivin A, Yakushenko E.  St Petersburg INCART 12-lead Arrhythmia Database.  PhysioBank PhysioToolkit and PhysioNet 2008;Doi: 10.13026/C2V88N.
  * [8] Bousseljot R, Kreiseler D, Schnabel A.  Nutzung der EKG-Signaldatenbank CARDIODAT der PTB über das Internet.  Biomedizinische Technik 1995;40(S1):317–318.
  * [9] Wagner P, Strodthoff N, Bousseljot RD, Kreiseler D, Lunze FI, Samek W, et al.  PTB-XL, a Large Publicly Available Electrocardiography Dataset.  Scientific Data 2020;7(1):1–15.
  * [10] Zheng J, Zhang J, Danioko S, Yao H, Guo H, Rakovski C.  A 12-lead Electrocardiogram Database for Arrhythmia Research Covering More Than 10,000 Patients.  Scientific Data 2020;7(48):1–8.
  * [11] Zheng J, Cui H, Struppa D, Zhang J, Yacoub SM, El-Askary H, et al.  Optimal Multi-Stage Arrhythmia Classification Approach.  Scientific Data 2020;10(2898):1–17.
  * [12] Szymański P, Kajdanowicz T.  A scikit-based Python environment for performing multi-label classification.  ArXiv e prints February 2017;.
  * [13] Wightman R.  Pytorch image models.  https://github.com/rwightman/pytorch-image-models, 2019.

Jonathan Rubin
222 Jacobs St Cambridge, MA 02141, United States
<EMAIL_ADDRESS>