# Anonymization for Skeleton Action Recognition

Saemi Moon
POSTECH
<EMAIL_ADDRESS>
&Myeonghyeon Kim 11footnotemark: 1
POSTECH
<EMAIL_ADDRESS>
&Zhenyue Qin
Australian National University
<EMAIL_ADDRESS>
&Yang Liu
Australian National University
<EMAIL_ADDRESS>
&Dongwoo Kim
POSTECH
<EMAIL_ADDRESS>
Equal contribution.

###### Abstract

Skeleton-based action recognition attracts practitioners and researchers due
to the lightweight, compact nature of datasets. Compared with RGB-video-based
action recognition, skeleton-based action recognition is a safer way to
protect the privacy of subjects while having competitive recognition
performance. However, due to improvements in skeleton estimation algorithms as
well as motion- and depth-sensors, more details of motion characteristics can
be preserved in the skeleton dataset, leading to potential privacy leakage. To
investigate the potential privacy leakage from skeleton datasets, we first
train a classifier to categorize sensitive private information from
trajectories of joints. Our preliminary experiments show that the gender
classifier achieves 87% accuracy on average and the re-identification task
achieves 80% accuracy on average for three baseline models: Shift-GCN, MS-G3D,
and 2s-AGCN. We propose an adversarial anonymization algorithm to protect
potential privacy leakage from the skeleton dataset. Experimental results show
that an anonymized dataset can reduce the risk of privacy leakage while having
marginal effects on action recognition performance.

##  1 Introduction

Action recognition has been widely studied in many applications such as sports
analysis [27], human-robot interaction [5], and intelligent healthcare
services [21]. Due to the success of convolutional neural networks, many
recognition approaches are proposed based on a sequence of video frames. The
action recognition can further be used for the public good. For example, with
surveillance cameras in a public area or a school, we can detect violent
actions.

To employ the recognition system appropriately, one must ensure that private
information is not abused before and after analysis. Skeleton-based action
recognition can be alternative to video-based recognition. Due to the advance
in depth and motion sensors, details of motion characteristics can be
preserved in the skeleton dataset. Compared with RGB videos, the skeleton
dataset seems to expose fewer details on participants. It is often challenging
to identify sensitive information such as gender or age from a skeleton to
compare with the RGB video to the naked eye.

We raise a question about the privacy-safeness of skeleton datasets. To check
potential privacy leakage from skeletons, we conduct experiments on
identifying gender or identity with Shift-GCN [2], MS-G3D [15], and 2s-AGCN
[23]. Based on our analysis, a properly trained classifier can predict the
private information accurately. Therefore, the skeletons are not safe from the
privacy leakage problem. A previous study [25] confirms the possibility of
identifying a person from skeletons extracted from Kinect. Also, prior work
[29] suggests an end-to-end framework to predict both action and identity
recognition tasks.

This work aims to develop a framework that can anonymize skeleton datasets
while preserving critical action features for recognition. To this end, we
propose a minimax framework to anonymize the skeletons. With RGB-video
datasets, object detection followed by blurring or inpainting with pre-trained
generative models is often employed to anonymize datasets [31, 11]. However,
these methods cannot be directly applied to the skeleton dataset.

The minimax framework consists of an anonymizer network with two sub-networks
designed to predict action and private information. The anonymizer removes
private information from skeletons, and then the output skeleton is fed into
action and privacy classifiers separately. We maximize the accuracy of the
action classifier while minimizing the identifiability of private information
with the other classifier. In addition, we enforce the anonymized skeleton
similar to the original one to make sure they are visually indistinguishable
from each other. An adversarial learning algorithm is proposed to solve the
minimax problem. Experimental results show that the proposed algorithm results
in an effective anonymizer.

We summarize our contributions as follows:

  * •

We empirically show potential privacy leakage from widely-used skeleton
datasets such as NTU60 [22] and ETRI-activity3D[12].

  * •

We develop a skeleton anonymization network based on action and sensitive
variable classifiers.

  * •

We propose a learning algorithm based on the adversarial learning method to
anonymize skeletons.

  * •

We show that the anonymized skeletons are more robust to privacy leakage while
still enjoying high action recognition accuracy.

Figure 1: Anonymization framework. The framework consists of three sub-
networks: 1) anonymizer $f_{\theta}$, 2) privacy classifier $g_{\phi}$, and 3)
action classifier $h_{\psi}$. The dashed box represents the losses used in
minimization and maximization steps with adversarial learning. Note that the
privacy classifier uses a separate loss for minimization and maximization in
an adversarial learning setup. The parameter of the action classifier $\psi$
is pre-trained and not updated during anonymizer training.

##  2 Skeleton Anonymization

In this section, we propose a framework for the skeleton anonymization model.

#### Anonymization framework.

Let $\mathbf{x}\in\mathbb{R}^{T\times D\times 3}$ be 3D coordinates of $D$
joints over $T$ frames, and $y\in\mathcal{Y}$ be an action label for a given
skeleton sequence $\mathbf{x}$, where $\mathcal{Y}$ is a set of actions to be
recognized. Let $z\in\mathcal{Z}$ be private information related to the
skeleton sequence $\mathbf{x}$, e.g., gender or identity, where $\mathcal{Z}$
is a set of possible private labels.

We aim to develop an anonymization network that can effectively remove private
information from skeleton datasets while maintaining the recognizability of
actions from the anonymized skeletons. To do this, we propose a minimax
framework consisting of three different neural network components. Let
$f_{\theta}:\mathbb{R}^{T\times D\times 3}\rightarrow\mathbb{R}^{T\times
D\times 3}$ be an anonymizer network aiming to remove sensitive information
from the input skeletons, $h_{\psi}:\mathbb{R}^{T\times D\times
3}\rightarrow\mathcal{Y}$ be an action classifier, and
$g_{\phi}:\mathbb{R}^{T\times D\times 3}\rightarrow\mathcal{Z}$ be a privacy
classifier that predicts sensitive personal information. Our goal is to train
an anonymizer $f_{\theta}$ whose output can maximally confuse the
classification performance on the private variables. On the other hand, the
output of the anonymizer should keep all relevant information for recognizing
action to preserve the performance of the action classifier $h_{\psi}$. In
other words, the output should not be very different from the original
skeletons since the anonymized skeletons can be recognizable by naked eyes. To
satisfy all requirements, we formalize the anonymization via the following
minimax objective:

$\displaystyle\min_{\theta}\max_{\phi}\mathbb{E}\bigg{[}\operatorname{CE}\left(y,h_{\psi}\left(f_{\theta}(\mathbf{x})\right)\right)$
$\displaystyle-\alpha\operatorname{CE}\left(z,g_{\phi}\left(f_{\theta}(\mathbf{x})\right)\right)+\beta||\mathbf{x}-f_{\theta}(\mathbf{x})||_{2}^{2}\bigg{]}\;,$
(1)

where $\operatorname{CE}$ is the cross entropy, and $\alpha$ and $\beta$ are
hyperparameters controlling the importance of the privacy classification and
the reconstruction error, respectively. The reconstruction error between the
original and anonymized skeleton data
$||\mathbf{x}-f_{\theta}(\mathbf{x})||_{2}^{2}$ ensures the anonymized
skeletons are similar to the original ones. To maximize the objective, the
private classifier needs to classify the private label $z$ correctly. To
minimize the objective, the anonymizer makes the actions easily identifiable
by action classifier $h_{\psi}$ while making the private classifier
misclassify the private label $z$. To simplify the learning process, we use a
pre-trained action classifier and fix the parameters of the action classifier
during training. The fixed action classifier constrains the anonymized
skeleton compatible with the pre-trained model. The anonymized skeletons are
also likely to work well with other pre-trained classifiers available.

Minimizing the objective w.r.t $\theta$ can make the anonymizer fool the
private classifier. However, one may exploit this fact to infer the true
label. For example, in a binary classification problem, the true label can be
obtained by choosing the opposite of the prediction. To avoid this issue, we
minimize the entropy of classified outputs during the minimization step:

$\displaystyle\min_{\theta}\mathcal{L}_{\text{adv}}=$
$\displaystyle\min_{\theta}\mathbb{E}\bigg{[}\operatorname{CE}\left(y,h_{\psi}\left(f_{\theta}(\mathbf{x})\right)\right)-\alpha{H}\left(g_{\phi}\left(f_{\theta}(\mathbf{x})\right)\right)+\beta||\mathbf{x}-f_{\theta}(\mathbf{x})||_{2}^{2}\bigg{]}\;,$
(2)

where ${H}\left(g_{\phi}\left(f_{\theta}(\mathbf{x})\right)\right)$ is the
entropy of the distribution of private labels predicted from the anonymized
skeleton. Therefore, the optimal anonymizer yields the most confusing
skeletons to the private classifier. In the maximization step, we still
maximize the negative cross entropy
$-\alpha\operatorname{CE}\left(z,g_{\phi}\left(f_{\theta}(\mathbf{x})\right)\right)$
w.r.t. $\phi$ to train the private classifier. Figure 1 shows the overall
framework for the data anonymization.

Algorithm 1 Adversarial Anonymization

Pre-trained classifiers $h_{\psi}$ and $g_{\phi}$, $E$: # of epochs, $m$:
minibatch size, $k$: # of minimization steps

while until convergence do

for $t\leftarrow 1$ to $k$ do

Sample minibatch of $m$ samples $\\{(\mathbf{x}_{i},y_{i},z_{i})\\}_{i=1}^{m}$

Compute $\nabla_{\theta}\mathcal{L}_{\text{adv}}$ with minibatch
$\triangleright$ Equation 2

Update $\theta\leftarrow\theta-\nabla_{\theta}\mathcal{L}_{\text{adv}}$

end for

Sample minibatch of $m$ samples $\\{(\mathbf{x}_{i},y_{i},z_{i})\\}_{i=1}^{m}$

Compute
$\nabla_{\phi}\alpha\operatorname{CE}\left(z,g_{\phi}\left(f_{\theta}(\mathbf{x})\right)\right)$
with minibatch

Update
$\phi\leftarrow\phi-\nabla_{\phi}\alpha\operatorname{CE}\left(z,g_{\phi}\left(f_{\theta}(\mathbf{x})\right)\right)$

end while

Alternating minimization and maximization are often employed to solve a
minimax objective as shown in the generative adversarial network [8].
Following previous work, we also use the alternating algorithm to optimize the
objective. Algorithm 1 shows the overall training algorithm. In this work, the
adversarial learning algorithm starts with pre-trained classifiers $g_{\phi}$
and $h_{\psi}$ to make the learning stable.

#### Anonymizer networks.

The anonymizer $f_{\theta}$ can be any prediction model that modifies
skeletons while preserving the original dimension. We employ two simple neural
network architectures for the anonymizer: 1) residual networks and 2) U-net
architectures.

First, the residual network [10] anonymizer adopts a simple residual
connection from the input skeletons to the output skeletons. Specifically, the
model can be formalized as

$\displaystyle
f_{\theta}(\mathbf{x})=\operatorname{MLP}_{\theta}(\mathbf{x})+\mathbf{x},$

where $\operatorname{MLP}_{\theta}:\mathbb{R}^{D\times
3}\rightarrow\mathbb{R}^{D\times 3}$ is a simple multi-layered perceptron
parameterized by $\theta$. The residual connection keeps the position similar
to the original skeleton while the $\operatorname{MLP}$ layer models the
disposition of joints to anonymize. We use two fully-connected layers to model
the disposition. The anonymizer is applied to each frame of a skeleton
sequence. Although the anonymizer is applied to each frame independently, the
back-propagated signals from action and private classifiers make the entire
sequence coherent. By initializing $\theta$ with weights close to zero, we
make the anonymizer add a small random noise to the original skeleton in the
early stage of learning.

Second, the U-Net [20] architecture is adopted to our anonymizer network. The
U-Net consists of two paths: the contracting path, and the expanding path. In
the contracting path, it repeats downsampling and maxpool an input skeleton
data to encode it to the feature map. In the expanding path, U-Net repeats
upsampling and concatenating feature map via skip connection. Especially, skip
connections concatenates the features from the contracting path to the
corresponding level in the expanding path. It makes the output skeleton
position similar to the original skeleton. The detailed architecture is
provided in Appendix B.

##  3 Experiments

In this section, we demonstrate the performance of the proposed framework for
anonymizing skeleton datasets. We use two publicly available datasets and
anonymize two different types of private information: gender and identity.

###  3.1 Datasets

We use two datasets: ETRI-activity3D [12] and NTU RGB+D 60 (NTU60) [22]. For
the ETRI-activity3D dataset, we anonymize the gender information from the
skeletons. For the NTU60 dataset, we anonymize the identity of the skeletons.
The detailed experimental setups for these datasets are as follows.

#### ETRI-activity3D.

ETRI-activity3D is an action recognition dataset originally published for
recognizing the daily activities of the elderly and youths. It contains
112,620 skeleton sequence samples observed from 100 people, half of whom were
between the ages of 64 and 88 and the rest were in their 20s. The elderly
consist of 33 females and 17 males, and the young adults consist of 25 females
and 25 males. The samples are categorized into 55 classes based on the
activity type. Each action is captured from 8 different Kinect v2 sensors to
provide multiple views. Each sequence consists of 3D locations of 25 joints of
the human body.

With the ETRI-activity3D dataset, we anonymize the gender information from the
skeletons. We drop samples from 5 classes for the following experiments, e.g.,
handshaking, containing two people, so only one person appears in the
remaining samples. After removing malformed and two-person samples, we split
the remaining samples into 68,788 and 34,025 training and validation,
respectively. We split the dataset according to the subject ID. In other
words, the subjects in the validation set do not appear in the training set.
Through this split, we measure the generalizability of the gender classifier
to the unknown subjects.

#### NTU60.

NTU60 is an action recognition dataset that contains 60 action classes and
56,880 skeleton sequences taken from 40 subjects. The format of skeleton data
is the same as ETRI-activity3D, which includes the 3D positions of 25 human
body joints. After removing malformed samples, we split the remaining samples
into 37,646 and 18,932 as training and validation sets, respectively.
Following the original work [22], we split it according to the camera ID so
that both sets contain identical subjects with different views.

###  3.2 Privacy Leakage

To verify privacy leakage from each dataset, we first check the performance of
gender classification and re-identification task. To train gender classifier
and re-identification task, three popular baseline models, Shift-GCN [2],
MS-G3D [15], and 2s-AGCN [23], are adopted. For gender classifier with MS-G3D,
we use MS-G3D without a G3D module. This makes training faster without losing
too much accuracy. We train multiple times for gender classifier and re-
identification task. Each model is trained with a different random
initialization.

After training, the gender classifier achieves 87% accuracy on average. The
re-identification task achieves 80% and 97% for top-1 and top-5 accuracy,
respectively. The detailed results are available in Appendix A. As the results
suggest, the privacy information can be easily predicted by a classification
model trained with private labels. Note that the test splits do not contain
the subject used for gender classification. This reveals the generalizability
of gender classification to the unseen subjects. Also, for the re-
identification task, train split and test split have different camera IDs, so
the same person appears in both sets with different views. This result
indicates that the joint trajectory contains personal traits that can be
easily exploited to identify a person.

###  3.3 Anonymization Results

(a) Re-identification task

(b) Gender classification

Figure 2: Action and privacy accuracy of three baseline models with two
different anonymizers after anonymization. $y$-axis is reversed. Note that,
before anonymization, the average top-1 re-identification accuracy is 80%, and
the average gender classification accuracy is 87%.

Our preliminary study indicates that gender and identification can leak enough
from training. Based on the results obtained in the previous experiments, we
evaluate the performance of anonymization with an adversarial learning
algorithm. As mentioned earlier, experiments are conducted with two anonymizer
networks (ResNet [10], U-Net [20]), two datasets (NTU60 [22], ETRI-activity3D
[12]), and three baseline models (Shift-GCN [2], MS-G3D [15], and 2s-AGCN
[23]). For each task, we use two pre-trained classifiers for action and
privacy, respectively. One classifier is used to initialize the adversarial
algorithm, and the other is used to measure the accuracy after anonymization.

Figure 2 shows the results of anonymization with re-identification task and
gender classification. In general, we observe that we can dramatically
decrease the privacy accuracy while minimally sacrificing the action
recognition accuracy. We also observe more leakage of private information when
the action accuracy is relatively higher. Note that we use a balanced test set
for identity anonymization. Since NTU60 has 40 subjects, one can achieve 2.5%
accuracy with random classification. For the gender classification, one can
achieve 50% accuracy with a random classifier on the test set.

One would expect the trade-off between action accuracy and privacy accuracy
based on the choice of hyperparameters $\alpha$ and $\beta$. The choice of the
best anonymization model may vary depending on the application. In this work,
we report the performance of the best model based on action accuracy $\times$
(1 - re-identification accuracy) from the results of various configurations.
According to our metric, ResNet ($\alpha$:1, $\beta$:10) and U-Net
($\alpha$:0.3, $\beta$:2) models are chosen as representative model. We
provide additional results with different configurations in the next section.
Note that one may use a different metric to select a model given different
application scenarios. We use the best configuration obtained from the
baseline model, Shift-GCN, to train the other baseline models. The detailed
configuration of hyperparameters used to plot Figure 2 are available in
Appendix B.

#### Trade-off analysis.

(a) NTU60 - ResNet

(b) NTU60 - U-Net

(c) ETRI-activity3D - ResNet

(d) ETRI-activity3D - U-Net

Figure 3: The trade-off between action accuracy and privacy accuracy based on
a different configuration of hyperparameter $\alpha$ and $\beta$ on NTU60 and
ETRI-activity3D with two anonymizer networks (legend: $\alpha$-$\beta$). Note
that the y-axis is reversed.

We vary the value of $\alpha$ and $\beta$ to check the trade-off between
action accuracy and privacy leakage based on different configurations of
hyperparameters. We use Shift-GCN as a baseline model for analysis. 3(a) and
3(b) show the result with various hyperparameter configurations on the
identity anonymization task. 3(c) and 3(d) show the result of the gender
anonymization task. We can observe that given a fixed $\alpha$, increasing
$\beta$ increases the chance of privacy leakage as well as the action accuracy
showing the presence of the trade-off between the action accuracy and privacy
leakage.

(a) Gender classification

(b) Re-identification task

Figure 4: Accuracy and reconstruction error over epochs with the residual
anonymizer. ‘Train:Action’ and ‘Val:Action’ indicate the training and
validation accuracy of the action classification, and ‘Train:Privacy’ and
‘Val:Privacy’ indicate the training and validation accuracy of the privacy
classification.

We additionally plot how the validation accuracy changes over training
procedure with adversarial learning in Figure 4. For both datasets, the action
accuracy remains high over the epochs on both training and validation sets.
However, there is a gap between the training and validation accuracy on
privacy, which shows the overfitting in the classifier. Specifically, for
gender classification, the gender accuracy starts similar value. Then gender
accuracy increases on the train set and decreases on the validation set. For
the re-identification task, the re-identification accuracy drops first on both
training and validation sets. Note that the re-identification task achieves
80% accuracy. The validation accuracy at the first epoch indicates that the
re-identification task is more sensitive to the additional noise introduced by
random weights of the residual network than the gender classification.

#### Reconstruction error analysis.

Figure 5: The trade-off between reconstruction error and accuracy with the
residual anonymizer on NTU60.

A reconstruction error directly shows the level of difference between the
original and anonymized skeletons. Although we cannot directly set the level
of reconstruction error, we vary the parameters to obtain different levels of
reconstruction error and corresponding prediction accuracy. Please check the
Appendix B for the detailed hyperparameter settings. As shown in Figure 5,
there is a trade-off between reconstruction error and re-identification
accuracy. As we increase the reconstruction error, we can reduce the re-
identification accuracy. However, high reconstruction error yields low action
accuracy as well.

#### Qualitative analysis.

To qualitatively understand the effect of anonymization, we visualize one
example from the ETRI-activity3D dataset before and after anonymization in
Figure 6. The top and bottom rows show five selected frames before and after
anonymization for each figure, respectively. We can find some interesting
patterns from the visualization. For example, the length of the neck bone is
slightly increased, and the bone is moved to the upright position after
anonymization. Given that an elderly female acts, we can conjecture that the
adjustment makes gender unrecognizable. More visualization results are
provided in Appendix D.

Figure 6: Five frames of the original (top) and the gender anonymized (bottom)
skeletons for an action “wiping face with a towel" from ETRI-activity3D. The
subject is an elderly female.

##  4 Related work

Our work lies in public data anonymization and skeleton-based action
recognition. In this section, we provide the previous attempts at datasets
anonymization and skeleton-based action recognition.

###  4.1 Public dataset anonymization

Researchers have pointed out privacy issues with public visual datasets and
tried to mitigate them. [1], [7], and [31] propose a blurring approach where
the privacy-sensitive regions are blurred with an object detection method. [6]
and [28] propose an inpainting method to remove potentially problematic
objects such as pedestrians and vehicles. [28] adopt a generative neural
network model for the inpainting task, and show more realistic results. A
large body of prior work has used GANs[8] to preserve visual private
information. [19], [17], and [11] uses GANs to generate fake faces to replace
real ones. Also, [9] proposes a face identity transformer that anonymizes face
information according to the given password.

There are also some works that exist for other domains. For example, [4] blurs
the voice of people recorded in a public space by using a U-Net based sound
source separation model. Sümer et al. focus on anonymizing students’ voices in
a classroom scenario [26]. In text domain, [14] and [3] propose a minimax
problem to remove private information from text. Mosallanezhad et al. further
advance the text anonymization with a reinforcement learning algorithm [18].
Malekzadeh et al. focus on data from mobile sensors such as accelerometers and
gyroscopes and show the potential leakage of private information therein [16].
The authors introduce an autoencoder-based anonymizer and adversarial
optimization method.

Similar concerns are also made for skeleton datasets. Sinha et al. propose a
method to recognize persons from skeleton data [25]. This work focuses on gait
patterns extracted from human skeletons. The authors build a model with some
predefined features and tested an adaptive neural network and naïve Bayes
classifier to recognize the identity of persons. This implies the potential
privacy leakage from public datasets.

###  4.2 Skeleton-based action recognition

Human skeleton data is a sequence of graphs, where joints and bones are
represented as nodes and edges separately within a graph. In early times,
skeleton motion trajectories are embedded into a manifold space as points. The
relative distances between these points acted as clues for action recognition.
However, these models do not exploit the internal spatial relationship between
joints. Later, convolution neural networks (CNNs) are utilized to extract
spatial co-occurrence patterns between joints. Nevertheless, CNNs cannot model
a skeleton’s topological information.

Then, graph convolution networks (GCNs) are introduced to model these
topological relations. Nonetheless, basic GCNs are not suitable for human
skeleton sequences because they contain not only the 3D position of joints but
also the time series. Yan et al. introduce the spatial-temporal graph
convolutional networks (ST-GCN) [30]. They conduct graph convolution for
extracting spatial features and perform $1\times 1$ convolution over each
joint for capturing temporal variations. Following this line, various graph
neural architectures are proposed to extract features from the graphs. AS-GCN
[13] applies parametric adjacency matrices to substitute for the fixed
skeleton graph. Subsequently, AGC-LSTM [24] incorporates graph convolution
layers into a long short-term memory network (LSTM) as gate operations to
capture long-range temporal movements in action sequences. The 2s-AGCN model
[23] proposes bone features and learnable residual masks to enhance more
flexibly extracting skeletons’ structural information and ensembles the models
trained separately with joints and bones to improve the classification
accuracy. MS-G3D [15] introduces cross-spacetime skip connections, which
additionally connect all 1-hop neighbors across all time frames of a dilated
sliding window for direct information flow, and shows improved recognition
performance. Shift-GCN [2] applies shift graph operations and lightweight
point-wise convolutions to overcome computational complexity and inflexible
receptive fields of prior GCN-based studies.

In this work, we use Shift-GCN[2], MS-G3D[15], and 2s-AGCN[23] as a baseline
recognition model for private information. Although the original model is
developed to recognize the actions of skeletons, we empirically show the model
can successfully classify the private information with a proper training
procedure.

##  5 Conclusion

In this work, we investigate privacy leakage from publicly available skeleton
datasets. We show that although skeleton data may seemingly be privacy-
protective, recently proposed skeletal action recognizers are surprisingly
capable of extracting sensitive and identity information from these data. To
address this privacy leakage problem, we propose a learning framework. Our
experimental results reveal that the proposed method effectively removes the
privacy information while preserving the movement patterns. Note that the
anonymizers used in this work employ relatively simple architectures.
Experiments show that private information can be removed effectively even with
simple architectures. We leave the study of more advanced architectures for
future work since our goal is to show the potential vulnerability of the
skeletons and to provide a general framework to overcome.

## References

  * [1] Holger Caesar, Varun Bankiti, Alex H Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom.  nuscenes: A multimodal dataset for autonomous driving.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11621–11631, 2020.
  * [2] Ke Cheng, Yifan Zhang, Xiangyu He, Weihan Chen, Jian Cheng, and Hanqing Lu.  Skeleton-based action recognition with shift graph convolutional network.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 183–192, 2020.
  * [3] Maximin Coavoux, Shashi Narayan, and Shay B. Cohen.  Privacy-preserving neural representations of text.  In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1–10, Brussels, Belgium, October-November 2018. Association for Computational Linguistics.
  * [4] Alice Cohen-Hadria, Mark Cartwright, Brian McFee, and Juan Pablo Bello.  Voice anonymization in urban sound recordings.  In 2019 IEEE 29th International Workshop on Machine Learning for Signal Processing (MLSP), pages 1–6. IEEE, 2019.
  * [5] Sean Ryan Fanello, Ilaria Gori, Giorgio Metta, and Francesca Odone.  Keep it simple and sparse: Real-time action recognition.  Journal of Machine Learning Research, 14, 2013.
  * [6] Arturo Flores and Serge Belongie.  Removing pedestrians from google street view images.  In 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition-Workshops, pages 53–58. IEEE, 2010.
  * [7] Andrea Frome, German Cheung, Ahmad Abdulkader, Marco Zennaro, Bo Wu, Alessandro Bissacco, Hartwig Adam, Hartmut Neven, and Luc Vincent.  Large-scale privacy protection in google street view.  In 2009 IEEE 12th international conference on computer vision, pages 2373–2380. IEEE, 2009.
  * [8] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio.  Generative adversarial nets.  Advances in neural information processing systems, 27, 2014.
  * [9] Xiuye Gu, Weixin Luo, Michael S Ryoo, and Yong Jae Lee.  Password-conditioned anonymization and deanonymization with face identity transformers.  In European Conference on Computer Vision, pages 727–743. Springer, 2020.
  * [10] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  * [11] Håkon Hukkelås, Rudolf Mester, and Frank Lindseth.  Deepprivacy: A generative adversarial network for face anonymization.  In International Symposium on Visual Computing, pages 565–578. Springer, 2019.
  * [12] Jinhyeok Jang, Dohyung Kim, Cheonshu Park, Minsu Jang, Jaeyeon Lee, and Jaehong Kim.  Etri-activity3d: A large-scale rgb-d dataset for robots to recognize daily activities of the elderly.  In 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 10990–10997. IEEE, 2020.
  * [13] Maosen Li, Siheng Chen, Xu Chen, Ya Zhang, Yanfeng Wang, and Qi Tian.  Actional-structural graph convolutional networks for skeleton-based action recognition.  In Conference on Computer Vision and Pattern Recognition (CVPR), pages 3595–3603, 2019.
  * [14] Yitong Li, Timothy Baldwin, and Trevor Cohn.  Towards robust and privacy-preserving text representations.  In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 25–30, Melbourne, Australia, July 2018. Association for Computational Linguistics.
  * [15] Ziyu Liu, Hongwen Zhang, Zhenghao Chen, Zhiyong Wang, and Wanli Ouyang.  Disentangling and unifying graph convolutions for skeleton-based action recognition.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
  * [16] Mohammad Malekzadeh, Richard G Clegg, Andrea Cavallaro, and Hamed Haddadi.  Mobile sensor data anonymization.  In Proceedings of the international conference on internet of things design and implementation, pages 49–58, 2019.
  * [17] Maxim Maximov, Ismail Elezi, and Laura Leal-Taixé.  Ciagan: Conditional identity anonymization generative adversarial networks.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5447–5456, 2020.
  * [18] Ahmadreza Mosallanezhad, Ghazaleh Beigi, and Huan Liu.  Deep reinforcement learning-based text anonymization against private-attribute inference.  In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2360–2369, 2019.
  * [19] Zhongzheng Ren, Yong Jae Lee, and Michael S Ryoo.  Learning to anonymize faces for privacy preserving action detection.  In Proceedings of the european conference on computer vision (ECCV), pages 620–636, 2018.
  * [20] Olaf Ronneberger, Philipp Fischer, and Thomas Brox.  U-net: Convolutional networks for biomedical image segmentation.  In International Conference on Medical image computing and computer-assisted intervention, pages 234–241. Springer, 2015.
  * [21] Alessia Saggese, Nicola Strisciuglio, Mario Vento, and Nicolai Petkov.  Learning skeleton representations for human action recognition.  Pattern Recognition Letters, 118:23–31, 2019.
  * [22] Amir Shahroudy, Jun Liu, Tian-Tsong Ng, and Gang Wang.  Ntu rgb+ d: A large scale dataset for 3d human activity analysis.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1010–1019, 2016.
  * [23] Lei Shi, Yifan Zhang, Jian Cheng, and Hanqing Lu.  Two-stream adaptive graph convolutional networks for skeleton-based action recognition.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 12026–12035, 2019.
  * [24] Chenyang Si, Wentao Chen, Wei Wang, Liang Wang, and Tieniu Tan.  An attention enhanced graph convolutional lstm network for skeleton-based action recognition.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1227–1236, 2019.
  * [25] Aniruddha Sinha, Kingshuk Chakravarty, and Brojeshwar Bhowmick.  Person identification using skeleton information from kinect.  In Proc. Intl. Conf. on Advances in Computer-Human Interactions, pages 101–108, 2013.
  * [26] Ömer Sümer, Peter Gerjets, Ulrich Trautwein, and Enkelejda Kasneci.  Automated anonymisation of visual and audio data in classroom studies.  arXiv preprint arXiv:2001.05080, 2020.
  * [27] Du Tran, Heng Wang, Lorenzo Torresani, Jamie Ray, Yann LeCun, and Manohar Paluri.  A closer look at spatiotemporal convolutions for action recognition.  In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pages 6450–6459, 2018.
  * [28] Ries Uittenbogaard, Clint Sebastian, Julien Vijverberg, Bas Boom, Dariu M Gavrila, et al.  Privacy protection in street-view panoramas using depth and multi-view imagery.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10581–10590, 2019.
  * [29] Hongsong Wang and Liang Wang.  Learning content and style: Joint action recognition and person identification from human skeletons.  Pattern Recognition, 81:23–35, 2018.
  * [30] Sijie Yan, Yuanjun Xiong, and Dahua Lin.  Spatial temporal graph convolutional networks for skeleton-based action recognition.  In Thirty-second AAAI conference on artificial intelligence, 2018\.
  * [31] Kaiyu Yang, Jacqueline Yau, Li Fei-Fei, Jia Deng, and Olga Russakovsky.  A study of face obfuscation in imagenet.  arXiv preprint arXiv:2103.06191, 2021.

##  Appendix A Privacy Leakage

Table 1 and Table 2 provide the detailed results of privacy leakage
experiments. We use three baseline models: Shift-
GCN111https://github.com/kchengiva/Shift-GCN,
MS-G3D222https://github.com/kenziyuliu/MS-G3D, and
2s-AGCN333https://github.com/lshiwjx/2s-AGCN. In this experiment, we set all
hyperparameter configurations as default values given baseline models. To
obtain a stable result, multiple training with random initialization is
conducted with two NVIDIA GeForce RTX 3090 or two NVIDIA RTX A5000.

| Top-1 | standard deviation | Top-5 | standard deviation
---|---|---|---|---
Shift-GCN | 0.7962 | 0.0070 | 0.9681 | 0.0009
MS-G3D | 0.8223 | 0.0087 | 0.9751 | 0.0007
2s-AGCN | 0.7689 | 0.0183 | 0.9656 | 0.0032
Table 1: Top-1 and Top-5 accuracy of re-identification task with NTU60 dataset. The re-identification task achieves 80% and 97% for top-1 and top-5 accuracy on average. | accuracy | standard deviation
---|---|---
Shift-GCN | 0.8599 | 0.0040
MS-G3D | 0.8790 | 0.0017
2s-AGCN | 0.8643 | 0.0047
Table 2: Accuracy of gender classification with ETRI-activity3D dataset. The
gender classifier achieves 87% accuracy on average.

##  Appendix B Detailed Experimental Results

In this section, detailed hyperparameter configurations and results of our
main experiments are provided. The code used in our experiments is available
at https://anonymous.4open.science/r/Skeleton-anonymization-90B7/. Since the
implementation is similar for all baselines, we only provide the code based on
Shift-GCN. Also, data related to ETRI-activity3D will not be provided, since
ETRI-activity3D can access by authorized users. If one wants to use this
dataset for research purposes, please visit
here444https://ai4robot.github.io/etri-activity3d-en/ and get permission.
Additionally, we use U-Net architecture from Pytorch-UNet
repository555https://github.com/milesial/Pytorch-UNet for our anonymizer
network. For all experiments, we set $k=1$ (see in Algorithm 1), i.e., one
minimization with one maximization. We use four NVIDIA GeForce RTX 3090 or
four NVIDIA RTX A5000 for training.

#### Anonymization results.

Table 3 and Table 4 provide the detailed hyperparameter configurations and the
entire results of Figure 2. For all experiments, we train Shift-GCN to find a
representative model at first. After finding the best $\alpha$ and $\beta$,
the same $\alpha$ and $\beta$ are applied to the other two baselines with
different learning rates.

Model | $\alpha$ | $\beta$ | learning rate | Re-iden. acc. | action acc. | recon. error
---|---|---|---|---|---|---
Shift-GCN | ResNet | 1.0 | 10 | 0.0100 | 0.04202 | 0.9175 | 0.00860
U-Net | 0.3 | 2 | 0.0100 | 0.05701 | 0.9145 | 0.01655
MS-G3D | ResNet | 1.0 | 10 | 0.0005 | 0.07770 | 0.9215 | 0.01092
U-Net | 0.3 | 2 | 0.0005 | 0.10110 | 0.9243 | 0.01115
2s-AGCN | ResNet | 1.0 | 10 | 0.0050 | 0.05889 | 0.8994 | 0.01446
U-Net | 0.3 | 2 | 0.0050 | 0.03514 | 0.8908 | 0.07536
Table 3: Detail hyperparameter configurations and results of our representative model with three baselines for anonymizing identity information using NTU60. Model | $\alpha$ | $\beta$ | learning rate | gender acc. | action acc. | recon. error
---|---|---|---|---|---|---
Shift-GCN | ResNet | 2.0 | 15 | 0.010 | 0.5875 | 0.8863 | 0.001273
U-Net | 0.3 | 0.5 | 0.010 | 0.5671 | 0.8708 | 0.010860
MS-G3D | ResNet | 2.0 | 15 | 0.001 | 0.6426 | 0.9073 | 0.001075
U-Net | 0.3 | 0.5 | 0.001 | 0.6046 | 0.9033 | 0.004032
2s-AGCN | ResNet | 2.0 | 15 | 0.005 | 0.4221 | 0.8758 | 0.003965
U-Net | 0.3 | 0.5 | 0.005 | 0.5645 | 0.8563 | 0.009198
Table 4: Detail hyperparameter configurations and results of our
representative model with three baselines for anonymizing gender information
using ETRI-activity3D.

#### Trade-off analysis results.

Table 5 and Table 6 show the detailed results of Figure 3. We vary
hyperparameter $\alpha$ and $\beta$ to observe the trade-off between action
accuracy and privacy accuracy. For these experiments, we use Shift-GCN and set
the learning rate as 0.01.

$\alpha$ | $\beta$ | Action acc. | Re-iden. acc. | Recon. error (RMSE) | Anonymization Network
---|---|---|---|---|---
1 | 5 | 0.9191 | 0.08303 | 0.00980 | ResNet
1 | 10 | 0.9175 | 0.04202 | 0.00860 | ResNet
1 | 15 | 0.9351 | 0.17070 | 0.00380 | ResNet
1 | 20 | 0.9235 | 0.12010 | 0.00550 | ResNet
2 | 5 | 0.8909 | 0.13190 | 0.02431 | ResNet
2 | 10 | 0.8838 | 0.09381 | 0.01822 | ResNet
2 | 15 | 0.9113 | 0.14690 | 0.01008 | ResNet
2 | 20 | 0.8728 | 0.07691 | 0.01020 | ResNet
3 | 5 | 0.8731 | 0.08266 | 0.02557 | ResNet
3 | 10 | 0.8814 | 0.07500 | 0.01340 | ResNet
3 | 15 | 0.8891 | 0.09286 | 0.01649 | ResNet
3 | 20 | 0.8595 | 0.10550 | 0.01410 | ResNet
0.3 | 0.5 | 0.9188 | 0.08657 | 0.02463 | U-Net
0.3 | 1 | 0.9206 | 0.07622 | 0.01945 | U-Net
0.3 | 1.5 | 0.9209 | 0.07891 | 0.01513 | U-Net
0.3 | 2 | 0.9145 | 0.05701 | 0.01655 | U-Net
0.5 | 0.5 | 0.9060 | 0.08045 | 0.02959 | U-Net
0.5 | 1 | 0.9074 | 0.06956 | 0.06496 | U-Net
0.5 | 1.5 | 0.9099 | 0.07791 | 0.02055 | U-Net
0.5 | 2 | 0.9071 | 0.05509 | 0.01750 | U-Net
0.7 | 0.5 | 0.8945 | 0.06597 | 0.12950 | U-Net
0.7 | 1 | 0.9022 | 0.05277 | 0.07635 | U-Net
0.7 | 1.5 | 0.8988 | 0.08494 | 0.02410 | U-Net
0.7 | 2 | 0.9006 | 0.04352 | 0.02232 | U-Net
Table 5: Results with varying hyperparameters for analyzing the trade-off
between action accuracy and re-identification accuracy with NTU60.

$\alpha$ | $\beta$ | Action acc. | Gender acc. | Recon. error (RMSE) | Anonymization Network
---|---|---|---|---|---
1 | 5 | 0.9028 | 0.6648 | 0.0017160 | ResNet
1 | 10 | 0.9032 | 0.6778 | 0.0007986 | ResNet
1 | 15 | 0.9050 | 0.7643 | 0.0004499 | ResNet
1 | 20 | 0.9032 | 0.6655 | 0.0003264 | ResNet
2 | 5 | 0.8940 | 0.6569 | 0.0061400 | ResNet
2 | 10 | 0.8925 | 0.6296 | 0.0023480 | ResNet
2 | 15 | 0.8863 | 0.5875 | 0.0012730 | ResNet
2 | 20 | 0.9011 | 0.7352 | 0.0011160 | ResNet
3 | 5 | 0.8890 | 0.6612 | 0.0061500 | ResNet
3 | 10 | 0.8934 | 0.6512 | 0.0035020 | ResNet
3 | 15 | 0.9031 | 0.7601 | 0.0025420 | ResNet
3 | 20 | 0.8958 | 0.6558 | 0.0020570 | ResNet
0.3 | 0.5 | 0.8708 | 0.5671 | 0.0108600 | U-Net
0.3 | 1 | 0.8754 | 0.5798 | 0.0094480 | U-Net
0.3 | 1.5 | 0.8337 | 0.5892 | 0.0264000 | U-Net
0.3 | 2 | 0.8758 | 0.5952 | 0.0108300 | U-Net
0.5 | 0.5 | 0.8569 | 0.5797 | 0.0156400 | U-Net
0.5 | 1 | 0.8303 | 0.5905 | 0.0276500 | U-Net
0.5 | 1.5 | 0.8341 | 0.5853 | 0.0249800 | U-Net
0.5 | 2 | 0.8567 | 0.5823 | 0.0117300 | U-Net
0.7 | 0.5 | 0.8239 | 0.6053 | 0.0294500 | U-Net
0.7 | 1 | 0.8265 | 0.5978 | 0.0326100 | U-Net
0.7 | 1.5 | 0.8269 | 0.5984 | 0.0277300 | U-Net
0.7 | 2 | 0.7504 | 0.6046 | 0.0469700 | U-Net
Table 6: Results with varying hyperparameters for analyzing the trade-off
between action accuracy and gender accuracy with ETRI-activity3D.

#### Reconstruction error analysis results.

Table 7 provides the detailed results of Figure 5. To obtain different levels
of reconstruction error, we vary $\beta$ with fixed $\alpha$ as 1. For these
experiments, we use Shift-GCN with NTU60 and set the learning rate as 0.01.
Multiple training with random initialization is conducted for stable results.

$\beta$ | Recon. error | Action | Re-identification
---|---|---|---
Accuracy | std | Accuracy | std
5 | 0.012920 | 0.9144 | 0.002025 | 0.08016 | 0.016120
10 | 0.008804 | 0.9138 | 0.005149 | 0.03964 | 0.006796
25 | 0.004570 | 0.9210 | 0.004780 | 0.08288 | 0.030340
50 | 0.002268 | 0.9327 | 0.002413 | 0.12080 | 0.038370
75 | 0.001636 | 0.9385 | 0.002549 | 0.15790 | 0.044330
Table 7: Detail results of reconstruction error analysis. std means standard
deviation.

##  Appendix C Hyperparameter Analysis

In this section, we analyze the hyperparameter used in our objective function
(See Equation 2) with action accuracy, re-identification accuracy, and
reconstruction error. There are two hyperparameters: $\alpha$ and $\beta$.
$\alpha$ adjusts the importance of privacy, one expects that large $\alpha$
may occur low privacy accuracy. Also, $\beta$ adjusts the reconstruction
error, which regularizes anonymized data to look similar to original data. We
conduct this experiment with Shift-GCN, NTU60, and ResNet.

#### Alpha term analysis.

Figure 7 is the result of different $\alpha$. In this experiment, we vary
$\alpha$ with fixed $\beta$ of 10. As shown in 7(a), action accuracy
decreases, as $\alpha$ increases. Also, re-identification accuracy drops
rapidly from 0.1 to 1.0, but increases at 2.0 and decreases again. Note that
$\alpha$: 1, $\beta$: 10 is the best model in our metric (See subsection 3.3).
This means that optimizing the minimax objective works well under a certain
$\alpha$ so that growing $\alpha$ produces low re-identification accuracy.
However, $\alpha$ that exceeds a particular value disturbs optimization due to
a trade-off with action accuracy and reconstruction error. Also, 7(b) shows
that reconstruction error increases, as $\alpha$ increases. Since the ratio of
$\beta$ to $\alpha$ decreases, so regularizing anonymized data can not
influence enough during training.

(a) Re-identification and Action accuracy

(b) Reconstruction error

Figure 7: Re-identification accuracy, action accuracy, and reconstruction
error with different $\alpha$

#### Beta term analysis.

Figure 8 is the result of according to different $\beta$. In this experiment,
we vary $\beta$ with fixed $\alpha$ of 1. As shown in 8(b), it is trivial that
reconstruction error increases by raising $\beta$. This means that large
$\beta$ makes anonymized data look similar to original data. This affects re-
identification accuracy and action accuracy. 8(a) shows that action accuracy
and re-identification accuracy increase, as $\beta$ increases.

(a) Re-identification and Action accuracy

(b) Reconstruction error

Figure 8: Re-identification accuracy, action accuracy, and reconstruction
error with different $\beta$

##  Appendix D Quantitative Analysis

(a) The original (top) and the gender anonymized (bottom) skeletons for an
action “wiping face with a towel" from ETRI-activity3D. The subject is an
elderly female.

(b) The original (top) and the gender anonymized (bottom) skeletons for an
action “drinking water" from ETRI-activity3D. The subject is an elderly male.

(c) The original (top) and the identity anonymized (bottom) skeletons for an
action “kicking something" from NTU60. The subject’s ID is 2.

Figure 9: Examples of anonymized skeletons. Five frames are visualized from a
sequence of action frames.


Huyen Le, Khiet Dang, Tien Lai, Nhung Nguyen, Mai Tran, and Hieu Pham
11institutetext: VinUni-Illinois Smart Health Center, VinUniversity
22institutetext: College of Engineering & Computer Science, VinUniversity
33institutetext: College of Health Science, VinUniversity
Corresponding address<EMAIL_ADDRESS>
* These authors contributed equally

# SarcNet: A Novel AI-based Framework to Automatically Analyze and Score
Sarcomere Organizations in Fluorescently Tagged hiPSC-CMs

Huyen Le 11**** Khiet Dang 11**** Tien Lai 22 Nhung Nguyen 33 Mai Tran 1122
Hieu Pham 1122**

###### Abstract

Quantifying sarcomere structure organization in human-induced pluripotent stem
cell-derived cardiomyocytes (hiPSC-CMs) is crucial for understanding cardiac
disease pathology, improving drug screening, and advancing regenerative
medicine. Traditional methods, such as manual annotation and Fourier transform
analysis, are labor-intensive, error-prone, and lack high-throughput
capabilities. In this study, we present a novel deep learning-based framework
that leverages cell images and integrates cell features to automatically
evaluate the sarcomere structure of hiPSC-CMs from the onset of
differentiation. This framework overcomes the limitations of traditional
methods through automated, high-throughput analysis, providing consistent,
reliable results while accurately detecting complex sarcomere patterns across
diverse samples. The proposed framework contains the SarcNet, a linear layers-
added ResNet-18 module, to output a continuous score ranging from one to five
that captures the level of sarcomere structure organization. It is trained and
validated on an open-source dataset of hiPSC-CMs images with the endogenously
GFP-tagged alpha-actinin-2 structure developed by the Allen Institute for Cell
Science (AICS). SarcNet achieves a Spearman correlation of 0.831 with expert
evaluations, demonstrating superior performance and an improvement of 0.075
over the current state-of-the-art approach, which uses linear regression. Our
results also show a consistent pattern of increasing organization from day 18
to day 32 of differentiation, aligning with expert evaluations. By integrating
the quantitative features calculated directly from the images with the visual
features learned during the deep learning model, our framework offers a more
comprehensive and accurate assessment, enhancing the further utility of hiPSC-
CMs in medical research and therapy development.

###### keywords:

hiPSC-CMs, sarcomere structure organization, SarcNet, linear regression, deep
learning.

##  1 Introduction

Human-induced pluripotent stem cell-derived cardiomyocytes (hiPSC-CMs) are
cells safely collected from the blood or skin of living humans and then
reprogrammed into human heart muscle cells [9]. Fully characterized hiPSC-CMs
have a range of critical applications, including disease modeling, drug
discovery, regenerative and precision medicine [3, 10, 12]. Specifically,
these cells have attracted considerable attention as a promising alternative
for modeling arrhythmogenic disorders and assessing the cardiac function of
patients [24]. In another instance, Vicente et al. (2018) introduced the
Comprehensive in vitro proarrhythmia Assay (CiPA) as a method to assess the
impact of drugs on various ion channels in hiPSC-CMs and to predict the risk
of proarrhythmia [22].

To effectively utilize these applications, hiPSC-CMs must acquire key features
and capacities that accurately reflect the electrical activity, calcium
dynamics, structure, and contractility of mature cardiomyocytes [3, 4]. During
the differentiation of hiPSC-CMs, gene expression varies, with immature
cardiomyocytes showing higher levels of the TNNI1 gene, while mature cells
primarily express TNNI3 [2, 8]. Therefore, it is necessary to establish a
framework to quantify hiPSC-CM maturation.

Quantitative approaches for tracking hiPSC-CMs development are vital but
currently restricted [3]. Traditional methods for assessing cardiomyocyte
structures include manual annotation, which is labor-intensive and lacks high-
throughput capabilities, and Fourier transform analysis on hiPSC-CMs
alignment, which is error-prone on images with thicker multi-cell layer [15].
Recently, various methodologies have been employed to assess the maturation of
hiPSC-CMs, focusing on distinct characteristics, such as sarcomere
configuration, electrophysiological attributes, metabolism, and gene
expression profiles [1]. For example, measuring the TNNI3 to TNNI1 ratio is
one way to quantify hiPSC-CMs based on gene expression [2]. The structure of
the sarcomere can be evaluated using hiPSC-CM imaging, an important technique
to consider. This is particularly relevant because the contraction
capabilities of adult cardiomyocytes are closely tied to the organization and
structure of the sarcomeres and myofilaments [21]. In particular, mature
hiPSC-CMs are notably longer and display a higher degree of structural
organization compared to their immature counterparts [17].

Several studies have employed artificial intelligence (AI) to quantify the
maturity of hiPSC-CMs by analyzing fluorescent images [19, 7]. The authors in
[19] (2015) first defined a set of 11 metrics to capture the increasing
organization of sarcomeres within striated muscle cells during their
developmental process and then used machine learning algorithms to score the
phenotypic maturity of hiPSC-CMs unbiasedly. However, due to the limited
availability of hiPSC-CMs at various developmental stages, the model was
trained on primary cardiomyocytes from neonate rats (rpCMs) and was tested on
immature hiPSC-CMs only. In [7] published in 2021, Gerbin, Kaytlyn A., et al.
used linear regression to distinguish stages of myofibrillar organization in
hiPSC-CMs at the single-cell level. They utilized 11 cell features, six of
which originated from deep learning. The model reached a Spearman correlation
of 0.63 and 0.67 on two test sets. Despite these advances, translation to
clinical application remains a distant goal.

To address the gap, in this study, we propose SarcNet, a cell-features
concatenated and linear layers-added ResNet-18 convolutional neural network
(CNN) for automatically quantifying sarcomere structure organizations on
single-cell images of hiPSC-CMs. In particular, this model leverages
predictions by concatenating the output from the ResNet-18 module and a
representation vector of quantitative single-cell measurements of subcellular
organization, followed by the addition of four linear layers to gain deeper
linear representations. To evaluate the proposed framework, we compare SarcNet
with a combined model, which integrates predictions from linear regression and
the ResNet-18 module to generate the final score. SarcNet obtains a higher
performance, with a Spearman correlation of 0.831. Furthermore, we assess the
performance of each approach using two different feature extraction protocols
as described in Section 2.3.1. While the performance of the combined model
drops significantly with Protocol 1, our SarcNet model maintains nearly
consistent performance. This result indicates that using SarcNet, the features
extracted by Protocol 1 can be sufficient for scoring sarcomere structure
organizations without a complicated deep learning-based feature extraction
method as in Protocol 2. In addition, SarcNet also outperforms linear
regression and other state-of-the-art CNN models. Finally, we evaluate the
proposed model using Gradient-weighted Class Activation Mapping (Grad-CAM)
[20], an explanatory algorithm. The heatmaps generated by Grad-CAM reveal that
the important regions in the correctly predicted images correspond to the
majority of sarcomere patterns within the cell. Our main contributions through
this work can be summarized as follows:

  1. 1.

We introduce a novel AI-based framework to effectively analyze and score
sarcomere organizations in fluorescently tagged hiPSC-CMs single cell images
by integrating feature extraction with the existing state-of-the-art CNN
model.

  2. 2.

We conduct thorough experiments to evaluate the efficacy of the proposed
solution and compare it to other methods. The experimental findings
demonstrated that our SarcNet model with 11 features extracted by Protocol 2
enhances performance by at least 0.075 and 0.022 for the Spearman correlation
metric compared to linear regression and the combined model. Notably, using
only five features from Protocol 1 does not significantly affect the model
performance.

  3. 3.

We can speed up the training and inference by simplifying the feature
extraction process and decreasing the number of features while maintaining
performance. Our codes and models will be released at
https://github.com/vinuni-vishc/sarcnet.

The remainder of the paper is organized as follows. Section 2 describes an
overview of the proposed framework for quantifying sarcomere structure
organization. In Section 3, we assess the databases and discuss implementation
specifics. Section 4 presents the experiment results, visualizations, and
limitations. Section 5 concludes the paper.

##  2 Methodology

This section introduces details of the proposed approach. We first give an
overview of our framework for predicting a continuous score of sarcomere
structure organization on single-cell imaging of hiPSC-CMs (Section 2.1). We
then provide a formulation of the problem as a regression task (Section 2.2).
Next, the framework architecture is described (Section 2.3).

###  2.1 Overall Framework

Figure 1: An illustration of our overall framework, which aims to quantify
sarcomere structure organization. The system takes the hiPSC-CM images as
input and outputs the scores of alpha-actinin-2 patterns, ranging from one to
five. For feature extraction, three sets with a total of 11 cell features can
be extracted using the techniques described in [7], including CellProfiler-
based feature extraction, Gray-Level Co-occurrence Matrix-based (GLCM-based)
feature extraction and deep learning-based feature extraction. Our model also
localizes the most relevant portions of the image using a heatmap generated by
the Grad-CAM approach and proclaims the increasing organization between day 18
and day 32 using the histogram.

Fig. 1 provides an overview of the proposed approach, which is a regression
framework using CNN. It predicts a continuous score of sarcomere structure
organization by taking fluorescent images of hiPSC-CMs single cells as input.
To train the proposed model, an expert-scored hiPSC-CMs single cells dataset
of 5,761 images has been used (Section 3.1). We enhance the prediction
performance by integrating feature extraction into the model architecture.
Model performance is evaluated using four metrics: Spearman correlation, mean
absolute error (MAE), mean squared error (MSE), and $R^{2}$ score (Section
3.3). A visual explanation module based on Grad-CAMs is also used for model
interpretation. Last but not least, we confirm the pattern of increasing
maturity levels of sarcomere structure between day 18 and day 32-time points.

###  2.2 Problem Formulation

In a regression problem setting, we are given a training set $\mathcal{D}$
consisting of N samples, $\mathcal{D}$ = {$(x^{(i)},y^{(i)}$); i = 1,…, N}
where each input image $x^{(i)}\in\mathcal{X}$ is associated with a continuous
value $y^{(i)}\in\mathcal{Y}$. Our goal is to approximate a mapping function
$f_{\theta}:\mathcal{X}\rightarrow\mathcal{Y}$ from input images, which are
fluorescently tagged single-cell images of hiPSC-CMs to continuous output
variables representing the organization of sarcomere structures. In general,
this learning task could be performed by training a CNN, parameterized by
weight $\theta$ that the MSE loss function is minimized over the training set
$\mathcal{D}$. As the desired output is a continuous quantity, the activation
function at the output layer is linear. The loss function MSE is given by

$\mathcal{L}(\theta)=\frac{1}{N}\sum_{i=1}^{N}|y^{(i)}-\hat{y}^{(i)}|^{2},$
(1)

where $y^{(i)}$ is the ground truth and $\hat{y}^{(i)}$ is the value predicted
by the network.

###  2.3 Framework Architecture

####  2.3.1 Feature Extraction

Three sets with a total of 11 cell features can be extracted for each input
image using techniques provided in [7]. More details, Set 1, the CellProfiler-
based feature extraction [18], generates two cell morphology measurement
metrics: cell area and aspect ratio. Set 2, the GLCM-based method, produces
three metrics describing sarcomere alignment: max coefficient of variation,
peak height, and peak distance. Finally, Set 3, the deep learning-based
method, provides six additional metrics representing the fraction of the cell
area covered by each class: background, diffuse/other, fibers, disorganized
puncta, organized puncta, and organized z-discs. The features extracted from
Set 1 and Set 2 together are referred to as Protocol 1. The features extracted
from Set 1, Set 2, and Set 3 together are referred to as Protocol 2.

####  2.3.2 ResNet-18 Module

ResNet-18 [11], one of the state-of-the-art image classification models, is
adapted for the single-cell image classification task. It receives input
images of size $3\times 224\times 224$ and generates labels ranging from one
to five, corresponding to the score of sarcomere structure organization. The
ResNet-18 module architecture used in this study consists of 18 convolution
layers, an average pooling layer, and three linear layers. Compared to the
original architecture, two fully connected layers are added at the end before
outputting the result.

####  2.3.3 SarcNet

Figure 2: SarcNet architecture for scoring sarcomere structure organization,
which are continuous values ranging from one to five.

In this study, we propose the SarcNet, which comprises two branches of the
ResNet-18 module and linear layers as depicted in Fig. 2. First, the former
requires an input image with a size of $3\times 224\times 224$ to pass through
the ResNet-18 module (described in 2.3 \- ResNet-18 module). Regarding the
second branch, the input feature vector would be passed through three linear
layers with rectified linear unit (ReLU) activation functions to extract more
deep linear representations. After concatenating with the output from the
ResNet-18 module, a new representation vector is created, consisting of the
information contributed equally from both input images and input features.
This vector is then processed through four linear layers to get the final
score.

##  3 Experiments

###  3.1 Dataset and Settings

####  3.1.1 Dataset

In this study, we use a publicly available dataset of alpha-actinin-2-mEGFP-
tagged hPSC-CMs single cells at days 18 and 32 time points since the
initiation of differentiation, provided by the Allen Institute for Cell
Science (AICS) [7]. The dataset consists of two components: images of single
cells and tabular data containing cell features, along with the overall alpha-
actinin-2 organization score corresponding to each image.

As described [7], each cell is manually scored for the structural maturity of
its sarcomere organization by two experts; thus, all cells are assessed twice.
Based on the majority of their organization, experts categorize cells into
five score groups, from one to five. A score of 1 indicates cells with sparse,
disorganized puncta; a score of 2 indicates cells with denser, more organized
puncta; a score of 3 indicates cells with a combination of puncta and other
structures like fibers and z-discs; a score of 4 indicates cells with regular
but not aligned z-discs and finally a score of 5 indicates cells with almost
organized and aligned z-discs. Cells lacking alpha-actinin-2 mEGFP protein
expression were assigned a score of “0” and subsequently excluded from further
analysis. After filtering, there are a total of 5,761 images of different
sizes. The number of cell images in each score category is shown in Table 1.
For consistency, we define ground truth (GT) in our downstream analysis as the
mean score from two experts.

Table 1: Number of cells in each score category | Score 1 | Score 2 | Score 3 | Score 4 | Score 5
---|---|---|---|---|---
Expert 1 | 293 | 708 | 3,868 | 798 | 94
Expert 2 | 115 | 1,007 | 2,979 | 1,527 | 113

The tabular data component provides cell features extracted by CellProfiler,
GLCM, and deep-learning model. In this work, we utilize a subset of 11
features from the tabular data in the dataset package, including cell area,
cell aspect ratio, max coefficient of variation, peak height, peak distance,
fraction cell area background, fraction cell area diffuse/other, fraction cell
area fibers, fraction cell area disorganized puncta, fraction cell area
organized puncta, and fraction cell area organized z-discs. Table 2 shows a
statistical summary of the mean and standard deviation (SD) of these 11
metrics. It is evident that cell area, max coefficient of variation, and peak
distance increase as cells mature. Also, the fraction cell area background is
highest in cells scored as 1, while the fraction cell area organized z-disc is
highest in cells scored as 5. These features are utilized differently in the
two protocols: Protocol 1 employs the first five features, whereas Protocol 2
incorporates all 11 features, as depicted in Section 2.3.1.

Table 2: Statistical summary of cell features in each score category

Mean (SD) 1.0 1.5 2.0 2.5 3.0 3.5 4.0 4.5 5.0 Count 81 234 440 622 2,543 1,108
572 120 41 Cell area 1,732 (1,040) 1,620 (1,016) 1,925 (1,070) 1,923 (993)
2,272 (1,207) 2,557 (1,489) 3,090 (2,086) 2,669 (1,394) 2,717 (1,300) Set 1
Cell aspect ratio 0.49 (0.16) 0.50 (0.18) 0.48 (0.19) 0.50 (0.19) 0.50 (0.18)
0.54 (0.18) 0.50 (0.18) 0.46 (0.18) 0.46 (0.19) Max coefficient of variation
0.26 (0.16) 0.25 (0.16) 0.24 (0.14) 0.25 (0.14) 0.30 (0.15) 0.32 (0.15) 0.37
(0.16) 0.49 (0.16) 0.57 (0.17) Peak height 0.47 (0.09) 0.48 (0.10) 0.44 (0.09)
0.44 (0.09) 0.44 (0.10) 0.44 (0.10) 0.46 (0.10) 0.55 (0.08) 0.60 (0.07) Set 2
Peak distance 1.50 (0.50) 1.72 (0.41) 1.69 (0.31) 1.72 (0.25) 1.76 (0.19) 1.77
(0.12) 1.78 (0.10) 1.79 (0.08) 1.79 (0.06) Fraction cell area background 0.20
(0.28) 0.10 (0.19) 0.04 (0.11) 0.04 (0.11) 0.03 (0.09) 0.02 (0.08) 0.02 (0.06)
0.04 (0.08) 0.05 (0.10) Fraction cell area diffuse/other 0.04 (0.06) 0.09
(0.09) 0.14 (0.11) 0.15 (0.12) 0.19 (0.14) 0.18 (0.14) 0.11 (0.10) 0.11 (0.09)
0.08 (0.07) Fraction cell area fiber 0.30 (0.22) 0.34 (0.20) 0.34 (0.19) 0.32
(0.19) 0.27 (0.18) 0.14 (0.12) 0.09 (0.10) 0.08 (0.09) 0.06 (0.07) Fraction
cell area disorganized puncta 0.09 (0.11) 0.13 (0.13) 0.20 (0.15) 0.15 (0.13)
0.08 (0.09) 0.03 (0.06) 0.01 (0.02) 0.01 (0.02) 0.01 (0.02) Fraction cell area
organized puncta 0.35 (0.23) 0.34 (0.20) 0.27 (0.17) 0.30 (0.20) 0.30 (0.18)
0.29 (0.18) 0.26 (0.16) 0.20 (0.13) 0.16 (0.09) Set 3 Fraction cell area
organized z-discs 0.01 (0.02) 0.01 (0.02) 0.01 (0.03) 0.03 (0.05) 0.13 (0.14)
0.34 (0.19) 0.49 (0.20) 0.56 (0.19) 0.63 (0.19)

####  3.1.2 Settings

To evaluate the effectiveness of the proposed method, several experiments have
been conducted. We divide the dataset into training, validation, and testing
sets, resulting in 3,686 cells for training, 922 cells for validation, and
1,153 cells for testing. For all experiments, we preprocess the images by
resizing them to a size of $3\times 224\times 224$ to align with the ResNet-18
input shape, followed by normalization; and preprocess the tabular data by
applying standard scaler from the Scikit-learn library. To investigate the
impact of feature extraction, we perform the proposed method on both Protocol
1 and Protocol 2.

###  3.2 Training Methodology

In this study, an Adam optimizer is used with a learning rate of 0.0005 to
optimize the MSE objective function (Section 2.2). The models are trained with
a batch size of 40 in 100 epochs using the Pytorch framework. The network with
the highest Spearman correlation on the validation dataset is selected and
continuously tested on the testing dataset. All models are trained on a
GeForce RTX 3090 GPU.

###  3.3 Evaluation Metrics

The model performance is evaluated and measured using four metrics, including
Spearman correlation, MAE, MSE, and $R^{2}$ score. First, Spearman correlation
[13] is a non-parametric statistical method to assess the relationships
between two variables based on ranks. To calculate this coefficient, we first
order the two variables $\hat{y}$ and $y$ from least to greatest. Let
$D^{(i)}$ be the distance between the two corresponding ranks. The Spearman
correlation is computed using the formula

$\displaystyle r_{S}=1-\frac{6{\sum_{i=1}^{N}{D^{(i)}}^{2}}}{N(N^{2}-1)},$ (2)

where $i=1,...,N$.

Secondly, MAE is the mean of the difference between $\hat{y}$ and $y$ in
absolute value, calculated as

$\displaystyle MAE=\frac{1}{N}{\sum_{i=1}^{N}|y^{(i)}-\hat{y}^{(i)}|}.$ (3)

MSE is the mean of the square of the difference between $\hat{y}$ and $y$ as
the equation

$\displaystyle MSE=\frac{1}{N}{\sum_{i=1}^{N}|y^{(i)}-\hat{y}^{(i)}|^{2}}.$
(4)

$R^{2}$ (also called the coefficient of determination) [23] is defined as the
fraction of the variation in the dependent variable $\hat{y}$ that is
predicted from the independent variable $y$. It has the worst value of
$-\infty$, the best value of $+1$ [5], and can be calculated with the formula

$\displaystyle
R^{2}=1-\frac{\sum_{i=1}^{N}|y^{(i)}-\hat{y}^{(i)}|^{2}}{\sum_{i=1}^{N}|y^{(i)}-\overline{y}|^{2}},$
(5)

where $\overline{y}$ is the mean value of $y$.

##  4 Result & Discussion

###  4.1 Model Performance

Table 3: Result summary of four approaches on the testing dataset | Protocol
---
Approach | Model |  | Spearman
---
Correlation
MAE | MSE |  | $R^{2}$
---
score
| 1 | Combined model | 0.689 | 0.373 | 0.249 | 0.483
1 | 2 | SarcNet | 0.825 | 0.310 | 0.159 | 0.671
| 3 | Combined model | 0.809 | 0.320 | 0.180 | 0.627
2 | 4 | SarcNet | 0.831 | 0.310 | 0.161 | 0.668

In this section, we compare the performance of SarcNet with a combined model,
which integrates the strengths of both the ResNet-18 module and linear
regression to enhance the prediction performance of the sarcomere structure
organization score. Given an input image of a single cell, the ResNet-18
module directly processes the raw image, while linear regression utilizes a
set of extracted features. Both component models generate a predicted score
ranging from one to five. The final score is computed as the mean of two
outputs derived from the individual predictions from the ResNet-18 module and
linear regression.

To understand the effectiveness of feature extraction and deep-learning models
on quantifying sarcomere structure organization, Table 3 provides the four
performance metrics, including Spearman correlation, MAE, MSE, and $R^{2}$
score, of four approaches on the testing dataset. The last two approaches (3,
4), which use Protocol 2, claim much superior outcomes (0.809-0.831) compared
to those with Protocol 1 (0.689-0.825). A highlighted point is that while the
combined model with Protocol 2 dramatically outperforms the performance in
Protocol 1, the SarcNet model reveals nearly identical results in all
performance metrics (Spearman: 0.83, MAE: 0.31, MSE: 0.16, and $R^{2}$ score:
0.67) between the two different methods. An explanation for this could be that
the linear regression model with fewer features leads to poor performance,
which significantly affects the performance of the combined model, lowering
the results from 0.809 to 0.689.

In addition, the SarcNet model claims strength in extracting useful
information from input images and features compared to the combined model.
While the combined model requires the six features extracted from the deep-
learning model to leverage the performance (Table 3), the former can forgo
this step and still maintain similar outcomes. This would effectively speed up
the framework and minimize the memory storage in assessing sarcomere structure
organization in hiPSC-CMs.

Figure 3: Visualization results for the GT and the corresponding predictions
from the SarcNet model. The first row illustrates the predicted examples with
$MAE\leq 0.1$, the second row with $0.1<MAE\leq 1$, and the last row with
$MAE>1$.

Fig. 3 presents prediction examples with different ranges of MAE. It is
noticeable that the predictions from the proposed architecture significantly
reach the original targets, especially those from the first row. The last row,
however, shows some incorrectly predicted results. In particular, Fig. 3(g) is
mistakenly predicted since this cell has a section with considerably organized
z-discs, which confuses the model. Furthermore, Fig. 3(h) is enlarged too
much, making the alpha-actinin-2 patterns in this image appear crisper with
higher density, resulting in a higher prediction score. In contrast, Fig. 3(i)
is shrunk too tiny relative to the original, resulting in blurrier alpha-
actinin-2 patterns and a substantially lower prediction score.

###  4.2 Model Interpretation

Figure 4: Visualization of Grad-CAM heatmaps for different examples.

Grad-CAM heatmaps (Fig. 4) are used to interpret the SarcNet model
predictions. Fig. 4(c), (d), and (e) produce the highest values focusing
mainly on the sarcomere structure, while Fig. 4(b) has the lowest value at the
center region of the cell. An explanation for this could be that the two sides
of the cell have clearer fiber patterns while the center does not; as a
result, it would be easier for the model to make decisions based on the two
sides of the cell. In contrast, Fig. 4(a) illustrates a mistaken case in which
the model seems to focus more on the regions with organized z-discs than the
diffuse regions.

###  4.3 Changes in Cell Organization Over Time

Figure 5: Histogram of the predicted scores for day 18 (blue) and day 32
(red).

Fig. 5 aims to confirm the pattern of increasing organization levels between
days 18 and 32. It is noticeable that the predicted scores from the SarcNet
model are consistent with the culture period and expert annotations, with a
slight shift to the right of the 32-day histogram. While most of the hiPSC-CM
organization on day 18 concentrates around level 3, with most of the regions
being disorganized puncta, day 32 indicates more organized regions with level
4. However, most of the cells on day 32 are still immature and need further
culture. This finding agrees with the research in [6], which has investigated
the culture of hiPSC-CMs on days 30, 90, and 200 and proclaimed that cell
contractility reaches the maximum level at the 200-day time point.

###  4.4 Comparison

Table 4: Comparison of other models on the testing dataset Model |  | Feature
---
extraction
| Spearman
---
Correlation
MAE | MSE |  | $R^{2}$
---
score
ResNet-18 [11] | No | 0.829 | 0.453 | 0.300 | 0.379
DenseNet [14] | No | 0.813 | 0.340 | 0.194 | 0.599
AlexNet [16] | No | 0.788 | 0.346 | 0.195 | 0.596
Linear Regression [7] | Protocol 2 | 0.756 | 0.373 | 0.234 | 0.514
Ours | Protocol 2 | 0.831 | 0.310 | 0.161 | 0.668

Table 4 compares the performance of our proposed framework to other models and
claims a noticeable result. As can be seen from the table, the ResNet-18 model
and SarcNet achieve an approximately similar Spearman correlation of 0.829 and
0.831, respectively, but are significantly different in the results of the
other metrics. An explanation for this finding could be that the Spearman
correlation only measures the association based on ranks between the GT and
predictions but does not consider the distance between them as MAE and MSE. As
a result, the ResNet-18 model seems to quantify the sarcomere structure
organization in the correct rank of scores but still achieves poor performance
in MAE and MSE (0.453 and 0.300, respectively). In contrast, thanks to feature
extraction, SarcNet can get more relevant information and score the sarcomere
structure with smaller values in MAE and MSE, which are 0.310 and 0.161,
respectively.

###  4.5 Limitations and Future Directions

Some limitations in this work need addressing before quantifying sarcomere
structure organization. One of the most noteworthy is the lack of single-cell
segmentation progress. Experts manually mark single-cell boundaries in the
AISC dataset [7], as there is no accessible automated framework. This work
should be considered challenging because the individual cells are not
completely separated from one another but overlap; also, high cell density
regions should be avoided. In the future, we intend to develop an automated
procedure that addresses this challenge and facilitates more accurate
quantification of sarcomere structure organization.

##  5 Conclusion

This work highlights key discoveries in single-cell imaging of hiPSC-CMs.
First, in this study, we develop SarcNet, a deep learning-based method for
quantifying sarcomere structure organization. In particular, we propose this
model to leverage predictions by concatenating the ResNet-18 module output
with a representation vector of quantitative single-cell measurements of
subcellular organization. Second, we conduct extensive experiments and compare
SarcNet to other approaches to illustrate the usefulness of the proposed
method, with the best performance in Spearman correlation of 0.831. Finally,
we speed up the training and interference processes by reducing the number of
features while retaining a performance of 0.825.

## Conflict of Interest

The authors have no conflicts of interest to declare.

## Acknowledgment

This study is supported by the VinUni-Illinois Smart Health Center,
VinUniversity.

## References

  * [1] Ahmed, R.E., Anzai, T., Chanthra, N., Uosaki, H.: A brief review of current maturation methods for human induced pluripotent stem cells-derived cardiomyocytes.  Frontiers in Cell and Developmental Biology 8, 178 (2020)
  * [2] Bedada, F.B., Chan, S.S., Metzger, S.K., Zhang, L., Zhang, J., Garry, D.J., Kamp, T.J., Kyba, M., Metzger, J.M.: Acquisition of a quantitative, stoichiometrically conserved ratiometric marker of maturation status in stem cell-derived cardiac myocytes.  Stem Cell Reports 3(4), 594–605 (2014)
  * [3] Bedada, F.B., Wheelwright, M., Metzger, J.M.: Maturation status of sarcomere structure and function in human iPSC-derived cardiac myocytes.  Biochimica et Biophysica Acta (BBA) - Molecular Cell Research 1863(7), 1829–1838 (2016)
  * [4] Cai, W., Zhang, J., de Lange, W.J., Gregorich, Z.R., Karp, H., Farrell, E.T., Mitchell, S.D., Tucholski, T., Lin, Z., Biermann, M., et al.: An unbiased proteomics method to assess the maturation of human pluripotent stem cell-derived cardiomyocytes.  Circulation Research 125(11), 936–953 (2019)
  * [5] Chicco, D., Warrens, M.J., Jurman, G.: The coefficient of determination R-squared is more informative than SMAPE, MAE, MAPE, MSE and RMSE in regression analysis evaluation.  PeerJ Computer Science 7, e623 (2021)
  * [6] Ebert, A., Joshi, A.U., Andorf, S., Dai, Y., Sampathkumar, S., Chen, H., Li, Y., Garg, P., Toischer, K., Hasenfuss, G., et al.: Proteasome-dependent regulation of distinct metabolic states during long-term culture of human iPSC-derived cardiomyocytes.  Circulation Research 125(1), 90–103 (2019)
  * [7] Gerbin, K.A., Grancharova, T., Donovan-Maiye, R.M., Hendershott, M.C., Anderson, H.G., Brown, J.M., Chen, J., Dinh, S.Q., Gehring, J.L., Johnson, G.R., Lee, H., Nath, A., Nelson, A.M., Sluzewski, M.F., Viana, M.P., Yan, C., Zaunbrecher, R.J., Cordes Metzler, K.R., Gaudreault, N., Knijnenburg, T.A., Rafelski, S.M., Theriot, J.A., Gunawardane, R.N.: Cell states beyond transcriptomics: Integrating structural organization and gene expression in hiPSC-derived cardiomyocytes.  Cell Systems 12(6), 670–687 e10 (2021).  10.1016/j.cels.2021.05.001.  URL https://doi.org/10.1016/j.cels.2021.05.001
  * [8] Guo, Y., Pu, W.T.: Cardiomyocyte maturation: New phase in development.  Circulation Research 126(8), 1086–1106 (2020)
  * [9] Hamledari, H., Asghari, P., Jayousi, F., Aguirre, A., Maaref, Y., Barszczewski, T., Ser, T., Moore, E., Wasserman, W., Klein Geltink, R., et al.: Using human induced pluripotent stem cell-derived cardiomyocytes to understand the mechanisms driving cardiomyocyte maturation.  Frontiers in Cardiovascular Medicine 9, 967,659 (2022)
  * [10] Häneke, T., Sahara, M.: Progress in bioengineering strategies for heart regenerative medicine.  International Journal of Molecular Sciences 23(7), 3482 (2022)
  * [11] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition.  In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)
  * [12] Hnatiuk, A.P., Briganti, F., Staudt, D.W., Mercola, M.: Human iPSC modeling of heart disease for drug development.  Cell Chemical Biology 28(3), 271–282 (2021)
  * [13] Hollander, M., Wolfe, D.A., Chicken, E.: Nonparametric statistical methods.  John Wiley & Sons (2013)
  * [14] Huang, G., Liu, Z., Van Der Maaten, L., Weinberger, K.Q.: Densely connected convolutional networks.  In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4700–4708 (2017)
  * [15] Khan, M., Xu, Y., Hua, S., Johnson, J., Belevych, A., Janssen, P.M., Gyorke, S., Guan, J., Angelos, M.G.: Evaluation of changes in morphology and function of human induced pluripotent stem cell derived cardiomyocytes (hiPSC-CMs) cultured on an aligned-nanofiber cardiac patch.  PLOS One 10(5), e0126,338 (2015)
  * [16] Krizhevsky, A.: One weird trick for parallelizing convolutional neural networks.  CoRR abs/1404.5997 (2014).  URL http://arxiv.org/abs/1404.5997
  * [17] Machiraju, P., Greenway, S.C.: Current methods for the maturation of induced pluripotent stem cell-derived cardiomyocytes.  World Journal of Stem Cells 11(1), 33 (2019)
  * [18] McQuin, C., Goodman, A., Chernyshev, V., Kamentsky, L., Cimini, B.A., Karhohs, K.W., Doan, M., Ding, L., Rafelski, S.M., Thirstrup, D., et al.: CellProfiler 3.0: Next-generation image processing for biology.  PLOS Biology 16(7), e2005,970 (2018)
  * [19] Pasqualini, F.S., Sheehy, S.P., Agarwal, A., Aratyn-Schaus, Y., Parker, K.K.: Structural phenotyping of stem cell-derived cardiomyocytes.  Stem Cell Reports 4(3), 340–347 (2015)
  * [20] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-CAM: Visual explanations from deep networks via gradient-based localization.  In: Proceedings of the IEEE International Conference on Computer Vision, pp. 618–626 (2017)
  * [21] Skorska, A., Johann, L., Chabanovska, O., Vasudevan, P., Kussauer, S., Hillemanns, M., Wolfien, M., Jonitz-Heincke, A., Wolkenhauer, O., Bader, R., et al.: Monitoring the maturation of the sarcomere network: a super-resolution microscopy-based approach.  Cellular and Molecular Life Sciences 79(3), 149 (2022)
  * [22] Vicente, J., Zusterzeel, R., Johannesen, L., Mason, J., Sager, P., Patel, V., Matta, M.K., Li, Z., Liu, J., Garnett, C., et al.: Mechanistic model-informed proarrhythmic risk assessment of drugs: review of the “CiPA” initiative and design of a prospective clinical validation study.  Clinical Pharmacology & Therapeutics 103(1), 54–66 (2018)
  * [23] Wright, S.: Correlation and causation.  Journal of Agricultural Research 20(7), 557 (1921)
  * [24] Yang, X., Ribeiro, A.J., Pang, L., Strauss, D.G.: Use of human iPSC-CMs in nonclinical regulatory studies for cardiac safety assessment.  Toxicological Sciences 190(2), 117–126 (2022)