# Analysis of Atom-level pretraining with QM data for Graph Neural Networks
Molecular property models

Jose Arjona-Medina
Janssen Research & Development
<EMAIL_ADDRESS>
Ramil Nugmanov
Janssen Pharmaceutica NV
<EMAIL_ADDRESS>

###### Abstract

Despite the rapid and significant advancements in deep learning for
Quantitative Structure-Activity Relationship (QSAR) models, the challenge of
learning robust molecular representations that effectively generalize in real-
world scenarios to novel compounds remains an elusive and unresolved task.
This study examines how atom-level pretraining with quantum mechanics (QM)
data can mitigate violations of assumptions regarding the distributional
similarity between training and test data and therefore improve performance
and generalization in downstream tasks. In the public dataset Therapeutics
Data Commons (TDC), we show how pretraining on atom-level QM improves
performance overall and makes the activation of the features distributes more
Gaussian-like which results in a representation that is more robust to
distribution shifts. To the best of our knowledge, this is the first time that
hidden state molecular representations are analyzed to compare the effects of
molecule-level and atom-level pretraining on QM data.

##  1 Introduction

Despite the rapid and significant advancements in deep learning for
Quantitative Structure-Activity Relationship (QSAR) models, the challenge of
learning robust molecular representations that effectively generalize in real-
world scenarios to novel compounds remains an elusive and unresolved task.

At the core of supervised learning with deep neural networks, there is the
underlying assumption that the function we aim to approximate is "smooth"
enough, meaning that small changes in the input should not result in large
changes in the output [12]. However, in the application field of QSAR, we
often experience regions where small chemical modifications drastically change
the biological response [9]. For example, the extreme forms of Structure-
Activity Relationship (SAR) discontinuity are called activity cliffs (AC) [1],
which are formed by pairs of structurally similar compounds with large
differences in potency [[32],[22]]. One way of overcoming this problem would
be to increase the number of datapoints since the smoothness assumption works
well as long as there are enough examples for the learning algorithm to
observe high points on most peaks and low points on most valleys of the true
underlying function to be learned [12]. However, it is not a general or
scalable solution because the costs of producing additional experimental
datapoints are high and would be applicable only locally for a specific target
property

Supervised Learning methods also assume that test data comes from the same
underlying distribution as training data [3]. Although novel techniques often
present impressive metrics, most often they do not suffice to meet the
practical needs in real-world drug discovery [10]. In the real world, data
production is by construction strongly biased: Chemists work with structurally
analogous series. In practice, target data is drifting [34], making these
models obsolete for real-world production environments. The fundamental
factors that influence molecular properties are still unexplored [10]. With
statistical analyses, fixed representations like fingerprints generally match
end-to-end deep learning (DL) models in most datasets [10, 2, 26], especially
in the presence of activity cliffs [35]. Much of the progress in molecular
property estimation has been driven by the strategic incorporation of
inductive biases [19, 38]. Techniques such as pretraining [39], the use of
equivariant networks, and graph neural networks have proven useful [29, 17].
The shift towards leveraging domain-specific knowledge through inductive
biases underlines the evolution in our approach to QSAR modeling.

#### Pretraining and Transfer Learning in QSAR Models.

Pretraining has emerged as a potent solution to overcome data scarcity and
enhance model generalizability in various domains, including image and natural
language processing. However, without appropriate domain expertise,
pretraining can sometimes lead to negative transfer or only marginal
performance gains [27]. In the realm of molecular modeling, pretraining
strategies have been successful, particularly when models are pretrained on
large-scale datasets [36, 7, 33, 40]. For instance, the large-scale
pretraining of models has been shown to improve sample efficiency in active
learning frameworks used for molecule virtual screening [5]. Or [30] which
enriches molecule representations with contextual knowledge from reference
molecules using a Modern Hopfield Network [25]. Combining node-level and
graph-level pre-training tasks to achieve superior generalization,
particularly in out-of-distribution scenarios was presented in [14]. ChemBERTa
utilizes large-scale pretraining datasets to explore the effects of dataset
size on downstream task performance [7].

#### Multitask learning.

Closely related to transfer learning, involves the simultaneous learning of
multiple related tasks, enhancing the model’s ability to generalize across
tasks [6]. This approach has been leveraged effectively in models like
MolPMoFiT, which was pretrained using bioactive molecules from ChEMBL for
QSPR/QSAR tasks [19].

#### Multimodal and Self-supervised Learning.

Recent advancements have also been made in multimodal learning and self-
supervised learning frameworks. The GIMLET model, for example, pretrained on a
dataset of molecule tasks with textual instructions, bridging the gap between
graph and text data modalities and paving the way for instruction-based
pretraining [44]. Similarly, the D&D framework utilizes a cross-modal
distillation approach, transferring knowledge from 3D to 2D molecular
structures, thereby significantly enhancing model performance in downstream
tasks [8]. Or denoising 3D structures as a pre-training objective, setting new
benchmarks in the widely used QM9 dataset [42].

#### Contrastive learning

approaches [18, 16] can overcome data limitations by integrating additional
data. For example, CLOOME [28] embeds bioimages and chemical structures into a
unified space, enabling highly accurate retrieval of bioimages based on
chemical structures. MolFeSCue [43] incorporates a dynamic contrastive loss
function tailored for class imbalance. MolCLR [37] uses large-scale unlabeled
data, graph-based augmentations, and a contrastive learning strategy to
improve model generalization. A recent overview of methods for multi-modal
contrastive learning is presented in [31].

Our approach effectively combines pretraining with QM data at node-level, to
improve the downstream task of property modeling. We analyze the effect of
this pretraining in the distribution of the features of the network, and we
observe that atom-level pretrained networks have more Gaussian-like
distributions which are more robust to distribution shifts in the input space,
establishing a direct connection between pretraining at node-level and
supervised learning theory of generalization.

Or contributions can be summarized as follows:

  * •

We empirically show that atom-level pretraining in Graph neural networks with
QM properties improves performance over scratch networks and molecular-level
pretraining, in the TDC public dataset.

  * •

We empirically show that atom-level pretraining produces a more normal
distribution of features, compared to scratch and molecular-level pretrained
networks.

  * •

We empirically show that atom-level pretraining produces a more robust
molecular representation against distribution shift from train to eval and
train to test datasplits, which could explain the performance gain of atom-
level pretraining approach.

To the best of our knowledge, this is the first study that aims to understand
how atom-level pretraining improves molecular representation.

##  2 Methods

#### Graphormer.

In 2021, Microsoft published Graphormer[41], a neural network specifically
engineered for processing graphs and, more pointedly, molecular structures.
This network has exhibited outstanding performance in molecular quantum
properties, adapting the architecture of BERT[11] to suit graph data
effectively. Graphormer introduces several novel features, the key among them
being "centrality encoding". This technique captures the importance of nodes
within the graph by integrating graph degree centrality, which is encoded as
embedding vectors added into the atom’s node type embeddings. Additionally,
Graphormer incorporates a "spatial encoding" strategy. This involves
representing the shortest path between pairs of nodes, which is then utilized
as a learnable bias in the attention matrix. This concept mirrors techniques
used in other advanced models like T5[24] and ALIBI[23], highlighting its
relevance and utility in enhancing the model’s attention mechanism within
graph networks.

#### Custom implementation.

Following [21], we have refined the "centrality encoder" of the original
Graphormer to encompass not just explicit neighbors but total neighbors,
integrating both explicit atom neighbors and implicit hydrogens. This approach
eliminates the need for an "edge encoder", leveraging a combination of the
centrality encoder and an atom type encoder to implicitly capture atom
hybridization. Moreover, we have streamlined the model by omitting the
encoding of atoms’ formal charges and radical states, allowing for the
representation of resonance structures in a unified form and dispensing with
the traditional concept of "aromatic" bonds typically applied to arenes.
Additionally, our model introduces a "spatial encoder" constrained by a
configurable maximum distance threshold to enhance computational efficiency
and model accuracy. By treating distances beyond this threshold uniformly, the
model focuses on more relevant short-range interactions understanding
molecular structures.

Figure 1: Different aromatic ring representations folded into a single form

#### Enabling multitask learning by task-specific virtual nodes.

Similar to the original BERT model, Graphormer utilizes a [CLS] token as a
global readout for estimating properties of graphs and molecules. This
methodology aligns with techniques used in graph convolutional networks
(GCNs), where virtual nodes connected to every node of the graph can serve a
comparable purpose [4]. Building on this concept, we propose a novel extension
tailored for multitask learning applications. In our approach, we employ the
same encoder and output head to estimate multiple properties, but
differentiate the tasks by using distinct virtual nodes for each. This allows
for task-specific processing while maintaining a unified architecture,
enhancing the model’s efficiency, adaptability, and scalability in handling
diverse learning tasks.

#### Quantitative Assessment of learned features.

To analyze the impact of pretraining methods on the molecular representation
captured by the model, we conducted a comprehensive analysis of learned
feature distributions. We run the models on the three different data splits
(training, validation, and test), and extract the distribution of features
after the first layer of the Graphormer architecture.

For each compound in the dataset, we aggregated the node representations after
the first Graphormer layer, resulting in a cumulative dataset-wide node
representation. This aggregation facilitated the derivation of an empirical
distribution of activations per model and data split. Using this empirical
distribution, we applied various statistical techniques to quantify the
differences in feature distributions across the different data splits for each
of the models. Furthermore, we undertook an evaluation of the normality of
these distributions using Shapiro-Wilk and Kolmogorov-Smirnov tests. This
assessment was used to determine whether significant deviations exist in the
distribution of learned features across different pretraining methods.

##  3 Experiments

#### Datasets

For pretraining, we used a publicly available dataset [13] consisting of 136k
organic molecules. Each molecule is represented by a single conformer
generated using the Merck Molecular Force Field (MMFF94s) in the RDKit
library. The initial geometry for the lowest-lying conformer was then
optimized at the GFN2-xtb level of theory followed by refinement of the
electronic structure with DFT (B3LYP/def2svp). The advantage of the described
dataset is several reported atomic properties: charge, electrophilicity,
nucleophilicity Fukui indexes, and an NMR shielding constant. Some additional
curation was carried out as described in the addendum. Another pretraining
dataset, PCQM4Mv2, consists of a single molecular property per molecule, a
HOMO-LUMO gap https://ogb.stanford.edu/docs/lsc/pcqm4mv2/. It was curated
under the PubChemQC project [20]. For the benchmarking of the obtained
pretrained models, we used the absorption, distribution, metabolism,
excretion, and toxicity (ADMET) group of the Therapeutics Data Commons (TDC)
dataset [15], consisting of 9 regression and 13 binary classification tasks
for modeling biochemical molecular properties.

#### Experiments setup

We train our Graphormer network from scratch on the set of tasks provided by
the TDC dataset. We also pretrained the same network using as main task HOMO-
LUMO gap (molecule-level) and the atomic properties charge, electrophilicity,
and nucleophilicity Fukui indexes and an NMR shielding constant (atom-level).
For atom-level pretrain, we pretrained 4 models, one per every single
property. We also pretrained a fifth model which is pretrained on 4 properties
via multi-task by task-specific virtual nodes. Main results reported in this
paper for atom-level uses this last model. Results for specific atomic
property pretrained are available in the Supplementary Material.

##  4 Results

This section provides a concise summary of the key findings of this study. For
an exhaustive elaboration of the results, please refer to the supplementary
materials.

#### Pretraining on atom-level node with QM significantly improves performance
in downstream tasks

In Table 1, we present the outcomes from benchmarking three distinct training
approaches: scratch, molecule-level QM pretrained, and atom-level QM
pretrained with all properties for 5 different seeds, as described in the
guidelines provided by the TDC dataset
https://tdcommons.ai/benchmark/overview/. We have excluded the results for
atom-level pretraining on individual QM properties from this table; these can
be found in the Supplementary Materials. These results show that atom-level
pretraining notably enhances model performance compared to training from
scratch for 21 of the 22 datasets.

Table 1: Graphormer results | Metric | Direction | scratch | mol-level | atom-level
---|---|---|---|---|---
|  |  |  | pretrained | pretrained
|  |  |  | HLgap | all (4)
caco2_wang | MAE | $\downarrow$ | 0.48 ± 0.06 | 0.53 ± 0.02 | 0.41 ± 0.03
lipophilicity_astrazeneca | MAE | $\downarrow$ | 0.58 ± 0.02 | 0.57 ± 0.02 | 0.42 ± 0.01
solubility_aqsoldb | MAE | $\downarrow$ | 0.89 ± 0.04 | 0.89 ± 0.02 | 0.75 ± 0.01
ppbr_az | MAE | $\downarrow$ | 8.38 ± 0.24 | 8.22 ± 0.23 | 7.79 ± 0.24
ld50_zhu | MAE | $\downarrow$ | 0.61 ± 0.02 | 0.60 ± 0.03 | 0.57 ± 0.02
hia_hou | ROC-AUC | $\uparrow$ | 0.96 ± 0.03 | 0.96 ± 0.02 | 0.94 ± 0.05
pgp_broccatelli | ROC-AUC | $\uparrow$ | 0.87 ± 0.04 | 0.86 ± 0.01 | 0.89 ± 0.02
bioavailability_ma | ROC-AUC | $\uparrow$ | 0.52 ± 0.01 | 0.55 ± 0.03 | 0.64 ± 0.05
bbb_martins | ROC-AUC | $\uparrow$ | 0.83 ± 0.01 | 0.82 ± 0.03 | 0.88 ± 0.02
cyp3a4_substrate_carbonmangels | ROC-AUC | $\uparrow$ | 0.63 ± 0.07 | 0.64 ± 0.03 | 0.64 ± 0.02
ames | ROC-AUC | $\uparrow$ | 0.72 ± 0.02 | 0.73 ± 0.01 | 0.80 ± 0.01
dili | ROC-AUC | $\uparrow$ | 0.86 ± 0.02 | 0.87 ± 0.01 | 0.88 ± 0.03
herg | ROC-AUC | $\uparrow$ | 0.78 ± 0.01 | 0.76 ± 0.04 | 0.77 ± 0.06
vdss_lombardo | Spearman | $\uparrow$ | 0.58 ± 0.04 | 0.59 ± 0.04 | 0.59 ± 0.03
half_life_obach | Spearman | $\uparrow$ | 0.39 ± 0.07 | 0.34 ± 0.07 | 0.48 ± 0.06
clearance_microsome_az | Spearman | $\uparrow$ | 0.49 ± 0.03 | 0.46 ± 0.03 | 0.60 ± 0.01
clearance_hepatocyte_az | Spearman | $\uparrow$ | 0.34 ± 0.04 | 0.31 ± 0.02 | 0.46 ± 0.03
cyp2d6_veith | PR-AUC | $\uparrow$ | 0.43 ± 0.03 | 0.47 ± 0.02 | 0.61 ± 0.02
cyp3a4_veith | PR-AUC | $\uparrow$ | 0.73 ± 0.01 | 0.74 ± 0.03 | 0.80 ± 0.03
cyp2c9_veith | PR-AUC | $\uparrow$ | 0.63 ± 0.02 | 0.66 ± 0.03 | 0.69 ± 0.02
cyp2d6_substrate_carbonmangels | PR-AUC | $\uparrow$ | 0.52 ± 0.01 | 0.54 ± 0.04 | 0.58 ± 0.03
cyp2c9_substrate_carbonmangels | PR-AUC | $\uparrow$ | 0.35 ± 0.02 | 0.33 ± 0.03 | 0.37 ± 0.04

#### Pretraining on atom-level node with QM results on smoother feature
distribution

We further examined the distribution of the features from the network’s first
layer for atom-level pretrained, molecular-level (HOMO-LUMO gap) and scratch
models across different data splits and datasets. Figure 2 illustrates the
feature distribution for the test split of one of the datasets
(lipophilicity), highlighting how scratch and molecular-level pretrain compare
to the atom-level pretrained network (for other datasets and splits, please
refer to the Supplementary Materials). Visually, we can observe that the atom-
level pretrained method results in a more Gaussian-like distribution of the
features. We conducted a Shapiro-Wilk test for each dimension in both scratch
and pretrained networks to assess the normality of the distributions. The
average p-value for the scratch network was 3.2E-06 with a standard deviation
of 5E-05, suggesting a significant deviation from normality. In contrast, the
pretrained network had an average p-value of 4E-04 with a standard deviation
of 0.003, indicating a closer approximation to a Gaussian distribution.

Figure 2: Distribution of first 20 features from the first layer of the
Graphormer network for three different training approaches —scratch, HOMO-LUMO
pretrained and atom-level pretrained— across test split of lipophilicity
dataset .

#### Pretraining on atom-level node with QM enhances robustness to input
distribution shifts

Pretrained networks show in general less distribution shift compared to
scratch network, as we can see in Figure 3. In this figure, we show the
distribution shifts differences between scratch and pretrained networks. As we
can appreciate, in most of the cases, specially in the Train-Test comparison,
atom-level pretrained networks have less distribution shift than scratch
networks.

Figure 3: This heatmap illustrates the differences in distribution shifts
between scratch and atom-levle pretrained networks, calculated across various
feature dimensions using metrics such as Kullback-Leibler Divergence, Jensen-
Shannon Divergence, Earth Mover’s Distance, Total Variation Distance, and
Hellinger Distance. Green hues indicate instances where the atom-levle
pretrained network exhibits smaller distribution shifts compared to the
scratch network, while purple hues denote the opposite. Notably, both heatmaps
comparison frequently show reduced distribution shifts in atom-level
pretrained networks, specially for Train-Test comparison, which likely helps
to explain atom-level pretrained network’s superior performance on the test
data split.

##  5 Conclusions

In this study, we have demonstrated that pretraining of graph-based neural
networks with atom-level quantum mechanics (QM) data significantly enhances
performance on downstream tasks related to ADMET properties within the TDC
dataset, as illustrated in Table 1. Furthermore, we showed the change in the
distributions of activations of the internal model’s features due to specific
pretraining. After atom-level pretraining with QM data, these distributions
become more Gaussian-like, which is known to be conducive to better learning
dynamics and thus improved performance (see Figure 2). Moreover, our findings
indicate that pretrained models exhibit smaller distribution shifts from
training to testing datasets, further supporting the efficacy of QM data
pretraining in enhancing model robustness (see Figure 2).

To our knowledge, this is the first study that elucidates how atom-level
pretraining can optimize molecular representations by analyzing the model’s
internal representation and robustness to distribution shifts.

## References

  * [1] J. Bajorath, L. Peltason, M. Wawer, R. Guha, M. S. Lajiness, and J. H. Van Drie.  Navigating structure-activity landscapes.  Drug Discov Today, 14(13-14):698–705, 2009.
  * [2] D. Baptista, J. Correia, B. Pereira, and M. Rocha.  Evaluating molecular representations in machine learning models for drug response prediction and interpretability.  J Integr Bioinform, 19(3):20220006, 2022.
  * [3] C. M. Bishop.  Pattern Recognition and Machine Learning.  Springer, 2006.
  * [4] C. Cai, T. S. Hy, R. Yu, and Y. Wang.  On the connection between mpnn and graph transformer.  In Proceedings of the 40th International Conference on Machine Learning, page 138. JMLR.org, 2023.
  * [5] Z. Cao, S. Sciabola, and Y. Wang.  Large-scale pretraining improves sample efficiency of active learning based molecule virtual screening.  In The NeurIPS 2023 Workshop on New Frontiers of AI for Drug Discovery and Development (AI4D3 2023), 2023.
  * [6] R. Caruana.  Multitask learning.  Machine Learning, 28(1):41–75, 1997.
  * [7] S. Chithrananda, G. Grand, and B. Ramsundar.  Chemberta: Large-scale self-supervised pretraining for molecular property prediction, 2020.
  * [8] S. Cho, D.-W. Jeong, S. M. Ko, J. Kim, S. Han, S. Hong, H. Lee, and M. Lee.  3d denoisers are good 2d teachers: Molecular pretraining via denoising and cross-modal distillation, 2023.
  * [9] M. Cruz-Monteagudo, J. L. Medina-Franco, Y. Pérez-Castillo, O. Nicolotti, M. N. D. S. Cordeiro, and F. Borges.  Activity cliffs in drug discovery: Dr jekyll or mr hyde?  Drug Discovery Today, 19(8):1069–1080, 2014.
  * [10] J. Deng, Z. Yang, and H. Wang et al.  A systematic study of key elements underlying molecular property prediction.  Nat Commun, 14:6395, 2023.
  * [11] J. Devlin, M. W. Chang, K. Lee, and K. Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding.  In Proc. Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT), pages 4171–4186, 2019.
  * [12] Ian Goodfellow, Yoshua Bengio, and Aaron Courville.  Deep Learning.  MIT Press, 2016.  http://www.deeplearningbook.org.
  * [13] Yanfei Guan, Connor W. Coley, Haoyang Wu, Duminda Ranasinghe, Esther Heid, Thomas J. Struble, Lagnajit Pattanaik, William H. Green, and Klavs F. Jensen.  Regio-selectivity prediction with a machine-learned reaction representation and on-the-fly quantum mechanical descriptors.  Chemical Science, 12(6):2198–2208, 2021.
  * [14] W. Hu, B. Liu, J. Gomes, M. Zitnik, P. Liang, V. Pande, and J. Leskovec.  Strategies for pre-training graph neural networks, 2020.
  * [15] Kexin Huang, Tianfan Fu, Wenhao Gao, Yue Zhao, Yusuf Roohani, Jure Leskovec, Connor W. Coley, Cao Xiao, Jimeng Sun, and Marinka Zitnik.  Artificial intelligence foundation for therapeutic science.  Nature Chemical Biology, 18(10):1033–1036, 2022.
  * [16] Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan.  Supervised contrastive learning.  In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 18661–18673. Curran Associates, Inc., 2020.
  * [17] T. Le, F. Noé, and D. Clevert.  Equivariant graph attention networks for molecular property prediction.  In Proceedings of the Learning on Graphs Conference 2022. LoG Conference, 2022.
  * [18] Phuc H. Le-Khac, Graham Healy, and Alan F. Smeaton.  Contrastive representation learning: A framework and review.  IEEE Access, 8:193907–193934, 2020.
  * [19] X. Li and D. Fourches.  Inductive transfer learning for molecular activity prediction: Next-gen qsar models with molpmofit.  J Cheminform, page 27, 2020.
  * [20] Maho Nakata and Tomomi Shimazaki.  PubChemQC Project: A Large-Scale First-Principles Electronic Structure Database for Data-Driven Chemistry.  Journal of Chemical Information and Modeling, 57(6):1300–1308, 2017.
  * [21] R. Nugmanov, N. Dyubankova, A. Gedich, and J. K. Wegner.  Bidirectional graphormer for reactivity understanding: Neural network trained to reaction atom-to-atom mapping task.  Journal of Chemical Information and Modeling, 62(14):3307–3315, 2022.
  * [22] L. Peltason and J. Bajorath.  Computational Analysis of Activity and Selectivity Cliffs, volume 672, pages 119–132.  Humana Press, 2011.
  * [23] O. Press, N. A. Smith, and M. Lewis.  Train short, test long: Attention with linear biases enables input length extrapolation.  In Proc. Annual Meeting of the Association for Computational Linguistics. ACL, 2021.
  * [24] C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu.  Exploring the limits of transfer learning with a unified text-to-text transformer.  In Proc. Annual Conference on Neural Information Processing Systems (NeurIPS), 2019.
  * [25] Hubert Ramsauer, Bernhard Schäfl, Johannes Lehner, Philipp Seidl, Michael Widrich, Thomas Adler, Lukas Gruber, Markus Holzleitner, Milena Pavlović, Geir Kjetil Sandve, Victor Greiff, David Kreil, Michael Kopp, Günter Klambauer, Johannes Brandstetter, and Sepp Hochreiter.  Hopfield networks is all you need.  In International Conference on Learning Representations, 2021.
  * [26] M. C. Robinson, R. C. Glen, and A. A. Lee.  Validating the validation: reanalyzing a large-scale comparison of deep learning and machine learning models for bioactivity prediction.  J Comput Aided Mol Des, 34:717–730, 2020.
  * [27] M. T. Rosenstein, Z. Marx, L. P. Kaelbling, and T. G. Dietterich.  To transfer or not to transfer.  In Advances in Neural Information Processing Systems (NeurIPS), Workshop on transfer learning, volume 898, pages 1–4, 2005.
  * [28] A. Sanchez-Fernandez, E. Rumetshofer, and S. Hochreiter et al.  Cloome: contrastive learning unlocks bioimaging databases for queries with chemical structures.  Nat Commun, 14:7339, 2023.
  * [29] V. Garcia Satorras, E. Hoogeboom, and M. Welling.  E(n) equivariant graph neural networks.  In Proceedings of the International Conference on Learning Representations (ICLR), 2021.
  * [30] Johannes Schimunek, Philipp Seidl, Lukas Friedrich, Daniel Kuhn, Friedrich Rippmann, Sepp Hochreiter, and Günter Klambauer.  Context-enriched molecule representations improve few-shot drug discovery.  In The Eleventh International Conference on Learning Representations, 2023.
  * [31] Philipp Seidl.  Multimodal Contrastive Learning for Drug Discovery.  Dissertation (phd), Johannes Kepler University Linz, Linz, 2024.  Includes illustrations.
  * [32] D. Stumpfe and J. Bajorath.  Exploring activity cliffs in medicinal chemistry miniperspective.  Journal of Medicinal Chemistry, 55(7):2932–2942, 2012.
  * [33] H. Stärk, D. Beaini, G. Corso, P. Tossou, C. Dallago, S. Günnemann, and P. Lió.  3d infomax improves gnns for molecular property prediction.  In Proceedings of the 39th International Conference on Machine Learning (ICML). PMLR, 2023.
  * [34] P. Tossou, C. Wognum, M. Craig, M. H, and E. Noutahi.  Real-world molecular out-of-distribution: Specification and investigation.  ChemRxiv, 2023.  This content is a preprint and has not been peer-reviewed.
  * [35] D. van Tilborg, A. Alenicheva, and F. Grisoni.  Exposing the limitations of molecular machine learning with activity cliffs.  Journal of Chemical Information and Modeling, 62(23):5938–5951, 2022.
  * [36] S. Wang, Y. Guo, Y. Wang, H. Sun, and J. Huang.  Smiles-bert: Large scale unsupervised pre-training for molecular property prediction.  In Proceedings of the 10th ACM International Conference on Bioinformatics, Computational Biology and Health Informatics (ACM-BCB), pages 429–436, Niagara Falls, NY, USA, 2019. ACM.
  * [37] Yuyang Wang, Jianren Wang, Zhonglin Cao, and Amir Barati Farimani et al.  Molecular contrastive learning of representations via graph neural networks.  Nat Mach Intell, 4:279–287, 2022.
  * [38] J. Xia, L. Zhang, X. Zhu, Y. Liu, Z. Gao, B. Hu, C. Tan, J. Zheng, S. Li, and S. Z. Li.  Understanding the limitations of deep models for molecular property prediction: Insights and solutions.  In Advances in Neural Information Processing Systems, volume 36, pages 64774–64792. Curran Associates, Inc., 2023.
  * [39] J. Xia, Y. Zhu, Y. Du, and S. Z. Li.  A systematic survey of chemical pre-trained models.  In Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence (IJCAI-23). IJCAI, 2023.
  * [40] L. Xu, L. Xia, S. Pan, and Z. Li.  Triple generative self-supervised learning method for molecular property prediction.  Int. J. Mol. Sci., 25(7):3794, 2024.
  * [41] C. Ying, T. Cai, S. Luo, S. Zheng, G. Ke, D. He, Y. Shen, and T.-Y. Liu.  Do transformers really perform badly for graph representation?  In M. Ranzato, A. Beygelzimer, Y. Dauphin, P. S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 28877–28888. Curran Associates, Inc., 2021.
  * [42] S. Zaidi, M. Schaarschmidt, J. Martens, H. Kim, Y. W. Teh, A. Sanchez-Gonzalez, P. Battaglia, R. Pascanu, and J. Godwin.  Pre-training via denoising for molecular property prediction, 2022.
  * [43] Ruochi Zhang, Chao Wu, Qian Yang, Chang Liu, Yan Wang, Kewei Li, Lan Huang, and Fengfeng Zhou.  Molfescue: enhancing molecular property prediction in data-limited and imbalanced contexts using few-shot and contrastive learning.  Bioinformatics, 40:btae118, 2024.
  * [44] H. Zhao, S. Liu, C. Ma, H. Xu, J. Fu, Z.-H. Deng, L. Kong, and Q. Liu.  Gimlet: A unified graph-text model for instruction-based molecule zero-shot learning, 2023.

##  Appendix A Supplementary Materials

###  A.1 Full table results

###  A.2 Distribution of activations of features for scratch, HOMO-LUMO
pretrained and Atom-level pretrained networks in TDC Lipophilicity Dataset

###  A.3 Distribution of first 20 features from the first layer of the
Graphormer network for three different training approaches —scratch, HOMO-LUMO
pretrained and atom-level pretrained— across test split of TDC datasets

Table 2: Graphormer results | Metric | Direction | scratch | HLgap | atom-level |  |  |  |  | mol-level
---|---|---|---|---|---|---|---|---|---|---
|  |  |  |  | pretrained |  |  |  |  | pretrained
|  |  |  |  | all (4) | fukui_e | nmr | fukui_n | charge | HLgap
caco2_wang | MAE | $\downarrow$ | 0.48 ± 0.06 | 0.53 ± 0.02 | 0.41 ± 0.03 | 0.45 ± 0.07 | 0.48 ± 0.06 | 0.39 ± 0.02 | 0.40 ± 0.08 | 0.53 ± 0.02
lipophilicity_astrazeneca | MAE | $\downarrow$ | 0.58 ± 0.02 | 0.57 ± 0.02 | 0.42 ± 0.01 | 0.49 ± 0.02 | 0.46 ± 0.01 | 0.48 ± 0.01 | 0.43 ± 0.01 | 0.57 ± 0.02
solubility_aqsoldb | MAE | $\downarrow$ | 0.89 ± 0.04 | 0.89 ± 0.02 | 0.75 ± 0.01 | 0.80 ± 0.02 | 0.78 ± 0.02 | 0.78 ± 0.02 | 0.75 ± 0.01 | 0.89 ± 0.02
ppbr_az | MAE | $\downarrow$ | 8.38 ± 0.24 | 8.22 ± 0.23 | 7.79 ± 0.24 | 7.92 ± 0.12 | 8.02 ± 0.40 | 7.79 ± 0.28 | 7.57 ± 0.32 | 8.22 ± 0.23
ld50_zhu | MAE | $\downarrow$ | 0.61 ± 0.02 | 0.60 ± 0.03 | 0.57 ± 0.02 | 0.60 ± 0.01 | 0.56 ± 0.02 | 0.60 ± 0.02 | 0.57 ± 0.01 | 0.60 ± 0.03
hia_hou | ROC-AUC | $\uparrow$ | 0.96 ± 0.03 | 0.96 ± 0.02 | 0.94 ± 0.05 | 0.93 ± 0.03 | 0.97 ± 0.02 | 0.94 ± 0.02 | 0.95 ± 0.02 | 0.96 ± 0.02
pgp_broccatelli | ROC-AUC | $\uparrow$ | 0.87 ± 0.04 | 0.86 ± 0.01 | 0.89 ± 0.02 | 0.89 ± 0.03 | 0.86 ± 0.03 | 0.90 ± 0.01 | 0.88 ± 0.02 | 0.86 ± 0.01
bioavailability_ma | ROC-AUC | $\uparrow$ | 0.52 ± 0.01 | 0.55 ± 0.03 | 0.64 ± 0.05 | 0.64 ± 0.02 | 0.66 ± 0.01 | 0.69 ± 0.05 | 0.62 ± 0.07 | 0.55 ± 0.03
bbb_martins | ROC-AUC | $\uparrow$ | 0.83 ± 0.01 | 0.82 ± 0.03 | 0.88 ± 0.02 | 0.86 ± 0.03 | 0.86 ± 0.02 | 0.85 ± 0.02 | 0.87 ± 0.01 | 0.82 ± 0.03
cyp3a4_substrate_carbonmangels | ROC-AUC | $\uparrow$ | 0.63 ± 0.07 | 0.64 ± 0.03 | 0.64 ± 0.02 | 0.66 ± 0.03 | 0.62 ± 0.02 | 0.61 ± 0.02 | 0.63 ± 0.02 | 0.64 ± 0.03
ames | ROC-AUC | $\uparrow$ | 0.72 ± 0.02 | 0.73 ± 0.01 | 0.80 ± 0.01 | 0.78 ± 0.02 | 0.80 ± 0.02 | 0.76 ± 0.01 | 0.80 ± 0.01 | 0.73 ± 0.01
dili | ROC-AUC | $\uparrow$ | 0.86 ± 0.02 | 0.87 ± 0.01 | 0.88 ± 0.03 | 0.86 ± 0.04 | 0.89 ± 0.03 | 0.82 ± 0.04 | 0.85 ± 0.04 | 0.87 ± 0.01
herg | ROC-AUC | $\uparrow$ | 0.78 ± 0.01 | 0.76 ± 0.04 | 0.77 ± 0.06 | 0.73 ± 0.06 | 0.77 ± 0.05 | 0.77 ± 0.02 | 0.79 ± 0.03 | 0.76 ± 0.04
vdss_lombardo | Spearman | $\uparrow$ | 0.58 ± 0.04 | 0.59 ± 0.04 | 0.59 ± 0.03 | 0.64 ± 0.02 | 0.61 ± 0.04 | 0.61 ± 0.01 | 0.63 ± 0.03 | 0.59 ± 0.04
half_life_obach | Spearman | $\uparrow$ | 0.39 ± 0.07 | 0.34 ± 0.07 | 0.48 ± 0.06 | 0.48 ± 0.04 | 0.42 ± 0.10 | 0.48 ± 0.03 | 0.47 ± 0.04 | 0.34 ± 0.07
clearance_microsome_az | Spearman | $\uparrow$ | 0.49 ± 0.03 | 0.46 ± 0.03 | 0.60 ± 0.01 | 0.47 ± 0.06 | 0.57 ± 0.01 | 0.58 ± 0.02 | 0.58 ± 0.01 | 0.46 ± 0.03
clearance_hepatocyte_az | Spearman | $\uparrow$ | 0.34 ± 0.04 | 0.31 ± 0.02 | 0.46 ± 0.03 | 0.42 ± 0.02 | 0.44 ± 0.04 | 0.41 ± 0.02 | 0.46 ± 0.04 | 0.31 ± 0.02
cyp2d6_veith | PR-AUC | $\uparrow$ | 0.43 ± 0.03 | 0.47 ± 0.02 | 0.61 ± 0.02 | 0.55 ± 0.03 | 0.56 ± 0.04 | 0.56 ± 0.02 | 0.58 ± 0.04 | 0.47 ± 0.02
cyp3a4_veith | PR-AUC | $\uparrow$ | 0.73 ± 0.01 | 0.74 ± 0.03 | 0.80 ± 0.03 | 0.77 ± 0.03 | 0.78 ± 0.01 | 0.79 ± 0.03 | 0.76 ± 0.04 | 0.74 ± 0.03
cyp2c9_veith | PR-AUC | $\uparrow$ | 0.63 ± 0.02 | 0.66 ± 0.03 | 0.69 ± 0.02 | 0.67 ± 0.02 | 0.69 ± 0.04 | 0.69 ± 0.01 | 0.69 ± 0.02 | 0.66 ± 0.03
cyp2d6_substrate_carbonmangels | PR-AUC | $\uparrow$ | 0.52 ± 0.01 | 0.54 ± 0.04 | 0.58 ± 0.03 | 0.53 ± 0.06 | 0.64 ± 0.06 | 0.57 ± 0.04 | 0.63 ± 0.03 | 0.54 ± 0.04
cyp2c9_substrate_carbonmangels | PR-AUC | $\uparrow$ | 0.35 ± 0.02 | 0.33 ± 0.03 | 0.37 ± 0.04 | 0.32 ± 0.04 | 0.34 ± 0.03 | 0.37 ± 0.04 | 0.36 ± 0.04 | 0.33 ± 0.03
Figure 4: Distribution plots Scratch for Lipophilicity test set Figure 5:
Distribution plots HOMO-LUMO pretrained for Lipophilicity test set Figure 6:
Distribution plots Atom-level pretrained for Lipophilicity test set Figure 7:
Distribution of first 20 features from the first layer of the Graphormer
network for three different training approaches —scratch, HOMO-LUMO pretrained
and atom-level pretrained— across test split of caco2 wang dataset. Figure 8:
Distribution of first 20 features from the first layer of the Graphormer
network for three different training approaches —scratch, HOMO-LUMO pretrained
and atom-level pretrained— across test split of dili dataset. Figure 9:
Distribution of first 20 features from the first layer of the Graphormer
network for three different training approaches —scratch, HOMO-LUMO pretrained
and atom-level pretrained— across test split of bioavailability ma dataset.
Figure 10: Distribution of first 20 features from the first layer of the
Graphormer network for three different training approaches —scratch, HOMO-LUMO
pretrained and atom-level pretrained— across test split of pgp broccatelli
dataset. Figure 11: Distribution of first 20 features from the first layer of
the Graphormer network for three different training approaches —scratch, HOMO-
LUMO pretrained and atom-level pretrained— across test split of ames dataset.
Figure 12: Distribution of first 20 features from the first layer of the
Graphormer network for three different training approaches —scratch, HOMO-LUMO
pretrained and atom-level pretrained— across test split of half life obach
dataset. Figure 13: Distribution of first 20 features from the first layer of
the Graphormer network for three different training approaches —scratch, HOMO-
LUMO pretrained and atom-level pretrained— across test split of herg dataset.
Figure 14: Distribution of first 20 features from the first layer of the
Graphormer network for three different training approaches —scratch, HOMO-LUMO
pretrained and atom-level pretrained— across test split of ppbr az dataset.
Figure 15: Distribution of first 20 features from the first layer of the
Graphormer network for three different training approaches —scratch, HOMO-LUMO
pretrained and atom-level pretrained— across test split of cyp2c9 substrate
carbonmangels dataset. Figure 16: Distribution of first 20 features from the
first layer of the Graphormer network for three different training approaches
—scratch, HOMO-LUMO pretrained and atom-level pretrained— across test split of
cyp2c9 veith dataset. Figure 17: Distribution of first 20 features from the
first layer of the Graphormer network for three different training approaches
—scratch, HOMO-LUMO pretrained and atom-level pretrained— across test split of
clearance microsome az dataset. Figure 18: Distribution of first 20 features
from the first layer of the Graphormer network for three different training
approaches —scratch, HOMO-LUMO pretrained and atom-level pretrained— across
test split of solubility aqsoldb dataset. Figure 19: Distribution of first 20
features from the first layer of the Graphormer network for three different
training approaches —scratch, HOMO-LUMO pretrained and atom-level pretrained—
across test split of clearance hepatocyte az dataset. Figure 20: Distribution
of first 20 features from the first layer of the Graphormer network for three
different training approaches —scratch, HOMO-LUMO pretrained and atom-level
pretrained— across test split of hia hou dataset. Figure 21: Distribution of
first 20 features from the first layer of the Graphormer network for three
different training approaches —scratch, HOMO-LUMO pretrained and atom-level
pretrained— across test split of bbb martins dataset. Figure 22: Distribution
of first 20 features from the first layer of the Graphormer network for three
different training approaches —scratch, HOMO-LUMO pretrained and atom-level
pretrained— across test split of lipophilicity astrazeneca dataset. Figure 23:
Distribution of first 20 features from the first layer of the Graphormer
network for three different training approaches —scratch, HOMO-LUMO pretrained
and atom-level pretrained— across test split of cyp3a4 veith dataset. Figure
24: Distribution of first 20 features from the first layer of the Graphormer
network for three different training approaches —scratch, HOMO-LUMO pretrained
and atom-level pretrained— across test split of ld50 zhu dataset. Figure 25:
Distribution of first 20 features from the first layer of the Graphormer
network for three different training approaches —scratch, HOMO-LUMO pretrained
and atom-level pretrained— across test split of cyp3a4 substrate carbonmangels
dataset. Figure 26: Distribution of first 20 features from the first layer of
the Graphormer network for three different training approaches —scratch, HOMO-
LUMO pretrained and atom-level pretrained— across test split of cyp2d6 veith
dataset. Figure 27: Distribution of first 20 features from the first layer of
the Graphormer network for three different training approaches —scratch, HOMO-
LUMO pretrained and atom-level pretrained— across test split of cyp2d6
substrate carbonmangels dataset. Figure 28: Distribution of first 20 features
from the first layer of the Graphormer network for three different training
approaches —scratch, HOMO-LUMO pretrained and atom-level pretrained— across
test split of vdss lombardo dataset.


# Gromit: Benchmarking the Performance and Scalability of Blockchain
Systems††thanks: This work was funded by NWO/TKI grant BLOCK.2019.004

Bulat Nasrulin, Martijn De Vos, Georgy Ishmaev, Johan Pouwelse  Delft
University of Technology
{b.nasrulin, m.a.devos-1, g.ishmaev<EMAIL_ADDRESS>
###### Abstract

The growing number of implementations of blockchain systems stands in stark
contrast with still limited research on a systematic comparison of performance
characteristics of these solutions. Such research is crucial for evaluating
fundamental trade-offs introduced by novel consensus protocols and their
implementations. These performance limitations are commonly analyzed with ad-
hoc benchmarking frameworks focused on the consensus algorithm of blockchain
systems. However, comparative evaluations of design choices require macro-
benchmarks for uniform and comprehensive performance evaluations of
blockchains at the system level rather than performance metrics of isolated
components. To address this research gap, we implement Gromit, a generic
framework for analyzing blockchain systems. Gromit treats each system under
test as a transaction fabric where clients issue transactions to validators.
We use Gromit to conduct the largest blockchain study to date, involving seven
representative systems with varying consensus models. We determine the peak
performance of these systems with a synthetic workload in terms of transaction
throughput and scalability and show that transaction throughput does not scale
with the number of validators. We explore how robust the subjected systems are
against network delays and reveal that the performance of permissoned
blockchain is highly sensitive to network conditions.

###### Index Terms:

benchmark, blockchain performance, reproducibility, stress testing

##  I Introduction

The rapid growth in the number of blockchain protocols in the past few years
has been boosted by the interest in crypto-currencies, decentralized finance,
and identity systems. The solutions are mostly empirically driven, with direct
economic incentives stimulating engineering experiments. To date, there are
more than 700 different blockchain and distributed ledger platforms offering
native digital assets and products.Many of these solutions deploy original
families of consensus protocols or significant modifications of popular
protocols, with variations in scalability, performance, and decentralization
guarantees. These developments outpace systematization and research on
inherent trade-offs of different design choices [1].

The absence of benchmarking solutions for comprehensive comparative analysis
of various protocols is a particularly problematic omission, given the
cumulative marketcap of 1,468 trillion $ for these projects. More
fundamentally, this systematization gap hampers our ability to tackle the
increasing complexity of blockchain systems and make conscious design choices
in blockchain engineering. Developers of these protocols often provide
performance metrics of blockchain solutions as declarative whitepapers that do
not pass the standards of peer review and reproducibility, calling into
question the reliability and objectivity of these measurements. For instance,
it is a common practice to provide performance metrics of an isolated
component and report them as a system-wide performance metric [2, 3, 4]. This
practice often leads to false impressions of the end-to-end system
performance.

Few available benchmarking solutions, such as Blockbench [5] and Hyperledger
Caliper [6] focus on narrow sets of permissioned consensus protocols or DAG-
based protocols as DAGBENCH [7]. There is also a noticeable deficit of
academic research in macro benchmarks for blockchain systems. Existing studies
are rather limited in scope either focusing on specific protocols such as
Hyperledger [8, 9] or Ripple [10], or reusing Hyperldeger Caliper [11] and
Blockbench [5]. BCTMark is one of the few comparative benchmark studies which
compares three different protocols, including permissionless Ethereum
blockchain [12]. The authors in this study highlight the necessity to extend
the comparison set and include more metrics such as partition tolerance. The
most recent systematic survey on performance evaluation of blockchain systems
demonstrates that available comparative studies are rather limited in scope
both in terms of compared systems and depth of analysis, focusing on isolated
layers of blockchain systems [1]. To address this research gap, we design a
benchmark that is comprehensive in scope, allowing us to stress-test the
system under different network conditions.

Figure 1: Our abstraction model of blockchain system as a transaction fabric,
comprising four stages. The dotted line highlights the focus of most
whitepaper benchmarks.

We introduce _Gromit_ , a generic benchmarking framework that allows a
performance evaluation of _any_ blockchain solution. Gromit treats each system
under test as a transaction fabric, which means a transaction processing
system where a group of peers continuously reach a consensus on transactions
submitted by clients. Gromit analyses performance metrics related to
transactional data, specifically throughput and latency. As we will show, this
data alone can reveal the limitations of various aspects of the system.

We show the applicability of Gromit and conduct the _largest blockchain
benchmarking study to date_. Our benchmark involves seven major blockchain
solutions with different consensus algorithms. We determine the peak
performance of these blockchain systems for different numbers of peers and
without any modifications to the source code. A key finding is that the
performance for most evaluated blockchain systems _degrades_ when the number
of peers increases. We also reveal the effect of network delays and an
increase in system load on the distribution of end-to-end transaction
latencies, yielding valuable insights into the real-life bottlenecks of
underlying consensus mechanisms. While most performance bottleneck research
focuses on the consensus layer, a system-wide stress test can reveal
bottlenecks in other layers, e.g., in the persistence layer.

The contribution of this work is two-fold:

  1. 1.

We design and implement _Gromit_ , a benchmarking framework that enables an
analysis of _any_ blockchain system (Section III).

  2. 2.

We conduct _the largest blockchain benchmark to date_ , measuring the
performance of seven prominent blockchain systems (Section V).

##  II Blockchain as a Transaction Fabric

In this work, we view a blockchain system as a _transaction fabric_. Figure 1
visualizes this abstraction model, which illustrates a typical transaction
life cycle.

In our model, we distinguish between clients and peers. Clients are instances
that create transactions and send them to peers. An example of a client is a
light wallet or lightweight nodes in a Bitcoin network. Peers in our model are
responsible for processing and validating transactions in a shared,
decentralized network. Thus, we call them _validator peers_. In some
blockchain systems, they are also referred to as miners.

###  II-A Transaction Life Cycle Model

####  II-A1 Transaction Creation

A transaction contains logic that modifies the system state, e.g., by
transferring an asset to another account. Each transaction is
cryptographically signed with the private key of the issuing client to ensure
authenticity. Blockchain solutions usually provide Wallet API’s for clients to
submit their transactions, e.g., with an RPC endpoint or REST endpoint.

####  II-A2 Transaction Sharing

Blockchain systems employ complex transaction sharing mechanisms.
Permissionless blockchains typically use a global gossip protocol to share
transactions over a structured or unstructured overlay. Permissoned
blockchains are deployed in a more controlled network environment and, as a
result, might share transactions using a broadcast algorithm.

The transactions are stored in a datastore, often called a transaction pool
(TxPool). The transaction pool is a temporary store used to queue or
preprocess transactions before the network validates them.

Figure 2: Architectural overview and process flow of Gromit, our benchmarking
framework for blockchain systems.

####  II-A3 Transaction Settlement

A consensus algorithm is a crucial part of the blockchain system as a
mechanism for achieving security and liveness[13]. The intermediate outcome of
the consensus process in blockchain systems is a set of valid transactions.
Valid transactions are stored in a tamper-proof distributed ledger, a
replicated data structure maintained by validators. The system discards
invalid transactions.

Blockchain solutions typically bundle valid transactions in blocks,
interlinked in a hash chain, and stored in a local database (Block DB). Each
block in a hash chain contains the cryptographic hash of the previous block,
making illegitimate modifications of the chain detectable. Some blockchain-
like systems adopt a different organization of the distributed ledger, e.g.,
by maintaining a Directed Acyclic Graph (DAG) [7].

####  II-A4 Transaction Status Response

After the transaction is settled the client waits for a transaction approval
(or rejection), received from validator peers. Some blockchains also include a
proof in the response that proves that a transaction is finalized.

###  II-B Transaction Performance Indicators

Our approach is to analyze the performance of current blockchain solutions
through _transaction benchmarking_. The speed at which a blockchain system
processes transactions is a defining metric for blockchains. High transaction
latencies directly impact end-users experience, e.g., the relatively high
finalization times of Bitcoin transactions (10 minutes) make it unsuitable for
interactive trade [14]. We include all stages of the transaction life cycle in
our measurements, rather than focusing on the performance of the consensus
layer only.

We obtain insights into the limitation of blockchain systems by measuring peak
performance and associated transaction latencies. We also measure performance
under different network conditions, such as changes in the geographical
distribution of the overlay network. Our experiments in Section V highlight
that these metrics can reveal performance bottlenecks and act as a guideline
for optimization efforts.

##  III Gromit: A Generic Framework for Blockchain Benchmarking

We design and implement _Gromit_ , the first generic framework for blockchain
benchmarking. Gromit is designed for quick experiment iterations, utilizing a
domain-specific language to devise experiments. Our framework, implemented in
Python, is based on the abstraction model discussed in Section II. To
encourage the adoption of Gromit, its source code is published on
Github.111See https://github.com/grimadas/gromit Gromit provides developers
and researchers with the necessary tools to design benchmarks targeting
blockchain performance. Gromit spawns client and validator processes on remote
servers and coordinates interactions between running processes, e.g.,
transaction issuing.

###  III-A Experiment Flow

Figure 2 shows the architecture of the framework and the flow of an
experiment. This diagram is described next.

Experiment Setup. Before an experiment starts, it spawns a dedicated process,
called the _orchestrator_ , that setups the environment on remote servers and
handles cross-server communication. The orchestrator reads the configuration
file associated with the experiment. This file specifies the network addresses
of remote servers and the number of clients and validators that should be
started (step 1). The orchestrator then copies the source code and all
necessary experiment files to the remote servers using rsync. Next, the
coordinator setups the environment on the specified servers over an SSH
connection (step 2). This step includes the installation of required system
packages and the generation of a genesis file. This file describes the initial
state of the blockchain system and can pre-load specific accounts with assets.
The orchestrator then starts a _instance_ for each client or validator on the
remote server and assigns an identifier to each running instance. The scenario
manager, part of the logic of a Gromit instance, parses a provided scenario
file and starts the experiment (step 3).

Scenario Files. A user describes the actions performed during an experiment
with a _scenario file_. The scenario manager parses this file and schedules
actions using the asyncio library.

⬇

@8 init_blockchain_config {1-10}

@12 start_validator {1-10}

@20 start_client {11-20}

@80 start_creating_transactions {11-20}

@100 stop

Listing 1: An example of a scenario file in Gromit, describing an experiment
with 10 clients and validators.

An example scenario file associated with a simple blockchain benchmark is
given in listing 1. Actions and their timestamps are explicitly denoted. A
user can schedule an action to execute on a subset of all validators. Scenario
files can be machine-generated and are a flexible approach to devising and
conducting experiments.

Experiment Runner. The experiment runner spawns validators and clients as a
subprocess during each experiment run (step 4). Clients interact with
validators through the exposed API endpoint. We simulate a client in Gromit as
a procedure that issues transactions. During an experiment, Gromit instances
can share data over a TCP connection through a message broker. We use the
message broker functionality to share the credentials of pre-defined
blockchain accounts with clients. Gromit contains tools to gracefully
terminate a running experiment, e.g., when a particular condition is not met.

Gromit also provides utilities to track system resource usage. Gromit
instances monitor CPU, memory, disk, and network usage using the procfs
library (step 5). These metrics allow us to estimate the system resource usage
of blockchain systems. Developers can easily extend Gromit to monitor specific
metrics, for example, the number of inbound network messages for a particular
blockchain system.

Collecting Experiment Results. When the experiment ends, the orchestrator
copies all generated artifacts from the remote nodes using rsync (step 6).
These artifacts include the data generated by the blockchain systems and the
data output by the Gromit instances, e.g., monitoring statistics. This data is
parsed by the orchestrator (step 7) and generates human-readable graphs.
Finally, the data is stored in a database (step 8) and is ready for analysis
by the user (step 9).

###  III-B Integrating Blockchain Systems into Gromit

To show practicality of Gromit, we have integrated seven prominent blockchain
systems into it. The integration requires no change in the source code of the
blockchain systems. This allows to benchmark the blockchain system as close to
the deployed systems as possible.

The design of Gromit is modular, and developers can implement _modules_ that
enrich an experiment with more functionality, for example, bandwidth
monitoring. Integration of a blockchain system requires a developer to
subclass the BlockchainModule and to implement the init_configuration,
start_validator, stop_validator, and parse_ledger methods. Gromit enables
developers to specify custom network topologies to connect validators among
each other. We refer readers who wish to integrate a particular blockchain
system to the Gromit documentation.

Besides supporting blockchain systems, Gromit has support for generic
experiments with distributed systems. For example, we have used Gromit to
conduct experiments with peer-to-peer protocols on custom infrastructure.

###  III-C Transaction Analysis

A workload during a Gromit benchmark consists of transactions issued by one or
more clients. All transactions can be submitted to one validator peer or can
be evenly spread among the peers. In line with related work, we mainly gauge
blockchain performance using two metrics, transaction throughput, and
transaction latency. Specifically, we consider the peak transaction
_throughput_ , which is the maximum rate at which the system can process
transactions before getting congested. Second, we analyze the _latency_ of
transactions, which is the time between submitting a transaction and its
irreversible inclusion in the distributed ledger.

We store the timestamp at which a client has submitted the transaction to a
validator to determine transaction latency. However, determining the
finalization time of transactions can be complicated since some blockchain
systems do not expose granular, temporal information. We use the following two
approaches to determine the latency of individual transactions. Our first
approach is to inspect the resulting distributed ledger after all transactions
have been issued (this logic should be part of the parse_ledger method). We
then compute transaction latency based on timestamps included in ledger data
elements (e.g., blocks). However, this approach is useful only if the
blockchain system annotates data elements in the distributed ledger with a
timestamp. The second approach suggests clients periodically poll the
blockchain system to determine if a transaction has been confirmed.

##  IV Blockchain Consensus Model

Since the introduction of Bitcoin in 2008, there have been many proposals for
new blockchains and consensus mechanisms. Some proposals have materialized
into operational systems, whereas other ones are only theoretically analyzed.
The proliferation of blockchain solutions makes it infeasible to conduct a
benchmarking study with all available systems.

Blockchain System |  Consensus Model |  Consensus Principle |  Validator Group Formation |  First commit (year)
---|---|---|---|---
Ethereum (v1.9.24) [15] |  Proof of Work (PoW) |  Resource-based lottery |  Resource mining |  2013
Algorand (v2.3.0) [16] |  Proof of Stake (PoS) |  Random selection of leaders |  Stake-based enrolment |  2019
BitShares (v5.0.0) [17] |  Delegated Proof of Stake (dPoS) |  Rotating leader |  Election by stakeholders |  2015
Diem (v1.1.0) [18] |  Practical Byzantine Fault Tolerance (PBFT), based on HotStuff [19] |  Leader-based |  Enrolment by an authority |  2019
Stellar (v15.1.0) [3] |  Federated Byzantine Agreement (FBA) |  Quorum Intersection |  User-defined quorums |  2014
Hyperledger Fabric (v1.4.9) [20] |  Crash-tolerant consensus (CFT), based on Raft [21] |  Leader-based |  Enrolment by an authority |  2016
Avalanche (v1.1.1) [2] |  Meta-Stable Consensus (MSC) |  Network subsampling |  Stake-based enrolment |  2020
TABLE I: The seven selected blockchain systems and consensus models analysed
in this work.

Based on taxonomy of different consensus mechanisms used in blockchain systems
[13] we consider seven prominent consensus models and select a representative
blockchain system for each consensus model. Our selection process is based on
the economic magnitude, adoption, maturity of the system, and the protocol’s
academic significance. Table I lists the representative system for each
considered consensus model. For each system, we also show the evaluated
version of the software, the principle underpinning each consensus model, how
the group of validators is formed, and the year in which the first software
commit has been made (as an indicator of matureness). We are aware of state-
of-the-art blockchains that use techniques such as layer-one scaling to
improve throughput, e.g., OmniLedger [22] and RapidChain [23], or layer-two
scaling solutions [24]. However, the primary focus of our benchmarking study
is on deployed layer-one blockchain systems. We now elaborate on each
consensus model and refer the reader to [13] for a more extensive overview of
blockchain consensus models.

Proof-of-Work (PoW) is the oldest consensus mechanisms designed explicitly for
blockchain systems. PoW is used in blockchains such as Bitcoin [25] and
Ethereum [15] and to date remains a standard approach to build open blockchain
systems. PoW scales well in the number of participants: the Bitcoin network
has over 11’000 operational miners. However, PoW is known to consume large
amounts of energy, and the achievable transaction throughput is theoretically
limited to around 60 transactions per second [26].

Proof-of-Stake (PoS), initially proposed by the Peercoin cryptocurrency [27],
is an alternative consensus mechanism that addresses the excessive resource
usage by PoW. PoS is typically based on a random, periodic selection of a
leader. This selection process is weighted by the participants’ stake in the
system, e.g., by the number of assets owned, or by the age of possessed
assets. Algorand is one of the most prominent PoS-based blockchains and
leverages Verifiable Random Functions (VRFs) for the leader election process
[16].

Delegated Proof-of-Stake (dPoS) is a consensus mechanism where stakeholders
elect a group of _delegates_ through voting. Voting decision can, for example,
be based on community engagement. Since the set of validators participating in
consensus remains relatively small compared to PoS-based solutions, dPoS-based
consensus has in theory a better potential to scale. BitShares is one of the
most mature blockchains using dPoS consensus [17].

Practical Byzantine Fault Tolerance (PBFT) is a consensus algorithm introduced
in 1999 [28]. PBFT is specifically designed for networks with static and pre-
approved membership and has been revised for adoption in blockchain
environments. Recent advancements have resulted in HotStuff [19], a consensus
protocol based on PBFT that reduces communication overhead and increases
throughput. HotStuff is at the core of Diem, a permissioned distributed ledger
maintained by a consortium led by Facebook [18].

Federated BFT (FBFT) is a consensus model that distinguishes itself from the
approaches mentioned earlier by having validators explicitly specifying trust
relations. Stellar is one of the first systems to adopt FBFT consensus [3].
The Stellar Consensus Protocol (SCP) leverages a federated voting approach in
which each validator votes on statements while ensuring that no two members of
an overlapping quorum can confirm contradicting statements.

Crash-tolerant Consensus (CFT) is a consensus approach widely used to achieve
fault tolerance, e.g., by Apache Kafka [29]. Unlike PBFT, CFT is not resistant
against arbitrary (Byzantine) behaviour but can withstand crash-stop failures
of participants. Notable algorithms achieving CFT are Paxos [30] and Raft
[21]. Hyperledger Fabric, one of the most prominent industrial blockchains, is
currently using Raft [20].

Metastable Consensus (MSC) is a family of consensus algorithms that leverage
network subsampling techniques to determine the validity of a transaction. The
idea is to repeatedly sample random validators in the network and to steer
correct nodes to a common decision. Avalanche is one of the most mature
blockchain solutions to leverage MSC consensus and maintains a DAG data
structure to store transactions [2].

(a) Peak transaction throughput

(b) Transaction latency

Figure 3: The peak throughput and transaction latencies of evaluated
blockchain systems with the number of validators.

##  V Blockchain Performance Evaluation

We conduct a diverse set of experiments with Gromit to reveal the performance
characteristics and limitations of seven blockchain solutions. To the best of
our knowledge, We are the first to perform a large-scale performance benchmark
of blockchain solutions.

Our experiments answer the following questions: How does the increase in the
number of validators affect the system performance? What is the peak
performance of each system under a heavy system load? What is the impact of a
network delay on systems’ performance? How consistent are our performance
results compared to previously reported values?

Throughout the section we use two variables for our experiments: $n$ indicates
the number of validators and $\lambda$ signifies the transaction throughput.

###  V-A Setup and Transaction Workload

All experiments are conducted on four HPE DL385 Gen10 servers, located within
the same data centre and inter-connected with 10GB Ethernet links. Each server
is equipped with 128 AMD EPYC 7452 CPUs, has 512GB of DDR4 memory, and runs
Debian 10. During our experiments, we deploy each blockchain system with its
source code unmodified. For each system, we use the default settings provided
by the systems. Each experiment starts with only the genesis block included in
the blockchain. We use a random network topology where each validator is
connected to 10 other random validators.

We use Gromit to subject each system to a synthetic transaction workload,
issued by up to 64 clients. To ensure an equal load on each validator, a
client submits transactions to the validator with ID $i\equiv c\
(\mathrm{mod}\ n)$ where $c$ is the ID of the client and $n$ the total number
of validators. Each transaction is submitted to exactly one validator.
Transactions are submitted during a two-minute period, after which we wait an
additional minute for all transaction to be finalized.

We use simple asset transfers as a performance baseline. In our workload, a
transaction issued by a client involves an asset transfer of a small, fixed
amount to another account; the client counterparty is fixed throughout the
experiment. We ensure that each client has sufficient funds to spend during
the experiment. For Ethereum and Hyperledger Fabric, transactions involve the
transfer of an ERC20 token.

###  V-B Determining Peak Transaction Throughput

To determine peak transaction throughput, we gradually increase the system
transaction rate in steps of 100 transactions per second (tx/s). Based on the
reported statistics by Gromit, we estimate the peak transaction load that each
system is still able to process during a sustained period. If the system has
any unconfirmed transactions after our two-minute period, we consider the
system as “saturated”. We evaluate the peak throughput of each system with an
increasing number of validators ($n$). We provide each system with an equal
amount of resources and ensure that the resource usage of evaluated systems
(CPU power, disk space, and memory) does not exceed the available resources.
Due to excessive resource usage of the Stellar software, we are only able to
run Stellar with up to $64$ validators. We run each experiment at least five
times and average all results. Appropriate graphs are annotated with 95%
confidence interval markers.

Finding 1. _Adding validators does not have a significant positive effect on
the achievable peak transaction throughput of the evaluated systems._

Figure 3 shows the result of our scalability experiment as $n$ grows, in terms
of peak transaction throughput and transaction latency. Figure 3(a) shows the
peak transaction throughput of evaluated systems (with a horizontal and
vertical log-axis). We notice that none of the evaluated systems can process
over 1’000 tx/s with $n=128$. In general, the transaction throughput of most
of the systems is capped between 500 and 1’500 tx/s. Except for Ethereum, the
peak transaction of all blockchains is _decreasing_ as $n$ increases.
Specifically, Hyperledger Fabric shows a severe degradation in performance
when $n>8$, and is just capable of processing 2 to 4 tx/s with $n=128$. We
believe that this is caused by the underlying consensus model of Hyperledger
Fabric, Raft, which does not scale well with the number of validators [21]. Of
all systems, Ethereum has the lowest transaction throughput (around 10-20
tx/s), yet manages to keep stable throughput with the increase in the number
of validators. Peak throughput of Avalanche increases up to $n=64$, but then
degrades for $n=128$.

Finding 2. _For Avalanche, BitShares and Hyperledger Fabric, we observe
significant discrepancies between the peak throughput found by us and
previously reported values._

Figure 4: The distribution of transaction latencies for systems, under peak
load ($\lambda$ tx/s) and moderate load ($\frac{\lambda}{2}$ tx/s) ($n=64$).

The previous experiment estimates the peak throughput of blockchain systems,
without modification to the source code and under default settings. We now
compare the performance results with values reported by other literature. The
performance, i.e. peak throughput, is typically determined through an
evaluation by system developers themselves. For Stellar, we could not find
reliable benchmarking results, making this work the first benchmarking study
of Stellar. We are interested to see if there are significant inconsistencies
with performance metrics reported by these studies.

Our performance results are comparable with results reported for Ethereum
(4-40 tx/s [31]), Algorand (880 tx/s [16]) and Diem (200-1’000 tx/s [32]).
However, we notice that previously reported values for some systems are
significantly higher than our findings, specifically for Avalanche (7’000
tx/s, 26x higher [2]), BitShares (3’300 tx/s, 3x higher [33]) and Hyperledger
Fabric (3’500 tx/s, 2 times higher [20]). For each system, we now explain
these inconsistencies with additional experiments and analysis.

Avalanche. The relative low throughput of Avalanche surprises us and warrants
further performance analysis. Since we noticed that each validator node is
fully utilizing a CPU core, even with $n=4$ and 32 tx/s, we perform a CPU
analysis of deployed validators using the pprof profiler. We find that around
60% of CPU time is spent on hash computations using the argon2 algorithm [34].
This CPU consumption originates from the API provided by Avalanche validators.
Specifically, each validator maintains a keystore with credentials that is
managed by end users; interactions with that keystore, e.g., accessing a
private key, requires the user to include the password hash in the request.
Consequently, many parallel requests to the API by clients cause severe
performance degradations.

To analyse the impact of password hashing, we recompile Avalanche with this
hash verification disabled and re-run our experiment. We do not observe a
significant increase in transaction throughput. However, this reveals another
performance bottleneck, originating from the verification of transactions,
consuming around 90% of CPU time. Since Avalanche transactions are linked in a
DAG structure, incoming transactions require the validation of parent
transactions, which is a resource-intensive, recursive operation. We believe
this can be addressed with further engineering efforts, e.g., queueing the
verification of transactions.

BitShares. We further analyse the reported throughput of BitShares and found
that the peak transaction throughput (3’300 tx/s) is not an accurate
performance indicator since the sustained throughput throughout the experiment
is only around 450 tx/s. Specifically, the achievable throughput of the
BitShares consensus algorithm seems to be predicated by the speed of the
slowest consensus participants in terms of connectivity and CPU resources.
When operating BitShares in a heterogeneous environment, this can result in
significant deviations in transaction throughput.

Hyperledger Fabric. We further analyse the results reported in the work of
Androulaki et al. [20] and Blockbench [5]. We find that their work evaluates
an early implementation of Hyperledger Fabric using a different consensus
model (Zookeeper or PBFT). As such, these results are not directly comparable.

We also present the following two reasons to explain the discrepancies in
reported and observed throughput numbers:

  1. 1.

_Experimental Software vs Production_. Many of the throughput numbers reported
by system developers are extracted using a premature or even incomplete
software implementation. As such, we argue that the values found by our
experiments are a more accurate reflection of the achievable throughput in a
production environment. Additionally, we noticed that some solutions (Diem and
Stellar) have built-in measures that artificially lower the achievable
throughput, likely to ensure safety properties or to prevent attacks in a
production environment. A similar insights were observed in [35].

  2. 2.

_Client-Validator Interaction_. In our experiments, clients submit
transactions to the API exposed by the system, whereas other studies might
directly inject transactions in the validator process. API-based interaction
adds additional overhead as the request needs to be processed, and this
approach therefore is likely to lower the peak throughput of the system.
However, this approach resembles how users interact with validators when a
blockchain system is Internet-deployed.

###  V-C Transaction Latency

Finding 3. _For all evaluated systems, the average transaction latency under
peak load is largely independent of the number of validators._

Figure 5: Transaction latencies without network modifications and in a geo-
distributed setting ($\lambda=64$ tx/s, $n=32$). The transaction latencies of
Ethereum are shown in the right plot.

Figure 3(b) shows the average transaction latency under peak load, as $n$
increases. Except for Ethereum, the average transaction latency of evaluated
systems is around or below ten seconds. The variance between different runs is
relatively low, except for Ethereum. In general, the average transaction
latency increases as the number of validators grows. The consensus algorithm
underlying BitShares, Algorand and Stellar progresses in five-second rounds,
theoretically resulting in an average transaction latency of 2.5 seconds. We
see that the transaction latency of Algorand increases from 5 seconds for
$n=4$ to 15 seconds with $n=128$, suggesting that consensus rounds take longer
to complete. For BitShares and Stellar, this increase is less pronounced.

Finding 4. _The variance of transaction latencies for Algorand and Diem
increases significantly under peak load, compared to a moderate load. However,
the transaction latencies of BitShares and Ethereum are largerly indifferent
towards the system load._

We visualize the distribution of transaction latencies for each system to
explore further the effects of increasing the system load on the transaction
latency. We consider both peak and moderate loads, the latter being defined as
half the determined peak load. Figure 4 shows this distribution in a violin
plot. We observe that Algorand, BitShares, Hyperledger Fabric, and Stellar
transaction latencies are roughly uniformly distributed under moderate load.
These systems adopt a round-based consensus approach, with a target of around
five seconds for Algorand, BitShares, and Stellar, and one second for
Hyperledger Fabric. For these systems, most transactions are usually confirmed
within the current or next consensus round relative to transaction submission.
The distribution of transaction latencies transforms as the system is
subjected to a peak load. Figure 4 shows that the finalization of Algorand
transactions is being deferred to later rounds: 6% of Algorand transactions
have a transaction latency above 20 seconds. This effect is less pronounced
for BitShares and Stellar.

Increasing the system load impacts the latency distribution of Diem
transactions. Further investigation reveals that the round duration in Diem
adjusts to the system load. As more validators join the network and as more
transactions are submitted to Diem validators, the round duration increases to
ensure transactions can be processed on time. Nonetheless, 30% of all issued
transactions in Diem are confirmed only after 10 seconds under peak load,
whereas the system can handle all submitted transactions within 7 seconds
under moderate load.

###  V-D Impact of Network Delays

Finding 5. _Adding network delays has a minimal effect on the transaction
latencies of Algorand and Stellar. However, Avalanche and Diem are extremely
sensitive to network delays._

Finally, we modify the network settings using the netem Linux kernel
module222TC documentation: https://www.linux.org/docs/man8/tc-netem.html and
measure the impact of a network delay on the transaction latency for
integrated blockchain systems. This experiment reveals how different
blockchain systems are sensitive to deployment in geo-distributed settings. To
replicate realistic network delays, we extract ping statistics for 32 cities
worldwide from WonderNetwork [36] and assign each validator to a city in a
round-robin fashion.

Figure 5 shows the distribution of transaction latencies of systems deployed
in one data center with unmodified network settings and when deployed with
geo-distributed settings. For each experiment run, we use 32 validators and
fix the transaction load to 64 tx/s, which all evaluated systems can handle
without congestion. Except for Algorand and Stellar, network delays visibly
impact transaction latencies.

The effect is the most pronounced for the permissoned systems Diem and
Hyperledger Fabric. This suggests that these systems can only operate in
controlled/closed network environments, as any change in the network
significantly affects the performance. We also observe a significant impact of
network conditions on Avalance. This is due to the poll-based nature of
metastable consensus. The poll rounds are visible in Figure 5.

For BitShares, we observe that validators occasionally fail to produce a block
with higher network latency. For Diem, consensus progress stales a few seconds
after starting the experiment, and rounds are timing out without confirming
any transaction, violating system liveness.

###  V-E Network and CPU Utilization

Finding 6. _Algorand, Stellar, Diem show high network utilization under idle
load. Ethereum, Stellar and Diem consume significant CPU resources under idle
load._

(a) Network usage

(b) CPU utilization

Figure 6: The resource utilization of the evaluated systems with increase in
transaction load ($n=32$).

(a) Network usage

(b) CPU utilization

Figure 7: The network and CPU usage for evaluated systems, when increasing the
number of validators ($\lambda=32$ tx/s).

We track the total network usage (inbound and outbound traffic) and average
CPU utilization for each system while increasing $\lambda$ and fixing $n$ to
32. To obtain insights into the system under idle load, we also run blockchain
systems with $\lambda=$ 0 tx/s. Figure 6(a) shows that the network usage per
validator quickly grows for Avalanche and Stellar as the transaction load
increases. For $\lambda=256$ tx/s, both Avalanche and Stellar use over 800 MB
of network traffic per validator process. BitShares is the most network-
efficient, using only 80 MB per validator for $\lambda=256$ tx/s. We also
observe that Algorand, Diem, and Stellar show 10x to 100x more bandwidth
consumption under no transaction load compared to other systems.

Figure 6(b) shows the average CPU utilization when increasing $\lambda$. The
mining process of Ethereum is continuously utilizing a single CPU. Under
$\lambda=256$ tx/s, BitShares and Hyperledger Fabric are the most CPU-
efficient compared to other systems. Our synthetic workload results are
consistent with real-world observations of liveness issues for Avalanche and
Stellar [37, 38].

Figure 7(a) shows that Diem is using significant network resources for $n=4$:
970 MB per validator process. This number decreases quickly when $n$
increases. We explain this behavior by the self-adjusting round times of the
Diem consensus mechanism: as $n$ increases, rounds take longer to complete,
lowering the bandwidth usage. For Stellar, we see the opposite effect: network
usage becomes significant for $n=128$. Although Stellar cannot process
transactions for $n=128$, we report its resource usage nonetheless. Inspection
of Stellar logs reveals that validators lose track of consensus, clogging the
network with resynchronization messages.

Figure 7(b) shows how CPU utilization behaves when increasing $n$. BitShares
is the most CPU-efficient for all evaluated values of $n$. The CPU load of
validators decreases for Avalanche and Diem as $n$ increases. Since we fix the
transaction load, adding more validators decreases the individual load.

##  VI Related Work

Blockchain benchmarking and its associated challenges has received attention
from other researchers. Fan et al. present an extensive survey outlining
methods for evaluating blockchain performance [1]. The work of Wang and Ye
describes benchmarking tools and consensus mechanisms and outlines techniques
to improve the throughput of blockchains [39]. The authors of these studies
summarize work on blockchain performance but do not conduct benchmarking
studies themselves.

Popular blockchain benchmarking tools are Blockbench [5], Hyperledger Caliper
[6], and DAGBench [7]. Blockbench, introduced in 2017, is the earliest
benchmarking framework for blockchain and is specifically designed to evaluate
permissioned blockchains [5]. Blockbench measures the performance of
components commonly found in blockchains, e.g., the transaction execution
engine). The authors of Blockbench evaluate the performance of Hyperledger
Fabric, Ethereum, and Parity. Hyperledger Caliper is a benchmarking tool for
the performance evaluation of specific systems with a set of pre-defined use
cases [6]. Hyperledger Caliper primarily supports projects by the Hyperledger
Foundation. DAGBench is a benchmarking tool for DAG-based blockchains [7]. The
authors evaluate three popular DAG-based blockchain implementations [40]:
IOTA, Nano, and Byteball. However, DAGBench does not allow for a comparison of
other blockchains. The authors of BCTMark [12]. present an Ethereum
performance evaluation.

##  VII Conclusion

We have presented Gromit, a generic benchmarking framework for blockchain
solutions. By treating each blockchain system as a transaction fabric, our
framework enables any blockchain system’s integration, benchmarking, and
performance analysis. We leverage the functionalities of Gromit and conduct
the largest blockchain benchmark to date, involving seven prominent blockchain
systems. Our main finding is that none of the evaluated solutions can handle
beyond 1’000 transactions per second as the number of validators increases.
Yet, they show relatively low transaction latencies on average. We also find
that synthetic workloads can provide accurate predictions of performance
limitations, as our findings are consistent with real-world observations on
performance incidents on Avalanche and Stellar networks.

## References

  * [1] C. Fan _et al._ , “Performance evaluation of blockchain systems: A systematic survey,” _Access_ , 2020.
  * [2] T. Rocket _et al._ , “Scalable and probabilistic leaderless bft consensus through metastability,” _arXiv preprint arXiv:1906.08936_ , 2019.
  * [3] M. Lokhava _et al._ , “Fast and secure global payments with stellar,” in _SOSP_. ACM, 2019.
  * [4] B. Cao _et al._ , “Performance analysis and comparison of pow, pos and dag based blockchains,” _Digital Communications and Networks_ , 2020.
  * [5] T. T. A. Dinh _et al._ , “Blockbench: A framework for analyzing private blockchains,” in _ICMD_ , 2017.
  * [6] “Hyperledger Caliper,” https://github.com/hyperledger/caliper, 2018.
  * [7] Z. Dong _et al._ , “Dagbench: A performance evaluation framework for dag distributed ledgers,” in _CLOUD_. IEEE, 2019.
  * [8] H. Sukhwani _et al._ , “Performance modeling of hyperledger fabric (permissioned blockchain network),” in _NCA_. IEEE, 2018.
  * [9] M. Kuzlu _et al._ , “Performance analysis of a hyperledger fabric blockchain framework: throughput, latency and scalability,” in _IEEE international conference on blockchain (Blockchain)_ , 2019.
  * [10] M. Touloupou _et al._ , “Towards a framework for understanding the performance of blockchains,” in _BRAINS_. IEEE, 2021, pp. 47–48.
  * [11] M. Dabbagh _et al._ , “Performance analysis of blockchain platforms: Empirical evaluation of hyperledger fabric and ethereum,” in _IEEE IICAIET_ , 2020, pp. 1–6.
  * [12] D. Saingre _et al._ , “Bctmark: a framework for benchmarking blockchain technologies,” in _AICCSA_. IEEE, 2020.
  * [13] S. Bano _et al._ , “Sok: Consensus in the age of blockchains,” in _AFT_ , 2019.
  * [14] T. Bamert _et al._ , “Have a snack, pay with bitcoins,” in _P2P_. IEEE, 2013.
  * [15] G. Wood _et al._ , “Ethereum: A secure decentralised generalised transaction ledger,” _Ethereum project yellow paper_ , 2014.
  * [16] Y. Gilad _et al._ , “Algorand: Scaling byzantine agreements for cryptocurrencies,” in _SOSP_. ACM, 2017.
  * [17] F. Schuh and D. Larimer, “Bitshares 2.0: General overview,” 2017.
  * [18] “Libra white paper,” https://libra.org/en-US/white-paper/, 2019.
  * [19] M. Yin _et al._ , “Hotstuff: Bft consensus with linearity and responsiveness,” in _SPDC_ , 2019.
  * [20] E. Androulaki _et al._ , “Hyperledger fabric: a distributed operating system for permissioned blockchains,” in _EuroSys_. ACM, 2018.
  * [21] D. Ongaro and J. Ousterhout, “In search of an understandable consensus algorithm,” in _USENIX ATC_ , 2014.
  * [22] E. Kokoris-Kogias, P. Jovanovic, L. Gasser, N. Gailly, E. Syta, and B. Ford, “Omniledger: A secure, scale-out, decentralized ledger via sharding,” in _2018 IEEE Symposium on Security and Privacy (SP)_. IEEE, 2018, pp. 583–598.
  * [23] M. Zamani, M. Movahedi, and M. Raykova, “Rapidchain: Scaling blockchain via full sharding,” in _Proceedings of the 2018 ACM SIGSAC Conference on Computer and Communications Security_ , 2018, pp. 931–948.
  * [24] L. Gudgeon _et al._ , “Sok: Layer-two blockchain protocols,” in _International Conference on Financial Cryptography and Data Security_. Springer, 2020, pp. 201–226.
  * [25] S. Nakamoto, “Bitcoin: A peer-to-peer electronic cash system,” Manubot, Tech. Rep., 2019.
  * [26] A. Gervais _et al._ , “On the security and performance of proof of work blockchains,” in _SIGSAC_ , 2016.
  * [27] S. King and S. Nadal, “Peercoin–secure & sustainable cryptocoin,” _Aug-2012 [Online]. Available: https://peercoin. net/whitepaper ()_ , 2012\.
  * [28] M. Castro, B. Liskov _et al._ , “Practical byzantine fault tolerance,” in _OSDI_ , 1999.
  * [29] “Apache Kafka,” https://kafka.apache.org, 2017.
  * [30] L. Lamport _et al._ , “Paxos made simple,” _ACM Sigact News_ , 2001.
  * [31] S. Pongnumkul _et al._ , “Performance analysis of private blockchain platforms in varying workloads,” in _ICCCN_. IEEE, 2017.
  * [32] J. Zhang _et al._ , “Performance analysis of the libra blockchain: An experimental study,” in _HotICN_. IEEE, 2019.
  * [33] “BitShares Industrial Performance and Scalability,” https://bitshareshub.io/industrial-performance-and-scalability/, 2017.
  * [34] A. Biryukov _et al._ , “Argon2: new generation of memory-hard functions for password hashing and other applications,” in _EuroS &P_. IEEE, 2016.
  * [35] J. A. Chacko, R. Mayer, and H.-A. Jacobsen, “Why do my blockchain transactions fail? a study of hyperledger fabric,” in _SIGMOD’21_ , 2021.
  * [36] WonderNetwork, “Global ping statistics,” https://wondernetwork.com/pings, accessed: 2022-05-12.
  * [37] P. O’Grady. Preliminary analysis of the invalid minting bug. Accessed: 2022-05-12. [Online]. Available: https://medium.com/avalancheavax/preliminary-analysis-of-the-invalid-minting-bug-bee940cbd9e9
  * [38] M. McSweeney, “Stellar hit by transaction issues, developers say ’network is still online’ despite node outage,” _The Block_ , 2021, accessed: 2022-05-12. [Online]. Available: https://www.theblockcrypto.com/linked/100649/stellar-network-stoppage-developers-investigation
  * [39] R. Wang _et al._ , “Performance benchmarking and optimization for blockchain systems: A survey,” in _ICBC_. Springer, 2019.
  * [40] H. Pervez _et al._ , “A comparative analysis of dag-based blockchain architectures,” in _ICOSST_. IEEE, 2018, pp. 27–34.