# NovoBench: Benchmarking Deep Learning-based
_De Novo_ Peptide Sequencing Methods in Proteomics

Jingbo Zhou∗1, Shaorong Chen∗1, Jun Xia∗1, Sizhe Liu2, Tianze Ling3
Wenjie Du1, Yue Liu1, Jianwei Yin4, Stan Z. Li†1
1Westlake University, 2University of Southern California
3Tsinghua Univerisity, 4Zhejiang University
albertnoend@gmail.com,{chenshaorong, xiajun<EMAIL_ADDRESS>
###### Abstract

Tandem mass spectrometry has played a pivotal role in advancing proteomics,
enabling the high-throughput analysis of protein composition in biological
tissues. Many deep learning methods have been developed for _de novo_ peptide
sequencing task, i.e., predicting the peptide sequence for the observed mass
spectrum. However, two key challenges seriously hinder the further advancement
of this important task. Firstly, since there is no consensus for the
evaluation datasets, the empirical results in different research papers are
often not comparable, leading to unfair comparison. Secondly, the current
methods are usually limited to amino acid-level or peptide-level precision and
recall metrics. In this work, we present the first unified benchmark NovoBench
for _de novo_ peptide sequencing, which comprises diverse mass spectrum data,
integrated models, and comprehensive evaluation metrics. Recent impressive
methods, including DeepNovo, PointNovo, Casanovo, InstaNovo, AdaNovo and
$\pi$-HelixNovo are integrated into our framework. In addition to amino acid-
level and peptide-level precision and recall, we evaluate the models’
performance in terms of identifying post-tranlational modifications (PTMs),
efficiency and robustness to peptide length, noise peaks and missing fragment
ratio, which are important influencing factors while seldom be considered.
Leveraging this benchmark, we conduct a large-scale study of current methods,
report many insightful findings that open up new possibilities for future
development. The benchmark will be open-sourced to facilitate future research
and application.

##  1 Introduction

Proteomics, the study of proteins within biological systems, relies heavily on
mass spectrometry for protein identification aebersold2003mass . Traditional
methods use existing databases to match observed peptide fragments with known
sequences. However, these methods may miss novel or modified peptides not
present in the databases ma2003peaks ; ma2015novor ; nesvizhskii2010survey .
_De novo_ peptide sequencing offers a solution by directly annotating mass
spectra to reconstruct peptide sequences without relying on databases. By
circumventing the need for predefined databases, _de novo_ sequencing enables
researchers to uncover new peptides and investigate post-translational
modifications (PTMs), contributing to a deeper understanding of cellular
processes uzozie2018advancing  and disease mechanisms mann2003proteomic .
Here, PTMs refers to chemically modified version of 20 naturally-occurring
amino acids, influencing many critical biomolecular processes including
central enzyme activities, protein turnover, and DNA repair ramazi2021post ;
duan2015roles ; pejaver2014structural . Deep learning has been widely used in
the _de novo_ peptide sequencing, where researchers “translate” the observed
mass spectrum to peptide sequence using the encoder-decoder architectures
tran2017novo ; qiao2021computationally ; yilmaz2022novo ; mao2023mitigating ;
xia2024adanovo ; liu2023accurate .

Despite the remarkable success, there exists three key challenges that
seriously hinder the further development of deep learning-based _de novo_
peptide sequencing:

  * •

The datasets for fair evaluation. Because the Peptide-Spectrum Matches (PSMs)
for training and evaluation are easily accessible in ProteomeXchange
vizcaino2014proteomexchange , researchers may download different parts for the
evaluation of their respective models in _de novo_ peptide sequencing. For
example, DeepNovo tran2017novo  and PointNovo qiao2021computationally  report
the performance on the seven-species dataset, while InstaNovo
Eloff2023.08.30.555055  evaluate the performance on the other data collected
by themselves. Furthermore, there are multiple versions of the dataset
available, and the datasets used by the models with the same name are actually
different. For example, PointNovo and CasaNovo use different versions of the
Nine-species dataset (MassIVE dataset identifier: MSV000090982, MSV000081382).
The inconsistency of the datasets used would potentially obscure the true
progress in the field.

  * •

The metrics for comprehensive evaluation. Although previous works in _de novo_
peptide sequencing share the metrics of peptide-level or amino acids-level
precision and recall, they fail to evaluate some important abilities of the
models. For example, PTMs play an essential role in elucidating protein
functions, however, we find that current methods struggle to identify them
compared to naturally-observing amnio acids. Hence, it is necessary to
establish some metrics to evaluate the models’ abilities in identifying PTMs.
Moreover, we observe that some models are computationally extensive, hindering
the practical deployment of these tools and necessitating the metrics for
training and inference efficiency.

  * •

The robustness to important influencing factors. Intuitively, longer peptide
sequences and a higher noise peaks ratio are expected to degrade the
performance of various models. However, the extent to which these factors
affect different models remains unknown. This underscores the importance of
selecting the appropriate model for specific scenarios. Additionally, another
crucial factor is the missing fragmentation ratio. Previous
studiesmcdonnell2022impact  have indicated that missing fragmentation during
peptide sequencing results in insufficient information for determining peptide
sequences in those regions, increasing the likelihood of errors in the
resulting peptide identifications. However, few previous works have evaluated
the impact of this factor on model performance.

To solve these challenges, we develop the first deep learning-based _de novo_
sequencing benchmark that supports unified, reproducible, and efficient
evaluations. We comprehensively integrated deep learning-based _de novo_
peptide sequencing models including DeepNovotran2017novo ,
PointNovoqiao2021computationally , Casanovoyilmaz2022novo ,
InstaNovoEloff2023.08.30.555055 , AdaNovoxia2024adanovo , and
$\pi$-HelixNovo10.1093/bib/bbae021  into our framework. Additionally, in
addition to amino acids-level or peptide-level precision and recall, we take
new metrics including PTMs precision and efficiency. Based on this benchmark,
we conduct extensive experiments to compare different models in a fair
fashion. We also investigate the models’ robustness to the three key
influencing factors, providing guidance for model selection according to
specific applications.

##  2 Background and Task Definition

Protein identification is a central objective in proteomics-related analyses.
The liquid chromatography-tandem mass spectrometry (LC-MS/MS) technique is
widely utilized for both the quantitative and qualitative analysis of
proteins. In the typical protein identification workflow of shotgun
proteomics, proteins are first digested by enzymes to generate a mixture of
peptides wolters2001automated .

These peptides are then separated using liquid chromatography. Each charged
peptide is analyzed by a mass spectrometer, producing first scan (MS1) spectra
that show the mass-to-charge (_m/z_) ratios of the intact peptides.
Subsequently, the peptides are fragmented in the mass spectrometer, yielding
second scan (MS2) spectra, which consist of multiple peaks, each defined by an
_m/z_ value and an intensity value. A crucial part of protein identification
is peptide sequencing, which involves predicting the peptide sequence from the
observed MS2 spectrum and precursors (the mass and charge of the peptide).
Unlike database search-based peptide sequencing methods, _de novo_ peptide
sequencing relies solely on the MS2 spectrum information without using any
pre-constructed databases. Finally, the complete protein sequence can be
deduced using assembly methods liu2015proteome .

The precise task definition of _de novo_ peptide sequencing is the process of
reconstructing a peptide’s amino acid sequence directly from an observed MS2
spectrum, along with the precursor charge and mass. The MS2 spectrum can be
denoted as as
$\mathbf{x}=\left\\{\left(m_{i},I_{i}\right)\right\\}_{i=1}^{M}$, where each
peak $\left(m_{i},I_{i}\right)$ forms a 2-tuple representing the m/z and
intensity value, and $M$ is the number of peaks that can be varied across
different mass spectra. The precursor charge and mass are represented as an
integer indicating the charge and a floating-point number indicating the mass,
respectively. Additionally, we represent the peptide sequence which we want to
identify as
$\mathbf{y}=\left\\{\left(y_{1},y_{2},\dots,y_{N}\right)\right\\}$, where
$y_{i}$ is the type of the $i$-th amino acid, $N$ is the peptide length and
can be varied across different peptides.

Figure 1: The overview of the NovoBench benchmark. The benchmark is organized
incrementally from datasets to models, to metrics. We color contents in green
and blue that are widely and partially considered by previous studies,
respectively. Newly introduced contents are colored in pink.

##  3 Datasets

In this paper, we selected three representative datasets: Seven-species, Nine-
species, and HC-PT. These datasets, varying in size, exhibit diversity in
spectrum resolution and peptide sources, enabling a more comprehensive and
accurate evaluation of current _de novo_ peptide sequencing models. The
detailed properties of the datasets are shown in Table 1.

Table 1: The datasets statistics of NovoBench.

Dataset |  _Avg._ | PTM class | min _m/z_ | max _m/z_ | train/valid/test num.
---|---|---|---|---|---
precusor m/z | precusor charge | peaks num. | intensity | peptide len.
Seven-species | 719.07 | 2.42 | 466.05 | 956.17 | 15.79 | 3 | 70.17 | 3997.66 | 317,009 / 17,740 / 17,049
Nine-species | 679.68 | 2.47 | 134.91 | 175082.65 | 15.01 | 3 | 53.03 | 35932.63 | 499,402 / 28,572 / 27,142
HC-PT | 635.32 | 2.31 | 184.21 | 143363.17 | 12.53 | 1 | 99.99 | 1999.99 | 213,284 / 25,718 / 26,536

Seven-species Dataset. The Seven-species dataset contains low-resolution mass
spectrum and their peptide labels from 7 different species. The previous work
DeepNovo tran2017novo  has evaluated its performance on these datasets with
the leave-one-out method, i.e., training the model on 6 species and testing on
the left one species, to mimic the real-world challenging cases where we have
to identify the never-before-seen peptide sequences for the observed mass
spectrum. In this paper, we conducted testing on the yeast species and
training on the remaining 6 species.

Nine-species Dataset. The Nine-species dataset is the most widely-used dataset
by previous works such as DeepNovo tran2017novo , PointNovo
qiao2021computationally , and Casanovo yilmaz2022novo , which contains high-
resolution mass spectrum and their peptide labels from 9 different species. We
adopt the Nine-species dataset used by the original publication of DeepNovo
(MassIVE dataset identifier: MSV000081382) for benchmarking. Similar to Seven-
species dataset, we train models on 8 species and evaluate the left yeast
species. Additionally, these datasets contain 3 PTMs (oxidation of methionine,
deamidation of asparagine or glutamine), enabling the fair evaluation of
various models’ performance in terms of identifying PTMs.

HC-PT Dataset. The HC-PT dataset, as detailed in the InstaNovo
paperEloff2023.08.30.555055 , includes synthetic tryptic peptides that span
all canonical human proteins and isoforms. It also encompasses peptides
generated by alternative proteases and HLA peptides. The key feature of the
HC-PT dataset is its high-resolution spectrum for human-origin peptides, and
the peptide labels are derived from the high-confidence search results of
MaxQuantPMID:27809316 .

##  4 Baseline models

Due to advancements in deep learning, many neural network-based models have
been developed for the _de novo_ peptide sequencing task. These methods can be
broadly classified into two categories: those based on traditional deep
learning techniques such as CNN and LSTM and those based on the Transformer
architecture. We selected six representative models from these two categories
as benchmark models for testing.

Traditional Deep Learning Methods. DeepNovo, as the pioneering model,
integrates deep learning into de novo peptide sequencing. It discretizes the
input mass spectrum into vector representation using predefined bin sizes and
channels them into ion-CNN for effective processing. The resulting output from
ion-CNN undergoes further feature extraction through LSTM, and the combined
outputs from both modules are fused to predict the succeeding amino acid in
the sequence. To ensure precise decoding, the algorithm employs a knapsack
algorithm that constrains the predicted peptide mass and the observed
precursor mass within a specific range. PointNovo enhances the discretization
approach of DeepNovo by utilizing sets of (_m/z_ , intensity) pairs as the
input to the model. It adopts an architecture similar to PointNet for accurate
prediction of the next amino acid.

Transformer Models. CasaNovo employs the Transformer as both the encoder and
decoder. The encoder takes variable-length sets of (_m/z_ , intensity) pairs
as input, while the precursor mass and charge are provided as additional
inputs to the decoder, which iteratively predicts the next amino acid.
InstaNovo uses a similar architecture to CasaNovo but adding a precursor
encoder. The output of the precursor encoder and the spectrum encoder are
concatenated into the decoder for decoding. AdaNovo, a novel framework for
addressing PTM (post-translational modification) identification problems, adds
a peptide decoder to calculate the spectrum-peptide and amino acid mutual
information for adaptive model training. $\pi$-HelixNovo addresses the issue
of missing ions in MS2 by introducing a complementary spectrum as a supplement
to each experimental spectrum.

##  5 Metrics

In this section, we introduce metrics that will be used for _de novo_ peptide
sequencing evaluation, including precision and recall at multiple levels
(peptide level, amino acid level, and amino acid with PTM level), confidence
scores, area under the precision-recall curve and efficiency.

Previous workstran2017novo ; qiao2021computationally ; yilmaz2022novo ;
mao2023mitigating ; xia2024adanovo ; liu2023accurate  mainly focus on
improving amino acid-level or peptide-level precision and recall, while
ignoring other metrics. However, we argue that recovery is not the only
important metric for _de novo_ peptide sequencing evaluation. Other metrics
introduced follows are also crucial for comprehensively revisiting current
approaches. For example, PTMs play an essential role in elucidating protein
functions but have lower accuracy than common amino acids. Therefore, this
paper introduces PTM level metrics.

In addition to metrics related to prediction accuracy, practical applications
of models often place significant emphasis on the model’s efficiency and the
confidence scores of its predictions. This paper also incorporates these
metrics into the evaluation.

Amino acid-level Precision and Recall. We first calculate the number of
matched amino acid predictions, $N_{\text{match}}^{aa}$, which are defined as
predicted amino acids that (1) exhibit a mass difference of $<0.1\mathrm{Da}$
from the corresponding ground truth amino acid and (2) have either a prefix or
suffix with a mass difference of $\leq 0.5\mathrm{Da}$ from the corresponding
amino acid sequence in the ground truth peptide. Amino acid-level precision
and recall is then defined as $N_{\text{match }}^{aa}/N_{\text{pred }}^{aa}$
and $N_{\text{match }}^{aa}/N_{\text{truth}}^{aa}$, where
$N_{\text{pred}}^{aa}$ and $N_{\text{truth}}^{aa}$ represent the number of
predicted amino acids in predicted peptide sequences and ground truth peptide
sequences, respectively.

Peptide-level Precision. Compared with amino acid-level performance, peptide-
level performance measures are the primary quantifier of the model’s practical
utility because the goal of _de novo_ peptide sequencing is to assign a
complete peptide sequence to each spectrum. For peptide prediction, a
predicted peptide is deemed a correct match only if all of its amino acids are
matched according to the definition mentioned above. In a collection of
$N_{\text{all}}^{p}$ spectra, if a model accurately predicts
$N_{\text{match}}^{p}$ peptides, the peptide-level precision are defined as
$N_{\text{match}}^{p}/N_{\text{all}}^{p}$.

PTM-level Precision and Recall. PTMs play an essential role in elucidating
protein functions. However, current models show significantly lower accuracy
for amino acids with PTMs compared to other common amino acids, and this
metric is also less frequently used as an evaluation. Similar to amino acid-
level metrics, PTMs identification precision and recall can be formulated as
$N_{\text{match }}^{ptm}/N_{\text{pred }}^{ptm}$ and $N_{\text{match
}}^{ptm}/N_{\text{orig }}^{ptm}$, where $N_{\text{match }}^{ptm}$,
$N_{\text{pred}}^{ptm}$ and $N_{\text{orig}}^{ptm}$ denote the number of
matched PTMs, predicted amino acids with PTMs and PTMs in ground truth peptide
sequence, respectively.

Confidence. Calculating precision and recall requires access to the reference
sequence, which is not always available in practice. When the ground-truth
sequence is unknown, measuring and ranking the quality of the predicted
sequence becomes more challenging. We introduce the confidence metric to
address this problem, which is the average predictive probability of predicted
amino acids, defined as:

$\texttt{Conf}=\frac{1}{N}\sum_{i=1}^{n}p\left(\hat{y}_{i}\right),$ (1)

We adopt the mean score of all amino acids as a peptide-level confidence
score, where $N$ is the peptide length, and $p\left(\hat{y}_{i}\right)$ is the
predicted probability of the amino acid type $\hat{y}_{i}$ output by the
model.

Area under the precision-recall curve (AUC). Given the availability of peptide
recall, precision, and confidence scores, it is reasonable to draw precision-
recall curves and use the area under the curve (AUC) as a summary of _de novo_
sequencing accuracy. To achieve this, sort the prediction results of each
model from high to low based on confidence scores. Starting from the highest
confidence prediction, accumulate the model’s peptide recall and precision
values. These cumulative values are then used as the x and y coordinates,
respectively, for the points on the precision-recall curve. The AUC of the
precision-recall curve is an effective metric for evaluating classification
models on imbalanced datasets, providing a comprehensive assessment of the
model’s performance under different confidence scores.

Efficiency. Efficiency measures the computational resources and time required
to generate the peptide sequence. This study reports the training time,
inference time, and model parameters of different methods over the standard
benchmarks. While efficiency may not be a crucial problem compared to
precision and recall, it is a useful metric for assessing the model’s
scalability and practicality.

##  6 Influencing Factors

In this section, we investigate the influence of various factors on
performance of _de novo_ peptide sequence models to evaluate the robustness of
all models under varying degrees of missing fragmentation, noise peaks and
peptide lengths.

Peptide Length. We first analyze the impact of peptide length. Peptide length
is defined as the number of amino acids in a peptide chain. Generally, the
longer the peptide length, the more challenging it is for the model to make
accurate predictions.

Noise Peaks. Noise is another crucial factor contributing to the complexity of
resolving spectra. It can originate from various sources, such as white noise
from the instruments, chemical contaminants, and unexpected fragment ions. To
evaluate the impact of noise, we reconstruct the theoretical spectrum
considering several types of ions based on the ground-truth peptide sequence.
Peaks from the experimental spectrum that appear in the theoretical spectrum
within the error tolerance are considered signal peaks, while others are
classified as noise peaks. We then calculate the ratio of noise peaks to
signal peaks in the spectrum, termed the Noise Signal Ratio (NSR), to measure
the noise degree of the spectrum.

Missing Fragmentation Ratio. To further explore the influence of missing
fragmentation, we assess the impact of varying degrees of missing
fragmentation on the performance of all models. Given that recall and
precision calculations are related to peptide length, we define the Missing
Fragmentation Ratio (MFR). MFR quantifies the extent of fragmentation
information loss in the spectrum and is calculated as the number of missing
fragmentations divided by the number of candidate fragmentation sites along
the peptide.

##  7 Results and Analysis

###  7.1 Experimental Settings

The model hyperparameters used in this paper are consistent with those in the
original papers. We set the batch size to 32 and train the models for 30
epochs. DeepNovo and PointNovo were validated every 3000 steps, while the
other models were validated every 50000 steps. The model with the lowest
validation loss was selected for testing. All experimental results were
obtained using a Nvidia A100 GPU (80GB). Please refer to the supplementary
material for the detailed settings of each model.

###  7.2 Main Results

Table 2: Empirical comparison of _de novo_ sequencing models using amino acid-
level and peptide-level metrics. The best and the second best are highlighted
with bold and underline, respectively.

Method | Amino acid-level performance | Peptide-level performance
---|---|---
Seven-species | Nine-species | HC-PT | Seven-species | Nine-species | HC-PT
Prec. | Recall | Prec. | Recall | Prec. | Recall | Prec. | AUC | Prec. | AUC | Prec. | AUC
DeepNovo | 0.492 | 0.433 | 0.696 | 0.638 | 0.531 | 0.534 | 0.204 | 0.136 | 0.428 | 0.376 | 0.313 | 0.255
PointNovo | 0.196 | 0.169 | 0.740 | 0.671 | 0.623 | 0.622 | 0.022 | 0.007 | 0.480 | 0.436 | 0.419 | 0.373
CasaNovo | 0.322 | 0.327 | 0.697 | 0.696 | 0.442 | 0.453 | 0.119 | 0.084 | 0.481 | 0.439 | 0.211 | 0.177
InstaNovo | 0.192 | 0.176 | 0.420 | 0.395 | 0.289 | 0.285 | 0.031 | 0.009 | 0.164 | 0.123 | 0.057 | 0.034
AdaNovo | 0.379 | 0.385 | 0.698 | 0.709 | 0.442 | 0.451 | 0.174 | 0.135 | 0.505 | 0.469 | 0.212 | 0.178
$\pi$-HelixNovo | 0.481 | 0.472 | 0.765 | 0.758 | 0.588 | 0.582 | 0.234 | 0.173 | 0.517 | 0.453 | 0.356 | 0.318

_De novo_ peptide sequencing. As can be observed from Table 2, de novo
sequencing models reveals varied performance at both the amino acid and
peptide levels across different datasets. At the amino acid level,
$\pi$-HelixNovo stands out with the high precision and recall across the Nine-
species and HC-PT datasets, achieving precision and recall values of 0.765 and
0.758, and 0.588 and 0.582, respectively. DeepNovo exhibits the highest
precision (0.492) on the Seven-species dataset but shows less impressive
performance on the other datasets. PointNovo demonstrated the highest
precision and recall (0.623 and 0.622) on the HC-PT dataset. However, compared
to its performance on the high-resolution datasets Nine-species and HC-PT, the
model did not perform as well on the low-resolution Seven-species dataset.
CasaNovo shows consistent but average performance across all datasets, whereas
InstaNovo demonstrates the weakest performance with notably low precision and
recall values in all cases. Notably, we found that InstaNovo have a lower
training loss than CasaNovo, which also uses a Transformer architecture,
across all three datasets. However, its performance on the validation set was
not as good, limiting its generalization capability. AdaNovo performs well on
the Nine-species dataset, with a recall of 0.709, but is otherwise
unremarkable.

At the peptide level, $\pi$-HelixNovo again demonstrates superior performance,
particularly in the Seven-species, with the highest precision and AUC values.
In contrast, DeepNovo and CasaNovo show moderate performance, with DeepNovo
slightly edging out in precision for the Seven-species dataset (0.204).
PointNovo excels on the HC-PT dataset, achieving the highest precision (0.419)
and AUC (0.373) among all models. InstaNovo remains the weakest model at the
peptide level, with significantly lower precision and AUC values compared to
other models. AdaNovo performs best on the nine-species dataset, achieving the
highest AUC (0.469). Regarding confidence score, as shown in Table 4,
$\pi$-HelixNovo has the highest confidence score, while PointNovo has the
lowest. The confidence scores of Transformer-based models are notably higher
compared to traditional methods.

Identifying PTMs. The results in Table 3 show that DeepNovo achieves the
highest PTM recall for the Seven-species dataset (0.373) and performs well on
both the Nine-species and HC-PT datasets. PointNovo excels in the HC-PT
dataset, attaining the highest PTM recall (0.740) and precision (0.676).
CasaNovo demonstrates high PTM precision on the Nine-species dataset (0.706).
AdaNovo performs competitively, achieving the second-highest PTM recall for
the nine-species dataset (0.570) and demonstrating high precision across the
datasets, particularly with a PTM precision of 0.448 for the seven-species
dataset. $\pi$-HelixNovo exhibits consistent performance across all datasets,
securing the top PTM recall for the Nine-species dataset (0.598) and achieving
notable precision scores.

Overall, $\pi$-HelixNovo consistently performs well across all datasets and
metrics, demonstrating the effectiveness of complementary spectrum in _de
novo_ peptide sequencing task. Moreover, comparing results between CasaNovo
and InstaNovo highlights the ineffectiveness of integrating precursor
information into the decoder cross attention. Additionly, traditional deep
learning-based methods remain highly competitive, and designing a better model
architecture is still worth exploring.

Table 3: Empirical comparison of _de novo_ sequencing models in terms of
identifying PTMs. The best results and the second best are highlighted with
bold and underline, respectively.

Method | PTM Recall | PTM Prec.
---|---|---
Seven-species | Nine-species | HC-PT | Seven-species | Nine-species | HC-PT
DeepNovo | 0.373 | 0.529 | 0.615 | 0.391 | 0.576 | 0.626
PointNovo | 0.094 | 0.546 | 0.740 | 0.117 | 0.629 | 0.676
CasaNovo | 0.251 | 0.566 | 0.460 | 0.360 | 0.706 | 0.501
InstaNovo | 0.115 | 0.294 | 0.261 | 0.126 | 0.443 | 0.350
AdaNovo | 0.321 | 0.570 | 0.482 | 0.448 | 0.652 | 0.552
$\pi$-HelixNovo | 0.366 | 0.598 | 0.667 | 0.473 | 0.680 | 0.568

Table 4: Confidence score of different models on various datasets.The best
results and the second best are highlighted with bold and underline,
respectively.

Model | Seven-species | Nine-species | HC-PT | _Avg._
---|---|---|---|---
Deepnovo | 0.515 | 0.697 | 0.503 | 0.572
Pointnovo | 0.264 | 0.735 | 0.620 | 0.540
CasaNovo | 0.760 | 0.905 | 0.751 | 0.805
AdaNovo | 0.798 | 0.914 | 0.761 | 0.824
InstaNovo | 0.665 | 0.782 | 0.652 | 0.700
$\pi$-HelixNovo | 0.846 | 0.935 | 0.843 | 0.875

###  7.3 Influencing Factors

In this section, we evaluate the robustness of all models under varying
degrees of three influencing factors: missing fragmentation, noise, and
peptide lengths.

Figure 2: Peptide-level precision curves of the benchmarking models under the
influence of three factors on Nine-species dataset. The first row to the sixth
row correspond to the models DeepNovo, PointNovo, CasaNovo, InstaNovo,
AdaNovo, $\pi$-HelixNovo, and the first column to the third column correspond
to the three influencing factors Peptide length, missing fragmentation ratio
(MFR) and noise peaks (NSR). More results of the other two datasets can be
found in the appendix.

Peptide Length. We first analyze the impact of peptide length, defined as the
number of amino acids in a peptide. For the nine-species datasets, peptides
lengths are highly variable. Although the mode of peptide length is 12, there
are still many peptides with lengths exceeding 25. Generally, the longer the
peptide length, the more challenging it becomes for the model to accurately
predict, resulting in lower peptide-level accuracy. However, the performance
of models remains relatively stable when the peptide length exceeds a certain
threshold. For instance, when the peptide length exceeds 14, the precision of
five models (excluding Instanovo) fluctuates rather than monotonically
decreasing. Notably, the precision for peptides of length 5 is slightly lower
or comparable to those of lengths 6 and 7 for these five models, which may be
due to the fewer training data for peptides of length 5. In contrast, the
Instanovo model exhibits poor robustness to peptide length, with performance
declining sharply as peptide length increases.

Missing Fragmentation Ratio. The Missing Fragmentation Ratio (MFR) measures
the degree of missing fragmentation by quantifying the portion of
fragmentation information lost in the spectrum. Statistical results show that
the majority (over 50%) of the mass spectra in the nine-species datasets have
an MFR in the range of [0.0, 0.1), suggesting a high level of reliability in
the training data. The results demonstrate a significant decline in peptide
precision as the Missing Fragmentation Ratio (MFR) increases. This trend is
expected, as the recall of amino acids is low in regions with missing
fragmentations, leading to the failure of predicting the entire peptide if any
single amino acid is inaccurately predicted. We can conclude that nearly all
_de novo_ peptide sequencing models are highly affected by the degree of
missing fragmentation.

Noise Peaks. The Noise Signal Ratio (NSR) is calculated as the ratio of noise
peaks to signal peaks in the spectrum to measure the noise profile. From the
NSR analysis, it is evident that in the vast majority of mass spectra, the
number of noise peaks significantly exceeds the number of signal peaks (NSR >
1), indicating the widespread presence of noise information in the spectra.
Generally, the performance is expected to degrade as NSR increases. However,
we observe an unexpected initial increasing trend. This can be attributed to
the data distribution: spectra with smaller NSR tend to have more missing
fragmentations. As NSR increases, the average number of missing fragmentations
decreases to a relatively stable point, leading to an initial improvement in
performance. In summary, as noise increases (indicated by a rising NSR), the
model’s performance initially improves and then declines. However, compared to
peptide length and missing fragmentation ratio, noise peaks have a relatively
smaller impact on model performance.

###  7.4 Computational Efficiency

Table 5: Computational efficiency comparison of various models on the same
device. The training time and inference time here refer to the averaged time
over a batch.

Model | Training Time (s) | Inference Time (s) | Trainable Params (M)
---|---|---|---
Seven-species | Nine-species | HC-PT | Seven-species | Nine-species | HC-PT | All dataset
DeepNovo | 0.31 | 0.38 | 0.30 | 0.04 | 0.07 | 0.02 | 8.63
PointNovo | 0.34 | 0.31 | 0.28 | 0.25 | 0.24 | 0.22 | 4.78
CasaNovo | 0.36 | 0.33 | 0.32 | 0.27 | 0.28 | 0.26 | 47.3
InstaNovo | 0.90 | 0.86 | 0.79 | 0.46 | 0.39 | 0.37 | 92.3
AdaNovo | 1.16 | 1.07 | 0.96 | 1.48 | 1.50 | 1.46 | 66.3
$\pi$-HelixNovo | 0.56 | 0.35 | 0.41 | 0.30 | 0.28 | 0.17 | 47.3

In this section, we compares the efficiency of various _de novo_ peptide
sequencing models in terms of training time, inference time, and the number of
trainable parameters across three datasets. We set the batch size as 32 and
recorded the time required for each model to train and infer a single step
over a batch on different datasets. Additionally, we documented the number of
trainable parameters for each model. From Table 5, we can observe that the
parameter counts for DeepNovo and PointNovo are lower compared to models based
on the Transformer architecture. Despite CasaNovo having nearly ten times the
parameters of PointNovo, its training and inference time are comparable.
InstaNovo significantly outperforms other models in training parameters.
AdaNovo, which includes an additional peptide decoder compared to CasaNovo,
has the highest training time and inference time. $\pi$-HelixNovo modifies the
input spectrum, resulting in similar levels for all three metrics as CasaNovo.

##  8 Conclusion

Although _De novo_ peptide sequencing has achieved remarkable advancement, the
lack of thorough comparisons across diverse datasets, metrics and influencing
factors hinders the progress toward practical applications. To address this
issue, we propose NovoBench, which consists of diverse datasets, models, and
metrics and provides a comprehensive view of deep learning-based _de novo_
peptide sequencing. In the future, We plan to build an automated end-to-end
computational proteomics pipeline that simplifies and standardizes the process
of PSMs data loading, experimental setup, and model evaluation for both _de
novo_ peptide sequencing and theoretical mass spectrum prediction. This will
promote the scientific research and practical application of computational
proteomics.

## References

  * (1) Ruedi Aebersold and Matthias Mann.  Mass spectrometry-based proteomics.  Nature, 422(6928):198–207, 2003.
  * (2) Christian Bartels.  Fast algorithm for peptide sequencing by mass spectroscopy.  Biomedical & environmental mass spectrometry, 19(6):363–368, 1990.
  * (3) Vlado Dančík, Theresa A Addona, Karl R Clauser, James E Vath, and Pavel A Pevzner.  De novo peptide sequencing via tandem mass spectrometry.  Journal of computational biology, 6(3-4):327–342, 1999.
  * (4) Yonathan Lissanu Deribe, Tony Pawson, and Ivan Dikic.  Post-translational modifications in signal integration.  Nature structural & molecular biology, 17(6):666–672, 2010.
  * (5) Ashok R Dongré, Jimmy K Eng, and John R Yates.  Emerging tandem-mass-spectrometry techniques for the rapid identification of proteins.  Trends in biotechnology, 15(10):418–425, 1997.
  * (6) Guangyou Duan and Dirk Walther.  The roles of post-translational modifications in the context of protein interaction networks.  PLoS computational biology, 11(2):e1004049, 2015.
  * (7) Kevin Eloff, Konstantinos Kalogeropoulos, Oliver Morell, Amandla Mabona, Jakob Berg Jespersen, Wesley Williams, Sam van Beljouw, Marcin Skwark, Andreas Hougaard Laustsen, Stan J. J. Brouns, Anne Ljungars, Erwin M. Schoof, Jeroen Van Goey, Ulrich auf dem Keller, Karim Beguir, Nicolas Lopez Carranza, and Timothy P. Jenkins.  De novo peptide sequencing with instanovo: Accurate, database-free peptide identification for large scale proteomics experiments.  bioRxiv, 2023.
  * (8) Bernd Fischer, Volker Roth, Franz Roos, Jonas Grossmann, Sacha Baginsky, Peter Widmayer, Wilhelm Gruissem, and Joachim M Buhmann.  Novohmm: a hidden markov model for de novo peptide sequencing.  Analytical chemistry, 77(22):7265–7273, 2005.
  * (9) Ari Frank and Pavel Pevzner.  Pepnovo: de novo peptide sequencing via probabilistic network modeling.  Analytical chemistry, 77(4):964–973, 2005.
  * (10) Ari M Frank.  Predicting intensity ranks of peptide fragment ions.  Journal of proteome research, 8(5):2226–2240, 2009.
  * (11) Jonas Grossmann, Franz F Roos, Mark Cieliebak, Zsuzsanna Lipták, Lucas K Mathis, Matthias Müller, Wilhelm Gruissem, and Sacha Baginsky.  Audens: a tool for automated peptide de novo sequencing.  Journal of proteome research, 4(5):1768–1774, 2005.
  * (12) Alexander S Hebert, Alicia L Richards, Derek J Bailey, Arne Ulbrich, Emma E Coughlin, Michael S Westphall, and Joshua J Coon.  The one hour yeast proteome.  Molecular & Cellular Proteomics, 13(1):339–347, 2014.
  * (13) Richard S Johnson and Klaus Biemann.  The primary structure of thioredoxin from chromatium vinosum determined by high-performance tandem mass spectrometry.  Biochemistry, 26(5):1209–1214, 1987.
  * (14) Korrawe Karunratanakul, Hsin-Yao Tang, David W Speicher, Ekapol Chuangsuwanich, and Sira Sriswasdi.  Uncovering thousands of new peptides with sequence-mask-search hybrid de novo peptide sequencing framework.  Molecular & Cellular Proteomics, 18(12):2478–2491, 2019.
  * (15) Guolin Ke, Di He, and Tie-Yan Liu.  Rethinking positional encoding in language pre-training.  In International Conference on Learning Representations, 2021.
  * (16) Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  arXiv preprint arXiv:1412.6980, 2014.
  * (17) Céline M Laumont, Krystel Vincent, Leslie Hesnard, Éric Audemard, Éric Bonneil, Jean-Philippe Laverdure, Patrick Gendron, Mathieu Courcelles, Marie-Pierre Hardy, Caroline Côté, et al.  Noncoding regions are the main source of targetable tumor-specific antigens.  Science translational medicine, 10(470):eaau5516, 2018.
  * (18) Eugene Lin, Chieh-Hsin Lin, and Hsien-Yuan Lane.  Relevant applications of generative adversarial networks in drug design and discovery: molecular de novo design, dimensionality reduction, and de novo peptide and protein design.  Molecules, 25(14):3250, 2020.
  * (19) Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár.  Focal loss for dense object detection.  In Proceedings of the IEEE international conference on computer vision, pages 2980–2988, 2017.
  * (20) Fan Liu, Dirk TS Rijkers, Harm Post, and Albert JR Heck.  Proteome-wide profiling of protein assemblies by cross-linking mass spectrometry.  Nature methods, 12(12):1179–1184, 2015.
  * (21) Kaiyuan Liu, Yuzhen Ye, Sujun Li, and Haixu Tang.  Accurate de novo peptide sequencing using fully convolutional neural networks.  Nature Communications, 14(1):7974, 2023.
  * (22) Bin Ma.  Novor: real-time peptide de novo sequencing software.  Journal of the American Society for Mass Spectrometry, 26(11):1885–1894, 2015.
  * (23) Bin Ma, Kaizhong Zhang, Christopher Hendrie, Chengzhi Liang, Ming Li, Amanda Doherty-Kirby, and Gilles Lajoie.  Peaks: powerful software for peptide de novo sequencing by tandem mass spectrometry.  Rapid communications in mass spectrometry, 17(20):2337–2342, 2003.
  * (24) Matthias Mann and Ole N Jensen.  Proteomic analysis of post-translational modifications.  Nature biotechnology, 21(3):255–261, 2003.
  * (25) Zeping Mao, Ruixue Zhang, Lei Xin, and Ming Li.  Mitigating the missing-fragmentation problem in de novo peptide sequencing with a two-stage graph-based deep learning model.  Nature Machine Intelligence, 5(11):1250–1260, 2023.
  * (26) Rupert L Mayer and Francis Impens.  Immunopeptidomics for next-generation bacterial vaccine development.  Trends in microbiology, 29(11):1034–1045, 2021.
  * (27) Kevin McDonnell, Enda Howley, and Florence Abram.  The impact of noise and missing fragmentation cleavages on de novo peptide identification algorithms.  Computational and Structural Biotechnology Journal, 20:1402–1412, 2022.
  * (28) Thilo Muth, Dirk Benndorf, Udo Reichl, Erdmann Rapp, and Lennart Martens.  Searching for a needle in a stack of needles: challenges in metaproteomics data analysis.  Molecular BioSystems, 9(4):578–585, 2013.
  * (29) Alexey I Nesvizhskii.  A survey of computational methods and error rate estimation procedures for peptide and protein identification in shotgun proteomics.  Journal of proteomics, 73(11):2092–2123, 2010.
  * (30) Vikas Pejaver, Wei-Lun Hsu, Fuxiao Xin, A Keith Dunker, Vladimir N Uversky, and Predrag Radivojac.  The structural and functional signatures of proteins that undergo multiple events of post-translational modification.  Protein Science, 23(8):1077–1093, 2014.
  * (31) Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas.  Pointnet: Deep learning on point sets for 3d classification and segmentation.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 652–660, 2017.
  * (32) Rui Qiao, Ngoc Hieu Tran, Lei Xin, Xin Chen, Ming Li, Baozhen Shan, and Ali Ghodsi.  Computationally instrument-resolution-independent de novo peptide sequencing for high-resolution devices.  Nature Machine Intelligence, 3(5):420–425, 2021.
  * (33) Shahin Ramazi and Javad Zahiri.  Post-translational modifications in proteins: resources, tools and prediction methods.  Database, 2021:baab012, 2021.
  * (34) Mengye Ren, Wenyuan Zeng, Bin Yang, and Raquel Urtasun.  Learning to reweight examples for robust deep learning.  In International conference on machine learning, pages 4334–4343. PMLR, 2018.
  * (35) CARNEGIE-MELLON UNIV PITTSBURGH PA DEPT OF COMPUTER SCIENCE.  Speech Understanding Systems. Summary of Results of the Five-Year Research Effort at Carnegie-Mellon University.  1977\.
  * (36) Matteo Stefanini, Marcella Cornia, Lorenzo Baraldi, Silvia Cascianelli, Giuseppe Fiameni, and Rita Cucchiara.  From show to tell: A survey on deep learning-based image captioning.  IEEE transactions on pattern analysis and machine intelligence, 45(1):539–559, 2022.
  * (37) J Alex Taylor and Richard S Johnson.  Implementation and uses of automated de novo peptide sequencing by tandem mass spectrometry.  Analytical chemistry, 73(11):2594–2604, 2001.
  * (38) Ngoc Hieu Tran, Rui Qiao, Lei Xin, Xin Chen, Baozhen Shan, and Ming Li.  Personalized deep learning of individual immunopeptidomes to identify neoantigens for cancer vaccines.  Nature Machine Intelligence, 2(12):764–771, 2020.
  * (39) Ngoc Hieu Tran, M Ziaur Rahman, Lin He, Lei Xin, Baozhen Shan, and Ming Li.  Complete de novo assembly of monoclonal antibody sequences.  Scientific reports, 6(1):31730, 2016.
  * (40) Ngoc Hieu Tran, Xianglilan Zhang, Lei Xin, Baozhen Shan, and Ming Li.  De novo peptide sequencing by deep learning.  Proceedings of the National Academy of Sciences, 114(31):8247–8252, 2017.
  * (41) Stefka Tyanova, Tikira Temu, and Juergen Cox.  The maxquant computational platform for mass spectrometry-based shotgun proteomics.  Nature protocols, 11(12):2301—2319, December 2016.
  * (42) Anuli Christiana Uzozie and Ruedi Aebersold.  Advancing translational research and precision medicine with targeted proteomics.  Journal of proteomics, 189:1–10, 2018.
  * (43) Martijn M VanDuijn, Lennard J Dekker, Wilfred FJ van IJcken, Peter AE Sillevis Smitt, and Theo M Luider.  Immune repertoire after immunization as seen by next-generation sequencing and proteomics.  Frontiers in Immunology, 8:1286, 2017.
  * (44) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  Advances in neural information processing systems, 30, 2017.
  * (45) Rui Vitorino, Sofia Guedes, Fabio Trindade, Inês Correia, Gabriela Moura, Paulo Carvalho, Manuel AS Santos, and Francisco Amado.  De novo sequencing of proteins by mass spectrometry.  Expert Review of Proteomics, 17(7-8):595–607, 2020.
  * (46) Juan A Vizcaíno, Eric W Deutsch, Rui Wang, Attila Csordas, Florian Reisinger, Daniel Ríos, José A Dianes, Zhi Sun, Terry Farrah, Nuno Bandeira, et al.  Proteomexchange provides globally coordinated proteomics data submission and dissemination.  Nature biotechnology, 32(3):223–226, 2014.
  * (47) Dirk A Wolters, Michael P Washburn, and John R Yates.  An automated multidimensional protein identification technology for shotgun proteomics.  Analytical chemistry, 73(23):5683–5690, 2001.
  * (48) Aaron D Wyner.  A definition of conditional mutual information for arbitrary ensembles.  Information and Control, 38(1):51–59, 1978.
  * (49) Jun Xia, Shaorong Chen, Jingbo Zhou, Tianze Lin, Wenjie Du, Sizhe Liu, and Stan Z Li.  Adanovo: Adaptive$\backslash$emph $\\{$De Novo$\\}$ peptide sequencing with conditional mutual information.  arXiv preprint arXiv:2403.07013, 2024.
  * (50) Hao Yang, Hao Chi, Wen-Feng Zeng, Wen-Jing Zhou, and Si-Min He.  pnovo 3: precise de novo peptide sequencing using a learning-to-rank framework.  Bioinformatics, 35(14):i183–i190, 2019.
  * (51) Tingpeng Yang, Tianze Ling, Boyan Sun, Zhendong Liang, Fan Xu, Xiansong Huang, Linhai Xie, Yonghong He, Leyuan Li, Fuchu He, Yu Wang, and Cheng Chang.  Introducing pi-HelixNovo for practical large-scale de novo peptide sequencing.  Briefings in Bioinformatics, 25(2):bbae021, 02 2024.
  * (52) Melih Yilmaz, William Fondrie, Wout Bittremieux, Sewoong Oh, and William S Noble.  De novo mass spectrometry peptide sequencing with a transformer model.  In International Conference on Machine Learning, pages 25514–25522. PMLR, 2022.
  * (53) Yifan Zhang, Bingyi Kang, Bryan Hooi, Shuicheng Yan, and Jiashi Feng.  Deep long-tailed learning: A survey.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.

## APPENDIX

### A. Influencing Factors

Figure 3: Peptide-level precision curves of the benchmarking models under the
influence of three factors on Seven-species dataset. The first row to the
sixth row correspond to the models DeepNovo, PointNovo, CasaNovo, InstaNovo,
AdaNovo, $\pi$-HelixNovo, and the first column to the third column correspond
to the three influencing factors Peptide length, missing fragmentation ratio
(MFR) and noise peaks (NSR).

Figure 4: Peptide-level precision curves of the benchmarking models under the
influence of three factors on HC-PT dataset. The first row to the sixth row
correspond to the models DeepNovo, PointNovo, CasaNovo, InstaNovo, AdaNovo,
$\pi$-HelixNovo, and the first column to the third column correspond to the
three influencing factors Peptide length, missing fragmentation ratio (MFR)
and noise peaks (NSR).


$\displaystyle\lim_{\omega}\frac{1}{n}d_{n}(p_{n,\lambda_{n}}(x_{n}),p_{n,\tau_{n}}(x_{n}))\leq\lim_{\omega}\frac{K}{n}|\lambda_{n}-\tau_{n}|=K\left|\lim_{\omega}\frac{\lambda_{n}}{n}-\lim_{\omega}\frac{\tau_{n}}{n}\right|=K|t-t|=0,$

which shows independence of the particular choice of sequence $(\lambda_{n})$.
Thus, $H$ is well-defined.

######  Claim.

$H$ is continuous.

By Lemma 4.1 $H$ is continuous if it is continuous when restricted to each
factor and $H$ is globally Lipschitz continuous with respect to one of the
factors. Let us start with the time parameter. Denote the induced metric on
$\operatorname{Cone}_{\omega}(X_{n},d_{n})$ simply by $d$. Fix $x=[x_{n}]$. By
property $(i)$ we calculate for $\lambda_{n}$ and $\tau_{n}$ such that
$\lim_{\omega}\frac{\lambda_{n}}{n}=t$ and
$\lim_{\omega}\frac{\tau_{n}}{n}=t^{\prime}$ that

$\displaystyle
d(H(x,t),H(x,t^{\prime}))=\lim_{\omega}\frac{d_{n}(p_{n,\lambda_{n}}(x_{n}),p_{n,\tau_{n}}(x_{n}))}{n}\leq\lim_{\omega}\frac{K|\lambda_{n}-\tau_{n}|}{n}=K|t-t^{\prime}|.$

For Lipschitz continuity with respect to a fixed time $t$ we calculate using
$(ii)$

$\displaystyle
d(H(x,t),H(y,t))=\lim_{\omega}\frac{d_{n}(p_{n,\lambda_{n}}(x_{n}),p_{n,\lambda_{n}}(y_{n}))}{n}\leq\lim_{\omega}\frac{d_{n}(x_{n},y_{n})}{n}=d(x,y)$

This establishes continuity of $H$.

Finally, $H(-,0)=id$ letting $\lambda_{n}=0$ for all $n$. Respectively,
letting $\lambda_{n}=n$ for all $n$. $H(-,1)\equiv x_{0}$, the unique point
$x_{0}\in X_{0}$. Therefore, $H$ is the desired contracting homotopy.

∎

###  6.2 Upper triangular matrices

######  Definition 6.3.

Let $R$ be a commutative ring. Define the rank norm of an element $g$ in the
infinite general linear group
$\operatorname{GL}_{\infty}(R)=\operatorname{colim}_{n}\operatorname{GL}_{n}(R)$
to be

$\|g\|_{rk}=\operatorname{rk}(g-id).$

Moreover, for $g\in\operatorname{GL}_{n}(R)$ we define the projectivised rank
norm to be
$\|g\|_{\overline{\operatorname{rk}}}=\frac{\|g\|_{\operatorname{rk}}}{n}$.

Observe, that the (projectivised) rank norms are conjugation invariant and
that by definition
$\|g\|_{\operatorname{rk}}=\dim(\operatorname{im}(g-id))=n-\dim\operatorname{Fix}(g)$
where $\operatorname{Fix}(g)$ denotes the subspace of fixed points of $g$.

######  Proposition 6.4.

Let $\operatorname{B}_{n}(R)$ be the group of invertible upper triangular
matrices with entries in a commutative ring $R$. Then the metric ultraproduct
$\prod_{n\to\omega}(B_{n},\|.\|_{\overline{\operatorname{rk}}})$ is
contractible.

###### Proof.

Clearly, the diameter of $B_{n}$ with the projectivised rank norm is bounded
by 1. The statement is equivalent to showing that the cone
$\operatorname{Cone}_{\omega}(\operatorname{B}_{n}(R),\|.\|_{\operatorname{rk}})$
for the standard scaling sequence $s_{n}=n$ is contractible. For
$n\in\mathbb{N}$ define the projection maps
$p_{n}\colon\operatorname{B}_{n}(R)\rightarrow\operatorname{B}_{n-1}(R)$ to be
the projections onto the upper left block with the convention that
$\operatorname{B}_{0}(R)$ is the trivial group. These maps satisfy

  1. (i)

$p_{n}$ is a group homomorphism for all $n$,

  2. (ii)

$\|p_{n}(g)g^{-1}\|_{\operatorname{rk}}\leq 1$ for all $n$,

  3. (iii)

$\|p_{n}(g)p_{n}(h)^{-1}\|_{\operatorname{rk}}\leq\|gh^{-1}\|_{\operatorname{rk}}$
for all $g,h$ and all $n$.

Indeed, it is clear that $p_{n}$ is a group homomorphism and that
$\|p_{n}(g)g^{-1}\|_{|rk}\leq 1$ since $p_{n}(g)g^{-1}$ differs from the
identity in at most one column. Moreover, for $g\in\operatorname{B}_{n}$ it
holds that

$\|p_{n}(g)\|_{\operatorname{rk}}=\operatorname{rk}(p_{n}(g)-id)\leq\operatorname{rk}(g-id)=\|g\|_{\operatorname{rk}},$

Then it follows for $h,g\in\operatorname{B}_{n}$ using the fact that $p_{n}$
is a group homomorphism that

$\|p_{n}(g)p_{n}(h)^{-1}\|_{\operatorname{rk}}=\|p_{n}(gh^{-1})\|_{\operatorname{rk}}\leq\|gh^{-1}\|_{\operatorname{rk}}.$

Consequently, the contractibility of
$\operatorname{Cone}_{\omega}(B_{n}(R),\|.\|_{\operatorname{rk}})$ follows
from Lemma 6.2. ∎

###  6.3 Symmetric positive definite matrices

Let $\operatorname{P}_{n}$ be the set of real symmetric positive definite
matrices equipped with the rank norm $\|.\|_{\operatorname{rk}}$ as a subspace
$\operatorname{P}_{n}\subset\operatorname{GL}_{n}(\mathbb{R})$. For $n\in N$
define the projection map

$\displaystyle
p_{n}\colon\operatorname{P}_{n}\rightarrow\operatorname{P}_{n-1},\hskip
14.22636ptA\rightarrow A^{(n-1)}$

by forgetting the last row and column. The projection maps $p_{n}$ are well-
defined since $A^{(n-1)}$ is clearly symmetric and by the criterion of
positive principle minors $A^{(n-1)}$ is still positive definite hence
invertible.

######  Proposition 6.5.

The metric ultraproduct
$\prod_{n\to\omega}(\operatorname{P}_{n},\|.\|_{\overline{\operatorname{rk}}})$
is contractible.

###### Proof.

Again, we will verify the conditions of Lemma 6.2 to conclude the
contractibility of
$\operatorname{Cone}_{\omega}(\operatorname{P}_{n},\|.\|_{\operatorname{rk}})=\prod_{n\to\omega}(\operatorname{P}_{n},\|.\|_{\overline{\operatorname{rk}}})$.
Let $A\in\operatorname{P}_{n}$. Then

$\|p_{n}(A)A^{-1}\|_{\operatorname{rk}}=\|A^{(n-1)}A^{-1}\|_{\operatorname{rk}}=\operatorname{rk}(A^{(n-1)}-A)\leq
2$

since the only possible non-zero entries of $A^{(n-1)}-A$ are in the $n$-th
row and column. Furthermore, $A,B\in\operatorname{P}_{n}$ satisfy

$\displaystyle\|p_{n}(A)p_{n}(B)^{-1}\|_{\operatorname{rk}}=\operatorname{rk}(A^{(n-1)}-B^{(n-1)})\leq\operatorname{rk}(A-B)=\|AB^{-1}\|_{\operatorname{rk}}$

since any collection of linearly independent vectors of $A^{(n-1)}-B^{(n-1)}$
stays linearly independent in $A-B$. Thus, Lemma 6.2 applies and it follows
that
$\operatorname{Cone}_{\omega}(\operatorname{P}_{n},\|.\|_{\operatorname{rk}})$
is contractible. ∎

###  6.4 Special orthogonal group

Call an element $g\in\operatorname{SO}(n)$ that belongs to a subgroup
conjugate to $\operatorname{SO}(2)\leq SO(n)$ an elementary rotation. That is,
elementary rotations in $\operatorname{SO}(n)$ are precisely the elements that
fix a subspace of codimension two.

######  Lemma 6.6.

The inclusions
$\iota_{n}\colon\operatorname{SO}(n)\rightarrow\operatorname{O}(n)$ are all
isometries which are quasi-surjective with constant 1 with respect to the rank
norm.

###### Proof.

Since all spaces are equipped with the same subspace norm of
$\operatorname{GL}_{n}(\mathbb{R})$ all inclusions are isometries by
definition. For any $g\in\operatorname{O}(n)$ we have $|\det(g)|=1$ and so the
matrix $a=(\det(g))$ lies in $\operatorname{O}(1)$. Hence, $h=ag$ satisfies
$\|hg^{-1}\|_{\operatorname{rk}}\leq 1$. ∎

######  Corollary 6.7.

$\operatorname{Cone}_{\omega}(\operatorname{SO}(n),\|.\|_{\operatorname{rk}})=\operatorname{Cone}_{\omega}(\operatorname{O}(n),\|.\|_{\operatorname{rk}})$.
That is, the metric ultraproducts of the special orthogonal groups and the
orthogonal groups with the projectivesed rank metric are equal.

######  Lemma 6.8.

There is the following identification of the rank norm using the standard
maximal tori of $\operatorname{SO}(n)$. For $g\in\operatorname{SO}(n)$ the
norm $\|g\|_{\operatorname{rk}}$ is twice the number of non-trivial rotations
of a representative $hgh^{-1}$ in the maximal torus of $\operatorname{SO}(n)$.
In particular, $\|g\|_{\operatorname{rk}}$ is always even.

###### Proof.

We have

$\displaystyle hgh^{-1}=\begin{pmatrix}R_{1}&&\\\ &\ddots&\\\ &&R_{n}\\\
\end{pmatrix},$

where all $R_{i}$ are rotations belonging to $\operatorname{SO}(2)$. An
element in $\operatorname{SO}(2)$ has a fixed point if and only if it is the
identity. Thus, the largest subspace in $\mathbb{R}^{n}$ without any fixed
point of $g$ has dimension twice the number of non-trivial rotations.
Consequently, $\|g\|_{\operatorname{rk}}$ is twice the number of non-trivial
rotations $R_{i}$. ∎

Recall, that the action of $\operatorname{SO}(n)$ on the unit sphere
$\mathbb{S}^{n-1}$ gives rise to a fibration with fibre
$\operatorname{SO}(n-1)$. In this spirit we want to define projection maps
$\operatorname{SO}(n)\rightarrow\operatorname{SO}(n-1)$ that rotate vectors
back into this fibre and do not increase the norm $\|.\|_{\operatorname{rk}}$.

######  Lemma 6.9.

Let $g,h\in\operatorname{SO}(n)$ such that $g(e_{n})\neq h(e_{n})$. Let
$R_{\alpha}$ and $R_{\beta}$ be elementary rotations such that
$R_{\alpha}g(e_{n})=e_{n}=R_{\beta}h(e_{n})$ and such that $R_{\alpha}$ (and
$R_{\beta}$ respectively) is the identity if $g(e_{n})=e_{n}$ (or $h(e_{n})$
respectively). Then
$\|R_{\alpha}g(R_{\beta}h)^{-1}\|_{\operatorname{rk}}\leq\|gh^{-1}\|_{\operatorname{rk}}$.

###### Proof.

By conjugation invariance
$\|R_{\alpha}g(R_{\beta}h)^{-1}\|_{\operatorname{rk}}=\|gh^{-1}R_{\beta}^{-1}R_{\alpha}\|_{\operatorname{rk}}$.
By our assumptions the rotations $R_{\alpha}$ and $R_{\beta}^{-1}$ either both
act non-trivially on $e_{n}$ or one of them is the identity. In both cases
their product is again an elementary rotation since every element of
$\operatorname{SO}(3)$ is an elementary rotation. Consequently,
$\|R_{\beta}^{-1}R_{\alpha}\|_{\operatorname{rk}}\leq 2$. Since
$R_{\alpha}g(e_{n})=e_{n}=R_{\beta}h(e_{n})$, it holds that
$g(e_{n})\in\operatorname{Fix}(gh^{-1}R_{\beta}^{-1}R_{\alpha})$, but
$R_{\beta}^{-1}R_{\alpha}(g(e_{n}))=h(e_{n})\neq g(e_{n})$. Let
$k\in\mathbb{N}$ be the norm $k=\|gh^{-1}\|_{\operatorname{rk}}$. Then
$\dim(\operatorname{Fix}(gh^{-1}))=n-k$ and thus

$\displaystyle\dim\Big{(}\operatorname{Fix}(gh^{-1}R_{\beta}^{-1}R_{\alpha})\Big{)}$
$\displaystyle\geq\dim\Big{(}\big{(}\operatorname{Fix}(gh^{-1})\cap\operatorname{Fix}(R_{\beta}^{-1}R_{\alpha})\big{)}\oplus\langle
g(e_{n})\rangle\Big{)}$ $\displaystyle=$
$\displaystyle\dim\Big{(}\operatorname{Fix}(gh^{-1})\cap\operatorname{Fix}(R_{\beta}^{-1}R_{\alpha})\Big{)}+1$
$\displaystyle=$
$\displaystyle\dim(\operatorname{Fix}(gh^{-1}))+\dim(\operatorname{Fix}(R_{\beta}^{-1}R_{\alpha}))-\dim(\operatorname{Fix}(gh^{-1})$
$\displaystyle+\operatorname{Fix}(R_{\beta}^{-1}R_{\alpha}))+1$
$\displaystyle\geq$ $\displaystyle n-k+n-2-n+1$ $\displaystyle=$
$\displaystyle n-(k+1).$

Therefore,
$\|R_{\alpha}g(R_{\beta}h)^{-1}\|_{\operatorname{rk}}\leq\|gh^{-1}\|_{\operatorname{rk}}+1$.
However, by Lemma 6.8 the rank norms
$\|R_{\alpha}g(R_{\beta}h)^{-1}\|_{\operatorname{rk}}$ and
$\|gh^{-1}\|_{\operatorname{rk}}$ are both even, so
$\|gh^{-1}\|_{\operatorname{rk}}+1$ is odd. Consequently, we must have
$\|R_{\alpha}g(R_{\beta}h)^{-1}\|_{\operatorname{rk}}\leq\|gh^{-1}\|_{\operatorname{rk}}$.
∎

With these preliminary considerations at hand we will define our desired
projection maps
$p_{n}\colon\operatorname{SO}(n)\rightarrow\operatorname{SO}(n-1)$ as follows.
For every $x\in\mathbb{S}^{n-1}$ we fix an elementary rotation
$R_{x}\in\operatorname{SO}(n)$ which rotates $x$ to $e_{n}$ and set
$R_{e_{n}}=id$. Note, that this choice is unique when $x\neq\pm e_{n}$ since
$x$ and $e_{n}$ span a 2 dimensional subspace in that case. Define

$\displaystyle
p_{n}\colon\operatorname{SO}(n)\rightarrow\operatorname{SO}(n-1)\hskip
5.69054pt\text{by}\hskip 5.69054ptp_{n}(g)=R_{g(e_{n})}g.$

Note, that $R_{g(e_{n})}g$ is indeed an element of $\operatorname{SO}(n-1)$
since it belongs to $SO(n)$ and satisfies $R_{g(e_{n})}g(e_{n})=e_{n}$. Thus,
it follows by orthogonality that the last entry in all other columns of
$R_{g(e_{n})}g$ vanishes and it lies in the image of $\operatorname{SO}(n-1)$.

######  Proposition 6.10.

The metric ultraproduct
$\prod_{n\to\omega}(\operatorname{SO}(n),\|.\|_{\overline{\operatorname{rk}}})$
of the special orthogonal groups with the projectivised rank metric is
contractible.

###### Proof.

We will verify the conditions of Lemma 6.2 for the cone with respect to the
standard scaling sequence
$\operatorname{Cone}_{\omega}(\operatorname{SO}(n),\|.\|_{\operatorname{rk}})=\prod_{n\to\omega}(SO(n),\|.\|_{\overline{\operatorname{rk}}})$
to obtain a contracting homotopy.

Clearly, $\|p_{n}(g)g^{-1}\|_{\operatorname{rk}}\leq 2$ for all
$n\in\mathbb{N}$ and $g\in\operatorname{SO}(n)$. Additionally, by Lemma 6.9
all $g,h\in\operatorname{SO}(n)$ satisfy
$\|p_{n}(g)p_{n}(h)^{-1}\|_{\operatorname{rk}}\leq\|gh^{-1}\|_{\operatorname{rk}}$
if $g(e_{n})\neq h(e_{n})$. However, if $g(e_{n})=h(e_{n})$ then
$R_{g(e_{n})}=R_{h(e_{n})}$ and so

$\displaystyle\|p_{n}(g)p_{n}(h)^{-1}\|_{\operatorname{rk}}=\|R_{g(e_{n})}gh^{-1}R_{h(e_{n})}^{-1}\|_{\operatorname{rk}}=\|gh^{-1}\|_{\operatorname{rk}}.$

Therefore, the result follows from Lemma 6.2. ∎

######  Remark 6.11.

The analogeous statement for unitary groups does not follow since the rank
norm on $\operatorname{SU}(n)$ can assume odd values.

##  7 Appendix: Non-contractible metric ultraproducts and asymptotic cones

The purpose of this appendix is to give some immediate examples for groups
that have non-contractible asymptotic cones stemming directly from the metric
ultraproduct of cyclic groups of order $n$. More background and much more
involved theory focussing mainly on the case of finitely generated groups can
be found in [Bur99], [Ril03] and [EO05].

######  Lemma 7.1.

Let $G_{n}=\mathbb{Z}/n$ be equipped with the word norm $\|.\|_{n}$ generated
by $1\in\mathbb{Z}/n$ for all $n$ and let $\|.\|_{\overline{n}}$ be the
projectivisation. Then $\prod_{n\to\omega}(\mathbb{Z}/n,\|.\|_{\overline{n}})$
is Lipschitz isomorphic to the unit circle $S^{1}$ with its standard metric.

###### Proof.

Recall, that the standard subspace metric on $S^{1}$ is Lipschitz equivalent
to the metric given by the arclength $d_{\operatorname{arc}}$. For each $n$
consider the homomorphism

$\varphi_{n}\colon\mathbb{Z}/n\rightarrow S^{1}\hskip 14.22636pt,\hskip
14.22636ptk\rightarrow e^{2\pi i\frac{k}{n}}$

with image equal to the $n$-th roots of unity. Then $a,b\in\mathbb{Z}/n$ with
$\|ab^{-1}\|_{n}=k$ satisfy
$d_{\operatorname{arc}}(\varphi_{n}(a),\varphi_{n}(b))=2\pi\frac{k}{n}$.
Hence, the sequence $(\varphi_{n})$ gives rise to a Lipschitz continuous
homomorphism
$\varphi\colon\operatorname{Cone}_{\omega}(\mathbb{Z}/n,\|.\|_{n})\rightarrow
S^{1}$ setting $\varphi(x)=\lim_{\omega}\varphi_{n}(x_{n})$ for $x=[x_{n}]$.

Conversely, define $\theta_{n}\colon S^{1}\rightarrow\mathbb{Z}/n$ by
$\theta_{n}(x)=k$ where $k$ is a fixed choice for each $x$ such such that
$d_{\operatorname{arc}}(x,e^{2\pi i\frac{k}{n}})$ is minimal. These (non-
continuous) maps satisfy

$\|\theta_{n}(x)\theta_{n}(y)^{-1}\|_{n}\leq d_{\operatorname{arc}}(x,y)n+2.$

Consequently, setting $\theta(x)=[\theta_{n}(x)]$ yields a map $\theta\colon
S^{1}\rightarrow\operatorname{Cone}_{\omega}(\mathbb{Z}/n,\|.\|_{n})$ that is
Lipschitz continous. Since for each $n$ the choice of $k$-th root of unity in
the definition of $\theta_{n}(x)$ is distance minimising, this sequence
converges again to $x\in S^{1}$ and so $(\varphi\circ\theta)=id$.

For $[x_{n}]\in\operatorname{Cone}_{\omega}(\mathbb{Z}/n,\|.\|_{n})$ let
$x^{\prime}=\lim_{\omega}\varphi_{n}(x_{n})$ and let
$y_{n}=\theta_{n}(x^{\prime})$ for all $n$. We claim that $[y_{n}]=[x_{n}]$.
Let $\epsilon>0$ be given. Then the following sequence of implications holds
where the first line follows from the definition of the ultralimit
$x^{\prime}=\lim_{\omega}\varphi_{n}(x_{n})$.

$\displaystyle\left\\{n\in\mathbb{N}\hskip 5.69054pt:\hskip
5.69054ptd_{\operatorname{arc}}(e^{2\pi
i\frac{x_{n}}{n}},x^{\prime})<\epsilon\right\\}\in\omega$
$\displaystyle\implies$ $\displaystyle\left\\{n\in\mathbb{N}\hskip
5.69054pt:\hskip 5.69054pt\|\theta_{n}(e^{2\pi
i\frac{x_{n}}{n}})\theta_{n}(x^{\prime})^{-1}\|_{n}<\epsilon
n+2\right\\}\in\omega$ $\displaystyle\implies$
$\displaystyle\left\\{n\in\mathbb{N}\hskip 5.69054pt:\hskip
5.69054pt\frac{\|x_{n}y_{n}^{-1}\|_{n}}{n}<\frac{\epsilon
n+2}{n}\right\\}\in\omega$ $\displaystyle\implies$
$\displaystyle\left\\{n\in\mathbb{N}\hskip 5.69054pt:\hskip
5.69054pt\frac{\|x_{n}y_{n}^{-1}\|_{n}}{n}<2\epsilon\right\\}\in\omega.$

Therefore, $[x_{n}]=[y_{n}]$ and so $(\theta\circ\varphi)=id$. Thus, $\varphi$
and $\theta$ are Lipschitz isomorphisms and
$\prod_{n\to\omega}(\mathbb{Z}/n,\|.\|_{\overline{n}})$ is Lipschitz
isomorphic to the unit circle $S^{1}$ with its standard metric. ∎

######  Proposition 7.2.

Let $G=\bigoplus_{n}\mathbb{Z}/n$ be equipped with the word norm $\|.\|$
generated by the unit vectors $\\{1_{n}\in\mathbb{Z}/n\\}_{n}$. Then
$\operatorname{Cone}_{\omega}(G,\|.\|)$ is Lipschitz isomorphic to
$S^{1}\times\operatorname{Cone}_{\omega}(X_{n},\|.\|_{n})$ for suitable
sequence $(X_{n},\|.\|_{n})$.

###### Proof.

Let $p_{n}\colon G\rightarrow\mathbb{Z}/n$ be the projection on the
$\mathbb{Z}/n$ factor and let $\iota_{n}\colon\mathbb{Z}/n\rightarrow G$ be
the inclusion. Then for all $n$ the inclusion $\iota_{n}$ is an isometric
embedding, $p_{n}$ is Lipschitz continuous with constant 1 and
$p_{n}\circ\iota_{n}=id_{\mathbb{Z}_{n}}$. On the level of asymptotic cones
this induces the maps
$p\colon\operatorname{Cone}_{\omega}(G,\|.\|)\rightarrow\operatorname{Cone}_{\omega}(\mathbb{Z}/n,\|.\|_{n})$
and
$\iota\colon\operatorname{Cone}_{\omega}(\mathbb{Z}/n,\|.\|_{n})\rightarrow\operatorname{Cone}_{\omega}(G,\|.\|)$.
Thus, the exact sequence of abelian groups

${0}$${\ker(p)}$${\operatorname{Cone}_{\omega}(G,\|.\|)}$${\operatorname{Cone}_{\omega}(\mathbb{Z}/n,\|.\|_{n})}$${0.}$$\scriptstyle{p}$$\scriptstyle{\iota}$

splits as a product of groups. The maps

$\displaystyle\varphi\colon\operatorname{Cone}_{\omega}(G,\|.\|)\rightarrow\operatorname{Cone}_{\omega}(\mathbb{Z}/n,\|.\|_{n})\times\ker(p),$
$\displaystyle g\rightarrow\left(p(g),g-p(g)\right),$
$\displaystyle\psi\colon\operatorname{Cone}_{\omega}(\mathbb{Z}/n,\|.\|_{n})\times\ker(p)\rightarrow\operatorname{Cone}_{\omega}(G,\|.\|),$
$\displaystyle(a,b)\rightarrow\iota(a)+b.$

display the product structure on the level of groups. However, all these
involved maps are identities or Lipschitz continuous and so $\varphi$ and
$\psi$ are Lipschitz continuous as well. The result follows from Lemma 7.1.

The kernel $\ker(p)$ is given precisely by equivalence classes having a
representative sequence $(x_{n})$ such that $x_{n}\notin\mathbb{Z}/n$ for all
$n$. Hence, setting $X_{n}=\bigoplus_{k\neq n}\mathbb{Z}/k$ and equipping this
space with the natural subspace metric $d_{n}$ generated by
$\\{1_{k}\in\mathbb{Z}/k\ :k\neq n\\}_{k}$ it holds that
$\ker(p)=\operatorname{Cone}_{\omega}(X_{n},d_{n})$. Finally, it follows that
$\psi$ is an isometric isomorphism. ∎

######  Corollary 7.3.

Let $G=\bigoplus_{n}\mathbb{Z}/n$ be equipped with the word norm generated by
$\\{1_{n}\in\mathbb{Z}/n\\}_{n}$. There is an injective map
$\pi_{1}(S^{1})\hookrightarrow\pi_{1}(\operatorname{Cone}_{\omega}(G,\|.\|))$.
In particular, $\operatorname{Cone}_{\omega}(G,\|.\|)$ is not contractible.

###### Proof.

Every map collapsing $S^{1}$ to a point and including it in another
topological space $Y$ is continuous. Therefore, by the universal property of
the product there are continuous maps $S^{1}\hookrightarrow
S^{1}\times\operatorname{Cone}_{\omega}(X_{n},\|.\|_{n})\rightarrow S^{1}$
whose composition is the identity. This induces an injective map
$\pi_{1}(S^{1})\hookrightarrow\pi_{1}(\operatorname{Cone}_{\omega}(G,\|.\|))$
on the level of fundamental groups. ∎

######  Proposition 7.4.

Let $\mathbb{Z}/n$ be equipped with the word norm $\|.\|_{n}$ generated by
$1\in\mathbb{Z}/n$ for all $n$ and let the free product
$\ast_{k\in\mathbb{N}}\mathbb{Z}/k$ be equipped with the associated
$\ell_{1}$-norm. Then $S^{1}$ is a topological retract of
$\operatorname{Cone}_{\omega}(\ast_{k\in\mathbb{N}}\mathbb{Z}/k,\|.\|_{\ell_{1}})$.

###### Proof.

By Lemma 4.19 the inclusions
$\iota_{n}\colon\mathbb{Z}/n\rightarrow\ast_{k}\mathbb{Z}/k$ are all
isometries. The projections
$p_{n}\colon\ast_{k}\mathbb{Z}/k\rightarrow\mathbb{Z}/n$ are all Lipschitz
continuous with constant one. On the level of asymptotic cones this gives rise
to the Lipschitz homomorphisms

$\displaystyle\operatorname{Cone}_{\omega}(\mathbb{Z}/n,\|.\|_{n})\xrightarrow{\iota=[\iota_{n}]}\operatorname{Cone}_{\omega}(\ast_{k}\mathbb{Z}/k,\|.\|_{\ell_{1}})\xrightarrow{p=[p_{n}]}\operatorname{Cone}_{\omega}(\mathbb{Z}/n,\|.\|_{n}).$

By Lemma 7.1 identify $\operatorname{Cone}_{\omega}(\mathbb{Z}/n,\|.\|_{n})$
with the unit circle. Since $p_{n}\circ\iota_{n}=id_{\mathbb{Z}/n}$ for all
$n$ we have $p\circ\iota=id_{S^{1}}$ and so
$\operatorname{Cone}_{\omega}(\ast_{k\in\mathbb{N}}\mathbb{Z}/k,\|.\|_{\ell_{1}})$
retracts onto $S^{1}$ via $p$. ∎

######  Corollary 7.5.

Let $\mathbb{Z}/n$ be equipped with the word norm $\|.\|_{n}$ generated by
$1\in\mathbb{Z}/n$ for all $n$ and let $\ast_{k\in\mathbb{N}}\mathbb{Z}/k$ be
equipped with the associated $\ell_{1}$-norm. Then
$\operatorname{Cone}_{\omega}(\ast_{k\in\mathbb{N}}\mathbb{Z}/k,\|.\|_{\ell_{1}})$
has nontrivial fundamental group. In particular, it is not contractible. ∎

## Acknowledgements

I like to thank Andreas Thom and Jakob Schneider for their helpful comments
during a research visit at the TU Dresden in January 2020. This work was
partly funded by the Leverhulme Trust Research Project Grant RPG-2017-159.

## References

  * [AKKK19] D. Álvarez-Gavela, V. Kaminker, A. Kislev, K. Kliakhandler, A. Pavlichenko, L. Rigolli, D. Rosen, O. Shabtai, B. Stevenson, J. Zhang, Embeddings of free groups into asymptotic cones of Hamiltonian diffeomorphisms, J. of Topol. and Anal., 2019.
  * [BDS11] J. Behrstock, C. Drutu, M. Sapir, Median structures on asymptotic cones and homomorphisms into mapping class groups, Proc. Lond. Math. Soc., Vol. 102, Iss. 3, 2011, 503 – 554.
  * [BIP07] D. Burago, S. Ivanov , L. Polterovich, Conjugation-invariant norms on groups of geometric origin, Groups of Diffeomorphisms, Adv. Stud. Pure Math., Vol 52, 2008, 221 – 250.
  * [Bur99] J. Burillo, Dimension and Fundamental Groups of Asymptotic Cones, J. Lond. Math. Soc., vol. 59, Iss. 2, 1999, 557 – 572.
  * [Bre78] J.L. Brenner, Covering theorems for FINASIGs. VIII. Almost all conjugacy classes in $\operatorname{A}_{n}$ have exponent $\leq 4$. J. Austral. Math. Soc. Ser. A. 25, no.2, 1978, 210 – 214.
  * [Cal09] D. Calegari, scl. Mathematical Society of Japan Memoirs: Volume 20, 2009.
  * [CK83] D. Carter, G. Keller, Bounded elementary generation of $\operatorname{SL}(n,\mathfrak{O})$, Amer. J. Math. 105, 1983, 673 – 687.
  * [CZ11] D. Calegari, D. Zhuang, Stable W-length. Contemporary Mathematics 560, 2011, 145 – 169.
  * [dDW84] L. Van den Dries, A. Wilkie, Gromov’s theorem on groups of polynomial growth and elementary logic. J. Algebra 89, 1984, 349 – 374.
  * [DK18] C. Drutu, M. Kapovich, Geometric Group Theory, AMS Colloquium Publications, Vol.63, 2018.
  * [EO05] A. Erschler, D. Osin, Fundamental groups of asymptotic cones, Topology, vol. 44, Iss. 4, 2005, 827 – 843.
  * [ES05] G. Elek, E. Szabó, Hyperlinearity,essentially free actions and $L^{2}$-invariants. The sofic property. Mathematische Annalen, Vol.332, 2, Springer, 2005, 421 – 441.
  * [Ghy01] É. Ghys, Groups acting on the circle. Enseign. Math. (2), 47(3-4), 329 – 407.
  * [Gro81] M. Gromov, Groups of polynomial growth and expanding maps, Inst. Hautes Etudes Sci. Publ. Math. 53, 1981, 53 – 73.
  * [HS16] T. Hartnick, P. Schweitzer, On quasioutomorphism groups of free groups and their transitivity properties. J. of Algebra, Vol. 450, 2016, 242 – 281.
  * [Kar21] B. Karlhofer, Aut-invariant quasimorphisms on free products, Ann. Math. Québec, 2021. https://doi.org/10.1007/s40316-021-00184-4.
  * [KLM21] J. Kędra, A. Libman, B. Martin, Strong and Uniform Boundedness of Groups. To appear in J. of Topol. and Anal. https://arxiv.org/abs/1808.01815, 2021.
  * [KSTT05] L. Kramer, S. Shelah, K. Tent, S. Thomas, Asymptotic cones of finitely presented groups, Adv. Math. 193, 2005, 142 – 173.
  * [LP97] M.W. Liebeck, L. Pyber, Upper Bounds for the Number of Conjugacy Classes of a Finite Group, Journal of Algebra, vol. 198, Issue 2, 1997, 538–562.
  * [LS01] M. W. Liebeck, A. Shalev, Diameters of Finite Simple Groups: Sharp Bounds and Applications, Ann. Math. (2), 154, 2001, 383 – 406.
  * [Mil99] G.A. Miller, On the commutators of a given group. Bull. Amer. Math. Soc. Vol. 6, No.3, 1899, 105 – 109.
  * [Mor07] D. Witte Morris, Bounded generation of $\operatorname{SL}(n,A)$ (after D. Carter, G. Keller, E. Paige). New York J. Math., 13, 2007, 383 – 421.
  * [OOS09] A.Y. Ol’shanskii, D.V. Osin, M.V. Sapir, Lacunary hyperbolic groups, Geom. Topol. 13, 2009, no. 4, 2051 – 2140, Appendix by M. Kapovich and B. Kleiner.
  * [OS07] A.Y. Ol’shanskii, M.V. Sapir, A Finitely Presented Group with Two Non-Homeomorphic asymptotic Cones, Int. J. Algebra Comput., Vol. 17, 2007, 421 – 426.
  * [Ore51] O. Ore, Some remarks on commutators, Proc. Amer. Math. Soc. 2, 1951, 307 – 314.
  * [Ril03] T.R. Riley, Higher connectedness of aymptotic cones, Topology, vol. 42, 2003, 1289 – 1352.
  * [TS14] A. Stolz, A. Thom, On the lattice of normal subgroups in ultraproducts ofcompact simple groups, Proc. London Math. Soc., 108(3), 2014, 73 – 102.
  * [TW14] A. Thom, J. Wilson, Metric ultraproducts of finite simple groups, C. R. Math. Acad. Sci. Paris 352, 2014, 463 – 466.
  * [Wei94] C. Weibel, An introduction to Homological Algebra. Cambridge Studies in Adv. Math., Vol. 38, Cambridge University Press 1994.
  * [Wil17] J. Wilson, Metric ultraproducts of classical groups, Arch. Math. 109, 2017, 407 – 412.