# Numerical Stability of DeepGOPlus Inference

Inés Gonzalez Pepe1, Yohan Chatelain1, Gregory Kiar2, Tristan Glatard1
1 Department of Computer Science and Software Engineering, Concordia
University, Montreal, Canada
2 Center for the Developing Brain, Child Mind Institute, New York, NY, USA

###### Abstract

Convolutional neural networks (CNNs) are currently among the most widely-used
neural networks available and achieve state-of-the-art performance for many
problems. While originally applied to computer vision tasks, CNNs work well
with any data with a spatial relationship, besides images, and have been
applied to different fields. However, recent works have highlighted how CNNs,
like other deep learning models, are sensitive to noise injection which can
jeopardise their performance. This paper quantifies the numerical uncertainty
of the floating point arithmetic inaccuracies of the inference stage of
DeepGOPlus, a CNN that predicts protein function, in order to determine its
numerical stability. In addition, this paper investigates the possibility to
use reduced-precision floating point formats for DeepGOPlus inference to
reduce memory consumption and latency. This is achieved with Monte Carlo
Arithmetic, a technique that experimentally quantifies floating point
operation errors and VPREC, a tool that emulates results with customizable
floating point precision formats. Focus is placed on the inference stage as it
is the main deliverable of the DeepGOPlus model that will be used across
environments and therefore most likely be subjected to the most amount of
noise. Furthermore, studies have shown that the inference stage is the part of
the model which is most disposed to being scaled down in terms of reduced
precision. All in all, it has been found that the numerical uncertainty of the
DeepGOPlus CNN is very low at its current numerical precision format, but the
model cannot currently be reduced to a lower precision that might render it
more lightweight.

##  I Introduction

Advances in the field of proteomics have resulted in a surplus of protein
sequences that cannot be identified by traditional experimental means, thus
leading to innovations in computational protein function detection methods.
Deep neural networks (DNNs) have become popular approaches for this task in
recent years due to their state-of-the-art performance [1, 2, 3]. Nonetheless,
while current models are able to achieve high performance when predicting
protein function, the numerical reliability of these predictions—defined as
the stability with respect to minor numerical perturbations—is still mainly
unknown and should be evaluated for multiple reasons. First, adversarial
attacks have shown that perturbing input data with small amounts of noise can
drastically impact classification outcomes [4, 5, 6], which suggests that
minor numerical perturbations in the model itself could similarly impact the
predictions. Second, numerical schemes adopted for the implementation of
neural networks may impact classification stability; for instance, Tensorflow
[7] utilises Winograd convolutions [8] to reduce the number of arithmetic
operations, which can result in a loss of precision for the model [9]. This
potential numerical instability is worrisome in any field, but particularly in
biology where the applications of such models extend to the healthcare sector
and confidence in the model’s predictions is critical for ultimately providing
correct diagnosis to patients that healthcare professionals can depend on.

Furthermore, numerical stability has strong implications on the numerical
precision required in model training and inference, and therefore on the
floating point number formats available for the model. In turn, floating point
formats critically impact the training and inference time of the model and its
memory consumption. In recent years, the use of reduced or mixed precision has
yielded substantial performance benefits for DNNs, including CNNs [10, 11, 12,
13]. These efforts do however come back to numerical stability, as reduced
precision can only be attempted on a model that is already stable at full
precision and must be similarly stable at a lower precision without a drop in
performance.

Focusing on DeepGOPlus [14], a state-of-the-art CNN protein function
classification model, this paper aims to (1) determine the stability of
DeepGoPlus inference as a result of minute numerical perturbations, (2)
evaluate the possibility of using reduced numerical precision formats during
the inference stage of DeepGOPlus. To achieve the first goal, we study
DeepGOPlus’ numerical stability by employing Monte Carlo Arithmetic (MCA)
[15]. MCA is a technique to empirically evaluate numerical stability by
injecting noise into floating point operations involved in programs and
quantifying the resulting numerical error at a given virtual precision. We
apply MCA to the model through software tools Verificarlo [16] and Verrou [17]
that dynamically replace floating point operations by perturbed ones at
program execution time. The perturbations introduced simulate noise
encountered when running code in different environments subject to library
updates, OS and/or hardware changes. To achieve the second goal, we emulate
reduced precision formats with the VPREC tool [18] and observe the
corresponding predictive performance. We evaluate reduced precision formats
that have existing hardware implementations to increase the practical value of
our results.

The next section discusses related work on the numerical stability of DNNs and
their use of reduced precision. The methods section introduces the tools
mentioned above such as MCA, Verificarlo, Verrou and VPREC, as well as the
dataset used for our evaluations and other experimental parameters. The
results section presents our evaluation of the numerical stability of
DeepGOPlus, as well as the possibility of implementing reduced precision
formats in the inference stage.

##  II Related Work

###  II-A Numerical Stability of DNNs

Several works have studied DNNs to improve their numerical stability or
identify numerical instabilities in their implementation. One such approach is
DeepStability, a public database of numerical stability issues and solutions
in PyTorch and TensorFlow [19]. DeepStability was created by studying the
PyTorch and Tensorflow code repositories for issues revolving around numerical
stability. However, an approach other to manually compiling a numerical
stability database is the Monte Carlo Deep Neural Network Arithmetic (MCDA)
[20]. This technique measures the sensitivity of a model in regards to the
quantization of its floating point operations. Like this project, this paper
also utilises MCA in order to experimentally determine the uncertainty in DNN
inference stages. In this respect, MCA is a valuable method as it allows one
to quantify uncertainty through different tools such as MCDA, Verrou or
Verificarlo and can be scaled up to analyse large code bases such as
Tensorflow. Nonetheless, the numerical uncertainty rendering DNNs unstable can
have different origins, among them the algorithms used to implement the model.
Algorithms in general can have different equivalent implementations, but some
implementations are more stable than others. As a result, more stable forward
propagation algorithms [21] and skip connections in ResNet models [22] have
been proposed. These works differ from the ones previously mentioned as they
propose stable implementations to specific unstable problems while the former
propose tools to be used across DNNs to identify said unstable problems.

This work is important, particularly in the context of adversarial attacks,
i.e., perturbations of the input data to mislead the model, which are
currently being considered the main source of noise for DNNs. When expanding
the scope of numerical stability of DNNs to include robustness against
adversarial attacks, we can see that popular strategies involve detecting
attacks, mitigating them through data manipulation or mitigating them through
network manipulation [23, 24]. Identifying unstable components of a model
falls under mitigating adversarial attacks through network manipulation, but
this approach can be used in conjunction with others. Another example of
mitigating an attack through network manipulation is adding a defense layer to
minimize perturbations [25] which preserves the current algorithmic
implementation by modifying the network’s architecture. Meanwhile, an example
of mitigating the attack through data manipulation is stability training,
i.e., exposing the model to noisy or adversarial data in order to render it
more robust. The primary difference between these approaches and improving the
stability of the algorithmic implementation of the model is that the
aforementioned approaches modify the model from the exterior, while addressing
the algorithmic implementation modifies the internal workings of the model.

In brief, there exist a variety of approaches to addressing the robustness of
DNNs, particularly in response to adversarial attacks. Nonetheless, it is
important to note that while a lot of these approaches have been attempted on
CNNs, the class of the model studied in this paper, most of the
implementations suggested are for image-classification tasks [26, 27], which
allows for this paper to make a contribution to the numerical stability of
DNNs in the sector of protein function classification.

###  II-B Reduced Precision DNNs

DNNs are becoming widely adopted, but one of their limitations is the amount
of computing resources required to train and run them. One approach to address
this problem is implementing reduced precision formats to replace the standard
double precision (binary64) and single precision (binary32) IEEE-754 formats
used. Benefits of reducing the precision involve a direct decrease in memory
usage and an indirect decrease in the energy and time it takes to run an
arithmetic operation.

There are many approaches to reducing precision in DNNs. One is to reduce the
precision whole sale for the entire model, that is to say, to train it
entirely on the chosen reduced format. Another is to employ mixed-precision
when training the model, a technique that selectively reduces the precision in
certain operations, while leaving others intact. Besides only reducing
precision however, attempts to improve computing efficiency also involve
investigating fixed point operations along with floating point operations.
While there has been successes training reduced precision DNNs with fixed
point operations using stochastic rounding to allow the model to converge with
low precision format [28], this paper focuses on floating point operations
which generally offer a wider range of numbers and a larger dynamic exponent
range than fixed point as well as being the default for the DeepGOPlus model.

Using mixed precision usually involves more instrumentation on the part of the
user while reducing precision across the board is more likely to cause a drop
in performance or in stability [29]. Studies have shown that reducing the
precision at the inference stage of the DNN with negligible loss in
performance is achievable, but training the DNN with reduced precision is more
difficult due to the importance of preserving the gradients during
backpropagation [30]. Nonetheless, many recent papers have shown that
comparable performance can be achieved with DNNs trained on 16-bit precision
and in some cases with 8-bit precision with either approach [10, 11, 12, 13].

##  III Methods

This section firstly outlines the theory behind Monte Carlo Arithmetic before
delving into Verificarlo and Verrou, which are the tools used to implement it.
Significant digits are also explored as another form of quantifying numerical
uncertainty before the focus is shifted to reduced precision formats and how
to simulate their implementation using the VPREC tool. Finally the dataset,
model and experimental setup are explained in detail in order to render the
methodology of the project as transparent as possible.

###  III-A Use of Monte Carlo Arithmetic for Numerical Uncertainty
Quantification

In order to complete the first goal of establishing the numerical stability of
DeepGOPlus, we need a method of determining said stability. To this end, we
quantify the numerical uncertainty of the floating point model using Monte
Carlo Arithmetic. We use Monte Carlo Arithmetic to inject numerical
perturbations into the DeepGOPlus model and quantify the resulting error in
the system. MCA leverages randomness to model loss of accuracy inherent to
floating point arithmetic due to its finite precision. MCA simulates floating
point roundoff and cancellation errors through random perturbations, allowing
for the estimation of error distributions from independent random result
samples. MCA simulates computations for a given virtual precision using the
following perturbation:

$inexact(x)=x+2^{e_{x}-t}\xi$ (1)

where $e_{x}$ is the exponent in the floating point representation of $x$, $t$
is the virtual precision and $\xi$ is a random uniform variable of
$(-\frac{1}{2},\frac{1}{2})$. MCA has three modes that allows perturbation to
be introduced in the function input (Precision Bounding—PB), the function
output (Random Rouding—RR) or both (full MCA). Random Rounding tracks rounding
errors, while Precision Bounding tracks catastrophic cancellations, and full
MCA is a combination of both approaches. Precision bounding or full MCA which
includes PB is not used in this project as it is known to be more invasive and
prone to execution failures, which prevents analysis of numerical stability
[31]. Only RR, which introduces perturbation in function outputs thus
simulating roundoff errors is used in the scope of this project.

$random\\_rounding(x\circ y)=round(inexact(x\circ y))$ (2)

We ran DeepGOPlus inference using a virtual precision of 24 bits for binary32
numbers and 53 bits for binary64 numbers which is the default setting for the
model. Noise injected at these virtual precision values reflects typical
perturbations caused by changes in the model’s environment. These changes may
originate from software package changes, different operating systems or
different architectures, such as moving from CPU to GPU which do not generally
have the same solver implementation. To this end, we wish to quantify the
numerical uncertainty associated with the CNN across different environments.
Therefore, according to equation 1, for binary32, $e_{x}$ ranges between -149
and 127 in terms of values and $t$ equals 24. Therefore, we can calculate the
relative inputted noise to be $2^{-24}\simeq 10^{-8}$. For binary64, $e_{x}$
ranges between -1075 and 1023, $t$ is 53 so the magnitude injected is
$2^{e_{x}-53}$ corresponding to a relative error of $2^{-53}\simeq 10^{-16}$.
The noise is then introduced into the model through Verificarlo and Verrou and
ten iterations are run in order to obtain an adequate sample size to quantify.

#### Verificarlo

Verificarlo [16] is a clang-based compiler that replaces floating point
operations by a generic call to one of several configurable floating point
models available. MCA is leveraged in the DeepGOPlus model through the use of
Fuzzy [32], a collection of software packages compiled with Verificarlo which
uses the the libinterflop_mca.so backend. Fuzzy enables the instrumentation of
different libraries by Verificarlo in order to quantify the numerical
uncertainty present in code using these libraries. Existing MCA-instrumented
libraries available in Fuzzy that were used in this project are BLAS, LAPACK
and Python. Fuzzy containers located at https://github.com/verificarlo/fuzzy
using Verificarlo v0.8.0 with instrumented versions of BLAS and LAPACK 3.9.0
and the Python 3.8.5 interpreter were used to simulate different containing
noise.

#### Verrou

Instrumenting Tensorflow as a whole with Verificarlo was unsuccessful since
Verificarlo is a clang-based compiler and Tensorflow uses the GCC compiler.
Therefore, attempts were made to instrument Tensorflow indirectly through the
math libraries used, that is to say either Eigen or MKL. Instrumenting
Tensorflow through MKL with Verificarlo didn’t work since MKL is a closed-
source library. Since it is closed source, the source code is not accessible
to recompile which is necessary for Verificarlo to instrument it. However, the
library provides the option of outsourcing certain operations to the BLAS and
LAPACK libraries which are possible to instrument. Nonetheless, even with the
outsourcing to BLAS and LAPACK, the majority of arithmetic operations is done
internally by Eigen, therefore attempting to instrument Tensorflow indirectly
through Eigen was also unsuccessful.

It is not possible to pass over the instrumentation of Tensorflow as it is the
primary software package used to build the DeepGOPlus model as well as the
source of most arithmetic operations. Therefore, the Verrou tool was used to
instrument the Tensorflow library. The Verrou tool is located at
https://github.com/edf-hpc/verrou in Github where v2.3.1 was used. Verrou is a
tool which also uses Monte Carlo Arithmetic to monitor the accuracy of
floating point operations without needing to instrument the source code or
recompile it. Verrou is based upon Valgrind [33], a memory-debugging tool
which uses dynamic binary instrumentation, i.e. the process of modifying
instructions of a binary program while it executes. By default, Verrou
instruments the entire executable and produces perturbed results, but
libraries can be excluded from instrumentation by the user whenever necessary,
such as instrumenting solely Tensorflow. Moreover, while still implementing
MCA, it uses its own backend also called Verrou to introduce RR perturbations.

###  III-B Use of Significant Digits as Numerical Uncertainty Quantification
Metric

Once the MCA samples are obtained, the uncertainty in the samples was
quantified by calculating the number of significant digits as well as the
standard deviation. The method used to calculate significant digits also
provides confidence intervals at varying degrees of confidence and probability
[34]. These confidence intervals allow for the number of significant digits to
be calculated at precise levels of confidence and can inform the user on how
many samples should be used in order to achieve high confidence. The
$significant\\_digits$ package, which can be found at
https://github.com/verificarlo/significantdigits with v0.0.0 being used, can
implement both non-parametric and the Centred Normality Hypothesis cases.
However, we only use the general non-parametric case for significant digits
since we don’t know that the output samples follow the Centred Normality
Hypothesis.

For a general distribution, we first determine a bit’s significance
$S^{k}_{i}$ where $k$ is the position of the bit and $i$ is the number of the
iteration of the samples $X$ being evaluated where the sample size is $n$.
Given $Z$, a random variable which represents the distance between $X$ and its
reference value, we can define the bit as significant if the absolute value of
$Z_{i}$ is less than $2^{-k}$.

$S^{k}_{i}=\mathbbm{1}_{|Z_{i}|<2^{-k}}$

Then, to calculate the number of significant bits, $\hat{s_{b}}$, we determine
the maximal index $k$ for which the first $k$ bits of all $n$ sampled results
coincide with the reference is determined by:

$\hat{s_{b}}=\textnormal{max}\\{k\in\\{1,2,...,53\\}\textnormal{ such that
}\forall i\in\\{1,2,...,n\\}S^{k}_{i}=\mathbbm{1}\\}$

This concludes the calculation of the number of significant bits which can
then be converted to significant digits. When converting from bits to digits,
we note the maximal achievable value for binary64 input samples is 15.95 given
$53\log_{10}(2)$ and 7.23 is the maximal value for binary32 given
$24\log_{10}(2)$. Therefore, we compare our obtained samples to these in order
to obtain a measure of how much significance was lost and what remains.
However, this method can also provide the confidence and the probability at
which the number of significant digits is likely to be accurate which is
linked to the number of samples from which the significant digits are
calculated from. In this project we choose a sample size of 10 which
corresponds to 10 MCA iterations and according to the paper, this corresponds
to a good tradeoff between a high confidence and a high probability.

This method was chosen because of the confidence intervals it provides and
because it is more reliable than other stochastic arithmetic methods that also
calculate significant digits such as MCA and CESTAC (Controle et Estimation
Stochastique des Arrondis de Calculs) [35].

###  III-C Use of VPREC Backend for Reduced Precision Emulation

The VPREC backend [18] is used to emulate reduced numerical formats and the
results they would produce, thus allowing for an evaluation of how DeepGOPlus
functions at lower precisions after numerical uncertainty is quantified. It is
important to note, that as previously mentioned, Verrou and Verificarlo both
use different backends with the same goal of introducing MCA perturbations
into DeepGOPlus. In contrast, the VPREC backend [18] can now be used as a
backend by either Verrou or Verificarlo. VPREC was developed by the
Verificarlo team originally for Verificarlo but was made compatible as a
backend for Verrou as an additional contribution of this project. It can
simulate any floating point format within the binary64 format by modifying the
length of the exponent and the mantissa, referred to as precision. VPREC
emulates results obtained at customised numerical formats by performing the
operation in binary64 precision and rounding the result to the custom
precision and exponent.

Table I: Numerical Formats Specifications Format | Precision (bits) | Exponent (bits)
---|---|---
binary32 | 24 | 8
binary16 | 11 | 5
bfloat16 | 8 | 8
bfloat8 | 3 | 5

The VPREC backend was used to simulate the reduced precision formats of
IEEE-754 binary32 [36], IEEE-754 binary16 [36], bfloat16 [37] and Microsoft’s
bfloat8 bfloat8s [38]. These are popular formats that are already implemented
in hardware and have already been used to reduce precision while maintaining
performance and accuracy in different code applications, including in some
cases, deep learning algorithms [13]. In Table I, we see the exact precision
and exponent range used to simulate each format in VPREC.

###  III-D Dataset

The data used in this paper can be found at http://deepgoplus.bio2vec.net/data
labelled as version 1.0.6. This dataset was originally used in the DeepGOPlus
paper to compare the model to other models in the fields, but the dataset has
since been updated since the original publication of the DeepGOPlus paper to
reflect new data releases. The dataset is composed of the Gene Ontology
released on 2021-10-26 and the SwissProt data version 2021_04. The data
consists of experimental annotations collected before a certain data as the
training set and confirmed experimental annotations discovered later as the
test set.

Moreover, the data directory also contains a model trained on this dataset as
well as the performance metrics obtained by the authors corresponding to the
data and which are used as references in this paper.

###  III-E DeepGOPlus Model

DeepGOPlus utilises the Gene Ontology (GO) [39], a hierarchical classification
that separates protein functions into annotations that each fall under three
super-classes; Biological Process (BP), Molecular Function (MF) and Cellular
Component (CC), in order to predict the protein’s function by inferring the
protein’s associated Gene Ontology annotations. Unlike other models,
DeepGOPlus only takes as input the protein sequence and does not require
additional features such as protein-protein interactions, contact maps,
protein structure, etc. This additional information about the protein can be
difficult to come by especially when trying to predict the function of
recently discovered proteins little is known about. DeepGOPlus is one of the
three best predictors in the Cellular Component class and the second best
performing method in the Biological Process and Molecular Function evaluations
compared to other models based on the evaluations from the CAFA3 challenge
[40].

The model is composed of 16 one-dimensional convolutional layers with
different filter lengths that are then sent to their respective max pooling
layer before coming together in a fully connected layer which returns a vector
with a specific probability score associated with each GO annotation. These
protein function class probabilities are then individually combined using a
weighted sum with their counterpart probabilities which were obtained using a
different protein function predictor called Diamond [41]. The model is trained
on data from the CAFA3 challenge, a community experiment to compare protein
prediction methods, and from SwissProt [42], a protein function database.

A model already trained on the v1.0.6 data is made available, as detailed in
III-D, which simplifies the task of studying the inference step. Moreover, we
also excluded the Diamond tool to determine the instability originating purely
from the CNN.

###  III-F Experimental Setup

The model was run in Singularity containers over ten MCA iterations in order
to obtain perturbed samples from which to get a measure of the instability
within the model. The container containing instrumented Python was
instrumented using the Fuzzy software tool while Tensorflow and the entirety
of the model were instrumented using the Verrou software tool. In order to
render the naming conventions clear in the following sections and figures,
Table II explains the naming schemes used.

Table II: Naming Conventions of Instrumented Results Names | Meaning
---|---
Verificarlo Python | Python Interpreter Instrumented with Verificarlo
Verificarlo Libmath | BLAS/LAPACK Libraries Instrumented with Verificarlo
Verrou Tensorflow/TF | Tensorflow Library Instrumented with Verrou
Verrou All | Entire CNN Instrumented with Verrou

From the obtained MCA samples, the standard deviation and the mean were
observed as well as the number of significant digits in order to measure the
variability of protein function class probabilities.

To assess the performance of the model, $S_{min}$, $F_{max}$ and $AUPR$ were
used as metrics as they are the metrics used to evaluate the model in the
DeepGOPlus paper [14]. Typical protein function predictors output
probabilities that indicate the probability of belonging to a specific
function. Therefore, when calculating metrics to measure the predictors’
performance, the uncertainty and misinformation linked to the predictions
needs to be considered. To this end, the semantic distance between the real
and predicted functions based on information content ($S_{min}$) is
calculated:

$S_{min}=min_{t}\sqrt{ru(t)^{2}+mi(t)^{2}}$

where $ru(t)$ is the average remaining uncertainty and $mi(t)$ is average
misinformation:

$ru(t)=\frac{1}{n}\sum_{i=1}^{n}\sum_{c\in T_{i}-P_{i}(t)}IC(c)$
$mi(t)=\frac{1}{n}\sum_{i=1}^{n}\sum_{c\in P_{i}(t)-T_{i}}IC(c)$

We define $IC(c)$ is the information content for ontology class $c$,
$P_{i}(t)$ is a set of predicted annotations for a protein $i$ and prediction
threshold $t$ and $T_{i}$ is a set of true annotations, $P_{i}$(t).

Moreover, precision and recall are considered and combined into the F-measure
to provide one metric. F-measure is the harmonic mean of precision and recall,
but to account for thresholding which is often used with protein function
predictors, the maximal F-measure over all possible threshold values
($F_{max}$) is ultimately used and defined as:

$F_{max}=max_{t}\left\\{\frac{2\cdot AvgPr(t)\cdot
AvgRc(t)}{AvgPr(t)+AvgRc(t)}\right\\}$

where $AvgPr(t)$ is defined as the average precision for a threshold value and
$AvgRc(t)$ is the average recall for the same threshold value.

Finally, the Area Under the Precision Recall curve ($AUPR$) was also chosen
since the authors of the DeepGOPlus reported this metric due to the high class
imbalance in the data.

Therefore, the $F_{max}$, $S_{min}$ and Area Under the Precision Recall curve
($AUPR$) metrics were calculated from the perturbed samples. These metrics
were chosen as they are the original measures of performance chosen by the
authors of the DeepGOPlus paper. The mean and standard-deviation statistics
about the number of significant digits are also reported.

##  IV Results

###  IV-A Sanity Check

Modified versions of the scripts that generate the DeepGOPlus protein function
class probabilities and metrics were used in order to remove instability
originating from the Diamond tool used, multithreading and any sources that
did not originate from the CNN itself when run without instrumentation
(IEEE-754). While there are metrics provided by the DeepGOPlus authors to use
as ground truth, it is more difficult to obtain ground truth for the protein
function class probabilities predicted by the model. Since binary32 is the
format currently used by DeepGOPlus, results calculated with IEEE in a
purposefully stable environment are considered the reference against which the
perturbed results are compared to. The results obtained from the original
DeepGOPlus paper are only given with three digits. However, the perturbed
results are identical to the reference results up to the second digit after
the decimal place, after which there is some variation observed. This is to be
expected considering the Diamond tool was removed from the model. Moreover,
given ten MCA iterations were used through this paper, we can conclude that
the number of significant digits calculated have a confidence level of 0.80
and a probability of 0.85 [34]. For the displayed instrumented libraries, PB
and full MCA weren’t used to instrument the model as it would cause it to
crash, instead what is seen is the average of the evaluation metrics
calculated from MCA iterations perturbed with RR.

Table III: DeepGOPlus Metrics From Original Paper, IEEE and Averaged
Instrumented Environments Across GO Classes MF, CC and BP

| Fmax | Smin | AUPR
---|---|---|---
| MF | CC | BP | MF | CC | BP | MF | CC | BP
DeepGOPlus | 0.664 | 0.695 | 0.530 | 9.591 | 10.442 | 44.370 | 0.671 | 0.738 | 0.496
IEEE | 0.6639578 | 0.6944381 | 0.5301602 | 9.5972334 | 10.4480327 | 44.3779111 | 0.6707084 | 0.7379463 | 0.4958606
Verificarlo Python | 0.6639578 | 0.6944381 | 0.5301602 | 9.5972334 | 10.4480327 | 44.3779111 | 0.6707084 | 0.7379463 | 0.4958606
Verrou TF | 0.6639578 | 0.6944381 | 0.5301602 | 9.5972334 | 10.4480327 | 44.3779111 | 0.6707084 | 0.7379463 | 0.4958607
Verrou All | 0.6639578 | 0.6944381 | 0.5301602 | 9.5972334 | 10.4480327 | 44.3779111 | 0.6707084 | 0.7379463 | 0.4958607

###  IV-B No Instabilities Found in DeepGOPlus at Virtual Binary32 Precision
24 and Virtual Binary64 Precision 53

Table IV: DeepGOPlus Metrics and Significant Digits Across Instrumented
Environments

|  | Fmax | Smin | AUPR
---|---|---|---|---
|  | Mean |  | Standard
---
Deviation
| Significant
---
Digits
Mean |  | Standard
---
Deviation
| Significant
---
Digits
Mean |  | Standard
---
Deviation
| Significant
---
Digits
Fuzzy Python | MF | $6.6395779\text{e-}1$ | 0.00 | 6.92 | $9.5972334\text{e}0$ | 0.00 | 6.92 | $6.7070835\text{e-}1$ | 0.00 | 6.92
CC | $6.9443807\text{e-}1$ | 0.00 | 6.92 | $1.0448033\text{e}1$ | 0.00 | 6.92 | $7.3794628\text{e-}1$ | 0.00 | 6.92
BP | $5.3016021\text{e-}1$ | 0.00 | 6.92 | $4.4377911\text{e}1$ | 0.00 | 6.92 | $4.9586063\text{e-}1$ | 0.00 | 6.92
Verrou TF | MF | $6.6395779\text{e-}1$ | 0.00 | 6.92 | $9.5972334\text{e}0$ | 0.00 | 6.92 | $6.7070835\text{e-}1$ | 0.00 | 6.92
CC | $6.9443807\text{e-}1$ | 0.00 | 6.92 | $1.0448033\text{e}1$ | 0.00 | 6.92 | $7.3794629\text{e-}1$ | $3.6136826\text{e-}8$ | 6.92
BP | $5.3016021\text{e-}1$ | 0.00 | 6.92 | $4.4377911\text{e}1$ | 0.00 | 6.92 | $4.9586067\text{e-}1$ | $3.3860007\text{e-}8$ | 6.92
Verrou All | MF | $6.6395779\text{e-}1$ | 0.00 | 6.92 | $9.5972334\text{e}0$ | 0.00 | 6.92 | $6.7070835\text{e-}1$ | 0.00 | 6.92
CC | $6.9443807\text{e-}1$ | 0.00 | 6.92 | $1.0448033\text{e}1$ | 0.00 | 6.92 | $7.3794628\text{e-}1$ | 0.00 | 6.92
BP | $5.3016021\text{e-}1$ | 0.00 | 6.92 | $4.4377911\text{e}1$ | 0.00 | 6.92 | $4.9586066\text{e-}1$ | $3.6831094\text{e-}8$ | 6.92

Figure 1: Standard Deviation Distribution Across Protein Class Probabilities
for Different Instrumented Libraries Figure 2: Mean and Standard Deviation of
Significant Digits Across Protein Class Probabilities for Different
Instrumented Libraries

#### Tensorflow and All Inclusive

Most of the model’s arithmetic operations occur within Tensorflow, therefore
it can be expected that the most numerical instabilities will be found when
using an instrumented Tensorflow library. When looking at the instability
found among the protein function class probabilities in Figure 1, we find
standard deviation with an average magnitude of $10^{-8}$. Despite some
outliers achieving a magnitude of $10^{-}7$, most probabilities are at a lower
magnitude as seen in the histogram when one considers the Y-axis is
logarithmically scaled. This noise which is found in the probabilities, is
equivalent to the magnitude of noise being introduced into the Tensorflow
library. Therefore, the magnitude of noise in the output is the same as the
magnitude of the noise input to the floating point operations which
demonstrates there is no internal instability in the model that is amplifying
the numerical error.

When observing the metrics obtained using the instrumented Tensorflow library
in Table IV, we can see that the level of noise previously found is close to
that of the precision of the binary32 format and is not large enough to
differentiate the results from the IEEE references. It is interesting to note
however, that while the $F_{max}$ and $S_{min}$ metrics are consistently
identical across all Gene Ontology super classes, the AUPR metric is less
stable, reaching a maximum standard deviation once again in the magnitude of
$10^{-8}$.

Moreover, when calculating the number of significant digits among protein
function class probabilities, there was approximately an average of 6.92
significant digits found as we can see in Figure 2, while in Table IV, we see
the metrics’ also have an average of 6.92 significant digits. Given that these
numbers were calculated in binary32, this is slightly less than the full 7.23
significant digits available, but not worrisome since both values round to 7
and this slight loss does not appear to impact the results. Finally, all
instrumented libraries are tested together in order to observe the instability
when everything is instrumented at the same time. To this end, Verrou was used
to instrument all libraries, including the Python interpreter. When observing
the results in Figure 1 and the metrics in Table IV, one sees that the
instability of the protein function class probabilities, the metrics, even the
distribution of the standard deviation of the protein function class
probabilities match the instability of just the instrumented Tensorflow
results quite similarly. Moreover, we also see virtually identical results
when calculating the number of significant digits. This makes sense
considering all instability found at this point originates from the
instrumented Tensorflow library.

#### Python Interpreter

As can be seen in Figure 1, the standard deviation of the protein function
probabilities is null and in Table IV the standard-deviation of the metrics is
also zero and the number of significant digits remained at approximately
sixteen digits meaning that no instability is detected in the final products
of the model. Like with all libraries instrumented, noise with a magnitude of
$10^{-8}$ was introduced to the model, though in this case exclusively through
the Python interpreter. However, no trace of it is seen in the final output of
the model as the Python interpreter does not appear to perform many if any of
the arithmetic operations used by the model.

(a) Biological Process (BP)

(b) Cellular Component (CC)

(c) Molecular Function (MF)

Figure 3: Difference in Performance between Reduced Precision Formats and IEEE
for GO Classes

###  IV-C DeepGOPlus Inference Can Be Reduced From Double Precision to Single
Precision

Given the stability of the DeepGOPlus model, this opens up the possibility of
exploring reduced precision formats for the DeepGOPlus model with VPREC. To
this end, the float32, float16, bfloat16 and bfloat8 low-precision numerical
formats are tested in order to provide benchmarks of the DeepGOPlus model’s
performance at each reduced format. Given that both the binary64 and binary32
format are used by DeepGOPlus, sixteen different combinations of the above
four reduced formats are tested though only six are displayed as DeepGOPlus
crashes when reducing the binary64 position to float16 or when reducing either
the binary64 or binary32 position to bfloat8.

Interestingly enough, setting binary64 to float16 or bfloat8 causes Tensorflow
to abort upon starting up while setting binary32 to bfloat8 only throws a
calculation error once the model has begun running. Considering most of the
arithmetic operations occur in binary32, when binary32 is set to bfloat8, this
can perhaps be attributed to numerical overflow/underflow. While studies have
utilised 8-bit formats with DNNs, the usual approach is leaving the
calculations to be performed in float32 and selectively using the 8-bit format
to store intermediate computations and the data input to the model. In this
case, all operations were reduced to bfloat8, including the arithmetic
operations, which explains why the model crashed despite bfloat8 being used
successfully in other scenarios. However, whenever binary64 is improperly
reduced, the error appears to be a failed start up sanity check. This implies
even if binary64 is not crucial to the inference stage of the model, it is
necessary in ensuring Tensorflow, the software running the model, works
correctly.

When examining the performance of these numerical formats in Figures 3a, 3b
and 3c, there is a substantive drop in performance with an average of -4% for
$F_{max}$, -6% for AUPR and -7% for $S_{min}$ across all protein classes,
excluding the combinations (float32, float32) and (bfloat16, float32) for
binary64 and binary32 respectively. These two combinations of numerical
formats achieve comparable performance to the full IEEE performance with
negligible loss or gain.

##  V Discussion

As seen previously, the DeepGOPlus CNN is incredibly stable as no variation is
seen among the metrics evaluated, despite the presence of some standard
deviation across the protein function class probabilities. This is primarily
due to the fact that the variation found is of a magnitude of $10^{-8}$ which
therefore only causes the last digits of the probabilities to vary. Therefore,
when thresholding is applied to the probabilities, these variations in the
trailing digits have no effect on the thresholding. Furthermore, the
probabilities themselves are just a means to obtain the protein function class
labels, so once the probabilities pass thresholding, the probability is
discarded in favor of the label it represents. These labels are classified as
true positive, true negative, false positive and false negative predictions in
order to compute the precision and recall metrics which are used in all of the
final metrics evaluated. Since the instability did not significantly modify
the value of the probabilities, the uncertainty was mostly unable to filter
through and affect the model’s performance metrics. Moreover, the $F_{max}$
and $S_{min}$ metrics involve the use of the max and min operations, which
also explains why $AUPR$ was the only metric where some variability was
evident.

Interestingly enough, after having instrumented all libraries together and
separately, we can observe that the MF ontology class is consistently the most
stable. Even in the presence of noise that managed to create a non-zero
standard deviation with CC and BP, the MF class remained completely stable.
This disparity can be partially explained by the fact that while CC and BP and
MF are all under the Gene Ontology umbrella, they are in fact separate,
unrelated protein function hierarchical ontologies. Moreover, it is not an
even split among the three classes and MF is least represented which might
also contribute to why no instability was seen.

###  V-A AUPR Instability

This difference in metric stability might be explained by the fact that the
AUPR is calculated by taking the precision and recall calculated at each
threshold value between 0 and 1 while the $F_{max}$ and $S_{min}$ displayed
are always the metrics found after finding the optimal threshold value.
However, another factor that might also be impacting AUPR’s stability is the
fact that it is calculated using a numerical integration with a trapezoidal
scheme. Doing so involves some level of approximation of the shape of the AUPR
curve, which could also factor into the instability found. Nonetheless, the
calculation is known to be considerably stable and would only introduce noise
in the magnitude of $10^{-16}$, which is much smaller than the magnitude found
suggesting the instability found is from the perturbed results.

###  V-B Necessity of Float32 for Binary32 Format

It is interesting to note that the precision of the binary64 position can be
halved with little to no effect, but the same cannot be said for the binary32
position. This is most likely due to the fact that the model mainly utilises
binary32 and binary64 accounts for a minimal amount of total instrumented
operations. When examining Verrou output of instrumented operations, besides
regular arithmetic computations, we know Fused Multiply-Add (FMA) operations
are also instrumented. This is essential as FMA operations are key hardware
components for training DNNs. Moreover, when looking at the instrumented
operation breakdown, binary64 accounts for less than 1% of additions,
subtractions and multiplications and approximately 30% of divisions. It would
look like double operations are not negligible, however, when addition,
subtraction, multiplication and division operations are put in perspective
relative to the total amount of instrumented operations, divisions account for
less than 1% of total operations while addition and multiplication jointly
account for approximately 95%. Considering additions and multiplications are
the key operations in convolutions and binary64 operations are barely present
along with the finding that binary64 precision can be halved with no impact,
it can be concluded that binary64 is not used in core operations such as
convolution layers and are most likely used to store intermediate values in
between operations.

Furthermore, another takeaway from this is the performance of bfloat16. As
seen, binary32 must be kept to nearly full precision, but binary64 can be
reduced. Nonetheless, while the model was able to run with bfloat16 as the
numerical format for binary64, it crashed when using float16. Float16 has a
mantissa of length ten and an exponent of length of five, while bfloat16 has a
smaller mantissa of length seven but a larger exponent of length eight.
Considering the model’s inability to perform with float16 in the binary64
position, it can be concluded that a crucial amount of information is stored
in the exponent versus the mantissa for this CNN and bfloat16 larger dynamic
exponent makes it better optimized to render machine learning more lightweight
[13].

###  V-C Verrou & Verificarlo Comparison

Both Verrou and Verificarlo are tools for numerical uncertainty quantification
and do so by implementing MCA. Moreover, both are set to use virtual precision
24 for binary32 and virtual precision 53 for binary64 in order to be
consistent across the numerical instability quantification. However, they
differ in their implementation. As a whole, Verrou can be used with programs
across languages with no instrumentation, unlike Verificarlo which requires
the programs targeted to be recompiled. Nonetheless, we observed Verrou can
run up to approximately 120x slower than Verificarlo, which can become a
bottleneck in any project if not utilised carefully.

As an additional sanity check, the numerical uncertainty quantification
results obtained from Verificarlo and Verrou were compared. In order to have a
complete comparison, results would be required from the Tensorflow library
instrumented by Verificarlo, but this was not possible due to compatibility
issues with the clang compiler. However, we can compare the results obtained
from instrumenting the Python interpreter with both Verrou and Verificarlo.
When examining Table III, we see that in both cases, instrumenting Python does
not result in any instability. We see that when Python is instrumented alone
with Verificarlo, no instability is found. Meanwhile, when Python is
instrumented with Tensorflow by Verrou, it appears that Tensorflow is the sole
source of instability. Therefore, we can confirm that the results obtained
with either Verificarlo or Verrou are comparable to one another and that the
tools can be used interchangeably at one’s discretion.

##  VI Conclusion

In brief, there were two goals in this paper. The first was to determine the
numerical stability of the DeepGOPlus model, in particular the CNN built to
predict protein function from protein sequences. The second was to evaluate
the performance of the DeepGOPlus inference stage with reduced precision
formats.

To this end, libraries used by DeepGOPlus were instrumented using MCA in order
to inject noise into the model, record the perturbed samples and quantify the
overall stability of the CNN. It was found that most of the arithmetic
operations performed by the model occurred in the Tensorflow library,
therefore, when perturbed, most noise originated from there. Nevertheless, the
instability found was negligible and overall DeepGOPlus can be considered a
stable model fairly impervious to noise. Despite its stability, when
instrumenting the model with VPREC in order to simulate performance at lower
precisions, it was found that most of the core arithmetic operations of the
inference stage could not be reduced to any of the formats tried. While
binary64 could be halved and reduced to the bfloat16 format, binary32, which
was used for most arithmetic operations, could not be reduced from its
original state. In conclusion, while the DeepGOPlus inference stage is stable
in its current state, it cannot afford to lose precision in binary32 without
suffering a substantive drop in performance.

This opens up new avenues of leveraging the model’s stability in order to
render it more lightweight overall and therefore easier to deploy. Considering
the DeepGOPlus model was pre-trained for binary32 and binary64, this might
explain as to why the inference stage, while stable, could not achieve
comparable performance with reduced precision formats, since it was not
trained to due so. There are several examples in the literature [13] that
demonstrate deep learning algorithms including CNNs can achieve high
performance with reduced precisions. Therefore, perhaps not all bits in the
binary32 mantissa are useful but as the model is currently trained to utilise
them all, future work might involve re-training it to use less. Other lines of
questioning involve exploring mixed precision techniques or studying deeper
CNNs than DeepGOPlus (which is composed of a single convolutional layer),
where the risk of exploding or vanishing gradients [43] is prevalent and
therefore its stability is potentially more fragile.

## Acknowledgements

Special thanks are extended to the authors of the DeepGOPlus paper and for the
support they provided answering queries about their work.

## References

  * [1] A. K. Tiwari and R. Srivastava, “A Survey of Computational Intelligence Techniques in Protein Function Prediction,” _International journal of proteomics_ , vol. 2014, 2014.
  * [2] K. Kadam, S. Sawant, U. Kulkarni-Kale, and V. K. Jayaraman, “Prediction of Protein Function Based on Machine Learning Methods: an Overview,” _genomics III: methods, techniques and applications_ , 2014.
  * [3] J. S Bernardes, “A Review of Protein Function Prediction Under Machine Learning Perspective,” _Recent patents on biotechnology_ , vol. 7, no. 2, pp. 122–141, 2013.
  * [4] C. Szegedy, W. Zaremba, I. Sutskever, J. Bruna, D. Erhan, I. Goodfellow, and R. Fergus, “Intriguing Properties of Neural Networks,” _arXiv preprint arXiv:1312.6199_ , 2013.
  * [5] A. Chakraborty, M. Alam, V. Dey, A. Chattopadhyay, and D. Mukhopadhyay, “A Survey on Adversarial Attacks and Defences,” _CAAI Transactions on Intelligence Technology_ , vol. 6, no. 1, pp. 25–45, 2021.
  * [6] X. Wang, J. Li, X. Kuang, Y.-a. Tan, and J. Li, “The Security of Machine Learning In an Adversarial Setting: A Survey,” _Journal of Parallel and Distributed Computing_ , vol. 130, pp. 12–23, 2019.
  * [7] M. Abadi, P. Barham, J. Chen, Z. Chen, A. Davis, J. Dean, M. Devin, S. Ghemawat, G. Irving, M. Isard _et al._ , “$\\{$TensorFlow$\\}$: a System for $\\{$Large-Scale$\\}$ Machine Learning,” in _12th USENIX symposium on operating systems design and implementation (OSDI 16)_ , 2016, pp. 265–283.
  * [8] A. Lavin and S. Gray, “Fast Algorithms for Convolutional Neural Networks,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 4013–4021.
  * [9] N. J. Higham, _Accuracy and Stability of Numerical Algorithms_. SIAM, 2002.
  * [10] P. Micikevicius, S. Narang, J. Alben, G. Diamos, E. Elsen, D. Garcia, B. Ginsburg, M. Houston, O. Kuchaiev, G. Venkatesh _et al._ , “Mixed Precision Training,” _arXiv preprint arXiv:1710.03740_ , 2017.
  * [11] N. Wang, J. Choi, D. Brand, C.-Y. Chen, and K. Gopalakrishnan, “Training Deep Neural Networks with 8-bit Floating Point Numbers,” _Advances in Neural Information Processing Systems_ , vol. 31, 2018.
  * [12] N. Mellempudi, S. Srinivasan, D. Das, and B. Kaul, “Mixed Precision Training with 8-bit Floating Point,” _arXiv preprint arXiv:1905.12334_ , 2019.
  * [13] D. Kalamkar, D. Mudigere, N. Mellempudi, D. Das, K. Banerjee, S. Avancha, D. T. Vooturi, N. Jammalamadaka, J. Huang, H. Yuen _et al._ , “A Study of BFLOAT16 for Deep Learning Training,” _arXiv preprint arXiv:1905.12322_ , 2019.
  * [14] M. Kulmanov and R. Hoehndorf, “DeepGOPlus: Improved Protein Function Prediction from Sequence,” _Bioinformatics_ , vol. 36, no. 2, pp. 422–429, 2020.
  * [15] D. S. Parker, _Monte Carlo Arithmetic: Exploiting Randomness in Floating-Point Arithmetic_. University of California (Los Angeles). Computer Science Department, 1997.
  * [16] C. Denis, P. D. O. Castro, and E. Petit, “Verificarlo: Checking Floating Point Accuracy through Monte Carlo Arithmetic,” in _2016 IEEE 23nd Symposium on Computer Arithmetic (ARITH)_. Los Alamitos, CA, USA: IEEE Computer Society, jul 2016, pp. 55–62. [Online]. Available: https://doi.ieeecomputersociety.org/10.1109/ARITH.2016.31
  * [17] F. Févotte and B. Lathuilière, “Verrou: Assessing Floating-Point Accuracy Without Recompiling,” 2016.
  * [18] Y. Chatelain, E. Petit, P. d. Oliveira Castro, G. Lartigue, and D. Defour, “Automatic Exploration of Reduced Floating-Point Representations in Iterative Methods,” in _European conference on parallel processing_. Springer, 2019, pp. 481–494.
  * [19] E. Kloberdanz, K. G. Kloberdanz, and W. Le, “DeepStability: A Study of Unstable Numerical Methods and Their Solutions in Deep Learning,” _arXiv preprint arXiv:2202.03493_ , 2022.
  * [20] J. Faraone and P. Leong, “Monte Carlo Deep Neural Network Arithmetic,” 2019\.
  * [21] “Stable Architectures for Deep Neural Networks, author=Haber, Eldad and Ruthotto, Lars,” _Inverse problems_ , vol. 34, no. 1, p. 014004, 2017.
  * [22] M. Li, L. He, and Z. Lin, “Implicit Euler Skip Connections: Enhancing Adversarial Robustness Via Numerical Stability,” in _International Conference on Machine Learning_. PMLR, 2020, pp. 5874–5883.
  * [23] G. Goswami, A. Agarwal, N. Ratha, R. Singh, and M. Vatsa, “Detecting and Mitigating Adversarial Perturbations for Robust Face Recognition,” _International Journal of Computer Vision_ , vol. 127, no. 6, pp. 719–742, 2019.
  * [24] F. Tramèr, A. Kurakin, N. Papernot, I. Goodfellow, D. Boneh, and P. McDaniel, “Ensemble Adversarial Training: Attacks and Defenses,” _arXiv preprint arXiv:1705.07204_ , 2017.
  * [25] A. Goel, A. Agarwal, M. Vatsa, R. Singh, and N. K. Ratha, “DNDNet: Reconfiguring CNN for Adversarial Robustness,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops_ , 2020, pp. 22–23.
  * [26] S. Zheng, Y. Song, T. Leung, and I. Goodfellow, “Improving the Robustness of Deep Neural Networks via Stability Training,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , June 2016\.
  * [27] J. Laermann, W. Samek, and N. Strodthoff, “Achieving Generalizable Robustness of Deep Neural Networks by Stability Training,” in _German conference on pattern recognition_. Springer, 2019, pp. 360–373.
  * [28] S. Gupta, A. Agrawal, K. Gopalakrishnan, and P. Narayanan, “Deep Learning with Limited Numerical Precision,” in _International conference on machine learning_. PMLR, 2015, pp. 1737–1746.
  * [29] C. Sakr, N. Wang, C.-Y. Chen, J. Choi, A. Agrawal, N. Shanbhag, and K. Gopalakrishnan, “Accumulation Bit-Width Scaling for Ultra-Low Precision Training of Deep Networks,” _arXiv preprint arXiv:1901.06588_ , 2019.
  * [30] R. R. Gupta and V. Ranga, “Comparative Study of Different Reduced Precision Techniques in Deep Neural Network,” in _Proceedings of International Conference on Big Data, Machine Learning and their Applications_. Springer, 2021, pp. 123–136.
  * [31] Y. Chatelain, N. Y. S. Young, G. Kiar, and T. Glatard, “PyTracer: Automatically Profiling Numerical Instabilities in Python,” _IEEE Transactions on Computers_ , pp. 1–12, 2022.
  * [32] G. Kiar, P. de Oliveira Castro, P. Rioux, E. Petit, S. T. Brown, A. C. Evans, and T. Glatard, “Comparing Perturbation Models for Evaluating Stability of Neuroimaging Pipelines,” _The International Journal of High Performance Computing Applications_ , vol. 34, no. 5, pp. 491–501, 2020.
  * [33] N. Nethercote and J. Seward, “Valgrind: A Framework for Heavyweight Dynamic Binary Instrumentation,” _ACM Sigplan notices_ , vol. 42, no. 6, pp. 89–100, 2007.
  * [34] D. Sohier, P. D. O. Castro, F. Févotte, B. Lathuilière, E. Petit, and O. Jamond, “Confidence Intervals for Stochastic Arithmetic,” _ACM Transactions on Mathematical Software (TOMS)_ , vol. 47, no. 2, pp. 1–33, 2021\.
  * [35] J. Vignes, “Discrete Stochastic Arithmetic for Validating Results of Numerical Software,” _Numerical Algorithms_ , vol. 37, no. 1, pp. 377–390, 2004.
  * [36] “IEEE Standard for Floating-Point Arithmetic,” _IEEE Std 754-2008_ , pp. 1–70, 2008.
  * [37] Intel, “BFLOAT16 – Hardware Numerics Definition,” Nov 2018. [Online]. Available: https://software.intel.com/sites/default/files/managed/40/8b/bf16-hardware-numerics-definition-white-paper.pdf
  * [38] Microsoft, “MSFP8 - Microsoft.” [Online]. Available: https://en.wikichip.org/wiki/microsoft/msfp8
  * [39] M. Ashburner, C. A. Ball, J. A. Blake, D. Botstein, H. Butler, J. M. Cherry, A. P. Davis, K. Dolinski, S. S. Dwight, J. T. Eppig _et al._ , “Gene Ontology: Tool for the Unification of Biology,” _Nature Genetics_ , vol. 25, no. 1, pp. 25–29, 2000.
  * [40] N. Zhou, Y. Jiang, T. R. Bergquist, A. J. Lee, B. Z. Kacsoh, A. W. Crocker, K. A. Lewis, G. Georghiou, H. N. Nguyen, M. N. Hamid _et al._ , “The CAFA Challenge Reports Improved Protein Function Prediction and New Functional Annotations for Hundreds of Genes Through Experimental Screens,” _Genome Biology_ , vol. 20, no. 1, pp. 1–23, 2019.
  * [41] B. Buchfink, C. Xie, and D. H. Huson, “Fast and Sensitive Protein Alignment Using DIAMOND,” _Nature Methods_ , vol. 12, no. 1, pp. 59–60, 2015.
  * [42] A. Bairoch and R. Apweiler, “The SWISS-PROT Protein Sequence Database and Its Supplement TrEMBL in 2000,” _Nucleic acids research_ , vol. 28, no. 1, pp. 45–48, 2000.
  * [43] Y. Bengio, P. Simard, and P. Frasconi, “Learning Long-Term Dependencies with Gradient Descent Is Difficult,” _IEEE Transactions on Neural Networks_ , vol. 5, no. 2, pp. 157–166, 1994.