# On structural and practical identifiability

Franz-Georg Wieland  Adrian L. Hauber  Marcus Rosenblatt  Christian Tönsing
Jens Timmer<EMAIL_ADDRESS>Institute of Physics, University of
Freiburg, Hermann-Herder-Str. 3, 79104 Freiburg, Germany  Freiburg Center for
Data Analysis and Modelling (FDM), University of Freiburg, Ernst-Zermelo-Str.
1, 79104 Freiburg, Germany  Centre for Integrative Biological Signalling
Studies (CIBSS), University of Freiburg, Schänzlestr. 18, 79104 Freiburg,
Germany

# Flowchart of the entire modelling process

Franz-Georg Wieland  Adrian L. Hauber  Marcus Rosenblatt  Christian Tönsing
Jens Timmer<EMAIL_ADDRESS>Institute of Physics, University of
Freiburg, Hermann-Herder-Str. 3, 79104 Freiburg, Germany  Freiburg Center for
Data Analysis and Modelling (FDM), University of Freiburg, Ernst-Zermelo-Str.
1, 79104 Freiburg, Germany  Centre for Integrative Biological Signalling
Studies (CIBSS), University of Freiburg, Schänzlestr. 18, 79104 Freiburg,
Germany

###### Abstract

We discuss issues of structural and practical identifiability of partially
observed differential equations which are often applied in systems biology.
The development of mathematical methods to investigate structural non-
identifiability has a long tradition. Computationally efficient methods to
detect and cure it have been developed recently. Practical non-identifiability
on the other hand has not been investigated at the same conceptually clear
level. We argue that practical identifiability is more challenging than
structural identifiability when it comes to modelling experimental data. We
discuss that the classical approach based on the Fisher information matrix has
severe shortcomings. As an alternative, we propose using the profile
likelihood, which is a powerful approach to detect and resolve practical non-
identifiability.
Highlights

  * •

With recent advances, identifying structural identifiability is no longer a
major issue

  * •

Practical identifiability is still challenging

  * •

Fisher information matrix is misleading

  * •

Profile likelihood can solve practical identifiability challenge

###### keywords:

Identifiability , Structural identifiability , Practical identifiability ,
Profile likelihood , Fisher information matrix , Non-linear dynamics , ODE
models , Experimental design , Model reduction , Observability

††journal: Current Opinion in Systems Biology

Graphical abstract

## Introduction

### Biological modelling and Box’s statement

Traditional biological reasoning often is rather qualitative, descriptive, and
static, which results e.g. in cell biology in so-called “pathway cartoons”.
Mathematical models based on differential equations can help to turn these
into a quantitative, predictive, and dynamic understanding of the underlying
system. Discussing modelling in general, in 1979, George E.P. Box coined his
famous statement: “All models are wrong, but some are useful” [1]. While the
former part of the quote is intuitively clear, since every model necessarily
poses a simplification of reality, the latter highlights the importance of
assessing what constitutes a _useful_ model.

### Bad, good and useful models

Three properties comprise a _useful_ model. Firstly, it has to capture the
main effects of the question of interest, i.e. describe the data with
reasonable accuracy, and neglect the rest. Secondly, a _useful_ model has to
make experimentally falsifiable predictions in order to be testable. Models
that exhibit these two properties are _good_ models. Thirdly, the model should
enable to gather insights about the biological system. In a typical modelling
process, one starts off with an initial model based on current biological
knowledge. Usually, this model cannot explain the data and therefore is a
_bad_ model. Based on biological intuition and trial-and-error, one increases
the model complexity until the data can be fitted. Often, this leads to an
over-parameterised model that over-fits the data. The parameters of such a
model and in turn its predictions are not well-determined and it thus remains
a _bad_ model.

The path from such a _bad_ model towards a _good_ model is laborious:
additional data needs to be measured and integrated, the model complexity
needs to be reduced and balanced to the available data, or a combination of
both. This process needs to be iterated until a _good_ model is found, which
has well determined parameters and predictions.

However, such a _good_ model also needs to deliver biological insights in
order to be _useful_. Only this third property turns a _good_ model into a
_useful_ model. In this sense, the final goal of mathematical modelling in
systems biology is not the model itself but to use the model to understand
biology. One example of how a model can be used to gain biological insight,
which would be unattainable by merely assessing the data by itself, was given
by Becker et al. [2].

### Parameter identifiability

The concept of identifiability is strongly linked to the transition from _bad_
models to _good_ models. Identifiability analysis is necessary to create
_good_ models that can describe the data and have well-determined parameters
and predictions. It is especially important when modelling biological systems
because the limited amount and quality of the experimental data with large
measurement noise in only partially observed systems often leads to _bad_
models during the modelling process. Concerning identifiability, one
distinguishes between structural identifiability dealing with inherently
indeterminable parameters due to the model structure itself, and practical
identifiability, dealing with insufficiently informative measurements to
determine the parameters with adequate precision.

## Partially observed dynamical systems

A biological system is translated into ordinary differential equations (ODEs)

$\dot{x}=f(x,p,u),$ (1)

comprising $n$ model states $x(t)$, unknown parameters $p$ to be estimated
from time-resolved experimental data, and external stimuli $u(t)$. Since data
is often recorded on a relative scale, scaling and offset parameters for
background corrections need to be estimated in parallel. Furthermore, in
typical applications not all components of a cell-biological system can be
measured, e.g. because of the limited availability or restricted capability of
antibodies to discriminate between un-phosphorylated, i.e. inactive, and
phosphorylated, i.e. active, proteins. Thus, an observation function
$g(\cdot)$ is required that maps the internal states $x$ to the observations:

$y=g(x,p,t).$ (2)

Typically, the dimension $m$ of $y$ is smaller than the dimension $n$ of $x$.
We are therefore dealing with parameter estimation in partially observed
systems. Moreover, in systems biology, these ODE models are typically stiff,
non-linear, sparse and non-autonomous, and the discrete time observations are
noisy.

Parameter estimation is usually performed based on the weighted residual sum
of squares, the negative log-likelihood assuming Gaussian errors

$\chi^{2}_{\text{res}}(p)=\sum^{m}_{k=1}\sum^{d_{k}}_{l=1}\left(\frac{y^{D}_{kl}-g_{k}(p,t_{l})}{\sigma^{D}_{kl}}\right)^{2},$
(3)

to determine the agreement of experimental data with the model trajectories,
where $y^{D}_{kl}$ and $\sigma^{D}_{kl}$ represent $d_{k}$ data points and
measurement errors at time points $t_{l}$ for each observable. A common point
estimate for the best parameter vector is the maximum likelihood estimator

$\hat{p}=\arg\min\left[\chi^{2}_{\text{res}}(p)\right].$ (4)

## Structural identifiability

### Definition of structural identifiability and connection to observability

Partially observed dynamical systems often exhibit structural non-
identifiability. A model is structurally identifiable, if a unique
parameterisation exists for any given model output. A parameter $p_{i}$ is
globally structurally identifiable [3], if for all parameter vectors $p$, it
holds

$y(p)=y(p^{\prime})\hskip 5.69046pt\Rightarrow\hskip
5.69046ptp_{i}=p^{\prime}_{i}\,.$ (5)

An individual parameter $p_{i}$ is structurally non-identifiable, if changing
the parameter does not necessarily alter the model trajectory $y$, because the
changes can be fully compensated by altering other parameters. Local
structural identifiability of a parameter is defined by reducing the
definition to a neighbourhood $v(p)$ instead of the entire parameter space. A
model is structurally identifiable, if all of its parameters are structurally
identifiable. Multiple related definitions for structural identifiability
exist, for a comprehensive discussion see a recent overview [4].

A structurally non-identifiable parameter implies the existence of a manifold
in parameter space upon which the trajectory $y$ is unchanged. However, on
this manifold the dynamic variables $x$ of the model can change, e.g. by a
scaling factor, and are thus not uniquely determinable. This is denoted as
non-observability, a concept closely related to parameter non-identifiability
[5, 6, 7, 8, 9].

### A priori analysis of structural identifiability

Two basic approaches exist to assess structural identifiability of non-linear
dynamic models. _A priori_ methods only use the model definition, while _a
posteriori_ methods use the available data to find non-identifiable
parameters. Many _a priori_ algorithms have been developed based on a variety
of approaches. Powerful methods use Lie group theory, since non-
identifiabilities are closely related to symmetries in the system [10, 11, 12,
13]. Furthermore, a variety of notable methods exist, which are based on power
series expansion [14], generating series [15, 16], semi-numerical approaches
[17, 18], differential algebra [19, 20, 21, 22, 23, 24, 25, 26, 27],
differential geometry [28], and numerical algebraic geometry [29]. For reviews
of some of these approaches, see [28, 30, 31]. Many of these approaches,
especially the early developed methods, can only be applied to rather low-
dimensional systems because of their computational complexity. Thus, recent
developments have mainly focused on improving the computational efficiency of
the algorithms, e.g. by local sensitivity calculations.

As a promising example, Joubert et al. [32] proposed a comprehensive and
computationally fast pipeline to cure structural non-identifiabilities by re-
parameterisation of the model in a five-step procedure: (i) a numerical
identifiability analysis based on sensitivities, (ii) symbolic identifiability
calculations for the low-dimensional candidates from (i), this renders the
procedure fast, (iii) computation of new model parameters, this step is not
unique, but requires decisions of the modeler, (iv) simplify the original
model leading to a lower dimensional parameter vector, and finally (v) check
the identifiability of the re-parameterised model. In an application to a
model with 21 states and 75 parameters, two groups of non-identifiable
parameters were detected and the model was re-parameterised within minutes.

### Analysis of structural identifiability using experimental data

In contrast to the aforementioned methods, _a posteriori_ methods use the
available data to perform identifiability analysis. They infer structural non-
identifiability based on model fits to experimental data. Similar to some
sensitivity-based _a priori_ approaches, these approaches only assess local
structural identifiability.

One approach by Hengl et al. [33] suggested to perform numerous fits and
investigate non-parametrically whether the final parameter estimates form a
low-dimensional manifold in parameter space. This approach also allows to
disentangle different sets of coupled non-identifiable parameters.

An informative and successful method is based on the profile likelihood [34].
The idea of the profile likelihood is to vary one parameter $p_{i}$ after the
other around the maximum likelihood estimate (Equation (4)) and re-optimise
the remaining ones

$\mathrm{PL}\left(p_{i}\right)=\min_{p_{j\neq
i}}\left[\chi^{2}_{\text{res}}\left(p\right)\right]\,.$ (6)

For the two-parameter examples in Figure 1, the blue dashed lines show the
path in the parameter space determined by Equation (6). Figure 1A shows the
profile likelihood of an identifiable parameter. For a structurally non-
identifiable parameter the profile likelihood yields a flat line as shown in
Figure 1B. Plotting the remaining parameters along the profiled parameter
reveals which parameters are coupled to the non-identifiable one [35]. The
profile likelihood was recently extended to include two-dimensional profiles
to allow for the identification of parameter interdependence [36].

Profile likelihood calculation can be computationally demanding for larger
systems due to the numerical re-optimisation. Addressing this issue, a fast _a
posteriori_ method to test identifiability without the need to calculate
complete profiles using radial penalisation was recently developed [37].

Figure 1: Illustrative example of likelihood contour plots and profile
likelihood for an identifiable parameter and structurally and practically non-
identifiable parameters. Subfigures (A), (B), and (C) show contour plots of
$\chi^{2}_{\text{res}}$ above as well as the profile likelihood versus the
parameter below. Lighter colours in the contour plots signify a lower value of
$\chi^{2}_{\text{res}}$. Thresholds for confidence intervals corresponding to
a confidence level of 95 % are shown in red and plotted both in the contour
plots and the profile likelihood plots. The lowest value of
$\chi^{2}_{\text{res}}$ is denoted by a grey asterisk in both the contour plot
and the profile likelihood plot. For the identifiable parameter (A) the
profile likelihood reaches both an upper and lower threshold thus leading to a
finite confidence interval. For the structurally non-identifiable parameter
(B) the profile likelihood is completely flat, thus yielding infinite
confidence intervals. In the contour plot this translates to a flat path,
along which $\chi^{2}_{\text{res}}$ does not change. The practically non-
identifiable parameter (C) shows an infinite extension of the low
$\chi^{2}_{\text{res}}$ region for lower values of the parameter, never
reaching the 95 % confidence interval threshold. In contrast, a finite upper
confidence bound can be derived.

Structural non-identifiability can also be investigated _a posteriori_ by a
Bayesian Markov chain Monte Carlo (MCMC) sampling approach. However, for non-
identifiable systems efficient mixing and thus convergence of the Markov
chains is difficult [38]. This problem can be cured by informative priors but
these would mask the problem and should only be implemented if they are based
on actual biological insights and prior information. One recent application in
the field identified a minimal subset of reactions in a signalling network
with a combination of parallel tempering and LASSO regression methods [39].

### Re-parameterising structurally non-identifiable models

Given the recent advances in the computational efficiency of methods, we
essentially consider determining structural identifiability no longer a
bottleneck in the modelling of non-linear dynamic systems with ODEs. For
models with a high number of connected structurally non-identifiable
parameters, finding and resolving these structural non-identifiabilities can
still be challenging. This is often the case if the number of observed states
is much lower than the number of dynamic states. When the structurally non-
identifiable parameters are determined, the problem is usually fixed by a re-
parameterisation of the model. In the simplest case this is accomplished by
fixing some of the involved parameters to a certain value. The price to be
paid is typically that the information about the scale of some components is
lost which can limit the predictive power of the model. Nevertheless,
biologically meaningful re-parameterisation of the models after finding non-
identifiabilites remains a challenging task (G. Massonis et al. ,
arXiv:2012.09826v2).

## Practical identifiability

### From structural to practical identifiability

Structural identifiability implies practical identifiability only for an
infinite amount of data with zero noise. Practical identifiability is
important for obtaining precise parameter estimates. Moreover, it is
especially crucial to ensure that model predictions are well-determined. It is
analyzed increasingly often to judge a model’s predictivity [40, 41, 42, 43,
44]. The notion of practical identifiability has been rather vague in the
literature, mainly referring to _large confidence intervals_ [45, 46, 47].
Some approaches exist that define practical identifiability as a combination
of model structure and experimental protocol without actual data [48, 49]. In
contrast, we consider a combination of model and data as practically
identifiable if the confidence intervals of all estimated parameters are of
finite size [35].

### Parameter confidence intervals and identifiability

The profile likelihood (Equation (6)) provides a proper assessment of
confidence intervals of estimated parameters in ODE models (Figure 1) by

$\mathit{CI}_{\mathrm{PL}}\left(p_{i}\right)=\bigl{\\{}p_{i}\>|\>\mathrm{PL}\left(p_{i}\right)\leq\chi^{2}_{\text{res}}\left(\hat{p}\right)+\Delta_{\alpha}\bigr{\\}},$
(7)

where $\Delta_{\alpha}$ denotes the $\alpha$ quantile of the $\chi^{2}$
distribution with $\mathit{df}=1$ degrees of freedom for point-wise confidence
intervals [34].

The traditional method for determining confidence intervals based on the
Fisher information matrix (FIM) leads to accurate confidence intervals for
linear regression models. Since the solutions of all nontrivial ODE models are
non-linear in their parameters, using this method for analysing
identifiability of such models is questionable [50]. Furthermore, in contrast
to FIM-based confidence intervals, profile likelihood-based confidence
intervals can be asymmetric and are invariant under re-parameterisations of
the model, e.g. the often applied logarithmic transformation of the
parameters. Figure 2 shows five parameters with FIM- and profile likelihood-
based confidence intervals, mainly taken from applications in synthetic
biology [51, 52].

Figure 2: Parameter confidence intervals based on Fisher information matrix
and profile likelihood. Confidence intervals for five parameters based on
profile likelihood (blue) and on quadratic approximation using the Fisher
information matrix (FIM) (orange). FIM-based confidence intervals have two
major problems. Firstly, due to the non-linearity of the underlying systems,
the Cramér-Rao bound on the error is invalid and thus the FIM-based confidence
intervals become uncontrollable for a finite amount of measurements. While in
(A) the FIM-based interval is larger than the profile likelihood-based
interval, in (B) it is smaller. Secondly, FIM based intervals are insensitive
to practical non-identifiabilities. In (C), the FIM-based confidence interval
is finite and thus the practically non-identifiable parameter is not detected.
In (D), the practically non-identifiable parameter leads to a flat FIM-based
interval, wrongly suggesting structural non-identifiability. While the
structurally non-identifiable parameter in (E) is correctly detected,
similarly to (D) the calculation of the FIM is challenging due to its
singularity in flat likelihood landscapes. The parameters (A)-(D) are adapted
from two applications in synthetic biology ((A),(B),(C) from Schneider et al.
[52], (D) from Ochoa-Fernandez et al. [51]). Parameter (E) is from a minimal
non-identifiable toy model. Grey asterisks signify the maximum likelihood
estimate of the parameter.

Identifiability is obtained if all estimated parameters are structurally and
practically identifiable, i.e. have finite confidence intervals. A non-
identifiable parameter is called practically non-identifiable if the
confidence interval becomes finite for a given confidence level by adding
additional measurements for the existing observables (Figure 1C). By adding
enough data, a practically non-identifiable parameter can be made
identifiable.

### Bayesian methods for identifiability analysis

Bayesian sampling approaches, e.g. MCMC, can be used to assess practical
identifiability [53, 54, 55]. This, however is only feasible if the model is
structurally identifiable, since structural non-identifiabilities will lead to
bad mixing of the sampling algorithms. Given a structurally identifiable
model, MCMC sampling yields similar results as the profile likelihood analysis
[38]. However, a recent application in a spatio-temporal reaction–diffusion
model showed that it is one order of magnitude slower than the profile
likelihood [56]. To the best of our knowledge, a comprehensive benchmark study
comparing the two methods is so far missing.

## Experimental design and model reduction

### Model predictions

To test the predictive power of a model, confidence intervals for the
predictions can be computed. For this purpose, forward evaluations of the
model are utilized, e.g. bootstrap approaches [57] or sensitivity analysis
[58]. They typically require large numerical efforts in the context of non-
linear biological models with a high-dimensional parameter space. A more
powerful approach is the prediction profile likelihood

$\mathrm{PPL}\left(z\right)=\min_{p\in\\{p\>|\>g_{\text{pred}}\left(p\right)=z\\}}\hskip
1.42271pt\left[\chi^{2}_{\text{res}}\left(p\right)\right]\,,$ (8)

which is obtained by minimising $\chi^{2}_{\text{res}}(p)$ (Equation (3))
under the constraint that the model response $g_{\text{pred}}\left(p\right)$
is equal to the prediction $z$. The prediction profile likelihood propagates
the uncertainty from the experimental data to the prediction by exploring the
prediction space instead of the parameter space [59].

Model predictions have to be sufficiently precise to produce insights. For
special cases, this can be achieved without identifiability [60, 61]. If the
model predictions are not of sufficient precision, one has two principal
options to tailor the model complexity to the information content of the data:
(i) measure additional data, corresponding to an increase of the dimension of
the observation function $g$ in Equation (2), or (ii) reduce the model
complexity according to the available data, corresponding to a decrease of the
dimension of the parameter space and/or of the ODE system $f$ in Equation (1).
Both options increase the practical identifiability of the model.

### Achieving practical identifiability by new measurements with optimal
experimental design

Practical identifiability can be achieved by adding new data [44, 62]. The
process of determining the most informative targets and time points for the
new measurements is known as optimal experimental design and is frequently
applied in different modelling fields, e.g. metabolic models [63], animal
science [64], linear perturbation networks [65] or synthetic biology [66]. The
task is related to the search of an additional measurement that contains the
maximal information about the system or parts of it. For improving the
identifiability of a specific parameter, the model trajectories along the
corresponding parameter profile can be investigated [67, 68]. Thereby,
measurement points with maximal information content for the parameter of
interest can be determined, which corresponds to trajectories with high
spread. Similarly, the prediction profile likelihood (Equation (8)) determines
the prediction uncertainty of the model at a potential new measurement time
point [59] thus promoting the identifiability of the whole model. Measurement
points with high prediction uncertainty are effective to constrain the model
further, whereas measurements with a low prediction uncertainty are better
suited for model selection purposes.

### Achieving practical identifiability by reducing model complexity

If measuring additional data is not feasible, the complexity of the model has
to be reduced. One way is to fix parameter values or ratios of parameters by
means of prior knowledge [69], sensitivity analysis [70, 71] or profile
likelihood [72]. However, fixing parameters can decrease the interpretative
relevance of the model’s predictions.

Taking this into account, a systematic model reduction strategy that tailors
model complexity to the available data was suggested by Maiwald et al. [73].
Based on likelihood profiles, they discuss four basic scenarios that are
discriminated based on the profile likelihood by the combinations of: either
(i) the profile flattens out for a logarithmised parameter going to infinity
or (ii) to minus infinity, and either other parameters are (a) coupled to the
investigated one or (b) not. For all four possible combinations, there is a
cure. For case (i/a), one differential equation is replaced by an algebraic
equation, for (i/b), states can be lumped, for (ii/a), a variable is fixed,
leading to a structural non-identifiability that can be cured by the methods
discussed above, and for (ii/b), a reaction can be removed from the model.
This model reduction strategy has been applied e.g. in [51, 52, 74].
Independent of the applied method, model reduction steps, and in particular
the conclusions thereof, should always be documented together with the model
according to good scientific practice to facilitate reproducibility.

## Conclusions

Given the multitude of recently developed methods [13, 16, 27, 32], we
consider the file of identifying structurally non-identifiable parameters as
closed. Future research in this field could focus on identifying biologically
plausible re-parameterisations of the model, for which no comprehensive method
yet exists to our knowledge. Furthermore, the extension of the concept of
identifiability to different model types, e.g. mixed effects models [75, 76],
is of interest.

Achieving practical identifiability for model and data is more laborious in
practice. Practical non-identifiabilities can be detected reliably, e.g. by
the profile likelihood method [31]. In order to achieve identifiability, the
model complexity has to be reduced or additional data must be added. Profile
likelihood-based model reduction [73] and optimal experimental design [68]
provide valuable methods for these purposes. A flowchart locating structural
and practical identifiability analysis as discussed in this review within the
entire modelling process is given in Figure 3.

Figure 3: Flowchart of the entire modelling process from initial to final
model including identifiability analysis. The modelling process begins with
the inception of an initial model based on prior knowledge and the underlying
biological research question. It ends with the final validated model and the
biological insights it provides. The flowchart shows, how identifiability
analysis is embedded into the overall modelling workflow. The topics discussed
in this review related to structural identifiability (blue) and practical
identifiability (red) are highlighted with colours in the flowchart. The
remaining tiles in gray represent aspects that are beyond the scope of this
review. The intricacy of the flowchart shows, that the path to biological
insights requires multiple iterations of different methods. Identifiability
analysis is an integral part of this workflow and should be performed to gain
insights from predictive models with well-determined parameters. Furthermore,
methods dealing with structural and practical identifiability should always be
focused on ultimately progressing along the path towards biological insights.

Although the availability of advanced methods for the detection and cure of
structural and practical non-identifiabilities is promising, two related
challenges remain. In many applications identifiability analysis is not
performed with state-of-the-art methods. Particularly, identifiability
analysis based on the Fisher information matrix can be misleading in typical
applications in systems biology. We propose a more consequent use of the
discussed methods for structural identifiability and especially profile
likelihood for practical identifiability analysis in order to check the
limitations and predictive power of mathematical models. In summary, we
believe the focal point of research in systems biology should always remain on
the biological insights that can be gained from mathematical models which are
structurally and practically identifiable.

## Conflict of interest statement

The authors declare that they have no known competing financial interests or
personal relationships that could have appeared to influence the work reported
in this paper.

## Acknowledgements

This work was supported by the German Research Foundation (DFG) under
Germany’s Excellence Strategy (CIBSS – EXC-2189 – Project ID 390939984), the
German Research Foundation (DFG) through grant 272983813/TRR 179, the Deutsche
Krebshilfe (grant 70112355), the German Federal Ministry for Education and
Research within the research network Systems Medicine of the Liver (LiSyM;
grant 031L0048), and by the state of Baden-Württemberg through bwHPC and the
German Research Foundation (DFG) through grant INST 35/1134-1 FUGG.

The authors thank Hans Stigter for insightful feedback on the manuscript and
Daniel Lill for his help in the writing process.

The graphical abstract and Figure 3 were created with draw.io.

## References

  * Box [1979] G. E. P. Box,  Robustness in the strategy of scientific model building,  in: R. L. Launer, G. N. Wilkinson (Eds.), Robustness in Statistics, Academic Press, New York, 1979, pp. 201–236. doi:10.1016/b978-0-12-438150-6.50018-2.
  * Becker et al. [2010] V. Becker, M. Schilling, J. Bachmann, U. Baumann, A. Raue, T. Maiwald, J. Timmer, U. Klingmüller,  Covering a broad dynamic range: Information processing at the erythropoietin receptor,  Science 328 (2010) 1404–1408. doi:10.1126/science.1184913.
  * Walter and Pronzato [1997] E. Walter, L. Pronzato, Identification of Parametric Models from Experimental Data, Springer, London, U.K., 1997\.
  * Anstett-Collin et al. [2020] F. Anstett-Collin, L. Denis-Vidal, G. Millérioux,  A priori identifiability: An overview on definitions and approaches,  Annual Reviews in Control 50 (2020) 139–149. doi:10.1016/j.arcontrol.2020.10.006.

> *Rigorously assessing 21 definitions of a priori identifiability, the
> authors provide an overview over analytical and algebraic definitions as
> well as local and global definitions and their dependencies.

  * Kalman [1959] R. Kalman,  On the general theory of control systems,  IRE Transactions on Automatic Control 4 (1959) 110–110. doi:10.1109/TAC.1959.1104873.
  * Bellman and Åström [1970] R. Bellman, K. Åström,  On structural identifiability,  Mathematical Biosciences 7 (1970) 329–339. doi:10.1016/0025-5564(70)90132-X.
  * Chappell et al. [1990] M. J. Chappell, K. R. Godfrey, S. Vajda,  Global identifiability of the parameters of nonlinear systems with spezified inputs: a comparison of methods,  Math. Biosciences 102 (1990) 41–73. doi:10.1016/0025-5564(90)90055-4.
  * Villaverde et al. [2019] A. F. Villaverde, N. Tsiantis, J. R. Banga,  Full observability and estimation of unknown inputs, states and parameters of nonlinear biological models,  Journal of The Royal Society Interface 16 (2019) 20190043. doi:10.1098/rsif.2019.0043.

> *A comprehensive approach to observability analysis is provided in this
> paper. This includes assessing inputs, states and parameters. Structural
> parameter identifiability is assessed with an approach based on differential
> geometry.

  * Schmitt and Ritter [2020] T. Schmitt, B. Ritter,  Data-based identifiability and observability assessment for nonlinear control systems using the profile likelihood method,  in: 21st IFAC World Congress, Germany, 2020. doi:10.25534/tuprints-00011713.
  * Yates et al. [2009] J. W. Yates, N. D. Evans, M. J. Chappell,  Structural identifiability analysis via symmetries of differential equations,  Automatica 45 (2009) 2585–2591. doi:10.1016/j.automatica.2009.07.009.
  * Merkt et al. [2015] B. Merkt, D. Kaschek, J. Timmer,  Higher-order Lie symmetries in identifiability and predictability analysis of dynamic models,  Physical Review E 92 (2015) 012920. doi:10.1103/PhysRevE.92.012920.
  * Villaverde et al. [2019] A. F. Villaverde, N. D. Evans, M. J. Chappell, J. R. Banga,  Input-dependent structural identifiability of nonlinear systems,  IEEE Control Systems Letters 3 (2019) 272–277. doi:10.1109/LCSYS.2018.2868608.
  * Massonis and Villaverde [2020] G. Massonis, A. F. Villaverde,  Finding and breaking Lie symmetries: Implications for structural identifiability and observability in biological modelling,  Symmetry 12 (2020) 469. doi:10.3390/sym12030469.

> **The authors provide an improved algorithm to find model symmetries using
> Lie algebra that is computationally more efficient and thus applicable to
> higher dimensional models.

  * Pohjanpalo [1978] H. Pohjanpalo,  System identifiability based on the power series expansion of the solution,  Mathematical Biosciences 41 (1978) 21–33. doi:10.1016/0025-5564(78)90063-9.
  * Walter and Lecourtier [1982] E. Walter, Y. Lecourtier,  Global approaches to identifiability testing for linear and nonlinear state space models,  Mathematics and Computers in Simulation 24 (1982) 472–482. doi:10.1016/0378-4754(82)90645-0.
  * Ligon et al. [2018] T. S. Ligon, F. Fröhlich, O. T. Chiş, J. R. Banga, E. Balsa-Canto, J. Hasenauer,  GenSSI 2.0: Multi-experiment structural identifiability analysis of SBML models,  Bioinformatics 34 (2018) 1421–1423. doi:10.1093/bioinformatics/btx735.

> *The authors provide an open-source Matlab toolbox for structural
> identifiability analysis and the reparametrization of structurally non-
> identifiable models. They use the generating series approach and support
> Systems Biology Markup Language import.

  * Sedoglavic [2002] A. Sedoglavic,  A probabilistic algorithm to test local algebraic observability in polynomial time,  Journal of Symbolic Computation 33 (2002) 735–755. doi:10.1006/jsco.2002.0532.
  * Karlsson et al. [2012] J. Karlsson, M. Anguelova, M. Jirstrand,  An efficient method for structural identifiability analysis of large dynamic systems,  IFAC Proceedings Volumes 45 (2012) 941–946. doi:10.3182/20120711-3-BE-2027.00381.
  * Ljung and Glad [1994] L. Ljung, T. Glad,  On global identifiability for arbitrary model parametrizations,  Automatica 30 (1994) 265–276. doi:10.1016/0005-1098(94)90029-9.
  * Saccomani et al. [2003] M. P. Saccomani, S. Audoly, L. D’Angiò,  Parameter identifiability of nonlinear systems: The role of initial conditions,  Automatica 39 (2003) 619–632. doi:10.1016/S0005-1098(02)00302-3.
  * Bellu et al. [2007] G. Bellu, M. P. Saccomani, S. Audoly, L. D’Angiò,  DAISY: A new software tool to test global identifiability of biological and physiological systems,  Computer Methods and Programs in Biomedicine 88 (2007) 52–61. doi:10.1016/j.cmpb.2007.07.002.
  * Meshkat et al. [2014] N. Meshkat, C. E.-Z. Kuo, J. DiStefano,  On finding and using identifiable parameter combinations in nonlinear dynamic systems biology models and COMBOS: A novel web implementation,  PLoS ONE 9 (2014) e110261. doi:10.1371/journal.pone.0110261.
  * Thomaseth and Saccomani [2018] K. Thomaseth, M. P. Saccomani,  Local identifiability analysis of nonlinear ODE models: How to determine all candidate solutions,  IFAC-PapersOnLine 51 (2018) 529–534. doi:10.1016/j.ifacol.2018.03.089.
  * Varghese et al. [2018] A. Varghese, S. Narasimhan, N. Bhatt,  A priori parameter identifiability in complex reaction networks,  IFAC-PapersOnLine 51 (2018) 760–765. doi:10.1016/j.ifacol.2018.09.162.
  * Saccomani et al. [2019] M. P. Saccomani, G. Bellu, S. Audoly, L. D’Angió,  A new version of DAISY to test structural identifiability of biological models,  in: Computational Methods in Systems Biology, 2019, pp. 329–334. doi:10.1007/978-3-030-31304-3_21.
  * Hong et al. [2019] H. Hong, A. Ovchinnikov, G. Pogudin, C. Yap,  SIAN: Software for structural identifiability analysis of ODE models,  Bioinformatics 35 (2019) 2873–2874. doi:10.1093/bioinformatics/bty1069.
  * Hong et al. [2020] H. Hong, A. Ovchinnikov, G. Pogudin, C. Yap,  Global identifiability of differential models,  Communications on Pure and Applied Mathematics 73 (2020) 1831–1879. doi:10.1002/cpa.21921.

> **Combining a probabilistic and analytical criterion in one algorithm, the
> authors perform a priori structural identifiability analysis efficiently and
> quickly. The algorithm is available in the SIAN Maple software package.

  * Villaverde [2019] A. F. Villaverde,  Observability and structural identifiability of nonlinear biological systems,  Complexity 2019 (2019) 1–12. doi:10.1155/2019/8497093.
  * Bates et al. [2019] D. J. Bates, J. D. Hauenstein, N. Meshkat,  Identifiability and numerical algebraic geometry,  PLOS ONE 14 (2019) e0226299. doi:10.1371/journal.pone.0226299.
  * Chis et al. [2011] O. T. Chis, J. R. Banga, E. Balsa-Canto,  Structural identifiability of systems biology models: A critical comparison of methods,  PLoS ONE 6 (2011) e27755. doi:10.1371/journal.pone.0027755.
  * Raue et al. [2014] A. Raue, J. Karlsson, M. P. Saccomani, M. Jirstrand, J. Timmer,  Comparison of approaches for parameter identifiability analysis of biological systems,  Bioinformatics 30 (2014) 1440–1448. doi:10.1093/bioinformatics/btu006.
  * Joubert et al. [2020] D. Joubert, J. D. Stigter, J. Molenaar,  An efficient procedure to assist in the re-parametrization of structurally unidentifiable models,  Mathematical Biosciences 323 (2020) 108328. doi:10.1016/j.mbs.2020.108328.

> **Using a combination of numerical and algebraic methods, the authors
> provide an incredibly fast algorithm for assessing structural
> identifiability. Their approach uses sensitivity analysis to provide
> candidates for further symbolic analysis, before computing new algebraic
> relations for the non-identifiable parameters.

  * Hengl et al. [2007] S. Hengl, C. Kreutz, J. Timmer, T. Maiwald,  Data-based identifiability analysis of non-linear dynamical models,  Bioinformatics 23 (2007) 2612–2618. doi:10.1093/bioinformatics/btm382.
  * Murphy and van der Vaart [2000] S. A. Murphy, A. W. van der Vaart,  On profile likelihood,  Journal of the American Statistical Association 95 (2000) 449. doi:10.2307/2669386.
  * Raue et al. [2009] A. Raue, C. Kreutz, T. Maiwald, J. Bachmann, M. Schilling, U. Klingmüller, J. Timmer,  Structural and practical identifiability analysis of partially observed dynamical models by exploiting the profile likelihood,  Bioinformatics 25 (2009) 1923–1929. doi:10.1093/bioinformatics/btp358.
  * Brastein et al. [2019] O. M. Brastein, B. Lie, R. Sharma, N.-O. Skeie,  Parameter estimation for externally simulated thermal network models,  Energy and Buildings 191 (2019) 200–210. doi:10.1016/j.enbuild.2019.03.018.

> *Extension of the profile likelihood method for 2D likelihood profiles that
> allow for analysis of pairwise parameter interdependence.

  * Kreutz [2018] C. Kreutz,  An easy and efficient approach for testing identifiability,  Bioinformatics 34 (2018) 1913–1921. doi:10.1093/bioinformatics/bty035.

> **Fast alternative method to test structural and practical identifiability
> without the need to calculate the profile likelhood for each model
> parameter.

  * Raue et al. [2013] A. Raue, C. Kreutz, F. J. Theis, J. Timmer,  Joining forces of Bayesian and frequentist methodology: A study for inference in the presence of non-identifiability,  Phil. Trans. Roy. Soc. A 371 (2013) 20110544. doi:10.1098/rsta.2011.0544.
  * Gupta et al. [2020] S. Gupta, R. E. C. Lee, J. R. Faeder,  Parallel tempering with LASSO for model reduction in systems biology,  PLOS Computational Biology 16 (2020) e1007669. doi:10.1371/journal.pcbi.1007669.
  * Busch et al. [2015] K. Busch, K. Klapproth, M. Barile, M. Flossdorf, T. Holland-Letz, S. M. Schlenner, M. Reth, T. Höfer, H.-R. Rodewald,  Fundamental properties of unperturbed haematopoiesis from stem cells in vivo,  Nature 518 (2015) 542–546. doi:10.1038/nature14242.
  * David and Ricard [2019] I. David, A. Ricard,  A unified model for inclusive inheritance in livestock species,  Genetics 212 (2019) 1075–1099. doi:10.1534/genetics.119.302375.
  * Duchesne et al. [2019] R. Duchesne, A. Guillemin, F. Crauste, O. Gandrillon,  Calibration, selection and identifiability analysis of a mathematical model of the in vitro erythropoiesis in normal and perturbed contexts,  In Silico Biology 13 (2019) 55–69. doi:10.3233/ISB-190471.
  * Zhou et al. [2020] W. Zhou, R. Huang, K. Liu, W. Zhang,  A novel interval‐based approach for quantifying practical parameter identifiability of a lithium‐ion battery model,  International Journal of Energy Research 44 (2020) 3558–3573. doi:10.1002/er.5118.
  * Johnson et al. [2020] K. E. Johnson, G. R. Howard, D. Morgan, E. A. Brenner, A. L. Gardner, R. E. Durrett, W. Mo, A. Al’Khafaji, E. D. Sontag, A. M. Jarrett, T. E. Yankeelov, A. Brock,  Integrating transcriptomics and bulk time course data into a mathematical framework to describe and predict therapeutic resistance in cancer,  Physical Biology 18 (2020) 016001. doi:10.1088/1478-3975/abb09c.

> *The authors emphasise practical identifiability as a central requirement
> for the validity of their model predictions. It is achieved by inclusion of
> single-cell RNA-Seq data into the ODE model via a machine learning
> algorithm.

  * Nihtilä and Virkkunen [1977] M. Nihtilä, J. Virkkunen,  Practical identifiability of growth and substrate consumption models,  Biotechnology and Bioengineering 19 (1977) 1831–1850. doi:10.1002/bit.260191208.
  * Holmberg [1982] A. Holmberg,  On the practical identifiability of microbial growth models incorporating Michaelis-Menten type nonlinearities,  Mathematical Biosciences 62 (1982) 23–43. doi:10.1016/0025-5564(82)90061-X.
  * Miao et al. [2011] H. Miao, X. Xia, A. S. Perelson, H. Wu,  On identifiability of nonlinear ODE models and Aapplications in viral dynamics,  SIAM Review 53 (2011) 3–39. doi:10.1137/090757009.
  * Gontier and Pfister [2020] C. Gontier, J.-P. Pfister,  Identifiability of a binomial synapse,  Frontiers in Computational Neuroscience 14 (2020). doi:10.3389/fncom.2020.558477.

> *An alternative definition of practical identifiability is provided by
> Gontier and Pfister which is independent of the specific data realization.
> This method allows to study how practical identifiability is affected by
> different experimental protocols. However, it requires a nested hierarchy of
> competing models which is not clear a priori for complex models as used
> frequently in systems biology.

  * Saccomani and Thomaseth [2018] M. P. Saccomani, K. Thomaseth,  The union between structural and practical identifiability makes strength in reducing oncological model complexity: A case study,  Complexity 2018 (2018) 1–10. doi:10.1155/2018/2380650.
  * Neale and Miller [1997] M. C. Neale, M. B. Miller,  The use of likelihood-based confidence intervals in genetic models,  Behavior Genetics 27 (1997) 113–120. doi:10.1023/A:1025681223921.
  * Ochoa-Fernandez et al. [2020] R. Ochoa-Fernandez, N. B. Abel, F.-G. Wieland, J. Schlegel, L.-A. Koch, J. B. Miller, R. Engesser, G. Giuriani, S. M. Brandl, J. Timmer, W. Weber, T. Ott, R. Simon, M. D. Zurbriggen,  Optogenetic control of gene expression in plants in the presence of ambient white light,  Nature Methods 17 (2020) 717–725. doi:10.1038/s41592-020-0868-y.
  * Schneider et al. [2021] N. Schneider, F.-G. Wieland, D. Kong, A. A. M. Fischer, M. Hörner, J. Timmer, H. Ye, W. Weber,  Liquid-liquid phase separation of light-inducible transcription factors increases transcription activation in mammalian cells and mice,  Science Advances 7 (2021) eabd3568. doi:10.1126/sciadv.abd3568.
  * Siekmann et al. [2012] I. Siekmann, J. Sneyd, E. J. Crampin,  MCMC can detect nonidentifiable models,  Biophysical Journal 103 (2012) 2275–2286. doi:10.1016/j.bpj.2012.10.024.
  * Hines et al. [2014] K. E. Hines, T. R. Middendorf, R. W. Aldrich,  Determination of parameter identifiability in nonlinear biophysical models: A Bayesian approach,  Journal of General Physiology 143 (2014) 401–416. doi:10.1085/jgp.201311116.
  * Zuo et al. [2019] Y. Zuo, S. Sarkar, M. T. Corwin, K. Olson, R. D. Badawi, G. Wang,  Structural and practical identifiability of dual-input kinetic modeling in dynamic PET of liver inflammation,  Physics in Medicine & Biology 64 (2019) 175023. doi:10.1088/1361-6560/ab1f29.
  * Simpson et al. [2020] M. J. Simpson, R. E. Baker, S. T. Vittadello, O. J. Maclaren,  Practical parameter identifiability for spatio-temporal models of cell invasion,  Journal of The Royal Society Interface 17 (2020) 20200055. doi:10.1098/rsif.2020.0055.

> *By means of spatio-temporal models, the authors show that profile
> likelihood leads to the same results as the established MCMC approach while
> being one order of magnitude faster to compute.

  * Joshi et al. [2006] M. Joshi, A. Seidel-Morgenstern, A. Kremling,  Exploiting the bootstrap method for quantifying parameter confidence intervals in dynamical systems,  Metabolic Engineering 8 (2006) 447–455. doi:10.1016/j.ymben.2006.04.003.
  * Sachs [1984] L. Sachs, Applied Statistics, Springer Series in Statistics, Springer New York, New York, NY, 1984. doi:10.1007/978-1-4612-5246-7.
  * Kreutz et al. [2012] C. Kreutz, A. Raue, J. Timmer,  Likelihood based observability analysis and confidence intervals for predictions of dynamic models,  BMC Systems Biology 6 (2012) 120. doi:10.1186/1752-0509-6-120.
  * Cedersund [2016] G. Cedersund,  Prediction uncertainty estimation despite unidentifiability: An overview of recent developments,  in: Uncertainty in Biology, 2016, pp. 449–466. doi:10.1007/978-3-319-21296-8_17.
  * Rateitschak et al. [2012] K. Rateitschak, F. Winter, F. Lange, R. Jaster, O. Wolkenhauer,  Parameter identifiability and sensitivity analysis predict targets for enhancement of STAT1 activity in pancreatic cancer and stellate cells,  PLoS Computational Biology 8 (2012) e1002815. doi:10.1371/journal.pcbi.1002815.
  * Bhonsale et al. [2020] S. S. Bhonsale, B. Stokbroekx, J. Van Impe,  Assessment of the parameter identifiability of population balance models for air jet mills,  Computers & Chemical Engineering 143 (2020) 107056. doi:10.1016/j.compchemeng.2020.107056.
  * Frøysa et al. [2020] H. G. Frøysa, H. J. Skaug, G. Alendal,  Experimental design for parameter estimation in steady-state linear models of metabolic networks,  Mathematical Biosciences 319 (2020) 108291. doi:10.1016/j.mbs.2019.108291.
  * Muñoz-Tamayo et al. [2018] R. Muñoz-Tamayo, L. Puillet, J. B. Daniel, D. Sauvant, O. Martin, M. Taghipoor, P. Blavy,  Review: To be or not to be an identifiable model. Is this a relevant question in animal science modelling?,  Animal 12 (2018) 701–712. doi:10.1017/S1751731117002774.
  * Gross and Blüthgen [2020] T. Gross, N. Blüthgen,  Identifiability and experimental design in perturbation studies,  Bioinformatics 36 (2020) i482–i489. doi:10.1093/bioinformatics/btaa404.
  * Bandiera et al. [2020] L. Bandiera, D. Gomez-Cabeza, J. Gilman, E. Balsa-Canto, F. Menolascina,  Optimally designed model selection for synthetic biology,  ACS Synthetic Biology 9 (2020) 3134–3144. doi:10.1021/acssynbio.0c00393.
  * Raue et al. [2010] A. Raue, V. Becker, U. Klingmüller, J. Timmer,  Identifiability and observability analysis for experimental design in nonlinear dynamical models,  Chaos 20 (2010) 45105\. doi:10.1063/1.3528102.
  * Steiert et al. [2012] B. Steiert, A. Raue, J. Timmer, C. Kreutz,  Experimental design for parameter estimation of gene regulatory networks,  PLoS ONE 7 (2012) e40052. doi:10.1371/journal.pone.0040052.
  * Tivay et al. [2020] A. Tivay, X. Jin, A. K.-Y. Lo, C. G. Scully, J.-O. Hahn,  Practical use of regularization in individualizing a mathematical model of cardiovascular hemodynamics using scarce data,  Frontiers in Physiology 11 (2020). doi:10.3389/fphys.2020.00452.
  * He and Wang [2020] Q. P. He, J. Wang,  Application of systems engineering principles and techniques in biological big data analytics: A review,  Processes 8 (2020) 951. doi:10.3390/pr8080951.
  * Zi [2011] Z. Zi,  Sensitivity analysis approaches applied to systems biology models,  IET Systems Biology 5 (2011) 336–346. doi:10.1049/iet-syb.2011.0015.
  * Pironet et al. [2019] A. Pironet, P. D. Docherty, P. C. Dauby, J. G. Chase, T. Desaive,  Practical identifiability analysis of a minimal cardiovascular system model,  Computer Methods and Programs in Biomedicine 171 (2019) 53–65. doi:10.1016/j.cmpb.2017.01.005.
  * Maiwald et al. [2016] T. Maiwald, H. Hass, B. Steiert, J. Vanlier, R. Engesser, A. Raue, F. Kipkeew, H. H. Bock, D. Kaschek, C. Kreutz, J. Timmer,  Driving the model to its limit: Profile likelihood based model reduction,  PLoS ONE 11 (2016) e0162366. doi:10.1371/journal.pone.0162366.
  * Tönsing et al. [2018] C. Tönsing, J. Timmer, C. Kreutz,  Profile likelihood-based analyses of infectious disease models,  Statistical Methods in Medical Research 27 (2018) 1979–1998. doi:10.1177/0962280217746444.

> *This study applies profile likelihood-based identifiability analysis to
> infectious diseases models, which are based on ODE models similarly to
> systems biology context. Using profile likelihood, a stepwise data-based
> model reduction procedure is shown for a Zika virus model.

  * Cucurull‐Sanchez et al. [2019] L. Cucurull‐Sanchez, M. J. Chappell, V. Chelliah, S. Y. Amy Cheung, G. Derks, M. Penney, A. Phipps, R. S. Malik‐Sheriff, J. Timmis, M. J. Tindall, P. H. Graaf, P. Vicini, J. W. T. Yates,  Best practices to maximize the use and reuse of quantitative and systems pharmacology models: Recommendations from the United Kingdom quantitative and systems pharmacology network,  CPT: Pharmacometrics & Systems Pharmacology 8 (2019) 259–272. doi:10.1002/psp4.12381.
  * Janzén et al. [2019] D. L. Janzén, M. Jirstrand, M. J. Chappell, N. D. Evans,  Three novel approaches to structural identifiability analysis in mixed-effects models,  Computer Methods and Programs in Biomedicine 171 (2019) 141–152. doi:10.1016/j.cmpb.2016.04.024.


# Performance Comparison of DAOS and Lustre for Object Data Storage Approaches

Nicolau Manubens  European Centre for Medium-Range
Weather Forecasts
Bonn, Germany
<EMAIL_ADDRESS>Simon D. Smart  European Centre for Medium-Range
Weather Forecasts
Reading, United Kingdom
<EMAIL_ADDRESS>Tiago Quintino  Adrian Jackson  European Centre for
Medium-Range
Weather Forecasts
Reading, United Kingdom
<EMAIL_ADDRESS>EPCC, The University of Edinburgh
Edinburgh, United Kingdom
<EMAIL_ADDRESS>
###### Abstract

High-performance object stores are an emerging technology which offers an
alternative solution in the field of HPC storage, with potential to address
long-standing scalability issues in traditional distributed POSIX file systems
due to excessive consistency assurance and metadata prescriptiveness.

In this paper we assess the performance of storing object-like data within a
standard file system, where the configuration and access mechanisms have not
been optimised for object access behaviour, and compare with and investigate
the benefits of using an object storage system.

Whilst this approach is not exploiting the file system in a standard way, this
work allows us to investigate whether the underlying storage technology
performance is more or less important than the software interface and
infrastructure a file system or object store provides.

###### Index Terms:

scalable object storage, next-generation I/O, storage class memory, numerical
weather prediction, DAOS, Lustre

##  I Introduction

Object stores are a candidate to address long-standing scalability issues in
POSIX file systems, including excessive consistency assurance and
prescriptiveness[1].

Numerical Weather Prediction (NWP) usually entails object-like data access, as
global weather fields are currently of the order of 1 MiB in size, relatively
small if compared to traditional high-performance I/O sizes, and an advanced
indexing mechanism is required for high-performance semantic discovery and
access, which involves several metadata operations. Domain-specific object
stores have been developed to implement this semantic indexing on traditional
distributed file systems in a way that satisfies current operational NWP
performance requirements[2].

With the planned resolution increases in NWP simulations, resulting in one to
two orders of magnitude larger data sets, and the advent of general-purpose
high-performance object stores, which are specially designed for the type of
object-like operations common in NWP, adapting the domain-specific store
currently in use at ECMWF becomes an increasingly appealing pathway. In a
recent study, as prior research to validate such effort, we have assessed the
performance that DAOS, a high-performance object store which has been recently
gaining traction, can provide together with Storage Class Memory (SCM) when
tested with an ad-hoc benchmark which mimics I/O patterns in our operational
NWP use case[3].

In this paper we review these DAOS performance results, and compare with
corresponding performance results obtained using Lustre, one of the most
popular distributed file systems in HPC, after adapting the ad-hoc benchmark
to carry out the object operations on top of a file system.

This work allows us to discriminate the benefits achieved from using specific
storage hardware as compared to the benefits from the object store design and
implementation. It also allows us to draw conclusions of general interest on
the benefits of object stores, and gives some real use-case same-hardware
same-software data points for comparison of Lustre and DAOS.

##  II DAOS

The Distributed Asynchronous Object Store (DAOS)[4] is an open-source high-
performance object store designed for massively distributed non-volatile
memory (NVM) including SCM and NVMe. It provides a low-level key-value storage
interface on top of which other higher-level APIs, also provided by DAOS, are
built. Its features include transactional non-blocking I/O, fine-grained I/O
operations with zero-copy I/O to SCM, end-to-end data integrity and advanced
data protection. The OpenFabrics Interfaces (OFI) library is used for low-
latency communications over a wide range of network back-ends.

DAOS is deployable as a set of I/O processes or engines, one per physical
socket in a server node, each managing access to SCM and NVMe devices within
the socket. An engine partitions the storage it manages into targets to
optimize concurrency, each target being managed and exported by a dedicated
group of threads. DAOS allows reserving space distributed across targets in
so-called pools, a form of virtual storage. A pool can serve multiple
transactional object stores called containers.A container is a private object
address space, which can be modified transactionally and independently of the
other containers in the same pool. An application first needs to connect to
the pool and then open the desired container. If successfully authorised, the
application obtains a handle it can use for its processes to interact with the
container.

Upon creation, objects in a container are assigned a 128-bit unique object
identifier, of which 96 bits are user-managed. Objects can be configured for
replication and striping across pool targets by specifying their object class.
An object configured with striping is stored in parts, distributed across
targets, enabling concurrent access.

##  III Lustre

Lustre[5] is the foremost parallel file system used at HPC site globally. It
is an open-source file system, that aims to provide high bandwidth and high
availability for many users across a wide range of hardware. Lustre provides a
POSIX-compliant interface to distributed data storage that enables large
numbers of clients to connect and use the file system concurrently.

##  IV Methodology

In this work, DAOS and Lustre have been deployed on the NEXTGenIO research HPC
system, exploiting the same underlying hardware, and the deployments have been
benchmarked with the community-developed IOR benchmark[6] and the Field I/O
benchmark.

In brief, the Field I/O benchmark consists of a pair of functions that perform
writing and reading of weather fields to and from a DAOS object store, using
the DAOS C API. Their design closely mimics the domain-specific object store
already employed within ECWMF, and they can be combined and run in parallel in
different ways, resulting in two different data access patterns of interest:

  * •

pattern A: in a first phase, writer programs are run on a number client nodes
(typically more than one per node), and issue a sequence of write operations.
Once all writers have finished, a second phase runs, where an equal number of
reader programs are executed, issuing a sequence of corresponding read
operations. This pattern aims to assess the maximum write or read throughput
the storage can provide to applications, mimicking a scenario where the NWP
writer applications are run separately from the post-processing reader
applications.

  * •

pattern B: firstly, the storage is pre-populated with some data. Following
this initialisation step, half of the client nodes employed for the benchmark
issue a sequence of write operations, while the other half issue corresponding
read operations. This pattern aims to assess the throughput storage can
provide in more realistic scenarios with contention between writing and
reading processes.

Due to its design, Field I/O is representative of the types of I/O workload
exhibited from real NWP workflows. Typically, operational NWP workflows at
ECMWF operate as in pattern B, with approximately 250 HPC nodes writing
simulation output to storage while another 250 nodes read and run post-
processing tasks. However many users also run the forecasting models or post-
processing applications independently in a manner that is equivalent to
pattern A.

Field I/O is described in more detail in[3] along with a methodology to assess
object storage performance for NWP applications, which is also adopted here.
Following that methodology, the IOR benchmark has been configured so that each
I/O process issues a single, large, I/O operation comprising a sequence of
data parts (we refer to this mode of operation as segments mode). This enables
assessment of the maximum achievable performance if the developed application
were optimised to gather and transfer all relevant data in a single I/O
operation, and provides insight on to what extent the storage server is able
to exploit available network bandwidth and/or storage capability when not
having to deal with new operations.

In order to execute the object-store-oriented Field I/O benchmark against
POSIX Lustre, a helper library has been developed which implements the DAOS
API using POSIX file system concepts. DAOS Pools have been implemented as
directories hosted on the distributed file system. Containers have been
implemented as directories under the corresponding pool directories. Key-Value
objects have been implemented as directories under the corresponding container
directory. A key is implemented as a file in the Key-Value directory, named
with the key name, and a value is stored as content in that key file. Array
objects are implemented as files under the corresponding container directory,
named with the object ID of the Array and containing the array data.

Following this design, every write of a weather field performed by the Field
I/O benchmark using the helper library will usually involve: a) write of an
Array file in a Container directory exclusive for every client process, b)
check existence of a Key-Value directory in a Container directory shared with
all processes in the client node, c) creation and write of a Key file in a
Key-Value directory shared with all processes in the client node.

A field read will usually involve: a) check existence of a Key-Value directory
in a Container directory shared with all client processes, b) open and read of
a Key file in a Key-Value directory shared by all client processes, c) check
existence of a Key-Value directory in a Container directory shared with all
processes in the client node, d) open and read of a Key file in a Key-Value
directory shared with all processes in the client node, e) open and read an
Array file in a Container directory exclusive for the process.

Whereas the bandwidth metric used to quantify performance in IOR runs has been
the one provided by IOR itself, referred to as synchronous bandwidth here, a
custom bandwidth metric for non-synchronised applications is used in Field I/O
runs, the global timing bandwidth. Both metrics are described in[3].

###  IV-A NEXTGenIO

The benchmarks we present have been conducted in NEXTGenIO[7], a research HPC
system composed of 34 dual-socket nodes with Intel Xeon Cascade Lake
processors. Each socket has six 256 GiB first-generation Intel’s Optane Data
Centre Persistent Memory Modules (DCPMMs) configured in AppDirect interleaved
mode, although there are no NVMe devices. Each processor is connected with its
own integrated network adapter to a low-latency OmniPath fabric. Each of these
adapters has a maximum bandwidth of 12.5 GiB/s.

For the Lustre benchmarking, Lustre has been deployed on 8 storage nodes
(providing 16 OSTs, one per socket), plus one node devoted to the metadata
service. Both the OSTs and the MDTs used in the file system mount an ext4 file
system on the SCM attached to their respective sockets, providing 1.5 TB of
high-performance storage per OST and MDT.

To run the benchmarks against configurable amounts of Lustre storage nodes,
Lustre pools have been set up with 1, 2, 4 and 8 nodes. Before running a test,
a folder is created in the file system where all test files will be generated,
and the `setstripe` command is used to bind that folder to the pool with the
desired number of storage nodes.

DAOS deployments have been conducted separately in an ad-hoc basis, removing
and re-deploying with the desired amount of storage nodes as needed. Each node
used for DAOS storage deploys a single DAOS engine per socket, using the full
ext4 file system on the Optane SCM for that socket, and has access to all the
cores available on the associated processor. For instance, to compare to a
two-node Lustre configuration (which uses two OST nodes and one MDT node,
giving 4 OSTs overall), a two-node DAOS deployment is created and provisioned
for the benchmark.

Up to 16 nodes were employed to execute the benchmark client processes using
both sockets and network interfaces. SCM in the client nodes was not used and
did not have any effect on I/O performance.

##  V Results

Performance results obtained from running the described benchmarks against
Lustre and DAOS deployments in NEXTGenIO are discussed next, starting with an
assessment of potential achievable performance with IOR, following with real
achieved performance with Field I/O without and with contention between writer
and reader applications, and concluding with an assessment of the impact of
file or object size in such applications.

###  V-A IOR results, potential performance

In this test set, the IOR benchmark has been run against different amounts of
Lustre and DAOS storage server nodes with the intent to analyse maximum write
and read performance a client application can potentially achieve, and to
analyse the behaviour of the storage performance as more server nodes are
added. IOR has been run in segments mode as described above, following access
pattern A.

For each server node count the benchmark has been run using as many client
nodes, twice as many and, where possible, four times as many, with the aim to
effectively make use of the available bandwidth theoretically provided by the
server network interfaces. For each of these combinations, the benchmark has
been run with 36, 48, 72 and 96 IOR processes per client node as these were
found to result in the best performance in preliminary tests, for both Lustre
and DAOS. Each run has been repeated 5 times to account for variability.

The segment count has been set to 100 as it was found to be the minimum
segment count that results in reduced bandwidth variability. The segment size
has been set to 1 MiB, to match the object or field size in the NWP use case.
This segment configuration results in files or objects of 100 MiB in size
being written and read during the benchmark runs.

The results are shown in Fig. 1. Each dot represents the mean synchronous
bandwidth obtained with the best performing number of client nodes and IOR
process count for the corresponding server node count. Hollow dots are used to
indicate cases where it was only possible to use up to twice as many client
nodes as server nodes, not four times as many as in the rest of the
benchmarks.

(a) Write

(b) Read

Figure 1: Mean synchronous write and read bandwidth results for Lustre and
DAOS for access pattern A with IOR in segments mode.

Lustre and DAOS perform similarly, resulting in comparable write and read
benchmark bandwidths, with a similar scaling pattern. These results indicate
that both storage servers and the benchmarks have been properly configured to
exploit available storage and network resources, which has been further
verified with monitoring of resource usage during preliminary test runs.

Slightly higher bandwidths are achieved overall with Lustre in this scenario,
except in the read phase in the configuration with 8 server nodes, which was
tested with only up to twice as many client nodes running the benchmark. The
performance limitation in that configuration is explained by the fact that, in
our test platform and with our configuration, four times as many client nodes
as Lustre server nodes are required to exploit all available server interface
bandwidth whereas, with DAOS, only twice as many client nodes are required.
Results showing optimal client to server node ratio with Lustre have been
omitted in favour of space. For DAOS, this ratio is addressed in[3].

Excluding this special case, write and read IOR bandwidths obtained with both
Lustre and DAOS scale linearly. For Lustre, the benchmark bandwidth increases
at a rate of approximately 6 GiB/s for write and 9 GiB/s for read per
additional server node. For DAOS, the increase is of approximately 5 GiB/s for
write and 7.5 GiB/s for read.

###  V-B Field I/O results

####  V-B1 Application performance with DAOS vs. Lustre

The Field I/O benchmark, in contrast to the IOR benchmark in segments mode,
entails object-store-like I/O operations. The data elements are small, and
writing or reading a single data object usually involves multiple I/O
operations.

In this test set, Field I/O has been run with access patterns A and B
following a similar strategy to the IOR tests above, using both Lustre and
DAOS with the similar amounts of server and client nodes. In the runs here, a
maximum of twice as many client nodes as server nodes have been employed for
all configurations.

The benchmark has been run in its default full mode, and also in a mode which
avoids use of DAOS containers called no-containers[3]. When run against
Lustre, using the helper library to adapt Field I/O to POSIX file systems,
every field write in the benchmark in no-containers mode will involve less
metadata operations but the Array files will be written to a single directory
shared by all client processes. Field reads will equally read Array files from
that shared directory.

The object size has been set to 1 MiB, and the number of I/O iterations per
process to 2000, to reduce the impact of potential parallel process start-up
delays on bandwidth measurements. The chosen amounts of processes per client
node, which have been found to result in best performance, have been 24 and 36
(slightly lower than the 36 and 48 found for DAOS). Each run has been repeated
5 times.

Results for pattern A and B are shown in Fig. 2 and Fig. 3, respectively. Note
that, in both figures, Lustre results on the left side and DAOS results on the
right side use a different y-axis scale.

(a) Lustre, write

(b) Lustre, read

(c) DAOS, write

(d) DAOS, read

Figure 2: Access pattern A, global timing write and read bandwidth results
with the Field I/O benchmark.

(a) Lustre, write

(b) Lustre, read

(c) DAOS, write

(d) DAOS, read

Figure 3: Access pattern B, global timing write and read bandwidth results
with the Field I/O benchmark.

Looking at DAOS results for pattern A, in Fig. 2 (c) and (d), we can see that
Field I/O in no-containers mode performs better than the mode with containers.
This is possibly due to inefficiencies in the use of DAOS containers, as
discussed in [3]. The mode without containers scales linearly, and the
application bandwidth increases at an approximate rate of 4.5 GiB/s write and
5.5 GiB/s read per additional server node.

Lustre results in Fig. 2 (a) show that Field I/O in no-containers mode, using
the helper library to adapt to POSIX, performs poorly for write. This is
likely due to performing all the Array file writes and reads in a single
directory, as explained above, which suffers from Lustre contention or locking
on that directory.

The full mode, which has the Array files distributed in several Container
directories, performs better and reaches up to 7.5 GiB/s for write, but hits a
limit at 4 server nodes. In benchmark runs with more server nodes, more client
nodes are also employed, and every client node is set to execute an additional
fixed amount of I/O operations. The limitation observed here is due to
reaching to IOPs limits on the Lustre metadata server, which we have
benchmarked at around 100 KIOPs using IOR. Measured IOPs rates for the Field
I/O benchmark runs approach this limit where the scaling limit is reached. In
IOR benchmarks Lustre can scale up to much larger bandwidths because there are
less IOPs are involved.

Fig. 2 (b) shows that both Field I/O modes perform similarly for read, and
they similarly hit a limit beyond 4 server nodes, again due to reaching
maximum IOPs on the Lustre metadata server. We postulate that both modes
provide similar performance due to the lack of locking/coherency required for
read operations in Lustre.

For results in Fig. 3 for access pattern B, where writer applications are run
concurrently with reader applications, the write and read bandwidths need to
be combined for an approximate comparison with results with pattern A. With
DAOS, Field I/O in no-containers mode performs remarkably well, with linear
scaling and an aggregated bandwidth of approximately 50 GiB/s with 8 server
nodes, higher than the separate bandwidths in pattern A.

With Lustre, the write and read performance behaviour of the full mode with
access pattern B is similar to that observed in pattern A, with slightly
better scaling beyond 4 nodes. The no-containers mode performs poorly not only
for write but also for read, likely due to locking/coherency issues caused by
concurrent writes.

In terms of achieved aggregate application bandwidth, Lustre with Field I/O
pattern B in full mode reaches up to approximately 20 GiB/s with 8 server
nodes, close to the read bandwidth obtained in pattern A, but far from the
higher aggregate bandwidth obtained with DAOS with pattern B in any of the
modes.

From the performance results for access pattern A, it can be observed that the
best application bandwidths obtained with DAOS are in the same order of
magnitude as those obtained with IOR, whereas with Lustre they are
approximately a fifth of those obtained with IOR.

We have identified four factors that could contribute to this performance
difference:

  * •

Lustre is designed for large file I/O, and has a bottleneck in the metadata
server when exposed to object-store-like applications.

  * •

the Field I/O application, when run against Lustre, uses the helper library to
adapt to POSIX, which implements a custom indexing mechanism with Key-Value
directories and files. An implementation that made use of Lustre’s own
directory and file names for indexing would involve less file metadata
operations and may perform better.

  * •

Lustre needs four times as many client nodes as server nodes to saturate
network bandwidth, but we have run the benchmarks with only twice as many.

  * •

DAOS has been optimised for new memory technologies such as SCM and can bypass
the block storage interface for some operations.

####  V-B2 Impact of object and file size

Fig. 4 shows bandwidth results from runs of access pattern A with Field I/O in
full mode with Lustre, and no-containers mode with DAOS. This time the
benchmark has been run with varying object sizes of 1, 5, 10 and 20 MiB, to
assess the impact of future NWP model resolution increases on I/O performance.

All tests have been run with a fixed configuration of 2 server nodes and 4
client nodes, and 100 I/O operations per client process. Benchmarks are
repeated 5 times, with the same client process counts as in previous Field I/O
runs. The results for the 5 repetitions have been averaged, with the average
bandwidth for the highest performing number of client processes per client
node shown for each I/O size tested.

(a) Write

(b) Read

Figure 4: Global timing write and read bandwidth results for access pattern A,
with the Field I/O benchmark, using 2 server nodes and 4 client nodes.

It can be observed that increasing object or file size from 1 MiB to 5 MiB has
a substantial benefit for both write and read, with both Lustre and DAOS.
Beyond 5 MiB, the bandwidth stabilizes except in Lustre reading, where it
continues to increase.

In this benchmark configuration, the bandwidths obtained with DAOS are higher
than with Lustre when using an object or file size of 1 MiB. The two storage
systems provide similar write bandwidths for larger object or file sizes. For
read, DAOS performs better than Lustre with objects up to 10 MiB in size, and
Lustre performs better than DAOS with larger files.

This test set could be repeated using access pattern B to investigate whether
application bandwidth behaves differently as object or file size increase
under write and read contention. Likewise, different server and client node
configurations could be investigated as well.

##  VI Related Work

There has been research into object stores for high-performance I/O, including
CEPH [9] and CORTX Motr [10]. However, these have so far seen less adoption
for very intensive data creation or processing workloads in large-scale
systems. DAOS is one of the first production-ready object storage technologies
targeting HPC, with remarkable results in recent IO-500 benchmarks[11].

Science communities have investigated object stores for I/O operations[12],
including using non-volatile memory[2]. These investigations have demonstrated
the performance and functionality benefits, however they highlight direct
porting relies on custom management of objects on storage mediums. Domain-
agnostic object stores like DAOS simplify the use of NVM in production
environments.

The work presented in this paper builds on previous research in the areas of
exploiting object storage and NVM technologies for HPC I/O, extending
understanding and knowledge on the impact of the object store approach versus
the benefits of using high-performance I/O hardware. It also extends community
understanding of how DAOS and Lustre perform under workload patterns common in
many applications.

##  VII Conclusions

Through our benchmarking and evaluation work we have demonstrated that Lustre
and DAOS, using the same underlying storage hardware, can provide similar
performance for large-scale bulk data operations, such as those the IOR
benchmark mimics and many applications have been tuned to utilise. However,
when moving to object-store-like I/O operations and workflows, DAOS shows
significant performance benefits. We conclude that DAOS is a strong target for
a future storage platform as it demonstrates comparative performance with
traditional files systems whilst enabling object-store functionality at high-
performance.

The access patterns produced with the Field I/O benchmark are not particularly
specific to the NWP use case or bound to any particular file format, and
therefore the findings are applicable to other use cases repeatedly writing
and/or reading any objects or files with sizes in the order of 1 to 10 MiB
from several processes and nodes.

The results suggest that the good performance results obtained with DAOS in
the object-store-like NWP use case are not purely due to the use of high-
performance storage hardware such as SCM, but also due to the design
characteristics of an object store and their implementation in DAOS.

Nevertheless, performance and scalability of Lustre in this scenario should be
further explored and validated by benchmarking with larger amounts of server
and client nodes, and possibly implementing additional optimisations in the
benchmarks to better exploit file system capabilities.

## Acknowledgment

The work presented in this paper has been produced in the context of the
European Union’s Destination Earth Initiative and relates to tasks entrusted
by the European Union to the European Centre for Medium-Range Weather
Forecasts implementing part of this Initiative with funding by the European
Union (cost center DE3100, code 3320). Views and opinions expressed are those
of the author(s) only and do not necessarily reflect those of the European
Union or the European Commission. Neither the European Union nor the European
Commission can be held responsible for them. This work was also supported by
EPSRC, grant EP/T028351/1.

The NEXTGenIO system was funded by the European Union’s Horizon 2020 Research
and Innovation program under Grant Agreement no. 671951, and supported by
EPCC, The University of Edinburgh.

## References

  * [1] G. Lockwood, ”What’s so bad about POSIX I/O?”. The Next Platform 2017. https://www.nextplatform.com/2017/09/11/whats-bad-posix-io/
  * [2] S. Smart, T. Quintino, and B. Raoult. ”A High-Performance Distributed Object-Store for Exascale Numerical Weather Prediction and Climate”. In Proceedings of the Platform for Advanced Scientific Computing Conference (PASC ’19). Association for Computing Machinery, New York, NY, USA, Article 16, 1–11. DOI:https://doi.org/10.1145/3324989.3325726
  * [3] N. Manubens, T. Quintino, S. D. Smart, E. Danovaro, and A. Jackson, “DAOS as HPC Storage, a view from Numerical Weather Prediction”, arXiv e-prints, https://arxiv.org/abs/2208.06752, 2022.
  * [4] Z. Liang, J. Lombardi, M. Chaarawi, M. Hennecke, ”DAOS: A Scale-Out High Performance Storage Stack for Storage Class Memory” In: Panda, D. (eds) Supercomputing Frontiers. SCFA 2020. Lecture Notes in Computer Science(), vol 12082. Springer, Cham. DOI:https://doi.org/10.1007/978-3-030-48842-0_3
  * [5] P. Braam ”The Lustre Storage Architecture” 2019 arXiv. DOI:https://doi.org/10.48550/arXiv.1903.01955
  * [6] ”HPC IO Benchmark Repository”, (2022), GitHub repository, https://github.com/hpc/ior
  * [7] A. Jackson, M. Weiland, M.Parsons, and B. Homölle. ”An Architecture for High Performance Computing and Data Systems Using Byte-Addressable Persistent Memory”. In High Performance Computing: ISC High Performance 2019 International Workshops, Frankfurt, Germany, June 16-20, 2019. Springer-Verlag, Berlin, Heidelberg, 258–274. DOI:https://doi.org/10.1007/978-3-030-34356-9_21
  * [8] M. Weiland, H. Brunst, T. Quintino, et al. ”An early evaluation of Intel’s optane DC persistent memory module and its impact on high-performance scientific applications”. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis . Association for Computing Machinery, New York, NY, USA, Article 76, 1–19. DOI:https://doi.org/10.1145/3295500.3356159
  * [9] K. Jeong, C. Duffy, J. -S. Kim and J. Lee, ”Optimizing the Ceph Distributed File System for High Performance Computing,” 2019 27th Euromicro International Conference on Parallel, Distributed and Network-Based Processing (PDP), 2019, pp. 446-451, DOI: 10.1109/EMPDP.2019.8671563.
  * [10] S. Narasimhamurthy, N. Danilov, S. Wu, G. Umanesan, S. Chien, S. Rivas-Gomez, I. Peng, E. Laure, S. Witt, D. Pleiter, and S. Markidis. ”The SAGE project: a storage centric approach for exascale computing: invited paper”. In Proceedings of the 15th ACM International Conference on Computing Frontiers (CF ’18). Association for Computing Machinery, New York, NY, USA, 287–292. DOI:https://doi.org/10.1145/3203217.3205341
  * [11] A. Dilger, D. Hildebrand, J. Kunkel, J. Lofstead, and G. Markomanolis, ”IO500 10 node list Interational Supercomputing 2021, June 2021, [online] Available:https://doi.org/10.5281/zenodo.5171694
  * [12] S. Smart, T. Quintino, and B. Raoult ”A Scalable Object Store for Meteorological and Climate Data”. In Proceedings of the Platform for Advanced Scientific Computing Conference (PASC ’17). Association for Computing Machinery, New York, NY, USA, Article 13, 1–8. DOI:https://doi.org/10.1145/3093172.3093238
  * [13] MA. Vef, N. Moti, T. Süß et al. ”GekkoFS — A Temporary Burst Buffer File System for HPC Applications”. J. Comput. Sci. Technol. 35, 72–91 (2020). https://doi.org/10.1007/s11390-020-9797-6
  * [14] O. Tatebe, K. Obata, K. Hiraga, and H. Ohtsuji. 2022. ”CHFS: Parallel Consistent Hashing File System for Node-local Persistent Memory”. In International Conference on High Performance Computing in Asia-Pacific Region (HPCAsia2022). Association for Computing Machinery, New York, NY, USA, 115–124. DOI:https://doi.org/10.1145/3492805.3492807
  * [15] A. Miranda, A. Jackson, T. Tocci, I. Panourgias and R. Nou, ”NORNS: Extending Slurm to Support Data-Driven Workflows through Asynchronous Data Staging” 2019 IEEE International Conference on Cluster Computing (CLUSTER), 2019, pp. 1-12, doi: 10.1109/CLUSTER.2019.8891014.