# Methods to Evaluate Lifecycle Models for Research Data Management

Tobias Weber, Dieter Kranzlmüller

Lifecycle models for research data are often abstract and simple. This comes
at the danger of oversimplifying the complex concepts of research data
management. The analysis of 90 different lifecycle models lead to two
approaches to assess the quality of these models. While terminological issues
make direct comparisons of models hard, an empirical evaluation seems
possible.

Lebenszyklus-Modelle für Forschungsdaten sind oft abstrakt und einfach. Hierin
liegt die Gefahr, ein zu einfaches Bild der komplexen Forschungsdaten-
Landschaft zu zeichnen. Die Analyse von 90 dieser Modelle führt zu zwei
Ansätzen, die Qualität dieser Modelle zu bewerten. Die Uneinheitlichkeit in
der Terminologie erschwert einen direkten Vergleich zwischen den Modellen,
wohingegen eine empirische Evaluierung der Modelle in Reichweite liegt.

##  1 Introduction

Advances in science are usually the product of a team rather than individuals.
It is obvious that more than one researcher is needed to further science,
since new insights are based on the work of others, and scientific
publications are reviewed by peers. Maybe less obvious is the necessity for a
number of other actors: research software developers help to develop state-of-
the art tools, communication specialists disseminate important scientific
findings, and data librarians support researchers in data management tasks.
These three professions gain the more importance, as the role of digital
methods and forms of communication increases.

Both aspects of modern research, its collaborative nature and the fast-
evolving technical possibilities, are best exemplified by the task to manage
research data. A large number of services, tools, protocols, best practices,
and policies have been created and are currently competing for adoption. This
state of creolization111cf. [1] leads itself to a research question: How can
we describe, explain, assess, and maybe even predict phenomena in research
data management? Of what nature is the interaction between researchers and
other professionals? The most prominent answer to this question is to model
research data phenomena along a lifecycle.

While the term ”research data lifecycle” is used in many books, papers, blogs,
a commonly shared definition is not available. Most of these models break down
the phenomena of research data management into a series of tasks or states of
data and relate them to different roles or actors. As [2] indicates, these
models are often not evaluated in a manner that allows to reproducibly derive
the same model for a certain purpose (explaining, educating, etc.). A model
un-evaluated is, scientifically speaking, of doubtful quality. If it remains
unclear _how_ the quality of these models can be assessed, their contribution
to a better theoretical understanding of research data management remains an
open question.

The contribution of this paper is the analysis of 90 data lifecycles, in order
to identify ways to evaluate these models. Two approaches are presented:

  * •

One approach focuses on the comparison of data lifecycle models and tries to
derive common quality indicators from the literature (and data lifecycle
models published in non-classical ways.

  * •

The alternative approach abstracts from the usage of the models found in the
literature survey, suggests a classifcation with regard to the purposes the
models is developed for and derives empirical evaluation criteria from these
purposes.

The rest of the paper is structured as follows: in Section 2 we will examine
the related work. Our methodological approach is discussed in Section 3. The
results are presented in Section 4 and discussed in Section 5.

##  2 Related Work

[3] is the one of the first research data lifecycle models in the sense
indicated above. Despite the early publication date, very few practical
aspects have been added to the description of research data management tasks
by later lifecycle models. It is derived from a literature review and
interviews with 18 leaders of contemporary ”cutting edge” projects.
Unfortunately this publication is rarely used in the literature as a reference
evaluate against and check the consistency of terminology.

[4] shows an approach very similar to ours: Based on a survey of lifecycle
models, an abstract data lifecycle model is derived and a classification
scheme is developed. In contrast to [4], we do not define a lifecycle model
but a common scheme shared by all found lifecycle models. One of features by
which [4] classifies, is the distinction between prescriptive and descriptive
models, which comes very close to our proposal to classify along the purpose
the model was designed for. Our method is more focused on evaluation and the
resulting classification is therefore more fine-grained with regard to that.
[4] provides more classifications of features, of which some are irrelevant
for evaluation (e.g. the distinction betwen homogeneous and heterogeneous
lifecycles).

[5], [6] and [7] are alike to [4] in the approach to review existing models
and deriving an own lifecycle model based on a gap analysis. None of the three
publications offer generic and empirical evaluation criteria or a metamodel
for the existing models. Their lifecycle model is designed to supersede the
existing approaches for a specific context.

The model of [5] is not targeted at scientific data, but at open data in
governmental context. The authors clearly state the empirical methods, how the
model was derived, but the paper does not include an evaluation of the
lifecycle model.

[6] and [7] both propose a lifecycle model for Big Data. Although they model
the same phenomena, the models are not similar. While [7] does not describe
evaluation criteria of the model, [6] proposes the 6Vs of Big Data (Value,
Volume, Variety, Velocity, Variability, Veracity) as a base to evaluate data
lifecycle models in the context of Big Data. This evaluation222carried out in
[8] is also applied to evaluate other data lifecycle models for their aptness
to describe Big Data challenges. This evaluation is the most rigorous we found
in the literature, but it is limited to the context of Big Data and itself is
based on a theoretical concept instead of empirical evaluation.

[2] provides a scoped review of 301 articles and 10 companion documents
discussing research data management practices in academic institutions between
1995 and 2016. The review is not limited to, but includes publications
discussing data lifecycle models. The discussion includes the observation,
that of the papers reviewed, only a view provided empirical evidence for their
results, which is in accordance to our findings. The study classifies the
papers based on the UK data
lifecycle333https://www.ukdataservice.ac.uk/manage-data/lifecycle, which
fortunately is preserved as an attachement to this paper (its ”official”
version has changed since the original publication).

##  3 Methods

A survey was executed, to derive a framework to compare data lifecycle models
to each other and to find the purposes for which those models are designed
for, Since not every research data lifecycle is described in an accademic
publication444e.g. the UK data research lifecycle, see above, our approach was
to use a combination of methods of a classical literature review with a
”snowball” method (following references from a first set of models to enlarge
the number of found models). Starting from a research in May 2017, which
facilitated search engines ( google scholar555https://scholar.google.de,
BASE666https://www.base-search.net) and literature databases ( ACM digital
library777https://dl.acm.org and IEEE Xplore888https://ieeexplore.ieee.org),
and a list of already known articles, a first set of 35 data lifecycles was
collected.

The search terms used included any combination of two out of the three
following words: ”research”, ”data” and ”lifecycle”. This deliberately
included lifecycle models which are not specifically dedicated to research
data (e.g. governmental data, linked open data), but we found no essential
differences in both conceptualization and evaluation of these models in
comparsion to research data lifecycle models in the strict sense. Decisive
inclusion criteria for a resource was a check for a textual or graphical
representation of a set of actions regarding data or states of data. Following
the references to other resources (either links or citations), we stopped to
collect further models when we reached 90 data lifecycle and our analyses did
not reveal new aspects.

After an evaluation of 35 lifecycle models a common pattern emerged, which was
successfully applied to the following 55 models, and therefore positively
evaluated. All models included at least one of the following characteristics,
which are the building blocks of the metamodel:

  * •

A set of states in which data are during their scientific processing (such as
creation, analysis, preservation, etc.)

  * •

A connection between these states (in the sense of edges in a directed graph)

  * •

A set of roles in the context of to research data management (researchers,
data stewards/librarians, funders, etc.)

  * •

A set of actions with regard to research data managment (collecting,
documenting, annotating, etc.)

  * •

A mapping of roles, actions and states to each other (e.g. ”in state creation
researchers describe their methods”)

Since the lifecycle models differ widely with regard to their representation
(differnt graphical and textual representation) a homogeneous processing was
not possible at first. To ease the analysis and comparisons between the
models, they were transcribed into an XML representation.999 These XML
representations are published together with bibliographic data of the 90
lifecycle models and other resources: 10.25927/002 A schema to validate the
XML representations was used to guarantee quality of the representations.

During the processing of the sources for the data lifecycle models, excerpts
stating the purpose the lifecycle models were collected. The classification of
purposes is a result of an abstraction from these excerpts and the context in
which some of the data lifecycle models were published at (e.g. training
material, service advertisements).

###  3.1 Threats to Validity

The collection method does not guarantee completeness, which means there might
be research data lifecycle models not captured by our analysis. Since [2]
already provides a scoping review of the relevant literature this is an
acceptable defect. Our approach was focused on finding criteria to compare
data lifecycle models to each other and to evaluate the fitness of lifecycle
models in general for certain purposes, which does not necessitate
completeness.

The list of purposes a model can be designed for is probably not complete,
too, at least in a generic sense (models could for example also be used to
exemplify). But the list should include all relevant applications of models in
the context of research data management.

We only included English and German resources describing data lifecycles. As
far as models had been described in other languages, they often seemed to be
translations. When German models would bias our results, we excluded them from
the statistics (this is clearly stated in the text).

Figure 1: Publication types of found research data management
models/lifecycles

##  4 Results

The heterogenity of the sources for data lifecycle models can be seen in
Figure 1. 62% of the models are published in a medium that is citeable in the
classical sense (journals, proceedings, or books). 78% (70) of the found
models have a graphical representation.

The remainder of this section is divided into two parts: The first part
presents our statistical evaluation results, based on the metamodel presented
in Section 3. The presented numbers will be the basis for the discussion how
the ”dimensions” of the metamodel could facilitate a comparison between data
lifecycle models. The second part proposes a classification of data life cycle
models via their application, a derived evaluation method, and an example for
this application.

###  4.1 Comparison of Lifecycle Models along the Metamodel

39% (35) of the models define actions, 14% (13) define roles and only 13% (12)
define both. Some of the models that only include states, ”encode” an action
into the state the data is currently in (e.g. ”Analysing” or ”data
cleansing”), which makes them hard to compare with other models that separate
state and actions. 11% (10) of the models provide a mapping of activities and
roles to specific states. For partial mappings, Table 1 can be consulted.

The five characteristics listed in Section 3 allow us to define classes of
data life cycle models. Each class extension is defined by the characteristics
fulfilled by its members, i.e. there is a class for all models which define
states and actions, but no roles etc. This classification determines a partial
order that allows to realise a partial comparison. The following data
lifecycle models provide all five characteristics and are therefore members of
the ”highest” class with regard to the partial order:

[9], [10], [11], [4], [5], [7], [12], [13].

The number of states in the data lifecycles ranges between three and thirteen,
the number of actions between zero and 42, the number of roles between zero
and eight. We found 399 different terms for a state, 54 different terms for
roles, and 454 different terms for a research data related activity (case-
insensitive matching, non-English resources were ignored). All these numbers
give evidence to the obvious heterogenity in the existing terminology in
research data management.

To derive a total order from the partial order would allow us to compare all
data lifecycle models to each other (and not only the classes). To achieve
this we would need to have the criteria of completeness for each
characterists, i.e. to answer the question, whether a model includes all
essential states, actions, roles, mappings and connections in the finest
resolution. Given the already stated heterogenity this task is virtually
impossible to accomplish on the collected model descriptions alone: The
semantical mapping between to terms is often not possible, since they lack a
rigorous definition and the models differ in their granularity.

Table 1: Mappings of states, actions and roles to each other (right upper half: absolute number of mappings, left lower half: percentage of mappings) Mapping | States | Actions | Roles
---|---|---|---
States | - | 35 | 11
Actions | 39 % | - | 10
Roles | 12 % | 11 % | -

###  4.2 Evaluation Criteria based on Model Application

These are the classes abstracted from the 90 data lifecycle models. Each class
corresponds to the purpose a model was designed for:

  * •

Documentation: Models can be used to describe certain aspects of reality,
hence document it. If a model is used to document the state of research data
management practices, its main evaluation criteria is its correspondence with
actual research data management practices. Since these practices differ widely
with regard to tools, standards, protocols and policies, there is certainly
not one model that can claim to be _the_ research data lifecycle.
Methodologically speaking, the evaluation of a model designed to document is
executed by the same approach by which it is (or should have been) created:
interviewing experts is an appropriate method to test the accurateness of such
a model. Examples for models which are used to document the actual state of
research data management include the DataOne data lifecycle model101010[14]
and the lifecycle of CENS data.111111[15]

  * •

Explanation: A model explains a set of phenomena, if its usage leads to a
better understanding of it. Explanatory models are to documenting models as
tutorials are to manuals. Data lifecycle models which explain certain aspects
of research data management, are evaluated along the success in educative
outcome. The evaluation how apt a model is to explain to researchers, for
example, how they can make data more reuseable, is therefore a task that
should use the methods of empirical educational theory. The ”lifecycle stages
of environmental datasets” is an example of this kind of purpose.121212[12]

  * •

Design: Designing a desired state with a model is the (re-)arrangement of
components that could be also part of a documenting model. In the context of
research data management, a model that arranges states of data items, roles,
and actions can be evaluated according to the set of features such a desired
state would have. This is comparable to the model that depicts the layout of a
house: one can show, how this specific layout would facilitate the usage by a
family, a bachelor, or old persons in need of care. This indicates that an
evaluation of a model is only possible, if the model is assessed together with
a set of objectives (a use case or a set of generic principles). An example
for a data lifecycle model that can be subsumed under this category is the
data lifecycle of the Inter-University Consortium for Policitical and Social
Research (ICPSR).131313[16]

  * •

Assessment: To assess means to map the actual state to a desired state and
qualify or quantify the conformance. Either the model is used to describe the
actual state or the desired state or there are two models for each of the
states. Such an assessment is implicetly carried out when statements are made
that a certain service ”supports the research data lifecycle”. Whether or not
a set of models are suitable for assessment depends on their specific
evaluation of how well they are equipped to document or to design
respectively. An example for such a usage is the United States geological
survey science data lifecycle model.141414[17]

  * •

Instruction: Another way to relate documenting and designing models to each
other, is to use them to steer and execute transitions from the actual state
to a desired state. Such a transition typically includes the orchestration of
tasks and the allocation of resources as done in classical project management.
A prominent example is to use a lifecycle model as tool to plan a data-
intensive project. Whether or not a couple of models (one documenting, one
designing) is suitable for planning and executing such a transition is not
only determined by the composite evaluation of the two models, but also by the
success of the transition. An example for a research data lifecycle model that
claims to support this activity is the DCC curation lifecycle model151515[18]
or the community-driven open data lifecycle model.161616[5]

##  5 Discussion

This section is structured in the same way as the previous one: First, the
model comparisons will be discussed and after that the empirical evaluation
criteria.

Models providing all five aspects of research data management should be
considered of higher quality than models which only provide them partially.
While this is a first start to compare the quality between data lifecycle
models, it does not take into account whether the states, actions, roles,
their connections and mappings are complete. It is obvious from the numbers
presented in Section 4 that handling of heterogenity of the terms for states,
actions and roles is a very complex task. As stated, another problem is
handling the different resolutions of the lifecycle models: there is no
obvious way to handle mereological relations between states, actions and roles
of two distinct models. A core set of states, actions and role which typically
are part of a data life cycle is therefore not deducible objectively with the
methods presented in this paper. These ”canonical sets” would allow to answer
the question of completeness of a lifecycle model, define a total order on the
set of data lifecycle model, and therefore a means to compare the models with
regard to quality.

An option to come to such an evaluation criterion would be to postulate
canonical sets. If this turns out to be a viable option, it is recommendable
to start with the 50 states, 35 roles and 84 actions that are part of the
models in the highest class according to the partial order. A good starting
point to converge the terminology would be the ontology produced by the RDA
Data Foundation and Terminology Interest Group.171717https://smw-
rda.esc.rzg.mpg.de

The evaluation methods proposed in Section 4 on the other hand are ready to be
used by researchers. It is to be expected that a positive evaluation according
to one purpose might imply a conflict to another one. Take the example of
documentation and explanation: typically, good explanations place greater
emphasis on certain aspects compared to others, if this helps grasping central
concepts. This might entail simplifcations or incompleteness in the model that
are not acceptable in the context of documentation.

Examples for evaluating a design model with regard to objectives are
Findability, Accessibility, Interoperability or Reusability (the FAIR
prinicples).181818[19] Although a convergence on these principles is a goal
embraced by many, there is no common agreement with regard to all aspects of
these objectives. Whether these principles or maturity models191919[20] are
more apt as a means to assess research data management practices202020[21] or
to instruct actors in certain data-related tasks is a question that only
rigorous empirical evaluation can answer..

##  6 Conclusion

Wheras a systematic comparisons of data lifecycle models is not easy, based on
the approach proposed, the evaluation criteria for models based on the purpose
they were designed for is a viable option. Scientific papers proposing a model
for research data management should clearly state the purpose of the model and
consequently include an evaluation with regard to this purpose. This would
bring evidence-based methods into the field of scientific infrastructure
research. Evidence-based statements improve the quality of the research,
foster reproducibility of findings and ease comparison between different
theoretical approaches. A more rigorous definition or reusage of definitions
of terms will furthermore ease comparability between different models in the
future.

These considerations do not only apply for research data models, but could be
extended to other tasks of scientific infrastructure research, including, but
not limited to models for research software development or standards with
regard to technical scientific infrastructures. The improvement of methods of
this research field will have impact to all disciplines, since they will
profit from new insights gained that lead to improved services of research
service providers.

##  7 Acknowledgements

We like to thank Richard Grunzke for his feedback on first ideas for this
paper. This work was supported by the DFG (German Research Foundation) with
the GeRDI project (Grant No. BO818/16-1).

## References

  * [1] P. Wittenburg and G. Strawn. (2018, February) Common Patterns in Revolutionary Infrastructures and Data. https://www.rd-alliance.org/sites/default/files/Common_Patterns_in_Revolutionising_Infrastructures-final.pdf. [Online; Accessed 2018-Nov-13]. [Online]. Available: https://www.rd-alliance.org/sites/default/files/Common_Patterns_in_Revolutionising_Infrastructures-final.pdf
  * [2] L. Perrier, E. Blondal, A. P. Ayala, D. Dearborn, T. Kenny, D. Lightfoot, R. Reka, M. Thuna, L. Trimble, and H. MacDonald, “Research data management in academic institutions: A scoping review,” _PLOS ONE_ , vol. 12, no. 5, pp. 1–14, 05 2017. [Online]. Available: https://doi.org/10.1371/journal.pone.0178261
  * [3] G. M. Hodge, “Best Practices for Digital Archiving - An Information Life Cycle Approach,” _D-Lib Magazine_ , vol. 6, no. 1, Jan 2000. [Online]. Available: http://www.dlib.org/dlib/january00/01hodge.html
  * [4] K. Möller, “Lifecycle models of data-centric systems and domains,” _Semantic Web_ , vol. 4, no. 1, pp. 67–88, 2013. [Online]. Available: http://content.iospress.com/articles/semantic-web/sw060
  * [5] A. F. van Veenstra and T. van den Broek, _Case Studies in e-Government 2.0_. Cham: Springer International Publishing, 2015, ch. A Community-driven Open Data Lifecycle Model Based on Literature and Practice, pp. 183–198. [Online]. Available: http://dx.doi.org/10.1007/978-3-319-08081-9_11
  * [6] A. Sinaeepourfard, J. Garcia, X. Masip-Bruin, and E. Marìn-Torder, “Towards a Comprehensive Data Lifecycle Model for Big Data Environments,” in _Proceedings of the 3rd IEEE/ACM International Conference on Big Data Computing, Applications and Technologies_ , ser. BDCAT ’16. New York, NY, USA: ACM, 2016, pp. 100–106. [Online]. Available: http://doi.acm.org/10.1145/3006299.3006311
  * [7] L. Pouchard, “Revisiting the data lifecycle with big data curation,” _International Journal of Digital Curation_ , vol. 10, no. 2, pp. 176–192, 2016.
  * [8] A. Sinaeepourfard, J. Garcia, X. Masip-Bruin, and E. Marìn-Torder, “A comprehensive scenario agnostic data lifecycle model for an efficient data complexity management,” in _e-Science (e-Science), 2016 IEEE 12th International Conference on_. IEEE, 2016, pp. 276–281.
  * [9] H. James, R. Ruusalepp, S. Anderson, and S. Pinfield, “Feasibility and Requirements Study on Preservation of E-Prints,” Joint Information Systems Committess (JISC), Tech. Rep., October 2003, ”[Online; accessed 26-May-2017]”. [Online]. Available: http://www.sherpa.ac.uk/documents/feasibility_eprint_preservation.pdf
  * [10] G. Knight, “A lifecycle model for an e-print in the institutional repository,” https://pdfs.semanticscholar.org/30ea/1260b787b2ca3cc04e0878aa4495f3513043.pdf, February 2006, ”[Online; accessed 26-May-2017] could be a submission to the DCC Conference on November 22nd, 2006”. [Online]. Available: https://pdfs.semanticscholar.org/30ea/1260b787b2ca3cc04e0878aa4495f3513043.pdf
  * [11] M. Kuberek, “Die Forschungsdaten-Infrastruktur der TU Berlin,” _Bibliotheksdienst_ , vol. 47, no. 11, pp. 833–846, 2013.
  * [12] G. Peng, N. A. Ritchey, K. S. Casey, E. J. Kearns, J. L. Privette, D. Saunders, P. Jones, T. Maycock, and S. Ansari, “Scientific stewardship in the Open Data and Big Data era—Roles and responsibilities of stewards and other major product stakeholders,” _D-Lib Magazine_ , vol. 22, no. 5/6, May 2016, ”[Online; Accessed 2018-Nov-13]”. [Online]. Available: http://dlib.org/dlib/may16/peng/05peng.html
  * [13] A. Sarmiento Soler, M. Ort, J. Steckel, and J. Nieschulze, “An Introduction to Data Management,” Feb. 2016. [Online]. Available: https://doi.org/10.5281/zenodo.46715
  * [14] W. K. Michener and M. B. Jones, “Ecoinformatics: supporting ecology as a data-intensive science,” _Trends in ecology & evolution_, vol. 27, no. 2, pp. 85–93, 2012.
  * [15] J. C. Wallis, A. Pepe, M. S. Mayernik, and C. L. Borgman, “An exploration of the life cycle of escience collaboratory data,” in _iConference 2008_ , 2008\. [Online]. Available: https://www.ideals.illinois.edu/handle/2142/15122
  * [16] ICPSR, “Guide to Social Science Data Preparation and Archiving: Best Practice Throughout the Data Life Cycle,” http://www.icpsr.umich.edu/icpsrweb/content/deposit/guide/, 2012, [Online; Accessed 2018-Nov-13]. [Online]. Available: http://www.icpsr.umich.edu/icpsrweb/content/deposit/guide/
  * [17] J. L. Faundeen, T. E. Burley, J. A. Carlino, D. L. Govoni, H. S. Henkel, S. L. Holl, V. B. Hutchison, E. Martín, E. T. Montgomery, C. Ladino _et al._ , “The United States geological survey science data lifecycle model,” US Geological Survey, Tech. Rep., 2014. [Online]. Available: https://pubs.usgs.gov/of/2013/1265/pdf/of2013-1265.pdf
  * [18] S. Higgins, “The DCC Curation Lifecycle Model,” _The International Journal of Digital Curation_ , vol. 3, no. 1, pp. 134–140, June 2008. [Online]. Available: http://www.ijdc.net/index.php/ijdc/article/viewFile/69/48..
  * [19] M. D. Wilkinson, M. Dumontier, I. J. Aalbersberg, G. Appleton, M. Axton, A. Baak, N. Blomberg, J.-W. Boiten, L. B. da Silva Santos, P. E. Bourne _et al._ , “The FAIR Guiding Principles for scientific data management and stewardship,” _Scientific data_ , vol. 3, 2016.
  * [20] K. Crowston and J. Qin, “A capability maturity model for scientific data management: Evidence from the literature,” _Proceedings of the American Society for Information Science and Technology_ , vol. 48, no. 1, pp. 1–9, 2011\. [Online]. Available: http://dx.doi.org/10.1002/meet.2011.14504801036
  * [21] T. Weber and D. Kranzlmüller, “How FAIR can you get? Image Retrieval as a Use Case to calculate FAIR metrics,” in _2018 IEEE 14th International Conference on e-Science_ , 2018.