# Experiments as Code:
A Concept for Reproducible, Auditable, Debuggable, Reusable, & Scalable
Experiments

Leonel Aguilar1,2,† Michal Gath-Morad1,† Jascha Grübel1,3,† Jasper Ermatinger1
Hantao Zhao1 Stefan Wehrli5 Robert W. Sumner3 Ce Zhang2 Dirk
Helbing${}^{5},^{6}$ Christoph Hölscher1,5

###### Abstract

A common concern in experimental research is the auditability and
reproducibility of experiments. Experiments are usually designed, provisioned,
managed, and analyzed by diverse teams of specialists (e.g., researchers,
technicians and engineers) and may require many resources (e.g. cloud
infrastructure, specialized equipment). Even though researchers strive to
document experiments accurately, this process is often lacking, making it hard
to reproduce them. Moreover, when it is necessary to create a similar
experiment, very often we end up “reinventing the wheel” as it is easier to
start from scratch than trying to reuse existing work, thus losing valuable
embedded best practices and previous experiences. In behavioral studies this
has contributed to the reproducibility crisis. To tackle this challenge, we
propose the “Experiments as Code” paradigm, where the whole experiment is not
only documented but additionally the automation code to provision, deploy,
manage, and analyze it is provided. To this end we define the Experiments as
Code concept, provide a taxonomy for the components of a practical
implementation, and provide a proof of concept with a simple desktop VR
experiment that showcases the benefits of its “as code” representation, i.e.,
reproducibility, auditability, debuggability, reusability, & scalability.

_K_ eywords Online Experiment $\cdot$ Experiment Infrastructure $\cdot$
Experiment Data Collection $\cdot$ Cloud Architecture $\cdot$ Experiments as
Code $\cdot$ Virtual Reality

Contact: Leonel Aguilar
<EMAIL_ADDRESS>
1 Chair of Cognitive Science, ETH Zürich
2 Data Science, Systems and Services Group, ETH Zürich
3 Game Technology Center, ETH Zürich
5 Decision Science Laboratory, ETH Zürich
6 Chair of Computational Social Science, ETH Zr̈ich
$\dagger$ These authors provided equivalent contributions

##  1 Introduction

Over the last decade, we have observed a crisis of scientific reproducibility
in behavioral experimental research (Ioannidis, 2005; Pashler and Wagenmakers,
2012; Collaboration et al., 2015; Camerer et al., 2018). This reproducibility
crisis has exposed several short-comings of how experiments are communicated
within the scientific community and beyond. The crisis is very much rooted in
the fact that implicit information about the experimental process often gets
communicated as _viva voce_ (Hassani and Idrissi, 2018). Third parties
attempting to reproduce an experiment may fail due to missing such implicit
information.

The COVID-19 pandemic is exacerbating the crisis of reproducibility due to the
shift towards remote work and limiting in person experiments. This, in turn,
further limited the ability to communicate such information directly within a
research context. The diminished opportunities for interaction during the
pandemic caused major side-and-after effects resulting in a decrease of
experimental data quality and unnecessary extra costs within a research group
and the research community at large.

As a way to achieve reproducibility, Stark (2018) introduced the term
“Preproducibility”. Preproducibility, or scientific recipes, requires
researchers to provide the complete set of necessary elements and procedures
to replicate an experiment that “cannot with advantage be omitted” (Popper,
1992, p.44). In contrast, current efforts to achieve reproducibility tend to
provide only sufficient information to _understand_ the experiment in detail.
Understanding an experiment is not enough in the process of re-implementing
the experiment from scratch. Nuanced details may be implemented differently,
which ultimately changes the outcome (e.g., Reardon, 2016). Any future
research needs to address these concerns to underpin the validity of their
claims or explain deviations from previous results like behavioral drifts or
changes in implicit experimental conditions (e.g., Reardon, 2016).

The relatively new field of _meta-science_ (Ioannidis et al., 2015) and
initiatives across all sciences (Stojnic et al., 2021; Swiss Data Science
Center, 2021; Aguilar et al., 2021; Orchard et al., 2003; Welter et al., 2014)
aim to improve experimental research (Baumgaertner et al., 2018) and enable
both reusability and composability (Nierstrasz and Meijler, 1995). However,
these important initiatives and suggestions disjunctively solve partial
problems. To bridge this gap, we require a holistic approach that ties
together the benefits of these initiatives. Behavioral experiments in VR are a
useful testing ground to adopt a more holistic approach because they pose
unique reproducibility challenges and an opportunity to implement
Preproducibility. On the one hand, the susceptibility of human behavior to
nuanced changes can drastically alter experiment outcomes (Kuliga et al.,
2015; Gath-Morad et al., 2020b). On the other, VR experiments are often fully
virtual and offer themselves to automation and thus formal documentation
(e.g., Grübel et al., 2016).

As a holistic approach to the reproducibility of VR experiments and as a
practical implementation of Preproducbility we propose the paradigm
_Experiments as Code_ (ExaC) inspired by the cloud computing world and in
particular the “ as Code” and “ as a Service” revolutions. Given the ubiquity
and accessibility of cloud computing services, researchers who use ExaC could
deploy their experiments without cloud computing expertise. This would allow
the reuse of published working configurations, effectively transferring and
inheriting implicit knowledge essential for reproducibility. ExaC implements
preproducibility in VR experiments by documenting all its elements,
procedures, information, and implemented best practices through code. This
includes the whole experimental pipeline, protocol, analysis tools, VR
software, and the configuration of the required computing infrastructure. The
current publishing standards and disjoint initiatives already encourage better
practices in the experimental process. Nevertheless, the ExaC paradigm ties
them together and overcomes the limitations imposed by _viva voce_ knowledge
transfer. ExaC transforms the implicit elements and procedures required for
reproducibility into explicitly documented and automated code procedures.
Additionally, versioning of the code allows keeping track not only of the
final working configurations but also intermediate steps, failed attempts, and
configuration testing. These steps are all recorded and annotated with
information concerning the decisions or criteria that lead to changes in the
code base through a version control system.

The challenges of reproducibility are often specific to the experiment type,
nature and requirements. For example for local experiments, APIs interfacing
infrastructure with specialized hardware needs to be built, e.g., interfacing
with skin conductance sensors, specialized immersive equipment or CAVE (Cruz-
Neira et al., 1992) environments. For remote data collection, Ma et al. (2018)
identified challenges in crowd-sourced remote immersive VR experiments, e.g.,
the heterogeneity of hardware, participants trying to trick the experiment,
and participant pool size and composition. The Human Computer Interaction
(HCI) community has documented several reproducibility challenges, e.g.,
Ratcliffe et al. (2021) collected experiences and challenges of remote
experimentation in HCI research, while Echtler and Häußler (2018) documented
and discusses the reproducibility crisis and the extent of use of open source
and open science concepts in HCI. Feger et al. (2019) glances over the roles
and challenges of HCI on research reproducibility, e.g., the incentives for
making research reproducible and the potential role of gamification
mechanisms. Concrete recommendations such as the ones provided for remote VR
data collection by (Mottelson et al., 2021) could be embedded and actionable
in an ExaC representation. We argue that, although ExaC by itself does not
make an experiment reproducible or solve the challenges to achieve
reproducibility it provides a path to it through a way to document, automate,
instantiate and transfer proven working solutions.

To showcase the features of an ExaC representation we provide a proof-of-
concept implementation of a desktop VR wayfinding experiment. Our Simple
Experiment as Code Template (SimpleExaCT) repository enables the
reproducibility of this experiment. The implementation uses several “ as Code”
components along with an ad-hoc VR framework, SimpleVR. However, SimpleExaCT’s
modular approach makes it compatible with other VR frameworks. Consequently,
we review existing frameworks in the light of the ExaC paradigm. We envision a
new third generation of VR frameworks that goes beyond VR hardware integration
(first generation) and experiment design (second generation) towards
conducting reproducible, auditable, debuggable, reusable and scalable
experiments.

The main contributions of this paper are three-fold. First, we contribute
towards the ExaC paradigm by providing a definition and demonstrating it with
the full experimental pipeline of a desktop VR behavioral experiment in the
domain of Cognitive Science. Second, we provide an open source proof of
concept implementation of this Experiment as Code, to scaffold data collection
in our online desktop VR experiment. This enables third parties to audit and
reuse our experimental design and showcases the benefits of the ExaC
representation. Third, we provide a detailed comparison of notable VR
frameworks and review them through the lens of ExaC emphasising their capacity
to support Preproducibility. Although we showcase the ExaC paradigm with a
desktop VR experiment, we envision its potential for improving experimental
research in general.

##  2 Background, Definitions, and Related Work

###  2.1 Meta-science, Reproducibility and the “ as Code” revolution

The ExaC paradigm is rooted in three larger developments of the last decade.
First is the rise of the field of _meta-science_ (Ioannidis et al., 2015),
second are initiatives to overcome the reproducibility crisis, and third are
developments in cloud computing such as the “ as Code” and “ as a Service”
revolutions.

Meta-science tries to improve the processes of scientific work by making the
scientific inquiry itself the object of their study. Suggestions include pre-
registration (van’t Veer and Giner Sorolla, 2016), a complex systems approach
(Wallot and Kelty-Stephen, 2018), replication as teaching (Grahe et al.,
2012), change in attitudes to p-values (Colquhoun, 2014), larger samples
(Maxwell et al., 2015), raw data access, triangulation instead of replication
(Munafò and Smith, 2018), and standards of presentation (Gosselin, 2020).

There are several initiatives that tackle challenges impacting directly
reproducibility. For instance, there is an indexing service that specializes
in indexing papers accompanied by code (Stojnic et al., 2021). Another
initiative provides multidisciplinary, collaborative and reproducible data
pipelines (Swiss Data Science Center, 2021). There are also modular ecosystems
for improving machine learning reusability by managing and automating the
complete machine learning application development life-cycle with reusable
models and components (Aguilar et al., 2021). Some disciplines have pushed
even further, creating domain specific standards (Orchard et al., 2003) and
ontological databases (Welter et al., 2014) that enable more efficient and
systematic knowledge extraction. Moreover, publishers are now encouraging the
publication of protocols in specialized services such as protocols.io, where
protocols are converted to modifiable and reusable checklists.

Tightly interrelated with reproducibility are the notions of auditability,
debuggability, reusability, & scalability. We define auditability as providing
a transparent and comprehensive view into the experiments’ mechanics, tools
and components. Thereby, any third party is able to evaluate the software
(Weiss, 1980) and by extension the experiment’s reproducibility. We define
debuggability as providing the means to identify and modify aspects of the
experiment affecting its reproducibility (Rabinovich, 2014). We define
reusability as the ability to reuse parts or variations of an existing
software to create a new one (Prieto-Diaz, 1993) and in our context software
extends to experiments. Here we can exploit the benefits of open source
software development in science, (Echtler and Häußler, 2018). These three are
ubiquitous tasks in open source software development and as such are enabled
through the ExaC representation by inheriting the same good practices.
Finally, in systems scalability refers the ability to either expand a system
in one dimension (structural) or an increase in the amount of activity (load)
the system receives (Bondi, 2000). In the context of experiments, this refers
to the ability to increase the sample size either through repetition
(horizontal) or the use of larger participant pools (vertical), e.g., crowd-
sourced and remote experiments.

In cloud computing, the complexities and requirements of current web services
have pushed advances and paradigm shifts such as the “ as Code” and “ as a
Service” revolutions. Paradigms such as “Infrastructure as Code” (IaC; e.g.,
Terraform, AWS Cloud formation), “Environments as Code” (e.g., Vagrant,
Docker), “Configuration as Code” (e.g., Ansible, Puppet Chef), “Data Pipelines
as Code” (e.g., Airflow) are now ubiquitous in web service development. There,
a single service requires the deployment of hundreds to thousands of micro-
services, their infrastructure to be dynamically provisioned, and their data
to be continuously analysed.

Figure 1: Six pillars taxonomy of behavioral experimentation: 1)
documentation, 2) infrastructure & environment, 3) data collection, 4) data
assembly, 5) data analysis, 6) management. Each pillar consists of multiple
essential tasks that need to be completed. Each pillar is represented
distinctively in the ExaC paradigm as a block for automation of an experiment.

For example, IaC allows storing and versioning the infrastructure and
configuration requirements of complex web-services. These code representations
have enabled simpler deployment and provisioning, rolling back to previous
configurations, mirroring services, planning, scaling, and even debugging
problems in the infrastructure provisioning. We believe that many of the
features of the “ as Code” movement and their tools could enhance experiments
in general and VR experiments in particular.

Despite this substantial drive for improving the reproducibility of behavioral
experiments, Preproducibility is still a challenge. However, the boom of “ as
Code” provides a foundation to implement Preproducibility and achieve
reproducibility in behavioral experiments. To the best of our knowledge, the
term “Experiment(s) as Code” has only been used in two research articles.
First, in EnosStack (Cherrueau et al., 2018) that proposes a software stack
composed of Python, Ansible, Docker and their library EnosLib to enable
reproducible experimental workflows. Second and similarly, Almaatouq et al.
(2020) identified the need for reproducible virtual labs and created a full
platform. In contrast to these works, we define the ExaC paradigm and scaffold
it on to a taxonomy describing how to conduct VR experiments as an
implementation of Preproducibility. This entails combining a wide variety of
approaches from “ as Code” with VR frameworks and other complementary software
components of the experiment. Moreover, the components are only loosely
coupled allowing to exchange them as services evolve.

###  2.2 Behavioral experiments in VR and ExaC definition

Behavioral experiments in VR can be composed of several elements (e.g., tasks,
procedures, information, and documentation) beyond the experiment content.
They can span from its pre-registration over the acquisition of participants
to the monitoring of the infrastructure status. We classify these elements as
components in groups and subgroups under the proposed six pillars taxonomy
(see Fig. 1). The pillars cover the essential aspects of an experiment by
subdividing it into groups: documentation, infrastructure & environment, data
collection, data assembly, data analysis, and management. We collect and
exemplify these common components required to enable reproducibility. The
exemplified components may vary in their classification and extent depending
on the details of their implementation and experiment requirements, e.g.,
sensitive data anonymization could be desirable to occur at collection time.
While some components of each pillar are strictly required to produce an
experiment, others are optional but desired to improve reproducibility.

We base our definition of ExaC on the idea of preproducibility (Stark, 2018),
but instead of providing _scientific recipes_ we aim at providing _automated
and digitally documented scientific recipes_ inspired by the advances in cloud
computing. Thus, we define the ExaC paradigm as providing imperative, or
declarative automation code and digital documentation to the components of the
respective essential pillars.

We borrow the imaging concept from Environments as Code. An image of how a
run-time environment should look like serves as a specification of how actual
containers are deployed. The static ExaC code base serves as the image (see
Fig 2, Base). The whole live experimental setup can be automatically
instantiated from the image including the supporting components and the
components building the experiment itself. This entails but is not limited to
provisioning the right hardware, deploying the software stack, instantiating
abstraction interface services, acquiring participants, hardware monitoring
services, serving the content to participants, and ultimately collecting the
experiments data.

Figure 2: ExaC paradigm application scheme: The paradigm serves as the Base
image to instantiate the Support (infrastructure and services) and then
conduct the Experiment itself. The ExaC (gray box) covers the 6 pillars
taxonomy of behavioral experimentation (see Fig. 1).

###  2.3 VR Frameworks in the context of ExaC

The pillars used in ExaC provide defined boundaries for modularization. VR
frameworks and specialized software can be used to provide automation code and
digital documentation for them. Many existing VR Frameworks already provide
support or potential for the automation of many of the common components
described within the pillars. We provide a summary of the features of these VR
Frameworks through the lens of ExaC and the six pillars in Table 1.
Additionally, we collect the documented efforts to address each of the
specific components of the pillars in the Supplementary Information Section 1
and an in-depth discussion of the VR frameworks through the lens of ExaC in
the Supplementary Information Section 2.

|  | Experiments as Code |  |
---|---|---|---|---
|  | Doc. |  Infrastructure |  Data Collection |  Data Assembly |  Data Analysis | Managem. | Access
Framework |  Platform |  Protocol† |  Multi-user Server |  Immersive VR |  Provisioning† |  Deployment† |  Prefab. tasks |  Task config. files |  Game engine code |  Questionnaires |  Remote Storage |  Database |  Analysis |  Replay |  Participants |  Monitoring |  github/bitbucket |  Unity package
AGENT (Moulec et al., 2017) | Any |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
bmlTUX (Bebko and Troje, 2020) | Unity |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
EVE (Grübel et al., 2016) | Unity |  |  | * |  |  |  | * |  |  |  |  |  |  |  |  |  |
DeFINE (Tiwari et al., 2021) | Unity |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
Landmarks (Starrett et al., 2020) | Unity |  |  |  |  |  |  |  |  |  |  |  | * |  |  |  |  |
NVR-DeSciL (Zhao et al., 2018) | Unity |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
OpenMaze (Alsbury-Nealy et al., 2020) | Unity |  |  | * |  |  |  |  |  |  |  |  |  |  |  |  |  |
Toggle Toolkit (Ugwitz et al., 2021) | Unity |  |  | * |  |  | * |  |  |  |  |  |  |  |  |  |  |
VO (Howie and Gilardi, 2020) | Unity |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
VRate (Regal et al., 2018) | Unity |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
VREVAL (Schneider et al., 2018) | Unreal |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  | -
VREX (Vasser et al., 2017) | Unity |  |  |  |  |  | * | * | * |  |  |  |  |  |  |  |  |
VR-Rides (Wang et al., 2020) | Unity |  |  |  |  |  |  |  |  |  |  |  | * |  |  |  |  |
USE (Watson et al., 2019) | Unity |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
UXF (Brookes et al., 2019) | Unity |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
Symbols: | : Feature, *: Partial Feature, -: Not possible, †: Never implemented |

Table 1: Comparison of current VR frameworks since 2015. The features are
grouped by correspondence to the 6 pillars taxonomy of the _ExaC_ paradigm.
Features are evaluated on the description of the frameworks in their
accompanying paper. We also report the availability of the frameworks.

In the early 2000s, a first set of VR frameworks (Bierbaum et al., 2001;
Tramberend, 1999; Ayaz et al., 2008; Annett and Bischof, 2009; Mossel et al.,
2012) focused on simplifying the technical setup. However, these frameworks
were rapidly eclipsed by game engines (i.e., Unity and Unreal) and became
quickly obsolete. Over the last five years, a second generation of VR
frameworks (Grübel et al., 2016; Moulec et al., 2017; Schneider et al., 2018;
Zhao et al., 2018; Brookes et al., 2019; Watson et al., 2019; Alsbury-Nealy et
al., 2020; Bebko and Troje, 2020; Starrett et al., 2020; Wang et al., 2020;
Ugwitz et al., 2021) has put the experiment design at the core of the
framework. Technical issues such as rendering, physics and specialized
hardware have been delegated to the underlying game engines such as Unity or
Unreal and third-party libraries to integrate Human Interface Devices (HIDs;
e.g. Oculus Touch, 3D mice).

The second generation of VR frameworks provide better support to the different
components required for experimentation. However, there is still a
considerable gap in terms of hardware provisioning and deployment, support for
data analysis, multi-user setups, and participant management and monitoring.
To bridge these gaps, we envision a third generation of VR frameworks that
either natively support all six pillars of ExaC or that are modularly designed
to better integrate with specialized tools that cover them. This third
generation would extend their focus from experimental design towards
reproducibility, auditability, debuggability, reusability, and scalability.
General frameworks for behavioral experimentation recently started focusing on
reproducibility (Cherrueau et al., 2018; Almaatouq et al., 2020). However, we
believe that for the future development of VR frameworks with reproducibility
as the corner stone, a robust definition of the underlying paradigm to achieve
and document reproducibility is required. Hence this paper proposes and
defines ExaC as the underlying paradigm and proposes a taxonomy to support its
modular development.

##  3 Method

To demonstrate the ExaC paradigm we provide as a proof-of-concept automation
code covering each of the 6 pillars of experimentation (see Fig. 1). For this,
we used open source tools (i.e., terraform, jupyter-lab and docker) coupled
with a minimal ad-hoc framework we named SimpleVR written in C#/Unity, Python,
JavaScript and Golang. This section goes through each of the pillars and
provides a concise description of our automation code and technologies
covering the respective pillars. Finally, we present an example workflow. Even
though the provided implementation of ExaC has been used for several of our
experiments, it should be considered a minimal working example to demonstrate
the concept and not a mature code base. One of the added advantages of this
modular design is the potential evolutive improvement. Accordingly, successful
ExaC code bases and code components will be replicated, modified, and
improved.

In our ExaC paradigm implementation, we distinguish between three layers (see
Fig. 3, A). The top layer consists of the software stack that describes what
the experiment is (see Fig. 3, A3, A4, A5, & A6). The middle layer consists of
the software stack that describes how the experiment is run (see Fig. 3, A2).
The bottom layer consists of the digital documentation management (see Fig. 3,
A1). The distinction between the stacks is similar to the differentiation
between dev-stack and a devops-stack in software service development. In the
top layer, we provide an implementation of the four pillars with our proof-of-
concept framework SimpleVR, analogous to the dev-stack. In the middle layer,
we provide software defining run-time environments, automation, provisioning,
and deployment, analogous to the devops-stack. Our ExaC implementation is the
composite of these three layers, which we refer to as SimpleExaCT (Aguilar
Melgar and GathMorad, 2021). The middle layer of the ExaC implementation
supports Preproducibility by automating the deployment through cloud computing
concepts, namely EaC and IaC. We containerized the top layer services using
EaC (e.g., a Docker image). The middle layer provisions the infrastructure and
deploy the containerized services using IaC (e.g., Terraform).

Figure 3: Overview of the ExaC implementation and instantiation for the
purpose of our case-study experiment (see Sec. 3.2). The illustration
integrates the ExaC implementation (Base), the instantiated services and
infrastructure (Support) and the data collection client (Experiment).

Since the system is meant to support reproducible experiments, we pay
particular attention to the top layer that contains the experiment defining
code. To that end, we address its design considerations in light of our case
study needs, specifically remote VR experimentation (Rivu et al., 2021). For
the most part, conducting VR studies has been the privilege of well-funded
institutions because of the prohibitive costs of large-scale experimental data
collection. We address this issue with a minimalistic ad-hoc VR framework,
SimpleVR, by reducing the computational cost of large-scale data collection.
In traditional lab-based VR experiments, the researchers need to provide the
computational power to run each of the VR experimental trials. This is done
through a central server or offloaded to the lab computers.

In contrast, SimpleVR offloads the VR computations to the participants’
browser and requires the researchers to provide only the infrastructure needed
to capture the transferred data. This is supported by new web browsers that
provide local browser storage, service workers, access to 3D rendering
capabilities, in-browser databases, and the ability to execute code with low
overhead through web-assembly. It needs to be noted that the purpose of this
paper is not to present the development of a feature rich VR framework.
Instead, we focus on the novelty of the ExaC paradigm, and this minimal
framework (i.e., SimpleVR) was developed to bring the experiment to the
browser and minimally cover the top layer in the 6 pillar taxonomy.

SimpleVR represents the top layer of the ExaC code Base (see Fig. 3, A3, A4,
A5 & A6). When deployed, the instances of the four composable component groups
serve a number of purposes. The first component group (see Fig. 3, A3) defines
the data collection software which is displayed in the client browser (see
Fig. 3, E). Trials of independent human participants run within their web
browser to collect data. The second component group (see Fig. 3, A4) defines
the data assembly service which is deployed in the cloud (see Fig. 3, D). This
component group provides an API to the first component detailing how to
collect and store the data. The third component (see Fig. 3, A5) defines the
data analysis service that allows to process data and gain insights (see Fig.
3, B). This component group interacts with the data assembly. The fourth
component group (see Fig. 3, A6) defines the data management service that
allows the researcher to monitor and manage the experiment (see Fig. 3, B & C)
and interacts with all of the three previous component groups. Once the data
collection and data assembly services are ready, the experiment deployment
infrastructure distributes the data collection software to the participants.

###  3.1 ExaC implementation – A

The SimpleExaCT code base consists of the digital documentation, the IaC and
EaC technologies and SimpleVR. We refer to it as the _Base_ (see Fig. 3, A). A
description of the implementation of the proof of concept is provided in
Section 1 of the Supplementary Materials.

###  3.2 Demonstrative Case Study

To demonstrate the power of the ExaC paradigm, we use the SimpleExaCT
implementation in a desktop VR behavioral experiment in the domain of
Cognitive Science. The experiment consists in participants finding their way
to different targets within a building. The experiment evaluates the effect of
two architectural variations ‘Treatment A’ and ‘Treatment B’ on the wayfinding
performance of participants in comparison to the unmodified building
‘Control’. We selected this remote destkop VR experiment for our proof of
concept as it doesn’t require specialized equipment beyond a capable computer
and a browser, and is robust under a wide range of uncontrolled variations,
e.g., variations in screen sizes, or distance to the screen. This enabled us
to easily showcase the use remote data collection and demonstrate this kind of
vertical scaling. This experiment was performed in April 2020 in three
sessions, during the COVID-19 pandemic and lockdowns. The use of SimpleVR
allowed us to bring the VR experiments to the browser and our SimpleExaCT
implementation allowed us to communicate and document the procedure following
the preproducbility concept.

The data collection in the original experiment was spread out across days to
obtain enough participants according to the power analysis. Although for the
original experiment, all sessions where aggregated in the data pool, in terms
of independent instantiations, the three sessions can be understood as three
independent experiments. The third session did not collect data on all
treatments because only some treatments required more data. Hence, the results
presented in this paper include only a comparison between the two full
instantiations, sessions 1 and 2. For demonstration purposes and to avoid
conflict with the detailed publications of the experiment, here we refer in
general terms to: 1) the control and applied treatments which were
architectural variations on a building, and 2) the dependent variable used to
measure the participant’s performance. Details on the experiment and the
behavioral analysis of the MTurk participant data can be found here (GathMorad
et al., 2021). Details on the validation of an Agent-Based Model (ABM) using
this data can be found here (Gath-Morad et al., 2020a).

Participants were recruited online through MTurk Human Intelligence Tasks
(HITs). The participants were filtered according to the external requirements
of informed consent, technical ability to display the experiment and
completion of the experiment. We only used data from the participants whose
web-browser was deemed capable of displaying the experiment, who agreed on the
consent screen, and who completed the task with the correct response to the
challenge. 149 participants fulfilled these criteria. Participants’ age ranged
between 18 years to 59 years. The mean age was 33.7 years with a Standard
Deviation of 6.8 years. The methods and experiment described in this section
were performed in accordance with the relevant guidelines and regulations
following the approval of the study by ETH Zurich’s Research Ethics Committee.
Two inclusion criteria for the study were English proficiency, corrected-to-
normal or normal vision. Color blind individuals were excluded from
participating in the study. All participants signed an informed consent form.
The average time taken to complete the experiment was 20 minutes and the
average monetary compensation was 4.5 USD. If participants completed the
experiment in less than 20 minutes they received 1 USD bonus. The average
compensation was 5.4 USD.

#### Experimental Protocol

The protocol of the experiment can be found in the SimpleExaCT repository,
(Aguilar Melgar and GathMorad, 2021) and a complete description of the
demonstrative experiment content, the experimental protocol, materials, and
data analysis can be found at Gath-Morad et al. (2020a). Overall, the
experiment tested how changes in the configuration of buildings may affect
human navigation performance.

Figure 3 shows how ExaC was used to deploy the experiment. An abstraction
layer (Figure3, C) was in interfacing with the participant procurement
service. For logistical reasons, we opted for the DeSciL and MTurk as specific
service providers in charge of communication with both the researcher and with
the study participants. Nonetheless, we provide scripts to interface with
MTurk directly within the SimpleExaCT implementation. Communication with
participants mainly consisted of deploying the HITs for them to perform the
experiments.

Participants had a a first person perspective of the virtual scene in each
building. Participants’ origin was set to the same location across the three
building groups (see Fig. 3, E). Navigation was performed using keyboard and
mouse. Participants had to digitally sign a consent form and complete the
onboarding before the experimental trials and the offboarding afterwards. The
onboarding ensured the correct assignment to one of the three building
treatments and the offboarding the appropriate reward. The experiment itself
mainly consisted of two parts, a training phase and 6 navigation trials. The
mean duration of the training phase 5 minutes. During training, participants
learned how to move in the virtual environment using a mouse and keyboard.
This is a common procedure to ensure that participants are familiar with the
controls to reduce adverse effects (Thrash et al., 2015; Grübel et al., 2017).
Upon successful completion of the practice scene, participants had to complete
6 trials in which they were asked to find a specific destination inside the
building.

The participants’ positions and orientations were recorded every frame
(roughly every 0.02 seconds) using a cloud infrastructure to deploy and
retrieve participants’ data (see Fig. 3, C & D). The research team (see Fig.
3, B) was able to manage and monitor participants, hardware and software
status from their computers. The abstraction services provided interactions
with the management of participants (see Fig. 3, B & C). The cloud
infrastructure provided interactions with both the deployment of the
experiment content and access to the collected data (see Fig.3, B & D). Using
the analysis scripts, the research team was able to visualize and analyze the
data collected immediately after the experiment was concluded (see Fig. 3, B).

###  3.3 Results

#### Reproducibility

is assessed by independently evaluating the results of the two complete
instantiations with the Linear Mixed Effect Model,
$\mbox{DependentVariable}\sim\mbox{Treatment}+(1|\mbox{Participant})$. Figure
4 shows that in the complete instantiations of the experiment, the same
pattern of results can be observed, treatment A has no significant effect,
while treatment B provided a significant ($p<0.05$) performance improvement.

Figure 4: The same pattern of results is shown by the two complete
instantiations of the experiment, lower value on the dependent variable means
and improvement in performance. Session 1 (n=275), Session 2 (n=495). For an
in-depth data analysis of the experimental results see (GathMorad et al.,
2021).

#### Auditability, Debuggability, & Reusability

are shown through a simulated third party auditing the experiment and
describing bugs found while using the code. This is documented in a GitHub
issue ‘Example 1 - Bug fix’. As a response to this GitHub issue, the
maintainer identifies the cause of the bug and creates a new version reusing
most of the code. Additionally, the GitHub issue ‘Example 2 - Reuse’, shows a
simulated interaction with a researcher reusing SimpleExaCT for an online
version of another desktop VR wayfinding experiment (Gath-Morad et al.,
2020b). Here, the researcher adds the experiment assets, i.e., the building
and building variations, and implements the experiment mechanics, e.g., door
frames changing in color. These modifications are limited to the data
collection module, while all the other modules are reused without changes. All
Github issues presented here as examples are linked with their respective
code. Interested third parties are encouraged to check the repository Issues
section in addition to auditing and forking the repository.

#### Scalability

We investigate the access to the experiment beyond the 149 participants that
successfully took part in the experiment by reviewing the raw accesses to the
experiment. This includes all the interactions with the experimental data
collection (i.e., MTurkers that clicked on the experiment link, see Fig. 3,
E). In total, 462 participants accessed the experiment, 316 (68%) had
computers that were deemed capable of displaying the experiment, while the
remaining 146 (32%) were not. Of the participants starting the experiment, 149
(47%) participants also completed the experiment. The participants’ inferred
Operating System and browsers are reported in Table 2.

| Succeeded | Failed
---|---|---
| Chrome | Firefox | Other | Chrome
Chrome OS | 12 | - | - | -
Linux | 3 | 1 | 1 | 2
MacOS X 10 | 22 | 1 | - | 5
Windows 10 | 208 | 20 | 1 | 70
Windows 8 | 9 | 1 | - | 60
Windows 7 | 36 | 1 | - | 9
Total | 290 | 24 | 2 | 146

Table 2: Excerpt from the participant monitoring. Operating system and browser
are inferred from the self reported browser user agents. The client was
successfully rendered on 316 computers and failed to render on 146 computers.

##  4 Discussion

Behavioral experiments in VR often require time consuming setup and diverse
expertise in specialized hardware (e.g., physiological sensors (Weibel et al.,
2018; Hackman et al., 2019)) or infrastructure well beyond the capacity of
smaller research groups and researchers (Fischer et al., 2020; Spielhofer et
al., 2020). This difficulty, coupled with the implicit information required to
reproduce an experiment, has contributed to the reproducibility crisis across
many disciplines. We believe that this crisis can be mitigated through the use
of emergent technologies such as the ones behind the “ as Code” and “ as a
Service” movements. The “ as Code” technologies provide a tool-set for
automation and reproducible software environments. Together with VR
frameworks, this enables the implementation of the so far conceptual notion of
‘Preproducibility’. The “ as a Service” technologies provide outsourcing and
abstraction layers that can be used to interface with real world needs such as
the recruiting and rewarding of participants.

To explore the applicability of “ as Code” and “ as a Service” to behavioral
experiments in VR, we first define a taxonomy for behavioral studies in VR
(see Fig. 1) to define a scope in which automation could and should occur. We
argue that there is a need for such a taxonomy to create a common framework
for shareable work and modularity that can enable the reuse of individual
components. Second, we proposed a definition of the ExaC paradigm as providing
declarative or imperative automation code and digital documentation that
corresponds to each of the experiment taxonomy pillars (see Fig. 2). By
publishing their codebase, researchers offer concrete means of reproducing
their research. Hence, enabling their experiments to be reproduced, audited,
debugged, reused, and scaled by other researchers. Beyond replication, their
research can then be systematically varied to uncover robustness and
generalizability. Additionally, different experimental approaches could be
adapted to the same experiment, thus enabling triangulation (Munafò and Smith,
2018). In triangulation, the same concept is tested with different approaches
to ensure generalizability.

To elucidate the applicability of the ExaC paradigm to support Preproduciblity
in VR experiments, we present a proof-of-concept implementation of the ExaC
and apply it in a VR case-study in the domain of cognitive science. The proof-
of-concept implementation of ExaC consists of the digital documentation, the
infrastructure & environment definitions, and the SimpleVR framework.

Our proof-of-concept implementation of ExaC requires a plethora of
technologies. This might seem counterproductive at first sight as it could
require strong expertise to master. The technologies and software stacks range
from Docker, Terraform, Jupyter-Lab, Unity, to WebAssembly, and WebGL. On top,
they require using several programming languages, Golang, Python, JavaScript,
and C#. To add to this complexity, it has to integrate services that serve as
an abstract interface to the physical world, such as DeSciL and MTurk.
However, they are some of the best technologies available for each specific
task and there is currently no one-size-fits-all solution. The additional
effort will be amortized by reusing ExaC code bases. Even though monolithic
highly coupled software can provide a more homogeneous user experience, we
believe it would be difficult for it to keep up with the rapid advancement of
the technology. Furthermore, these loosely coupled modular components can be
exchanged as technologies progress or as requirements change.

Beyond the fast-paced development of technology, the loose coupling actually
reduces end-user complexity. End users are only required to change the
components they work on, enabling them to consider all other components _black
boxes_. In the context of VR, this means that using ExaC allows experimenters
to focus on experiment content rather than any aspect of deployment and
provisioning. Through loose coupling and automation we can tame the complexity
and implement for the first time the concept of Preproducibility.

There remain many technical challenges to be tackled for more general ExaC
implementations beyond the proof of concept. For example, interfaces for
specialized local or remote hardware, and specific protocols for composition,
description and management of services in the context of experiments need to
be developed.

In our case study, we leveraged the ExaC paradigm to scale and communicate the
experimental efforts. This enabled us to overcome the limitations of the
COVID-19 pandemic lockdown that put on hold in-person experiments. The
challenges regarding reproducibility are specific to the experiment’s nature,
requirements and constraints. We chose this simple desktop VR experiment as
proof-of-concept to showcase all the benefits of its as code representation.
We showcase the reproducibility of our proof-of-concept experiment by
analyzing two complete instantiations of the experiment. We show that the
pattern of results remains the same, i.e., ‘Treatment A’ displays a null
effect while ‘Treatment B’ displays a statistically significant improvement in
comparison to the ‘Control’ building. We showcase the auditability,
debuggability, and reusability of the experiment by performing a task common
in open source projects, i.e., a simulated auditor opened a GitHub issue and
provided a description of a bug and a fix request, while the repository
maintainer identified the required changes in the code and developed that fix.
Additionally, we show the implementation of a second experiment by changing
the data collection module and reusing the rest of the components. We
demonstrate the experiment scaling in two ways, by enabling several sessions
of data collection (horizontal scaling) and by recruiting participants from a
big remote participant pool in comparison to the usually smaller local
participant pools (vertical scaling). Even though our VR experiment was
graphically demanding and data heavy, we reached 462 participants of whom the
majority (68%) had computers that were deemed capable of running the desktop
VR experiment. From this pool, 149 participants successfully completed our
experiment. This showcases the scaling potential of this experiment.

Many experiments pose more challenging reproducibility requirements, e.g.,
interfacing with specialized local hardware, or behavioural response changes
due to nuanced variations in the setup, sensors or equipment. Researchers can
document as ExaC the solutions required to successfully run the experiment,
e.g., local server setup and hardware interfaces, sensor configuration, or
specialized hardware setup. This would enable themselves and third parties to
reuse the solutions and would make the nuances transparent. With an ExaC
ecosystem, researchers could focus on the specifics of their experiments while
reusing parts of existing and tested experimental setups.

##  5 Conclusion

Communicating scientific results requires enumerating, recording, and
reporting those things that “cannot with advantage be omitted” (Popper, 1992,
p.44). Depending on the discipline, Preproducibility (Stark, 2018) might
require information about materials (including organisms and their care),
instruments, and procedures; experimental design; raw data at the instrument
level; algorithms used to process the raw data; computational tools used in
analyses, including any parameter settings or ad-hoc choices; code, processed
data and software build environments; or even analyses that were tried and
abandoned. In this paper we provide a conceptual grounding for ExaC based on
the Preproducibility. We provide a taxonomy of the components for a practical
implementation and provide a proof-of-concept implementation showcasing its
reproducibility, auditability, debuggability, reusability, and scalability of
the ExaC paradigm.

There are other systems envisioned to define precise courses of action,
including what is known as _Smart Contracts_ a fundamental building block of
_decentralized apps_ (Wang et al., 2019), based on blockchain technology.
However, in contrast to what is attempted in the area of _Social Computing_
(Messinger et al., 2009), we are not primarily interested in automation but in
reproducibility. A desirable positive reward-based system could take advantage
of both technologies. Stake holders could create rewards pools that encourage
the reproduction of experiments in a decentralized manner. Interested parties
could earn these rewards by providing proof of reproducibility or reproducing
experiments. All this could be scaffolded on _decentralized apps_ , blockchain
technology and ExaC.

It is well-known that automation of social processes can also have undesirable
implications (Helbing, 2015). These problems range from loss of creative
freedoms to new forms of exploitative, or alienating labor. We would like to
distance ourselves from these practices. We believe that ExaC based
experiments can mitigate these problems by ensuring transparent and auditable
interactions with a society where fair practices such as rewards based on
decent wages are enforced.

The ExaC paradigm is applicable beyond the narrow frame of VR experiments if
we can define consistent abstraction interfaces to non-virtual components.
Going beyond outsourcing research to specialized groups and facilities, the
world advances with services that provide additional layers of abstraction.
Such services allow us to interface with non-virtual components (e.g., MTurk,
DeSciL). Moreover, the ability to interface with reality from virtual worlds
would enable innovative experiments based on virtual components, e.g., digital
twins (Grieves and Vickers, 2017; Grübel et al., 2021), enhanced ABMs, and
multi-agent-systems (Helbing, 2012; Grübel et al., 2019; Schamann et al.,
2016; Aguilar et al., 2014, 2015, 2016, 2019; Gath-Morad et al., 2020a; Zhao
et al., 2020, 2022) and medical applications (Brander et al., 2021). In
combination with the ExaC paradigm, it becomes possible to define reproducible
virtual(ized) and mixed-virtual experiments in the real world.

There is a great potential in extending the ecosystem of “ as Code” and “ as a
Service” to research facilities. Complex experiments in physics, chemistry,
biology, and engineering could be transformed to a specification within the
ExaC paradigm. This specification could be automatically sent to specialized
facilities possessing unique equipment and domain knowledge of the best
practices, ethical requirements, and efficient procedures. However, there are
clear limitations as to how far this interfacing with non-virtual components
can reach. For example, there could be sensitive interventions that could be
dangerous or unethical to automate. Nonetheless, when the ExaC paradigm can be
applied, it contributes to the democratization and improvement of the quality
of research.

The ExaC paradigm and implementation could also provide an answer to
challenging questions about reproducibility. For example, the inability to
reproduce certain results in behavioral science could potentially be due to
behavioral drifts. The individuals participating in a study could represent a
group that has gradually changed its behavioral patterns due to societal,
technological, environmental, ecological, or economical changes. When a study
is reproduced, the previous results no longer hold as the group has shifted
from previous behavioral patterns. By providing automation code, it would be
possible for stakeholders to easily and continuously reproduce experiments and
consequently observe the behavioral drifts over longer periods and across
participants’ compositions.

The ExaC paradigm can also provide answers beyond behavioral drift by
improving the generalizability of research results. Reusing flawed experiments
over and over again could encourage the prevalence and perpetuation of bad
practices. To counter this, we believe the ExaC ecosystem would need to adopt,
document, and introduce best practices. However, the evolutive aspect of ExaC,
if widely adopted, should also on its own weed out bad experiments and replace
them with better versions. Additionally and beyond replication, triangulation
(Munafò and Smith, 2018) can be applied to systematically vary experiments to
uncover the underlying fundamental principles. Given the ability to easily
reuse existing experiments, triangulated versions can be produced to shed
light on phenomena from a different perspective.

In the future, we plan to further develop our proof of concept, providing more
features to better achieve reproducibility. Previous work on VR has already
produced robust frameworks for data collection (e.g., EVE (Grübel et al.,
2016), (Zhao et al., 2018)). We would like to integrate these frameworks to
provide improved participant management but also tools to aid the quick and
effective design of experiments and multi-user environments (Zhao et al.,
2018). Additionally, we would like to improve the analysis and data management
of experiments by integrating complete machine learning lifecycles (Aguilar et
al., 2021). Our current implementation of ExaC, SimpleExaCT, limited itself to
a descriptive non-automated digital documentation. However, the digital
documentation still holds a lot of potential for automation and enhancements.
For instance, the pre-registration of a study (e.g. with the Open Science
Framework; OSF) could be automated. DOIs for the composable components can be
minted using the API of services such as CERN’s Zenodo making the experiment
easily available to other researchers. The file services.json could serve as
an information source to procure and manage the required external abstraction
services including opening accounts, procuring resources, and exchanging
information. Finally, we want to extend the proposed taxonomy to more complete
ontological relations (Noy and Hafner, 1998; Soldatova and King, 2006) and
follow best practices in their design (Soldatova and King, 2005). Combining
all these ideas could enable the creation of robust knowledge databases
containing ExaC experiments and results.

In summary, we believe that the ExaC paradigm has a great potential to enhance
cooperation, enable equitable access to research resources, underpin research
in VR, especially in times of limited physical interaction. Moreover, we built
a proof of concept based on the proposed taxonomy and demonstrate the
possibility to implement Preproducibility in a scalable online web-based VR
behavioral experiment. Through this proof of concept ExaC implementation we
showcase reproducibility, auditability, debuggability, reusability and
scalability. The adoption of the ExaC paradigm can improve the quality of
research across several domains, by providing reproducibility, auditability,
debuggability, reusability, and scalability to experimental research.

##  6 Acknowledgements

We would like to thank Giordano Giannoccolo and Joris Stemmle from DeSciL for
their support in the data collection and MTurk experiment deployment
experience.

###  6.1 Funding

This research has been funded by the Chair of Cognitive Science and the Data
Science, Services and Systems Laboratory at ETH Zürich.

###  6.2 Conflicts of interest/Competing interests

The authors declare that they have no conflict of interest.

###  6.3 Availability of data and material (data transparency)

For the interested reader, a constrained reproducible demonstration is
provided. This constrained reproducible demonstration diverges from the case
study as it uses free/test services where possible, and is fully automated
directly through MTurk instead of using DeSciL. To reproduce the constrained
demonstrative experiment, an account in Amazon’s Mechanical Turk and Amazon
Web Services are required. The total cost of reproducing the full experiment
is estimated below 800 USD (with an average payout of 5.4 USD per participant
and about 140 participants) as of January, 2021.

###  6.4 Code Availability

The workflow required to reproduce the experiment is quite simple. The
instructions are provided in the README of the github repository (Aguilar
Melgar and GathMorad, 2021). The checklist is summarized here:

  1. 1.

Fill in or customize the variables in the Terraform script to access cloud
services (e.g., AWS, MTurk etc).

  2. 2.

Provision and deploy experiment infrastructure:
# terraform init, and # terraform apply

  3. 3.

Release experiment trials (HITs) for participants to complete through the
provided MTurk’s scripts

  4. 4.

Manage and Monitor the experiment through the management scripts.

  5. 5.

Download your data and analysis results.

###  6.5 Authors’ contributions

Contributions are organized by CRediT (Contributor Roles Taxonomy) and authors
are listed in alphabetical order of the surname.

_Conceptualization_ : L. Aguilar, M. Gath-Morad, J. Grübel, D. Helbing, C.
Hölscher, R. W. Sumner, C. Zhang

_Data curation_ : L. Aguilar

_Formal Analysis_ : L. Aguilar

_Funding acquisition_ : C. Hölscher, C. Zhang

_Investigation_ : L. Aguilar, M. Gath-Morad, S. Wehrli

_Methodology_ : L. Aguilar, M. Gath-Morad, J. Grübel

_Project administration_ : C. Hölscher

_Resources_ : C. Hölscher, S. Wehrli, C. Zhang

_Software_ : L. Aguilar, J. Ermatinger, S. Wehrli, H. Zhao

_Supervision_ : C. Hölscher, R. W. Sumner, C. Zhang

_Validation_ : L. Aguilar

_Visualization_ : L. Aguilar, M. Gath-Morad, J. Grübel

_Writing – original draft_ : L. Aguilar, M. Gath-Morad, J. Grübel

_Writing – review & editing_: L. Aguilar, M. Gath-Morad, J. Grübel, D.
Helbing, C. Hölscher, R. W. Sumner, S. Wehrli, H. Zhao, C. Zhang

###  6.6 Compliance with Ethical Standards

The research with human participants was approved by the Research Ethics
Committee of ETH Zürich (2020-N-24). The participants were informed on the
study goal and gave informed consent and accepted the publication of
appropriately anonymized data.

## References

  * Aguilar et al. (2014) Aguilar L, Lalith M, Hori M, Ichimura T, Tanaka S (2014) A scalable workbench for large urban area simulations, comprised of resources for behavioural models, interactions and dynamic environments. In: International Conference on Principles and Practice of Multi-Agent Systems, Springer, pp 166–181
  * Aguilar et al. (2015) Aguilar L, Wijerathne L, Ichimura T, Hori M, Tanaka S (2015) Mixed mode large urban area tsunami evacuation considering car-pedestrian interactions. Proceedings of the Japan Society of Civil Engineers, A2 (Applied Mechanics) 71(2):I_633–I_641
  * Aguilar et al. (2016) Aguilar L, Lalith M, Ichimura T, Hori M (2016) Automatic evacuation management using a multi agent system and parallel meta-heuristic search. In: International Conference on Principles and Practice of Multi-Agent Systems, Springer, pp 387–396
  * Aguilar et al. (2019) Aguilar L, Wijerathne L, Jacob S, Hori M, Ichimura T (2019) Mass evacuation simulation considering detailed models: behavioral profiles, environmental effects, and mixed-mode evacuation. Asia Pacific Management Review 24(2):114–123
  * Aguilar et al. (2021) Aguilar L, Dao D, Gan S, Gürel NM, Hollenstein N, Jiang J, Karlaš B, Lemmin T, Li T, Li Y, et al. (2021) Ease.ml: A lifecycle management system for machine learning. In: 11th Annual Conference on Innovative Data Systems Research (CIDR 2021)(virtual), CIDR
  * Aguilar Melgar and GathMorad (2021) Aguilar Melgar L, GathMorad M (2021) SimpleExaCT. URL https://github.com/leaguilar/SimpleExaCT
  * Almaatouq et al. (2020) Almaatouq A, Becker J, Houghton JP, Paton N, Watts DJ, Whiting ME (2020) Empirica: a virtual lab for high-throughput macro-level experiments. arXiv preprint arXiv:200611398
  * Alsbury-Nealy et al. (2020) Alsbury-Nealy K, Wang H, Howarth C, Gordienko A, Schlichting M, Duncan K (2020) Openmaze: An open-source toolbox for creating virtual environment experiments
  * Annett and Bischof (2009) Annett M, Bischof WF (2009) Vr for everybody: the snap framework. In: Proceedings of IEEE VR 2009 Workshop on Software Engineering and Architectures for Realtime Interactive Systems, pp 131–132
  * Ayaz et al. (2008) Ayaz H, Allen SL, Platek SM, Onaral B (2008) Maze suite 1.0: A complete set of tools to prepare, present, and analyze navigational and spatial cognitive neuroscience experiments. Behavior research methods 40(1):353–359
  * Baumgaertner et al. (2018) Baumgaertner B, Devezer B, Buzbas EO, Nardin LG (2018) Openness and reproducibility: Insights from a model-centric approach. arXiv preprint arXiv:181104525
  * Bebko and Troje (2020) Bebko AO, Troje NF (2020) bmltux: Design and control of experiments in virtual reality and beyond. i-Perception 11(4):2041669520938400
  * Bierbaum et al. (2001) Bierbaum A, Just C, Hartling P, Meinert K, Baker A, Cruz-Neira C (2001) Vr juggler: A virtual platform for virtual reality application development. In: Proceedings IEEE Virtual Reality 2001, IEEE, pp 89–96
  * Bondi (2000) Bondi AB (2000) Characteristics of scalability and their impact on performance. In: Proceedings of the 2nd international workshop on Software and performance, pp 195–203
  * Brander et al. (2021) Brander M, Egger S, Hürlimann N, Seifritz E, Sumner R, Vetter S, Magnenat S (2021) Virtual reality human-human interface to deliver psychotherapy to people experiencing auditory verbal hallucinations, a development and usability study. DOI 29/12/2020:26820, jMIR Preprints
  * Brookes et al. (2019) Brookes J, Warburton M, Alghadier M, Mon-Williams M, Mushtaq F (2019) Studying human behavior with virtual reality: The unity experiment framework. Behavior research methods pp 1–9
  * Camerer et al. (2018) Camerer CF, Dreber A, Holzmeister F, Ho TH, Huber J, Johannesson M, Kirchler M, Nave G, Nosek BA, Pfeiffer T, et al. (2018) Evaluating the replicability of social science experiments in nature and science between 2010 and 2015. Nature Human Behaviour 2(9):637–644
  * Cherrueau et al. (2018) Cherrueau RA, Simonin M, Van Kempen A (2018) Enosstack: A lamp-like stack for the experimenter. In: IEEE INFOCOM 2018-IEEE Conference on Computer Communications Workshops (INFOCOM WKSHPS), IEEE, pp 336–341
  * Collaboration et al. (2015) Collaboration OS, et al. (2015) Estimating the reproducibility of psychological science. Science 349(6251)
  * Colquhoun (2014) Colquhoun D (2014) An investigation of the false discovery rate and the misinterpretation of p-values. Royal Society open science 1(3):140216
  * Cruz-Neira et al. (1992) Cruz-Neira C, Sandin DJ, DeFanti TA, Kenyon RV, Hart JC (1992) The cave: audio visual experience automatic virtual environment. Communications of the ACM 35(6):64–73
  * Echtler and Häußler (2018) Echtler F, Häußler M (2018) Open source, open science, and the replication crisis in hci. In: Extended Abstracts of the 2018 CHI Conference on Human Factors in Computing Systems, pp 1–8
  * Feger et al. (2019) Feger SS, Dallmeier-Tiessen S, Woźniak PW, Schmidt A (2019) The role of hci in reproducible science: Understanding, supporting and motivating core practices. In: Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems, pp 1–6
  * Fischer et al. (2020) Fischer J, Wissen Hayek U, Galleguillos Torres M, Weibel B, Grêt-Regamey A (2020) Investigating effects of animated 3d point cloud simulations on emotional responses. Journal of Digital Landscape Architecture 5:295–304
  * Gath-Morad et al. (2020a) Gath-Morad M, Aguilar Melgar LE, Conroy-Dalton R, Holscher C (2020a) Towards a cognitive agent simulation in architecture
  * Gath-Morad et al. (2020b) Gath-Morad M, Thrash T, Schicker J, Holscher C, Helbing D, Aguilar Melgar LE (2020b) How visibility matters for wayfinding in the vertical
  * GathMorad et al. (2021) GathMorad M, Melgar A, Conroy-Dalton R, Holscher C (2021) Spatial convergence predicts wayfinding performance in multilevel environments
  * Gosselin (2020) Gosselin RD (2020) Statistical analysis must improve to address the reproducibility crisis: The access to transparent statistics (acts) call to action. BioEssays 42(1):1900189
  * Grahe et al. (2012) Grahe JE, Reifman A, Hermann AD, Walker M, Oleson KC, Nario-Redmond M, Wiebe RP (2012) Harnessing the undiscovered resource of student research projects. Perspectives on Psychological Science 7(6):605–607
  * Grieves and Vickers (2017) Grieves M, Vickers J (2017) Digital twin: Mitigating unpredictable, undesirable emergent behavior in complex systems. In: Transdisciplinary Perspectives Complex Systems, Springer, Heidelberg, pp 85–113
  * Grübel et al. (2016) Grübel J, Weibel R, Jiang MH, Hölscher C, Hackman DA, Schinazi VR (2016) Eve: A framework for experiments in virtual environments. In: Spatial Cognition X, Springer, pp 159–176
  * Grübel et al. (2017) Grübel J, Thrash T, Hölscher C, Schinazi VR (2017) Evaluation of a conceptual framework for predicting navigation performance in virtual reality. PloS one 12(9):e0184682
  * Grübel et al. (2019) Grübel J, Wise S, Thrash T, Hölscher C (2019) A cognitive model for routing in agent-based modelling. In: AIP Conference Proceedings, AIP Publishing LLC, Melville, NY, USA, vol 2116
  * Grübel et al. (2021) Grübel J, Gath-Morad M, Aguilar L, Thrash T, Sumner RW, Hölscher C, Schinazi V (2021) Fused twins: A cognitive approach to augmented reality media architecture. In: Media Architecture Biennale 20, pp 215–220
  * Hackman et al. (2019) Hackman DA, Robert SA, Grübel J, Weibel RP, Anagnostou E, Hölscher C, Schinazi VR (2019) Neighborhood environments influence emotion and physiological reactivity. Scientific reports 9(1):1–11
  * Hassani and Idrissi (2018) Hassani R, Idrissi YEBE (2018) Communication and software project management in the era of digital transformation. In: Proceedings of the International Conference on Geoinformatics and Data Analysis, pp 22–26
  * Helbing (2012) Helbing D (2012) Agent-based modeling. In: Social self-organization, Springer, pp 25–70
  * Helbing (2015) Helbing D (2015) The automation of society is next: How to survive the digital revolution. Available at SSRN 2694312
  * Howie and Gilardi (2020) Howie S, Gilardi M (2020) Virtual observations: a software tool for contextual observation and assessment of user’s actions in virtual reality. Virtual Reality pp 1–14
  * Ioannidis (2005) Ioannidis JP (2005) Why most published research findings are false. PLoS medicine 2(8):e124
  * Ioannidis et al. (2015) Ioannidis JP, Fanelli D, Dunne DD, Goodman SN (2015) Meta-research: evaluation and improvement of research methods and practices. PLoS Biol 13(10):e1002264
  * Kuliga et al. (2015) Kuliga SF, Thrash T, Dalton RC, Hölscher C (2015) Virtual reality as an empirical research tool—exploring user experience in a real building and a corresponding virtual model. Computers, environment and urban systems 54:363–375
  * Ma et al. (2018) Ma X, Cackett M, Park L, Chien E, Naaman M (2018) Web-based vr experiments powered by the crowd. In: Proceedings of the 2018 World Wide Web Conference, pp 33–43
  * Maxwell et al. (2015) Maxwell SE, Lau MY, Howard GS (2015) Is psychology suffering from a replication crisis? what does “failure to replicate” really mean? American Psychologist 70(6):487
  * Messinger et al. (2009) Messinger PR, Stroulia E, Lyons K, Bone M, Niu RH, Smirnov K, Perelgut S (2009) Virtual worlds—past, present, and future: New directions in social computing. Decision support systems 47(3):204–228
  * Mossel et al. (2012) Mossel A, Schönauer C, Gerstweiler G, Kaufmann H (2012) Artifice-augmented reality framework for distributed collaboration. International Journal of Virtual Reality 11(3):1–7
  * Mottelson et al. (2021) Mottelson A, Petersen GB, Lilija K, Makransky G (2021) Conducting unsupervised virtual reality user studies online. Frontiers in Virtual Reality 2:66, DOI 10.3389/frvir.2021.681482, URL https://www.frontiersin.org/article/10.3389/frvir.2021.681482
  * Moulec et al. (2017) Moulec GL, Argelaguet F, Gouranton V, Blouin A, Arnaldi B (2017) Agent: automatic generation of experimental protocol runtime. In: Proceedings of the 23rd ACM Symposium on Virtual Reality Software and Technology, pp 1–10
  * Munafò and Smith (2018) Munafò MR, Smith GD (2018) Robust research needs many lines of evidence
  * Nierstrasz and Meijler (1995) Nierstrasz O, Meijler TD (1995) Research directions in software composition. ACM Computing Surveys (CSUR) 27(2):262–264
  * Noy and Hafner (1998) Noy NF, Hafner CD (1998) Representing scientific experiments: Implications for ontology design and knowledge sharing. In: AAAI/IAAI, pp 615–622
  * Orchard et al. (2003) Orchard S, Hermjakob H, Apweiler R (2003) The proteomics standards initiative. PROTEOMICS: International Edition 3(7):1374–1376
  * Pashler and Wagenmakers (2012) Pashler H, Wagenmakers EJ (2012) Editors’ introduction to the special section on replicability in psychological science: A crisis of confidence? Perspectives on Psychological Science 7(6):528–530
  * Popper (1992) Popper KR (1992) The open universe: An argument for indeterminism, vol 2. Psychology Press
  * Prieto-Diaz (1993) Prieto-Diaz R (1993) Status report: Software reusability. IEEE software 10(3):61–66
  * Rabinovich (2014) Rabinovich M (2014) The reproducibility versus debuggability of research. IEEE Internet Computing 18(6):4–6
  * Ratcliffe et al. (2021) Ratcliffe J, Soave F, Hoover M, Ortega FR, Bryan-Kinns N, Tokarchuk L, Farkhatdinov I (2021) Remote xr studies: Exploring three key challenges of remote xr experimentation. In: Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems, pp 1–4
  * Reardon (2016) Reardon S (2016) A mouse’s house may ruin experiments. Nature News 530(7590):264
  * Regal et al. (2018) Regal G, Schatz R, Schrammel J, Suette S (2018) Vrate: a unity3d asset for integrating subjective assessment questionnaires in virtual environments. In: 2018 Tenth International Conference on Quality of Multimedia Experience (QoMEX), IEEE, pp 1–3
  * Rivu et al. (2021) Rivu R, Mäkelä V, Prange S, Rodriguez SD, Piening R, Zhou Y, Köhle K, Pfeuffer K, Abdelrahman Y, Hoppe M, et al. (2021) Remote vr studies–a framework for running virtual reality studies remotely via participant-owned hmds. arXiv preprint arXiv:210211207
  * Schamann et al. (2016) Schamann D, Morad MG, Zinger E, Pilosof NP, Sopher H, Brodeschi M, Date K, Kalay YE (2016) A computational framework to simulate human spatial behavior in built environments. In: SimAUD 2016 Symposium on Simulation for Architecture and Urban Design
  * Schneider et al. (2018) Schneider S, Kuliga S, Weiser R, Kammler O, Fuchkina E (2018) Vreval-a bim-based framework for user-centered evaluation of complex buildings in virtual environments. In: Proceedings of the 36th eCAADe Conference, CUMINCAD
  * Soldatova and King (2005) Soldatova LN, King RD (2005) Are the current ontologies in biology good ontologies? Nature biotechnology 23(9):1095–1098
  * Soldatova and King (2006) Soldatova LN, King RD (2006) An ontology of scientific experiments. Journal of the Royal Society Interface 3(11):795–803
  * Spielhofer et al. (2020) Spielhofer R, Thrash T, Hayek UW, Grêt-Regamey A, Salak B, Grübel J, Schinazi V (2020) Physiological and behavioral reactions to renewable energy systems in various landscape types. Renewable and Sustainable Energy Reviews 135:110410
  * Stark (2018) Stark PB (2018) Before reproducibility must come preproducibility. Nature 557(7706):613–614
  * Starrett et al. (2020) Starrett MJ, McAvan AS, Huffman DJ, Stokes JD, Kyle CT, Smuda DN, Kolarik BS, Laczko J, Ekstrom AD (2020) Landmarks: A solution for spatial navigation and memory experiments in virtual reality. Behavior Research Methods pp 1–14
  * Stojnic et al. (2021) Stojnic R, Ross T, Kardas M, Kerkez V, Viaud L (2021) Papers with Code indexing service. https://www.paperswithcode.com/sota
  * Swiss Data Science Center (2021) Swiss Data Science Center (2021) Renku. https://renkulab.io
  * Thrash et al. (2015) Thrash T, Kapadia M, Moussaid M, Wilhelm C, Helbing D, Sumner RW, Hölscher C (2015) Evaluation of control interfaces for desktop virtual environments. Presence: Teleoperators and Virtual Environments 24(4):322–334
  * Tiwari et al. (2021) Tiwari K, Kyrki V, Cheung A, Yamamoto N (2021) Define: Delayed feedback based immersive navigation environment for studying goal-directed human navigation. 2003.03133
  * Tramberend (1999) Tramberend H (1999) Avocado: A distributed virtual reality framework. In: Proceedings IEEE Virtual Reality (Cat. No. 99CB36316), IEEE, pp 14–21
  * Ugwitz et al. (2021) Ugwitz P, Šašinková A, Šašinka Č, Stachoň Z, Juřík V (2021) Toggle toolkit: A tool for conducting experiments in unity virtual environments. Behavior research methods pp 1–11
  * Vasser et al. (2017) Vasser M, Kängsepp M, Magomedkerimov M, Kilvits K, Stafinjak V, Kivisik T, Vicente R, Aru J (2017) Vrex: an open-source toolbox for creating 3d virtual reality experiments. BMC psychology 5(1):1–8
  * van’t Veer and Giner Sorolla (2016) van’t Veer AE, Giner Sorolla R (2016) Pre-registration in social psychology—a discussion and suggested template. Journal of Experimental Social Psychology 67:2–12
  * Wallot and Kelty-Stephen (2018) Wallot S, Kelty-Stephen DG (2018) Interaction-dominant causation in mind and brain, and its implication for questions of generalization and replication. Minds and Machines 28(2):353–374
  * Wang et al. (2019) Wang S, Ouyang L, Yuan Y, Ni X, Han X, Wang FY (2019) Blockchain-enabled smart contracts: architecture, applications, and future trends. IEEE Transactions on Systems, Man, and Cybernetics: Systems 49(11):2266–2277
  * Wang et al. (2020) Wang Y, Ijaz K, Yuan D, Calvo RA (2020) Vr-rides: An object-oriented application framework for immersive virtual reality exergames. Software: Practice and Experience 50(7):1305–1324
  * Watson et al. (2019) Watson MR, Voloh B, Thomas C, Hasan A, Womelsdorf T (2019) Use: An integrative suite for temporally-precise psychophysical experiments in virtual environments for human, nonhuman, and artificially intelligent agents. Journal of neuroscience methods 326:108374
  * Weibel et al. (2018) Weibel RP, Grübel J, Zhao H, Thrash T, Meloni D, Hölscher C, Schinazi VR (2018) Virtual reality experiments with physiological measures. JoVE (Journal of Visualized Experiments) 1(138):e58318
  * Weiss (1980) Weiss IR (1980) Auditability of software: A survey of techniques and costs. MIS Quarterly pp 39–50
  * Welter et al. (2014) Welter D, MacArthur J, Morales J, Burdett T, Hall P, Junkins H, Klemm A, Flicek P, Manolio T, Hindorff L, et al. (2014) The nhgri gwas catalog, a curated resource of snp-trait associations. Nucleic acids research 42(D1):D1001–D1006
  * Zhao et al. (2018) Zhao H, Thrash T, Wehrli S, Hölscher C, Kapadia M, Grübel J, Weibel RP, Schinazi VR (2018) A networked desktop virtual reality setup for decision science and navigation experiments with multiple participants. JoVE (Journal of Visualized Experiments) 1(138):e58155
  * Zhao et al. (2020) Zhao H, Thrash T, Kapadia M, Wolff K, Hölscher C, Helbing D, Schinazi VR (2020) Assessing crowd management strategies for the 2010 love parade disaster using computer simulations and virtual reality. Journal of the Royal Society Interface 17(167):20200116
  * Zhao et al. (2022) Zhao H, Schwabe A, Schläfli F, Thrash T, Aguilar L, Dubey RK, Karjalainen J, Hölscher C, Helbing D, Schinazi VR (2022) Fire evacuation supported by centralized and decentralized visual guidance systems. Safety Science 145:105451