# D-VRE: From a Jupyter-enabled Private Research Environment to Decentralized
Collaborative Research Ecosystem

Yuandou Wang1, Sheejan Tripathi12, Siamak Farshidi1, Zhiming Zhao13
1Multiscale Networked System, University of Amsterdam, The Netherlands
2Department of Computer Science, Vrije Universiteit Amsterdam, The Netherlands
3LifeWatch ERIC Virtual Lab and Innovation Center, Amsterdam, The Netherlands

###### Abstract

Today, scientific research is increasingly data-centric and compute-intensive,
relying on data and models across distributed sources. However, it still faces
challenges in the traditional cooperation mode, due to the high storage and
computing cost, geo-location barriers, and local confidentiality regulations.
The Jupyter environment has recently emerged and evolved as a vital virtual
research environment for scientific computing, which researchers can use to
scale computational analyses up to larger datasets and high-performance
computing resources. Nevertheless, existing approaches lack robust support of
a decentralized cooperation mode to unlock the full potential of decentralized
collaborative scientific research, e.g., seamlessly secure data sharing. In
this work, we change the basic structure and legacy norms of current research
environments via the seamless integration of Jupyter with Ethereum blockchain
capabilities. As such, it creates a Decentralized Virtual Research Environment
(D-VRE) from private computational notebooks to decentralized collaborative
research ecosystem. We propose a novel architecture for the D-VRE and
prototype some essential D-VRE elements for enabling secure data sharing with
decentralized identity, user-centric agreement-making, membership, and
research asset management. To validate our method, we conducted an
experimental study to test all functionalities of D-VRE smart contracts and
their gas consumption. In addition, we deployed the D-VRE prototype on a test
net of the Ethereum blockchain for demonstration. The feedback from the
studies showcases the current prototype’s usability, ease of use, and
potential and suggests further improvements.

###### Index Terms:

Blockchain, Jupyter Integration, Secure Data Sharing, User-centric Policy-
Making, Virtual Research Environment, Web 3.0

##  I Introduction

In the contemporary era, data is at the center of most scientific research,
particularly in fields such as medical research [1], climate science [2], and
biodiversity [3], which usually require data and analysis models across
distributed sources. To tackle complex scientific problems, teams of
researchers used to travel to different institutes, reserving time slots to
discuss collaborations, collect data, and then analyze the data afterward [4].

The Jupyter111https://jupyter.org/, well-known as data scientists’
computational notebook of choice, has played a vital facilitating role in such
processes for scientific computing [5]. On the one hand, Jupyter is already
well-established in the scientific toolbox for small-scale, single-node
exploratory analysis with the Jupyter Notebook interface.
JupyterLab222https://github.com/jupyterlab/jupyterlab can support custom
applications and extensions that live alongside the core Notebook interface.
JupyterHub333https://github.com/jupyterhub/jupyterhub can serve a variety of
research environments that can be configured with authentication to provide
access to a subset of users to share data and scientific workflows remotely.
On the other hand, Jupyter shares some features with Virtual Research
Environments (VREs) and Science Gateways (SGs), for example, they are Web-
based platforms providing scientists with computational services. Over the
past couple of years, many studies have explored how it would be beneficial to
scale computational analyses up on larger datasets and compute resources using
Jupyter [6, 7, 8, 9, 10, 11, 12]. It is known that the current success of the
traditional cooperation mode should be attributed to the centralized research
data lifecycle governance pattern [13] that usually covers identity
management, sharing policies, research assets transfer, workflow
orchestration, and infrastructure management. There are a few works in which
the sharing policies are in whole or part based upon terms defined by the end-
users. For example, D4Science [14] allows users to define authorization
policies that satisfy the authorization requirements using the policy
language. WorkflowHub [15], based on the FAIRDOM-SEEK
framework444https://docs.seek4science.org/, https://fair-dom.org/, allows
users to choose different levels of sharing permissions pre-defined in the
framework. The fact is that the traditional centralized cooperation mode is
not a preferable choice today, with the advent of the explosive burst of big
data. The fully centralized pattern has inherent disadvantages, including
increased data traffic and concerns about data ownership, privacy, security,
and confidentiality [16]. Undoubtedly, the increasing data volumes
significantly require large storage and computing costs in the centralized
management paradigm [17]. Noticeably, data assets are naturally decentralized
and stored on geo-graphically distributed infrastructures or facilities. Data
policies and compliance require researchers or data scientists to include
robust and detailed plans for how the research data will be managed and shared
during the entire lifecycle of their research. Therefore, it still faces
challenges in research asset management due to geo-location barriers and local
confidential regulations.

We hypothesize that the decentralized cooperation mode would overcome the
current weaknesses of centralized ones, and accommodate inherently
decentralized data and sharing policies regarding confidentiality and workflow
efficiency in the lifecycle of scientific research. In the past few years,
impressive progress has been achieved by decentralized models using blockchain
[18, 19] that remove the central control elements from the system and
decentralize its role for various purposes, e.g., enabling controllable remote
computing for sharing based on a zero-trust decentralized identity
infrastructure [20] and confidential distributed machine learning [16].
Decentralized Science (DeSci) and Decentralized Artificial Intelligence (DeAI)
are hot topics emerging with the advent of the infrastructure of Web 3.0 in
recent years [17, 13, 21, 22]. DeSci [21] is an attempt to solve the
bottleneck problems about oligarchy and data silos, and make science more
fair, accessible, responsible, and sensitive. Blythman et al. [17] explore the
potential of decentralized technologies for DeAI to address the limitations
caused by current centralized solutions. However, since work is still in its
very early stages, not all aspects are managed in a decentralized manner for
collaborative research.

Our prior work [7, 23] has demonstrated how Jupyter Notebook as a VRE can
provide user-centric support in the lifecycle of research activities, e.g.,
discovering and accessing research assets or composing application workflows,
and executing them on Cloud infrastructures. However, it still lacks built-in
decentralized cooperation capabilities to speak various decentralized
computing platforms, applications, or custom sharing policies. In this
article, we target leveraging decentralized technologies and Jupyter to bridge
this gap and propose a decentralized VRE (D-VRE) framework to serve as a
reference for the development and implementation of such a decentralized
cooperation mode in Jupyter-enabled virtual research environments. The main
contribution of this work can be summarized as follows:

  * •

We characterize existing research environment exemplars by comparing the
management of identity, asset, and infrastructure, as well as workflow
orchestration.

  * •

We propose the D-VRE framework for decentralized collaborative research within
a human-in-the-loop scientific research lifecycle context.

  * •

We propose a detailed architecture design and description of the D-VRE
components and the custom smart contract-based sharing policy as its core
functionality.

  * •

We provide a Proof-of-Concept (PoC) implementation of the D-VRE architecture,
and its evaluation by an experimental study and a case study.

The reminder of this article is structured as follows: Section II review the
current state of existing VRE-related exemplars and analyzes research gaps.
Section III presents the conceptual framework, architecture design, and
technology choices of the proposed D-VRE. Section IV details the PoC
implementation, experimental and case studies, and its evaluation. In Section
V, we discuss open challenges and future improvements; and finally, Section VI
concludes this work.

##  II Related Works

This section characterizes the existing works into two dimensions of unity of
opposites: centralized vs. decentralized management of assets and co-located
vs. distributed infrastructure resources, as presented in the axis of Figure
1. To make the analysis deeper, we focus on the four main aspects: ( )
identity management, ( ) sharing policy, ( ) asset transfer, and ( ) workflow
orchestration.

Figure 1: An overview of comparisons of existing research environment
exemplars by comparing the ways of identity management ( ), sharing policy (
), assets transfer ( ), and workflow orchestration ( ).

Jupyter Notebook, as a computational notebook, is typically used for small-
scale and single-node analysis, such as on personal computers or co-located
infrastructure. Over the past few years, several studies have implemented
interactive large-scale services with Jupyter to scale computational analyses
on HPC, cloud, and hybrid HPC-Cloud scientific computing facilities. These
approaches utilize Jupyter as a companion application to gateways, lowering
the entry barrier to various computational resources on large servers,
clusters, or distributed infrastructures. Examples of these implementations
include the CyberGIS-Jupyter framework [10], HPC Notebooks services [24],
Jupyter at NERSC [4], and deployment of JupyterHub on XSEDE resources [11].
Although these works support assets (e.g., data and/ or analyses) transfers
across distributed sources, management remains centralized as the computing
center typically handles the operations and maintenance of the platform.
Besides, the authentication scheme was usually implemented by reverse HTTP
proxy with third-party providers for authentication. In some instances, the
back-end comprises a federated computing system involving multiple
organizations. Nevertheless, the decision-making layer remains centralized,
relying on the consensus mechanism among the participating parties.

Apart from the deployment of the Jupyter environment on HPC, VRE exemplars
such as D4Science [14] and CERN VRE [6] are often implemented as an integrated
environment, incl., a catalog of research assets, a workflow management
system, a data management layer, and a bunch of tools that enable user
collaboration. Both CERN VRE and D4Science integrate the Jupyter for
documenting and recording analytics processes. The D4Science computing
infrastructure is geographically distributed across four main sites [8] and
managed across different administrative domains. Since D4Science relies on a
single entity to store and manage user data, it should be attributed to
centralized identity management.

Google Colab is a hosted Jupyter Notebook service that provides free-of-charge
access to computing resources, including Graphics Processing Units (GPUs) and
Tensor Processing Units (TPUs), especially well suited to machine learning,
data science, and education555https://colab.research.google.com/. It has a
built-in mechanism in the Web-based interface to copy or share a Notebook.
Since the computing resources and the OAuth service for authentication are
mainly from Google, we place its identity management in the centralized zone.
Colab shares some features with D4Science about the sharing policy definition,
for instance, both can manage access control by defining who has what access
for which resource. Based on this, we pose them in the decentralized
management zone regarding the sharing policy aspect.

WorkflowHub [15], evolved by
myExperiment666https://www.myexperiment.org/about, is a collaborative
environment where scientists can safely publish their workflows and in silico
experiments, share them with groups and find those of others [25]. It is a
domain-agnostic workflow registry777https://workflowhub.eu/ designed around
FAIR (Findable, Accessible, Interoperable, and Reusable) principles.
WorkflowHub brings together scientists from different countries and across
individual research institutes, to contribute to the federated registry.
WorkflowHub and JupyterHub share some features that allow users can organize
space, teams, and their workflows; yet, managing JupyterHub on a cluster is
more centralized than WorkflowHub, in terms of the workflow orchestration and
user identities. Since the main development, hosting, and maintenance of the
WorkflowHub is done by a single institute, we see that identity, workflow
orchestration, and asset transfers are still in centralized cooperation mode.
However, it provides different levels of sharing permissions for users to
define access control to various research assets within or across teams or
organizations. Hence, the activities regarding sharing policies behave as
decentralized.

With the advent of the infrastructure of Web3, decentralized cooperation
solutions have emerged for data-centric scientific research, that aims to
address the shortcomings of the current centralized solutions. Integrating
decentralized technologies, such as blockchains, smart contracts,
decentralized storage and protocols, and Web3 wallet for research data
management marks a significant shift towards decentralized data governance and
autonomy over research lifecycle management [17, 16, 21]. Cao [22] describes a
vision of decentralized intelligence systems and services leveraging
intelligence at edge, blockchains, Metaverse, Web3, and DeSci. Blythman et al.
[17] explore Web3 wallets, Peer-to-Peer (P2P) marketplaces, decentralized
storage and compute, and decentralized autonomous organizations to address the
limits of the centralized management, decisions, and governance for DeAI.

Although these visions and approaches are close to the decentralized
capabilities in our work; there is a big difference between their works and
ours. This work, based on our prior work NaaVRE [7], aims to bridge the gap
between Jupyter-enabled VREs and decentralized cooperation modes with identity
management, asset sharing and custom policy definitions, and decentralized
workflows (See in Figure 1).

##  III Decentralized Virtual Research Environment

This section illustrates the system design and technical architecture of the
D-VRE that bridges the research gaps identified in Section II.

###  III-A Conceptual Framework for D-VRE

From the foundation of the embedded VRE solution using Jupyter [23, 7], we
have an attempt to create a decentralized collaborative environment in the
community by embedding essential D-VRE components and corresponding logical
layers. Figure 2 provides a high-level overview of the D-VRE conceptual
framework, starting from the Jupyter-enabled private research environment.

Figure 2: An overview of the D-VRE conceptual framework with the key
components and logical layers.

####  III-A1 Jupyter-enabled Private Research Environment

A research environment can be a collection of research assets, storage and
computing, and software tools supporting research activities and managing
research assets within the lifecycle context, e.g., the database, e-notebooks,
repositories, local storage and computing, and private cloud resources. In the
context of the research lifecycle, this space is usually responsible for
independent research and creating knowledge within a private zone. This work
has chosen JupyterLab as the interactive computing environment rooted in the
private research environment, because it allows users to customize the
behavior of the various pages for designing D-VRE components. Such a design
allows users to seamlessly access D-VRE features through Graphical User
Interfaces (GUIs) and services without an extra learning curve while reducing
the effort. It can also save storage and computing costs by avoiding creating
many comprehensive VREs. In other words, if the existing research platforms on
HPC or Clouds can use JupyterLab as the companion application, they will also
be able to seamlessly install and access to D-VRE components to activate the
blockchain capabilities instead of reinventing the wheel.

####  III-A2 D-VRE Components

We propose the following essential components for enabling decentralized
identity, custom sharing policy definition, secure data transfer or sharing,
and decentralized workflow management in the Jupyter-enabled private research
environment.

Auth via Digital Wallet shall ensure the decentralized identity function for
distributed users to engage into the D-VRE network via individual digital
wallets. It allows users and entities to join the D-VRE network effortlessly,
enabling interaction with online services as authenticated users without a
centralized oversight body, which emphasizes accessible yet secure engagement
in the blockchain networks.

Make Agreements, as one of D-VRE’s most novel features, shall empower end-
users to forge and customize the collaboration agreements, ensuring that
precise, mutually agreed-upon terms govern collaborations while automating
smart contracts. This aspect is vital for delineating the scope and rules of
engagement within collaborative research.

Membership shall allow a collaboration initiator or creator to assign several
users to become group members to access the research assets under the defined
access conditions quickly. This component facilitates the efficient management
of virtual research groups in the D-VRE network.

Asset Manager is the component that streamlines the administration of research
assets for end-users. This function shall facilitate users in managing the
assets with diverse groups while catering to both shared and private data.

Workflow Manager, coupled with the Asset Manager, is the module that enables
collaborative workflow management. Our use cases include federated learning
workflows with decentralized data stored on heterogeneous infrastructures [26]
and scheduling privacy-aware workflows over hybrid Clouds [27]. This function
shall be integral to the collaborative PoC workflows in the D-VRE ecosystem
and enhance the efficiency and effectiveness of research collaborations within
private groups.

Provenance Manager is an essential feature that shall maintain a transparent
record of any changes made to a digital research object. We consider a
combined solution to this layer, for example, utilizing secure transparent
records of blockchain and research metadata approaches, e.g., Research Object
Crate (RO-Crate) [28] and Schema.org [29].

####  III-A3 Logical Layers

In the framework, we consider the following five logical layers to achieve the
above D-VRE components.

Research Activity Layer is towards individual users, organizations, and
research communities. Research activities among distributed researchers in
scientific communities might vary across different disciplines. In this layer,
we point out several main activities related to the research collaboration —
viz discovery of resources, making agreements for sharing research assets that
cover ownership, access control, and data usage rights, as well as workflow
composition and publishing.

Access Policy Layer aims to collect users’ input for making agreements from
the human-in-the-loop interactive interface and translate it into the D-VRE
system’s agreement policy and access protocol. This layer shall facilitate
efficient agreement-making and enforce the access control policies.

Encryption and Decryption Layer is a safeguard of the secure connection
between the Jupyter-enabled private research environment and the D-VRE
ecosystem. Cryptography is crucial for data integrity, availability, and
confidentiality. We assume that decentralized signing and encryption can be
embedded in the Jupyter-enabled private research environment. Thus, this layer
shall introduce robust cryptographic solutions for the system with them.

Digital Object Storage Layer. Considering the long-live research activities,
the system shall provide robust data storage solutions and archiving
capabilities for various data types, e.g., documents, project code files,
datasets, AI models, and e-notebooks. In accordance with the FAIR principles,
the digital object shall be stored with the data itself, a variable amount of
metadata, and a globally unique identifier [30], making it findable,
accessible, interoperable, and reusable.

Distributed Ledger Layer. It is known that a distributed ledger is the
consensus of shared and synchronized across various sites, institutions, or
geographical locations, and accessible by multiple users. Any changes or
additions made to the ledger are instantly or within minutes reflected and
copied to all participants [31]. The most famous one is the blockchain
network. This layer shall support the underlying data management with such
features.

Figure 3: An overview of the D-VRE architecture, in conjunction with the
individual’s private research environment (left), D-VRE middleware layer and
the management of asset, workflow, and infrastructure above (middle), and
different potential customers in the blockchain network (right).

###  III-B D-VRE Architecture

Based on the conceptual framework design, we propose the D-VRE system
architecture, as illustrated in Figure 3. It visually represents the overview
of the D-VRE, and highlights the modules within the D-VRE middleware and their
interactions between the individual’s private research environment and the
decentralized network hosting different customers. Although users’ private
research environments may vary in the great research picture, the D-VRE
middleware shall be able to adapt to various environments via Jupyter. Thus,
individual users, institutes, or groups using Jupyter can easily engage in the
community.

In our design, the metadata of research objects, including research assets,
distributed workflows, computing resources, and access policies, shall be
published and stored on the blockchain, e.g., the Ethereum blockchain. An
Ethereum account could work across public networks (e.g., Ethereum Mainnet and
Testnets) and private networks (e.g., development and consortium networks).
Stakeholders can choose a preferable operational model to construct blockchain
networks based on their requirements and budgets. Leveraging blockchain, the
system can verify the processes and automatically deploy a number of immutable
smart contract instances in the blockchain network. It thus ensures that all
participants adhere to the agreed-upon terms. Consequently, it can pave a
powerful way to enhance the integrity, traceability, and security of the
digital objects.

Regarding the application scenarios, we refer to the prior research experience
and user requirements from several EU projects such as
CLARIFY888http://www.clarify-project.eu/, BlueCloud2026999https://blue-
cloud.org/about-blue-cloud-2026, ENVRI-Hub NEXT101010https://envri-
hub.envri.eu/, and LifeWatch ERIC111111 https://www.lifewatch.eu/. In general,
there are three main categories regarding collaboration and sharing: (1)
research asset sharing, (2) collaborative workflows in machine learning and
data science, and (3) federated infrastructures. This paper focuses on
research asset sharing. Regarding collaborative workflows with D-VRE, we have
considered potential applications for data quality control (ocean data and
medical images), digital twin applications, and distributed machine learning
workflows (at the training phase and inference phase, respectively).
Furthermore, participants in the D-VRE might want to share idle storage and
computing resources. From the perspective of federated infrastructure,
examples include outsourcing computation tasks to remote infrastructure in
multi-institute collaborations.

###  III-C Technology Choices

After the analysis of the literature survey and technology investigation, we
select the following key technologies.

Web Technology is critical for developing Web-based applications. It can be
utilized to design and implement the user interface and interactions that
present information to users, capture user input, and access functions and
services. Our D-VRE shall involve both Web 2.0 and Web 3.0 technologies. To
develop D-VRE GUIs, we have considered common Web 2.0 technologies such as
HTML, CSS, and JavaScript (JS), including Node.js, Express.js, Vue.js, and
React. We investigate Node.js and React for the webpage design. Regarding Web
3.0, we have investigated the following key technologies for further
implementation.

Blockchains are the first citizens of Web 3.0. We look at Ethereum Virtual
Machine (EVM)-compatible blockchains (e.g., Ethereum121212 Ethereum.
https://ethereum.org/, Polygon131313Polygon. https://polygon.technology/,
Avalanche141414Avalanche. https://www.avax.network/) and non-EVM blockchains
(e.g., Hyperledger Fabric151515Hyperledger Fabric.
https://www.hyperledger.org/projects/fabric and Cardano161616Cardano.
https://cardano.org/). We have chosen Ethereum in this work because of the
potential of full decentralization (e.g., public blockchain networks), native
cryptocurrency support, and a broad developer community. Ethereum provides a
comprehensive development toolchain for researchers and developers to
implement smart contracts, DApps, and secure transactions between nodes in the
network using Web3.js and Ethers.js. This is crucial for custom agreements or
sharing policies and automatically enforcing and executing contract terms when
predefined conditions are met.

Digital Wallet, which injects Web3 objects into the webpage’s JavaScript
content, enables D-VRE components to interact with the decentralized Web, via
individual user’s browsers (e.g., Chrome, Firefox, and Safari). We have chosen
Metamask [32], a self-custody digital wallet, for the first component
implementation. MetaMask can act as the user’s digital identity and bank. It
allows users to sign transactions and messages without revealing their private
keys and offers a secure way to prove ownership and authorize processes.

Lit Protocol. To enhance the distributed trust and security of the system, we
also investigate advanced security mechanisms for encryption and decryption
layer. It is well-known that MetaMask as a self-custody wallet might face the
issue of a single point of failure. To address this, we have chosen Lit
protocol, as the key management modular and enhance the secure data sharing
with access conditions. It distributes encrypted key shares across the Lit
network by harnessing multi-party computation, threshold secret schemes, and
trusted execution environments to ensure that node operators never have access
to key shares or computations processed within the nodes. Lit allows users to
manage sovereign identities on the open Web with self-defined flexible access
control171717Lit protocol. https://www.litprotocol.com/, ensuring robust,
efficient, and secure DApp solutions embedded in the system.

IPFS, a.k.a., InterPlanetary File System [33] can provide a set of composable,
P2P protocols for addressing, routing, and transferring content-addressed data
in a decentralized file system181818IPFS. https://docs.ipfs.tech/. The main
reasons why we choose IPFS include (1) it can lower the cost associated with
data storage on the blockchain, (2) the adoption of IPFS enhances data
availability and persistence, and (3) it aligns with the open-source and
academic communities’ preference for cost-effective decentralized data
management solutions.

##  IV Prototype, Experiments, and Evaluation

This section illustrates the current implementation of the D-VRE prototype,
details the experimental study, and demonstrates the visualized results with a
case study and its validation.

⬇

1

2 contract GroupContract {

3 struct ContractDetails {

4 string groupName;

5 address groupOwnerAddress;

6 string permissions;

7 string[] organizations;

8 string[] countries;

9 }

10 …

11 modifier onlyGroupOwner() {

12 require(msg.sender == contractDetails.groupOwnerAddress, ”Only group owner
can call this function”);

13 _;

14 }

15 function addFilesToGroup(FileDetails[] memory fileDetails) public {

16 for (uint256 i = 0; i < fileDetails.length; i++) {

17 FileDetails memory fDetail = fileDetails[i];

18 string memory IPFSHash = fDetail.IPFSHash;

19 sharedIPFSHashes[IPFSHash] = true;

20 addedFileDetails.push(fDetail);

21 emit Success(”Files successfullly shared in the group”);

22 }

23 }

24 function associateUsersToGroup(UserInput[] memory users) public
onlyGroupOwner{

25 for (uint i = 0; i < users.length; i++) {

26 setUserAccess(users[i].eoaAddress, users[i].accessFrom, users[i].accessTo);

27 }

28 emit Success(”Users successfully added to the group”);

29 }

30 }

Figure 4: Example of the GroupContract smart contract.

###  IV-A Prototype Implementation

We have configured INFURA191919INFURA. https://docs.infura.io/api and
Pinata202020Pinata. https://docs.pinata.cloud/api-reference/introduction API
keys to simplify the management of blockchain infrastructure and IPFS. We
implemented smart contracts in Solidity 0.8.0 and developed custom Web3
applications and extensions for D-VRE, which are seamlessly integrated into
the JupyterLab environment. To evaluate the D-VRE prototype in a realistic
setting, we deployed these smart contracts on one of the Ethereum Testnets —
viz the Sepolia network. Moreover, it is worth noting that the demonstration
and case study results presented here are, in whole or in part, based on the
collaborative research activities in the CLARIFY project.

####  IV-A1 D-VRE Smart Contracts

In this prototype, we implemented four smart contracts, namely
UserMetadataFactory, UserMetadata, PolicyManager, and GroupContract, by
leveraging the idea of human-in-the-loop smart contracts. The first two
contracts were designed for user identity management, for example, user
registration or login, authentication, and authorization. And the user
metadata factory is the parent contract of the user metadata contract.
Similarly, the group contract is the child contract of the policy manager.
Figure 4 is an example of Solidity implementation of the GroupContract. Any
user with an Ethereum Externally Owned Account (EOA) can create private groups
to become a group owner. In the ContractDetails struct, it contains the group
name, group owner’s address, permissions, organizations, and countries, as
well as a modifier, and two functions in the listings. The group owner can
invite other users to become group members via the function
associateUsersToGroup. Participants in the group can share digital objects via
IPFSHash to reduce the storage cost in the blockchain through the
addFilesToGroup function. Note that we also implement the access duration
control as some shared data might be timeliness within a research lifecycle
context.

####  IV-A2 Lit-based Encryption and Decryption

IPFS uses transport encryption but not content encryption, if anyone has the
content identifier (IPFSHash code) of a digital object, then he or she can
download and view that data via the IPFS gateway. To address this issue, we
leverage Lit to encrypt the content stored in IPFS and guarantee that only
registered users can decrypt it with qualified access control conditions.

Output: a zipped folder with the encrypted file and its metadata $c$ and the
response $r$.

1

Input: Key of a user’s local storage $k$, litNetwork.

2

3$f,state\leftarrow$ selected file from local storage with $k$;

4client $\leftarrow$ litNodeClient(litNetwork);

5accs $\leftarrow$ define the access control conditions;

6authSig $\leftarrow$ check and sign on Ethereum chain;

7$c\leftarrow$ encryptFileAndZipWithMetadata(accs, chain, $f$, authSig,
client, readme);

8$r\leftarrow$ respond to uploading $c$ and $f.name$ to IPFS gateway;

return $c$, $r$.

Algorithm 1 EncryptFileAndUpload

Output: a zipped folder with the decrypted file and its metadata $f$ and the
response $r$.

1

Input: IPFS gateway, $c$, litNetwork.

2

3client $\leftarrow$ litNodeClient(litNetwork);

4authSig $\leftarrow$ check and sign on Ethereum chain;

5$f\leftarrow$ decryptZipFileWithMetadata($c$, authSig, client)

6$r\leftarrow$ respond to downloading $f$ with metadata from IPFS gateway;

return $f$, $r$.

Algorithm 2 DecryptFileAndDownload

The main procedures for encrypting files and uploading them to IPFS have been
written in the algorithm pseudo-code 1. The user can create an access control
condition and combine it with selected files, e.g., stored in IPFS. The user
holds the encryption metadata, i.e., access control condition, IPFSHash, the
time window of the private data, and a ciphertext. This mechanism guarantees
that only authorized users can successfully decrypt the ciphertext and access
the shared data while reducing the data storage cost in the blockchain. More
decryption details have been presented in Algorithm 2.

For more complex security or access control requirements, we can customize the
access control condition definitions. For example, the security module be
adopted in various application scenarios by setting the parameters, such as
blockchain network or chain name, lit node network name, and so forth.

Figure 5: The screenshot of the embedded D-VRE components in the JupyterLab
environment: ❶ Auth via MetaMask, ❷ Make Agreements, ❸ Membership, ❹ Asset
Manager. Figure 6: An overview of the collection of the D-VRE GUIs: (a) Auth
via MetaMask, (b)Make Agreements, (c) Membership, and (d) Asset Manager.

####  IV-A3 Jupyter Integration

Figure 5 presents the screenshot of our current D-VRE prototype implementation
embedded in JupyterLab. Four out of six (4/6) essential D-VRE modules have
been seamlessly integrated into the launcher of the JupyterLab environment: ❶
Auth via MetaMask, ❷ Make Agreements, ❸ Membership, ❹ Asset Manager. In Figure
6, we present a collection of critical GUIs from these four components. Note
that the rest features of D-VRE, such as Workflow Manager and Provenance
Manager, are still under development and work in progress.

####  IV-A4 Auth via MetaMask

Figure 6(a) presents the screenshot of the login page in the ❶ Auth via
MetaMask component, where Jupyter users can register as D-VRE users and login
to the D-VRE system via Metamask. Specifically, each user can register with
the MetaMask’s publicAddress (e.g., 0x680Axxxx810A), the username,
organization, and country details. When interacting with D-VRE, the user’s
MetaMask public address is by default associated with the D-VRE account for
decentralized signing and encryption.

####  IV-A5 Make Agreements

In the ❷ Make Agreement component, we have implemented flexible group creation
with custom terms of agreement. Figure 6(b) presents the details of this
feature, including custom policies of data sharing, uploading assets (files)
to IPFS, and group creation details. The user can first upload an asset, for
example, an image titled #binary#mask.png from the local system to IPFS. In
the PoC implementation, the IPFS storage was configured with the free-of-
charge account of the Pinata IPFS gateway, which freely supports a total of
500 pinned files and 1 GB of storage per workplace member. Then, the user can
add one or multiple groups for data sharing purposes by customizing the name —
DataSharing, permission settings — Full Access, cross-organizational
collaboration among UvA, UiS, and UPV, and countries — Netherlands, Norway,
and Spain. The user saves the input by clicking on the button SAVE POLICIES.
Finally, the system will automatically generate a group smart contract
instance, written into code and deploy it to the Ethereum network.

####  IV-A6 Membership

As shown in Figure 6(c), a group owner can increase group members by adding
the EOA address of the users, which refers to the user’s publicAddress
introduced in the Metamask. Meanwhile, the user can customize the access
period of the group and shared research assets inside. For example, the user
(i.e., 0x680Axxxx810A) was selected by the DataSharing group owner and granted
unlimited access to the shared data, available from 2024-03-27 to 2024-03-29
in the ❸ Membership page.

####  IV-A7 Asset Manager

Figure 6(d) showcases the webpage of an authenticated user in the ❹ Asset
Manager interface. It lists the group and the shared files in the group. The
authenticated user or group member (e.g., 0x680Axxxx810A) can select local
files to upload to IPFS for group sharing. In addition, the user can view and
download the shared file #binary#mask.png. If a user has become a group member
and has been granted access permission, he will see all the involved groups
and shared files through the action buttons VIEW and DOWNLOAD.

###  IV-B Experimental Study

To simplify the analysis of gas usage of the smart contract implementations in
our D-VRE, we conducted an experimental study by deploying the implemented
smart contracts on the Ganache212121Ganache.
https://github.com/trufflesuite/ganache. As an essential tool in the Ethereum
development ecosystem, it provides a personal blockchain that developers can
use to deploy contracts, run tests, and execute commands without the need for
a live network. We leverage the retrieved first account from the local Ganache
node to execute the interfaces and estimate different types of gas according
to the D-VRE smart contract implementations. Figures 7 and 8 present the
results of this experimental study.

Figure 7: An overview of the gas usage for the D-VRE smart contract
deployment. Figure 8: An overview of the gas consumption of all functions in
smart contracts.

We compare the gas units consumed by the deployment of different smart
contracts, as presented in Figure 7. It is obvious that deploying parent
contracts i.e., PolicyManager and UserMetaFactory, consumes more than child
smart contracts, i.e., GroupContract and UserMetadata, respectively. The first
place (PolicyManager, 2,738,927 gas used) has 498,248 more than the second one
(UserMetaFactory, 2,249,679 gas used). In contrast, UserMetadata contract
costs the lowest (1,602,341 gas units), less than the GroupContract (1,917,322
gas usage), and their difference is 314,981. Figure 8 compares the gas used
for all functionalities in each smart contract. It is noticeable that the
function createGroupContract from the PolicyManager smart contract used the
most gas. Similarly, the function createUserContract places the second.
Notably, the gas costs for creating a group and user metadata contracts are
estimated at 1,832,050 and 1,535,460, respectively, far above (nearly 7 and 6
times) the used gas of other functions.

These differences in gas consumption highlight the operations’ efficiency and
resource use of the current implementations. There is still optimization space
to reduce gas consumption by optimizing interface design and implementation in
smart contracts.

###  IV-C Feedback from the Case Study

Analysis of the researchers’ feedback is beneficial for the usability, ease of
use, impact, and improvement suggestions of the D-VRE prototype. We
disseminated our D-VRE software prototype and surveyed the CLARIFY community
by asking the following four questions.

$Q_{1}:$ Do you think the D-VRE features are functional or helpful in the
research lifecycle? And why?

$Q_{2}:$ Are the D-VRE features easy to use? And why?

$Q_{3}:$ Will you be interested in using the proposed D-VRE in your daily
research practices, for example, in building decentralized collaboration and
data-sharing?

$Q_{4}:$ (optional) Do you have any suggestions for improvement for a research
collaboration scenario you’d like to facilitate during the D-VRE
dissemination?

##### Usability.

We collected five responses to $Q_{1}$ about usability and usefulness, and 3/5
of the participants gave very positive feedback. They like the idea of
defining policies by individual researchers, making data available only during
a specific period, and the encryption and decryption feature for sensitive
data. The rest of the participants think the D-VRE features may be useful
since it still looks a bit limited:

  * •

“The D-VRE looks like an institute-level feature not so functional for
individual researchers. Maybe organization design helps.”

  * •

“It is unclear how much leverage we get using D-VRE instead of the simpler
sharing option on Google Colab. Maybe comparing existing solutions can
highlight their usefulness in different cases.”

Note that we have compared existing solutions, incl., Google Colab, in Section
II.

##### Ease of Use.

Based on the responses to $Q_{2}$, all participants think the D-VRE features
are easy to use because (1) the GUI looks simple and understandable, (2)
buttons and features are clearly named and straight-to-the-point, and (3) the
video tutorials are well detailed. However, we also received some suggestions
about GUI improvements:

  * •

“Hopefully a better visualization in GUIs.”

  * •

“Maybe A/B testing with different features can help understand how different
groups can find it easier without knowing much about notebooks.”

##### Impact.

Regarding whether they will be interested in using the proposed D-VRE in their
daily research practices, we received 4/5 positive feedback about $Q_{3}$. One
participant commented that it needs to be robust, and he or she wanted to know
if the D-VRE supports large-scale datasets and training. The biggest concern
in the responses is the volume of data. Participants wonder how much it can
handle.

##  V Discussion and Open Challenges

This section discusses the limitations of the current D-VRE, open challenges,
and possible improvements in future directions.

D-VRE, based on a decentralized future, is expected to impact many scientific
domains that rely on research data and models from distributed sources, using
Jupyter for data analytics, and large-scale computation in the age of Web 3.0
— give power back to the users in the form of ownership. We argue that D-VRE
can serve as a novel framework for achieving a decentralized virtual research
environment ecosystem and our vision is to make it a working system that can
serve as a key driver for decentralized science. To achieve that goal, it must
be researched, developed, tested, deployed, and evaluated based on the
infrastructure of Web 3.0 and address core research questions in scientific
communities.

The primary objective of this study was to bridge the gap between an
individual’s private research platform and the decentralized collaborative
research ecosystem from three perspectives: asset management, workflow
management, and infrastructure management. We prototyped D-VRE with seamless
integration of JupyterLab. Despite this significant bridge, this study has
several limitations.

###  V-A Operation Model

First but foremost, there is still optimization space in the operation model.
Currently, the D-VRE is deployed on the Sepolia Testnet, which features a
Proof-of-Stake (PoS) consensus mechanism to mimic the Ethereum Mainnet, where
everyone can test running a validator. However, it might face an issue with
the cheating validators or other risks related to cheating behaviors. A
consortium network may be preferable because a pre-defined set of trusted
nodes controls the consensus process. For instance, a group of academic
institutes like universities can construct a private network, and each
institute governs a single node. As a result, the institute can verify
individual user identities and make research activities more reliable. In this
case, it can also combine with verifiable credential models to enhance the
decentralized identity management for decentralized authentication.

###  V-B Energy Consumption and Sustainability

According to the statistics of Crypto Carbon Ratings Institute
(CCRI)222222https://carbon-ratings.com/, retrieved on 19 June 2024., the
annualized electricity consumption of the Ethereum PoS is estimated
approximately 0.0060 TWh across the entire global network, which corresponds
to the estimated yearly carbon emissions of 0.0019MtCO2e. In contrast, the
annual global data centre electricity consumption in 2022 was 240-340TWh,
according to the tracking from the International Energy Agency
(IEA)232323https://www.iea.org/energy-system/buildings#tracking, retrieved on
19 June 2024.. Thus, we can say that Ethereum is green blockchain thanks to
the PoS consensus mechanism. However, there is still optimization space to
reduce the energy consumption by optimizing the consensus algorithms and/ or
the resource utility of distributed infrastructures. For example, there might
be a PoX (X is an unknown factor) consensus mechanism in the future that will
make the Ethereum blockchain greener than using PoS.

Apart from energy consumption and CO2 emissions, smart contracts or
transaction optimization is another aspect to facilitate sustainability. It is
known that every operation that performs a transaction or smart contract on
the Ethereum platform costs certain units of gas, and the gas used denotes the
metering unit for the use of the Ethereum virtual machines in the world
computer. Assume that we could decrease the gas used in smart contract
deployment and operations in the smart contract functionalities, it will also
reduce the use of the Ethereum world computer. As such, the corresponding
electricity energy and carbon emissions will be reduced.

###  V-C Community Support

Last but not least, our D-VRE needs the support from the community. However,
one of the biggest challenges to the sustainability of any scientific tools or
software is community support. There is a need to continue research and
development and engage in D-VRE dissemination, including a well-documented
manual, demonstration of D-VRE features, and tutorials for the target
communities.

The current prototype implementation may restrict the generalizability of our
findings since it only has been disseminated in one community, i.e., CLARIFY,
that shows significant potential for use in medical research, especially in
supporting sensitive data sharing. The need for D-VRE to adapt to the varied
demands of different communities highlights the importance of incorporating
flexibility into its design.

##  VI Conclusion and Future Work

This article presents D-VRE, a significant bridge between the Jupyter
environment and a decentralized ecosystem for scientific research. The PoC
prototype achieved seamless integration with blockchain-empowered D-VRE
applications and extensions, such as Auth via MetaMask, Make Agreements,
Membership, and Asset Manager into JupyterLab. We conducted an experimental
study to test all functionalities written in D-VRE smart contracts and their
gas consumption. In addition, the conducted case study demonstrates the
D-VRE’s usability, ease of use, and potential in P2P trustworthy data sharing
and collaboration scenarios. The promising results from the D-VRE prototype
bridge the gap between traditional Jupyter-related virtual research
environments and blockchain capabilities, paving the way for broader
collaborative applications in real-world situations using Jupyter.

In our future work, we will first complete the rest of the D-VRE components
such as workflow manager and provenance manager. Then we will investigate the
underlying metadata management to enhance the FAIRification in the context of
research lifecycles. Moreover, we will continuously improve our D-VRE by
engaging in more data-centric scientific communities and exploring more real-
world use cases.

## Acknowledgments

This research was made possible through partial funding from several European
Union projects: CLARIFY (860627), ENVRI-Hub Next (101131141), EVERSE
(101129744), BlueCloud-2026 (101094227), OSCARS (101129751), LifeWatch ERIC,
BioDT (101057437, through LifeWatch ERIC), and Dutch NWO LTER-LIFE project.

## CRediT authorship contribution statement

Yuandou Wang: Conceptualization, Methodology, Software (testing and revision
of existing code components), Investigation, Validation, Formal analysis,
Writing – Original draft, Writing – Review & Editing. Sheejan Tripathi:
Methodology, Software, Validation, Formal analysis, Investigation. Siamak
Farshidi: Writing – Review & Editing. Zhiming Zhao: Conceptualization, Writing
– Review & Editing, Supervision, Project administration, Funding acquisition.

## References

  * [1] J. A. Sidey-Gibbons and C. J. Sidey-Gibbons, “Machine learning in medicine: a practical introduction,” _BMC medical research methodology_ , vol. 19, pp. 1–18, 2019.
  * [2] C. Monteleoni, G. A. Schmidt, and S. McQuade, “Climate informatics: accelerating discovering in climate science with machine learning,” _Computing in Science & Engineering_, vol. 15, no. 5, pp. 32–40, 2013.
  * [3] P. Mathur and M. Mathur, “Machine learning ensemble species distribution modeling of an endangered arid land tree tecomella undulata: a global appraisal,” _Arabian Journal of Geosciences_ , vol. 16, no. 2, p. 131, 2023.
  * [4] M. L. Henderson, W. Krinsman, S. Cholia, R. Thomas, and T. Slaton, “Accelerating experimental science using jupyter and nersc hpc,” in _Tools and Techniques for High Performance Computing: Selected Workshops, HUST, SE-HER and WIHPC, Held in Conjunction with SC 2019, Denver, CO, USA, November 17–18, 2019, Revised Selected Papers 6_. Springer, 2020, pp. 145–163.
  * [5] J. M. Perkel, “By jupyter, it all makes sense,” _Nature_ , vol. 563, no. 7729, pp. 145–146, 2018.
  * [6] E. Gazzarrini, E. G. Garcia, D. Gosein, and X. Espinal, “The virtual research environment: A multi-science analysis platform,” in _EPJ Web of Conferences_ , vol. 295. EDP Sciences, 2024, p. 08023.
  * [7] Z. Zhao, S. Koulouzis, R. Bianchi, S. Farshidi, Z. Shi, R. Xin, Y. Wang, N. Li, Y. Shi, J. Timmermans _et al._ , “Notebook-as-a-vre (naavre): From private notebooks to a collaborative cloud virtual research environment,” _Software: Practice and Experience_ , vol. 52, no. 9, pp. 1947–1966, 2022.
  * [8] M. Assante, L. Candela, D. Castelli, R. Cirillo, G. Coro, A. Dell’Amico, L. Frosini, L. Lelii, M. Lettere, F. Mangiacrapa _et al._ , “Virtual research environments co-creation: The d4science experience,” _Concurrency and Computation: Practice and Experience_ , vol. 35, no. 18, p. e6925, 2023.
  * [9] S. Wang, L. Anselin, B. Bhaduri, C. Crosby, M. F. Goodchild, Y. Liu, and T. L. Nyerges, “Cybergis software: a synthetic review and integration roadmap,” _International Journal of Geographical Information Science_ , vol. 27, no. 11, pp. 2122–2145, 2013.
  * [10] D. Yin, Y. Liu, A. Padmanabhan, J. Terstriep, J. Rush, and S. Wang, “A cybergis-jupyter framework for geospatial analytics at scale,” _ACM International Conference Proceeding Series_ , vol. Part F1287, 2017.
  * [11] A. Zonca and R. S. Sinkovits, “Deploying jupyter notebooks at scale on xsede resources for science gateways and workshops,” in _Proceedings of the Practice and Experience on Advanced Research Computing_. ACM, 2018, pp. 1–7.
  * [12] K. A. Lawrence, M. Zentner, N. Wilkins-Diehr, J. A. Wernert, M. Pierce, S. Marru, and S. Michael, “Science gateways today and tomorrow: positive perspectives of nearly 5000 members of the research community,” _Concurrency and Computation: Practice and Experience_ , vol. 27, no. 16, pp. 4252–4268, 2015.
  * [13] G. Huang, C. Luo, K. Wu, Y. Ma, Y. Zhang, and X. Liu, “Software-defined infrastructure for decentralized data lifecycle governance: principled design and open challenges,” in _2019 IEEE 39th International Conference on Distributed Computing Systems (ICDCS)_. IEEE, 2019, pp. 1674–1683.
  * [14] M. Assante, L. Candela, D. Castelli, R. Cirillo, G. Coro, L. Frosini, L. Lelii, F. Mangiacrapa, P. Pagano, G. Panichi _et al._ , “Enacting open science by d4science,” _Future Generation Computer Systems_ , vol. 101, pp. 555–563, 2019.
  * [15] C. Goble, S. Soiland-Reyes, F. Bacall, S. Owen, A. Williams, I. Eguinoa, B. Droesbeke, S. Leo, L. Pireddu, L. Rodríguez-Navas _et al._ , “Implementing fair digital objects in the eosc-life workflow collaboratory,” _Zenodo_ , 2021.
  * [16] S. Warnat-Herresthal, H. Schultze, K. L. Shastry, S. Manamohan, S. Mukherjee, V. Garg, R. Sarveswara, K. Händler, P. Pickkers, N. A. Aziz _et al._ , “Swarm learning for decentralized and confidential clinical machine learning,” _Nature_ , vol. 594, no. 7862, pp. 265–270, 2021.
  * [17] R. Blythman, M. Arshath, J. Smékal, H. Shaji, S. Vivona, and T. Dunmore, “Libraries, integrations and hubs for decentralized ai using ipfs,” _arXiv preprint arXiv:2210.16651_ , 2022.
  * [18] Z. Zheng, S. Xie, H.-N. Dai, X. Chen, and H. Wang, “Blockchain challenges and opportunities: A survey,” _International journal of web and grid services_ , vol. 14, no. 4, pp. 352–375, 2018.
  * [19] Z. Zheng, S. Xie, H. Dai, X. Chen, and H. Wang, “An overview of blockchain technology: Architecture, consensus, and future trends,” in _2017 IEEE international congress on big data (BigData congress)_. Ieee, 2017, pp. 557–564.
  * [20] C. Rong, J. Geng, T. J. Hacker, H. Bryhni, and M. G. Jaatun, “Openiac: open infrastructure as code-the network is my computer,” _Journal of Cloud Computing_ , vol. 11, no. 1, p. 12, 2022.
  * [21] W. Ding, J. Hou, J. Li, C. Guo, J. Qin, R. Kozma, and F.-Y. Wang, “Desci based on web3 and dao: A comprehensive overview and reference model,” _IEEE Transactions on Computational Social Systems_ , vol. 9, no. 5, pp. 1563–1573, 2022.
  * [22] L. Cao, “Decentralized ai: Edge intelligence and smart blockchain, metaverse, web3, and desci,” _IEEE Intelligent Systems_ , vol. 37, no. 3, pp. 6–19, 2022.
  * [23] Y. Wang, S. Koulouzis, R. Bianchi, N. Li, Y. Shi, J. Timmermans, W. D. Kissling, and Z. Zhao, “Scaling notebooks as re-configurable cloud workflows,” _Data Intelligence_ , vol. 4, no. 2, pp. 409–425, 2022.
  * [24] M. B. Milligan, “Jupyter as common technology platform for interactive hpc services,” in _Proceedings of the Practice and Experience on Advanced Research Computing_. ACM, 2018, pp. 1–6.
  * [25] D. De Roure, C. Goble, J. Bhagat, D. Cruickshank, A. Goderis, D. Michaelides, and D. Newman, “myexperiment: defining the social virtual research environment,” in _2008 IEEE fourth international conference on EScience_. IEEE, 2008, pp. 182–189.
  * [26] C. Kontomaris, Y. Wang, and Z. Zhao, “Cwl-flops: A novel method for federated learning operations at scale,” in _2023 IEEE 19th International Conference on e-Science (e-Science)_. IEEE, 2023, pp. 1–2.
  * [27] Y. Wang, N. Kanwal, K. Engan, C. Rong, P. Grosso, and Z. Zhao, “Price: Privacy-preserving and cost-effective scheduling for parallelizing the large medical image processing workflow over hybrid clouds,” _arXiv preprint arXiv:2405.15398_ , 2024.
  * [28] S. Soiland-Reyes, P. Sefton, M. Crosas, L. J. Castro, F. Coppens, J. M. Fernández, D. Garijo, B. Grüning, M. La Rosa, S. Leo _et al._ , “Packaging research artefacts with ro-crate,” _Data Science_ , vol. 5, no. 2, pp. 97–138, 2022.
  * [29] P. Barker and L. M. Campbell, “What is schema. org,” _LRMI. Retrieved April_ , vol. 21, p. 2015, 2014.
  * [30] E. Schultes and P. Wittenburg, “Fair principles and digital objects: Accelerating convergence on a data infrastructure,” in _Data Analytics and Management in Data Intensive Domains: 20th International Conference, DAMDID/RCDL 2018, Moscow, Russia, October 9–12, 2018, Revised Selected Papers 20_. Springer, 2019, pp. 3–16.
  * [31] M. Rauchs, A. Glidden, B. Gordon, G. C. Pieters, M. Recanatini, F. Rostand, K. Vagneur, and B. Z. Zhang, “Distributed ledger technology systems: A conceptual framework,” _Available at SSRN 3230013_ , 2018.
  * [32] W.-M. Lee and W.-M. Lee, “Using the metamask chrome extension,” _Beginning Ethereum Smart Contracts Programming: With Examples in Python, Solidity, and JavaScript_ , pp. 93–126, 2019.
  * [33] J. Benet, “Ipfs-content addressed, versioned, p2p file system,” _arXiv preprint arXiv:1407.3561_ , 2014.