# Collaborative Software Modeling in Virtual Reality

Enes Yigitbas, Simon Gorissen, Nils Weidmann, Gregor Engels  Paderborn
University, Germany
<EMAIL_ADDRESS>
###### Abstract

Modeling is a key activity in conceptual design and system design. Through
collaborative modeling, end-users, stakeholders, experts, and entrepreneurs
are able to create a shared understanding of a system representation. While
the Unified Modeling Language (UML) is one of the major conceptual modeling
languages in object-oriented software engineering, more and more concerns
arise from the modeling quality of UML and its tool-support. Among them, the
limitation of the two-dimensional presentation of its notations and lack of
natural collaborative modeling tools are reported to be significant. In this
paper, we explore the potential of using Virtual Reality (VR) technology for
collaborative UML software design by comparing it with classical collaborative
software design using conventional devices (Desktop PC / Laptop). For this
purpose, we have developed a VR modeling environment that offers a natural
collaborative modeling experience for UML Class Diagrams. Based on a user
study with 24 participants, we have compared collaborative VR modeling with
conventional modeling with regard to efficiency, effectiveness, and user
satisfaction. Results show that the use of VR has some disadvantages
concerning efficiency and effectiveness, but the user’s fun, the feeling of
being in the same room with a remote collaborator, and the naturalness of
collaboration were increased.

###### Index Terms:

Collaborative Modeling, Virtual Reality, UML

##  I Introduction

In modern software development collaboration between developers is one of the
driving factors that determines the quality and speed at which the projects
can be realized. One central artifact of communication and discussion in
software engineering are models [1]. The Unified Modeling Language (UML) with
its associated diagrams is one of the most well known general-purpose modeling
languages in software engineering and is considered by many as ”lingua franca”
for software engineers [2]. However, researchers and software designers have
realized the insufficiency of UML in its expressiveness since it is restricted
to a two-dimensional plane. These insufficiencies include a lack of dynamic
expression and interaction ability between groups of remote designers [3], [4]
as well as the complexity for large models [5]. Furthermore, the authors in
[6] argue that the dissatisfaction of developers with UML tools is one of the
reasons it has not been adopted more universally, exemplifying the need for
improving the tool support. Since the COVID-19 pandemic has spread around the
world, this need for good UML tool support has only increased. Many
educational institutions, like schools and universities, around the world have
been forced to switch to online education settings to support social
distancing. Likewise, millions of workplaces wherever possible were
transitioned to home office. To enable collaboration for software engineers in
such situations, tools are needed that offer support for creating and
discussing models from remote locations. While classical modeling
applications, like Lucidchart [7] or GenMyModel [8], support remote
collaboration, they do not overcome the mentioned issues with regard to
visualization and collaboration as they are mostly relying on a 2D UML
notation and do not support a natural way of collaboration comparable to
editing a model on a whiteboard while situated together in one room.

Virtual Reality technology, on the other hand, is becoming increasingly
sophisticated and cost-effective and can be applied to many areas such as
training [9], robotics [10], education [11], healthcare [12], and even
Information Systems (IS) research, to simulate a real environment or represent
complicated scenarios. Modern Head Mounted Display (HMD) Virtual Reality (VR)
devices have several technological capabilities that are not present on
conventional devices (Desktop PC / Laptop): (1) Stereoscopic 3D Images, (2)
Six Degrees of Freedom, (3) Hand Presence Support. and (4) 3D Spatial Voice
Chat. Since HMD devices show a slightly different image to each eye, they can
invoke the perception of a truly 3D virtual world that the user inhabits.
These Stereoscopic 3D Images are more in line with the visual experience of
the real world compared to images on a 2D screen, since people also perceive
the real world in 3D, not in 2D. In VR, users can also look around in the
virtual world by simply moving their heads, like they would in a real
environment. In addition to this rotational movement, modern VR devices can
also track the positional movement of a user, independent of the direction the
user moves in. This means, the user has Six Degrees of Freedom (three
rotational and three positional axes) in her movement and the VR system can
adapt the view and position of the user in the virtual world accordingly.
Additionally, typical VR controllers are used single-handedly with each
controller representing one hand in the virtual world. This Hand Presence
Support allows a user to make natural gestures like grabbing an object and
pointing at things to interact with the virtual world. With the means of 3D
Spatial Voice Chat, it is possible to make users feel like the voices of their
collaborators come with the volume level (distance-based) and from the
direction that their virtual representations (i.e. Avatars) are in. Overall,
the fast development in VR concerning prototyping [13] and engineering of VR
applications [14] as well as the mentioned technological advances allow us to
extend the research field to improve the quality of UML and collaborative
modeling. Thus, the main goal of this paper is to explore the potential of
using Virtual Reality (VR) technology for collaborative UML software design by
comparing it with classical collaborative software design using conventional
devices (Desktop PC / Laptop). For this purpose, we have developed a VR
modeling environment, called VmodlR, that offers a natural collaborative
modeling experience for UML Class Diagrams. Based on a user study with 24
participants, we have compared collaborative VR modeling with conventional
modeling with regard to efficiency, effectiveness, and user satisfaction.

The rest of the paper is structured as follows. In Section 2, we present and
discuss the related work. In Section 3, we describe the conceptual solution of
our VR-based collaborative modeling environment VmodlR. In Section 4, we show
the details of the implementation of VmodlR. In Section 5, we present and
discuss the main results of the usability evaluation. In Section 6, we
conclude the paper and give an outlook for future work.

##  II Related Work

Model-based and model-driven development methods have been discussed in the
past for various application domains such as intelligent user interfaces [15,
16, 17], usability engineering [18, 19, 20], digital twins [21, 22] or
business model development [23, 24]. In the following, we draw on prior
research into Collaborative Modeling, 3D Modeling, and Immersive Modeling.

###  II-A Collaborative Modeling

In previous work, the topic of collaborative software modeling was already
researched from different perspectives [25]. In [26], the authors present a
distributed UML editor that aims to transfer collaborative discussion and
editing of UML models from regular meeting rooms to remote settings using
regular computers. In addition, a collaborative learning environment for UML
modeling is presented in [27]. Furthermore, [28] introduces an approach to a
real-time synchronous collaborative modeling of software systems using 3D UML.
Besides these approaches from research, there are also many commercial tools
supporting collaborative software modeling. Lucidchart is a web-based
commercial tool for collaboratively creating diagrams [7]. It includes support
for many diagram types and modeling languages like UML, Business Process Model
Notation (BPMN), Enterprise Relationship models, and more. The tool shows a 2D
canvas where standard 2D diagrams can be created. Since Lucidchart is web-
based, it can be used by any device that runs a modern browser, although as
with many web-based contents the experience can be assumed to be best suited
to the use on desktop and laptop computers and is not specifically adopted to
the possibilities of VR devices. Released by Axellience in 2014, GenMyModel
[8] is a web-based tool similar to Lucidchart. GenMyModel supports a subset of
the most commonly used UML diagrams, i.e., Class, Use Case, Component, Object,
State, Deployment, Activity, and Sequence Diagrams. Axellience describes the
mode of collaboration as similar to how Google Drive or Microsoft Office
online collaboration works. So users can see where their co-workers are
editing the model but do not have an integrated voice chat to discuss those
changes. This has to be done via an external tool. The system aims to support
the language’s visual representations according to their official definitions
and thus only presents 2D models to the users.

While the above-described approaches mostly support collaborative modeling
based on standard 2D UML diagrams, they do not support immersive VR.

###  II-B 3D Modeling

The authors in [29] presented a conceptual system that visualizes UML Class
and Sequence Diagrams in 3D. The Sequence Diagrams are displayed in the
context of the Classes they belong to and animated to emphasize their
connection. Furthermore, the authors in [30] have also discussed a conceptual
approach to extending 2D UML Diagrams to the third dimension. They introduce
some modeling quality attributes and explain how VR can improve the UML
modeling quality according to these attributes. For example, they argue that
the model’s ”understandability” can be improved through VR’s ”Immersion” and
”Stereopsis” (Stereoscopic 3D presentation) features. Subsequently, they
provide some examples of a Class Diagram and a Sequence Diagram that are
visualized in 3D and demonstrate their advantages in comparison to a 2D
representation. In [31], the authors proposed VisAr3D, a 3D visualization tool
for UML models to make large models easier to comprehend, especially by
inexperienced modelers like students. The system automatically converts a 2D
UML model from an .xmi file into its 3D representation. The virtual
environment the 3D UML models are placed in can be viewed through a web-based
app and does not support HMDs. Their prototype is not a model editor, however,
it only visualizes UML models in 3D. This approach was evaluated based on a
user study assessing the effect of the 3D compared to a 2D representation. The
main results show that the 3D representation aided the understandability of
large models, ”increased students’ interest” and supported teaching purposes
[31]. In [32], the authors described an implementation that visualized a
process model in a 3D virtual world on a desktop computer. It is a training
system intended to aid a single user in understanding Business Process Models
through 3D visualization. They evaluate how the 3D representation assists the
user in learning the modeled process compared to a standard 2D model
representation. They conclude that the 3D representation provides a noticeable
learning benefit. It is not a modeling tool, however, only a visualization and
training tool. Further examples for approaches which make use of a 3D
representation of UML models can be found in [33, 34, 35, 36]. In summary, a
modeling approach that combines and integrates the aspects of 3D modeling,
collaboration, and VR in one solution is not fully covered and yet existing.

###  II-C Immersive Modeling in AR and VR

In the following, AR- and VR-based approaches for modeling purposes will be
briefly described and discussed. Although VR is our main focus, we included AR
approaches to cover immersive modeling approaches on the whole.

Augmented Reality (AR) is closely related to VR with the main difference being
that VR immerses a user in a completely virtual world while AR does not
isolate the user from the real world by displaying virtual objects in the real
environment. In general, AR has been already applied for different aspects
such as robot programming [37], product configuration (e.g., [38], [39]),
planning and measurements [40] or for realizing smart interfaces (e.g., [41],
[42]). To be more specific, example approaches that apply AR for software
modeling are as follows. The authors in [43] have presented a framework that
is supposed to allow editing and viewing UML models in a 3D space through the
Microsoft Hololens AR Glasses. Their approach is to overlay 3D model elements
over real scenery so the user can move around and inside the model while not
being shut off from the real world. In this way, their prototype only displays
static objects that are meant to represent UML models. Similarly, in [44], the
authors proposed a system that aims to make learning UML more accessible by
displaying it in 3D as overlays to the natural environment using the Microsoft
Hololense. The system supports creating and editing UML Class Diagrams but is
only intended for single users and does not support collaboration.
Furthermore, in [45] the authors introduce the concept of ”HoloFlows” to
support the modelling of processes for the internet of things in mixed
reality. A similar solution is introduced in [46] where the authors present an
approach for supporting domain-specific modelling environments based on AR.
The main drawback of AR-based solutions for modeling is the small field of
view which narrows down the possibilities for modeling support. Therefore, we
have explored an alternative solution in VR.

Focusing on VR-related approaches for software modeling, we can see that many
previous works already have seen the potential in using VR for improving
modeling activities. In [47], for example, the authors have presented a system
that analyses an object-orientated code base and visualizes its classes,
attributes, and relations automatically in a 3D virtual environment that users
can inspect inside a Cave Automatic Virtual Environment (CAVE). So it supports
VR only in a broader sense. The application is not networked and does not
support editing the model, it is only a visualization tool. In [48], the
authors have proposed a system for HMD VR devices that can import a Finite
State Machine (FSM) and visualize it in a game-like environment where players
stand on islands representing states and can change islands via different
boats representing the possible transitions. It is a single-player game
environment meant for educational purposes and does not support UML modeling
with actual UML Diagram elements or remote collaboration. In [49], the authors
have presented a HMD VR system for visualizing process models in 3D. Therein,
the models can be annotated but not edited. They evaluated the effectiveness,
efficiency, and intuitiveness of the VR visualization in comparison to (1)
paper and (2) desktop tool based BPMN and found that the effectiveness was
equivalent between VR and desktop but task completion in VR was faster than
with the BPMN tool and the users found the VR controls very intuitive.

##  III Solution Overview

The system overview of our VR-based collaborative modeling environment VmodlR
is shown in Fig. 1. The top half of this figure represents User A and the
Virtual Environment she accesses through a VR Device. The bottom half
symmetrically shows a remote collaborator, User B, and the VR Device she uses
to access the same shared Virtual Environment. Through the VR Device, each
User is represented in the Virtual Environment as a Virtual Character. The
View Orientation and Position of the VR Device together with the Controller
Input controls the Virtual Character while the Virtual Environment with its
content is displayed from the Virtual Character’s perspective inside the VR
Device.

Figure 1: System overview of VmodlR

Through the Virtual Character, each User interacts with the elements inside
the Virtual Environment: The 3D Model can be edited by either directly editing
model elements (e.g. creating, deleting, or moving them), or through the VR
Text Input component that is used to edit the text inside the 3D Model, for
example, the Class names. All those changes to the model are synchronized
between users through the Network Synchronization component. In the case of
this solution, the 3D Model is a three-dimensional UML Class Diagram, but
theoretically, this could be adapted to any kind of conceptual model. Through
their Virtual Characters, Users can also interact with each other via Network
Synchronization which transmits their voices and synchronizes their body and
hand movements. This is visualized in Fig. 1 as the Synchronize Movements &
Communication interface between the Virtual Characters and the Network
Synchronization. The Network Synchronization component then ensures that the
Virtual Environment and its content is synchronized between the Users through
the Cloud. It offers several different services that can be used by other
components to synchronize all necessary aspects of the Virtual Environment. In
the following, each of the main components Virtual Environment, Network
Synchronization, Virtual Character, 3D Models, and VR Text Input will be
described in more detail.

###  III-A Virtual Environment

The Virtual Environment consists of all virtual elements that are needed to
provide a collaborative software modeling environment. Within this
environment, each user can see and move around via their VR Device in 3D. As a
design decision for the Virtual Environment, we opted to use an open space
instead of a closed one to not introduce some unrealistic environment behavior
or limit the user’s ability to create large models. Since there is no open
environment that could be considered natural for creating conceptual models,
any space that offers a planar ground for the user to walk on could be chosen.
We decided on a grass field under a blue sky because it does not limit the 3D
space available to the user and depicts a pleasant real-world environment that
users are familiar with.

###  III-B Network Synchronization

All elements inside the Virtual Environment have to be synchronized through
the Network Synchronization component. We will briefly describe how this
networking generally works in this conceptual solution. The networking has a
server-based architecture where all users connect to a server and the server
synchronizes instances of a Virtual Environment between all users that are
currently in that environment. It is important to understand that a networked
environment is not a singular environment. Instead, on each user’s VR Device
(clients) a local version of the networked environment exists, so the user can
look around in it and interact with it. All the changes the user can thereby
make inside the Virtual Environment, like moving their Virtual Character, are
communicated to the server so it can update its reference representation of
the networked environment and forward the changes to the local environments of
all other clients. These clients then apply the changes accordingly to their
local copies of the environment. This way, the local environments on all
clients are always kept in sync with the server’s networked environment. Three
components can be used to synchronize objects inside the Virtual Environment:
The Movement Synchronization Service, the Event Synchronization Service, and
the Voice Synchronization Service. The Movement\- and Event Synchronization
Services can be more generally used. The difference between them is that the
Movement Synchronization Service is dedicated to rapidly and frequently
changing information, that need to be synchronized many times per second. In
the case of movement, this is needed to show the movement of an object that is
moved by a remote collaborator fluently to the local user. The Movement
Synchronization Service also deals with tracking if a synchronized object that
can be moved gets created or deleted. The Event Synchronization Service, on
the other hand, is supposed to synchronize arbitrary events that happen rather
infrequently and therefore only have to be synchronized occasionally, once
they occur, instead of the constant synchronization needed for movement. This
provides flexibility where something that only occasionally changes (like the
name of a Class for example) can be synchronized via the Event Synchronization
Service and otherwise does not consume network bandwidth while things that
often change and have to be synchronized many times per second, like the
movement of a Class or a Virtual Character’s Virtual Hands, can be
synchronized fluently via the Movement Synchronization Service. Finally, the
Voice Synchronization Service is dedicated to synchronizing the voices of
users to enable voice chat inside our solution.

###  III-C Virtual Character

In VR, people can be represented by 3D characters through an avatar with a
body, head, and hands. This way, a user can, for example, move around in the
virtual environment and point with their finger in real life and the VR
Controller can reproduce this gesture on the Virtual Character’s hands. Since
this has the possibility to make discussions about models in VR much more
natural than possible on PCs, these features were also included in the design
of our solution and the hands’ gestures are synchronized across the network
for each user, along with the positional audio of voices and the positions and
orientations of Virtual Characters. Our solution supports this form of natural
movement, where the Virtual Character and therefore the user’s view into the
world changes according to the physical VR Headset movement. This is the ideal
scenario for movement, where there is enough physical space available to the
user to move anywhere she would want to go in the virtual world. However, this
is hardly a realistic scenario since the models creatable in our solution can
theoretically become arbitrarily large, meaning that the user would need an
infinitely large physical space to move around in. Therefore, a secondary
movement method, namely teleportation is required, that allows a user to move
their Virtual Character through the environment without moving in the physical
world. Teleportation involves the user entering a teleportation mode, for
example through pressing or holding a specific button on one of the VR
Controllers, and then aiming the controller at a spot that she wishes to
teleport to. When either releasing the button or pressing it again the target
position is selected and the Virtual Character is instantly teleported to the
aimed location. With these two movement methods—natural walking and
teleportation movement—the player can reach any position in the Virtual
Environment independent of the size of physical space available to her while
still moving in a rather natural way.

###  III-D 3D Models

Usually, modeling languages are only specified with 2D visual representations.
UML is no exception from that rule. We could stick to those same 2D shapes
inside a 3D world with users being able to position the 2D shapes freely in
3D. However, we believe using 3D shapes instead of 2D ones will likely result
in a more natural experience for users because the real world only consists of
3D objects and we do not want the model elements to seem like foreign bodies
in the Virtual Environment. The main shapes used to visualize UML elements in
the 2D specification are rectangles and lines with different forms of
arrowheads at the end of those lines. To ensure that users familiar with the
2D representations can learn the 3D ones easily we tried to find natural
equivalents of the 2D shapes in 3D. The equivalent of rectangles in 3D are
cubes and cuboids, while lines are best represented by thin tubes (see Fig. 2
(c)). A challenge of a 3D visualization of UML Classes is the text
representation inside them. A 2D Class only has one side that the text is
displayed on which always faces the user. Since the Class rectangles known
from 2D are most similar to Cuboids in 3D, every UML Class is visualized as a
cuboid in our solution (see Fig. 2 (a)). It can be assumed that the users can
see at most three of the six sides of a cuboid at any given time. If the
Class’s cuboid only displays the text on one side, this side could, thus, be
hidden depending on the view direction that the user has towards the Class.
Rotating the cubes automatically so the text-side always faces the user would
be a possible solution to that. However, we wanted the user to be the only
entity changing the model’s appearance, so the model seems more stable and
thus natural. For these reasons, we chose to display the text associated with
a Class on all sides of the Class. This means that every Class side basically
shows the same 2D Class in a notation similar to 2D UML. Since all these sides
belong to a single Class it is important that all sides always show the same
content.

###  III-E VR Text Input

In VR, text input is more difficult compared to traditional computers due to
the lack of a physical keyboard that provides haptic feedback on whether a key
was hit or not. Since the user cannot see and use a real keyboard while in VR,
virtual keyboards are often used instead. There are two basic types of
keyboards regularly found in VR applications: Laser pointer and drum-style
keyboards. Laser pointer style keyboards are usually displayed as a vertical
plane in front of the user with each hand representing one laser pointer that
can be used to press a key with a dedicated button. Since the buttons have to
be rather large to be easily hittable, this input method requires a lot of
space. An alternative to laser pointer keyboards are drum-style keyboards
which we have chosen in our solution (see Fig. 2 (b)). These are shown in a
horizontal, slightly tilted, form in front of the user like many real
keyboards are as well. One virtual mallet is attached to each of the user’s
hands that can be used to hit a key similar to how a person would hit a drum
in the real world. Because of the obvious drum analogy, it can be assumed that
this control scheme is also easy to understand for most users and could be
more natural to use than laser pointers.

##  IV Implementation

As a target implementation platform for our collaborative VR modeling
environment, we have chosen the Oculus Quest 2 which is a cable-free, mobile
VR HMD. For the implementation of the VR environment, we have used the Unity
3D Game Engine developed by Unity Technologies [50] as it includes a fully
featured Graphics, Sound and Physics Engine and has easy-to-use APIs for many
different aspects like controller input. Furthermore, Unity also provides
support for the Oculus Quest 2 among other Oculus VR headsets through an
Oculus VR SDK. To enable remote collaboration, a networking system is needed
that synchronizes aspects like model element- and user avatar positions, user
hand gestures, etc. For the implementation of the networking system, we have
used the Photon Unity Networking 2 (PUN2) plug-in for Unity. This is a third-
party system developed by Exit Games specifically for use in Unity multiplayer
projects and offers easy-to-use high-level components. PUN2 realizes the
network architecture discussed in the previous section by providing its own
Cloud servers that automatically work with PUN2 without the need for any
custom server-side development. The Oculus Software Development Kit (SDK),
used to realize the interaction between the VR Device and the Virtual
Environment, comes with a variety of assets that can be used to quickly
implement common functionalities inside a VR application. Most aspects needed
to realize the Virtual Character are covered by those assets provided by
Oculus. Therefore, to save development time on those basics, we used Oculus’
OVRAvatar and OVRCameraRig assets and adjusted them slightly for our
application. The OVRCameraRig deals with tracking the position and rotation of
the head and controllers and rendering the world accordingly into the headset.
It does not include animated hands by default but there are separate assets
for that which can simply be placed under the empty GameObjects that track the
controller positions to enable users to see their hands. The gestures of the
hands are tracked locally by default through an animator so we only had to add
an animator synchronizer provided by PUN2 to synchronize the hand’s animation
states across the network [51]. Since all Virtual Characters are represented
equally in this prototype, we chose to display every user’s name above their
character. An example of how a remote user’s Virtual Character looks inside
the VR app is shown in Figure 2.

Figure 2: Screenshots of the collaborative VR modeling environment

##  V Evaluation

To evaluate the efficiency, effectiveness, and user satisfaction of VmodlR and
to compare it with conventional collaborative modeling approaches, we have
conducted a user study which will be presented in the following.

###  V-A Setup and Participants

The user study was organized in such a way that two people in different rooms
had to collaboratively create a UML Class Diagram. We have chosen a within-
subjects design [52] for our user study where the participants were asked to
use both modeling approaches, a conventional modeling tool, and the developed
VR modeling tool. Two different small UML Class Diagram modeling tasks
(consisting of five classes and based on a textual description) were provided
to the participants, while the sequence and type of task which was carried
with the help of a modeling tool were evenly distributed to avoid potential
bias in the collected data. As a reference application for comparing with our
own VR environment, we have decided to use an existing commercially available
tool for collaborative UML modeling. The tool we used is the Web application
Lucidchart [7]. This was chosen because it offers a mode of collaboration many
users are already familiar with from services like Google Docs, it offers a
free version that could be used for this study, and supported UML Class
Diagrams. The evaluated applications focus on remote collaboration, therefore,
we simulated a remote setting by placing each participant in a different room
during their collaborative tasks where they could only communicate through
computing devices (conventional and VR device). For the VR application, each
participant was positioned in a free space of approximately the same size and
equipped with an Oculus Quest 2 VR Headset including its two VR Controllers.
The participants were able to communicate over the application’s voice chat
feature using the Oculus Quest’s built-in microphone and speakers. The
respective task was displayed inside the VR app on a panel that could be
opened and closed through an icon on the user’s menu. The panel was positioned
slightly to the left of the user so they could leave the panel open and edit
the model in front of them at the same time. While using the Web task, each
participant was sitting at a desk in the same room that the VR free space was
in, equipped with a laptop and a wired mouse. The participants could use the
laptop’s keyboard, its trackpad, and the mouse as they saw fit. Since the Web
application does not include voice chat, the participants communicated via the
voice chat application Skype that was running in the background on their
laptops. The app used the built-in speakers and microphones of the laptops.
The tasks were supplied to users on a sheet of paper so, like in the VR app,
it would not occlude their modeling environment unnecessarily. Since this
study involved participants having to create two small Class Diagrams, these
participants had to bring a basic knowledge about what a Class Diagram is and
which purpose it serves for software modeling. Therefore, we had to rely on
participants who either had lectures or school classes on this topic, for
example in computer science lectures or subjects and/or who knew Class
Diagrams from a different source, like working as a software developer who
uses them. Therefore, we primarily tried to acquire participants with
educational backgrounds, like university students and recent graduates. Our
main source was a lecture on Model-Based Software Engineering designed for
undergraduate students of computer science in their third year of studies. In
this lecture, we presented the study and asked students to participate in it.
We also reached out to students we were still in contact with who took part in
this lecture during the prior year to widen our pool of possible participants
while still ensuring comparable credentials among them. In total, 24
participants took part in the user study, meaning that there were 12 groups of
two people each.

###  V-B Procedure

The user study was conducted during one week in February 2021. The
experimental setting was kept as equal as possible for all pairs of
participants. First, the participants were greeted and introduced to the user
study. Then, the basic procedure of how they will take part in the study and
what they will do during their participation were explained. Afterward,
depending on whether the VR or Web app was used first, the collaboration
environment was set up (e.g., splitting the participants, starting Skype,
etc.), and they received a short introduction to the respective program. In
the case of the Web application, this was done through the study supervisor
explaining the main functionality from a pre-written script to ensure all
participants were given the same information. In the VR app, we have
additionally implemented a tutorial that served as an on-boarding tutorial at
the beginning of the user study. After the tutorials, the participants were
provided with the respective task and instructed to solve the task
collaboratively in the sense that they should only create one Class Diagram
together in each application. They were then asked to indicate to the study
executor once they think they are done with their task. When the participants
finished the first task, the procedure was repeated for the second task. After
both tasks were finished, the collaborative applications were closed so
participants would not be able to talk to each other anymore. They were then
given the questionnaire hosted via Google Forms [53] and were asked to fill it
out using the laptops that were used for the Web application as well. During
this process, the users stayed in the different rooms they were in while
working on the tasks to ensure that they would not influence each other’s
answers.

###  V-C Usability Measurements

To extract meaningful results from the study we had to choose certain
measurements that we would take during the execution. Figure 3 shows an
overview of the data that was collected during the user study and which
measurements were derived from that.

Figure 3: An overview of what was measured within the user study.

Efficiency was measured by recording the participants’ execution of the task
and tracking the time from the point where they started reading the task to
the time when they told the study executor that they finished it. From these
same recordings, the effectiveness was measured by counting the number of
operating errors that the participants made during the execution of the task.
From this, an error rate was calculated by dividing the number of errors by
the time in minutes that was measured as the efficiency. This error rate was
our final score for the effectiveness in errors per minute. The recordings
used for tracking efficiency and effectiveness were screen-captures including
the voice chat audio. In each group, only one participant was recorded to
reduce the amount of video data that had to be manually evaluated. Another
reason for that was the network infrastructure that did not allow us to
capture and record two video feeds from the VR headsets in parallel. The user
satisfaction was evaluated through a questionnaire that participants were
asked to fill out after they finished both tasks. We chose to use the System
Usability Scale (SUS) questionnaire [54] since it is a well-proven and
reliable questionnaire that provides comparability with SUS evaluations of
other applications and that can be quickly filled out due to its low number of
questions [55]. While efficiency and effectiveness were measured for both
applications, the SUS questionnaire was only asked in the context of the VR
application. This is because the SUS questionnaire’s main purpose in this work
is to provide a proof of concept that the VR solution has a rather good
usability. Furthermore, the participants were asked to answer a custom-
developed questionnaire that included mainly 5 point Likert scale questions
(like the SUS questions) and some free text questions asking for more specific
impressions with regard to collaboration and interaction.

###  V-D Results

In the following, we present the main results of the user study.

####  V-D1 Demographic Statistics

24 people in 12 collaborating groups participated in the study. All
participants were between 22 and 30 with a median of 25 and a mean of 25.5. 21
of the 24 participants were male and the others female. Most of the
participants were software engineering practitioners or had a background in
computer science. As a consequence, many participants reported advanced
experience with UML and UML modeling tools. The average rating scores for
prior experience with UML and UML modeling tools were 3.25 and approximately
3, respectively. With regard to prior experience with VR, our results show
that half of the participants have never used a VR device before this study
while 8 used them at least reasonably often.

####  V-D2 Efficiency

The times participants needed to complete their tasks are shown in Figure 4.

Figure 4: The times each group took to execute the task in the Web and the VR
app respectively.

All but two groups (Group 7 and 11) needed more time in VR than in the Web
application to complete their task. From the recordings, it could be observed
that almost all groups split up the work on the Class Diagram by each modeling
one part of the diagram and putting both parts together in the end. During
this process, they occasionally discussed how to model certain aspects if they
were unsure and checked the part of the model their collaborator created in
the end. The two groups that took longer in the Web application than in VR had
a slightly different approach that could explain this anomaly: In VR they
split up their task as outlined above. In the Web application, however, they
mainly followed a pair-programming style approach: For each part of the task
they discussed how to model it, and then only one of them modeled the aspect
accordingly. The first approach obviously saves time in comparison to the
latter one which could explain why they were able to complete the task faster
in VR than in the Web application when everyone else needed longer in VR. This
makes the times for these two groups not comparable since they did not use a
similar organizational method in both conditions. Therefore, we excluded them
from time-related analyses like the box plots shown on the right of figure 4.
Besides this anomaly, it can be said that the spread with respect to time was
way smaller for the Web application than for VR, i.e. they vary less as shown
on the right of figure 4. In VR, times range from almost 20 minutes to as low
as 9:30 min, while in the Web condition, the times vary from approximately
5:30 to 10 minutes. The difference of each group between Web and VR had a mean
of 4:46min and a median of 5:26. This time-data may be normally distributed
around its mean but does not have to be. To perform a significance test, we
used the Wilcoxon Signed-Rank test [56] since it does not assume a normal
distribution and based on our test we cannot confirm that both samples are
normally distributed. This test reports a $Z=-2,589$ with $p<.05$, meaning
that VR had statistically significantly longer task times than Web.

####  V-D3 Effectiveness

In order to consistently track errors across all recordings we had to define
what ”error” means in the context of this usability evaluation. Our definition
of a Usability Problem is based on [57] where the authors define a Usability
Problem as ”a set of negative phenomena, such as user’s inability to reach
his/her goal, inefficient interaction and/or user’s dissatisfaction, caused by
a combination of user interface design factors and factors of usage context”
[57]. Based on this definition, we extracted 3 types of errors that we
analyzed in the recordings:

  1. 1.

Missed Interaction Point: The user tried to interact with an element of the
application but did not hit said element (for example a button) or used the
wrong control for interaction. Thus, the user needs to repeat the interaction.

  2. 2.

Accidental Interaction: The user did not intend to interact at all or not with
this specific element but accidentally interacted with it anyway. Thus, the
user needs to revert the interaction.

  3. 3.

False Interaction: The user tried to do an interaction that is not possible at
all or not possible at that specific element. Thus, the user experiences a
loss of time and needs to find out the correct interaction to achieve the
desired effect.

During the evaluation of the recordings, we tracked each error by its type and
a time stamp. Finally, the effectiveness was measured in errors per minute and
is shown in Figure 5. Most errors belonged to the Missed Interaction Point or
Accidental Interaction types with only a few False Interactions.

Figure 5: The error rates of the recorded participant in each group.

In VR, we observed a mean error rate of approximately 1.5 (error per minute)
with a median of 1.25 (error per minute), while these values lied at 0.66 and
0.61 (error per minute) respectively in the Web application. The Wilcoxon
Signed-Rank test resulted in a test statistic of $Z=-2.599$ with a
significance $<.05$ which shows that VR had a statistically significantly
higher error rate than Web.

####  V-D4 User Satisfaction

The basic user satisfaction was measured in the VR application using the 10
items SUS questionnaire for each participant. In total, our collaborative VR
modeling environment reached an average SUS score of 78 out of 100 which
indicates according to [58] good usability. As mentioned earlier, we did not
ask the participants to fill out the SUS questionnaire for the Web application
as our focus was more on the acceptance of our own VR solution than on the
acceptance of a mature and commercial modeling tool like Lucidchart.

####  V-D5 Additional Questionnaire Results

To assess the perceived naturalness of both applications, the Web and our VR
solution, the similarity of interactions inside the applications to face-to-
face interactions and the feeling of co-presence were analyzed. The results
are depicted in Fig. 6.

Figure 6: Results of additional questions concerning interactivity and co-
presence

Concerning the similarity of interactions to face-to-face interactions (see
Fig. 6 (a)), the average score of the Web application is 2.33 and noticeably
lower than the VR’s 3.58. Additionally, in VR 14 out of 24 people selected the
highest (5/5 on the Likert scale) answers whereas, in the Web, nobody chose
the highest and only two people chose the second highest answer (4/5 on the
Likert scale). A Wilcoxon Signed-Rank test revealed that this observed
difference is indeed statistically significant with $Z=-3.208$ and a
significance of $<0.05$. Overall, this means that users found interactions
with their teammates to be significantly more similar to face-to-face
interactions in the VR modeling environment compared to the Web application. A
further important aspect for natural interaction and collaboration is co-
presence which denotes the feeling of being in the same place despite the
remote setting. Fig. 6 (b) shows how the participants assessed their feeling
of co-presence in both applications. It shows that in the Web, this feeling is
very different from user to user, with larger bulks at both ends of the scale.
The median, however, with a value of 2.6 points more towards the lower end of
the scale. In the VR app, this is very different. While two people rated the
co-presence rather low (2 on the 1-5 scale), every other participant at least
somewhat agreed with the sentiment that there was a feeling of co-presence
with 9 participants selecting 4/5 and 13 choosing 5/5. The Wilcoxon Signed-
Rank test confirmed that this difference is statistically significant with a
$Z=-3.587$ and a significance of $<0.05$. This implies that the VR application
can more successfully make users feel like they are collaborating in the same
room compared to the Web app. Apart from the above-mentioned questions, we
asked the participants to provide us general feedback and remarks (what have
you liked/disliked most) on the developed collaborative VR modeling
environment. The most notable result is that 14 out of 24 people mentioned
that moving the model elements through the grabbing feature felt natural.
Furthermore, most comments were praising the ”collaboration” in the VR
modeling environment. Some comments explicitly mentioned a like for the
collaboration (3 out of 24), some indicated a positive impression for the
ability of talking to the teammate (4 out of 24), being able to see the
teammate (4 out of 24) or the feeling of the teammate’s presence (4 out of
24). Some participants mentioned multiple of these aspects. In total, 11 out
of 24 noted that at least one of these collaboration-related aspects felt
natural or intuitive. Concerning the negative feedback comments, typing on the
keyboard in VR was the most mentioned aspect (9 out of 24) while some of the
participants stated that this would only need some time to get used to it.
Furthermore, the missing of an auto-alignment or snapping feature, like known
from most diagramming and modeling tools (e.g. Microsoft Visio [59]), was
complained about by five users. However, five out of 24 people said that VR
was entertaining or fun to use and two added that it is specifically useful
for home office scenarios since it makes people feel more together and would
”definitely improve motivation and team spirit”. Finally, the participants had
to choose which application they would prefer to use for collaborative UML
modelling: They could choose either one or state that they would want to use
both depending on the situation. A follow up question to this one subsequently
asked in which situations they would want to use which application if they
selected ”Both”. 13% (3 out of 24) wanted to use the VR app over the Web
application and 37% (9 out of 24) vice versa. Half of the participants,
however, indicated that they would want to use both applications, each for
specific situations. When asked in which situations they would want to use
which application, participants gave rather diverse answers. Five users stated
that they would want to use the Web application for complex or longer tasks
and the VR one for shorter ones. Two mentioned that they would prefer the Web
application for time critical work. Brainstorming and planning was mentioned
by three people to be more suited for the VR application and one person stated
that she would like to use ”the web application when you are working on one
device with your partner” and the VR app when in ”different locations”.

####  V-D6 Discussion

Based on previous research and the participants’ familiarity with traditional
computer programs, we expected one downside of VR to be that users’ task
executions are slower and more error-prone in VR. The data from our study
shows that this was indeed the case. These measures could have been influenced
by the universal familiarity of participants with PC applications in general
and UML tools specifically. It is therefore possible that speed and error
rates in VR improve as users get more experience with a specific VR UML tool.
However, the current state-of-art text entry can be seen as a major bottleneck
for the efficiency of text-intensive VR applications. The results of the SUS
evaluation additionally showed that our VR implementation is already quite
usable even though it still lacks many features that users expect from such an
application like automatic aligning of model elements and copy & paste
functionalities. The various data points gathered about the naturalness of
different aspects of the application gave a clearer insight on what concrete
advantages such a VR application can have compared to traditional tools: On
average, users found the interactions and especially the collaboration related
aspects of the VR application significantly more natural than in the Web
application. This is especially important with respect to this work’s focus on
remote collaboration settings as the study showed that the feelings of being
together and collaborating face to face with a co-worker were much higher in
VR compared to the traditional PC alternative. Another aspect that we expected
VR to be beneficial for is the motivation and fun users are having while using
it. Our study shows that it is indeed true that users were a lot more
motivated and had a lot more fun using the VR application compared to the Web
app. It is important to note that this could be influenced by the fact that VR
is a relatively new and therefore possibly more interesting technology, so
these values might align more over time when a user regularly uses a VR
application for modeling. The fact that using VR can be more exhausting for
people was one downside of VR we expected to observe in this study as well.
While we did see that some participants experienced significant discomfort
(due to heavy headsets, pressure against the face, or cybersickness), most
people did not have any issues with that. Considering the suggestions made by
participants about when they would like to use which application, it is
evident, however, that a VR UML tool would not simply replace the desktop
ones. Rather it would be an amendment, so users have the option to use the
tool that is most suited for a given situation. In remote collaboration
settings, where two or more people need to brainstorm or discuss how something
should be modeled, the VR application could be used. When collaborating in the
same room or when creating a model alone, a PC-based application could be more
appropriate. This implies that such apps need seamless interoperability
between PCs and VR when they ought to be used in actual modeling work outside
of usability studies. Summarizing, it can be said that VR can offer a more
natural collaborative modeling experience compared to PC-based tools but that
both techniques have certain advantages and drawbacks. These make the use of
both tool-types, depending on the concrete situation, most sensible instead of
using only one of them exclusively.

####  V-D7 Threats to Validity

With regard to collaboration, this study only evaluated two-person teams.
Therefore, it is unclear if the findings can be generalized to larger numbers
of simultaneous collaborators. The number of participants and thereby
collaboration teams was also rather limited. With more participants, a
between-subjects design [52] could have been chosen which has the possibility
to provide data and thus findings that are more generalizable across many
different users. Due to the participant’s demographics (e.g., age, profession,
background, etc.), the findings of this study should be taken carefully and
further user studies with other user groups are needed to get insights about
the general usability. Additionally, this study compared a fully-featured
commercially available Web application with a VR modeling prototype, missing
many of the features that the Web application supported, like automatic
aligning of model elements. This study therefore could not achieve a strict
like-for-like comparison between the two types of applications.

##  VI Conclusion and Future Work

While the Unified Modeling Language (UML) is one of the major conceptual
modeling languages for software engineers, more and more concerns arise from
the modeling quality of UML and its tool-support. Among them, the limitation
of the two-dimensional presentation of its notations and lack of natural
collaborative modeling tools are reported to be significant. In this paper, we
have explored the potential of using Virtual Reality (VR) technology for
collaborative UML software design by comparing it with classical collaborative
software design using conventional devices (Desktop PC / Laptop). For this
purpose, we have presented a VR modeling environment that offers a natural
collaborative modeling experience for UML Class Diagrams. Based on a user
study with 24 participants, we have compared collaborative VR modeling with
conventional modeling with regard to efficiency, effectiveness, and user
satisfaction.

In future work, we plan to extend our collaborative VR modeling environment to
further UML modeling diagram types and even to further modeling languages
(e.g., BPMN, SysML, etc.). In addition, further improvements of the VR
modeling environment are planned to support a multi-modal (e.g. speech
in-/output or haptic feedback) interaction and a more realistic representation
of the virtual character (e.g. realistic avatars or tracking of users’ facial
expressions). Features like automatic aligning of model elements, and tracking
of which collaborator created which parts of the model are also useful aspects
that could be integrated into the VR modeling environment. In addition,
further usability evaluation studies with larger groups of heterogeneous
participants and more complex modeling tasks should be conducted to analyze in
more detail the benefit of collaborative modeling in VR. Finally, we believe
that a cross-device mixed reality collaborative modeling approach is a
promising way to support modeling across different AR/VR capable and
conventional devices.

## References

  * [1] J. Whitehead, “Collaboration in software engineering: A roadmap,” in _Future of Software Engineering (FOSE '07)_. IEEE, May 2007.
  * [2] M. Petre, “UML in practice,” in _35th International Conference on Software Engineering, ICSE ’13, San Francisco, CA, USA, May 18-26, 2013_ , D. Notkin, B. H. C. Cheng, and K. Pohl, Eds. IEEE Computer Society, 2013, pp. 722–731. [Online]. Available: https://doi.org/10.1109/ICSE.2013.6606618
  * [3] M. Gogolla, O. Radfelder, and M. Richters, “Towards three-dimensional representation and animation of uml diagrams,” in _Proceedings of the 2nd International Conference on The Unified Modeling Language: Beyond the Standard_ , ser. UML’99. Berlin, Heidelberg: Springer-Verlag, 1999, p. 489–502.
  * [4] J. I. Maletic, J. Leigh, and A. Marcus, “Visualizing software in an immersive virtual reality environment,” in _in Proceedings of ICSE’01 Workshop on Software Visualization_. Society Press, 2001, pp. 12–13.
  * [5] J. Erickson and K. Siau, “Unified modeling language: Theoretical and practical complexity,” in _9th Americas Conference on Information Systems, AMCIS 2003, Tampa, FL, USA, August 4-6, 2003_. Association for Information Systems, 2003, p. 164. [Online]. Available: http://aisel.aisnet.org/amcis2003/164
  * [6] O. Badreddin, R. Khandoker, A. Forward, O. Masmali, and T. C. Lethbridge, “A decade of software design and modeling: A survey to uncover trends of the practice,” in _Proceedings of the 21th ACM/IEEE International Conference on Model Driven Engineering Languages and Systems, MODELS 2018, Copenhagen, Denmark, October 14-19, 2018_ , A. Wasowski, R. F. Paige, and Ø. Haugen, Eds. ACM, 2018, pp. 245–255. [Online]. Available: https://doi.org/10.1145/3239372.3239389
  * [7] L. S. Inc., “Lucidchart,” https://www.lucidchart.com/, Dec. 2020.
  * [8] Axellience, “Genmymodel,” https://www.genmymodel.com/, Nov. 2020.
  * [9] E. Yigitbas, I. Jovanovikj, J. Scholand, and G. Engels, “VR training for warehouse management,” in _VRST ’20: 26th ACM Symposium on Virtual Reality Software and Technology_ , R. J. Teather, C. Joslin, W. Stuerzlinger, P. Figueroa, Y. Hu, A. U. Batmaz, W. Lee, and F. R. Ortega, Eds. ACM, 2020, pp. 78:1–78:3.
  * [10] E. Yigitbas, K. Karakaya, I. Jovanovikj, and G. Engels, “Enhancing human-in-the-loop adaptive systems through digital twins and VR interfaces,” in _16th International Symposium on Software Engineering for Adaptive and Self-Managing Systems, SEAMS@ICSE 2021, Madrid, Spain, May 18-24, 2021_. IEEE, 2021, pp. 30–40. [Online]. Available: https://doi.org/10.1109/SEAMS51251.2021.00015
  * [11] E. Yigitbas, C. B. Tejedor, and G. Engels, “Experiencing and programming the ENIAC in VR,” in _Mensch und Computer 2020_ , F. Alt, S. Schneegass, and E. Hornecker, Eds. ACM, 2020, pp. 505–506.
  * [12] E. Yigitbas, J. Heindörfer, and G. Engels, “A context-aware virtual reality first aid training application,” in _Proc. of Mensch und Computer 2019_ , F. Alt, A. Bulling, and T. Döring, Eds. GI / ACM, 2019, pp. 885–888.
  * [13] I. Jovanovikj, E. Yigitbas, S. Sauer, and G. Engels, “Augmented and virtual reality object repository for rapid prototyping,” in _Human-Centered Software Engineering - 8th IFIP WG 13.2 International Working Conference, HCSE 2020, Eindhoven, The Netherlands, November 30 - December 2, 2020, Proceedings_ , ser. Lecture Notes in Computer Science, R. Bernhaupt, C. Ardito, and S. Sauer, Eds., vol. 12481. Springer, 2020, pp. 216–224. [Online]. Available: https://doi.org/10.1007/978-3-030-64266-2\\_15
  * [14] E. Yigitbas, J. Klauke, S. Gottschalk, and G. Engels, “VREUD - an end-user development tool to simplify the creation of interactive VR scenes,” _CoRR_ , vol. abs/2107.00377, 2021. [Online]. Available: https://arxiv.org/abs/2107.00377
  * [15] E. Yigitbas, S. Grün, S. Sauer, and G. Engels, “Model-driven context management for self-adaptive user interfaces,” in _Ubiquitous Computing and Ambient Intelligence - 11th International Conference, UCAmI 2017, Philadelphia, PA, USA, November 7-10, 2017, Proceedings_ , ser. Lecture Notes in Computer Science, S. F. Ochoa, P. Singh, and J. Bravo, Eds., vol. 10586\. Springer, 2017, pp. 624–635. [Online]. Available: https://doi.org/10.1007/978-3-319-67585-5\\_61
  * [16] E. Yigitbas, S. Sauer, and G. Engels, “Adapt-ui: an IDE supporting model-driven development of self-adaptive uis,” in _Proceedings of the ACM SIGCHI Symposium on Engineering Interactive Computing Systems, EICS 2017, Lisbon, Portugal, June 26-29, 2017_ , J. C. Campos, N. Nunes, P. Campos, G. Calvary, J. Nichols, C. Martinie, and J. L. Silva, Eds. ACM, 2017, pp. 99–104. [Online]. Available: https://doi.org/10.1145/3102113.3102144
  * [17] E. Yigitbas, I. Jovanovikj, K. Biermeier, S. Sauer, and G. Engels, “Integrated model-driven development of self-adaptive user interfaces,” _Softw. Syst. Model._ , vol. 19, no. 5, pp. 1057–1081, 2020. [Online]. Available: https://doi.org/10.1007/s10270-020-00777-7
  * [18] E. Yigitbas, A. Anjorin, I. Jovanovikj, T. Kern, S. Sauer, and G. Engels, “Usability evaluation of model-driven cross-device web user interfaces,” in _Human-Centered Software Engineering - 7th IFIP WG 13.2 International Working Conference, HCSE 2018, Sophia Antipolis, France, September 3-5, 2018, Revised Selected Papers_ , ser. Lecture Notes in Computer Science, C. Bogdan, K. Kuusinen, M. K. Lárusdóttir, P. A. Palanque, and M. Winckler, Eds., vol. 11262. Springer, 2018, pp. 231–247. [Online]. Available: https://doi.org/10.1007/978-3-030-05909-5\\_14
  * [19] E. Yigitbas, A. Hottung, S. M. Rojas, A. Anjorin, S. Sauer, and G. Engels, “Context- and data-driven satisfaction analysis of user interface adaptations based on instant user feedback,” _Proc. ACM Hum. Comput. Interact._ , vol. 3, no. EICS, pp. 19:1–19:20, 2019. [Online]. Available: https://doi.org/10.1145/3331161
  * [20] E. Yigitbas, B. Mohrmann, and S. Sauer, “Model-driven UI development integrating HCI patterns,” in _Proceedings of the 1st Workshop on Large-scale and Model-based Interactive Systems: Approaches and Challenges, LMIS 2015, co-located with 7th ACM SIGCHI Symposium on Engineering Interactive Computing Systems (EICS 2015), Duisburg, Germany, June 23, 2015_ , ser. CEUR Workshop Proceedings, R. Seiger, B. Altakrouri, A. Schrader, and T. Schlegel, Eds., vol. 1380. CEUR-WS.org, 2015, pp. 42–46. [Online]. Available: http://ceur-ws.org/Vol-1380/paper6.pdf
  * [21] K. Josifovska, E. Yigitbas, and G. Engels, “A digital twin-based multi-modal UI adaptation framework for assistance systems in industry 4.0,” in _Human-Computer Interaction. Design Practice in Contemporary Societies - Thematic Area, HCI 2019, Held as Part of the 21st HCI International Conference, HCII 2019, Orlando, FL, USA, July 26-31, 2019, Proceedings, Part III_ , ser. Lecture Notes in Computer Science, M. Kurosu, Ed., vol. 11568\. Springer, 2019, pp. 398–409. [Online]. Available: https://doi.org/10.1007/978-3-030-22636-7\\_30
  * [22] ——, “Reference framework for digital twins within cyber-physical systems,” in _Proceedings of the 5th International Workshop on Software Engineering for Smart Cyber-Physical Systems, SEsCPS@ICSE 2019, Montreal, QC, Canada, May 28, 2019_ , T. Bures, B. R. Schmerl, J. S. Fitzgerald, and D. Weyns, Eds. IEEE / ACM, 2019, pp. 25–31. [Online]. Available: https://doi.org/10.1109/SEsCPS.2019.00012
  * [23] S. Gottschalk, E. Yigitbas, A. Nowosad, and G. Engels, “Situation-specific business model development methods for mobile app developers,” in _Enterprise, Business-Process and Information Systems Modeling - 22nd International Conference, BPMDS 2021, and 26th International Conference, EMMSAD 2021, Held at CAiSE 2021, Melbourne, VIC, Australia, June 28-29, 2021, Proceedings_ , ser. Lecture Notes in Business Information Processing, A. Augusto, A. Gill, S. Nurcan, I. Reinhartz-Berger, R. Schmidt, and J. Zdravkovic, Eds., vol. 421. Springer, 2021, pp. 262–276. [Online]. Available: https://doi.org/10.1007/978-3-030-79186-5\\_17
  * [24] S. Gottschalk, E. Yigitbas, and G. Engels, “Model-based hypothesis engineering for supporting adaptation to uncertain customer needs,” in _Business Modeling and Software Design - 10th International Symposium, BMSD 2020, Berlin, Germany, July 6-8, 2020, Proceedings_ , ser. Lecture Notes in Business Information Processing, B. Shishkov, Ed., vol. 391. Springer, 2020, pp. 276–286. [Online]. Available: https://doi.org/10.1007/978-3-030-52306-0\\_18
  * [25] M. Renger, G. L. Kolfschoten, and G. de Vreede, “Challenges in collaborative modelling: a literature review and research agenda,” _Int. J. Simul. Process. Model._ , vol. 4, no. 3/4, pp. 248–263, 2008. [Online]. Available: https://doi.org/10.1504/IJSPM.2008.023686
  * [26] N. Boulila, “Supporting distributed software development with rd-uml,” in _GI-Edition: Lecture Notes in Informatics_ , Jan. 2002.
  * [27] W. Chen, R. Pedersen, and Ø. Pettersen, “Colemo: A collaborative learning environment for UML modelling,” _Interact. Learn. Environ._ , vol. 14, no. 3, pp. 233–249, 2006. [Online]. Available: https://doi.org/10.1080/10494820600909165
  * [28] M. Ferenc, I. Polásek, and J. Vincur, “Collaborative modeling and visualization of software systems using multidimensional UML,” in _IEEE Working Conference on Software Visualization, VISSOFT 2017, Shanghai, China, September 18-19, 2017_. IEEE, 2017, pp. 99–103. [Online]. Available: https://doi.org/10.1109/VISSOFT.2017.19
  * [29] O. Radfelder and M. Gogolla, “On better understanding uml diagrams through interactive three-dimensional visualization and animation,” in _Proceedings of the Working Conference on Advanced Visual Interfaces_ , ser. AVI ’00. New York, NY, USA: Association for Computing Machinery, 2000, p. 292–295. [Online]. Available: https://doi.org/10.1145/345513.345358
  * [30] B. Zhang and Y. sho Chen, “Enhancing UML conceptual modeling through the use of virtual reality,” in _Proceedings of the 38th Annual Hawaii International Conference on System Sciences_. IEEE, 2005.
  * [31] C. S. C. Rodrigues, C. M. L. Werner, and L. Landau, “VisAr3d: an innovative 3d visualization of uml models,” in _Proceedings of the 38th International Conference on Software Engineering Companion - ICSE '16_. ACM Press, 2016\.
  * [32] M. Leyer, R. Brown, B. Aysolmaz, I. Vanderfeesten, and O. Turetken, “3d virtual world BPM training systems: Process gateway experimental results,” in _Advanced Information Systems Engineering_. Springer International Publishing, 2019, pp. 415–429.
  * [33] K. Casey and C. Exton, “A java 3d implementation of a geon based visualisation tool for UML,” in _Proceedings of the 2nd International Symposium on Principles and Practice of Programming in Java, PPPJ 2003, Kilkenny City, Ireland, June 16-18, 2003_ , ser. ACM International Conference Proceeding Series, J. F. Power and J. Waldron, Eds., vol. 42. ACM, 2003, pp. 63–65. [Online]. Available: https://dl.acm.org/citation.cfm?id=957309
  * [34] J. von Pilgrim and K. Duske, “Gef3d: a framework for two-, two-and-a-half-, and three-dimensional graphical editors,” in _Proceedings of the ACM 2008 Symposium on Software Visualization, Ammersee, Germany, September 16-17, 2008_ , R. Koschke, C. D. Hundhausen, and A. C. Telea, Eds. ACM, 2008, pp. 95–104. [Online]. Available: https://doi.org/10.1145/1409720.1409737
  * [35] J. von Pilgrim, K. Duske, and P. McIntosh, “Eclipse GEF3D: bringing 3d to existing 2d editors,” _Inf. Vis._ , vol. 8, no. 2, pp. 107–119, 2009. [Online]. Available: https://doi.org/10.1057/ivs.2009.9
  * [36] P. McIntosh and M. Hamilton, “X3D-UML: 3d UML mechatronic diagrams,” in _21st Australian Software Engineering Conference (ASWEC 2010), 6-9 April 2010, Auckland, New Zealand_. IEEE Computer Society, 2010, pp. 85–93. [Online]. Available: https://doi.org/10.1109/ASWEC.2010.14
  * [37] E. Yigitbas, I. Jovanovikj, and G. Engels, “Simplifying robot programming using augmented reality and end-user development,” _CoRR_ , vol. abs/2106.07944, 2021. [Online]. Available: https://arxiv.org/abs/2106.07944
  * [38] S. Gottschalk, E. Yigitbas, E. Schmidt, and G. Engels, “Model-based product configuration in augmented reality applications,” in _Human-Centered Software Engineering - 8th IFIP WG 13.2 International Working Conference, HCSE 2020, Eindhoven, The Netherlands, November 30 - December 2, 2020, Proceedings_ , ser. Lecture Notes in Computer Science, R. Bernhaupt, C. Ardito, and S. Sauer, Eds., vol. 12481. Springer, 2020, pp. 84–104. [Online]. Available: https://doi.org/10.1007/978-3-030-64266-2\\_5
  * [39] ——, “Proconar: A tool support for model-based AR product configuration,” in _Human-Centered Software Engineering - 8th IFIP WG 13.2 International Working Conference, HCSE 2020, Eindhoven, The Netherlands, November 30 - December 2, 2020, Proceedings_ , ser. Lecture Notes in Computer Science, R. Bernhaupt, C. Ardito, and S. Sauer, Eds., vol. 12481\. Springer, 2020, pp. 207–215. [Online]. Available: https://doi.org/10.1007/978-3-030-64266-2\\_14
  * [40] E. Yigitbas, S. Sauer, and G. Engels, “Using augmented reality for enhancing planning and measurements in the scaffolding business,” in _EICS ’21: ACM SIGCHI Symposium on Engineering Interactive Computing Systems, virtual, June 8-11, 2021_. ACM, 2021\. [Online]. Available: https://doi.org/10.1145/3459926.3464747
  * [41] S. Krings, E. Yigitbas, I. Jovanovikj, S. Sauer, and G. Engels, “Development framework for context-aware augmented reality applications,” in _EICS ’20: ACM SIGCHI Symposium on Engineering Interactive Computing Systems, Sophia Antipolis, France, June 23-26, 2020_ , J. Bowen, J. Vanderdonckt, and M. Winckler, Eds. ACM, 2020, pp. 9:1–9:6. [Online]. Available: https://doi.org/10.1145/3393672.3398640
  * [42] E. Yigitbas, I. Jovanovikj, S. Sauer, and G. Engels, “On the development of context-aware augmented reality applications,” in _Beyond Interactions \- INTERACT 2019 IFIP TC 13 Workshops, Paphos, Cyprus, September 2-6, 2019, Revised Selected Papers_ , ser. Lecture Notes in Computer Science, J. L. Abdelnour-Nocera, A. Parmaxi, M. Winckler, F. Loizides, C. Ardito, G. Bhutkar, and P. Dannenmann, Eds., vol. 11930. Springer, 2019, pp. 107–120. [Online]. Available: https://doi.org/10.1007/978-3-030-46540-7\\_11
  * [43] A. Mikkelsen, S. Honningsøy, T.-M. Grønli, and G. Ghinea, “Exploring microsoft hololens for interactive visualization of UML diagrams,” in _Proceedings of the 9th International Conference on Management of Digital EcoSystems - MEDES '17_. ACM Press, 2017.
  * [44] R. Reuter, F. Hauser, D. Muckelbauer, T. Stark, E. Antoni, J. Mottok, and C. Wolff, “Using augmented reality in software engineering education? first insights to a comparative study of 2d and AR UML modeling,” in _Proceedings of the 52nd Hawaii International Conference on System Sciences_. Hawaii International Conference on System Sciences, 2019.
  * [45] R. Seiger, R. Kühn, M. Korzetz, and U. Aßmann, “Holoflows: modelling of processes for the internet of things in mixed reality,” _Software and Systems Modeling_ , pp. 1–25, 2021.
  * [46] L. Brunschwig, R. Campos-López, E. Guerra, and J. de Lara, “Towards domain-specific modelling environments based on augmented reality,” in _43rd IEEE/ACM International Conference on Software Engineering: New Ideas and Emerging Results, ICSE (NIER) 2021, Madrid, Spain, May 25-28, 2021_. IEEE, 2021, pp. 56–60. [Online]. Available: https://doi.org/10.1109/ICSE-NIER52604.2021.00020
  * [47] J. Maletic, J. Leigh, A. Marcus, and G. Dunlap, “Visualizing object-oriented software in virtual reality,” in _Proceedings 9th International Workshop on Program Comprehension. IWPC 2001_. IEEE Comput. Soc, 2001.
  * [48] A. Dengel, “Seeking the treasures of theoretical computer science education: Towards educational virtual reality for the visualization of finite state machines,” in _2018 IEEE International Conference on Teaching, Assessment, and Learning for Engineering (TALE)_. IEEE, Dec. 2018.
  * [49] R. Oberhauser, C. Pogolski, and A. Matic, “VR-BPMN: Visualizing BPMN models in virtual reality,” in _Lecture Notes in Business Information Processing_. Springer International Publishing, 2018, pp. 83–97.
  * [50] U. Technologies, “Unity engine,” https://unity.com, Feb. 2021.
  * [51] Exit Games, “Pun — player networking,” https://doc.photonengine.com/en-us/pun/v2/demos-and-tutorials/pun-basics-tutorial/player-networking, Feb. 2021.
  * [52] C. M. Barnum, _Usability Testing Essentials: Ready, Set…Test!_ , 1st ed. San Francisco, CA, USA: Morgan Kaufmann Publishers Inc., 2010.
  * [53] Google LLC, “Google forms,” https://www.google.de/intl/en/forms/about/, Mar. 2021.
  * [54] J. Brooke, _”SUS - A quick and dirty usability scale.” Usability evaluation in industry_. CRC Press, June 1996, iSBN: 9780748404605. [Online]. Available: https://www.crcpress.com/product/isbn/9780748404605
  * [55] A. Bangor, P. Kortum, and J. Miller, “Determining what individual sus scores mean: Adding an adjective rating scale,” _J. Usability Stud._ , vol. 4, pp. 114–123, 04 2009.
  * [56] Ahmed Sabbir Arif, University of California, “A brief note on selecting & reporting the right statistical test,” https://www.asarif.com/notes/HypothesisTesting.html, Mar. 2021.
  * [57] P. Manakhov and V. D. Ivanov, “Defining usability problems,” in _Proceedings of the 2016 CHI Conference Extended Abstracts on Human Factors in Computing Systems_. ACM, May 2016.
  * [58] J. Sauro, _A practical guide to the system usability scale: background, benchmarks & best practices_. Denver, Colorado: Measuring Usability LLC, 2011.
  * [59] Microsoft, “Visio — the ultimate tool for diagramming,” https://www.microsoft.com/en-us/microsoft-365/visio/flowchart-software/, Feb. 2021.

  *[UML]: Unified Modeling Language
  *[HMD]: Head Mounted Display
  *[VR]: Virtual Reality
  *[BPMN]: Business Process Model Notation
  *[AR]: Augmented Reality
  *[CAVE]: Cave Automatic Virtual Environment
  *[FSM]: Finite State Machine
  *[PUN2]: Photon Unity Networking 2
  *[SDK]: Software Development Kit
  *[SUS]: System Usability Scale