* Guyon (2003) Guyon, Isabelle (2003), “Design of experiments of the nips 2003 variable selection benchmark.” In _NIPS 2003 workshop on feature extraction and feature selection_ , volume 253.
  * Hinton (1888) Hinton, Charles Howard (1888), _A new era of thought_. S. Sonnenschein & Company.
  * Kimeldorf and Wahba (1971) Kimeldorf, George and Grace Wahba (1971), “Some results on tchebycheffian spline functions.” _Journal of mathematical analysis and applications_ , 33, 82–95.
  * Koltchinskii (2009) Koltchinskii, Vladimir (2009), “Sparsity in penalized empirical risk minimization.” In _Annales de l’IHP Probabilités et statistiques_ , volume 45, 7–57.
  * Rennie (2005) Rennie, Jason DM (2005), “Smooth hinge classification.”
  * Rudi et al. (2015) Rudi, Alessandro, Raffaello Camoriano, and Lorenzo Rosasco (2015), “Less is more: Nyström computational regularization.” In _Advances in Neural Information Processing Systems_ , 1657–1665.
  * Salzo and Suykens (2016) Salzo, Saverio and Johan AK Suykens (2016), “Generalized support vector regression: duality and tensor-kernel representation.” _arXiv preprint arXiv:1603.05876_.
  * Salzo et al. (2017) Salzo, Saverio, Johan AK Suykens, and Lorenzo Rosasco (2017), “Solving $\ell^{p}$-norm regularization with tensor kernels.” _arXiv preprint arXiv:1707.05609_.
  * Smola and Schölkopf (1998) Smola, Alex J and Bernhard Schölkopf (1998), _Learning with kernels_ , volume 4. Citeseer.
  * Steinwart and Christmann (2008) Steinwart, Ingo and Andreas Christmann (2008), _Support vector machines_. Springer Science & Business Media.
  * Stone (1977) Stone, Charles J. (1977), “Consistent nonparametric regression.” _Ann. Statist._ , 5, 595–620, URL https://doi.org/10.1214/aos/1176343886.
  * Von Luxburg and Schölkopf (2011) Von Luxburg, Ulrike and Bernhard Schölkopf (2011), “Statistical learning theory: Models, concepts, and results.” In _Handbook of the History of Logic_ , volume 10, 651–706, Elsevier.
  * Zhang et al. (2009) Zhang, Haizhang, Yuesheng Xu, and Jun Zhang (2009), “Reproducing kernel banach spaces for machine learning.” _Journal of Machine Learning Research_ , 10, 2741–2775.
  * Zhang (2004) Zhang, Tong (2004), “Solving large scale linear prediction problems using stochastic gradient descent algorithms.” In _Proceedings of the twenty-first international conference on Machine learning_ , 116.
  * Zou and Hastie (2005) Zou, Hui and Trevor Hastie (2005), “Regularization and variable selection via the elastic net.” _Journal of the Royal Statistical Society, Series B_ , 67, 301–320.

##

  * Acknowledgements

I would like to thank my advisor of thesis, Saverio, who helped me a lot in
each step of this thesis. I want to express my gratitute to prof. Pelillo who
keeps providing me with amazing opportunities since my bachelor, I totally
appreciate everything, thank you! In particular thanks for this collaboration
with IIT of Genoa, thanks to Massimiliano. Thanks to the representative group
for providing a friendly working environment (in particular thanks to prof.
Raffaetà for being a lovely person).

My gratitute goes also to my family, that provided me with the best conditions
and environment that allowed me to study for all of this time. I would like to
give a big thanks to my ethiopian buddies (manu) Leulee, Yoshua and Yonathan
who introduced me to the humanities library in Venice and to late night
studying and shared their hard working habits with me. Thanks for all the time
spent together, thanks for the time in via roma apartment (thanks Carmelo),
Hiwot Betam des Titalech (with arms wide open)! A warm thought goes to my
library friends, Mattia (because every great friendship begins with a
lighter), Giorgio (thanks for all the great music and great time), Marco
(thanks for writing my name on public bathrooms around the world) and Giacomos
(little Giacomo: thanks for the laughs, big Giacomo: thanks for the amazing
discussions while having dinner). Thanks for the philosophy I didn’t ask for,
but greatly appreciated. Thanks for filling my head full of philosophical
reasoning after long days of studying. Thanks for all the amazing time spent
in Campo s. Margherita (probably this line includes all the people cited
here). Thanks to Laura and Flavia for sharing an amazing late night library
time, followed by a well-deserved cold beer. In particular to Flavia for the
beer part, while Laura was a great canteen companion :). So thanks to the
canteen for all the delicious food and the amazingly funny time spent on those
tables. Thanks for laughing tears while eating, thanks to all the people who
ate with me. Thanks to Sebastiano that introduced me to the fantastic world of
research. Thanks for spending the christmas break for working on that paper. I
feel honored to be the first student having you in the graduation commission.
Thanks to all the PhDs of the group, in particular to Ismail for all the time
spent having fun and sharing your deep learning wisdom. Thanks to Federico for
graduating together with me, for the second time, in particular thanks for
last minute commentary about last minute changes (just as this line :). Thanks
to Shadow and Greg who spent the first summer in that library with me, working
on projects and creating socks coloured heatmaps.

Thanks to the stunning city of Venice for providing thoughtful walks and
inspiring views at every corner. It was a complete honor to be part of Ca’
Foscari, to live in Venice and meet all these fantastic people inside and
outside of university. Thanks to all the hundreds and thousands of people from
all around the world visiting Venice who stopped by to share some happiness
with me. Thanks for teaching me how to say:

\- Shenghuo feichang piaoliang - Jizn prekrasnaya - Elämä on todella kaunista
- $\check{Z}$ivot je lijep - Az élet nagyon szép - A vide é muito bonita -
Jeeveethey hongak lassamai - Livet är sa snygg - Dzive ir loti skaista -
Zendegi zibast
Jeta është shumë e bukur!


language=XML, extendedchars=true, breaklines=true, breakatwhitespace=true,
emph=, emphstyle=, basicstyle=, columns=fullflexible, commentstyle=,
morestring=[b]", morecomment=[s]<??>, morecomment=[s][]<!---->, keywordstyle=,
stringstyle=, tagstyle=, morekeywords=attribute,xmlns,version,type,release,
otherkeywords=attribute=, xmlns=,

11institutetext: Paderborn University
Zukunftsmeile 2, 33102 Paderborn, Germany
11email<EMAIL_ADDRESS>
Authors’ Instructions

# Virtual Reality in University Teaching: Experiences from a Computer Science
Seminar

Enes Yigitbas

###### Abstract

Due to the corona pandemic, numerous courses were held using digital solutions
in order to be able to continue teaching. Conventional collaboration tools
(Zoom, Big Blue Button, etc.) were used in particular to digitally map a
synchronous session for teaching and learning purposes. While these
conventional collaboration tools offer a solid basis for communication between
learners and teachers, aspects such as presence or a realistic type of
interaction are neglected. In this work, we report on the experiences from a
computer science seminar where virtual reality (VR) technology was used as an
alternative solution for teaching and group work. The benefits of VR compared
to conventional collaboration tools were examined using questionnaires and
interviews with the participants. On the one hand, the results show the high
potential of VR to increase the clarity and experienceability of learning
content and to promote cooperation through social presence. On the other hand,
the use of VR brings with it some technical and organizational difficulties
that should be taken into account in the didactic implementation.

###### Keywords:

Virtual Reality, Remote Education, Teaching and Learning

##  1 Introduction

Digital devices and media are increasingly being used for learning and
educational purposes these days. This was already evident in the period
1997-2006 when networked computers were used extensively for collaborative
learning, and in the period 2007-2016 when so-called online digital learning
became widespread [20]. In recent years, virtual reality (VR) technology has
been actively integrated into education, teaching, and training in various
application domains [7]. Although VR is not new and was defined as ”a real or
simulated environment in which a perceiver experiences telepresence” [24],
recent developments in display technology and computer graphics have made VR
more affordable and available to a wider spectrum made accessible by humans.
The latest VR head-mounted displays (HMDs) such as Valve Index or Oculus Quest
2 allow users to immerse themselves in a realistic virtual world. The
potential of VR technology for education is compelling according to experts
and according to some experts like Sol Rogers VR is the learning aid of the
21st century [21]. This is mainly due to the fact that learning activities and
practical experiments, which are often too expensive, too dangerous, or simply
too time-consuming in the real-physical learning space, can be replaced by
virtual experiences. Based on Pirker et al. [19], translating these
experiences into immersive virtual realities can be of great benefit to
learners, as the sense of being physically, socially, and personally present
in the virtual environment contributes to the learning process.

In view of these promises of VR technology, the question arises as to what the
advantages and disadvantages of using VR are compared to classic collaboration
tools such as Zoom or Big Blue Button. Due to the corona pandemic and the
associated restrictions, numerous courses on digital solutions were held in
order to overcome the challenges of classroom teaching and to offer
alternative ways of being able to continue teaching. In particular, the
classic collaboration tools mentioned were increasingly used to digitally map
a synchronous session for teaching and learning purposes. While these offer a
solid basis for communication and interaction between learners and teachers,
aspects such as presence (i.e. the feeling of ”being there”) or a realistic
type of interaction (e.g. through natural movement, gestures, etc.) are
neglected. In order to explore an alternative solution to this problem, a new
teaching concept was developed in a computer science master’s seminar ”Mixed
Reality Software and Technology (XRST)” (SS 2021) at the University of
Paderborn, in which VR technology is used as the primary medium for the course
and the associated group work. In the course of the seminar, various plenary
sessions (workshops, presentations, discussions, etc.) and group work were
carried out using a VR learning room. Using questionnaires and interviews with
the participants, the user-friendliness, interaction, collaboration and
learning support of VR were examined in comparison to conventional
collaboration tools. In this work, the advantages and disadvantages of using
VR in university teaching are analyzed and discussed using the example of a
computer science seminar. The focus is on promoting collaboration,
interaction, and learning support from VR compared to conventional
collaboration tools. In this context, evaluation results based on the
questionnaires and interviews are presented and critically reflected upon.

This work is structured as follows: Chapter 2 describes the basic concepts on
the subject of Virtual Reality. Chapter 3 deals with related work in the
context of VR in university teaching. Chapter 4 then presents the research
questions that are to be answered in the teaching research project. Chapter 5
explains the new teaching concept and the course of the event, which was
primarily based on VR. The evaluation of the event with reference to the
research questions is discussed in Chapter 6. A discussion and critical
reflection of the evaluation results and findings can be found in Chapter 7.
The article ends with a summary and outlook in Chapter 8.

##  2 Background and Related Work

This chapter described relevant background information. First, a brief
introduction to Virtual Reality technology is given. Then, VR-based
collaboration tools are presented, with a particular focus on the VR
collaboration platform ”Spatial” used in the seminar under consideration.

###  2.1 Introduction to Virtual Reality Technology

Virtual Reality (VR) is the representation and simultaneous perception of
reality and its physical properties in a real-time, computer-generated,
interactive virtual environment. According to Biocca and Delaney [3], VR can
be defined as ”the total sum of hardware and software systems attempting to
perfect an all-encompassing, sensory illusion of presence in another
environment.” Immersion, presence, and interactivity are considered core
features of VR technology ([22], [25]). According to Slater and Wilbur [23],
immersion is defined as ”a perception of being physically present in a non-
physical world by surrounding users of a VR environment with images, sound or
stimuli” so that they feel to actually be ”there”. Immersion thus describes
the embedding of the user in the virtual world. The perception of oneself in
the real world is reduced and the user feels more like a person in the virtual
world. The more immersive a VR experience is, the more realistic it feels to
the user. Presence is defined as ”the subjective experience of being in one
place or environment, even when physically located elsewhere” [26]. Finally,
the term interactivity can be described as the extent to which users can
change the VR environment in real-time [24]. VR technology is becoming
increasingly sophisticated and cost-effective and can be applied to many areas
such as training [33], prototyping (e.g., [14], [35]), robotics [34],
education (e.g., [39], [38]), healthcare (e.g., [30], [36]), finance [29], or
even software modeling (e.g., [27], [28]).

While VR realizes the idea of a virtual world in which the user feels present
through the manipulation of their senses and is completely immersed in a
virtual world without having any relation to the outside world, augmented
reality (AR) is an extension of reality. With augmented reality, the real-
physical world is enriched with digital information such as text, 3D objects,
images, etc. [1]. Similar to VR, AR has already been applied in various
application domains such as robot programming (e.g., [31], [16]), product
configuration (e.g., [10], [11]), planning and measurements [37] or for
realizing smart interfaces (e.g., [15], [32]). While AR also has great
potential for teaching and learning purposes, VR technology is the main focus
of this work.

As the already introduced definition of VR suggests, special hardware and
software are required for an immersive VR experience. In order to create a
feeling of immersion, special input and output devices called VR headsets are
required to display virtual worlds. The Oculus Quest 2 was used to carry out
this work. This mobile, wireless standalone VR headset consists of a head-
mounted display (HMD) and two hand controllers. When using the HMD, two images
are generated and displayed from different perspectives in order to convey a
spatial impression. The HMD is also used to track head movements. This
tracking serves to map the movements, for example by changing the viewing
angle, in VR. Analogously, inputs can be made via the hand controller and hand
and arm movements can be tracked, which overall enables natural control of VR
applications. In addition to special hardware, special software is also
required to support a VR experience. To create virtual reality, you need
software specially developed for this purpose. These programs must be able to
calculate complex three-dimensional worlds in real-time, in stereo (separately
for the left and right eye). Special software development tools (SDKs) and
game engines are available for this purpose to support the development of
complex virtual 3D worlds. In practice, however, there are already existing VR
applications and tools for many areas of application that simplify work in the
respective specialist domain.

###  2.2 VR Collaboration Tools

A collaboration tool is a digital tool for working together in a team. This
involves communication, joint project work, and topics such as the
collaborative processing of documents. Similar to the traditional
collaboration tools (Big Blue Button (BBB), Teams, Zoom, etc.), numerous VR
collaboration tools enable a synchronous session in Virtual Reality. Engage
VR111https://engagevr.io/, Altspace VR222https://altvr.com/ and
Spatial333https://spatial.io/ are some of these VR collaboration tools among
many others. Based on a technological evaluation, the VR collaboration
platform ”Spatial” was selected to carry out this work, which is described in
more detail below.

”Spatial” was originally developed for companies. Customers include well-known
corporations such as Mattel, Nestlé, and Pfizer. In the wake of the Corona
crisis, the start-up opened its platform to the masses by making its services,
including enterprise functions, available to end users without restrictions
and free of charge.

In order to support collaboration in VR as realistically as possible,
”Spatial” provides a number of features. Based on a selfie photo or profile
picture, users can have a realistic 3D avatar created of themselves, which
characterizes the respective user as a digital twin during a virtual session.
This avatar is brought to life during a VR session by reflecting the user’s
movements onto the avatar. In addition, different prefabricated virtual rooms
can be selected for a virtual session (office, seminar room, lecture hall,
campfire, etc.). Furthermore, digital elements can be created in the selected
virtual space by adding 3D objects, photos, images from external sources (hard
disk or web). Other features include free-hand drawing in VR, searching in the
VR web browser or the integration of other services such as Slack or Google
Drive. In addition, the work artifacts (text, images, 3D objects, etc.) can be
viewed and edited together with other participants. In addition to the
avatars, communication options via ”spatial audio” are available for
collaboration, or the option of also seeing each other via the webcam. The
idea of ”Spatial Audio” is to enable auditory communication as in presence,
i.e. the closer you are to a person virtually, the better you can hear the
person. ”Spatial” aims at a collaboration in VR over different device classes.
In addition to typical VR headsets such as Oculus Quest or Valve Index, AR
glasses such as Hololens, Magicleap or Nreal are also supported to access the
virtual collaboration environment. In addition, a web interface is provided
for non-immersive device classes such as desktop PCs, laptops or smartphones,
which can be used to show the virtual collaboration environment as a 3D space
on the screen. This form of access or use of ”Spatial” is referred to in this
article as desktop VR (DVR). While the use of ”Spatial” via an HMD enables an
immersive experience where the collaboration space and other participants can
be experienced as realistically as possible, the use via non-immersive device
classes (DVR) is a 3D representation that is based on a 2D screen.

##  3 Initial Situation: Use and Evaluation of VR Technology in University
Teaching

The early adoption of VR-based solutions for teaching and educational purposes
began with desktop-based 3D virtual environments. For example, a very popular
virtual world called “Second Life” [8] has been introduced to create digital
twins of real-world locations in which users, represented digitally in the
form of avatars, actively participate in realistic activities that stimulate
learning. Although these desktop-based virtual environments (DVR) cannot
provide a fully immersive experience, their photorealistic computer graphics
have been shown to improve learner engagement [6].

A survey of VR usage in teaching and education [20] shows that VR is most
commonly used in the following educational application areas: Engineering
(24%), Computer Science (10%), Astronomy (7%), and Biology (5%). Most of the
existing approaches to using VR in teaching are very specific or limited to
single-user scenarios. In this context, [12] presents, for example,
experiences with VR in a geography lecture in which only one student uses a VR
headset and the rest of the class watches the streamed VR video.

Similarly, the authors in [9] have developed a VR environment in which
individual participants can engage in a virtual chemistry lab. In addition,
projects such as the Erasmus+ project ”Virtual Reality in Higher Education:
Application Scenarios and Recommendation”444https://www.uni-
due.de/proco/erasmusplus.php (2018-2020) or the launched project
”AR/VR.NRW”555https://www.hshl.de/forschung-
unternehmen/forschungsprojekte/forschungsprojekte-im-themenfeld-
materialwissenschaften/ar-vr-nrw/ underline the importance and potential of VR
for university scenarios.

However, existing approaches in this area do not adequately cover the aspects
of multi-user collaboration and interaction in a VR setting and their impact
on the learning outcome. In this context, it can be observed that VR has so
far mostly been part of the experimental and development work and has not been
used regularly in teaching. In particular, pilot projects in which VR is used
throughout a semester and student perception is analyzed in order to evaluate
the user-friendliness of this medium in comparison to traditional
collaboration tools such as Big Blue Button or Zoom are missing in teaching at
universities.

##  4 Research Questions

With the advent of VR technology, it is possible to involve teachers and
learners in a virtual classroom where immersion and presence, key factors of
VR, can contribute to learning processes by creating a sense of ”being there”.
In addition to immersion and presence, VR technology has great potential to
create collaborative learning experiences for modern educational institutions
[40]. Although the advancement of VR technology expands collaborative
learning, related research (see Section 3) on VR-based collaborative learning
is still limited. Therefore, it is important to investigate how the use of VR
affects collaborative learning and how VR influences collaboration and
interaction in learning processes. From this we derive the following research
questions and hypotheses for this work:

(RQ1) How do the students perceive the use of VR in terms of user-friendliness
compared to conventional collaboration tools? How does the use of VR
technology differ in terms of usability when accessed via HMDs or non-
immersive device classes (desktop, laptop, tablet, etc.)? My hypothesis in
terms of usability is that traditional collaboration tools have higher
acceptance and generally perform better because many users are familiar with
them. With regard to the distinction between VR via HMDs and non-immersive
device classes, I expect a significantly higher level of user-friendliness for
the immersive version where HMDs are used.

(RQ2) How do the students perceive the use of VR in terms of collaboration
compared to conventional collaboration tools? My hypothesis is that the use of
VR enables a more natural (realistic) form of collaboration and that this is
positively reflected in aspects such as motivation and fun. On the other hand,
it could be that the efficiency and effectiveness of collaboration suffer
since VR is a new medium for many, where the input and output options are not
familiar when used.

(RQ3) How do the students perceive the use of VR in terms of interaction
compared to conventional collaboration tools? Here is my hypothesis that VR
increases the perception of presence, since the depiction of the participants
as 3D avatars and the enabling of realistic forms of interaction support an
immersive experience that is not possible with conventional collaboration
tools.

(RQ4) How do students perceive the use of VR in terms of learning support
compared to conventional collaboration tools? Here it is to be expected that
the use of VR could be particularly useful for learning processes where visual
and spatial perception is required and should be underpinned by interactive
learning steps.

##  5 Teaching Concept and Course of the Event

The basis of the teaching concept was the original seminar concept of the
Institute for Computer Science. According to the description of the module
manual, the participants work on a topic in a seminar, which is presented in a
lecture followed by a discussion and a written elaboration. The focus here is
on the independent or joint development of research-related sub-areas of
computer science. In the master’s computer science seminar ”Recent Advances in
Mixed Reality Software and Technology”, which deals with the topics of
augmented and virtual reality and their basics and fields of application,
virtual reality was used for the first time in the 2021 summer semester as the
primary medium for conducting the event carried out. At the university, there
is (to the best of our knowledge and belief) no comparable format in which the
advantages and disadvantages of using VR in terms of collaboration,
interaction, and learning support have been analyzed in detail.

Twelve participants were registered for the course. The seminar was designed
and carried out as follows. At the content level, the participants had the
task of selecting a topic from a list of offered topics in the context of
augmented and virtual reality and creating an elaboration and a presentation
on this topic. The seminar was organized as shown in Figure 1.

Figure 1: Seminar Schedule

First, the seminar started with a kick-off meeting via BBB. Since this was the
very first meeting and most of the participants did not have VR headsets, it
was conducted via BBB. The main purpose of this meeting was to encourage the
participants to get to know each other and to provide them with organizational
information and a list of seminar topics. In addition, the participants were
informed about the SoTL project [13].

After the kick-off meeting, the VR participants in the seminar were randomly
selected as the number of available VR headsets (Oculus Quest 2) was limited
to six. After borrowing the VR headsets, there was a VR setup and warm-up
phase, which served to set up the VR environment and test it with the
participants so that everything worked technically flawlessly. The 2nd session
was then carried out in VR, with half of the participants connecting to the VR
learning room ”Spatial” via the VR headsets (hereinafter referred to as VR
users) and the other half connecting via their desktops or laptops based on
the web application of ”Spatial” (hereinafter referred to as DVR users). In
this session, there was an elevator pitch, where the participants were
supposed to give a short (max. 3 min.) presentation on their seminar topic. A
screenshot from the ”Spatial” VR environment during the elevator pitch is
shown in Figure 2.

Figure 2: Screenshot from the Elevator Pitch in VR

The participants should then form groups for the upcoming work phase in the
seminar. The method of speed dating in VR was used to form the groups, where
different zones were provided. When the groups were formed, the rule was that
three different groups should be created where BBB users, VR users, and DVR
users could meet exclusively. The aim of this rule was that the participants
with different media uses did not mix with each other, thereby enabling a
better comparison between the BBB, VR, and DVR users in the seminar.

After the groups had been formed in the 2nd session (3 VR groups of 2
participants each, a DVR group of 2 participants, and a BBB group of 3
participants) the group work phase began, in which each group shared a
presentation of the respective seminar topics.

For the creation of the final presentation, the task of the VR and DVR
participants was to create a VR environment to present their results, while
the BBB teams could use all media formats allowed in BBB. When carrying out
the group work, there was a strict requirement that each group should use the
respective medium BBB, VR or DVR for the group exchange as a meeting. After
the end of the group work phase, there was the next session in VR, where the
final presentations of the groups were presented.

Very creative and interesting VR environments were created (see Figure 3) to
present the results of the individual and group work. Some screenshots from
the VR environments are shown in Figure 3 as an example. In Figure 3 (a), for
example, a group developed a VR learning environment for the medical field,
where body parts of humans and animals can be explored and learned in a
virtual world. In Figure 3 (b), another group explained how 3D printing and
additive manufacturing works in VR. Another group presented in VR how
augmented reality-based drone control can be used in the context of sports
(see Figure 3 (c)). Finally, another group created a VR learning environment,
where the topic of product recommendation systems based on 3D product models
is explained in VR (see Figure 3 (d)).

Figure 3: Screenshots from the VR environments during the final presentation

After completing the final presentations in VR, there was a peer review and
feedback phase where the participants were asked to read and review a seminar
draft by another participant. For this purpose, the well-known conference
management system ”Easychair” was used to familiarize the participants with
the organization and implementation of conferences and review procedures.
Finally, the seminar ended with the 4th session ”Wrap-up”. This last session
took place via BBB in order to gather all participants again in the same
medium and to offer them the opportunity to give final feedback on the content
and organization of the seminar.

##  6 Evaluation

In this chapter, the results of the use of virtual reality in the underlying
seminar are examined in more detail. For this purpose, the data collected
during the seminar is considered. Within the analysis, the research questions
defined in Chapter 4 are answered.

###  6.1 Data Collection

In the course of the event, various data were collected in order to analyze
the respective research questions. First, in the “1st Session: Kick-off
meeting (BBB)” (see Figure 1) a questionnaire was filled out by the twelve
participants, which dealt with the usability, quality of collaboration and
interaction, as well as the participants’ perception of learning via BBB.
There was a similar questionnaire after the “2nd Elevator Pitch Meeting” (see
Figure 1) for the VR and DVR media. There were six VR and five DVR
participants. As a supplement, there were additional questions about the
perception of presence in order to be able to analyze the differences between
VR and DVR with regard to immersion. The questionnaires mentioned also
contained open questions on the general advantages and disadvantages of the
respective media (BBB, VR, DVR). In addition, the participants were asked to
write minutes of their group work meetings in the respective media, which were
also taken into account in the analysis. This form of approach to teaching
research was chosen to provide a comprehensive view of the use of VR by
combining a quantitative and qualitative analysis. The quantitative approach
based on the evaluation of the questionnaires enables a comparative analysis
of the media examined with regard to the relevant criteria. The open questions
and minutes of the group work opened up the possibility of assessing the user-
friendliness and the strengths and weaknesses of the respective media using a
qualitative form of analysis.

###  6.2 Usability and Presence Perception

The System Usability Scale (SUS) questionnaire [18] was used to analyze the
usability of the media used in the seminar (BBB, VR, DVR). The SUS
questionnaire is a standardized questionnaire for assessing the user-
friendliness of an interactive system. It consists of a total of 10 questions
that are rated on a Likert scale from 1-5. Based on these answers, a SUS score
between 0-100 can be determined, which is an indication of the usability of
the interactive system as a whole. As can be seen in the box-plot diagram in
Figure 4 on the left, the average ease of use is highest for Big Blue Button
(BBB) with a SUS score of around 78. The medium of virtual reality based on
the ”spatial” application (VR SUS score 75) takes second place in terms of
user-friendliness, while the use of the same application via desktop or laptop
(DVR SUS score: 68.5 ) performs worst. Based on the evaluation scheme
according to Bangor et al. [2], the SUS scores for BBB and VR can be
classified as ”good” with minimal differences, while the SUS score of DVR can
be classified as ”ok”. (A SUS score above 85 is expected for an “excellent”
rating.) These observations are not surprising as most participants are very
familiar with BBB and have been using it for a long time. On the other hand,
according to their own statements, more than half of the VR participants were
not familiar with the use of a VR HMD or the VR collaboration platform
”Spatial”. For this reason, the user-friendliness of VR should not be viewed
as negatively, even if there is still room for improvement in some areas.
However, the user-friendliness from the perspective of the DVR user is
significantly worse, since the use of a VR application without a VR HMD does
not result in great added value and the display in 3D alone is not sufficient
to enable a realistic meeting. This is also underlined again based on the data
collected with regard to the presence questionnaire. Here, too, an established
questionnaire for the perception of presence was used by Witmer and Singer
[26]. The results show that the average presence perception with VR is
significantly better compared to DVR (see Figure 4 right).

Figure 4: Usability and presence perception of the different media used

In summary, with a view to the research question (Q1), it can be said that the
user-friendliness of established conventional tools such as BBB is very high
and that most users can handle them well, which confirms the hypothesis.
However, the use of virtual reality is associated with initial hurdles and
difficulties, since most users need an introduction or training in the
required hardware or software, which they are usually not completely familiar
with at the beginning. Nevertheless, after overcoming this entry problem, VR
can serve as a medium for a seminar to hold presentations, meetings, etc. It
is important to emphasize that the availability of VR hardware is extremely
important and access to VR via non-immersive devices such as desktop PCs,
laptops or tablets does not offer a great perception of presence and therefore
the potential of VR cannot fully unfold.

###  6.3 Collaboration

To analyze the strengths and weaknesses of the respective media in terms of
collaboration opportunities, a separate questionnaire was created based on
literature research on the topics of ”usability evaluation” and ”collaboration
in VR”. This contains a total of eight questions or statements CQ1-CQ8, which
are shown in Figure 5 and were answered on a Likert scale from 1-5, similar to
the SUS questionnaire. With regard to the first statement ”CQ1 - The
collaboration via the APP felt very natural” it quickly becomes apparent that
VR achieves the best result on average. BBB is second in terms of natural
collaboration, and DVR has the worst results on average. This can possibly be
explained by the inherent properties of the modern VR interaction technology,
which not only reproduces or simulates the virtual space realistically, but
also makes it possible to experience and touch it naturally through movement,
gesture control, etc. BBB and DVR perform relatively poorly here since they
are based on a 2D-based user interface that is operated via mouse/keyboard.

Figure 5: Evaluation results concerning collaboration

However, if we look at collaboration in terms of the perceived efficiency
(CQ2), it quickly becomes clear that the picture is changing. BBB achieves
significantly better results here compared to VR and DVR, which perform
similarly poorly. This can be explained by the fact that direct manipulation
of the display and control elements is possible in classic 2D-based user
interfaces such as BBB, while real control operations such as movements take
longer in VR and DVR.

A similar picture can be seen with regard to the perceived effectiveness (CQ3)
of collaboration, where BBB achieves the best and VR the worst rating on
average. This result can be explained by the fact that precise input options
in VR and DVR are only possible to a limited extent. In VR, inputs are usually
made using the hand controllers, which are not yet as precise to use as a
regular keyboard. In DVR you have the keyboard of the non-immersive device
used, but the processing of 3D elements is error-prone and time-consuming.

Regarding the statement ”CQ4 - The collaboration via the APP was motivating to
increase my concentration on the seminar content” it becomes apparent that VR
performs slightly better than BBB and DVR. This may be due to the increased
perceptual presence described in Section 6.2.

With regard to the fun factor (CQ5) when conducting the seminar, it becomes
clear that VR and DVR achieved significantly higher results on average than
BBB. This may be explained by the fact that most were new to VR as a new
technology while already familiar with BBB. The immersive and dynamic
exploration and experience of the VR environment could also have contributed
to the fun factor in VR being felt so positively.

Based on the CQ6 statement, it becomes clear that BBB was classified on
average as a more helpful medium for getting to know the participants and
seminar content. This could be related to the chat function, where
participants can write to and contact each other directly. In VR/DVR, this was
not directly possible in the ”Spatial” collaboration platform used and one
would have had to withdraw with one of the participants in order to be able to
have a private conversation.

As statement CQ7 shows, most participants can imagine using BBB and VR as a
medium for group work in the seminar. On the other hand, the rating from DVR
was rather negative in this respect, which shows that many do not see this
solution as the right alternative for collaboration in the seminar. Finally,
the evaluation of the CQ8 statement makes it clear that on average BBB is seen
as the most effective collaborative learning environment among the three media
compared, because on the one hand it is probably known and on the other hand
it provides the necessary features for collaboration in a pragmatic and simple
manner.

In summary, with a view to the research question (RQ2), it can be deduced that
VR and DVR bring no real perceived improvement in comparison to conventional
collaboration tools (BBB, Zoom, etc.) in terms of efficiency and
effectiveness. However, it can be seen that VR can have added value if it is
used in a targeted manner to increase the motivation of the participants and
to promote playful learning through doing with a fun factor. Overall, with
reference to the research question (RQ2), the hypothesis can also be confirmed
here.

###  6.4 Interaction

Analogously to the aspect of collaboration, a separate questionnaire was
created to examine the interaction properties. This contains a total of four
questions or statements IQ1-IQ4, which are shown in Figure 6 and were answered
on a Likert scale from 1-5, similar to the SUS questionnaire.

Figure 6: Evaluation results concerning interaction

Regarding the statement ”IQ1 - The interactions with other participants of the
APP felt very similar to the personal (face-to-face) interactions.”, on
average, VR received a significantly higher rating than BBB and DVR. This can
be explained, for example, with the 3D avatars that each person can create for
themselves. By controlling this avatar, for example, with hand, arm and head
movements, the impression of face-to-face interaction in VR is conveyed more.
This is only possible to a limited extent in DVR since the avatar can only be
moved forwards, backwards, left, and right, but other forms of movement are
not possible. In BBB this is completely missing.

The results for the statement ”IQ2 - It felt like the other participants and I
shared the same place while we interacted in the APP.”, where VR was also
perceived as the clear winner, can be explained in a similarly. By
characterizing the virtual environment with 3D objects and depicting oneself
with an avatar, the feeling of sharing the same space with others is conveyed
more strongly. This in turn also strengthens the awareness of the perception
of the virtual space and changes occurring in it (IQ3). Therefore, regarding
”IQ3 - I was always aware of what the participants were presenting or editing
during the session.” VR was perceived most positively. However, BBB takes
second place here, since notifications and visual feedback elements alert
users of BBB to changes in the system status. Finally, statement IQ4 asked
whether the interaction options are appealing to exchange ideas and artifacts.
From this point of view, too, VR was felt to be more beneficial than BBB and
”Spatial”. This emphasizes the importance of 3D representation of artifacts
for the exchange of ideas, allowing for better content delivery and visual
representations.

With a view to the research question (RQ3), it can be summarized that VR
offers many advantages over conventional collaboration tools in terms of
interaction properties and thus enables realistic communication that is closer
to usual face-to-face events. Here, too, the hypothesis can be confirmed.

###  6.5 Learning Support

In order to analyze the perception of the learning support provided by the
respective media (BBB, VR, and DVR), the participants in the seminar were
asked for their assessment of the learning support of the taxonomy levels
according to Bloom [4]. As shown in Figure 7, there were a total of five
questions LQ1-LQ5. The LQ1 question aims to have the participants’ perception
of the learning conduciveness of taxonomy level 1 (knowledge) evaluated.
Similarly, in LQ2 and LQ3, taxonomy levels 2 (understanding) and 3
(application) are asked about their learning support. In LQ4, the higher
taxonomy levels 4-6, i.e. analysis, synthesis, and evaluation, were combined
in one question and asked about their ability to promote learning. Finally,
the final question LQ5 addresses how tiring the medium is to learn new
concepts and ideas. LQ5 was asked intentionally negated in order to explicitly
examine the cognitive load of the respective media.

Figure 7: Evaluation results concerning interaction

Looking at the results in Figure 7, it becomes clear that specifying,
describing or reproducing known content is perceived as easier on average via
BBB than in VR or DVR. This is due to the fact that many well-known features
such as chat function, screen sharing, freehand drawing and shared notes are
possible in BBB, which simplify the presentation and transfer of known
knowledge. In the ”Spatial” collaboration platform used for VR/DVR, some of
the features such as chat functions or screen sharing were not possible.

However, with regard to the support of the learning conduciveness of taxonomy
level 2 (understanding), it becomes apparent that VR was perceived slightly
more positively on average than BBB and DVR. This can be explained with the
three-dimensional VR environment, which facilitates the understanding and
comparison of different concepts through the combination of different
multimodal media objects such as 3D objects, images, audio, video, etc. (see
Figure 3). VR was also rated significantly more positively than DVR and BBB
with regard to the support of the learning conduciveness of taxonomy level 3
(application). This is strongly related to the interactivity of the VR
learning environments, which make learning new ideas and concepts through
movement and exploration naturally tangible and closer.

A similar trend can be seen with regard to the support of the learning
conduciveness of the higher taxonomy levels 4-6 (analysis, synthesis and
assessment). Again, participants felt that VR was, on average, more useful
than BBB and DVR in analyzing, synthesizing and evaluating new ideas. On the
one hand, this can be explained by the aforementioned point of interactivity
and, on the other hand, by the additional factors such as fun and motivation,
which have strengthened cooperation with VR and thus triggered a positive
learning experience among the participants.

Looking at the results of the last question LQ5 regarding learning support, it
can be seen that VR was rated as a tiring medium when it comes to learning new
content or concepts. On average, VR also performs significantly worse than BBB
and DVR. On the one hand, this can be explained by the nature of the VR
headsets, which were not considered very ergonomic for many participants
because they pressed on the nose and caused headaches during longer VR stays.
On the other hand, the natural form of input control in VR via movement,
gestures, etc. requires a higher expenditure of energy, which is not necessary
when using BBB and DVR.

In summary, with a view to the research question (RQ4), it can be concluded
that VR was not felt to be very conducive to learning support for the low
taxonomy level 1. On the other hand, the learning conduciveness of the higher
taxonomy levels, especially levels 4-6, was rated more positively than BBB and
DVR due to the diverse possibilities for interactivity in VR.

###  6.6 General Feedback and Group Work Logs

In addition to the questionnaires described above, there were various open
questions and group work protocols to collect general feedback from the
participants on the VR seminar. These are summarized below and linked to the
results described above. In the group work protocols, the findings from the
questionnaires regarding the perception of collaboration, interaction, and
learning support were confirmed. It became clear several times in the minutes
that the collaboration and the design of the VR presentation rooms were very
time-consuming and the students overall had the impression that things were
not running as efficiently and effectively as if they were creating or using a
classic PowerPoint slide. Nevertheless, the notes of the group work protocols
show that the participants had a lot of fun working together and that they
were able to find free rein in their creativity in order to be able to present
their ideas in a VR environment. The open questions showed that, despite the
very positive assessment of the interaction options in VR, it was difficult to
make precise entries or type in longer text sequences on the virtual keypad.
In addition, technical issues were mentioned where participants were thrown
out of the VR room. This shows that a stable and fast internet connection must
be ensured in order for the audio and graphic quality in VR to be adequate and
to work properly despite the synchronization of larger 3D data models. In
addition, some participants (2/6 of the VR participants) indicated that after
longer stays in VR they experienced a feeling of discomfort related to the
phenomenon of cyber sickness. In DVR, on the other hand, there were no
complaints in this regard. Despite the suggestions for improvement mentioned,
the majority opinion was that VR increases creativity and enables interactive
exchange meetings despite physical distance.

###  6.7 Student Feedback

Every semester, students are asked about their opinion of the respective
courses by a student event criticism (VKrit). For this purpose, the VKrit
provides a written course evaluation from the faculty with a standardized
evaluation form, which the students should fill out online. The VKrit is
carried out anonymously for each course. Although the evaluation criteria of
the VKrit are general and not tailored to specific events, some findings that
are relevant to the research questions can be taken into account. The seminar
did very well in the VKrit. Under the heading of overall impression, the
participants in the survey voted ”very good” (the highest grade on a scale
consisting of five items from very good to very bad). Eleven of a total of
twelve participants took part in the VKrit and provided their feedback. Many
of the participants (especially the VR participants, who were provided with VR
headsets) were positively impressed by the format and would like to see more
events using VR technology: ”It was very exciting to be almost exclusively in
VR. Especially since the content of the seminar is closely linked to VR, the
content and technology complemented each other very well. Even if face-to-face
events are possible again, there should still be events like these that use VR
as a medium.” Some participants gave feedback that VR meetings are a nice
alternative to video calls, but only for shorter meetings (up to 60 min) as
they become strenuous from a certain point in time (”VR headsets press against
the face”). Some of the participants wished that in the future all
participants should be provided with VR headsets so that they can experience
the event in full immersion.

##  7 Findings

This section summarizes various findings that were gained from the use of
virtual reality in the computer science seminar. Different perspectives such
as organization, collaboration, and interaction as well as learning success
are discussed.

###  7.1 Organization

Due to the interactive discussion and workshop formats (idea generation,
discussion, and presentation of ideas) at the organizational level and the
direct content connection of the offered seminar topics to VR, a promising
basis was given to assess the quality of VR as a medium for teaching and
learning in the university context. However, from an organizational
perspective, organizing the seminar using VR took significantly more time and
effort compared to conducting a seminar in person or using traditional
collaboration tools such as BBB or Zoom. This meant that participants had to
be familiarized with the VR technology, including hardware and software. In
addition, suitable virtual learning rooms had to be selected and designed for
the respective sessions, which often involved a lot of preparation time. The
reason for this is that the web interface of ”Spatial” only offers limited
possibilities for the precise design of the room and many details could often
only be set and refined when entering the VR room via a VR HMD. Even if in
some cases it is possible to reuse the learning rooms that have already been
created in future learning scenarios, a cost-benefit calculation is very
important in order to be able to precisely estimate the effort and thus be
able to weigh up whether the use of the VR medium is really worthwhile brings
additional profit. In addition to the design and implementation effort, which
was also mentioned by the participants for the creation of the final
presentation rooms, accessibility to the VR medium is an important aspect.
While some of the participants were familiar with VR and had prior experience
with Spatial, some participants were unfamiliar with some of Spatial’s
features. There was also the point that “Spatial” as a collaboration platform
revealed some weaknesses that only became apparent when used with a larger
group of participants. There were occasional breaks during the VR meetings,
which occurred due to the connection quality of individual participants or
because the VR space was getting larger and larger due to the objects created
(text, graphics, 3D models, etc.) so scalability was not given during
synchronization. Some of the features such as ”spatial” audio or the general
sound quality could still be improved and were often mentioned by the
participants in discussions or in the group work reports. In the course of the
seminar, there were also various updates to ”Spatial”, where changes were made
to some interaction concepts, which the participants had to get used to again.
It can therefore be observed that ”Spatial” and many other VR collaboration
tools are still in an experimental state and there are no technical standards
on which such VR collaboration tools are based.

###  7.2 Collaboration and Interaction

Through the use of VR, the cooperation between the participants could be
improved with regard to some factors. In particular, the results regarding the
perception of presence show that social presence was able to greatly promote
cooperation. On the one hand, this is related to the realistic representation
of the participants through 3D avatars and, on the other hand, to the
possibility of a realistic face-to-face interaction among the participants.
This enabled a collaborative environment that was felt by many participants to
be motivating for the learning process.

In addition to these positive insights, some negative insights made
collaboration and thus joint learning more difficult. For example,
collaboration in VR was rated poorer in terms of efficiency and effectiveness
compared to conventional collaboration tools. This was also confirmed as
feedback in the open questions and in the group work protocols.

###  7.3 Learning Success

The use of VR has shown that the teaching and learning experience can be
enhanced through the increased perception of presence. The results regarding
the perception of presence show, in particular, that VR has clear strengths
here compared to conventional collaboration tools and can therefore positively
promote the learning process, especially in group work. In addition, VR offers
the possibility of multi-modal learning via different media artifacts (text,
images, audio, video, 3D models, etc.), which appeal to different stimuli and
senses. This makes it possible to address different types of learners equally.
In addition, VR offers greater clarity and experience of learning content
through virtual learning rooms, which is not supported in this way in
conventional collaboration tools. These findings were also confirmed in the
assessment of learning support, particularly with regard to the higher
taxonomy levels. This illustrates the potential of VR to promote learning,
especially in the application, analysis, synthesis, and assessment of
knowledge.

In addition to the aforementioned opportunities, there are also some risks to
learning success. The use of VR involves additional costs for purchasing the
hardware. The results from Chapter 6 showed that an immersive experience in VR
cannot be replaced by an on-screen experience on a non-immersive device class.
In addition, some of the participants stated that VR caused dizziness and
headaches, especially in longer sessions of up to two hours. This problem,
also known as cybersickness ([17], [5]), occurs particularly in VR
applications where a lot of fast and sudden movements happen. In addition,
wearing a heavy HMD can also be seen as an obstacle to learning, since some of
the participants also stated that the VR headsets can be ergonomically
upgraded, that they press on the nose if worn for too long and cause a feeling
of discomfort, which has a negative effect on concentration. In addition to
these technical aspects, it was observed with regard to the learning success
in the organization of the event that there is a general lack of
teaching/learning concepts or a conceptual didactic basis for the use of VR.
In addition, most teachers and learners do not yet have the curricular media
competence for VR, so systematic training, introduction, planning, and use of
the VR medium are required. Finally, the conditions for success and hurdles
for the use of VR are briefly summarized in a tabular overview of Figure 8.

Figure 8: Conditions for success and hurdles for the use of VR

##  8 Conclusion and Outlook

Due to the constant digitization of learning and teaching media, educational
institutions such as universities are faced with the challenge of finding a
balance between the technical possibilities of the media and didactically
meaningful use. Virtual reality is said to have great potential to increase
the learning success of students and to positively enrich teaching. Since the
use of virtual reality in university education and in particular its effective
use in teaching has not yet been examined in depth and there are no concrete
pilot projects in this context, a computer science seminar based on VR as the
primary medium was carried out in this work. Plenary sessions were held for
workshops, discussions, and presentations in VR. In addition, there was also
group work that was carried out entirely in VR. Based on the course of the
seminar carried out here in this article, it was possible to compare the use
of VR with conventional collaboration tools such as BBB in terms of usability,
presence, collaboration, interaction, and learning success.

The results show that VR hardware and VR collaboration tools are not yet very
well known to many of the participants and that the VR collaboration tool used
cannot yet keep up with conventional, mature collaboration tools such as BBB
in terms of user-friendliness. Nevertheless, VR has the potential to increase
the clarity and experience of learning content through the increased
perception of presence. With regard to collaboration, it was found that VR
still has a lot of potential for improvement in terms of efficient and
effective collaboration, for example, to support the precise input and editing
of virtual rooms. Nevertheless, the possibilities of interaction in VR and in
particular the face-to-face interaction with other participants were perceived
as very positive, which is the greatest advantage over conventional
collaboration tools. In addition, the fun factor in connection with VR was
also strongly emphasized, as many participants found the use of VR to be
motivating for exchange and learning success. It should not go unmentioned
that there are limitations to this study. For example, the small number of VR
participants should be mentioned. In addition, there is the aspect that some
participants had previous experience in the field of VR, which could have had
a positive effect on the results of the evaluation. It should also be
mentioned that the course was based on ”Spatial” and that there are other VR
collaboration tools that could influence the analysis of the results. Seen
from this perspective, the results presented here should be regarded as
initial assessments. Overall, however, the knowledge gathered shows that the
strengths of VR in terms of interaction and increasing motivation to learn can
be used in a targeted manner to create benefits through the digital medium VR
even in the post-corona period. VR is therefore not only to be seen as a
medium that enables realistic interaction and communication for distance
teaching, but also as an additional enrichment of the known teaching formats.

As an outlook, it can be summarized briefly that the developments in hardware
and display technology are constantly advancing and give hope that the
required hardware will become more immersive, ergonomic, and affordable so
that an even broader use of this technology can be made possible. In addition,
the latest software advances in the field of VR show that more and more
context-sensitive learning aids are emerging that enable complex and dangerous
learning scenarios and enable users to transfer learned (psycho)motor skills
into reality. Due to the trend in the field of artificial intelligence, AI-
supported methods are increasingly being used in VR-based learning and
training environments in order to support the learning process even more
effectively and efficiently. In this context, it will be even more important
in the future to continuously analyze the effects of VR on teaching and
learning processes and to use this medium in a targeted manner.

## References

  * [1] Azuma, R.T.: A survey of augmented reality. Presence: teleoperators & virtual environments 6(4), 355–385 (1997)
  * [2] Bangor, A., Kortum, P., Miller, J.: Determining what individual sus scores mean: Adding an adjective rating scale. Journal of usability studies 4(3), 114–123 (2009)
  * [3] Biocca, F., Delaney, B.: Immersive virtual reality technology. Communication in the age of virtual reality 15(32), 10–5555 (1995)
  * [4] Bloom, B.S., Engelhart, M.D., Furst, E., Hill, W.H., Krathwohl, D.R.: Handbook i: cognitive domain. New York: David McKay (1956)
  * [5] Davis, S., Nesbitt, K., Nalivaiko, E.: A systematic review of cybersickness. In: Proceedings of the 2014 conference on interactive entertainment. pp. 1–9 (2014)
  * [6] Dickey, M.D.: Teaching in 3d: Pedagogical affordances and constraints of 3d virtual worlds for synchronous distance learning. Distance education 24(1), 105–121 (2003)
  * [7] Doerner, R., Broll, W., Grimm, P., Jung, B. (eds.): Virtual and Augmented Reality (VR/AR): Foundations and Methods of Extended Realities (XR). Springer International Publishing (2022). https://doi.org/10.1007/978-3-030-79062-2, https://doi.org/10.1007/978-3-030-79062-2
  * [8] Esteves, M., Fonseca, B., Morgado, L., Martins, P.: Improving teaching and learning of computer programming through the use of the second life virtual world. British Journal of Educational Technology 42(4), 624–637 (2011)
  * [9] García-Hernández, R.J., Kranzlmüller, D.: Nomad vr: Multiplatform virtual reality viewer for chemistry simulations. Computer Physics Communications 237, 230–237 (2019)
  * [10] Gottschalk, S., Yigitbas, E., Schmidt, E., Engels, G.: Model-based product configuration in augmented reality applications. In: Bernhaupt, R., Ardito, C., Sauer, S. (eds.) Human-Centered Software Engineering - 8th IFIP WG 13.2 International Working Conference, HCSE 2020, Eindhoven, The Netherlands, November 30 - December 2, 2020, Proceedings. Lecture Notes in Computer Science, vol. 12481, pp. 84–104. Springer (2020)
  * [11] Gottschalk, S., Yigitbas, E., Schmidt, E., Engels, G.: Proconar: A tool support for model-based AR product configuration. In: Bernhaupt, R., Ardito, C., Sauer, S. (eds.) Human-Centered Software Engineering - 8th IFIP WG 13.2 International Working Conference, HCSE 2020, Eindhoven, The Netherlands, November 30 - December 2, 2020, Proceedings. Lecture Notes in Computer Science, vol. 12481, pp. 207–215. Springer (2020)
  * [12] Hagge, P.: Student perceptions of semester-long in-class virtual reality: Effectively using “google earth vr” in a higher education classroom. Journal of Geography in Higher Education 45(3), 342–360 (2021)
  * [13] Huber, L.: Scholarship of teaching and learning: Konzept, geschichte, formen, entwicklungsaufgaben. Forschendes Lehren im eigenen Fach. Scholarship of teaching and learning in Beispielen pp. 19–36 (2014)
  * [14] Jovanovikj, I., Yigitbas, E., Sauer, S., Engels, G.: Augmented and virtual reality object repository for rapid prototyping. In: Bernhaupt, R., Ardito, C., Sauer, S. (eds.) Human-Centered Software Engineering - 8th IFIP WG 13.2 International Working Conference, HCSE 2020, Eindhoven, The Netherlands, November 30 - December 2, 2020, Proceedings. Lecture Notes in Computer Science, vol. 12481, pp. 216–224. Springer (2020)
  * [15] Krings, S., Yigitbas, E., Jovanovikj, I., Sauer, S., Engels, G.: Development framework for context-aware augmented reality applications. In: Bowen, J., Vanderdonckt, J., Winckler, M. (eds.) EICS ’20: ACM SIGCHI Symposium on Engineering Interactive Computing Systems, Sophia Antipolis, France, June 23-26, 2020. pp. 9:1–9:6. ACM (2020)
  * [16] Krings, S.C., Yigitbas, E., Biermeier, K., Engels, G.: Design and evaluation of ar-assisted end-user robot path planning strategies. In: Winckler, M., Quigley, A. (eds.) EICS ’22: ACM SIGCHI Symposium on Engineering Interactive Computing Systems, Sophia Antipolis, France, June 21 - 24, 2022, Companion Volume. pp. 14–18. ACM (2022)
  * [17] LaViola Jr, J.J.: A discussion of cybersickness in virtual environments. ACM Sigchi Bulletin 32(1), 47–56 (2000)
  * [18] Lewis, J.R.: Ibm computer usability satisfaction questionnaires: psychometric evaluation and instructions for use. International Journal of Human-Computer Interaction 7(1), 57–78 (1995)
  * [19] Pirker, J., Dengel, A., Holly, M., Safikhani, S.: Virtual reality in computer science education: A systematic review. In: 26th ACM symposium on virtual reality software and technology. pp. 1–8 (2020)
  * [20] Radianti, J., Majchrzak, T.A., Fromm, J., Wohlgenannt, I.: A systematic review of immersive virtual reality applications for higher education: Design elements, lessons learned, and research agenda. Computers & Education 147, 103778 (2020)
  * [21] Rogers, S.: Virtual reality: The learning aid of the 21st century. Secondary Virtual reality: The learning aid of the 21st century (2019)
  * [22] Ryan, M.L.: Narrative as virtual reality 2: Revisiting immersion and interactivity in literature and electronic media. JHU press (2015)
  * [23] Slater, M., Wilbur, S.: A framework for immersive virtual environments (five): Speculations on the role of presence in virtual environments. Presence: Teleoperators & Virtual Environments 6(6), 603–616 (1997)
  * [24] Steuer, J.: Defining virtual reality: dimensions determining telepresence. communication in the age of virtual reality, 33–56 (1993)
  * [25] Walsh, K.R., Pawlowski, S.D.: Virtual reality: A technology in need of is research. Communications of the Association for Information Systems 8(1), 20 (2002)
  * [26] Witmer, B.G., Singer, M.J.: Measuring presence in virtual environments: A presence questionnaire. Presence 7(3), 225–240 (1998)
  * [27] Yigitbas, E., Gorissen, S., Weidmann, N., Engels, G.: Collaborative software modeling in virtual reality. In: 24th International Conference on Model Driven Engineering Languages and Systems, MODELS 2021, Fukuoka, Japan, October 10-15, 2021. pp. 261–272. IEEE (2021)
  * [28] Yigitbas, E., Gorissen, S., Weidmann, N., Engels, G.: Design and evaluation of a collaborative uml modeling environment in virtual reality. Software and Systems Modeling pp. 1–29 (2022)
  * [29] Yigitbas, E., Gottschalk, S., Nowosad, A., Engels, G.: Development and evaluation of a collaborative stock trading environment in virtual reality (2022)
  * [30] Yigitbas, E., Heindörfer, J., Engels, G.: A context-aware virtual reality first aid training application. In: Alt, F., Bulling, A., Döring, T. (eds.) Proc. of Mensch und Computer 2019. pp. 885–888. GI / ACM (2019)
  * [31] Yigitbas, E., Jovanovikj, I., Engels, G.: Simplifying robot programming using augmented reality and end-user development. In: Ardito, C., Lanzilotti, R., Malizia, A., Petrie, H., Piccinno, A., Desolda, G., Inkpen, K. (eds.) Human-Computer Interaction - INTERACT 2021 - 18th IFIP TC 13 International Conference, Bari, Italy, August 30 - September 3, 2021, Proceedings, Part I. Lecture Notes in Computer Science, vol. 12932, pp. 631–651. Springer (2021)
  * [32] Yigitbas, E., Jovanovikj, I., Sauer, S., Engels, G.: On the development of context-aware augmented reality applications. In: Abdelnour-Nocera, J.L., Parmaxi, A., Winckler, M., Loizides, F., Ardito, C., Bhutkar, G., Dannenmann, P. (eds.) Beyond Interactions - INTERACT 2019 IFIP TC 13 Workshops, Paphos, Cyprus, September 2-6, 2019, Revised Selected Papers. Lecture Notes in Computer Science, vol. 11930, pp. 107–120. Springer (2019)
  * [33] Yigitbas, E., Jovanovikj, I., Scholand, J., Engels, G.: VR training for warehouse management. In: Teather, R.J., Joslin, C., Stuerzlinger, W., Figueroa, P., Hu, Y., Batmaz, A.U., Lee, W., Ortega, F.R. (eds.) VRST ’20: 26th ACM Symposium on Virtual Reality Software and Technology. pp. 78:1–78:3. ACM (2020)
  * [34] Yigitbas, E., Karakaya, K., Jovanovikj, I., Engels, G.: Enhancing human-in-the-loop adaptive systems through digital twins and VR interfaces. In: 16th International Symposium on Software Engineering for Adaptive and Self-Managing Systems, SEAMS@ICSE 2021, Madrid, Spain, May 18-24, 2021. pp. 30–40. IEEE (2021)
  * [35] Yigitbas, E., Klauke, J., Gottschalk, S., Engels, G.: VREUD - an end-user development tool to simplify the creation of interactive VR scenes. In: Harms, K.J., Cunha, J., Oney, S., Kelleher, C. (eds.) IEEE Symposium on Visual Languages and Human-Centric Computing, VL/HCC 2021, St Louis, MO, USA, October 10-13, 2021. pp. 1–10. IEEE (2021)
  * [36] Yigitbas, E., Krois, S., Renzelmann, T., Engels, G.: Technical report: Comparative evaluation of ar-based, vr-based, and traditional basic life support training. arXiv preprint arXiv:2207.03306 (2022)
  * [37] Yigitbas, E., Sauer, S., Engels, G.: Using augmented reality for enhancing planning and measurements in the scaffolding business. In: EICS ’21: ACM SIGCHI Symposium on Engineering Interactive Computing Systems, virtual, June 8-11, 2021. ACM (2021), https://doi.org/10.1145/3459926.3464747
  * [38] Yigitbas, E., Schmidt, M., Bucchiarone, A., Gottschalk, S., Engels, G.: Gamification-based UML learning environment in virtual reality. In: Kühn, T., Sousa, V. (eds.) Proceedings of the 25th International Conference on Model Driven Engineering Languages and Systems: Companion Proceedings, MODELS 2022, Montreal, Quebec, Canada, October 23-28, 2022. pp. 27–31. ACM (2022)
  * [39] Yigitbas, E., Tejedor, C.B., Engels, G.: Experiencing and programming the ENIAC in VR. In: Alt, F., Schneegass, S., Hornecker, E. (eds.) Mensch und Computer 2020. pp. 505–506. ACM (2020)
  * [40] Zheng, L., Xie, T., Liu, G.: Affordances of virtual reality for collaborative learning. In: 2018 International Joint Conference on Information, Media and Engineering (ICIME). pp. 6–10. IEEE (2018)