# Towards Immersive Collaborative Sensemaking

Ying Yang<EMAIL_ADDRESS>Monash UniversityWellington Road,
ClaytonMelbourneVictoriaAustralia3800 ,  Tim Dwyer  Monash
UniversityMelbourneAustralia<EMAIL_ADDRESS>,  Michael Wybrow  Monash
UniversityMelbourneAustralia<EMAIL_ADDRESS>,  Benjamin Lee  Monash
UniversityMelbourneAustralia<EMAIL_ADDRESS>,  Maxime Cordeil  The
University of QueenslandSt LuciaBrisbaneQueenslandAustralia4072
<EMAIL_ADDRESS>,  Mark Billinghurst  University of South AustraliaMawson
Lakes BlvdAdelaideSouth AustraliaAustralia5095<EMAIL_ADDRESS>and  Bruce H. Thomas  University of South AustraliaMawson Lakes
BlvdAdelaideSouth AustraliaAustralia5095<EMAIL_ADDRESS>
(2022)

###### Abstract.

When collaborating face-to-face, people commonly use the surfaces and spaces
around them to perform sensemaking tasks, such as spatially organising
documents, notes or images. However, when people collaborate remotely using
desktop interfaces they no longer feel like they are sharing the same space.
This limitation may be overcome through collaboration in immersive
environments, which simulate the physical in-person experience. In this paper,
we report on a between-groups study comparing collaborations on image
organisation tasks, in an immersive Virtual Reality (VR) environment to more
conventional desktop conferencing. Collecting data from 40 subjects in groups
of four, we measured task performance, user behaviours, collaboration
engagement and awareness. Overall, the VR and desktop interface resulted in
similar speed, accuracy and social presence rating, but we observed more
conversations and interaction with objects, and more equal contributions to
the interaction from participants within groups in VR. We also identified
differences in coordination and collaborative awareness behaviours between VR
and desktop platforms. We report on a set of systematic measures for assessing
VR collaborative experience and a new analysis tool that we have developed to
capture user behaviours in collaborative setting. Finally, we provide design
considerations and directions for future work.

Collaborative Sensemaking, Virtual Reality

††ccs: Human-centered computing User studies††ccs: Human-centered computing
Interaction devices††ccs: Human-centered computing Collaborative and social
computing systems and tools††ccs: Human-centered computing Empirical studies
in collaborative and social computing††copyright: acmlicensed††price:
15.00††doi: 10.1145/3567741††journalyear: 2022††submissionid: iss22main-
id9657-p††journal: PACMHCI††journalvolume: 6††journalnumber: ISS††article:
588††publicationmonth: 12

##  1\. Introduction

Remote collaboration continues to grow in popularity, particularly since the
COVID-19 pandemic. Using a wide range of technologies people are engaging in a
greater variety of remote collaboration tasks than ever before. A key example
of a task that needs better support for remote collaboration is collaborative
sensemaking, i.e., working together to make sense of information (Paul and
Reddy, 2010). Standard practice for collaborative sensemaking is video
conferencing (e.g., Zoom, https://zoom.us/) and online tools like shared
whiteboards (e.g., Miro, https://miro.com/), but these do not provide a strong
sense of spatial awareness or embodied presence. Previous research has claimed
that Virtual Reality (VR) can overcome these limitations (Witmer and Singer,
1998). The emerging field of Collaborative Immersive Analytics (CIA)
(Billinghurst et al., 2018) explores how VR can be used for analysing data in
a group setting. For example, Leigh et al. (1999) describe early work in using
VR for collaborative visualisation, and Donalek et al. (2014) provide a
summary of collaborative data visualisation using VR. Marai et al. (2016)
reviewed several Immersive Analytics projects by groups using a large
cylindrical projection environment where people reported on getting more work
done in two days than in six months of desktop collaboration tools.

VR potentially offers an embodied experience closer to in-person collaboration
than is afforded by remote collaboration using a desktop interface. However,
although CIA systems are promising, there has been little research that
compares collaborative sensemaking in VR to a desktop interface and—to the
best of our knowledge—none with groups of more than three people at a time.
Studying collaboration among groups larger than three is important because it
enables different dynamics than with smaller groups, e.g., a group of four may
subdivide into two teams of two in order to discuss separately and work in
parallel.

Our research explores whether shared immersive environments can better support
group sensemaking tasks compared to a desktop environment. Our study asked
groups of four people to arrange and cluster images using remote collaboration
via either a VR or desktop interface. Compared to a desktop condition, we
found that teams in VR had similar task effectiveness but that VR provided
several benefits, with the teams on each platform engaging in significantly
different behaviours in terms of interaction, conversation, coordination and
awareness. From our observation of these different behaviours we derive
suggestions for the design of future collaborative VR applications in various
aspects, such as communication, notification, navigation, environment and
virtual elements, and provide directions for future research.

Our study focused on spatial sensemaking tasks, which are fundamental
sensemaking behaviour that uses space to make sense of information and manage
data, and is a way to externalise thoughts and reduce cognitive affordances
(Robertson et al., 1998; Andrews et al., 2010). Our results showed that, when
collaborating with others in VR, people employ various space-use strategies
rather than preferring to use the space around them when performed tasks
individually, as found in previous research (Batch et al., 2019; Satriadi et
al., 2020; Lisle et al., 2021). Interestingly, we still identified common
patterns from these different space usages. These findings can inform future
researchers of how the space could be used in collaborative sensemaking,
especially when designing and providing automatic spacial layout for
organising information and data.

We also present a new tool to capture and analyse group behaviour in a
collaborative sensemaking task. Past remote collaboration research has found
that behavioural measures are often a better measure of technology impact than
performance measures (McCarthy and Monk, 1994; Billinghurst et al., 2003).
However, behaviour of groups of more than two people can be complex and
difficult to analyse. We have created new techniques for analysing multiway
communication and interactions between group members and objects in the
environment.

In summary, our paper makes the following novel contributions: (1) the first
comparative study of a four-person collaborative sensemaking task between a VR
condition and a desktop interface condition; (2) results suggesting benefits
to collaborative sensemaking in immersive environments over desktop
environments; (3) novel systematic measures for understanding and analysing
the behaviours of larger groups (four people or more) in an immerse
collaborative sensemaking task; and (4) design considerations for immersive
collaborative sensemaking applications and directions for future works.

In the next section, we survey past work informing our research. In Section 3,
we describe our group sensemaking user study and the measures. In Sections 4,
5 and 6, we present the study outcomes and discuss the results. Finally, we
summarise and reflect on findings, limitations and directions for future work.

##  2\. Related Work

###  2.1. Immersive Spatial Sensemaking

Commercial immersive collaborative techniques and applications are emerging to
support sensemaking, such as Microsoft’s Mesh (https://www.microsoft.com/en-
us/mesh) or Meta’s Workrooms (https://www.oculus.com/workrooms/), which allow
multiple people to join and perform tasks such as presenting data or
brainstorming. While there is momentum in the immersive industry to provide
collaboration platforms, it is unclear however, how those tools can support
sensemaking. Spatial sensemaking uses spatial organisation to manage data to
support understandability and memorability, which has been researched on
various platforms (Barsalou, 1983; Robertson et al., 1998; Cockburn and
McKenzie, 2002; Andrews et al., 2010; Endert et al., 2012; Wenskovitch and
North, 2020). Immersive techniques extend traditional desktop interfaces into
immersive environments, and enable use of embodied representations and
intuitive interaction to improve cognition and facilitate sensemaking
(Chandler et al., 2015; Marriott et al., 2018).

Recent research has studied how individuals and groups perform spatial
sensemaking on data and documents in immersive spaces. For example, Batch et
al. (2019) found that for immersive data analytics, data scientists tended to
use the space in front of them to explore different visualisations while they
used the space around them to present their findings to an audience.
Similarly, other research also showed that people tended to layout maps and
documents spherically around themselves (Satriadi et al., 2020; Lisle et al.,
2021). Liu et al. (2020) found that semi-circular layouts in 3D space improve
visual search compared to flat layout. However, all of this research only
examined single-user scenarios.

Fewer researchers have explored collaborative sensemaking in immersive
analytics. For example, Lee et al. (Lee et al., 2020) evaluated a CIA
prototype with three simultaneous users, and found that people still organise
visualisations in curved layouts around them, but also placed them in flat
arrangements to facilitate group discussion and collaboration. Most recently,
Luo et al. (2021, 2022) conducted an empirical study to investigate the impact
of physical surroundings on spatial arrangement with paired users in Augmented
Reality (AR). They found that some users produced cylindrical layouts around
them while others used the walls and physical furniture as separation for
different clusters.

The main results regarding the use of space for sensemaking in immersive
environments are that people tend to arrange information around them in a
semi-circular shape or a flat manner. More work is needed to understand how
users collaborate and interact with data items, and how this compares to
traditional collaborative 2D desktop systems.

###  2.2. Collaboration Proximity and Territoriality

The collaborative sensemaking process is affected by two spatial factors: the
(physical) proximity of collaborators to one another, and the use of
territories to manage resources in the workspace. Hawkey et al. (Hawkey et
al., 2005) found that collaboration is more effective when groups physically
stand close to each other when using a large vertical display. In contrast,
Prouzeau et al. (Prouzeau et al., 2017) found that participants using separate
desktops performed a collaborative path-finding task faster than when using a
shared large vertical display. However, they note that the latter produces
consistently higher quality results, due to there being more verbal
communication in the shared environment.

How groups manage resources in the common workspace has been shown to
influence collaboration styles (Scott and Carpendale, 2010; Isenberg et al.,
2011; Chung et al., 2014). Most notably are the notions of personal, shared
and storage territories (Scott et al., 2004). Bradel et al. (Bradel et al.,
2013) found that groups who predominantly used shared territories on a large
vertical display communicated more than those who mainly used personal
territories. The partitioning of the shared workspace into territories is
natural and rarely requires verbal communication (Tse et al., 2004). Personal
territories are commonly directly in front of users (Tang, 1991; Kruger et
al., 2004; Liu et al., 2016), and the shift between personal and shared
territories can easily shift as users physically move around the workspace
(Tang et al., 2006; Jakobsen and Hornbæk, 2014).

Previous research shows that personal and shared territories affect
collaboration and communication. It is important to explore how such
collaboration proximity and territory pattern differ in immersive 3D and 2D
flat shared space.

###  2.3. Awareness and Presence in Collaborative Workspaces

It is common to perform sensemaking tasks with flat screens, mouse and
keyboard setups. For example, using a combination of collaborative production
and communication tools such as Miro or Zoom. These provide a shared 2D
virtual space for the joint manipulation of virtual objects (e.g., documents
and images) (Durlach and Slater, 2000), while use of audio and video enables
communication, awareness and social presence (Gutwin et al., 2004; Sallnäs,
2005; Bente et al., 2008). Workspace awareness (Gutwin and Greenberg, 1996) is
key to recognising others’ behaviours in a cooperative environment. While
collaborators’ activities are not difficult to realise in a co-located
situation, it is a challenge to maintain awareness via remote collaboration
(Gutwin et al., 2004). Social presence, or the sense of “being with another”
(Biocca et al., 2003), is desirable as it can enable collaborators to freely
converse and seek help (Weinel et al., 2011).

Early research has investigated how to increase awareness and presence by
showing others’ cursors and viewports on a desktop interface such as
Telepointers (Greenberg et al., 1996). Hauber et al. (Hauber et al., 2005)
compared physical face-to-face meetings, 2D desktop, and non-stereoscopic 3D
desktop setups for a ranking task. They found that while the physical meeting
was superior to the computer-mediated collaborative tools, the non-
stereoscopic 3D desktop improved awareness and presence over the 2D desktop
setup. Later, embodied virtual representations have been employed in
distributed tabletop and big display applications (Tang et al., 2007;
Tuddenham and Robinson, 2009). Zillner et al. (Zillner et al., 2014) presented
3D whole-body telepresence of remote collaborators on a digital whiteboard,
and compared this technique to a 2D representation and co-collocated
collaboration. Their findings showed that 3D embodiment representation greatly
improved collaboration effectiveness. Similarly, Higuch et al. (Higuchi et
al., 2015) employed 3D-processed immersive telepresence over a digital
whiteboard and found that, compared to traditional video conferencing, 3D
telepresence provided better co-presence and a more enjoyable environment.
Pejsa et al. (Pejsa et al., 2016) developed an AR technique that projected a
remote collaborators’ virtual replication onto the physical environment. They
found that this life-sized telepresence provided a stronger sense of being
together than a Skype video conferencing.

In stereoscopic immersive environments, collaboration and presence is usually
enhanced by the use of embodied avatars (Gerhard et al., 2004; Piumsomboon et
al., 2018; Ens et al., 2019). While realistic full body avatars are favoured
for collaboration, using non-realistic avatars still produces a good sense of
presence and awareness (Steed et al., 1999; Heidicker et al., 2017; Yoon et
al., 2019; Lee et al., 2020). Notably, Cordeil et al. (Cordeil et al., 2016),
in a collaborative visualisation task, found no difference in communication
and perceived sense of presence in a CAVE environment (where participants see
each other physically) compared to a pair of connected VR headsets. However,
direct comparison of collaboration in immersive VR to using a desktop
interface remains to be addressed, and research especially needs to be done on
whether users collaborate, and perceive awareness and presence differently in
these two scenarios.

###  2.4. Collaboration Measures

While previous research has studied and measured how people use online
collaborative tools for group work and educational purposes (e.g., (Roberts et
al., 2006; Hernández-Sellés et al., 2019; Bulu, 2012)), and has focused on
modelling collaboration behaviours (Gutwin and Greenberg, 2000; Baker et al.,
2002; Janssen et al., 2007; Meier et al., 2007; Wang et al., 2017; Chen et
al., 2018), little research has investigated the comparison between non-
immersive and immersive platforms for sensemaking.

Some research has investigated collaboration in a hybrid system with one
person in VR and two on desktop interfaces (Tromp et al., 1998), and a hybrid
of VR, AR, desktop and wall display with three users (Cavallo et al., 2019),
but none of these made a comparison between different interfaces and
quantified user behaviours and collaboration. Radu and Schneider (Radu and
Schneider, 2019) proposed an AR system for physics learning in pairs and
compared different levels of virtual involvements, from non-AR to including
all AR elements. However, their measures mainly focused on learning outcome,
user attitude and oral communications. Billinghurst et al. (Billinghurst et
al., 2003) conducted series experiments to compare paired users in face-to-
face, projection screen and AR settings, and proposed a set of measures
covering oral and gestural communications, while the quantification of user
interactions with virtual contents during collaboration was still missing.
This previous research shows that user experience and behaviours have yet to
be systematically measured in immersive environment collaborations, and how
different these behaviours could be in VR and with a desktop interface is
still unknown.

###  2.5. Summary

Taken together, to the best of our knowledge, there has been no comparative
study using non-immersive and immersive environments for sensemaking in groups
of four, and systematically measuring the similarity and difference of user
interactions, collaborations, communications and space usage in these
environments. Thus our research is novel, and addresses an important research
gap. In the next section we describe the user study developed to explore this
space.

##  3\. User Study

Our study aimed to explore how people use a 3D immersive space for
collaborative sensemaking when organising images. We wanted to investigate the
advantages and disadvantages of collaborating and communicating in a 3D
immersive environment compared with collaborating via a 2D desktop tool and
communicating using Zoom.

###  3.1. Study Design

We set up two conditions: (1) a remote VR collaboration system which allows
participants to jointly arrange images (VR), and (2) a remote desktop
collaboration system that allows participants to arrange images together while
on Zoom (Desktop). We used a between-subjects design for the study where each
group of four completed the study with one of the conditions. In the VR
condition, participants were each in their own room with at least $4\times 4$
metres of space, sound isolated from one another but in the same building. In
the Desktop condition (conducted during a strict COVID-19 lockdown),
participants joined from home using their own computer.

###  3.2. Tasks and Data Set

To investigate sensemaking for image organisation, we designed tasks requiring
participants to arrange a set of images into groups around a set of affective
labels, i.e., emotion terms such as “glad”, “distressed”, etc. In both Desktop
and VR, images and labels were on moveable “cards” that could be freely placed
within the environment. Participants were required to discuss their feelings
about the images and reach a consensus to place the images near the labels
with which they felt the images had the strongest correspondence.

We used the Open Affective Standardized Image Set (OASIS) (Kurdi et al., 2017)
for the tasks, which contains 900 images. Each image in the data set has two
dimensions: valence (the degree of pleasantness or unpleasantness) and arousal
(the degree of excitement or calm), where the images’ values in these
dimensions have been rated by 822 participants. These two dimensions of
emotion served as the ground truth for our image grouping task to measure
performance accuracy. We designed four tasks and varied the number of image
groups and images for each task. The first task, which had the least groups
and fewest images, was considered as a trial task to allow participants to get
used to collaborating in the shared environment used for their condition. The
number of groups in the other tasks ranged from 4 to 7, and the number of
images in each group ranged from 3 to 8. We plotted the OASIS image
distribution in a valence-arousal matrix to select the images for each task
and tried to maximise the distance between the groups and minimise the
distance between the images within a group.

The emotion terms used as group labels were selected from the affect model
from Russell (1980), which organises the terms by valence-arousal. We matched
the distribution of the images in the valence-arousal matrix to the emotion
word coordinates to select the terms for representing each image group. To
avoid a learning effect between tasks, we selected different terms and images
for each task.

Figure 1. Setup of VR and Desktop conditions. Participants were presented as
embodied avatars in VR. On the desktop they see others’ mouse cursors in the
platform while communicating via conferencing video calls (omitted in the
figure). Name tags were visible to participants but are hidden in the figure
for anonymity. Individual images show sample layouts created by participants.
For Desktop: (a) Rows, (b) Columns, (c) Clusters in corners, and (d) Clusters
around canvas border. For VR: (e) Rows (f) Columns, (g) Clusters on a 2D
“panel” (h) Clusters on multiple 2D “panels”, (i) spherical Clusters, and (j)
Convex hull.

figure description

###  3.3. VR Condition

Our VR condition allows participants to place the image and label cards
hanging freely within a 3D space. The VR system was implemented with the
Unity3D engine. We used the standalone Oculus Quest 2 VR headsets and
controllers in the study. The headsets are self-contained and require no cord
connection to a PC, allowing participants to freely move and rotate their
bodies in the space. The controllers have a trigger that allows participants
to naturally grasp and manipulate objects with either hand. The locations used
in the VR condition were empty rooms with a cleared area of $4\times
4$$\text{\,}\frac{\mathrm{m}}{}$ within which each participant had complete
freedom of movement.

Initially, the image cards were distributed evenly in the room at waist-height
facing up, with randomly generated $x$\- and $z$-axis positions and 360°
rotations around their $y$-axis. The same approach was used to distribute the
label cards. Participants were randomly distributed in the space when they
started the tasks (by walking to press a “START” button displayed at a random
location within the space).

To find the most suitable size for images and labels we conducted a pilot
study with four participants with varying image and text sizes. Based on
participant feedback, our final study design used image and label cards
measuring $20\times 20$ $\text{\,}\frac{\mathrm{cm}}{}$. The label font size
was chosen such that the longest label width filled the card. We used
“LiberationSans SDF” in the TextMeshPro package as the font style and used the
most common yellow colour (R: 255, G: 237, B: 45) as the background for labels
(such that they resembled ‘post-it notes’).

The controllers were represented as virtual hands in the environment to give a
sense of self-presence to participants. To allow participants to focus on the
spatial arrangement, we kept interaction as simple as possible. Participants
could freely move or rotate objects with a simple grip interaction activated
with the controller trigger. When the trigger was pressed, participants would
see their virtual hands perform a pinch gesture, and visual feedback of a
thick blue outline was provided for objects under manipulation. Since we were
most interested in natural “embodied” interaction, we did not support ranged
interactions.

Participants could see their collaborators as embodied avatars in the system.
We used the Oculus Avatar SDK to implement the avatars, which show head, upper
body and hands (as seen in Figure 1). To distinguish different collaborators,
the head and hands were given unique colours, and name tags were displayed
above the avatar’s head and hands, always facing the viewer’s camera. To
support the awareness of collaborators’ activities, we provided head gaze by
casting a pointing ray from the centre of the avatar’s two eyes. Whenever a
pointing ray hit an image or label card, a reticle with a name tag was shown
at the hit point.

To achieve remote collaboration and communication in the VR system, we used
the Photon Unity Networking framework to synchronise the movement of avatars
and objects. The Photon Voice package was used to transmit participants’ voice
through the network. We implemented 3D spatial sound for the audio of the VR
system, so participants could hear the voices of nearby collaborators as
loader than the those further away in the shared virtual space. We used a
customised logarithmic roll-off curve for the spatial sound, with maximum
audible distance of 4 meters.

###  3.4. Desktop Condition

For the Desktop condition, we developed a system with the functionally of the
VR system, inspired by the Miro online whiteboard, but with the minimal
functionality required for the study tasks. The desktop system was implemented
using Unity3D and run on Windows or macOS. The prototype presentation used a
windowed mode set to a $2560\times 1600$ pixels resolution as the default
window size. The application window was resizable. We asked participants to
maximise the application to fit their screen. A square 2D canvas was displayed
on the centre of the application and used the full window length.

Like in a Miro board, participants in the Desktop condition could see the
positions of their collaborators’ mouse cursors (see Figure 1). Visualising
users’ mouse cursors in shared space is a conventional approach to support
group awareness (Engelbart and English, 1968). This cursor representation is
the same as the mouse cursor used in typical desktop systems, familiar for
computer users. Also, it is simple but effective enough to share
collaborators’ location and movement on the limited desktop screen (Greenberg
et al., 1996). Each mouse cursor had a unique colour and a name tag above it.
The movement of mouse cursors and the image and label cards were synchronised
using the Photon Unity Networking framework.

We used the same approach as the VR condition to initially distribute the
images and label cards on the canvas. The $x$\- and $z$-axis positions of the
objects in the VR condition were projected to the $x$\- and $y$-axis positions
on the desktop canvas. Since people look at the screen from a fixed
orientation, the rotation of the objects in this condition is fixed to the
orientation of the screen. We conducted a pilot study and found that the most
suitable scale of the (square) image and label cards as a fraction of canvas
width was 0.067. To avoid extra navigation on canvas, since the cards can be
clearly seen with a normal sized 23 inch monitor, we did not provide zoom
in/out functions in the application. We used the same font style, font size
relative to card size and label note background colour as the VR condition.

Participants used a mouse to interact with the desktop application. The only
interaction required in the study is moving objects. They could move image or
label cards by left-clicking on them and dragging with the mouse. As with the
VR condition, during interaction the image or label card was highlighted with
a blue outline.

During the study, participants were connected via Zoom video calls, and their
voices were transmitted via Zoom, so they could easily speak to one another.
As we want to compare embodied VR and traditional desktop environment, as a
convention, the loudness of users voice was naturally controlled by the
distance from a user to the camera and microphone.

###  3.5. Participants

We recruited 40 participants (13 female, 27 male), resulting in 20 people (5
groups) for each condition. We balanced the age and gender for the VR
condition (6 female and 14 male, aged 18-34, M = 26.5, SD = 4.2) and the
Desktop condition (7 female and 13 male, aged 18-44, M = 25.8, SD = 5.3). In
the VR condition, 20% of the participants self-reported no experience in
VR/AR, 55% reported having 10 hours or less of experience, and 25% reported
having more than 10 hours experience. Only three participants reported using
social VR/AR systems before. The participants in the Desktop condition were
asked about previous experience with video conferencing and desktop
collaborative tools (e.g., Miro, Google Docs). Over 90% of participants
reported frequent usage of video conferencing, and 55% reported having used
desktop collaborative tools. We offered the participants gift cards for their
participation. The experiment lasted 45 minutes.

###  3.6. Procedure

Each condition followed the same general procedure. In the VR condition, to
prevent hearing each other, the participants were in separate rooms, each of
which had an empty space of at least $4\times 4$
$\text{\,}\frac{\mathrm{m}}{}$. The participants communicated via the VR
system. They were supervised physically by two experimenters and also
monitored via Zoom video calls set up in each room. For the Desktop condition,
due to COVID-19, the study was run completely online. The participants
communicated via Zoom calls, supervised by the experimenter.

The participants needed to enter their names to log into the VR/Desktop
system. After login, the participants completed a training protocol
individually before meeting their collaborators. The training guided the
participants to practise grabbing, moving and placing objects in the system.
In the VR condition, the objects were placed at particular positions to force
participants to walk around and become familiar with the environment. After
training, the participants were shown task instructions. When all four
participants finished reading the instructions, they joined the meeting room
together—the VR/Desktop environment where they would complete the tasks. In
the meeting room, the participants were asked to introduce themselves and do
an ice-breaker activity: sharing which animal they wanted to be if they could.
When the participants finished the ice-breaker activity and were ready for the
tasks, they asked the experimenter to show them the first task. The
experimental tasks were fully controlled by the experimenter, who used a
server application to control the progress of the study and communicate with
the participants. When the participants finished each task, they needed to ask
the experimenter to change the task for them.

In the first task of one group, the participants finished the task without any
discussion or collaboration, and so the experimenter prompted them to review
the arrangement of the images. All other groups naturally discussed and
collaborated on all tasks.

After the study, all participants were asked to complete a questionnaire
regarding demographic information and their previous experience with VR, video
conferencing systems and collaboration tools. They were also asked to complete
a post-experiment questionnaire, answering short-answer questions about their
strategies for image arrangement and collaboration during the study and
overall feedback, and rating their perception of Social Presence on a 7-point
Likert scale from strongly disagree to strongly agree (Harms and Biocca,
2004).

###  3.7. Data Collection

The server application used by the experimenter also collected the study log
data: time stamps of actions, positions of image and label cards, and
participants’ head and hands/mice positions. In addition, we developed an
audio recording system to record participants’ voice. Four audio recording
systems were run during the study, each followed one participant to record
their audio.

The Desktop/VR systems that were run by the participants were connected to a
Photon server used by the experimenter on a desktop PC, with high-speed
Ethernet to minimise the latency of the networking. The server application
includes the ability to replay the visuals and synced audio from experimental
sessions to facilitate data analysis.

###  3.8. Measures and Research Questions

Our experimental design is exploratory rather than using formal hypothesis
testing. Therefore, we collected as much data as possible to support a variety
of post-hoc analysis. In the following, we describe the research questions we
addressed and the measures that we analyse to do so. The definitions of
measure terms refer to Appendix A.1. We also make note, where applicable, of
some of our prior expectations.

Regarding task performance, we measured accuracy and completion time. We
explored activities in terms of interaction, conversation and coordination,
and analysed engagement with respect to awareness and social presence. We also
investigated spatial usage and territories.

In both conditions, the sources of our analyses came from the system logs,
playback of log files, audio recordings and post-experiment questionnaires.
Firstly, we computed the occurrence of a number of objective measures from the
playback of log files and audio recordings to identify conversation contents
(e.g., planning or greeting), conversation types (task and social),
conversation participants (initial participant and responding participants)
and conversation targets (label and image). As these measures were objective
rather than subjective, one author went through the playback manually to
identify behaviour types of interest. To increase the rigor of these results,
the author took another pass and did a double-check. Then we went through the
playback and identified criteria for each behaviour, and developed a program
to count the occurrence of these behaviours from logs. Some example behaviours
are: for interaction (grab and regroup), for coordination (monitoring,
protection, etc.), and for awareness (no response, shaking, etc.). We checked
the results captured by the program against a sample of the playback to make
sure the results matched the behaviours present in the playback. For the
samples of results we checked, the program accurately counted all occurrences
of the behaviours. The definitions and criteria to identify these behaviours
are discussed below.

####  3.8.1. Task

RQ1: Is there a difference in sensemaking task accuracy and completion time
across VR and Desktop? We did not expect that the conditions would affect the
task accuracy. We believed the participants in VR would take more time than
the Desktop condition to complete the tasks as they needed to walk physically
in the environment.

####  3.8.2. Activity

Interaction. RQ2: Are there differences in frequency, duration and type of
people’s interactions with objects across VR and Desktop? From our logs we can
identify precisely which participants interacted with which objects, where
they grabbed and moved those objects to, how many times, and how much time
they spent moving them. We regard the most significant movements as
regrouping, where those movements change the grouping (the label to which an
image is closest). We did not expect any major difference in the number of
interaction operations, but one might expect differences in how participants
interact with objects during communication or perhaps through embodied
decision making (e.g., externalising their thought process by different usage
of space for objects being discussed or left for later discussion). An
important measure of effectiveness of collaborative activities is whether all
team members are able to contribute equally to the task. We wondered RQ3: Do
people collaborate more equally in VR or Desktop in terms of interaction? We
used the Gini coefficient (David, 1968; Dixon et al., 1987) to do the
calculation, which ranges from 0 to 1, with 0 meaning all participants of a
group contributed to the interaction equally, and 1 indicating they did not
contribute equally—typically one of the participants led the whole group
during a task.

Conversation. RQ4: Do people spend a different amount of time on discussion in
VR than Desktop? We analysed the conversation proportion to explore this
question, using the procedure from Cordeil et al. (Cordeil et al., 2016). Same
as for interaction, we also explored equality of conversation among
participants. RQ5: Do people collaborate more equally in VR or Desktop in
terms of discussion?

RQ6: Does conversation differ in VR from Desktop? We categorised the
conversation into task-related and social conversations. We expected roughly
equal task-related conversation between the two conditions, but for the
participants in the VR condition to have more social conversations due to
being more aware of the people around them.

Coordination. RQ7: What are the differences in coordination between
participants across Desktop and VR in terms of communication and action
activities? From previous studies (Gutwin and Greenberg, 2000; Janssen et al.,
2007) and our observations, we identified six behaviours of coordination from
the playback and audio recordings: Planning, Assistance (question and
exchange), Monitoring, Protection, Respect and Accommodation. For detailed
definitions of these terms refer to Appendix A.1.

####  3.8.3. Engagement

Awareness. As mentioned, we measured the study audio and visual playback to
get a sense of the participants’ collaborative awareness to address: RQ8: Is
there a difference in the type and amount of communication between users based
on the distance between them? To investigate how participants acknowledged
where the others were, we measured the relationship between the discussion and
the distance between participants. We retrieved three behaviours related to
awareness from the visual playback and audio recordings: No response, Think
aloud and Conversation conflict. We expected that in the VR condition the
participants would communicate more with those close to them in VR, and that
there would be no such correlation between mouse cursor separation and
communication in the Desktop condition.

Another analysis related to awareness concerns gestures to get others’
attention. RQ9: Is there a difference in the amount of gestures between the
two conditions of VR and Desktop? The program retrieved gestures from the
visual playback and audio recordings for the analysis. We expected there would
be more gestures in the VR condition than the Desktop condition as the VR
condition allowed more embodied presence of others in the environment.

Social Presence Rating. RQ10: Do people experience Social Presence differently
for VR and Desktop? We gathered the subjective rating of Social Presence to
evaluate participants’ perception of how aware they were of others during
collaboration, using the Harms and Biocca’s questionnaire (Harms and Biocca,
2004). The questionnaire measures the factors of Social Presence in four sub-
categories: Co-presence, Attentional Allocation, Perceived Understanding
(Message & Affective) and Perceived Interdependence (Emotional & Behavioral),
covering the aspects of communication, coordination and awareness. We expected
that the VR condition would be more similar to face-to-face circumstances and
the participants would be less aware of each other during tasks in the Desktop
condition.

####  3.8.4. Spatial Usage

One interesting question investigates the spatial usage based on observations.
RQ11: Do people use the space in VR and Desktop differently? Do they have
different strategies to use these space? We analysed the visual replay and
plotted participant and object positions over time to address this question.

##  4\. Behavioural Measure Results

The following is a detailed analysis of the various measures in terms of the
first nine research questions (RQs). We use the assumption-free non-parametric
Wilcoxon’s rank-sum test for this analysis (Field et al., 2012). All the
statistical results of the measures are presented in Table 2. A concise
summary of the significant results and their relationships to the research
questions is provided in Section 6. Some figures are combined regardless of
their order to minimise space.

Figure 2. Table of study results.

###  4.1. Task

We did not find significant differences on task accuracy or completion time
between two conditions. Regarding RQ1 we cannot accept the VR or Desktop
condition as being faster or more accurate. However Figure 7 (left, middle)
shows that there is a weak trend that the VR condition was slightly faster and
more accurate across most tasks.

###  4.2. Activity

Interaction. Figure 3 (left) shows the proportion of time that participants
spent on grabbing (and moving) objects and performing regrouping actions in
both conditions (RQ2). Participants spent a significantly greater percentage
of their time on both grabbing and regrouping in VR than in the Desktop
condition.

Figure 3. Left: Proportion of time that participants grab and regroup objects
(data points represent individual participants). Middle: The average number of
participants grabbing each object, and the average number of participants
changing the grouping of each image. The data points represent objects/images.
Right: Equality of interactions across tasks (each data point represents one
group in one task).

To compare participants’ interactions with objects across groups and
conditions, we computed networks showing which participants grabbed objects
(labels and images) and which regrouped images. A sample of the graphs
computed for each of the four tasks across one VR group and one Desktop group
is shown in Figure 4. We saw a common pattern across all groups that in VR
participants interacted with more objects than in Desktop and all four members
of each VR group tended to interact more uniformly. This is evident from the
more highly connected and symmetric graphs for the rows of VR graphs. An
analysis of the degrees of object and image nodes (Figure 3 middle) also shows
the greater degree of interaction in VR. In VR, objects are interacted with by
significantly more participants than in Desktop for both grabbing and
regrouping.

Figure 4. Indicative graphs of participants’ interactions with objects in one
VR and one Desktop group. In the “Grabs” graphs there is a link between a
participant and an object (label/image) if the object was grabbed and moved by
that user. In the “Regroup Actions” graphs a link shows the image was moved to
a different group by the user. The link thickness indicates the number of
interactions. In VR, we see objects are both manipulated and regrouped by more
participants than Desktop.

Figure 3 (right) shows the equality of participants’ interactions with the
objects in each task for both conditions in terms of duration (time spent
grabbing and regrouping), count of grabs and regroup operations (RQ3). The
only statistically significant difference is in grab duration. The lower Gini
coefficient in the VR condition shows participants in VR interacted with the
objects more equally in terms of duration, and significantly more equally than
those in the Desktop condition.

Conversation. Figure 5 (left) shows the proportion of time that participants
have conversations (someone was speaking) in both conditions (RQ4).
Participants spent significantly more time speaking in VR than in Desktop.
Figure 5 (middle) shows the equality of contributions by participants to the
conversation as a proportion of task time in both conditions (RQ5). There is
no significant difference between the two conditions. Where there is
conversation, Gini coefficient scores between 0.3–0.6 indicate that there is
an imbalance between participants in both conditions, although the condition
does not effect this significantly.

Figure 5. Left, Middle: Oral collaboration performance. Each data point
represents one group in one task. Right: Coordination activities boxplots. For
Plan, Question, and Exchange activities, each data point represents one group
in one task, and the percentage is duration of each activity against task
completion time. For Monitoring, Protection, Respect and Contention, data
points represent individual participants. The Monitoring percentage relates to
duration, while for Protection, Respect, and Contention percentage indicates
the number of times that these activities happened to a participant relative
to the total conversation/interaction times across all tasks.

We identified various topics of social-related conversations in VR: Greeting
(2 times), Emotion (4 times), Fun (5 times), Interaction (2 times), Distance
(6 times) and Height (2 times) across all groups. The social conversations
rarely happened in Desktop, once for Greeting and once for Fun (RQ6). Some
examples for these topics are as follows. For Fun, participants played with
the hand animation and explored it with others. For Interaction, participants
tried to “steal” objects from others’ hands. For Distance, as the participants
were remotely located, some participants were surprised that they could pass
through others and discussed the feeling of this. Some participants also
expressed that the height of some objects placed by other participants were
out of their reach. The duration of these conversations were mostly between 10
to 45 seconds. We identified three topics for the task-related conversations,
as shown in Figure 5 (right): Planning, Question and Exchange, which also
related to Coordination (discussed below).

Coordination. Figure 5 (right) shows there are significant differences of
Monitoring, Protection, Respect and Contention (RQ7). The participants in
Desktop spent much more time on monitoring than in VR. They also tended to
protect their work more than in VR. Participants respected others’ work more
in VR than in Desktop. For Accommodation, we found participants exhibited the
Voting activity once in VR and twice in Desktop, and they tried to interact
with the same objects significantly more in Desktop than in VR. There is no
significant difference of Planning and Assistance (Question and Exchange)
activities between two conditions (RQ7).

###  4.3. Engagement

Awareness. Figure 6 (left) shows there are significant differences of No
response, Think aloud and Conversation conflict behaviours. There are more
occurrences of No response in Desktop than in VR, but less Think aloud and
Conversation conflict happened in Desktop than in VR.

We produced heatmaps of participants’ $x$\- and $z$-axis positions to explore
the relationship between distance and communication between participants
(RQ8). In VR, we observed that the parallel conversations happened wherever
the participants were. For example, Figure 6 (right (a)) is the heatmap of a
group in one task. It shows most of the time the participants worked in pairs
(blue and magenta, red and cyan) and stood on the opposite sides of the
“object wall” they built in the center of the room (see Figure 1 (e)). Besides
talking to the person on the same side, participants also talked to the others
across the “object wall”. The conversations with no responses happened when
the other participants stood far away and had some participants close to them,
as shown in Figure 6 (right (b)) that the participants freely walked around
the room during the task. The heatmaps for the Desktop condition did not
reveal that there was a relationship between the distance between participants
(in terms of mouse cursor position) and the communication pattern.

Figure 6. Left, Middle: Awareness behaviours boxplots. The data points for No
response, Think aloud and Conversation conflict behaviours represent tasks.
The percentage values indicate the relative number of times these behaviours
occurred. For the Shake behaviour each data point represents individuals, and
the count value indicates the number of times that participants shook their
controller/mouse in all tasks. Right: Sample heatmaps of participants’ $x$\-
and $z$-axis positions over time. Each heatmap is generated for one group in
one task, and each colour represents a participant.

Two gestures were identified that participants performed to attract others’
attention: Pointing and Shaking. Since Pointing happened in every conversation
in Desktop condition (i.e., mouse hovering over objects) as the only awareness
indication, we did not include it into the discussion. The Shaking gesture was
counted during speaking. In VR, participants shook their virtual hands or
objects, and in Desktop participants quickly moved their mouse cursor around
on or with objects. As can be seen from Figure 6 (middle) there is no
significant difference across two conditions (RQ9).

Social Presence Rating. Except for the sub-category Attentional Allocation,
the Cronbach’s Alpha of other sub-categories ranges from 0.90 to 0.96 in VR
and from 0.78 to 0.95 in Desktop, showing high internal consistency. The
Cronbach’s Alpha of Attentional Allocation is 0.65 and 0.64 for VR and
Desktop, which are in an accepted range. Figure 7 (right) shows there is no
significant difference in social presence in each sub-category across both
conditions (RQ10).

Figure 7. Left, Middle: Task accuracy and completion time across all groups.
Right: Boxplots for Social Presence results of overall rating and each sub-
categories: Co-presence (CP), Attentional Allocation (AA), Perceived Message
Understanding (PMU), Perceived Affective Understanding (PAU), Perceived
Emotional Interdependence (PEI) and Perceived Behavioral Interdependence
(PBI). The data points represent individual participants.

##  5\. Observation Results

To explore RQ11, we analysed the study visual replay in detail in terms of:
Strategies, Organisations and Territories, which were widely used to analyse
collaboration behaviours in previous research (Bradel et al., 2013; Clayphan
et al., 2016; Lee et al., 2020; Luo et al., 2022).

Strategies. Participants’ strategies for space use were quite similar in both
conditions, where the participants divided the space into regions: grouped
region, unsure region and ungrouped region. The empty space between the
regions tended to be used temporarily by the participants for placing the
objects for discussion.

Collaboration strategies were also quite similar across conditions. We
identified two main strategies: Strategy 1 Participants worked individually to
organise images first (Phase One). Some groups put images they were unsure of
in an empty space, then subsequently discussed the images in the unsure pile
(Phase Two). When they finished an initial grouping of all images, they
started reviewing the groups (Phase Three). This basic strategy was used by
four groups in both VR and Desktop. Strategy 2 Participants gathered all the
images to the centre of the space, and discussed and grouped them together.
One group in VR and one group in Desktop used this strategy.

Organisation. There were two types of organisation in VR condition: (1) 2D
space (two groups) and (2) 3D space (three groups). The images in the
organisation using 2D space were basically formed into a kind of “panel” and
these panels were either placed in the centre of the room (see Figures 1(e)
and (f)) or on one side of the room (see Figure 1(g)). There were three
patterns of layout of the objects on these 2D panels, e.g., Figure 1(e) the
images followed the labels and formed Rows, Figure 1(f) the images followed
the labels and formed Columns, and Figure 1(g) the images were placed around
the labels into Clusters. We identified three patterns from the organisations
using 3D space. The first is depicted in Figure 1(h), the participants
organised the labels around multiple sides of the room and put the images
around these labels to form some Clusters. Each cluster was organised as a
small 2D “panel”. In the second, the participants fully used the 3D space and
organised the labels far away from one another as possible. For example in
Figure 1(i), four labels were put into four corners and one label was placed
in the centre of the room. The images placed around them formed a spherical
layout in each Cluster. For the third, one group created a Convex hull and
built a kind of “console” in the centre of the room (see Figure 1(j)). The
labels were placed around a circle in the centre of the console and the images
followed the labels and extended from the centre to the outside of the
console.

Spatial organisations in the Desktop condition were more uniform than in VR.
We observed three patterns of the layouts in Desktop: (1) Rows (one group),
(2) Columns (two groups), and (3) Clusters (two groups). The rows and columns
layouts were similar to the 2D “panel” layouts in VR condition as shown in
Figures 1(a) and (b). For the cluster layouts, each cluster occupied a
separate space on the 2D canvas, either on the corners (when number of labels
= 4 (see Figure 1(c)) or around the border of the canvas (see Figure 1(d)).

In both conditions, participants either planned the arrangements (two groups
in VR, three groups in Desktop) or instinctively organised the objects without
discussion (three groups in VR and two groups in Desktop). The organisation
planning normally happened at the beginning of the tasks, but some groups
refined the plan in the middle of the tasks when they found the initial
organisation did not work for the task. The groups that had spatial
organisation plans usually created more organised layouts in both conditions,
such as Row and Column layouts. When there was no spatial organisation
planning, the participants just followed others’ strategies. This normally
happened in the groups using Cluster layouts in both conditions.
Interestingly, the VR group that produced the Convex hull layout (see Figure
1(j)) did not plan the organisation. They collected the labels in the center
of the room at the beginning of Task 1, and stood in a circle around the
labels, facing others to discuss the word meanings. When one participant
rotated a label towards themselves to see the text, others used the same
behaviour. As a result, they naturally extended the grouping of images
outwards from the labels in the center and built a “console”, and used the
same layout throughout the subsequent tasks.

Territories. We observed territory patterns from Figure 8, which show the
positions of objects when picked and placed. Each sub-figure is generated for
one group, and each column in the sub-figures represents each task. A dot is
drawn when a participant picked (first row in each sub-figure) or placed
(second row in each sub-figure) the object. The dots are colour-coded by
participants. Figures 8(a)–(e) are the heatmaps for the groups in VR, and
Figures 8(f)–(h) shows three patterns in Desktop. These territories match the
object organisation discussed previously.

Figure 8(a) reveals how participants in VR built a 2D “panel” in the centre of
the room, and Figure 8(b) shows how they built a panel along the sides of the
room. Figures 8(c)–(e) show use of territories in 3D space. Figures 8(c) and
(d) depict building a spherical layout, and Figure 8(e) is for the “console”
built in the room centre.

The territory heatmap for Desktop condition is uniform, e.g., Figure 8(f)
shows the participants built the row layout, Figure 8(g) shows is the heatmap
for the column layout, and Figure 8(h) is the heatmap for the cluster layouts.
Interestingly, Figure 8(h) column 3 and 4 also reveals that in Task 3 and 4
the participants collected all the objects into the canvas centre at first to
make space for organising the grouped images.

Figure 8. Sample participant interaction heatmaps that indicate the
territories of participants. Two groups of Desktop condition are omitted as
their heatmap patterns are similar to (h). The dots represented the positions
when an object was picked/placed by a participant, coloured the same as the
participant. Sub-figures (a)–(e) are $x$\- and $z$-axis positions of objects
in VR and (f)–(h) are $x$\- and $y$-axis positions of objects in Desktop.

Feedback. The questionnaire prompted participants for their open-ended
thoughts and suggestions. For both Desktop and VR conditions, feedback
overwhelmingly described the systems as being fun to use and easy to
collaborate in. For the Desktop condition, some comments were made by single
participants: auto-alignment tools for the cards; a highlight button to
attract attention of group members; and the ability to see each other within
the application itself, akin to online services such as Gather
(https://www.gather.town/). For the VR condition, three participants pointed
out that they felt the headsets were uncomfortable after some time. Two
participants had requested more communication methods such as symbols or
emoji-like responses, particularly to show agreement in group decisions. The
remaining comments were given by single participants: poor provenance as there
was no way to identify who moved the cards without asking; difficulties caused
by height differences between group members; a feature to avoid collisions
with each other; and a way to select and move multiple cards at once.

##  6\. Discussion

There are several notable findings from the results of the user study.
Firstly, VR prompted more interactions with virtual objects and more
communications between participants than Desktop, especially much more social
conversations. In the Desktop condition, participants spent more time
monitoring others’ behaviours, but interestingly they had more contention
while interacting with objects than in VR. VR participants initiated more
discussions when they tried to modify others’ work, while Desktop participants
were involved in more discussions concerning their own work. There was more
parallel talking in VR than in Desktop, and more no-response conversation
happened in Desktop than in VR. Detailed discussions on the measures follow.

###  6.1. Task

The results of accuracy match our expectation discussed in Section 3.8 (RQ1),
where the condition did not have effects on accuracy. Unexpectedly,
participants in the Desktop condition did not complete the tasks faster. From
observation, we found that in VR participants often used two hands to grab two
images at the same time, which potentially saved time for moving images.
Another explanation for the similar completion time is that the space in VR is
not large, so participants could easily walk in the virtual room. As the study
was designed to examine natural embodiment experience, we did not include
extra navigation techniques (e.g., teleportation for VR, panning and zooming
for Desktop). It would be interesting to investigate in the future whether the
navigation techniques affect task completion time in a large virtual space.

###  6.2. Activity

Interaction. Participants grabbed and regrouped objects more frequently and
spent significantly longer time on these in VR than in Desktop (RQ2), and VR
participants contributed more equally on grabbing objects (RQ3). From
observation, the reason for this could be that aligning objects in 3D space
requires more effort than in 2D space, so participants adjusted the placement
of objects more in VR than on the Desktop to make the organisation neater.
This indicates that there is a desire to have auto-alignment mechanisms for
objects placed in an immersive environment. Also, we observed that when there
was not enough space to add more images into individual groups, participants
spent longer time in VR than Desktop on moving the images one-by-one to make
more space for the existing image groups. Providing functions to allow
grouping and moving multiple images together will facilitate this object
arrangement process. Another reason for more interaction and longer
interaction time in VR could be that the animated hand representation in VR
gave strong self-presence and embodiment sense to participants, which may
increase the enjoyment of interactions and stimulate participants to interact
more with objects.

Conversation. Participants spent significantly more time speaking in VR than
Desktop (RQ4). This is surprising but could be attributed to VR increasing the
sense of physically being in the space and working with other people. However,
there is no significant difference in conversation equality of participants
between the two conditions (RQ5). As expected, we observed that the task-
related conversations were similar in the two conditions, and there were much
more social conversations in VR than Desktop (RQ6). This might suggest that
people in VR feel more like speaking one-to-one rather than one-to-many in a
Zoom like interface. Also, VR might provide a more natural and less
professional shared space feeling than video conferencing, as the latter has
already been widely adopted in various professional occasions. One possibility
to shape a more professional workplace in VR could be providing professional
scenes, avatars and objects to simulate an office or conference venue. Another
interesting finding is about the conversation on height. In VR when there was
a great disparity in the heights of the participants, sometimes the short
participants commented there were some images outside their accessible range,
and then others helped them to reach or place the images. While providing
distance interaction can solve this issue, another possibility is employing
navigation techniques to “scroll” the whole virtual space, which not only
supports interaction but also give everyone the power to glimpse the whole
space.

Coordination. Regarding coordination activities (RQ7), participants in the
Desktop condition monitored others’ behaviours and subsequently protected
their own works more than in VR. It is harder to get an overview of the space
in VR than Desktop, especially to notice what is happening outside your own
field of view in an immersive environment. One possibility is to provide
notifications of changes made by others to inform users. Participants in VR
respect others’ work more than in Desktop. Although there is no difference in
the social presence perception rated by participants across the two
conditions, we suspect that the greater presence and awareness of working with
others made the participants respect others more in VR than in the Desktop
condition. Also, the lack of presence led to Desktop participants having more
contentions on interaction with objects than in VR.

###  6.3. Engagement

Awareness. As shown in Figure 6, there were more Think aloud and Conversation
conflicts in VR than Desktop, as separate parallel conversations can occur in
the VR environment but are impossible in Zoom calls. However, we did not
observe any relationship between distance and communication for awareness
(RQ8). As discussed in Section 4.3, the parallel conversations in VR happened
regardless of the collaborators’ positions in the virtual space. This is an
interesting phenomenon as the spatial sound did not work as in the physical
world where people tend to only talk to others standing close to them. To
support a better experience of parallel conversations, VR systems could
provide sound isolation mechanisms to allow users to choose whom they want to
talk with. There were more occurrences of No response occurring in Desktop
than VR, which could be due to people in Desktop being uncertain who a
question was directed towards or perhaps because of lower engagement. Also,
there were slightly more Shaking behaviours in Desktop than VR (RQ9), which is
contrary to our expectation. We suspect that it was more difficult for
participants in Desktop to get others’ attention than in VR.

Social Presence Rating. Surprisingly, participants did not give a
significantly lower rating on Social Presence in Desktop than VR (RQ10) as we
expected. Perhaps people are now so familiar with remote Desktop collaboration
that they feel relatively comfortable communicating, despite the impediments
we observe above.

###  6.4. Observations

Spatial Organisation. Regarding RQ11, although the spatial organisations in
Desktop are neater, we can find common patterns of the layouts across two
conditions, like Rows, Columns and Clusters. This suggests that even in
immersive 3D space, users tend to organise objects as conventionally as they
do in 2D space. Future research could be done to design and provide automatic
layout options in immersive sensemaking systems to facilitate object
organisations. We also found diverse space usage from different groups in VR,
which is different from previous studies (Batch et al., 2019; Satriadi et al.,
2020; Lisle et al., 2021; Lee et al., 2020). This suggests that the nature of
tasks, devices, and individual differences need to be considered when
designing an automatic organisational layout system. We did not observe which
spatial usage and organisations facilitated or impaired task performance and
collaboration, and we note that this could be due to the limited number of
appearances of each organisation pattern.

Territories. There is no clear boundary of individual working spaces in either
condition. Although there were individual and collaborative working phases,
participants freely walked within the room in VR or moved their mice on the
canvas on the Desktop to interact with objects or communicate with others
during the whole task. This leads to an interesting question for future
research: should there be functions to claim individual territories in the
shared space and would this positively or negatively impact collaboration?

Other Concerns. Although we did not ask for physical workload rate from
participants, participants reported discomfort with VR headsets. However, the
physical affordance seems not to impede participants’ engagement and activity
in the VR condition.

###  6.5. Design Considerations for Immersive Collaboration

Here we summarise from the detailed analysis above the key considerations that
may inform design of and research into future immersive collaboration systems:

Communication (based on the discussion on Awareness in Section 6.3): active
(manual or automatic) control over participant volume levels is essential to
support parallel conversations. In our VR system we used spatial audio with a
custom logarithmic roll-off (see 3.3). Participants indicated they wanted
greater control, so either better automatic sound attenuation needs to be
developed or systems need to consider providing functions to mute certain
collaborators.

Notification (based on the discussion on Coordination in Section 6.2):
consider providing notifications of collaborators’ activity, since
participants wanted to know of changes made by others in order to protect
against destructive changes. This is both a workaround for the limited field
of view in current VR headsets, but also the ability to be aware of changes
made behind one’s back is a potential visualisation “superpower” (Willett et
al., 2021).

Navigation (based on the discussion on Conversation in Section 6.2): consider
providing navigation mechanisms, to allow users to “scroll” the whole
immersive space in all the three dimensions (x, y, z). Participants wanted the
ability to move the surrounding space to access out-of-reach objects without
modifying layout and to get an overview of the environment. We can also
consider providing a miniature space to give an overview and inform users
about current field of view.

Environment (based on the discussion on Conversation in Section 6.2): consider
providing options for users to choose different types of environments,
decorative elements and avatars, to support different collaborative scenarios
and atmosphere, or to support management of territories (as per the discussion
on Territories in Section 6.4). The ability to adapt working environment for
various requirements of collaborative relationships and feelings can improve
efficiency, effectiveness and engagement in collaboration (Raziq and
Maulabakhsh, 2015). For example, participants who spent time playing with
avatars or gestures in VR, might have done so less in a more formal VR
environment like a virtual classroom or conference room.

Automatic placement (based on the discussions on Interaction in Section 6.2
and Spatial Organisation in Section 6.4): consider providing auto-alignment
mechanisms to facilitate placing virtual objects, and providing automatic
layout options to organise the objects. While layout and alignment tools are
frequently found in desktop diagramming tools, in immersive environments they
still reduce physical work required to place objects but are arguably more
important because this work is increased due to the third dimension. For
example, allowing snapping to 2D panels to facilitate alignment to planes and
spherical surfaces, which were tasks we observed users doing manually in our
study (Fig. 1).

##  7\. Limitations, Conclusions and Future Work

Our results suggest many positive outcomes and potential advantages for
performing distributed collaborative sensemaking tasks that involve spatial
organisation in VR over more standard Desktop environment. Our main conjecture
is that many of the advantages stem from the natural embodied interaction and
communication that is possible in VR, and this would lead to increased
engagement in the task. However, we note that there could also be other
sources of increased engagement in VR, such as the isolation effect of VR
headsets reducing environmental distractions, or the novelty effect of using
VR compared to more familiar desktop communication and collaboration tools.

Our study employed basic and natural techniques that are suitable for novice
users. In the future, these techniques could be used as the basis for a suite
of studies varying the parameters of the shared immersive environment, such as
the introduction of table and wall surfaces, ranged interactions with objects,
and navigation techniques such as teleport and zoom to enable larger shared
virtual workspaces. Collaboration in immersive sensemaking is an area that has
not been well studied. This research involved groups of four subjects. In the
future, the measures that we have developed to investigate group user
behaviours could be extended to fit larger group settings. Also, we used a
variety of objective measures for behavioural analysis, but only one
subjective measure, Social Presence. In the future, these measures could be
supplemented with more subjective surveys around group dynamics (such as who
the perceived leader was), and communication behaviours (such as subject
perception of how easy it was to communicate with each other). It would also
be good to use interviews to capture more overall feedback about the
experience.

In summary, we believe that group-based immersive sensemaking is a rich area
for future research, and hope that the measures we have developed could be of
use to broader research community.

###### Acknowledgements.

This research was supported under the Australian Research Council’s Discovery
Projects funding scheme (DP180100755).

##  Appendix A Appendix

###  A.1. Measure Definitions

####  A.1.1. Task

Accuracy. The proportion of the number of images that were categorised into
the correct emotion terms to the total number of images.

Completion Time. Between the time that the experimenter started showing the
task content to the participants and the time that the experimenter changed
the content to the next task.

####  A.1.2. Interaction

Grab. The interaction that moves an object but does not change the grouping of
an image (change the label to which an image is closest).

Regroup. The interaction that moves an object and changes the grouping of an
image.

Interaction Proportion. The proportion of interaction duration in each task to
the task completion time.

Interaction Count. The number of times that interactions are applied to
objects.

Interaction Equality. The balance between the duration of interaction that
each participant contributed (equality of duration) and the balance between
the interaction count of each participant (equality of count). The Gini
coefficient (David, 1968) that is a commonly used measure of inequality was
used to do the calculation. As the sample size was not large, a refined bias-
corrected method of the Gini coefficient (Dixon et al., 1987) was used.

####  A.1.3. Conversation

Conversation Proportion. The proportion of conversation duration in each task
to the task completion time.

Conversation Equality. The balance between the duration of conversation that
each participant contributed. Again the bias-corrected Gini coefficient was
used to do the calculation.

####  A.1.4. Coordination

Planning. Planning refers to the high-level decisions about the tasks and
collaborations. The discussions would relate to the strategies for organising
the image and label cards, and dividing the responsibilities among
participants.

Assistance. Participants help each other during collaboration. It is further
divided into two categories: Question (directly requesting help from others,
asking/answering questions), and Exchange (exchanging information).

Monitoring. Gutwin and Greenberg (Gutwin and Greenberg, 2000) defined
monitoring as the awareness of others in the shared workspace, such as the
information of who are the others, where are the others and what the others
are doing. Monitoring here is considered as explicit observation of others and
focus on others’ activities during collaboration. The Monitoring activity was
retrieved automatically from audio and visual session playback when
participants did not speak or interact with any objects longer than 5 seconds.

Protection. Protection refers to that participants defended their own work.
This activity was retrieved automatically from visual playback and audio
recordings, when participants made a response to the discussion of the images
that they had grouped.

Respect. Respect refers to a participant only modified other’s work after
asking for permission. This activity was captured automatically from visual
playback and audio recordings. The criteria for retrieving this was when
participants wanted to changed the grouping of an image, they asked others’
opinions.

Accommodation. Accommodation relates to dealing with conflicts during
collaboration, such as resolving different opinions (e.g. Voting) and
interacting to the same object at the same time (e.g. Contention).

####  A.1.5. Awareness

No Response. No response when a participant initialised a question.

Think Aloud. A participant talked to self.

Conversation Conflict. More than one conversation happened concurrently.

Shaking. When speaking, a participant shook the virtual hand (VR) / mouse
cursor (desktop) or objects.

## References

  * (1)
  * Andrews et al. (2010) Christopher Andrews, Alex Endert, and Chris North. 2010\.  Space to Think: Large High-Resolution Displays for Sensemaking. In _Proceedings of the SIGCHI Conference on Human Factors in Computing Systems_. 55–64.  https://doi.org/10.1145/1753326.1753336
  * Baker et al. (2002) Kevin Baker, Saul Greenberg, and Carl Gutwin. 2002\.  Empirical Development of a Heuristic Evaluation Methodology for Shared Workspace Groupware. In _Proceedings of the 2002 ACM Conference on Computer Supported Cooperative work_. 96–105.  https://doi.org/10.1145/587078.587093
  * Barsalou (1983) Lawrence W Barsalou. 1983\.  Ad hoc categories.  _Memory & Cognition_ 11, 3 (1983), 211–227.  https://doi.org/10.3758/BF03196968
  * Batch et al. (2019) Andrea Batch, Andrew Cunningham, Maxime Cordeil, Niklas Elmqvist, Tim Dwyer, Bruce H Thomas, and Kim Marriott. 2019.  There Is No Spoon: Evaluating Performance, Space Use, and Presence with Expert Domain Users in Immersive Analytics.  _IEEE Transactions on Visualization and Computer Graphics_ 26, 1 (2019), 536–546.  https://doi.org/10.1109/TVCG.2019.2934803
  * Bente et al. (2008) Gary Bente, Sabine Rüggenberg, Nicole C. Krämer, and Felix Eschenburg. 2008.  Avatar-Mediated Networking: Increasing Social Presence and Interpersonal Trust in Net-Based Collaborations.  _Human Communication Research_ 34, 2 (04 2008), 287–318.  https://doi.org/10.1111/j.1468-2958.2008.00322.x arXiv:https://academic.oup.com/hcr/article-pdf/34/2/287/22325251/jhumcom0287.pdf
  * Billinghurst et al. (2003) Mark Billinghurst, Daniel Belcher, Arnab Gupta, and Kiyoshi Kiyokawa. 2003.  Communication Behaviors in Colocated Collaborative AR Interfaces.  _International Journal of Human-Computer Interaction_ 16, 3 (2003), 395–423.  https://doi.org/10.1207/S15327590IJHC1603_2
  * Billinghurst et al. (2018) Mark Billinghurst, Maxime Cordeil, Anastasia Bezerianos, and Todd Margolis. 2018.  Collaborative immersive analytics.  In _Immersive Analytics_. Springer, 221–257.  https://doi.org/10.1007/978-3-030-01388-2_8
  * Biocca et al. (2003) Frank Biocca, Chad Harms, and Judee K. Burgoon. 2003.  Toward a More Robust Theory and Measure of Social Presence: Review and Suggested Criteria.  _Presence: Teleoperators and Virtual Environments_ 12, 5 (10 2003), 456–480.  https://doi.org/10.1162/105474603322761270 arXiv:https://direct.mit.edu/pvar/article-pdf/12/5/456/1623957/105474603322761270.pdf
  * Bradel et al. (2013) Lauren Bradel, Alex Endert, Kristen Koch, Christopher Andrews, and Chris North. 2013\.  Large high resolution displays for co-located collaborative sensemaking: Display usage and territoriality.  _International Journal of Human-Computer Studies_ 71, 11 (2013), 1078–1088.  https://doi.org/10.1016/j.ijhcs.2013.07.004
  * Bulu (2012) Saniye Tugba Bulu. 2012\.  Place presence, social presence, co-presence, and satisfaction in virtual worlds.  _Computers and Education_ 58, 1 (2012), 154–161.  https://doi.org/10.1016/j.compedu.2011.08.024
  * Cavallo et al. (2019) Marco Cavallo, Mishal Dolakia, Matous Havlena, Kenneth Ocheltree, and Mark Podlaseck. 2019\.  Immersive Insights: A Hybrid Analytics System for Collaborative Exploratory Data Analysis. In _25th ACM Symposium on Virtual Reality Software and Technology_. 1–12.  https://doi.org/10.1145/3359996.3364242
  * Chandler et al. (2015) Tom Chandler, Maxime Cordeil, Tobias Czauderna, Tim Dwyer, Jaroslaw Glowacki, Cagatay Goncu, Matthias Klapperstueck, Karsten Klein, Kim Marriott, Falk Schreiber, et al. 2015\.  Immersive Analytics. In 2015 Big Data Visual Analytics (BDVA), Vol. 1. Institute of Electrical and Electronics Engineers (IEEE), Hobart, TAS, USA, 1–8.  https://doi.org/10.1109/BDVA.2015.7314296
  * Chen et al. (2018) Juanjuan Chen, Minhong Wang, Paul A Kirschner, and Chin-Chung Tsai. 2018.  The Role of Collaboration, Computer Use, Learning Environments, and Supporting Strategies in CSCL: A Meta-Analysis.  _Review of Educational Research_ 88, 6 (2018), 799–843.  https://doi.org/10.3102/0034654318791584
  * Chung et al. (2014) Haeyong Chung, Chris North, Jessica Zeitz Self, Sharon Chu, and Francis Quek. 2014.  VisPorter: facilitating information sharing for collaborative sensemaking on multiple displays.  _Personal and Ubiquitous Computing_ 18, 5 (2014), 1169–1186.  https://doi.org/0.1007/s00779-013-0727-2
  * Clayphan et al. (2016) Andrew Clayphan, Roberto Martinez-Maldonado, Martin Tomitsch, Susan Atkinson, and Judy Kay. 2016\.  An in-the-wild study of learning to brainstorm: comparing cards, tabletops and wall displays in the classroom.  _Interacting with Computers_ 28, 6 (2016), 788–810.  https://doi.org/10.1093/iwc/iww001
  * Cockburn and McKenzie (2002) Andy Cockburn and Bruce McKenzie. 2002.  Evaluating the Effectiveness of Spatial Memory in 2D and 3D Physical and Virtual Environments. In _Proceedings of the SIGCHI Conference on Human Factors in Computing Systems_. 203–210.  https://doi.org/10.1145/503376.503413
  * Cordeil et al. (2016) Maxime Cordeil, Tim Dwyer, Karsten Klein, Bireswar Laha, Kim Marriott, and Bruce H Thomas. 2016\.  Immersive Collaborative Analysis of Network Connectivity: CAVE-style or Head-Mounted Display?  _IEEE transactions on visualization and computer graphics_ 23, 1 (2016), 441–450.  https://doi.org/10.1109/TVCG.2016.2599107
  * David (1968) Herbert A. David. 1968\.  Miscellanea: Gini’s mean difference rediscovered.  _Biometrika_ 55, 3 (1968), 573–575.  https://doi.org/10.1093/biomet/55.3.573
  * Dixon et al. (1987) Philip M Dixon, Jacob Weiner, Thomas Mitchell-Olds, and Robert Woodley. 1987.  Bootstrapping the Gini Coefficient of Inequality.  _Ecology_ 68, 5 (1987), 1548–1551.  https://doi.org/10.2307/1939238
  * Donalek et al. (2014) Ciro Donalek, S. G. Djorgovski, Alex Cioc, Anwell Wang, Jerry Zhang, Elizabeth Lawler, Stacy Yeh, Ashish Mahabal, Matthew Graham, Andrew Drake, Scott Davidoff, Jeffrey S. Norris, and Giuseppe Longo. 2014.  Immersive and collaborative data visualization using virtual reality platforms. In _2014 IEEE International Conference on Big Data (Big Data)_. 609–614.  https://doi.org/10.1109/BigData.2014.7004282
  * Durlach and Slater (2000) Nat Durlach and Mel Slater. 2000.  Presence in Shared Virtual Environments and Virtual Togetherness.  _Presence: Teleoperators and Virtual Environments_ 9, 2 (04 2000), 214–217.  https://doi.org/10.1162/105474600566736 arXiv:https://direct.mit.edu/pvar/article-pdf/9/2/214/1623414/105474600566736.pdf
  * Endert et al. (2012) Alex Endert, Seth Fox, Dipayan Maiti, Scotland Leman, and Chris North. 2012.  The Semantics of Clustering: Analysis of User-Generated Spatializations of Text Documents. In _Proceedings of the International Working Conference on Advanced Visual Interfaces_. 555–562.  https://doi.org/10.1145/2254556.2254660
  * Engelbart and English (1968) Douglas C Engelbart and William K English. 1968.  A research center for augmenting human intellect. In _Proceedings of the December 9-11, 1968, fall joint computer conference, part I_. 395–410.  https://doi.org/10.1145/1476589.1476645
  * Ens et al. (2019) Barrett Ens, Joel Lanir, Anthony Tang, Scott Bateman, Gun Lee, Thammathip Piumsomboon, and Mark Billinghurst. 2019.  Revisiting collaboration through mixed reality: The evolution of groupware.  _International Journal of Human-Computer Studies_ 131 (2019), 81–98.  https://doi.org/10.1016/j.ijhcs.2019.05.011
  * Field et al. (2012) Andy Field, Jeremy Miles, and Zoë Field. 2012.  _Discovering Statistics Using R_.  SAGE Publications.  https://doi.org/10.1111/insr.12011_21
  * Gerhard et al. (2004) Michael Gerhard, David Moore, and Dave Hobbs. 2004.  Embodiment and copresence in collaborative interfaces.  _International Journal of Human-Computer Studies_ 61, 4 (2004), 453–480.  https://doi.org/10.1016/j.ijhcs.2003.12.014
  * Greenberg et al. (1996) Saul Greenberg, Carl Gutwin, and Mark Roseman. 1996\.  Semantic Telepointers for Groupware. In _Proceedings sixth Australian conference on computer-human interaction_. IEEE, 54–61.  https://doi.org/10.1109/OZCHI.1996.559988
  * Gutwin and Greenberg (1996) Carl Gutwin and Saul Greenberg. 1996.  Workspace Awareness for Groupware. In _Conference Companion on Human Factors in Computing Systems_. 208–209.  https://doi.org/10.1145/257089.257284
  * Gutwin and Greenberg (2000) Carl Gutwin and Saul Greenberg. 2000.  The Mechanics of Collaboration: Developing Low Cost Usability Evaluation Methods for Shared Workspaces. In _Proceedings IEEE 9th International Workshops on Enabling Technologies: Infrastructure for Collaborative Enterprises (WET ICE 2000)_. IEEE, 98–103.  https://doi.org/10.1109/ENABL.2000.883711
  * Gutwin et al. (2004) Carl Gutwin, Reagan Penner, and Kevin Schneider. 2004\.  Group Awareness in Distributed Software Development. In _Proceedings of the 2004 ACM conference on Computer supported cooperative work_. 72–81.  https://doi.org/10.1145/1031607.1031621
  * Harms and Biocca (2004) Chad Harms and Frank Biocca. 2004.  Internal Consistency and Reliability of the Networked Minds Measure of Social Presence. In _Proceedings of the 7th Annual International Workshop: Presence_. 1–7.
  * Hauber et al. (2005) Jörg Hauber, Holger Regenbrecht, Aimèe Hills, Andy Cockburn, and Mark Billinghurst. 2005\.  Social presence in two-and three-dimensional videoconferencing.  (2005).
  * Hawkey et al. (2005) Kirstie Hawkey, Melanie Kellar, Derek Reilly, Tara Whalen, and Kori M Inkpen. 2005.  The Proximity Factor: Impact of Distance on Co-located Collaboration. In _Proceedings of the 2005 international ACM SIGGROUP conference on Supporting group work_. 31–40.  https://doi.org/10.1145/1099203.1099209
  * Heidicker et al. (2017) Paul Heidicker, Eike Langbehn, and Frank Steinicke. 2017\.  Influence of Avatar Appearance on Presence in Social VR. In _2017 IEEE Symposium on 3D User Interfaces (3DUI)_. IEEE, 233–234.  https://doi.org/10.1109/3DUI.2017.7893357
  * Hernández-Sellés et al. (2019) Nuria Hernández-Sellés, Pablo-César Muñoz-Carril, and Mercedes González-Sanmamed. 2019\.  Computer-supported collaborative learning: An analysis of the relationship between interaction, emotional support and online collaborative tools.  _Computers & Education_ 138 (2019), 1–12.  https://doi.org/10.1016/j.compedu.2019.04.012
  * Higuchi et al. (2015) Keita Higuchi, Yinpeng Chen, Philip A Chou, Zhengyou Zhang, and Zicheng Liu. 2015.  ImmerseBoard: Immersive Telepresence Experience using a Digital Whiteboard. In _Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems_. 2383–2392.  https://doi.org/10.1145/2702123.2702160
  * Isenberg et al. (2011) Petra Isenberg, Danyel Fisher, Sharoda A Paul, Meredith Ringel Morris, Kori Inkpen, and Mary Czerwinski. 2011.  Co-located Collaborative Visual Analytics Around a Tabletop Display.  _IEEE Transactions on Visualization and Computer Graphics_ 18, 5 (2011), 689–702.  https://doi.org/10.1109/TVCG.2011.287
  * Jakobsen and Hornbæk (2014) Mikkel R Jakobsen and Kasper Hornbæk. 2014.  Up Close and Personal: Collaborative Work on a High-Resolution Multitouch Wall Display.  _ACM Transactions on Computer-Human Interaction (TOCHI)_ 21, 2 (2014), 1–34.  https://doi.org/10.1145/2576099
  * Janssen et al. (2007) Jeroen Janssen, Gijsbert Erkens, Gellof Kanselaar, and Jos Jaspers. 2007.  Visualization of participation: Does it contribute to successful computer-supported collaborative learning?  _Computers and Education_ 49, 4 (2007), 1037–1065.  https://doi.org/10.1016/j.compedu.2006.01.004
  * Kruger et al. (2004) Russell Kruger, Sheelagh Carpendale, Stacey D. Scott, and Saul Greenberg. 2004.  Roles of Orientation in Tabletop Collaboration: Comprehension, Coordination and Communication.  _Comput. Supported Coop. Work_ 13, 5-6 (Dec. 2004), 501–537.  https://doi.org/10.1007/s10606-004-5062-8
  * Kurdi et al. (2017) Benedek Kurdi, Shayn Lozano, and Mahzarin R Banaji. 2017\.  Introducing the Open Affective Standardized Image Set (OASIS).  _Behavior research methods_ 49, 2 (2017), 457–470.  https://doi.org/10.3758/s13428-016-0715-3
  * Lee et al. (2020) Benjamin Lee, Xiaoyun Hu, Maxime Cordeil, Arnaud Prouzeau, Bernhard Jenny, and Tim Dwyer. 2020\.  Shared Surfaces and Spaces: Collaborative Data Visualisation in a Co-located Immersive Environment.  _IEEE Transactions on Visualization and Computer Graphics_ 27, 2 (2020), 1171–1181.  https://doi.org/10.1109/TVCG.2020.3030450
  * Leigh et al. (1999) J. Leigh, A.E. Johnson, M. Brown, D.J. Sandin, and T.A. DeFanti. 1999.  Visualization in teleimmersive environments.  _Computer_ 32, 12 (1999), 66–73.  https://doi.org/10.1109/2.809253
  * Lisle et al. (2021) Lee Lisle, Kylie Davidson, Edward JK Gitre, Chris North, and Doug A Bowman. 2021.  Sensemaking Strategies with Immersive Space to Think. In _2021 IEEE Virtual Reality and 3D User Interfaces (VR)_. IEEE, 529–537.  https://doi.org/10.1109/VR50410.2021.00077
  * Liu et al. (2016) Can Liu, Olivier Chapuis, Michel Beaudouin-Lafon, and Eric Lecolinet. 2016.  Shared Interaction on a Wall-Sized Display in a Data Manipulation Task. In _Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems_. 2075–2086.  https://doi.org/10.1145/2858036.2858039
  * Liu et al. (2020) Jiazhou Liu, Arnaud Prouzeau, Barrett Ens, and Tim Dwyer. 2020\.  Design and Evaluation of Interactive Small Multiples Data Visualisation in Immersive Spaces. In _2020 IEEE Conference on Virtual Reality and 3D User Interfaces (VR)_. IEEE, 588–597.  https://doi.org/10.1109/VR46266.2020.00081
  * Luo et al. (2022) Weizhou Luo, Anke Lehmann, Hjalmar Widengren, and Raimund Dachselt. 2022.  Where Should We Put It? Layout and Placement Strategies of Documents in Augmented Reality for Collaborative Sensemaking. In _CHI Conference on Human Factors in Computing Systems_. 1–16.  https://doi.org/10.1145/3491102.3501946
  * Luo et al. (2021) Weizhou Luo, Anke Lehmann, Yushan Yang, and Raimund Dachselt. 2021.  Investigating Document Layout and Placement Strategies for Collaborative Sensemaking in Augmented Reality. In _Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems_. 1–7.  https://doi.org/10.1145/3411763.3451588
  * Marai et al. (2016) G Elisabeta Marai, Angus G Forbes, and Andrew Johnson. 2016\.  Interdisciplinary Immersive Analytics at the Electronic Visualization Laboratory: Lessons Learned and Upcoming Challenges. In _2016 Workshop on Immersive Analytics (IA)_. IEEE, 54–59.  https://doi.org/10.1109/IMMERSIVE.2016.7932384
  * Marriott et al. (2018) Kim Marriott, Falk Schreiber, Tim Dwyer, Karsten Klein, Nathalie Henry Riche, Takayuki Itoh, Wolfgang Stuerzlinger, and Bruce H Thomas. 2018.  _Immersive Analytics_. Vol. 11190.  Springer.  https://doi.org/10.1007/978-3-030-01388-2
  * McCarthy and Monk (1994) John C McCarthy and Andrew F Monk. 1994.  Measuring the quality of computer-mediated communication.  _Behaviour & Information Technology_ 13, 5 (1994), 311–319.  https://doi.org/10.1080/01449299408914611
  * Meier et al. (2007) Anne Meier, Hans Spada, and Nikol Rummel. 2007.  A rating scheme for assessing the quality of computer-supported collaboration processes.  _International Journal of Computer-Supported Collaborative Learning_ 2, 1 (2007), 63–86.  https://doi.org/10.1007/s11412-006-9005-x
  * Paul and Reddy (2010) Sharoda A Paul and Madhu C Reddy. 2010.  Understanding together: sensemaking in collaborative information seeking. In _Proceedings of the 2010 ACM conference on Computer supported cooperative work_. 321–330.  https://doi.org/10.1145/1718918.1718976
  * Pejsa et al. (2016) Tomislav Pejsa, Julian Kantor, Hrvoje Benko, Eyal Ofek, and Andrew Wilson. 2016.  Room2Room: Enabling life-Size Telepresence in a Projected Augmented Reality Environment. In _Proceedings of the 19th ACM conference on computer-supported cooperative work & social computing_. 1716–1725.  https://doi.org/10.1145/2818048.2819965
  * Piumsomboon et al. (2018) Thammathip Piumsomboon, Gun A Lee, Jonathon D Hart, Barrett Ens, Robert W Lindeman, Bruce H Thomas, and Mark Billinghurst. 2018\.  Mini-Me: An Adaptive Avatar for Mixed Reality Remote Collaboration. In _Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems_. 1–13.  https://doi.org/10.1145/3173574.3173620
  * Prouzeau et al. (2017) Arnaud Prouzeau, Anastasia Bezerianos, and Olivier Chapuis. 2017\.  Trade-offs between a vertical shared display and two desktops in a collaborative path-finding task. In _Proceedings of Graphics Interface 2017_. CHCCS.
  * Radu and Schneider (2019) Iulian Radu and Bertrand Schneider. 2019.  What Can We Learn from Augmented Reality (AR)? Benefits and Drawbacks of AR for Inquiry-based Learning of Physics. In _Proceedings of the 2019 CHI conference on human factors in computing systems_. 1–12.  https://doi.org/10.1145/3290605.3300774
  * Raziq and Maulabakhsh (2015) Abdul Raziq and Raheela Maulabakhsh. 2015.  Impact of Working Environment on Job Satisfaction.  _Procedia Economics and Finance_ 23 (2015), 717–725.  https://doi.org/10.1016/S2212-5671(15)00524-9
  * Roberts et al. (2006) Tom L Roberts, Paul B Lowry, and Paul D Sweeney. 2006\.  An Evaluation of the Impact of Social Presence Through Group Size and the Use of Collaborative Software on Group Member ”Voice” in Face-to-Face and Computer-Mediated Task Groups.  _IEEE Transactions on Professional Communication_ 49, 1 (2006), 28–43.  https://doi.org/10.1109/TPC.2006.870460
  * Robertson et al. (1998) George Robertson, Mary Czerwinski, Kevin Larson, Daniel C Robbins, David Thiel, and Maarten Van Dantzich. 1998.  Data Mountain: Using Spatial Memory for Document Management. In _Proceedings of the 11th Annual ACM Symposium on User Interface Software and Technology_. 153–162.  https://doi.org/10.1145/288392.288596
  * Russell (1980) James A Russell. 1980\.  A Circumplex Model of Affect.  _Journal of Personality and Social Psychology_ 39, 6 (1980), 1161.  https://doi.org/10.1037/h0077714
  * Sallnäs (2005) Eva-Lotta Sallnäs. 2005\.  Effects of Communication Mode on Social Presence, Virtual Presence, and Performance in Collaborative Virtual Environments.  _Presence: Teleoperators and Virtual Environments_ 14, 4 (08 2005), 434–449.  https://doi.org/10.1162/105474605774785253 arXiv:https://direct.mit.edu/pvar/article-pdf/14/4/434/1624506/105474605774785253.pdf
  * Satriadi et al. (2020) Kadek Ananta Satriadi, Barrett Ens, Maxime Cordeil, Tobias Czauderna, and Bernhard Jenny. 2020\.  Maps Around Me: 3D Multiview Layouts in Immersive Spaces.  _Proceedings of the ACM on Human-Computer Interaction_ 4, ISS (2020), 1–20.  https://doi.org/10.1145/3427329
  * Scott et al. (2004) Stacey D. Scott, M. Sheelagh T. Carpendale, and Kori Inkpen. 2004.  Territoriality in Collaborative Tabletop Workspaces _(CSCW ’04)_. Association for Computing Machinery, New York, NY, USA, 294–303.  https://doi.org/10.1145/1031607.1031655
  * Scott and Carpendale (2010) Stacey D Scott and Sheelagh Carpendale. 2010.  Theory of Tabletop Territoriality.  In _Tabletops-Horizontal Interactive Displays_. Springer, 357–385.  https://doi.org/10.1007/978-1-84996-113-4_15
  * Steed et al. (1999) Anthony Steed, Mel Slater, Amela Sadagic, Adrian Bullock, and Jolanda Tromp. 1999.  Leadership and collaboration in shared virtual environments. In _Proceedings IEEE Virtual Reality (Cat. No. 99CB36316)_. IEEE, 112–115.  https://doi.org/10.1109/VR.1999.756941
  * Tang et al. (2007) Anthony Tang, Carman Neustaedter, and Saul Greenberg. 2007\.  VideoArms: Embodiments for Mixed Presence Groupware.  In _People and Computers XX—Engage_. Springer, 85–102.  https://doi.org/10.1007/978-1-84628-664-3_8
  * Tang et al. (2006) Anthony Tang, Melanie Tory, Barry Po, Petra Neumann, and Sheelagh Carpendale. 2006.  Collaborative Coupling over Tabletop Displays. In _Proceedings of the SIGCHI Conference on Human Factors in Computing Systems_ _(CHI ’06)_. ACM, New York, NY, USA, 1181–1190.  https://doi.org/10.1145/1124772.1124950
  * Tang (1991) John C. Tang. 1991\.  Findings from observational studies of collaborative work.  _International Journal of Man-Machine Studies_ 34, 2 (1991), 143 – 160.  https://doi.org/10.1016/0020-7373(91)90039-A Special Issue: Computer-supported Cooperative Work and Groupware. Part 1.
  * Tromp et al. (1998) Jolanda Tromp, Adrian Bullock, Anthony Steed, Amela Sadagic, Mel Slater, and Emmanuel Frecon. 1998\.  Small Group Behaviour Experiments in the Coven Project.  _IEEE Computer Graphics and Applications_ 18, 6 (1998), 53–63.  https://doi.org/10.1109/38.734980
  * Tse et al. (2004) Edward Tse, Jonathan Histon, Stacey D. Scott, and Saul Greenberg. 2004.  Avoiding Interference: How People Use Spatial Separation and Partitioning in SDG Workspaces. In _Proceedings of the 2004 ACM Conference on Computer Supported Cooperative Work_ (Chicago, Illinois, USA) _(CSCW ’04)_. Association for Computing Machinery, New York, NY, USA, 252–261.  https://doi.org/10.1145/1031607.1031647
  * Tuddenham and Robinson (2009) Philip Tuddenham and Peter Robinson. 2009.  Territorial Coordination and Workspace Awareness in Remote Tabletop Collaboration. In _Proceedings of the SIGCHI Conference on Human Factors in Computing Systems_. 2139–2148.  https://doi.org/10.1145/1518701.1519026
  * Wang et al. (2017) Minhong Wang, Bo Cheng, Juanjuan Chen, Neil Mercer, and Paul A Kirschner. 2017.  The use of web-based collaborative concept mapping to support group learning and interaction in an online environment.  _The Internet and Higher Education_ 34 (2017), 28–40.  https://doi.org/10.1016/j.iheduc.2017.04.003
  * Weinel et al. (2011) Miriam Weinel, Maria Bannert, Jörg Zumbach, H. Ulrich Hoppe, and Nils Malzahn. 2011\.  A Closer Look on Social Presence as a Causing Factor in Computer-Mediated Collaboration.  _Comput. Hum. Behav._ 27, 1 (Jan. 2011), 513–521.  https://doi.org/10.1016/j.chb.2010.09.020
  * Wenskovitch and North (2020) John Wenskovitch and Chris North. 2020.  An Examination of Grouping and Spatial Organization Tasks for High-Dimensional Data Exploration.  _IEEE Transactions on Visualization and Computer Graphics_ (2020).  https://doi.org/10.1109/TVCG.2020.3028890
  * Willett et al. (2021) Wesley Willett, Bon Adriel Aseniero, Sheelagh Carpendale, Pierre Dragicevic, Yvonne Jansen, Lora Oehlberg, and Petra Isenberg. 2021\.  Perception! Immersion! Empowerment! Superpowers as Inspiration for Visualization.  _IEEE Transactions on Visualization and Computer Graphics_ 28, 1 (2021), 22–32.  https://doi.org/10.1109/TVCG.2021.3114844
  * Witmer and Singer (1998) Bob G Witmer and Michael J Singer. 1998.  Measuring Presence in Virtual Environments.  _Presence_ 7, 3 (1998), 225–240.  https://doi.org/MeasuringPresenceinVirtualEnvironments
  * Yoon et al. (2019) Boram Yoon, Hyung-il Kim, Gun A Lee, Mark Billinghurst, and Woontack Woo. 2019.  The Effect of Avatar Appearance on Social Presence in an Augmented Reality Remote Collaboration. In _2019 IEEE Conference on Virtual Reality and 3D User Interfaces (VR)_. IEEE, 547–556.  https://doi.org/10.1109/VR.2019.8797719
  * Zillner et al. (2014) Jakob Zillner, Christoph Rhemann, Shahram Izadi, and Michael Haller. 2014.  3D-Board: A Whole-body Remote Collaborative Whiteboard. In _Proceedings of the 27th annual ACM symposium on User interface software and technology_. 471–479.  https://doi.org/10.1145/2642918.2647393