#  Portobello: Extending Driving Simulation
from the Lab to the Road

Fanjun Bu  Cornell Tech2 W Loop RoadNew YorkU.S<EMAIL_ADDRESS>,  Stacey
Li  Cornell Tech2 W Loop RoadNew YorkU.S<EMAIL_ADDRESS>,  David Goedicke
Cornell Tech2 W Loop RoadNew YorkU.S<EMAIL_ADDRESS>,  Mark Colley
<EMAIL_ADDRESS>Ulm UniversityUlmGermany Cornell Tech2 W Loop RoadNew
YorkU.S. ,  Gyanendra Sharma  Woven by ToyotaLos AltosCaliforniaUSA
<EMAIL_ADDRESS>,  Hiroshi Yasuda  Toyota Research InstituteLos
AltosCaliforniaUSA<EMAIL_ADDRESS>and  Wendy Ju  Cornell Tech2 W
Loop RoadNew YorkU.S<EMAIL_ADDRESS>
###### Abstract.

In automotive user interface design, testing often starts with lab-based
driving simulators and migrates toward on-road studies to mitigate risks.
Mixed reality (XR) helps translate virtual study designs to the real road to
increase ecological validity. However, researchers rarely run the same study
in both in-lab and on-road simulators due to the challenges of replicating
studies in both physical and virtual worlds. To provide a common
infrastructure to port in-lab study designs on-road, we built a platform-
portable infrastructure, Portobello, to enable us to run twinned physical-
virtual studies. As a proof-of-concept, we extended the on-road simulator XR-
OOM with Portobello. We ran a within-subjects, autonomous-vehicle crosswalk
cooperation study (N=32) both in-lab and on-road to investigate study design
portability and platform-driven influences on study outcomes. To our
knowledge, this is the first system that enables the twinning of studies
originally designed for in-lab simulators to be carried out in an on-road
platform.

Figure 1. Portobello enables platform portability, so that the same study can
be run on in-lab (left) and on-road (right) driving simulation platforms.

The picture is split into two halves. In the left half, a black Fiat 500
stands in front of screens on which a driving scene is displayed. The driving
scene shows a pedestrian crossing on a crosswalk. In the Fiat 500, there is a
participant holding a smartphone in the passenger’s seat. In the right half, a
Toyota Prius stands at the real crosswalk. Again, a participant sits in the
passenger’s seat and wears a Varjo XR 1 video-passthrough headset. No real
pedestrian is visible but the headset shows a virtual pedestrian crossing.

Figure 2. Complete execution pipeline using the Portobello infrastructure.
During the design phase, researchers can drag and drop virtual objects on the
point cloud map. At run time, the LiDAR-based navigation system locates the
vehicle’s position within the same map and sends the relative vehicle position
to Unity. As a result, passengers wearing the video-see-through headset can
see virtual objects at their corresponding real-world locations.

This figure shows the execution pipeline using the Portobello infrastructure.
It is divided into two phases: the design phase and the runtime phase.
Interactions are staged in the point cloud map of the study route. The map has
to be recorded prior and is visualized in purple. During runtime, the
application uses LiDAR to localize itself within the same map, and stage
scenarios are rendered in mixed reality based on the vehicle’s real-time
location. The far-right figure shows the scene of the virtual pedestrian
crossing the real crosswalk. Note that the point cloud map is only rendered in
Unity in the design phase. At runtime, it is only loaded into ROS localization
algorithms for tracking purposes. It will not be rendered in Unity for mixed-
reality display.

##  1\. Introduction

Driving simulators play a critical role in human-centered automotive research
applications because they allow people to experience different driving
scenarios in a safe and repeatable fashion. Researchers have full control over
the simulation setup and can program complex events in virtual environments.
However, one of the major challenges for driving simulation has always been
the difficulty of replicating the inertial forces and vehicle dynamics present
in on-road driving (Pinto et al., 2008)–even the most high-end motion platform
driving simulators only replicate a fraction of the forces felt in real-world
driving (Greenberg et al., 2003). These forces are more critical in testing
automated vehicle (AV) scenarios, where participants are often given non-
driving-related tasks that keep them from looking at the screens of the
simulations (e.g., (Walch et al., 2019)). Without the physical sensation and
the visual engagement of the simulation environment, the immersiveness of the
environment drops, making the evaluation of novel user experience and
interaction techniques such as VEmotion (Bethge et al., 2021) or SoundsRide
(Kari et al., 2021) difficult. One way to address this problem is to
incorporate driving simulation into a real vehicle driving on real streets
(on-road simulators) (Baltodano et al., 2015).

On-road simulators are possible due to the maturity of XR, where digital
displays blend reality and virtuality to increase the level of a user’s
immersion. The use of XR-in-the-car using simulated or actual dash-mounted
heads-up displays have been explored by prior researchers such as Tonnis et
al. (2005), Kim and Dey (2009), Schall Jr et al. (2013), Ghiurãu et al.
(2020), McGill et al. (2022), Colley et al. (2021a), von Sawitzky et al.
(2019), Narzt et al. (2006), and Bark et al. (2014).

Despite this influx of on-road simulators under development, it remains
challenging to migrate studies from in-lab simulators to on-road simulators.
From a study design perspective, the key difference between in-lab and on-road
simulators is that with in-lab simulators, designers have full access not only
to the virtual vehicle but also to the virtual environment. The position and
orientation of every brick are available to the designer with high precision,
which makes event staging as simple as dragging-and-dropping modules into a
map. However, researchers do not have easy access to objects outside the
vehicle in on-road simulators. As a result, no on-road driving simulation
system to date considers the surrounding context outside the vehicle for event
staging, which limits the range of applicable studies. To replicate in-lab
simulation on on-road platforms, access to the out-of-vehicle surrounding
context is crucial.

In this paper, we describe a novel driving simulation infrastructure called
Portobello, which enables platform portability in virtual driving simulation
by incorporating localization technology and software from robotics. We define
platform portability as the ability to run the same study on different (in-lab
vs. on-road) platforms, an approach which we refer to as the twinning of
studies. For this demonstration, we extended XR-OOM, a state-of-art XR driving
simulation (Goedicke et al., 2022), to support on-road, location-based event
staging (see Figure 2). To validate Portobello’s platform portability, we
developed a within-subjects crosswalk-cooperation study (N=32) to be run on
both an in-lab fixed-based vehicle chassis driving simulator and the on-road
driving simulator built on top of the Portobello system (Walch et al., 2020).
As part of this work, we investigated how the different simulation platforms
may affect the design process and results of user studies. The primary
contribution of this work is the technical infrastructure system of
Portobello, as validated by the proof-of-concept study. In addition, we
provide a definition of platform-portability in virtual reality driving
simulation and contribute insights into the process needed to develop twinned
studies whose deployment is intended across multiple platforms. Finally, we
demonstrate the relative strengths of different study platforms in the course
of automotive research.

##  2\. Related Work

Driving simulation platforms are intended to be proxy environments that enable
researchers to conduct studies where real-world experiments are dangerous or
impossible. The standard for such platforms is face validity (Dole and Ju,
2019): when participants take a simulation seriously, researchers can have
greater faith that the study’s results will be applicable to the real world.
It is more important that the simulation allows participants to behave as if
they are in a realistic setting than it is for the simulation itself to
replicate reality in fine detail. In-lab and on-road simulators provide
different approximations of driving scenarios.

###  2.1. In-Lab Driving Simulators

In-lab driving simulators are used to test interactions between drivers and
the vehicle, the driving environment, and other in-world agents (Jansen et
al., 2022); simulation allows researchers to observe in-vehicle behaviors
safely, without physical risk to study participants (Tateyama et al., 2010).
Simulations can be implemented with low fidelity but still provide insights
into how drivers will react sitting in a real vehicle (Colley et al., 2021b).
When used to simulate outdoor environments, in-lab driving simulators can be
used to test the usage of augmented reality (AR) on roads (Tonnis et al.,
2005). In-lab simulators can also be used to test how other road users, such
as pedestrians and cyclists, interact with autonomous vehicles (Hou et al.,
2020; Mahadevan et al., 2019). However, simulator sickness remains a large
risk for in-lab driving simulators because a user’s vestibular senses do not
align with their visual senses when taking part in a simulation (Bertolini and
Straumann, 2016). Researchers have tried to address this with methods such as
aligning vehicular motion with VR content (Cho and Kim, 2020) or simulating
movement (Colley et al., 2022b; Hock et al., 2022), but even the highest-end
simulators replicate a fraction of the forces felt in normal on-road driving
(Greenberg et al., 2003).

###  2.2. XR On-Road Driving Simulators

XR technology enables simulating AV driving either by allowing users to move
through completely virtual spaces with real vehicle dynamics or by overlaying
virtual objects on top of real-time video footage of the surroundings to
increase the immersion of pre-programmed interfaces or interactions (Quinlan
et al., 2010; Yeo et al., 2019; Goedicke et al., 2022; Riegler et al., 2020;
Blissing et al., 2013; Hock et al., 2017; Goedicke et al., 2018; Sawano and
Okada, 2005). Recently, XR systems have been deployed on-road to take
advantage of the realistic road environment and vehicle dynamics. The XR-OOM
system developed by Goedicke et al. (2022) employed an XR headset for drivers
to drive through virtual, external obstacles in a parking lot. The MAXIM
system developed by Yeo et al. (2019) utilized a virtual reality headset
coupled with 360∘ cameras for subjects to experience an autonomous virtual
vehicle situated in a live environment created from live streamed 360∘ videos.
Ghiurãu et al. (2020) showed a proof-of-concept headset-based XR driving
experience revealing that original equipment manufacturers (OEMs) such as
Volvo use such technology. Finally, McGill et al. (2022) presented PassengXR,
an open-source toolkit to create passenger XR experiences. While providing XR
experiences, they did not compare their system to an in-lab simulator.

Although all the systems mentioned above track the vehicles’ dynamics to
accurately position virtual objects related to the participant, they do not
natively support high-precision interaction staging based on the surroundings
outside of the vehicle, which we refer to as surrounding context in this
paper. In essence, previous approaches are mostly concerned with aligning
virtual and physical motions, such as CarVR, but not the worlds themselves
(Hock et al., 2017). The authors of PassengXR introduced a hypothetical
application that requires high-precision alignment between virtual and
physical worlds, where passengers on a bus tour could view AR-style
information overlaid on historic buildings outside the vehicle, but their
demonstration was still carried out indoors without discussing how feasible it
was to implement such application on-road (McGill et al., 2022). Potential
issues for designing such applications with existing platforms are two-fold.
First, current systems make it challenging for designers to stage
interactions, where they need to manually locate event trigger positions (in
the bus tour example, buildings’ coordinates either in GPS coordinates or
relative coordinates in the vehicle frame) and program corresponding AR
information boards to be at those precise coordinates. Second, the GPS-based
tracking system may not provide enough accuracy for small-scale interaction,
especially in cities where buildings can disrupt GPS signals. The authors for
PassengXR commented that their ”approach prioritizes perception of motion over
location accuracy,” which is not best suited for location-based XR experience
(McGill et al., 2022). These hardships limit the capability of these on-road
systems to act as participant testing platforms compared to traditional in-lab
simulators. Most autonomous driving studies require sufficient staging and
surrounding context (Colley et al., 2022a). For example, driver-to-driver
communications mostly happen at intersections, and pedestrian-vehicle
interactions usually occur at crosswalks (Houtenbos, 2009; Lemmer et al.,
2020; Schneemann and Gohl, 2016; Krome et al., 2019). Our work makes high
prevision interaction staging possible while keeping the design process
intuitive.

###  2.3. Platform Portability

Previous research has focused on how to replicate on-road scenarios in in-lab
simulators, which is crucial when studying problems that are dangerous to
experiment on the road, such as near-collision scenarios and passive rail
level crossing (Demmel et al., 2011; Larue et al., 2018). However, compromises
are necessary to compensate for the lack of motion and sensory cues in in-lab
simulators, and little research has been conducted to reduce the performance
gap between on-the-road and in-simulators (Green, 2005). As such, merely
pursuing statistical significance with in-lab simulators may result in
overlooking issues of practical relevance in real-world contexts. (Green,
2005).

Because in-lab and on-road driving simulation environments offer different
strengths when it comes to control and realism of driving scenarios, it can be
desirable to run the same study in both when possible— an approach we call
twinning of studies — to understand how study results from different
environments relate to each other. Hammel et al. (2002) found that, when they
replicated an on-road study in a fixed-based simulator, participants’ eye-
scanning behavioral patterns were similar, which demonstrates fix-based
simulators’ ecological validity. In a systematic review of validation studies
featuring comparisons of driving simulation and on-road driving between 1977
and 2017, authors Wynne et al. (2019) found only 44 validation studies
comparing simulation to real driving. This is out of the 21,312 found by the
same researchers to be English-language publications of original research
having to do with driving simulation. Such studies are so rare that the 44
represent less than 0.25% of the published driving simulation research
surveyed by Wynne et al. (2019). They report that ”There was little
consistency in the dependent measures used to assess differences between the
simulator and on-road drive…Of particular concern is the fact that only half
of the driving simulators were found to be valid and some were valid for one
measure but not others.” They note that since policy, legislation, and
training are built off of simulator studies, a better understanding of which
aspects of simulated studies are likely to carry over to real road conditions,
and which are not critical (Wynne et al., 2019).

Frequently, we believe, the lack of validation studies is due to the
significant challenge of creating ”twinned” studies in both environments. The
advent of on-road mixed reality simulation (Baltodano et al., 2015; Yeo et
al., 2019, 2020; Goedicke et al., 2022; McGill et al., 2022) makes it possible
to bridge the divide using software events in the real world. However, no
system has yet ported the same study course, code, and event design from one
environment to the other. By making it possible to port studies developed for
in-lab simulators to be run on-road–what we call platform portability–we
improve the ability for automotive researchers to extend their in-lab studies
to the real road, and thereby improve the validity of simulation research.
This would improve the ability of researchers to validate their simulation
studies in on-road environments, as recommended by Wynne et al. (2019):
”Ideally this would see authors report empirical validation evidence for their
own simulator, and not relying on other simulators as support for validity.
Even if modeled on a previously validated simulator, each set-up is unique and
should be validated for those specifications.”

##  3\. Systems

Figure 3. The Portobello system uses a LiDAR-based navigation stack to
localize the vehicle’s runtime position within a given map. The location
information is transmitted to the Unity Desktop through ROS. The black dotted
line indicates a virtual divider appearing only in the headset.

This figure shows the schematic overview of Portobello. A top-down view of the
vehicle is given. The schematic view shows the researcher sitting in the
driver’s seat, and the participant wearing the Varjo XR 1 headset in the
passenger’s seat. A mini-display is mounted next to the driver displaying the
headset’s view. The SMARTTRACK3 system is attached to where usually the glove
compartment is located. The LiDAR connection is visualized and it is shown
that this is connected to Unity via ROS. The equipment that is part of the XR-
OOM system is highlighted in green, and the equipment that is part of the
Portobello infrastructure is highlighted in orange.

In this work, we present a study we developed meant to run on two driving
simulator environments: a lab-based driving simulator and an on-road XR
driving simulator equipped with our Portobello system. Here, we recap the
features of both environments and introduce the key adjustments made to
accommodate the Portobello system.

###  3.1. In-Lab Driving Simulator

Our in-lab fixed-base driving simulator features a modified Fiat 500 in front
of three projector screens (see Figure 1). The projector screens cover
participants’ visual field when they sit in either the driver’s seat or the
passenger’s. The three projectors are DLP-based and can produce an image with
low latency on the projector screens. The projectors are connected to the
computer over HDMI and use the TripleHead2Go to split one DisplayPort signal
into the three outputs.

The vehicle is coated with non-reflective material to reduce the backscatter
onto the projector screens, increasing the contrast of the projector screens.
ButtKicker haptic transducers are installed under the front seats to provide
realistic tactile feedback from road noise and the engine. The simulation
software is run by an Alienware Area-51 R4 computer with two NVIDIA GTX1080 in
SLI. The vehicle’s side mirrors are small digital displays rendered by the
same computer.

In our simulation software, the simulated vehicle is extended out of the
Genivi Vehicle Simulator, which has been widely used in driving simulation
studies (Alliance, 2021). The simulated vehicle uses standard Unity physics
wheel colliders with a built-in engine simulation. To enable autonomous
driving, we replace the original steering wheel input with a waypoint-based
navigation system.

###  3.2. On-Road XR System

To enable cross-environment study deployment, we use Portobello with the XR-
OOM system designed by Goedicke et al. (2022). In the original XR-OOM system,
tracking and positioning of virtual objects are managed by a ZED 2 camera (for
visual-SLAM) and the ART SMARTTRACK3111https://ar-
tracking.com/products/tracking-systems/smarttrack3/, accessed Jan 20, 2023
(for headset tracking within the vehicle) (Goedicke et al., 2022). An onboard
desktop running the Unity 3D game engine in version 2020.3.26f1 overlays
virtual objects on top of the ”passed-through” video of surroundings in the XR
headset.

###  3.3. Portobello System

In the Portobello system, we use a LiDAR-based navigation system on the car
roof driven by the Robot Operating System (ROS 1 Noetic) (Quigley et al.,
2009); this replaces the XR-OOM’s ZED 2 camera in front of the vehicle. The
Portobello navigation system features an Ouster OS-1 3D LiDAR with a built-in
Inertial Measurement Unit (IMU) and a ZED-F9R u-Blox GPS module. The
communication between the navigation system and the Unity Desktop is managed
through the ROS-TCP-Connector222https://github.com/Unity-Technologies/ROS-TCP-
Connector, accessed Oct 20, 2022 provided by Unity (Juliani et al., 2018).

Platform portability also drives augmentations to the XR-OOM software
structure. Whereas XR-OOM uses real-time visual SLAM to compute short-term
vehicle odometry, the Portobello LiDAR system enables global vehicle
localization within a given map. As a result, virtual objects’ positions are
no longer associated with the vehicle position directly as its children.
Rather, the vehicle and virtual objects share a common parent — the world
frame — which is introduced by a map of the environment. Instead of staging
virtual events around the vehicle, as is done in XR-OOM (and in all the other
XR-based driving simulation systems mentioned in (Ghiurãu et al., 2020)),
Portobello can stage virtual events in a static shared map through which the
vehicle drives. As Portobello replaces the car-centered reference frame with a
map-based global frame, out-of-vehicle virtual objects remain fixed with
respect to the map instead of to the arbitrary starting position of the
vehicle. We detail the map generation process and staging process in Section
4.

At runtime, Portobello’s LiDAR-based navigation system updates the position of
a virtual vehicle in Unity. The relative position and orientation between the
virtual vehicle and the participant’s headset are managed by SMARTTRACK3. The
virtual vehicle has the same shape as our research vehicle and is aligned with
the research vehicle throughout the ride. To simulate proper depth ordering,
the virtual vehicle is transparent with the alpha clipping option enabled. By
acting as a cutout shader, the virtual vehicle occludes virtual objects
outside. From the participants’ view, the virtual objects are occluded by the
real car they sit in. As we are rendering virtual events in XR over a large
area (half of an island), we cap the headset’s maximum rendering distance to
45 meters in order to improve motion parallax.

From a system design perspective, the computer running Portobello with ROS is
a separate computer from the computer running XR-OOM with Unity. In essence,
the communication between the ROS localization algorithm and Unity is achieved
through the ROS transform package (TF), which is a hierarchical tree structure
that tracks the relative position of multiple coordinate frames (map, LiDAR,
etc.). These coordinate frames can be accessed from Unity as game objects.
Isolating the system on a hardware level allows researchers to develop ROS and
Unity in their own environments and for one driving simulator to be swapped
out for another. Designers can focus on designing studies by placing objects
in the course map in Unity, rather than being concerned with low-level ROS
localization of objects. Another benefit of this practice is that the
Portobello system consumes no computational resources in the original on-road
platform at run time. The LiDAR-generated point-cloud map is rendered into
Unity in the design phase. At runtime, the on-road platform computer imports
the Unity map at start-up, and does not require the additional computational
resources that would be needed to manage the point cloud map data. (This does
mean that the Unity map might be missing physical features that change between
the design phase and runtime.)

##  4\. Enabling Portability of Study Design Using Common Models

To run twinned studies across platforms, we must keep portability in mind
during the design of the study. We outline the necessary components to
guarantee equivalent performance in cross-environment twinned studies and
detail our system pipeline to showcase the components’ connections using our
system.

###  4.1. Course Design

On-road simulators are limited by real-world road infrastructures. Based on
the study focus, researchers should carefully consider test routes to ensure
efficiency and reproducibility.

As our study focused on interactions at crosswalks, we chose the southern loop
of Roosevelt Island as our study course. Most of the 0.9-mile long route is
single-track and has a high density of crosswalks, 15 in total. The drive
takes about 8 minutes. The route has no traffic lights and overlaps with two
bus lines.

###  4.2. Map

A map (model of the study area) is the starting point of event design, and its
precision and resolution greatly affect design complexity and quality. In
cross-environment twinned studies, a map is a bridge between simulation and
the real world, and it is also the shared common ground on every simulator
platform.

To generate a high-quality map, we used the LiDAR-based navigation system to
scan the entire test area. Specifically, we ran the real-time LiDAR-inertial
odometry package (LIO-SAM) developed by Shan et al. (2020) to create a true-
to-scale point cloud mapping of the study area. We drove through the testing
route multiple times to ensure loop closure. The resulting map is a
monochromatic digital twin that includes over one million points and captures
the test area in fine detail. Researchers can manually contextualize the point
cloud map in Unity with colored assets and use the map as the background in
in-lab simulators.

###  4.3. Event Design

Staging events along the planned course requires researchers to consider two
major questions: where and when events occur. Staging is relatively easy for
in-lab simulations, where agents’ movement and speed profiles can be carefully
controlled to guarantee timing and location. In this section, we discuss how
we stage events using on-road simulators.

####  4.3.1. Planned Events

##### Where?

A one-to-one scaled map is necessary to plan events for on-road simulations
because the vehicle will drive through the real world during the study. Any
scaling or shifting on the map will cause significant errors at runtime. With
a loaded digital twin in Unity, researchers can drag and drop virtual agents
and objects just like they would for in-lab simulators.

##### When?

Timing of events for on-road simulators can be controlled through the
placement of collision-based triggers. Triggers in Unity are colliders that
trigger events upon external contact. For example, invisible triggers can be
placed at some distance $x$ in front of a virtual traffic light. Once the
vehicle collides with the trigger, the traffic light starts changing colors,
and participants should react accordingly. The distance $x$ governs the start
of the interaction, which essentially affects the maximum response time for
participants.

Of course, a vehicle on-road cannot ”collide” with the virtual collider in
simulation. In our system, the LiDAR-based navigation system synchronizes the
position of the real vehicle with a virtual vehicle in the digital twin
through the $hdl\\_localization$ and ROS-TCP-Connector packages in real-time
(Koide et al., 2019). (By employing the $hdl_{l}ocalization$ algorithm, the
offset between the vehicle’s actual and estimated locations is maintained
within a 0.2-meter range.) As the real vehicle drives through the world, the
virtual vehicle simultaneously moves through the digital twin to trigger
planned events. Note that this is the same virtual vehicle mentioned in
section 3.2 for proper depth ordering.

Figure 4. Our study area for on-road simulator on Roosevelt Island. The pre-
determined test route is highlighted in blue. The crosswalks with staged
interactions are highlighted in the red bounding boxes. The start and end
locations are denoted by the green and purple dots, respectively.

This figure shows a top-down satellite image of Anonymous. The pre-determined
test route is highlighted in blue. It spans an approximate rectangle.

####  4.3.2. Unplanned Events

We define events outside the simulation, which researchers have no control
over, as unplanned events. In in-lab simulations, unplanned events are rare
and typically caused by system failures or external interruptions. However, in
on-road simulators, unplanned events are common and can even be valuable for
ecological validity; they let researchers know if their findings are robust to
real-world variation. Findings that are only true in the tightly controlled
environment of a study have little practical application.

Nevertheless, researchers must factor in potential unplanned events during the
design phase to ensure safety and preserve meaningful study results. Unplanned
events come in different forms, from unexpected appearances of pedestrians to
weather changes. For instance, in our study, we encountered the following
unplanned events: real pedestrians and geese crossing the street, other
vehicles passing the research vehicle from the bicycle lanes, and rain.

###  4.4. Platform Measures

Another aspect of platform portability is whether researchers can obtain the
same set of measures from twinned studies. While measurements should be
equally attainable across all simulation platforms, the characteristics of
each simulator naturally encourage and discourage different sets of measures.

####  4.4.1. Behavioral Response

Behavioral responses refer to the participants’ elicited behavior during the
study. Examining behavioral responses is crucial when studying interactions
between drivers, vehicles, and infrastructure (Ruscio et al., 2017). For
example, Jansen et al. (2023) was interested in differences in participants’
responses to different stimuli in automotive user interfaces. With appropriate
sensors, collecting behavioral responses in in-lab and on-road simulators is
possible.

####  4.4.2. Performance Response

Paas and Van Merriënboer (1993) define performance as efficiency in completing
tasks. We distinguish performance from behavioral responses based on the
availability of ground truth. Researchers can collect performance responses
during the study when participants are assigned tasks with general guidelines
and standards. One example of performance response is the lateral vehicle
position when the driver is distracted (Kountouriotis and Merat, 2016). While
extra sensors might be needed for on-road simulators to obtain vehicle-related
performance measures (e.g., vehicle speed, acceleration, or trajectories are
not easily attainable in on-road simulators as they are in in-lab simulators),
we do not anticipate significant challenges in obtaining performance responses
in both in-lab and on-road simulators.

####  4.4.3. Survey Response

Surveys can be conducted through different devices (pen and paper, tablets) in
various formats (interviews, multiple choice, open-ended questions). In
portable study design, it is important to consider the timing of the survey.
One natural advantage of in-lab simulators over on-road simulators is the
ability to pause at any point of the study and prompt participants with
questions in situ (De Winter et al., 2012). On-road simulators cannot be
paused easily, so surveys need to be planned so that participants can take
them when it is safe to do so.

###  4.5. Additional Instrumentation

Detailed runtime recording of the environment is crucial for post-facto data
analysis, particularly of unplanned events, for both on-road and in-lab
simulators. Some measures need to be recorded differently in the different
platforms and translated. For example, geo-location data from the on-road
vehicle GPS must be correlated with the virtual world coordinates in the lab
simulator. Head orientation and gaze direction obtained from the XR headset in
the on-road simulator can be correlated with camera-tracked head-pose in the
in-lab driving simulator.

##  5\. Twinning of Studies

As proof of concept that we can run the same study design in the lab and on-
road (twinning of studies), we conducted a within-subjects experiment with
N=32 participants (under IRB#1806008105). We described the cross-platform
deployment of twinned studies and compared the differences in between. We
counterbalanced the experiment conditions, where half of the participants
experienced the study in the indoor simulator first and in the on-road
simulator second, and the other 16 participants experienced the simulators in
the reverse order.

We employed the in-lab and on-road driving simulators to run twinned Crosswalk
Cooperation studies, which we adapted from Walch et al. (2020). In this
previous study, Walch et al. used an in-lab driving simulator to evaluate the
usability of a novel car UI and staged interactions around a driving loop. In
this current work, we are not seeking to validate the results of the
previously published study; we are not expecting or arguing that our study
results would be the same. Rather, we are merely using this study design to
evaluate the capacities and key influences of both systems.

###  5.1. Study Setup

####  5.1.1. Protocol

The experiment is a within-participants experiment design; each participant
experiences two study sessions in counterbalanced order. In one session,
participants experience the crosswalk cooperation study in the in-lab driving
simulator. Afterwards, they fill out a post-session questionnaire, which
collects information on their experience with the simulator. In the other
session, the participants are escorted to the curbside and experience the
crosswalk cooperation study in the on-road driving simulator. Afterwards, they
fill out the same post-session questionnaire. Finally, they fill out a post-
study questionnaire, which collects information on their perceived differences
between the two simulators.

During each simulation session, participants are seated in the front passenger
seat and informed that they will experience automated driving: The vehicle
will stop at all crosswalks automatically. The vehicle will proceed
autonomously when road conditions are clear (e.g. crosswalks without virtual
pedestrians) and will ask for input from the participant via a smartphone
interface, on how to proceed in unclear situations (e.g. virtual pedestrians
walking towards crosswalks). Specifically, the vehicle will ask, ”Is now safe
to proceed?” on the phone interface while waiting at the crosswalk; when the
participant feels it is safe to proceed, they press the ”Proceed” button, and
the car resumes its predefined route. If the participants decide it is not
safe to proceed, they should wait until it is safe to press the button. The
researcher driving the vehicle monitors when the participant presses the
button and manually proceeds with the course if it is safe.

####  5.1.2. Differences in Simulator Setup

As much as possible, we maintained identical setups for the twinned studies.
The key difference was that during the on-road simulation session, the
researcher driving the car was mindful of the actual road conditions before
proceeding with driving.

To maintain the autonomous driving narrative, the researcher driving during
the on-road simulation was masked by a black divider in the video pass-through
headset (shown in Figure 3). A similar black divider was also installed in the
in-lab simulator to maintain setup parity. During pilot sessions, we found it
difficult to disguise the on-road vehicle as an AV due to differences in sound
profile. In complex on-road conditions, the sound of pressing the pedal and
rotating the steering wheel broke the illusion quickly. Thus, we decided to
inform the participants of the divider’s purpose and that there was a real
researcher in the car with them operating the vehicle.

While a virtual map environment is required in in-lab simulation, the point
cloud map is not required for on-road simulation due to the benefit of the
video-pass-through headset. Therefore, after the event staging phase, we
disabled the point cloud rendering in Unity to save computation power.

####  5.1.3. Scenarios

We recreated four scenarios from the original Crosswalk Cooperation study by
Walch et al. (2020). Each scenario was engineered so that pedestrian
interactions would only happen at crosswalks. In each scenario, virtual
pedestrians interacted with each other on the sidewalks near the crosswalks.
In half of the scenarios, one pedestrian walked to the stop sign and crossed
the street after giving clear body language signals that they intended to
cross (looking left and right). In the other half of the scenarios, the
pedestrian stopped at or walked past the crosswalk. In our version of the
Crosswalk Cooperation study, we constrained our study area to the southern
loop of Roosevelt Island.

###  5.2. Participants

Out of our 32 participants ranging from 20 to 47 years old (age: M = 27.38
$\pm$ 5.92), 19 participants identified themselves as male, 11 as female, and
two as non-binary. Six participants had experience with AV simulations and/or
AV research, and five had experience with commercialized AVs (Tesla, demos at
car shows). The others had little experience with AVs.

###  5.3. Study Measures

After each session, participants filled out a post-session questionnaire,
which collected information on their experience with the simulator. After
completing both sessions and corresponding questionnaires, participants filled
out a the post-study questionnaire, which collected information on their
perceived differences between the two simulators.

We also recorded video and audio for all sessions run in both in-lab and on-
road driving simulations to investigate participants’ behavioral responses.
For the in-lab driving simulator, a go-pro camera was pointed towards the
participant to record their upper body. The simulated virtual environment was
recorded using screen recording software. For the on-road driving simulator,
similar to the setup in the in-lab simulator, a camera was mounted in the
glove compartment to record the participant’s upper body. An additional camera
was mounted near the rear mirror facing forward to record the road condition
ahead. The participant’s XR view (video pass-through with overlay) was also
recorded using the Varjo Base software.

##  6\. Results and Analysis

We analyzed the video footage and the questionnaires from both sessions with
the goal of investigating the influences of both driving simulators on the
user study experience and understanding what results from one platform predict
for results on the other. Our evaluation of Portobello is based on being able
to run and gather comparable results from the studies on both the in-lab and
on-road platforms and not on the originality, validity, or significance of the
study itself.

Figure 5. Participants’ ratings of each simulator on a five-point Likert scale
across seven distinct dimensions where 1=low and 5=high.

This figure shows the ratings of each simulator on a five-point scale across
anxiety, system enjoyment, discomfort, safety, immersion, car performance, and
trust. The actual values are given in the text.

###  6.1. Study Results

####  6.1.1. Measures

To compare the overall participant measures in the study on both simulators,
we asked the participants (N = 32) to rate their feelings of anxiety, safety,
and trust on a 5-point Likert scale for each of the platforms. We designed the
questions based on the questionnaire used in Walch et al. (2020)’s original
Crosswalk Cooperation study. We ran a Bayesian factor analysis on the captured
measures with the null hypothesis that there is no difference between the
platforms (Kass and Raftery, 1995) with R version 4.3.2 and the BayesFactor
package (Morey, 2022) using Jeffreys-Zellner-Siow (JZS) priors, i.e., the
default non-informative Jeffreys prior. Interpretations were made according to
Jeffreys (1998). All packages were up to date in November 2023. Since we have
limited data, and we are not interested in how different variables (ratings)
interact with each other, we chose to report single variate analysis over
multivariate analysis. Nonetheless, our claims hold under multivariate
analysis as well. We will provide R script for both analyses.

One participant left multiple answers empty, so we dropped their results in
the following analysis.

##### Anxiety

More participants reported reduced anxiety with the in-lab simulator (M = 1.32
$\pm$ 0.60) than with the outdoor simulator (M = 1.45 $\pm$ 0.72). We found
moderate evidence (BF = 0.32) in favor of the null model, suggesting that
there is no significant difference in the anxiety generated by the simulators.

##### Safety

Participants considered the in-lab simulator (M = 4.58 $\pm$ 0.76) safer than
the on-road simulator (M = 4.16 $\pm$ 0.86). We found moderate evidence (BF =
3.53) against the null model, suggesting a moderate difference in favor of the
in-lab simulator.

This may have been because the in-lab simulator’s roads did not involve any
real vehicles or people, e.g., P6 explained that ”… there were more actual
obstacles…to take into account [in the outdoor simulator] whereas the in-lab
[simulator] had a preset number.”

##### Trust

Overall, participants reported their trust in the simulated autonomous driving
to be higher in the on-road simulator (M = 3.71 $\pm$ 0.86) than in the in-lab
simulator (M = 3.52 $\pm$ 1.06). We found moderate evidence (BF = 0.31) in
favor of the null model, suggesting there is no significant difference in
trust.

The preference for the on-road simulator may have been influenced by people’s
perception of each vehicle’s performance. At the same time, this may have been
a breakdown in face validity. For example, P14 said, ”One of the survey
questions asked ’how much do you trust this car’ and I think I forgot to
pretend that this was an AI driving the car when answering that…” Participants
were informed during the on-road simulator session that a researcher would be
driving the car; their trust rating might have been an indicator that they
trusted the driver to obey local traffic laws, rather than an indicator of
their trust in the simulated autonomous driving, as we had intended.

####  6.1.2. Cooperation Behavior

From recorded videos, we analyzed 32 participants’ cooperative behaviors with
the vehicle at crosswalks. Video recordings from the same session were
synchronized before the analysis. One researcher watched recordings for each
participant and labeled their behaviors in terms of the timing of cooperation
behaviors for each scenario. The researcher then noted down the behavioral
changes, if any, for each participant between different simulators.

15 participants cooperated with the vehicle perfectly in both simulation
platforms in all scenarios, where they waited until the pedestrians fully
crossed the street or waited for clear non-crossing signals before instructing
the vehicle to proceed. During the first crossing scenario, 11 participants
instructed the vehicle to proceed while the pedestrian was about to cross;
this led to six virtual collisions. One of these participants made the vehicle
run over the same virtual pedestrian in both in-lab and on-road simulators.
Three participants did not wait for any virtual pedestrian to cross in both
in-lab and on-road simulators. Six participants who had cooperated perfectly
in the first session made different decisions in the second session; they
chose not to wait for the crossing participants when they believed it was
safe, and one of them ended up running over the virtual pedestrian. One
participant who ignored both crossing pedestrians in in-lab simulators waited
for one of the crossing pedestrians in the on-road simulator. Overall, 14
participants made different decisions in the second session from the first
session.

We feel compelled to point out that the fact that participants made poor
crossing decisions is not a sign that the study or the driving simulation
platforms were designed poorly; instead, it is precisely these sorts of
outcomes that indicate the necessity for simulation platforms that enable
studies with virtual pedestrians to be conducted prior to putting real
pedestrians in harm’s way. We do not expect this means that participants would
run over real people in subsequent tests with real cars, but this does point
out that participants are aware that they are not exposed to real danger in
driving simulators (Jamson and Jamson, 2010); impatience and lack of
conscientiousness amongst some portion of the population are factors that any
cooperative autonomous driving system would have to account for.

####  6.1.3. Ordering Effect

While we counterbalanced our study, we noticed some differences in cooperation
behavior that may be attributed to the ordering of simulators. Of the six
participants who made mistakes in the first crossing scenario, four
participants were experiencing the on-road driving simulator. We hypothesize
that the on-road driving simulator is more overwhelming than the in-lab
simulator to familiarize the participants with the study setup. P28 mentioned
in their post-study questionnaire that ”Visual noise in outdoor sim [made]
task completion more difficult but [was] more realistic in that regard.” It is
worth mentioning that the three participants who made the mistake in the first
crossing scenario had limited (e.g., they had only their learner’s permits) to
no driving experience.

###  6.2. Simulation Evaluation Results

####  6.2.1. Simulator Measures

To compare the overall experience between the two simulators, we also asked
participants to rate their feelings about car performance, system enjoyment,
discomfort, and immersion on a 5-point Likert scale. These questions were
based on questionnaires used during the validation process of Goedicke et al.
(2022)’s XR-OOM system. We again ran a Bayesian factor analysis on the
captured measures with the null hypothesis that there was no difference
between the platforms (Kass and Raftery, 1995).

##### Car Performance

Participants considered the autonomous vehicle in the on-road simulator (M =
4.35 $\pm$ 0.71) to perform better than in the in-lab simulator (M = 3.42
$\pm$ 0.96). We found extreme evidence (BF = 1.67e+04) against the null model,
suggesting a significant difference in favor of the on-road simulator.
Participants felt that the in-lab car simulator did not appear to drive
smoothly and stopped rather abruptly at times and thus thought that the
driving felt more natural in the on-road simulator.

##### System Enjoyment

More participants reported increased levels of system enjoyment with the in-
lab simulator (M = 3.84 $\pm$ 0.97) than with the outdoor simulator (M = 3.03
$\pm$ 0.98). We found extreme evidence (BF = 225.00) against the null model,
suggesting a significant difference in favor of the in-lab simulator.
Participants seemed to have preferred the in-lab simulator because of the
graphics quality and comfort. For example, when asked to describe their
experience with both simulators in the post-study questionnaire, P14 said,
”The pedestrians seemed to ”appear out of nowhere” in the [on-road simulator],
whereas it seemed like they were always part of the scenery in the [in-lab]
simulator (i.e., came into view naturally in the simulator). … [The] turns in
the [in-lab simulator’s] road felt unnatural/like the scenery was clicked and
dragged in front of my eyes, instead of me moving through the scenery.”

##### Discomfort

Participants reported less discomfort with the in-lab simulator (M = 1.71
$\pm$ 0.86) than the on-road simulator (M = 2.35 $\pm$ 1.14). We found
anecdotal evidence (BF = 2.24) against the null model, suggesting a mild
difference in favor of the in-lab simulator. For the in-lab simulator,
discomfort mainly arose from the unrealistic vehicle dynamics. For the on-road
simulator, many comments were related specifically to the XR headset (n = 11),
which people found heavy and uncomfortable. The occasional misalignment of
virtual objects caused by bumps in the road also induced a considerable amount
of motion discomfort. For example, P13 said, ”…both [simulators] cause some
discomfort, but I think the outdoor one is more uncomfortable due to the
[pass-through] being very [shaky] and more motion-sickness-inducing.”

##### Immersion

As shown in Figure 5, participants considered the in-lab simulator (M = 4.00
$\pm$ 0.77) to be more immersive than the on-road simulator (M = 3.74 $\pm$
1.06). We found anecdotal evidence (BF = 0.0.40) in favor of the null model.
However, in the post-study questionnaire, when both simulators were presented
on the same Likert scale, the numbers of participants in favor of either
simulator were the same. Figure 6 shows that 14 participants thought the in-
lab simulator was more immersive, and 14 thought the on-road simulator was
more immersive. Three participants considered both simulators equally
immersive.

Two participants reported verbally during the on-road session that it was
difficult to distinguish virtual from real pedestrians.

Many participants reported difficulty with the weight and technical maturity
of the XR headset in the outdoor simulator, which impeded their attention and
may have contributed to its lower immersion rating. P8 said, ”headset jitter
made visuals blurry, which impaired decision-making/attention.” P2 said,
”[Putting] something on my head is so uncomfortable. I couldn’t [focus on] the
view or be relaxed. The [in-lab] simulator is not so real[,] but I could be so
relaxed.”

####  6.2.2. In simulator behavior

We weighed the observed and reported behavior of participants to assess
differences in behavior introduced by each platform.

##### Natural Head Motions

We noticed that some participants’ head pose motion patterns were different
when experiencing the two simulation platforms. When sitting in the in-lab
static simulator, participants tilted their heads mostly during staged events
to track the motion of virtual pedestrians. For the rest of the ride, they
faced forward. However, for the on-road simulator, the participants’ head
motions were more varied, and participants naturally looked to view the
surroundings more often.

##### Decision-making

For the crosswalk cooperation study, the key measure was the participant’s
decision-making around whether the vehicle should go or not. Participants
reflected that the level of complexity and severity in decision-making was
greater in the on-road simulator. The staged interactions were identical in
both sessions, but researchers had less control over surrounding factors
during the on-road sessions. Other road users, including real pedestrians,
other vehicles, and geese native to Roosevelt Island complicated the staged
scenarios.

P5 said, ”I think that in the on-road simulator, I was a little more nervous
because real people were on the street. The in-lab simulator does not deal
with real people, so any mistake I make does not have as much weight.” P28
said, ”Outdoor provides [a] more generally immersive feeling and is the only
one in which I can realistically feel unsafe, which is a positive in terms of
validity. Visual noise in outdoor sim makes task completion more difficult but
is more realistic in that regard.” P31 said, ”I felt anxious after saying to
proceed and wondering how the pedestrians would move afterwards.” In contrast,
P11 reflected: ”Perhaps due to its nature of being [an in-lab] simulator, I
felt at ease at all times.” P20 reflected: ”The [polished nature] of the [in-
lab] simulator makes the experience [feel] more entertaining/performative
rather than realistic. … The [on-road] simulator feels more like a functional
approximation of an actual autonomous driving car, while the indoor simulator
feels more like a fun, polished experience.” P21 reflected: ”I felt much
safer/less anxious in the indoor simulator which also probably means it was
less realistic.” P29 reflected: ”[The] indoor [simulator] seems to have lower
stakes, even though it was the same virtual people.

Figure 6. After experiencing both simulators, participants also directly
compared the in-lab simulator and the on-road simulator on the same Likert
scale. Participants’ responses are shown as histograms.

This figure shows the histograms of the participants’ responses to different
questions comparing the two simulators on 5-point Likert scales. The questions
were: Which vehicle did you feel safer in? Here, the tendency was clear
towards the in-lab simulator. Which vehicle made you feel more immersed? Here,
the tendency wasn’t that clear with 8 claiming totally in-lab but 10 stating
more on-road. Which vehicle performed better? Here, the tendency was clear
toward the on-road simulator. Which vehicle did you trust? Here, 10
participants stated “same” but also 8 stated on-road. Which vehicle did you
feel more comfortable in? Here, the tendency was clear towards the in-lab
simulator.

##  7\. Technical Validation

Since the Portobello system is an infrastructure meant to be used in
conjunction with existing platforms, the technical performance largely depends
on the system on top of Portobello. Therefore, we investigated the change in
performance of the on-road platform after adapting Portobello. Rendering a
total of 12 virtual pedestrians, the headset ran at 60 FPS consistently with a
display latency of around 35ms, which was on par with the original XR-OOM
system. The localization frequency was 10 Hz, limited by the 10Hz LiDAR. The
fact that there is no change in performance is expected since the Portobello
system operates on a computer separated from the original XR-OOM system.

##  8\. Discussion

###  8.1. LiDAR-based vs. GPS/IMU-based systems

Our LiDAR-based Portobello system resolves the two challenges posed by
traditional GPS/IMU-based systems in surrounding context-based interaction
staging: localization accuracy and design hardships. LiDAR-based localization
systems work reliably in cities where buildings serve as landmarks instead of
GPS signal blockers. A point cloud 3D map of the environment generated from
LiDAR-based SLAM algorithm saves designers from staging using hardcoded
coordinates, and simplifies the design process to drag-and-drop within the
map. Lastly, we want to point out that the sensors are not mutually exclusive.
We can fuse in GPS data as an additional data source into the LiDAR-based
algorithm if necessary.

###  8.2. Platform Portability Challenges

The primary goal of this research effort is to establish a proof-of-concept
demonstration of platform portability through our Portobello system. Platform
portability is necessary to run twinned studies across different platforms,
which is desirable because the in-lab simulator can help establish causal
differences across experimental conditions using well-controlled studies, and
the on-road simulator can help validate the ecological validity of such study
results when the same study conditions are moved into the less-controlled
environment of the real world. By incorporating robotics mapping, sensing, and
localization capability in the Portobello system, we can set up twinned
studies to run in different environments. We believe that this is of relevance
to any of the simulators discussed in Section 2.2, which could be deployed on
top of Portobello.

####  8.2.1. Randomness in the Wild

For on-road simulators, randomness persists throughout the entire study. For
example, during mapping, we generated a snapshot of the test area. While major
landmarks such as buildings and land topology will not change significantly
over time, the map also captures transient objects (e.g., parked cars). Such
randomness may cause inaccuracy in real-time localization.

During the study, unplanned events were the most salient form of randomness.
Unplanned occurrences and interruptions from the real world may increase
immersion. P19 reported that ”the extra people and cars in the outdoor study
made the experience feel more immersive and interesting.” However, randomness
also brings concerns regarding study reproducibility. We tried to eliminate
the co-occurrence of planned and unplanned events by staging events in less
populated areas. In general, we encourage researchers to plan for all possible
unplanned events during the study design phase.

####  8.2.2. Event Timing

One major challenge we faced with the on-road simulator is the trigger and
timing of staged events, and we foresee such a problem persisting in future
similar studies. For in-lab simulators, vehicle speed and travel distances can
be coded in detail. For on-road simulators, it can be difficult to maintain
the same speed curve as in-lab simulators due to obstacles and unplanned
events. In our study, we expect the vehicle to stop at the crosswalk
simultaneously as a virtual pedestrian reaches the stop sign on the sidewalk.
The researcher who operates the car has access to a mini-display monitoring
the location of the virtual pedestrians and adjusting vehicle speed
accordingly. However, we have noticed that participants made different
decisions across platforms due to event timing differences.

The timing misalignment between the simulators is the natural consequence of
the intentional difference between running studies in a controlled environment
(the lab) and an uncontrolled environment (the real world). We are arguing
that it is desirable to run both kinds of studies and that it is easier to do
this if platform portability exists. The in-lab simulator is more suitable for
quantitative analysis, and the on-road simulator is more suitable for
qualitative analysis of the factors that complicate the outcomes learned from
the more controlled in-lab simulator.

###  8.3. Platform Effects

Our study results were intended to help us understand whether and how our
twinned studies were the same across the two platforms and to help us
understand the differences across the platforms, a comparison made possible by
the Portobello system. For research purposes, it would be best if the study
results between the two platforms were similar (i.e., that the platform
effects were negligible), or at least that the results were biased
consistently across the platforms (i.e., that the platform effects were
predictable).

From the study results, we can see that around half of the participants made
similar decisions in both the in-lab simulator and the on-road system.
Notably, one participant made the same mistake in each simulator. The fact
that some participants made different decisions in different simulators
indicates that participants’ decision-making was not affected by their
existing knowledge about the study. Even if they were aware of the
pedestrians’ behaviors from the first session, they took into account the
event timing difference and made the most appropriate decision at the moment.

Some notable differences between the platforms were centered around the
participant’s decision-making behavior and their resulting trust in the
autonomous driving system. In some ways, there were indications that the on-
road simulator might have failed to maintain face validity for at least one of
the participants; their trust rating might have shown their trust in the
research driver rather than their trust in the simulated autonomous driving,
as we had intended. On the other hand, many participants reported greater
weight in the decision-making around whether the vehicle should go or not go
in the on-road simulator, a sign that face validity is higher in the real-
world environment.

While our participants favored the experience of the in-lab simulation
platform on a whole, most of their complaints pertained to aspects of the XR
system–the weight of the headset, the jitter in the display–which are likely
to improve with advancing technology. It seems like the naturalism of the on-
road environment is more likely to yield naturalistic behaviors than the in-
vehicle environment, and hence this platform can help industrial and academic
researchers better understand how people will engage with autonomous vehicle
technologies in the real world.

###  8.4. Limitations and Future Work

Some limitations in the study results are inherent to a driving simulation
study. In this section, we focus on limitations in the design and execution of
studies using the XR driving simulation system augmented by Portobello that
should be accounted for and discuss future developments that could improve
such systems.

####  8.4.1. Real-time adjustment of Depth Ordering

Although we render the occlusion of virtual objects caused by the research
vehicle (e.g. we do not render the pedestrians over the front pillar of the
car), the current system does not provide the same occlusion for runtime
dynamic objects. If a bus drives between the research vehicle and the location
where the virtual pedestrians are supposed to appear, for example,
participants would see the virtual pedestrians in front of the bus. To correct
the depth order, future systems could use real-time LiDAR scans of the
environment.

####  8.4.2. Pedestrian Appearance

We cap the maximum rendering distance for on-road virtual objects for
technical reasons; distant virtual objects are less salient and require better
alignment between the in-vehicle and out-of-vehicle reference frames to be
placed believably in the mixed-reality view. The artifice of having
pedestrians suddenly appear, however, may affect study results. One
participant (P24) said, ”I think being able to see the passengers from further
away in the in-lab simulator made a big difference because, by the time I got
to the intersection, it was easier to anticipate their movements.” Future
technology could improve the motion parallax issues, enabling longer rendering
distances and smoother transitions when virtual objects approach the rendering
threshold.

####  8.4.3. Headset Discomfort

Many participants complained about the bulkiness and narrow field-of-view of
the headset. This platform-level discomfort was pronounced enough that it
drowned out our ability to measure experiential aspects (system enjoyment,
discomfort) of the autonomous driving scenario. While the weight and
limitations of the XR headsets are beyond our control, we believe that
anticipated advancements in XR headset technology are necessary to use these
systems in experiments wherein the experiential aspects of automated driving
are critical.

####  8.4.4. Simulating Autonomous Driving on Road

In our current study, we informed the participants that there was an actual
driver behind the scenes in the on-road simulator. The driver’s maneuver sound
easily breaks the AV illusion for participants who have previous experience
with AV. Future research can benefit from disguising the driver by playing the
recorded AV sound profile during acceleration and deceleration to cover the
driver’s maneuver sound.

####  8.4.5. Consistent Driver Performance

While the same researcher operated the vehicle throughout the study, and
attempted to maintain a consistent driving style from one run to the next,
there were natural variations in the driving, in part in response to
uncontrolled environmental factors. While it is not feasible nor desirable to
force the driver to operate the vehicle the same way for all sessions on the
road, future systems should collect data to enable later analysis of variance.

##  9\. Conclusion

Driving simulations can be used to create scenarios for driving interactions,
which enable researchers to better understand how people will behave and
respond to future driving scenarios. In this work, we present the Portobello
system, an on-road driving simulation infrastructure that enables platform
portability. By advancing the capabilities of driving simulators, we can
better anticipate what aspects of driving interaction will work well or
poorly.

This paper outlines the first-ever deployment of twinned studies across in-lab
and on-road simulators. We found that participants preferred the experience of
the in-lab simulator but displayed more natural head movements in the on-road
simulator; they also reported that the decisions made in the on-road system
carried more weight. Based on our findings, we suggest researchers working in
driving simulations also take the twinning of studies approach: they should
first run studies within a controlled, in-lab environment to collect
statistical measures and form hypotheses and then port their studies to a
less-controlled, on-road simulator and test their hypotheses in a more
complex, realistic environment.

This experiment looking at the platform-driven influences on study outcomes
demonstrates the utility of platform portability, as the same study design was
able to be run both in-lab and on-road. This was made possible by the
Portobello system’s common model and vehicle localization; using robotics
mapping and localization technology, we were able to capture surrounding
environments for study and event staging for our on-road simulator. We
anticipate that Portobello will advance the state of open-source and
accessible driving simulation by extending the reach and translational
capabilities of VR and XR driving simulation systems and thus, in turn, enable
wider-scale development and testing of safe driving systems.

## Open Science

The source code for Portobello has been made publicly available. It can be
accessed via the following link: https://github.com/FAR-Lab/Portobello.git.

###### Acknowledgements.

This research was made possible by sponsorship by Toyota Research Institute
and Woven by Toyota. Studies were conducted under Cornell Tech’s IRB Protocol
#IRB0008105. This work was also supported by a fellowship of the German
Academic Exchange Service (DAAD).

## References

  * (1)
  * Alliance (2021) GENIVI Alliance. 2021.  Genivi Vehicle Simulator.  https://github.com/GENIVI/genivi-vehicle-simulator.
  * Baltodano et al. (2015) Sonia Baltodano, Srinath Sibi, Nikolas Martelaro, Nikhil Gowda, and Wendy Ju. 2015.  The RRADS platform: a real road autonomous driving simulator. In _Proceedings of the 7th International Conference on Automotive User Interfaces and Interactive Vehicular Applications_. ACM, New York, NY, USA, 281–288.
  * Bark et al. (2014) Karlin Bark, Cuong Tran, Kikuo Fujimura, and Victor Ng-Thow-Hing. 2014.  Personal navi: Benefits of an augmented reality navigational aid using a see-thru 3d volumetric hud. In _Proceedings of the 6th International Conference on Automotive User Interfaces and Interactive Vehicular Applications_. ACM, New York, NY, USA, 1–8.
  * Bertolini and Straumann (2016) Giovanni Bertolini and Dominik Straumann. 2016.  Moving in a moving world: a review on vestibular motion sickness.  _Frontiers in neurology_ 7 (2016), 14.
  * Bethge et al. (2021) David Bethge, Thomas Kosch, Tobias Grosse-Puppendahl, Lewis L. Chuang, Mohamed Kari, Alexander Jagaciak, and Albrecht Schmidt. 2021.  VEmotion: Using Driving Context for Indirect Emotion Prediction in Real-Time. In _The 34th Annual ACM Symposium on User Interface Software and Technology_ (Virtual Event, USA) _(UIST ’21)_. Association for Computing Machinery, New York, NY, USA, 638–651.  https://doi.org/10.1145/3472749.3474775
  * Blissing et al. (2013) Björn Blissing, Fredrik Bruzelius, and Johan Ölvander. October 23-25, 2013.  Augmented and mixed reality as a tool for evaluation of vehicle active safety systems. In _RSS2013 Road Safety and Simulation-International Conference_. Aracne, Rome, Italy, 7 pages.
  * Cho and Kim (2020) Hyung-jun Cho and Gerard J. Kim. 2020.  RoadVR: Mitigating the Effect of Vection and Sickness by Distortion of Pathways for In-Car Virtual Reality. In _Proceedings of the 26th ACM Symposium on Virtual Reality Software and Technology_ _(VRST ’20)_. Association for Computing Machinery, New York, NY, USA, 1–3.  https://doi.org/10.1145/3385956.3422115
  * Colley et al. (2022a) Mark Colley, Julian Britten, Simon Demharter, Tolga Hisir, and Enrico Rukzio. 2022a.  Feedback Strategies for Crowded Intersections in Automated Traffic — A Desirable Future?. In _Proceedings of the 14th International Conference on Automotive User Interfaces and Interactive Vehicular Applications_ (Seoul, Republic of Korea) _(AutomotiveUI ’22)_. Association for Computing Machinery, New York, NY, USA, 243–252.  https://doi.org/10.1145/3543174.3545255
  * Colley et al. (2021a) Mark Colley, Benjamin Eder, Jan Ole Rixen, and Enrico Rukzio. 2021a.  Effects of Semantic Segmentation Visualization on Trust, Situation Awareness, and Cognitive Load in Highly Automated Vehicles. In _Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems_ (Yokohama, Japan) _(CHI ’21)_. Association for Computing Machinery, New York, NY, USA, Article 155, 11 pages.  https://doi.org/10.1145/3411764.3445351
  * Colley et al. (2021b) Mark Colley, Pascal Jansen, Enrico Rukzio, and Jan Gugenheimer. 2021b.  Swivr-car-seat: Exploring vehicle motion effects on interaction quality in virtual reality automated driving using a motorized swivel seat.  _Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies_ 5, 4 (2021), 1–26.
  * Colley et al. (2022b) Mark Colley, Pascal Jansen, Enrico Rukzio, and Jan Gugenheimer. 2022b.  SwiVR-Car-Seat: Exploring Vehicle Motion Effects on Interaction Quality in Virtual Reality Automated Driving Using a Motorized Swivel Seat.  _Proc. ACM Interact. Mob. Wearable Ubiquitous Technol._ 5, 4, Article 150 (dec 2022), 26 pages.  https://doi.org/10.1145/3494968
  * De Winter et al. (2012) Joost De Winter, Peter M van Leeuwen, Riender Happee, et al. 2012\.  Advantages and disadvantages of driving simulators: A discussion. In _Proceedings of measuring behavior_ , Vol. 2012. 8th.
  * Demmel et al. (2011) Sebastien Demmel, Gregoire S Larue, Andry Rakotonirainy, and Dominique Gruyer. 2011.  Using cooperative vehicle systems to collect near collision incidents events: a simulation study. In _2nd International Workshop on Traffic Data Collection & its Standardisation_.
  * Dole and Ju (2019) Lorin Dole and Wendy Ju. 2019.  Face and Ecological Validity in Simulations: Lessons from Search-and-Rescue HRI. In _Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems_ (Glasgow, Scotland Uk) _(CHI ’19)_. Association for Computing Machinery, New York, NY, USA, 1–8.  https://doi.org/10.1145/3290605.3300681
  * Ghiurãu et al. (2020) Florin-Timotei Ghiurãu, Mehmet Aydın Baytaş, and Casper Wickman. 2020.  ARCAR: On-Road Driving in Mixed Reality by Volvo Cars. In _Adjunct Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology_ (Virtual Event, USA) _(UIST ’20 Adjunct)_. Association for Computing Machinery, New York, NY, USA, 62–64.  https://doi.org/10.1145/3379350.3416186
  * Goedicke et al. (2022) David Goedicke, Alexandra W.D. Bremers, Sam Lee, Fanjun Bu, Hiroshi Yasuda, and Wendy Ju. 2022.  XR-OOM: MiXed Reality Driving Simulation with Real Cars for Research and Design. In _Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems_ (New Orleans, LA, USA) _(CHI ’22)_. Association for Computing Machinery, New York, NY, USA, Article 107, 13 pages.  https://doi.org/10.1145/3491102.3517704
  * Goedicke et al. (2018) David Goedicke, Jamy Li, Vanessa Evers, and Wendy Ju. 2018.  Vr-oom: Virtual reality on-road driving simulation. In _Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems_. ACM, New York, NY, USA, 1–11.
  * Green (2005) Paul Allan Green. 2005.  _How driving simulator data quality can be improved_.  University of Michigan Transportation Research Institute, Human Factors Division.
  * Greenberg et al. (2003) Jeff Greenberg, Bruce Artz, and Larry Cathey. 2003.  The effect of lateral motion cues during simulated driving.  _Proceedings of DSC North America_ (2003), 8–10.
  * Hammel et al. (2002) Kim R Hammel, Donald L Fisher, and Anuj K Pradhan. 2002.  Verbal and spatial loading effects on eye movements in driving simulators: A comparison to real world driving. In _Proceedings of the Human Factors and Ergonomics Society Annual Meeting_ , Vol. 46. SAGE Publications, Los Angeles, CA, USA, 2174–2178.
  * Hock et al. (2017) Philipp Hock, Sebastian Benedikter, Jan Gugenheimer, and Enrico Rukzio. 2017.  CarVR: Enabling In-Car Virtual Reality Entertainment. In _Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems_ (Denver, Colorado, USA) _(CHI ’17)_. Association for Computing Machinery, New York, NY, USA, 4034–4044.  https://doi.org/10.1145/3025453.3025665
  * Hock et al. (2022) Philipp Hock, Mark Colley, Ali Askari, Tobias Wagner, Martin Baumann, and Enrico Rukzio. 2022.  Introducing VAMPIRE – Using Kinaesthetic Feedback in Virtual Reality for Automated Driving Experiments. In _Proceedings of the 14th International Conference on Automotive User Interfaces and Interactive Vehicular Applications_ (Seoul, Republic of Korea) _(AutomotiveUI ’22)_. Association for Computing Machinery, New York, NY, USA, 204–214.  https://doi.org/10.1145/3543174.3545252
  * Hou et al. (2020) Ming Hou, Karthik Mahadevan, Sowmya Somanath, Ehud Sharlin, and Lora Oehlberg. 2020.  Autonomous Vehicle-Cyclist Interaction: Peril and Promise. In _Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems_ (Honolulu, HI, USA) _(CHI ’20)_. Association for Computing Machinery, New York, NY, USA, 1–12.  https://doi.org/10.1145/3313831.3376884
  * Houtenbos (2009) Maura Houtenbos. 2009.  Interacting drivers at intersections: what can make them more safe and more efficient. In _Proceedings of the 21st International Technical Conference on the Enhance Safety Vehicles_. Citeseer, Citeseer.
  * Jamson and Jamson (2010) Samantha L Jamson and A Hamish Jamson. 2010.  The validity of a low-cost simulator for the assessment of the effects of in-vehicle information systems.  _Safety Science_ 48, 10 (2010), 1477–1483.
  * Jansen et al. (2023) Pascal Jansen, Julian Britten, Thilo Segschneider, Mark Colley, and Enrico Rukzio. 2023.  AutoVis: Enabling Mixed-Immersive Analysis of Automotive User Interface Interaction Studies. In _Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems_ (Hamburg, Germany) _(CHI ’23)_. Association for Computing Machinery, New York, NY, USA, 15 pages.  https://doi.org/10.1145/3544548.3580760 conditionally accepted.
  * Jansen et al. (2022) Pascal Jansen, Mark Colley, and Enrico Rukzio. 2022.  A Design Space for Human Sensor and Actuator Focused In-Vehicle Interaction Based on a Systematic Literature Review.  _Proc. ACM Interact. Mob. Wearable Ubiquitous Technol._ 6, 2, Article 56 (jul 2022), 51 pages.  https://doi.org/10.1145/3534617
  * Jeffreys (1998) Harold Jeffreys. 1998.  _The theory of probability_.  OUP Oxford.
  * Juliani et al. (2018) Arthur Juliani, Vincent-Pierre Berges, Ervin Teng, Andrew Cohen, Jonathan Harper, Chris Elion, Chris Goy, Yuan Gao, Hunter Henry, Marwan Mattar, et al. 2018\.  Unity: A general platform for intelligent agents.  _arXiv preprint arXiv:1809.02627_ (2018).
  * Kari et al. (2021) Mohamed Kari, Tobias Grosse-Puppendahl, Alexander Jagaciak, David Bethge, Reinhard Schütte, and Christian Holz. 2021.  SoundsRide: Affordance-Synchronized Music Mixing for In-Car Audio Augmented Reality. In _The 34th Annual ACM Symposium on User Interface Software and Technology_ (Virtual Event, USA) _(UIST ’21)_. Association for Computing Machinery, New York, NY, USA, 118–133.  https://doi.org/10.1145/3472749.3474739
  * Kass and Raftery (1995) Robert E Kass and Adrian E Raftery. 1995.  Bayes factors.  _Journal of the American statistical association_ 90, 430 (1995), 773–795.
  * Kim and Dey (2009) SeungJun Kim and Anind K Dey. 2009.  Simulated augmented reality windshield display as a cognitive mapping aid for elder driver navigation. In _Proceedings of the SIGCHI Conference on Human Factors in Computing Systems_. ACM, New York, NY, USA, 133–142.
  * Koide et al. (2019) Kenji Koide, Jun Miura, and Emanuele Menegatti. 2019.  A portable three-dimensional LIDAR-based system for long-term and wide-area people behavior measurement.  _International Journal of Advanced Robotic Systems_ 16, 2 (2019), 1729881419841532.
  * Kountouriotis and Merat (2016) Georgios K Kountouriotis and Natasha Merat. 2016.  Leading to distraction: Driver distraction, lead car, and road environment.  _Accident Analysis & Prevention_ 89 (2016), 22–30.
  * Krome et al. (2019) Sven Krome, David Goedicke, Thomas J Matarazzo, Zimeng Zhu, Zhenwei Zhang, JD Zamfirescu-Pereira, and Wendy Ju. 2019.  How people experience autonomous intersections: taking a first-person perspective. In _Proceedings of the 11th International Conference on Automotive User Interfaces and Interactive Vehicular Applications_. ACM, New York, NY, USA, 275–283.
  * Larue et al. (2018) Grégoire S Larue, Christian Wullems, Michelle Sheldrake, and Andry Rakotonirainy. 2018.  Validation of a driving simulator study on driver behavior at passive rail level crossings.  _Human factors_ 60, 6 (2018), 743–754.
  * Lemmer et al. (2020) Markus Lemmer, Stefan Schwab, and Sören Hohmann. 2020.  Driver interaction at intersections: A hybrid dynamic game based model. In _2020 IEEE International Conference on Systems, Man, and Cybernetics (SMC)_. IEEE, IEEE, New York, NY, USA, 2269–2276.
  * Mahadevan et al. (2019) Karthik Mahadevan, Elaheh Sanoubari, Sowmya Somanath, James E. Young, and Ehud Sharlin. 2019.  AV-Pedestrian Interaction Design Using a Pedestrian Mixed Traffic Simulator. In _Proceedings of the 2019 on Designing Interactive Systems Conference_ (San Diego, CA, USA) _(DIS ’19)_. Association for Computing Machinery, New York, NY, USA, 475–486.  https://doi.org/10.1145/3322276.3322328
  * McGill et al. (2022) Mark McGill, Graham Wilson, Daniel Medeiros, and Stephen Anthony Brewster. 2022.  PassengXR: A Low Cost Platform for Any-Car, Multi-User, Motion-Based Passenger XR Experiences. In _Proceedings of the 35th Annual ACM Symposium on User Interface Software and Technology_ (Bend, OR, USA) _(UIST ’22)_. Association for Computing Machinery, New York, NY, USA, Article 2, 15 pages.  https://doi.org/10.1145/3526113.3545657
  * Morey (2022) Richard Morey. 2022.  _Richarddmorey/BayesFactor_.  Retrieved 6 September, 2022 from https://github.com/richarddmorey/BayesFactor
  * Narzt et al. (2006) Wolfgang Narzt, Gustav Pomberger, Alois Ferscha, Dieter Kolb, Reiner Müller, Jan Wieghardt, Horst Hörtner, and Christopher Lindinger. 2006.  Augmented reality navigation systems.  _Universal Access in the Information Society_ 4 (2006), 177–187.
  * Paas and Van Merriënboer (1993) Fred GWC Paas and Jeroen JG Van Merriënboer. 1993.  The efficiency of instructional conditions: An approach to combine mental effort and performance measures.  _Human factors_ 35, 4 (1993), 737–743.
  * Pinto et al. (2008) Maria Pinto, Viola Cavallo, and Théophile Ohlmann. 2008.  The development of driving simulators: Toward a multisensory solution.  _Le travail humain_ 71, 1 (2008), 62–95.
  * Quigley et al. (2009) Morgan Quigley, Ken Conley, Brian Gerkey, Josh Faust, Tully Foote, Jeremy Leibs, Rob Wheeler, Andrew Y Ng, et al. 2009\.  ROS: an open-source Robot Operating System. In _ICRA workshop on open source software_ , Vol. 3. Kobe, Japan, Japan, 5.
  * Quinlan et al. (2010) Michael Quinlan, Tsz-Chiu Au, Jesse Zhu, Nicolae Stiurca, and Peter Stone. 2010.  Bringing simulation to life: A mixed reality autonomous intersection. In _2010 IEEE/RSJ International Conference on Intelligent Robots and Systems_. IEEE, New York, NY, USA, 6083–6088.  https://doi.org/10.1109/IROS.2010.5651993
  * Riegler et al. (2020) Andreas Riegler, Andreas Riener, and Clemens Holzmann. 2020.  A research agenda for mixed reality in automated vehicles. In _19th International Conference on Mobile and Ubiquitous Multimedia_. ACM, New York, NY, USA, 119–131.
  * Ruscio et al. (2017) Daniele Ruscio, Luca Bascetta, Alessandro Gabrielli, Matteo Matteucci, Dedy Ariansyah, Monica Bordegoni, Giandomenico Caruso, and Lorenzo Mussone. 2017.  Collection and comparison of driver/passenger physiologic and behavioural data in simulation and on-road driving. In _2017 5th IEEE International Conference on Models and Technologies for Intelligent Transportation Systems (MT-ITS)_. IEEE, New York, NY, USA, 403–408.
  * Sawano and Okada (2005) Hiroaki Sawano and Minoru Okada. 2005.  A car-navigation system based on augmented reality.  In _ACM SIGGRAPH 2005 Sketches_. ACM, New York, NY, USA, 119–es.
  * Schall Jr et al. (2013) Mark C Schall Jr, Michelle L Rusch, John D Lee, Jeffrey D Dawson, Geb Thomas, Nazan Aksan, and Matthew Rizzo. 2013.  Augmented reality cues and elderly driver hazard perception.  _Human factors_ 55, 3 (2013), 643–658.
  * Schneemann and Gohl (2016) Friederike Schneemann and Irene Gohl. 2016.  Analyzing driver-pedestrian interaction at crosswalks: A contribution to autonomous driving in urban environments. In _2016 IEEE intelligent vehicles symposium (IV)_. IEEE, IEEE, New York, NY, USA, 38–43.
  * Shan et al. (2020) Tixiao Shan, Brendan Englot, Drew Meyers, Wei Wang, Carlo Ratti, and Rus Daniela. 2020.  LIO-SAM: Tightly-coupled Lidar Inertial Odometry via Smoothing and Mapping. In _IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_. IEEE, IEEE, New York, NY, USA, 5135–5142.
  * Tateyama et al. (2010) Yoshisuke Tateyama, Hiroki Yamada, Junpei Noyori, Yukihiro Mori, Keiichi Yamamoto, Tetsuro Ogi, Hidekazu Nishimura, Noriyasu Kitamura, and Harumi Yashiro. 2010.  Observation of drivers’ behavior at narrow roads using immersive car driving simulator. In _Proceedings of the 9th ACM SIGGRAPH Conference on Virtual-Reality Continuum and its Applications in Industry_ _(VRCAI ’10)_. Association for Computing Machinery, New York, NY, USA, 391–396.  https://doi.org/10.1145/1900179.1900263
  * Tonnis et al. (2005) Marcus Tonnis, Christian Sandor, Gudrun Klinker, Christian Lange, and Heiner Bubb. 2005.  Experimental evaluation of an augmented reality visualization for directing a car driver’s attention. In _Fourth IEEE and ACM International Symposium on Mixed and Augmented Reality (ISMAR’05)_. IEEE, New York, NY, USA, 56–59.
  * von Sawitzky et al. (2019) Tamara von Sawitzky, Philipp Wintersberger, Andreas Riener, and Joseph L. Gabbard. 2019.  Increasing Trust in Fully Automated Driving: Route Indication on an Augmented Reality Head-up Display. In _Proceedings of the 8th ACM International Symposium on Pervasive Displays_ (Palermo, Italy) _(PerDis ’19)_. Association for Computing Machinery, New York, NY, USA, Article 6, 7 pages.  https://doi.org/10.1145/3321335.3324947
  * Walch et al. (2020) Marcel Walch, Stacey Li, Ilan Mandel, David Goedicke, Natalie Friedman, and Wendy Ju. 2020.  Crosswalk cooperation: A phone-integrated driver-vehicle cooperation approach to predict the crossing intentions of pedestrians in automated driving. In _12th International Conference on Automotive User Interfaces and Interactive Vehicular Applications_. ACM, New York, NY, USA, 74–77.
  * Walch et al. (2019) Marcel Walch, Marcel Woide, Kristin Mühl, Martin Baumann, and Michael Weber. 2019.  Cooperative Overtaking: Overcoming Automated Vehicles’ Obstructed Sensor Range via Driver Help. In _Proceedings of the 11th International Conference on Automotive User Interfaces and Interactive Vehicular Applications_ (Utrecht, Netherlands) _(AutomotiveUI ’19)_. Association for Computing Machinery, New York, NY, USA, 144–155.  https://doi.org/10.1145/3342197.3344531
  * Wynne et al. (2019) Rachael A. Wynne, Vanessa Beanland, and Paul M. Salmon. 2019.  Systematic review of driving simulator validation studies.  _Safety Science_ 117 (2019), 138–151.  https://doi.org/10.1016/j.ssci.2019.04.004
  * Yeo et al. (2019) Dohyeon Yeo, Gwangbin Kim, and SeungJun Kim. 2019.  MAXIM: Mixed-reality Automotive Driving XIMulation. In _2019 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct)_. IEEE, IEEE, New York, NY, USA, 460–464.
  * Yeo et al. (2020) Dohyeon Yeo, Gwangbin Kim, and Seungjun Kim. 2020.  Toward immersive self-driving simulations: Reports from a user study across six platforms. In _Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems_. ACM, New York, NY, USA, 1–12.