# Gotta Go Fast: Measuring Input/Output Latencies of Virtual Reality 3D
Engines for Cognitive Experiments

Taeho Kang, Christian Wallraven

###### Abstract

Virtual Reality (VR) is seeing increased adoption across many fields. The
field of experimental cognitive science is also testing utilization of the
technology combined with physiological measures such as electroencephalography
(EEG) and eye tracking. Quantitative measures of human behavior and cognition
process, however, are sensitive to minuscule time resolutions that are often
overlooked in the scope of consumer-level VR hardware and software stacks. In
this preliminary study, we implement VR testing environments in two prominent
3D Virtual Reality frameworks (Unity and Unreal Engine) to measure latency
values for stimulus onset execution code to Head-Mount Display (HMD) pixel
change, as well as the latency between human behavioral response input to its
registration in the engine environment under a typical cognitive experiment
hardware setup. We find that whereas the specifics of the latency may further
be influenced by different hardware and software setups, the variations in
consumer hardware is apparent regardless and report detailed statistics on
these latencies. Such consideration should be taken into account when
designing VR-based cognitive experiments that measure human behavior.

###### Index Terms:

Virtual reality, VR, EEG, cognitive experiments, human behavioral, behavioral
measurements, eye-tracking, latency, response time

##  I Introduction

The idea of utilizing naturalistic stimuli in cognitive experiments is
increasingly gaining traction, and its importance has been recognized in an
increasing number of studies [1, 2, 3, 4, 5, 6, 7]. 3D environments such as
Virtual and Mixed Reality (VR/MR) can provide an excellent platform for
implementing experimental paradigms where immersive, interactive and
naturalistic stimuli presentation is desired [8, 9, 10]. Especially for VR,
behavioral and cognitive investigative experiments performed in virtual
reality have the advantage of being able to control and manipulate
environmental variables that in real-life settings would be nearly impossible
to control [11, 9]. Possibly due to this, virtual reality has seen increased
utilization in the area of behavioral and cognitive investigations, from
simple behavioral experiments [12], to neuroimaging studies [13], to even
timing sensitive studies involving physiological signal measurements such as
EEG [14, 15, 16, 17, 18, 19, 20, 21].

Due to the nature of cognitive processes of interest, behavioral and cognitive
studies investigating timing-critical brain processes have been historically
sensitive to latency in experimental hardware and software [22, 23]. It has
been suggested, however, that specialized behavioral input devices may not be
as crucial for even time-critical experiments, as the variability from human
behavior itself is generally larger in scale than the input lag occurring from
individual hardware devices [24]. For ease of experimental equipment
acquisition that may in turn be relevant to the easy replication of studies,
usage of adequately performant consumer hardware may be preferable to limited
costly specialized equipment for behavioral input.

Nonetheless, especially in studies measuring time-sensitive behaviors, there
is importance in measuring expected latency in hardware and software setups
used for experimental paradigms. Wimmer et al. [25] used opto-couplers to
measure latency of 36 different serial input devices connected to a Raspberry
Pi device and formed probability distribution models for each of the devices,
and reported different input latency distributions per device; suggesting the
need for measuring input latency levels in interactive experimental setups
that make use of serial device based user input device. Furthermore, due to
higher graphical computation requirements than conventional displays arising
from not only generally higher refresh rates but also other factors such as
needing to render twice for stereo vision, the current state of VR hardware
suffers from latency greater than of conventional user interface devices [26].

A final point of consideration in this context concerns the use of higher-
level APIs for generating three-dimensional, interactive environments that
afford realistic levels of sensory realism and interactivity. While it is
possible to create well-controlled low-level stimuli with relative ease in
computer graphics languages, the amount of work necessary to create
environments which, for example, contain objects that interact with each other
in a physically-realistic fashion from scratch is beyond the capabilities of
standard cognitive and behavioral research labs. For this reason, many
researchers have increasingly turned to 3D game engines for creating such
environments. One drawback of this development is that these engines offer
only a reduced degree of control over their timing internals given that much
of the behind-the-scenes calculation remains hidden from the API user
(examples include the calculation of graphic primitives, the determination of
collisions in physics-aware simulations, etc.). This raises the question of
how much timing accuracy and precision is possible in game engine simulation
programming environments. In this context, Wiesing et al. [27] have measured
stimuli duration and onset measurements in Unreal Engine with a dedicated
response pad and reported increased average latency, compared to dedicated
cognitive experiment software such as Psychopy and Psychtoolbox. While Unreal
Engine as a serious 3D engine has been used for VR based behavioral
experiments [28], possibly due to the relative ease of implementation in
comparison to the former, Unity Engine has been seeing increased applications
in cognitive experiments [29, 30, 31, 32, 33, 34, 17]. While they are suited
for similar purposes, due to differences in implementation detail Unity and
Unreal Engine often exhibit different behaviors even when the same effect is
intended, especially in frame and I/O related latency performances [35].

In light of these considerations, to ultimately implement and execute
experiments investigating brain processes in a naturalistic VR environment, we
deem it worth investigating the expected latency values for hardware and
software setups that would (commonly) be used in VR-based behavioral
experiments. In this study, we aim to achieve this by utilizing a measuring
apparatus with oscilloscopes, as well as a bare-bone experimental paradigm
implemented in two widely-used VR capable 3D engines, Unity Engine and Unreal
Engine. In the bare bone paradigm, we create stimulus onsets that send trigger
codes to the measuring apparatus before the actual displaying of stimuli is
performed in the Head Mounted Device (HMD), and measure latency between the
the onset code and the actual pixel change in the HMD. Furthermore, we measure
the latency between a physical input action on consumer-level user interface
hardware (keyboard) that can be used for behavioral response, and the
registration of the input in the 3D engines. Lastly, we measure the latency
between the physical input action and the pixel change from the resultant
feedback code execution.

##  II Materials and Methods

###  II-A Experiment Design

Figure 1: Diagram of the experimental data collection paradigm trial design.

We were interested in the measurement of the following events: 1) latency
between stimulus onset code execution in the 3D engine and the actual HMD
pixel changes as the stimulus was presented (Stim2Disp), 2) latency between
participant behavioral response by a keypress and the code execution performed
immediately upon the register of the key event in the 3D engine (Key2Led), and
3) latency between participant behavioral response by keypress and the pixel
change in HMD caused by the 3D engine code that presents a visual feedback
stimulus upon the key event register (Key2Disp). To measure these events, we
implemented experimental paradigms capable of measuring these events in both
Unity Engine and Unreal Engine, as can be seen in Figure 1.

We decided to measure both 3D Engines as their implementations differ, as well
as the scripting implementation language for user code: Unity utilizes C# for
this, whereas Unreal uses C++. Both engines have seen usage in cognitive
experimental designs in 3D environments (see Introduction).

The 3D VR experiment environment consists of a basic 3D spherical object that
can move around based on the user’s input. Upon a stimulus onset code or
recognition of a participant’s behavioral response by a keyboard, a chess-
board grid of black and white covers the entire screen for one frame, then
with the colors inverted for another frame. The experiment code embodies a
bare-bone basic form of cognitive experimental paradigm using 3D Engines, and
we send programmatic triggers upon stimulus onset and behavioral input
registration, as one would for experiments requiring high temporal resolution
like in EEG or other cognitive experiments that involve some form of time
series physiological measurement. We measure the latency of the above 3
scenarios (Stim2Disp, Key2Disp, Key2Led) by sending programmatic markers to
Arduino upon stimulus onset code execution, and behavioral response
registration in the 3D engine, which triggers an LED. Furthermore, we use
pressure sensors and photodiodes connected to the Arduino board to have
quantifiable measures of when participant behavior and stimuli display happen
in the real world.

####  II-A1 Unity-specific setup

For Unity, we implemented the paradigm in Unity Engine version 2019.4.20f.
Following Unity Engine’s manual on order of execution for event functions
(seehttps://docs.unity3d.com/Manual/ExecutionOrder.html), the FixedUpdate()
function handles ticks on computations focused on the physics engine
calculations, and may render more than once per rendered frame depending on
the computation load and settings, while the Update() function ticks per every
frame that is rendered (after FixedUpdate() calls are complete). The events
are called serially, and between the Update() call and the actual rendering
calls of the scene on display there are several other calls; as such, in the
interest of sending the trigger for stimulus onset in experimental behavior
measurement database as close to the onset of the actual stimulus in the
display, it is preferable to send the marker code sometime after the Update()
call but before the actual display rendering process. We achieve this by
calling a coroutine that waits execution of sending stimulus onset triggers
until the rendering computation is complete, but before the displaying is
performed:

⬇

private IEnumerator HandleTriggers(GameObject grid1, GameObject grid2)

{

// because the first EOF is still on the same frame, wait

yield return new WaitForEndOfFrame();

// send LED signal RIGHT BEFORE FRAME RENDERING

duino_port.Write(sevent_keypress, 0, 1);

yield return new WaitForEndOfFrame();

// the stimuli is two frames of visual stimuli

// as such, disable the first frame and enable the other

grid1.SetActive(false);

grid2.SetActive(true);

grid2.SetActive(false);

yield return null;

}

Figure 2: Code for sending trigger to Arduino in Unity

Furthermore, as the FixedUpdate() call executes at the beginning of the game
tick and executes at a higher rate, it is preferable to process keyboard input
events (i.e. behavioral response) in there. For Key2Led and Key2Disp events,
the function handling keyboard input events can also call for the co-routines
to send markers subsequently:

⬇

private void FixedUpdate()

{

// updating the speed for the cueball

mvVec3.Set(mvVec.x, 0.0f, mvVec.y);

// actual position update and physicis

// are handled internally by unity

rb.AddForce(mvVec3 * mvspeed);

// check update for keypresses in fixedupdate

// for fastest response possible

if (keypressed)

{

mrklsl_keypress.Write("Keypress");

// handle stimulus triggering

HandleTriggers(gridquad_1, gridquad_2);

}

// reset keypress flag

keypressed = false;

}

Figure 3: Unity code for sending trigger on behavioral response

####  II-A2 Unreal-specific setup

For the Unreal Engine, we implemented the paradigm in version 4.26. Unreal
Engine logic ticks are separated in tick groups (PrePhysics, DuringPhysics,
PostPhysics, and PostUpdateWork) that are serially run as per the
documentation (https://docs.unrealengine.com/5.1/en-US/actor-ticking-in-
unreal-engine/). Processing of user input is handled in the PrePhysics segment
of the ticking, and as such binding the input of specific keys to a method
that sends trigger is sufficient. By calling stimulus onset triggers in a code
that is executed on the PostPhysics or PostUpdateWork segment, we can also set
it to be as close to the timing of the actual display as possible. To ensure
tweak Unreal Engine for optimal performance, several project settings were
changed in addition: First, in the Rendering-¿VR settings, Instanced Stereo
was enabled while mobile HDR was disabled, as per recommendations by [27].
Second, the following console variables were were changed: R.GTSyncType to 1,
R.Vsync to 0, and R.OneFrameThreadLag to 0. R.GTSyncType determines which
thread in game processes sync to: 0 if they sync with the rendering thread, if
1 they sync to the RHI(render hardware interface=d3dx or opengl) thread. As
per the Unreal documentation syncing to the RHI thread helps with input
latency, so we set it as 1. VSync renders the frames at the pace at which the
display device is capable of, but it often leads to more dropped frames when
enabled than otherwise [36]. When OneFrameThreadLag is enabled, the graphics
drivers keep the game thread from processing further than one frame worth of
computations than what is currently being displayed. We deemed this
undesirable as our purpose was to minimize lags stemming from computations not
being ahead enough, along with minimization of the input latency.

⬇

void Aball3d_426Ball::TriggerStim()

{

FTimerHandle dispTrigger;

// send trigger to arudino

if (!WriteFile(hSerial, &ledcode, 1, &bytesw, 0))

{

//In case it don’t work get comm error and return false

std::cout << "Writefile failed" << std::endl;

} //!Stim1TimerFlag

GridPlane1->SetVisibility(true);

Stim1TimerFlag = true;

lastRender = FDateTime::Now();

lastRender64 = lastRender.ToUnixTimestamp();

lastRender32 = lastRender.GetMillisecond();

}

Figure 4: Code for sending Arduino trigger on Unreal

###  II-B Measuring apparatus

To measure timings of 1) behavioral response onset, 2) stimulus onset code
execution, 3) feedback code execution in response to behavior, and 4) pixel
changes on the HMD as precisely as possible, a circuit apparatus using that
can bee seen in Figure 5 was implemented. The inspiration for the circuit
board was based on a schematic from class material in Aachen university’s
system design course [37]. Specific components of the circuitry included an
Arduino Uno Rev.3, a BPW-34 photosensitive diode developed by Vishay
Semiconductors, a pressure sensor FSR402 developed by Interlink Electronics.
For registering the behavior response, a Wooting One keyboard developed by
Wooting was used. For running the 3D Engine based experimental paradigms, a
Windows-10 based computer running on AMD’s 5900x CPU with Nvidia RTX 3090Ti
was used.

(a) Circuit diagram

(b) Setup

Figure 5: Circuit diagram of the latency measuring apparatus, setup pictures

As the actual display of the VR environment, an Oculus DK2 headset from Meta
Inc. was used, in which the photodiode was attached to next to the display. An
USB-based oscilloscope developed by Pico Technology (Picoscope series 2205A)
with two probes was used for measuring the changes in voltage. The
oscilloscope’s sampling frequency was set too 240kHz. The ground clamps of
both probes were connected to the ground pin cable of the Arduino board. As
the Oculus DK2’s refresh rate is at 75Hz, we band-pass filtered the probe
connected to the photodiode to [60 80]Hz. Sample collection length per trial
was set to 200ms, with 20ms pre-trigger and 180ms post-trigger. For the
Stim2Disp measurements, the first probe was clamped to the LED diode that
would be toggled on and off by the stimulus control code in the 3D engine,
while the second probe was clamped to the cable connected to the HMD-attached
photodiode. The scope data collection trigger was set to a rising threshold of
1.5V for Unity and 115mV for Unreal with a hysteresis of 5.87% on the first
probe. All scope measure trigger thresholds were set manually set after trial
and error for catching the events of interest, and the difference in threshold
was made due to the probe attenuator settings changed to different levels
between the two sets of measurements, the difference in thresholds, however,
did not interfere with the trigger adequately capturing the point where the
event was occurring. This was verified after measurements by visually
inspecting the probe waveforms for peaks from LED and keypress actions. In
Key2Disp and Key2LED measurements, the first probe was clamped to the cable
attached to the pressure sensor attached to the keyboard. Here again, due to
selecting different probe attenuator settings, the probe trigger threshold was
set to 450mV rise with 2.44% hysteresis for Unreal, and a 4.7V rise for unity.
The second probe was connected to the LED in Key2LED measurements, and to the
photodiode in Key2Disp measurements. The Arduino would be connected to the
experiment PC via USB, through which LED trigger communications would be sent
from the 3D engines via serial communication. For each latency event of
interest, we made at least 300 repetitions of the measurement in order to
collect sufficient sample size.

###  II-C Data processing

Data preprocessing and analysis were performed with Matlab 2021b by Mathworks
Inc.. As the scope sampling rate was rather high considering our time epochs
of interest, data was first downsampled to 20kHz. As the data epochs were
temporally zero-centered to the triggering event of the first probe, the
timing of the events of interest on the second probe (photodiode voltage
change, LED power on) had to be found by peak detection, as the onset of
events of interest would result in significant changes in the probe voltage.
In photodiode measurements this meant voltage troughs that were far greater
than the baseline pixels (as the black and white grids would trigger a greater
change in the luminosity of the display, leading to greater voltage changes).
For all measurements on the second probe, as we were looking for the latency
for the onset of the event of interest, finding the timing of only the first
significant peak detected was necessary. Finding the position of the peaks was
performed with the findpeak() function provided in the Signal Processing
Toolbox of Matlab. The resultant peaks were plotted and manually inspected for
enough number of trials (¿100) in each condition to ensure the function was
performing as desired. Once the timing of the events of interest were found,
we calculated latency for the three events of interest.

##  III Results

From stimulus onset marker code execution to the actual onset of the chess-
board grid on the HMD pixels, on Unity Engine there was an average latency of
10.777ms (SD 0.672), while on Unreal Engine an average latency of 21.059ms (SD
0.671) was observed. From behavioral keypress onset detection to chess board
grid onset on the HMD, an average latency of 47.026ms (SD 6.156) was observed
on Unity while 46.682ms (SD 4.499) was observed on Unreal Engine. In a
separate session measuring latency between physical keypress detection and LED
onset upon keypress register in the 3D engine, we found an average latency of
36.948ms (SD 4.911) on Unity and 25.161ms (SD 5.087) on Unreal. Table I shows
the summarized results. Figure 6 (Stim2Disp), 7 (Key2Disp), and 8 (Key2Led)
each shows probe measurements for all individual trials superimposed on the
top plot (as well as the detected response peaks as black scatterplots), and
the averaged out measurements on the lower plot.

Mean latency in ms ($\pm$SD) | Unity 2019.4.20f | Unreal Engine 4.26
---|---|---
Stim2Disp |  10.777($\pm$0.672) |  21.059($\pm$0.671)
Key2Disp |  47.026($\pm$6.156) |  46.682($\pm$4.499)
Key2Led |  36.948($\pm$4.911) |  25.161($\pm$5.087)
TABLE I: Average and standard deviation values for the measured latency in all
conditions, for both 3D engines shown in milliseconds.

(a) Stimulus onset - HMD pixel change(Stim2Disp), Unity 2019.4.20f

(b) Stimulus onset - HMD pixel change(Stim2Disp), Unreal Engine 4.26

Figure 6: Latency measured between stimulus onset marker execution code and
actual stimuli onset pixel change in HMD (Stim2Disp), for both Unity and
Unreal Engine. Higher plot in each set of plots shows all measurements from
the two probes superimposed, along with the delayed response peaks determined
by Matlab’s findpeak() function. Lower plot shows the average waveforms, as
well as the average of the automatically detected first onset of peaks in the
second probe.

(a) Keypress Behavior - HMD pixel change(Key2Disp), Unity 2019.4.20f

(b) Keypress Behavior - HMD pixel change(Key2Disp), Unreal Engine 4.26

Figure 7: Latency measured between keyboard press behavior and feedback
stimuli onset on HMD pixel in 3D engine (Key2Disp), for both Unity and Unreal
Engine. Higher plot in each set of plots shows all measurements from the two
probes superimposed, along with the delayed response peaks determined by
Matlab’s findpeak() function. Lower plot shows the average waveforms, as well
as the average of the automatically detected first onset of peaks in the
second probe.

(a) Keypress Behavior - feedback code execution LED(Key2Led), Unity 2019.4.20f

(b) Keypress Behavior - feedback code execution LED(Key2Led), Unreal Engine
4.26

Figure 8: Latency measured between keyboard press behavior and marker
execution for keyboard event recognition code in 3D engine (Key2Led), for both
Unity and Unreal Engine. Higher plot in each set of plots shows all
measurements from the two probes superimposed, along with the delayed response
peaks determined by Matlab’s findpeak() function. Lower plot shows the average
waveforms, as well as the average of the automatically detected first onset of
peaks in the second probe.

Two-sample T-tests between the two 3D engines for the Stim2Disp condition
showed a significant difference in latency between Unity and Unreal
($t_{df=641}=60.537,p<1e^{-100},SD=0.665$). Similarly, a significant
difference was observed between Unity and Unreal’s latecny in the Key2Led
condition ($t_{df\textasciiacute 735}=31.900,p<1e^{-100},SD=4.991$). No
significant difference was found in latency between Unity and Unreal for the
Key2Disp condition ($t_{df=713}=0.833,p=0.405,SD=5.484$).

##  IV Discussion

This study aimed to make precise measurements of latencies that may occur
during time-critical cognitive behavioral experiments in 3D engine-based
virtual reality environments. To achieve this, we implemented a bare-bone 3D
environment in both Unity and Unreal Engine, two prominent 3D engines that are
used to develop Virtual reality and 3D scenarios for general purposes. We
implemented latency measurement in three different scenarios: a scenario in
which a marker event for stimuli presentation was sent and rendered on the
display, a scenario in which a physical key press by a participant happened
and registered in the 3D engine, and a scenario where a key press happened and
the resultant feedback occurred on the display. We used oscilloscope probes
combined with photodiodes on display events, serial-communication triggered
LEDs for software events, and pressure sensors for physical keypress events in
order to make precise timing measurements on when each of these events were
occurring.

We first discuss the difference in average latency values between Key2Disp and
Key2Led conditions: although measurements for the two conditions were made
separately (as the Key2Led involved sending a LED trigger to arduino upon 3D
engine recognition of the key event), considering our experiments were
designed to be as basic as possible in terms of code implementation (as well
as the sufficient sample size for each condition), at a glance one would
expect the sum of Stim2Disp and Key2Led conditions to match up or be somewhat
less than the average Key2Disp values. In Unity, the sum of the average of two
conditions exceed the Key2Disp slightly. We believe this is understandable
considering the communication time between PC software and the actual Arduino
interface itself. In a previous study by Schubert et al. [38], downstream
communication from an experimental computer to an Arduino was measured to be
an average of 1.251ms with a low standard deviation. Considering the
difference between the sum of mean Key2Led and Stim2Disp, and the Key2Disp
condition itself, the difference appears to be enough to explain the somewhat
larger mean latency in the combined latency.

###  IV-A Registering behavioral response without Unity or Unreal with LSL

From our current set of results, it appears the largest issue in maintaining a
reliable latency for cognitive experiment occurs from registering behavioral
responses from the user I/O device. As can be observed from results in the
Key2Led condition, in both 3D engines this latency is the largest (and the
most variable) in registering the physical response event to the software
stack.

It is possible that the nature of serial port devices contribute a large part
in this variance: parallel-port connected devices have been known to be
favored over serial port connections for participant I/O in timing critical
experimental design [39, 40, 22]. However serial port device technology has
come a long way, and it has been suggested that the imprecision arising from
user input devices may not be as critical as believed previously [24]. In
older studies comparing serial and PS/2 devices, serial input devices were
reported to have a much higher input latency with high variance [40]. In
modern devices however, the latency gap between serial port based devices and
parallel port devices may have become less considerable: response pad hardware
specifically used for cognitive experiments such as Cedrus pads use serial USB
connections. Furthermore, the choice of keyboard hardware that was used in our
experiment was a mechanical keyboard that uses optic-based switches for faster
input recognition on the hardware’s part, along with high polling rates over
100Hz.

The software stack also plays as much as a large part in the mean and
variation of the latency as the hardware stack does. It has been reported that
the experimental software framework as well as the operating system can
contribute to differently distributed latency and missed frame counts [23]. We
look into lowering the input variance and latency further by utilizing
software stack independent from 3D engines in this section.

In light of these considerations, we performed another set of measurements,
but this time using a software outside of 3D engines for key event
recognition. Lab Streaming Layer (LSL) [41] is a C++ based system for
synchronizing experimental data from multiple sources through a unified clock
(https://github.com/sccn/labstreaminglayer for more info). LSL has been used
for cognitive studies involving physiological signal measurements in which
timing was critical [42, 43, 44, 45, 46]. It supports language bindings in
multiple programming languages, as well as writing functions for adding custom
data source to the data streaming system. We modified a C++ callback code
available in the LSL Github Repository to catch certain key events and send
Arduino LED events similar to Key2Led condition, but bypassing 3D engines for
the key event recognition and getting them directly from the OS level:

⬇

// LSL outlet definition

static lsl::stream_outlet* outlet = nullptr;

// Keyboard hook

static HHOOK kbdHook = nullptr;

static bool isPressed[256] = {0};

//const WCHAR FileFullPath[] = { L"COM4" };

HANDLE hSerial;

char ledcode;

DCB dcbSerialParams = { 0 }; //FILE_ATTRIBUTE_NORMAL

DWORD byteswritten;

LRESULT CALLBACK keyboard_callback

(int code, WPARAM wParam, LPARAM lParam) {

if (code >= 0 && outlet) {

unsigned char key = 0;

switch (wParam) {

case WM_KEYDOWN:

case WM_SYSKEYDOWN:

key = ((KBDLLHOOKSTRUCT *)lParam)->vkCode & 0xFF;

if (!isPressed[key]) {

std::string evstr{key_names[key] + " pressed"};

// push key event to LSL stream

outlet->push_sample(&evstr);

std::cout << evstr << std::endl;

isPressed[key] = true;

// send LED trigger to Arduino

if (!WriteFile(hSerial, &key, 1, &byteswritten, 0))

{

//In case it don’t work get comm error and return false

std::cout << "Writefile failed" << std::endl;

}

}

break;

case WM_KEYUP:

case WM_SYSKEYUP:

key = ((KBDLLHOOKSTRUCT *)lParam)->vkCode & 0xFF;

{

std::string evstr{key_names[key] + "released"};

// push key release event to LSL

outlet->push_sample(&evstr);

}

isPressed[key] = false;

break;

default:;

}

}

return CallNextHookEx(kbdHook, code, wParam, lParam);

}

Figure 9: Code for sending markers after catching Key events on LSL, and not
Unreal or Unity

The results from the set of measurements using LSL and a C++ callback function
for key events can be seen in Figure 10. With a mean latency of 9.950ms (SD
1.700) from physical key press event to the Arduino sensor trigger, we are
seeing much lower average latency levels that compare to older PS/2 devices,
as well as more stable variations in the latency. By logging keypress events
or participant behavioral input through separately run programs such as LSL,
we believe some of the issues regarding input lag variation in experiments
using graphics and compute-heavy 3D engines can be alleviated somewhat.

Figure 10: Keyboard input to C++ LSL code register. Higher plot in each set of
plots shows all measurements from the two probes superimposed, along with the
delayed response peaks determined by Matlab’s findpeak() function. Lower plot
shows the average waveforms, as well as the average of the automatically
detected first onset of peaks in the second probe.

###  IV-B Stimulus presentation code to auditory stimulus onset delay in Unity

In interactive experiments utilizing VR technology, especially in those that
aim to create naturalistic experimental environment with immersion, it is
often worth considering a multisensory presentation of stimuli. The addition
of auditory components to the visual stimuli presentation would create a much
more immersive VR simulation. And like visual stimuli, presentation of
auditory stimuli needs to be considerably precise in timing as well for event
related design experiments measuring physiological and behavioral response to
stimuli.

(a) Sound file used for latency measurement

(b) Oscilloscope probe setup of sound latency measurement

Figure 11: Circuit diagram of the latency measuring apparatus, setup pictures

We deemed it was also worth investigating the latency of auditory stimuli
onset code and the physical propagation of the stimuli sound. In the case of
Unity, one can utilize the base sound functionality provided by the engine,
or, use 3rd-party sound engines that are compatible with the 3D engine such as
FMOD (https://www.fmod.com/). While the default sound library from Unity does
not provide a lot of tweaking options to optimize for performance, FMOD allows
manually setting sound playback buffer sizes. For this study, we used a
soundfile from [47] as the stimulus to playback, either using Unity’s default
sound library, or using FMOD with a buffer size of either 512 or 1024. A line-
out cable (3.5mm M/M) was plugged into the speaker jack of the experimental
computer, with the other end being connected to the oscilloscope probe as can
be seen in Figure 11. We did similar measurements like in Stim2Disp conditon,
measuring the latency between stimuli onset code and the actual propagation of
the sound in the sound cable. We report the results in Figure 12. While using
FMOD with a buffer size of 512 yielded the best results, we observe that
latency for auditory stimuli presentation is much worse compared to visual
stimuli both in mean accuracy and in variation. Based on this observation, we
believe caution is warranted when using auditory stimuli, especially when in
the absence of a concurrent visual stimuli.

(a) Unity default sound library

(b) FMOD Buffer Size 512

(c) FMOD Buffer Size 1024

Figure 12: Stimuli onset code Led to actual auditory stimuli propagation.
Higher plot in each set of plots shows all measurements from the two probes
superimposed, along with the delayed response peaks determined by Matlab’s
findpeak() function. Lower plot shows the average waveforms, as well as the
average of the automatically detected first onset of peaks in the second
probe.

###  IV-C Future works

As Wimmer et al. [25] reported, specific latency measures for experimental
setup are strongly dependent on the specifics of the hardware and software one
acquires for the experiment. Considering the continuously developing landscape
in VR and its related hardware/software stack, simply measuring the latency of
each setup is not only insufficient, but it is a fruitless endeavor long-term.
Instead, it would be more prudent to develop a framework capable of measuring
delays for configurable setup on the go: this is our most immediate next step.
Furthermore, the purpose of establishing latency value distributions are to
ultimately utilize them in developments of VR-based behavioral experiments in
event-related designs to collect synchronized time-dependent behavioral and
physiological data; for our purposes of investigating underlying brain
processes, we are especially interested in utilizing these findings to create
latency-optimized VR EEG experiments in immersive, naturalistic 3D.

## Acknowledgments

This study was supported by the National Research Foundation of Korea under
project BK21 FOUR and grants NRF-2022R1A2C2092118, NRF-2022R1H1A2092007,
NRF-2019R1A2C2007612, as well as by Institute of Information & Communications
Technology Planning & Evaluation (IITP) grants funded by the Korea government
(No. 2017-0-00451, Development of BCI based Brain and Cognitive Computing
Technology for Recognizing User’s Intentions using Deep Learning; No.
2019-0-00079, Department of Artificial Intelligence, Korea University; No.
2021-0-02068, Artificial Intelligence Innovation Hub).

## References

  * [1] U. Hasson, Y. Nir, I. Levy, G. Fuhrmann, and R. Malach, “Intersubject synchronization of cortical activity during natural vision,” _science_ , vol. 303, no. 5664, pp. 1634–1640, 2004.
  * [2] B. de Gelder, J. Kätsyri, and A. W. de Borst, “Virtual reality and the new psychophysics,” _British Journal of Psychology_ , vol. 109, no. 3, pp. 421–426, 2018.
  * [3] M. Bacha-Trams, Y. I. Alexandrov, E. Broman, E. Glerean, M. Kauppila, J. Kauttonen, E. Ryyppö, M. Sams, and I. P. Jääskeläinen, “A drama movie activates brains of holistic and analytical thinkers differentially,” _Social cognitive and affective neuroscience_ , vol. 13, no. 12, pp. 1293–1304, 2018.
  * [4] I. P. Jääskeläinen, M. Sams, E. Glerean, and J. Ahveninen, “Movies and narratives as naturalistic stimuli in neuroimaging,” _NeuroImage_ , vol. 224, p. 117445, 2021.
  * [5] E. S. Finn, E. Glerean, A. Y. Khojandi, D. Nielson, P. J. Molfese, D. A. Handwerker, and P. A. Bandettini, “Idiosynchrony: From shared responses to individual differences during naturalistic neuroimaging,” _NeuroImage_ , vol. 215, p. 116828, 2020.
  * [6] J. Salmi, E. Glerean, I. P. Jääskeläinen, J. M. Lahnakoski, J. Kettunen, J. Lampinen, P. Tikka, and M. Sams, “Posterior parietal cortex activity reflects the significance of others’ actions during natural viewing,” _Human brain mapping_ , vol. 35, no. 9, pp. 4767–4776, 2014.
  * [7] M. A. Schmuckler, “What is ecological validity? a dimensional analysis,” _Infancy_ , vol. 2, no. 4, pp. 419–436, 2001.
  * [8] M. J. Tarr and W. H. Warren, “Virtual reality in behavioral neuroscience and beyond,” _Nature neuroscience_ , vol. 5, no. 11, pp. 1089–1092, 2002.
  * [9] X. Pan and A. F. d. C. Hamilton, “Why and how to use virtual reality to study human social interaction: The challenges of exploring a new research landscape,” _British Journal of Psychology_ , vol. 109, no. 3, pp. 395–417, 2018.
  * [10] S. de la Rosa and M. Breidt, “Virtual reality: A new track in psychological research,” _British Journal of Psychology_ , vol. 109, no. 3, pp. 427–430, 2018.
  * [11] A. N. Stephens and J. A. Groeger, “Situational specificity of trait influences on drivers’ evaluations and driving behaviour,” _Transportation research part F: traffic psychology and behaviour_ , vol. 12, no. 1, pp. 29–39, 2009.
  * [12] R. S. Rosenberg, S. L. Baughman, and J. N. Bailenson, “Virtual superheroes: Using superpowers in virtual reality to encourage prosocial behavior,” _PloS one_ , vol. 8, no. 1, p. e55003, 2013.
  * [13] C. Mueller, M. Luehrs, S. Baecke, D. Adolf, R. Luetzkendorf, M. Luchtmann, and J. Bernarding, “Building virtual reality fmri paradigms: a framework for presenting immersive virtual environments,” _Journal of neuroscience methods_ , vol. 209, no. 2, pp. 290–298, 2012.
  * [14] A.-M. Brouwer, M. A. Neerincx, V. Kallen, L. van der Leer, and M. ten Brinke, “Eeg alpha asymmetry, heart rate variability and cortisol in response to virtual reality induced stress,” _J. Cyberther. Rehabil_ , vol. 4, no. 1, pp. 21–34, 2011.
  * [15] S. B. i Badia, A. G. Morgade, H. Samaha, and P. Verschure, “Using a hybrid brain computer interface and virtual reality system to monitor and promote cortical reorganization through motor activity and motor imagery training,” _IEEE Transactions on Neural Systems and Rehabilitation Engineering_ , vol. 21, no. 2, pp. 174–181, 2012.
  * [16] J. Faller, B. Z. Allison, C. Brunner, R. Scherer, D. Schmalstieg, G. Pfurtscheller, and C. Neuper, “A feasibility study on ssvep-based interaction with motivating and immersive virtual and augmented reality,” _arXiv preprint arXiv:1701.03981_ , 2017.
  * [17] J. W. Kuziek, A. R. Tayem, J. I. Burrell, E. X. Redman, J. Murray, J. Reinen, A. Sipolins, and K. E. Mathewson, “Real brains in virtual worlds: Validating a novel oddball paradigm in virtual reality,” _bioRxiv_ , p. 749192, 2020\.
  * [18] S. Tarng, D. Wang, Y. Hu, and F. Merienne, “Towards eeg-based haptic interaction within virtual environments,” in _2019 IEEE Conference on Virtual Reality and 3D User Interfaces (VR)_. IEEE, 2019, pp. 1179–1180.
  * [19] S. J. Jang, Y. J. Yang, S. Ryun, J. S. Kim, C. K. Chung, and J. Jeong, “Decoding trajectories of imagined hand movement using electrocorticograms for brain–machine interface,” _Journal of Neural Engineering_ , vol. 19, no. 5, p. 056011, 2022.
  * [20] J. Tarrant, J. Viczko, and H. Cope, “Virtual reality for anxiety reduction demonstrated by quantitative eeg: a pilot study,” _Frontiers in psychology_ , vol. 9, p. 1280, 2018.
  * [21] J. D. Bayliss and D. H. Ballard, “A virtual reality testbed for brain-computer interface research,” _IEEE Transactions on Rehabilitation Engineering_ , vol. 8, no. 2, pp. 188–190, 2000.
  * [22] R. R. Plant, N. Hammond, and T. Whitehouse, “How choice of mouse may affect response timing in psychological studies,” _Behavior Research Methods, Instruments, & Computers_, vol. 35, no. 2, pp. 276–284, 2003.
  * [23] P. Garaizar, M. A. Vadillo, D. López-de Ipiña, and H. Matute, “Measuring software timing errors in the presentation of visual stimuli in cognitive neuroscience experiments,” _PloS one_ , vol. 9, no. 1, p. e85108, 2014.
  * [24] M. F. Damian, “Does variability in human performance outweigh imprecision in response devices such as computer keyboards?” _Behavior Research Methods_ , vol. 42, no. 1, pp. 205–211, 2010.
  * [25] R. Wimmer, A. Schmid, and F. Bockes, “On the latency of usb-connected input devices,” in _Proceedings of the 2019 CHI conference on human factors in computing systems_ , 2019, pp. 1–12.
  * [26] M. S. Elbamby, C. Perfecto, M. Bennis, and K. Doppler, “Toward low-latency and ultra-reliable virtual reality,” _IEEE Network_ , vol. 32, no. 2, pp. 78–84, 2018.
  * [27] M. Wiesing, G. R. Fink, and R. Weidner, “Accuracy and precision of stimulus timing and reaction times with unreal engine and steamvr,” _PloS one_ , vol. 15, no. 4, p. e0231152, 2020.
  * [28] K. A. Shapcott, M. Weigand, I. Glukhova, M. N. Havenith, and M. L. Scholvinck, “Domevr: A setup for experimental control of an immersive dome virtual environment created with unreal engine 4,” _bioRxiv_ , 2022.
  * [29] U. Ju, J. Kang, and C. Wallraven, “To brake or not to brake? personality traits predict decision-making in an accident situation,” _Frontiers in psychology_ , vol. 10, p. 134, 2019.
  * [30] F. N. Nezami, M. A. Wächter, N. Maleki, P. Spaniol, L. M. Kühne, A. Haas, J. M. Pin, L. Tiemann, F. Nienhaus, L. Keller _et al._ , “Westdrive x loopar: An open-access virtual reality project in unity for evaluating user interaction methods during takeover requests,” _Sensors_ , vol. 21, no. 5, p. 1879, 2021.
  * [31] U. Ju, L. L. Chuang, and C. Wallraven, “Acoustic cues increase situational awareness in accident situations: A vr car-driving study,” _IEEE Transactions on Intelligent Transportation Systems_ , 2020.
  * [32] J. Brookes, M. Warburton, M. Alghadier, M. Mon-Williams, and F. Mushtaq, “Studying human behavior with virtual reality: The unity experiment framework,” _Behavior research methods_ , vol. 52, no. 2, pp. 455–463, 2020\.
  * [33] L.-M. Vortmann, F. Kroll, and F. Putze, “Eeg-based classification of internally-and externally-directed attention in an augmented reality paradigm,” _Frontiers in human neuroscience_ , vol. 13, p. 348, 2019.
  * [34] L. Quintero, P. Papapetrou, J. E. Muñoz, J. De Mooij, and M. Gaebler, “Excite-o-meter: an open-source unity plugin to analyze heart activity and movement trajectories in custom vr environments,” in _2022 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW)_. IEEE, 2022, pp. 46–47.
  * [35] A. Šmíd, “Comparison of unity and unreal engine,” _Czech Technical University in Prague_ , pp. 41–61, 2017.
  * [36] K. Raaen and I. Kjellmo, “Measuring latency in virtual reality systems,” in _International Conference on Entertainment Computing_. Springer, 2015, pp. 457–462.
  * [37] J. Borchers, S. Hueber, and S. Schröder, “Designing interactive systems course,” 2022, course website for schematics. [Online]. Available: https://hci.rwth-aachen.de/dis2-ss22
  * [38] T. W. Schubert, A. D’Ausilio, and R. Canto, “Using arduino microcontroller boards to measure response latencies,” _Behavior research methods_ , vol. 45, no. 4, pp. 1332–1346, 2013.
  * [39] J. Beringer, “Timing accuracy of mouse response registration on the ibm microcomputer family,” _Behavior Research Methods, Instruments, & Computers_, vol. 24, no. 3, pp. 486–490, 1992.
  * [40] C. D. Chambers and M. Brown, “Timing accuracy under microsoft windows revealed through external chronometry,” _Behavior Research Methods, Instruments, & Computers_, vol. 35, no. 1, pp. 96–108, 2003.
  * [41] C. Kothe _et al._ , “Lab streaming layer (lsl),” 2014.
  * [42] Y. Wang, C. Markham, and C. Deegan, “Assessing the time synchronisation of eeg systems,” in _2019 30th Irish Signals and Systems Conference (ISSC)_. IEEE, 2019, pp. 1–6.
  * [43] K. Gramann, D. P. Ferris, J. Gwin, and S. Makeig, “Imaging natural cognition in action,” _International Journal of Psychophysiology_ , vol. 91, no. 1, pp. 22–29, 2014.
  * [44] S. Blum, D. Hölle, M. G. Bleichner, and S. Debener, “Pocketable labs for everyone: synchronized multi-sensor data streaming and recording on smartphones with the lab streaming layer,” _Sensors_ , vol. 21, no. 23, p. 8135, 2021.
  * [45] N. Bigdely-Shamlo, K. Kreutz-Delgado, K. Robbins, M. Miyakoshi, M. Westerfield, T. Bel-Bahar, C. Kothe, J. Hsi, and S. Makeig, “Hierarchical event descriptor (hed) tags for analysis of event-related eeg studies,” in _2013 IEEE Global Conference on Signal and Information Processing_. IEEE, 2013, pp. 1–4.
  * [46] D. H. Lee and T.-P. Jung, “A virtual reality game as a tool to assess physiological correlations of stress,” _arXiv preprint arXiv:2009.14421_ , 2020.
  * [47] Y. Chen and C. Wallraven, “Pop or not? eeg correlates of risk-taking behavior in the balloon analogue risk task,” in _2017 5th International Winter Conference on Brain-Computer Interface (BCI)_. IEEE, 2017, pp. 16–19.