# SIM2VR: Towards Automated Biomechanical Testing in VR

Florian Fischer  0000-0001-7530-6838 University of CambridgeUK ,  Aleksi
Ikkala  0000-0001-9118-1860 Aalto UniversityFinland ,  Markus Klar
0000-0003-2445-152X University of GlasgowScotland ,  Arthur Fleig
0000-0003-4987-7308 ScaDS.AI Dresden/Leipzig, Leipzig UniversityGermany ,
Miroslav Bachinski  0000-0002-2245-3700 University of BergenNorway ,  Roderick
Murray-Smith  0000-0003-4228-7962 University of GlasgowScotland ,  Perttu
Hämäläinen  0000-0001-7764-3459 Aalto UniversityFinland ,  Antti Oulasvirta
0000-0002-2498-7837 Aalto UniversityFinland and  Jörg Müller
0000-0002-4971-9126 University of BayreuthGermany

(2018)

###### Abstract.

Automated biomechanical testing has great potential for the development of VR
applications, as initial insights into user behaviour can be gained in silico
early in the design process. In particular, it allows prediction of user
movements and ergonomic variables, such as fatigue, prior to conducting user
studies. However, there is a fundamental disconnect between simulators hosting
state-of-the-art biomechanical user models and simulators used to develop and
run VR applications. Existing user simulators often struggle to capture the
intricacies and nuances of real-world VR applications, reducing ecological
validity of user predictions. In this paper, we introduce SIM2VR, a system
that aligns user simulation with a given VR application by establishing a
continuous closed loop between the two processes. This, for the first time,
enables training simulated users directly in the same VR application that real
users interact with. We demonstrate that SIM2VR can predict differences in
user performance, ergonomics and strategies in a fast-paced, dynamic arcade
game. In order to expand the scope of automated biomechanical testing beyond
simple visuomotor tasks, advances in cognitive models and reward function
design will be needed.

biomechanical simulation, interaction design, VR simulation alignment problem,
automated testing, virtual reality, VR development, VR application, deep
reinforcement learning

††copyright: acmcopyright††journalyear: 2018††copyright:
rightsretained††conference: Make sure to enter the correct conference title
from your rights confirmation emai; June 03–05, 2018; Woodstock,
NY††booktitle: Woodstock ’18: ACM Symposium on Neural Gaze Detection, June
03–05, 2018, Woodstock, NY††doi: XXXXXXX.XXXXXXX††isbn:
978-1-4503-XXXX-X/18/06††ccs: Human-centered computing Human computer
interaction (HCI)††ccs: Human-centered computing Systems and tools for
interaction design††ccs: Human-centered computing Virtual reality††ccs: Human-
centered computing User models

Figure 1.  Simulating interactive user behaviour in VR typically requires
reimplementing the game dynamics, logic and graphics of the VR application in
another simulation engine, which is time-consuming, redundant and error-prone
(left).
SIM2VR (middle) enables VR designers for the first time to run state-of-the-
art biomechanical user simulations directly in the same VR application real
users are interacting with (right). By aligning simulated and real interaction
with respect to VR input and output, as well as application dynamics, SIM2VR
allows for more ecologically valid predictions of user performance, effort and
strategies, supporting developers in the early stages of the design process.

Three boxes illustrating the simulated interaction loops without SIM2VR and
with SIM2VR together with the real interaction loop. The ”Without SIM2VR” loop
(left) shows a biomechanical model with one eye (”Simulated User”) and three
targets in the same environment (”Simulated Environment”). The ”With SIM2VR”
loop (middle) shows a biomechanical model equipped with VR controller and HMD
(”Simulated User”), a ”VR Application” block, and the SIM2VR logo in between.
An orange frame highlights this loop. The ”Real Interaction” loop (right)
shows a real user (”User”) and the same ”VR Application” block.

##  1\. Introduction

Recent advances in deep reinforcement learning (deep RL) and biomechanical
models have enabled the creation of simulated users for modelling how humans
may behave in interaction tasks. Using advanced biomechanical models that
capture the intricacies of human body movement beyond linked-segment models,
such simulated users allow prediction of time-continuous movements and effort-
related quantities such as muscle energy expenditure (Umberger et al., 2003)
and fatigue (Xia and Law, 2008; Rashedi and Nussbaum, 2015). In HCI, simulated
users have been recently demonstrated for various tasks, including pointing
(Klar et al., 2023; Cheema et al., 2020), typing (Hetzel et al., 2021; Jokinen
et al., 2021), tracking (Fischer et al., 2021), steering and choice reaction
(Ikkala et al., 2022). This modeling approach is promising, as it enables
designers and developers to pose “what-if?” questions to assess how changes in
design choices affect users’ performance and ergonomics.

This bears particular promise for HCI domains where human data is scarce or
costly to collect, such as in virtual reality (VR) interactions. In VR
development, testing prototypes typically involves wearing a headset, standing
up and moving around in space, which incurs significant overhead and renders
iterative testing slow and cumbersome (Liu et al., 2023). Biomechanical user
simulations show significant potential to remedy this as a means of automated
testing. In particular, they allow VR designers to evaluate and compare
prototypes without the need for expensive user studies, which can be lengthy
and stressful, especially for vulnerable participants. By using an appropriate
range of different user models, a simulation-based approach can increase
diversity during the testing process, thereby promoting ability-based design
(Murray-Smith et al., 2022).

To realise the potential of simulated users, we need to address the challenge
of integrating model-based simulations into VR development workflows. State-
of-the-art biomechanical models are not readily available in commonly used VR
development environments like Unity111https://unity.com/, or they cannot be
combined with deep RL approaches to generate forward simulations in those
environments. For instance, while there exists a MuJoCo Unity
plugin222https://github.com/google-deepmind/mujoco/tree/main/unity that
provides access to biomechanical models implemented in MuJoCo (Todorov et al.,
2012), an efficient physics engine suitable for biomechanical simulation, the
models can only be controlled manually using the Unity Editor.

As a result, VR designers who wish to use automated testing must recreate the
dynamics and graphics of the VR application within the simulation engine used
for the biomechanical model. This, however, is impractical for three reasons.
First, it introduces implementation overhead, in that two VR environments need
to be maintained and updated in parallel. Second, building the replica is
often tedious and cumbersome, and requires detailed knowledge of the
subtleties of the considered application. Third, the resulting simulation is
unlikely to create ecologically valid predictions of interactive user
behaviour, as the replica will typically correspond to a low-fidelity version
of the original VR application. Such an approach thus considerably complicates
alignment of the user input sensed by the application, the (visual) output
generated by the application, and the underlying application dynamics
themselves. We call this the VR Simulation Alignment Problem and describe it
in detail in Section 3.1. Simulations that do not adequately address this
problem will provide inferior predictions of how real users interact with VR
interfaces or techniques, limiting the validity of in silico testing and
compromising the comparative analysis of design choices.

To overcome these limitations, we propose SIM2VR, a system that integrates
biomechanical user simulations directly into a given Unity application.
Alignment is improved by simulating hardware within the physics engine used
for biomechanical simulation, tracking virtual VR input and output using
OpenXR, accessing all relevant application data directly in Unity, and
synchronising the user simulation and the VR application. From a research
perspective, this enables a more accurate and reliable development and
validation of user simulation models by ensuring that real and simulated users
interact with exactly the same virtual environment. From an interactive
software development perspective, our work provides a new solution to the
long-standing problem of automated testing of VR interaction.

To summarise, this paper contributes

  * •

a novel definition and comprehensive discussion of the VR Simulation Alignment
Problem encountered when simulating movement-based interaction in VR (Section
3);

  * •

SIM2VR, a system for integrating biomechanical simulations into Unity
applications, which partially addresses the VR Simulation Alignment Problem
(Section 4);

  * •

a demonstration of SIM2VR’s capability to predict differences in performance
and ergonomics, and to reveal potential user strategies, verified by an
accompanying user study (Section 5).

##  2\. Background and Context

Below, we situate our work in relation to previous efforts in both automated
testing in VR development, and computational user modeling and simulation.

###  2.1. VR Development and Automated Testing

Because of the effort and time needed to don and doff equipment during
prototype development, iterative testing and development in VR has a higher
cost than traditional interface design for desktop or mobile software, making
it typically slow and cumbersome (Liu et al., 2023). This underlines the need
to automate testing as much as possible. Automated testing allows prototypes
to be automatically evaluated and compared without the need for expensive user
studies (Bierbaum et al., 2003; Karakaya et al., 2022), by predicting
performance estimates such as completion times and error rates.

While the need for automated testing has been clear for decades, a recent
study of over 300 VR projects found that 79% of the projects did not use any
automatic tests, and that test automation followed common software testing
practices without considering the user’s body movements, focusing on aspects
such as evaluating the correctness of a response after an event was triggered
(Rzig et al., 2023). So far, even automated testing approaches designed
specifically for VR do not emulate or simulate the user’s moving body.
Instead, they focus on moving and rotating the viewpoint to inspect VR scenes
(Wang, 2022), or operate on higher-level action events such as “click UI
button” or “grab object” which abstract away the details of the user’s
movements (Bierbaum et al., 2003; Harms, 2019; Karakaya et al., 2022). In the
domain of Natural User Interfaces (NUIs), a generative motion model was
proposed for automatic testing of gestural interaction in (Hunt et al., 2014),
but unlike our work, the model was not embedded in a learning loop; instead,
it generated random movement sequences without the ability to adapt to the
tested interface.

###  2.2. Computational User Modelling and Simulation

####  2.2.1. Simulation in HCI

As discussed in (Murray-Smith et al., 2022), user-centred design could move to
a more rigorous, safer and predictable process via a simulation-based approach
to interaction design. The same paper highlights general potential benefits of
simulation in HCI for efficiency of design, safety, supporting diversity and
transparency. Simulations can be used to predict task performance in
interaction, such as time taken to finish the task and how often tasks can be
successfully accomplished. For example, very early work on keystroke-level
models in (Card et al., 1983), and more recent work (Kristensson and Müllners,
2021) used modelling to replace extensive experimentation, optimising text
entry system parameters.

The general tools for simulation are improving rapidly, opening up new
opportunities. As pointed out in (Lavin et al., 2021), we have seen recent
developments in probabilistic, differentiable programming, high-performance
computing and causal modelling. This, combined with the rapidly improving
ability of machine learning to emulate complex aspects of human perception and
behaviour (e.g. deep neural nets can simulate human-level visual perception
(He et al., 2016), speech recognition (Graves et al., 2013) and control (Mnih
et al., 2015)), means that these new tools for simulation have an increased
potential to be efficiently and usefully applied to domains such as HCI.

####  2.2.2. Biomechanical User Simulation

Biomechanical models and computer-based simulations are well-established
(Ayoub et al., 1974). Increases in computational power have allowed them to
evolve from simple models, limited to computation of mechanical loads in
static postures (Winter, 1984), to more physiologically-accurate
musculoskeletal models (Delp et al., 2007; Damsgaard et al., 2006).
Biomechanical models were typically used for inverse simulation, e.g. using
the OpenSim ecosystem (Seth et al., 2011). Such inverse simulation methods,
namely inverse kinematics, inverse dynamics and static optimisation or
computed muscle control, allow the estimation of mechanical loads within the
human musculoskeletal system, and neural controls of the muscles given motion
tracking data of a given user’s movement as input (Delp et al., 2007). In HCI,
inverse dynamics have been used to identify fundamental ergonomics issues,
such as “gorilla arm” (Jang et al., 2017). Forward simulation methods, in
contrast, are less frequently used in standalone situations, as they require
muscle controls as inputs, which are extremely complex to measure
experimentally. However, these have become more useful when used in
conjunction with computational controllers (Lee and Umberger, 2016; Dembia et
al., 2021; Schumacher et al., 2023) and efficient physics engines (Klar et
al., 2023), which we will exploit in this paper.

####  2.2.3. Optimal Feedback Control and Reinforcement Learning Models

Control-theoretic models of human motion for mouse movements were developed in
(Müller et al., 2017) and supplemented by the assumption of Optimal Feedback
Control (Todorov and Jordan, 2002) in (Fischer et al., 2022). In such
optimization-based forward simulations, users are assumed to continuously
interact with their environment by selecting controls such that a task-
specific utility (e.g., a game score) is maximised, given the perceptual
information available at decision time. By including intermittent control
behaviour typical of humans, simulations could also capture more realistic
models of human motor variability (Martín et al., 2021).

Reinforcement learning has recently emerged as a suitable framework for
modelling human behaviour in a flexible way: one only needs to define the
states, actions and rewards, and then use RL methods to estimate the optimal
policy (Chen et al., 2015, 2017). In some cases, the reward function and other
parameters can be inferred from human data (Banovic et al., 2016; Kangasrääsiö
et al., 2017). If the reward function and state-action space, including their
key limitations, are similar to those of a human, increasingly human-like
behaviour has been shown to emerge through learning (Oulasvirta et al., 2022).

Recent applications in HCI include models of typing, menu selection,
multitasking and visual decision-making (Oulasvirta et al., 2022). Using the
assumptions of signal-dependent control noise and movement time minimization,
Fischer et al. (Fischer et al., 2021) have shown that an RL agent can generate
human-like mid-air trajectories that are in accordance with well-established
phenomena such as Fitts’ Law (Fitts, 1954) and the Two-Thirds Power Law
(Lacquaniti et al., 1983). RL-based simulation may also provide valuable
information for predicting usability- and ergonomics-related criteria and to
aid in interface design. For instance, Cheema et al. trained a torque-actuated
biomechanical arm model in a mid-air pointing task, and used the model to
predict fatigue of real human subjects performing the task (Cheema et al.,
2020). In this paper, we build on these developments and use RL to model
interaction in VR.

###  2.3. Creating Simulated Users

By combining perception models, musculoskeletal models, and physically
simulated input devices, we can train RL agents, or “simulated users”, to
model and simulate intricate interaction tasks, such as those requiring
visuomotor control. An early example of such control was presented in (Nakada
et al., 2018), where Nakada et al. introduced a virtual human model and used
deep learning to learn reaching and tracking tasks.

The development of computationally efficient simulation software, such as
MuJoCo (Todorov et al., 2012), has permitted the use of deep RL for flexible
problem formalisations of interaction tasks. This allows a researcher to guide
an agent’s learning through reward functions, as was demonstrated in (Ikkala
et al., 2022) by Ikkala et al. With their User-in-the-Box (UitB) approach,
Ikkala et al. trained a muscle-actuated model of the upper torso and right arm
to complete a series of interaction tasks. UitB allows combining biomechanical
models with perception models and movement-based interaction tasks through a
modular software structure, which facilitates integrating custom model
components. In our work, we exploit this modularity and extend UitB to VR
environments.

MyoSuite (Caggiano et al., 2022a), another user simulator implemented in
MuJoCo, focuses on dexterous hand, lower- and full-body movements. MyoSuite
also provides a conversion tool for converting OpenSim models into MuJoCo
format (MyoConverter333https://github.com/MyoHub/myoconverter (Wang et al.,
2022; Ikkala and Hämäläinen, 2020)), which can also be readily added into the
UitB framework.

Simulated users could promote ability-based design in VR, as diverse user
populations, i.e., populations with different physical or cognitive
properties, could be simulated without the risk of physical harm (Murray-Smith
et al., 2022; Quek et al., 2011; Quek, 2013). However, existing approaches
that create and evaluate simulated users in the context of HCI are limited to
“mock-up” interaction environments that require a re-implementation of the
application dynamics real users face during interaction (Fischer et al., 2021;
Ikkala et al., 2022; Hetzel et al., 2021). These approaches are thus
fundamentally unsuitable for automated biomechanical testing of a given VR
prototype. To the best of our knowledge, our work is the first to create
simulated users in an ecologically valid VR context by training biomechanical
user models directly in the same VR environment that human users interact
with.

##  3\. The VR Simulation Alignment Problem

Table 1. Simulated user interaction in VR needs to be aligned with real user
interaction in six dimensions. For each dimension, the table provides a
description and examples of corresponding variables to be aligned.

Alignment Dimension Description Examples of Alignment Targets User Perceptual
Environmental Perception Visual & Proprioceptive Observations Cognitive User
Behaviour Muscle Control Signals Physical Biomechanics & Hardware Dynamics Arm
Posture, VR Controller Position Appl. Input Channel Input Data Transfer Sensed
Position of VR Controller Application VR Application Dynamics Virtual Object
Size & Position, Game Score Output Channel Output Data Transfer Rendered
Image, Controller Vibrations, Audio Signal

The table contains the columns ”Alignment Dimension”, ”Description” and
”Examples of Alignment Targets” for the following six rows (the first three
corresponding to user-related dimensions, the latter three to application-
related dimensions): 1\. Perceptual - Environmental Perception - Visual &
Proprioceptive Observations. 2\. Cognitive - User Behaviour - Muscle Control
Signals. 3\. Physical - Biomechanics Hardware Dynamics - Arm Posture, VR
Controller Position. 4\. Input Channel - Input Data Transfer - Sensed Position
of VR Controller. 5\. Application - VR Application Dynamics - Virtual Object
Size Position, Game Score. 6\. Output Channel - Output Data Transfer -
Rendered Image, Controller Vibrations, Audio Signal.

As depicted in Figure 1 (right), movement-based interaction in VR can be
regarded as a time-continuous closed loop between the two core components of
interaction: the user and the VR application. During this interaction, the
user obtains multi-sensory information from its environment444Usually, this
includes information from the peripersonal and physical space of the user
(e.g., proprioceptive or haptic input signals) as well as the virtual space
(visual input signals)., then produces a neuromuscular control signal to
generate body movements, hence manipulating the input device to complete the
given interaction task. The VR application, on the other hand, conveys an
interactive, virtual environment that can be manipulated via one or multiple
input devices. The state of the input device is actively sensed by the VR
system, measuring, e.g., how the VR controller is moved, or whether a physical
controller button is pressed. As a final step to close the loop, the VR
application provides sensory output to the user, such as a rendered image
displayed on an HMD, or haptic output in the form of temporary controller
vibrations.

###  3.1. Aligning Simulated VR Interaction

The VR interaction loop described above not only characterises the movement-
based interaction between a real user and a given VR system, but can also be
used as a framework for creating model-based simulations of VR interaction. To
ensure that the simulated interaction loop acts as a reasonable proxy for real
user interaction, i.e., to ensure external validity of the predictions made,
the two interaction loops need to be aligned. We coin this problem the VR
Simulation Alignment Problem.

As summarised in Table 1 and described in detail below, VR simulation
alignment evolves in six dimensions, three of which focus on the alignment of
the simulated user and three on the alignment of the simulated VR system.

#### Physical Alignment

The physics of the interaction, i.e., the biomechanics of the user’s body and
the dynamics of the input and output devices, need to be captured adequately
by the simulation. This requires a sophisticated biomechanical model that
captures the complexity of the human musculoskeletal system, at least for the
body parts relevant for the considered interaction (e.g., a model of the
shoulder, arm and hand), as well as simulated input and output devices (e.g.,
a simulated VR controller and an HMD) that align with the real hardware in
terms of static and dynamic properties such as mass or size. Too simple a
model of the user’s body (e.g., a linked-segment model of the arm) may produce
unrealistic movement trajectories, resulting in severe deviations from how
real users interact with a given application.

#### Input Channel Alignment

The simulation needs to sense the state of the input device (e.g., the
location and orientation of the VR controller) in the same way as the real VR
system. In particular, it has to account for coordinate frame transformations
and update rates (i.e., the measuring frequency).

#### Application Alignment

The simulated VR application should have the same internal states as the
target VR application (e.g., it needs to provide the same virtual objects and
affordances) and show the same responsive behaviour, given that both
applications are provided the same input signals. If application alignment is
low, e.g., because a game constraint is missing or is incorrectly re-
implemented, applying the same body movements in the simulation as in reality
may result in substantially different states of the VR application.

#### Output Channel Alignment

The simulated VR system needs to provide the same output signals as the real
VR system, given that both are in the same state. This ensures that the
simulated user has the chance to perceive the same information from the
application as a real user. In particular, the simulated user needs to be
provided the same rendered image as is shown on the the real user’s HMD.

#### Perceptual Alignment

The simulated user should perceive its environment (including its own body,
physical input devices and the VR application) in the same way a real user
does. This requires appropriate modeling of all sensory modalities relevant
for the considered interaction, including vision, proprioception and haptics.
Note that perceptual alignment focuses on the observations that users make of
their environment, whereas output channel alignment is about the information
that the VR application makes available to the user in the first place.

#### Cognitive Alignment

Given an interaction task (e.g., to successfully complete a specific game
level as fast as possible), the simulated user should generate the same muscle
control signals as a real user, when provided with the same perceptions. In
the context of reinforcement learning, this requires defining an appropriate
task-specific reward function such that an optimal control policy resembles
real user behaviour.

Figure 2.  To obtain valid and reliable predictions of interactive user
behaviour in VR, a simulated user needs to perceive and control exactly the
same VR application as real users. Our SIM2VR system addresses this VR
Simulation Alignment Problem by establishing a closed loop between the
simulation process and the desired VR application, ensuring that the same
input and output signals are generated as with real hardware. The resulting
time-continuous integration allows to train and evaluate a biomechanical user
simulation directly in a given VR environment.

A closed-loop between the simulated user (shown as biomechanical model with VR
controller and HMD) and the VR application (visualised by two example frames
from the Whac-A-Mole game) is established through the SIM2VR system
(visualised by an arrow from the simulated user to the VR application and an
arrow in the opposite direction, with SIM2VR logo in between). This loop
consists of four steps (the following texts are annotated in corresponding
text boxes): 1) [Simulated User] ”Select muscle control signals to generate
body movement and update position and orientation of VR controller and HMD.”
2) [SIM2VR] ”Send simulated sensor input to the VR application.” 3) [VR
Application] ”Update game physics and game logic, compute new reward signal
and render the scene.” 4) [SIM2VR] ”Send rendered image, game reward and
stateful information to the simulated user.”

As the behaviour of the user and the VR application depend on each other in a
closed loop manner, errors can easily cascade in time. This results in an
increasing divergence between the simulation and the real VR interaction, once
an error is introduced. For example, in the Whac-A-Mole game from Section 5, a
different grasp (e.g., due to a slight offset in the game dynamics) may result
in fewer/more/different targets being hit, which impacts the movements
generated by the simulated user at later times, leading to even greater
differences in target hits, and so on. The same holds for a missing or
incorrectly implemented game constraint: If the simulated user has to hit a
target with less power or less often to receive the same game reward, it will
learn a totally different control policy, leading to even more divergent
application states. Depending on the complexity and the amount of
discontinuities in the interaction dynamics, these errors may pollute any
variable of interest obtained from the simulation. It is thus essential to
ensure alignment in as many dimensions as possible.

##  4\. SIM2VR: Enabling Biomechanical User Simulations in VR

In this section, we propose SIM2VR, a system that allows two simulators, the
simulated user and the original VR application, to run in parallel while being
synchronised temporally and spatially. The aim of this system is to establish
a closed interaction loop between the simulated user and the VR application
(see Figure 2) that closely resembles the interaction loop between the real
user and the VR application described in Section 3.

In this simulated interaction loop, the simulated user generates muscle
control signals based on its sensory perception of the body, the input
devices, and the VR application (e.g., visual information of the scene and
proprioceptive information of the body posture). These control signals are
used to actuate the muscles, driving the movement of the body, and,
consequently, the simulated states of the hardware devices. The position and
rotation of the VR controllers and HMD are then measured by virtual sensors
and sent to the VR application. The VR application processes this input,
computes a reward based on the updated application state (e.g., the current
game score) and renders the image perceived through the virtual HMD. Finally,
the rendered image is provided to the simulated user, along with the reward
and optional “stateful” information (such as time remaining in an interaction
task), thus closing the interaction loop.555Note that the reward signal and
the stateful information are only part of the simulated interaction loop. This
is mainly for technical reasons: The reward signal is required for an RL
method to learn a control policy, and the stateful information can be used to
improve the training.

Crucially, this approach ensures application alignment by giving the simulated
user access to exactly the same VR application that real users interact with.
Based on this approach, we developed the SIM2VR system, which additionally
addresses the input channel and output channel alignment problems to provide
more ecologically valid predictions of VR interaction.

###  4.1. Implementation Principles & Details

SIM2VR is an extension of the User-in-the-Box (UitB) framework (Ikkala et al.,
2022) to Unity applications, which we built based on the following principles
outlined in the remainder of this section:

  * •

Simulated Hardware

  * •

Standardised VR Interaction Library

  * •

Direct Access to the VR Environment

  * •

Synchronised Simulators

We opted to use Unity, as it provides an extensive set of VR/AR features and
has a large community of VR developers. The VR application in the simulated
interaction loop can be any existing Unity application or a prototype thereof,
either built as a standalone app or run within the Unity Editor.

Figure 3. With SIM2VR, simulated users for the first time have access to
exactly the same visual output as humans. The top figure depicts how real
users perceive a VR environment (in this case, the Whac-A-Mole game from
Section 5), while the bottom figure shows the downsampled RGB-D image (120x80
pixels) that the simulated user perceives.

Sample renders of the Whac-A-Mole game as shown to real users (top;
annotation: ”Real User View (1280x960)”) and the simulated user (bottom;
annotation: ”Simulated User View (120x80)”), respectively.

The simulated user can be created within UitB, which requires selecting a
biomechanical model implemented in MuJoCo, as well as one or more perception
modules. These perception modules define how the simulated user perceives its
surroundings, e.g., via visual or proprioceptive signals. To model how users
specifically perceive the virtual environment when wearing an HMD, SIM2VR adds
a novel vision module denoted as UnityHeadset to UitB. This module enables the
simulated user to observe the rendered image sent to the virtual HMD as an
RGB-D array (see Figure 3).

In the UitB framework, the interaction environment and the rewards that
incentivise completing the given interaction task need to be defined in a so-
called task module. To enable using Unity applications as interaction
environments, SIM2VR provides a UnityEnv task module on the UitB side, and a
SIM2VR Asset on the Unity side. These are connected via ZeroMQ, which creates
a communication channel between the two simulators. A detailed overview of the
components of SIM2VR and their dependencies is given in Figure 4.

Figure 4. SIM2VR enables efficient data transfer between the simulated user
(implemented in UitB) and the VR application (implemented in Unity). This is
achieved by establishing a ZeroMQ connection between the generic UnityEnv task
module and the SIM2VR Asset, both provided by SIM2VR. The UnityEnv equips the
biomechanical model with the VR controller and the HMD. Visual perception of
the VR application is received through the UnityHeadset vision module, based
on the rendered HMD images sent to the UnityEnv. The SIM2VR Asset also allows
application-dependent rewards, such as game scores, to be defined directly in
the VR application using the RLEnv Unity script. The rewards can be used by
the simulated user to improve its control policy while interacting with the VR
application.

Components of the SIM2VR platform. The ”Simulated User” consists of a
”Biomechanical Model”, a ”Task Module”, a ”Learned Control Policy” and one or
multiple ”Perception Modules”, and is implemented in the UitB framework (which
makes use of MuJoCo). The ”VR Application” is implemented in Unity. SIM2VR
connects these two components through a ZeroMQ connection that allows to
transfer sensor input data (symbolised by a VR controller icon) from the
”Simulated User” to the ”VR Application”, and visual perceptions (symbolised
by an eye icon) in the opposite direction. This is achieved by adding the
”UnityEnv” task class and the ”UnityHeadset” vision module to the simulated
user, and the ”SIM2VR Asset” to the VR application. The latter also provides a
template for setting up the application’s ”RLEnv”. Additional arrows are drawn
from ”UnityEnv” to ”Biomechanical Model”, from ”UnityEnv” to ”UnityHeadset”
and from ”SIM2VR Asset” to ”RLEnv”. The ”RLEnv” block has a dashed border.

The following principles are essential for the functionality of SIM2VR:

#### Simulated Hardware

The simulated user needs to be equipped with VR controllers and an HMD in
order to be able to interact with the VR application in the same way as a real
user. To this end, the UnityEnv task module augments the biomechanical MuJoCo
model of the simulated user with respective hardware objects.666The module
currently comes with mesh files of the Meta Quest 1 & 2
(https://www.meta.com/quest/), however, these can be easily replaced with
models of other VR equipment. In particular, it ensures that the VR
controllers and HMD are rigidly attached to the hands and head of the
biomechanical model.

#### Standardised VR Interaction Library

To achieve input channel alignment, the VR application needs to sense the
location and orientation of simulated hardware just as if a physical
controller and HMD were used. SIM2VR enables such hardware-agnostic input
handling by building on the OpenXR777https://www.khronos.org/openxr/. SIM2VR
requires the Unity application to handle interaction with VR devices using the
OpenXR plugin (version 1.5.3 or later). standard.

#### Direct Access to the VR Environment

By adding the SIM2VR Asset to the desired Unity application, SIM2VR gains
direct and genuine access to all Unity variables relevant for the interaction.
This has several advantages. First, it permits capturing the output signals,
such as the rendered image displayed on the HMD, and directly forwards these
signals to the simulated user.888While SIM2VR is currently limited to the
transmission of visual output signals, we plan to support other feedback
modalities such as auditory and haptic output.  It is thus key in ensuring
output channel alignment. Second, it can be used to define application-
specific rewards, e.g., based on a game score. This is done in the RLEnv
script provided by the SIM2VR Asset. Third, it allows to improve the RL
training by providing further stateful information (e.g., the time passed in
an episode, or the number of completed subtasks) as observation to the
simulated user, and to log999To facilitate troubleshooting, we have added
Weights & Biases (https://wandb.ai/site) integration, which allows real-time
monitoring of the simulated user’s performance. any Unity variables of
interest (e.g., the current performance in terms of specific subtasks).

#### Synchronised Simulators

The physics engine of the user simulation and the VR application need to be
temporally synchronised. SIM2VR achieves this by including the current as well
as the next timestamp of the user simulation as metadata in the data packages
sent to Unity. These timestamps are used to halt the VR application updates
until the same amount of time has elapsed as in the user simulation. To avoid
computational overhead, the Unity scene is rendered exactly once per UitB time
step, and only relevant data is transferred between the two simulators. Faster
than real time simulations are achieved by modifying the Unity timescale
parameter (in our experiments we ran Unity at 5 times real time). The UnityEnv
module can also trigger the reset function of the SIM2VR Asset and vice versa.
This is essential to ensure that both the simulated user and the VR
application are reset together when necessary, for example, to start a new
episode during RL training, or after completing a game level.

###  4.2. Creating the Simulated User

The remaining three alignment problems – physical, perceptual, and cognitive –
need to be addressed by improving the quality of the simulated user. While
this is not the main focus of SIM2VR, we also provide tools that aim to
support VR designers and researchers in this regard.

#### Reach Envelope Tool

Figure 5. The reach envelope tool provided by SIM2VR visualises the range that
the simulated user (black cross, facing in z-axis direction) can theoretically
reach with a maximally extended arm (blue lines) along with target positions
relevant for the considered VR application (coloured dots). This example shows
that in the Whac-A-Mole game (Section 5), all target positions in all three
target area placements can be reached by the simulated user, albeit some of
the low targets (those close to the boundary of the envelope) require a fully
extended arm.

In a 3D plot, the area that can be reached by the simulated user with
maximally extended arm is shown as blue lines from a bird’s eye view. A black
cross symbolises the position of the simulated user. The figure suggests that
the target positions of all three target area placement conditions in the
Whac-A-Mole game (visualised by cyan, green and magenta dots for low, mid and
high conditions, respectively) are within this reach envelope. Figure title:
”ReachEnvelope Check”

As the biomechanical model typically has fewer degrees-of-freedom (DOFs) than
a real human body, it is important to ensure that it is still able to perform
all the movements required by the interaction. For a model of the upper
extremity used to simulate visuomotor interaction in VR, this mainly involves
verifying that all target positions are within the model’s reach envelope. The
reach envelope tool provided by SIM2VR allows interactive exploration and
visualisation of the body-centred positions that a given simulated user can
theoretically reach. This can be used to verify that all target positions that
occur in a given VR interaction task can be reached by the simulated user. The
tool works as follows. Given a simulated user, its reach envelope is computed
by identifying all VR controller positions that can be reached with a
maximally extended arm. These positions are then transformed into Unity
coordinates and plotted in 3D along with desired target positions (see Figure
5 for an example plot; the tool can also create an animated 360° plot for
better spatial perception). Target positions can either be provided manually
(in Unity coordinates) or can be directly extracted from a Unity log file. The
tool also allows one to distinguish between the envelope of the pure
biomechanical model and the envelope when equipped with a VR controller, which
can be used to verify correct placement of the controller mesh.

Running this tool for a given simulated user and a given VR interaction task
can help improve the physical alignment.

#### Reward Scaling Tool

Reward functions required to learn biomechanically plausible behaviour
typically involve multiple components. For example, effort costs are usually
added to the reward to incentivise accomplishing the task with as little
muscle effort as possible (Todorov and Jordan, 2002; Croxson et al., 2009).
However, balancing these reward terms is a non-trivial task and typically
requires some RL experience.

To this end, we provide a reward scaling tool that facilitates reward function
design for visuomotor VR interaction tasks. This tool takes the selected
reward components (e.g., a one-time bonus reward given at task completion, a
dense reward continuously awarded based on the current distance to all active
targets, and the effort costs) as input, and then predicts how the
contribution of each component to the total reward evolves over time.
Predictions are generated by evaluating the reward components for different
scenarios such as worst case (i.e., when the simulated user has not learned to
make any movements yet), best case (i.e., when the desired target position is
already reached at beginning of an episode), and linear or quadratic
interpolation between initial and target position. The resulting plots can be
used to identify poorly scaled reward components that either have too large of
an effect on the simulated user or no effect at all, in which case these
components need to be differently weighted.

By helping to identify an appropriate reward function, this tool represents a
first step towards better cognitive alignment between simulated and real
users, which is certainly the most difficult dimension of the alignment
problem to address. A discussion on the current capabilities of simulated
users for modelling different VR interaction tasks, and how these might
benefit from more sophisticated RL methods recently proposed, can be found in
Section 6.

##  5\. Predicting Interactive User Behaviour with SIM2VR

In this section, we show how SIM2VR can be used to predict movement-based
interaction in VR and demonstrate its utility in evaluating VR interface
design choices. We developed a VR version of the popular, visuomotorically
demanding, fast-paced and dynamic arcade game Whac-A-Mole in Unity with
different interface variations. With SIM2VR, we train a UitB model to play the
game, and then use this model to predict differences in performance, effort
and strategies between different game variations. To assess the quality of
these predictions, we compare them to reference data from a user study, where
real humans were asked to play the same VR game.

We re-implemented our own version of the Whac-A-Mole game, as this allows us
to investigate in a controlled way the extent to which SIM2VR can predict the
effect of design choices such as difficulty or target area placement. In the
supplementary material, we show how to integrate SIM2VR into an existing
(third-party) Unity application and discuss how to create a simulated user in
UitB.

###  5.1. Game Design

(a) Front View with Target Grid

(b) Target Area Placements

Figure 6. (a) In Whac-A-Mole, targets appear randomly at one of nine fixed
positions and must be hit with a hammer within one second to score a point.
The transparent targets are shown for visualization purposes only and are not
visible during the game. (b) The game allows for three different placements of
the target area. Top (subcaption: ”Front View of Target Grid”): Front view of
the target plane and the target grid in the Whac-A-Mole game, together with
the controlled hammer hitting a target. Bottom (subcaption: ”Target Area
Placements”): Three target areas (low, mid, high) differing in terms of
position and orientation are shown in 3D space together with the controlled
hammer.

In the Whac-A-Mole game the goal is to hit targets (the ”moles”) with a hammer
to score points. The targets appear randomly on a $3\times 3$ grid as shown in
Figure 6a. If a target is successfully hit within one second, it explodes and
disappears (see Figure 2) and the game score is increased by one. Otherwise,
the target collapses after one second and no points are given. One round of
game play lasts for one minute.

We explore three design choices. First, the game has three difficulty levels
defining the maximum number of simultaneously displayed targets (easy: 1,
medium: 3, hard: 5). The second design choice, target area placement,
determines the position and orientation of the target area with respect to the
HMD (low, mid, high), as shown in Figure 6b. For the third design choice, we
create two versions of the game. In the constrained version, a velocity
threshold needs to be exceeded in order to successfully hit a target, whereas
in the unconstrained version, no such threshold exists. Further details on the
game design are provided in the supplementary material.

###  5.2. Training Simulated Users

Figure 7. Still frames from an evaluation of a simulated user as it is hitting
a target on the right side of the target area. The top row shows the simulated
user in MuJoCo, while the bottom row is a video capture from the Unity
application.

2x3 still frames of a biomechanical user model equipped with the right VR
controller and HMD playing the Whac-A-Mole game. The simulated user (top) and
the corresponding VR application view (bottom) are shown below each other for
three different time frames.

We train a UitB user model to play the different variations of the game. We
use MoblArmsWrist, a MuJoCo version of the MoBL ARMS model (Saul et al., 2014)
of the right arm, as the biomechanical model. This model includes shoulder,
elbow and wrist joints, and the muscles actuating these joints, resulting in a
total of 7 DOFs and 32 muscles. Since the model does not actively model neck
and head movements, the position and orientation of the HMD were kept constant
throughout the simulation. We use the UnityHeadset vision module provided by
SIM2VR, along with the pre-implemented UitB proprioception module, to generate
observations during training. The UnityHeadset module is configured to include
the green colour and depth channels, without stacking visual observations. The
proprioception module provides observations regarding the biomechanical model,
such as the joint positions, velocities, and accelerations, as well as
internal muscle activation states.

For training, we use a reward function consisting of multiple components:

(1) $r(s,a)=w_{s}*S+w_{c}*v_{h}*C_{c}+w_{d}*C_{d}+w_{e}*C_{e}$

The component $S$ is the game score provided by the game; $C_{c}$ is a term to
penalize target contacts with too low a hitting velocity in the constrained
version (to encourage faster hits); $C_{d}$ penalises the distance between the
hammer and each active target (to incentivise movement toward any target); and
$C_{e}$ is an effort term based on the 3CC-r fatigue model (Looft et al.,
2018; Cheema et al., 2020). The reward components are weighted with
$w_{s}=10$, $w_{c}=2.5$, $w_{d}=1$ and $w_{e}=0.1$. The target contact term
$C_{c}$ is additionally weighted by the current hammer velocity $v_{h}$.

Simulated users were trained separately for the constrained and unconstrained
versions, while keeping the difficulty level fixed to medium and sampling the
target area placements randomly at the beginning of each episode.101010While
it may be reasonable to continue training the user models on either the easy
or hard difficulty levels, we observed that the models trained solely on the
medium level already generalise well to the remaining difficulties. For each
version we train two instances of the simulated user, called SIMu and SIMa.
SIMu is trained on the default game dynamics, i.e., the position of the ”mole”
is sampled uniformly randomly among the nine possible positions. SIMa, on the
other hand, is trained using an adaptive automated curriculum manually
implemented in Unity. Such a curriculum can be thought of as a “personal
trainer” that creates a customised training plan for the simulated user based
on its current strengths and weaknesses.

The learned control policies were evaluated for 1) all three difficulty levels
while keeping the target area placement fixed to mid, and 2) for all target
area placements while keeping the difficulty level fixed to medium (six
configurations in total). Further details on the training and evaluation
procedure can be found in the supplementary material. A frame sequence of the
simulated user playing the game with the learned control policy is shown in
Figure 7.

###  5.3. User Study Method

In order to compare the simulated user behaviour against real player
behaviour, we conducted a user study where we collected game playing data from
18 right-handed participants. Twelve participants played the the constrained
version of the game, while six participants played the unconstrained version.
Each participant played the same six difficulty $\times$ target area placement
configurations used to evaluate the simulated users.

#### Participants

The participants were mostly local students (graduate or post-graduate) and
faculty members at Aalto University, Finland. The average age was 28.8 years,
with a standard deviation of 6.2 years. Eight of the participants identified
as female, nine as male, and one as non-binary.

#### Task

The participants first played the three difficulty levels, while keeping the
target area placement fixed to mid, in a counterbalanced order. Participants
were then given a rest period of at least 5 minutes, with the option to rest
longer if necessary to ensure their right arm was not fatigued. After resting,
participants played the game in all three target area placements, in a
counterbalanced order, with the difficulty level fixed to medium, and reported
the Borg Rating of Perceived Exertion (Borg RPE) (Borg, 1982) score after each
round.

#### Procedure

The experiments were conducted in an office room. The participants were
seated, but not restricted to the chair. Participants were asked to try to
avoid bending and rotating their torso (as the simulated users have a fixed
torso), and focus on their arm movements. Participants were allowed to train
with the medium difficulty level with mid target area placement. This was to
reduce learning effects, and to provide the participants a chance to
familiarise themselves with the notion of hitting non-physical floating
targets. The experiment length was approximately 30 minutes, and participants
were rewarded with a 15€ gift card to a local restaurant.

###  5.4. Results

We demonstrate the predictive capabilities of SIM2VR by comparing the
performance and effort estimates of simulated and real users in the different
variations of the game. We also show that SIM2VR is capable of predicting
specific strategies that users may exploit when game dynamics allow.

Unless otherwise stated, the following results were obtained with the
constrained game version.

####  5.4.1. Performance

Figure 8. The simulated users SIMu and SIMa (left and mid bars) show similar
numbers of target hits and misses in the Whac-A-Mole game as the 12 real users
from our study on average (right bars). Differences between the three
difficulty levels are predicted well.

Bar plot showing the average number of target hits and misses (y-axis; hits
shown as green bars, misses shown as orange bars stacked on top) per
difficulty level (x-axis; ”Easy” on the left, ”Medium” in the middle, ”Hard”
on the right) and for the two simulated users ”SIM_u” and ”SIM_a” and averaged
across all real users (denoted as ”Users”). Figure title: ”Target Hits and
Misses”

Figure 9. Both SIMu and SIMa predict target hit rates that are within the
between-user variability observed from real players. The simulations also show
a similar decrease in hit rate as the game difficulty increases as most real
players.

Scatter plot separately showing the target hit rate (y-axis) per difficulty
level (”Easy” shown in blue, ”Medium” shown in orange, ”Hard” shown in green)
for the 12 real users and the two simulated users ”SIM_u” and ”SIM_a”, which
are highlighted by vertical red stripes (x-axis). The 14 users are sorted in
ascending order by the average hit rate across the three levels (08, 04, 06,
09, SIM_u, 01, 07, 11, 12, SIM_a, 10, 03, 02, 05). Figure title: ”Target Hit
Rates – Simulated vs. Real Users”

In Whac-A-Mole, the performance (and thus the difficulty) of a game variant
can be assessed by the number of target hits and misses users achieve per
round. In Figure 8, the mean number of hits and misses from 12 rounds of game
play is shown for the two simulated users SIMu and SIMa for each level, along
with the means of the 12 real users. For real players (solid, right bars),
both target hits and misses increase from the easy to the medium level.
However, this trend does not continue when proceeding to the hard level.
Instead, the number of hits remains approximately constant (green), while the
number of misses (orange) increases. A similar trend can be observed for both
SIMu and SIMa (shown as left and mid bars, respectively). Thus, if used early
in the design process, the simulated users could have suggested that
increasing the task difficulty beyond the medium level would not lead to
higher game scores for average users, but only to more target misses.

In Figure 9, the target hit rate is plotted separately for all users in each
difficulty level. While there is a considerable variability in the performance
of real users, both simulations consistently predict hit rates that lie within
this range. In addition, the decrease in hit rate with increasing game
difficulty predicted by both simulated users is also observed for almost all
real users (the only exceptions are users 9 and 11, who achieved higher hit
rates on the hard than on the medium level).

####  5.4.2. Effort

User Variable Placement Mean/ Median† Std./ IQR† Alternative Hypothesis
Wilcoxon Signed Rank Z-score p-value SIMu Max. Fatig. MUs Low 0.232342
0.001100 Low ¡ Mid 3.059 1.0 (n.s.) Mid 0.203537 0.001641 Mid ¡ High -3.059
0.0002 (***) High 0.238448 0.001772 Low ¡ High -3.059 0.0002 (***) SIMa Max.
Fatig. MUs Low 0.265645 0.002151 Low ¡ Mid -2.51 0.0046 (**) Mid 0.269120
0.002360 Mid ¡ High 3.059 1.0 (n.s.) High 0.229852 0.001937 Low ¡ High 3.059
1.0 (n.s.) User Study Borg RPE Low 9† 1.25† Low ¡ Mid -1.508 0.0658 (n.s.) Mid
9† 3† Mid ¡ High -2.414 0.0079 (**) High 9.5† 4.25† Low ¡ High -2.213 0.0134
(*)

Table 2.  Overview of both descriptive and inferential statistics for effort
predictions. Descriptive statistics include the mean and standard deviation
for the maximum fatigued motor units per round (averaged over all muscles), as
predicted by the simulation, and the median and interquartile range (IQR) for
the Borg RPE as reported in the user study (†)). Kolmogorov-Smirnov tests
showed that none of the considered variables are normally distributed (all
p-values ¡ 0.001); we thus used one-sided Wilcoxon Signed rank tests to infer
significant differences between the three target area placements.
Explanations: *: $p\leq 0.05$; **: $p\leq 0.01$; ***: $p\leq 0.001$; n.s.:
$p>0.05$.
Abbreviations: Std. for standard deviation, IQR for interquartile range, Max.
Fatig. MUs for maximum fatigued motor units and n.s. for not significant.

The table consists of three parts. The left part has the columns ”User” and
”Variable” and the following rows: 1\. SIM_u - Max. Fatig. MUs. 2\. SIM_a -
Max. Fatig. MUs. 3\. User Study - Borg RPE. The second part, which is placed
right next to the first part, has the columns ”Placement”, ”Mean/Median †” and
”Std./IQR †” and the following rows (three subrows per row of the first part):
1a. Low - 0.232342 - 0.001100. 1b. Mid - 0.203537 - 0.001641. 1c. High -
0.238448 - 0.001772. 2a. Low - 0.265645 - 0.002151. 2b. Mid - 0.269120 -
0.002360. 2c. High - 0.229852 - 0.001937. 3a. Low - 9 † - 1.25 †. 3b. Mid - 9
† - 3 †. 3c. High - 9.5 † - 4.25 †. The third part, which is placed right next
to the first second, has the columns ”Alternative Hypothesis”, ”Wilcoxon
Signed Rank – Z-score” and ”Wilcoxon Signed Rank – p-value” and the following
rows (three subrows per row of the first part): 1a. Low ¡ Mid - 3.059 - 1.0
(n.s.). 1b. Mid ¡ High - -3.059 - 0.0002 (***). 1c. Low ¡ High - -3.059 -
0.0002 (***). 2a. Low ¡ Mid - -2.51 - 0.0046 (**). 2b. Mid ¡ High - 3.059 -
1.0 (n.s.). 2c. Low ¡ High - 3.059 - 1.0 (n.s.). 3a. Low ¡ Mid - -1.508 -
0.0658 (n.s.). 3b. Mid ¡ High - -2.414 - 0.0079 (**). 3c. Low ¡ High - -2.213
- 0.0134 (*). In the ”Wilcoxon Signed Rank – p-value” column, all cells with
at least one ”*” (i.e., those showing significant differences) are highlighted
in green.

To assess the impact of target area placement on movement effort, participants
in the user study were asked to report their perceived exertion in terms of
the Borg RPE scale, after playing each 1-minute round. The Borg RPE reported
for the high placement was significantly higher than for both the low and mid
placements, while no significant difference could be found between low and mid
placements (summary statistics as well as details on the statistical testing
are given in Table 2). This also agrees well with qualitative remarks of most
of the participants, stating that the arm felt considerably more fatigued
after playing the high placement as compared to the low and mid placements. To
estimate the level of exertion from our simulation, we measured the percentage
of fatigued motor units as predicted by the 3CC-r fatigue model for each
simulation time frame, calculated the maximum value per round for each muscle
separately, and then took the average over all 32 muscles.

As shown in Table 2, SIMu predicts a significant increase in fatigue between
the mid and high as well as between the low and high placements, showing the
same trend as observed for the Borg RPE scale for real players. Interestingly,
the fatigue predicted for the low placement is considerably higher than in mid
placement. Comparing the simulation videos for these placements, we found that
in the low placement SIMu follows a strategy that hardly flexes the elbow, but
rather keeps the arm extended for almost the entire movement, which may lead
to higher muscle fatigue over time.

For SIMa, the mid placement exhibits significantly higher fatigue values than
the low placement ($p=0.0046$), whereas no significant increase from mid to
high or from low to high could be found. We found that this is mainly due to
different strategies for each of the three placements, despite being trained
together and sharing the same network weights. In particular, SIMa has learned
to efficiently reduce unnecessary shoulder movements mainly in the high
placement, resulting in a lower number of fatigued motor units than in the low
and mid placements.

Our findings can be seen as an example of how small changes in the training
may have a considerable impact on which strategies are learned and predicted
by the user simulation. While we demonstrate that characteristic differences
in subjective levels of muscle exertion can be predicted in principle by our
simulation-based approach, effort-related predictions must currently be
treated with caution, as the effort can vary with small changes in the
movement strategy.

####  5.4.3. Strategy

Figure 10. In the unconstrained variant our simulation predicts a different
strategy, with the hammer kept close to the target plane during the entire
movement (orange line in left plot). This strategy was also observed among
real players (orange line in right plot). The hammer depth trajectories of
(different) simulated and real players interacting with the constrained
version are shown for comparison (blue lines). All movements are in the hard
level, but similar results were also observed for the medium level.

Left (Subfigure title: ”Simulated Users – Hammer Depth During Movement”):
Hammer depth in meters (y-axis) plotted over time (x-axis) for an entire round
of 60 seconds, once for a simulated user movement in the constrained variant
(blue line) and once in the unconstrained variant (orange line). Right
(Subfigure title: ”Real Users – Hammer Depth During Movement”): Hammer depth
in meters (y-axis) plotted over time (x-axis) for an entire round of 60
seconds, once for a real user movement in the constrained variant (blue line)
and once in the unconstrained variant (orange line).

Simulations trained via SIM2VR can anticipate the effect of a simple game
design choice on the strategies employed by users. In Figure 10 this is shown
for a trajectory comparison between the constrained and the unconstrained
version of the game. In the unconstrained version, an entirely different
strategy is predicted by our simulation; in particular, the unconstrained
policy retracts the arm considerably less, and instead keeps the hammer
relatively close to the target area during movements, which indeed allows to
hit many more targets in the same time.111111Applying the two simulation
strategies to the hard level, 195 target hits were observed in the
unconstrained variant, and 102 target hits and 35 additional target contacts
with too low hitting velocity were observed in the constrained variant.
Interestingly, one of the six users who played the unconstrained version of
the game exploited the same strategy, as shown in Figure 10 (right, orange
line) along with a reference user trajectory from the constrained version of
the game (blue line).121212Another user pursued a similar strategy but hit the
targets more from above, while the remaining four users followed essentially
the same strategy as the users in the constrained version.

This demonstrates the value of biomechanical user models in discovering
specific strategies that users may exploit for a given game design. By making
such insights available early in the VR development process, SIM2VR can help
find game dynamics that meet the designers’ requirements for performance,
effort, comfort, space limitations and more.

###  5.5. Summary

This demonstration shows that SIM2VR can be utilised to predict differences in
terms of performance, effort and strategies between different versions of a VR
game. Our approach enables the evaluation and comparison of different
prototypes and design choices, and thus could be used in addition to (or as a
partial replacement for) user studies in the early stages of application
development.

##  6\. Discussion

Computational approaches to modelling and simulation of users have the
potential to be of value in design and analysis of interactive systems.
However, in addition to the challenges of modelling the complexity of human
behaviour, a significant challenge is modelling the sensory experience of
users, who can be in complex environments and subject to unmodelled external
disturbances to all senses, which can reduce the potential benefits of a “sim-
to-real” transfer (a term mainly used in the field of robotics (Dimitropoulos
et al., 2022)).

VR interaction, however, is a special case of human–computer interaction in
that significant aspects of the user’s perception (namely vision and audio)
can be controlled by the VR application to a greater extent than other cases.
In particular, the (currently) low number of different VR systems, the “full-
screen mode” of most VR applications and the typically high level of immersion
(Seinfeld et al., 2020) reduces the impact of external disturbances when
assessing performance. VR interaction thus offers ideal conditions for
creating ecologically valid simulations and performing automated user testing.
This makes VR a good starting point for research into simulation models which
have well-aligned information from visual and auditory displays.

With SIM2VR, we have built on these ideal conditions and developed a system
that does not only give a biomechanical user model access to the same VR
application as real users, but also ensures that it can interact with the
application just as if a physical VR controller and HMD were used. This
constitutes a decisive step towards automated biomechanical testing in VR. In
the future, biomechanical user simulations could lead to more comfortable and
health-promoting VR interactions, as well as VR environments that specifically
consider the abilities of individual user groups.

###  6.1. Scope and Limitations

SIM2VR mainly addresses the application-related dimensions of the alignment
problem. Aligning the user-related dimensions, i.e., physics, perception,
cognition, however, remains challenging. This particularly includes finding an
appropriate biomechanical user model, modelling multisensory noisy perception,
and designing a task-specific reward function, as we discuss in the following.

Due to its dependency on UitB, SIM2VR currently only provides the standard
MoBL ARMS model (Saul et al., 2014), which may be unsuitable for simulating
dexterous hand movements or body rotations. However, more sophisticated models
of the human body focusing on either the elbow, hand, single finger or lower
extremity have recently been released as part of the MyoSuite
project131313https://sites.google.com/view/myosuite and can be integrated
directly into UitB. In addition, the MyoConverter (Ikkala and Hämäläinen,
2020; Wang et al., 2022) can be used to transfer OpenSim models to MuJoCo.

Regarding the interaction technique, in this paper we have focused on one-
handed VR interaction using a standard VR controller and HMD. An extension of
SIM2VR to other movement-based methods such as full body movements sensed via
Motion Capture is straightforward. The same applies to currently missing
output modalities such as auditory or haptic feedback.

The vision module provided by SIM2VR currently only models the user’s visual
perception of the virtual environment at a very rudimentary level. To better
reflect the complexity and subtlety of visuomotor VR interaction, this module
should be extended in the future to include, for example, binocular, foveal,
noisy and/or delayed visual observations. Initial ideas for modelling these
well-established phenomena might be derived from (Strasburger et al., 2011;
Kowler, 2011; Fischer et al., 2022).

Simulating user cognition is known to be particularly challenging. In the
context of reinforcement learning, a major part of cognition modelling is to
identify appropriate reward terms that incentivise “human-like” behaviour.
However, no general guidelines have yet been proposed on how to define these
reward terms appropriately for a given interaction task. Additionally, reward
functions can easily induce unintended behaviour, if not designed carefully.
For example, a reward signal that penalises the remaining distance to multiple
positions at the same time (e.g., to all currently active targets) will
encourage an ”intermediate” strategy that opts for a particular target
position later rather than sooner. Such a “strategic bias” may also degrade
the quality of simulation-based predictions. The availability of highly
skilled and approved simulated users for a variety of interaction tasks will
therefore be critical to the extent to which biomechanical simulation finds
its way into future automated VR testing routines.

In order to expand the scope of automated biomechanical testing beyond simple
visuomotor interaction tasks, further research into (and validation of) more
complex RL-based training methods and design choices of reward functions and
metaparameters is certainly needed. Promising RL methods and frameworks in
this regard include Hierarchical RL (Li et al., 2019; Nachum et al., 2018), in
which a complex learning problem is decomposed into simpler subtasks that are
easier to learn, and Transfer Learning (Zhu et al., 2020), in which models are
first exposed to a data-rich domain before being fine-tuned in a data-scarce
or otherwise more challenging environment. Moreover, recent successes in
training user models for cognitively and biomechanically challenging motor
control and interaction tasks, such as rotating multiple Baoding balls over
the palm of the hand (Caggiano et al., 2022b) or parking a remote-controlled
car using a joystick (Ikkala et al., 2022), raise hopes for versatile
simulated users that can be integrated directly into a range of existing VR
applications without extensive refinement.

##  7\. Conclusion

The SIM2VR system presented in this paper enables the training of RL-based
simulated users directly in the same VR environments that real users interact
with. By partially addressing the VR Simulation Alignment Problem, our
approach increases the ecological validity of biomechanical user simulations
in VR. Using Whac-A-Mole as an example, we have demonstrated that SIM2VR can
predict characteristic differences between different game variants in terms of
performance and effort, and reveal potential user strategies. By lowering the
barrier to the use of biomechanical simulation, we hope that SIM2VR will
increase the adoption of automated testing methods in VR development. Given
that biomechanical user models can help identify bugs, safety risks and health
issues prior to user testing, we expect user modeling to assume a position as
an indispensable part of the VR development process in the future.

## References

  * (1)
  * Ayoub et al. (1974) M. A. Ayoub, M. M. Ayoub, and A. G. Walvekar. 1974.  A Biomechanical Model for the Upper Extremity using Optimization Techniques.  [Human Factors] 16, 6 (1974), 585–594.  https://doi.org/10.1177/001872087401600603 arXiv:https://doi.org/10.1177/001872087401600603  PMID: 4442903.
  * Banovic et al. (2016) Nikola Banovic, Tofi Buzali, Fanny Chevalier, Jennifer Mankoff, and Anind K. Dey. 2016\.  [Modeling and Understanding Human Routine Behavior].  Association for Computing Machinery, New York, NY, USA, 248–260.  https://doi.org/10.1145/2858036.2858557
  * Bierbaum et al. (2003) Allen Bierbaum, Patrick Hartling, and Carolina Cruz-Neira. 2003\.  Automated testing of virtual reality application interfaces. In [Proceedings of the workshop on Virtual Environments 2003]. 107–114.
  * Borg (1982) Gunnar A Borg. 1982\.  Psychophysical bases of perceived exertion.  [Medicine and science in sports and exercise] 14, 5 (1982), 377–381.
  * Caggiano et al. (2022a) Vittorio Caggiano, Huawei Wang, Guillaume Durandau, Massimo Sartori, and Vikash Kumar. 2022a.  MyoSuite–A contact-rich simulation suite for musculoskeletal motor control.  [arXiv preprint arXiv:2205.13600] (2022).
  * Caggiano et al. (2022b) Vittorio Caggiano, Huawei Wang, Guillaume Durandau, Seungmoon Song, Yuval Tassa, Massimo Sartori, and Vikash Kumar. 2022b.  MyoChallenge: Learning contact-rich manipulation using a musculoskeletal hand.  https://sites.google.com/view/myochallenge.
  * Card et al. (1983) Stuart K. Card, Thomas P. Moran, and Allen Newell. 1983\.  [The psychology of human-computer interaction].  Crc Press.
  * Cheema et al. (2020) Noshaba Cheema, Laura A. Frey-Law, Kourosh Naderi, Jaakko Lehtinen, Philipp Slusallek, and Perttu Hämäläinen. 2020.  Predicting Mid-Air Interaction Movements and Fatigue Using Deep Reinforcement Learning. In [Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems] (Honolulu, HI, USA) [(CHI ’20)]. Association for Computing Machinery, New York, NY, USA, 1–13.  https://doi.org/10.1145/3313831.3376701
  * Chen et al. (2015) Xiuli Chen, Gilles Bailly, Duncan P Brumby, Antti Oulasvirta, and Andrew Howes. 2015\.  The Emergence of Interactive Behaviour: A Model of Rational Menu Search.  (2015).
  * Chen et al. (2017) Xiuli Chen, Sandra Dorothee Starke, Chris Baber, and Andrew Howes. 2017.  A cognitive model of how people make decisions through interaction with visual displays. In [Proceedings of the 2017 CHI conference on human factors in computing systems]. 1205–1216.
  * Croxson et al. (2009) Paula L. Croxson, Mark E. Walton, Jill X. O’Reilly, Timothy E. J. Behrens, and Matthew F. S. Rushworth. 2009.  Effort-Based Cost–Benefit Valuation and the Human Brain.  [Journal of Neuroscience] 29, 14 (2009), 4531–4541.  https://doi.org/10.1523/JNEUROSCI.4515-08.2009 arXiv:https://www.jneurosci.org/content/29/14/4531.full.pdf
  * Damsgaard et al. (2006) Michael Damsgaard, John Rasmussen, Søren Tørholm Christensen, Egidijus Surma, and Mark de Zee. 2006\.  Analysis of musculoskeletal systems in the AnyBody Modeling System.  [Simulation Modelling Practice and Theory] 14, 8 (2006), 1100–1111.  https://doi.org/10.1016/j.simpat.2006.09.001 SIMS 2004.
  * Delp et al. (2007) Scott Delp, Frank Anderson, Allison Arnold, Peter Loan, A. Habib, Chand John, Eran Guendelman, and Darryl Thelen. 2007\.  OpenSim: Open-Source Software to Create and Analyze Dynamic Simulations of Movement.  [Biomedical Engineering, IEEE Transactions on] 54 (12 2007), 1940 – 1950.  https://doi.org/10.1109/TBME.2007.901024
  * Dembia et al. (2021) Christopher L. Dembia, Nicholas A. Bianco, Antoine Falisse, Jennifer L. Hicks, and Scott L. Delp. 2021\.  OpenSim Moco: Musculoskeletal optimal control.  [PLOS Computational Biology] 16, 12 (12 2021), 1–21.  https://doi.org/10.1371/journal.pcbi.1008493
  * Dimitropoulos et al. (2022) Konstantinos Dimitropoulos, Ioannis Hatzilygeroudis, and Konstantinos Chatzilygeroudis. 2022\.  A Brief Survey of Sim2Real Methods for Robot Learning. In [Advances in Service and Industrial Robotics], Andreas Müller and Mathias Brandstötter (Eds.). Springer International Publishing, Cham, 133–140.
  * Fischer et al. (2021) Florian Fischer, Miroslav Bachinski, Markus Klar, Arthur Fleig, and Jörg Müller. 2021\.  Reinforcement Learning Control of a Biomechanical Model of the Upper Extremity.  [Scientific Reports] 11, 1 (2021), 1–15.
  * Fischer et al. (2022) Florian Fischer, Arthur Fleig, Markus Klar, and Jörg Müller. 2022.  Optimal Feedback Control for Modeling Human-Computer Interaction.  [ACM Trans. Comput.-Hum. Interact.] 29, 6, Article 51 (april 2022), 70 pages.  https://doi.org/10.1145/3524122
  * Fitts (1954) Paul M Fitts. 1954\.  The information capacity of the human motor system in controlling the amplitude of movement.  [Journal of experimental psychology] 47, 6 (1954), 381\.
  * Graves et al. (2013) Alex Graves, Abdel-rahman Mohamed, and Geoffrey Hinton. 2013\.  Speech recognition with deep recurrent neural networks. In [2013 IEEE international conference on acoustics, speech and signal processing]. Ieee, 6645–6649.
  * Harms (2019) Patrick Harms. 2019\.  Automated usability evaluation of virtual reality applications.  [ACM Transactions on Computer-Human Interaction (TOCHI)] 26, 3 (2019), 1–36.
  * He et al. (2016) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016\.  Deep residual learning for image recognition. In [Proceedings of the IEEE conference on computer vision and pattern recognition]. 770–778.
  * Hetzel et al. (2021) Lorenz Hetzel, John Dudley, Anna Maria Feit, and Per Ola Kristensson. 2021.  Complex Interaction as Emergent Behaviour: Simulating Mid-Air Virtual Keyboard Typing using Reinforcement Learning.  [IEEE Transactions on Visualization and Computer Graphics] (2021), 1–1.  https://doi.org/10.1109/TVCG.2021.3106494
  * Hunt et al. (2014) Chris J Hunt, Guy Brown, and Gordon Fraser. 2014.  Automatic testing of natural user interfaces. In [2014 IEEE Seventh International Conference on Software Testing, Verification and Validation]. IEEE, 123–132.
  * Ikkala et al. (2022) Aleksi Ikkala, Florian Fischer, Markus Klar, Miroslav Bachinski, Arthur Fleig, Andrew Howes, Perttu Hämäläinen, Jörg Müller, Roderick Murray-Smith, and Antti Oulasvirta. 2022.  Breathing Life Into Biomechanical User Models. In [Proceedings of the 35th Annual ACM Symposium on User Interface Software and Technology] (Bend, OR, USA) [(UIST ’22)]. Association for Computing Machinery, New York, NY, USA, Article 90, 14 pages.  https://doi.org/10.1145/3526113.3545689
  * Ikkala and Hämäläinen (2020) Aleksi Ikkala and Perttu Hämäläinen. 2020.  Converting Biomechanical Models from OpenSim to MuJoCo.  , 277–281 pages.  arXiv:2006.10618 [q-bio.QM] https://arxiv.org/abs/2006.10618
  * Jang et al. (2017) Sujin Jang, Wolfgang Stuerzlinger, Satyajit Ambike, and Karthik Ramani. 2017.  Modeling Cumulative Arm Fatigue in Mid-Air Interaction Based on Perceived Exertion and Kinetics of Arm Motion. In [Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems] (Denver, Colorado, USA) [(CHI ’17)]. Association for Computing Machinery, New York, NY, USA, 3328–3339.  https://doi.org/10.1145/3025453.3025523
  * Jokinen et al. (2021) Jussi Jokinen, Aditya Acharya, Mohammad Uzair, Xinhui Jiang, and Antti Oulasvirta. 2021.  Touchscreen typing as optimal supervisory control. In [Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems]. 1–14.
  * Kangasrääsiö et al. (2017) Antti Kangasrääsiö, Kumaripaba Athukorala, Andrew Howes, Jukka Corander, Samuel Kaski, and Antti Oulasvirta. 2017.  Inferring Cognitive Models from Data Using Approximate Bayesian Computation. In [Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems] (Denver, Colorado, USA) [(CHI ’17)]. Association for Computing Machinery, New York, NY, USA, 1295–1306.  https://doi.org/10.1145/3025453.3025576
  * Karakaya et al. (2022) Kadiray Karakaya, Enes Yigitbas, and Gregor Engels. 2022\.  Automated UX Evaluation for User-Centered Design of VR Interfaces. In [International Conference on Human-Centred Software Engineering]. Springer, 140–149.
  * Klar et al. (2023) Markus Klar, Florian Fischer, Arthur Fleig, Miroslav Bachinski, and Jörg Müller. 2023\.  Simulating Interaction Movements via Model Predictive Control.  [ACM Trans. Comput.-Hum. Interact.] 30, 3, Article 44 (jun 2023), 50 pages.  https://doi.org/10.1145/3577016
  * Kowler (2011) Eileen Kowler. 2011\.  Eye movements: The past 25 years.  [Vision Research] 51, 13 (2011), 1457–1483.  https://doi.org/10.1016/j.visres.2010.12.014 Vision Research 50th Anniversary Issue: Part 2.
  * Kristensson and Müllners (2021) P. O. Kristensson and Th. Müllners. 2021.  Design and Analysis of Intelligent Text Entry Systems with Function Structure Models and Envelope Analysis. In [Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems]. 1–12.
  * Lacquaniti et al. (1983) Francesco Lacquaniti, Carlo Terzuolo, and Paolo Viviani. 1983\.  The law relating the kinematic and figural aspects of drawing movements.  [Acta Psychologica] 54, 1 (1983), 115 – 130.
  * Lavin et al. (2021) Alexander Lavin, Hector Zenil, Brooks Paige, David Krakauer, Justin Gottschlich, Tim Mattson, Anima Anandkumar, Sanjay Choudry, Kamil Rocki, Atılım Güneş Baydin, et al. 2021\.  Simulation Intelligence: Towards a New Generation of Scientific Methods.  [arXiv preprint arXiv:2112.03235] (2021).
  * Lee and Umberger (2016) Leng-Feng Lee and Brian R Umberger. 2016.  Generating optimal control simulations of musculoskeletal movement using OpenSim and MATLAB.  [PeerJ] 4 (2016), e1638.
  * Li et al. (2019) Alexander C Li, Carlos Florensa, Ignasi Clavera, and Pieter Abbeel. 2019.  Sub-policy adaptation for hierarchical reinforcement learning.  [arXiv preprint arXiv:1906.05862] (2019).
  * Liu et al. (2023) Qing Liu, Gustavo Alves, and Jian Zhao. 2023.  Challenges and Opportunities for Software Testing in Virtual Reality Application Development.  [Graphics Interface 2023 -second deadline] (2023).
  * Looft et al. (2018) John M Looft, Nicole Herkert, and Laura Frey-Law. 2018\.  Modification of a three-compartment muscle fatigue model to predict peak torque decline during intermittent tasks.  [Journal of biomechanics] 77 (2018), 16–25.
  * Martín et al. (2021) J. Alberto Álvarez Martín, Henrik Gollee, Jörg Müller, and Roderick Murray-Smith. 2021.  Intermittent control as a model of mouse movements.  [ACM Transactions on Computer-Human Interaction (TOCHI)] 28, 5 (2021), 1–46.
  * Mnih et al. (2015) Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. 2015\.  Human-level control through deep reinforcement learning.  [nature] 518, 7540 (2015), 529–533.
  * Müller et al. (2017) Jörg Müller, Antti Oulasvirta, and Roderick Murray-Smith. 2017\.  Control theoretic models of pointing.  [ACM Transactions on Computer-Human Interaction (TOCHI)] 24, 4 (2017), 1–36.
  * Murray-Smith et al. (2022) Roderick Murray-Smith, Antti Oulasvirta, Andrew Howes, Jörg Müller, Aleksi Ikkala, Miroslav Bachinski, Arthur Fleig, Florian Fischer, and Markus Klar. 2022\.  What simulation can do for HCI research.  [Interactions] 29, 6 (2022), 48–53.
  * Nachum et al. (2018) Ofir Nachum, Shixiang Shane Gu, Honglak Lee, and Sergey Levine. 2018.  Data-efficient hierarchical reinforcement learning.  [Advances in neural information processing systems] 31 (2018).
  * Nakada et al. (2018) Masaki Nakada, Tao Zhou, Honglin Chen, Tomer Weiss, and Demetri Terzopoulos. 2018.  Deep Learning of Biomimetic Sensorimotor Control for Biomechanical Human Animation.  [ACM Trans. Graph.] 37, 4, Article 56 (jul 2018), 15 pages.  https://doi.org/10.1145/3197517.3201305
  * Oulasvirta et al. (2022) Antti Oulasvirta, Jussi P. P. Jokinen, and Andrew Howes. 2022\.  Computational Rationality as a Theory of Interaction. In [Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems] [(CHI ’22)]. Association for Computing Machinery, New York, NY, USA, Article 359, 14 pages.  https://doi.org/10.1145/3491102.3517739
  * Quek (2013) Melissa Quek. 2013\.  [The role of simulation in developing and designing applications for 2-class motor imagery brain-computer interfaces].  Ph. D. Dissertation. University of Glasgow.
  * Quek et al. (2011) M. Quek, D. Boland, J. Williamson, R. Murray-Smith, M. Tavella, S. Perdikis, Martijn Schreuder, and Michael Tangermann. 2011.  Simulating the feel of brain-computer interfaces for design, development and social interaction.  [Proceedings of the 2011 Annual conference on Human factors in computing systems – CHI ’11] (2011), 25\.
  * Rashedi and Nussbaum (2015) Ehsan Rashedi and Maury A Nussbaum. 2015.  Mathematical models of localized muscle fatigue: sensitivity analysis and assessment of two occupationally-relevant models.  [PloS one] 10, 12 (2015), e0143872.
  * Rzig et al. (2023) Dhia Elhaq Rzig, Nafees Iqbal, Isabella Attisano, Xue Qin, and Foyzul Hassan. 2023.  Virtual Reality (VR) Automated Testing in the Wild: A Case Study on Unity-Based VR Applications. In [Proceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis] (Seattle, WA, USA) [(ISSTA 2023)]. Association for Computing Machinery, New York, NY, USA, 1269–1281.  https://doi.org/10.1145/3597926.3598134
  * Saul et al. (2014) Katherine R. Saul, Xiao Hu, Craig M. Goehler, Meghan E. Vidt, Melissa Daly, Anca Velisar, and Wendy M. Murray. 2014.  Benchmarking of dynamic simulation predictions in two software platforms using an upper limb musculoskeletal model.  [Computer methods in biomechanics and biomedical engineering] 5842, May 2016 (2014), 1–14.  https://doi.org/10.1080/10255842.2014.916698
  * Schumacher et al. (2023) Pierre Schumacher, Daniel Haeufle, Dieter Büchler, Syn Schmitt, and Georg Martius. 2023\.  DEP-RL: Embodied Exploration for Reinforcement Learning in Overactuated and Musculoskeletal Systems. In [The Eleventh International Conference on Learning Representations].  https://openreview.net/forum?id=C-xa_D3oTj6
  * Seinfeld et al. (2020) Sofia Seinfeld, Tiare Feuchtner, Antonella Maselli, and Jörg Müller. 2020.  User Representations in Human-Computer Interaction.  [Human–Computer Interaction] 0, 0 (2020), 1–39.  https://doi.org/10.1080/07370024.2020.1724790 arXiv:https://doi.org/10.1080/07370024.2020.1724790
  * Seth et al. (2011) Ajay Seth, Michael Sherman, Jeffrey A. Reinbolt, and Scott L. Delp. 2011.  OpenSim: a musculoskeletal modeling and simulation framework for in silico investigations and exchange.  [Procedia IUTAM] 2 (2011), 212–232.  https://doi.org/10.1016/j.piutam.2011.04.021 IUTAM Symposium on Human Body Dynamics.
  * Strasburger et al. (2011) Hans Strasburger, Ingo Rentschler, and Martin Jüttner. 2011\.  Peripheral vision and pattern recognition: A review.  [Journal of Vision] 11, 5 (12 2011), 13–13.  https://doi.org/10.1167/11.5.13 arXiv:https://arvojournals.org/arvo/content_public/journal/jov/933487/jov-11-5-13.pdf
  * Todorov et al. (2012) Emanuel Todorov, Tom Erez, and Yuval Tassa. 2012.  MuJoCo: A physics engine for model-based control. In [2012 IEEE/RSJ International Conference on Intelligent Robots and Systems]. 5026–5033.  https://doi.org/10.1109/IROS.2012.6386109
  * Todorov and Jordan (2002) Emanuel Todorov and Michael I. Jordan. 2002.  Optimal feedback control as a theory of motor coordination.  [Nature Neuroscience] 5, 11 (2002), 1226–1235.  https://doi.org/10.1038/nn963
  * Umberger et al. (2003) Brian R. Umberger, Karin G.M. Gerritsen, and Philip E. Martin. 2003.  A Model of Human Muscle Energy Expenditure.  [Computer Methods in Biomechanics and Biomedical Engineering] 6, 2 (2003), 99–111.  https://doi.org/10.1080/1025584031000091678 arXiv:https://doi.org/10.1080/1025584031000091678  PMID: 12745424.
  * Wang et al. (2022) Huawei Wang, Vittorio Caggiano, Guillaume Durandau, Massimo Sartori, and Vikash Kumar. 2022\.  MyoSim: Fast and physiologically realistic MuJoCo models for musculoskeletal and exoskeletal studies. In [2022 International Conference on Robotics and Automation (ICRA)]. IEEE, 8104–8111.
  * Wang (2022) Xiaoyin Wang. 2022\.  VRTest: an extensible framework for automatic testing of virtual reality scenes. In [Proceedings of the ACM/IEEE 44th International Conference on Software Engineering: Companion Proceedings]. 232–236.
  * Winter (1984) DA Winter. 1984\.  Biomechanics of human movement with applications to the study of human locomotion.  [Critical reviews in biomedical engineering] 9, 4 (1984), 287—314.  http://europepmc.org/abstract/MED/6368126
  * Xia and Law (2008) Ting Xia and Laura A Frey Law. 2008.  A theoretical approach for modeling peripheral muscle fatigue and recovery.  [Journal of biomechanics] 41, 14 (2008), 3046–3052.
  * Zhu et al. (2020) Zhuangdi Zhu, Kaixiang Lin, Anil K Jain, and Jiayu Zhou. 2020\.  Transfer learning in deep reinforcement learning: A survey.  [arXiv preprint arXiv:2009.07888] (2020).