# WheelPose: Data Synthesis Techniques to Improve Pose Estimation Performance
on Wheelchair Users

William Huang  0000-0001-7651-2190 University of California, Los
AngelesCaliforniaUSA<EMAIL_ADDRESS>,  Sam Ghahremani
0009-0007-5844-4834 University of California, Los AngelesCaliforniaUSA
<EMAIL_ADDRESS>,  Siyou Pei  0000-0003-3802-8298 University of
California, Los AngelesCaliforniaUSA<EMAIL_ADDRESS>and  Yang Zhang
0000-0003-2472-6968 University of California, Los AngelesCaliforniaUSA
<EMAIL_ADDRESS>
(2024)

###### Abstract.

Existing pose estimation models perform poorly on wheelchair users due to a
lack of representation in training data. We present a data synthesis pipeline
to address this disparity in data collection and subsequently improve pose
estimation performance for wheelchair users. Our configurable pipeline
generates synthetic data of wheelchair users using motion capture data and
motion generation outputs simulated in the Unity game engine. We validated our
pipeline by conducting a human evaluation, investigating perceived realism,
diversity, and an AI performance evaluation on a set of synthetic datasets
from our pipeline that synthesized different backgrounds, models, and
postures. We found our generated datasets were perceived as realistic by human
evaluators, had more diversity than existing image datasets, and had improved
person detection and pose estimation performance when fine-tuned on existing
pose estimation models. Through this work, we hope to create a foothold for
future efforts in tackling the inclusiveness of AI in a data-centric and
human-centric manner with the data synthesis techniques demonstrated in this
work. Finally, for future works to extend upon, we open source all code in
this research and provide a fully configurable Unity Environment used to
generate our datasets. In the case of any models we are unable to share due to
redistribution and licensing policies, we provide detailed instructions on how
to source and replace said models. All materials can be found at
https://github.com/hilab-open-source/wheelpose.

Accessibility, Data Synthesis, Wheelchair Users, Pose Estimation

††journalyear: 2024††copyright: rightsretained††conference: Proceedings of the
CHI Conference on Human Factors in Computing Systems; May 11–16, 2024;
Honolulu, HI, USA††booktitle: Proceedings of the CHI Conference on Human
Factors in Computing Systems (CHI ’24), May 11–16, 2024, Honolulu, HI,
USA††doi: 10.1145/3613904.3642555††isbn: 979-8-4007-0330-0/24/05††ccs: Human-
centered computing Accessibility systems and tools

Figure 1. Overview of the full WheelPose data generation pipeline. Developers
can choose different motion sources. Motion sequences are modified according
to the specification stated below before being evaluated by human evaluators.
Developers can regenerate, filter, and clean motion sequences from human
evaluations before all motions are converted into Unity readable
AnimationClips. Converted motion sequences, selected background images, and
parameters are used in simulation to generate synthetic images and their
related annotations for use in model boosting.

Eight components: Initial Motion Generation, Pose Modification, Human
Evaluations, Animation Conversion, Background Generation, Unity Simulation
Environment, Pipeline Output, and Model Boosting. Initial motion generation
consists of a choice between generative models, motion datasets, or self
captured data. This data is in the form of 3D joint motion sequences and fed
into Pose Modification which consists of a low pass filter, lower body
modification, and joint position to rotation conversion. This feeds into Human
Evaluations which returns user feedback back to Initial Motion Generation and
passes data into Animation Conversion which consists of FBX conversion and
AnimationClip conversion. AnimationClips are fed into the Unity Simulation
Environment along with the user choice of real world background images or
simulated background images in Background Generation. Unity Simulation
environment outputs Pipeline Outputs consisting of synthetic images and
annotations which inputted into model boosting.

##  1\. Introduction

The inclusiveness of AI depends on the quality and diversity of data used to
train AI models. We focus on pose estimation models, which have found
widespread use in health care, environmental safety, entertainment, context-
aware smart environments, and more. These models are a major concern in the
push for AI fairness due to the disparity in their accuracy of predicted
postures between able-bodied users and users with disabilities (Guo et al.,
2020; Trewin, 2018; Whittaker et al., 2019). Focusing on human movement,
Olugbade et al. (Olugbade et al., 2022) surveyed 704 open datasets and found a
major lack of diversity. The authors found no datasets that included people
with disabilities performing sports, engaging in artistic expressions, or
simply performing everyday tasks. We suspect that the lack of diversity has
contributed to biases and poor performance on users with disabilities in many
popular AI models trained on common human movement datasets like Detectron2
ImageNet (Wu et al., 2019). This is especially apparent in users who use
mobility-assistive technologies (Figure 2). The lack of disability
representation in training data can be directly attributed to poor
accessibility in the data collection process for people with disabilities
(Park et al., 2021; Guo et al., 2020). People with disabilities often must
overcome more challenges in data collection during their commute and
communications in the recruitment and participation process. One such example
and the focus of our work are wheelchair users, who may not be able to
navigate through motion capture rigs easily. Additionally, certain poses that
able-bodied users could easily perform might be difficult or even dangerous
for people with motor impairments. To improve disability representation in
training data in the push for inclusive AI, we must make data collection
equitable across people with all levels of capabilities.

A side-view of a wheelchair user pushing. Both her arms are on the wheels.

A side-view of a wheelchair user opening a door. The wheelchair is directly in
front of the door and both her hands are placed on the handle.

A side-view of a wheelchair user putting on a graduation hat. Both their hands
are holding a hang which is over her head.

A 45-degree side-front ground-up view of a wheelchair user doing a wheelie.
Both hands are on the wheels of the wheelchair.

A wheelchair dancer facing the camera with their wheelchair parallel to the
plane of the camera. They are in front of four crowd members in a white dress.

Figure 2. Examples of poor keypoint prediction performance from Detectron2
ImageNet (Wu et al., 2019). Figure 2 Poor prediction of the legs and torso
which are slightly occluded by the wheelchair. Figure 2 Both shins are
predicted to be on the hand due to the occlusion of the wheelchair. Figure 2
Legs are predicted to be on the upper body. Figure 2 The right shin is
predicted to be on the wheels of the wheelchair. Figure 2 The wheelchair
dancer is completely undetected by ImageNet.

Prior work has combated this issue by proposing guidelines in the design of
studies (Bennett and Rosner, 2019) and online data collection systems that
could be more accessible to people with disabilities (Park et al., 2021). We
propose an alternative solution to the data collection problem: synthetic
data. Like how synthesized materials could help preserve scarce natural
resources – synthetic rubbers were developed in part because of concerns about
the availability of natural rubber – synthetic data could be valuable in
supplementing insufficient data collection from people with disabilities.
While traditional motion capture procedures lie on a spectrum of difficulty
dictated by the motion of the user and exacerbated by the innate difficulties
of data collection with users with assistive technologies, synthetic data
offers a more accessible alternative where different actions, settings, and
assistive technologies, ranging from cooking at home to performing back-flips
in the forest, are equal in the difficulty of implementation and feasibility.

In this research, we present a novel data synthesis pipeline which leverages
motion generation models to simulate highly customizable image data of
wheelchair users. Our approach includes steps for user-defined parameters,
data screening, and developer feedback. This pipeline yields image data which
can be used to improve the performance of AI models for wheelchair users. We
evaluate this in the case of pose estimation by fine-tuning common pose
estimation models, trained on common human movement datasets, with our
synthetic data. Fine-tuned models are then tested on a new dataset of
wheelchair users to analyze the degree of improvement from adding synthetic
data to training datasets.

Finally, as we are cautious about the problematic simulation of disabilities
(e.g., blindfolded participants to simulate people without vision or with low
vision), synthesized wheelchair user postures are carefully reviewed in human
evaluations to avoid inadvertently exacerbating existing equitability problems
our approach attempts (Bennett and Rosner, 2019). Our goal is not to exclude
wheelchair users from AI training, but rather present a data collection
solution that enables them to shepherd the synthesis of data. In doing so,
this research leverages data as an intuitive way for wheelchair users to
impact AI training, through which we hope to produce more fair and inclusive
AI models.

Through our work, we aim to answer the following research questions and
related sub-questions in the context of wheelchair users and pose estimation
problems:

  1. (1)

RQ1: How to effectively generate synthetic data?

     * •

RQ1.1: How can we model wheelchair users?

     * •

RQ1.2: What are the controlling parameters in synthetic data generation?

  2. (2)

RQ2: What are the efficacies of synthetic data?

     * •

RQ2.1: How do individual parameters of synthetic data generation affect pose
estimation performance?

     * •

RQ2.2: What are the benefits and drawbacks of using synthetic data?

To summarize, our contribution is three-fold:

  * •

adoption of data synthesis to improve inclusion of AI.

  * •

a custom data-synthesis pipeline for pose tracking with improved performance
for wheelchair users.

  * •

investigations of the efficacy of the overall approach.

##  2\. Related Work

###  2.1. Pose Estimation

Pose estimation plays a key role in fields like the animation and video
industry (SUTIL, 2015; Kin, 2019). Developments in deep learning have enabled
users to circumvent the need for cumbersome marker suits in traditional motion
capture and directly generate human postures from camera outputs (Xie et al.,
2022; Luvizon et al., 2023; Zheng et al., 2023; Dee, 2023). Of particular note
is 2D posture recognition, where the recent releases of easily accessible
pipelines like MediaPipe (Lugaresi et al., 2019), OpenPose (Cao et al., 2019),
and BlazePose (Bazarevsky et al., 2020) have enabled widespread access to
posture recognition in a wide variety of applications including biomechanics
(Mundt et al., 2022), autonomous driving (Way, 2022), sign language
interpretation (Moryossef et al., 2020), and more.

###  2.2. Sensing for People with Limited Mobility

Currently, over 8.5% of the population of the world is age 65 or over. This
number is projected to grow to nearly 17% of the world’s population by 2050
(He et al., 2016a). Given the direct correlation between age and and mobility
limitations and disabilities, this trend implies a growing need for mobility-
related technologies (Ferrucci et al., 2016; Freiberger et al., 2020; Gardener
et al., 2006). Our research focuses on the community of wheelchair users,
where technologies like SpokeSense (Carrington et al., 2020) have established
themselves as a part of a broader focus in research related to developing
smart wheelchairs (Leaman and La, 2017) which integrate different sensors,
including camera, lidar, and EEG, to make wheelchairs more comfortable and
safe. Other works focus on developing more accessible control systems
(Upadhyaya et al., 2022) or routing systems (Kirkham and Tannert, 2021) for
users with mobility impairments. Posture estimation techniques for wheelchair
users can reveal a user’s sitting habits, analyze their mood, and predict
health risks such as pressure ulcers or lower back pain (Ma et al., 2017).

###  2.3. Synthetic Data for Computer Vision

Computer vision models have traditionally been trained using large-scale
human-labeled datasets such as PASCAL VOC (Everingham et al., 2010), Microsoft
COCO (Lin et al., 2015), and ImageNet (Deng et al., 2009). While effective,
these datasets are costly to produce, requiring large amounts of publicly
available images, manpower, and time to create. Furthermore, these datasets
are often static and offer little in the form of customizability to allow
researchers and engineers to use data specific to their task. One solution to
these problems is the use of data simulators. SYNTHIA (Ros et al., 2016),
Synscapes (Wrenninge and Unger, 2018), Hypersim (Roberts et al., 2021), and
OpenRooms (Li et al., 2021b) provide synthetic datasets for computer vision
tasks related to object detection in different settings. Other robotics
simulators including AI-2THOR (Kolve et al., 2022), NVIDIA Isaac Sim (Liang et
al., 2018), Mujoco (Todorov et al., 2012), and iGibson (Shen et al., 2021)
offer a rich set of tools for embodied AI tasks. Other systems, like
BlenderProc (Denninger et al., 2019), BlendTorch (Heindl et al., 2020), NVISII
(Morrical et al., 2021), and Unity Perception (Bartolomé Villar et al., 2022)
prefer to instead enable the developer to generate their own data through
highly configurable simulators. These tools and datasets have already
demonstrated considerable success in deep learning-related training tasks (He
et al., 2023; Anderson et al., 2022; Tian et al., 2023).

Synthetic humans provide further challenges due to the complexities of human
bodies and the variations and limitations of a human’s appearance and posture.
Various approaches have been taken, using different types of simulators to
generate labeled datasets. Examples of different approaches include deriving
data from hand-crafted scenes (Bak et al., 2018), custom 3D scenes with SMPL
models (Yang et al., 2023; Pumarola et al., 2019; Varol et al., 2017a; Patel
et al., 2021; Bazavan et al., 2022; Purkrábek and Matas, 2023), existing games
like Grand Theft Auto V (Cao et al., 2020; Fabbri et al., 2018; Hu et al.,
2019, 2021; Cai et al., 2022), and game engines (Ebadi et al., 2022b, a). We
were inspired by this line of work and extended upon the existing
PeopleSansPeople (PSP) data generator with the Unity Perception package
(Bartolomé Villar et al., 2022) using domain randomization principles which
help AI models trained in simulated environments to effectively transfer to
real-world tasks (Tobin et al., 2017).

###  2.4. Evaluating the Quality of Synthetic Data

Despite the advantages of data synthesis, an implicit assumption of using
synthetic data is that it should be sufficiently high-quality to achieve
performance similar to real data. To evaluate the quality of synthetic data,
researchers have explored a wide range of metrics. Emam et al. (El Emam et
al., 2020) outlined three types of approaches to assess synthetic data utility
in their book - workload-aware evaluations, generic assessments, and
subjective assessments of data utility. Among them, workload-aware evaluations
check if synthetic data replicates the performance of real data, widely used
in data synthesis research (Varol et al., 2017b; Pishchulin et al., 2012;
Gaidon et al., 2016). Generic assessments measure the utility indicators of
real and synthetic data when the indicators are quantifiable and clear
(Kaloskampis et al., 2019; Howe et al., 2017) (e.g., the distance between
their statistical indicators such as mean, average, and distribution).
Furthermore, subjective assessments involve real users to evaluate data
realism. Some researchers investigate distinguishability, assuming highly-
realistic synthetic data leans to be perceived as real (Snoke et al., 2018),
similar to deploying a discriminator in algorithms (Tzikas, 2022). Other
researchers design Likert-Scale questionnaires for realism, which are broadly
adopted in clinical training simulation (Bartolomé Villar et al., 2022).
Another criterion is to collect user preferences between several synthetic
samples to form a high-quality dataset (Weng et al., 2023; Tevet et al.,
2022). With a consistent and valid examination of synthetic data, researchers
can obtain feedback to improve generation methods and understand how reliable
synthetic data is. In our work, we not only conduct workload-aware evaluation
but also involve subjective assessments by asking real users to evaluate the
data realism. The user-in-the-loop process provides filter handles for more
realistic datasets under various contexts and inspires key findings on how
data realism affects performance.

##  3\. WheelPose Dataset Synthesis

We address (RQ1.1) with our system, WheelPose, a data synthesis framework
where motion data is converted to wheelchair user animations and rigged on
human models in a Unity simulation environment to generate synthetic images
and annotations. We present a simple simulation environment, with human models
randomly placed in front of a background as the most primitive example of
synthetic data still capable of generating positive results in pose estimation
(Section 4.3). A visualization of the overall pipeline is found in Figure 1.
Our pipeline generates a set of datasets, each including 70,000 captured
frames ($1280\times 720$). Each frame is fully annotated using the COCO
17-keypoint 2D skeletal model, shown in Figure 5(b). Beyond the fact that our
dataset is the first fully annotated dataset of wheelchair users, the size of
our dataset is comparable with existing datasets like 3DPW with 51,000
captured frames (Marcard et al., 2018) or SMPLy with 24,428 captured frames
(Leroy et al., 2020). Example synthesized data is shown in Figure 3.

Figure 3. Example output from the full WheelPose data generation pipeline. The
top row includes the generated RGB images. The bottom row includes the
generated RGB images with the keypoint and bounding box annotations
superimposed on top. Keypoints outlined in black are labeled as ”occluded”
while keypoints outlined in white are labeled as ”visible”. Each image is
generated with randomized backgrounds, lighting conditions, humans, postures,
and occluders.

Six examples of WheelPose output. Each example features a set of different
wheelchair users in front of a unique interior setting. Geometric objects with
differing textures are placed around the scene. Below each image is the same
image with the annotation of keypoints and bounding boxes superimposed on top.

###  3.1. Generating Postures

Our data synthesis pipeline begins with pose generation, where motion data is
converted into animations to be used in data synthesis. We choose to use two
motion data sources, HumanML3D (Guo et al., 2022a) and Text2Motion (Guo et
al., 2022b) in our case study. Other motion sources can be easily adapted and
used within our pipeline. Figure 4 demonstrates 14 motion sequences and their
resulting postures from our pose generation, documented next. We separate
posture generation from the rest of our pipeline to allow developers to
iterate upon generated postures through human evaluation, regenerating and
filtering data as needed.

Figure 4. Example pose frames in their motion sequences resulting from our
pose frame generation. Each column is an individual animation with pose frames
selected in chronological order from back to front.

Models in wheelchairs performing different motions. Fourteen columns of five
humans each are in a scene. Each column features different upper body
movements with a large variety of ranges of motion across.

####  3.1.1. Human Skeletal Modeling

In order to enable a wide variety of different postures, we base the
implementation of WheelPose on a 23-keypoint skeletal model, which is easily
converted from commonly available human posture datasets including COCO (Lin
et al., 2015) and MPII (Andriluka et al., 2014) and the output of common pose
estimation algorithms including BlazePose (Bazarevsky et al., 2020), MediaPipe
(Lugaresi et al., 2019), and OpenPose (Cao et al., 2019). Detailed information
on these keypoints is shown in Figure 5(a). We note that our current pipeline
assumes users have all four limbs and acknowledge that data synthesis for
wheelchair users with amputation requires efforts beyond simple ad-hoc
removals of key points in our current model. Nonetheless, we interviewed two
participants with limb loss, leading to insights for future work which we will
discuss later in the paper (Section 4.1).

Visualization of the WheelPose keypoint designation. A human model sits on a
wheelchair in a T-pose facing forward. Twenty-three keypoints extend across
their body, going through the wrists, elbows, and shoulders to create the
arms, the top of the head, the top of the neck, the neck base, the upper
chest, the upper spine, the lower spine, and the middle hip to create a spine,
and hips, knees, ankles, and toes to create legs.

(a) WheelPose Animation Keypoints

Visualization of the COCO keypoint designation. A human model sits on a
wheelchair in a T-pose facing forward. Seventeen keypoints extend across their
body, going through the wrists, elbows, and shoulders to create arms, ears,
eyes, and nose to create a face, and hips, knees, and ankles to create legs.

(b) COCO Keypoints

Figure 5. Keypoint mappings used in WheelPose. Figure 5(a) WheelPose
23-keypoint animation format. Used as the input format of motion sequences
before pose modification. Figure 5(b) COCO 17-keypoint annotation scheme. Used
as the final output format of WheelPose annotations.

####  3.1.2. Motion Sequence Generation

We use HumanML3D (Guo et al., 2022a), a 3D human motion dataset collected from
real-world motion capture in the form of 3D joint positions as an example of
motion sequence generation from existing motion capture datasets. Motions with
high translational movement, high lower body movement, and broken animations
(e.g., jittering, limb snapping, unrealistic rotations) were filtered out from
HumanML3D. Individual motions were then randomly sampled and evaluated for
their uniqueness and range of motion compared to previously collected data
until 100 unique motions were collected.

Additionally, we leverage Text2Motion (Guo et al., 2022b), a motion generation
model that uses textual descriptions to generate motions, as a fully
generative alternative source of human motion sequences. Text2Motion is only
one example of human motion generation through textual descriptions (Zhang et
al., 2022; Azadi et al., 2023; Zhang et al., 2023) and can be easily
substituted in our data generation pipeline. Evaluators from the research team
assigned textual descriptions to each of the 100 selected HumanML3D motions
before inputting the descriptions into Text2Motion and generating 3 potential
motions for each description. The most realistic motion was selected and
evaluated on whether the motion would be possible for the evaluator to perform
and the absence of any noise or artifacts from the generation process that may
lead to unrealistic limb movements or positions. Our Text2Motion generation
process results in a new dataset of 100 human motions that directly mirror the
actions of the sampled HumanML3D motions and provide a direct comparison
between synthesized and real data. Our goal in enabling the use of generative
motion models like Text2Motion is to investigate the feasibility of using
generative deep learning models to further simplify the data collection
process for synthetic data generation and therefore improve the efficiency and
overall accessibility, especially in the context of users with disabilities
and assistive technologies.

In total, this process yielded 200 motion sequences from both HumanML3D and
Text2Motion (i.e., each yielded 100 sequences). On average, HumanML3D motions
had 60.26 (SD=42.05) unique frames per animation and Text2Motion has 146.32
(SD=50.84) frames per animation. Since Text2Motion has no set animation length
parameter, we choose to take the full animation output for each motion,
leading to the discrepancy in average motion lengths, to directly compare data
sourced from motion capture and deep learning models. Text2Motion outputs tend
to extend and slow down the described action, leading to a longer but not
necessarily more diverse animation compared to HumanML3D.

####  3.1.3. Pose Modification and Conversion

Both HumanML3D and Text2Motion represent motions through the 3D position of
joints. All motions are converted into the corresponding 23-keypoint skeletal
model used by WheelPose through a direct mapping of corresponding joints or
the positional average between the surrounding joints. As is common in
biomechanical analysis (Schreven et al., 2015; Bartlett, 2014), a 5Hz low pass
filter is then applied to the data to handle high-frequency noise generated
from the motion capture or data generation process. We convert all motions
from a 3D position to a joint rotation representation. In the process of this
conversion, internal and external rotations – rotations around the axis
parallel to the bone not able to be described in joint position notation – of
the arms are affixed to the rotation of the parent bone.

We modified the resulting pose sequences from the two generation methods by
affixing the model’s legs onto the wheelchair model with an additional
rotational noise applied independently on the flexion/extension,
abduction/adduction, and internal/external rotation on each of the lower body
bones (i.e. three joints in total) to simulate regular lower body movements
when in a wheelchair. The following steps document the procedure for
generating rotational noise on one joint. This procedure was motivated by the
need for smooth interpolated noise and inspired by the Poisson process.

Given an array of joint angles $F_{orig}$ expressed in degrees with length $n$
total frames in the animation, the noise is generated by first sampling a set
of frames indices $S$.

(1) $\textstyle S=\left\\{x_{0},x_{1},...,x_{n}\
\middle|\begin{array}[]{l}x_{0}=0,\\\ x_{i}=x_{i-1}+k_{i},\\\
k_{i}\sim\mathcal{N}(\frac{n}{4},(\frac{n}{32})^{2}),\\\ x_{i}<n,\\\
i=1,2,..,n\end{array}\right\\}$

A new array of joint angle noise values $N$ is then constructed. Let $f(i)$ be
a function that generates the $i$-th joint angle of the animation. Given
$U\sim\mathcal{U}(-10,10)$, representing a random angular noise added to
frames in $S$,

(2) $f(i)=\begin{cases}U&i\in S\\\ \text{NaN}&else\end{cases}$

(3) $N=[f(i)\mid 0\leq i<n]$

Linear interpolation is then applied to fill in all NaN values in $N$. The new
array of joint angles $F_{new}$ is thus the element-wise sum of $F_{orig}$ and
$N$. Our algorithm is motivated by the need for a simple and efficient noise
algorithm that does not jitter as the user iterates through frames of the
animation.

(4) $F_{new}=[f_{orig}[i]+N[i]\mid 0\leq i<n]$

All motion sequences are applied to a Blender111A free and open source 3D
modeling software. More information is found at https://www.blender.org/ model
before being imported into Unity and converted to Unity Perception-readable
AnimationClips files. Example outputs from our posture generation step
described above can be found in Figure 4.

###  3.2. Generating Wheelchair User Models

In order to capture synthetic images, we must rig the postures resulting from
previous sections onto Unity human models.

For human models, WheelPose enables both the default human models provided by
PeopleSansPeople (Ebadi et al., 2022b) and randomized humans leveraging the
Unity SyntheticHumans (Syn, 2023), a Unity Perception package using domain
randomization to generate unique human models from a sampling of different
clothes, body types, sexes, and more. This utilization allows WheelPose to
generate unique human models that better capture the wide variety of
appearances of real people. We use 8,750 unique human instances using the
default SyntheticHumans configuration limited to people over the age of 10 to
better reflect the general population of people in wheelchairs (Vignier et
al., 2008). Figure 6 shows 19 examples of these human models.

We also enable the human models to spawn with different objects (e.g.,
wheelchairs, crutches, canes, walkers, etc.) in user-defined positions when
placed into the environment. For the scope of this project, we focused on
wheelchair users and used a realistic wheelchair model, sourced from the Unity
Asset Store, scaled by the size of the human model. Finally, the posture of
each human model is randomly sampled from the AnimationClips generated in
Section 3.1.

Figure 6. Example human models used in the synthesis of wheelchair user
images.

Nineteen unique human models in wheelchairs. Each model has a different
combination of clothing, age, ethnicity, sex, hairdo, and body size.

###  3.3. Generating the Simulation Environment

We address (RQ1.2) by developing a highly configurable simulation environment.
We use the PeopleSansPeople (PSP) (Ebadi et al., 2022b) base template and its
related extension, PSP-HDRI (Ebadi et al., 2022a), as our baseline data
generator built in the Unity222More information found at https://unity.com/
game engine through the High Definition Render Pipeline (HDRP). PSP is a
parametric human image generator that contains a fully developed simulation
environment including rigged human models, parameterized lighting and camera
systems, occluders, synthetic RGB image outputs, and ground truth annotations.
PSP is built on the idea of _domain randomization_ (Tobin et al., 2017) where
different aspects of a simulation environment are independently randomized to
diversify the generated synthetic data, exposing models to a wider array of
different environments during training and improving testing accuracy
(Tremblay et al., 2018; Valtchev and Wu, 2021). All domain randomization is
implemented through the ”randomizer” paradigm designed in the Unity Perception
package (Borkman et al., 2021), a Unity toolkit to generate synthetic data for
computer vision training. Within each scene, individual randomizers are
assigned to a specific parameter (i.e. lighting, occluder positions, human
poses, etc.) and independently sample parameter values from a uniform
distribution. PSP was then updated to Unity 2021.3 and Unity Perception
1.0.0333First official release of Unity Perception which enabled more
annotations, more extendable randomizers, and flexibility with other Unity
packages. We then added a new background image parameter and its related
randomizer for the sampling of user-defined images to be used as a backdrop in
each scene. We use this parameter to enable three different sets of background
images: PSP default textures, 100 background images randomly sampled from the
BG-20k background image dataset (Li et al., 2022, 2021a, 2021c), and 100
generative images from Unity SynthHomes (Unity Technologies, 2022), a dataset
generator for photorealistic home interiors. Table 1 outlines the statistical
distributions of the environment parameters used.

####  3.3.1. Camera Configuration and Keypoint Annotations

A main camera in the Unity scene is used as the primary capture source of all
images and annotations. The Unity Perception _Perception Camera_ is used to
simulate realistic camera features including focal length and field of view
(FOV) and to capture all annotations. The position, rotation, focal length,
and FOV of the main camera are set through a series of randomizers with
default parameters found in Table 1. The main camera captures RGB, depth,
surface normal, and instance segmentation images in $1280\times 720$ for each
frame of captured data. Default Unity Perception annotation labelers are
placed on the main camera to capture 2D/3D bounding boxes for each human
model, object counts, rendered object metadata, semantic segmentation, 2D/3D
keypoint locations in COCO format, and percent of human model occluded. Out-
of-view and fully occluded human instances are automatically ignored in
annotation capture, recording only data on human instances within direct view
of the main camera.

###  3.4. Assembling WheelPose Datasets

Overall, our pipeline yields 70,000 images for each generated dataset. All
data was generated in a Unity 2021.3 project configured with parameters preset
to the values listed in Table 1. We ran our data synthesis pipeline on a PC
with a 4.2GHz 6-Core/12-Thread AMD Ryzen R5 3600, NVIDIA GTX 1070 8GB VRAM,
and 32 GB 3600MHz DDR4 memory for an average generation time of $\approx$ 1
hour and 45 minutes for 10,000 images – which translates to 12 hours and 15
minutes for each dataset. This time includes all steps of the generation
process including motion generation, parameter randomization, data capture,
label creation, and writing to disk. Examples of generated synthetic images
are shown in Figure 7. We open-source our data synthesis pipeline including
pose modification and the full configurable Unity 2021.3 project for data
generation in https://github.com/hilab-open-source/wheelpose.

Figure 7. Examples of a scene being generated. Notice the random placement of
wheelchair users, different occluder objects, different lighting conditions,
and various SynthHome backgrounds. The red, green, and blue arrows represent
the camera coordinates which we used to insert randomization in forms of
Cartesian translations and Euler rotations.

Three examples of scene generation in WheelPose. Each example features a
third-person example of wheelchair users in front of a backdrop with occluders
randomly suspended in the air. Arrows indicating the camera are all placed in
different locations and rotations for each image.

##  4\. Evaluation of WheelPose

To answer (RQ2) on the benefits and drawbacks of synthetic data, we evaluate
the WheelPose pipeline and generated data through three specific methods: 1)
human evaluation on realism, 2) statistical analysis of innate dataset
characteristics, and 3) evaluation of our generated datasets’ effects on AI
model performance. We document the results of these methods in the following
sections.

###  4.1. Human Evaluation

We involved real wheelchair users in the loop to evaluate the realism of our
synthetic data. In our study, “realism” manifests as ease and frequency. We
sent out online surveys and strictly verified users’ eligibility and
authenticity manually to prevent scammers. We recruited 13 daily wheelchair
users (5 F, 8 M), with ages ranging from 26 to 56 (M=32, SD=9.6), as shown in
Table 1. A key limitation of this research is the limited diversity among the
wheelchair user participants. All participants were individuals with spinal
cord injuries (SCI), a specific condition that has distinct movement patterns.
This represents only a small subsection of the broader wheelchair user
population. Despite our efforts to diversify the participant pool by including
participants with different levels of SCI, we acknowledge that our participant
population is not sufficiently representative to draw statistical insights for
wheelchair users with different conditions or bodies than those in this study
(e.g., muscular dystrophy, amputations, dwarfism, spinal deformities).

Table 1. Demographics of participants (P1-P13) in human evaluation.

ID | Age | Gender | Occupation | SCI Level |  Exercise Routines |  Full Mobility of Arms, Shoulders, and Hands
---|---|---|---|---|---|---
P1 | 56 | M | Professor | T-12/L-1 |  No |  Yes
P2 | 29 | F | Home-maker | T-3 |  Weight lifting, Strolling, and Stretches |  Yes
P3 | 40 | M | Entrepreneur | T-12 |  Swimming, Cycling, and Gym |  Yes
P4 | 40 | F | Self-employed | C-5 |  No |  No (DASH Score 79.2/100 (Hudak et al., 1996))
P5 | 36 | M | Customer Relations | C-5 |  Stretching and Strength training |  No
P6 | 26 | M | Software Developer | Lumbar spinal stenosis |  Wheelchair walking |  Yes
P7 | 51 | M | Proctor/Graphic Designer | T-12; L1 |  Gym workouts periodically, Wheelchair Basketball, Wheelchair Tennis, and Pushing long distances |  Yes
P8 | 26 | M | Librarian | C7 |  No |  No (DASH Score 40.0/100 (Hudak et al., 1996))
P9 | 27 | F | Remote Computer Programmer | L5 |  Arm training using a band |  Yes
P10 | 27 | F | Teacher | Lumbar SCI |  Aquatic therapy |  Yes
P11 | 32 | F | Receptionist | Thoracic SCI |  Aerobic exercise |  Yes
P12 | 34 | M | Marketing Manager | Sacral SCI |  No |  Yes
P13 | 29 | M | Freelancer | Lumbar SCI |  Water exercise |  Yes

This table shows demographic information about participants in the human
evaluation. The header includes ID, Age, Gender, Occupation, SCI Level,
Exercise Routines, and Full Upper Limb Mobility.

####  4.1.1. Procedure

Participation was conducted entirely online, allowing users to contribute at
their convenience. Users first answered a questionnaire consisting of
demographic information and mobility capability before then evaluating two
groups of synthetic motions - HumanML3D motions and Text2Motion motions, using
a browser-based user interface (Figure 8). The motions were presented as
animation GIFs of a human skeleton performing a certain movement. We chose to
use skeletons rather than a more photorealistic model as an embodiment
technique facilitating users to think of these skeletons as tracked motion of
their bodies. Related work (Fribourg et al., 2020) has shown that a dummy
avatar provides stronger senses of embodiment comparable to non-personalized
realistic avatars. Our uses of the skeletal model aim to shift the focus
towards motion, steering attention away from superficial cosmetic details.
Users observed one animation clip at a time, simultaneously from four
perspectives (45-degree oblique top view, top view, side view, and front
view). To navigate through motion clips, users click the previous/next buttons
or press the arrow keys. Buttons were designed for participants to select
scores for two Likert-Scale questions.

For each motion, users answered three questions:

  * •

Q1: “How difficult is it for you to do this motion?” - by rating from 1
(Cannot perform the sequence at all) to 7 (Without any difficulty).

  * •

Q2: “How often do you do this motion?” - by rating from 1 (Never) to 7
(Everyday).

  * •

Q3: “Have you seen or do you know of other wheelchair users who perform this
motion?” (Yes/No).

We used all 100 motions in both HumanML3D and Text2Motion converted motion
sets. After finishing the last animation GIF, users proceeded to the other
motion group. The question set is consistent for both groups. The order of
groups was random. Three users evaluated HumanML3D first, while the others saw
Text2Motion first.

After all animations were scored, we followed up with participants via email
to better understand the following:

  1. (1)

What are the criteria used in your evaluation of a motion’s difficulty?

  2. (2)

What determines the ”frequency” of performing a motion?

  3. (3)

What motion do most wheelchair users often perform and what motion do you
frequently perform, that did not show up in our dataset?

Participants were paid $20 per hour as compensation for their time. As the
study did not enforce a time limit and was purely online, users could take a
brief break whenever they wanted as long as the questionnaire remained open.
Excluding breaks, the study took 1.78 hours on average. The study was
evaluated and approved by the Institutional Review Board (IRB) at UCLA.

Figure 8. Screenshot of the human evaluation interface. Note that the four
subplots on the left are supposed to show an animation of a human skeleton
performing a motion sequence in loops from different perspectives (45-degree
oblique top view, top view, side view, and front view). Participants were
asked to observe an animation and give Likert-Scale scores and a binary
response before moving on to evaluation of the next motion sequence.

Example of human evaluation interface. Four different views of a skeletal
model performing one of the selected motions are on the left. On the right, a
set of questions on motion difficulty, frequency, and whether they have seen
the motion before with multiple choice answers are displayed.

####  4.1.2. Data Analysis

User responses were Likert-Scale scores (1-7 for Q1 and Q2) and binary
responses (yes/no for Q3). We first visualized the distribution of data across
users (Figure 9). Afterward, we separately ranked the motions based on average
ease and frequency scores. We also calculated the correlation between
difficulty and frequency. From follow-up emails, we collected their comments
and performed a thematic analysis of their perception of metrics, and the
validity of our dataset. The initial codes were the summary of their rating
reasons, which were later merged and discussed. In the end, we ranked
Text2Motion motions by the total frequency and difficulty score and identified
the bottom 10% for later use in the model performance evaluation.

Figure 9. From left to right, Figure (a)(b)(c)(d) are 100% stacked bar charts
showing the motion distribution in HumanML3D or Text2Motion, with (a) and (b)
depicting the perceived ease and frequency of HumanML3D, while (c) and (d)
depicting Text2Motion. In (a) and (c), a score of 1 denotes Cannot perform the
sequence at all while a score of 7 denotes Can perform without difficulty. In
(b) and (d), 1 means Never, and 7 means Everyday. The two scatter plots,
Figure (e) and (f), demonstrate the strong correlation between ease and
frequency in both motion groups. Figures (g) and (h) depict stacked bar graphs
showing whether participants have seen or known of a wheelchair user who has
performed this motion.

Four stacked bar graphs with percent of animations from 0% to 100% on the Y
axis and the participants on the X axis. Each bar graph has responses 1 - 7
stacked to add up to 100%. Two more scatterplots with frequency on the Y axis
from 1 to 7 and ease on the X axis from 1 to 7. Both scatter plots have strong
linear correlations. Two final stacked bar graphs with Yes or No frequency on
the Y axis and the participant on the X axis. A majority of participants
answered yes they have seen most motion.

####  4.1.3. Results and Findings

User perception of the generated dataset (i.e., perceived realism) is a strong
indicator of the efficacy of our data generation pipeline in that efficient
data generation should only yield data that wheelchair users perceive as being
realistic. Investigating deeper causes for the perceived realism of our
generated data also inevitably led to insights about wheelchair users. We
summarize the findings of our user evaluation in the rest of this section.

Participants found our synthetic motion sequences realistic. HumanML3D motions
received an average ease score of 4.742 (SD=0.943) and an average frequency
score of 4.079 (SD=1.085) across the dataset. Text2Motion motions performed
comparably, receiving an average ease score of 4.977 (SD=0.915) and an average
frequency score of 4.507 (SD=0.889).  Regarding Q3, for each motion we
presented, our participants had seen or knew of other wheelchair users who
performed that motion. Specifically, as Figure 9(g)(h) shows, most
participants have seen most motions performed by other wheelchair users.

Regarding the representativeness of datasets, most participants said they had
seen all of the common motions (e.g., “rolling forward”, “driving”, “writing”,
“drinking”, “cooking”) they knew about in our datasets. We suspect two factors
that account for the outcomes observed in these participants: 1) lack of
contextual cues made it challenging for them to recall specific motions, and
2) our embodiment technique facilitated their use of imagination that bridged
the gaps between the motions we demonstrated and those they executed in daily
tasks. Nevertheless, some other participants commented on popular but missing
motions, including “wearing some lower body clothes” (P10), “clapping hands”
(P11), “typing on a keyboard” (P13). The rest of the users were unsure about
their recollection of the datasets and used general phrasings e.g., “seen
almost all”. Our datasets were not comprehensive enough to cover all common
motions. We believe user feedback is the key to improving data completeness
and representativeness.

Ease (Q1) and frequency (Q2) of performing a motion vary across participants.
From Figure 9(a) to Figure 9(d), we observe diversity of perception on ease
and frequency across individuals. The horizontal axis is participant ID, the
vertical axis is the percentage of motions in the whole group. The sequential
color palette depicts the ease of doing a motion from 1 to 7, with 7 being
very easy. For example, P4 and P5, who had a higher injury position, rated
more motions into the difficult pool. The takeaway is that the ease and
frequency of performing a motion are highly personal. A realistic motion
dataset should include motions across the entire spectrum while eliminating
motions that no wheelchair users will ever perceive as being easy/frequent to
perform. However, the result of this user evaluation would change as the size
of the participant group grows, which lowers the likelihood of unrealistic
motions. Nonetheless, we argue that a realistic dataset should include motions
that most wheelchair users would perceive as easy and frequent, to ensure the
data quality and avoid introducing new biases, while addressing existing
issues in the inclusiveness of data collection. Our human evaluation serves as
a reference for conducting user assessments of data quality in motion
synthesis.

There was a strong correlation between ease and frequency. The last two
scatter plots (Figure 9(e)(f)) showcase a strong positive correlation between
ease and frequency. In other words, less difficulty was associated with a
higher frequency of usage. The Pearson correlation coefficients (Schober et
al., 2018) are respectively 0.956 and 0.959 for Text2Motion and HumanML3D.
This result was expected for motions that are difficult to perform for
wheelchair users being less frequently performed because they are often
circumvented by alternative motions, which was confirmed by comments from
participants later on.

Factors on ease: range of motion, pain, balance, and tiredness. Participants
evaluated how easy a motion was with several factors. 7 out of 13 people
highlighted their range of motion using keywords “range of motion”, “being
paralyzed”, “based on my abilities”, “considering the angles.” Six
participants mentioned that pain affected their decisions.  Regarding
tiredness, P2 explained “Since I have no core muscles, being paralyzed from
the nipple line down, a lot I can do, but can’t do for long”. Besides, P2, P4,
and P5 emphasized balance. For example, P2 commented “My left side is a little
higher on my injury, so I struggle a lot with that side, or even staying
upright when both hands are in use.” Their perception of ease was reflected in
Likert-scale scores in Figure 9 (a)(c).

Factors on frequency: utility and ease. How frequently one would make a motion
depended on both utility and ease. This explained why the correlation
coefficient was strong but not definite (e.g., R=0.99). An easy motion did not
necessarily lead to frequent usage. Participants determined frequency mainly
based on their routine. P1 commented, “Cooking, cleaning, driving, shopping…
based on where/when/how I might use the motion, I decided how often I might
actually use the motion.” Along with P1, six more users recalled their daily
routine when rating for the frequency. P3 and P8 also referred to
exercises/therapies to define the frequency of each motion.

On the other hand, when difficulty and utility had conflicts, participants
were experts in circumventing them with alternative motions if possible. P4
explained, “I’m limited in my arm functions with putting hands or arms over my
head and behind my back… so I use straps on shoulders and high back on my
wheelchair.” P5 also commented on how he used elbows and hands to support
himself without abdominal muscles when needed, “As a tetraplegia, I don’t
really have abs. So, I’m kinda ’crawling’ to get back up. Propping myself up
with elbows on knees and such.”

Sense of embodiment. Results show that our uses of the skeletal model
successfully facilitated the sense of embodiment – participants could think of
the human model as their own body, and imagine themselves performing those
motions when evaluating the ease and frequency. P1 commented, “I mimicked the
motions and tried to decide when/how I would use the motion”. Similarly,
another comment said, “I picture myself doing certain tasks throughout the
day, or motions I use for exercise.” On this note of sense of embodiment
facilitation, we also chose not to attach text descriptions to animation
clips, but encouraged users’ imagination by allowing them to interpret the
visuals on their own.

We expanded our study and recruited two new amputee participants (U1 and U2,
not from P1-P13). They (1F, 1M) both had an artificial leg and used
wheelchairs every day. We asked open-ended questions about their expectation
of pose estimation, including but not limited to an amputated skeleton, the
integration of artificial limbs, and the desired features/interfaces for pose
estimation. To increase the sense of embodiment, U1 and U2 both proposed using
an amputated model that accurately represented their body. For example, U1
commented, “To be accurate I think the skeleton should be amputated to
maintain accurate pose estimations.” Similarly, U2 said the model “should be
able to be customized to meet different users with different amputations.”
However, expectations changed when artificial limbs came in. U2 talked about
our research’s reflection of his artificial limb, and said “The skeleton
should adapt to work together with the artificial part that is added.”
Meanwhile, U1 insisted the integration of artificial limbs depended on users,
saying “I think should allow users to access settings and have a choice of
their own.” As for the feature/interface, both U1 and U2 mentioned a settings
panel to annotate where a person was amputated, and U1 further suggested an
option to turn on/off the integration of artificial limbs.

Histogram showing the density of images from 0 to 0.4 on the Y axis against
the number of bounding boxes from 0 to 15 on the X axis. Two lines are shown.
WheelPose-Gen was more evenly distributed from 0-12 bounding boxes at around
0.13 density while COCO peaked at 1 bounding box at 0.38 density and quickly
descended to 0.04 at 10 bounding boxes before slightly jumping to 0.10 at 15
bounding boxes.

(a) Number of Bounding Boxes per Image

Histogram showing the density of images from 0 to 35 on the Y axis and the
area of a bounding box relative to the image from 0 to 1 on the X axis. Two
lines are shown. Both WheelPose-Gen and COCO follow a similar exponential
decay-shaped distribution, starting at a density of 30 and 38 respectively,
and dropping to nearly 0 when the area relative to the image is greater than
0.2.

(b) Relative Size of Bounding Box

Two heatmaps showing the distribution of bounding boxes. WheelPose has a more
oblong distribution stretching upwards into the top edge of the heatmap. COCO
has a very circular distribution centered right in the middle of the heatmap
with little to no distribution at the edges of the heatmap.

(c) Bounding Box Spatial Location Heatmap

Figure 10. Bounding Box Statistics. All COCO statistics are computed for
images that contain at least one person instance in COCO. Figure 10(a) Number
of bounding boxes per image. Images with no human bounding boxes are not
counted as COCO is not a purely human dataset. Figure 10(b) Relative area of
each bounding box compared to the image area. Relative area is computed
through $\sqrt{\text{bounding box area}/\text{image area}}$. Figure 10(c)
Heatmap of bounding box location scaled by image size. The color of each pixel
maps to the likelihood of that corresponding coordinate in an image being
bounded by bounding boxes. The peak location of the heatmap is marked with a
green $+$.

###  4.2. Statistical Analyses

We performed a set of statistical analyses to examine how the diversity and
size of our dataset compare to the full COCO 2017 persons dataset (training
and validation) (Lin et al., 2015), selected as our benchmark for its ubiquity
in other 2D human pose estimation related research (Cao et al., 2017; Sun et
al., 2019; Xu et al., 2023). Greater diversity and size in training datasets
has been shown to improve machine learning performance (Gong et al., 2019) and
is a common solution to prevent overfitting. We consider the following
categories in our analyses: high-level dataset features, bounding box
location, size, and number in generated images, keypoint number and occlusion
per image and instance, diversity of human poses, and camera placement. For
all subsequent dataset statistics, we used the WheelPose-Gen dataset, which
was generated with no real-world data using SyntheticHumans models,
Text2Motion animations, and SynthHome background images. The other datasets
from WheelPose shared similar statistics and were skipped in this validation
to avoid redundancy.

####  4.2.1. High-Level Dataset Features

In total, our dataset has 70,000 images with bounding boxes and keypoint
annotations. We chose to generate 70,000 images to mimic the size of the COCO
dataset. For reference, a recent effort in synthetic data has proven that a
PSP dataset of as little as 49,000 images was found to have meaningful pose
estimation improvements in Detectron2 fine-tuning (Ebadi et al., 2022b).

Bar graph showing the probability of keypoints being not visible on the Y axis
and the 17 different COCO keypoints on the X axis. Two bars for each keypoint.
COCO had a much probability for all keypoints, hovering around 0.5 while
WheelPose-Gen stayed around 0.2.

(a) Not Visible

Bar graph showing the probability of keypoints being occluded on the Y axis
and the 17 different COCO keypoints on the X axis. Two bars for each keypoint.
WheelPose-Gen had a much higher probability for all keypoints, hovering around
0.3 while COCO stayed around 0.05.

(b) Occluded

Bar graph showing the probability of keypoints being visible on the Y axis and
the 17 different COCO keypoints on the X axis. Two bars for each keypoint.
WheelPose-Gen had a higher probability for most keypoints except the hips,
hovering around 0.5 while COCO stayed around 0.4. WheelPose-Gen hips had a
probability of around .35.

(c) Visible

Figure 11. Probability of Occlusion Labelings per Keypoint. Figure 11(a)
Probability of a keypoint being labeled as ”not visible.” Figure 11(b)
Probability of a keypoint being labeled as ”occluded.” Figure 11(c)
Probability of a keypoint being labeled as ”visible.” Labeling definitions are
taken directly from the COCO occlusion and keypoint labeling standard. (Lin et
al., 2015).

Figure 12. Heatmap of Five Keypoints Location. Top row: WheelPose-Gen. Bottom
row: COCO. All keypoints locations in the heatmap are computed by
$(\frac{\text{x}-\text{bounding box top corner x}}{\text{bounding box
width}},\frac{\text{y}-\text{bounding box top corner y}}{\text{bounding box
height}})$. The heatmap is normalized according to the size of the dataset.

Ten heatmaps depicting the spatial distribution of 5 keypoints with respect to
the bounding box for WheelPose-Gen and COCO. The distributions were as
follows. Nose: WheelPose-Gen and COCO both had a majority of the distribution
fall at the top of the heatmap. Wrists: WheelPose-Gen had a distribution
across the entire top half of the heatmap while COCO centered around the
left/right edge of the corresponding-sided wrist. COCO was also asymmetrical
where the distribution peaked on the edge of the corresponding sided wrist.
Hips: WheelPose-Gen had hips centered in the middle of the heatmap with little
deviation while COCO had hips centered around their corresponding side in the
middle of the heatmap and deviated all the way to the bottom of the heatmap in
a near T shape.

Our dataset contains 296,508 human instances, of which 271,803 instances have
annotated keypoint labeling. Note that not all human instances have annotated
keypoints because some human instances had no joints within the camera’s view
despite still being visible. In comparison, COCO has 66,808 images with
273,469 human instances, of which 156,165 have annotated keypoints. This
difference in human instances and annotated keypoints are both due to out-of-
view keypoints like in our dataset, and a lack of keypoint labels due to human
labeling errors.

####  4.2.2. Bounding Boxes

We analyze the bounding box annotations by generating a set of statistics
comparing WheelPose-Gen against COCO (Figure 10) to evaluate the diversity of
the frequency, placement, and size of human instances. We find that WheelPose-
Gen contains a more even distribution of the number of bounding boxes, or
human instances, per image compared to COCO, indicating a greater diversity of
images featuring wheelchair users in differently sized groups (Figure 10(a)).
The COCO dataset does however have a higher concentration of images with large
amounts of human instances within them, most likely due to the number of
images depicting crowds of people. We also calculated the area of the bounding
box relative to the overall image size to analyze the size of individual human
instances (Figure 10(b)). Here, we observe that our bounding boxes have
slightly more evenly distributed sizes in relation to the image size when
compared to COCO. Note that our dataset consists of images of a uniform size
($1280\times 720$), and COCO contains a wide variety of different image sizes
up to a maximum size of $640\times 640$. Thus, images with the same relative
size in relation to the image dimensions are still a higher definition in
WheelPose-Gen comparatively.

We then analyze the spatial distribution of bounding boxes by plotting a
heatmap of bounding box locations for both WheelPose-Gen and COCO (Figure
10(c)). We note that the location of bounding boxes is a direct product of the
human generation, occluder, and camera randomizers and acts as a
quantification of their effects. For both datasets, we overlay the bounding
boxes with their location scaled by the overall size of the image. For the
COCO dataset, we observe a majority of bounding boxes are centered in the
middle of the image. We also observe that WheelPose-Gen has a wider bounding
box distribution with more spreading into the top edge of the image compared
to COCO, indicating our randomization parameters create a more even spread of
human instances across the image with more examples of humans at the edge of
the camera. The center of the distribution of bounding boxes in WheelPose-Gen
is also slightly higher in the image than that of COCO.

####  4.2.3. Keypoints

We first measure the probability of a keypoint to be one of the three
predefined COCO occlusion levels (not visible, occluded, visible) in
WheelPose-Gen and COCO as further quantification of the effects of the
randomizers listed previously. In the context of WheelPose, not visible is
when a keypoint is not in the image and has no prediction, occluded is when a
keypoint is in the image but not visible (e.g., behind an object), and visible
is when a keypoint is seen in an image. Here we see that WheelPose-Gen
displays a significantly smaller probability of having nonvisible keypoints
and a more uniform distribution of keypoints compared to COCO (Figure 11(a)).
We also notice that WheelPose-Gen has a far higher probability for a keypoint
to be labeled as occluded compared to COCO, especially in the hips (Figure
11(b)). This can be explained by the different methods both datasets used in
keypoint annotation. While WheelPose-Gen uses a self-occlusion labeler defined
in PSP which computes the distance between each keypoint and the closest
visible part of the object within a threshold to determine occlusion labeling
(Ebadi et al., 2022b), COCO is a fully human-labeled dataset. Within human
annotators, there are variations between how an annotator might define and
classify as occluded and not visible. WheelPose does not suffer from the same
issue as the labeler has information on the full 3D scene and can precisely
identify every keypoint location over a human annotator who only has access to
a single 2D view with no additional context. This phenomenon can be further
seen in the probability of keypoint visibility, where WheelPose-Gen displayed
a higher probability for all keypoints except the two keypoints on hips
(Figure 11(c)) where many hip keypoints are labeled as occluded due to the
self-occlusion of the wheelchair.

We evaluate the diversity of our poses by creating a heatmap of keypoint
annotations locations scaled by the corresponding bounding box in WheelPose-
Gen and COCO (Figure 12). WheelPose-Gen displays a wider distribution of
potential keypoint locations in upper body keypoints compared to COCO. Lower
body keypoints display a smaller distribution due to the limitations of
postures in a wheelchair. We see for asymmetrical keypoints (left/right
wrist), WheelPose-Gen is far more evenly distributed across the X axis
compared to COCO, a clear indication of a more even distribution of front,
side, and back-facing human instances. We also notice a smaller Y-axis
distribution in WheelPose-Gen due to the presence of the wheelchair limiting
potential movements up and down.

####  4.2.4. Camera

Finally, we quantify the variations in our camera placement and rotation.
Recent studies have shown the critical impact of diverse camera angles on
model performance in 3D human recovery problems (Cai et al., 2022; Madan et
al., 2021; Park et al., 2023). We visualize our diversity of camera angles and
distances and observe a wide distribution of potential elevations, azimuths,
and distances (Figure 13). We then sample a set of camera locations relative
to individual human instances and visualize their angle around the instance to
observe full coverage of camera angles all around instances (Figure 13(d)).
All visualizations indicate a loosely followed Gaussian distribution with a
wide variety of different camera angles. We do not compare these statistics
with COCO since it does not include camera configuration information. Our
camera configuration parameters can be found in Table 1.

Histogram of camera elevation with density from 0 to 0.5 on the Y axis and
angles in radians from -1.57 to 1.57 on the X axis. The distribution roughly
follows a normal distribution but peaks twice to 0.5 density at -0.5 and 0.5
radians before dipping to 0.4 at 0 radians.

(a) Camera Elevation

Histogram of camera azimuth with density from 0 to 0.5 on the Y axis and
angles in radians from -1.57 to 1.57 on the X axis. The distribution roughly
follows a normal distribution but peaks twice to 0.5 density at -0.5 and 0.5
radians before dipping to 0.45 at 0 radians.

(b) Camera Azimuth

Histogram of camera distance with density from 0 to 0.12 on the Y axis and
distance in meters from 7.5 to 27.5 on the X axis. The distribution roughly
follows a normal distribution peaking at 20 meters with a density of about
0.11.

(c) Camera Distance

Visualization of potential camera angles. A human model in a wheelchair in
T-pose is surrounded by a set of points on the surface of a sphere
representing a potential camera angle. The human model has camera angles
surrounding all sides of it, creating an evenly distributed sphere of camera
angles.

(d) Camera Potential Angles

Figure 13. Distribution of Potential Camera Angles and Distances. Figure 13(a)
Distribution of elevation angle (up-down, positive indicating a camera above
the nose and looking down). Figure 13(b) Distribution of azimuth angle (left-
right, positive indicating a camera is to the right looking left)
distribution. Figure 13(c) Distribution of camera distance to a human
instance. Figure 13(d) Visualization of potential camera angles. Computed by
sampling camera location relative to human instance every 100 human instances
and visualizing the corresponding unit vector.

###  4.3. Model Performance Evaluation

####  4.3.1. Testing Dataset

There currently does not exist an image dataset focused primarily on
wheelchair users. For the testing of our system, we create a new dataset of
2,464 images of wheelchair users collected from 84 YouTube videos in a similar
data collection process as other computer vision works (Andriluka et al.,
2014; Wang et al., 2019). A set of 16 action classes consisting of common
daily tasks wheelchair and able-bodied users both perform and unique
wheelchair sports were selected (e.g., talking, basketball, rugby, dancing,
etc.). Action classes are listed in Table 2. Videos are collected through
keyword searches revolving around each of the action classes. Annotators
iterate through 500 equally spaced frames (minimum one-second intervals) from
each video and identify frames with poses and settings sufficiently different
from the previously collected images. Annotators ensure that there is a
wheelchair user within view for every collected frame. Crowd worker
involvement is then utilized to annotate bounding boxes and keypoint locations
on collected images. Researchers manually validated results from this process
for accuracy. Examples of this dataset can be found in our open-source
repository at https://github.com/hilab-open-source/wheelpose.

Table 2. Ablation trials with configurations of each dataset and results in terms of the mean Average Precision (mAP) in the bounding box (BB) and keypoint (KP) detection tasks. Dataset | Human Model | Background | Animation | BB mAP | KP mAP
---|---|---|---|---|---
WheelPose-base | PSP default | PSP default | HumanML3D | 68.68 | 65.18
WheelPose-SH | SyntheticHumans | PSP default | HumanML3D | 68.53 | 68.04
WheelPose-t2m | PSP default | PSP default | Text2Motion | 68.95 | 64.83
WheelPose-t2mR10 | PSP default | PSP default | Text2Motion random 10% removed | 68.97 | 65.19
WheelPose-t2mHE10 | PSP default | PSP default | Text2Motion HE 10% removed | 69.36 | 65.53
WheelPose-SB | PSP default | SynthHomes | HumanML3D | 69.75 | 66.60
WheelPose-RB | PSP default | BG-20K | HumanML3D | 64.44 | 63.89
WheelPose-Gen | SyntheticHumans | SynthHomes | Text2Motion | 69.71 | 67.53
WheelPose-Opt | SyntheticHumans | SynthHomes | Text2Motion HE 10% removed | 69.58 | 67.96
ImageNet (baseline) | N.A. | N.A. | N.A. | 35.19 | 63.11
PSP (baseline) | PSP Default | PSP Default | PSP Default Able-Bodied | 31.03 | 53.26

####  4.3.2. Training Strategy

Similar to training outlined in PSP-HDRI (Ebadi et al., 2022a), all of our
models in this evaluation utilized ResNet-50 (He et al., 2016b) plus Feature
Pyramid Network (FPN) (Lin et al., 2017) backbones. Additionally, these models
were fine-tuned using the starting weights and framework of the Detectron2
ImageNet Keypoint R-CNN R50-FPN variant (He et al., 2018).

We create each model in the same way: fine-tuning with the relevant dataset on
the backbone described previously. We opted for a simple training approach,
setting the initial learning rate to 0.000025 for 20 epochs, and then lowering
the learning rate by a factor of 10 for an additional 10 epochs. For the first
1000 iterations we also conduct a linear warm-up of the learning rate to its
starting value, slowly increasing the learning rate to its starting value. The
momentum was set to 0.9 and the weight decay was set to 0.0001. All training
runs were completed using a 4.2GHz 16-core/32-thread AMD Ryzen Threadripper
PRO 3955WX CPU, 2 NVIDIA RTX A5500 24GB VRAM, and 256 GB 3200MHz DDR4 memory
with a mini-batch size of 13 images per GPU, where each image was normalized
using the mean pixel value and standard deviation of the ImageNet base model.
For each model, we checkpointed the model weights during every epoch and
selected the epoch with the best-performing keypoint AP to report in
evaluation. This evaluation scheme was consistent across baseline datasets and
our synthetic datasets.

We note that WheelPose fine-tuned models and individual human evaluators may
not agree with each other on where a keypoint is due to the innate difference
between a modeled person and a real person. Changes in the way WheelPose
defines keypoints can alter where a keypoint is predicted and the metrics
computed in the coming sections as seen in Section A.5. We attempted to
minimize these differences as much as possible through realistic keypoint
definitions in Unity.

####  4.3.3. Ablation Testing Strategy

We address (RQ2.1) through ablation testing444Ablation testing involves the
removal of certain components to understand the contribution of the component
to the overall performance of an AI system on a set of selected domain
randomization parameters, including animations, backgrounds, and human models,
to better understand the performance impacts of select data generation
parameters.

Configuration. Regarding the human model, we analyze the effects on the model
performance of using: 1) PSP default human models (PSP Default), and 2)
SyntheticHuman human models (SyntheticHumans). As to the background, we
compare between 1) PSP default texture backgrounds (PSP Default), 2)
SynthHomes background images (SynthHomes), and 3) BG-20K real background
images (BG-20K). Finally, regarding the motion sequence, we compare between 1)
HumanML3D animations (HumanML3D), 2) Text2Motion animations (Text2Motion), 3)
Text2Motion with $10\%$ of animations randomly removed (Text2Motion random
$10\%$ removed), and 4) Text2Motion animations with the bottom $10\%$ of
animations in total ease and frequency from human evaluations (HE)
(Text2Motion HE $10\%$ removed). Table 2 shows this list of datasets and their
configurations.

We use combinations of these parameters to assemble a set of datasets of
70,000 images each. Each dataset consisted of $\approx 65,000$ images with at
least one human instance and was used to fine-tune the base ImageNet model
using the strategy listed in Section 4.3.2. We also include the original PSP
dataset with $\approx 65,000$ images with at least one human instance
generated from the provided Unity environment to test the efficacy of fine-
tuning with synthetic able-bodied user data (Ebadi et al., 2022b). For these
tests, all models and our baseline tests, ImageNet and PSP, were tested on our
real wheelchair data testing set using the industry standard metric for
detection and estimation accuracy, COCO mean Average Precision (Lin et al.,
2015).

Results. Results in terms of bounding box (BB) and keypoint (KP) mean APs
(mAP) across ablation trials are shown in Table 2.

Regarding person detection (BB) performance, all datasets demonstrated
significant performance boosts of varying degrees when compared to ImageNet
and PSP. Notably, our best-performing ablation test led to a $98.21\%$
improvement in BB mAP (WheelPose-SB) and a $7.81\%$ improvement in keypoint
mAP (WheelPose-SH) over the best baseline dataset (ImageNet). This indicates
the efficacy of our synthetic data and the large headroom for improvement in
detecting poses of wheelchair users with industry-standard deep learning
models.

Table 3. Bounding box AP performance comparison between base models and WheelPose-Opt. We list the mean of our seeded testing $\pm$ the maximum absolute deviation from the mean. Dataset | BB mAP | BB APIoU=.50 | BB APIoU=.75 | BB APsmall | BB APmedium | BB APlarge
---|---|---|---|---|---|---
WheelPose-Opt | $69.46\pm 0.22$ | $90.77\pm 0.20$ | $82.75\pm 0.61$ | $3.24\pm 0.76$ | $59.82\pm 0.85$ | $69.91\pm 0.21$
ImageNet (baseline) | 35.19 | 71.49 | 28.50 | 0.00 | 3.15 | 36.12
PSP (baseline) | 31.03 | 63.64 | 25.08 | 0.00 | 6.27 | 32.35
Table 4. Keypoint AP performance comparison between baseline and WheelPose-Opt. We list the mean of our seeded testing $\pm$ the maximum absolute deviation from the mean. We do not include APsmall since the human is too small to accurately assign keypoints. Dataset | KP mAP | KP APOKS=.50 | KP APOKS=.75 | KP APmedium | KP APlarge
---|---|---|---|---|---
WheelPose-Opt | $67.93\pm 0.02$ | $87.61\pm 0.19$ | $74.48\pm 0.25$ | $35.48\pm 0.40$ | $68.99\pm 0.06$
ImageNet (baseline) | 63.11 | 77.43 | 67.20 | 6.22 | 64.96
PSP (baseline) | 53.26 | 68.07 | 57.36 | 10.15 | 56.12

Poor performance from BG-20K may be explained by the detail of background
images. PSP default and SynthHomes data tend to feature a set of simple or
smooth textures while real-world images are often more detailed and consist of
more texture. These results may signal a preference for other background
characteristics over pure realism.

When examining keypoint performance, the most significant improvement was the
inclusion of Unity SyntheticHumans models. This makes intuitive sense, as the
diverse and more representative human models more closely match up with the
humans found in the real world rather than the generalized Unity humanoid
models. The variations between models also help combat overfitting issues by
introducing many different definitions of what is a ”human” to the model.

Finally, we examine the performance tradeoffs in our different animation sets.
We found motion generation outputs with random filtering (WheelPose-t2mR10)
performed comparably to motion capture data (WheelPose), indicating similar
motion quality between the two data sources. We also found that randomly
filtered Text2Motion animations (WheelPose-t2mR10) performed better than the
full motion set (WheelPose-t2m). This may imply that the number of animations
and poses is not directly correlated with model performance. It is important
to note these results may change depending on what animations have been
filtered. This idea is further shown in the HE model (WheelPose-t2mHE10),
which showed improvement in both BB and KP mAP over the randomly removed
$10\%$ model. We see that the removal of specific animations that are not
perceived as ”realistic” can improve the model performance of generated data.

From our ablation testing results, we assembled two new datasets: WheelPose-
Gen, a dataset created from fully generative parameters using SyntheticHuman
models, SynthHomes backgrounds, and all $100$ Text2Motion animations, and
WheelPose-Opt, a dataset created from the best performing parameters from
ablation testing which include SyntheticHuman models, SynthHomes backgrounds,
and Text2Motion HE 10% removed animations. Both datasets performed comparably
to the best performing ablation test in BB and KP mAP (WheelPose-SH) with
WheelPose-Opt. We note that WheelPose-Opt outperforms WheelPose-Gen in KP mAP
which follows our findings in the initial ablation test. Our findings indicate
that different combinations of domain randomization parameters can produce
better AI models than the best perform parameters individually.

####  4.3.4. WheelPose-Opt Model Performance Deep Dive

We further evaluate (RQ2.2) by conducting an in-depth quantitative and
qualitative analysis of the changes in performance in Detectron2 when fine-
tuned with WheelPose-Opt, the best performing dataset from ablation testing
using synthetic data and simple human evaluations.

Configuration. We trained and evaluated the results for WheelPose-Opt with the
same strategy described in Section 4.3.2 three separate times using different
model seeds (42, 4242, 424242). We then computed the mean and maximum absolute
deviation of a set of evaluation metrics, including BB AP, KP AP, and
individual keypoint metrics, across the three trials. We compute the same
metrics on ImageNet and PSP for use as our baseline.

Results. We first quantify our overall bounding box and keypoint performance
with AP and its related breakdowns to build an overarching view of our
dataset’s performance across different scenarios. AP at different IoU and
OKS555IoU and OKS perform the same fundamental purpose for bounding boxes and
keypoints respectively thresholds measure the prediction accuracy at varying
degrees of recall (Higher indicates a stricter ground truth definition).
Additionally, the overall AP score is split into small, medium, and large
based on the detection segment area to quantify the performance at different
camera distances and human instance sizes. More information is found in the
COCO documentation (Lin et al., 2015).

Table 3 lists the BB AP performance for WheelPose-Opt and the baselines. Our
dataset displays over a 98% improvement over both baseline scores in all
subcategories of AP. Furthermore, we notice a major drop in performance in the
baseline models as the IoU threshold becomes stricter. In contrast, our
WheelPose models maintain a high level of accuracy across a much wider range
of IoU thresholds. These results indicate WheelPose-Opt can not only identify
a wheelchair user but is capable of drawing an accurate bounding box around
the entire human instance compared to the baseline models which are only
capable of low-fidelity wheelchair user detections.

As shown in Table 4, pose estimation improved by up to 7.64% in KP mAP in
WheelPose-Opt when compared to ImageNet and up to 27.54% when compared to PSP.
Furthermore, we see a similar or greater magnitude of improvement across all
AP metrics, indicating the use of WheelPose-Opt improves pose estimation in
all scenarios. We note drastic improvements in AP on medium-sized human
instances, where ImageNet had noticeably poor performance (6.22). Our system,
thus, not only improves but even enables existing models to estimate the
postures of wheelchair chairs at further distances.

Table 5. Keypoint performance breakdown of our primary datasets. We list the mean of our seeded testing $\pm$ the maximum absolute deviation from the mean. Within the parentheses on the right is the percent difference from the base model, ImageNet. Percentage of Detected Joints (PDJ) @ 5 describes the percentage of correctly predicted joints within a 5% bounding box diagonal radius (Sapp and Taskar, 2013). Per Joint Position Error (PJPE) describes the mean Euclidean distance for each keypoint from the ground truth (Zwölfer et al., 2023). Object Keypoint Similarity Score (OKS) as described by COCO (Lin et al., 2015) and used in Chen et al. (Chen and Zwicker, 2021) is the mean precision per keypoint evaluated at both standard loose and strict similarity thresholds of 0.5 and 0.75 respectively. Superiority directions are noted as + and - next to each metric.  Keypoint | PDJ@5 (+) | PJPE (-) | OKS50 (+) | OKS75 (+)
---|---|---|---|---
nose | $0.91\pm 0.02(-2.15\%)$ | $8.73\pm 4.06(-77.38\%)$ | $0.90\pm 0.02(-3.23\%)$ | $0.89\pm 0.03(-1.83\%)$
eyes | $0.99\pm 0.00(+5.70\%)$ | $6.30\pm 0.64(-77.6\%)$ | $0.96\pm 0.01(+3.76\%)$ | $0.95\pm 0.01(+4.19\%)$
ears | $0.92\pm 0.01(+5.56\%)$ | $10.90\pm 0.03(-61.60\%)$ | $0.93\pm 0.00(+5.90\%)$ | $0.83\pm 0.01(+0.81\%)$
shoulders | $0.92\pm 0.02(+6.74\%)$ | $10.98\pm 0.26(-63.89\%)$ | $1.00\pm 0.01(+4.74\%)$ | $0.97\pm 0.01(+6.38\%)$
elbows | $0.86\pm 0.01(+2.78\%)$ | $14.54\pm 0.23(-69.38\%)$ | $0.97\pm 0.00(+5.80\%)$ | $0.92\pm 0.00(+6.55\%)$
wrists | $0.85\pm 0.01(-0.19\%)$ | $16.22\pm 1.65(-76.22\%)$ | $0.92\pm 0.02(+6.13\%)$ | $0.88\pm 0.02(+1.73\%)$
hips | $0.65\pm 0.01(-18.96\%)$ | $22.96\pm 0.61(-51.24\%)$ | $0.97\pm 0.00(-2.02\%)$ | $0.89\pm 0.01(-6.63\%)$
knees | $0.89\pm 0.02(+6.63\%)$ | $14.99\pm 0.97(-78.85\%)$ | $0.97\pm 0.01(+5.25\%)$ | $0.94\pm 0.01(+7.01\%)$
ankles | $0.78\pm 0.01(+76.14\%)$ | $18.16\pm 0.95(-69.04\%)$ | $0.94\pm 0.00(+87.67\%)$ | $0.90\pm 0.01(+91.49\%)$

Finally, we compute a set of per-keypoint metrics shown in Table 5 to analyze
the differences in specific keypoint predictions between WheelPose-Opt and the
baseline. We only showcase percent change with respect to ImageNet as the PSP
fine-tuned model performs drastically worse in nearly all BB and KP metrics.
Upon examining the percentage of detected joint (PDJ) (Sapp and Taskar, 2013)
at a 5% threshold, a measure of a model’s ability to identify a joint, we
notice a 76.14% improvement in ankle detection, attributed to more information
on foot placement in wheelchairs, and a 18.96% decrease in detected hips,
attributed to the wheelchair obstructing most of the lower torso and hip area.
We then compute the per joint position error (PJPE), a simple accuracy metric
measuring the Euclidean distance error in detected joints, and found that as
long as a joint is detected, WheelPose-Opt predicts keypoints over 51.24%
(hips) more accurately. Finally, we compute the meaned per-keypoint precision
values at OKS thresholds of 0.5 and 0.75 as another measure of individual
keypoint prediction accuracy. We see slight improvements across most joints.
Similar to PDJ, we can see similar trends in the ankles and hips, improving a
significant amount or slightly worse respectively.

####  4.3.5. Key Prediction Changes

We conduct a visual analysis of the changes in predicted keypoints between
ImageNet and WheelPose-Opt to analyze the information transfer between our
synthetic data and the real world across different scenarios to identify
specific situations where we perform better or worse. We ignore the PSP
dataset as it demonstrated noticeably worse performance compared to ImageNet
in both bounding box and pose estimation (Tables 3 and 4). Examples of
different trends were plotted with the predictions from both WheelPose-Opt, in
green, and ImageNet, in red, overlaid on top.

Improvements in wheelchair user detection. As shown in the BB AP improvements
between ImageNet and WheelPose-Opt in Table 3, we notice major improvements in
wheelchair user detection. Figure 14 shows some examples of these improvements
in a variety of different environments. Figures 14, 14 and 14 shows two
examples of proper wheelchair user detection through WheelPose-Opt. Figures 14
and 14 shows two examples of wheelchair users being detected in low visibility
settings due to both extremely bright and dark lighting conditions. Of
particular note is Figure 14, where even the background poster of a wheelchair
image has been detected, which human annotators had missed. We also notice
that even if ImageNet had detected a wheelchair user, its bounding box
prediction still was not accurate. Figures 14, 14, 14, 14 and 14 shows some
examples of this, where ImageNet tends to cut off portions of the wheelchair
user’s full body while WheelPose-Opt captures a more accurate and
representative bounding box.

A wheelchair dancer gets ready for her performance in front of a crowd at a
45-degree angle from the camera.

A wheelchair leans back while holding hands with an able bodied dancer.

A side view of a wheelchair user in a dark lighting condition against a
projector.

A side view of a wheelchair dancer with their back hunched down. In the
background, there are four judges against a promotional poster featuring a
wheelchair user.

A top down view of a wheelchair tennis player in a court. The wheelchair user
is holding a tennis racket. Another person is standing on the court.

A back view of a wheelchair user going towards a car. Both hands are on their
wheels as they push towards the vehicle.

A back view of a wheelchair user dancing with an able-bodied user. The able
bodied user is sitting on the ground as the wheelchair user looks at them.

A close-up 45-degree front view of a wheelchair user at a table. The camera
cuts off part of their head as they use their phone. The camera captures their
legs under the table.

A back-view of a wheelchair user entering a slope at a skate park. The user is
pushing fast with both hands in front of them. They are wearing a helmet and
elbow pads.

A front view of a wheelchair user holding a basketball in their lap. They are
looking directly at the camera.

Figure 14. Examples of wheelchair user detection improvements with WheelPose-
Opt over ImageNet. Red represents ImageNet predictions while green represents
WheelPose-Opt predictions. Figures 14, 14, 14, 14 and 14 all show wheelchair
users in different scenarios who were completely undetected by ImageNet but
detected with WheelPose-Opt fine-tuning. Figures 14, 14, 14, 14 and 14 all
show wheelchair users in different scenarios who were detected by ImageNet,
but had poor bounding box predictions which were improved in WheelPose-Opt
fine-tuning.

Similar performance in front-facing scenarios. In front-facing scenarios,
ImageNet and other pose estimation models often perform very well on
wheelchair users. This is because, at this angle, the user can simply be
interpreted to be sitting, with all limbs in full view of the camera. Thus in
practice, the front-facing wheelchair user is very similar to a front-facing
able-bodied user that is sitting. We find that fine-tuning with WheelPose-Opt
performs comparably with the base ImageNet models in front-facing scenarios.
Thus, our system maintains crucial information learned from the initial
training of ImageNet that has proven to work well on wheelchair users already.
Examples of this phenomenon are shown in Figure 15.

A wheelchair user in a motorized wheelchair facing forward in the middle of a
tiled room. The user’s arms are moving in front of them.

A wheelchair user pushing forward towards the camera in the middle of a bright
room with mirrors. The overall brightness of the image is high.

A wheelchair user pushing forward towards the camera in a parking lot. Both
hands are placed on the front of the wheel.

A wheelchair user is stationary in front of a camera on a field and in front
of the road. Both hands are placed on the wheels. The user is not wearing
shoes.

Figure 15. Examples of similar performance between WheelPose-Opt and ImageNet
in front-facing wheelchair users. Red represents ImageNet predictions while
green represents WheelPose-Opt predictions. Figures 15, 15, 15 and 15 all
depict examples of front-facing wheelchair users in a variety of different
settings. The WheelPose-Opt fine-tuned model and ImageNet both performed
similarly, generating relatively matching keypoint predictions.

Improvements in keypoint estimation in wheelchair self-occlusion scenarios.
While existing pose estimation models may work well when the wheelchair user
is facing directly forward, they often break down when the user is turned away
and the wheelchair begins obstructing the view of the full human body. We find
that the additional synthetic data from WheelPose-Opt helps Detectron2 discern
between what is a part of the wheelchair and what is a part of the user’s body
for a more accurate prediction. Figure 16 illustrates a few examples of such
poor performance in ImageNet and improved predictions enabled through
WheelPose-Opt. In situations where the legs are fully occluded by the
wheelchair like in Figures 16 and 16, our system generates more reasonable
predictions compared to those of ImageNet, which placed the legs onto the
wheels of the wheelchair or even the elbow. Figure 16 shows an example of how
WheelPose-Opt can improve the detection of self occluded keypoints like the
user’s left knee and ankle. Figures 16 and 16 shows examples of where ImageNet
has mistakenly classified parts of the wheelchair as a keypoint.

A wheelchair user near the corner of the room with their back to the camera.
Both hands are placed on the wheels of the wheelchair.

A side-view of a wheelchair user traveling down an outdoor path. Both hands
are on the wheels doing a slight wheelie.

A wheelchair athlete at the skate park. The athlete is at a 45-degree angle to
the camera in the middle of the air. They are wearing protective gear
including a helmet and gloves.

A wheelchair user on the street traveling at a 45-degree angle away from the
camera towards a parked car. Both hands are placed on the wheels as they move
towards the car.

Figure 16. Examples of improvements in keypoint estimation in WheelPose-Opt
over ImageNet in scenarios where the wheelchair occludes part of the user’s
body. Red represents ImageNet predictions while green represents WheelPose-Opt
predictions. Figures 16, 16, 16 and 16 all display improvements from
WheelPose-Opt on keypoint predictions, specifically in the lower body.

Overfitting on wheelchairs. Upon examining the predictions made by ImageNet
and ImageNet fine-tuned with WheelPose-Opt, we notice that both systems
perform poorly on users with no lower limbs. As shown in Figures 17 and 17
While ImageNet tends to classify the wheelchair as the missing legs, we notice
that our system instead ”fills in the blanks” and predicts legs in reference
to the wheelchair where they might usually be for a wheelchair user. We
further notice that our system tends to have more false positives in detecting
what can be defined as a wheelchair. As seen in Figure 17, objects that
resemble a wheelchair, like a grocery cart, may affect the keypoint estimation
of a wheelchair user. In other cases like the one shown in Figure 17, we see
that our system can even detect empty wheelchairs and fill them in with humans
when there are not any. While this shows great promise in the information
transfer between digitally modeled mobility assistive devices and real-world
data, we find these tendencies can obfuscate the real postures of wheelchair
users.

A young child without lower limbs at a 45-degree angle in a wheelchair playing
pickleball. The child is partially occluded by the pickleball net.

A wheelchair basketball player without lower limbs is dribbling while facing
the camera. The players right hand is on the wheel and the left hand is
dribbling the basketball.

A person is at a 45-degree angle in a motorized grocery cart behind other
grocery carts. They are explaining something with both hands sticking out.

A game of wheelchair tennis. The camera is at a 45-degree angle in an overhead
view. A wheelchair user is raising their racket up in front of an empty
wheelchair and a person crouches on the ground. There is a crowd of spectators
beyond the field of the game.

Figure 17. Examples of overfitting on wheelchairs from WheelPose-Opt. Red
represents ImageNet predictions while green represents WheelPose-Opt
predictions. Figures 17 and 17 Examples of poor predictions on wheelchair
users without lower limbs. Figure 17 An example of overfitting onto any
wheelchair resembling object, like a grocery cart. Figure 17 An example of
overfitting where even an empty wheelchair is detected to have a user.

##  5\. Discussion and Future Work

Generalizable knowledge and transferability to HCI research. Data generation
approaches have been widely adopted in projects in HCI for problems involving
thermal imaging (Hu et al., 2020), IMU (Jain et al., 2023; Rey et al., 2019),
stroke gesture (Leiva et al., 2017), and RF data (Ahuja et al., 2021; Cai et
al., 2020). Given the popularity of data generation in HCI, we believe our
techniques could be easily transferable to related and future works in
accessibility, motion generation, and pose estimation. Furthermore, the
modularization in our pipeline could improve transferability by facilitating
segmented changes – a flexible way for data synthesis to experiment with
different components. Finally, methods shown in our statistical analysis and
model performance evaluation could be highly reusable in future work that
adopts our technique. That being said, our pipeline provides the baseline
framework for futures efforts in research that require different humanoid
models, motion synthesis techniques for upper body and lower body,
environmental factors, and VR toolchains.

Configurability to cater needs of developers and end users. We believe the
results shown in Table 2, Section A.1, and Section A.5 demonstrate a clear
need for developers to have an interactive tool to modify training data, where
slight changes in the data modeling can have major effects on AI performance.
Currently, there are few tools to do so, with many developers choosing to
leverage static motion datasets which may not perfectly fit their needs. This
has been one of the dominant reasons for the rise of inequitable AI models.
For this reason, WheelPose is a pipeline specifically designed to enable a
high degree of configurability. This allows for the creation of personalized
synthetic datasets that cater to developers’ needs, thereby increasing the
likelihood of catering to the needs of end users. This would lead to more
effective and inclusive AI models, which can be tuned to the individual needs
of the user instead of a one-size-fits-all solution for better performance in
real-world applications.

Improvements on the realism of generated data. We hope the use of Unity
enables future research into this idea and enables developers to build a
synthetic data generator that extends beyond our simple simulation environment
using 3D modeled scenes and rooms instead of flat background images. Recent
achievements in Neural Radiance Fields (NeRF) could be leveraged to synthesize
photorealistic background images that adapt in response to changes in the
virtual camera’s perspective. Future work in realistic 3D environment modeling
will enable research in surface semantics for more realistic configurations –
having wheelchair users positioned at ground surfaces. Moreover, physics could
be incorporated to simulate the locomotion of wheelchair users which would
enable the modeling of more realistic motions and collisions with other models
compared to our simple animations.

Efficacy of human evaluation. We found that motions from motion generation
models and motion capture were largely perceived similarly by participants in
the human evaluation (Section 4.1). We further found that these two types of
motions after being filtered for human-perceived realism performed comparably
in model performance evaluation (Table 2). However, whether due to current
generative AI performance or the lack of training data for disability-related
movement, our human evaluations indicate that generative AI models are still
not able to represent users with disabilities accurately without external
human feedback. This is shown in the improved performance with the addition of
human evaluations (Table 2). Due to this, we believe that human evaluators are
still vital to ensure that generated data is representative. We recognize that
our method for assessing motions, although not exhaustive or entirely free of
bias, contributes as an initial stride towards creating datasets that are more
inclusive. Future research should extend upon this work to integrate a more
involved human-centered system which will, instead, allow evaluators to
meaningfully guide the generation of data, including motions, simulation
scenarios, and user modeling, through iterative feedback (e.g., guided
prompting) to create a more representative dataset. We also found little
literature on motion synthesis for people with disabilities and recognized
this vacuum as both a challenge and an opportunity for generative AI.

Comprehensive model performance evaluation. Our model testing did not involve
a serious grid search for data generation hyperparameters, model seeds,
initializations, or model configurations. We also used the same training
strategy for all tested datasets. We held all these values constant to focus
on the quality and impact of the synthetic dataset on model performance. Even
with this naïve approach to training, WheelPose still yielded promising
results in the improvement of pose estimation models through synthetic data on
wheelchair users. We found noticeable improvements in both person detection
and pose estimation problems. This indicates that the specific synthetic
modeling of users in wheelchairs in WheelPose can make existing computer
vision models more equitable by improving performance on wheelchairs. We
believe our findings pave the way for future works in synthetic data on humans
with other mobility-assistive technologies to improve pose estimation
equitability.

Diverse participant groups. A key limitation in this work is that we only
analyze the digitalized representation of users with all four limbs and feet
fixed on the foot rest, which does not express the full range of wheelchair
users with different bodies such as those with amputations, dwarfism, spinal
deformities, and other conditions. Thus the findings may not be reflective of
the wheelchair population at large. We hope our data generation pipeline
provides a framework and will enable future developers to leverage 3D modeling
and Unity tools to create a more diverse body of wheelchair user models. For
instance, developers can easily add new bones to existing models to more
realistically represent spinal deformities. We hope that these tools will also
enable future works analyzing different disabilities and mobility assistive
devices which were not addressed in our current research.

Improve inclusiveness of AI for more recognition modalities. Furthermore,
WheelPose enables more work beyond 2D bounding boxes and pose estimation.
Annotations on depth, surface normals, object segmentation, occlusion, 3D
bounding boxes, and 3D keypoints are fully implemented in our current data
synthesis pipeline but still unexplored. These annotations can be even more
difficult and costly to collect compared to RGB images and 2D pose
annotations, often requiring the use of specialized equipment and data
collection processes. Synthetic data has no such problem, where any desired
annotation and labeling are all equally accessible to collect. Thus, we
believe that WheelPose can be adapted to potentially address problems in
wheelchair pedestrian detection with object segmentation and occlusion
annotations (Dollár et al., 2011), 3D pose estimation using 3D bounding box
and pose annotations, and robotics detection of people and mobility aids
through depth data (Vasquez et al., 2017) among other accessibility-related
problems cheaply and efficiently.

Pitfalls of exclusion in data generation vs. data collection. WheelPose is a
pipeline for both AI developers and wheelchair users to circumvent existing
inaccessible data collection methods and meaningfully improve the training
process of AI models by generating data for wheelchair users. However, we are
cautious that by circumventing existing inaccessible data collection methods
with our tool, we could run the potential risk of furthering exclusion, which
echoes long-standing debates within the accessibility community. Our paper is
based on the assumption that making AI equitable requires pursuing multiple
approaches together – effective approaches to improving representations of
training data from people with disabilities leveraging both data collection
and data generation. We advocate that new tools for data generation and the
existing data collection methods are not mutually exclusive. Their synergy
could lead to a more practical approach to resolving accessibility challenges
than what could be offered by either of the two approaches alone. We believe
that the following characteristics are vital in future works to avoid pitfalls
of exclusion: 1) representative and diverse participant groups, for which we
have conducted studies around spinal injuries of various levels and recommend
future work to consider participants from wider backgrounds; 2) realistic
generated data, for which we invented several data generation techniques
optimizing data realism; and 3) effective tools for human evaluation, for
which we adopted embodiment in our human evaluation interface allowing
participants to seamlessly transfer the presented motion sequences to their
own bodies for a more intuitive evaluation.

##  6\. Conclusion

We introduce WheelPose, an extension of the highly-parameterized synthetic
data generator PeopleSansPeople, for wheelchair users with the possibility of
use in other mobility-assistive technologies to improve the performance of
common pose estimation algorithms in the traditionally underrepresented group
of wheelchair users. WheelPose includes a full end-to-end pipeline to convert
existing motion capture and motion generation model outputs into wheelchair
user animations for use in a complete Unity Simulation scene (RQ1.1)
containing a range of 3D human models from Unity SyntheticHumans in
wheelchairs, backgrounds, occluders, and unique lighting conditions. We
provide full control over all related parameters, including keypoint labeling
schema, for computer vision tasks (RQ1.2). We tested our pipeline using two
different motion sequence sources: motion capture data from HumanML3D and
motion generation outputs from Text2Motion. These motions underwent a set of
human evaluations. We then analyzed the impacts of different domain
randomization parameters and motions on model performance, finding an
”optimal” combination of parameters and comparable performance between our
motion sources (RQ2.1). Finally, we tested the model performance of the
dataset generated through WheelPose with no real-world data using the optimal
parameters found previously on a dataset of real wheelchair users to find
noticeable improvements in model performance when compared against existing
pose estimation models (RQ2.2). We expect WheelPose to enable a new range of
research in using synthetic data to model users with disabilities in improving
the equity of AI.

## References

  * (1)
  * Kin (2019) 2019\.  The Kinesthetic Index: Video Games and the Body of Motion Capture – InVisible Culture.  https://ivc.lib.rochester.edu/the-kinesthetic-index-video-games-and-the-body-of-motion-capture/.
  * Way (2022) 2022\.  Waypoint - The Official Waymo Blog: Utilizing Key Point and Pose Estimation for the Task of Autonomous Driving.  https://waymo.com/blog/2022/02/utilizing-key-point-and-pose-estimation.html.
  * Dee (2023) 2023\.  DeepMotion - AI Motion Capture & Body Tracking.  https://www.deepmotion.com/.
  * Syn (2023) 2023\.  SyntheticHumans Package (Unity Computer Vision).  Unity Technologies.
  * Ahuja et al. (2021) Karan Ahuja, Yue Jiang, Mayank Goel, and Chris Harrison. 2021\.  Vid2Doppler: Synthesizing Doppler Radar Data from Videos for Training Privacy-Preserving Activity Recognition. In _Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems_ (¡conf-loc¿, ¡city¿Yokohama¡/city¿, ¡country¿Japan¡/country¿, ¡/conf-loc¿) _(CHI ’21)_. Association for Computing Machinery, New York, NY, USA, Article 292, 10 pages.  https://doi.org/10.1145/3411764.3445138
  * Anderson et al. (2022) Jason W. Anderson, Marcin Ziolkowski, Ken Kennedy, and Amy W. Apon. 2022.  Synthetic Image Data for Deep Learning.  arXiv:2212.06232 [cs.CV]
  * Andriluka et al. (2014) Mykhaylo Andriluka, Leonid Pishchulin, Peter Gehler, and Bernt Schiele. 2014.  2D Human Pose Estimation: New Benchmark and State of the Art Analysis. In _IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * Azadi et al. (2023) Samaneh Azadi, Akbar Shah, Thomas Hayes, Devi Parikh, and Sonal Gupta. 2023.  Make-An-Animation: Large-Scale Text-conditional 3D Human Motion Generation.  arXiv:2305.09662 [cs.CV]
  * Bak et al. (2018) Slawomir Bak, Peter Carr, and Jean-Francois Lalonde. 2018\.  Domain Adaptation through Synthesis for Unsupervised Person Re-identification.  arXiv:1804.10094 [cs.CV]
  * Bartlett (2014) Roger Bartlett. 2014\.  _Introduction to Sports Biomechanics: Analysing Human Movement Patterns_.  Routledge.
  * Bartolomé Villar et al. (2022) Begoña Bartolomé Villar, Irene Real Benlloch, Ana De la Hoz Calvo, and Gleyvis Coro-Montanet. 2022\.  Perception of Realism and Acquisition of Clinical Skills in Simulated Pediatric Dentistry Scenarios.  _International Journal of Environmental Research and Public Health_ 19, 18 (2022), 11387.
  * Bazarevsky et al. (2020) Valentin Bazarevsky, Ivan Grishchenko, Karthik Raveendran, Tyler Zhu, Fan Zhang, and Matthias Grundmann. 2020.  BlazePose: On-device Real-time Body Pose tracking.  arXiv:2006.10204 [cs.CV]
  * Bazavan et al. (2022) Eduard Gabriel Bazavan, Andrei Zanfir, Mihai Zanfir, William T. Freeman, Rahul Sukthankar, and Cristian Sminchisescu. 2022\.  HSPACE: Synthetic Parametric Humans Animated in Complex Environments.  arXiv:2112.12867 [cs.CV]
  * Bennett and Rosner (2019) Cynthia L Bennett and Daniela K Rosner. 2019.  The promise of empathy: Design, disability, and knowing the” other”. In _Proceedings of the 2019 CHI conference on human factors in computing systems_. 1–13.
  * Borkman et al. (2021) Steve Borkman, Adam Crespi, Saurav Dhakad, Sujoy Ganguly, Jonathan Hogins, You-Cyuan Jhang, Mohsen Kamalzadeh, Bowen Li, Steven Leal, Pete Parisi, Cesar Romero, Wesley Smith, Alex Thaman, Samuel Warren, and Nupur Yadav. 2021.  Unity Perception: Generate Synthetic Data for Computer Vision.  arXiv:2107.04259 [cs.CV]
  * Bridson (2007) Robert Bridson. 2007\.  Fast Poisson Disk Sampling in Arbitrary Dimensions. In _ACM SIGGRAPH 2007 Sketches_ (San Diego, California) _(SIGGRAPH ’07)_. Association for Computing Machinery, New York, NY, USA, 22–es.  https://doi.org/10.1145/1278780.1278807
  * Cai et al. (2020) Hong Cai, Belal Korany, Chitra R. Karanam, and Yasamin Mostofi. 2020.  Teaching RF to Sense without RF Training Measurements.  _Proc. ACM Interact. Mob. Wearable Ubiquitous Technol._ 4, 4, Article 120 (dec 2020), 22 pages.  https://doi.org/10.1145/3432224
  * Cai et al. (2022) Zhongang Cai, Mingyuan Zhang, Jiawei Ren, Chen Wei, Daxuan Ren, Zhengyu Lin, Haiyu Zhao, Lei Yang, Chen Change Loy, and Ziwei Liu. 2022\.  Playing for 3D Human Recovery.  arXiv:2110.07588 [cs.CV]
  * Cao et al. (2020) Zhe Cao, Hang Gao, Karttikeya Mangalam, Qi-Zhi Cai, Minh Vo, and Jitendra Malik. 2020\.  Long-term Human Motion Prediction with Scene Context.  arXiv:2007.03672 [cs.CV]
  * Cao et al. (2019) Zhe Cao, Gines Hidalgo, Tomas Simon, Shih-En Wei, and Yaser Sheikh. 2019.  OpenPose: Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields.  arXiv:1812.08008 [cs.CV]
  * Cao et al. (2017) Zhe Cao, Tomas Simon, Shih-En Wei, and Yaser Sheikh. 2017\.  Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields.  arXiv:1611.08050 [cs.CV]
  * Carrington et al. (2020) Patrick Carrington, Gierad Laput, and Jeffrey P. Bigham. 2020\.  SpokeSense: Developing a Real-Time Sensing Platform for Wheelchair Sports.  _SIGACCESS Access. Comput._ 124, Article 2 (mar 2020), 1 pages.  https://doi.org/10.1145/3386308.3386310
  * Chen and Zwicker (2021) Shuhong Chen and Matthias Zwicker. 2021.  Transfer Learning for Pose Estimation of Illustrated Characters.  arXiv:2108.01819 [cs.CV]
  * Deng et al. (2009) Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009\.  ImageNet: A large-scale hierarchical image database. In _2009 IEEE Conference on Computer Vision and Pattern Recognition_. 248–255.  https://doi.org/10.1109/CVPR.2009.5206848
  * Denninger et al. (2019) Maximilian Denninger, Martin Sundermeyer, Dominik Winkelbauer, Youssef Zidan, Dmitry Olefir, Mohamad Elbadrawy, Ahsan Lodhi, and Harinandan Katam. 2019.  BlenderProc.  arXiv:1911.01911 [cs.CV]
  * Dollár et al. (2011) Piotr Dollár, Christian Wojek, Bernt Schiele, and Pietro Perona. 2011.  Pedestrian Detection: An Evaluation of the State of the Art.  _IEEE transactions on pattern analysis and machine intelligence_ 34 (07 2011), 743–61.  https://doi.org/10.1109/TPAMI.2011.155
  * Ebadi et al. (2022a) Salehe Erfanian Ebadi, Saurav Dhakad, Sanjay Vishwakarma, Chunpu Wang, You-Cyuan Jhang, Maciek Chociej, Adam Crespi, Alex Thaman, and Sujoy Ganguly. 2022a.  PSP-HDRI$+$: A Synthetic Dataset Generator for Pre-Training of Human-Centric Computer Vision Models.  arXiv:2207.05025 [cs]
  * Ebadi et al. (2022b) Salehe Erfanian Ebadi, You-Cyuan Jhang, Alex Zook, Saurav Dhakad, Adam Crespi, Pete Parisi, Steven Borkman, Jonathan Hogins, and Sujoy Ganguly. 2022b.  PeopleSansPeople: A Synthetic Data Generator for Human-Centric Computer Vision.  arXiv:2112.09290 [cs]
  * El Emam et al. (2020) Khaled El Emam, Lucy Mosquera, and Richard Hoptroff. 2020\.  _Practical synthetic data generation: balancing privacy and the broad availability of data_.  O’Reilly Media.
  * Everingham et al. (2010) Mark Everingham, Luc Gool, Christopher K. Williams, John Winn, and Andrew Zisserman. 2010\.  The Pascal Visual Object Classes (VOC) Challenge.  _Int. J. Comput. Vision_ 88, 2 (jun 2010), 303–338.  https://doi.org/10.1007/s11263-009-0275-4
  * Fabbri et al. (2018) Matteo Fabbri, Fabio Lanzi, Simone Calderara, Andrea Palazzi, Roberto Vezzani, and Rita Cucchiara. 2018\.  Learning to Detect and Track Visible and Occluded Body Joints in a Virtual World.  arXiv:1803.08319 [cs.CV]
  * Ferrucci et al. (2016) Luigi Ferrucci, Rachel Cooper, Michelle Shardell, Eleanor M. Simonsick, Jennifer A. Schrack, and Diana Kuh. 2016.  Age-Related Change in Mobility: Perspectives From Life Course Epidemiology and Geroscience.  _The Journals of Gerontology Series A: Biological Sciences and Medical Sciences_ 71, 9 (Sept. 2016), 1184–1194.  https://doi.org/10.1093/gerona/glw043
  * Freiberger et al. (2020) Ellen Freiberger, Cornel Christian Sieber, and Robert Kob. 2020.  Mobility in Older Community-Dwelling Persons: A Narrative Review.  _Frontiers in Physiology_ 11 (2020).
  * Fribourg et al. (2020) Rebecca Fribourg, Ferran Argelaguet, Anatole Lécuyer, and Ludovic Hoyet. 2020.  Avatar and sense of embodiment: Studying the relative preference between appearance, control and point of view.  _IEEE transactions on visualization and computer graphics_ 26, 5 (2020), 2062–2072.
  * Gaidon et al. (2016) Adrien Gaidon, Qiao Wang, Yohann Cabon, and Eleonora Vig. 2016\.  Virtual worlds as proxy for multi-object tracking analysis. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 4340–4349.
  * Gardener et al. (2006) Elizabeth A Gardener, Felicia A Huppert, Jack M Guralnik, and David Melzer. 2006.  Middle-Aged and Mobility-Limited: Prevalence of Disability and Symptom Attributions in a National Survey.  _Journal of General Internal Medicine_ 21, 10 (Oct. 2006), 1091–1096.  https://doi.org/10.1111/j.1525-1497.2006.00564.x
  * Gong et al. (2019) Zhiqiang Gong, Ping Zhong, and Weidong Hu. 2019.  Diversity in Machine Learning.  _IEEE Access_ 7 (2019), 64323–64350.  https://doi.org/10.1109/access.2019.2917620
  * Guo et al. (2020) Anhong Guo, Ece Kamar, Jennifer Wortman Vaughan, Hanna Wallach, and Meredith Ringel Morris. 2020\.  Toward fairness in AI for people with disabilities SBG@ a research roadmap.  _ACM SIGACCESS Accessibility and Computing_ 125 (2020), 1–1.
  * Guo et al. (2022a) Chuan Guo, Shihao Zou, Xinxin Zuo, Sen Wang, Wei Ji, Xingyu Li, and Li Cheng. 2022a.  Generating Diverse and Natural 3D Human Motions From Text. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_. 5152–5161.
  * Guo et al. (2022b) Chuan Guo, Shihao Zou, Xinxin Zuo, Sen Wang, Wei Ji, Xingyu Li, and Li Cheng. 2022b.  Generating Diverse and Natural 3D Human Motions From Text.  (2022).
  * He et al. (2018) Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. 2018.  Mask R-CNN.  arXiv:1703.06870 [cs.CV]
  * He et al. (2016b) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016b.  Deep Residual Learning for Image Recognition. In _2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_. 770–778.  https://doi.org/10.1109/CVPR.2016.90
  * He et al. (2023) Ruifei He, Shuyang Sun, Xin Yu, Chuhui Xue, Wenqing Zhang, Philip Torr, Song Bai, and Xiaojuan Qi. 2023\.  Is synthetic data from generative models ready for image recognition?  arXiv:2210.07574 [cs.CV]
  * He et al. (2016a) Wan He, Daniel Goodkind, and Paul Kowal. 2016a.  International Population Reports.  (2016).
  * Heindl et al. (2020) Christoph Heindl, Lukas Brunner, Sebastian Zambal, and Josef Scharinger. 2020.  BlendTorch: A Real-Time, Adaptive Domain Randomization Library.  arXiv:2010.11696 [cs.CV]
  * Howe et al. (2017) Bill Howe, Julia Stoyanovich, Haoyue Ping, Bernease Herman, and Matt Gee. 2017.  Synthetic data for social good.  _arXiv preprint arXiv:1710.08874_ (2017).
  * Hu et al. (2020) Fang Hu, Peng He, Songlin Xu, Yin Li, and Cheng Zhang. 2020.  FingerTrak: Continuous 3D hand pose tracking by deep learning hand silhouettes captured by miniature thermal cameras on wrist.  _Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies_ 4, 2 (2020), 1–24.
  * Hu et al. (2019) Yuan-Ting Hu, Hong-Shuo Chen, Kexin Hui, Jia-Bin Huang, and Alexander G. Schwing. 2019.  SAIL-VOS: Semantic Amodal Instance Level Video Object Segmentation – A Synthetic Dataset and Baselines. In _2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_. 3100–3110.  https://doi.org/10.1109/CVPR.2019.00322
  * Hu et al. (2021) Yuan-Ting Hu, Jiahong Wang, Raymond A. Yeh, and Alexander G. Schwing. 2021.  SAIL-VOS 3D: A Synthetic Dataset and Baselines for Object Detection and 3D Mesh Reconstruction from Video Data.  arXiv:2105.08612 [cs.CV]
  * Hudak et al. (1996) Pamela L Hudak, Peter C Amadio, Claire Bombardier, Dorcas Beaton, Donald Cole, Aileen Davis, Gillian Hawker, Jeffrey N Katz, Matti Makela, Robert G Marx, et al. 1996\.  Development of an upper extremity outcome measure: the DASH (disabilities of the arm, shoulder, and head).  _American journal of industrial medicine_ 29, 6 (1996), 602–608.
  * Jain et al. (2023) Yash Jain, Hyeokhyen Kwon, and Thomas Ploetz. 2023\.  On the Effectiveness of Virtual IMU Data for Eating Detection with Wrist Sensors. In _Adjunct Proceedings of the 2022 ACM International Joint Conference on Pervasive and Ubiquitous Computing and the 2022 ACM International Symposium on Wearable Computers_ (Cambridge, United Kingdom) _(UbiComp/ISWC ’22 Adjunct)_. Association for Computing Machinery, New York, NY, USA, 50–52.  https://doi.org/10.1145/3544793.3560337
  * Kaloskampis et al. (2019) Ioannis Kaloskampis, David Pugh, Chaitanya Joshi, and Louisa Nolan. 2019.  Data science for the public good.  (2019).
  * Kirkham and Tannert (2021) Reuben Kirkham and Benjamin Tannert. 2021.  Using Computer Simulations to Investigate the Potential Performance of ’A to B’ Routing Systems for People with Mobility Impairments.  arXiv:2107.01570 [cs.HC]
  * Kolve et al. (2022) Eric Kolve, Roozbeh Mottaghi, Winson Han, Eli VanderBilt, Luca Weihs, Alvaro Herrasti, Matt Deitke, Kiana Ehsani, Daniel Gordon, Yuke Zhu, Aniruddha Kembhavi, Abhinav Gupta, and Ali Farhadi. 2022.  AI2-THOR: An Interactive 3D Environment for Visual AI.  arXiv:1712.05474 [cs.CV]
  * Leaman and La (2017) Jesse Leaman and Hung M. La. 2017.  A Comprehensive Review of Smart Wheelchairs: Past, Present and Future.  arXiv:1704.04697 [cs.RO]
  * Leiva et al. (2017) Luis A. Leiva, Daniel Martín-Albo, and Radu-Daniel Vatavu. 2017\.  Synthesizing Stroke Gestures Across User Populations: A Case for Users with Visual Impairments. In _Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems_ (Denver, Colorado, USA) _(CHI ’17)_. Association for Computing Machinery, New York, NY, USA, 4182–4193.  https://doi.org/10.1145/3025453.3025906
  * Leroy et al. (2020) Vincent Leroy, Philippe Weinzaepfel, Romain Brégier, Hadrien Combaluzier, and Grégory Rogez. 2020\.  SMPLy Benchmarking 3D Human Pose Estimation in the Wild.  arXiv:2012.02743 [cs.CV]
  * Li et al. (2021a) Jizhizi Li, Sihan Ma, Jing Zhang, and Dacheng Tao. 2021a.  Privacy-Preserving Portrait Matting.  arXiv:2104.14222 [cs.CV]
  * Li et al. (2022) Jizhizi Li, Jing Zhang, Stephen J Maybank, and Dacheng Tao. 2022\.  Bridging composite and real: towards end-to-end deep image matting.  _International Journal of Computer Vision_ 130, 2 (2022), 246–266.
  * Li et al. (2021c) Jizhizi Li, Jing Zhang, and Dacheng Tao. 2021c.  Deep Automatic Natural Image Matting.  arXiv:2107.07235 [cs.CV]
  * Li et al. (2021b) Zhengqin Li, Ting-Wei Yu, Shen Sang, Sarah Wang, Meng Song, Yuhan Liu, Yu-Ying Yeh, Rui Zhu, Nitesh Gundavarapu, Jia Shi, Sai Bi, Zexiang Xu, Hong-Xing Yu, Kalyan Sunkavalli, Miloš Hašan, Ravi Ramamoorthi, and Manmohan Chandraker. 2021b.  OpenRooms: An End-to-End Open Framework for Photorealistic Indoor Scene Datasets.  arXiv:2007.12868 [cs.CV]
  * Liang et al. (2018) Jacky Liang, Viktor Makoviychuk, Ankur Handa, Nuttapong Chentanez, Miles Macklin, and Dieter Fox. 2018.  GPU-Accelerated Robotic Simulation for Distributed Reinforcement Learning.  arXiv:1810.05762 [cs.RO]
  * Lin et al. (2017) Tsung-Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. 2017\.  Feature Pyramid Networks for Object Detection.  arXiv:1612.03144 [cs.CV]
  * Lin et al. (2015) Tsung-Yi Lin, Michael Maire, Serge Belongie, Lubomir Bourdev, Ross Girshick, James Hays, Pietro Perona, Deva Ramanan, C. Lawrence Zitnick, and Piotr Dollár. 2015.  Microsoft COCO: Common Objects in Context.  arXiv:1405.0312 [cs.CV]
  * Lugaresi et al. (2019) Camillo Lugaresi, Jiuqiang Tang, Hadon Nash, Chris McClanahan, Esha Uboweja, Michael Hays, Fan Zhang, Chuo-Ling Chang, Ming Guang Yong, Juhyun Lee, Wan-Teh Chang, Wei Hua, Manfred Georg, and Matthias Grundmann. 2019\.  MediaPipe: A Framework for Building Perception Pipelines.  arXiv:1906.08172 [cs.DC]
  * Luvizon et al. (2023) Diogo Luvizon, Marc Habermann, Vladislav Golyanik, Adam Kortylewski, and Christian Theobalt. 2023\.  Scene-Aware 3D Multi-Human Motion Capture from a Single Camera.  arXiv:2301.05175 [cs.CV]
  * Ma et al. (2017) Congcong Ma, Wenfeng Li, Raffaele Gravina, and Giancarlo Fortino. 2017.  Posture Detection Based on Smart Cushion for Wheelchair Users.  _Sensors (Basel, Switzerland)_ 17, 4 (March 2017), 719.  https://doi.org/10.3390/s17040719
  * Madan et al. (2021) Spandan Madan, Timothy Henry, Jamell Dozier, Helen Ho, Nishchal Bhandari, Tomotake Sasaki, Frédo Durand, Hanspeter Pfister, and Xavier Boix. 2021.  When and how CNNs generalize to out-of-distribution category-viewpoint combinations.  arXiv:2007.08032 [cs.CV]
  * Marcard et al. (2018) Timo Marcard, Roberto Henschel, Michael Black, Bodo Rosenhahn, and Gerard Pons-Moll. 2018.  _Recovering Accurate 3D Human Pose in the Wild Using IMUs and a Moving Camera: 15th European Conference, Munich, Germany, September 8-14, 2018, Proceedings, Part X_.  614–631.  https://doi.org/10.1007/978-3-030-01249-6_37
  * Morrical et al. (2021) Nathan Morrical, Jonathan Tremblay, Yunzhi Lin, Stephen Tyree, Stan Birchfield, Valerio Pascucci, and Ingo Wald. 2021.  NViSII: A Scriptable Tool for Photorealistic Image Generation.  arXiv:2105.13962 [cs.CV]
  * Moryossef et al. (2020) Amit Moryossef, Ioannis Tsochantaridis, Roee Aharoni, Sarah Ebling, and Srini Narayanan. 2020\.  Real-Time Sign Language Detection using Human Pose Estimation.  arXiv:2008.04637 [cs.CV]
  * Mundt et al. (2022) Marion Mundt, Zachery Born, Molly Goldacre, and Jacqueline Alderson. 2022.  Estimating Ground Reaction Forces from Two-Dimensional Pose Data: A Biomechanics-Based Comparison of AlphaPose, BlazePose, and OpenPose.  _Sensors (Basel, Switzerland)_ 23, 1 (Dec. 2022), 78.  https://doi.org/10.3390/s23010078
  * Olugbade et al. (2022) Temitayo Olugbade, Marta Bieńkiewicz, Giulia Barbareschi, Vincenzo D’Amato, Luca Oneto, Antonio Camurri, Catherine Holloway, Mårten Björkman, Peter Keller, Martin Clayton, Amanda Williams, Nicolas Gold, Cristina Becchio, Benoît Bardy, and Nadia Bianchi-Berthouze. 2022.  Human Movement Datasets: An Interdisciplinary Scoping Review.  _Comput. Surveys_ 55 (05 2022).  https://doi.org/10.1145/3534970
  * Park et al. (2021) Joon Sung Park, Danielle Bragg, Ece Kamar, and Meredith Ringel Morris. 2021.  Designing an online infrastructure for collecting AI data from people with disabilities. In _Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency_. 52–63.
  * Park et al. (2023) Sola Park, Seungjin Yang, and Hyuk-Jae Lee. 2023.  MVDet: multi-view multi-class object detection without ground plane assumption.  _Pattern Analysis and Applications_ 26 (06 2023).  https://doi.org/10.1007/s10044-023-01168-6
  * Patel et al. (2021) Priyanka Patel, Chun-Hao P. Huang, Joachim Tesch, David T. Hoffmann, Shashank Tripathi, and Michael J. Black. 2021.  AGORA: Avatars in Geography Optimized for Regression Analysis.  arXiv:2104.14643 [cs.CV]
  * Pishchulin et al. (2012) Leonid Pishchulin, Arjun Jain, Mykhaylo Andriluka, Thorsten Thormählen, and Bernt Schiele. 2012\.  Articulated people detection and pose estimation: Reshaping the future. In _2012 IEEE Conference on Computer Vision and Pattern Recognition_. IEEE, 3178–3185.
  * Pumarola et al. (2019) Albert Pumarola, Jordi Sanchez, Gary Choi, Alberto Sanfeliu, and Francesc Moreno-Noguer. 2019\.  3DPeople: Modeling the Geometry of Dressed Humans. In _International Conference in Computer Vision (ICCV)_.
  * Purkrábek and Matas (2023) Miroslav Purkrábek and Jiří Matas. 2023.  Improving 2D Human Pose Estimation across Unseen Camera Views with Synthetic Data.  arXiv:2307.06737 [cs.CV]
  * Rey et al. (2019) Vitor Fortes Rey, Peter Hevesi, Onorina Kovalenko, and Paul Lukowicz. 2019.  Let There Be IMU Data: Generating Training Data for Wearable, Motion Sensor Based Activity Recognition from Monocular RGB Videos. In _Adjunct Proceedings of the 2019 ACM International Joint Conference on Pervasive and Ubiquitous Computing and Proceedings of the 2019 ACM International Symposium on Wearable Computers_ (London, United Kingdom) _(UbiComp/ISWC ’19 Adjunct)_. Association for Computing Machinery, New York, NY, USA, 699–708.  https://doi.org/10.1145/3341162.3345590
  * Roberts et al. (2021) Mike Roberts, Jason Ramapuram, Anurag Ranjan, Atulit Kumar, Miguel Angel Bautista, Nathan Paczan, Russ Webb, and Joshua M. Susskind. 2021\.  Hypersim: A Photorealistic Synthetic Dataset for Holistic Indoor Scene Understanding.  arXiv:2011.02523 [cs.CV]
  * Ros et al. (2016) German Ros, Laura Sellart, Joanna Materzynska, David Vazquez, and Antonio M. Lopez. 2016\.  The SYNTHIA Dataset: A Large Collection of Synthetic Images for Semantic Segmentation of Urban Scenes. In _2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_. 3234–3243.  https://doi.org/10.1109/CVPR.2016.352
  * Sapp and Taskar (2013) Ben Sapp and Ben Taskar. 2013.  MODEC: Multimodal Decomposable Models for Human Pose Estimation.  _Proceedings / CVPR, IEEE Computer Society Conference on Computer Vision and Pattern Recognition. IEEE Computer Society Conference on Computer Vision and Pattern Recognition_ , 3674–3681.  https://doi.org/10.1109/CVPR.2013.471
  * Schober et al. (2018) Patrick Schober, Christa Boer, and Lothar A Schwarte. 2018\.  Correlation coefficients: appropriate use and interpretation.  _Anesthesia & analgesia_ 126, 5 (2018), 1763–1768.
  * Schreven et al. (2015) Sander Schreven, Peter J. Beek, and Jeroen B. J. Smeets. 2015\.  Optimising Filtering Parameters for a 3D Motion Analysis System.  _Journal of Electromyography and Kinesiology_ 25, 5 (Oct. 2015), 808–814.  https://doi.org/10.1016/j.jelekin.2015.06.004
  * Shen et al. (2021) Bokui Shen, Fei Xia, Chengshu Li, Roberto Martín-Martín, Linxi Fan, Guanzhi Wang, Claudia Pérez-D’Arpino, Shyamal Buch, Sanjana Srivastava, Lyne P. Tchapmi, Micael E. Tchapmi, Kent Vainio, Josiah Wong, Li Fei-Fei, and Silvio Savarese. 2021.  iGibson 1.0: a Simulation Environment for Interactive Tasks in Large Realistic Scenes.  arXiv:2012.02924 [cs.AI]
  * Snoke et al. (2018) Joshua Snoke, Gillian M Raab, Beata Nowok, Chris Dibben, and Aleksandra Slavkovic. 2018.  General and specific utility measures for synthetic data.  _Journal of the Royal Statistical Society Series A: Statistics in Society_ 181, 3 (2018), 663–688.
  * Sun et al. (2019) Ke Sun, Bin Xiao, Dong Liu, and Jingdong Wang. 2019\.  Deep High-Resolution Representation Learning for Human Pose Estimation.  arXiv:1902.09212 [cs.CV]
  * SUTIL (2015) NICOLÁS SALAZAR SUTIL. 2015\.  _Motion and Representation: The Language of Human Movement_.  The MIT Press.  http://www.jstor.org/stable/j.ctt17kk8zx
  * Tevet et al. (2022) Guy Tevet, Sigal Raab, Brian Gordon, Yonatan Shafir, Daniel Cohen-Or, and Amit H Bermano. 2022\.  Human motion diffusion model.  _arXiv preprint arXiv:2209.14916_ (2022).
  * Tian et al. (2023) Yonglong Tian, Lijie Fan, Phillip Isola, Huiwen Chang, and Dilip Krishnan. 2023.  StableRep: Synthetic Images from Text-to-Image Models Make Strong Visual Representation Learners.  arXiv:2306.00984 [cs.CV]
  * Tobin et al. (2017) Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojciech Zaremba, and Pieter Abbeel. 2017\.  Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World.  arXiv:1703.06907 [cs.RO]
  * Todorov et al. (2012) Emanuel Todorov, Tom Erez, and Yuval Tassa. 2012.  MuJoCo: A physics engine for model-based control. In _2012 IEEE/RSJ International Conference on Intelligent Robots and Systems_. 5026–5033.  https://doi.org/10.1109/IROS.2012.6386109
  * Tremblay et al. (2018) Jonathan Tremblay, Aayush Prakash, David Acuna, Mark Brophy, Varun Jampani, Cem Anil, Thang To, Eric Cameracci, Shaad Boochoon, and Stan Birchfield. 2018\.  Training Deep Networks with Synthetic Data: Bridging the Reality Gap by Domain Randomization.  arXiv:1804.06516 [cs.CV]
  * Trewin (2018) Shari Trewin. 2018\.  AI fairness for people with disabilities: Point of view.  _arXiv preprint arXiv:1811.10670_ (2018).
  * Tzikas (2022) Rigas Tzikas. 2022\.  _How realistic is my synthetic data? A qualitative approach._ Master’s thesis.
  * Unity Technologies (2022) Unity Technologies. 2022\.  Unity SynthHomes: A Synthetic Home Interior Dataset Generator.  https://github.com/Unity-Technologies/SynthHomes.
  * Upadhyaya et al. (2022) Skanda Upadhyaya, Shravan Bhat, Siddhanth P. Rao, V Ashwin, and Krishnan Chemmangat. 2022.  A cost effective eye movement tracker based wheel chair control algorithm for people with paraplegia.  arXiv:2207.10511 [cs.HC]
  * Valtchev and Wu (2021) Svetozar Zarko Valtchev and Jianhong Wu. 2021.  Domain Randomization for Neural Network Classification.  _Journal of Big Data_ 8, 1 (July 2021), 94\.  https://doi.org/10.1186/s40537-021-00455-5
  * Varol et al. (2017a) Gül Varol, Javier Romero, Xavier Martin, Naureen Mahmood, Michael J. Black, Ivan Laptev, and Cordelia Schmid. 2017a.  Learning from Synthetic Humans. In _CVPR_.
  * Varol et al. (2017b) Gul Varol, Javier Romero, Xavier Martin, Naureen Mahmood, Michael J Black, Ivan Laptev, and Cordelia Schmid. 2017b.  Learning from synthetic humans. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 109–117.
  * Vasquez et al. (2017) Andres Vasquez, Marina Kollmitz, Andreas Eitel, and Wolfram Burgard. 2017.  Deep Detection of People and their Mobility Aids for a Hospital Robot.  arXiv:1708.00674 [cs.RO]
  * Vignier et al. (2008) Nicolas Vignier, Jean-François Ravaud, Myriam Winance, François-Xavier Lepoutre, and Isabelle Ville. 2008.  Demographics of wheelchair users in France: Results of National community-based handicaps-incapacités-dépendance surveys.  _Journal of rehabilitation medicine : official journal of the UEMS European Board of Physical and Rehabilitation Medicine_ 40 (04 2008), 231–9.  https://doi.org/10.2340/16501977-0159