1352 Research  Motions in the future one second generated from different
methods on the MoGaze dataset [33] with best prediction boxed in green. Our
method can generate human motions that are more close to the ground truth (GT)
than other methods.

# GazeMoDiff: Gaze-guided Diffusion Model for
Stochastic Human Motion Prediction

Haodong Yan
Xi’an Jiaotong University  e-mail<EMAIL_ADDRESS>Zhiming Hu
University of Stuttgart  e-mail<EMAIL_ADDRESS>Syn Schmitt
University of Stuttgart  e-mail<EMAIL_ADDRESS>Andreas
Bulling
University of Stuttgart e-mail<EMAIL_ADDRESS>
###### Abstract

Human motion prediction is important for virtual reality (VR) applications,
e.g., for realistic avatar animation. Existing methods have synthesised body
motion only from observed past motion, despite the fact that human gaze is
known to correlate strongly with body movements and is readily available in
recent VR headsets. We present GazeMoDiff – a novel gaze-guided denoising
diffusion model to generate stochastic human motions. Our method first uses a
graph attention network to learn the spatio-temporal correlations between eye
gaze and human movements and to fuse them into cross-modal gaze-motion
features. These cross-modal features are injected into a noise prediction
network via a cross-attention mechanism and progressively denoised to generate
realistic human full-body motions. Experimental results on the MoGaze and GIMO
datasets demonstrate that our method outperforms the state-of-the-art methods
by a large margin in terms of average displacement error (15.03% on MoGaze and
9.20% on GIMO). We further conducted an online user study to compare our
method with state-of-the-art methods and the responses from $23$ participants
validate that the motions generated by our method are more realistic than
those from other methods. Taken together, our work makes a first important
step towards gaze-guided stochastic human motion prediction and guides future
work on this important topic in VR research.

Human-centered computingHuman computer interactionInteraction paradigmsVirtual
reality; Computing methodologiesMachine learningMachine learning
approachesNeural networks;

Introduction

Generating realistic human body movements is a key research challenge in the
area of virtual reality (VR) and is the basis for safe, smooth, and immersive
human-environment [12] and human-human interactions [29, 22]. Human motion
prediction (HMP) enables a number of exciting applications, such as redirected
walking to create the illusion of unlimited virtual interaction spaces [3, 53]
or to steer users away from physical boundaries, such as walls, and thus avoid
collisions [17, 66]. Human motion prediction can also provide users with a
low-latency experience by preparing VR content in advance based on the
predicted future human poses [23, 9] and it has been used to generate human-
like motions for virtual agents to enhance the interaction experience [6, 14].

Previous work on HMP has typically generated human motions in a deterministic
way, i.e. by producing only a single prediction at a time [41, 7, 8, 19].
Recently, in light of the fact that human motion is stochastic by nature [10,
4], researchers have turned to stochastic human motion prediction, i.e.
generating a number of reasonable human motions at a time [10, 62, 4, 5].
Stochastic human motion prediction suits the needs of many VR applications.
For example, to minimise the collision risk in a virtual environment, it is
necessary to consider multiple possible future trajectories to warn the users.
To produce realistic virtual agents, it is also beneficial to synthesise
diverse human motions that users can peruse and select from according to their
personal preferences. However, existing stochastic HMP methods typically
generate human motions using only past observed motions and neglect other
modalities, in particular human eye gaze. With rapid advances in eye tracking
technology, human eye gaze information has become readily available in many VR
head mounted displays (HMDs), such as HTC Vive Pro Eye, Varjo VR-3, and Vision
Pro, and has demonstrated its potential for gaze-based interaction [50] and
gaze-contingent rendering [27, 26] in VR. In addition, a large body of work in
the cognitive sciences and human-centred computing has shown that human body
movements are closely coordinated with human gaze behaviour [24, 25, 49].
Despite this close coordination, information on eye gaze has not been used for
stochastic human motion prediction so far.

To address this limitation we propose GazeMoDiff111 Code and pretrained models
will be released upon acceptance – the first Gaze-guided human Motion
Diffusion model to generate realistic and diverse human motions. At the core
of our method is a spatio-temporal graph attention network (ST-GAT) that
models the correlations between eye gaze and different body joints across each
time step and fuses them into cross-modal gaze-motion features. The gaze-
motion features are then injected into a noise prediction network via a cross-
attention mechanism to generate realistic and diverse human motions through
progressive denoising. We report extensive evaluations of our method using
common metrics for stochastic HMP and visualisation analysis on two public
datasets: MoGaze [33] and GIMO[66]. We show that our method significantly
outperforms state-of-the-art methods by 15.03% on MoGaze and 9.20% on GIMO in
terms of average displacement error, and 15.24% on MoGaze and 8.92% on GIMO in
final displacement error. To evaluate the realism of the motions generated by
our method, we further perform an on-line user study to compare the
performances of different methods by human intuition. The 414 responses from
23 participants demonstrate that our method can generate motions that are more
realistic than the state of the art.

The main contributions of our work are three-fold:

  * •

We propose a novel gaze-guided diffusion model for stochastic HMP: Our model
first extracts cross-modal gaze-motion features using a ST-GAT, and then
injects these features into a noise prediction network via a cross-attention
mechanism, and finally generates diverse human motions through progressive
denoising.

  * •

We provide extensive evaluations of our method using common metrics for
stochastic HMP and visualisation analysis on two publicly available benchmarks
and show that our method significantly outperforms the state-of-the-art
methods.

  * •

We perform an on-line user study for subjective assessment and the responses
from 23 participants validate that the motions generated by our method are
more realistic than that from other methods.

##  1 Related Work

###  1.1 Human Motion Prediction

Human motion prediction is a fundamental research topic in virtual reality
(VR). Early studies commonly considered HMP as a deterministic sequence
prediction task, tackling it with recurrent neural networks (RNN)[41, 18, 37,
16], graph neural networks (GNN)[36, 35] and Transformers[1, 42]. Noticing
that human motion trends are inherently stochastic, recent works began to
predict human motions in a stochastic way using generative models such as
variational autoencoder (VAE) [2, 39], generative adversarial network
(GAN)[34, 5, 28], and Flow networks[62]. Generative approaches model motion
possible distribution rather than a single trajectory. It meets the demands of
many VR applications such as collision warning during human-human interaction
and human-environment interaction. These methods often design diversity-aware
loss or sampling[39, 62] to enforce diversity. However, the predicted
sequences were not plausible. Recently, the emergence of the denoising
diffusion model[20], which has shown both diversity and realism in
generalisation in the fields of image generation[45, 43, 64] without any
explicit diversity constraints. Several works [48, 4, 10] have applied
diffusion in the field of stochastic motion prediction and have achieved more
realistic results. However, those methods only generate predictions based on
observed poses and neglect other potentially exploitable information in VR
scenarios from other modalities. To fill this gap, our work introduces a
naturally available modal, gaze, into a diffusion-based model for the first
time.

###  1.2 Correlations between Eye Gaze and Body Motions

Intuitively, a period of gaze information seems related to intention[31],
which can drive future motion trends. Extensive work in cognitive science and
the human-computer interaction has also demonstrated this correlation between
gaze and subsequent motion.Some research [63, 27, 32] have shown that in many
everyday activities, such as free viewing or object searching, head movements
are closely associated with eye movements. Emery et al.[15] focused on how
eye, hand, and head movements coordinate in a virtual world, and then
leveraged that coordination to enhance gaze estimation. Sidenmark et al.[51,
49] also have revealed the coordination of eye, head and torso movements
during gaze shifts in VR.

Although the aforementioned works have indicated the strong relationship
between eye movements and human motion trends, existing stochastic HMP methods
have not considered this correlation yet. We are the first to leverage the
correlation between gaze and motion, which enhances the realism and
trustworthiness of stochastic HMP. This gaze-motion correlation ultimately
leads to superior quantitative performance, as we show in this work.

###  1.3 Denoising Diffusion Models

Denoising Diffusion Models, or more precisely, Denoising Diffusion
Probabilistic Models (DDPM)[20, 52, 13, 47] are a group of the most ingenious
generative models. They aim to model reversing a Markov chain of the diffusion
process illustrated in Figure 1. During training, noisy samples are obtained
by incrementally adding noise to raw samples. The DDPM model then
progressively reverses the diffusion process by predicting noise and denoising
the samples. The loss is computed as the difference between predicted noise
and Gaussian noise added during the diffusion process. In the inference stage,
given a well-trained DDPM, it can generate diverse samples from a Gaussian
distribution via the reversed diffusion process.

Figure 1: Diffusion process and reversed process in DDPM.

Owing to the power of generating diverse and high-quality samples, these
models have been applied in image/video generation[47, 21], anomaly
detection[60], objection detection[11], 3D reconstruction[61], time series
forecasting[46] and imputation[54]. In a similar context, Tevet et al.[55]
proposed a text-driven human motion synthesis method with diffusion models.
Recently, Barquero et al. [4] utilised a latent diffusion model to sample the
diverse behaviour code to predict stochastic motions. Chen et al.[10]
presented an end-to-end motion prediction framework based on diffusion models
without complicated loss constraints and training processes. Their work has
demonstrated competitive performance in this domain. Although these diffusion-
based methods have made significant contributions to stochastic HMP,
unrealistic results can be yielded by only focusing on past human poses and
neglecting other important information from other modalities. To address these
problems, our gaze-guided diffusion model takes into account the coordination
between gaze and human movements. By incorporating gaze information, we aim to
generate more reasonable and plausible human motion predictions. This
integration of gaze and human movement data allows for a more holistic
understanding of human behaviour, resulting in improved motion synthesis that
better aligns with intentions.

##  2 The GazeMoDiff Model

Our diffusion model for gaze-guided stochastic human motion forecasting takes
past body poses and gaze information as input. We encode the human pose
$p\in\mathbb{R}^{j\times 3}$ using the 3D positions of all human joints, where
$j$ denotes the number of human joints. Gaze $g_{i}\in\mathbb{R}^{1\times 3}$
is defined as the unit direction vector. Given a $H$-frame observed sequence
$\mathbf{x}=[(g_{1},p_{1}),(g_{2},p_{2}),...,(g_{H},p_{H})]$, our goal is to
train a model that can predict $k$ different following $F$-frame future human
motions. It can be formulated as $\mathbf{P}=f_{p}(\mathbf{x},k)$, where
$\mathbf{p}=\\{\mathbf{p}_{0},\mathbf{p}_{1},...,\mathbf{p}_{k}\\}$ and
$\mathbf{p}_{i}=[p_{H+1},p_{H+2},...,p_{H+F}]$. An overview of our model is
shown in Figure 2. At a high level, our model consists of a cross-modal fusion
module to fuse human gaze and full body poses as well as a motion generation
module.

Figure 2: Overview of the proposed method GazeMoDiff. GazeMoDiff first uses a
spatio-temporal graph attention network to extract and fuse features from eye
gaze and human motions, then injects these features into a noise prediction
module via a cross-attention mechanism, and finally generates diverse human
motions through progressive denoising.

###  2.1 Gaze-motion Fusion

Recent works have demonstrated that a graph structure is a particularly
effective representation to learn abstract correlations between body joints
[36, 35, 38]. We treated gaze as a “virtual joint” and combined gaze and poses
together as a fully connected spatial-temporal graph
$X\in\mathbb{R}^{3\times(j+1)\times(H+F)}$, where later $F$-frame poses were
padded with the last observed frame.

We further employed an approximate discrete cosine transform (DCT) [39] that
selected the first $L$ components. By leveraging this transformation, the time
dimension was reduced to $L$ from $H+F$, which both improved the smoothness of
generation and reduced computational complexity[40] Given the original
spatial-temporal graph $X\in\mathbb{R}^{3\times(j+1)\times(H+F)}$, the DCT
spatial-temporal graph $Y\in\mathbb{R}^{3\times(j+1)\times L}$ was obtained by
DCT.

Previous work on graph attention networks (GATs) has shown good results for
human pose aggregation [57]. We designed the Gaze-motion Fusion module
${GazePoseFuse}\left(\cdot\right)$, illustrated in Figure 2, to model the
spatio-temporal correlations in the defined graph and fuse the gaze-motion
features. The Gaze-motion Fusion module was composed of a stack of temporal
GAT layers and spatial attention GAT layers with skip connections. In this
module, the number of middle blocks $m$ is 4 and there is a layer
normalisation, tanh activation function and a dropout layer after each middle
block. The attention head of each spatial GAT layer and temporal GAT layer is
8. The output feature dimension of the start, middle, and end block is 16, 16,
and 3, respectively. The gaze-motion feature is calculated as:

$H^{\prime\prime}=GazePoseFuse\left(Y\right)$ (1)

Given the input $H\in\mathbb{R}^{f\times(j+1)\times L}$ of each block, where
$f$ is the feature dimension of input, we first aggregated features along the
temporal dimension and obtain
$H^{\prime}=[{h}_{1}^{\prime},{h}_{2}^{\prime},...,{h}_{L}^{\prime}]\in\mathbb{R}^{f\times(j+1)\times
L}$ in the following way:

${h}_{i}^{\prime}=\operatorname{LeakyReLU}\left(\frac{1}{N_{head}}\sum_{n=1}^{N_{head}}\sum_{k=1}^{L}\alpha_{ik}^{n}{h}_{k}\right)$
(2)

where ${h}_{i}^{\prime}\in\mathbb{R}^{f\times(j+1)}$ is the output feature of
node $i$, ${h}_{k}$ is the input feature of node $k$, and $N_{head}$ denotes
the number of heads for attention. We fused different output features from
each head by averaging them. For each head, the attention matrix
$\alpha_{ik}^{n}$ represents interactions between each timestamp, calculated
as follows:

$\alpha_{ik}^{n}=\frac{\exp\left(\operatorname{LeakyReLU}\left({\mathbf{a}^{n}}\left[{h}_{i}\oplus{h}_{k}\right]\right)\right)}{\sum_{l=1}^{T}\exp\left(\operatorname{LeakyReLU}\left({\mathbf{a}^{n}}\left[{h}_{i}\oplus{h}_{l}\right]\right)\right)}$
(3)

where ${\mathbf{a}^{n}}$ is a parameter vector $\in\mathbb{R}^{2f(j+1)\times
1}$

A weigh matrix $\mathbf{W}\in\mathbb{R}^{f\times f^{\prime}}$ was then used to
transfer feature dimension from $H^{\prime}\in\mathbb{R}^{f\times(j+1)\times
L}$ to $\bar{H}^{\prime}\in\mathbb{R}^{f^{\prime}\times(j+1)\times L}$.

To further capture correlations between joints and the gaze feature, we
proposed a spatial graph attention network layer. Specifically, the spatial
GAT layer was in a similar manner to the temporal GAT layer, differing
primarily in how attention coefficients were computed. Rather than aggregating
along the temporal axis as in the temporal GAT, the spatial GAT aggregated
features across spatial dimensions as follows:

${h}_{i}^{\prime\prime}=\operatorname{LeakyReLU}\left(\frac{1}{\bar{N}_{head}}\sum_{n=1}^{\bar{N}_{head}}\sum_{k=1}^{j+1}\bar{\alpha}_{ik}^{n}\bar{h}_{k}^{\prime}\right)$
(4)

where ${h}_{i}^{\prime\prime}\in\mathbb{R}^{f^{\prime}\times L}$ is the output
feature of node $i$ and $\bar{N}_{head}$ denotes the number of heads for
attention. We also applied an average function to fuse different output
features from each head in spatial GAT. For each head, the attention matrix
$\bar{\alpha}_{ik}^{n}$ represents interactions between each timestamp,
calculated as follows:

$\bar{\alpha}_{ik}^{n}=\frac{\exp\left(\operatorname{LeakyReLU}\left({\bar{\mathbf{a}}^{n}}\left[\bar{h}_{i}^{\prime}\oplus\bar{h}_{k}^{\prime}\right]\right)\right)}{\sum_{l=1}^{j+1}\exp\left(\operatorname{LeakyReLU}\left({\bar{\mathbf{a}}^{n}}\left[\bar{h}_{i}^{\prime}\oplus\bar{h}_{l}^{\prime}\right]\right)\right)}$
(5)

where $\bar{\mathbf{a}}^{n}$ is a parameter vector
$\in\mathbb{R}^{2f^{\prime}L\times 1}$

###  2.2 Motion Generation

We formulated the motion generation task as an inverse diffusion process,
which contains iterative noise prediction and denoising steps.

Noise prediction. Inspired by text-driven noise prediction network[65], we
designed a noise prediction network ${\boldsymbol{\epsilon}_{\theta}}$. This
network is developed to predict the noise at the current time step given the
gaze-motion features $H^{\prime\prime}$ and noisy motion sequence
$Y^{t}\in\mathbb{R}^{3\times(j+1)\times L}$ from the previous time step. As
illustrated in Figure 2, ${\boldsymbol{\epsilon}_{\theta}}$ consists of $n$
stacked self-attention block, cross-attention block and multilayer
perception(MLP) block with skip connections. The predicted noise at time step
$t$ is formulated as follows:

$\bar{\boldsymbol{\epsilon}}=\boldsymbol{\epsilon}_{\boldsymbol{\theta}}\left(Y^{t},H^{\prime\prime},t\right)$
(6)

To capture more complex representations and better model the temporal
correlations within the noisy gaze-motion sequence
$Y^{t}\in\mathbb{R}^{3\times(j+1)\times L}$, we first employed a linear
transformation to project the sequence into a higher-dimension latent space
$Y^{t}\in\mathbb{R}^{3\times(j+1)\times L^{\prime}}$. We then apply an
efficient self-attention block[65] to further model temporal correlations
between each frame:

$\mathbf{Y}=\operatorname{Dropout}\left(\operatorname{softmax}\left(\mathbf{Q}\right)\operatorname{softmax}\left(\mathbf{K}^{\top}\right)\right)\operatorname{LN}\left(\mathbf{V}\right)+Y^{t}$
(7)

where $\operatorname{LN}$ is layer normalisation,
$\mathbf{Q}\in\mathbb{R}^{L\times d}$, $\mathbf{K}\in\mathbb{R}^{L\times d}$,
and $\mathbf{V}\in\mathbb{R}^{L\times d}$ are calculated using the original
self-attention mechanism[56]:

$\mathbf{Q}=\mathbf{W}_{q}Y^{t},\mathbf{K}=\mathbf{W}_{k}Y^{t},\mathbf{V}=\mathbf{W}_{v}Y^{t}$
(8)

where $\mathbf{W}_{q}$, $\mathbf{W}_{k}$ and $\mathbf{W}_{v}$ are learnable
parameter matrices.

We then applied a step hint module to inform about how many steps of noise
have been added thus far. We first obtained the timestep embedding
$\mathbf{e}_{t}$ using position embedding[56]. The gaze-motion historical
features $H^{\prime\prime}$ were also fused with a learned linear projection
of the timestep embedding,
$\mathbf{e}=\mathbf{e}_{t}+\mathbf{W}^{\prime}H^{\prime\prime}$. This fused
embedding $\mathbf{e}$ was then injected into the output of the self-attention
block using a feature-wise linear modulation:

$w=\phi_{w}\left(\psi\left(\mathbf{e}_{t}\right)\right),\
b=\phi_{b}\left(\psi\left(\mathbf{e}_{t}\right)\right),\
\mathbf{Y}^{\prime}=\operatorname{SiLU}\left(w\cdot\mathbf{Y}+b\right)$ (9)

where $\left(\cdot\right)$ denotes element-wise multiplication, $\phi_{w}$ and
$\phi_{b}$ are linear projections, and $\psi$ is a single layer MLP with SiLU
activation function. This modulation allowed the step hint embedding to
influence the self-attention features. We also applied this block after each
cross-attention and MLP block, enabling the timestep embedding to provide
hints throughout the network.

To incorporate the historical gaze-motion features $H^{\prime\prime}$ and
leverage their impact on noise prediction, we utilised cross-attention blocks.
These blocks enable a deeper exploration of how the historical features
influence the noise predictions at different denoising steps. In addition,
computing the contributions of different attention heads in parallel better
integrates information from both modalities (gaze and motion) from multiple
perspectives. Potentially, the noisy motions can also provide feedback to
update $H^{\prime\prime}$, leading to a collaborative learning process. In the
cross-attention block, $\mathbf{Q}_{c}$ and $\mathbf{V}_{c}$ were calculated
by gaze-motion historical features $H^{\prime\prime}$ while $\mathbf{K}_{c}$
was calculated by the output of self-attention block $\mathbf{Y}^{\prime}$.
The output of the cross-attention block was calculated as follows:

$\mathbf{Y}_{c}=\operatorname{Dropout}\left(\operatorname{softmax}\left(\mathbf{Q}_{c}\right)\operatorname{softmax}\left(\mathbf{K}_{c}^{\top}\right)\right)\operatorname{LN}\left(\mathbf{V}_{c}\right)+\mathbf{Y}^{\prime}$
(10)
$\mathbf{Q}_{c}=\mathbf{W}^{\prime}_{q}H^{\prime\prime},\mathbf{K}_{c}=\mathbf{W}^{\prime}_{k}\mathbf{Y}^{\prime},\mathbf{V}_{c}=\mathbf{W}^{\prime}_{v}H^{\prime\prime}$
(11)

Finally, we employed a two-layer MLP with GELU activation function and dropout
to further extract features. An output projection layer was added at the end
to align with the noise dimension.

In our implementation, the noise prediction network contained 4 self-attention
blocks, 4 cross-attention blocks, and 4 MLP blocks. Each attention operation
used 8 attention heads. The timestep embedding dimension and the latent space
dimension were both set to 512.

Denoising. In a vanilla approach, the denoised sequence was generated directly
from the Gaussian noise input. However, due to the accumulation of prediction
noise errors, the information of observation in the padded sequence is far
away from the truth in the latter steps. In each denoising step, the observed
sequence was also available to guide the generation in the original space.
Thus, we employed an ingenious prediction mask mechanism[10]:

${Y}^{t-1}=\operatorname{DCT}\left(\mathbf{M}\odot\operatorname{iDCT}\left(Y^{t-1}_{orig}\right)+\mathbf{\left(1-M\right)}\odot\operatorname{iDCT}\left(Y^{t-1}_{pred}\right)\right)$
(12)

where $\mathbf{M}=[\underbrace{1,1,\ldots,1}_{H},\underbrace{0,0,\ldots
0}_{F}]^{\top}\in\mathrm{R}^{\left(H+F\right)\times 1}$ is a mask vector
indicating which frames are observed. $Y^{t-1}_{orig}$ is obtained by adding
$t-1$ iterations of Gaussian noise, and $Y^{t-1}_{pred}$ is obtained by
denoising the output ${Y}^{t}$ from the previous iteration:

$Y^{t-1}_{orig}=\sqrt{\bar{\beta}_{t-1}}Y+\sqrt{1-\bar{\beta}_{t-1}}\boldsymbol{\epsilon},\
\bar{\beta}_{t}=\prod_{i=1}^{t}\beta_{i},\beta_{i}\in[0,1]$ (13)

${Y}^{t-1}_{pred}=\frac{1}{\sqrt{\beta_{t}}}\left({Y}^{t}-\frac{1-\beta_{t}}{\sqrt{1-\bar{\beta}_{t}}}\boldsymbol{\epsilon}_{\boldsymbol{\theta}}\left({Y}^{t},H^{\prime\prime},t\right)\right)+\left(1-\beta\right)\boldsymbol{\epsilon}$
(14)

where $t$ denotes the $t$-th noise iteration, $\beta_{i}$ controls noise
level, and $\boldsymbol{\epsilon}\sim N(0,I)$. At the start, $Y^{t}$ is
sampled from Gaussian noise.

Through this iteration process, we can obtain the full generated sequence
$\mathbf{\bar{p}}=\operatorname{iDCT}(Y^{0})$. The last $H$ frames were
predicted motions.

The detailed inference procedure is illustrated in the supplementary material.

###  2.3 Training

In the training stage, the observation and prediction sequences are both
available. Thus, we trained the model on the full-motion sequence
$X_{full}\in\mathbb{R}^{3\times(j+1)\times(H+F)}$ without pre-padding. First,
we also transfer into the DCT space $Y_{full}\in\mathbb{R}^{3\times(j+1)\times
L}$. We then add noise to $Y_{full}$ to generate a noisy sequence
$Y_{full}^{t}$ via Equation 13. Then we predict the noise through Equation 1
and Equation 6

We then optimised all parameters in our pipeline by minimising the $l_{2}$
loss between the predicted noise
$\boldsymbol{\epsilon}_{\boldsymbol{\theta}}\left(Y^{t},H^{\prime\prime},t\right)$
and true noise $\epsilon$:

$\mathcal{L}=\mathbb{E}_{\boldsymbol{\epsilon},t}\left[\left\|\boldsymbol{\epsilon}-\boldsymbol{\epsilon}_{\boldsymbol{\theta}}\left(Y_{full}^{t},H^{\prime\prime},t\right)\right\|^{2}\right]$
(15)

The detailed training procedure is shown in the supplementary material.

##  3 Experiments

###  3.1 Datasets

Only a limited number of datasets contain synchronised recordings of both eye
gaze and full-body human motion. We evaluated our method on two such public
datasets: MoGaze[33] and GIMO[66].

MoGaze. The MoGaze dataset provides motion capture and eye-tracking data
recorded simultaneously from 6 participants performing $place$ and $pick$
actions. It contains over 3 hours of movement and gaze recordings captured at
30 Hz. The pose of the human body is expressed using 3D coordinates and Euler
angles of 21 joints, while the gaze is represented as a direction vector.
Following prior work[10, 4, 62] on stochastic HMP, we represented the pose
using only the 3D coordinates of each joint. The gaze vector was treated
similarly to a virtual joint, with coordinates computed as the vector sum of
the head joint coordinates and the gaze direction vector. This unified
representation of both pose and gaze solely in joint coordinate space allowed
us to apply existing stochastic models to compare easily. Without special
notes, all $baselines+gaze$ donate to this implementation. Pose sequences from
$p1$, $p2$, $p4$, $p5$, and $p6$ were used for training while that from $p7$
was used for testing.

GIMO. The GIMO dataset contains motion and gaze data captured from 11
participants. The action categories include ${sitting}$ or ${laying}$ ${on}$
${objects}$, ${touching}$, ${holding}$, ${reaching}$ ${to}$ ${objects}$,
$opening$, $pushing$, $transferring$, $throwing$, $swapping$ $objects$, etc.
The skeletal model consists of 23 joints. As with MoGaze, we represented the
pose using only the 3D coordinates of each GIMO skeleton joint, and the gaze
was represented as a direction vector. To enable training and evaluation, we
followed the official dataset splits in GIMO[66]. The training set comprises
motion and gaze recordings from 12 scenes, while the test set contains data
from 14 scenes, including 12 known environments and 2 new, unseen
environments. This evaluation protocol allows the assessment of generalisation
capabilities to novel scenes.

###  3.2 Evaluation Metrics

To evaluate the performance of our model, we employed five commonly used
metrics following prior stochastic HMP works[62, 4, 10].

Average Pairwise Distance (APD) measures diversity as the average L2 distance
between all motion examples generated by the model. Higher APD indicates more
diverse predictions.

Average Displacement Error (ADE) and Final Displacement Error (FDE) measure
accuracy. ADE is the average L2 distance between ground truth and predicted
motions over the whole future sequence. FDE is the L2 distance at the final
timestep. Lower ADE and FDE indicate more accurate predictions.

Multi-Modal Average Displacement Error (MMADE) and Multi-Modal Final
Displacement Error (MMFDE) are are metrics specifically designed to handle the
multi-modal nature of predictions in stochastic HMP tasks. These metrics take
into account the fact that there can be multiple plausible ground truth
sequences for a given input. The multiple plausible ground truth sequences are
obtained by clustering similar past motions. Lower MMADE and MMFDE indicate a
better ability to generate multi-modal predictions.

###  3.3 Baselines

We compared our approach with the following state-of-the-art methods in
stochastic HMP.

  * •

DLow[62]: DLow is a latent flow-based model to generate diverse motion via
diversity-promoting sampling and loss.

  * •

CVAE[62]: CVAE is a conditional variational autoencoder utilised in DLow[62]
as a pre-trained generative model which can also forecast stochastic motions.

  * •

BeLFusion[4]: BeLFusion is a latent diffusion-based model to predict diverse
motion benefited from disentangling the behavioural representation from past
poses and motions.

  * •

HumanMac[10]: HumanMac is an end-to-end single-loss stochastic HMP model with
a DCT Completion fashion in the inference.

###  3.4 Implementation Details

We set the observation and prediction time window to 0.5 seconds (15 frames)
and 2 seconds (60 frames) for both the MoGaze and GIMO datasets, following
common practices in stochastic human motion prediction tasks.[10, 4, 62] As
baselines are not evaluated on MoGaze and GIMO, We need to train them from
scratch. For a fair comparison, all training hyperparameters and settings were
kept at their default values. All experiments were conducted in an Nvidia
TITAN X GPU with 12GB memory using the PyTorch 1.7.1 framework. For both
datasets, we applied the same training and hyperparameter settings, we jointly
trained the gaze-motion fusion network and the noise prediction network using
Adam optimization[30] with an initial learning rate of 0.0003 and the multi-
step learning rate scheduler ($\gamma=0.9$). The batch size was 32. We applied
a standard diffusion process as proposed by [52] that degraded 1000 steps in
the training and sampled 100 steps in the inference. We selected the Cosine
scheduler[44] in the diffusion following HumanMAC[10]. The multi-modal ground
truth threshold was set as 0.4 referred to previous work[4].

###  3.5 Quantitative Results

Table 1: Comparison of GazeMoDiff with the state-of-the-art methods on both MoGaze and GIMO. The best results are in bold and the second best results are underlined. |  Results on MoGaze [33] |  Results on GIMO [66]
---|---|---
|  APD $\uparrow$ |  ADE $\downarrow$ |  FDE $\downarrow$ |  MMADE $\downarrow$ |  MMFDE $\downarrow$ |  APD $\uparrow$ |  ADE $\downarrow$ |  FDE $\downarrow$ |  MMADE $\downarrow$ |  MMFDE $\downarrow$
CVAE[62] | 17.900 | 1.070 | 1.644 | 1.091 | 1.667 | 20.297 | 1.292 | 2.059 | 1.294 | 2.054
DLow[62] | 22.277 | 0.807 | 1.261 | 0.840 | 1.274 | 24.538 | 1.084 | 1.688 | 1.090 | 1.687
BeLFusion[4] | 17.099 | 0.899 | 1.306 | 0.908 | 1.313 | 17.959 | 0.840 | 1.220 | 0.845 | 1.221
HumanMAC [10] | 11.867 | 0.732 | 1.089 | 0.779 | 1.144 | 15.040 | 0.815 | 1.121 | 0.821 | 1.118
Ours | 14.037 | 0.622 | 0.923 | 0.632 | 0.930 | 14.907 | 0.749 | 1.021 | 0.754 | 1.026

The quantitative results on MoGaze[33] and GIMO[66] are shown in Table 1,
Overall, our model outperforms state-of-the-art methods on all accuracy
metrics.

Results on MoGaze. As shown in Table 1, GazeMoDiff achieves the lowest ADE,
FDE, MMADE and MMFDE compared to all other methods, substantially surpassing
the best state-of-the-art HumanMAC. For average displacement error(ADE), our
method achieved an improvement of 15.03% (0.622 $vs.$ 0.732) over
HumanMAC[10]. On final displacement error (FDE), we obtain a 15.24%
performance gain (0.923 $vs.$ 1.089). These significant improvements
demonstrate the value of jointly modeling historical gaze and joint movements
for guiding human motion prediction. On multi-modal version accuracy metrics,
our method improves the MMADE performance by 18.87% (0.632 $vs.$ 0.779) and
the MMFDE performance by 18.81% (0.930 $vs.$ 1.144). The greater gains on the
multi-modal metrics reveal our model is more capable of producing multimodal
predictions aligned with the stochasticity of human nature. A Wilcoxon signed-
rank test[59] indicates a statistically significant ADE difference ($p<0.01$)
between our proposed method and HumanMAC. Although diversity reduces due to
gaze-guided predictions, compared to DLow with the highest APD, our method
outperforms by 22.92%, 26.80%, 30.40% and 27.00% in terms of ADE, FDE, MMADE
and MMFDE. It implies the predictions of DLow are not plausible so the
diversity is meaningless somehow.

Results on GIMO. As shown in Table 1, our method GazeMoDiff is still superior
among the state -of-the-art methods in all accuracy metrics. Our model lowers
ADE by 9.20% compared to HumanMAC (0.815 to 0.749) and reduces FDE by 8.92%
(1.212 to 1.021). On multi-modal accuracy, we attain 8.16% and 8.23%
improvements in MMADE and MMFDE respectively. This demonstrates the strong
generalisation ability of our approach to a new dataset. A Wilcoxon signed-
rank test again showed a significant ADE difference ($p<0.01$) versus
HumanMAC. DLow also shows the highest APD values but particularly low accuracy
on this dataset. Our method is superior to DLow by 30.9%, 39.51%, 30.83% and
39.18% in terms of ADE, FDE, MMADE and MMFDE, respectively. Additionally,
previous work[55] pointed out that methods with high diversity metrics but low
accuracy may suffer failure predictions. We will further discuss this
phenomenon in subsection 3.6.

###  3.6 Visualisation Analysis

We visualised the predicted pose in the future one second. We compared our
method to the best diversity baseline, DLow[62], and the best accuracy
approach, HumanMAC[10]. For each method, we randomly generate 10 predictions
for comparison.

Visualisation results on the MoGaze dataset. We illustrated a representative
visualisation from the MoGaze dataset [66] in GazeMoDiff: Gaze-guided
Diffusion Model for Stochastic Human Motion Prediction. The observed motion
sequence involves a turn to the right, and the ground truth pose shows a
second later, this person continues the turn by about 90 degrees further to
the right. We can observe that predictions of HumanMAC[10] and DLow[62]
generally continue walking in the same forward direction as the observed last
frame, failing to anticipate the full turning trajectory. The best predictions
from baselines labelled in green also differ from the ground truth obviously.
In contrast, by incorporating gaze guidance, our method is able to recognise
that the person intends to keep turning right. As a result, the best
prediction from our method is precisely facing the true direction of motion in
the ground truth and other predictions generally align with the ground truth.
Due to the inherent uncertainty in predicting future speed, the body
orientations of our predictions differ slightly from one another, reflecting
the range of plausible orientation angles given the observed turning motion.
In addition, generated poses are plausible without any angle distortions or
strange limb lengths.

Visualisation results on the GIMO dataset. We also illustrated a typical
visualisation from the GIMO dataset [66]. As depicted in Figure 3, predictions
from our methods are more similar to the ground truth. Furthermore, it is
evident that all predictions from our method are generally realistic. In
contrast, other methods yield predictions that include some implausible
outcomes boxed in red. Corresponding with prior findings[10], we argued that
some baselines like DLow achieved high APD but suffered from implausible
cases. Despite the high APD calculated for these implausible prediction
samples, they are meaningless and misleading for downstream tasks in fact. In
contrast, predictions from our approach look natural with diversity.

Figure 3: Motions in the future one second generated from different methods on
the GIMO dataset [66] with best prediction boxed in green and implausible
cases marked in red. Our method can generate realistic motions while other
methods produce some implausible predictions.

###  3.7 Ablation Study

We conducted an ablation study to comprehensively analyse the individual
contributions of various modules within our framework and evaluate the
specific impact of incorporating gaze information.

We thoroughly assessed the performance of our model by comparing it to
variants that exclude specific modules:

  * •

$Ours_{w/o\ gat}$ : Without spatio-temporal attention graph network

  * •

$Ours_{w/o\ CA}$ : Without cross attention in the noise predicting network

  * •

$Ours_{w/o\ gaze}$ : Without gaze as a virtual joint

Furthermore, we also compared our method to two diffusion baselines that
incorporate gaze guidance.

  * •

$BeLFusion_{g}$: We introduce gaze into BeLFusion by regarding gaze as a
virtual joint as described in subsection 3.1.

  * •

$HumanMAC_{g}$: We introduce gaze into HumanMac in the same way.

As we report in Table 2, our model outperforms all variants and baseline
methods with gaze. It reveals that each module integrated into our framework
significantly contributes to the generation of accurate predictions. There are
three notable findings worth discussing. 1) Compared to GazeMoDiff, the
variant without gaze presents obviously lower accuracy in terms of both ADE
and FDE, which demonstrates gaze information is effective. 2) Despite the
variant without gaze not surpassing the performance GazeMoDiff, it notably
outperforms other baselines without gaze, as presented in Table 1. This
finding underscores the superiority of our proposed framework, whether or not
gaze information is involved. 3) Our approach outperforms both BeLFusion and
HumanMAC which incorporate gaze information in terms of all accuracy metrics,
showing the effective utilisation of gaze information to enhance overall
performance.

Table 2: Ablation study on MoGaze[33]. The best results are in bold.

|  APD $\uparrow$ |  ADE $\downarrow$ |  FDE $\downarrow$ |  MMADE $\downarrow$ |  MMFDE $\downarrow$
---|---|---|---|---|---
$Ours_{w/o\ gat}$ | 11.088 | 0.678 | 1.002 | 0.691 | 1.011
$Ours_{w/o\ CA}$ | 13.252 | 0.676 | 0.957 | 0.686 | 0.966
$Ours_{w/o\ gaze}$ | 12.478 | 0.677 | 1.048 | 0.710 | 1.063
$BeLFusion_{g}$ | 15.711 | 0.654 | 1.018 | 0.659 | 1.012
$HumanMAC_{g}$ | 13.783 | 0.650 | 0.959 | 0.660 | 0.967
$Ours$ | 14.037 | 0.622 | 0.923 | 0.632 | 0.930

##  4 User Study

To perform a comprehensive evaluation of the perceived realism of the
generated samples, we conducted a user study to assess model performance based
on human perception and intuition. Participants were tasked with ranking
samples predicted by different motion forecasting models. By collecting
subjective rankings directly from individuals, the user study provided an
intuitive approach to evaluate aspects of the predictions that are challenging
to assess objectively, such as naturalness, continuity, and the overall
plausibility of the motions to a human observer.

###  4.1 Assessment Details

We randomly selected 6 sequences from the MoGaze dataset and 12 sequences from
the GIMO dataset. The difference in the number of selections between the two
datasets can be attributed to the difference in the variety of actions
included. While the MoGaze dataset consists of only two actions, the GIMO
dataset contains a larger number of diverse actions. We compared our method
with the best accuracy baseline HumanMAC and the best diversity baseline DLow.
Each method generated 10 predictions from random noise. The participants in
our study were then instructed to rank these predictions based on overall
realism, considering the following three key aspects:

  * •

If these motions appeared plausible. Participants can identify any angle
distortions, excessively short or long limbs, or poses that seem implausible.

  * •

If these motions are continuous. Participants can check if there are any
sudden or unreasonable changes during the whole motion.

  * •

If these motions align with the ground truth. Participants can measure the
similarity between each motion and ground truth.

The online questionnaire utilised in our study was created using the Jotform
platform222· https://www.jotform.com/. The interface of our user study can be
found in the supplementary material. A total of 23 individuals participated in
our user study. This user study is approved by the University.

###  4.2 Evaluation

As shown in Table 3, it is evident that our method outperformed both DLow and
HumanMAC, achieving the highest average ranking of 1.691. This indicates that
our method was perceived as the most realistic by the participants overall.
Remarkably, our method received the highest percentage of first-place
rankings, with 49.52%, indicating that it was consistently ranked as the most
realistic option. Additionally, it obtained a respectable 31.88% of second-
place rankings. On the other hand, our method had the lowest percentage of
third-place rankings at 18.60%, suggesting that it was least frequently
perceived as the least realistic option. In comparison, DLow obtained the
highest percentage of second-place rankings at 39.13%, but its first-place
percentage was lower, at 26.33%. This reveals that DLow was less frequently
considered the most realistic option but often ranked as the second most
realistic. HumanMAC, on the other hand, had the highest percentage of third-
place rankings at 46.60%, indicating that it was frequently perceived as the
least realistic prediction. Moreover, its first and second-place percentages
were the lowest among the three methods. To further validate our findings, we
conducted a paired Wilcoxon signed-rank test, which revealed a statistically
significant difference ($p<0.01$) between our method and the second-best
method, DLow, in terms of average rankings from all responses.

Table 3: Results of the on-line user study. The lower number of average rank is better. |  Average Rank $\downarrow$ | 1st | 2nd | 3rd
---|---|---|---|---
$DLow$ | 2.082 | 26.33% | 39.13% | 34.54%
$HumanMAC$ | 2.227 | 24.15% | 28.99% | 46.86%
$Ours$ | 1.691 | 49.52% | 31.88% | 18.60%

##  5 Discussion

Our study marks a notable advancement in the emerging field of incorporating
eye gaze cues for stochastic human motion prediction in VR research. We have
successfully demonstrated the effectiveness of our proposed method in
generating realistic human motion predictions.

Introducing gaze information. Introducing gaze as a modality allows the model
to generate more realistic and goal-directed poses. Gaze cues provide valuable
information about a person’s intentions and planned actions. In addition, the
integration of gaze modality is highly practical. Modern VR headsets are
equipped with built-in eye-tracking modules that can capture gaze data in real
time. This means that gaze signals are readily available when using these
devices, without the need for additional peripheral hardware. As presented in
Table 1 and Table 2, gaze information indeed improves the accuracy of our
model.

Potential for extension to other modalities. Our proposed approach,
GazeMoDiff, has the potential to be extended to incorporate additional
modalities beyond gaze. It serves as a multimodal human motion prediction
framework, where various guidance information can be utilised to enhance the
generation of reasonable motions. By incorporating modalities such as facial
expressions, hand gestures, and physiological signals, the GazeMoDiff
framework can be expanded to capture a more comprehensive range of cues for
motion prediction. This multimodal approach can potentially lead to more
contextually appropriate motion generation.

Limitations. We followed prior works[10, 4, 62] to select APD as our diversity
metric. But measuring different prediction distances in this way is too simple
and crude, and it is susceptible to extreme values. As shown in Figure 3, we
observed cases where HumanMAC[10] and DLow[62] were prone to produce distorted
poses. Movement trends were diverse measured by a high APD But extremely
implausible. For accurate metrics, ADE and FDE, the error is the smallest one
among all predictions. Image an extreme case in a model that generated 1 very
accurate prediction and 99 distorted and unrealistic predictions. It would be
evaluated as a state-of-the-art model in the APD-ADE-FDE metrics system.
However, the most of predictions are useless and misleading for downstream
analysis. That is also why we evaluated further through visualisation analysis
and user study. Besides, in our work, we predicted future motions by gaze and
pose data. Gaze data is naturally available while pose needs to be captured by
the wearable device. This may limit the application scenarios and be
detrimental to the user experience.

Future work. In our future research endeavours, we aim to explore the
potential for extending our framework to incorporate a variety of modalities,
such as gaze, emotional text, physical parameters, and scene information.
Existing datasets only consist of partial modalities like MoGaze[33] (Motion
and Gaze), GIMO [66] (Motion and Gaze) and EBEDB (Gesture and Emotional
label)[58]. It is worth exploring new datasets aggregating more modalities to
enable more comprehensive multi-modal motion modeling and prediction.
Additionally, to address the limitations associated with device dependency, we
plan to develop a device-free approach. This approach would leverage different
modalities, including gaze, motion, emotion, and scene information directly
captured from video through learning-based estimation methods. By eliminating
the need for users to wear specific devices or deploy complex sensors, this
device-free method holds the potential to be applied in augmented reality or
mixed reality scenarios, enhancing user experience and portability.
Furthermore, we recognise the need for improved metrics for stochastic human
motion prediction. While metrics such as ADE, FDE, and APD have been widely
used, there is room for further refinement and development of metrics that
better capture the quality and usefulness of predicted motions. Enhancing the
metrics will contribute to more accurate and comprehensive evaluations of
human motion prediction methods.

##  6 Conclusion

In this work, we were the first to explore the effectiveness of eye gaze on
generating realistic and diverse human motions. We proposed a novel gaze-
guided diffusion model that first extracts spatio-temporal features from gaze
and motion data and then uses these features to generate realistic human
motions through a denoising diffusion process. Extensive experiments
demonstrated that our method outperforms the state-of-the-art methods by a
large margin in terms of prediction accuracy and an on-line user study
validated that our method can generate motions that are more realistic than
other methods. As such, our work makes an important step towards generating
more realistic human motions for virtual agents and guides future work on
cross-modal human behaviour generation.

## References

  * [1] E. Aksan, M. Kaufmann, P. Cao, and O. Hilliges.  A spatio-temporal transformer for 3d human motion prediction.  In 2021 International Conference on 3D Vision (3DV), pp. 565–574. IEEE, 2021.
  * [2] S. Aliakbarian, F. Saleh, L. Petersson, S. Gould, and M. Salzmann.  Contextually plausible and diverse 3d human motion prediction.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 11333–11342, 2021.
  * [3] M. Azmandian, T. Grechkin, M. Bolas, and E. Suma.  Automated path prediction for redirected walking using navigation meshes.  In 2016 IEEE Symposium on 3D User Interfaces (3DUI), pp. 63–66. IEEE, 2016.
  * [4] G. Barquero, S. Escalera, and C. Palmero.  Belfusion: Latent diffusion for behavior-driven human motion prediction.  arXiv preprint arXiv:2211.14304, 2022.
  * [5] E. Barsoum, J. Kender, and Z. Liu.  Hp-gan: Probabilistic 3d human motion prediction via gan.  In Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pp. 1418–1427, 2018.
  * [6] U. Bhattacharya, N. Rewkowski, A. Banerjee, P. Guhan, A. Bera, and D. Manocha.  Text2gestures: A transformer-based network for generating emotive body gestures for virtual agents.  In 2021 IEEE virtual reality and 3D user interfaces (VR), pp. 1–10. IEEE, 2021.
  * [7] J. Butepage, M. J. Black, D. Kragic, and H. Kjellstrom.  Deep representation learning for human motion prediction and classification.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 6158–6166, 2017.
  * [8] Z. Cao, H. Gao, K. Mangalam, Q.-Z. Cai, M. Vo, and J. Malik.  Long-term human motion prediction with scene context.  In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part I 16, pp. 387–404. Springer, 2020.
  * [9] H. Chen, L. Wei, H. Liu, B. Shi, G. Zhang, and H. Zha.  Mount: Learning 6dof motion prediction based on uncertainty estimation for delayed ar rendering.  IEEE Transactions on Visualization and Computer Graphics, pp. 1–12, 2022. doi: 10 . 1109/TVCG . 2022 . 3228807
  * [10] L.-H. Chen, J. Zhang, Y. Li, Y. Pang, X. Xia, and T. Liu.  Humanmac: Masked motion completion for human motion prediction.  arXiv preprint arXiv:2302.03665, 2023.
  * [11] S. Chen, P. Sun, Y. Song, and P. Luo.  Diffusiondet: Diffusion model for object detection.  arXiv preprint arXiv:2211.09788, 2022.
  * [12] B. David-John, C. Peacock, T. Zhang, T. S. Murdison, H. Benko, and T. R. Jonker.  Towards gaze-based prediction of the intent to interact in virtual reality.  In ACM Symposium on Eye Tracking Research and Applications, pp. 1–7, 2021.
  * [13] P. Dhariwal and A. Nichol.  Diffusion models beat gans on image synthesis.  Advances in Neural Information Processing Systems, 34:8780–8794, 2021.
  * [14] Y. Du, R. Kips, A. Pumarola, S. Starke, A. Thabet, and A. Sanakoyeu.  Avatars grow legs: Generating smooth human motion from sparse tracking inputs with diffusion model.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 481–490, 2023.
  * [15] K. J. Emery, M. Zannoli, J. Warren, L. Xiao, and S. S. Talathi.  Openneeds: A dataset of gaze, head, hand, and scene signals during exploration in open-ended vr environments.  In ACM Symposium on Eye Tracking Research and Applications, pp. 1–7, 2021.
  * [16] K. Fragkiadaki, S. Levine, P. Felsen, and J. Malik.  Recurrent network models for human dynamics.  In Proceedings of the IEEE international conference on computer vision, pp. 4346–4354, 2015.
  * [17] N. M. Gamage, D. Ishtaweera, M. Weigel, and A. Withana.  So predictable! continuous 3d hand trajectory prediction in virtual reality.  In The 34th Annual ACM Symposium on User Interface Software and Technology, pp. 332–343, 2021.
  * [18] P. Ghosh, J. Song, E. Aksan, and O. Hilliges.  Learning human motion models for long-term predictions.  In 2017 International Conference on 3D Vision (3DV), pp. 458–466. IEEE, 2017.
  * [19] A. Gopalakrishnan, A. Mali, D. Kifer, L. Giles, and A. G. Ororbia.  A neural temporal model for human motion prediction.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12116–12125, 2019.
  * [20] J. Ho, A. Jain, and P. Abbeel.  Denoising diffusion probabilistic models.  Advances in neural information processing systems, 33:6840–6851, 2020.
  * [21] J. Ho, T. Salimans, A. Gritsenko, W. Chan, M. Norouzi, and D. J. Fleet.  Video diffusion models.  arXiv preprint arXiv:2204.03458, 2022.
  * [22] J. E. Holm.  Collision prediction and prevention in a simultaneous multi-user immersive virtual environment.  PhD thesis, Miami University, 2012.
  * [23] X. Hou, J. Zhang, M. Budagavi, and S. Dey.  Head and body motion prediction to enable mobile vr experiences with low latency.  In 2019 IEEE Global Communications Conference (GLOBECOM), pp. 1–7. IEEE, 2019.
  * [24] Z. Hu, A. Bulling, S. Li, and G. Wang.  Ehtask: Recognizing user tasks from eye and head movements in immersive virtual reality.  IEEE Transactions on Visualization and Computer Graphics, 2021.
  * [25] Z. Hu, A. Bulling, S. Li, and G. Wang.  Fixationnet: Forecasting eye fixations in task-oriented virtual environments.  IEEE Transactions on Visualization and Computer Graphics, 27(5):2681–2690, 2021.
  * [26] Z. Hu, S. Li, C. Zhang, K. Yi, G. Wang, and D. Manocha.  Dgaze: Cnn-based gaze prediction in dynamic scenes.  IEEE Transactions on Visualization and Computer Graphics, 26(5):1902–1911, 2020.
  * [27] Z. Hu, C. Zhang, S. Li, G. Wang, and D. Manocha.  Sgaze: A data-driven eye-head coordination model for realtime gaze prediction.  IEEE transactions on visualization and computer graphics, 25(5):2002–2010, 2019.
  * [28] D. K. Jain, M. Zareapoor, R. Jain, A. Kathuria, and S. Bachhety.  Gan-poser: an improvised bidirectional gan model for human motion prediction.  Neural Computing and Applications, 32(18):14579–14591, 2020.
  * [29] Y. Kim, J. An, M. Lee, and Y. Lee.  An activity-embedding approach for next-activity prediction in a multi-user smart space.  In 2017 IEEE International Conference on Smart Computing (SMARTCOMP), pp. 1–6. IEEE, 2017.
  * [30] D. P. Kingma and J. Ba.  Adam: A method for stochastic optimization.  arXiv preprint arXiv:1412.6980, 2014.
  * [31] F. Koochaki and L. Najafizadeh.  Predicting intention through eye gaze patterns.  In 2018 IEEE Biomedical Circuits and Systems Conference (BioCAS), pp. 1–4. IEEE, 2018.
  * [32] R. Kothari, Z. Yang, C. Kanan, R. Bailey, J. B. Pelz, and G. J. Diaz.  Gaze-in-wild: A dataset for studying eye and head coordination in everyday activities.  Scientific reports, 10(1):2539, 2020.
  * [33] P. Kratzer, S. Bihlmaier, N. B. Midlagajni, R. Prakash, M. Toussaint, and J. Mainprice.  Mogaze: A dataset of full-body motions that includes workspace geometry and eye-gaze.  IEEE Robotics and Automation Letters, 6(2):367–373, 2020.
  * [34] J. N. Kundu, M. Gor, and R. V. Babu.  Bihmp-gan: Bidirectional 3d human motion prediction gan.  In Proceedings of the AAAI conference on artificial intelligence, vol. 33, pp. 8553–8560, 2019.
  * [35] M. Li, S. Chen, Y. Zhao, Y. Zhang, Y. Wang, and Q. Tian.  Multiscale spatio-temporal graph neural networks for 3d skeleton-based motion prediction.  IEEE Transactions on Image Processing, 30:7760–7775, 2021.
  * [36] Q. Li, G. Chalvatzaki, J. Peters, and Y. Wang.  Directed acyclic graph neural network for human motion prediction.  In 2021 IEEE International Conference on Robotics and Automation (ICRA), pp. 3197–3204. IEEE, 2021.
  * [37] Z. Li, Y. Zhou, S. Xiao, C. He, Z. Huang, and H. Li.  Auto-conditioned recurrent networks for extended complex human motion synthesis.  arXiv preprint arXiv:1707.05363, 2017.
  * [38] T. Ma, Y. Nie, C. Long, Q. Zhang, and G. Li.  Progressively generating better initial guesses towards next stages for high-quality human motion prediction.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6437–6446, 2022.
  * [39] W. Mao, M. Liu, and M. Salzmann.  Generating smooth pose sequences for diverse human motion prediction.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 13309–13318, 2021.
  * [40] W. Mao, M. Liu, M. Salzmann, and H. Li.  Learning trajectory dependencies for human motion prediction.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9489–9497, 2019.
  * [41] J. Martinez, M. J. Black, and J. Romero.  On human motion prediction using recurrent neural networks.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2891–2900, 2017.
  * [42] A. Martínez-González, M. Villamizar, and J.-M. Odobez.  Pose transformers (potr): Human motion prediction with non-autoregressive transformers.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 2276–2284, 2021.
  * [43] A. Nichol, P. Dhariwal, A. Ramesh, P. Shyam, P. Mishkin, B. McGrew, I. Sutskever, and M. Chen.  Glide: Towards photorealistic image generation and editing with text-guided diffusion models.  arXiv preprint arXiv:2112.10741, 2021.
  * [44] A. Q. Nichol and P. Dhariwal.  Improved denoising diffusion probabilistic models.  In International Conference on Machine Learning, pp. 8162–8171. PMLR, 2021.
  * [45] A. Ramesh, P. Dhariwal, A. Nichol, C. Chu, and M. Chen.  Hierarchical text-conditional image generation with clip latents.  arXiv preprint arXiv:2204.06125, 1(2):3, 2022.
  * [46] K. Rasul, C. Seward, I. Schuster, and R. Vollgraf.  Autoregressive denoising diffusion models for multivariate probabilistic time series forecasting.  In International Conference on Machine Learning, pp. 8857–8868. PMLR, 2021.
  * [47] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer.  High-resolution image synthesis with latent diffusion models.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10684–10695, 2022.
  * [48] S. Saadatnejad, A. Rasekh, M. Mofayezi, Y. Medghalchi, S. Rajabzadeh, T. Mordan, and A. Alahi.  A generic diffusion-based approach for 3d human pose prediction in the wild.  In 2023 IEEE International Conference on Robotics and Automation (ICRA), pp. 8246–8253. IEEE, 2023.
  * [49] L. Sidenmark and H. Gellersen.  Eye, head and torso coordination during gaze shifts in virtual reality.  ACM Transactions on Computer-Human Interaction (TOCHI), 27(1):1–40, 2019.
  * [50] L. Sidenmark and H. Gellersen.  Eye&head: Synergetic eye and head movement for gaze pointing and selection.  In Proceedings of the 2019 ACM Symposium on User Interface Software and Technology, pp. 1161–1174, 2019.
  * [51] L. Sidenmark, D. Mardanbegi, A. R. Gomez, C. Clarke, and H. Gellersen.  Bimodalgaze: Seamlessly refined pointing with gaze and filtered gestural head movement.  In ACM Symposium on Eye Tracking Research and Applications, pp. 1–9, 2020.
  * [52] J. Song, C. Meng, and S. Ermon.  Denoising diffusion implicit models.  arXiv preprint arXiv:2010.02502, 2020.
  * [53] Q. Sun, A. Patney, L.-Y. Wei, O. Shapira, J. Lu, P. Asente, S. Zhu, M. McGuire, D. Luebke, and A. Kaufman.  Towards virtual reality infinite walking: dynamic saccadic redirection.  ACM Transactions on Graphics (TOG), 37(4):1–13, 2018.
  * [54] Y. Tashiro, J. Song, Y. Song, and S. Ermon.  Csdi: Conditional score-based diffusion models for probabilistic time series imputation.  Advances in Neural Information Processing Systems, 34:24804–24816, 2021.
  * [55] G. Tevet, S. Raab, B. Gordon, Y. Shafir, D. Cohen-Or, and A. H. Bermano.  Human motion diffusion model.  arXiv preprint arXiv:2209.14916, 2022.
  * [56] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin.  Attention is all you need.  Advances in neural information processing systems, 30, 2017.
  * [57] P. Veličković, G. Cucurull, A. Casanova, A. Romero, P. Lio, and Y. Bengio.  Graph attention networks.  arXiv preprint arXiv:1710.10903, 2017.
  * [58] E. Volkova, S. De La Rosa, H. H. Bülthoff, and B. Mohler.  The mpi emotional body expressions database for narrative scenarios.  PloS one, 9(12):e113647, 2014.
  * [59] F. Wilcoxon.  Individual comparisons by ranking methods.  In Breakthroughs in Statistics: Methodology and Distribution, pp. 196–202. Springer, 1992.
  * [60] J. Wyatt, A. Leach, S. M. Schmon, and C. G. Willcocks.  Anoddpm: Anomaly detection with denoising diffusion probabilistic models using simplex noise.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 650–656, 2022.
  * [61] J. Xu, X. Wang, W. Cheng, Y.-P. Cao, Y. Shan, X. Qie, and S. Gao.  Dream3d: Zero-shot text-to-3d synthesis using 3d shape prior and text-to-image diffusion models.  arXiv preprint arXiv:2212.14704, 2022.
  * [62] Y. Yuan and K. Kitani.  Dlow: Diversifying latent flows for diverse human motion prediction.  In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part IX 16, pp. 346–364. Springer, 2020.
  * [63] W. H. Zangemeister and L. Stark.  Gaze latency: variable interactions of head and eye latency.  Experimental Neurology, 75(2):389–406, 1982.
  * [64] L. Zhang and M. Agrawala.  Adding conditional control to text-to-image diffusion models.  arXiv preprint arXiv:2302.05543, 2023.
  * [65] M. Zhang, Z. Cai, L. Pan, F. Hong, X. Guo, L. Yang, and Z. Liu.  Motiondiffuse: Text-driven human motion generation with diffusion model.  arXiv preprint arXiv:2208.15001, 2022.
  * [66] Y. Zheng, Y. Yang, K. Mo, J. Li, T. Yu, Y. Liu, C. K. Liu, and L. J. Guibas.  Gimo: Gaze-informed human motion prediction in context.  In European Conference on Computer Vision, pp. 676–694. Springer, 2022.