11institutetext: Carnegie Mellon University, Pittsburgh, PA 15213, USA
11email<EMAIL_ADDRESS>
# 3D Human Motion Estimation via
Motion Compression and Refinement

Zhengyi Luo  0000-0002-1842-7622 Carnegie Mellon University, Pittsburgh, PA
15213, USA
11email<EMAIL_ADDRESS>S. Alireza Golestaneh
0000-0002-3230-2797 Carnegie Mellon University, Pittsburgh, PA 15213, USA
11email<EMAIL_ADDRESS>Kris M. Kitani
0000-0002-9389-4060 Carnegie Mellon University, Pittsburgh, PA 15213, USA
11email<EMAIL_ADDRESS>
###### Abstract

We develop a technique for generating smooth and accurate 3D human pose and
motion estimates from RGB video sequences. Our method, which we call Motion
Estimation via Variational Autoencoder (MEVA), decomposes a temporal sequence
of human motion into a smooth motion representation using auto-encoder-based
motion compression and a residual representation learned through motion
refinement. This two-step encoding of human motion captures human motion in
two stages: a general human motion estimation step that captures the coarse
overall motion, and a residual estimation that adds back person-specific
motion details. Experiments show that our method produces both smooth and
accurate 3D human pose and motion estimates.

Figure 1: Given an in-the-wild video, state-of-the-art methods (e.g., VIBE
[1]) can achieve high per joint accuracy, but also suffers from high
acceleration error. To tackle this, we develop a two-stage motion estimation
method, MEVA, that is able to produce both accurate and smooth human motion.

##  1 Introduction

Estimating the 3D pose sequence of a person from a single video requires a
computational model that can extract the underlying kinematics of human motion
while also preserving motions that are unique to the person being captured.
Since people share a similar body structure (_e.g.,_ same number of joints)
and similar physical constraints (_e.g.,_ joint limitations), it is possible
to learn a generalized kinematic model that can be matched against the image
to infer the general motion of a person. However, since generalized models of
motion can also fail to model person-specific motions, it may also be
necessary to ‘add back in’ or refine the general motion estimates using image
evidence. In this work, we present a two-stage 3D motion estimation method
that first extracts coarse kinematic sequences of a person in a video and then
refines that sequence to produce a more accurate 3D motion estimate. We show
that by decomposing the inference process into (1) a general model of motion
and (2) a person-specific model of motion, we are able to obtain more accurate
and smooth estimates.

Over the past years, significant progress has been made on improving the
accuracy of 3D human pose estimation. Impressive results have been obtained
through human mesh recovery from single images [2, 3, 4, 5, 6] and videos [7,
1, 8, 9]. The main metric used to evaluate these methods is the Mean Per Joint
Position Error (MPJPE), which measures the performance in terms of the
relative joint positions computed for each frame of a video. However, less
emphasis has been given to the temporal smoothness of the estimated motion.
Optimizing for this metric, the tendency is to generate pose estimates that
‘jitter’ near the true pose. This is expected as the MPJPE only penalizes for
spatial errors and is not designed to account for temporal consistency. As the
methods for 3D pose estimation have improved in recent years, the ‘jitter’ has
become less pronounced, especially when applied to dynamic scenes with vibrant
moving backgrounds and camera motion. However, by rendering the estimated 3D
pose of state-of-the-art methods on a plain background, the ‘jitter’ can still
be observed, resulting in an overall unnatural motion estimation.

The issue of temporal smoothness is a known problem and has been addressed in
part by prior work. Large-scale motion datasets such as Archive of Motion
Capture as Surface Shapes (AMAAS) [10]) and adversarial loss have enabled
methods to improve both pose accuracy and temporal smoothness [1, 7]. Other
methods have been developed to enforce temporal smoothness [2] by letting the
model predict frame ordering. However, using prior knowledge only in the loss
function, it is hard to find the balance between smoothness and accuracy.

In this work, we argue that striking such balance between smoothness and
accuracy can be done through an explicit breakdown of coarse and fine motion.
First, we learn a coarse motion model by observing a large dataset of human
motion–since human motion is typically smooth (_e.g._ , we usually do not
shake as we walk), if one were to fit a model to a large set of human motions,
most of the data would lie in a sub-space in which motions are smooth. This
implies that if we were to compress human motion data, it should learn a
latent subspace in which the motions are inherently smooth and coarse. Using
this latent space as the regression target, we can directly infer coarse human
motion from the input videos. The problem of using such a human motion
subspace, of course, is that rare motions (_e.g.,_ sudden motions) are removed
from the motion data. To retain such motion, we also argue that producing the
final 3D motion estimate can be treated as a refinement step to ”add back” the
fine details to the coarse motion sequence.

To validate our arguments, we propose a two-stage 3D human motion estimation
method that first estimates a coarse human pose sequence through data
compression using a Variational Autoencoder (VAE), which we call the
Variational Motion Estimator (VME). Then we take the output of the VME and
refine the pose estimate using the image evidence with a pose regressor, which
we call the Motion Refinement Regressor (MRR).

In summary, we propose a video-based 3D human motion estimation method that
focuses on producing smooth and accurate human motion sequences. Our main
contributions are as follows: (1) we propose a two-stage motion estimation
method for ensuring temporal smoothness and accurate pose estimates; (2) we
describe a procedure to learn a robust Variational Autoencoder that can serve
as a latent human motion subspace for estimating coarse 3D human motion from
videos; (3) we demonstrate state-of-the-art pose/motion estimation performance
on challenging in-the-wild dataset such as 3DPW [11], reducing acceleration
error by $54.3\%$ while achieving state-of-the-art MPJPE results.

##  2 Related Works

In this section, we will first review the relevant work in human shape and
pose recovery from a single image and from videos–human motion recovery can be
treated as a subset of human pose estimation, as human motion is a sequence of
human poses. Then we will review how existing methods use human motion as a
prior and how popular methods map motion sequences to a low-dimensional space.

###  2.1 Recovering 3D human pose and shape from a single image

Here we focus on model-based methods that jointly recover human shape and
pose. We choose to use a parametric 3D human body model [12, 13, 14] since it
can be easily turned into a 3D human mesh that is usable for downstream tasks
such as animation. Directly fitting a parametric 3D human body to image input
has gained substantial traction over the years, morphing from methods that
require silhouette or human input [15, 16, 17, 18, 19], to ones that can
directly fit model parameters to 2D joint positions [20], and to ones that can
directly estimate shape and pose from images [8, 21, 22, 12, 23, 24, 3]. Due
to the lack of ground truth 3D labels, these methods use a weakly supervised
approach to fit the 3D human body to 2D joint positions [8, 23, 24], body part
segmentation [21, 25], or dense pixel correspondence [6]. Although these
methods achieve amazing results, their extracted motion tends to be unstable
due to the lack of temporal information.

###  2.2 Recovering 3D human pose and shape from video

Using temporal information to aid 3D human pose estimation is a natural
extension to single frame methods. [9, 26, 27, 28] focus on ”lifting”
predicted joint positions from 2D to 3D, and uses LSTM [9], Temporal
Convolution [26], and fully connected layers [27] to exploit temporal
information. [29, 30], on the other hand, predict 3D joint positions directly
from images and use a temporal filter to postprocess the motion sequence. For
methods that jointly recover shape and pose. HMMR [7], Sun et al. [2] and VIBE
[1] are the best performing models that exploit temporal information. HMMR [7]
proposes to enforce temporal consistency by letting the model predict future
and past frames of motion. Sun et al. [2] learns temporal information by
predicting the ordering of shuffled frames. VIBE [1] utilizes temporal
information by employing a Gated Recurrent Unit (GRU) to convert the input
frames of features into a sequence of temporally correlated latent features.

###  2.3 Human Pose and Motion Prior

Using prerecorded human motion sequences as a prior has also been explored in
various tasks related to human motion. Earlier work like [31] tries to
quantify unnaturalness in animated human motion by statistically analyzing
existing motion capture (MoCap) sequences, and [32, 33] propose to use learned
MoCap motions to aid 3D motion tracking. More recently, [8, 7] use an
adversarial discriminator at a per-frame level to ensure that the recovered
pose is a valid human pose. [12] uses a pretrained pose VAE’s latent space for
a similar purpose. [1] proposes to use the discriminator at a temporal level,
discriminating against a whole motion sequence. All above methods use a pose
or motion prior in an adversarial way, utilizing the prior knowledge in the
loss function.

###  2.4 Human Motion Representation

Compressing human motion into a compact latent representation plays an
important role in tasks such as human motion generation [34, 35, 36], human
motion generation across different modalities [37, 38, 39, 40], and trajectory
forecasting [41, 42, 43, 44]. Existing methods leverage different generative
models such as VAE [42, 43], generative flow [45], or generative adversarial
networks [34] to achieve a compact motion representation in the latent space.

##  3 Approach

As discussed in Sections 1 and 2, existing human motion estimation methods
often find it difficult to achieve a balance between temporal smoothness and
accuracy. To tackle this, we propose MEVA, Motion Estimation via Variational
Autoencoder, a framework that learns the overall coarse motion first then adds
back fine detailed motion as a residual. MEVA processes the inputs in three
steps: it first extracts correlated temporal features using a spatio-temporal
feature extractor (STE), and then captures the overall coarse motion through a
variational motion estimator (VME), and finally uses a motion residual
regressor (MRR) to add back the fine motion details. Our overall framework can
be shown in Fig. 2. In this section, we will first setup the overall problem
and then present the details of our framework. Finally, we will discuss the
training procedures.

Figure 2: Overall Architecture. MEVA estimates motion from videos by first
extracting temporal features using Spatio-Temporal Feature Extractor (STE) and
then estimates the overall coarse motion inside the video with Variational
Motion Estimator (VME). Finally, a Motion Residual Regressor (MRR) is used to
refine the motion estimates.

###  3.1 Problem Formulation

Given an input video $V_{T}=\\{I_{t}\\}_{t=1}^{T}$, where $I_{t}$ denotes the
$t^{th}$ frame, our task is to recover coherent human motion sequences
$M_{T}=\\{\theta_{t}\\}_{t=1}^{T}$ where each $\theta_{t}$ represents the
human pose for the $t^{th}$ frame. To represent the human motion, we utilize
the SMPL 3D human mesh model [13]. We choose SMPL parametrization over 3D
joint positions or other human models due to its versatility: SMPL parameters
can be easily converted to 3D joint positions and human mesh. Specifically, a
human body is represented by its shape $\beta$ and pose $\theta$, denoted by
$\Theta=\\{\beta,\theta\\}$. Given $\theta$ and $\beta$, let $S$ denote the
pretrained SMPL function, where $S(\Theta):\beta,\theta\rightarrow
R^{6890\times 3}$ ( $6890$ is the number of vertices of the resulting
triangular human mesh). The pose parameter $\theta\in R^{24\times N}$ stands
for the joint angles for the 23 joints plus the root orientation. $N$ is the
dimension of the chosen rotation representation ($N=3$ for axis/euler angle,
$N=4$ for quaternions, $N=6$ for a 6 degrees-of-freedom rotation
representation [46]). The shape parameters $\beta\in R^{10}$ represent the
linear coefficient for the principal component of the parametric human shape
space. Given a set of $\beta$ and $\theta$, $S$ can recover the 3D joint
positions through a pretrained mesh vertex regressor
$P:jp^{3d}=P\\{S(\beta,\theta)\\}\in R^{N\times 3}$. To project the 3D joint
positions back to 2D images, a weak perspective camera
$\pi=\\{s,t_{x},t_{y}\\}$ needs to be estimated:
$jp^{2d}=\Pi(P\\{S(\beta,\theta)\\})\in R^{N\times 2}$.

Intuitively, recovering human motion from video frames does not require
recovering the human shape: one can directly learn a mapping from input video
frames to the estimated human motion if sufficient paired ground truth data
exist. However, videos paired with ground truth motion annotation (SMPL
sequences) require professional capture equipment such as a motion capture
(MoCap) rig, which is still rather rare compared to annotated 2D pose
datasets. In the absence of 3D data samples, it is critical for our model to
learn motion sequences in a semisupervised fashion (i.e., from videos with 3D
or 2D labeled joint positions following the approach utilized in [8, 7, 1, 3,
2]).

Overall, our motion estimation objective is to learn a function
$\text{MEVA}(V):V_{T}\rightarrow{M_{T}}$ where $V_{T}=\\{I_{t}\\}_{t=1}^{T}$
and $M_{T}=\\{\theta_{t}\\}_{t=1}^{T}$.

###  3.2 Spatio-Temporal Feature Extractor (STE)

Human motion is inherently temporal and correlated, and past movement can give
cues about future motion. Thus, instead of extracting per-frame visual
features using a feed-forward convolutional network independently, we can
produce temporally correlated features that lead to better motion sequence
modeling. Similar to [1], we use a GRU-based temporal feature extractor (STE)
that encodes the input video frames $I_{1},I_{2},I_{3},...I_{T}$ into a
sequence of temporally correlated features
$f^{\prime}_{1},f^{\prime}_{2},f^{\prime}_{3},...f^{\prime}_{T}$.

###  3.3 Variational Motion Estimator (VME)

####  3.3.1 Human Motion VAE

To learn a human motion subspace that can encapsulate a broad spectrum of
human motion, we choose to use a Variational Autoencoder (VAE). VAEs can
effectively capture a large number of possible data modes by explicitly
mapping each data point to a latent code, and by imposing a Gaussian prior on
the learned latent space, similar motions’ latent codes will be near each
other. Thus, the VAE’s latent space allows for more overlap between codes and
therefore enforces smoothness in the latent space. Having a smooth latent code
is essential in improving the generalizability of the model since the space of
possible human motion is highly correlated and limited. Formally, following
the previous work on VAEs [43, 47, 48, 48], the objective is to maximize the
evidence lower bound of the log-likelihood $p_{\lambda}(x)$ ($\lambda$ and
$\phi$ denotes the function parametrization):

$L_{VAE}=E_{q_{\phi}}[\log p(x|z)]-\text{KLD}(q_{\phi}(z|x)||p_{\phi}(z)),$
(1)

where $x$ is the input and the latent code
$z\sim\mathcal{N}(\mathbf{0},\mathbf{I})$.

In the context of encoding human motion via VAE, the encoder $E_{vae}$ takes
in a sequence of $W$ frames of human motion represented in terms of SMPL pose
parameters: $x=M_{W}=[\theta_{w1},\theta_{w2},\theta_{w3},...]\in R^{W\times
144}$ and outputs the latent code $z$. A single frame of SMPL pose is
represented in joint rotations, resulting by a $24\times 6=144$ dimensional
input (a 6 degrees-of-freedom rotation representation[46] is used for
continuity purpose). The decoder $D_{vae}$ takes in the latent code $z$ and
reconstructs the motion $\hat{M}_{W}$. Both the encoder $E_{vae}$ and decoder
$D_{vae}$ are implemented as GRUs, and the detailed architectures are given in
Fig.3. Based on the Gaussian parameterization of the VAE, the objective
function of Eq.(1) can be written Eq.(2)

Figure 3: Motion VAE Architecture.

$\mathcal{L}_{vae}(\mathbf{x};\theta,\phi)=-\frac{1}{S}\sum_{s=1}^{S}\left\|\tilde{\mathbf{x}}-\mathbf{x}\right\|^{2}+\beta\cdot\frac{1}{S_{z}}\sum_{j=1}^{S_{z}}\left(1+2\log\sigma_{j}-\mu_{j}^{2}-\sigma_{j}^{2}\right),$
(2)

where $S$ is the number of samples for the current batch, $S_{z}$ is the
dimension of the current latent variable, and $\beta$ is the weighing
parameter. Once the VAE is trained and converged to a desirable reconstruction
accuracy, the decoder $D_{vae}$ is frozen for later use. During inference,
given a latent code $z\in R^{1\times S_{z}}$, $D_{vae}$ can decode it back
into a sequence of human motion: $M_{W}\in R^{w\times 144}$.

####  3.3.2 Human Motion Data augmentation

Our learned VAE should be able to generalize to unseen human motion sequences
and achieve high reconstruction accuracy to ensure that the learned latent
space can indeed serve as a comprehensive human motion subspace. Using an
already large-scale human motion dataset AMASS [10] (13k motion samples with
varying length), our trained VAE still has poor generalizability on unseen
sequences (for details refer to Sec.4.3.1). Thus, we devise an elaborate data
augmentation scheme that can augment the existing motion and produce viable
yet distinct human motion sequences. While data augmentation has been studied
extensively in image processing, to the best of our knowledge, few attempts
have been done in augmenting a human motion dataset. When used in trajectory
forecasting [43] and human motion generation [34], the generalizability of the
VAE latent space has not been discussed extensively since the models only need
to generate new motion sequences and do not emphasize on the ability to encode
unseen motion sequences.

Given a $T$ frame human motion sequence in SMPL parameters $M_{T}\in
R^{T\times 144}$ with a frame-rate $F_{amass}$, we employ the following data
augmentation scheme:

  * •

Speeding up and slowing down: based on $F_{amass}$, we can uniformly up-sample
or downsample the frames and produce novel sequences that are still plausible
and natural human motion.

  * •

Flipping left and right: The same action, performed using either the left or
right hand, will remain a valid human motion. Thus, we can follow the
kinematic tree of the SMPL model and mirror the motion across the left and
right and generate a new motion sequence.

  * •

Random root rotation: We randomly sample a root rotation from a unit sphere to
capture different root orientations for possible human motion. Different pose
estimators may assume different ground planes and coordinate systems, so SMPL
parameters usually come in different root orientations. Sampling random root
rotation helps the model cope with different possible coordinate frame
choices.

####  3.3.3 Learning Smooth Motion from videos

After learning a comprehensive human motion subspace using the VAE, we learn
an additional encoder $E_{motion}$ that can directly extract coarse motion
sequences from video features, mapping to the same latent space as $E_{vae}$.
Given an input sequence of video features $f_{W}=\\{f_{w}\\}_{w=1}^{W}$, the
encoder $E_{motion}$’s task is to compress the input features into a latent
code $z$ that best summarizes the current observation as a coarse human motion
sequence. We use the pretrained decoder $D_{vae}$ from the motion VAE to force
$E_{motion}$ to sample from our pretrained motion subspace. Constraining the
latent space of the $E_{motion}$ to a pretrained human motion subspace
provides a strong human motion prior that greatly aids the learning process of
$E_{motion}$. Combining $E_{motion}$ and $D_{vae}$, we form our Variational
Motion Estimator (VME).

###  3.4 Motion Residual Regressor (MRR)

As noted in the previous section, the learned motion sequences using the VAE’s
latent space as the target are inherently smooth and coarse, capturing the
overall motion signature of the current video frames through information
compression. To capture the details, we utilize a SMPL regressor from [8] that
can iteratively refine the estimated poses. The regressor takes in an initial
pose and shape estimation $\Theta_{t}$ and the visual feature $\text{f}_{t}$
for a single frame to calculate its estimation $\Theta_{t}^{\prime}$ for $k$
iterations. Notice that though [8, 7, 1, 3] all utilize the same regressor,
MEVA uses it in a fundamentally different way–in [8, 7, 1, 3], the regressor
is initialized with mean SMPL pose $\Theta_{mean}$. At a sequence level, a
regressor initialized uniformly with the mean pose $\Theta_{mean}$ is trying
to capture the overall motion in one pass, while in MEVA, the regressor is
initialized with the computed poses from VME. Thus, the regressor is only
tasked to do small cosmetic changes to the coarse estimation, adding back the
fine details of the motion lost during our compression step. Similar to [1],
the input visual features ${f}_{t}$ to the regressor are encoded using a
temporal visual encoder, so even though each frame’s estimation is calculated
separately at this stage, the visual feature is already temporally correlated.
The VME computes the overall coarse motion from videos by using a general
model of motion, and the regressor jointly refines motion and human shape
estimates, which amounts to adding back the person-specific motion details at
a per frame level. We call this regressor the Motion Residual Regressor (MRR).
MRR completes the overall framework of our proposed method, as shown in Fig.
2.

###  3.5 Training and Losses

Our framework is trained in two stages. At first, the motion VAE is
pretrained. Then STE, VME, and MRR are trained jointly end-to-end. Using
videos with various levels of annotation (2D joint positions, 3D joint
positions, SMPL parameters), similar to in [7, 1, 3], the networks are trained
with losses consisting of $L_{2D}$, $L_{3D}$, $L_{SMPL}$, as long as
respective data is available. Specifically:

$\displaystyle L_{meva}$ $\displaystyle=L_{3D}+L_{2D}+L_{SMPL},$ (3)
$\displaystyle L_{3D}$
$\displaystyle=\Sigma_{t=1}^{T}||jp^{3d}_{t}-\hat{jp^{3d}_{t}}||_{2},$ (4)
$\displaystyle L_{2D}$
$\displaystyle=\Sigma_{t=1}^{T}||jp^{2d}_{t}-\hat{jp^{2d}_{t}}||_{2},$ (5)
$\displaystyle L_{SMPL}$
$\displaystyle=||\beta-\hat{\beta}||_{2}+\Sigma_{t=1}^{T}||\theta_{t}-\hat{\theta}_{t}||_{2}.$
(6)

For implementation details, please refer to the supplementary material.

##  4 Experiments

To demonstrate the effectiveness of our proposed method, we evaluate our
method in terms of the overall accuracy and smoothness of the estimated motion
on MPI-INF-3DPH [49], 3DPW [11], and human 3.6M [50]. In the following
sections, we will first describe the main datasets used to train and evaluate
MEVA. Then in Section 4.2 we provide extensive evaluation results. Finally, in
Section 4.3, we provide the ablation studies for our proposed method.

###  4.1 Datasets

For the motion VAE, we use motion sequences from AMASS [10] for training and
sequences from 3DPW [11] for evaluation. For training with videos, in addition
to the train split of MPI-INF-3DPH [49], 3DPW [11], and human 3.6M [50], which
have 3D joint annotation, we also use InstaVariety [7] and PennAction [51]
which contain 2D joint annotation.

For training our motion VAE, we use AMASS [10]. It is a recent dataset that
contains a large sample of human motion sequences in SMPL parameters. These
sequences are fitted from MoCap sequences using Mosh++ [10]. There are in
total 13k motion sequences with varying length. We use this dataset only for
training our motion VAE.

For training with videos, 3DPW is the only dataset that contains paired SMPL
parameters and video sequences (which provide direct supervision to MEVA). The
videos from this dataset are mostly outdoors and in-the-wild. It uses paired
IMU sensors and video input to compute the near ground truth SMPL pose and
shape parameters. This is a relatively small dataset and we use the official
split in [11] for train, validation, and test. There are in total 60 videos
with varying length (24 train, 24 test, 12 val). MPI-INF-3DHP is a dataset
that contains 3D joint position annotation. It is captured using a multiview
camera setup, and the 3D joint annotation is calculated through multiview
methods [49]. There are 8 subjects and 16 videos per subject, in total 128
videos with varying length. We use the official test and train split. H3.6M is
a popular pose estimation dataset that contains 3D joint position annotations,
captured indoors with MoCap markers. Notice that a number of previous works
[8, 7, 3, 1, 2] had access to a near-ground truth SMPL pose and shape
parameters calculated by the Mosh [52] method. However, this annotation has
since been removed from public access due to legal issues. SMPL parameters
provide the best supervision, as noted in [53], so for a fair comparison we
retrain some of the state-of-the-art methods without such supervision. There
are 840 videos in total across 7 subjects in H3.6M and we use the official
train/test subject split ([S1, S5, S6, S7, S8] vs [S9, S11]). During
preprocessing, we subsample every 5 frames from the dataset. The PennAction
dataset contains human annotated ground truth 2D keypoints paired with video
sequences. There are in total 2326 videos with varying length. InstaVariety
dataset contains human annotated pseudo ground truth 2D keypoints paired with
video sequences. The 2D keypionts are estimated using openpose[54]. There are
in total 28,272 videos with varying length.

###  4.2 Evaluation Results and Analysis

####  4.2.1 Metrics

To best capture human motion, we utilize three popular metrics that measure
the overall accuracy and smoothness of the motion. Mean per joint position
error (MPJPE) and MPJPE after Procrustes Alignment (PA-MPJPE) measure the 3D
joint positional discrepancy between the predicted and ground truth 3D joint
positions in millimeters ($mm$), and are calculated after aligning the root
position (human pelvis) of the 3D joint positions. Both MPJPE and PA-MPJPE
serve as the accuracy indicator of the motion estimator. Acceleration error
(ACC-ERR), proposed in [7], measures the difference between the predicted and
ground truth 3D acceleration for each keypoint in $mm/s^{2}$. ACC-ERR serves
as the major smoothness indicator for the estimated motion sequences.
Acceleration is calculated using the finite difference between individual
frames. It is imperative to view these metrics jointly: a low position error
indicates overall correctness in motion capture and a better acceleration
error marks a smooth and natural estimation of human motion.

####  4.2.2 Generalization of Motion VAE

In this section, we study the genealizability of our learned motion VAE. We
report the reconstruction error of the VAE on unseen motion sequences from the
3DPW dataset. Table 1 shows VAE motion reconstruction error over the different
splits of 3DPW. The Motion VAE model is the best performing model that is
trained with all three forms of data augmentation techniques. Detailed
analysis about the effects of data augmentation can be found in 4.3.1. The
result shows that our VAE generalizes well to unseen sequences and the learned
subspace can represent the human motion space with reasonable quality.

Table 1: VAE Reconstruction Test Error on 3DPW dataset Here, the VAE is tasked
to encode and decode unseen motion sequences from the 3DPW dataset, and we
calculate our metrics between the ground truth and reconstructed sequences.
Motion sequences from different splits of 3DPW have varying difficulties, but
are all unseen by our VAE.

| 3DPW Train Split | 3DPW Val Split | 3DPW Test Split
---|---|---|---
MPJPE $\downarrow$ | PA-MPJPE $\downarrow$ | ACC-ERR $\downarrow$ | MPJPE $\downarrow$ | PA-MPJPE $\downarrow$ | ACC-ERR $\downarrow$ | MPJPE $\downarrow$ | PA-MPJPE $\downarrow$ | ACC-ERR $\downarrow$
Motion VAE | 72.7 | 52.3 | 8.9 | 72.1 | 52.9 | 9.3 | 58.7 | 43.9 | 8.2

####  4.2.3 Quantitative Results

The result in Table 2 shows that our method obtains state-of-the-art results
on video motion estimation across all three test datasets. Overall, our method
achieves comparable results in terms of position error (MPJPE and PA-MPJPE)
while significantly improving the smoothness (acceleration error), signifying
a smoother and more natural motion estimation without sacrificing accuracy.
Notice that all methods in italics have access to SMPL parameter annotation to
the H3.6M dataset, which has since been removed from the web due to legal
reasons. The SMPL parameters provide the most direct supervision for the task,
so the performance gain is significant especially on the H3.6M test set. For a
more direct comparison, we retrain the previous state-of-the-art method, VIBE
[1], using the official implementation with the exact same datasets as ours.
On 3DPW, under the same training condition, MEVA outperforms VIBE on almost
all three metrics while reducing the acceleration error by 54.3%, 59.3%, and
41.3 %, respectively. Even compared to VIBE trained with extra data, our model
achieves comparable results in accuracy while sporting a great reduction in
acceleration error, except for the H3.6M dataset. Note that the H3.6M dataset
contains mainly indoor scenes with limited background variation and models
trained with direct SMPL supervision tend to perform well on this dataset.
Compared to HMMR [7], which is the state-of-the-art on smoothness, our model
still achieves a smoother result (23.7% reduction in acceleration error) while
improving greatly in MPJPE by 25.4%.

Table 2: Testing error of state-of-the-art models on 3DPW, MPI-INF-3DHP, and
H3.6M. Here we compare with state-of-the-art methods on video pose estimation,
and report metrics on positional accuracy (MPJPE and PA-MPJPE) as well as
acceleration error. Notice that since an important annotation of H3.6M dataset
has since been made unavailable, we put all methods that are trained with such
supervision in italics. The most fair comparison is between our method and
VIBE [1], the previous best-performing model (trained using the same
datasets).

| 3DPW | MPI-INF-3DHP | H3.6M
---|---|---|---
MPJPE $\downarrow$ | PA-MPJPE $\downarrow$ | ACC-ERR $\downarrow$ | MPJPE $\downarrow$ | PA-MPJPE $\downarrow$ | ACC-ERR $\downarrow$ | MPJPE $\downarrow$ | PA-MPJPE $\downarrow$ | ACC-ERR $\downarrow$
HMR (w/ H3.6M SMPL) [8] | 130.0 | 76.7 | 37.4 | 124.2 | 89.8 | - | 88 | 56.8 | -
HMMR (w/ H3.6M SMPL) [7] | 116.5 | 72.6 | 15.2 | - | - | - | - | 56.9 | -
SPIN (w/ H3.6M SMPL) [3] | 96.9 | 59.2 | 29.8 | 105.2 | 67.5 | - | - | 41.1 | -
Sun et al. (w/ H3.6M SMPL) [2] | - | 69.5 | - | - | - | - | 59.1 | 42.4 | -
VIBE (w/ H3.6M SMPL) [1] | 82.9 | 51.9 | 23.4 | 96.6 | 64.6 | 31.2 | 65.6 | 41.4 | 27.3
VIBE (w/o H3.6M SMPL) [1] | 91.9 | 57.6 | 25.4 | 103.9 | 68.9 | 27.3 | 78.0 | 53.3 | 27.3
MEVA (w/o H3.6M SMPL) | 86.9 | 54.7 | 11.6 | 96.4 | 65.4 | 11.1 | 76.0 | 53.2 | 15.3

####  4.2.4 Qualitatively Results

As motions are best seen in videos, please refer to the supplementary video
for qualitative results. Overall, our model achieves better acceleration error
while preserving high joint position accuracy, resulting in an overall smooth
and natural motion.

###  4.3 Ablation Experiments

####  4.3.1 Effect of data augmentation for training the motion VAE

As mentioned in Sec.3.3.2, data augmentation performed on the AMASS dataset
significantly improves the generalizability of our motion VAE. Table 3
demonstrates the VAE reconstruction error on the unseen sequences from the
3DPW dataset (train/test/val), with various levels of data augmentation
techniques. Overall, RR (random root) rotation is essential since the motion
sequences in AMASS dataset, captured mostly in MoCap studio, have a single
fixed initial root rotation. Model trained only on AMASS would suffer greatly
when it encounters variation in root rotation. Changing the sampling frame
rate (FR) and flip left and right (LR) also provide a significant boost to
generalizability. Using all three augmentation techniques results in our best
performing motion VAE.

Table 3: Ablation of Data augmentation for VAE. Here we show the effects of
data augmentation on the VAE. RR: random root rotation, FR: different sampling
framerate, LR: left and right flip

| Train | Val | Test
---|---|---|---
MPJPE $\downarrow$ | PA-MPJPE $\downarrow$ | ACC-ERR $\downarrow$ | MPJPE $\downarrow$ | PA-MPJPE $\downarrow$ | ACC-ERR $\downarrow$ | MPJPE $\downarrow$ | PA-MPJPE $\downarrow$ | ACC-ERR $\downarrow$
No Aug | 200.2 | 86.0 | 15.9 | 181.7 | 90.5 | 15.8 | 172.8 | 75.2 | 14.1
RR | 146.3 | 78.0 | 12.1 | 133.9 | 81.2 | 13.0 | 115.0 | 70.4 | 11.0
FR + LR | 291.0 | 63.03 | 14.7 | 308.8 | 66.1 | 16.1 | 261.9 | 58.9 | 13.3
FR + LR + RR | 72.7 | 52.3 | 8.9 | 72.1 | 52.9 | 9.3 | 58.7 | 43.9 | 8.2

####  4.3.2 Coarse motion vs fine motion retrieval

MEVA benefits from an explicit breakdown of coarse and fine motion retrieval,
using a temporal compressive step that captures the overall motion in a given
human motion sequence. Just how much coarse/smooth motion information is
retrieved in the framework? Table 4 shows the result of MEVA if trained only
using the VME (capturing only coarse motion). Notice that MEVA with only VME
achieves result similar to HMMR [7], the previous state-of-the-art in
producing low acceleration error motion estimation. An illustrative
visualization of coarse and fine motion decomposition can be found in Fig.4.

Figure 4: Breakdown of coarse and fine human motion. The first row of
estimated human is the coarse part of the motion (output of VME), while the
second row adds back the fine details (output of MRR). Table 4: Ablation of
MEVA. Here we show the MEVA model trained with only Variational Motion
Estimator (without Motion Residual Regressor) or without using pretrained VAE.

| 3DPW
---|---
MPJPE $\downarrow$ | PA-MPJPE $\downarrow$ | ACC-ERR $\downarrow$
MEVA-VME only | 118.1 | 73.7 | 15.4
MEVA-without using pretrained VAE | 95.9 | 59.7 | 14.1
MEVA | 86.9 | 54.7 | 11.6

####  4.3.3 Effects of the pretrained Motion VAE

MEVA benefits from using a pretrained motion VAE’s latent space. As argued in
Sec.3.3, using a pretrained VAE provides a human motion subspace that assists
in constraining the estimated motion sequences to be natural and plausible
human motion. Table 4 shows the result of MEVA trained without using a
pretrained VAE (not using the pretrained $D_{vae}$). In this case, the whole
framework is trained end-to-end from scratch. Here we observe that the model
performed relatively well in both accuracy and smoothness, demonstrating the
power of our two-stage estimation framework. However, upon visual inspection,
as shown in Fig.5, a few kinematically invalid human poses are estimated
during the sequence, resulting in an overall accurate but flawed estimation.

Figure 5: MEVA results without using pretrained VAE. Here we show an ablative
study where we do not use a pretrained VAE. The second row shows MEVA result
without using a pretrained VAE and the first row shows our full model
estimation. Notice that the model estimates unnatural human poses in the first
few frames.

##  5 Conclusion

We have shown that to achieve temporally smooth and accurate 3D human pose
estimates, it is important to learn a compressive model that encodes the
smoothness of general human motion while also learning an image-based
regression model that can capture person-specific motion. We propose a two-
stage model that first trains a Variational Autoencoder to model the general
statistics of coarse/smooth human motion and then learns a person-specific
motion refinement regression module to retain motions not captured by the
general motion model. Through comprehensive experiments, we demonstrate that
our method produces both smooth and accurate motion.

Acknowledgements: This project was sponsored in part by IARPA
(D17PC00340), and JST AIP Acceleration Research Grant (JPMJCR20U1).

## References

  * [1] Kocabas, M., Athanasiou, N., Black, M.J.:  Vibe: Video inference for human body pose and shape estimation.  In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. (2020) 5253–5263
  * [2] Xu, Y., Zhu, S.C., Tung, T.:  Denserac: Joint 3d pose and shape estimation by dense render-and-compare.  2019 IEEE/CVF International Conference on Computer Vision (ICCV) (2019) 7759–7769
  * [3] Kolotouros, N., Pavlakos, G., Black, M.J., Daniilidis, K.:  Learning to reconstruct 3d human pose and shape via model-fitting in the loop.  2019 IEEE/CVF International Conference on Computer Vision (ICCV) (2019) 2252–2261
  * [4] Georgakis, G.V., Li, R., Karanam, S., Chen, T., Kosecka, J., Wu, Z.:  Hierarchical kinematic human mesh recovery.  ArXiv abs/2003.04232 (2020)
  * [5] Güler, R.A., Neverova, N., Kokkinos, I.:  Densepose: Dense human pose estimation in the wild.  2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (2018) 7297–7306
  * [6] Xu, Y., Zhu, S.C., Tung, T.:  Denserac: Joint 3d pose and shape estimation by dense render-and-compare.  2019 IEEE/CVF International Conference on Computer Vision (ICCV) (2019) 7759–7769
  * [7] Kanazawa, A., Zhang, J.Y., Felsen, P., Malik, J.:  Learning 3d human dynamics from video.  2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2019) 5607–5616
  * [8] Kanazawa, A., Black, M.J., Jacobs, D.W., Malik, J.:  End-to-end recovery of human shape and pose.  2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (2018) 7122–7131
  * [9] Hossain, M.R.I., Little, J.:  Exploiting temporal information for 3d human pose estimation.  In: ECCV. (2018)
  * [10] Mahmood, N., Ghorbani, N., Troje, N.F., Pons-Moll, G., Black, M.J.:  Amass: Archive of motion capture as surface shapes.  2019 IEEE/CVF International Conference on Computer Vision (ICCV) (2019) 5441–5450
  * [11] von Marcard, T., Henschel, R., Black, M.J., Rosenhahn, B., Pons-Moll, G.:  Recovering accurate 3D human pose in the wild using IMUs and a moving camera.  Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) 11214 LNCS (2018) 614–631
  * [12] Pavlakos, G., Choutas, V., Ghorbani, N., Bolkart, T., Osman, A.A.A., Tzionas, D., Black, M.J.:  Expressive body capture: 3d hands, face, and body from a single image.  2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2019) 10967–10977
  * [13] Loper, M., Mahmood, N., Romero, J., Pons-Moll, G., Black, M.J.:  Smpl: a skinned multi-person linear model.  ACM Trans. Graph. 34 (2015) 248:1–248:16
  * [14] Anguelov, D., Srinivasan, P., Koller, D., Thrun, S., Rodgers, J., Davis, J.:  SCAPE: Shape Completion and Animation of People.  ACM Transactions on Graphics 24 (2005) 408–416
  * [15] Grauman, K., Shakhnarovich, G., Darrell, T.:  Inferring 3D structure with a statistical image-based shape model.  Proceedings of the IEEE International Conference on Computer Vision 1 (2003) 641–648
  * [16] Agarwal, A., Triggs, B.:  Recovering 3D human pose from monocular images.  IEEE Transactions on Pattern Analysis and Machine Intelligence 28 (2006) 44–58
  * [17] Sigal, L., Balan, A., Black, M.J.:  Combined discriminative and generative articulated pose and non-rigid shape estimation.  Advances in Neural Information Processing Systems 20 - Proceedings of the 2007 Conference (2009) 1–8
  * [18] Zhou, S., Fu, H., Liu, L., Cohen-Or, D., Han, X.:  Parametric reshaping of human bodies in images.  ACM SIGGRAPH 2010 Papers, SIGGRAPH 2010 29 (2010) 1–10
  * [19] Peng Guan, Weiss, A., Bãlan, A.O., Black, M.J.:  Estimating human shape and pose from a single image.  In: 2009 IEEE 12th International Conference on Computer Vision. (2009) 1381–1388
  * [20] Bogo, F., Kanazawa, A., Lassner, C., Gehler, P., Romero, J., Black, M.J.:  Keep it SMPL: Automatic estimation of 3D human pose and shape from a single image.  Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) 9909 LNCS (2016) 561–578
  * [21] Omran, M., Lassner, C., Pons-Moll, G., Gehler, P., Schiele, B.:  Neural body fitting: Unifying deep learning and model based human pose and shape estimation.  Proceedings - 2018 International Conference on 3D Vision, 3DV 2018 (2018) 484–494
  * [22] Guler, R.A., Kokkinos, I.:  Holopose: Holistic 3D human reconstruction in-the-wild.  Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition 2019-June (2019) 10876–10886
  * [23] Tan, J.K.V., Budvytis, I., Cipolla, R.:  Indirect deep structured learning for 3D human body shape and pose prediction.  British Machine Vision Conference 2017, BMVC 2017 (2017) 1–11
  * [24] Tung, H.Y.F., Tung, H.W., Yumer, E., Fragkiadaki, K.:  Self-supervised learning of motion capture.  Advances in Neural Information Processing Systems 2017-December (2017) 5237–5247
  * [25] Pavlakos, G., Zhu, L., Zhou, X., Daniilidis, K.:  Learning to Estimate 3D Human Pose and Shape from a Single Color Image.  Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition (2018) 459–468
  * [26] Pavllo, D., Feichtenhofer, C., Grangier, D., Auli, M.:  3D human pose estimation in video with temporal convolutions and semi-supervised training.  Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition 2019-June (2019) 7745–7754
  * [27] Dabral, R., Mundhada, A., Kusupati, U., Afaque, S., Sharma, A., Jain, A.:  Learning 3D human pose from structure and motion.  Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) 11213 LNCS (2018) 679–696
  * [28] Xu, J., Yu, Z., Ni, B., Yang, J., Yang, X., Zhang, W.:  Deep kinematics analysis for monocular 3d human pose estimation.  In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). (2020)
  * [29] Mehta, D., Sridhar, S., Sotnychenko, O., Rhodin, H., Shafiei, M.M.h.M.M.h.M.M.h., Seidel, H.p., Casas, D., Theobalt, C., Shafiei, M.M.h.M.M.h.M.M.h., Seidel, H.p., Xu, W.:  VNect Real-time 3D Human Pose Estimation with a Single RGB Camera ACM Reference format VNect Real-time 3D Human Po.pdf.  Tog 36 (2017) 1–13
  * [30] Mehta, D., Sotnychenko, O., Mueller, F., Xu, W., Elgharib, M.A., Fua, P., Seidel, H.P., Rhodin, H., Pons-Moll, G., Theobalt, C.:  Xnect: Real-time multi-person 3d motion capture with a single rgb camera.  In: SIGGRAPH 2020. (2020)
  * [31] Ren, L., Patrick, A., Efros, A.A., Hodgins, J.K., Rehg, J.M.:  A data-driven approach to quantifying natural human motion.  In: SIGGRAPH 2005. (2005)
  * [32] Urtasun, R., Fleet, D.J., Fua, P.:  3D people tracking with Gaussian process dynamical models.  Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition 1 (2006) 238–245
  * [33] Ormoneit, D., Sidenbladh, H., Black, M.J., Hastie, T.:  Learning and tracking cyclic human motion.  Advances in Neural Information Processing Systems (2001)
  * [34] Wang, Z., Yu, P., Zhao, Y., Zhang, R., Zhou, Y., Yuan, J., Chen, C.:  Learning diverse stochastic human-action generators by learning smooth latent transitions.  ArXiv abs/1912.10150 (2020)
  * [35] Cai, H., Bai, C., Tai, Y.W., Tang, C.K.:  Deep video generation, prediction and completion of human action sequences.  In: The European Conference on Computer Vision (ECCV). (2018)
  * [36] Walker, J., Marino, K., Gupta, A., Hebert, M.:  The pose knows: Video forecasting by generating pose futures.  In: The IEEE International Conference on Computer Vision (ICCV). (2017)
  * [37] Plappert, M., Mandery, C., Asfour, T.:  Learning a bidirectional mapping between human whole-body motion and natural language using deep recurrent neural networks.  Robotics and Autonomous Systems 109 (2018) 13–26
  * [38] Ahuja, C., Morency, L.P.:  Language2pose: Natural language grounded pose forecasting.  2019 International Conference on 3D Vision (3DV) (2019) 719–728
  * [39] Yamada, T., Matsunaga, H., Ogata, T.:  Paired recurrent autoencoders for bidirectional translation between robot actions and linguistic descriptions.  IEEE Robotics and Automation Letters 3 (2018) 3441–3448
  * [40] Lee, H.Y., Yang, X., Liu, M.Y., Wang, T.C., Lu, Y.D., Yang, M.H., Kautz, J.:  Dancing to music.  In: NeurIPS. (2019)
  * [41] Fragkiadaki, K., Levine, S., Felsen, P., Malik, J.:  Recurrent network models for human dynamics.  Proceedings of the IEEE International Conference on Computer Vision 2015 Inter (2015) 4346–4354
  * [42] Yan, X., Rastogi, A., Villegas, R., Sunkavalli, K., Shechtman, E., Hadap, S., Yumer, E., Lee, H.:  MT-VAE: Learning Motion Transformations to Generate Multimodal Human Dynamics.  Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) 11209 LNCS (2018) 276–293
  * [43] Yuan, Y., Kitani, K.:  Diverse trajectory forecasting with determinantal point processes.  ArXiv abs/1907.04967 (2020)
  * [44] Butepage, J., Black, M.J., Kragic, D., Kjellstrom, H.:  Deep representation learning for human motion prediction and classification.  In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR). (2017)
  * [45] Yuan, Y., Kitani, K.M.:  Dlow: Diversifying latent flows for diverse human motion prediction.  ArXiv abs/2003.08386 (2020)
  * [46] Zhou, Y., Barnes, C., Lu, J., Yang, J., Li, H.:  On the continuity of rotation representations in neural networks.  2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2019) 5738–5746
  * [47] Kingma, D.P., Welling, M.:  Auto-encoding variational bayes.  2nd International Conference on Learning Representations, ICLR 2014 - Conference Track Proceedings (2014) 1–14
  * [48] Walker, J., Marino, K., Gupta, A., Hebert, M.:  The Pose Knows: Video Forecasting by Generating Pose Futures.  Proceedings of the IEEE International Conference on Computer Vision 2017-October (2017) 3352–3361
  * [49] Mehta, D., Rhodin, H., Casas, D., Fua, P., Sotnychenko, O., Xu, W., Theobalt, C.:  Monocular 3d human pose estimation in the wild using improved cnn supervision.  2017 International Conference on 3D Vision (3DV) (2017) 506–516
  * [50] Ionescu, C., Papava, D., Olaru, V., Sminchisescu, C.:  Human3.6M: Large scale datasets and predictive methods for 3D human sensing in natural environments.  IEEE Transactions on Pattern Analysis and Machine Intelligence 36 (2014) 1325–1339
  * [51] Zhang, W., Zhu, M., Derpanis, K.G.:  From actemes to action: A strongly-supervised representation for detailed action understanding.  Proceedings of the IEEE International Conference on Computer Vision (2013) 2248–2255
  * [52] Loper, M., Mahmoody, N., Blackz, M.J.:  MoSh: Motion and shape capture from sparse markers.  ACM Transactions on Graphics 33 (2014)
  * [53] Wang, Z., Yu, P., Zhao, Y., Zhang, R., Zhou, Y., Yuan, J., Chen, C.:  Learning diverse stochastic human-action generators by learning smooth latent transitions.  ArXiv abs/1912.10150 (2020)
  * [54] Cao, Z., Simon, T., Wei, S.E., Sheikh, Y.:  Realtime multi-person 2d pose estimation using part affinity fields.  In: CVPR. (2017)
  * [55] Huang, Y.:  Towards accurate marker-less human shape and pose estimation over time.  2017 International Conference on 3D Vision (3DV) (2017) 421–430
  * [56] Peng, X.B., Kanazawa, A., Malik, J., Abbeel, P., Levine, S.:  SFV: Reinforcement learning of physical skills from videos.  SIGGRAPH Asia 2018 Technical Papers, SIGGRAPH Asia 2018 37 (2018)

3D Human Motion Estimation via
Motion Compression and Refinement
Supplementary Material Zhengyi Luo S. Alireza Golestaneh Kris M. Kitani

This supplementary material is organized as follows. In Section 1, we provide
qualitative results for our proposed method (MEVA). In Section 2, we provide
complementary ablation studies. In Section 3, we will discuss the failure
modes of our method. In the last section, we include the details of our
implementation.

##  6 Qualitative Results

To best view our motion estimation and compare it with state-of-the-art,
please refer to the supplementary video.

Specifically, in the supplementary video, we first show a visual demonstration
of our two-stage decomposition of coarse and fine motion from a given video
sequence. Next, we demonstrate the qualitative comparison between our
algorithm and the prior state-of-the-art (VIBE[1]) and show that our method
achieves smoother, more natural, and accurate motion estimation. Finally, we
will discuss the implementation details of our method.

##  7 Additional Ablation Studies

###  7.1 Comparison with Average Filtering

While our method has significantly reduced the acceleration error and achieves
state-of-the-art accuracy, one can still apply postprocessing to existing
sequences to further improve the prediction. To best study its effects, here
we implement a simple average filter using spherical linear interpolation
(slerp) in quaternion. Specifically, for each joint rotation in SMPL $q^{i}$
at timestep $t$, we apply slerp with a ratio of 0.5:
$q^{i}_{t}=slerp(q^{i}_{t},q^{i}_{t+1},0.5)$. Table 5 shows the result of
applying averaging filtering as postprocessing on both VIBE [1] and MEVA. From
the results, it is clear that average filtering can help reduce the
acceleration error of both VIBE and MEVA while slightly affecting accuracy. It
is also conceivable that more sophisticated methods such as solving a
constrained optimization problem [55, 56, 29] can further improve results.
Nonetheless, in the paper we only compare with feed-forward methods without
any postprocessing, since postprocessing approaches are complementary to feed-
forward methods and would be beneficial to all of them.

Table 5: Ablation study on average filtering. Here we show the result of
applying the average filter on the output from VIBE [1] and MEVA.

| 3DPW
---|---
MPJPE $\downarrow$ | PA-MPJPE $\downarrow$ | ACC-ERR $\downarrow$
VIBE (w/o H3.6M SMPL) [1] | 91.9 | 57.6 | 25.4
VIBE (w/o H3.6M SMPL) [1] \+ Average Filtering | 91.6 | 57.8 | 13.5
MEVA (w/o H3.6M SMPL) (ours) | 86.9 | 54.7 | 11.6
MEVA (w/o H3.6M SMPL) + Average Filtering (ours) | 87.6 | 55.5 | 8.2

###  7.2 Effects of a long temporal window

MEVA uses a significantly longer temporal window (90 frames) than prior art
(HMMR [7]: 20 frames, VIBE[1]: 16 frames). To show that our MEVA framework
benefits more from this setting, we retrain VIBE with a 90 frames temporal
window. As shown in Table 6, using the same size temporal window, MEVA
produces better results on all three metrics and maintains a significant
advantage in acceleration error. Notice that VIBE trained with a longer
temporal window shows a slight improvement against the ones that use a shorter
window, validating our intuition that a longer temporal window provides a more
substantial context for motion estimation. Nonetheless, our two-stage
decomposition method is more effective in utilizing a longer temporal window
due to its separate motion compression and refinement stages.

Table 6: Ablation study on temporal window size. Here we show the results of
using different temporal windows in VIBE [1] and MEVA

| 3DPW
---|---
MPJPE $\downarrow$ | PA-MPJPE $\downarrow$ | ACC-ERR $\downarrow$
VIBE (w/o H3.6M SMPL) + 16 frames [1] | 91.9 | 57.6 | 25.4
VIBE (w/o H3.6M SMPL) + 90 frames [1] | 88.1 | 56.6 | 21.2
MEVA (w/o H3.6M SMPL) + 90 frames (ours) | 86.9 | 54.7 | 11.6

###  7.3 Effects of STE and VME on MEVA

Here we take a further look into the effects of different components (STE,
VME, and MRR) of our proposed method. Notice that without the Variational
Motion Estimator (VME), our method will collapse into a single-stage estimator
that only relies on the SMPL regressor, which has been studied extensively in
prior art. Thus, here we only study the effects of Spatial Temporal Feature
Extractor (STE) and Motion Refinement Regressor (MRR). Table 7 shows the
results of our framework trained without STE or MRR. Without the STE, MEVA
obtains high accuracy but suffers from high acceleration error. This indicates
that STE produces correlated features that impart the necessary temporal
consistency information to MRR. We reason that without STE, even although
initialized with coarse estimation from VME, MRR will be biased by the input
visual features and produce a temporally inconsistent refinement pose that
negatively affects the overall estimation. On the other hand, without MRR, our
method is reduced to one stage and only estimates the coarse motion. As shown
in the result, using only VME will lead to an overly smoothed motion
estimation and result in a higher acceleration error (underestimating movement
also leads to high acceleration error).

Table 7: Ablation of MEVA components. Here we show MEVA trained without STE
(with both VME and MRR) and without MRR (with both STE and VME).

| 3DPW
---|---
MPJPE $\downarrow$ | PA-MPJPE $\downarrow$ | ACC-ERR $\downarrow$
MEVA w/o STE | 89.7 | 55.4 | 29.0
MEVA w/o MRR | 118.1 | 73.7 | 15.4
MEVA | 86.9 | 54.7 | 11.6

##  8 Failure Modes

Although MEVA shows promising results in producing smooth and accurate human
motion, there is still room for improvement.

###  8.1 Sliding window processing

MEVA processes videos using a sliding window: input video sequences are
splited into chunks of 90 frames for processing. Due to the natural of this
sliding window approach, inconsistency can sometimes be observed at a 3 second
interval (videos are assumed to be at 30 fps). The explanation is as follows:
the coarse motion estimated by VME can be quite different between each
temporal window and MRR sometimes is unable to make enough adjustments to
account for a smooth transition. Each temporal window also has their own STE,
so the features from each window are longer correlated. Fig. 6 shows an
instance of this behavior. For visual inspection, please refer to our
supplement video.

Figure 6: Sliding window failure mode. This plot shows that at the
intersection of temporal windows, MEVA can result in a inaccurate transition
and bring a large acceleration error. Each green line in the plot marks a
temporal window, and there are large spikes of acceleration error at the first
two intersections.

###  8.2 Occluded body parts

Occluded body parts can still be challenging for MEVA. During occlusion, the
lack of visual indicators will compel MEVA to rely on coarse motion estimation
over the whole sequence and leads to a misscapture of detailed motion. Please
refer to the supplementary video for an example.

###  8.3 Missing hands and face movement

Since the original SMPL[13] model does not contain joints for the hand and
face, all methods using SMPL do not capture hand movements and facial
expressions. Moreover, there is not enough high quality 3D data that provides
hand and face annotations. A recent work [12] develops an enhanced SMPL model
that jointly models body pose, hands, and face, but this model has not gained
significant traction in the pose estimation community. We believe that
capturing hands and face movement in motion estimation is an essential
direction for future work.

##  9 Implementation Details

###  9.1 Human Motion VAE

The motion VAE’s encoder, $E_{vae}$, is a bidirectional Gated Recurrent Unit
(bi-GRU) with average pooling to obtain the temporal encoding $h$ of the
overall input motion sequence $M_{W}\in R^{W\times 144}$. We pass the temporal
encoding $h$ into a multilayer perceptron (MLP) with two hidden layers (1024,
512) and two heads to obtain the mean $\mu$ and variance $\sigma$ for the
latent code $z$. For the decoder $D_{vae}$, a forward GRU is used to decode
the output motion sequence. At each time step, the GRU takes in the previous
step estimation $\theta_{t-1}$ and the current latent code $z\in R^{1\times
S_{z}}$ to output a 512 latent feature. The feature is then passed through an
MLP with two hidden layers (1024, 512) to generate the reconstructed pose
$\theta\in R^{1\times 144}$

###  9.2  Spatio-Temporal Feature Extractor

For a video input, we first preprocess the video frames using a pretrained
ResNet-59 network [8]. The feature extractor outputs $f_{i}\in R^{2048}$ for
each frame. The extracted features within the same temporal window $W$ (we
choose $W=90$) are stacked together $[f_{t}]^{90}_{t=1}\in R^{90\times 2048}$
and are encoded by STE into a sequence of temporally correlated features
$[f^{\prime}_{t}]^{90}_{t=1}\in R^{90\times 2048}$. STE is a 2 layer bi-GRU
with hidden size 1024 that outputs a feature encoding at each timestep.
$E_{motion}$ that shares the same architecture as $E_{feat}$ except for the
final average pooling step to come up with a latent code $z\in R^{1\times
512}$ that represents the whole motion sequence.

###  9.3 Motion Residual Regressor

The MRR consists of 2 fully connected layers, each with 1024 neurons. It takes
in per-frame features and a set of initializing parameters (pose, shape, and
camera) and iterative refines its predictions (pose, shape, and camera) for
$k$ iterations.


# Dynamical Transition of Quantum Vortex-Pair Annihilation in a Bose-Einstein
Condensate

Toshiaki Kanai  Department of Physics, Washington University, St. Louis,
Missouri 63130, USA  Chuanwei Zhang  Email<EMAIL_ADDRESS>Department of Physics, Washington University, St. Louis, Missouri 63130, USA

###### Abstract

Understanding the elementary mechanism for the dissipation of vortex energy in
quantum liquids is one central issue in quantum hydrodynamics, such as quantum
turbulence in systems ranging from neutron stars to atomic condensates. In a
two-dimensional (2D) Bose-Einstein condensate (BEC) at zero temperature,
besides the vortex drift-out process from the boundary, vortex-antivortex pair
can annihilate in the bulk, but controversy remains on the number of vortices
involved in the annihilation process. We find there exists a dynamical
transition from four-body to three-body vortex annihilation processes with the
time evolution in a boundary-less uniform quasi-2D BEC. Such dynamical
transition depends on the initial vortex pair density, and occurs when the
sound waves generated in the vortex annihilation process surpass a critical
energy. With the confinement along the third direction is relaxed in a
quasi-2D BEC, the critical sound wave energy decreases due to the 3D vortex
line curve and reconnection, shifting the dynamical transition to the early
time. Our work reveals an elementary mechanism for the dissipation of vortex
energy that may help understand exotic matter and dynamics in quantum liquids.

Quantum vortices, ubiquitous topological excitations, play an important role
in diverse quantum phenomena such as Berezinskii-Kosterlitz-Thouless
transition [1, 4, 3, 2, 5, 6], Kibble–Zurek mechanism [7, 13, 14, 11, 8, 9,
10, 12], and quantum turbulence [15]. In quantum liquids, the quantization of
the superflow circulation around the vortex makes the vorticity robust against
decay. The understanding of vortex motion and their decaying mechanism is thus
fundamentally important and has attracted much attention recently,
particularly in 2D quantum turbulence [19, 28, 27, 29, 18, 17, 21, 23, 32, 31,
30, 24, 20, 25, 26, 22, 16]. In a 2D BEC, quantum vortices may disappear via
two processes: vortex drifting-out from the boundary and vortex-pair
annihilation in the bulk [33]. Here the vortex pair represents a pair of
vortex and antivortex with positive and negative single quantum of
circulations.

Figure 1: Schematics of the vortex-pair annihilation process. (a) Four-body
process: the first and second vortices form a soliton-like structure. The
third vortex is needed to conserve the vortex energy and momentum. This
soliton-like structure is robust in uniform 2D BECs, and the fourth vortex is
required to break the structure into sound waves. (b) Three-body process: the
pair of the first and second vortices dissipates to sound waves via the
interaction with the third vortex and sound waves.

So far the basic mechanism for the vortex-pair annihilation remains
inconclusive. While vortex-pair annihilation was initially considered as a
two-body process [33, 34], later studies utilizing the purely 2D Gross-
Pitaevskii equation (GPE) showed that it is a four-body process at zero
temperature [21, 35, 20], where the vortex pair forms a soliton-like structure
[37, 38, 36, 39] (i.e., a pair with a short vortex distance) through a three-
body interaction with the third vortex. This soliton-like structure is robust,
and the fourth vortex is needed to dissipate it to sound waves, as illustrated
in Fig. 1(a). However, recent work utilizing 3D GPE demonstrated that the
vortex-pair annihilation is a three-body process (Fig. 1(b)) in a uniform
quasi-2D BEC [40] with a strong confinement along the third direction. These
different results raise the natural question regarding the fundamental
mechanism behind the vortex-pair annihilation, particularly in the dimensional
crossover from 2D to 3D for a realistic quasi-2D BEC.

Figure 2: (a) Snapshots of density profile in a representative case with
$L_{z}=2\xi$ and the initial vortex number $N_{\mathrm{vor}}(0)=20^{2}$. These
plots are on the cross-section at $z=0$. The maximum value of the color bar is
$n_{\mathrm{M}}=2\times 10^{-5}N/\xi^{3}$. (b-d) Time evolution of the vortex
number decay. The initial vortex number is $N_{\mathrm{vor}}(0)=20^{2}$ (b),
$N_{\mathrm{vor}}(0)=16^{2}$ (c), and $N_{\mathrm{vor}}(0)=24^{2}$ (d). The
red and blue dashed lines correspond to four-body and three-body processes,
respectively.

In this Letter, we address this important question by simulating vortex decay
dynamics in a boundary-less uniform quasi-2D BEC utilizing the 3D GPE with
different confinements along the third direction. Here the boundary-less
condition prohibits the vortex drifting-out process in the 2D plane. Our main
results are:

i) The vortex annihilation process is governed by the sound wave intensity
characterized by a critical energy $E_{c}$, below (above) which the
annihilation process is four-body (three-body). With a low initial vortex
density, the vortex-pair annihilation remains a four-body process during the
dynamical evolution due to the low sound wave energy generated in the
annihilation process.

ii) In the high initial vortex density region, the vortex annihilation starts
from the four-body process. However, the generated sound wave energy from the
vortex annihilation exceeds $E_{c}$ after certain time $t$, leading to a
dynamical transition [41] from four-body to three-body processes.

iii) The critical sound wave energy decreases for weakening confinement along
the third direction. As the confinement becomes weaker, the vortex lines can
curve and reconnect along the third dimension, which reduce the effective
critical sound wave energy for annihilating the vortex pairs, shifting the
dynamical transition to an early time.

## Description of a quasi-2D BEC and numerical method.

Figure 3:  (a) Time evolution of the compressible kinetic energy
$E_{\mathrm{com}}$ with various initial vortex numbers $N_{\mathrm{vor}}(0)$
under the same $L_{z}=2\xi$. The dashed line represents the critical energy
$E_{\mathrm{C}}$ determined from $N_{\mathrm{vor}}(0)=20^{2}$. The three-body
scaling is observed when $E_{\mathrm{com}}>E_{\mathrm{C}}$. (b) Dependence of
$E_{\mathrm{C}}$ on the $z$-axis confinement $L_{z}$.
$N_{\mathrm{vor}}(0)=20^{2}$. The boundary between three and four-body
processes is drawn for illustration by connecting $E_{\mathrm{C}}$. With
larger $E_{\mathrm{com}}$, the vortex-pair annihilation may involve the two-
body process, where $\alpha$ is smaller than $3$.

We simulate the dynamics of a quasi-2D BEC by the 3D GPE [42]:

$i\hbar\frac{\partial\Psi(\bm{r},t)}{\partial
t}=-\frac{\hbar^{2}}{2m}\nabla^{2}\Psi+U(z)\Psi+C|\Psi|^{2}\Psi-\mu\Psi,$ (1)

where $\Psi$ is the macroscopic wave function, $\hbar$ is the reduced Plank
constant, $m$ is the atom mass, and $C=4\pi\hbar^{2}a_{s}/m$ is the
interaction strength with the $s$-wave scattering length $a_{s}$. For
simplicity, we assume the quasi-2D BEC is confined in the $z$-direction by a
box potential

$U(z)=\left\\{\begin{array}[]{ll}\infty&\hskip 10.0pt\left|z\right|\geq
L_{z}/2\\\ 0&\hskip 10.0pt\left|z\right|<L_{z}/2.\end{array}\right.$ (2)

The periodic boundary condition is assumed in the $x$-$y$ plane so that
quantum vortices may disappear only through the vortex-pair annihilation
process. Kelvin waves, i.e., transverse excitations along a vortex line, are
suppressed in thin BECs with a small $L_{z}$ [43], and such suppression is
often considered as the criteria of 2D in the context of quantum turbulence
[44].

The BEC density $n$ is given by $n=\left|\Psi\right|^{2}$, and the velocity
field $\bm{v}$ is computed by
$\bm{v}(\bm{r},t)=\frac{\hbar}{2mi}\frac{\Psi^{\ast}\bm{\nabla}\Psi-\Psi\bm{\nabla}\Psi^{\ast}}{n}$.
The BEC dynamics has the healing length scale $\xi={\hbar}/\sqrt{2m\mu_{0}}$
and time scale $\tau={\hbar}/{\mu_{0}}$ with the equilibrium chemical
potential $\mu_{0}=Cn_{0}$ and the average BEC density $n_{0}$. We solve the
GPE by the fourth-order Runge-Kutta-Gill’s method [45] with the spatial step
$\Delta x=0.25\xi$ and the time step $\Delta t=0.1\times 0.25^{2}\tau$. The
computational domain size is $L_{x}\times L_{y}\times
L_{z}=\left[0,256\xi\right]\times\left[0,256\xi\right]\times\left[-L_{z}/2,L_{z}/2\right]$.
In our simulations, $L_{z}$ varies from $\xi$ to $8\xi$, which are thin enough
for the 2D criteria. For the initial state, we arrange $N_{\mathrm{vor}}(0)/2$
positive and $N_{\mathrm{vor}}(0)/2$ negative vortices alternatively as a
square lattice in the $x$-$y$ plane, yielding the vortex density
$N_{\mathrm{vor}}(0)/\left(256\xi\right)^{2}$. To reduce the initial density
fluctuations, we generate the initial state by the imaginary time evolution
with the pinning potential located at vortex locations, hence the initial
state has no sound waves [46].

## Dynamical transition from four-body to three-body processes.

Fig. 2 (a) shows the density evolution of a BEC with $L_{z}=2\xi$ and the
initial vortex number $N_{\mathrm{vor}}(0)=20^{2}$. At $t=0$, the BEC has no
sound waves, and the vortices are placed as a square lattice with the lattice
constant $\sqrt{L_{x}L_{y}/N_{\mathrm{vor}}}=256\xi/20$. Here we slightly
shift the vortices randomly to break the discrete translational symmetry. As
the BEC evolves, the vortex pairs annihilate, emitting sound waves. The sound
waves are less in the early time $t=250\tau$, but the BEC becomes strongly
wavy in a later time $t=750\tau$.

The number of vortices involved in the vortex decay process can be
characterized by the scaling of the vortex number decay [35, 21]. For a
$\alpha$-body vortex decay process, the rate equation of the vortex number
$N_{\mathrm{vor}}$ is given by

$\frac{dN_{\mathrm{vor}}(t)}{dt}\propto-N_{\mathrm{vor}}^{\alpha},$ (3)

therefore the vortex number scales as

$N_{\mathrm{vor}}(t)\propto t^{-{1}/\left(\alpha-1\right)}.$ (4)

Fig. 2(b) shows the vortex number decay corresponding to Fig. 2(a). After
vortex pairs start to annihilate, the vortex number follows the scaling
$N_{\mathrm{vor}}(t)\propto t^{-1/3}$ at the early time, corresponding to the
four-body process. After certain time of evolution, the vortex number scaling
changes to $N_{\mathrm{vor}}\propto t^{-1/2}$, corresponding to the three-body
process. The transition depends on the initial vortex number (i.e., vortex
density): there is only four-body process for $N_{\mathrm{vor}}(0)=16^{2}$
(Fig. 2(c)), while the transition occurs earlier for
$N_{\mathrm{vor}}(0)=24^{2}$ (Fig. 2(d)). Such dynamical transition from four-
body to three-body processes have not been discussed before and understanding
the mechanism behind it is the main focus hereafter.

## Dynamical transition induced by increasing sound waves.

At finite temperatures, the thermal excitations dampening the kinetic energy
may change the scaling of $N_{\mathrm{vor}}$ [35, 21], but the GPE model here
has no thermal excitations. Sound waves can extract energy from quantum
vortices [36], and a complementary test of vortex-pair annihilation shown
later verifies the damping effect. In our case, intense sound waves generated
by the pair-annihilation in the early stage dampen the vortex energy later,
varying the scaling to a three-body process illustrated in Fig. 1(b).

Figure 4:  Time evolution of the fraction of the $z$ component contribution
$E_{\mathrm{com},z}/E_{\mathrm{com}}$ with various confinement $L_{z}$.

To characterize the sound wave intensity, we decompose the velocity field as
$\bm{v}=\bm{v}_{\mathrm{com}}+\bm{v}_{\mathrm{inc}}$ with the incompressible
velocity $\bm{v}_{\mathrm{inc}}$ and the compressible velocity
$\bm{v}_{\mathrm{com}}$, satisfying
$\bm{\nabla}\cdot\left(\sqrt{n}\bm{v}_{\mathrm{inc}}\right)=0$ and
$\bm{\nabla}\times\left(\sqrt{n}\bm{v}_{\mathrm{com}}\right)=0$, respectively.
Then the compressible kinetic energy $E_{\mathrm{com}}$ and the incompressible
kinetic energy $E_{\mathrm{inc}}$ can be computed by [49, 47, 48]

$\displaystyle E_{\mathrm{com}}$ $\displaystyle=$
$\displaystyle\frac{m}{2}\int n\left|\bm{v}_{\mathrm{com}}\right|^{2}dV$ (5)
$\displaystyle E_{\mathrm{inc}}$ $\displaystyle=$
$\displaystyle\frac{m}{2}\int n\left|\bm{v}_{\mathrm{inc}}\right|^{2}dV.$ (6)

$E_{\mathrm{com}}$ and $E_{\mathrm{inc}}$ are associated with sound waves and
vortices, respectively [49, 48]. Fig. 3 (a) shows the $E_{\mathrm{com}}$
evolution with various initial vortex numbers. $E_{\mathrm{com}}$ steadily
increases as more vortex-pairs annihilate. In the case of
$N_{\mathrm{vor}}(0)=20^{2}$, the three-body process emerges after about
$t=520\tau$, when the compressible kinetic energy is about
$E_{\mathrm{com}}\approx 9.7\times 10^{-3}N\mu_{0}$ with the total particle
number $N=\int ndV$. This compressible kinetic energy is defined as the
critical energy $E_{\mathrm{C}}$. With $N_{\mathrm{vor}}(0)=16^{2}$,
$E_{\mathrm{com}}$ is always lower than $E_{\mathrm{C}}$ in our simulation,
and the scaling of $N_{\mathrm{vor}}$ follows the four-body process during the
evolution (Fig. 2 (c)). For $N_{\mathrm{vor}}(0)=24^{2}$, $E_{\mathrm{com}}$
quickly increases and becomes higher than $E_{\mathrm{C}}$, inducing the
dynamical transition early in time (Fig. 2 (d)). $E_{\mathrm{C}}$ is estimated
from $N_{\mathrm{vor}}(0)=20^{2}$ but matches the transition time for
$N_{\mathrm{vor}}(0)=24^{2}$, indicating that the critical energy is mostly
independent of the initial vortex number in our parameter range.

## Confinement dependence in the quasi-2D BEC.

When $L_{z}$ increases, we expect a crossover from 2D to quasi-2D and then 3D
for the BEC. For each $L_{z}$, we sample three cases from different initial
states with the same initial vortex number $N_{\mathrm{vor}}(0)=20^{2}$ and
compute the arithmetic-averaged critical energy $\bar{E}_{\mathrm{C}}$ shown
in Fig. 3 (b). We see $\bar{E}_{\mathrm{C}}$ is almost a constant below
$L_{z}=2\xi$ and starts decreasing beyond $L_{z}=2\xi$. The decrease of
$\bar{E}_{\mathrm{C}}$ cannot attribute to Kelvin waves since they are still
suppressed even for $L_{z}=8\xi$. One intuitive guess is the increase of the
sound waves in the $z$-direction, which may be evaluated by the compressible
kinetic energy in the $z$-direction $E_{\mathrm{com},z}=\frac{m}{2}\int
n\left|\bm{v}_{\mathrm{com}}\cdot\bm{\hat{e}}_{z}\right|^{2}dV$ with the unit
vector in the $z$-direction $\hat{e}_{z}$. Fig. 3 (b) shows the fraction
$E_{\mathrm{com},z}/E_{\mathrm{com}}$. In our parameter range $\xi\leq
L_{z}\leq 8\xi$, $E_{\mathrm{com},z}$ is always less than $0.2\%$ of
$E_{\mathrm{com}}$, demonstrating that the $z$-component of sound waves is
negligible. Therefore, the change of the critical energy should come from the
vortex contribution.

Figure 5:  Vorticity evolution of a vortex reconnection event in a BEC
simulation. $L_{z}=8\xi$. The red, white, and blue lines represent the
vorticity at $t=350\tau$, $355\tau$, and $360\tau$, respectively.

We find that the decrease of $\bar{E}_{\mathrm{C}}$ with increasing $L_{z}$ is
due to the emergence of vortex reconnection channel. As $L_{z}$ increases,
vortices can be easily curved, and the vortex distance may vary in the
$z$-direction. Thus the soliton-like structure becomes unstable, enhancing the
vortex reconnection channel that makes the sound wave easier to annihilate the
vortex pair. Fig. 5 shows snapshots of a vortex reconnection event in a BEC
with $L_{z}=8\xi$. To quantitatively characterize this effect, we analyze the
incompressible kinetic energy of the $z$-component

$E_{\mathrm{inc},z}=\frac{m}{2}\int
n\left|\bm{v}_{\mathrm{inc}}\cdot\bm{\hat{e}}_{z}\right|^{2}dV.$ (7)

If the vortices do not curve, the incompressible velocity is confined in the
$x$-$y$ plane, and $E_{\mathrm{inc},z}$ is negligible. Fig. 6 (a) shows the
time evolution of the ratio between $E_{\mathrm{inc},z}$ and
$E_{\mathrm{inc}}$, which can be over $10\%$ with $L_{z}=8\xi$ that is a
significant contribution. Fig. 6 (b) shows the $L_{z}$ dependence of the ratio
at $t=600\tau$. The contribution of the z-component is suppressed below
$L_{z}=2\xi$ and rapidly increases above $L_{z}=3\xi$. This result means that
the vortex reconnection effect becomes significant above $L_{z}=3\xi$,
agreeing with the observation in the critical energy observed in Fig. 3 (b).

Figure 6:  (a) Time evolution of the fraction of the $z$ component
contribution $E_{\mathrm{inc},z}/E_{\mathrm{inc}}$ with various $L_{z}$ for
$N_{\mathrm{vor}}=20^{2}$. (b) Dependence of the fraction
$E_{\mathrm{inc},z}/E_{\mathrm{inc}}$ on $L_{z}$ at $t=600\tau$.

## Complementary test of vortex-pair annihilation by sound waves.

To directly demonstrate the energy damping caused by sound waves, the dynamics
of a vortex-pair are analyzed for different cases. In case 1, the initial BEC
has no sound waves, and we implement a vortex-pair with a vortex distance
$d_{\mathrm{V}}(t=0)=3.5\xi$. Here the initial density profile is given by the
two-point Padé approximation [50]. In case 2, the BEC contains the
compressible kinetic energy $E_{\mathrm{com}}\approx 2.5\times
10^{-2}N\mu_{0}$, which is high enough so that the vortex-pair annihilation
may be a three-body process. We then implement a vortex-pair in the same way
as case 1. Fig. 7 shows the time evolution of the vortex distance
$d_{\mathrm{V}}(t)$, which is associated with the vortex energy.
$d_{\mathrm{V}(t)}$ is almost constant in case 1, but rapidly varies over a
short period and gradually decays over a long time in case 2, indicating that
sound waves may fluctuate the vortex-pair energy and gradually dissipate it.
Note that the energy dissipation due to sound waves does not need to be strong
enough to dampen out the vortex-pair energy by itself since their role is to
destabilize the metastable state. In fact, weak energy dissipation of thermal
excitations’ scattering may alter the vortex-pair annihilation to a two-body
process [35, 21].

Figure 7: Time evolution of the vortex distance $d_{V}(t)$ in two cases. Case
1 has no initial sound waves, and case 2 has intense sound waves of the
compressible kinetic energy $E_{\mathrm{C}}\approx 2.5\times 10^{-2}N\mu_{0}$
at $t=0$. $L_{z}=4\xi$. Here $d_{V}$ is computed for every $t$, and a moving
average is taken within [$t-10\tau$, $t+10\tau$].

## Discussion and conclusion.

Past studies utilizing the 2D GPE used relatively low initial vortex
densities, therefore the three-body process does not emerge (e.g.,
$N_{vor}(0)/\left(L_{x}L_{y}\right)=10^{3}/10^{4}\xi^{-2}\approx
153/256^{2}\xi^{-2}$ in [35]). On the other hand, the 3D GPE study of the
quasi-2D BEC [40] did not observe the four-body process because of the
relatively weak confinement in the third direction, leading to a very small
critical energy. As a result, the vortex decay dynamics quickly pass the
transition point from four-body to three-body processes at an early time.

Our simulations focus on the dynamics at the absolute zero temperature, but
rich physics is expected at finite temperatures due to the scattering with
thermal excitations [35, 21]. Quasi-2D BECs have been realized in many
experiments [2, 51] with the tunable third dimension confinement. The uniform
BEC on the 2D plane can be achieved using the box types of potential [52, 53].
It is also worth noting that vortex collision can be controlled by localized
optical potentials [54], which allow us to measure the sound wave dissipation
to the vortex-pair directly and locally.

In summary, we have demonstrated that the vortex-pair annihilation process
undergoes a dynamical transition from four-body to three-body processes due to
increasing sound waves generated in annihilation process in the early stage.
We characterize the role of the initial vortex density and the third dimension
confinement in the quasi-2D BEC for such dynamical transition. Our work
reveals an elementary mechanism for vortex energy dissipation through pairing
annihilation that has been a long-standing issue in the field. Understanding
such vortex dissipation dynamics allows better characterization of dynamical
phenomena (e.g., quantum turbulence) of quantum liquids for new quantum
technologies applications.

###### Acknowledgements.

_Acknowledgement:_ The authors acknowledge the support of the Air Force Office
of Scientific Research under Grant No. FA9550-20-1-0220 and the National
Science Foundation under Grant No. PHY-2409943, OSI-2228725, ECCS-2411394.

## References

  * [1] S. Stock, Z. Hadzibabic, B. Battelier, M. Cheneau, and J. Dalibard, Observation of Phase Defects in Quasi-Two-Dimensional Bose–Einstein Condensates, Phys. Rev. Lett. 95, 190403 (2005).
  * [2] Z. Hadzibabic, P. Krüger, M. Cheneau, B. Battelier, and J. Dalibard, Berezinskii–Kosterlitz–Thouless crossover in a trapped atomic gas, Nature 441, 1118-1121 (2006).
  * [3] V. Schweikhard, S. Tung, and E. A. Cornell, Vortex Proliferation in the Berezinskii-Kosterlitz-Thouless Regime on a Two-Dimensional Lattice of Bose-Einstein Condensates, Phys. Rev. Lett. 99, 030401 (2007).
  * [4] P. Krüger, Z. Hadzibabic, and J. Dalibard, Critical Point of an Interacting Two-Dimensional Atomic Bose Gas, Phys. Rev. Lett. 99, 040402 (2007).
  * [5] S. Nazarenko, M. Onorato, and D. Proment, Bose-Einstein condensation and Berezinskii-Kosterlitz-Thouless transition in the two-dimensional nonlinear Schrödinger model, Phys. Rev. A 90, 013624 (2014).
  * [6] S. Sunami, V. P. Singh, D. Garrick, A. Beregi, A. J. Barker, K. Luksch, E. Bentine, L. Mathey, and C. J. Foot, Observation of the Berezinskii-Kosterlitz-Thouless Transition in a Two-Dimensional Bose Gas via Matter-Wave Interferometry, Phys. Rev. Lett. 128, 250402 (2022).
  * [7] D. R. Scherer, C. N. Weiler, T. W. Neely, and B. P. Anderson, Vortex Formation by Merging of Multiple Trapped Bose-Einstein Condensates, Phys. Rev. Lett. 98, 110402 (2007).
  * [8] C. N. Weiler, T. W. Neely, D. R. Scherer, A. S. Bradley, M. J. Davis, and B. P. Anderson, Spontaneous vortices in the formation of Bose–Einstein condensates, Nature 455, 948–951 (2008).
  * [9] T. Kanai, W. Guo, and M. Tsubota, Flows with fractional quantum circulation in Bose-Einstein condensates induced by nontopological phase defects, Phys. Rev. A 97, 013612 (2018).
  * [10] T. Kanai, W. Guo, and M. Tsubota, Merging of Rotating Bose–Einstein Condensates, J. Low Temp. Phys. 195, 37–50 (2019).
  * [11] B. Ko, J. W. Park, and Y. Shin, Kibble–Zurek universality in a strongly interacting Fermi superfluid, Nat. Phys. 15, 1227–1231 (2019).
  * [12] T. Kanai, W. Guo, M. Tsubota, and D. Jin, Torque and Angular-Momentum Transfer in Merging Rotating Bose-Einstein Condensates, Phys. Rev. Lett. 124, 105302 (2020).
  * [13] J. Goo, Y. Lim, and Y. Shin, Defect Saturation in a Rapidly Quenched Bose Gas, Phys. Rev. Lett. 127, 115701 (2021).
  * [14] J. Rysti, J. T. Mäkinen, S. Autti, T. Kamppinen, G. E. Volovik, and V. B. Eltsov, Suppressing the Kibble-Zurek Mechanism by a Symmetry-Violating Bias, Phys. Rev. Lett. 127, 115702 (2021).
  * [15] C. F. Barenghi, L. Skrbek, and K. R. Sreenivasan, Quantum turbulence, (Cambridge University Press, Cambridge, 2024).
  * [16] J. Schole, B. Nowak, and T. Gasenzer, Critical dynamics of a two-dimensional superfluid near a nonthermal fixed point, Phys. Rev. A 86, 013624 (2012).
  * [17] T. P. Billam, M. T. Reeves, B. P. Anderson, and A. S. Bradley, Onsager-Kraichnan Condensation in Decaying Two-Dimensional Quantum Turbulence, Phys. Rev. Lett. 112, 145301 (2014).
  * [18] T. Simula, M. J. Davis, and K. Helmerson, Emergence of Order from Turbulence in an Isolated Planar Superfluid, Phys. Rev. Lett. 113, 165302 (2014).
  * [19] A. Lucas and P. Surówka, Sound-induced vortex interactions in a zero-temperature two-dimensional superfluid, Phys. Rev. A 90, 053617 (2014).
  * [20] A. Cidrim, F. E. A. dos Santos, L. Galantucci, V. S. Bagnato, and C. F. Barenghi, Controlled polarization of two-dimensional quantum turbulence in atomic Bose-Einstein condensates, Phys. Rev. A 93, 033651 (2016).
  * [21] A. J. Groszek, T. P. Simula, D. M. Paganin, and K. Helmerson, Onsager vortex formation in Bose–Einstein condensates in two-dimensional power-law traps, Phys. Rev. A 93, 043614 (2016).
  * [22] X. Yu, T. P. Billam, J. Nian, M. T. Reeves, and A. S. Bradley, Theory of the vortex-clustering transition in a confined two-dimensional quantum fluid, Phys. Rev. A 94, 023602 (2016).
  * [23] S. W. Seo, B. Ko, J. H. Kim, and Y. Shin, Observation of vortex-antivortex pairing in decaying 2D turbulence of a superfluid gas, Sci. Rep. 7, 4587 (2017).
  * [24] M. Karl and T. Gasenzer, Strongly anomalous non-thermal fixed point in a quenched two-dimensional Bose gas, New J. Phys. 19, 093014 (2017).
  * [25] A. J. Groszek, M. J. Davis, D. M. Paganin, K. Helmerson, and T. P. Simula, Vortex Thermometry for Turbulent Two-Dimensional Fluids, Phys. Rev. Lett. 120, 034504 (2018).
  * [26] A. Frishman and C. Herbert, Turbulence Statistics in a Two-Dimensional Vortex Condensate, Phys. Rev. Lett. 120, 204505 (2018).
  * [27] G. Gauthier, M. T. Reeves, X. Yu, A. S. Bradley, M. A. Baker, T. A. Bell, H. Rubinsztein-Dunlop, M. J. Davis, and T. W. Neely, Giant vortex clusters in a two-dimensional quantum fluid, Science 364, 1264-1267 (2019).
  * [28] S. P. Johnstone, A. J. Groszek, P. T. Starkey, C. J. Billington, T. P. Simula, and K. Helmerson, Evolution of large-scale flow from turbulence in a two-dimensional superfluid, Science 364, 1267-1271 (2019).
  * [29] Y. P. Sachkou, C. G. Baker, G. I. Harris, O. R. Stockdale, S. Forstner, M. T. Reeves, X. He, D. L. McAuslan, A. S. Bradley, M. J. Davis, and W. P. Bowen, Coherent vortex dynamics in a strongly interacting superfluid on a silicon chip, Science 366, 1480-1485 (2019).
  * [30] A. J. Groszek, M. J. Davis, and T. P. Simula, Decaying quantum turbulence in a two-dimensional Bose-Einstein condensate at finite temperature, SciPost Phys. 8, 039 (2020).
  * [31] M. T. Reeves, K. Goddard-Lee, G. Gauthier, O. R. Stockdale, H. Salman, T. Edmonds, X. Yu, A. S. Bradley, M. Baker, H. Rubinsztein-Dunlop, M. J. Davis, and T. W. Neely, Turbulent Relaxation to Equilibrium in a Two-Dimensional Quantum Vortex Gas, Phys. Rev. X 12, 011031 (2022).
  * [32] T. Easton, M. Kokmotos, and G. Barontini, Vortex clustering in trapped Bose–Einstein condensates, Sci. Rep. 13, 19432 (2023).
  * [33] W. J. Kwon, G. Moon, J. Choi, S. W. Seo, and Y. Shin, Relaxation of superfluid turbulence in highly oblate Bose-Einstein condensates, Phys. Rev. A 90, 063627 (2014).
  * [34] G. W. Stagg, A. J. Allen, N. G. Parker, and C. F. Barenghi, Generation and decay of two-dimensional quantum turbulence in a trapped Bose-Einstein condensate, Phys. Rev. A 91, 013612 (2015).
  * [35] A. W. Baggaley and C. F. Barenghi, Decay of homogeneous two-dimensional quantum turbulence, Phys. Rev. A 97, 033601 (2018).
  * [36] S. Nazarenko and M. Onorato, Freely decaying Turbulence and Bose–Einstein Condensation in Gross–Pitaevski Model, J. Low Temp. Phys. 146, 31-46 (2007).
  * [37] L. A. Smirnov and V. A. Mironov, Dynamics of two-dimensional dark quasisolitons in a smoothly inhomogeneous Bose-Einstein condensate, Phys. Rev. A 85, 053620 (2012).
  * [38] L. A. Smirnov and A. I. Smirnov, Scattering of two-dimensional dark solitons by a single quantum vortex in a Bose-Einstein condensate, Phys. Rev. A 92, 013636 (2015).
  * [39] E. Rickinson, N. G. Parker, A. W. Baggaley, and C. F. Barenghi, Diffusion of quantum vortices, Phys. Rev. A 98, 023608 (2018).
  * [40] T. Kanai and W. Guo, True Mechanism of Spontaneous Order from Turbulence in Two-Dimensional Superfluid Manifolds, Phys. Rev. Lett. 127, 095301 (2021).
  * [41] T. Ma and S. Wang, Phase Transition Dynamics, (Springer International Publishing, Cham, 2019).
  * [42] C. J. Pethick and H. Smith, Bose-Einstein condensation in dilute gases, 2nd ed., (Cambridge University Press, Cambridge, 2008).
  * [43] S. J. Rooney, P. B. Blakie, B. P. Anderson, and A. S. Bradley, Suppression of Kelvon-induced decay of quantized vortices in oblate Bose-Einstein condensates, Phys. Rev. A 84, 023637 (2011).
  * [44] L. Chomaz, L. Corman, T. Bienaimé, R. Desbuquois, C. Weitenberg, S. Nascimbène, J. Beugnon, and J. Dalibard, Emergence of coherence via transverse condensation in a uniform quasi-two-dimensional Bose gas, Nat Commun 6, 6162 (2015).
  * [45] M. Abramowitz and I. A. Stegun (Eds.), Handbook of mathematical functions: with formulas, graphs, and mathematical tables, (Dover Publ, New York, NY, 2013).
  * [46] R. Boral, S. Sarkar, and P. K. Mishra, Structural Transformation and Melting of the Vortex Lattice in the Rotating Bose Einstein Condensates. In: Banerjee, S., Saha, A. (eds) Nonlinear Dynamics and Applications. Springer Proceedings in Complexity. Springer, Cham. https://link.springer.com/10.1007/978-3-030-99792-2_106
  * [47] R. Numasato, M. Tsubota, and V. S. L’vov, Direct energy cascade in two-dimensional compressible quantum turbulence, Phys. Rev. A 81, 063630 (2010).
  * [48] A. S. Bradley and B. P. Anderson, Energy Spectra of Vortex Distributions in Two-Dimensional Quantum Turbulence, Phys. Rev. X 2, 041001 (2012).
  * [49] T.-L. Horng, C.-H. Hsueh, S.-W. Su, Y.-M. Kao, and S.-C. Gou, Two-dimensional quantum turbulence in a nonuniform Bose–Einstein condensate, Phys. Rev. A 80, 023618 (2009).
  * [50] C. Rorai, K. R. Sreenivasan, and M. E. Fisher, Propagating and annihilating vortex dipoles in the Gross-Pitaevskii equation, Phys. Rev. B 88, 134522 (2013).
  * [51] C.-L. Hung, X. Zhang, N. Gemelke, and C. Chin, Observation of scale invariance and universality in two-dimensional Bose gases, Nature 470, 236 (2011).
  * [52] A. L. Gaunt, T. F. Schmidutz, I. Gotlibovych, R. P. Smith, Z. Hadzibabic, Bose-Einstein condensation of atoms in a uniform potential, Phys. Rev. Lett. 110, 200406 (2013).
  * [53] B. Mukherjee, Z. Yan, P. B. Patel, Z. Hadzibabic, T. Yefsah, J. Struck, and M. W. Zwierlein, Homogeneous Atomic Fermi Gases, Phys. Rev. Lett. 118, 123401 (2017).
  * [54] W. J. Kwon, G. Del Pace, K. Xhani, L. Galantucci, A. Muzi Falconi, M. Inguscio, F. Scazza, and G. Roati, Sound emission and annihilations in a programmable quantum vortex collider, Nature 600, 64 (2021).


An adaptive model reduction method leveraging
locally supported basis functions

rvt1]Han Gaofn1

rvt2]Matthew J. Zahrfn2cor1

[rvt1]School of Engineering and Applied Science, Harvard University, Cambridge, MA  02138, United States

[rvt2]Department of Aerospace and Mechanical Engineering, University
of Notre Dame, Notre Dame, IN 46556, United States

[cor1]Corresponding author

[fn1]Postdoctoral Scholar, School of Engineering and Applied Science, Harvard University
[fn2]Assistant Professor, Department of Aerospace and Mechanical
Engineering, University of Notre Dame

model reduction,
domain decomposition,
parametric robustness,
finite element method,
error estimation,
basis adaptation

We propose a new method, the continuous Galerkin method with globally and locally supported basis functions (CG-GL), to address the parametric robustness issues of reduced-order models (ROMs) by incorporating solution-based adaptivity with locally supported finite element basis functions. The CG-GL method combines the accuracy of locally supported basis functions with the efficiency of globally supported data-driven basis functions. Efficient output-based dual-weighted residual error estimates are derived and implemented for the CG-GL method and used to drive efficient online trial space adaptation. An empirical quadrature procedure is introduced for rapid evaluation of nonlinear terms that does not require retraining throughout the adaptation process. Two numerical experiments demonstrate the potential of the CG-GL method to produce accurate approximations with limited training and its tunable tradeoff between accuracy and computational cost.

§ ACKNOWLEDGMENTS

This material is based upon work supported by the Air Force Office of
Scientific Research (AFOSR) under award numbers FA9550-20-1-0236
and FA9550-22-1-0004. The content of this publication does not necessarily
reflect the position or policy of any of these supporters, and no official
endorsement should be inferred.

[1]
Anshul Agarwal and Lorenz T. Biegler.
A trust-region framework for constrained optimization using reduced
order modeling.
Optimization and Engineering, 14(1):3–35, March 2013.

[2]
David Amsallem and Charbel Farhat.
Interpolation method for adapting reduced-order models and
application to aeroelasticity.
AIAA journal, 46(7):1803–1813, 2008.

[3]
David Amsallem, Matthew J Zahr, and Charbel Farhat.
Nonlinear model order reduction based on local reduced-order bases.
International Journal for Numerical Methods in Engineering,
92(10):891–916, 2012.

[4]
E. Arian, M. Fahl, and E. W. Sachs.
Trust-Region Proper Orthogonal Decomposition for Flow
Technical Report ICASE-2000-25, Institue for Computer Applications in
Science and Engineering, May 2000.

[5]
Joshua Barnett and Charbel Farhat.
Quadratic approximation manifold for mitigating the kolmogorov
barrier in nonlinear projection-based model order reduction.
Journal of Computational Physics, 464:111348, 2022.

[6]
Roland Becker and Rolf Rannacher.
An optimal control approach to a posteriori error estimation in
finite element methods.
Acta numerica, 10:1–102, 2001.

[7]
Peter Benner, Serkan Gugercin, and Karen Willcox.
A survey of projection-based model reduction methods for parametric
dynamical systems.
SIAM Review, 57(4):483–531, 2015.

[8]
Annalisa Buffa, Yvon Maday, Anthony T. Patera, Christophe Prud'homme, and
Gabriel Turinici.
A priori convergence of the Greedy algorithm for the
parametrized reduced basis method.
ESAIM: Mathematical Modelling and Numerical Analysis,
46(3):595–603, May 2012.

[9]
Kevin Carlberg.
Adaptive $h$-refinement for reduced-order models.
International Journal for Numerical Methods in Engineering,
102(5):1192–1210, 2015.

[10]
Saifon Chaturantabut and Danny Sorensen.
Nonlinear model reduction via discrete empirical interpolation.
SIAM Journal on Scientific Computing, 32(5):2737–2764, 2010.

[11]
Markus Dihlmann, Martin Drohmann, and Bernard Haasdonk.
Model reduction of parametrized evolution problems using the reduced
basis method with adaptive time-partitioning.
In Proceedings of ADMOS, page 64, 2011.

[12]
Charbel Farhat, Philip Avery, Todd Chapman, and Julien Cortial.
Dimensional reduction of nonlinear finite element dynamic models with
finite rotations and energy-based mesh sampling and weighting for
computational efficiency.
International Journal for Numerical Methods in Engineering,
98(9):625–662, 2014.

[13]
Krzysztof J Fidkowski and David L Darmofal.
Review of output-based error estimation and mesh adaptation in
computational fluid dynamics.
AIAA journal, 49(4):673–694, 2011.

[14]
Bernard Haasdonk, Markus Dihlmann, and Mario Ohlberger.
A training set and multiple bases generation approach for
parameterized model reduction based on adaptive grids in parameter space.
Mathematical and Computer Modelling of Dynamical Systems,
17(4):423–442, August 2011.

[15]
Cheng Huang and Karthik Duraisamy.
Predictive reduced order modeling of chaotic multi-scale problems
using adaptively sampled projections.
arXiv preprint arXiv:2301.09006, 2023.

[16]
Pierre Kerfriden, Pierre Gosselet, Sondipon Adhikari, and Stephane Pierre-Alain
Bridging proper orthogonal decomposition methods and augmented
newton–krylov algorithms: an adaptive model order reduction for highly
nonlinear mechanical problems.
Computer Methods in Applied Mechanics and Engineering,
200(5-8):850–866, 2011.

[17]
Kookjin Lee and Kevin T Carlberg.
Model reduction of dynamical systems on nonlinear manifolds using
deep convolutional autoencoders.
Journal of Computational Physics, 404:108973, 2020.

[18]
David J. Lucia, Paul I. King, and Philip S. Beran.
Reduced order modeling of a two-dimensional flow with moving shocks.
Computers & Fluids, 32(7):917–938, August 2003.

[19]
Marzieh Alireza Mirhoseini and Matthew J Zahr.
Model reduction of convection-dominated partial differential
equations via optimization-based implicit feature tracking.
Journal of Computational Physics, 473:111739, 2023.

[20]
Nirmal J. Nair and Maciej Balajewicz.
Transported snapshot model order reduction approach for parametric,
steady-state fluid flows containing parameter-dependent shocks: Model order
reduction for fluid flows containing shocks.
International Journal for Numerical Methods in Engineering,
117(12):1234–1262, March 2019.

[21]
Mario Ohlberger and Stephan Rave.
Reduced Basis Methods: Success, Limitations and Future
In Proceedings of the Conference Algoritmy, 2016.

[22]
Benjamin Peherstorfer.
Model reduction for transport-dominated problems via online adaptive
bases and adaptive sampling.
SIAM Journal on Scientific Computing, 42(5):A2803–A2836,
January 2020.

[23]
Sébastien Riffaud, Michel Bergmann, Charbel Farhat, Sebastian Grimberg, and
Angelo Iollo.
The dgdd method for reduced-order modeling of conservation laws.
Journal of Computational Physics, 437:110336, 2021.

[24]
Lawrence Sirovich.
Turbulence and the dynamics of coherent structures. i. coherent
Quarterly of applied mathematics, 45(3):561–571, 1987.

[25]
Tommaso Taddei.
A registration method for model order reduction: Data compression and
geometry reduction.
SIAM Journal on Scientific Computing, 42(2):A997–A1027,
January 2020.

[26]
Karen Veroy, Christophe Prud'Homme, Dimitrios Rovas, and Anthony Patera.
A posteriori error bounds for reduced-basis approximation of
parametrized noncoercive and nonlinear elliptic partial differential
In 16th AIAA Computational Fluid Dynamics Conference, page
3847, 2003.

[27]
Kyle Michael Washabaugh.
Faster fidelity for better design: A scalable model order
reduction framework for steady aerodynamic design applications.
PhD thesis, Stanford University, 2016.

[28]
Tianshu Wen and Matthew J Zahr.
A globally convergent method to accelerate large-scale optimization
using on-the-fly model hyperreduction: application to shape optimization.
Journal of Computational Physics, 484:112082, 2023.

[29]
Masayuki Yano.
Discontinuous galerkin reduced basis empirical quadrature procedure
for model reduction of parametrized nonlinear conservation laws.
Advances in Computational Mathematics, 45:2287–2320, 2019.

[30]
Matthew J Zahr and Charbel Farhat.
Progressive construction of a parametric reduced-order model for
pde-constrained optimization.
International Journal for Numerical Methods in Engineering,
102(5):1111–1135, 2015.

[31]
Victor Zucatti and Matthew J Zahr.
An adaptive, training-free reduced-order model for
convection-dominated problems based on hybrid snapshots.
arXiv preprint arXiv:2301.01718, 2023.