# Video-Infinity: Distributed Long Video Generation

https://video-infinity.tanzhenxiong.com
Zhenxiong Tan 11footnotemark: 1 Xingyi Yang Songhua Liu Xinchao Wang
National University of Singapore
<EMAIL_ADDRESS><EMAIL_ADDRESS>
Equal ContributionCorresponding Author

###### Abstract

Diffusion models have recently achieved remarkable results for video
generation. Despite the encouraging performances, the generated videos are
typically constrained to a small number of frames, resulting in clips lasting
merely a few seconds. The primary challenges in producing longer videos
include the substantial memory requirements and the extended processing time
required on a single GPU. A straightforward solution would be to split the
workload across multiple GPUs, which, however, leads to two issues: (1)
ensuring all GPUs communicate effectively to share timing and context
information, and (2) modifying existing video diffusion models, which are
usually trained on short sequences, to create longer videos without additional
training. To tackle these, in this paper we introduce Video-Infinity, a
distributed inference pipeline that enables parallel processing across
multiple GPUs for long-form video generation. Specifically, we propose two
coherent mechanisms: _Clip parallelism_ and _Dual-scope attention_. Clip
parallelism optimizes the gathering and sharing of context information across
GPUs which minimizes communication overhead, while Dual-scope attention
modulates the temporal self-attention to balance local and global contexts
efficiently across the devices. Together, the two mechanisms join forces to
distribute the workload and enable the fast generation of long videos. Under
an 8 $\times$ Nvidia 6000 Ada GPU (48G) setup, our method generates videos up
to 2,300 frames in approximately 5 minutes, enabling long video generation at
a speed 100 times faster than the prior methods.

Figure 1:  Multiple GPUs parallelly generate a complete video, producing 2300
frames in 5 minutes.

##  1 Introduction

A long-standing pursuit of human being is to replicate the dynamic world we
live in, in the digital system. Traditionally dominated by physics and
graphics, this effort has recently been enhanced by the emergence of data-
driven generative models [20, 9, 5, 7], which can create highly realistic
images and videos indistinguishable from reality. However, these models
typically produce very short video segments, with most limited to 16-24 frames
[4, 1, 2]. Some models extend to 60 or 120 frames [30, 10], but compromise
heavily on resolution and visual quality.

Generating long video poses substantial challenges, primarily due to the
extensive resource demands for model training and inference. Current models,
constrained by available resources, are often trained on brief clips, making
it difficult to sustain quality over longer sequences. Moreover, generating a
minute-long video in one go can overwhelm GPU memory, making the task seem
elusive.

Existing solutions, including autoregressive, hierarchical, and short-to-long
methods, offer partial remedies but have significant limitations.
Autoregressive methods [6, 29] produce frames sequentially, dependent on
preceding ones. Hierarchical methods [3, 29, 31] create keyframes first, then
fill in transitional frames. Furthermore, some approaches treat a long video
as multiple overlapping short video clips [19, 25]. These methods are not end-
to-end; they often miss global continuity, require extensive computation,
especially in regions of overlap, and struggle with consistency across
segments.

To bridge these gaps, we introduce a novel framework for distributed long
video generation, termed Video-Infinity. On the high level, it work in a
divide-and-conquer principle. It breaks down the task of long video generation
into smaller, manageable segments. These segments are distributed across
multiple GPUs, allowing for parallel processing. All clients should work
collaboratively to ensure the final video is coherent in semantics.

This setup, while straightforward, faces two principal challenges: ensuring
effective communication among all GPUs to share and contextual information,
and adapting existing models—typically trained on shorter sequences—to
generate longer videos without requiring additional training.

To overcome these challenges, we introduce two synergistic mechanisms: _Clip
parallelism_ and _Dual-scope attention_. Clip parallelism enables efficient
collaboration among multiple GPUs by splitting contextual information into
three parts. It uses an interleaved communication strategy to complete the
sharing in three steps. Building on the capabilities of Clip parallelism,
Dual-scope attentionmeticulously adjusts the temporal self-attention
mechanisms to achieve an optimal balance between local and global contexts
across devices. This balance allows a model trained on short clips to be
extended to long video generation with overall coherence.

Even more exciting, by leveraging both strategies, Video-Infinity reduces
memory overhead from a quadratic to a linear scale. With the power of multiple
device parallelism and sufficient VRAM, our system can generate videos of any,
potentially even infinite length.

As a results, our method significantly extends the maximum length of videos
that can be generated and accelerates the speed of long video generation.
Specifficly, on an 8 $\times$ Nvidia 6000 Ada (48G) setup, our method manages
to generate videos up to 2300 frames in just 5 minutes. Our contributions are
summarized as follows: (1) We are the first to address long video generation
using distributed parallel computation, enhancing scalability and reducing
generation times. (2) We introduce two interconnected mechanisms: Clip
parallelism, which optimizes context information sharing across GPUs, and
Dual-scope attention, which adjusts temporal self-attention to ensure video
coherence across devices. (3) Our experiments show that, compared to the
existing ultra-long text-to-video method Streaming T2V [6], our approach can
be up to 100 times.

##  2 Related works

###  2.1 Diffusion models

Diffusion models have gained significant attention in recent years due to
their impressive ability to generate high-quality media. Originally introduced
for image synthesis, models like Denoising Diffusion Probabilistic Models
(DDPM) [8] and Latent Diffusion Models (LDM) [20] have demonstrated state-of-
the-art performance in image generation. These models progressively denoise a
Gaussian noise distribution by learning a sequence of reverse transformations.
Beyond images [8, 20], diffusion models have also shown promise in audio [12,
28, 14] and 3D generation [15, 18]. Adaptations of diffusion models for video
generation incorporate temporal modules to capture the sequential nature of
video frames. For instance, Video Diffusion Models (VDM) [9] and Flexible
Diffusion Model (FDM) [5] effectively extend diffusion frameworks to video
data, overcoming challenges like temporal consistency and quality degradation.
More recent models such as AnimateDiff [4], ModelScope [26], and VideoCrafter
[1, 2] can now produce video clips with better dynamics and improved visual
quality.

###  2.2 Techniques for long video generation

Streaming T2V [6] introduces a method that relies on a conditional attention
module to ensure smooth transitions between video segments and a scene-
preserving mechanism for content consistency. However, this method requires
training and is not end-to-end, posing limitations on its practicality.
FreeNoise [19] utilizes rescheduled noise sequences and window-based temporal
attention to improve video continuity. Despite these innovations, the
rescheduled noise contributes to limited dynamics in the generated videos, and
the overlapping attention windows introduce additional computational overhead.
NUWA-XL [29] from the NUWA series employs a “coarse-to-fine” autoregressive
approach, using a global diffusion model to generate keyframes and local
models to fill the intermediate frames. Although promising, NUWA-XL has been
trained only within a narrow domain and has not yet made its models and code
available, limiting both its evaluation and reproducibility. Gen-L-Video [25]
adapts short video diffusion models to handle long videos conditioned on
multiple texts without requiring additional training. This approach cleverly
uses latent overlaps to extend video length, which is a common strategy among
recent methodologies. SEINE [3] leverages a random-mask diffusion method to
automate the generation of transition videos between scenes, guided by textual
descriptions. Like other models, SEINE employs an autoregressive approach and
requires image conditioning to facilitate the generation process.

###  2.3 Distributed diffusion

Recently, to reduce the latency of each denoising step in diffusion models,
various distributed parallel methods have been applied to image diffusion
models. ParaDiGMS [23] utilizes step-based parallelism, where each denoising
step is executed on a different GPU device in parallel. However, this approach
tends to waste much computation. Another method, DistriFusion [13], employs a
technique of dividing images into patches, allowing different patches to be
denoised on separate GPUs. This approach ensures synchronization among patches
and achieves minimal computational waste. However, it is designed specifically
for image diffusion and requires significant communication overhead and
specialized hardware support to achieve low latency.

##  3 Preliminaries

Diffusion Models in Video Generation

The process of generating videos using diffusion models involves progressively
denoising the latent representation, denoted as $x_{t}$, where $t$ ranges from
0 to $T$. The initial noisy video latent is represented by a random noise
tensor $x_{T}$. With each denoising step, $x_{t}$ is updated to a clearer
latent $x_{t-1}$. This iterative process continues until $x_{T}$ is denoised
to $x_{0}$, which is then fed into a decoder to generate the final video. The
key aspect of updating $x_{t}$ to $x_{t-1}$ is computing the noisy prediction
$\epsilon_{t}$, given by:

$\epsilon_{t}=\mathcal{E}_{\theta}(x_{t}),$ (1)

where $\mathcal{E}_{\theta}$ represents the diffusion model.

The diffusion model $\mathcal{E}_{\theta}$ can be implemented using various
architectures, such as U-Net [21, 9, 5, 4, 1] or DiT [17, 10, 30]. These
diffusion models are generally composed of several similar layers. More
specifically, the initial random noise tensor is written as
$x_{T}\in\mathbb{R}^{F\times H\times W\times C}$, where $F$ represents the
number of frames, $H$ and $W$ denote the height and width of each frame,
respectively, and $C$ is the number of channels.

The latent tensor $v$ in each layer generally maintains a consistent shape,
$v\in\mathbb{R}^{F\times H^{\prime}\times W^{\prime}\times C^{\prime}}$, where
$F$ remains constant across layers. The dimensions $H^{\prime}$, $W^{\prime}$,
and $C^{\prime}$ can vary due to the down-sampling and up-sampling operations
of the U-Net architecture.

These layers in the diffusion model $\mathcal{E}_{\theta}$ are usually
composed of two main types of modules: spatial and temporal. The spatial
modules receive slices of the latent $v$ shaped
$v\in\mathbb{R}^{H^{\prime}\times W^{\prime}\times C^{\prime}}$ (a single
frame), representing tokens for each video frame in the latent space. They
independently process spatial features within each frame. The temporal modules
receive elongated strips of the latent tensor $v$ shaped
$v\in\mathbb{R}^{F\times C^{\prime}}$, representing tokens containing temporal
information across frames at specific spatial locations. They capture temporal
dependencies between frames at each location.

##  4 Distributed Long Video Generation

Figure 2:  (a) Pipeline of Video-Infinity: The latent tensor is split into
clips and distributed to different devices. The diffusion model predicts noise
in parallel with communication, and the noises are concatenated to produce the
final output. (b) Illustration of Clip parallelism: In each layer of the video
diffusion module, spatial modules operate independently, whereas temporal
modules synchronize context elements $c^{i}_{\text{pre}}$,
$c^{i}_{\text{post}}$, and $c^{i}_{\text{global}}$. Peer-to-peer and
collaborative communications are employed.

At the core of our pipeline, Video-Infinity segments the video latent into
chunks, which are then distributed across multiple devices. An overview of our
method is shown in Figure 3, where we divide the video latent along the
temporal dimension. Such partitioning allows for parallel denoising on
different devices, each handling non-overlapping frames. To facilitate this,
we propose Clip parallelism, detailed in in Section 4.1, a mechanism that
efficiently synchronizes temporal information across devices. Additionally, we
incorporate Dual-scope attention in Section 4.2, which modulates temporal
attention to ensure training-free long video coherence.

Formally, Video-Infinity splits the noisy latent $x_{T}\in\mathbb{R}^{F\times
H\times W\times C}$ into $N$ sub-latent clips
$x_{T}^{i}\in\mathbb{R}^{F_{\text{clip}}\times H\times W\times C}$, where
$i\in[1,N]$, $F_{\text{clip}}={F}/{N}$ represents the number of frames in each
clip, and $N$ represents the total number of clips. This structured
segmentation facilitates an even load distribution across $N$ devices.
Additionally, the spatial modules of video diffusion models operate
independently across frames, which eliminates the need for inter-device
communication and maintains consistency in the outputs across different
devices.

###  4.1 Clip parallelism for video diffusion

To ensure coherence among clips distributed on different devices, we propose
Clip parallelism, shown in Figure 3. It parallelizes the temporal layers for
video diffusion models and enables efficient inter-device communication.

Parallelized temporal modules. In the standard diffusion model, a temporal
module aggregates features across frames, which could be simplified as

$v_{\text{out}}=\text{temporal}\left(v_{\text{in}}\right),$ (2)

where $v_{\text{in}}\in\mathbb{R}^{F\times*\times C^{\prime}}$ is the input
feature of this temporal layer.

However, Video-Infinity distributes input feature tensors $v_{\text{in}}$
across multiple devices, dividing them into several clips
$v_{\text{in}}^{i}\in\mathbb{R}^{{F}_{\text{clip}}\times*\times C^{\prime}}$,
each placed on device(i). To facilitate distributed inference _without
modifying the original structure_ of the temporal modules, we redefine the
temporal operation. This modified operation now considers not only the current
clip, but also adjacent clips and global semantics. Conceptually, the
parallelized temporal modules are defined as follows:

$\displaystyle v_{\text{out}}^{i}$
$\displaystyle=\text{temporal}_{\text{Parallel}}\left(v_{\text{in}}^{i},c^{i}\right),$
(3) $\displaystyle c^{i}$
$\displaystyle=\left\\{c^{i}_{\text{pre}},c^{i}_{\text{post}},c_{\text{global}}\right\\}$
(4)

where $c^{i}$ stands for the temporal information that enriches each device’s
computation by incorporating inter-device context. Each $c^{i}$ includes
temporal information from the preceding device(i-1) via $c^{i}_{\text{pre}}$,
and from the succeeding device(i+1) via $c^{i}_{\text{post}}$. Furthermore,
$c_{\text{global}}$ is a selective aggregate of inputs from all devices,
optimizing global information coherence and reducing overhead.

The output for each device, $v_{\text{out}}^{i}$, reflects localized
computations augmented by these contextual inputs. The complete output of the
layer, $v_{\text{out}}$, is obtained by concatenating the outputs from all
devices:

$v_{\text{out}}=\text{Concat}\left(\left\\{v_{\text{out}}^{i}|i\in\left[1,N\right]\right\\}\right)$
(5)

This concatenation provides a holistic view of the processed features,
maintaining temporal coherence across the distributed system. Further details
on how these temporal modules integrate context will be discussed in Section
4.2.

Figure 3: Tree different stages in the communication process of Clip
parallelism

Three-round context communication. Redefining the temporal modules
necessitates efficient communication of the context components
$c^{i}=\left\\{c^{i}_{\text{pre}},c^{i}_{\text{post}},c_{\text{global}}\right\\}$.
This is achieved through a three-stage synchronization process, where each
stage addresses a specific part of the context, as illustrated in Figure 3.

In the first stage, T1, each device(i) broadcasts its global context
$c^{i}_{\text{global}}$ with all other devices through an all_gather()
operation. This operation ensures that every device receives the global
context, maintaining global consistency across the entire video.

The subsequent stages, T2 and T3 focus on exchanging neighboring contexts. Due
to connection limits111Only one device can communicate with another at a
time., we employ an interleaved strategy. In T2, odd-numbered nodes send their
$c^{i+1}_{\text{pre}}$ to their subsequent device(i+1), and even-numbered
nodes send their $c^{i-1}_{\text{post}}$ to device(i-1). In T3, this pattern
reverses—odd-numbered devices receive context from their predecessors, and
even-numbered devices from their successors. This approach prevents
bottlenecks, optimizes channel usage, and minimizes deadlock risks.

Finally, all nodes complete context synchronization, ensuring that each device
has the full context needed to perform its computations. More details can be
found in the pseudocode in Appendix A.2.

###  4.2 Putting each module in parallel

Building upon the Clip parallelism, this section details that how information
is synchronized in each temporal module. A key technique here is the _Dual-
scope attention_ , which facilitates training-free long video generation and
reduces the communication cost.

There are typically three temporal modules in video diffusion models:
attention module [24] Attention(), convolution module [16] Conv(), and group
normalization module [27] GroupNorm(). We have tailored these modules to
integrate into Clip parallelism, enabling distributed processing across
multiple devices for efficient and coherent video content synchronization.

DualScope attention. Applying attention in parallel inference incurs new
challenges. The original attention module require simultaneous access to all
input tokens [22]. To adopt it under Clip parallelism, it necessitates
aggregating tokens across devices, resulting in tremendous communication
costs. Additionally, those attention trained on shorter video clips often
degrade in quality when applied to longer sequences.

To address these issues, we introduce the _DualScope attention_ module. It
revises the computation of K-V pairs to incorporate both local and global
contexts into the attention. For each query token from frame $a$, its
corresponding keys and values are computed from tokens in the frame set
$\mathcal{A}^{a}=\mathcal{N}^{a}\cup\mathcal{G}$:

  * •

Local Context ($\mathcal{N}^{a}$). This includes the $|\mathcal{N}^{a}|$
neighboring frames of $a$, from which the keys and values are derived to
capture the local context. This local setup is typically achieved through
window attention, focusing on the nearby frames to enhance the temporal
coherence.

  * •

Global Context ($\mathcal{G}$). In contrast, the global context consists of
frames uniformly sampled from videos across all devices. This context provides
keys and values from a broader range, giving the model access to long-range
information

In practice, the keys $K$ and values $V$ are constructed by concatenating the
tokens from both contexts
$K=\text{Concat}(K_{\text{local}},K_{\text{global}})$ and
$V=\text{Concat}(V_{\text{local}},V_{\text{global}})$, where
$K_{\text{local}}$ and $Q_{\text{local}}$ is derived from $\mathcal{N}^{a}$
and $K_{\text{global}}$ and $Q_{\text{global}}$ from $\mathcal{G}$. We find
that this modified key-value computation can be easily incorporated into
existing temporal attention without additional training, enhancing the
coherence of long videos.

In the implementation of Clip parallelism, above reformulated attention
largely reduce the communication overhead. Comparing to gathering all tokens
of length $F$, we only synchronize constant number of tokens. Specifically, we
set $|c^{i}_{\text{pre}}|=|c^{i}_{\text{post}}|=\frac{|\mathcal{N}^{a}|}{2}$
and $|c_{\text{global}}|=|\mathcal{G}|$, with both $|\mathcal{N}^{a}|$ and
$|\mathcal{G}|$ configured to 16. This reduces data synchronization demands
while still capturing essential local and global information.

Convolution module. The temporal convolution module Conv() applies convolution
along the temporal dimension to its input
$v_{\text{in}}^{i}\in\mathbb{R}^{F_{\text{clip}}\times C^{\prime}}$. In Clip
parallelism, the context $c^{i}$ of the Conv() includes $c^{i}_{\text{pre}}$
and $c^{i}_{\text{post}}$. They are padded to the original sequences.
Specifically, $c^{i}_{\text{pre}}$ consists of the last $n$ frames of
$v_{\text{in}}^{i-1}$, and $c^{i}_{\text{post}}$ consists of the first $n$
frames of $v_{\text{in}}^{i+1}$, where $n$ is the receptive field size of the
convolution.

Group normalization. In video diffusion model, group normalization is applied
to the input tensor $v_{\text{in}}^{i}\in\mathbb{R}^{F_{\text{clip}}\times
H\times W\times C^{\prime}}$ to maintain consistent feature scaling across
different frames.

In Clip parallelism, each device first computes the group mean $\mu^{i}$ of
its respective video clip. These means are aggregated to compute the global
mean $\bar{\mu}=\frac{\sum_{i=1}^{N}\mu^{i}}{N}$, where N is the number of
devices. Subsequently, using $\bar{\mu}$, each device computes its standard
deviation $\bar{\sigma}^{i}$, which is shared to calculate the global standard
deviation $\bar{\sigma}$. The global mean $\bar{\mu}$ and global standard
deviation $\bar{\sigma}$, serving as $c_{\text{global}}$, are used for
normalization 222Note that simply averaging the individual standard deviations
$\sigma^{i}$ does not yield the true global standard deviation
$\bar{\sigma}$..

##  5 Experiments

###  5.1 Setups

Base model. In the experiments, the text to video model VideoCrafter2 [2] (320
x 512) is selected as the base model of our method. VideoCrafter2, which was
trained on 16-frame videos, excels at generating video clips that are both
consistent and of high quality. It is also the highest-scoring open-source
video generation model under the VBench [11] evaluation, achieving the top
total score.

Metrics evaluation. VBench [11] is utilized as a comprehensive video
evaluation tool, featuring a broad array of metrics across various video
dimensions. For each method, videos are generated using the prompts provided
by VBench for evaluation. The metrics measured encompass all the indicators
under the Video Quality category in VBench, including subject consistency,
background consistency, temporal flickering, motion smoothness, dynamic
degree, aesthetic quality and imaging quality. Given that VBench’s evaluation
is typically performed on video clips of 16 frames, we have modified the
evaluation method for videos longer than 16 frames: we randomly sample five
16-frame clips from each video to evaluate separately, and then calculate the
average score of these assessments.

Baslines. Our approach is benchmarked against several other methods:

  * •

FreeNoise [19]: We chose FreeNoise as a baseline because it is also a
training-free method that can base the VideoCrafter2 [2] model, which also
serves as our base model, to generate long videos. It employs a rescheduling
technique for the initialization noise and incorporates Window-based Attention
Fusion to generate longer videos.

  * •

Streaming T2V [6]: To assess our method’s effectiveness in generating longer
videos, StreamingT2V was chosen as our baseline. Streaming T2V involves
training a new model that uses an auto-regressive approach to produce long-
form videos. Like our approach, it also has the capability to generate videos
exceeding 1000 frames.

OpenSora V1.1 [10], a video diffusion model based on DiT [17], supports up to
120 frames, can generate videos at various resolutions, and has been
specifically trained on longer video sequences to enhance its extended video
generation capabilities.

Dual-scope attentionsetting. In the implementation of the Dual-scope
attention, the number of neighboring frames $\mathcal{N}^{i}$ is set to 16,
with 8 frames coming from the preceding clip and 8 frames from the subsequent
clip. The number of global frames, $\mathcal{G}$, is set to 16. To balance
consistency and dynamics during the denoising process, the weights of frames
in $\mathcal{G}$ and $\mathcal{N}^{i}$ are dynamically adjusted. Specifically,
the weight of $\mathcal{G}$ increases by 10 for timesteps $t$ greater than
800, whereas the weight of $\mathcal{N}^{i}$ increases by 10 for timesteps $t$
less than or equal to 800.

Implementation details. By default, all parameters of the diffusion are kept
consistent with the original inference settings of VideoCrafter2 [2], with the
number of denoising steps set to 30. Our experiments are conducted on 8
$\times$ Nvidia 6000 Ada (with 48G memory) . To implement the temporal module
in Clip parallelism, we utilized the torch.distributed tool package, employing
Nvidia’s NCCL as the backend to facilitate efficient inter-GPU communication.
Additionally, all fps conditions are set to 24, and the resolution is set to
$512\times 320$. Note that the resolution for Streaming T2V cannot be
modified; thus, videos are generated at its default resolution ($256\times
256$ for preview videos and $720\times 720$ for final videos).

Method | Max | Time Coast (second)
---|---|---
| Frames | 128 frames | 1024 frames
Ours | 2300 | 21 | 131
ST2V (preview) | - | 277 | 2,196
ST2V (final cut) | - | 1730 | 13,726
FreeNoise [19] | 128 | 201 | $\times$
Open-Sora v1.1 [10] | 120 | 234 | $\times$

Table 1: Comparison of maximum frames and generation times for different
methods.

###  5.2 Main results

Capacity and efficiency.

We evaluated the capabilities of our method on an 8 $\times$ Nvidia 6000 Ada
(48G) setup. Our approach successfully generated videos of 2300 frames at a
resolution of 512 $\times$ 320, equivalent to a duration of 95 seconds at 24
frames per second. Remarkably, the entire computation process took
approximately 5 minutes (312s), benefiting from efficient communication and
the leveraging of multi-GPU parallel processing.

Table 1 presents the capacities for long video generation of various methods,
all measured under the same device specifications. To ensure comparability, we
standardized the resolution of the videos generated by all methods to 512x320.
For StreamingT2V, we provide two sets of data: one for generating preview
videos at 256x256 resolution, and another for final videos produced at a
resolution of 720x720. The results demonstrate that our method is the most
capable within the end-to-end category, generating the longest videos of up to
2300 frames — 8.2 times more than OpenSora V1.1. Additionally, our method
consistently produces the final videos in the shortest time, both for short
videos of 128 frames and long videos of 1024 frames. Notably, in the
generation of 1024-frame videos, our method is over 100 times faster than
StreamingT2V, the only baseline method capable of producing videos of this
length. Even when compared to the speed of generating smaller, lower-
resolution preview videos by StreamingT2V, our method is 16 times faster.

Video quality. We compared the videos generated by our method with those
produced by FreeNoise [19] and StreamingT2V [6] for long video generation.
Figure 4 visualizes some frames from videos generated by different methods
using the same prompt. Additionally, Table 2 displays the quality of the
videos produced by these methods, evaluated across various metrics in VBench
[11].

Figure 4:  Comparison of frame images from sample videos generated by
different methods.

Figure 4 shows that while the StreamingT2V [6] method generates long videos
with sufficient dynamism, they lack consistency between the beginning and end.
Conversely, videos generated by FreeNoise [19] maintain consistency in object
placement throughout but exhibit minimal variation in visuals. For example, as
shown in Figure 4, the video of the person playing the guitar maintains a
single pose with only minimal movement. Similarly, the dog on the left remains
intently focused on the camera, with no changes in the position of its ears,
nose, or body. OpenSora V1.1 [10] failed to generate the first video and the
second video’s background was not smooth. In contrast, our method not only
ensures better consistency but also features more significant motion in the
generated videos.

Method | Number of | Subject | Background | Temporal | Motion | Dynamic | Aesthetic | Imaging | Overall
---|---|---|---|---|---|---|---|---|---
| Frames | consistency | consistency | flickering | smoothness | degree | quality | quality | Score
V.C.2 | 16 | 96.85% | 98.22% | 98.41% | 97.73% | 42.50% | 63.13% | 67.22% | 80.58%
FreeNoise | 64 | 94.16% | 96.63% | 98.37% | 97.04% | 44.44% | 60.53% | 67.44% | 79.80%
OpenSora v1.1 | 64 | 86.18% | 95.83% | 98.47% | 97.27% | 73.61% | 51.69% | 50.61% | 79.09%
Ours | 64 | 97.77% | 93.90% | 97.77% | 96.84% | 81.94% | 59.38% | 67.90% | 85.07%
ST2V | 192 | 75.02% | 87.93% | 95.96% | 94.71% |  80.56% | 48.08% | 57.85% | 77.16%
Ours | 192 | 91.32% | 92.93% | 97.40% | 95.55% | 77.78% | 55.49% | 66.93% | 82.49%

Table 2: Evaluation metrics: Comparison of performance metrics for various
video generation methods as benchmarked by VBench. Bold values represent the
best performance within each group.

Table 2 reveals that our method, when compared to our base model VideoCrafter
2 [2], experiences a slight decrease in most metrics except for the metric of
dynamic. In the generation of 64-frame videos, the performance of our method
shows mixed results compared to other methods, with both advantages and
disadvantages noted. However, our average metric scores are higher than those
of both FreeNoise and OpenSora V1.1. In the generation of longer 192-frame
videos, our method outperforms StreamingT2V, the only other method capable of
producing videos of this length, across the majority of evaluated metrics.

###  5.3 Ablation

Figure 5:  Visualization of ablation studies on temporal module communication
and context effects in video generation. Top panel: Ablation of communication
between the ResLayer module and the Attention module, showcasing two adjacent
frames from the video sequence generated on different GPUs. Bottom panel:
Effects of ablating different contexts within the Attention module, displaying
frames from videos generated post-ablation.

As mentioned in Section 4.2, three types of temporal modules (Conv(),
GroupNorm(), and Attention()) are adapted to synchronize context in Clip
parallelism. To demonstrate the effectiveness of context synchronization by
these modules, we conducted ablation experiments and visualized in Figure 5
the impact of removing certain parts of the context synchronization on the
quality of the generated videos. We performed ablation on the communication
between the temporal Attention() module and the temporal ResNet() module in
the video diffusion model, where the ResNet module includes temporal Conv()
and temporal GroupNorm() as submodules. Subsequently, we conducted ablations
on the global context $c_{\text{global}}$and the local context
$c^{i}_{\text{pre}},c^{i}_{\text{post}}$within the Attention() module.

Removing local context. From the top panel of Figure 5, it can be observed
that the absence of synchronized information from the ResNet() leads to
discrepancies in detail between the last frame on device(1) (frame 23) and the
first frame on device(2) (frame 24), which are highlighted in red. These
discrepancies, such as differences in the color of the clothes of the person
behind the robot and the shape of the parts in the robot’s hands on the table,
do not appear in the original inference. When context of the Attention()
module is absent, frame 23 and frame 24 become markedly different images,
illustrating a significant discontinuity between video segments generated on
adjacent devices. These observations suggest that synchronization in both
ResNet() and Attention() modules is crucial for preserving visual coherence
and continuity across video frames generated on different devices.

Removing global context. The bottom panel of Figure 5 demonstrates that when
synchronization of the global context is absent, content consistency within
the video is difficult to maintain. For example, in frames 12 and 16, the
horizon remains high, but in frames beyond 20, there is a noticeable rise in
the horizon. Furthermore, when the local context synchronization is removed,
although the content across different device clips remains consistent, the
lack of shared context in the transition areas leads to anomalies. For
instance, the content of snow in frame 22 abruptly transitions to a dog,
highlighted in red. These examples highlight the importance of global and
local context synchronization for video generation.

##  6 Conclusion

We presented Video-Infinity, a distributed inference pipeline that leverages
multiple GPUs for long-form video generation. We present two mechanisms, Clip
parallelism and _Dual-scope attention_ , to addressed key challenges
associated with distributed video generation. Clip parallelism reduces
communication overhead by optimizing the exchange of context information,
while _Dual-scope attention_ modified self-attention to ensure coherence
across devices. Together, these innovations enable the rapid generation of
videos up to 2,300 frames long, vastly improving generation speeds compared to
existing methods. This approach not only extends the practical utility of
diffusion models for video production but also sets a new benchmark for
efficiency in long-form video generation.

##  7 Limitation

To fully harness the potential of our method, it relies on the availability of
multiple GPUs. Additionally, our approach does not effectively handle video
generation involving scene transitions.

## References

  * [1] Haoxin Chen, Menghan Xia, Yingqing He, Yong Zhang, Xiaodong Cun, Shaoshu Yang, Jinbo Xing, Yaofang Liu, Qifeng Chen, Xintao Wang, Chao Weng, and Ying Shan.  Videocrafter1: Open diffusion models for high-quality video generation, 2023.
  * [2] Haoxin Chen, Yong Zhang, Xiaodong Cun, Menghan Xia, Xintao Wang, Chao Weng, and Ying Shan.  Videocrafter2: Overcoming data limitations for high-quality video diffusion models, 2024.
  * [3] Xinyuan Chen, Yaohui Wang, Lingjun Zhang, Shaobin Zhuang, Xin Ma, Jiashuo Yu, Yali Wang, Dahua Lin, Yu Qiao, and Ziwei Liu.  Seine: Short-to-long video diffusion model for generative transition and prediction.  In The Twelfth International Conference on Learning Representations, 2023.
  * [4] Yuwei Guo, Ceyuan Yang, Anyi Rao, Yaohui Wang, Yu Qiao, Dahua Lin, and Bo Dai.  Animatediff: Animate your personalized text-to-image diffusion models without specific tuning.  arXiv preprint arXiv:2307.04725, 2023.
  * [5] William Harvey, Saeid Naderiparizi, Vaden Masrani, Christian Weilbach, and Frank Wood.  Flexible diffusion modeling of long videos.  Advances in Neural Information Processing Systems, 35:27953–27965, 2022.
  * [6] Roberto Henschel, Levon Khachatryan, Daniil Hayrapetyan, Hayk Poghosyan, Vahram Tadevosyan, Zhangyang Wang, Shant Navasardyan, and Humphrey Shi.  Streamingt2v: Consistent, dynamic, and extendable long video generation from text.  arXiv preprint arXiv:2403.14773, 2024.
  * [7] Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik P Kingma, Ben Poole, Mohammad Norouzi, David J Fleet, et al.  Imagen video: High definition video generation with diffusion models.  arXiv preprint arXiv:2210.02303, 2022.
  * [8] Jonathan Ho, Ajay Jain, and Pieter Abbeel.  Denoising diffusion probabilistic models.  Advances in neural information processing systems, 33:6840–6851, 2020.
  * [9] Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J Fleet.  Video diffusion models.  Advances in Neural Information Processing Systems, 35:8633–8646, 2022.
  * [10] hpcaitech.  Open-sora.  https://github.com/hpcaitech/Open-Sora, 2024.
  * [11] Ziqi Huang, Yinan He, Jiashuo Yu, Fan Zhang, Chenyang Si, Yuming Jiang, Yuanhan Zhang, Tianxing Wu, Qingyang Jin, Nattapol Chanpaisit, et al.  Vbench: Comprehensive benchmark suite for video generative models.  arXiv preprint arXiv:2311.17982, 2023.
  * [12] Zhifeng Kong, Wei Ping, Jiaji Huang, Kexin Zhao, and Bryan Catanzaro.  Diffwave: A versatile diffusion model for audio synthesis.  arXiv preprint arXiv:2009.09761, 2020.
  * [13] Muyang Li, Tianle Cai, Jiaxin Cao, Qinsheng Zhang, Han Cai, Junjie Bai, Yangqing Jia, Ming-Yu Liu, Kai Li, and Song Han.  Distrifusion: Distributed parallel inference for high-resolution diffusion models.  arXiv preprint arXiv:2402.19481, 2024.
  * [14] Haohe Liu, Zehua Chen, Yi Yuan, Xinhao Mei, Xubo Liu, Danilo Mandic, Wenwu Wang, and Mark D Plumbley.  Audioldm: Text-to-audio generation with latent diffusion models.  arXiv preprint arXiv:2301.12503, 2023.
  * [15] Shitong Luo and Wei Hu.  Diffusion probabilistic models for 3d point cloud generation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2837–2845, 2021.
  * [16] Keiron O’shea and Ryan Nash.  An introduction to convolutional neural networks.  arXiv preprint arXiv:1511.08458, 2015.
  * [17] William Peebles and Saining Xie.  Scalable diffusion models with transformers.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4195–4205, 2023.
  * [18] Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Mildenhall.  Dreamfusion: Text-to-3d using 2d diffusion.  arXiv preprint arXiv:2209.14988, 2022.
  * [19] Haonan Qiu, Menghan Xia, Yong Zhang, Yingqing He, Xintao Wang, Ying Shan, and Ziwei Liu.  Freenoise: Tuning-free longer video diffusion via noise rescheduling.  arXiv preprint arXiv:2310.15169, 2023.
  * [20] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022.
  * [21] Olaf Ronneberger, Philipp Fischer, and Thomas Brox.  U-net: Convolutional networks for biomedical image segmentation.  In Medical image computing and computer-assisted intervention–MICCAI 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceedings, part III 18, pages 234–241. Springer, 2015.
  * [22] Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani.  Self-attention with relative position representations.  arXiv preprint arXiv:1803.02155, 2018.
  * [23] Andy Shih, Suneel Belkhale, Stefano Ermon, Dorsa Sadigh, and Nima Anari.  Parallel sampling of diffusion models.  Advances in Neural Information Processing Systems, 36, 2024.
  * [24] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  Advances in neural information processing systems, 30, 2017.
  * [25] Fu-Yun Wang, Wenshuo Chen, Guanglu Song, Han-Jia Ye, Yu Liu, and Hongsheng Li.  Gen-l-video: Multi-text to long video generation via temporal co-denoising.  arXiv preprint arXiv:2305.18264, 2023.
  * [26] Jiuniu Wang, Hangjie Yuan, Dayou Chen, Yingya Zhang, Xiang Wang, and Shiwei Zhang.  Modelscope text-to-video technical report.  arXiv preprint arXiv:2308.06571, 2023.
  * [27] Yuxin Wu and Kaiming He.  Group normalization.  In Proceedings of the European conference on computer vision (ECCV), pages 3–19, 2018.
  * [28] Dongchao Yang, Jianwei Yu, Helin Wang, Wen Wang, Chao Weng, Yuexian Zou, and Dong Yu.  Diffsound: Discrete diffusion model for text-to-sound generation.  IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2023.
  * [29] Shengming Yin, Chenfei Wu, Huan Yang, Jianfeng Wang, Xiaodong Wang, Minheng Ni, Zhengyuan Yang, Linjie Li, Shuguang Liu, Fan Yang, et al.  Nuwa-xl: Diffusion over diffusion for extremely long video generation.  arXiv preprint arXiv:2303.12346, 2023.
  * [30] Zhang Zhaoyang, Yuan Ziyang, Ju Xuan, Gao Yiming, Wang Xintao, Yuan Chun, and Shan Ying.  Mira: A mini-step towards sora-like long video generation.  https://github.com/mira-space/Mira, 2024.
  * [31] Yupeng Zhou, Daquan Zhou, Ming-Ming Cheng, Jiashi Feng, and Qibin Hou.  Storydiffusion: Consistent self-attention for long-range image and video generation.  arXiv preprint arXiv:2405.01434, 2024.

##  Appendix A Appendix

###  A.1 Communication overhead

Table 3 demonstrates the additional time overhead caused by communication
between different temporal modules. The experiments were conducted on multiple
Nvidia A5000 GPUs, with two settings: a dual-GPU configuration and an eight-
GPU configuration.

Sync | Inference Time (s)
---|---
| 2$\times$GPU | 8 $\times$ GPU
Plain | 145.4 | 149.5
\+ Conv() |  152.9 ($5.1\%\uparrow$) |  157.1 ($5.1\%\uparrow$)
\+ GroupNorm() |  158.3 ($8.9\%\uparrow$) |  160.1 ($7.1\%\uparrow$)
\+ Attention() |  170.7 ($17.4\%\uparrow$) |  180.2 ($20.5\%\uparrow$)
Full Sync |  182.3 ($25.3\%\uparrow$) |  192.3 ($28.6\%\uparrow$)
Table 3: Effect of Synchronization on Inference Time

###  A.2 Communication Algorithm

Algorithm 1 Distributed Temporal Module Communication

0: $i$ (the ID of the device), $v_{\text{in}}^{i}$ (the input latent segment)

0: Seamless and efficient distribution of frames for video processing.

1: Prepare the global context $c^{i}_{\text{global}}$ using
$v_{\text{in}}^{i}$

2: dist.all_gather($c^{i}_{\text{global}}$)

3: if $i$ mod 2 == 1 then

4: $c^{i}_{\text{pre}}$ = dist.recv(i+1)

5: Prepare the local context for device(i+1) using $v_{\text{in}}^{i}$

6: dist.send($c^{i+1}_{\text{post}}$)

7: $c^{i}_{\text{post}}$ = dist.recv(i-1)

8: Prepare the local context for device(i-1) using $v_{\text{in}}^{i}$

9: dist.send($c^{i-1}_{\text{pre}}$)

10: else

11: $c^{i}_{\text{post}}$ = dist.recv(i-1)

12: Prepare the local context for device(i-1) using $v_{\text{in}}^{i}$

13: dist.send($c^{i-1}_{\text{pre}}$)

14: $c^{i}_{\text{pre}}$ = dist.recv(i+1)

15: Prepare the local context for device(i+1) using $v_{\text{in}}^{i}$

16: dist.send($c^{i+1}_{\text{post}}$)

17: end if

##  Appendix B Gallery

More videos are available in the supplementary materials and at the following
link. Link


#  Glass: Geometric Latent Augmentation for Shape Spaces

Sanjeev Muralikrishnan  University College London  Siddhartha Chaudhuri  Adobe
Research  IIT Bombay  Noam Aigerman  Adobe Research  Vladimir G. Kim  Adobe
Research  Matthew Fisher  Adobe Research  Niloy J. Mitra  University College
London  Adobe Research

###### Abstract

We investigate the problem of training generative models on very sparse
collections of 3D models. Particularly, instead of using difficult-to-obtain
large sets of 3D models, we demonstrate that geometrically-motivated energy
functions can be used to effectively augment and boost only a sparse
collection of example (training) models. Technically, we analyze the Hessian
of the as-rigid-as-possible (ARAP) energy to adaptively sample from and
project to the underlying (local) shape space, and use the augmented dataset
to train a variational autoencoder (VAE). We iterate the process, of building
latent spaces of VAE and augmenting the associated dataset, to progressively
reveal a richer and more expressive generative space for creating
geometrically and semantically valid samples. We evaluate our method against a
set of strong baselines, provide ablation studies, and demonstrate application
towards establishing shape correspondences. Glass produces multiple
interesting and meaningful shape variations even when starting from as few as
3-10 training shapes. Our code is available at
https://sanjeevmk.github.io/glass_webpage/.

Figure 1: Starting from just 10 shapes (larger), our method iteratively
augments the collection by alternating between training a VAE, and exploring
random perturbations in its low-dimensional latent space guided by a purely
geometric deformation energy. Here we show the 1000 most diverse shapes from
the first 5K discovered by our method, positioned according to their latent
embedding (projected to 2D via t-SNE). Shapes are colored according to the
initial landmark they trace back to, with shapes added in later iterations
lighter (greyer) in color. The augmentation effectively fills in the space
between the sparse initial landmarks, and even extrapolates beyond them. It
manages to also interpolate global rotations for samples near the back-facing
exemplar, and yields shapes with larger feet-strides (far left), and crossed
arms or feet (front, left and center) even though there are no such initial
landmarks.

##  1 Introduction

This paper is concerned with generating plausible deformations of a 3D shape
from a very sparse set of examples. Fig. 1 shows an input of 10 human 3D
meshes in different poses, and the additional deformations generated by our
method. 3D deformations have a strong _semantic_ element to them – e.g., a
human’s limbs should only bend at the joints, and then, under normal
circumstances, not beyond certain angular ranges. Arguably, this can only be
deduced in general via learning by example from a dataset.

Unfortunately, in contrast to 2D images, the 3D domain poses several
challenges for data-driven frameworks. Probably the most significant one is
that data acquisition is complex and tedious, making datasets both scarcer and
sparser.

Given this data paucity, we tackle the challenge of generating additional
meaningful deformations from a given (very) sparse set of landmark
deformations. Our method meaningfully augments the sparse sets to create
larger datasets that, in turn, can be leveraged by other techniques that
cannot operate on sparse datasets.

Producing plausible deformations from a few landmarks is difficult. Linearly
interpolating the vertices of two landmarks yields highly implausible
intermediates. A key insight is that while meaningful deformations are
semantic, they often have a very strong pure-geometric element, e.g., they are
smooth (i.e., preserve local details) and don’t distort the shape too much
(i.e., local distances are preserved). However, simply perturbing vertices
while minimizing a geometric energy (e.g., smoothness or metric distortion)

generates artifacts such as smooth global bending or surface ripples because
by itself, the energy is not a sufficient constraint. Interpolating landmark
pairs, while preserving the energy, fares better but produces limited
variations [1, 25]. Our paper, like other recent approaches [12, 20],
advocates learning a low-dimensional generative latent space which maps out
the underlying manifold jointly defined by the landmarks, while simultaneously
minimizing a deformation energy. However, these prior methods still require a
large dataset to learn a rich set of variations.

Our core contribution is to address this difficulty with a novel data
augmentation approach that alternates between latent space training and
energy-guided exploration. We employ supervised learning of a generative space
from a training dataset, a very sparse one, but augment that set in an
unsupervised, geometry-aware way. Specifically, we train a Variational
Autoencoder (VAE) on the given dataset.

After training, we use the eigenmodes of a deformation energy’s Hessian to
perturb and project latent codes of training shapes in a way that ensures they
yield smooth, low-distortion deformations, which we add back as data
augmentation to the input set. We then re-train the VAE on the augmented
dataset and repeat the process iteratively until the space has been densely
sampled. In addition to reducing spurious deformations, the use of a low-
dimensional, jointly-trained latent space allows low-energy perturbations of
one landmark to be influenced by other landmarks, yielding richer variations.
We call our method Glass.

We evaluate Glass on several established datasets and compare performance
against baselines using multiple metrics. The experiments show the
effectiveness of Glass to recover meaningful additional deformations from a
mere handful of exemplars. We also evaluate the method in the context of shape
correspondence, demonstrating that our sampling process can be used as a data
augmentation technique to improve existing strongly-supervised correspondence
algorithms (e.g., 3D-CODED [18]).

##  2 Related Work

#### Geometric shape deformation.

Parametric deformation methods express 2D or 3D shapes as a known function of
a set of common parameters, and model deformations as variations of these
parameters. Such methods include cages [24], blendshapes [29], skinned
skeletons [23] and Laplacian eigenfunctions [32]. In contrast, variational
methods model deformations as minimizers of an energy functional – e.g.
Dirichlet [19], isometric [25], conformal [28], Laplacian [7], As-Rigid-As-
Possible (ARAP) [36], or As-Consistent-As-Possible (ACAP) [15] – subject to
user constraints. In our work we focus on minimizing the ARAP energy, although
our method supports any twice-differentiable energy function. There are strong
connections between the parametric and variational approaches, for instance
biharmonic skinning weights [22] (parametric) are equivalent to minimizing the
Laplacian energy (variational). Please see surveys [27, 42] for a complete
discussion. We are also inspired by work on modal analysis [21], which
linearize the deformation space of a shape in terms of the least-significant
eigenvectors of the Hessian of some energy functional. In the current paper,
we effectively perform learned non-linear modal analysis: starting with a
variational formulation – the implicitly-defined manifold of low-energy
perturbations of a few landmark shapes – we learn the corresponding parametric
representation as the latent space of an autoencoder by iteratively exploring
locally linear perturbations.

Our work on data augmentation from a sparse set of landmark shapes is related
to interpolation/morphing between, and extrapolation from, sets of shapes. As
in our scenario, the set typically comprises articulations of a common
template. See e.g. [39] for a survey of classical (non-learning-based) methods
for shape interpolation. Plausible extrapolation is less well-defined, and
less studied, in the classical literature. Kilian et al. [25] extend geodesics
of an isometric energy in the deformation space, though it is restricted to
exploring (and extrapolating) paths between shapes rather than the full
deformation space.

#### Learned deformation models.

Various types of generative models based on graphical models, GANs, VAEs etc
have been developed to probabilistically synthesize shape variations. A full
treatment is beyond the scope of this paper, please see surveys such as [11].
Here, we focus on models which capture the space of smooth deformations of a
given shape. The best-studied domain is that of virtual humans, beginning with
seminal works capturing face [4], bodyshape [2] and pose [3] variations in a
data-driven fashion from scanned exemplars. These works, like several
subsequent ones, rely on variations of principal component analysis (PCA) to
parameterize the deformation space. Yumer et al. [43] learn a common set of
deformation handles for a dataset. More recent work uses deep neural networks
to learn shape deformation models from training sets [31, 37, 16, 13, 40], and
use them for applications such as non-rigid correspondences [18]. Tan et al.
[38] and Huang et al. [20] regularize a VAE with an energy-based loss. We use
the latter [20] as our choice for energy. However, the primary role of the
energy in our method is to guide exploration for data augmentation.

Crucially, all the above methods rely on extensive training data. In contrast,
we specifically aim to learn meaningful data-driven deformation models under
extreme sparsity constraints, from just a handful of landmarks indicating
modes of the distribution. While this is broadly related to few-shot learning
scenarios, only a few other papers consider these requirements in the context
of geometric shape synthesis, or without any auxiliary data from other
domains. LIMP [12] is an important recent work that tries to regularize the
latent space of a 3D shape VAE by requiring points sampled on the line segment
between two latent codes to minimize geometric distortion relative to the
endpoints. Unlike our method, LIMP does not explore the full volume of the
hull bounding the training landmarks, or extrapolate beyond it –
regularization is limited to the web of pairwise paths. We modified LIMP to
work with ARAP energy, and demonstrate that our method significantly
outperforms their approach on a variety of metrics.

#### Unsupervised data augmentation.

Our work is part of a wide class of methods for synthetically increasing the
size of training datasets for data-hungry machine learning, without additional
supervision. For broad coverage, we refer the reader to surveys on images
[35], time series [41], and NLP [10]. A particularly relevant recent technique
is Deep Markov Chain Monte Carlo [33], which samples perturbations of training
data using MCMC on an energy functional, trains an autoencoder on these
samples, and uses the resulting latent space for lower-dimensional (and hence
faster) MCMC. We observed that on very sparse and high-dimensional datasets
(only a few landmark 3D shapes), the initial samples of Deep MCMC do not
capture meaningful variations, and hence it does not adequately augment the
dataset. Also related are methods that augment classification datasets with
adversarial perturbations along the gradients of loss functions [17, 34]. In
contrast, we seek to preserve an energy-based loss, and hence eliminate the
gradient and other high-change directions from consideration.

##  3 Method

###  3.1 Problem Setup

We assume all shapes in a particular input dataset are meshes with consistent
topology. Given a mesh with $N$ vertices $V\in\mathbb{R}^{N\times 3}$ and
triangle faces $T$, a mesh deformation is simply an assignment of a new
position to each vertex, denoted as $W\in\mathbb{R}^{N\times 3}$. We consider
the input dataset itself as deformations of a base topology and we are given a
sparse set of $n$ deformation “examples”, $W^{1},\dots W^{n}$. We assume
access to a deformation energy $f(W,W^{\prime})$ which measures the distortion
of candidate deformation $W$ with respect to an exemplar deformation
$W^{\prime}$, with higher values indicating more severe distortion induced by
the candidate. For brevity, we omit $W^{\prime}$ and simply write $f(W)$ to
mean energy with respect to the relevant base shape. We use the As-Rigid-As-
Possible (ARAP) energy [36] and its latent-space approximation ARAPReg [20] to
measure the deviation of a deformation from isometry, i.e., how much do
geodesic lengths change with respect to the rest pose $V$.

Figure 2: We present Glass to iteratively build a deformation-aware VAE latent
space and analyzing it to generate new training samples to augment the
original training set. This enables generation of diverse yet plausible shape
variations starting from very few input examples.

We devise a subspace-sampling strategy that adheres to two properties: (i) it
should be data-driven, and contain deformations from the given sparse set; and
(ii) it should be geometrically-meaningful, i.e., the deformations should have
low energy, wrt the given deformation energy $f(W)$.

Our main contribution is a method for online data augmentation during the
training of a variational autoencoder (VAE) [26]. Namely, during training, our
method explores the current sample space, guided by the deformation energy
$f(W)$, to discover additional meaningful deformation samples. These are
progressively used as additional sample points to form an augmented dataset
that is used to retrain the VAE, and the process is iterated until
convergence.

###  3.2 Deformation-Aware VAE

Let $E:\leavevmode\nobreak\ \mathbb{R}^{N\times 3}\to\mathbb{R}^{K}$ be the
encoder in the standard VAE architecture, mapping a deformation $W$ into
vectors of mean $\mu$ and variance $\Sigma$ into a distribution
$E(W)\sim\mathcal{N}(\mu,\Sigma)$. These vectors define the mean and variance
of a multivariate Gaussian distribution from which the latent code $z$, of
dimension $K$, is sampled, i.e., $z\sim\mathcal{N}(\mu,\Sigma)$. Similarly,
let $D:\mathbb{R}^{K}\to\mathbb{R}^{N\times 3}$ be the decoder mapping the
latent code to a deformation, $D(z)=W$. We shall slightly abuse notation and
use $D(E(W))$ to denote the full autoencoding process of $W$, including the
step of sampling from the Gaussian. We define three losses to be used in
training.

Figure 3:  tSNE embedding of generated samples shows progressive augmentation
of the shape space. Sample color indicates originating (parent) shape. See
also Fig. 1.

(i) Reconstruction Loss: We require that the VAE reduces to an identity map
for any sample deformation,

$L_{\text{Reconstruction}}:=\lVert D(E(W))-W\rVert^{2}.\vspace{-2mm}$ (1)

(ii) Gaussian Regularizer Loss: Instead of the standard the KL divergence
regularizer used in VAEs, we constrain the sample mean and covariance of the
mini-batch to that of a unit Gaussian, as proposed in [14]. We found that for
a small sample size this batch-based loss leads to faster convergence,
compared to the standard KL-Divergence. We denote this loss as,

$L_{\text{Gaussian}}:=\frac{1}{b}\sum_{i=1}^{b}\left(\lVert\mu_{i}\rVert^{2}+\lVert\Sigma_{i}-\mathbb{I}\rVert^{2}\right),\vspace{-2mm}$
(2)

where $b$ is the mini-batch size, $\mu_{i},\Sigma_{i}$ are the predicted mean
and covariance for the $i$-th sample in the mini-batch, and $\mathbb{I}$ is
the identity matrix.

(iii) Deformation Energy: Lastly, we require the resulting deformation to have
low deformation energy,

$L_{\text{Deformation}}:=f(D(E(W))).\vspace{-2mm}$ (3)

In summary, our network training loss is

$L:=L_{\text{Reconstruction}}+L_{\text{Gaussian}}+\sigma
L_{\text{Deformation}}\vspace{-2mm}$ (4)

where $\sigma$ is a scalar weight applied to the energy function.

1:procedure LatentAugment($W$,$E$,$D$,$f$,$R$)

2:$\triangleright$ E = Encoder, D= Decoder

3:$\triangleright$ W = Initial deformation, f = Energy

4:$\triangleright$ R = All previously input or generated shapes

5: $l=E(W)$ $\triangleright$ latent code

6: $\overline{H}\approx\nabla_{l}\nabla_{l}f(D(l))$ $\triangleright$
approximate Hessian

7: $\lambda,U^{\uparrow}(H)=\text{EigenDecomposition}(H)$

8: $\lambda_{k},U^{\uparrow}_{k}(H)\leftarrow\lambda,U^{\uparrow}(H)$
$\triangleright$ retain k components

9: $W_{d}=\emptyset$

10: for j $\in$ [1,s]: do

11: $\beta\sim\mathcal{N}(0,\mathbb{I})\in R^{k}$ $\triangleright$ sample
$\beta$, $k\ll K$

12: $\hat{\beta}=\beta/\sum_{i=1}^{k}\beta_{i}^{2}$

13: $\alpha=\sqrt{2\delta/\sum_{i=1}^{i=k}\hat{\beta}_{i}^{2}\lambda_{i}}$

14: $\hat{W}_{j}=D(l+\alpha\sum_{i=1}^{k}\hat{\beta}_{i}U^{\uparrow}_{i}(H))$

15: $W_{d}\leftarrow W_{d}\cup\hat{W}_{j}$ $\triangleright$ add to candidates

16: end for

17: $W^{*}=\text{MMR}(W,R,W_{d})$ $\triangleright$ prune candidates

18: $W_{\text{Projected}}=\arg\min f(W^{*})$ $\triangleright$ project

19: $R\leftarrow R\cup\\{W_{\text{Projected}}\\}$ $\triangleright$ augment
training set

20:end procedure

Algorithm 1 Pseudocode for searching the latent space, starting from
deformation $W$, for a new augmenting shape.

###  3.3 Augmenting via Latent Space Exploration

We now describe the main step of our technique – adding additional deformation
examples $W^{j}$ to the latent space to reinforce training (Algorithm 1).
Simply optimizing (4) is not enough to cover the deformation space. Instead,
we continuously introduce new low-energy deformations into the training set,
by which we make the _data term aware of the deformation energy_. We achieve
this through three steps: (i) deformation-aware _perturbation_ of the latent
code in directions that locally least modify the deformation energy; (ii)
data-driven _pruning_ of perturbed codes that do not introduce variance to the
current dataset; and (iii) deformation-aware _projection_ of the new codes to
further lower their deformation energy, with an optional _high-resolution
projection_ to transfer the deformation from a low-resolution mesh to a higher
resolution one. Figure 3 illustrates how the latent space is progressively
populated with new deformations over iterations, where colors indicate the
base shapes.

#### (i) Deformation-aware perturbation in latent space.

Our goal is to create variations of a given code in latent space without
modifying its deformation energy significantly: let $W$ be a deformation, and
$l=E(W)\in\mathbb{R}^{K}$ a latent code achieved from encoding it. We aim to
find the low-energy perturbation modes of $l$. In short, we aim to perturb the
deformation while not changing the deformation energy too much, or in other
words – we wish to stay on the current level set of the energy. To achieve
that, we can restrict ourselves to perturbations on the local _tangent space_
of the energy’s level set. This tangent space simply comprises of all
directions orthogonal to the gradient $\nabla_{l}f(D(l))$.

In the tangent space, we can pick better directions using a second order
analysis. Let $H$ denote the Hessian of the deformation energy with respect to
the latent code,

$\begin{split}H:=\nabla_{l}\nabla_{l}f(D(l)).\end{split}\vspace{-2mm}$ (5)

Let $\lambda_{i}$, and $U^{\uparrow}_{i}(H)$ respectively denote the
eigenvalues and eigenvectors of $H$, in ascending order of eigenvalues. Since
smaller eigenvalues correspond to directions of less change in energy, we
retain only the $k$ ($k<<K$) smallest $\lambda_{i}$ and $U^{\uparrow}_{i}(H)$.
We then draw a random perturbation, by sampling a random vector
$\beta\in\mathbb{R}^{k}$ from a normal distribution. Each $\beta_{i}\in\beta$
corresponds to step along the eigenvector $U^{\uparrow}_{i}(H)$. We normalize
$\beta$ to $\hat{\beta}$ such that
$\sum_{i=1}^{k}{\hat{\beta}_{i}U^{\uparrow}_{i}(H)}$ is a unit vector, i.e.,
$\sum_{i=1}^{k}\hat{\beta}_{i}^{2}=1$. We then take a step,

$l_{t}:=l+\alpha\sum_{i=1}^{k}{\hat{\beta}_{i}U^{\uparrow}_{i}(H)},\vspace{-2mm}$
(6)

where $\alpha$ denotes the step-size and $l_{t}$ is in the tangent plane. We
repeat this process $s$ times for each latent code $l$ to get $s$ perturbed
codes $\tilde{l}^{1},\tilde{l}^{2},\dots\tilde{l}^{s}$. Let
$\\{\tilde{W}^{1},\tilde{W}^{2}\dots\tilde{W}^{s}\\}$ denote the decoded
perturbed deformations where $\tilde{W}^{j}=D(\tilde{l}^{j})$.

Variable step size: Different regions of the latent space have different local
deformation landscapes, e.g., curvatures along different directions on the
tangent plane. Hence, we should adapt $\alpha$ to the nature of the local
landscape around $l$. To that end, we formulate the step size $\alpha$ in
terms of the eigenvalues $\lambda_{i}$ and a user-prescribed threshold
$\delta$ on the allowed deformation energy $f$ (i.e., $f()\leq\delta$).
Assuming C2 continuity for deformation energy $f()$, we can obtain the
following bound on the step size (see supplemental),

$\alpha\leq\sqrt{\frac{2\delta}{\sum_{i=1}^{k}\hat{\beta}_{i}^{2}\lambda_{i}}}.$

This gives an upper bound on the step size along any deformation direction.

#### (ii) Data-driven pruning of the perturbed deformations.

In order to add diverse samples, given the set of candidate deformations
$\\{\tilde{W}^{j}\\}$, we select one example to be added to the dataset, via
Maximal Marginal Relevance (MMR) ranking [9]. Specifically, MMR gives a higher
score to perturbations that are similar to the unperturbed $W$, but different
from the existing deformations set $R$, containing the landmarks and
deformations generated so far. We compute as,

$\displaystyle\begin{split}F(w)=\gamma M(w,W)-(1-\gamma)\max_{r\in
R}M(w,r),\end{split}\vspace{-2mm}$ (7)

where $M(x,y)$ is the cosine similarity between $x$ and $y$. Thus, we choose
the deformation ${W^{*}\in W_{d}}$ that maximizes the MMR function with its
latent code denoted $l^{*}$.

#### (iii) Deformation-aware projection to smooth, low-energy deformations.

Although the deformation-aware perturbation somewhat avoids high-energy
states, the perturbed deformation $W^{*}$ may still exhibit undesirable
artifacts such as lack of smoothness or high deformation energy. Hence, we
project the code to have lower deformation energy _with respect to the
unperturbed $W$_. We achieve this by treating $W$ as the rest pose, defining a
deformation energy with respect to it, $f_{W}$. We perform a fixed number of
gradient descent steps starting from $W^{*}$ to lower the energy, which yields
the final deformation $W_{\text{Projected}}$. In our experiments, we optimize
up to the threshold of $10^{-5}$.

#### (iv) Augment and iterate:

Finally, we append the newly generated deformations to the current (training)
set, and continue training. We repeat this augmentation and retraining several
times, until we reach a target training set size.

### Implementation Choices

Choice of energy $f$. For training the VAE, we set $f$ as the $L2$ formulation
of the ARAPReg energy [20]. ARAPReg is an efficient approximation of the ARAP
energy [36] that is conducive for network training. This energy computes an
approximate Hessian $\overline{H}$ of ARAP with respect to the latent space,
and directly minimizes the eigenvalues of $\overline{H}$.
Approximation of Hessian. Similarly, for step (i) in Section 3.3, we use the
approximate Hessian $\overline{H}\approx J^{T}HJ$ proposed in [20] in our
Equations 5 and 6, where $H$ is the exact Hessian of ARAP and $J$ is the
stochastic Jacobian of the ARAP with respect to the $\mathbb{R}^{K}$ latent
space.
The above choices speed up training and yield better results than classical
ARAP. We still use ARAP for step (iii) where Hessian approximation is not
needed.
Upper bound for $\alpha$. The eigenvalues allow us to choose an appropriate
local step-size, but they only yield a local approximation of the deformation
energy. As our method continues adding low-energy shapes, the eigenvalues
become lower, leading to an extremely large step size $\alpha$. Thus, we set
an upper bound of $2$ on $\alpha$.
High-resolution projection. To speed up training and avoid deformations that
are too high-frequency, we use low-resolution meshes (low vertex count). We
decimate the high-resolution meshes by preserving a subset of the vertices,
thus retaining correspondence from low to high. The generated low-res
deformations can later be projected back to original high-res meshes by
treating the chosen subset of vertices as deformation constraints in the ARAP
optimization proposed in [36].

Figure 4: Generation results evaluated by coverage. We train different methods
on the same training data (col 1) and generate comparable numbers of shapes.
Given two shapes from the holdout data (col 2), we evaluate the methods by
finding the closest generated shape (cols 3-9). Note how the baselines exhibit
strong artifacts and usually do not match the query shape. Figure 5: Training
Glass on the human, centaur, and horse meshes using the 3 examples each (top).
(Bottom) We show random samples from the latent space, which combine different
properties learned from the example deformations.

##  4 Experiments

We evaluate Glass on public data of humans (FAUST [5]) and creatures (TOSCA
[8]) in different poses. In our experiments, we sample $X$ landmark poses of a
model from a dataset and train our method. We evaluate quality and diversity
of newly generated poses as well as interpolation sequences between landmark
poses. We denote our experiments as “SubjectName-$X$” to indicate the type of
the subject and the number of landmark poses provided as an input to our
method; most results use between 3 and 10 landmarks.

Figure 6: We compare the interpolation results between our method, several
ablations of our method, and prior work.

#### Evaluation metrics.

We use three metrics - Coverage, Mesh smoothness and Interpolaiton smoothness
\- to evaluate our performance. For each metric, lower values are better. The
metrics are detailed in the supplemental.

Figure 7: _Interpolation results._ In gray, we show two landmark shapes. In gold, we show the decoded meshes after we linearly interpolate the latent space between these two landmarks. All models are trained on only 5 landmarks. Table 1: Surface smoothness and coverage with respect to excluded set, of generated samples. Lower is better.  Data |  Vanilla VAE |  +Interpolation |  +Interp. +Energy | LIMP | Deep-MCMC |  +ARAP SI [1] | Glass
---|---|---|---|---|---|---|---
Faust-3 | 1.0 / 1.0 | 0.73 / 0.96 | 0.75 / 0.89 | 1.06 / 1.01 | 1.04 / 0.95 | 0.77 / 1.09 | 0.63 / 0.77
Faust-5 | 1.0 / 1.0 | 0.65 / 1.04 | 0.62 / 0.88 | 1.0 / 0.96 | 1.06 / 0.94 | 0.62 / 0.97 | 0.59 / 0.78
Faust-7 | 1.0 / 1.0 | 1.00 / 1.51 | 0.57 / 1.61 | 1.07 / 0.93 | 1.01 / 0.96 | 0.85 / 1.84 | 0.54 / 0.81
Centaurs-3 | 1.0 / 1.0 | 0.83 / 0.96 | 0.85 / 0.94 | 1.03 / 0.97 | 1.04 / 0.98 | 0.85 / 0.99 | 0.69 / 0.84
Centaurs-4 | 1.0 / 1.0 | 0.81 / 0.97 | 0.84 / 0.95 | 1.01 / 0.99 | 1.08 / 0.96 | 0.81 / 1.0 | 0.69 / 0.84
Horses-3 | 1.0 / 1.0 | 0.66 / 0.95 | 0.73 / 0.91 | 1.06 / 1.03 | 1.02 / 1.05 | 0.65 / 0.94 | 0.61 / 0.89
Horses-4 | 1.0 / 1.0 | 0.65 / 0.94 | 0.68 / 0.95 | 1.03 / 0.98 | 1.04 / 1.04 | 0.62 / 1.0 | 0.60 / 0.80
Table 2: L2 Error wrt excluded DFaust frames and reconstruction error of those excluded frames. Lower is better.  Data |  Vanilla VAE |  +Interpolation |  +Interp. +Energy | LIMP |  +ARAP SI [1] | Glass
---|---|---|---|---|---|---
DFaust-1 | 1.0 / 1.0 | 0.48 / 0.51 | 0.48 / 0.40 | 0.53 / 0.56 | 0.94 / 0.92 | 0.45 / 0.40
DFaust-2 | 1.0 / 1.0 | 0.56 / 0.48 | 0.51 / 0.42 | 1.06 / 1.07 | 0.84 / 0.81 | 0.47 / 0.40
DFaust-3 | 1.0 / 1.0 | 0.61 / 0.57 | 0.51 / 0.44 | 1.08 / 2.0 | 0.77 / 0.8 | 0.45 / 0.41
DFaust-4 | 1.0 / 1.0 | 0.59 / 0.35 | 0.57 / 0.5 | 0.84 / 1.73 | 0.87 / 1.14 | 0.46 / 0.32
DFaust-5 | 1.0 / 1.0 | 0.27 / 0.38 | 0.3 / 0.27 | 0.53 / 1.28 | 0.67 / 1.2 | 0.22 / 0.24

#### Method Comparison

While existing methods are not designed to learn generative latent spaces from
sparse data, we adapt them as baselines. Since ARAP projection to high
resolution is specific to our method (see 3), we compare methods using the
low-res version that we train with.

(i) Vanilla VAE: We train a VAE using only the sparse set of shapes, with no
data augmentation. (ii) +Interpolation: We generate new shapes by
interpolating between all pairs of available landmarks by simply averaging
coordinates of corresponding vertices. We interpolate such that the amount of
augmented data is equivalent to what is generated with Glass (2500 shapes).
Then we train a VAE with these poses. (iii)  +Interp. +Energy: This is an
extension of the previous method. Since raw interpolation can deviate from the
underlying shape space, in addition to interpolation, we perform projection by
minimizing the sum of ARAP energies with respect to both shapes in the pair.
(iv)  LIMP-ARAP [12]: This method is motivated by the training strategy
proposed in LIMP [12]. They train a VAE with pairs of shapes in every
iteration - for each pair, they pick a random latent code on the line between
the two, decode it to a new shape, and minimize its energy. Since we only want
to compare augmentation strategies we adapt LIMP to use ARAP energy. (v) \+
ARAP Shape Interpolation [1]: This method morphs a shape from a source pose to
a target pose. The morph is rigid in the sense that local volumes are least-
distorting as they vary from their source to target configurations. (vi) Deep-
MCMC [33]: This method explores parameter variations via a latent space and
generate samples by performing HMC steps in the latent space and decoding the
generated codes.

Figure 8: In gray, are 2 subsets of 3 and 6 facial expressions from the COMA
dataset. Training Glass on each of them, correspondingly generates the novel
expressions in gold.

#### Generation Experiments

After training, we sample latent codes from a Unit Gaussian in
$\mathbb{R}^{K}$, and decode with our decoder to generate samples (see Figures
1, 5 and 8). Our generated poses look substantially different from the
training data and combine features from multiple input examples.

We compare our approach to all the baselines. We sample from the unit Gaussian
for all VAE-based techniques, where the only exception is Deep-MCMC where we
use latent-space HMC as proposed in their work. We show qualitative results in
Figure 4 and quantitative evaluations in Table 1. Each cell reports smoothness
and coverage errors, normalized based on the corresponding errors for Vanilla-
VAE. Note that our method outperforms all baselines in its ability to generate
novel and plausible poses (i.e., the poses from the hold-out set of the true
poses).

Table 3: Surface smoothness, ARAP energy, and standard deviation of inter-frame spacing between landmarks by interpolation across different datasets. All results are normalized such that Vanilla VAE is 1.0, and lower numbers are better. Data |  Vanilla VAE |  +Interpolation |  +Interp. +Energy |  LIMP |  +ARAP SI | Glass
---|---|---|---|---|---|---
Faust-3 | 1.0 / 1.0 / 1.0 | 0.47 / 0.22 / 0.43 | 0.44 / 0.24 / 0.29 | 0.45 / 0.32 / 0.7 | 0.95 / 1.49 / 0.91 |  0.42/0.20 / 0.21
Faust-5 | 1.0 / 1.0 / 1.0 | 0.53 / 0.25 / 0.49 | 0.53 / 0.25 / 0.41 | 0.52 / 0.27 / 0.5 | 0.99 / 1.13 / 0.87 |  0.51 / 0.23 / 0.38
Faust-7 | 1.0 / 1.0 / 1.0 | 0.64 / 0.31 / 0.49 |  0.57 / 0.26 / 0.62 | 0.62 / 0.32 / 0.72 | 0.84 / 0.57 / 0.86 |  0.57 / 0.3 / 0.23
Faust-10 | 1.0 / 1.0 / 1.0 | 0.71 / 0.37 / 0.3 | 0.67 / 0.43 / 0.27 | 0.66 / 0.36 / 1.52 | 0.78 / 0.48 / 0.55 |  0.55 / 0.27 / 0.54
Centaurs-3 | 1.0 / 1.0 / 1.0 | 0.53 / 0.31 / 0.22 | 0.51 / 0.31 / 0.48 | 0.52 / 0.34 / 0.24 | 0.89 / 1.28 / 2.12 |  0.5 / 0.28 / 0.18
Centaurs-4 | 1.0 / 1.0 / 1.0 | 0.53 / 0.24 / 0.27 | 0.52 / 0.26 / 0.17 |  0.51 / 0.25 / 0.48 | 0.88 / 0.88 / 0.8 |  0.49 / 0.25 / 0.1
Centaurs-6 | 1.0 / 1.0 / 1.0 |  0.59 / 0.26 / 0.33 | 0.65 / 0.38 / 0.5 | 0.6 / 0.27 / 0.47 | 0.98 / 1.02 / 0.44 |  0.58 / 0.22 / 0.43
Horses-3 | 1.0 / 1.0 / 1.0 | 0.44 / 0.39 / 0.33 |  0.46 / 0.33 / 0.28 | 0.44 / 0.4 / 0.43 | 0.9 / 1.41 / 1.28 |  0.42 / 0.24 / 0.43
Horses-4 | 1.0 / 1.0 / 1.0 | 0.48 / 0.29 / 0.28 | 0.47 / 0.3 / 0.45 |  0.45 / 0.36 / 0.85 | 0.94 / 1.25 / 1.42 |  0.48 / 0.22 / 0.44
Horses-8 | 1.0 / 1.0 / 1.0 | 0.55 / 0.31 / 0.55 | 0.56 / 0.38 / 0.7 |  0.53 / 0.43 / 1.29 | 0.83 / 0.76 / 0.74 |  0.53 / 0.25 / 0.28
Table 4: Ablation study results. Data | 1: Vanilla VAE |  2: (1)+$L_{\text{deform}}$ | 3: (1) + perturb | 4: (2) + perturb | 5: (3)+project | 6: (4)+project
---|---|---|---|---|---|---
Faust-10 | 1.0 / 1.0 / 1.0 | 0.63 / 0.39 / 0.6 | 0.61 / 0.35 / 0.6 | 0.59 / 0.32 / 0.59 | 0.56 / 0.32 / 0.57 |  0.55 / 0.27 / 0.54
Centaurs-6 | 1.0 / 1.0 / 1.0 | 0.69 / 0.34 / 0.54 | 0.62 / 0.31 / 0.47 | 0.59 / 0.29 / 0.47 | 0.59 / 0.27 / 0.46 |  0.58 / 0.22 / 0.43
Horses-8 | 1.0 / 1.0 / 1.0 | 0.66 / 0.37 / 0.43 | 0.59 / 0.37 / 0.33 | 0.56 / 0.29 / 0.32 | 0.54 / 0.27 / 0.29 |  0.53 / 0.25 / 0.28

#### Interpolation Experiments

We compare our method and the first five baselines by evaluating the quality
of interpolations produced between all pairs of landmark shapes (we omit Deep-
MCMC since it is not suitable for interpolation). We show our results in
Figure 7 and comparisons in Figure 6 with corresponding stats in Table 3. Each
cell reports smoothness, ARAP score, and interpolation quality, and to make
results more readable we normalize the scores using the corresponding value
for Vanilla VAE.

Ours performs the best with respect to ARAP score and also yields consistently
smoother shapes with fewer artifacts, both in terms of individual surfaces, as
well as discontinuities in interpolation sequences. ARAP Shape Interpolation
is performing consistently worse than all baselines except Vanilla-VAE.
Interpolation-based baselines sometimes yield smoother interpolations,
something we would expect to be true for simpler, linear motions. However, as
seen in Figure 6, an outlier global rotation can disturb the otherwise smooth
interpolation, which does not happen with Glass. Additionally, we demonstrate
that they are limited in their ability to synthesize novel plausible poses.

Table 5: Correspondence error on the Faust INTRA benchmark, by Glass-augmenting 3D-CODED with deformations sampled from our method. Data |  +Glass augmentation |  Error (cm)
---|---|---
Faust-3 |  0 | 26.90
Faust-3 | 3,065 | 13.18
Faust-7 |  0 | 22.10
Faust-7 | 3,573 | 11.78
SMPL-280 |  0 | 14.59
SMPL-280 | 40,000 | 6.85

#### Dynamic Faust Interpolation

Dynamic Faust [6] (DFaust) provides meshed data of captured motion sequences,
from which we selected 5 sequences with the most variance in vertex positions.
Each sequence contains 100-200 shapes from which we select $\approx$ 5
keypoints and train Glass and the baselines on these. We then evaluate
interpolation by measuring the $L2$ distance between the generated
interpolations and the ground-truth frames in the sequence that were excluded
from training. Additionally, we measure the reconstruction error of the
excluded frames. The results are compared in Table 2. Our method significantly
outperforms the baselines.

#### Using Glass for Learning Correspondences

We now evaluate our data augmentation technique on the practical task of
learning 3D correspondences between shapes. We pick a state-of-the-art
correspondence learning method, 3D-CODED [18], as a reference. Originally,
this method was trained on 230k deformations, most of them sampled from the
SMPL model [30] and 30k synthetic augmentations. We evaluate how well this
method could perform with a smaller training set, with and without the
proposed augmentation.

We train 3D-CODED using 280 sampled shapes from SMPL [30] that are augmented
with a number of additional deformations generated from our model (see Table
5). Ours consistently provides a significant improvement over training
3D-CODED with the original landmarks. For reference, the correspondence error
of 3D-CODED trained on the full 230k pose dataset is 1.98cm.

#### Ablation study

In this section we evaluate the contribution of various steps in Glass to our
interpolation metrics. We evaluate on the Faust-10, Centaurs-6 and Horses-8
datasets. Since these samples are all that is available we do not have a hold-
out set, so we do not measure coverage. Starting with Vanilla-VAE (that only
uses $L_{\text{Reconstruction}}$ and $L_{\text{Gaussian}}$), we first add the
deformation energy loss ($L_{\text{Deformation}}$), which is ARAPReg [20].
Table 4(1,2) shows this improves all metrics.

Next, we consider our perturbation strategy (Section 3.3 i, ii) and add it to
both vanilla and energy-guided VAE (Table 4.3, 4.4). We observe that
$L_{\text{Deformation}}$ performs better because it makes the latent-space
conducive for sampling low energy shapes. Having $L_{\text{Deformation}}$
helps our perturbation strategy find low energy shapes that are suitable for
our projection step. Due to this, we discover shapes with energy as low as
$0.001$, while without it, the discovered shapes can have energy $>0.1$. This
difference helps the subsequent projection step converge faster to our
required threshold of $10^{-5}$.

Finally, we look at the projection step (Section 3.3 iii). We add it to both
baseline techniques that have perturbation, and report results in Table 4.5,
4.6, where column (6) corresponds to our final method. Adding the projection
step improves the smoothness and ARAP scores. After projection, our shapes
have very low ARAP, in the order of $10^{-5}$. Since these are added back to
the training set, we observe that perturbation steps in future iterations find
lower energy shapes. This further improves convergence of the projection step
in future iterations. Overall $L_{\text{Deformation}}$ helps both the
perturbation and projection steps converge faster to low energy shapes, and
since projected shapes are encoded again by training, both perturbation and
projection steps require fewer iterations.

##  5 Conclusion

Glass is shown to be an effective generative technique for 3D shape
deformations, relying solely on a handful of examples and a given deformation
energy. The main limitation of our method is its reliance on a given mesh with
vertex correspondences, preventing its use on examples with different
triangulations, and we set the goal of generalizing it to arbitrary geometries
as important future work.

We believe our proposed technique opens many future directions. There are many
other deformation energies that could be explored; e.g., densely sampling
conformal (or quasi-conformal) deformations from a given sparse set can be an
extremely interesting followup. More broadly, replacing the deformation energy
with learned energies, such as the output of an image-discriminator, may
enable generating plausible images, given a very sparse set of examples.

#### Acknowledgement

This project has received funding from the UCL AI Center, gifts from Adobe
Research, and EU Marie Skłodowska-Curie grant agreement No 956585.

## References

  * [1] Marc Alexa, Daniel Cohen-Or, and David Levin.  As-rigid-as-possible shape interpolation.  In SIGGRAPH, 2000.
  * [2] Brett Allen, Brian Curless, and Zoran Popović.  The space of human body shapes: Reconstruction and parameterization from range scans.  In SIGGRAPH, 2003.
  * [3] Dragomir Anguelov, Praveen Srinivasan, Daphne Koller, Sebastian Thrun, Jim Rodgers, and James Davis.  SCAPE: Shape completion and animation of people.  In SIGGRAPH, 2005.
  * [4] Volker Blanz and Thomas Vetter.  A morphable model for the synthesis of 3D faces.  In SIGGRAPH, 1999.
  * [5] Federica Bogo, Javier Romero, Matthew Loper, and Michael J. Black.  FAUST: Dataset and evaluation for 3D mesh registration.  In CVPR, 2014.
  * [6] Federica Bogo, Javier Romero, Gerard Pons-Moll, and Michael J. Black.  Dynamic FAUST: Registering human bodies in motion.  In CVPR, 2017.
  * [7] Mario Botsch and Olga Sorkine.  On linear variational surface deformation methods.  TVCG, 14(1), 2008.
  * [8] Alexander M Bronstein, Michael M Bronstein, and Ron Kimmel.  Numerical geometry of non-rigid shapes.  Springer Science & Business Media, 2008.
  * [9] Jaime Carbonell and Jade Goldstein.  The use of MMR, diversity-based reranking for reordering documents and producing summaries.  In SIGIR, 1998.
  * [10] Amit Chaudhary.  A visual survey of data augmentation in NLP, 2020.  https://amitness.com/2020/05/data-augmentation-for-nlp.
  * [11] Siddhartha Chaudhuri, Daniel Ritchie, Jiajun Wu, Kai Xu, and Hao Zhang.  Learning generative models of 3D structures.  Comput. Graph. For. (Eurographics STAR), 2020.
  * [12] Luca Cosmo, Antonio Norelli, Oshri Halimi, Ron Kimmel, and Emanuele Rodolà.  LIMP: Learning latent shape representations with metric preservation priors.  ECCV, 2020.
  * [13] Matheus Gadelha, Giorgio Gori, Duygu Ceylan, Radomir Měch, Nathan Carr, Tamy Boubekeur, Rui Wang, and Subhransu Maji.  Learning generative models of shape handles.  In CVPR, 2020.
  * [14] Matheus Gadelha, Rui Wang, and Subhransu Maji.  Multiresolution tree networks for 3D point cloud processing.  CoRR, abs/1807.03520, 2018.
  * [15] Lin Gao, Yu-Kun Lai, Jie Yang, Ling-Xiao Zhang, Shihong Xia, and Leif Kobbelt.  Sparse data driven mesh deformation.  TVCG, 27(3), 2021.
  * [16] Lin Gao, Jie Yang, Tong Wu, Yu-Jie Yuan, Hongbo Fu, Yu-Kun Lai, and Hao Zhang.  SDM-NET: Deep generative network for structured deformable mesh.  ACM Trans. Graph., 38(6), 2019.
  * [17] Ian Goodfellow, Jonathon Shlens, and Christian Szegedy.  Explaining and harnessing adversarial examples.  In ICLR, 2015.
  * [18] Thibault Groueix, Matthew Fisher, Vladimir G. Kim, Bryan C. Russell, and Mathieu Aubry.  3D-CODED: 3D correspondences by deep deformation.  ECCV, 2018.
  * [19] Frédéric Hélein and John C. Wood.  Handbook of Global Analysis, chapter Harmonic Maps.  2008\.
  * [20] Qixing Huang, Xiangru Huang, Bo Sun, Zaiwei Zhang, Junfeng Jiang, and Chandrajit Bajaj.  ARAPReg: An as-rigid-as possible regularization loss for learning deformable shape generators.  CoRR, abs/2108.09432, 2021.
  * [21] Qi-Xing Huang, Martin Wicke, Bart Adams, and Leonidas Guibas.  Shape decomposition using modal analysis.  Computer Graphics Forum, 28(2), 2009.
  * [22] Alec Jacobson, Ilya Baran, Jovan Popović, and Olga Sorkine.  Bounded biharmonic weights for real-time deformation.  ACM Trans. Graph., 30(4), 2011.
  * [23] Alec Jacobson, Zhigang Deng, Ladislav Kavan, and J. P. Lewis.  Skinning: Real-time shape deformation.  In SIGGRAPH Courses, 2014.
  * [24] Tao Ju, Scott Schaefer, and Joe Warren.  Mean value coordinates for closed triangular meshes.  In SIGGRAPH, 2005.
  * [25] Martin Kilian, Niloy J. Mitra, and Helmut Pottmann.  Geometric modeling in shape space.  ACM Trans. Graph., 26(3), 2007.
  * [26] Diederik P. Kingma and Max Welling.  Auto-encoding variational bayes.  In ICLR, 2014.
  * [27] Hamid Laga.  A survey on non-rigid 3D shape analysis.  CoRR, abs/1812.10111, 2018.
  * [28] Bruno Lévy, Sylvain Petitjean, Nicolas Ray, and Jérôme Maillot.  Least squares conformal maps for automatic texture atlas generation.  In SIGGRAPH, 2002.
  * [29] J. P. Lewis, Ken Anjyo, Taehyun Rhee, Mengjie Zhang, Fred Pighin, and Zhigang Deng.  Practice and theory of blendshape facial models.  Eurographics State of the Art Reports, 2014.
  * [30] Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J. Black.  SMPL: A skinned multi-person linear model.  ACM Trans. Graphics (Proc. SIGGRAPH Asia), 34(6), 2015.
  * [31] Anurag Ranjan, Timo Bolkart, Soubhik Sanyal, and Michael J Black.  Generating 3D faces using convolutional mesh autoencoders.  In ECCV, 2018.
  * [32] Guodong Rong, Yan Cao, and Xiaohu Guo.  Spectral mesh deformation.  The Visual Computer, 24, 2008.
  * [33] Babak Shahbaba, Luis Martinez Lomeli, Tian Chen, and Shiwei Lan.  Deep Markov Chain Monte Carlo.  CoRR, abs/1910.05692, 2019.
  * [34] Shiv Shankar, Vihari Piratla, Soumen Chakrabarti, Siddhartha Chaudhuri, Preethi Jyothi, and Sunita Sarawagi.  Generalizing across domains via cross-gradient training.  In ICLR, 2018.
  * [35] Connor Shorten and Taghi M. Khoshgoftaar.  A survey on image data augmentation for deep learning.  Journal of Big Data, 6(1), 2019.
  * [36] Olga Sorkine and Marc Alexa.  As-rigid-as-possible surface modeling.  In SGP, 2007.
  * [37] Qingyang Tan, Lin Gao, Yu-Kun Lai, and Shihong Xia.  Variational autoencoders for deforming 3D mesh models.  In CVPR, 2018.
  * [38] Qingyang Tan, Zherong Pan, Lin Gao, and Dinesh Manocha.  Realtime simulation of thin-shell deformable materials using CNN-based mesh embedding.  IEEE Robotics and Automation Letters, 5(2), 2020.
  * [39] Christoph Von-Tycowicz, Christian Schulz, Hans-Peter Seidel, and Klaus Hildebrandt.  Real-time nonlinear shape interpolation.  ACM Trans. Graph., 34(3), 2015.
  * [40] Yifan Wang, Noam Aigerman, Vladimir G. Kim, Siddhartha Chaudhuri, and Olga Sorkine-Hornung.  Neural cages for detail-preserving 3D deformations.  In CVPR, 2020.
  * [41] Qingsong Wen, Liang Sun, Fan Yang, Xiaomin Song, Jingkun Gao, Xue Wang, and Huan Xu.  Time series data augmentation for deep learning: A survey.  CoRR, abs/2002.12478, 2021.
  * [42] Yu-Jie Yuan, Yu-Kun Lai, Tong Wu, Lin Gao, and Ligang Liu.  A revisit of shape editing techniques: from the geometric to the neural viewpoint.  CoRR, abs/2103.01694, 2021.
  * [43] Mehmet Ersin Yumer and Levent Burak Kara.  Co-constrained handles for deformation in shape collections.  ACM Trans. Graph., 33(6), 2014.

Appendix for Glass: Geometric Latent Augmentation for Shape Spaces

##  Appendix A Evaluation metrics.

We use the following to evaluate performance:
(i) Coverage: While it is difficult to evaluate whether generated poses are
meaningful and diverse, we propose using the holdout data ($S_{H}$) that was
not part of the input exemplars to see if the newly generated poses $L_{G}$
cover every holdout example:
$M_{\text{coverage}}:=\sum_{s\in S_{H}}{\min_{g\in L_{G}}{D(s,g)}}/{|S_{H}|},$
where $D(s,g)$ is the average Euclidean distance between corresponding
vertices of shapes $s$ and $g$.
(ii) Mesh smoothness: This metric measures how well a method preserves the
original intrinsic structure of the mesh. We compute the mean curvature
obtained from the discrete Laplace-Beltrami operator:
$M_{\text{smoothness}}:=\sum_{i=1}^{N}\lVert\Delta(V_{i})\rVert/2$ where
$\Delta$ is the area-normalized cotangent Laplace-Beltrami operator and $N$ is
the number of mesh vertices.
(iii) Interpolation smoothness: In addition to measuring quality of individual
meshes, we also evaluate the quality of interpolations between pairs of
shapes. Since none of the existing methods guarantee a clear relationship
between the distances in the latent space and differences between their 3D
counterparts, we first densely sample 1000 poses between pairs of landmark
deformations and then keep a subset of 30 poses so that they have
approximately equal average Euclidean distance between subsequent frames. We
then measure the standard deviation of these Euclidean distances, as a way to
penalize interpolations that yield significant jumps between frames. While
this measure is imperfect (e.g., variance could decrease as we increase the
sampling rate), we found it to stabilize in practice after 1000 poses (we
sampled up to 3000), which suggests that denser sampling would not reveal new
poses in the latent space.

##  Appendix B Variable step-size $\alpha$

Let $f$ be the energy function and $l$ be the latent code corresponding to a
zero/low energy shape in the training set, and $l_{t}$ be the latent code
obtained from the update in Equation 6 in the main paper under a small
(update) step (i.e., $\|l_{t}-l\|$ can be considered to be infinitesimal).
Assuming that the deformation energy is C2 continuous (and single valued),
$f(l_{t})$ can be approximated using Taylor series expansion of the up to the
2 order as,

$f(l_{t})\approx
f(l)+(l_{t}-l)^{T}\nabla_{l}f(l_{t})+\frac{1}{2}(l_{t}-l)^{T}H(l_{t}-l),$ (8)

where $H$ is the Hessian from Equation 5. Note that $H$ denotes the Hessian
term with respect to the current pose $l$. Since $(l_{t}-l)$ and
$\nabla_{l}f(l_{t})$ are orthogonal, $(l_{t}-l)^{T}\nabla_{l}f(l_{t})=0$. The
update step can be expressed in terms of the local eigenvectors as (see
Equation 6 in the main paper),

$(l_{t}-l)=\alpha\sum_{i=1}^{i=k}{\hat{\beta}_{i}U^{\uparrow}_{i}(H)}.$

Further, since eigenvalues and eigenvectors are related as
$HU^{\uparrow}_{i}(H)=\lambda_{i}U^{\uparrow}_{i}(H)$ and the vectors
$U^{\uparrow}_{i}(H)$ have unit length, we obtain,

$f(l_{t})\approx
f(l)+\frac{1}{2}\alpha^{2}\sum_{i=1}^{i=k}\hat{\beta}_{i}^{2}\lambda_{i}.$

By setting an upper bound on the change in deformation energy
$f(l_{t})-f(l)\leq\delta$, we obtain the relation for $\alpha$ as,

$\alpha\leq\sqrt{\frac{2\delta}{\sum_{i=1}^{k}\hat{\beta}_{i}^{2}\lambda_{i}}}.$

##  Appendix C Additional Generated Samples

Figures 9, 10, and 11 show some example generated deformations starting from
sparse sets of Faust, Centaur, and Horse models, respectively.

Figure 9: We show some of the samples generated by Glass (gold), from only 10
Faust poses (gray). Several poses of the limbs are unseen in the training set
- crossed arms, long leg strides, half-lowered arms. Figure 10: We show some
of the samples generated by Glass (gold), from only 6 Centaur poses (gray). We
see novel poses like bent back legs and torso facing upwards. Figure 11: We
show some of the samples generated by Glass (gold), from only 8 Horse poses
(gray). We see novel poses like front legs raised beyond what’s seen in the
training set, upright torso and back legs stretching farther.

##  Appendix D Additional Interpolation Examples

Figures 12, 13 and 14 show example interpolated shapes between end poses
(shown in gray), using the discovered latent space revealed by Glass.

Figure 12: Interpolated shapes (gold) between 2 Centaurs Poses (gray) inside
the latent space generated using Glass. Figure 13: Interpolated shapes (gold)
between 2 Horse Poses (gray) inside the latent space generated using Glass.
Figure 14: Interpolated shapes (gold) between 2 Faust Poses (gray) inside the
latent space generated using Glass.

##  Appendix E Dataset Images

We include images of the various input datasets we used to highlight the
diversity of pose variations in the input. Note that the datasets are all very
sparse, consisting of 3-10 models.

Figure 15: The Faust-10 dataset with 10 poses. Please refer to Figure 16 for
the corresponding even sparser versions of the dataset used in our
experiments.  Figure 16: Faust-3 (top left), Faust-5 (top right) and Faust-7
(bottom). Figure 17: Centaurs-6 (top), Centaurs-3 (bottom left), Centaurs-4
(bottom right). Figure 18: Horses-8 (top), Horses-3 (bottom left), Horses-4
(bottom right). Figure 19: The keypoints of 5 different Dynamic Faust
sequences used for training, for Table 2 of main paper.

##  Appendix F Network Architecture

Figure 20 shows the VAE architecture used by Glass. The main pseudocode for
Glass is provided in the main paper.

Figure 20: The VAE architecture used by GLASS.

##  Appendix G Extrapolation

We extrapolate beyond the endpoints of interpolation for the same length as
interpolation and measure the smoothness of the mesh, and report the results
in table 6. We observe that extrapolated meshes have a smoother surface for
Glass than the baselines.

Table 6: Ablation study. Surface smoothness of extrapolated shapes. All results are normalized such that Vanilla VAE is 1.0, and lower numbers are better. Data |  Vanilla VAE |  +Interpolation |  +Interp. +Energy |  LIMP |  +ARAP SI [1] | Glass
---|---|---|---|---|---|---
Faust-3 | 1.0 | 0.74 | 0.79 | 0.87 | 1.18 | 0.69
Faust-5 | 1.0 | 0.64 | 0.6 | 0.63 | 1.07 | 0.59
Faust-7 | 1.0 | 0.66 | 0.66 | 0.64 | 0.94 | 0.63
Faust-10 | 1.0 | 0.65 | 0.6 | 0.62 | 0.93 | 0.6
Centaurs-3 | 1.0 | 1.13 | 1.03 | 1.11 | 1.21 | 0.79
Centaurs-4 | 1.0 | 0.88 | 0.88 | 0.9 | 1.09 | 0.76
Centaurs-6 | 1.0 | 0.7 | 0.69 | 0.79 | 1.07 | 0.69
Horses-3 | 1.0 | 1.1 | 1.06 | 1.1 | 1.22 | 0.75
Horses-4 | 1.0 | 0.87 | 0.82 | 0.8 | 1.09 | 0.71
Horses-8 | 1.0 | 0.62 | 0.62 | 0.61 | 1.0 | 0.59