# Animated Stickers: Bringing Stickers to Life with Video Diffusion

David Yan*, Winnie Zhang*, Luxin Zhang, Anmol Kalia, Dingkang Wang, Ankit
Ramchandani, Miao Liu, Albert Pumarola, Edgar Schönfeld, Elliot Blanchard,
Krishna Narni, Yaqiao Luo, Lawrence Chen, Guan Pang, Ali Thabet, Peter Vajda,
Amy Bearman†, Licheng Yu† GenAI, MetaMenlo Park, CaliforniaUSA

###### Abstract.

We introduce animated stickers, a video diffusion model which generates an
animation conditioned on a text prompt and static sticker image. Our model is
built on top of the state-of-the-art Emu text-to-image model, with the
addition of temporal layers to model motion. Due to the domain gap, i.e.
differences in visual and motion style, a model which performed well on
generating natural videos can no longer generate vivid videos when applied to
stickers. To bridge this gap, we employ a two-stage finetuning pipeline –
first with weakly in-domain data, followed by human-in-the-loop (HITL)
strategy which we term ensemble-of-teachers. It distills the best qualities of
multiple teachers into a smaller student model. We show that this strategy
allows us to specifically target improvements to motion quality while
maintaining the style from the static image. With inference optimizations, our
model is able to generate an eight-frame video with high-quality, interesting,
and relevant motion in under one second.

Figure 1. Examples of animated stickers generated by our model. Motions
display a high degree of quality, consistency, expressiveness and relevance to
the subject. Images are shown with transparent regions replaced by a white
background. **footnotetext: Indicates equal contribution††footnotetext:
Corresponding authors

##  1\. Introduction

There has recently been an surge of interest in generative text (and image) to
video (T2V) modeling (Blattmann et al., 2023b, a; Girdhar et al., 2023; Ho et
al., 2022; Rakhimov et al., 2020; Hong et al., 2022; Yu et al., 2023; Guo et
al., 2023). Generated videos from current state-of-the-art models are
typically short (under 3 seconds), and typically use text (text-to-video, or
T2V), an image (image-to-video, or I2V), or both. In this work, we use a text-
and-image-to-video architecture to target a natural application of short video
generation: animating stickers for social expression. Our model uses the same
kind of text-to-image-to-video generation pipeline as Emu video (Girdhar et
al., 2023), where the image used as conditioning is produced by a text-to-
image (T2I) sticker model (Sinha et al., 2023), which we refer to as the
static stickers model. Some examples of our model’s outputs can be seen in
Figure 1.

Leveraging the existing T2I stickers model achieves the desired style “for
free”. However, we find that using a general-purpose I2V model (i.e. one
trained only on a general video dataset) does not produce high-quality motion
when applied to stickers, and frequently generates videos with static or
trivial motion (e.g. only a “bobbing" effect) and/or introduces
inconsistencies and motion artifacts (e.g. warping). This is due to the visual
and motion differences between natural (photorealistic) videos and sticker-
style animations, i.e. a domain gap. One example of this gap is that our
stickers are entirely contained in the center of the canvas, with a solid
color background, which must then be masked to be transparent. Figure 2 shows
visual examples of our pretrain (natural videos), weakly in-domain data (short
animations) and in-domain (human-filtered videos) sets, highlighting the large
differences between the source (natural videos) and target (sticker
animations) domains.

In this work, we bridge the domain gap using an ensemble-of-teachers human-in-
the-loop (HITL) training strategy. First, a number of “teacher” models are
trained using different “recipes” of datasets and frame sampling rates, so
that collectively, the teacher models are capable of producing high quality
diverse motion, though only rarely. Next, an HITL dataset is constructed by
performing inference using teacher models on a large prompt set covering a
wide range of concepts and motions, and then filtered manually for videos with
the highest quality motion. “Student” models are then trained directly using
the HITL dataset. This two-stage approach produces better results than single-
stage finetuning on short animations, regardless of quality of the dataset
used for single-stage finetuning.

Our model is intended for use in production, and so needs to be fast at
inference-time, without sacrificing visual or motion quality. To speed up the
model, we utilize three approaches: first, we allow student models to be
architecturally smaller than teacher models, using fewer weights and/or fewer
text encoders. Second, we use a variety of optimizations which don’t require
training, i.e. lowering floating point precision, reducing the number of model
evaluations needed during sampling, and serializing the model with
Torchscript. Finally, we use model distillation to even further reduce the
number of sampling steps and model evaluations. The optimized model produces
eight frames of four-channel video (RGB plus transparency) in less than 1
second per batch on an H100 GPU, with minimal degradation in quality, and is
currently deployed for public testing.

Figure 2. An example of the types of training data used, showing the domain
gap between natural videos (a), short animations (b), and HITL-filtered in-
domain videos (c).

In summary, our contributions are as follows:

  1. (1)

We present our end-to-end process for creating, training, finetuning and
optimizing a domain-specific generative video model

  2. (2)

We describe our ensemble-of-teachers HITL finetuning strategy, and show that
it dramatically improves motion quality and relevance

  3. (3)

We describe two video-specific train-time improvements to the data and model –
middle frame conditioning and motion bucketing, and show that these
improvements further increase model quality

###  1.1. Related work

Video generation models. With the success of diffusion-based image generation
models, there has been a great deal of recent work in generating videos. Video
generative models take conditioning signals from text (Singer et al., 2022;
Girdhar et al., 2023; Blattmann et al., 2023b; Ho et al., 2022; Khachatryan et
al., 2023; Hong et al., 2022; Yu et al., 2023), images (Girdhar et al., 2023;
Guo et al., 2023; Yu et al., 2023), sketches (Dhesikan and Rajmohan, 2023),
pre-extracted depth maps and optical flows (Liang et al., 2023), and others,
and generate videos which align with their respective conditioning. In
general, video generation models can be classified into those based on
generative adversarial networks (GANs) (Clark et al., 2019; Luc et al., 2021;
Aldausari et al., 2020), auto-regressive models (Villegas et al., 2022),
transformer-based models (Yu et al., 2023; Rakhimov et al., 2020; Hong et al.,
2022) and diffusion-based models (Singer et al., 2022; Girdhar et al., 2023;
Blattmann et al., 2023b). In this work, we use a diffusion architecture due to
its wide applicability in text-to-image (T2I) (Rombach et al., 2022), text-to-
video (T2V) (Singer et al., 2022; Girdhar et al., 2023; Blattmann et al.,
2023b), video editing (Zhang et al., 2023; Wang et al., 2023), text-to-3D
(Poole et al., 2022) and text-to-4D (Singer et al., 2023; Ling et al., 2024),
as well as its capability of generating diverse outputs with high fidelity.

Diffusion models generate images and videos by adding noise to an input and
learn to iteratively denoise using neural network predictions (Dhariwal and
Nichol, 2021; Ho et al., 2020; Nichol and Dhariwal, 2021; Shaul et al., 2023).
Latent diffusion models (LDMs) operate in latent space, reducing the heavy
computational burden from training on a high-resolution pixel space. In this
work, we train a latent video diffusion model (Blattmann et al., 2023b) on our
video datasets. We follow the factorized design from Emu-video (Girdhar et
al., 2023), where video generation is decomposed into static image generation
given the text prompt followed by generating a video conditioned on the image
and prompt.

Finetuning to bridge domain gaps. The training dataset plays a key role in
determining the quality of generative AI models. Emu Video (Girdhar et al.,
2023) shows that the motion of the generated videos can be improved by
finetuning the model on a small subset of high motion and high quality videos.
However, as mentioned previously, the key challenge we are facing is the
domain gap between real videos and animated stickers. Furthermore, AnimateDiff
(Guo et al., 2023) points out that collecting sufficient personalized videos
on target domains is costly; they instead train a generalizable motion module
on a large video dataset and plug it into the personalized T2I to stay
faithful to a specific domain. While the paper aims to generate valid
animations in different personalized domains, they observed failure cases with
apparent artifacts and inability to produce proper motion when the domain of
the personalized T2I model is too far from that of the real video dataset.
Animated stickers is one such case, as the image domain we are trying to
animate is from a previous static stickers model (Sinha et al., 2023), which
are specifically personalized with dedicated style controlling.

Although training on general animations is a way to learn motion priors on
animations, it is still not sufficient to close the domain gap with our
desired sticker style. DreamBooth (Ruiz et al., 2022) attempts to close the
domain gap by using a rare string as the indicator to represent the target
domain and augments the dataset by adding images generated by the original T2I
model. Here we follow the same inspiration by taking advantage of high quality
generated videos in the target domain. We first finetune several teacher
models on animation videos to try to learn motion prior on animation style.
Then we apply those teacher models to generate videos conditioned on sticker
image. Instead of blending those generated videos with animation finetuning
videos, we only use generated video clips in sticker style to finetune a
student model. This one-hop domain transfer from pretrained model proves to
stabilize motion prior in sticker space, without catastrophic forgetting of
the motion prior learned from the original pretraining set.

##  2\. Model and Data

###  2.1. Model architecture

Figure 3 shows an overview of our model architecture. We employ a latent
diffusion model (LDM) with additional temporal layers, similar to the approach
taken by VideoLDM (Blattmann et al., 2023b). Practically, our model
architecture is the same as Emu-Video (Girdhar et al., 2023), which itself
extends Emu (Dai et al., 2023), with the only difference being that we use an
IP2P-style conditioning instead of masked conditioning. We briefly summarize
our model architecture here.

Figure 3. Overall architecture of our animated stickers model (left), and
addition of temporal layers to transformer and convolutional blocks (right).
We employ a spatiotemporal latent diffusion model (LDM), where The UNet
consists of convolutional stages and attention stages, where the attention
stages perform both self and cross-attention to text embeddings (CLIP is
always used, FLAN-T5XL is optional depending on the architecture). Temporal
layers are added after convolution and spatial transformers, with identity-
initialization so that a newly initialized model can load T2I weights and
reproduce the T2I model.

Our model consists of a variational autoencoder (VAE), UNet, and one or more
text encoders. The UNet uses the layers and weights from Emu (Dai et al.,
2023), with 1D convolutions across the time axis inserted after each 2D
convolutional layer in ResNet blocks, and temporal attention layers inserted
after each spatial attention block. Temporal layers are identity-initialized,
so that a newly initialized model with only T2I weights can exactly reproduce
text-to-image generation. We use the same VAE as the text-to-image model,
including the alpha-channel prediction in (Sinha et al., 2023), which allows
animations with transparent backgrounds. Our model uses two conditioning
signals. Image conditioning is applied by cloning the image latent across the
time dimension and appended along the channel axis to the noise, in a similar
way as InstructPix2Pix (IP2P) (Brooks et al., 2023). Text conditioning is
applied by encoding a prompt using CLIP (Radford et al., 2021) and Flan-T5-XL
(Chung et al., 2022) (where the latter may be omitted in more efficient
architectures), and fed into cross-attention layers. As we use two
conditioning signals, we enable classifier-free guidance (CFG) by dropping
text and image conditioning each separately between 5 and 10% of the time and
together between 5 and 10% of the time during training, and use the IP2P CFG
formulation at inference,

(1) $\displaystyle\tilde{\epsilon}_{\theta}(z_{t},c_{I},c_{T})=$
$\displaystyle\epsilon_{\theta}(z_{t},\varnothing,\varnothing)$ (2)
$\displaystyle+\sigma_{I}(\epsilon_{\theta}(z_{t},c_{I},\varnothing))-\epsilon_{\theta}(z_{t},\varnothing,\varnothing))$
(3)
$\displaystyle+\sigma_{T}(\epsilon_{\theta}(z_{t},c_{I},c_{T}))-\epsilon_{\theta}(z_{t},c_{I},\varnothing))$

where $z_{t}$ is the noisy latent, $c_{I}$ and $c_{T}$ are the image and text
conditionings, respectively, and $\sigma_{I}$ and $\sigma_{T}$ are the image
and text classifier-free guidance scales. In practice, we use $\sigma_{I}$ in
the range 7.5 to 9 and $\sigma_{T}$ in the range 1.5 to 3.

###  2.2. Pretraining data

35 million natural videos from Shutterstock were used for pretraining video
models. Data used for in-domain finetuning included two large (15-60k) short
animation datasets, as well as a high quality, professionally-animated sticker
set.

  * •

Keyword Based Sourcing + Manual Filtering. We initially used keyword matching
to source 15000 animated videos. These were then manually downselected for
samples which were on-style and had high motion quality, resulting in a
dataset of 4000 sticker-style short videos.

  * •

Artist set. We collected a set of artist-animated sticker packs, and manually
removed stickers with text overlaid. In total, 1829 animated stickers were
curated. Though the artist dataset has the highest quality in-domain videos
(created by professional creatives specifically to be used as social
stickers), even this set contains some examples with low quality motion, such
as very fast “jittering” or videos which alternate between only two frames.
These types of motion make sense in the context of their respective sticker
packs, but were undesirable for our model, so we further manually filtered out
20% of videos from this set.

  * •

Video KNN. To further expand pretraining data, we used video embeddings to
perform KNN searches of short videos, using the human-curated sticker-style
videos and artist set as seeds. This resulted in a further 62000 medium-to-
good quality animated sticker videos. We used an internal video understanding
model that was trained using temporal attention and considered different
modalities like visual frames, audio, OCR and other signals to produce a
multimodal video embedding. We experimentally validated that this model
significantly outperformed other simpler video embeddings extracted using only
the thumbnail of the video or just visual frames.

The artist set has human-written captions which provide detailed descriptions
of both motion and content, but the original captions for KNN and keyword-
sourced videos tend to be far noisier, and often do not describe the video. To
improve these captions, we utilize an internal video captioning model and an
entity extraction model. Specifically, we train a video captioning model
bootstrapped from the BLIP model (Li et al., 2022) (trained with the
Shutterstock image dataset and additional data) using the divided spatial-
temporal self-attention mechanism from (Bertasius et al., 2021). We adopt a
two-stage training strategy to train our video captioning model: a pre-
training stage using the Shutterstock dataset, and a finetune stage using the
animated videos from Shutterstock and the aforementioned artist dataset. To
extract the named entities from each video’s original caption, we leverage an
entity linking system built on the knowledge base of Wikipedia. By
concatenating the outputs from both models, we are able to generate richer
descriptions that capture both motion and visual objects in detail.

###  2.3. HITL data

Data for human-in-the-loop (HITL) was created by first curating a set of 15000
prompts, and then sending the prompts into the static stickers model to
generate two images per prompt. The prompts came from two main sources: a
previous static sticker HITL prompt set, and generations using LLAMA (Touvron
et al., 2023). The prompts generated from LLAMA were curated to describe
dynamic motions in order to optimize for large motion in the animated
stickers.

The prompt-image pairs were then used to generate videos using a number of
teacher models, the details of which will be discussed in Section 3. Generated
videos were sent to human annotators for filtering that fulfilled shareability
guidelines, which are defined by three main criteria:

  * •

Motion quality. Motion quality is defined as the amount of motion, smoothness
of the motion, and if the motion is natural and expressive. A shareable
animated sticker will have large motions that is smooth and natural.

  * •

Relevance. Relevance looks to see if the purpose of the animated sticker is
clear with no room for misinterpretation. The movement in the animated sticker
is expected to be related the subject and prompt.

  * •

Consistency. A shareable animated sticker should not distort or morph in any
way.

A first round of filtering was performed by trained third party vendors with
two-out-of-three annotator agreement, followed by a second round of filtering
done by internal experts, to ensure the highest quality data. Each round of
filtering included jobs that showed six different animated stickers. The
annotators were instructed to select all of the animated stickers that
fulfilled the shareability criteria. Figure 4 shows an example of the
interface that the annotators saw. The final selection of animated stickers
was used to train the student models.

Figure 4. A mock-up of the annotation interface. To the left, annotators
select any number out of the available videos, or select “I wouldn’t share any
of these image” if none of the videos are acceptable. To the right, annotators
can see the caption, and auto-looped animated sticker videos.

The prompts were split into three different buckets: action prompts, which
focused on various actions, ranging from simple actions, such as "a person
waving", to dynamic actions, such as "a speeding car navigating a winding
road", emotion prompts, which capture a wide variety of emotions, and ranged
from simple prompts, such as "a girl crying", to complex prompts, such as "a
dejected-looking puppy with its ears drooping and its tail between its legs",
and open-ended prompts, which describe any prompts that do not fall into the
emotion and action prompt buckets, such as those about scenery and single=word
prompts. In total, out of 15000 starting prompts, about 1500 remained in the
post-human-filtering HITL training set.

##  3\. Training details

###  3.1. Pretraining

After initializing with text-to-image weights, we pretrain our I2V
architecture using a 35M Shutterstock video dataset. We find that the highest
quality general-purpose I2V models are ones which are trained using a multi-
stage process, where at each stage, we change one or more of the following
hyperparameters, in addition to tuning normal training parameters such as
learning rate and number of training iterations: whether spatial weights are
frozen or unfrozen, the spatial resolution (256p or 512p), frame sample rate
(either 4 or 8 fps, or dynamic – see Section 3.1.1), and which quantity the
UNet predicts, either the noise $\epsilon$ or the phase velocity $v$ (Salimans
and Ho, 2022). Additionally, when using $v$ prediction, we always rescale the
noise schedule for zero terminal SNR. (Lin et al., 2024). An example training
recipe is [256p, freeze spatial, 4 fps, $\epsilon$-prediction] $\rightarrow$
[512p, freeze spatial, 4 fps, $\epsilon$-prediction] $\rightarrow$ [512p,
unfreeze spatial, 8 fps, $v$-prediction].

Using different training recipes such as this one allows us to trade off
between motion size and consistency. Empirically, we find that training with
$\epsilon$-prediction in early stages increases motion size, and starting from
a smaller spatial resolution increases motion quality of the final model. We
always train with $v$-prediction and zero terminal SNR in the final stage, as
videos generated with $v$-prediction have dramatically better color saturation
compared to $\epsilon$. We trained our models on A100 and H100 GPUs with batch
size between 128 and 512, learning rate between $2.5\text{e-}5$ and
$1\text{e-}4$, and number of iterations between a few thousand and a 150
thousand, depending on whether we were finetuning or pretraining. Videos were
resized and center-cropped during training, and we randomly selected 1-second
(sample rate of 8fps) or 2-second (sample rate of 4fps) clips and uniformly
sampled 8 frames from the clips as training examples.

####  3.1.1. Motion bucketing

When sampling training clips from videos, we usually sample all videos from a
dataset at the same framerate, with uniform spacing between frames in the
clip. For example, when sampling a 24 FPS video at 4 frames per second, we
sample every sixth frame, with the general spacing between frames given by
$\text{min}\left(\text{round}\left(\frac{\text{video fps}}{\text{desired
fps}}\right),\left\lfloor\frac{\text{video frames}}{\text{desired
frames}}\right\rfloor\right)$.

However, real-world video datasets will typically contain videos with
artificial speed-ups and slow-downs. Additionally, the true level of motion
varies widely between videos, and even between different parts of the same
video. For applications like sticker animation, a consistent level of motion
(neither too much or too little) is key, so we introduced a method to
normalize sampling frame rate against actual motion size.

To do this, we compute a motion score for a video, then put scores into FPS
“buckets” via manual inspection of videos within each bucket. For a first
version of this score, we used the vmafmotion (Li et al., 2018) score, which
is a measure of the temporal difference between adjacent frames; for an
updated version, we averaged the norm of the motion vectors from H.264/MPEG-4
AVC standard (kak Kwon et al., 2006), which are designed for inter-prediction
of macroblock offsets to reference frames, over all frames. FPS bucketing
results in a mapping between scores and sampling FPS, which we use to
dynamically sample videos at train-time. This method is only applicable to
longer videos, where it is possible to sample at different framerates – the
HITL data, for example, has only eight frames and does not permit motion
bucketing.

Practically, we find that in-domain fine-tuning with motion bucketing improves
motion consistency and reduces variance in motion size.

####  3.1.2. First vs. middle frame conditioning

When choosing which frame to use as conditioning during training, the most
obvious choice is the first frame. That is, when sampling clips from videos at
train-time, use the first frame of the sampled clip as image conditioning.
However, we must also consider that, first, at inference-time, the image
generated from a prompt with an action (e.g. two people high-fiving) will
typically render an image depicting the middle or end of the action. Second,
generated frames further in time from the conditioning frame have been
empirically found to be more likely to be inconsistent or introduce artifacts.
For these reasons, we experimented with using the middle frame (in practice,
the fourth frame out of eight) as image conditioning, and find that motion
consistency is improved.

Other possible choices for frame conditioning are last frame, and randomly
selecting a frame. When experimenting with these, we found that using the last
frame gave similar results as using the first frame, and using a random frame
gave noticeably worse results. A visual comparison between first-frame and
middle-frame model generations is shown in Section 4.2.2.

###  3.2. Ensemble-of-teachers HITL

Static stickers used a human-in-the-loop (HITL) finetuning strategy to improve
text faithfulness and style adherence. Since the style and text faithfulness
for the content of the video is overwhelmingly determined by the image used as
conditioning, we wanted to use an HITL approach tailored specifically to
improve motion quality and diversity. Our HITL finetuning strategy has three
objectives:

  1. (1)

Distill high quality motion from large models into smaller models, for
efficient inference

  2. (2)

Bridge the domain gap between the pretrained models, which were trained on
general videos, and static stickers

  3. (3)

Maximize the diversity, relevance, and interestingness of animated sticker
motion

We take an approach we call ensemble-of-teachers HITL finetuning, which we
outline in Figure 5. This approach differs from the HITL used for static
stickers in two ways:

  1. (1)

We use multiple expert-selected models to generate the HITL data for human
annotation

  2. (2)

The models which generate the HITL data have different architectures
(typically larger) than the models which train on it

Figure 5. Ensemble-of-teachers finetuning, where a number of pretrained, large
general-purpose video models are finetuned using finetuning data and different
recipes, which vary by data order and sampling framerate. This results in a
set of “teacher” models, which are used to generate videos with the HITL
prompt set. After human filtering, high-quality HITL data is used to finetune
a set of small, efficient pretrained models and downselected into student
model candidates.

We begin with several pretrained foundational models, selected for different
levels of motion size vs. consistency, and finetune them using finetuning
recipes on the datasets discussed in Section 2.2. This produces a number of
teacher models which are then downselected by human experts according to two
factors:

  * •

High peak quality. Since we were human-annotating a large number of
generations for each model, we judged models on their best generated videos,
rather than the average. This allows us to filter for very high quality,
interesting animations from each teacher model even if the majority of the
videos produced by that model are poor.

  * •

Diversity of motion. We want teacher models to have minimal overlap between
each other in terms of motion speed, novelty, and quality in different
actions. For example, one teacher model may be great at producing running and
walking motions but poor at others. Notably, we are able to train on the same
data while sampling at different framerates, so that models trained at
different framerates have different distributions of motion speed.

These teacher models are used to generate videos from an HITL prompt and image
set which is filtered by human annotators, engineers, and creatives (covered
in the previous section). The downselected high quality HITL set is then used
to train a number of pretrained student models, some of which may be
architecturally different than the teacher models – these architectures are
covered in detail in Section 3.3.2. Finetuning on data which is more aligned
with the output distribution makes the model generate more stable, consistent,
and higher quality motion. Also, teacher models trained on noiser data (e.g.
the KNN data) often produces large but low-quality and inconsistent motion.
However, this is tolerable for the purposes of HITL, since we can filter for
“lucky” instances where the motion is both large and consistent.

###  3.3. Model optimizations

Since the animated stickers model is intended to be used in production, it
needs to perform inference quickly in addition to producing high quality
motion. We applied three strategies to trade off between inference time and
quality: training-free optimizations, reducing the number of UNet weights, and
model distillation. These are detailed in the remainder of this section.

####  3.3.1. Training-free optimizations

We first employed some general optimizations which are applicable to any
latent diffusion model at inference, independent of architecture or
distillation. This included:

  * •

Halving the floating point precision. Converting the model from Float32 to
Float16 speeds up the inference time for two reasons. First, the memory
footprint of the model is halved. Second, 16 floating point operations can be
executed faster. For all models, we use BFloat16 (a float16 variant with a
smaller mantissa) (Kalamkar et al., 2019) for training and inference.

  * •

Torchscripting and freezing. Torchscript (DeVito, 2022) is a serialized format
for easy deployment of PyTorch models. Converting a model from pure PyTorch to
TorchScript involves automatic optimizations that can increase inference
speed, such as fusing multiple operations, constant folding, and techniques to
reduce the complexity of the computational graph. Additionally, freezing
(referring to jit.freeze, not weight freezing) allows further automatic speed
optimizations in Torchscript, by converting dynamic parts of the graph into
constants to remove unnecessary operations. Importantly, we freeze with the
flag to preserve the numerics and prevent quality degradation.

  * •

Optimized temporal attention expansion. Temporal attention layers (attending
between the time axis and text conditioning) require the context tensors to be
replicated to match the number of frames (the time dimension). In a naive
implementation, this would be done before passing to cross-attention layers.
The optimized version takes advantage of the fact that the repeated tensors
are identical, and expands after passing through the cross-attention’s linear
projection layers, reducing compute and memory.

  * •

DPM-solver. Rather than use the DDPM (Ho et al., 2020) or DDIM (Song et al.,
2020) solvers for inference, which typically require more sampling steps for
good quality generation, we use DPM-solver (Lu et al., 2022) and a linear-in-
logSNR time schedule at inference to reduce the number of sampling steps to
15.

  * •

Adaptive guidance. A novel technique that reduces the number of network
evaluations from three to two one for a subset of the sampling steps (Castillo
et al., 2023). In effect, less forward passes through the network are executed
and memory usage is reduced. These two effects result in faster inference
speed without any quality degradation. In practice, we perform full guidance
for the first eight (out of 15) sampling steps, and no guidance for the
remaining seven.

With these optimizations, we are able to reduce inference time by an order of
magnitude when compared to a fully unoptimized model (e.g. DDIM 50 steps, full
precision, full guidance) with minimal change in quality.

####  3.3.2. Efficient model architectures

Our pretrained video model, with CLIP and T5 text encoders, which, as
mentioned previously, is roughly the same architecture as Emu Video. To reduce
the number of weights, we targeted the following areas:

  1. (1)

Number of UNet channels

  2. (2)

UNet spatial and temporal transformer depth

  3. (3)

Number of UNet resnet blocks per UNet block

  4. (4)

Whether to include the T5 text encoder or only use CLIP

Notably, we do not reduce the number of latent channels (which stays at 8 for
all experiments), as we empirically found that having at least 8 channels is
important to reducing visual artifacts and morphing. As an example, at 512p,
the foundational UNet has 4.3 B weights and uses 23.5 teraFLOPs, whereas a
more efficient UNet ("sm") has 1.2 B weights and uses 5.6 teraFLOPs.

For our students, we narrowed potential models to four UNet architectures:
“lg” (4.3 B UNet weights), “lg-e” a.k.a. lg-efficient (fewer res blocks, and
no T5 encoder, 3.5 B UNet weights), “med” (2.4 B UNet weights) and “sm” (1.2 B
UNet weights). These models were pretrained using similar recipes as used for
the teacher models in Section 3.1, with the notable difference being student
models are all trained up to a maximum of 256p, since that is the required
output size.

####  3.3.3. Distillation

To speed inference up further, we use two distillation techniques that reduce
the number of forward passes through the UNet without affecting the parameter
count:

  * •

Guidance distillation. Diffusion models use classifier-free guidance for
conditional image generation, which requires a conditional and unconditional
forward pass per solver step. Guidance distillation reduces these two forward
passes into one. However, in the case of the animated stickers model,
classifier-free guidance requires three forward passes per step: a full
conditional (text and image), unconditional, and an image-conditional step.
Applying guidance distillation to reduce three forward passes into one has not
yet been described in the literature, but we found that it works well in
practice, reducing inference time threefold.

  * •

Step-distillation. In step distillation, a teacher and student are initialized
with the same weights, and the student is trained to match multiple teacher
steps in a single step.

  * •

Guidance+step distillation. We combine guidance and step distillation can by
training a student to imitate classifier-free-guidance and multiple steps at
the same time with just one forward pass through the UNet. We find that a
four-to-one ratio of teacher to student steps works best. Distilling 32
teacher steps into 8 student steps during training. Our final model only
requires eight solver steps, with one forward pass through the UNet per step.

For each of the four efficient UNet architectures (sm, med, lg, lg-e), we
evaluated training-free optimization, guidance distillation, and guidance+step
distillation. Benchmark times for the models ranged from 300 ms on an H100 for
the smallest model with both guidance and step distillation, to 2000 ms for
the largest model with only training-free optimizations.

Ultimately, we select the lg model with guidance+step distillation, which has
an H100 inference time of 726 ms, for evaluation and public testing as the
best compromise between inference time and quality. Heavily-distilled smaller
models were found to have more frequent artifacts and worse motion, and more
expensive models had slightly better motion, but at a too-heavy computational
cost. The models with only training-free optimizations were most faithful to
the original model, but still significantly slower than the distilled models.

##  4\. Evaluation and results

###  4.1. Evaluation

In order to evaluate the quality of the model, we created an annotation
guideline to preform standalone evaluations for the different versions of the
animated sticker model. Standalone evaluations mean that we show the
annotators one animated sticker. The annotation guideline provides questions
to evaluate animated stickers based on motion quality and prompt similarity.
Annotators were instructed to place a stronger focus on motion quality due to
the fact that prompt similarity strongly correlates with the content of the
sticker image used for conditioning.

The motion quality task has ten questions pertaining to the motion in the
animated sticker:

  1. (1)

Existence: Is there any motion in the animated sticker?

  2. (2)

Relevance: Is the motion in the animated sticker expected and relevant to the
prompt?

  3. (3)

Consistency: Do the characters and objects remain in proportion and consistent
throughout the animation?

  4. (4)

Distortions: Is the animated sticker free of any flicker or distortion?

  5. (5)

Motion curves: Does the animation utilize smooth motion curves that reference
real physics/ gravity?

  6. (6)

Outline: Does the linework/ white outline of the sticker move with the motion
of the sticker?

  7. (7)

Looping: Does the animation play in a continuous loop?

  8. (8)

Clipping: Are there no integral elements of the animated sticker clipped?

  9. (9)

Expression: Are the expressions in the animated sticker clear?

  10. (10)

Background: Does background motion complement and not distract from the
primary animation?

For each question, the annotators were instructed to either select "yes" or a
reason why the animated sticker failed; they were able to select multiple
failure reasons.

The prompt similarity task has four questions pertaining to how well the
animated sticker matches the prompt.

  1. (1)

Subjects: Does the animated sticker clearly represent all subjects or objects
intended in the prompt?

  2. (2)

Actions: Does the animated sticker effectively depict all specific actions or
movements, as stated in the prompt?

  3. (3)

Composition: Does the animated sticker depict the desired arrangement and
quantity of the subjects or objects specified in the prompt?

  4. (4)

Attributes: Does the animated sticker include all attributes (e.g., colors,
shapes, sizes) and emotions of subjects or objects specified in the prompt?

Similar to the motion quality task, the annotators were instructed to either
select "yes" or a reason why the animated sticker failed. The annotators were
also instructed to fail the animated sticker if one of the frames failed for
the question.

Table 1 show standalone evaluation results with three annotator multi-review
for the optimized student model. Some of the criteria, such as existence and
relevance, have high pass rates because the guidelines for these criteria were
not strict. For example, the raters were asked to choose "yes" for the
existence of motion as long as there was primary motion in the sticker, which
is defined as the main movement in the animation. The animated sticker did not
need to have secondary motion, which are animation that amplifies the action
by supporting the primary motion with secondary characteristic detail, in
order to pass. However, if we tighten the guidelines and require both primary
and secondary motion, the pass rate will fall to 0.857. There are plans to
tighten the guidelines for future iterations.

We also observe that distortions and consistency have the lowest pass rate out
of all of the criteria. We hypothesize that more motion have a larger chance
of having distortions and less consistency, which lead to a lower pass rate.

Category | Consensus count | Pass rate
---|---|---
Existence | 1890 | 0.969
Relevance | 1928 | 0.992
Consistency | 1772 | 0.786
Distortions | 1800 | 0.673
Motion curves | 1888 | 0.934
Outline | 1894 | 0.920
Looping | 1894 | 0.999
Clipping | 1894 | 0.994
Expression | 1894 | 0.954
Background | 1928 | 0.999
Table 1. Standalone evaluation results on the optimized student model for all
evaluation categories. Consensus count refers to samples where all three
annotators agreed on the label, and pass rate is the percentage of samples
with consensus where the animated stickers passed the criteria.

####  4.1.1. Effect of Distillation on Model Quality

In order to demonstrate the effects of distillation on the model quality, we
ran evaluation on the distilled and non-distilled (normal Pytorch model with
DDIM 50 steps sampling) versions of the final student model. Table 2 show
evaluation results for both versions. While existence, motion curves, and
clipping has a slightly higher pass rater for the non-distilled student model,
the distilled student model has a higher pass rate for all the other
categories. Furthermore, it has much higher pass rate for consistency,
distortions, outline, and expression.

Category | Non-distilled Pass Rate | Distilled Pass Rate
---|---|---
Existence | 0.978 | 0.969
Relevance | 0.978 | 0.992
Consistency | 0.572 | 0.786
Distortions | 0.488 | 0.673
Motion curves | 0.977 | 0.934
Outline | 0.791 | 0.920
Looping | 0.993 | 0.999
Clipping | 0.998 | 0.994
Expression | 0.707 | 0.954
Background | 0.995 | 0.999
Table 2. Standalone evaluation results for the non-distilled student model and
distilled student model

###  4.2. Results and Visualizations

More general examples can also be found in Appendix A.

####  4.2.1. Pretrained vs. finetuned model

In order to demonstrate the significant improvement in motion from in-domain
and HITL finetuning, we show some examples in Figure 6 of the same image and
prompt conditioning, animated with a 256p-trained general-purpose video model
(trained on Shutterstock only) versus the student animated stickers model.
Note that the general-purpose model is capable of correctly animating natural
images.

Figure 6. Examples showing the effect of finetuning versus a general-purpose
(out-of-domain) video model trained on natural videos. In-domain and HITL
finetuning has the effect of a) increasing secondary motion (e.g. in faces,
background objects, etc.), b) giving the subject a relevant animation rather
than adding a bulk motion, and c) reducing motion artifacts and morphing. Top:
the general-purpose model gives the cat an up-and-down bobbing motion, whereas
the finetuned model animates a correct running movement. Bottom: the general-
purpose model adds morphing to the video, whereas the finetuned model
correctly animates dancing.

In addition to adding motion to many stickers where the general-purpose model
generates a static video, the full finetuning pipeline makes large
improvements in three areas: (1) increasing secondary motion, for example in
faces, background objects, and body parts (the general-purpose video model
very rarely generates secondary motion in sticker-style videos, which reduces
expressiveness), (2) giving the subject a relevant animation rather than bulk
motion (e.g. only rotation or translation of the sticker) – the HITL-finetuned
model is able to correctly animate a number of actions, such as running,
jumping, laughing, hugging, etc, but even when there isn’t full prompt-action
alignment, the HITL-finetuned model tends to give the subject correct and
relevant motion relative to itself, e.g. limbs and faces moving correctly
relative to the body, and (3) reducing motion artifacts, such as morphing and
blurring.

####  4.2.2. First vs. middle-frame conditioning

To highlight the effects of middle-frame conditioning, we trained two models:
one which was pretrained on Shutterstock using and then finetuned on the
artist sticker set using first frame conditioning for both pretraining and
finetuning, and another with the same pipeline but using middle-frame
conditioning for both. Figure 7 shows some comparisons between the two models.

Figure 7. Examples showing the difference between training using the first
frame as conditioning and using the middle (fourth) frame as conditioning. In
general, as shown in the examples, middle-frame conditioning tends to produce
larger motion, since the action in the prompt generally produces a static
sticker which depicts the middle of the action rather than the beginning.

In general, we find that both motion consistency and size are improved with
middle-frame conditioning. In the examples in Figure 7, both cases show larger
and more natural motion for middle-frame conditioning, where first-frame
conditioning only shows some “bobbing" motion.

##  5\. Conclusion

We presented our animated stickers model, which uses a spatiotemporal latent
diffusion model conditioned on a text-image pair to animate sticker images.
Our pretrain-to-production pipeline started with the Emu model, which was
fine-tuned on a large set of natural videos, followed by in-domain datasets.
We then use an ensemble-of-teachers HITL fine-tuning strategy to further
improve the motion quality, consistency, and relevance. We use a number of
architectural, distillation-based optimizations, and post-training
optimizations to speed up the inference to one second per batch. We show that
our fine-tuning strategy improves motion size and quality significantly over a
model trained on natural videos only, demonstrating the effectiveness of the
ensemble-of-teachers approach, and our other train-time improvements, such as
middle-frame conditioning and motion bucketing. Our model is currently in
public testing.

There are several areas for future work. First, our current model only outputs
8 frames, which limits the potential range of motion; increasing the number of
frames while maintaining inference time would be an important improvement.
Second, modifying model outputs to ensure smooth looping would improve user
experience, since stickers are automatically looped for users, and large jumps
between the first and last frame cause an unpleasant effect. Finally, there is
still room to improve the overall quality of primary and secondary motion by
expanding and further filtering datasets, tweaking model architecture, and
further reducing quality loss in inference optimizations.

###### Acknowledgements.

We would like to thank Anthony Chen, Ishan Misra, Mannat Singh, Rohit Girdhar,
Andrew Brown, Saketh Rambhatla, Quentin Duval, Samaneh Azadi, Samyak Datta,
Kapil Krishnakumar, Tsahi Glik, Jeremy Teboul, Shenghao Lin, Milan Zhou,
Karthik Sivakumar, Ashley Ngo, Thai Quach, Jiabo Hu, Yinan Zhao, Bichen Wu,
Ching-Yao Chuang, Arantxa Casanova Paga, Roshan Sumbaly, and Tali Zvi for
their helpful discussions, guidance, and support which made this work
possible.

## References

  * (1)
  * Aldausari et al. (2020) Nuha Aldausari, Arcot Sowmya, Nadine Marcus, and Gelareh Mohammadi. 2020.  Video Generative Adversarial Networks: A Review.  arXiv:2011.02250 [cs.CV]
  * Bertasius et al. (2021) Gedas Bertasius, Heng Wang, and Lorenzo Torresani. 2021.  Is space-time attention all you need for video understanding?. In _ICML_ , Vol. 2. 4.
  * Blattmann et al. (2023a) Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, et al. 2023a.  Stable video diffusion: Scaling latent video diffusion models to large datasets.  _arXiv preprint arXiv:2311.15127_ (2023).
  * Blattmann et al. (2023b) Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, and Sanja Fidler. 2023b.  Align Your Latents: High-Resolution Video Synthesis With Latent Diffusion Models. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_. 22563–22575.
  * Brooks et al. (2023) Tim Brooks, Aleksander Holynski, and Alexei A. Efros. 2023.  InstructPix2Pix: Learning to Follow Image Editing Instructions. In _CVPR_.
  * Castillo et al. (2023) Angela Castillo, Jonas Kohler, Juan C Pérez, Juan Pablo Pérez, Albert Pumarola, Bernard Ghanem, Pablo Arbeláez, and Ali Thabet. 2023.  Adaptive Guidance: Training-free Acceleration of Conditional Diffusion Models.  _arXiv preprint arXiv:2312.12487_ (2023).
  * Chung et al. (2022) Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. 2022\.  Scaling instruction-finetuned language models.  _arXiv preprint arXiv:2210.11416_ (2022).
  * Clark et al. (2019) Aidan Clark, Jeff Donahue, and Karen Simonyan. 2019.  Adversarial Video Generation on Complex Datasets.  arXiv:1907.06571 [cs.CV]
  * Dai et al. (2023) Xiaoliang Dai, Ji Hou, Chih-Yao Ma, Sam Tsai, Jialiang Wang, Rui Wang, Peizhao Zhang, Simon Vandenhende, Xiaofang Wang, Abhimanyu Dubey, Matthew Yu, Abhishek Kadian, Filip Radenovic, Dhruv Mahajan, Kunpeng Li, Yue Zhao, Vladan Petrovic, Mitesh Kumar Singh, Simran Motwani, Yi Wen, Yiwen Song, Roshan Sumbaly, Vignesh Ramanathan, Zijian He, Peter Vajda, and Devi Parikh. 2023.  Emu: Enhancing Image Generation Models Using Photogenic Needles in a Haystack.  arXiv:2309.15807 [cs.CV]
  * DeVito (2022) Zachary DeVito. 2022.  Torchscript: Optimized execution of pytorch programs.  _Retrieved January_ (2022).
  * Dhariwal and Nichol (2021) Prafulla Dhariwal and Alex Nichol. 2021.  Diffusion Models Beat GANs on Image Synthesis.  arXiv:2105.05233 [cs.LG]
  * Dhesikan and Rajmohan (2023) Rohan Dhesikan and Vignesh Rajmohan. 2023.  Sketching the Future (STF): Applying Conditional Control Techniques to Text-to-Video Models.  _arXiv preprint arXiv:2305.05845_ (2023).
  * Girdhar et al. (2023) Rohit Girdhar, Mannat Singh, Andrew Brown, Quentin Duval, Samaneh Azadi, Sai Saketh Rambhatla, Akbar Shah, Xi Yin, Devi Parikh, and Ishan Misra. 2023.  Emu Video: Factorizing Text-to-Video Generation by Explicit Image Conditioning.  arXiv:2311.10709 [cs.CV]
  * Guo et al. (2023) Yuwei Guo, Ceyuan Yang, Anyi Rao, Yaohui Wang, Yu Qiao, Dahua Lin, and Bo Dai. 2023.  Animatediff: Animate your personalized text-to-image diffusion models without specific tuning.  _arXiv preprint arXiv:2307.04725_ (2023).
  * Ho et al. (2022) Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik P Kingma, Ben Poole, Mohammad Norouzi, David J Fleet, et al. 2022\.  Imagen video: High definition video generation with diffusion models.  _arXiv preprint arXiv:2210.02303_ (2022).
  * Ho et al. (2020) Jonathan Ho, Ajay Jain, and Pieter Abbeel. 2020.  Denoising Diffusion Probabilistic Models.  arXiv:2006.11239 [cs.LG]
  * Hong et al. (2022) Wenyi Hong, Ming Ding, Wendi Zheng, Xinghan Liu, and Jie Tang. 2022.  Cogvideo: Large-scale pretraining for text-to-video generation via transformers.  _arXiv preprint arXiv:2205.15868_ (2022).
  * kak Kwon et al. (2006) Soon kak Kwon, A. Tamhankar, and K.R. Rao. 2006.  Overview of H.264/MPEG-4 part 10.  _Journal of Visual Communication and Image Representation_ 17, 2 (2006), 186–216.  https://doi.org/10.1016/j.jvcir.2005.05.010 Introduction: Special Issue on emerging H.264/AVC video coding standard.
  * Kalamkar et al. (2019) Dhiraj Kalamkar, Dheevatsa Mudigere, Naveen Mellempudi, Dipankar Das, Kunal Banerjee, Sasikanth Avancha, Dharma Teja Vooturi, Nataraj Jammalamadaka, Jianyu Huang, Hector Yuen, et al. 2019\.  A study of BFLOAT16 for deep learning training.  _arXiv preprint arXiv:1905.12322_ (2019).
  * Khachatryan et al. (2023) Levon Khachatryan, Andranik Movsisyan, Vahram Tadevosyan, Roberto Henschel, Zhangyang Wang, Shant Navasardyan, and Humphrey Shi. 2023.  Text2Video-Zero: Text-to-Image Diffusion Models are Zero-Shot Video Generators.  arXiv:2303.13439 [cs.CV]
  * Li et al. (2022) Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. 2022.  Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In _International Conference on Machine Learning_. PMLR, 12888–12900.
  * Li et al. (2018) Zhi Li, Christos Bampis, Julie Novak, Anne Aaron, Kyle Swanson, Anush Moorthy, and JD Cock. 2018.  VMAF: The journey continues.  _Netflix Technology Blog_ 25, 1 (2018).
  * Liang et al. (2023) Jingyun Liang, Yuchen Fan, Kai Zhang, Radu Timofte, Luc Van Gool, and Rakesh Ranjan. 2023.  MoVideo: Motion-Aware Video Generation with Diffusion Models.  arXiv:2311.11325 [cs.CV]
  * Lin et al. (2024) Shanchuan Lin, Bingchen Liu, Jiashi Li, and Xiao Yang. 2024.  Common diffusion noise schedules and sample steps are flawed. In _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_. 5404–5411.
  * Ling et al. (2024) Huan Ling, Seung Wook Kim, Antonio Torralba, Sanja Fidler, and Karsten Kreis. 2024.  Align Your Gaussians: Text-to-4D with Dynamic 3D Gaussians and Composed Diffusion Models.  arXiv:2312.13763 [cs.CV]
  * Lu et al. (2022) Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. 2022.  Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps.  _Advances in Neural Information Processing Systems_ 35 (2022), 5775–5787.
  * Luc et al. (2021) Pauline Luc, Aidan Clark, Sander Dieleman, Diego de Las Casas, Yotam Doron, Albin Cassirer, and Karen Simonyan. 2021.  Transformation-based Adversarial Video Prediction on Large-Scale Data.  arXiv:2003.04035 [cs.CV]
  * Nichol and Dhariwal (2021) Alex Nichol and Prafulla Dhariwal. 2021.  Improved Denoising Diffusion Probabilistic Models.  arXiv:2102.09672 [cs.LG]
  * Poole et al. (2022) Ben Poole, Ajay Jain, Jonathan T. Barron, and Ben Mildenhall. 2022.  DreamFusion: Text-to-3D using 2D Diffusion.  _arXiv_ (2022).
  * Radford et al. (2021) Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021\.  Learning transferable visual models from natural language supervision. In _International conference on machine learning_. PMLR, 8748–8763.
  * Rakhimov et al. (2020) Ruslan Rakhimov, Denis Volkhonskiy, Alexey Artemov, Denis Zorin, and Evgeny Burnaev. 2020.  Latent video transformer.  _arXiv preprint arXiv:2006.10704_ (2020).
  * Rombach et al. (2022) Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. 2022.  High-Resolution Image Synthesis with Latent Diffusion Models.  arXiv:2112.10752 [cs.CV]
  * Ruiz et al. (2022) Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. 2022.  DreamBooth: Fine Tuning Text-to-image Diffusion Models for Subject-Driven Generation.  (2022).
  * Salimans and Ho (2022) Tim Salimans and Jonathan Ho. 2022.  Progressive Distillation for Fast Sampling of Diffusion Models. In _International Conference on Learning Representations_.  https://openreview.net/forum?id=TIdIXIpzhoI
  * Shaul et al. (2023) Neta Shaul, Juan Perez, Ricky TQ Chen, Ali Thabet, Albert Pumarola, and Yaron Lipman. 2023.  Bespoke solvers for generative flow models.  _arXiv preprint arXiv:2310.19075_ (2023).
  * Singer et al. (2022) Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, et al. 2022\.  Make-A-Video: Text-to-Video Generation without Text-Video Data. In _The Eleventh International Conference on Learning Representations_.
  * Singer et al. (2023) Uriel Singer, Shelly Sheynin, Adam Polyak, Oron Ashual, Iurii Makarov, Filippos Kokkinos, Naman Goyal, Andrea Vedaldi, Devi Parikh, Justin Johnson, and Yaniv Taigman. 2023.  Text-To-4D Dynamic Scene Generation.  arXiv:2301.11280 [cs.CV]
  * Sinha et al. (2023) Animesh Sinha, Bo Sun, Anmol Kalia, Arantxa Casanova, Elliot Blanchard, David Yan, Winnie Zhang, Tony Nelli, Jiahui Chen, Hardik Shah, Licheng Yu, Mitesh Kumar Singh, Ankit Ramchandani, Maziar Sanjabi, Sonal Gupta, Amy Bearman, and Dhruv Mahajan. 2023.  Text-to-Sticker: Style Tailoring Latent Diffusion Models for Human Expression.  arXiv:2311.10794 [cs.CV]
  * Song et al. (2020) Jiaming Song, Chenlin Meng, and Stefano Ermon. 2020.  Denoising diffusion implicit models.  _arXiv preprint arXiv:2010.02502_ (2020).
  * Touvron et al. (2023) Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023\.  Llama: Open and efficient foundation language models.  _arXiv preprint arXiv:2302.13971_ (2023).
  * Villegas et al. (2022) Ruben Villegas, Mohammad Babaeizadeh, Pieter-Jan Kindermans, Hernan Moraldo, Han Zhang, Mohammad Taghi Saffar, Santiago Castro, Julius Kunze, and Dumitru Erhan. 2022.  Phenaki: Variable Length Video Generation From Open Domain Textual Description.  arXiv:2210.02399 [cs.CV]
  * Wang et al. (2023) Xiang Wang, Hangjie Yuan, Shiwei Zhang, Dayou Chen, Jiuniu Wang, Yingya Zhang, Yujun Shen, Deli Zhao, and Jingren Zhou. 2023.  VideoComposer: Compositional Video Synthesis with Motion Controllability.  arXiv:2306.02018 [cs.CV]
  * Yu et al. (2023) Lijun Yu, Yong Cheng, Kihyuk Sohn, José Lezama, Han Zhang, Huiwen Chang, Alexander G Hauptmann, Ming-Hsuan Yang, Yuan Hao, Irfan Essa, et al. 2023\.  Magvit: Masked generative video transformer. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 10459–10469.
  * Zhang et al. (2023) Zhixing Zhang, Bichen Wu, Xiaoyan Wang, Yaqiao Luo, Luxin Zhang, Yinan Zhao, Peter Vajda, Dimitris Metaxas, and Licheng Yu. 2023.  AVID: Any-Length Video Inpainting with Diffusion Model.  arXiv:2312.03816 [cs.CV]

##  Appendix A More animation examples

Figure 8 shows some additional examples of animated stickers generated by the
optimized student model. Since motion is more apparent when viewed as a looped
video, examples were selected which had motion that was more visible when
viewed as a sequence of images.

Figure 8. More examples of animated stickers generated by the optimized, HITL-
finetuned model. Examples were selected which had motion which was more
obvious when viewed as a series of images. Images are shown with transparent
regions replaced by a white background.