# Convolutional Cobweb: A Model of Incremental
Learning from 2D Images

Christopher J. MacLellan  Harshil Thakur

###### Abstract

This paper presents a new concept formation approach that supports the ability
to incrementally learn and predict labels for visual images. This work
integrates the idea of convolutional image processing, from computer vision
research, with a concept formation approach that is based on psychological
studies of how humans incrementally form and use concepts. We experimentally
evaluate this new approach by applying it to an incremental variation of the
MNIST digit recognition task. We compare its performance to Cobweb, a concept
formation approach that does not support convolutional processing, as well as
two convolutional neural networks that vary in the complexity of their
convolutional processing. This work represents a first step towards unifying
modern computer vision ideas with classical concept formation research.

<EMAIL_ADDRESS><EMAIL_ADDRESS>
College of Computing and Informatics, Drexel University, Philadelphia, PA
19104 USA

##  1 Introduction

Recent years have seen an explosion in the availability of image data and, as
a result, there has been rapid advances in the development of computer vision
approaches. One kind of approach that has achieved state-of-the-art
performance on a large number of computer vision tasks are Convolutional
Neural Networks, or CNNs (LeCun et al., 2015). These models operate by
convolving relatively small (e.g., 3x3) filters over an image to generate
transformed feature maps. Since each filter is applied at every location in
the image, the model is able to recognize and respond to patterns that occur
both within and across input images. Convolutional filters can represent
useful transformations, such as edge detection, sharpening, embossing, and
blurring.

Although CNNs tend to achieve high performance, they have multiple limitations
that are the focus of the current work. First, although it is possible to
train them incrementally, most research efforts exclusively utilize batch
training paradigms. Second, they require a large amount of training data to
achieve high performance (e.g., tens to hundreds of thousands of examples). If
sufficient labeled data is not available, then their performance suffers.
Finally, CNNs have a static structure that is fixed at creation time, only the
network weights are updated during training.

To overcome these limitations, we draw on ideas from the concept formation
literature (Fisher et al., 2014; Gennari et al., 1989). Concept formation
approaches, such as Cobweb (Fisher, 1987), were developed based on
psychological studies of how humans incrementally learn concepts. As a result,
they make incremental processing a necessary characteristic. Further, unlike
CNN’s which are tuned using stochastic gradient descent—a process that
requires many iterative updates—most concept formation approaches use closed-
form optimization (e.g., closed-form maximum likelihood estimation), which
does not require iterative training to converge. Lastly, concept formation
approaches support the ability to update both their structure and their
parameters during online training, whereas CNNs only update their parameters.

Despite these benefits, there do not yet exist many examples of research that
apply concept formation approaches to image data, such as the kind of data
typically employed by CNNs. For example, researchers have not typically
explored the use of approaches like Cobweb (Fisher, 1987; McKusick & Thompson,
1990; MacLellan et al., 2016) or SAGE (McLure et al., 2015) for learning from
pixel-level feature representations. Additionally, unlike CNNs, concept
formation approaches do not make use of convolutional filters to process their
inputs.

This work aims to bring together ideas from both of these lines of research.
In particular, we explore the development of a new concept formation approach
that we call Convolutional Cobweb. This approach utilizes convolutional
processing to support incremental concept formation over 2D images. We
evaluate our approach on an incremental variant of the MNIST digit recognition
task and compare it to three other approaches: Cobweb/3 (a variation of Cobweb
that supports continuous features; McKusick & Thompson, 1990) as well as two
CNNs that vary in terms of the complexity of their convolutional processing.
Based on our empirical study, we make three explicit claims: (1) our new
approach’s ability to engage in convolutional processing lets it outperform a
comparable Cobweb approach that does not support convolutional processing; (2)
by utilizing the Cobweb algorithm for categorization and learning, our new
approach is able to outperform a simple CNN that utilizes a similar
convolutional structure; and (3) our approach outperforms the CNNs when
learning incrementally from just a few examples (e.g., less than 100
examples).

##  2 The Cobweb Algorithm

Cobweb (Fisher, 1987) is an incremental concept formation approach that was
developed based on how people incrementally learn concepts. This approach
accepts a series of instances described in an attribute-value representation
(e.g., {color: red, shape: triangle, …}). Given these examples, Cobweb
produces a conceptual hierarchy like the one shown in Figure 1. Each concept
in this hierarchy contains a table of attribute-value counts that describe how
frequently each attribute-value pair occurs in the instances labeled with the
respective concept.

Given a concept hierarchy, Cobweb generates predictions about a new instance
by first sorting it down the concept tree. At each point in the sorting
process, Cobweb uses a measure called category utility to determine whether it
is better to terminate at the current node or recursively sort the instance
into a child concept. The category utility measure is similar to the
information gain measure used in decision tree learning (Quinlan, 1986), but
optimizes for the ability to predict the values of all attributes rather than
a single attribute. This measure was developed based on psychological studies
of human concept learning (Corter & Gluck, 1992). Cobweb uses the attribute-
value count table from the final concept the instance was sorted into to make
predictions about the instance (e.g., to predict the shape of an instance
given only its color).

Figure 1: An example of a conceptual hierarchy that Cobweb might learn. Each
concept stores a table of attribute-value counts (shown as probabilities) that
describe all of the instances stored under the concept.

To update its conceptual hierarchy in light of a new instance, Cobweb uses a
procedure similar to the one used at performance time. Specifically, it sorts
the through the instance into its categorization tree using an expanded set of
four operations: add, which adds the instances to the best child concept
(updating the attribute value counts); merge, which merges the two child
concepts that most closely match the instance; split, which splits the concept
that most closely matches the instance; and create, which creates a new child
concept to match the instance. At each point Cobweb chooses the operation that
maximizes the category utility measure.

While Cobweb only supports the ability to process instances that contain
nominal attribute values, there have been many extensions that offer support
of other kinds of attributes. For example, Cobweb/3 supports the ability to
handle continuous attribute values (McKusick & Thompson, 1990) by
approximating them with Gaussian distributions. This approach stores the mean
and standard deviation of each continuous attribute in each concept
description (rather than the attribute-value counts) and uses a modified
version of category utility that supports continuous attributes. Other
variations, such as Labyrinth (Thompson & Langley, 1991) and Trestle
(MacLellan et al., 2016), support the ability to handle component and
relational attributes.

##  3 Convolutional Cobweb

At its core, our new approach is a variation of Cobweb/3 that supports the
ability to engage in convolutional processing over 2D image data. However, we
do not simply make use of the convolutional processing approach used by neural
networks. Instead, we have developed a more unified way to integrate
convolutional processing in a fashion that is compatible with the spirit of
Cobweb. Prior Cobweb research generally assumes higher-level input
representation (e.g., features refer to shapes and colors rather than pixels)
and centers the attention on learning concept hierarchies over these higher-
level representations. In contrast, Convolutional Cobweb accepts lower-level
pixel features and learns a hierarchy of convolutional filters to represent
these inputs as well as a classification hierarchy for making predictions
about instances. In this section, we describe Convolutional Cobweb’s
representations for example instances as well as learned knowledge. We also
describe its performance and learning mechanisms.

###  3.1 Instance Representation

Single channel (black and white) image data is generally represented as a 2D
array of pixel intensity values. Convolutional Cobweb translates data input in
this format through multiple representations during processing, see Figure 2.
First, it translates the 2D array of pixels into a dictionary of attribute-
values, where the attribute names represent the indices of the pixels in the
original 2D array. This representation is also the one that we use later in
our baseline Cobweb/3 approach. Next, Convolutional Cobweb converts this
representation into a dictionary of all possible patches from the input image
that have a user-specified convolutional filter size. In the current work, we
use a filter size of 3x3, so this structure holds all 3x3 patches from the
input image. Each patch is indexed by the location of the upper left pixel in
the patch. Thus, a 28x28 image will generate 26x26 entries, each containing a
3x3 patches. Each 3x3 patch contains the intensity values for the pixels it
covers, indexed by where the pixel occurs in the patch. Each of these patches
is then categorized using a variant of Cobweb that learns a hierarchy of
convolutional filters. This categorization process (described in the following
performance mechanism subsection) returns a filter label for each patch. The
final representation, which replaces each patch with its respective filter
label, is then used for categorization and for making inferences about the
image itself (e.g., for predicting the digit label).

Figure 2: Representations used by Convolutional Cobweb to encode and
categorize images.

###  3.2 Knowledge Representation

Figure 3: Examples of the two internal hierarchies learned by Convolutional
Cobweb. Shown on the top is a hierarchy of 3x3 convolutional filters learned
directly from the images. Shown on the bottom is a hierarchy of concepts
learned over the convolutional filter representations.

Unlike Cobweb which consists of a single concept hierarchy, Convolutional
Cobweb consists of two hierarchies: a convolutional filter hierarchy and a
classification hierarchy (see Figure 3). The first hierarchy stores a
collection of 3x3 convolutional filters learned from the input images. Similar
to Cobweb/3, each node stores the means and standard deviations of the pixel
features for all the patches that have been categorized under it. The second
hierarchy stores each image (represented in terms of filters from the
convolutional filter hierarchy) and any other features of the image, such as
its digit label. As shown in Figure 3, each node maintains an table of
attribute-value counts111Although we only show nominal attributes in this
example, our implementation can also support continuous attributes using the
same approach adopted by Cobweb/3..

###  3.3 Performance Mechanism

Leveraging these representations, Convolutional Cobweb uses a variant of
Cobweb’s performance mechanism to make predictions about each input image.
When it is presented with an input image, such as the image of a 7 shown in
Figure 2, it first converts this image into a dictionary of 3x3 patch
dictionaries (second to last representation from Figure 2). Next, each patch
dictionary is sorted through the convolutional filter hierarchy. At each step
in the sorting process, Convolutional Cobweb chooses between stopping at the
current filter or sorting recursively into one of the children nodes using the
category utility measure. The instance description is then updated by
replacing each 3x3 patch dictionary with the label of the filter it was
categorized as (producing the final instance representation from Figure 2).

The resulting description is sorted through the classification hierarchy to
determine the final concept for the instance. Convolutional Cobweb sorts the
instance through the classification hierarchy using category utility to
determine when to stop and when to recursively categorize the instance into a
child node. For each category utility calculation, Convolutional Cobweb
dynamically replaces all filter references in the attribute-value count tables
with references to filters that are one level below the root of the
convolutional filter hierarchy. For example, references to Filters 9 or 75
from the convolutional filter hierarchy in Figure 3 would be replaced with
references to Filter 7 and any associated attribute-value counts would be
merged when computing category utility. This remapping procedure ensures that
all concepts use the same level of representation for filters. Additionally,
by using a dynamic remapping approach, Convolutional Cobweb ensures that all
filter references are valid, even in situations where the convolutional filter
hierarchy is modified (intermediate filters might be created or destroyed)
during incremental learning . Once a final concept has been identified in the
classification hierarchy, Convolutional Cobweb uses its attribute-value table
to make predictions about the image (e.g., to predict its digit label).

###  3.4 Learning Mechanism

During learning, Convolutional Cobweb goes through a similar process to
performance. It first categorizes each image patch into its convolutional
filter hierarchy using the four standard Cobweb operations (add, merge, split,
create). As it incorporates each patch into the hierarchy, it updates the
parameters (mean and std of pixels) of relevant filters to reflect the new
patch. After categorizing each patches, it replaces the patch dictionaries
with their respective filter labels and adds any additional attribute values
of the image (e.g., its digit label) to the instance description.
Convolutional Cobweb then uses the add, merge, split, and create operations to
incorporate this updated instance into its classification hierarchy. To decide
which operation is best it uses the modified category utility calculation that
dynamically remaps all filter references in the attribute-value count tables
to point to the filters that are one level below the root of the convolutional
filter hierarchy.

During learning, every unique patch from each image will be represented by a
leaf in the convolutional filter hierarchy and every unique image will be
represented as a leaf in the classification hierarchy (categorization always
ends in the creation of a new leaf or incorporation of the instance into an
existing leaf that is identical). As a result, all references to filters in
the classification hierarchy always point to leaves of the convolutional
filter hierarchy. Thus, information about filters is never lost, even when
intermediate (non-leaf) filters in the convolutional hierarchy are deleted.
Convolutional Cobweb dynamically remaps all filter references to point to
valid intermediate representations. Figure 3 shows examples of a convolutional
filter hierarchy and classification hierarchy that results from applying this
approach to a sequence of MNIST images that represent the digits 0-9. We can
see that the convolutional filter hierarchy represents lower-level pixel
patterns, whereas the categorization hierarchy uses the filter representation
to learn categories of images.

##  4 Experimental Evaluation

Our approach combines ideas from CNNs and Cobweb, so we were interested in how
it compares to these baselines. We conducted experiments to explore whether
our approach could outperform Cobweb/3, which uses a similar concept formation
approach, but does not support convolutional processing. We also conducted a
set of experiments to evaluate how our approach compares to CNNs, which
support convolutional processing, but use gradient descent to update their
parameters and do not support the ability to modify their internal structure.

###  4.1 Incremental MNIST Digit Recognition Task

To evaluate our approach, we chose to test it using images from the Modified
National Institute of Standards and Technology (MNIST) data set, which is
widely used to test image-based classification approaches (Deng, 2012). This
data set contains images of hand written digits that are labeled with values
ranging from 0 through 9. Each image contains 28x28 black and white pixel
values (784 continuous feature values), which we normalized to have a mean of
0 and a standard deviation of 1. See Figure 4 for a set of example images used
in our analysis.

Figure 4: Examples of MNIST images with their labels

The conventional approach to evaluating classification approaches on MNIST
images is to train on a batch of images and then to test a learned model on
another batch of held-out cases (e.g., training on 80% of the images and
testing on 20%). However, a goal of the current work is to evaluate how well
our proposed approach supports incremental prediction and learning. Thus, we
used an incremental prediction paradigm to evaluate our approach and to
compare it to baselines. In this framework, we present each approach with a
sequence of images. The first time we present an image from the sequence, we
withhold the image label and predict the missing label. Once a prediction has
been generated, we then provide each system with both the image and its label
and we incrementally update the system’s knowledge to account for the new
training example. This process is repeated for each image in the sequence
until all images have been processed. To estimate the average performance of
each approach, we applied this evaluation approach 50 times. For each
iteration, we used a sequence of 300 images (30 images of each digit) that was
randomly sampled from the MNIST data set and randomly shuffled. To reduce
performance differences due to differences in training sequences, we trained
each system on the same set of 50 sequences.

###  4.2 Comparison Approaches

To evaluate Convolutional Cobweb, we applied it to the incremental MNIST digit
recognition task and compared it to three baseline approaches: (1) Cobweb/3,
(2) a simple CNN that has a convolutional structure similar to Convolutional
Cobweb, and (3) a more complex CNN with multiple convolutional layers and
filters.

####  4.2.1 Cobweb/3

Our first baseline approach is Cobweb/3, which accepts a sequence of images
that are represented as a dictionaries of attribute-values (see second
representation in Figure 2). Cobweb/3 does not use any specialized approaches
for image processing, it simply treats the 784 pixels from each image as
continuous features. For each input image, it attempts to predict the label
for the image using the standard Cobweb prediction algorithm. Specifically,
the instance is sorted into the classification tree and the probability table
from the final concept the instance is sorted into is used to generate a
prediction of the image label. While generating predictions, the
classification tree is not modified. However, once a prediction has been
generated, the image label is added as an attribute value of the instance and
inserted into the tree using the standard Cobweb learning algorithm. This
incrementally updates the tree structure and attribute-value counts by adding,
merging, splitting, and creating concepts during instance categorization. Once
the example has been incorporated into the tree, the model is presented with
the next image in the sequence.

####  4.2.2 CNN-Simple

Our second approach is CNN-Simple, which uses a neural network structure
consisting of 1 convolutional layers and two fully connected layers. The
convolutional layer applies a single 3x3 convolutional filter to each single
channel (black and white) image using a padding of 0 and a stride of 1. The
output from the convolutional layer is passed through a non-linear ReLU
activation function before being flattened into a vector of features. These
features are then passed through two fully connected layers. The first layer
contains 128 hidden nodes and utilizes an ReLU activation function. The second
layer contains 10 output nodes and utilizes a Softmax activation function,
which translate each output into a probability value. These 10 output
probabilities correspond to the probability that each input image represents
one of the ten digits (0-9).

As a preliminary test of this approach, we trained it using a standard CNN
training paradigm. We divided the 70,000 MNIST training images into into
56,000 training images and 14,000 validation images (an 80-20 split). We used
stratification while creating this split to ensure the distribution of each
digit was similar in both the training and validation sets. We then trained
the model using stochastic gradient decent with the Adam optimizer (Kingma &
Ba, 2014). We trained the model on the entire training set 150 times (i.e.,
150 epochs) using a batch size of 1024 images and a cross-entropy loss
function. Once training completed, we then applied the trained model to
generate predictions for each of the 14,000 validation images. We compared
these prediction to the ground truth labels to compute the error rate of the
model. This batch training approach shows that CNN-Simple achieves an an error
rate of only 2.48% (97.52% accurate) on the validation set.

To support the incremental MNIST digit recognition task, we utilized a non-
standard approach to train the neural network. For each image in the training
sequence, we applied the neural network to predict the label of the image.
Once a prediction had been made, we added the image and its associated label
to a buffer. This buffer holds 16 images and their associated labels using a
first-in-first-out queue. After inserting the image into the buffer, we apply
the neural network to predict the labels for all 16 images, compute the cross-
entropy loss over these predictions, and then apply a single stochastic
gradient decent update (using the Adam optimizer) to adjust the network
weights. Thus, when processing an incremental training sequence of 300 images,
we apply a total of 300 weight updates to the network.

We designed CNN-Simple to be similar to our Convolutional Cobweb approach,
which uses a single convolutional processing step followed by a classification
step. Similarly, CNN-Simple uses a single 3x3 convolutional layer followed by
two fully connected layers for classification. While it is standard practice
within the computer vision community to use use multiple convolutional layers
that each have multiple convolutional filters (LeCun et al., 2015), we believe
that this provides a reasonable baseline for comparison to our new Cobweb
variant.

####  4.2.3 CNN

Our last approach uses a CNN structure that is more complex than CNN-Simple
and is more representative of the kinds of CNNs that typically appear in
practice. This network utilizes 2 convolutional layers, a max pool layer, and
two fully connected layers. The first convolutional layer maps a single (black
and white) input channel to 32 output channels. Each output channel is
constructed by passing 1 of 32 3x3 convolutional filters over the input image
with a stride of 1 and padding of 0 (each filter generates its own output
channel). Each output channel is then passed through an ReLU activation
function. The second convolutional layer maps from 32 channels (output from
the first convolutional layer) to 64 output channels. To handle the multi-
channel input, this approach has 32 3x3 convolutional filters for each of the
64 output channels—for a total of 2,048 3x3 convolutional filters. Each output
is generated by applying the output channel’s 32 3x3 convolutional filters to
their respective 32 input channels and summing the results. These 64 output
channels are then passed through an ReLU activation function and a 2x2 max
pool layer, which reduces the height and width of the 64 output channels by
half. The final 64 channels are flattened into a single vector of features and
then passed through two fully connected layers that have the same structure as
those used by CNN-Simple (128 hidden nodes with an ReLU activation function
and 10 output nodes with a Softmax activation function).

Similar to CNN-Simple, we tested this approach using conventional CNN batch
training. Specifically, we divided the MNIST data into training and validation
sets using a stratified 80-20 split. We trained the CNN on the entire training
set 150 times (150 epochs) using stochastic gradient decent with the Adam
optimizer and then tested it on the validation sets to determine the error
rate. This analysis shows that the more complex CNN is able to achieve an
error rate of only 1.26% (98.74% accurate) on the validation set. Thus, when
using batch training, this approach achieves better performance on the MNIST
data set than the Simple-CNN approach.

To apply this approach to incrementally predict the MNIST image labels, we use
the same approach we employed for CNN-Simple. Specifically, we applied the CNN
to predict each image in the sequence, after generating a prediction we added
the image and its label to a buffer containing the past 16 images and their
associated labels, and then we do a stochastic gradient decent update using
the 16 examples in the buffer. This approach uses a much more complex
structure than Convolutional Cobweb or CNN-Simple. Whereas these other
approaches use a single convolutional layer with a single output channel (a
single 3x3 filter in the case of CNN-Simple), this approach uses two
convolutional layers containing 2,080 3x3 convolutional filters (32 in first
layer and 2,048 in the second). We chose to include this approach in our
evaluation because it is representative of typical CNNs.

Figure 5: Overall error of the each approach on the incremental MNIST task,
averaging over 50 runs. For each run, each system was presented with 300
randomly ordered images (30 images from each digit 0-9). The whiskers denote
bootstrapped 95% confidence intervals over runs. Figure 6: Learning curve
showing the error of each approach at each point over the course of the
incremental MNIST task. The lines were generated by applying Lowess smoothing
to the 0-1 error data and the shaded regions denote 95% confidence intervals.

###  4.3 Results

We applied each of our four approaches to the incremental MNIST digit
recognition task. As mentioned previously, each approach was trained 50 times
using a randomly sampled and shuffled sequence of 300 images (30 images from
each digit). After training, we computed the average error rate over all of
the predictions. These error rates are shown in Figure 5 along with 95%
confidence intervals estimated using bootstrapping (Wood,
2005).222Bootstrapped confidence intervals provide an alternative to
statistical significance tests that are generally viewed as more transparent.
In addition to computing the overall error rates for each approach, we also
wanted to understand how the performance changed over the course of
incremental training. Thus, we computed the learning curves for each approach,
as presented in Figure 6. These curves show the average error rates at each
point over the course of the training (e.g., the average error on the 100th
training opportunity).

###  4.4 Discussion

Our overall results show that our new Convolutional Cobweb approach
outperforms two of our baselines (see Figure 5). Specifically, we see that our
approach outperforms Cobweb/3, which uses a similar incremental learning
paradigm, but lacks a convolutional layer. We also see that it outperforms
CNN-Simple, which has a similar convolutional structure, but differs in terms
of its structure and learning paradigm. When we compare Convolutional Cobweb
to the more complex CNN, we see that there is not a meaningful difference in
the overall error.

When we review the learning curve results (see Figure 6), we find that, with
the exception of CNN, these results largely hold over the entire duration of
training. We find that Convolutional Cobweb slightly outperforms Cobweb/3 over
the entire course of training. While it might not seem that the performance of
Convolutional Cobweb is substantially better than Cobweb/3, models on the
MNIST dataset generally tend to differ by fractions of a percent (Kowsari et
al., 2018). Thus, this difference is a substantial improvement. This claim is
supported by the overall error results, where we can see that Convolutional
Cobweb makes fewer errors than Cobweb/3.

Next, we look at the how our approach compares to the neural networks. First,
we find that our approach outperforms CNN-Simple over the entire range,
suggesting it achieves better performance when compared to a CNN with a
similar convolutional structure (one layer with one 3x3 convolutional filter).
However, we find that while Convolutional Cobweb outperforms the more complex
CNN approach over the first 100 training cases, the CNN outperforms
Convolutional Cobweb over the latter two thirds of training. This suggests
that the more complex CNN will likely outperform Convolutional Cobweb if we
were to train it on a larger number of training cases. This CNN may achieve
such good performance because it has a substantially more complex
convolutional structure (two layers with 2,080 3x3 convolutional filters).
This suggests that future work should explore variants of Convolutional Cobweb
that utilize more complex convolutional structures.

We know from our batch training evaluations of CNN-Simple and CNN that they
can both achieve substantially better performance on the non-incremental MNIST
digit recognition task (CNN-Simple has an error of 2.48% and CNN has an error
of 1.26%) than we observe on the incremental MNIST task. This suggests that
there is room for substantial improvement on the incremental MNIST digit
recognition task. Based on our analysis, we believe the CNN approaches
generally need more data and more stochastic gradient decent updates (i.e.,
more passes over the data) to achieve better performance. In contrast, the
Cobweb approaches tend to have faster learning rates and better performance on
the earlier training opportunities, suggesting they do better when learning
incrementally from fewer examples.

##  5 Conclusion and Future Work

Our findings show that Convolutional Cobweb demonstrates improved performance
on the incremental MNIST digit recognition task over comparable baselines. It
outperforms one baseline model that uses a similar modeling paradigm, but that
lacks convolutional capabilities (Cobweb/3). It also outperforms a CNN model
that has a similar convolutional structure (CNN-Simple). Lastly, we find that
while our approach outperforms a more complex CNN when training on fewer than
100 examples, the more complex CNN has better performance given a larger
number of training cases. These findings suggest that integrating
convolutional processing with incremental concept learning is a promising
research direction; however, more work is needed to develop approaches at the
intersection of these two lines of research. In particular, future work should
explore variants of Convolutional Cobweb that utilize multiple convolutional
layers (similar to our more complex CNN model). Our hope is that by utilizing
more complex convolutional structures we can exceed the performance of the
more complex CNN.

One major barrier to increasing the complexity of Convolutional Cobweb is its
runtime performance. Our CNNs were implemented using the PyTorch library,
which lets them take advantage of matrix libraries and GPUs to greatly
decrease the time it takes to process images and generate predictions. For
this work, we used one NVIDIA A40 GPUs to accelerate the CNN models. These
GPUs made it possible to do 150 epochs over 56,000 MNIST images in less than 7
minutes. In contrast, Convolutional Cobweb does not utilize matrix libraries
and does not support GPU acceleration. As a result, it took approximately 5
hours to incrementally process 300 MNIST images on a 2GHz AMD EPYC 7662 CPU.
Increasing the convolutional complexity was computationally prohibitive.
Future work should explore how to implement the Cobweb algorithm in a library
such as PyTorch, which would enable the use of both matrix libraries and GPU
acceleration.

Additionally, the current work explores the application of Convolutional
Cobweb to MNIST images, which only have a single input channel (images are
black and white). Future works should explore how to extend the approach to
support multi-channel inputs, such as three channel RGB input. There are
multiple possible options for implementing multi-channel support that should
be explored. For example, one extension might apply separate Convolutional
Cobweb layers to each input channel, generating three output channels. An
alternative extension might apply a single convolutional layer to all three
inputs and generate a single output channel. Future work should explore these
variants and evaluate them on multi-channel images, such as those found in the
CIFAR-10 data set (Krizhevsky et al., 2009).

In conclusion, our work explores the development of incremental approaches for
learning from 2D visual images. We show that CNN models do not perform as well
on incremental prediction tasks as they do on non-incremental prediction
tasks. Additionally, we present Convolutional Cobweb, which merges the CNN
concept of convolutions with the incremental learning approach used by Cobweb.
This approach performs well on an incremental MNIST digit recognition task,
outperforming a non-convolutional Cobweb approach and a CNN approach that uses
a similar convolutional structure. This work represents a first step towards
the unification of CNN advances with classic concept formation models.

Acknowledgements We thank Pat Langley for his guidance, suggestions, and
feedback while developing the Convolutional Cobweb model; this work would not
have been possible without his input. We also thank the Drexel University STAR
program for providing support for Harshil Thakur to work on this project over
the summer term. Lastly, we thank the other members of the Teachable AI Lab
(Qiao Zhang, Adit Gupta, and Glen Smith) for their suggestions and feedback on
this work.

## References

  * Corter & Gluck (1992) Corter, J. E., & Gluck, M. A. (1992).  Explaining basic categories: Feature predictability and information.  Psychological bulletin, 111, 291.
  * Deng (2012) Deng, L. (2012).  The mnist database of handwritten digit images for machine learning research.  IEEE Signal Processing Magazine, 29, 141–142.
  * Fisher (1987) Fisher, D. H. (1987).  Knowledge acquisition via incremental conceptual clustering.  Machine learning, 2, 139–172.
  * Fisher et al. (2014) Fisher, D. H., Pazzani, M. J., & Langley, P. (2014).  Concept formation: Knowledge and experience in unsupervised learning.  Morgan Kaufmann.
  * Gennari et al. (1989) Gennari, J. H., Langley, P., & Fisher, D. (1989).  Models of incremental concept formation.  Artificial intelligence, 40, 11–61.
  * Kingma & Ba (2014) Kingma, D. P., & Ba, J. (2014).  Adam: A method for stochastic optimization.  arXiv preprint arXiv:1412.6980.
  * Kowsari et al. (2018) Kowsari, K., Heidarysafa, M., Brown, D. E., Meimandi, K. J., & Barnes, L. E. (2018).  Rmdl: Random multimodel deep learning for classification.  Proceedings of the 2nd International Conference on Information System and Data Mining (pp. 19–28).
  * Krizhevsky et al. (2009) Krizhevsky, A., Hinton, G., et al. (2009).  Learning multiple layers of features from tiny images.  Technical report, University of Toronto.
  * LeCun et al. (2015) LeCun, Y., Bengio, Y., & Hinton, G. (2015).  Deep learning.  nature, 521, 436–444.
  * MacLellan et al. (2016) MacLellan, C. J., Harpstead, E., Aleven, V., Koedinger, K. R., et al. (2016).  Trestle: a model of concept formation in structured domains.  Advances in Cognitive Systems, 4, 131–150.
  * McKusick & Thompson (1990) McKusick, K., & Thompson, K. (1990).  Cobweb/3: A portable implementation.  Technical report, NASA Ames Research Center.
  * McLure et al. (2015) McLure, M., Friedman, S., & Forbus, K. (2015).  Extending analogical generalization with near-misses.  Proceedings of the AAAI Conference on Artificial Intelligence.
  * Quinlan (1986) Quinlan, J. R. (1986).  Induction of decision trees.  Machine learning, 1, 81–106.
  * Thompson & Langley (1991) Thompson, K., & Langley, P. (1991).  Concept formation in structured domains.  In Concept formation, 127–161. Elsevier.
  * Wood (2005) Wood, M. (2005).  Bootstrapped confidence intervals as an approach to statistical inference.  Organizational Research Methods, 8, 454–470.


# Speech Driven Video Editing via an Audio-Conditioned Diffusion Model

Dan Bigioi
University of Galway
<EMAIL_ADDRESS>Shubhajit Basak
University of Galway
<EMAIL_ADDRESS>Both Authors Contributed Equally to the Paper Michał
Stypułkowski11footnotemark: 1
University of Wrocław
<EMAIL_ADDRESS>Maciej Ziȩba
Wrocław University of Science and Technology
Tooploox
<EMAIL_ADDRESS>Hugh Jordan
Trinity College Dublin
<EMAIL_ADDRESS>Rachel McDonnell
Trinity College Dublin
<EMAIL_ADDRESS>Peter Corcoran
University of Galway
<EMAIL_ADDRESS>
###### Abstract

Taking inspiration from recent developments in visual generative tasks using
diffusion models, we propose a method for end-to-end speech-driven video
editing using a denoising diffusion model. Given a video of a talking person,
and a separate auditory speech recording, the lip and jaw motions are re-
synchronized without relying on intermediate structural representations such
as facial landmarks or a 3D face model. We show this is possible by
conditioning a denoising diffusion model on audio mel spectral features to
generate synchronised facial motion. Proof of concept results are demonstrated
on both single-speaker and multi-speaker video editing, providing a baseline
model on the CREMA-D audiovisual data set. To the best of our knowledge, this
is the first work to demonstrate and validate the feasibility of applying end-
to-end denoising diffusion models to the task of audio-driven video editing.
111All code, datasets, and models used as part of this work are made publicly
available here: https://danbigioi.github.io/DiffusionVideoEditing/

##  1 Introduction

The idea behind audio-driven video editing is to provide a means to re-
synchronise the lip and jaw movements of an actor in a video, in response to a
new speech input signal. This new speech signal may come from the original
speaker, or a voice actor. Regardless of the source of the input speech, a key
objective is that the performance of the actor is never diminished. No matter
how the lip and jaw movements change in response to the new audio, the facial
expressions, and emotions portrayed by the actor should remain consistent with
the original performance.

Achieving such seamless audio-driven video editing is an exciting prospect for
the entertainment industry, one with the potential of being applied to movies,
TV shows, live streaming, and even homemade content uploaded to platforms such
as YouTube, TikTok, and others. Giving video content creators the ability and
option to edit their work without having to go through time-consuming, and
expensive re-shoots, allows them to work with a greater tolerance for error
during filming.

Furthermore, the realisation of true audio-driven video editing would bring
about a significant transformation in the world of cinema and television,
allowing for more accessible and cost-effective dubbing of English-language
movies/TV shows/videos into other languages and vice versa, allowing for the
further democratisation of video content by making it more engaging and
personalized for audiences worldwide. Recent advancements in deep learning and
talking head generation techniques are bringing us closer to this exciting
possibility, where audio and video will be seamlessly synchronized in real-
time.

Current approaches for speech driven video editing, and the related task of
talking head generation can be grouped into two distinct types: structured,
and unstructured. Structured generation refers to techniques that use the
speech signal to first extract an intermediate structural representation of
the face (facial landmarks, 3D model expression parameters), before utilizing
it to render the photo-realistic frame [70, 88, 31, 7, 85]. On the other hand,
unstructured generation techniques [72, 18, 29, 87], utilise image
reconstruction techniques to generate the photo-realistic frame directly in an
end-to-end manner.

Diffusion models [61, 16, 24, 48] are a relatively new class of generative
model that have recently been gaining traction due to their strong performance
on image synthesis tasks, often outperforming state-of-the-art GAN (Generative
Adversarial Network) [20]-based methods [16]. Utilising conditioning signals
such as text and even images, diffusion models have shown that they can be
trained and conditioned towards generating a specific desired output at
inference time with relative ease [55]. They achieve high mode coverage unlike
GANs, maintain high sample quality, and are stable during training. These
properties make them an ideal candidate for application towards the task of
unstructured audio-driven video editing, a task that has thus far been
dominated by GAN-based approaches [72, 11, 8].

We present an approach for automatic speech driven video editing using a
denoising diffusion model. We utilise a U-Net backbone modifying it for the
task of video editing, and introduce a feature concatenation mechanism for
conditioning the network with information related to the previously generated
frame in the sequence so that the network can generate temporally coherent
frames. We further condition the network on speech by feeding spectrogram
feature embeddings combined with the noise signal throughout the residual
layers of the U-Net as demonstrated by Diffused Heads [66], though unlike
their approach, we use spectral features rather than features extracted from a
pretrained speech encoder in order to capture as much information about the
signal as possible. To the best of our knowledge, this is the first work that
applies denoising diffusion models to the task of audio-driven video editing.
As part of this work, we state the following contributions to the field:

  * •

A novel unstructured end-to-end approach for audio-driven video editing using
a denoising diffusion model. We condition the network on speech and train it
to modify the face such that the lip and jaw movements are synchronised to the
conditioning audio signal on a frame-by-frame basis. We train both single, and
multi-speaker proof-of-concept models using the GRID [14], and CREMA-D [6]
datasets respectively, achieving strong proof-of-concept results when tested
on unseen speakers. The project code, datasets, and trained models will be
made freely available to the public.

  * •

We demonstrate the applicability of our approach on the video editing task,
achieving competitive results thanks to our conditional inpainting strategy
which gathers information from previous frames and audio spectral embeddings,
to generate the current frame. Our method achieves near state-of-the-art
results when measured on traditional image quality metrics such as SSIM, PSNR,
FID, and competitive SyncNet [13] lip synchronisation scores compared to other
relevant methods from the field.

##  2 Related Works

###  2.1 Audio Driven Video Generation

Audio-driven video generation methods can generally be categorised by whether
they are generated by leveraging an audio-driven structural representation of
the face, or without.

There have been numerous approaches over the years relating to the former.
Taylor et al. [69] and Karras et al. [32] among the first to apply machine
learning techniques to the facial animation task, the former learning facial
expression parameters of a 3D face model from phoneme labels, and the latter
predicting 3D vertex positions of a face mesh from a speech audio window.
Suwajanakorn et al. [67] trained a speaker specific network to output sparse
mouth key-points, using them to modify videos of President Obama. Eskimez et
al [17] presented a recurrent architecture capable of taking in speech as
input and outputting 2D landmark face co-ordinates, with Chen et al. [9]
utilising cascaded GANs to translate those landmark features into
photorealistic frames. Cudeiro et al. [15] introduced a 4D facial dataset, and
trained a network to generate animations from speech with it. [88, 73, 40, 5]
generated intermediate landmark features from audio, also exploring the
related task of extracting realistic headpose. Thies et al. [70] generated 3D
facial expression parameters using features from a pretrained audio encoder,
using these parameters to generate a photorealistic video via a neural
renderer, with [75] and [63] following a similar approach but operating on
videos instead. [7] and [81] presented methods to generate 3D face animation
parameters, in addition to realistic head pose from speech, using these
features to generate photorealistic frames. Ji et al. [31] approached the
problem of video editing, generating emotion-controllable talking head
portraits using both intermediate landmark structures, and 3D model
parameters. [82, 84, 76, 37, 54, 62] are other approaches from the literature
which predict expression parameters from audio to drive a 3D face model.

What these approaches all have in common is that they use these intermediate
structural representations as input to a separate neural rendering model which
is typically trained as an image-to-image translation task to generate the
final photo-realistic image frame. As of the date of this submission, GAN-
based [20] approaches such as Pix2Pix [28], CycleGAN [90], and other
variations have proved immensely popular for this task. However, diffusion-
based techniques show big promise for the future, especially given recent
developments in various image-to-image translation tasks [58].

Nonstructural/end-to-end methods on the other hand utilise latent feature
learning and image reconstruction techniques to generate a photo-realistic
video sequence from an input speech signal and reference image/video in an
end-to-end manner. Approaches such as [50, 44, 89, 18, 8, 87, 72, 36, 86, 65,
29] have seen much success in recent times. Each of these approaches differs
from the one used in this paper as they are all GAN/VAE (variational
autoencoder)[34] based probabilistic methods while ours leverages a denoising
diffusion model. While current end-to-end approaches suffer from low output
resolution quality compared to structural methods, there is a lot of potential
for improvement, especially by exploiting diffusion models’ ability to
synthesise high-quality samples while maintaining good mode
coverage/diversity.

###  2.2 Diffusion Models

Denoising diffusion models [61, 64] have seen great success on a wide variety
of different challenges, ranging from image-to-image translation tasks like
inpainting, colorisation, image upscaling, uncropping [58, 25, 4, 51, 55, 60,
42, 43], audio generation [10, 35, 78, 68, 49, 33, 27, 38], text-based image
generation [2, 57, 19, 47, 53, 59, 21], video generation [26, 83, 22, 80], and
many others. Recently, diffusion models have also been applied to the related
task of talking head generation, with the work of [66] a concurrent approach
to our own. For a thorough review of diffusion models and all of their recent
applications, we recommend [79].

Diffusion models are a class of generative probabilistic models that consist
of two steps: 1) the forward diffusion process that destroys data by steadily
adding small amounts of random Gaussian noise over a series of time steps
until the data becomes a sample from a standard Gaussian distribution. 2) The
reverse diffusion process where a denoising model is trained to restore
structure in the data by steadily removing noise over a series of time steps.
The trained model can then sample information from random Gaussian noise and
steadily denoise it over a series of time steps to attain the desired output.

Sohl-Dickstein et al. [61] developed the first diffusion model and coined the
term, followed by Ho et al. [24] combining denoising score matching with
Langevin dynamics [64] and diffusion models to synthesise images. This ignited
a steady interest in diffusion models, with Nichol et al. [48] showing that by
making small adjustments to the diffusion process, they could sample data
faster and achieve better log-likelihoods to models trained explicitly to
minimise it with minimal impact to sample quality. They also found that
training diffusion models with more computational power typically lead to
better sample quality. Chen et al. [10] and Kong et al. [35] applied diffusion
models to the task of audio synthesis, succeeding in generating high-quality
samples. Dhariwal and Nichol [16] demonstrated that diffusion models beat GANs
on image synthesis, also introducing the concept of "classifier guidance" for
a conditional generation.

As diffusion models are trained under a single loss and do not rely on a
discriminator, they are more stable during training and do not suffer from
typical issues associated with training GANs such as mode collapse, and
vanishing gradients. They produce high-quality output samples and display high
mode coverage unlike GANs [77]. Despite these advantages, their sampling speed
is slow due to the need to run the inverse diffusion process thousands of
times on the same sample to denoise it completely. Xiao et al. [77] and
Rombach et al [55] attempted at speeding up the sampling and training times
associated with diffusion models with the former proposing a method to model
the denoising distribution using a complex multi-modal distribution in order
to facilitate larger diffusion steps, and the latter applying diffusion models
in the latent space of a pre-trained autoencoder to reduce the complexity.
This is an ongoing focus of research in the field, and it is a certainty that
more works tackling the inference/training speed problem will emerge.

##  3 Methodology

A diffusion model is defined as having two steps, the forward diffusion
process where the data is gradually destroyed, and the learned inverse
diffusion process which reconstructs the data, and is used during training and
inference. In our case, we condition a denoising U-Net on image and speech
features to denoise a masked portion of the target frame into the desired
output. A high-level overview of this process is depicted in figure 1.

###  3.1 Diffusion process

####  3.1.1 Forward diffusion process

As defined by [61], the forward diffusion process is a Markov chain that adds
small amounts of noise to the data $y$ over a predefined number of time steps
T, until the data is completely destroyed at time step t=T. This state is
represented as $y_{T}$ with $y_{0}$ representing the data before any noise was
added to it. The Markov chain is defined by:

$q\left(y_{1:T}|y_{0}\right):=\prod_{y=1}^{T}q\left(y_{t}|y_{t-1}\right)$ (1)

where at each step, Gaussian noise is added by:

$q\left(y_{t}|y_{t-1}\right):=\mathcal{N}\left(y_{t};\sqrt{\alpha_{t}}y_{t-1},(1-\alpha_{t})I\right),$
(2)

with $\alpha_{t}:=(1-\beta_{t})$, representing the hyperparameters of our
fixed noise scheduler. [24] show that it is possible to sample $y_{t}$ at any
step $t$ in closed form:

$q\left(y_{t}|y_{0}\right):=\mathcal{N}\left(y_{t};\sqrt{\bar{\alpha_{t}}}y_{0},(1-\bar{\alpha_{t}})I\right),$
(3)

with $\bar{\alpha_{t}}:=\prod_{s=1}^{t}\alpha_{s}$. This is an important
observation, as it significantly speeds up the forward diffusion process, and
can be used to train a model on the fly with random noise levels at each
forward step.

####  3.1.2 Inverse diffusion process

Given a noisy image $\bar{y}$ defined as:

$\bar{y}:=\sqrt{\bar{\alpha}}y_{0}+\sqrt{1-\bar{\alpha}}\epsilon,\epsilon\sim\mathcal{N}(0,I)$
(4)

the goal of the Inverse diffusion process is to learn an algorithm that can
denoise and restore the noisy image to its original image $Y_{0}$. Following
the approach in [58], we train a neural network
$f_{\theta}(x,\bar{y},\bar{\alpha},\omega)$, a 2D U-Net[56], to predict the
noise generated at time $t$, optimising the $L_{simple}$ objective proposed by
[24]:

$\mathbb{E}_{t,y{{}_{0}},\epsilon}\bigg{[}\left\|f_{\theta}(x,\sqrt{\bar{\alpha}}y_{0}+\sqrt{1-\bar{\alpha}}\epsilon,\bar{\alpha},\omega)-\epsilon\right\|^{2}\bigg{]}$
(5)

where $x$ represents the identity and previous frame input to our network,
$\bar{y}$ the noisy image, $\bar{\alpha}$ the noise level, and $\omega$ the
audio features. During training, we only calculate the loss for the masked
region of the face to conserve computational resources, following the approach
in [58].

Following [24], to run inference, each step of the inverse diffusion process
can then be computed by:

$y_{t-1}\leftarrow\frac{1}{\sqrt{\alpha_{t}}}\left(y_{t}-\frac{1-\alpha_{t}}{\sqrt{1-\bar{\alpha_{t}}}}f_{\theta}(x,y_{t},\bar{\alpha_{t}})\right)+\sqrt{1-\alpha_{t}}\epsilon_{t},$
(6)

where $\epsilon\sim\mathcal{N}(0,I)$. The inverse diffusion step is then
repeated T times. Please see figure 1 for a high-level view of our network
architecture, and to better understand where each equation is used. For a more
detailed discussion behind these equations, and how they are derived, please
see [24, 61, 64].

###  3.2 Model Architecture

Figure 1: High-level overview of the network architecture. Left of the dashed
line indicates the training procedure, right of it depicts the inference
procedure. $\bigoplus$ represents the concatenation operator, and
$\dashrightarrow$ represents a skip connection. The current frame is passed
through the forward diffusion process where the noise is computed and added to
the masked region of the face, obtaining noisy frame $Y_{t}$ (Equation 3). The
previous and identity frames are then concatenated channel-wise to it, forming
a 128x128x9 feature and passed to the U-net directly. Audio features and noise
level information are fed into the U-net through conditional residual blocks
as described in equation 7. During inference, the predicted noise is removed
from noisy image $Y_{t}$, obtaining $Y_{t-1}$. The previous and identity
frames are concatenated to $Y_{t-1}$, and the process is repeated until the
image is fully denoised (Equation 6).

Figure 1 depicts the overall architecture of our model. We frame the problem
of audio-driven video editing as a conditional inpainting task with a few key
changes. Traditionally, inpainting is an image-to-image translation task where
a neural network must learn to fill in a masked out region of the image with
realistic content. For video editing, we must provide the network with
additional context, to help guide its generation process. To do this, we split
the conditioning step into two categories, frame-based, and audio-based
conditioning.

Frame-Based Conditioning: For a given frame $y^{i}$ extracted from a video
consisting of frames ($y^{0}$,…,$y^{n}$), our model takes three images as
input: 1) the current masked noisy frame $y^{i}_{T}$ that is to be inpainted,
2) the previous frame $y^{(i-1)}$ in the video sequence, and 3) a constant
identity frame $y^{0}$. As our approach is auto-regressive and works on a
frame-by-frame basis, the purpose of the previous frame is to ensure that
there is temporal stability between consecutive frames. Omitting it causes the
model to output jittery, unstable frames. The identity frame is there to
encourage the model not to deviate away from the target identity during the
generation process, as so often is the case with auto-regressive models. While
the identity frame can be omitted if training a single-speaker model with
little to no adverse effects, we found that its inclusion was key to having a
model that could generalise well to unseen subjects when training on multiple
identities. These three frames are concatenated channel-wise, and fed into the
U-Net as an input feature of size [128x128x9], as depicted on the left hand
side of figure 1.

Audio-Based Conditioning:

For a given video sequence of frames ($y^{0}$ ,…, $y^{n}$), there is a
corresponding sequence of audio spectral features ($spec^{0}$ ,…, $spec^{2n}$)
extracted from the original speech signal. Each audio feature spans a 40ms
window, overlapping every 20ms. Details on how we compute these features are
provided in section 3.3. In order to provide the audio information to the
network, we extract a window of audio from ($spec^{2i-2}$ to $spec^{2i+2}$)
spanning 120 ms denoted as $z^{i}$ that is centered around the current video
frame $y^{i}$. We do this so that audio information from both the preceding
and following frames is captured within the window to guarantee the accurate
production of lip movements for plosive sounds ("p, t, k, b, d, g") by taking
into consideration that these lip movements precede the sound production. We
introduce this information to the U-net via the use of conditional residual
blocks that condition the network on audio and noise level embeddings, scaling
and shifting the hidden states of the U-net following the approach of [66]:

$h_{s+1}=z^{i}_{s}(t_{s}GN(h_{s}+t_{b}))+z^{i}_{b}$ (7)

where $h_{s}$ and $h_{s+1}$ represent consecutive hidden states of the U-Net,
($z^{i}_{s},z^{i}_{b}$) = MLP$(z^{i})$, and ($t_{s},t_{b}$) =
MLP$(\bar{\alpha}_{t})$. MLP represents a shallow neural network with a couple
of linear layers separated by a SiLu() activation function, and GN is a group
normalisation layer. This is shown in figure 1.

U-Net Set Up: In order to denoise the current noisy frame, we use a denoising
U-net [56], following the general architecture described by [58], which in
turn is based on the model proposed by [24] with modifications inspired by the
works of [16, 60]. For this work we use a lightweight 128x128 version of the
256x256 U-net architecture described by [16], omitting the class conditioning
mechanism. Like [58] we condition the model to generate the desired frames via
the concatenation of the previous and identity frames to the masked frame. We
drive the facial animation by sending audio features throughout conditional
residual blocks within the U-Net as detailed by [66], described by equation 7.
We include all details related to our U-Net configuration in table 1.

Table 1 displays the hyperparameters we use to train our diffusion model for
the task of audio-driven video editing. We train two models, a single-speaker
model trained on identity S1 of the GRID dataset, and a multispeaker model
trained on the train set of the CREMA-D dataset. A notable difference between
the two models is the use of attention. For the single-speaker model, we
omitted it from the up/downsampling layers of the U-Net, using it only within
the middle block in an effort to boost training speed. Despite this, we still
obtain pleasing results, as shown both in table 2, and in the videos provided
as part of the supplementary materials. During our experiments, we discovered
that the use of attention within the multi-speaker model was crucial for it to
generalise well to both seen and unseen speakers. We apply it at resolutions
of 32x32 within the up/downsampling layers of the U-Net. We provide more
discussion on this in section 4. To perform training we used a server of 4
32GB V100 GPUs, allowing us a batch size of 40 per GPU.

| Single ID | Multi-ID
---|---|---
Image Size | $128x128$ | $128x128$
Total Frames | $73704$ | $432000$
Diffusion Steps | $2000$ | $1000$
Noise Schedule | $Linear$ | $Cosine$
Linear Start | $1e-06$ | $NA$
Linear End | $0.01$ | $NA$
Input Channels | $10$ | $10$
Inner Channels | $64$ | $64$
Channels Multiple | $1,2,4,8$ | $1,2,3$
Attention Resolution | $NA$ | $32$
Res Blocks | $2$ | $2$
Head Channels | $32$ | $32$
Drop Out | $0.2$ | $0.2$
Batch Size | $10$ | $40$
Training Epochs | $2000$ | $735$
Learning Rate | $5e-05$ | $5e-05$
Table 1: U-Net Training Hyperparameters

###  3.3 Data Processing

####  3.3.1 Dataset

We rely on the GRID [14], and CREMA-D [6] audio-visual speech data sets to
carry out the work in this paper. GRID is a multi-speaker data set consisting
of 34 speakers (18 male, 16 female), with each speaker uttering 1000 short
6-word sentences. CREMA-D is a multi-speaker dataset consisting of 7,442
talking head clips of 91 speakers from diverse ethnic backgrounds. We present
two models: 1) A single speaker model trained on 950 videos from the speaker 1
of the GRID dataset, with the model’s performance being evaluated on the
remaining 50 videos on the task of video editing. 2) A multi-speaker model
trained on a majority of the CREMA-D dataset, with videos from identities 15,
20, 21, 30, 33, 52, 62, 81, 82, and 89 kept hidden from the model for testing
and evaluation purposes.

####  3.3.2 Audio Preprocessing

From each video within the GRID and CREMA-D datasets, we extract the audio
files and resample them at 16Khz. From the audio we compute overlapping mel-
spectrogram features with n-fft 2048, window length 640, hop length 320, and
256 mel bands. With these values, a 1-second audio feature has a shape
[50,256] that can be easily aligned to a sequence of video frames.

####  3.3.3 Video Preprocessing

First, we perform a 128x128 pixel crop centered around the face on every video
frame. We do this by aligning the face in the video to the canonical face with
a smoothing window of 7 frames, following the approach of [71]. We do this for
two reasons: To get rid of any irrelevant background, and to reduce the image
size to facilitate faster training and convergence speeds. In our initial
experiments, we used an image size of 256x256 however the model was too
expensive to train on our limited resources. It is worth noting that a video
super-resolution technique such as [39] may be applied on top of our solution
to achieve high-resolution samples.

Next, every video frame needs to have a rectangular region of the face masked
out. Using an off-the-shelf facial landmark extractor [41], we extract facial
landmark coordinates to determine the position of the jaw. Using this
information, we mask out a rectangular portion of the face that covers a
region just below the nose, as within figure 1. This face mask is computed and
applied to the frames at train time within the data loader on the fly.

During training, it is critical to hide the speaker’s jawline with a
rectangular face mask. This is because the network can easily pick up on the
strong correlation between lip and jaw movements, leading it to ignore the
speech input entirely. By hiding the jawline, we compel the model to learn to
generate lip movements based solely on the accompanying speech. As the
diffusion process relies on a single loss function, applying the rectangular
face-mask is the easiest way to prevent the frame-based input dominating over
the speech input.

####  3.3.4 Audio Video Alignment:

As described previously in section 3.2, given a video sequence with frames
($y^{0}$ ,…, $y^{n}$), there is a corresponding sequence of audio spectral
features ($spec^{0}$ ,…, $spec^{2n}$) extracted from the original speech
signal. Each audio feature spans a 40ms window, overlapping every 20ms. For
any given frame $Y^{i}$, it is aligned to audio features spanned by
($spec^{2i-2}$ to $spec^{2i+2}$). To align the first and last video frames, we
simply append silence to the start, and ends of their respective audio
features. Care must be taken when choosing the audio window, too large and the
network won’t use the most meaningful information available to it, too small
and there may not be enough context for the network to generate more complex
lip movements caused by plosives.

##  4 Experiments & Results

Method |  LSE-C$\uparrow$ |  LSE-D$\downarrow$ | FID |  SSIM$\uparrow$ |  PSNR$\uparrow$ | CPBD
---|---|---|---|---|---|---
Ground Truth CREMA-D | 5.45 | 8.12 | - | - | - | -
EAMM (Actual) | 3.98 | 8.92 | 22.52 | 0.74 | 29.43 | 0.1
EAMM (Random) | 3.95 | 8.98 | 23.04 | 0.72 | 29.21 | 0.124
PC-AVS (Actual) | 6.12 | 7.8 | 38.46 | 0.61 | 28.47 | 0.127
PC-AVS (Randon) | 6.07 | 7.82 | 40.05 | 0.59 | 28.42 | 0.11
SpeechDrivenAnimation | - | - | 155.63 | $0.844^{*}$ | $27.98^{*}$ | $0.277^{*}$
Wav2Lip(Actual) | 5.89 | 7.57 | 16.21 | 0.886 | 34.23 | 0.253
Wav2Lip(random) | 5.6 | 7.89 | 20.23 | 0.872 | 34.04 | 0.247
Make It Talk | 3.5 | 9.71 | 27.35 | 0.75 | 31.37 | 0.152
Ours (MultiSpeaker - 100) | 3.53 | 9.74 | $2.362^{\dagger}$ | 0.893 | 34.32 | 0.26
Ours (MultiSpeaker - 500) | 3.5 | 9.68 | $2.13^{\dagger}$ | 0.902 | 34.4 | 0.26
Ours (MultiSpeaker - 1000) | 3.49 | 9.69 | $2.369^{\dagger}$ | 0.863 | 34.12 | 0.242
Ours (Single Speaker) | 4.98 | 7.59 | $2.312^{\dagger}$ | 0.92 | 32.47 | 0.29
Table 2:  Quantitative comparison with previous works on image quality and lip
synchronization metrics. Most previous works we compare to require a driving
video to guide the pose of the generated speaker. For these approaches
(Actual) indicates whether we provided the ground truth video to their model
in addition to the ground truth audio to generate the new video, while
(Random) indicates that we used a random audio file instead. We report their
results under both configurations to maintain fairness. For our models we also
indicate how many diffusion timesteps were used to generate the frames during
inference. We report results for 100, 500, and 1000 inference steps. †
indicates that this metric was computed on the full frame. ∗ indicates that
these results are reported from their paper.

In this section we present two models. A single-speaker video editing model
trained on speaker S1 from the GRID dataset, and a multi-speaker model trained
on the train-split of the CREMA-D dataset. We evaluate and compare our results
to other recent audio-driven video generation methods, namely EAMM [30], PC-
AVS [87], MakeItTalk [88], Speech Driven Animation [72], and Wav2Lip [50]. All
models we test against are relevant end-to-end image-reconstruction based
methods, except for MakeItTalk, a landmark-based method we compare against for
reference purposes. We evaluate these models on the CREMA-D multispeaker test
set, reporting their scores along with our own in table 2. We generate the
videos for each model using the official publicly available implementations
with the recommended parameters.

As our models are trained explicitly for video editing, they generate only a
small portion of the overall frame, while keeping the rest as is. Therefore,
to maintain fairness, all metrics that rely on comparing the generated frame
to the ground truth are computed only on the generated portion of the image.
This limitation could also create bias in the perceptual metrics and readers
should consider this when comparing our model scores to others within the
literature.

We emphasise that the objective of this paper is to serve as a proof-of-
concept demonstrating the potential of applying denoising diffusion models to
the task of audio-driven video editing. As such, while we do not achieve
state-of-the-art in some of the metrics we report, our results still show
promising improvements over existing methods and highlight the potential of
using denoising diffusion models for this task instead of traditional GAN-
based methods.

###  4.1 Evaluation Metrics

Figure 2: Multi-speaker failure case: Over time the appearance of the speaker
slowly drifts away from the original.

We use a number of objective metrics to measure the quality of our generated
videos, allowing us to compare them directly to other state-of-the-art audio-
driven video generation methods from the literature. We compute SSIM [74]
(Structural Similarity Index Measure) $\uparrow$, PSNR (Peak Signal to Noise
Ratio) $\uparrow$, and FID [23] (Frechet Inception Distance) $\downarrow$
scores for the generated videos against their corresponding ground truth to
measure the overall quality of the generated frames. We also compute CPBD [46]
(Cumulative Probability Blur Detection) $\uparrow$ scores, and SyncNet [50,
13] Confidence (LSE-C) $\uparrow$ and Distance (LSE-D) $\downarrow$ scores. We
reiterate the point that in order to maintain fairness when computing the
image quality metrics, we only compute them on the generated portion of the
image where possible.

###  4.2 Single Speaker

We train our single speaker model on identity S1 using data from the GRID
audio-visual corpus [14]. There are 1000 videos in total, each of them roughly
3 seconds in length totaling about 50 minutes of audio-visual content for
training. We train our model on 950 videos, withholding 50 of them for testing
purposes. We train this model for 895 Epochs. As we mentioned previously, we
did not use any attention layers within the up/downsampling blocks of this
model, using it just within the middle block of the U-Net. We did this to save
on training time, however, for stronger results we recommend using it, as we
show within our multi-speaker model.

###  4.3 Multi-Speaker

We train our multi-speaker model on all identities of the CREMA-D data set
except for speakers 5, 20, 21, 30, 33, 52, 62, 81, 82, and 89, choosing to
keep them hidden from the model for testing purposes. We train the model for
735 Epochs. There are a number of key changes we make to train the multi-
speaker model. First, we use self-attention layers within the U-Net at the
32x32 resolution, as well as in the middle block. Second, we switch to a
cosine noise schedule and decrease the number of diffusion steps taken by the
model during training to 1000. Finally, we decrease the number of channel
multiples to [1,2,3]. We also experimented with training a model without
attention in the up/downsampling blocks. It failed to converge on even train
set identities. We speculate that increasing the number of inner channels used
by our U-Net from 64 to 128 or 256 would significantly improve the results, as
well as training the model for a longer amount of time. Please see table 2 for
a summary of our experiments and evaluations, compared to other popular works
in the literature, and section 4.4 for a detailed discussion surrounding the
results.

###  4.4 Results Discussion

Table 2 depicts the results our models score when tested on their unseen test
sets versus other approaches in the literature. While the results we obtain
are not state-of-the-art in all metrics, they successfully demonstrate that
using a denoising diffusion model to do audio-driven video editing, is indeed
quite feasible, and produces high-quality results comparable to other relevant
methods in the literature.

The multi-speaker model generalises quite well to unseen speakers, scoring
highly on image quality metrics, managing to outperform all other methods
except for Wav2Lip on SSIM and CPBD. The single speaker model also achieving
similarly strong results. We believe that this is due to the diffusion models
inherent ability to model complex, high-dimensional data distributions,
allowing it to learn the statistical properties of the dataset and generate
images that are similar to those in the training set. Further, as diffusion
models are trained to gradually remove noise from the target image over time,
this may help it generate smoother, and more visually pleasing results than
those generated by a GAN-based model which generates the frame in one shot.
Within the context of audio-driven video editing, achieving visually pleasing
results is a key requirement that our model fulfills. Please see the videos
attached in the supplementary material for a visual comparison between our
method and existing ones.

When evaluated on SyncNet [13] confidence (LSE-C) and distance (LSE-D) scores,
our multi-speaker results are comparable to other popular methods from the
literature, slightly outperforming MakeItTalk, but scoring lower than EAMM.
PC-AVS and Wav2Lip score the highest in that order. Notably, their approaches
significantly outperform the ground truth. We believe that this is because all
other methods are specifically trained to optimise a loss function designed to
penalise their models for poor lip synchronisation. In the case of PC-AVS and
Wav2Lip, they both rely on a strong lip sync discriminator, to encourage their
models to generate distinct, clear lip movements given speech. Our approach
uses no such losses or discriminators, inherently learning the relationship
between speech and lip movement during training. As such while our lip
synchronisation scores on unseen speakers are lower, we offer a novel approach
to the task as we do not explicitly train the model to improve lip
synchronisation.

It is also worth noting that our single-speaker model performs very well on
the synchronisation metrics mentioned above, leading us to speculate that with
more time spent learning the data distribution, our multi-speaker model could
also theoretically achieve such results.

During inference, we noticed that the multi-speaker model occasionally
struggled to maintain the identity consistent throughout the generation
process, with the problem especially prevalent if there were extreme changes
in head pose present in the original video. This is due to a buildup of small
errors, as our approach is completely auto-regressive at inference time,
relying entirely on just the previously generated frame, and identity frame to
modify the current frame. Figure 2 highlights one such instance of failure,
and the phenomenon is noticeable in some of the videos we provide in our video
abstract. We speculate that this could be alleviated in three ways 1)
introducing small amounts of face warping on the previous frame during
training in order to simulate the distortion that naturally occurs over the
generation process. This would encourage the model to look at the identity
frame in order to correct itself. 2) Simply train the model for longer. 3)
Train on a more diverse dataset of speakers captured in unconstrained
conditions such as VoxCeleb or LRS.

When testing on identities seen by the network during training by replacing
the original audio with a new one, the model achieves strong lip
synchronisation, and the identity deviation seen when testing on unseen
identities is significantly diminished, or simply does not occur over the
course of the video. This problem is also non-existent in our single-speaker
model.

We also observed that the multi-speaker model is highly sensitive to speaker
volume, and intonation, especially when exposed to speech from unseen
speakers. In instances where the speaker shouts or speaks loudly and clearly
at the microphone, the lip movement is highly accurate and appears well-
synchronised. When the volume is low, the speaker appears to be mumbling, and
the full range of lip motion is not correctly generated. Analysing the
synchronisation metrics confirmed this for us, with videos generated using
audio labelled as being "angry" or "happy", scoring significantly higher than
instances where the portrayed emotion was "sad", "fearful", or "disappointed".
We suspect that this is due to our use of spectral feature embeddings when
conditioning our network, and could be alleviated or significantly diminished
with the use of a pretrained audio encoder for speech recognition. This is
because such models are typically trained to extract the content from speech,
disregarding information considered irrelevant such as pitch, or tone, and
intonation.

##  5 Future Work

Model Speed and In The Wild Training: It is no secret that diffusion models
are slow, both to train and to sample from. Our models are no exception,
taking us approximately 6 minutes/epoch to train the single-speaker model, and
40 minutes/epoch for the multi-speaker one. We briefly experimented with
training in the latent space to speed up training following the approach of
[55], however, sample quality suffered, so we decided to operate in the pixel
space. We intend to revisit this however as improving our models training
speed is a top priority for us as it would allow us to train on larger, more
diverse, "in-the-wild" datasets such as VoxCeleb [45], or LRS [12].

Appearance Consistency: As previously discussed, our multi-speaker model’s
generated output appearance for unseen identities occasionally deviates from
the original. To investigate this phenomenon, we intend to delve deeper into
the underlying causes. Specifically, we will explore whether this effect is
due to inadequate training or insufficient diversity in the training dataset,
or a combination of both. By conducting a more detailed analysis, we hope to
gain a better understanding of how to optimize our model’s performance for a
wider range of identities. Further, we intend to fully train a model that
utilises the face warping augmentation to determine whether this truly
provides a positive impact on the generated samples.

Speech Conditioning: We plan to explore the potential of conditioning our
model with a wider range of speech features, such as experimenting with larger
or smaller window sizes when computing spectral features or using pre-trained
audio encoders such as Wav2Vec2 [3], Whisper [52], or DeepSpeech2 [1]. We
believe that incorporating such features could potentially improve the lip
synchronization performance of our model and generate even more realistic and
expressive lip movements.

##  6 Conclusion

Our results showcase the versatility of denoising diffusion models in
capturing complex relationships between audio and video signals and generating
coherent video sequences with accurate lip movements for the task of speech-
driven video editing. We are encouraged by the strong performance achieved by
our proof-of-concept approach, scoring highly on all tested metrics,
comparable to existing state of the art in end-to-end video generation.

However, our work is not without limitations. The CREMA-D dataset is
relatively small compared to other publicly available speech and video
datasets, which limits the generalizability of our approach to other domains.
Additionally, our approach requires a significant amount of computational
resources and time to train. This is a challenge for real-time applications or
for training on large-scale datasets.

We are confident that our work will inspire further research and development
in this area, leading to more efficient and effective methods for speech-
driven video editing. With the continuing advancements in machine learning and
computer vision, we believe that denoising diffusion models will play an
increasingly important role in enabling high-quality and immersive multimedia
experiences that can better reflect the diversity and richness of human
communication.

## References

  * [1] Dario Amodei, Sundaram Ananthanarayanan, Rishita Anubhai, Jingliang Bai, Eric Battenberg, Carl Case, Jared Casper, Bryan Catanzaro, Qiang Cheng, Guoliang Chen, et al.  Deep speech 2: End-to-end speech recognition in english and mandarin.  In International conference on machine learning, pages 173–182. PMLR, 2016.
  * [2] Omri Avrahami, Dani Lischinski, and Ohad Fried.  Blended diffusion for text-driven editing of natural images.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18208–18218, 2022.
  * [3] Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli.  wav2vec 2.0: A framework for self-supervised learning of speech representations.  Advances in neural information processing systems, 33:12449–12460, 2020.
  * [4] Georgios Batzolis, Jan Stanczuk, Carola-Bibiane Schönlieb, and Christian Etmann.  Conditional image generation with score-based diffusion models.  arXiv preprint arXiv:2111.13606, 2021.
  * [5] Sandika Biswas, Sanjana Sinha, Dipanjan Das, and Brojeshwar Bhowmick.  Realistic talking face animation with speech-induced head motion.  In Proceedings of the Twelfth Indian Conference on Computer Vision, Graphics and Image Processing, pages 1–9, 2021.
  * [6] Houwei Cao, David G. Cooper, Michael K. Keutmann, Ruben C. Gur, Ani Nenkova, and Ragini Verma.  Crema-d: Crowd-sourced emotional multimodal actors dataset.  IEEE Transactions on Affective Computing, 5(4):377–390, 2014.
  * [7] Lele Chen, Guofeng Cui, Celong Liu, Zhong Li, Ziyi Kou, Yi Xu, and Chenliang Xu.  Talking-head generation with rhythmic head motion.  In European Conference on Computer Vision, pages 35–51. Springer, 2020.
  * [8] Lele Chen, Zhiheng Li, Ross K Maddox, Zhiyao Duan, and Chenliang Xu.  Lip movements generation at a glance.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 520–535, 2018.
  * [9] Lele Chen, Ross K Maddox, Zhiyao Duan, and Chenliang Xu.  Hierarchical cross-modal talking face generation with dynamic pixel-wise loss.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 7832–7841, 2019.
  * [10] Nanxin Chen, Yu Zhang, Heiga Zen, Ron J Weiss, Mohammad Norouzi, and William Chan.  Wavegrad: Estimating gradients for waveform generation.  arXiv preprint arXiv:2009.00713, 2020.
  * [11] Sen Chen, Zhilei Liu, Jiaxing Liu, and Longbiao Wang.  Talking head generation driven by speech-related facial action units and audio-based on multimodal representation fusion.  arXiv preprint arXiv:2204.12756, 2022.
  * [12] Joon Son Chung, Andrew Senior, Oriol Vinyals, and Andrew Zisserman.  Lip reading sentences in the wild.  In 2017 IEEE conference on computer vision and pattern recognition (CVPR), pages 3444–3453. IEEE, 2017.
  * [13] J. S. Chung and A. Zisserman.  Out of time: automated lip sync in the wild.  In Workshop on Multi-view Lip-reading, ACCV, 2016.
  * [14] Martin Cooke, Jon Barker, Stuart Cunningham, and Xu Shao.  An audio-visual corpus for speech perception and automatic speech recognition.  The Journal of the Acoustical Society of America, 120(5):2421–2424, 2006.
  * [15] Daniel Cudeiro, Timo Bolkart, Cassidy Laidlaw, Anurag Ranjan, and Michael J Black.  Capture, learning, and synthesis of 3d speaking styles.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10101–10111, 2019.
  * [16] Prafulla Dhariwal and Alexander Nichol.  Diffusion models beat gans on image synthesis.  Advances in Neural Information Processing Systems, 34:8780–8794, 2021.
  * [17] Sefik Emre Eskimez, Ross K Maddox, Chenliang Xu, and Zhiyao Duan.  Generating talking face landmarks from speech.  In International Conference on Latent Variable Analysis and Signal Separation, pages 372–381. Springer, 2018.
  * [18] Sefik Emre Eskimez, Ross K Maddox, Chenliang Xu, and Zhiyao Duan.  End-to-end generation of talking faces from noisy speech.  In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1948–1952. IEEE, 2020.
  * [19] Wan-Cyuan Fan, Yen-Chun Chen, DongDong Chen, Yu Cheng, Lu Yuan, and Yu-Chiang Frank Wang.  Frido: Feature pyramid diffusion for complex scene image synthesis.  arXiv preprint arXiv:2208.13753, 2022.
  * [20] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio.  Generative adversarial networks.  Communications of the ACM, 63(11):139–144, 2020.
  * [21] Shuyang Gu, Dong Chen, Jianmin Bao, Fang Wen, Bo Zhang, Dongdong Chen, Lu Yuan, and Baining Guo.  Vector quantized diffusion model for text-to-image synthesis.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10696–10706, 2022.
  * [22] William Harvey, Saeid Naderiparizi, Vaden Masrani, Christian Weilbach, and Frank Wood.  Flexible diffusion modeling of long videos.  arXiv preprint arXiv:2205.11495, 2022.
  * [23] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter.  Gans trained by a two time-scale update rule converge to a local nash equilibrium.  Advances in neural information processing systems, 30, 2017.
  * [24] Jonathan Ho, Ajay Jain, and Pieter Abbeel.  Denoising diffusion probabilistic models.  Advances in Neural Information Processing Systems, 33:6840–6851, 2020.
  * [25] Jonathan Ho, Chitwan Saharia, William Chan, David J Fleet, Mohammad Norouzi, and Tim Salimans.  Cascaded diffusion models for high fidelity image generation.  J. Mach. Learn. Res., 23:47–1, 2022.
  * [26] Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J Fleet.  Video diffusion models.  arXiv preprint arXiv:2204.03458, 2022.
  * [27] Rongjie Huang, Zhou Zhao, Huadai Liu, Jinglin Liu, Chenye Cui, and Yi Ren.  Prodiff: Progressive fast diffusion model for high-quality text-to-speech.  In Proceedings of the 30th ACM International Conference on Multimedia, pages 2595–2605, 2022.
  * [28] Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros.  Image-to-image translation with conditional adversarial networks.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1125–1134, 2017.
  * [29] Amir Jamaludin, Joon Son Chung, and Andrew Zisserman.  You said that?: Synthesising talking faces from audio.  International Journal of Computer Vision, 127(11):1767–1779, 2019\.
  * [30] Xinya Ji, Hang Zhou, Kaisiyuan Wang, Qianyi Wu, Wayne Wu, Feng Xu, and Xun Cao.  Eamm: One-shot emotional talking face via audio-based emotion-aware motion model.  In ACM SIGGRAPH 2022 Conference Proceedings, pages 1–10, 2022.
  * [31] Xinya Ji, Hang Zhou, Kaisiyuan Wang, Wayne Wu, Chen Change Loy, Xun Cao, and Feng Xu.  Audio-driven emotional video portraits.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 14080–14089, 2021.
  * [32] Tero Karras, Timo Aila, Samuli Laine, Antti Herva, and Jaakko Lehtinen.  Audio-driven facial animation by joint end-to-end learning of pose and emotion.  ACM Transactions on Graphics (TOG), 36(4):1–12, 2017.
  * [33] Sungwon Kim, Heeseung Kim, and Sungroh Yoon.  Guided-tts 2: A diffusion model for high-quality adaptive text-to-speech with untranscribed data.  arXiv preprint arXiv:2205.15370, 2022.
  * [34] Diederik P Kingma and Max Welling.  Auto-encoding variational bayes.  arXiv preprint arXiv:1312.6114, 2013.
  * [35] Zhifeng Kong, Wei Ping, Jiaji Huang, Kexin Zhao, and Bryan Catanzaro.  Diffwave: A versatile diffusion model for audio synthesis.  arXiv preprint arXiv:2009.09761, 2020.
  * [36] Neeraj Kumar, Srishti Goel, Ankur Narang, and Mujtaba Hasan.  Robust one shot audio to video generation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pages 770–771, 2020.
  * [37] Avisek Lahiri, Vivek Kwatra, Christian Frueh, John Lewis, and Chris Bregler.  Lipsync3d: Data-efficient learning of personalized 3d talking faces from video using pose and lighting normalization.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2755–2764, 2021.
  * [38] Alon Levkovitch, Eliya Nachmani, and Lior Wolf.  Zero-shot voice conditioning for denoising diffusion tts models.  arXiv preprint arXiv:2206.02246, 2022.
  * [39] Chengxu Liu, Huan Yang, Jianlong Fu, and Xueming Qian.  Learning trajectory-aware transformer for video super-resolution.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5687–5696, 2022.
  * [40] Yuanxun Lu, Jinxiang Chai, and Xun Cao.  Live speech portraits: real-time photorealistic talking-head animation.  ACM Transactions on Graphics (TOG), 40(6):1–17, 2021.
  * [41] Camillo Lugaresi, Jiuqiang Tang, Hadon Nash, Chris McClanahan, Esha Uboweja, Michael Hays, Fan Zhang, Chuo-Ling Chang, Ming Guang Yong, Juhyun Lee, et al.  Mediapipe: A framework for building perception pipelines.  arXiv preprint arXiv:1906.08172, 2019.
  * [42] Andreas Lugmayr, Martin Danelljan, Andres Romero, Fisher Yu, Radu Timofte, and Luc Van Gool.  Repaint: Inpainting using denoising diffusion probabilistic models.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11461–11471, 2022.
  * [43] Chenlin Meng, Yang Song, Jiaming Song, Jiajun Wu, Jun-Yan Zhu, and Stefano Ermon.  Sdedit: Image synthesis and editing with stochastic differential equations.  arXiv preprint arXiv:2108.01073, 2021.
  * [44] Gaurav Mittal and Baoyuan Wang.  Animating face using disentangled audio representations.  In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 3290–3298, 2020.
  * [45] Arsha Nagrani, Joon Son Chung, and Andrew Zisserman.  Voxceleb: a large-scale speaker identification dataset.  arXiv preprint arXiv:1706.08612, 2017.
  * [46] Niranjan D Narvekar and Lina J Karam.  A no-reference image blur metric based on the cumulative probability of blur detection (cpbd).  IEEE Transactions on Image Processing, 20(9):2678–2683, 2011.
  * [47] Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen.  Glide: Towards photorealistic image generation and editing with text-guided diffusion models.  arXiv preprint arXiv:2112.10741, 2021.
  * [48] Alexander Quinn Nichol and Prafulla Dhariwal.  Improved denoising diffusion probabilistic models.  In International Conference on Machine Learning, pages 8162–8171. PMLR, 2021.
  * [49] Vadim Popov, Ivan Vovk, Vladimir Gogoryan, Tasnima Sadekova, and Mikhail Kudinov.  Grad-tts: A diffusion probabilistic model for text-to-speech.  In International Conference on Machine Learning, pages 8599–8608. PMLR, 2021.
  * [50] KR Prajwal, Rudrabha Mukhopadhyay, Vinay P Namboodiri, and CV Jawahar.  A lip sync expert is all you need for speech to lip generation in the wild.  In Proceedings of the 28th ACM International Conference on Multimedia, pages 484–492, 2020.
  * [51] Konpat Preechakul, Nattanat Chatthee, Suttisak Wizadwongsa, and Supasorn Suwajanakorn.  Diffusion autoencoders: Toward a meaningful and decodable representation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10619–10629, 2022.
  * [52] Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever.  Robust speech recognition via large-scale weak supervision.  arXiv preprint arXiv:2212.04356, 2022.
  * [53] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen.  Hierarchical text-conditional image generation with clip latents.  arXiv preprint arXiv:2204.06125, 2022.
  * [54] Alexander Richard, Michael Zollhöfer, Yandong Wen, Fernando De la Torre, and Yaser Sheikh.  Meshtalk: 3d face animation from speech using cross-modality disentanglement.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1173–1182, 2021.
  * [55] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10684–10695, 2022.
  * [56] Olaf Ronneberger, Philipp Fischer, and Thomas Brox.  U-net: Convolutional networks for biomedical image segmentation.  In International Conference on Medical image computing and computer-assisted intervention, pages 234–241. Springer, 2015.
  * [57] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman.  Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation.  arXiv preprint arXiv:2208.12242, 2022.
  * [58] Chitwan Saharia, William Chan, Huiwen Chang, Chris Lee, Jonathan Ho, Tim Salimans, David Fleet, and Mohammad Norouzi.  Palette: Image-to-image diffusion models.  In ACM SIGGRAPH 2022 Conference Proceedings, pages 1–10, 2022.
  * [59] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S Sara Mahdavi, Rapha Gontijo Lopes, et al.  Photorealistic text-to-image diffusion models with deep language understanding.  arXiv preprint arXiv:2205.11487, 2022.
  * [60] Chitwan Saharia, Jonathan Ho, William Chan, Tim Salimans, David J Fleet, and Mohammad Norouzi.  Image super-resolution via iterative refinement.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022\.
  * [61] Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli.  Deep unsupervised learning using nonequilibrium thermodynamics.  In International Conference on Machine Learning, pages 2256–2265. PMLR, 2015.
  * [62] Luchuan Song, Bin Liu, Guojun Yin, Xiaoyi Dong, Yufei Zhang, and Jia-Xuan Bai.  Tacr-net: Editing on deep video and voice portraits.  In Proceedings of the 29th ACM International Conference on Multimedia, pages 478–486, 2021.
  * [63] Linsen Song, Wayne Wu, Chen Qian, Ran He, and Chen Change Loy.  Everybody’s talkin’: Let me talk as you want.  IEEE Transactions on Information Forensics and Security, 17:585–598, 2022.
  * [64] Yang Song and Stefano Ermon.  Generative modeling by estimating gradients of the data distribution.  Advances in Neural Information Processing Systems, 32, 2019.
  * [65] Yang Song, Jingwen Zhu, Dawei Li, Andy Wang, and Hairong Qi.  Talking face generation by conditional recurrent adversarial network.  In Proceedings of the 28th International Joint Conference on Artificial Intelligence, pages 919–925, 2019.
  * [66] Michał Stypułkowski, Konstantinos Vougioukas, Sen He, Maciej Zięba, Stavros Petridis, and Maja Pantic.  Diffused heads: Diffusion models beat gans on talking-face generation.  arXiv preprint arXiv:2301.03396, 2023.
  * [67] Supasorn Suwajanakorn, Steven M Seitz, and Ira Kemelmacher-Shlizerman.  Synthesizing obama: learning lip sync from audio.  ACM Transactions on Graphics (ToG), 36(4):1–13, 2017.
  * [68] Jaesung Tae, Hyeongju Kim, and Taesu Kim.  Editts: Score-based editing for controllable text-to-speech.  arXiv preprint arXiv:2110.02584, 2021.
  * [69] Sarah Taylor, Taehwan Kim, Yisong Yue, Moshe Mahler, James Krahe, Anastasio Garcia Rodriguez, Jessica Hodgins, and Iain Matthews.  A deep learning approach for generalized speech animation.  ACM Transactions on Graphics (TOG), 36(4):1–11, 2017.
  * [70] Justus Thies, Mohamed Elgharib, Ayush Tewari, Christian Theobalt, and Matthias Nießner.  Neural voice puppetry: Audio-driven facial reenactment.  In European conference on computer vision, pages 716–731. Springer, 2020.
  * [71] Konstantinos Vougioukas, Stavros Petridis, and Maja Pantic.  End-to-end speech-driven facial animation with temporal gans.  ArXiv, abs/1805.09313, 2018.
  * [72] Konstantinos Vougioukas, Stavros Petridis, and Maja Pantic.  Realistic speech-driven facial animation with gans.  International Journal of Computer Vision, 128(5):1398–1413, 2020\.
  * [73] Suzhen Wang, Lincheng Li, Yu Ding, Changjie Fan, and Xin Yu.  Audio2head: Audio-driven one-shot talking-head generation with natural head motion.  arXiv preprint arXiv:2107.09293, 2021.
  * [74] Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli.  Image quality assessment: from error visibility to structural similarity.  IEEE transactions on image processing, 13(4):600–612, 2004.
  * [75] Xin Wen, Miao Wang, Christian Richardt, Ze-Yin Chen, and Shi-Min Hu.  Photorealistic audio-driven video portraits.  IEEE Transactions on Visualization and Computer Graphics, 26(12):3457–3466, 2020.
  * [76] Haozhe Wu, Jia Jia, Haoyu Wang, Yishun Dou, Chao Duan, and Qingshan Deng.  Imitating arbitrary talking style for realistic audio-driven talking face synthesis.  In Proceedings of the 29th ACM International Conference on Multimedia, pages 1478–1486, 2021.
  * [77] Zhisheng Xiao, Karsten Kreis, and Arash Vahdat.  Tackling the generative learning trilemma with denoising diffusion gans.  arXiv preprint arXiv:2112.07804, 2021.
  * [78] Dongchao Yang, Jianwei Yu, Helin Wang, Wen Wang, Chao Weng, Yuexian Zou, and Dong Yu.  Diffsound: Discrete diffusion model for text-to-sound generation.  arXiv preprint arXiv:2207.09983, 2022.
  * [79] Ling Yang, Zhilong Zhang, Yang Song, Shenda Hong, Runsheng Xu, Yue Zhao, Yingxia Shao, Wentao Zhang, Bin Cui, and Ming-Hsuan Yang.  Diffusion models: A comprehensive survey of methods and applications.  arXiv preprint arXiv:2209.00796, 2022.
  * [80] Ruihan Yang, Prakhar Srivastava, and Stephan Mandt.  Diffusion probabilistic modeling for video generation.  arXiv preprint arXiv:2203.09481, 2022.
  * [81] Ran Yi, Zipeng Ye, Juyong Zhang, Hujun Bao, and Yong-Jin Liu.  Audio-driven talking face video generation with learning-based personalized head pose.  arXiv preprint arXiv:2002.10137, 2020.
  * [82] Chenxu Zhang, Yifan Zhao, Yifei Huang, Ming Zeng, Saifeng Ni, Madhukar Budagavi, and Xiaohu Guo.  Facial: Synthesizing dynamic talking face with implicit attribute learning.  In Proceedings of the IEEE/CVF international conference on computer vision, pages 3867–3876, 2021.
  * [83] Mingyuan Zhang, Zhongang Cai, Liang Pan, Fangzhou Hong, Xinying Guo, Lei Yang, and Ziwei Liu.  Motiondiffuse: Text-driven human motion generation with diffusion model.  arXiv preprint arXiv:2208.15001, 2022.
  * [84] Zhimeng Zhang, Lincheng Li, Yu Ding, and Changjie Fan.  Flow-guided one-shot talking face generation with a high-resolution audio-visual dataset.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3661–3670, 2021.
  * [85] Ruiqi Zhao, Tianyi Wu, and Guodong Guo.  Sparse to dense motion transfer for face image animation.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1991–2000, 2021.
  * [86] Hang Zhou, Yu Liu, Ziwei Liu, Ping Luo, and Xiaogang Wang.  Talking face generation by adversarially disentangled audio-visual representation.  In Proceedings of the AAAI conference on artificial intelligence, volume 33, pages 9299–9306, 2019.
  * [87] Hang Zhou, Yasheng Sun, Wayne Wu, Chen Change Loy, Xiaogang Wang, and Ziwei Liu.  Pose-controllable talking face generation by implicitly modularized audio-visual representation.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4176–4186, 2021.
  * [88] Yang Zhou, Xintong Han, Eli Shechtman, Jose Echevarria, Evangelos Kalogerakis, and Dingzeyu Li.  Makelttalk: speaker-aware talking-head animation.  ACM Transactions on Graphics (TOG), 39(6):1–15, 2020.
  * [89] Hao Zhu, Huaibo Huang, Yi Li, Aihua Zheng, and Ran He.  Arbitrary talking face generation via attentional audio-visual coherence learning.  In Proceedings of the Twenty-Ninth International Conference on International Joint Conferences on Artificial Intelligence, pages 2362–2368, 2021.
  * [90] Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros.  Unpaired image-to-image translation using cycle-consistent adversarial networks.  In Proceedings of the IEEE international conference on computer vision, pages 2223–2232, 2017.