# Evolving Storytelling: Benchmarks and Methods for New Character
Customization with Diffusion Models

Xiyu Wang<EMAIL_ADDRESS>Nanyang Technological
UniversitySingaporeSingapore ,  Yufei Wang<EMAIL_ADDRESS>Nanyang
Technological UniversitySingaporeSingapore ,  Satoshi Tsutsui
<EMAIL_ADDRESS>Nanyang Technological UniversitySingaporeSingapore
,  Weisi Lin<EMAIL_ADDRESS>Nanyang Technological
UniversitySingaporeSingapore ,  Bihan Wen<EMAIL_ADDRESS>Nanyang
Technological UniversitySingaporeSingapore and  Alex C. Kot<EMAIL_ADDRESS>Nanyang Technological UniversitySingaporeSingapore

###### Abstract.

Diffusion-based models for story visualization have shown promise in
generating content-coherent images for storytelling tasks. However, how to
effectively integrate new characters into existing narratives while
maintaining character consistency remains an open problem, particularly with
limited data. Two major limitations hinder the progress: (1) the absence of a
suitable benchmark due to potential character leakage and inconsistent text
labeling, and (2) the challenge of distinguishing between new and old
characters, leading to ambiguous results. To address these challenges, we
introduce the NewEpisode benchmark, comprising refined datasets designed to
evaluate generative models’ adaptability in generating new stories with fresh
characters using just a single example story. The refined dataset involves
refined text prompts and eliminates character leakage. Additionally, to
mitigate the character confusion of generated results, we propose EpicEvo, a
method that customizes a diffusion-based visual story generation model with a
single story featuring the new characters seamlessly integrating them into
established character dynamics. EpicEvo introduces a novel adversarial
character alignment module to align the generated images progressively in the
diffusive process, with exemplar images of new characters, while applying
knowledge distillation to prevent forgetting of characters and background
details. Our evaluation quantitatively demonstrates that EpicEvo outperforms
existing baselines on the NewEpisode benchmark, and qualitative studies
confirm its superior customization of visual story generation in diffusion
models. In summary, EpicEvo provides an effective way to incorporate new
characters using only one example story, unlocking new possibilities for
applications such as serialized cartoons.

Generative Diffusion Model, Story Visualization, Generative Model
Customization

††copyright: none Figure 1. Given a text-to-image story generation model
trained on cartoons, we aim to customize the pretrained model so that end
users can create new storylines featuring unseen new characters specified by
only one example story (i.e., 5 frames). For example, the model pretrained on
The Flintstones is capable of generating visual stories for existing
characters such as Fred and Wilma, who are frequently depicted in the training
datasets (row (a)). However, it falls short when generating stories featuring
unseen new characters like Slaghoople because it has little prior knowledge of
her (row (b)). Our customized model can take only one story of Slaghoople and
generate new stories involving both new and existing characters (row (c)).

##  1\. Introduction

Diffusion models have demonstrated impressive performance in visual story
generation tasks (Li et al., 2019), holding promise for visual narratives,
such as generating a new episode of existing comic books while maintaining the
storyline continuity. However, existing generative models (Li et al., 2019;
Maharana and Bansal, 2021; Maharana et al., 2021; Rahman et al., 2023; Pan et
al., 2022; Maharana et al., 2022) are highly limited when visualizing a story
involving characters that appear infrequently or are absent from the training
data. As illustrated in Fig. 1, existing visual story generation models
struggle to depict Slaghoople (a character we held out from the training set)
consistently, resulting in varied appearances due to the lack of character
priors. Moreover, this generalization ability to unseen characters (referred
to as story character customization in our paper) has not been adequately
evaluated due to the absence of proper benchmarks in previous works (Li et
al., 2019; Maharana and Bansal, 2021; Maharana et al., 2021; Rahman et al.,
2023; Pan et al., 2022; Maharana et al., 2022). In this paper, we
systematically investigate story character customization by curating an
appropriate dataset and designing a new method tailored to this task.

Our NewEpisode Benchmark: To systematically address the challenge of
customizing visual story generation models, we identified a critical issue:
the absence of datasets featuring new characters in the testing split. To
tackle this, we introduce the NewEpisode benchmark, whose test set contains
unseen characters that have not appeared in the pretraining111We use
pretraining to refer to the process of training a visual story generation
model and customization to refer to the process of adding new characters. set.
Refining existing datasets (Maharana and Bansal, 2021; Li et al., 2019), we
exclusively utilize stories with main characters for the pretraining and
introduce minor supporting characters as new characters in the test set. It is
worth noting that this process is not merely a re-dividing of existing
datasets due to the ambiguous textual descriptions in these datasets. Minor
characters are often referred to by a generic name in texts (e.g., ”an alien”
instead of its character name), often causing the accidental inclusion of the
unseen test characters in the pretraining set. To prevent this leakage, we
manually investigated every single image-text pair of existing datasets, and
even modified some textual descriptions to disambiguate the referent
characters (see Fig. 2 for details). In contrast to datasets for general
customization (Ruiz et al., 2023; Kumari et al., 2023; Gal et al., 2022; Po et
al., 2023; Gu et al., 2024), which focus on personalizing a single concept
and/or merging multiple concepts based on a pretrained general text-to-image
model (Rombach et al., 2021; Saharia et al., 2022), our NewEpisode benchmark
is built for visual story generation models and offers testing samples that
focus more on story character dynamics. This allows for in-depth exploration
of how new characters can be integrated into existing storylines while
avoiding disrupting the established character dynamics. NewEpisode benchmark
thus provides a more nuanced approach to story character customization,
emphasizing the seamless integration of new characters into complex
narratives. We summarize some differences between NewEpisode and datasets used
in other model customization works in Table. 1.

Technical Challenges: How do existing off-the-shelf methods (Ruiz et al.,
2023; Gal et al., 2022; Kumari et al., 2023) perform on our new dataset? Our
preliminary studies indicate that existing methods might fall short within the
context of story character customization. Specifically, we identify the core
challenge is that: compared to a general text-to-image model (Rombach et al.,
2021), visual story generation models (Rahman et al., 2023; Pan et al., 2022)
pretrained on a specific story dataset have stronger priors of existing
characters. For instance, in Fig. 4-b, the model might be confused by the
appearance of the lizard-like creature, Rockzilla. As a result, the new
character might be straightly ignored, such as the case in Fig. 2-h, or the
new character might be rendered in a corrupted way, such as the case in
Fig.2-g.

Our EpicEvo Model: To address the aforementioned challenge, we propose a story
character customization method, namely EpicEvo, which is featured in Fig. 3.
In order to mitigate the overly strong influence of existing characters, we
design an adversarial character alignment mechanism. As the name suggests,
this mechanism is inspired by GANs(Goodfellow et al., 2020), but we have
devised it to be compatible with modern diffusion models. Specifically, we
train a discriminator to judge whether the images generated by the visual
story generation network contain a certain character in the latent space of
the final step of the diffusion process. Our experiments show that this
alignment encourages the generation of both new and existing characters
whenever they appear in the captions. Additionally, we empirically found that
the diversity of output stories can decrease if the model is extensively tuned
on the customization samples. To mitigate this decline in diversity, we
introduce a distillation loss, which effectively leverages both the pretrained
model and customized models to enhance the generation quality even further.

In summary, our main contributions are threefold:

  1. (1)

We propose the NewEpisode benchmark, which contains refined datasets for
pertaining visual story generation networks and a group of new characters
available for the training and testing of story character customization
methods. The NewEpisode benchmark presents a non-trivial challenge to existing
visual story generation models and customization methods.

  2. (2)

We introduce EpicEvo, our story character customization method that enables
the evolution of existing storylines by allowing the model to learn to
generate stories featuring existing and/or new characters. EpicEvo encourages
the model to generate characters distinctively using the adversarial character
alignment module and it preserves model priors via distilling knowledge from a
pretrained model.

  3. (3)

To the best of our knowledge, NewEpisode is the first benchmark specifically
built for story character customization within the context of visual story
generation. Our method, EpicEvo, is designed to tackle this challenging
benchmark such that it enables end-users to create their branch of new stories
with new characters using a few samples, making it possible to continue the
legend described by the canon.

Figure 2. Illustration of the dataset construction. The proposed dataset has
two main contributions compared with the original datasets (FlintStonesSV
(Maharana and Bansal, 2021) and PororoSV (Li et al., 2019)): first, as shown
on the left side, the original text prompts lack description for supporting
characters, making the training/evaluation less tractable while our dataset
provides more comprehensive and consistent annotations. Second, the original
data can not well establish the adapting performance of models on new
characters since “the leakage of character information” is shown on the right
side. We reorganize the dataset based on our previously more detailed
annotated dataset so that there is no leakage of new characters in the
customization set to the pertaining set.

##  2\. Related Works

Text-to-image Generation has been a heated topic in recent years. Earlier
generative models were mostly based on GAN (Goodfellow et al., 2020). Follow-
up works (Brock et al., 2019; Zhang et al., 2017; Karras et al., 2021, 2019)
based on GAN (Goodfellow et al., 2020) improved the model complexity and led
to noticeable improvements in the synthesized image quality. Some other
technical approaches featuring VAE (Kingma and Welling, 2014) and VQ-VAE
(Razavi et al., 2019) also demonstrated the abilities to generate realistic
images. Later, diffusion-based models (Saharia et al., 2022; Ramesh et al.,
2022; Dhariwal and Nichol, 2021; Ho et al., 2020; Nichol et al., 2022) emerged
as promising ways of composing more complex and diverse images based on text
prompts. Recently, the emergence of latent diffusion models, e.g., Stable
Diffusion (Rombach et al., 2021), demonstrated unprecedented generative
ability by performing diffusion operations in the latent space of a VAE
encoder (Kingma and Welling, 2014).

Visual Story Generation involves creating sequences of images that form a
coherent visual narrative, i.e., a series of images with consistent contents
such as characters, objects, background, style, etc. StoryGAN (Li et al.,
2019) first introduced the concept of visual story generation. (Li et al.,
2019) synthesizes several images based on the same number of prompts. The
model processes contexts, such as prompts for all images, to generate more
coherent content. Following works (Maharana et al., 2021; Maharana and Bansal,
2021; Li, 2022) further enhanced the model capability based on other
techniques. With the rise of latent diffusion models (Rombach et al., 2021),
(Rahman et al., 2023; Maharana et al., 2022) propose to synthesize visual
stories based on pretrained diffusion models (Rombach et al., 2021).
Specifically, Make-A-Story (Rahman et al., 2023) constructs a memory module to
store historical information such as previously generated images and prompts
and utilize such information to condition the latent diffusion model (Rombach
et al., 2021). Another work (Pan et al., 2022) proposes a more complex
architecture that leverages multi-modal encoders (Radford et al., 2021; Li et
al., 2022) to better encode historical information. Recently, (Liu et al.,
2024) proposed the task of open-ended story generation where the task is to
generate a series of images with 1-2 characters recurring. But it falls short
for story with more complex dynamics, such as dynamics between characters. In
sum, visual story generation models are good at generating stories for
characters it has frequently seen during training, but struggle to generalize
for new characters.

Model Customization enables end users to generate images that contain unique
concepts, e.g., an object, a certain art style. Previous model customization
methods (Ruiz et al., 2023; Kumari et al., 2023; Gal et al., 2022) primarily
focus on customizing an off-the-shelf text-to-image model, such as Stable
Diffusion (Rombach et al., 2021) . Specifically, (Ruiz et al., 2023) and
(Kumari et al., 2023) train the text-to-image model to associate a unique
concept with an uncommon token, and (Gal et al., 2022) tries to invert the
text embedding vector that could prompt the model to generate a unique
concept. Following works (Po et al., 2023; Gu et al., 2024) further combines
customization with efficient adaptation methods (Hu et al., 2022) and proposes
generating multiple concepts in one image. Still, our preliminary research
finds efficient tuning methods (Hu et al., 2022) or tuning a smaller portion
of the diffusion model can be less efficient for the story character
customization task. In this paper, we focus on customizing visual story-
generation models such that the customized model could generate stories for
new characters.

Table 1. Differences between NewEpisode and other general model customization
datasets. NewEpisode focuses on customizing a visual story generation model
with new characters filtered from the original story visualization datasets.
To ensure content-consistent storytelling, the underlying model of our method
is a Visual Story Generation model (Pan et al., 2022) tuned on the pretraining
datasets of the NewEpisode instead of a general stable diffusion model.

Method | Task | Backbone | New Concept Types | Number of New Concepts | Scale of Evaluation
---|---|---|---|---|---
DreamBooth (Ruiz et al., 2023) | Single Image Generation | SD (Rombach et al., 2021) / Imagen (Saharia et al., 2022) | Daily Objects, Pets | 30 | 5
Textual Inversion (Gal et al., 2022) | SD (Rombach et al., 2021) | Daily Objects, Pets, Humans, Styles | N/A | N/A
Custom Diffusion (Kumari et al., 2023) | SD (Rombach et al., 2021) | Daily Objects, Pets, Cars, etc. | 101 | 10
Mix of Show (Gu et al., 2024) | SD (Rombach et al., 2021) | Movie Characters, Cartoon Characters | N/A | N/A
Orthaganol Adaptation (Po et al., 2023) |  | SD (Rombach et al., 2021) | Movie Characters, Pets | 12 | 16
Ours (NewEpisode) | Visual Story Generation | Visual Story Generation Model (Pan et al., 2022) | Cartoon Characters | 15 | 40 / 200

##  3\. Dataset and Benchmark

In this section, we illustrate how the NewEpisode benchmark is constructed.
For simplicity, without further notice, a story is assumed to have 5 images
and 5 corresponding text prompts. Each image can feature different characters
and images can re-appear in multiple stories.

###  3.1. Dataset

NewEpisodeFlintStones. The proposed dataset NewEpisodeFlintStones is based on
FlintStonesSv (Maharana and Bansal, 2021) dataset that originated from a text-
to-video dataset proposed by (Gupta et al., 2018). The authors of (Maharana
and Bansal, 2021) sampled images from consecutive video clips in the text-to-
video dataset and there are 20132, 2071, and 2309 samples for training,
validation, and testing. Each story contains a fixed length of 5 images and 5
corresponding text captions. The occurrence of seven main characters is
annotated in the text captions, and there is a group of supporting characters
only appearing a few times without being carefully annotated as shown in Fig.
2. To create datasets free of these supporting characters for pretraining and
customization, we manually went through the entire dataset with a supporting
LLM tool (Liu et al., 2023). In sum, we identify nine supporting characters in
the original training, validation, and testing split. There are 382 different
images containing these characters and 829 stories containing these new
characters (the same image can appear in different stories). We construct the
customization dataset based on these stories that depict interactions between
these new characters and existing characters. We take one story for each new
character during customization and subsequently test the customized model on
all other stories about these new characters.

NewEpisodePororo. Similarly, PororoSV (Li et al., 2019) is also sampled from a
dataset originally used for other purposes (Kim et al., 2017). In the PororoSV
dataset, each story also contains 5 images and 5 corresponding text captions
illustrating the character actions. There are 10191, 2334, and 2208 stories
for training, validation, and testing. Upon close inspection, we found the
captions for PororoSV (Li et al., 2019) are more organized, i.e., supporting
characters are also referred to by their names. Therefore, we performed a
rule-based text matching for the text captions of each story and identified
2976 images related to these supporting characters and a total of 4976 stories
that contain these images. Excluding them from the training split, we obtain
the pretraining datasets for NewEpisodePororo that do not contain these
supporting characters, i.e., new characters. For customization and testing the
customized model, we follow the same protocol for NewEpisodeFlintstones.

###  3.2. Benchmarking

As illustrated in Table 1, we have a larger scale of evaluation stories
available for quantitative and qualitative evaluation of the customized visual
story generation model. We show the average number of available individual
images for each character. During customization, the model is provided with
$1$ story for each new character (such stories are ensured to have images all
relevant to the new character for efficient customization). For
NewEpisodeFlintstones, our evaluation protocol lets the model generate all the
relevant stories for each new character. For NewEpisodePororo, we randomly
select a maximum of 100 stories for each new character using a fixed random
seed as many images are reused multiple times in many stories. For
quantitative evaluation, we focus on the images that contain the new character
and calculate automatic metrics such as Fréchet inception distance (FID)
(Heusel et al., 2017), CLIP-T, and CLIP-I (Radford et al., 2021) for them.
Specifically, we leverage the FID score as we have a larger scale of samples
compared to general model customization works. Meanwhile, we also evaluate the
CLIP-I and CLIP-T scores as it is widely used in various model customization
works (Ruiz et al., 2023; Po et al., 2023; Kumari et al., 2023; Gal et al.,
2022; Gu et al., 2024).

##  4\. Methodology

Figure 3. Illustration of EpicEvo. When generating the $i$-th image, the model
takes all text inputs and previously generated images, encodes them using CLIP
(Radford et al., 2021) and BLIP (Li et al., 2022) text and visual encoders,
and conditions the denoising network (Ronneberger et al., 2015). To enable
better story character customization, the denoising network has three training
objectives: 1) predict the noise $\epsilon$ added to noisy latent $z_{t}$ at
time step t such that the estimated noise $\epsilon_{\theta}$ is close to the
ground truth noise $\epsilon$. This is reflected by the mean square loss (MSE)
loss $\mathcal{L}_{mse}$; 2) maximizing the probability that the discriminator
network will classify the generated image as a matching image w.r.t. to the
reference image, i.e., minimize the adversarial character alignment loss
$\mathcal{L}_{adv}^{G}$; 3) aligning with the pretrained model by minimizing a
distillation loss  $\mathcal{L}_{distill}$ calculated as the L2 distance
between the noises estimated by the pretrained model and the customized model.
We denote the latent denoised by the customized model and pretrained model as
$\hat{z}_{t-1}^{\theta}$ and $\hat{z}_{t-1}^{\psi}$. Notably, both the
diffusion process and adversarial alignment process operate in the latent
space of the VAE (Kingma and Welling, 2014) network. Thus, for the adversarial
alignment process, we estimate $\hat{z}_{0}$ instead of directly computing
$\hat{z}_{0}$ as this is too computationally expensive. The dotted line here
indicates that the estimated $\hat{z}_{0}$ can be decoded as a prediction of
the generated image in the pixel space. We omit the complete process of adding
noise to the image to obtain $z_{t}$ and the iterative nature of the reverse
diffusion process for simplicity. Best viewed in color.

Given $M$ lines of text prompts $S_{txt}=\\{L^{1},...,L^{M}\\}$, the visual
story generation task aims to generate $M$ images
$S_{img}=\\{I^{1},...,I^{M}\\}$ based on these prompts. Standard text-to-image
models could handle this task by generating each image in $S_{img}$ based on
each text in $S_{txt}$. However, this can overlook the contextual information
contained in other prompts as well as previously generated images. Thus, the
visual story generation method (Pan et al., 2022; Rahman et al., 2023;
Maharana et al., 2022) instead utilizes all text prompts and previously
generated images to guide the underlying diffusion model. In other words, the
$i$-th image is conditioned as $P(I^{i}|L^{1},...,L^{M},I^{1},...,I^{i-1})$.
Notably, previous works assume the model only needs to learn to generate
stories that contain a closed set of characters. This is reflected during the
dataset construction process as only these characters will be referred to by
their names consistently. For supporting characters that appear much less
frequently, they are referred to in various different ways. In this paper, we
instead assume the model will first learn to generate visual stories with a
close set of $N$ characters, i.e., $C_{ext}=\\{c_{1},c_{2},...,c_{N}\\}$. We
regard this process as pretraining as the model requires a considerable amount
of iterations to learn to generate coherent images by considering contextual
information. Subsequently, in order to generate stories with new characters,
we customize the visual story generation model to learn to generate content
related to a group of new characters $C_{new}=\\{c_{N+1},...,c_{N+K}\\}$. We
consider the customization process a few-shot finetuning process, i.e., the
model learns to generate an image containing the new character by only taking
a single story about the new characters. Notably, we use finetuning stories
that contain only the new characters themselves, which is closer to real-world
cases where the end users cannot provide versatile images of the new
characters, e.g., how they interact with other characters.

###  4.1. Visual Story Generation Preliminaries

Diffusion Models approximate a sample distribution $p(x)$ by denoising from a
base distribution in multiple steps. To learn such a model with parameter
$\theta$, we define the forward diffusion process as adding noise to an image
$x_{0}\sim p(x)$ sampled from the sample distribution $p(x)$ following a
Markov process for $T$ times. In practice, most recent works operate in the
latent space of a VAE encoder (Kingma and Welling, 2014) for various benefits
(Rombach et al., 2021). Thus, we denote the VAE-encoded $x$ as $z$, and the
forward and backward diffusion process can be described as follows:

(1)
$\begin{split}q(z_{1:T}|z_{0})&=\prod_{i=1}^{T}\mathcal{N}(z_{t};\sqrt{1-\beta_{t}}z_{t-1},\beta_{t}I)\\\
p_{\theta}(z_{0:T})&=p_{\theta}(z_{T})\prod_{i=1}^{T}p(z_{t-1}|z_{t}),\end{split}$

where $\\{\beta_{t}\in(0,1)\\}^{T}_{t=1}$ is a predefined time-dependent
variance schedule. The network is learned to estimate the noise $\epsilon$
added at each time step $t$ to reconstruct $z_{0}$ progressively and this can
be described as minimizing the L2 distance between the ground truth noise
$\epsilon$ and estimated noise $\epsilon_{\theta}$ by the denoising network:

(2)
$\mathcal{L}_{mse}=\mathbb{E}_{\epsilon,z,t}\left[\|\epsilon-{\epsilon}_{\theta}(z_{t},s,t)\|^{2}_{2}\right],$

where $\epsilon\sim\mathcal{N}(0,1)$. $s$ refers to a conditioning vector that
makes the generation process conditional, i.e., the generation process is
dependent on the input conditions, such as text, images, or information from
other modalities.

Diffusion-based Visual Story Generation Models (Rahman et al., 2023; Pan et
al., 2022) that achieved state-of-the-art performance recently are based on
the latent diffusion model described above. The main difference is that
instead of only encoding a single line of prompt $L$ as the condition $s$,
i.e.,

(3) $s=\mathcal{E}_{txt}(L),$

visual story generation models encode all text prompts in $S_{txt}$ and all
generated images ${S}_{img}^{i}=\\{I_{1},...,I_{i-1}\\}$ before the $i$-th
image as the condition. Formally, this can be described as:

(4)
$s_{i}=F\left(\mathcal{E}_{txt}(S_{txt}),\mathcal{E}_{img}(S_{img}^{i})\right),$

where $\mathcal{E}_{txt}$ and $\mathcal{E}_{img}$ are text and visual
encoders, $s_{i}$ is the conditioning vector for the $i$-th image to be
generated,$F$ is a function that fuses the encoded text and visual
information.

###  4.2. Story Character Customization

Pretraining. Existing customization methods (Ruiz et al., 2023; Kumari et al.,
2023; Gal et al., 2022; Gu et al., 2024; Po et al., 2023) aim to customize a
general text-to-image model (Rombach et al., 2021). For visual story
generation models (Rahman et al., 2023; Pan et al., 2022), the model needs to
be first trained on a story dataset, such as the FlintStonesSV (Maharana and
Bansal, 2021) dataset. Such a dataset contains a large number of stories that
have a fixed length of 5 images and 5 lines of text captions. In the context
of story character customization, we regard this process as pretraining.
Notably, previous models are trained using a dataset that contains both
$C_{ext}$ and $C_{new}$, whereas we train the underlying visual story
generation network using the refined pretraining splits of NewEpisode and
these splits are free of $C_{new}$.

Customization. Common customization methods customize a text-to-image model by
tuning the model to generate unique concepts whenever prompted by a special
word, or more precisely, a special token (Ruiz et al., 2023; Pan et al., 2022;
Gal et al., 2022). Within the context of story character customization, we
customize the visual story generation model by having the model generate a new
character whenever such a character is mentioned by the prompt. Specifically,
we refer to new characters with a short phrase of descriptive words. We found
this sufficient to prompt the model to generate stories containing these new
characters in the case of story character customization. We use such a
prompting strategy for our method, i.e., EpicEvo, in the rest of this paper.
Additionally, we also leverage a small group of stories of existing characters
as regularization terms to mitigate overfitting. The optimization goal can be
therefore formulated as minimizing the following loss:

(5)
$\mathcal{L}_{mse}=\mathbb{E}_{\epsilon,z^{\prime},t}\left[\|\epsilon-{\epsilon}_{\theta}(z^{\prime}_{t},s,t)\|^{2}_{2}\right]+\mathbb{E}_{\epsilon,z,t}\left[\|\epsilon-{\epsilon}_{\theta}(z_{t},s,t)\|_{2}^{2}\right],$

where $z^{\prime}$ corresponds to stories of existing characters and $z$
corresponds to stories of new characters.

###  4.3. Adversarial Character Alignment

A previous study (Kumari et al., 2023) alleviates the challenge of multi-
concept customization through either naive joint training or merging model
weights trained on individual concepts. Story character customization poses
more difficulties due to the similarities between new and existing characters.
In this work, we jointly train with multiple new characters. We noticed that
the customized model might be unable to capture the visual traits of the new
characters. For example, in Fig. 4-g, the model can be confused about the
appearance of the new character, leading to unsatisfactory generation results.

In view of this, inspired by the adversarial learning scheme featured by
(Sauer et al., 2023), we propose the adversarial character alignment module
designed for the diffusion-based visual story generation model. The key
objective of the adversarial finetuning process is to regulate the model such
that it generates each character distinctively and mitigates the confusion
between each existing and new character. Following the common formulation of
adversarial learning (Goodfellow et al., 2020), we regard the diffusion model
as the generator and we construct a discriminator $D_{\phi}$ with parameters
$\phi$ to discriminate between the ground truth latent $z_{0}$ and the
estimated latent $\hat{z}_{0}$. The key to enforcing better character
alignment lies in the way we construct the positive and negative samples for
the discriminator, that is:

  * •

We first select several visual references for a certain character and encode
them as $z_{r}$.

  * •

We generate the positive samples by fusing the ground truth latent $z_{0}$
with the latent $z_{r}$ of the reference character whenever a certain frame in
the story contains the reference character.

  * •

We generate the negative samples by fusing the estimated latent $\hat{z}_{0}$
and the latent $z_{r}$ of the reference character whenever a certain frame in
the story contains the reference character.

This forces the network to generate images containing the reference character.
In practice, the fusion process is implemented as concatenation. The noisy
latent $z_{t}$ is derived from a group of ground truth images $z_{0}$, i.e.,
$z_{t}=\alpha_{t}z_{0}+\sigma_{t}\epsilon$ where $\alpha_{t}=1-\beta_{t}$. The
generated data from the model could be denoted as $\hat{z}_{0}$. Therefore,
the optimization objective for the denoising network can be derived as:

(6)
$\mathcal{L}=\mathcal{L}_{mse}+\lambda\mathcal{L}_{adv}^{G}(\hat{z}_{0},z_{r},\phi),$

where $\lambda_{1}$ is the coefficient for the adversarial loss for the
diffusion network. In practice, we set the optimization goal for the
generation as minimizing the following loss:

(7)
$\mathcal{L}_{adv}^{G}=-\mathbb{E}_{\hat{z}_{0},z_{r}}\left[\log(D_{\phi}(\hat{z}_{0},z_{r}))\right]$

whereas the discriminator is trained to maximize:

(8)
$\begin{split}\mathcal{L}_{adv}^{D}=\mathbb{E}_{z_{0},z_{r}}\left[\log(D_{\phi}(z_{0},z_{r}))\right]+\mathbb{E}_{\hat{z}_{0},z_{r}}\left[\log(1-D_{\phi}(\hat{z}_{0},z_{r}))\right]\end{split}$

In practice, the discriminator $D_{\phi}$ contains 4 convolutional layers and
1 linear layer to process the positive and negative samples and it operates in
the latent space instead of the pixel space. Instead of stepping the model
from the $t$-th time step to $0$ to obtain $\hat{z}_{0}$, an alternative way
is to substitute $z_{0}$ based on (Song et al., 2020), i.e.,

(9)
$\tilde{z}_{0}=\frac{x_{t}-\sqrt{1-\overline{\alpha}_{t}}\cdot\epsilon_{t}}{\sqrt{\overline{\alpha}}_{t}},$

where $\overline{\alpha}_{t}=\prod^{T}_{i=1}\alpha_{i}$, and $\tilde{z}_{0}$
is an estimated version of $z_{0}$. Intuitively, the generator network, i.e.,
the diffusion model, generates images that contain the reference character
such that it maximizes the likelihood that these images are deemed a ‘match’
by the discriminator, while the discriminator tries to distinguish whether the
generated images contain the reference character.

###  4.4. Story Prior Preservation via Distillation

Finetuning based on a few training samples introduces risks of overfitting,
resulting in a decrease in diversity and many other undesirable outcomes such
as language drift (Ruiz et al., 2023; Kumari et al., 2023; Lee et al., 2019;
Lu et al., 2020). Within the context of visual story generation, we also
observe that the model could associate characters with certain types of
backgrounds, further decreasing the output diversity. Meanwhile, the
pretrained network possesses a higher level of output diversity despite it has
little knowledge of the new characters prior, making them sub-optimal for
directly generating these characters. Therefore, we design a distillation loss
where the pretrained story generation model with parameter $\psi$ is regarded
as a teacher and the distillation loss is formulated as:

(10)
$\mathcal{L}_{\text{distill}}=\mathbb{E}_{z_{0},s,t}\left[\|\epsilon_{\psi}(z_{t},s,t)-\epsilon_{\theta}(z_{t},s,t)\|^{2}_{2}\right]$

To sum up, the overall optimization objective for the story visualization
network can be formulated as:

(11)
$\mathcal{L}=\mathcal{L}_{mse}+\lambda_{1}\mathcal{L}_{adv}^{G}+\lambda_{2}\mathcal{L}_{distill},$

where $\lambda_{1},\lambda_{2}$ are two coefficients for the generator
adversarial loss and distillation loss.

##  5\. Experiments

In this section, we describe the details of the NewEpisode benchmark, followed
by implementation details of baselines and our method, quantitative analysis,
and qualitative results.

Dataset. We collected a dataset that contains thousands of visual stories for
a total of 15 new characters from the FlintStonesSV (Maharana and Bansal,
2021) and PororoSV (Li et al., 2019) dataset. Readers can refer to Sec. 3,
Fig. 2, and Table 1 for a detailed illustration of how we derived our dataset
from existing datasets and benchmark model customization methods.

Figure 4. Qualitative story character customization results from different
baselines. For (a) and (b), we highlight the original visual story and
highlight one of the generated frames containing the new character. For the
rest of the image, we present sampled individual frames from stories to better
demonstrate the effectiveness of EpicEvo under various conditions.

Training Details. We use the model proposed by (Pan et al., 2022) as the
backbone for visual story generation. During the pretraining stage, we keep
most of the settings the same as (Pan et al., 2022) but train the network at a
resolution of $256\times 256$ as the training samples from FlintStonesSV
(Maharana and Bansal, 2021) and PororoSV (Li et al., 2019) are $128\times
128$. During model customization, we freeze the CLIP (Radford et al., 2021)
and BLIP (Li et al., 2022) encoders and only train the denoising network. The
batchsize during customization is 2 and the learning rate is $1\times
10^{-5}$. The model is tuned for 100 epochs using the Adam (Kingma and Ba,
2015) optimizer. For sampling, we step the model for 50 steps using the DDIM
(Song et al., 2020) sampler with a guidance scale of $6.0$.
$\lambda_{1},\lambda_{2}$ is set to $0.75,0.5$ and $0.25,0.25$ for the
Flintstones and Pororo customization benchmark in NewEpisode. We consider
multiple model customization works (Ruiz et al., 2023; Kumari et al., 2023;
Gal et al., 2022) as competitive baselines. For DreamBooth (Ruiz et al.,
2023), we randomly select rare tokens and finetune the entire model following
the original paper. Since we use a small portion of training samples during
customization, we disable the prior-preservation loss of DreamBooth (Ruiz et
al., 2023). For Custom Diffusion, we employ the same group of tokens as
DreamBooth while following its experiment setup. For Textual Inversion (Gal et
al., 2022), we initialize new token embedding using the embedding mean and
variance.

Table 2. FID, CLIP-I, and CLIP-T scores on the FlintStones and Pororo split of
the NewEpisode benchmark

Methods | $\text{NewEpisode}_{FlintStones}$ | $\text{NewEpisode}_{Pororo}$
---|---|---
FID$\downarrow$ | CLIP-I $\uparrow$ | CLIP-T $\uparrow$ | FID$\downarrow$ | CLIP-I $\uparrow$ | CLIP-T $\uparrow$
Pretrain Model | 210.45 | 0.8203 | 0.2510 | 143.98 | 0.7690 | 0.2358
Textual Inversion | 207.36 | 0.8027 | 0.2399 | 150.08 | 0.7603 | 0.2335
DreamBooth | 192.95 | 0.8360 | 0.2510 | 134.94 | 0.8010 | 0.2522
Custom Diffusion | 190.97 | 0.8250 | 0.2480 | 131.88 | 0.7935 | 0.2487
EpicEvo | 188.30 | 0.8380 | 0.2573 | 130.4 | 0.7954 | 0.2551

Evaluation Metrics. To quantitatively evaluate our method and various
baselines, we employ three widely adapted metrics, including Fréchet inception
distance (FID) (Heusel et al., 2017), CLIP-I, and CLIP-T (Radford et al.,
2021; Ruiz et al., 2023; Kumari et al., 2023). CLIP-I refers to image-to-image
similarity score and CLIP-T refers to text-to-image similarity score.
Specifically, we focus on the images related to new characters (not every
image in one story contains the new characters) the FID score measures the
cluster distance between the generated images and ground truth in the latent
space of the Inception V3 model (Szegedy et al., 2016). We average across the
FID score for each character and report the average character FID. For CLIP-
based (Radford et al., 2021) metrics, CLIP-I is the average pairwise cosine
similarity between the CLIP features of generated images and ground truth
images. CLIP-T is the average pairwise cosine similarity between the CLIP
features of generated images and the text captions. When calculating the
CLIP-T score, we use descriptive phrases to refer to each new character
because the modifier tokens and inverted tokens are out-of-distribution for
the pretrained CLIP (Radford et al., 2021) model.

Quantitative Analysis. We show the FID scores in Table. 2. Notably, the scale
of the FID score highly relies on the sample size, with a smaller sample size,
it is expected to have larger FID scores. Nevertheless, the reduction in FID
scores on both NewEpisodeFlintstones and NewEpisodePororo indicate that our
story character customization method, i.e., EpicEvo, could on average achieve
better customization results compared to the baselines. In terms of the CLIP-I
and CLIP-T scores, we found our model achieves a higher CLIP-I score on the
NewEpisodeFlintstones and our method reaches the second place on the test set
of NewEpisodePororo. Still, for the CLIP-T score, our model obtains better
similarities compared to the baselines. This indicates the generated stories
of new characters are more semantically aligned with the text prompts. The
results of Textual Inversion (Gal et al., 2022) and the pretrained model
validate that the model has never seen these new characters, making it hard
for them to generate stories related to these characters.

Table 3. Ablation studies for the distillation process and adversarial
character alignment process

Methods | $\text{NewEpisode}_{FlintStones}$ | $\text{NewEpisode}_{Pororo}$
---|---|---
FID$\downarrow$ | CLIP-I $\uparrow$ | CLIP-T $\uparrow$ | FID$\downarrow$ | CLIP-I $\uparrow$ | CLIP-T $\uparrow$
Pretrained Model | 210.45 | 0.8203 | 0.2510 | 143.98 | 0.7690 | 0.2358
$\mathcal{L}_{mse}$ | 192.76 | 0.8360 | 0.2542 | 132.51 | 0.7940 | 0.2546
$\mathcal{L}_{mse}+\mathcal{L}_{distill}$ | 192.04 | 0.8345 | 0.2550 | 130.73 | 0.7915 | 0.2537
$\mathcal{L}_{mse}+\mathcal{L}_{distill}+\mathcal{L}_{adv}^{G}$ | 188.30 | 0.8380 | 0.2573 | 130.40 | 0.7954 | 0.2551

Ablation Study. We ablate our method, EpicEvo, by removing three losses, i.e.,
the reconstruction loss $\mathcal{L}_{mse}$, the adversarial character
alignment loss $\mathcal{L}_{adv}$, and the distillation loss
$\mathcal{L}_{distill}$. Starting from a pretrained model trained on the
pretraining split of NewEpisode, we validate the model performance using three
of our testing metrics. By finetuning the model using a few examples of each
new character, we notice a significant drop in terms of the FID score, meaning
the model is learning to generate content more relevant to the new characters.
Next, enabling the distillation loss decreases the FID score but increases the
CLIP-I and CLIP-T scores. We empirically found the reason might be the
pretrained model has no prior of the new character. Therefore, learning from
the pretrained model has the risk of misguiding the customized model despite
it could better prevent overfitting. We recommend using a smaller
$\lambda_{2}$ such that the distillation loss could encourage the model to
generate diverse contents without disturbing learning of the visual features
of new characters. Lastly, we enable the character alignment loss we designed
for the diffusion-based visual story generation model, i.e.,
$\mathcal{L}_{adv}^{G}$. We observe a further reduction in the FID score and
an increase in the CLIP-I and CLIP-T scores, validating that the proposed
adversarial character alignment method could encourage the model to learn to
generate new characters more consistently.

Qualitative Analysis. While our method reached better customization
performances on both customization testing datasets of the NewEpisode, it is
still an open debate regarding to what extent FID score (Heusel et al., 2017),
CLIP-I and CLIP-T (Radford et al., 2021), etc. can represent human
perceptions. . Thus, to empirically validate the performance of EpicEvo, we
display the generated images of new characters in Fig. 4, and we show stories
and images relevant to the new characters in Fig. 4. Overall, Fig. 4
demonstrates that our method can generate the new characters alone, e.g., Fig.
4-a,b,c,d,e or with other characters, e.g., Fig. 4-f,g,h,i,j more
consistently. We also find Dreambooth (Ruiz et al., 2023) achieves decent
results, specifically on NewEpisodePororo. We hypothesized that the Custom
Diffusion (Kumari et al., 2023) is limited because simply tuning the cross-
attention layer might not be sufficient to learn the novel appearance of new
characters, this also aligns with our preliminary studies that show LoRA (Hu
et al., 2022) offers unsatisfactory results due to its limited ability to
learn representations. For Textual Inversion (Gal et al., 2022), users rarely
pick images generated by it, and this is expected as the model will struggle
to invert concepts it has never seen before, especially under the condition
that we carefully removed all these new characters from the pertaining data.

##  6\. Conclusion

In this paper, we tackle the challenging problem of story character
customization. We aim to customize a visual story generation model so that it
can generate stories for new characters it has never seen before. We first
propose the NewEpisode benchmark. NewEpisode leverages supporting characters
in previous story generation datasets as new characters. It contains carefully
refined pretraining data to train visual story generation models and plenty of
data for training and testing story character customization methods. We
identify the core challenge is that visual story generation contains complex
priors such as character dynamics, making customizing more challenging. In
view of this, we propose EpicEvo, our method to tackle story character
customization. EpicEvo takes a few images of the new character and customizes
the model to generate stories for the new character. It contains an
interesting adversarial character alignment module and it utilizes knowledge
distillation to prevent overfitting. Compared to previous methods, we
quantitatively and qualitatively validate that EpicEvo can generate visual
stories that have better new character consistency. This indicates that
EpicEvo can better tackle the problem of story character customization, making
downstream tasks such as the creation of serialized cartoons, and TV series,
possible.

## References

  * (1)
  * Brock et al. (2019) Andrew Brock, Jeff Donahue, and Karen Simonyan. 2019.  Large Scale GAN Training for High Fidelity Natural Image Synthesis. In _7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019_. OpenReview.net, New Orleans, US.  https://openreview.net/forum?id=B1xsqj09Fm
  * Dhariwal and Nichol (2021) Prafulla Dhariwal and Alexander Quinn Nichol. 2021.  Diffusion Models Beat GANs on Image Synthesis. In _Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual_ , Marc’Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan (Eds.). 8780–8794.  https://proceedings.neurips.cc/paper/2021/hash/49ad23d1ec9fa4bd8d77d02681df5cfa-Abstract.html
  * Gal et al. (2022) Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H. Bermano, Gal Chechik, and Daniel Cohen-Or. 2022.  An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion.  https://doi.org/10.48550/ARXIV.2208.01618
  * Goodfellow et al. (2020) Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron C. Courville, and Yoshua Bengio. 2020.  Generative adversarial networks.  _Commun. ACM_ 63, 11 (2020), 139–144.  https://doi.org/10.1145/3422622
  * Gu et al. (2024) Yuchao Gu, Xintao Wang, Jay Zhangjie Wu, Yujun Shi, Yunpeng Chen, Zihan Fan, Wuyou Xiao, Rui Zhao, Shuning Chang, Weijia Wu, et al. 2024\.  Mix-of-show: Decentralized low-rank adaptation for multi-concept customization of diffusion models.  _Advances in Neural Information Processing Systems_ 36 (2024).
  * Gupta et al. (2018) Tanmay Gupta, Dustin Schwenk, Ali Farhadi, Derek Hoiem, and Aniruddha Kembhavi. 2018.  Imagine This! Scripts to Compositions to Videos. In _Computer Vision - ECCV 2018 - 15th European Conference, Munich, Germany, September 8-14, 2018, Proceedings, Part VIII_ _(Lecture Notes in Computer Science, Vol. 11212)_ , Vittorio Ferrari, Martial Hebert, Cristian Sminchisescu, and Yair Weiss (Eds.). Springer, 610–626.  https://doi.org/10.1007/978-3-030-01237-3_37
  * Heusel et al. (2017) Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. 2017.  GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium. In _Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA_ , Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett (Eds.). 6626–6637.  https://proceedings.neurips.cc/paper/2017/hash/8a1d694707eb0fefe65871369074926d-Abstract.html
  * Ho et al. (2020) Jonathan Ho, Ajay Jain, and Pieter Abbeel. 2020.  Denoising diffusion probabilistic models.  _Advances in neural information processing systems_ 33 (2020), 6840–6851.
  * Hu et al. (2022) Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022.  LoRA: Low-Rank Adaptation of Large Language Models. In _The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022_. OpenReview.net.  https://openreview.net/forum?id=nZeVKeeFYf9
  * Karras et al. (2021) Tero Karras, Miika Aittala, Samuli Laine, Erik Härkönen, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. 2021.  Alias-free generative adversarial networks.  _Advances in neural information processing systems_ 34 (2021), 852–863.
  * Karras et al. (2019) Tero Karras, Samuli Laine, and Timo Aila. 2019.  A Style-Based Generator Architecture for Generative Adversarial Networks. In _IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019_. Computer Vision Foundation / IEEE, 4401–4410.  https://doi.org/10.1109/CVPR.2019.00453
  * Kim et al. (2017) Kyung-Min Kim, Min-Oh Heo, Seong-Ho Choi, and Byoung-Tak Zhang. 2017.  DeepStory: Video Story QA by Deep Embedded Memory Networks. In _Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence, IJCAI 2017, Melbourne, Australia, August 19-25, 2017_ , Carles Sierra (Ed.). ijcai.org, 2016–2022.  https://doi.org/10.24963/IJCAI.2017/280
  * Kingma and Ba (2015) Diederik P. Kingma and Jimmy Ba. 2015.  Adam: A Method for Stochastic Optimization. In _3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings_ , Yoshua Bengio and Yann LeCun (Eds.).  http://arxiv.org/abs/1412.6980
  * Kingma and Welling (2014) Diederik P. Kingma and Max Welling. 2014.  Auto-Encoding Variational Bayes. In _2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings_ , Yoshua Bengio and Yann LeCun (Eds.).  http://arxiv.org/abs/1312.6114
  * Kumari et al. (2023) Nupur Kumari, Bingliang Zhang, Richard Zhang, Eli Shechtman, and Jun-Yan Zhu. 2023.  Multi-Concept Customization of Text-to-Image Diffusion. In _IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023, Vancouver, BC, Canada, June 17-24, 2023_. IEEE, 1931–1941.  https://doi.org/10.1109/CVPR52729.2023.00192
  * Lee et al. (2019) Jason Lee, Kyunghyun Cho, and Douwe Kiela. 2019.  Countering Language Drift via Visual Grounding. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019_ , Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan (Eds.). Association for Computational Linguistics, 4384–4394.  https://doi.org/10.18653/V1/D19-1447
  * Li (2022) Bowen Li. 2022.  Word-Level Fine-Grained Story Visualization. In _Computer Vision - ECCV 2022 - 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXXVI_ _(Lecture Notes in Computer Science, Vol. 13696)_ , Shai Avidan, Gabriel J. Brostow, Moustapha Cissé, Giovanni Maria Farinella, and Tal Hassner (Eds.). Springer, 347–362.  https://doi.org/10.1007/978-3-031-20059-5_20
  * Li et al. (2022) Junnan Li, Dongxu Li, Caiming Xiong, and Steven C. H. Hoi. 2022.  BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation. In _International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA_ _(Proceedings of Machine Learning Research, Vol. 162)_ , Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato (Eds.). PMLR, 12888–12900.  https://proceedings.mlr.press/v162/li22n.html
  * Li et al. (2019) Yitong Li, Zhe Gan, Yelong Shen, Jingjing Liu, Yu Cheng, Yuexin Wu, Lawrence Carin, David E. Carlson, and Jianfeng Gao. 2019.  StoryGAN: A Sequential Conditional GAN for Story Visualization. In _IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019_. Computer Vision Foundation / IEEE, 6329–6338.  https://doi.org/10.1109/CVPR.2019.00649
  * Liu et al. (2024) Chang Liu, Haoning Wu, Yujie Zhong, Xiaoyun Zhang, Yanfeng Wang, and Weidi Xie. 2024.  Intelligent Grimm – Open-ended Visual Storytelling via Latent Diffusion Models. In _The IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * Liu et al. (2023) Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023.  Visual Instruction Tuning.
  * Lu et al. (2020) Yuchen Lu, Soumye Singhal, Florian Strub, Aaron C. Courville, and Olivier Pietquin. 2020.  Countering Language Drift with Seeded Iterated Learning. In _Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event_ _(Proceedings of Machine Learning Research, Vol. 119)_. PMLR, 6437–6447.  http://proceedings.mlr.press/v119/lu20c.html
  * Maharana and Bansal (2021) Adyasha Maharana and Mohit Bansal. 2021.  Integrating Visuospatial, Linguistic, and Commonsense Structure into Story Visualization. In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2021_ , Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (Eds.). Association for Computational Linguistics, 6772–6786.  https://doi.org/10.18653/V1/2021.EMNLP-MAIN.543
  * Maharana et al. (2021) Adyasha Maharana, Darryl Hannan, and Mohit Bansal. 2021.  Improving generation and evaluation of visual stories via semantic consistency.  _arXiv preprint arXiv:2105.10026_ (2021).
  * Maharana et al. (2022) Adyasha Maharana, Darryl Hannan, and Mohit Bansal. 2022.  StoryDALL-E: Adapting Pretrained Text-to-Image Transformers for Story Continuation. In _Computer Vision - ECCV 2022 - 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXXVII_ _(Lecture Notes in Computer Science, Vol. 13697)_ , Shai Avidan, Gabriel J. Brostow, Moustapha Cissé, Giovanni Maria Farinella, and Tal Hassner (Eds.). Springer, 70–87.  https://doi.org/10.1007/978-3-031-19836-6_5
  * Nichol et al. (2022) Alexander Quinn Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. 2022.  GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models. In _International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA_ _(Proceedings of Machine Learning Research, Vol. 162)_ , Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato (Eds.). PMLR, 16784–16804.  https://proceedings.mlr.press/v162/nichol22a.html
  * Pan et al. (2022) Xichen Pan, Pengda Qin, Yuhong Li, Hui Xue, and Wenhu Chen. 2022.  Synthesizing Coherent Story with Auto-Regressive Latent Diffusion Models.  _CoRR_ abs/2211.10950 (2022).  https://doi.org/10.48550/ARXIV.2211.10950 arXiv:2211.10950
  * Po et al. (2023) Ryan Po, Guandao Yang, Kfir Aberman, and Gordon Wetzstein. 2023.  Orthogonal adaptation for modular customization of diffusion models.  _arXiv preprint arXiv:2312.02432_ (2023).
  * Radford et al. (2021) Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021.  Learning Transferable Visual Models From Natural Language Supervision. In _Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event_ _(Proceedings of Machine Learning Research, Vol. 139)_ , Marina Meila and Tong Zhang (Eds.). PMLR, 8748–8763.  http://proceedings.mlr.press/v139/radford21a.html
  * Rahman et al. (2023) Tanzila Rahman, Hsin-Ying Lee, Jian Ren, Sergey Tulyakov, Shweta Mahajan, and Leonid Sigal. 2023.  Make-A-Story: Visual Memory Conditioned Consistent Story Generation. In _IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023, Vancouver, BC, Canada, June 17-24, 2023_. IEEE, 2493–2502.  https://doi.org/10.1109/CVPR52729.2023.00246
  * Ramesh et al. (2022) Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. 2022.  Hierarchical Text-Conditional Image Generation with CLIP Latents.  _CoRR_ abs/2204.06125 (2022).  https://doi.org/10.48550/ARXIV.2204.06125 arXiv:2204.06125
  * Razavi et al. (2019) Ali Razavi, Aäron van den Oord, and Oriol Vinyals. 2019.  Generating Diverse High-Fidelity Images with VQ-VAE-2. In _Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada_ , Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d’Alché-Buc, Emily B. Fox, and Roman Garnett (Eds.). 14837–14847.  https://proceedings.neurips.cc/paper/2019/hash/5f8e2fa1718d1bbcadf1cd9c7a54fb8c-Abstract.html
  * Rombach et al. (2021) Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. 2021.  High-Resolution Image Synthesis with Latent Diffusion Models.  arXiv:2112.10752 [cs.CV]
  * Ronneberger et al. (2015) Olaf Ronneberger, Philipp Fischer, and Thomas Brox. 2015.  U-Net: Convolutional Networks for Biomedical Image Segmentation. In _Medical Image Computing and Computer-Assisted Intervention - MICCAI 2015 - 18th International Conference Munich, Germany, October 5 - 9, 2015, Proceedings, Part III_ _(Lecture Notes in Computer Science, Vol. 9351)_ , Nassir Navab, Joachim Hornegger, William M. Wells III, and Alejandro F. Frangi (Eds.). Springer, 234–241.  https://doi.org/10.1007/978-3-319-24574-4_28
  * Ruiz et al. (2023) Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. 2023.  DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation. In _IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023, Vancouver, BC, Canada, June 17-24, 2023_. IEEE, 22500–22510.  https://doi.org/10.1109/CVPR52729.2023.02155
  * Saharia et al. (2022) Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L. Denton, Seyed Kamyar Seyed Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, Jonathan Ho, David J. Fleet, and Mohammad Norouzi. 2022.  Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding. In _Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 - December 9, 2022_ , Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh (Eds.).  http://papers.nips.cc/paper_files/paper/2022/hash/ec795aeadae0b7d230fa35cbaf04c041-Abstract-Conference.html
  * Sauer et al. (2023) Axel Sauer, Dominik Lorenz, Andreas Blattmann, and Robin Rombach. 2023.  Adversarial Diffusion Distillation.  _CoRR_ abs/2311.17042 (2023).  https://doi.org/10.48550/ARXIV.2311.17042 arXiv:2311.17042
  * Song et al. (2020) Jiaming Song, Chenlin Meng, and Stefano Ermon. 2020.  Denoising Diffusion Implicit Models.  _arXiv:2010.02502_ (October 2020).  https://arxiv.org/abs/2010.02502
  * Szegedy et al. (2016) Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jonathon Shlens, and Zbigniew Wojna. 2016.  Rethinking the Inception Architecture for Computer Vision. In _2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016_. IEEE Computer Society, 2818–2826.  https://doi.org/10.1109/CVPR.2016.308
  * Zhang et al. (2017) Han Zhang, Tao Xu, and Hongsheng Li. 2017.  StackGAN: Text to Photo-Realistic Image Synthesis with Stacked Generative Adversarial Networks. In _IEEE International Conference on Computer Vision, ICCV 2017, Venice, Italy, October 22-29, 2017_. IEEE Computer Society, 5908–5916.  https://doi.org/10.1109/ICCV.2017.629