(eccv) Package eccv Warning: Package ‘hyperref’ is loaded with option
‘pagebackref’, which is *not* recommended for camera-ready version

11institutetext: Shenzhen Campus of Sun Yat-sen University 22institutetext:
The Chinese University of Hong Kong
33institutetext: Lenovo Research
Corresponding Author<EMAIL_ADDRESS>
https://howe140.github.io/theatergen.io/

# TheaterGen : Character Management with
LLM for Consistent Multi-turn Image Generation

Junhao Cheng 11 Baiqiao Yin 11 Kaixin Cai 11 Minbin Huang 22 Hanhui Li 11
Yuxin He 11 Xi Lu 11 Yue Li 11 Yifei Li 11 Yuhao Cheng 33 Yiqiang Yan 33
Xiaodan Liang 11**

###### Abstract

Recent advances in diffusion models can generate high-quality and stunning
images from text. However, multi-turn image generation, which is of high
demand in real-world scenarios, still faces challenges in maintaining semantic
consistency between images and texts, as well as contextual consistency of the
same subject across multiple interactive turns. To address this issue, we
introduce TheaterGen, a training-free framework that integrates large language
models (LLMs) and text-to-image (T2I) models to provide the capability of
multi-turn image generation. Within this framework, LLMs, acting as a
"Screenwriter", engage in multi-turn interaction, generating and managing a
standardized prompt book that encompasses prompts and layout designs for each
character in the target image. Based on character prompts and layouts, we
generate a list of character images and extract guidance information from
them, akin to the "Rehearsal". Subsequently, through incorporating the prompt
book and guidance information into the reverse denoising process of T2I
diffusion models, we generate the final image, as conducting the "Final
Performance". With the effective management of prompt books and character
images, TheaterGen significantly improves semantic and contextual consistency
in synthesized images. Furthermore, we introduce a dedicated benchmark,
CMIGBench (Consistent Multi-turn Image Generation Benchmark) with 8000 multi-
turn instructions. Different from previous multi-turn benchmarks, CMIGBench
does not define characters in advance, and hence it is of great diversity.
Both the tasks of story generation and multi-turn editing are included on
CMIGBench for comprehensive evaluation. Extensive experimental results show
that TheaterGen outperforms state-of-the-art methods significantly. For
example, it raises the performance bar of the cutting-edge Mini DALL·E 3 model
by 21% in average character-character similarity and 19% in average text-image
similarity. Our code and CMIGBench can be found in supplementary materials.
Our codes and benchmark will be available at
https://github.com/donahowe/Theatergen.

###### Keywords:

Diffusion models Consistency Multi-turn image generation

##  1 Introduction

Figure 1: Visual comparison between Mini DALL·E 3 [46] and our proposed
TheaterGen in multi-turn story generation and multi-turn editing.

In recent years, significant progress has been observed in text-to-image (T2I)
generation tasks, attributed to the advancements in diffusion models [36].
State-of-the-art T2I models [7, 25] are capable of synthesizing diverse and
detailed images that are truly astonishing. However, there is a growing need
for multi-turn image generation. As shown in Figure 1, this encompasses the
demand to produce coherent narrative images, such as sequential storylines,
across multi-turn dialogue. Additionally, there may exist a requirement for
iterative image refinement, necessitating multi-turn editing feedback to
achieve satisfactory results. These demands present substantial challenges to
text comprehension capability and the maintenance of consistency in multi-turn
image generation within T2I models.

To address this challenge, existing work attempts to utilize large language
models (LLM) [42, 9, 5] to extract pertinent information from multi-turn
dialogues, thereby guiding diffusion models. Mini DALLE·3 [46] references to
the recent T2I model DALLE 3 [30], and employs LLM to generate image
descriptions or image embeddings during multi-turn dialogues, subsequently
supplying these to the T2I model for image generation. MiniGPT-5 [49] equips
LLMs with the current story texts and corresponding images to generate the
subsequent image embedding of the story. SEED-LLaMA [8] integrates LLM with
the SEED image tokenizer, enabling it to process and generate text and images
interchangeably, leading to enhanced multimodal comprehension and generation
capabilities.

Despite these advancements in handling complex texts within multi-turn
dialogues, as shown in Figure 1, they still grapple with two primary issues:
(1) Semantic Consistency \- existing methods encounter difficulties in
processing complex descriptions such as spatial relations, quantities, or
referential expressions such as "it, they" during multi-turn dialogues,
resulting in the generation of images that are semantically inconsistent with
user requests. (2) Contextual Consistency \- during image generation within
multi-turn dialogues, the same entity often struggles to maintain consistent
features across different turns, and may even be forgotten. For example, the
same dog looks different in different turns without user edits.

To tackle the above issues, we propose TheaterGen in this paper. In the
context of multi-turn image generation, the consideration of information from
preceding turns is important, which is an ability that current T2I models
lack. Inspired by [26], we develop an LLM-based character designer that
leverages an LLM to establish and manage a standardized prompt book comprising
prompts and bounding boxes for each foreground character. This method serves
to enhance comprehension and arrangement of user intents within a multi-turn
interactive environment, effectively disentangling the intricate generation
task into amalgamations of multiple characters. This process results in the
production of more manageable text for multi-turn generation tasks, thereby
promoting the achievement of semantic consistency.

To effectively leverage the prompt book, TheaterGen generates and maintains a
reference image for each character. To ensure contextual consistency, upon the
introduction of a new character, we generate an image using the character
description with an off-the-shelf T2I model and preserve it as the reference
of the character. The reference image is considered as the contextual prior,
and will be amalgamated with the character prompt of the ongoing turn to
derive the corresponding on-stage character image. With on-stage character
images of multiple characters, TheaterGen further extracts lineart and latent
guidance along with the bounding boxes from the prompt book to generate the
final image with high consistency. In this way, even without extra training on
a specific dataset, TheaterGen can align existing LLMs with T2I models to
achieve consistent multi-turn image generation.

To evaluate both the semantic and contextual consistency in multi-turn image
generation, we propose a new benchmark called CMIGBench. Our benchmark focuses
on two common types found in multi-turn image generation: story generation and
multi-turn editing. It consists of 8000 multi-turn scripted dialogues, with
each dialogue containing 4 turns of natural language instructions. As in real-
world scenarios, generation models might not know specific characters before
the conversation starts, CMIGBench does not pre-define the characters. This
makes our benchmark differ from previous benchmarks significantly and avoids
the risk of loss in diversity. We compare TheaterGen with several cutting-edge
models on CMIGBench, and the qualitative and quantitative results show that
our method outperforms existing methods by large margins.

To summarize, our main contributions are as follows:

  1. 1.

We propose TheaterGen, which is a training-free framework that utilizes a
large language model to drive a text-to-image generation model, effectively
addressing the issues of semantic consistency and contextual consistency in
multi-turn image generation tasks without specialized training.

  2. 2.

TheaterGen can engage in multi-turn natural language interactions with users
to accomplish tasks such as story generation and multi-turn editing.

  3. 3.

We propose a new benchmark, CMIGBench, to evaluate both the semantic and
contextual consistency in multi-turn image generation and demonstrate the
superior performance of TheaterGen.

##  2 Related Works

###  2.1 Text to Image Generation

Text-to-image generation is currently a widely researched and applied field.
Mainstream image generation methods include Variational AutoEncoder [39],
flow-based models [28], Generative Adversarial Networks (GANs) [10, 16, 17,
18], and diffusion models [13, 36, 32]. Recently, the stunning results of
generating images from text descriptions using diffusion models have led to
the emergence of many excellent diffusion-based image generation models [45,
47, 35, 1]. These models encode text descriptions through multimodal models
such as CLIP [34, 15, 23, 22] and incorporate them into the image generation
process through cross-modal attention mechanisms. Our TheaterGen primarily
explores the potential for multi-turn image generation from text-to-image
models.

###  2.2 Multi-turn Image Generation with LLMs

Text-to-image (T2I) generation models traditionally require precise and highly
specific textual prompts, posing a challenge for natural language
compatibility. Recent research efforts have focused on bridging this gap by
integrating large language models (LLMs) with T2I generation models to enhance
interaction through natural language to achieve multi-turn image generation[8,
49, 3, 26, 42, 9, 21, 14]. Mini DALL·E 3 extends the multi-turn interactive
capabilities of its predecessors by categorizing user interaction into
generation and editing modes. Diffusion GPT [33] introduces a "Tree-of-Thought
of Models "[44] for model selection, guiding LLMs to parse user inputs and
match the most suitable T2I model. Next GPT [43] achieves multimodal alignment
by encoding inputs across modalities uniformly, adjusting and decoding through
LLMs. MiniGPT-5 [49] uses LLM to understand images and text, inferring the
next steps in the story and generating visual features to guide the generation
of the next story image. In our work, TheaterGen uses LLM to maintain the
layout and foreground character prompts in multi-turn interactions, thus
achieving high-quality generation.

###  2.3 Compositional Image Generation Task for Multi-turn Image Editing

There exists another type of compositional image generation task [40, 4, 6,
29] aimed at achieving more consistent completion of image editing tasks based
on a given series of instructions without LLM. Among these efforts, [40]
involves fine-tuning the editing instructions by incorporating attention
mechanisms and introducing loss guidance. GeNeVA-GAN [4] uses a GRU [2] to
combine the previous condition and the current text embedding to produce the
current context, guiding the generator to generate the current image from the
previous round’s image embedding. Finally, adversarial training with images
from two rounds completes the construction of the iterative editor. SSCR [6]
extends the GeNeVA-GAN [4] iterative editor by adding a GRU-based iterative
interpreter, which generates semantic differences in text across different
rounds as guidance. LatteGAN [29] utilizes attention for text guidance,
introducing a visually guided language attention module to ensure images
better align with editing instructions.

##  3 Method

Figure 2: The overall structure of TheaterGen. TheaterGen utilizes three key
components to generate an image in each interaction turn: (a) an LLM-based
character designer that interacts with the user and maintains a structured
prompt book for all character prompts and layouts, which serves as the
“screenwriter”; (b) a character image manager for “rehearsal”, which generates
character images and extracts guidance based on the prompt book; (c) a
character-guided generator that conducts the “final performance”, i.e.,
generates the final image for the current turn by combining the prompt book
and guidance information.

Figure 2 shows the overall framework of our proposed TheaterGen for multi-turn
interactive image generation. We first introduce an LLM-based character
designer (Section 3.1) to interact with the user and generate a prompt book
consisting of character prompts and layouts. After that, a character image
manager (Section 3.2) is employed to generate on-stage character images and
their corresponding lineart and latent guidance. At last, these two types of
guidance and the prompt book are fed into a character-guided generator to
synthesize the final image (Section 3.3). In this way, our training-free
framework can leverage the advantages of existing LLMs and T2I models in
natural interaction and diverse image generation, while eliminating the
computationally expensive training process.

###  3.1 LLM-based Character Designer

Although existing T2I models demonstrate the capacity to produce high-quality
images, they require appropriate textural descriptions and focus on generating
single-character images. Hence, we first introduce the LLM-based character
designer, to generate the structured character-oriented prompt book to encode
user intentions and descriptions of multiple characters. The details of the
user-inputted prompt and the prompt book can be seen in the Appendix.

Formally, the proposed prompt book $P$ can be formulated as,

$P=\\{p_{bg},~{}p_{neg},~{}(ID_{i},p_{i},b_{i})_{i=1}^{k}\\},$ (1)

where $p_{bg}$ is the background prompt, $p_{neg}$ is the negative prompt, and
$k$ is the number of characters existing in the current interaction turn. The
triple $(ID_{i},p_{i},b_{i})$ represents the unique ID, the prompt, and the
layout (bounding box) of the $i$-th character, respectively. With the unique
ID assigned by the LLM, we can distinguish and track different characters in
multiple turns effectively. This also allows us to edit images of target
characters, even if they exhibit significant changes in appearance. Moreover,
we can exploit spatial relationship priors from the LLM through the character
layout, which helps to arrange multiple characters in the synthesized image
reasonably and improve semantic consistency.

###  3.2 Character Image Manager

The prompt book serves to reify multi-turn dialogues with users into a more
sustainable format. However, employing the prompt book in T2I models directly
still poses challenges in attaining semantic and contextual coherence. Thus,
we devised the character image manager, which (i) generates on-stage character
images according to the prompt book, and (ii) exploits guidance to ensure
semantic and contextual consistency in generated images.

On-stage character image generation. For each character, we consider two types
of images, namely, a reference image that is used to maintain contextual
consistency throughout the entire interaction process, and an on-stage image
that represents the character in the current stage. Specifically, for an
arbitrary character with $ID_{i}$, let $I_{ref}^{(ID_{i})}$ and
$I_{on}^{(ID_{i})}$ denote its reference image and on-stage image,
respectively. We generate $I_{on}^{(ID_{i})}$ with an off-the-self diffusion-
based T2I model $\mathcal{M}$ as follows:

$\displaystyle
I_{on}^{(ID_{i})}=\begin{cases}\mathcal{M}(p_{i},~{}\mathcal{A}(I_{ref}^{(ID_{i})})),&\text{if
}I_{ref}^{(ID_{i})}\in D,\\\ \\\ \mathcal{M}(p_{i}),&\text{if
}I_{ref}^{(ID_{i})}\notin D,\end{cases}$ (2)

where $D$ is a list maintaining the reference images of all characters, i.e.,

$D=\\{I_{ref}^{(ID_{1})},...,I_{ref}^{(ID_{k})}\\}.$ (3)

$\mathcal{A}$ is an adapter that injects the reference image into
$\mathcal{M}$. With $\mathcal{A}$, we encourage the generated image to exhibit
similar characteristics to the reference image and achieve contextual
consistency. We adopt a pre-trained IP-Adapter [45] to implement
$\mathcal{A}$. For a character that appears for the first time, we generate
both its reference image and on-stage image via $\mathcal{M}$ directly.

Guidance extraction. To consolidate all on-stage images into one that is
coherent with the layouts provided by the prompt book and suitable for the
subsequent generation, we propose to extract two types of guidance, as shown
in Figure 3. To begin with, we construct a segment processor comprising an
open-vocabulary detection model [27] and a segmentation model [19] to detect
characters and remove backgrounds from the on-stage images. We then resize all
on-stage images and merge them into the same one according to the character
layouts, yielding a middle-state image $I_{mid}$ with a blank background.

Figure 3: The proposed guidance extractor. It first extracts subjects from
character images and rearranges them into the same image according to the
layout. Then the lineart guidance and the latent guidance for subsequent image
generation are obtained via a lineart processer and the forward diffusion
process, respectively.

We consider extracting the latent guidance and the lineart guidance from
$I_{mid}$. The former type of guidance is constructed by applying the forward
diffusion process of $\mathcal{M}$ on $I_{mid}$, so that we can project
$I_{mid}$ into the latent space of $\mathcal{M}$, complete and revise it via
the reverse diffusion process. Particularly, the latent guidance $G_{latent}$
is generated as follows,

$G_{latent}=\\{g_{l,t}|t=0,...,T-1\\},~{}g_{l,t}=\mathcal{FD}(I_{mid},t),$ (4)

where $\mathcal{FD}$ is the forward diffusion process of $T$ steps and
$g_{l,t}$ represents the latent guidance at the $t$-th step. In this way, all
generated images are from the same latent space. Hence, with $G_{latent}$, we
can maintain the characteristics of on-stage images and improve contextual
consistency.

In addition, to attain more precise manipulation of character layout and image
details, a lineart processor [47] is employed to extract lineart guidance
$G_{lineart}$. This method, as opposed to the direct utilization of bounding
boxes like in conventional methods [12, 48], provides stronger constraints on
character positions. Moreover, this facilitates the preservation of finer
edges and structural details, thereby contributing to the production of high-
quality and consistent images.

###  3.3 Character-Guided Generator

Given the prompt book $P$ and two types of guidance, our devised character-
guided generator is responsible for synthesizing the final image for each
turn. Specifically, we first concatenate the character prompts and the
background prompt in $P$ to obtain a global prompt $p_{global}$. $p_{global}$
contains the semantic information of the entire image, which we inject into
the cross-attention modules of $\mathcal{M}$ along with the guidance
information to control image generation. The negative prompt is utilized
following the common practice of diffusion models [36], and here we focus on
exploiting lineart guidance and latent guidance.

Lineart guidance. In each U-Net block of $\mathcal{M}$, we follow [47] to
encode the lineart guidance $G_{lineart}$ and inject it into the denoising
process. As mentioned above, this strengthens the layout constraint and
incorporates more details into synthesized characters.

Latent guidance. While the global prompt $p_{global}$ and the lineart guidance
$G_{lineart}$ are important for establishing semantic consistency, they are
insufficient in guaranteeing contextual consistency. Therefore, we utilize the
latent guidance $G_{latent}$ to enhance the consistency of a character across
multiple turns. To this end, let $z_{t}$ represent the denoised latent code at
the $t$-th time step, we incorporate $G_{latent}$ into $z_{t}$ as follows,

$\displaystyle\hat{z}_{t}=\begin{cases}z_{t}\odot(1-M)+g_{l,t}\odot
M,&\text{if }t\geq rT,\\\ z_{t},&\text{if }t<rT,\\\ \end{cases}$ (5)
$M=m^{(ID_{1})}\oplus m^{(ID_{2})},...,\oplus m^{(ID_{k})},$ (6)

where $\odot$ denotes the element-wise product and $\oplus$ denotes the OR
operator. $m^{(ID_{i})}\in\\{0,1\\}^{H\times W}$ denotes the $H\times W$-sized
segmentation mask of the $i$-th character obtained by the segment processor.
$r$ is the radio to control the number of steps that are required to input
latent guidance. We retain a portion of the denoising steps without
replacement by $r$, so that we can better balance the integration between the
characters and the background in the final image. The augmented $\hat{z}_{t}$
will be processed as the input of the next denoising step. With $G_{latent}$,
reference images and on-stage images can be merged effectively to guide the
generation of the final image.

##  4 Experiments

###  4.1 Setup

####  4.1.1 CMIGBench

As shown in Table 1, existing benchmarks have certain limitations for multi-
turn T2I generation. Primarily, they fail to comprehensively consider story
generation and multi-turn editing, focusing instead on specific types of
multi-turn generation. Secondly, some benchmarks disregard contextual
consistency, while others fail to involve multi-turn instructions. In
addition, most multi-turn benchmarks pre-define the images of characters,
requiring models to undergo fine-tuning to generate specific forms of
characters to achieve contextual consistency, which is hard to meet the
demands of real-world applications.

Therefore, we propose a consistent multi-turn image generation benchmark
(CMIGBench). This benchmark focuses on two common types found in multi-turn
image generation: story generation and multi-turn editing, comprising 8000
multi-turn scripted dialogues (4000 for each task) with each consisting of 4
turns of natural language instructions. This allows us to evaluate the
semantic consistency and contextual consistency of different models in multi-
turn image generation in a zero-shot manner.

CMIGBench is constructed automatically through GPT-4 [31] and then manually
curated. To ensure the diversity of dialogues, we first randomly select the
characters and a background scene for each dialogue from a pre-defined pool.
Then, GPT-4 is prompted to generate image-generation instructions for 4 turns
and a ground-truth prompt book. We encourage GPT-4 to use unambiguous
references to make the instructions more human-like. In the multi-turn editing
task, to avoid ambiguity, we have specifically fixed the editing type for each
turn, namely, editing spatial locations, attributes, negative prompts, and
numbers of characters. This facilitates the generation of high-quality
dialogues and enables the evaluation of alignment for each specific edit
instruction.

Table 1: Comparative analysis of benchmarks on diversity and functionality
metrics. "Char. Predefined" means characters are pre-defined.

Benchmarks | Vocabulary Diversity | Test Prompts | Char. Pre-defined | Multi-turn | Story | Edit
---|---|---|---|---|---|---
LMD[25] | 10 objects, 11 colors, 4 spatial | 400 | ✗ | ✗ | ✗ | ✓
ConsiStory[38] | 31 nouns | 500 | ✗ | ✗ | ✓ | ✗
CLEVR-SV[24] | 8 colors, 3 shapes, 2 sizes | 12000 | ✓ | ✓ | ✗ | ✓
CoDraw[41] | 58 nouns | 4250 | ✓ | ✓ | ✗ | ✓
PororoSV[24] | 13 characters | 11680 | ✓ | ✓ | ✓ | ✗
FlintstonesSV[11] | 3897 characters | 2518 | ✓ | ✓ | ✓ | ✗
CMIGBench | 157 nouns, 16 colors, 6 spatial | 32000 | ✗ | ✓ | ✓ | ✓

####  4.1.2 Evaluation metrics

To evaluate the contextual consistency, in the absence of pre-defined
character images, we utilize character reference images for evaluation. For
every dialogue, we use Grounding DINO to identify character regions in the
final image of each turn. Each character’s initial appearance is set as its
reference. For detected character regions and their corresponding reference
image, the assessment is conducted by calculating: (1) the average character-
to-character similarity (aCCS) through the CLIP image-image similarity method,
(2) the average Frechet Inception Distance (aFID).

Moreover, for evaluating semantic consistency, we leverage CLIP to determine
the average text-image similarity (aTIS) for the final image of each turn and
the associated global prompt. Additionally, to measure editing alignment
accuracy in the context of editing tasks, we follow [26] to compute metrics
w.r.t. our defined 4 editing types ("spatial", "attribute", "negative", and
"numeracy").

####  4.1.3 Inference Details

For LLM Character Designer, we use GPT-4 for prompt book generation while also
evaluating the ability of other LLMs (_i.e._ GPT-3.5 and Gemini Pro [37]) to
generate prompt books, which determines the upper limit of our model’s
generation quality. For image generation, Stable Diffusion version 1.5 and
version XL are selected as the basic generation models for TheaterGen. For the
Segment Processor, we utilize the GroundingDINO-T with a box-threshold setting
of 0.5 and a text-threshold setting of 0.25 and SAM [20] to predict mask. The
$r$ to control the injection of latent guidance is set to 0.1.

Table 2: Model performances on contextual and semantic consistency metrics.

Diffsion version | Model | Metrics
---|---|---
Contexual consistency | Sementic consistency
aFID$\downarrow$ | aCCS(%)$\uparrow$ | Human | aTIS(%)$\uparrow$ | Human
Story | Editing | Story | Editing | Story | Editing | Story | Editing | Story | Editing
SD1.5 | Mini DALLE·3 | 451.59 | 443.71 | 54.11 | 52.51 | 1.72 | 1.86 | 29.81 | 28.13 | 2.48 | 2.12
MiniGPT-5 | 528.3 | 480.7 | 43.09 | 44.08 | 1.24 | 1.08 | 24.93 | 22.95 | 1.94 | 1.41
SEED-LLaMA | 316 | 357.55 | 64.78 | 59.83 | 1.52 | 1.42 | 26.41 | 25.18 | 2.11 | 2.26
Ours | 252.31 | 240.32 | 78 | 84.31 | 2.64 | 3.63 | 31.52 | 29.67 | 3.06 | 3.55
SDXL | Mini DALLE·3 | 286.21 | 402.21 | 67.59 | 54.4 | 2.72 | 1.69 | 32.77 | 29.86 | 3.22 | 2.47
Ours | 209.45 | 222.56 | 81.05 | 93.52 | 3.18 | 4.18 | 38.91 | 37.72 | 3.24 | 4.11

Table 3: Model performances on alignment accuracy metrics.

Diffsion version | Model | Alignment Accuracy $\uparrow$
---|---|---
Spatial(%) | Attribute(%) | Negative(%) | Numeracy(%) | Human
SD1.5 | Mini DALLE·3 | 38.32 | 64.72 | 85.04 | 7.83 | 1.87
MiniGPT-5 | 34.17 | 49.6 | 87.68 | 3.65 | 1.12
SEED-LLaMA | 44.99 | 71.92 | 72.05 | 9.51 | 1.35
Ours | 87.27 | 88.21 | 89.31 | 47.74 | 3.24
SDXL | Mini DALLE·3 | 40.8 | 67.6 | 87.33 | 5.83 | 1.71
Ours | 87.33 | 90.66 | 94.92 | 70.67 | 3.41

###  4.2 Comparison with SOTA methods

We do a comprehensive evaluation of TheaterGen with the chosen baseline models
on CMIGBench.

As shown in Table 2 and Table 3, our approach surpasses other comparative
models across all metrics in both the story generation task and the multi-turn
image editing task. Specifically, our model showcases a substantial
enhancement of 27% and 21% respectively in the aFID and aCCS metrics about
semantic consistency, when contrasted with the previous SOTA model Mini
DALLE·3 with diffusion version XL. Additionally, in terms of contextual
consistency, our model demonstrates an improvement of 19% in the aTIS metrics
compared to the previous SOTA model. These results substantiate the notion
that our model exhibits a heightened capacity to generate images on multi-turn
dialog that upholds both semantic and contextual consistency.

For the alignment accuracy, Table 3 demonstrates that our model not only
performs well in maintaining dialogue consistency but also outperforms other
baselines in terms of completion for each editing type. A significant
improvement in our model’s performance is observed, particularly in cases
involving spatial relationships and quantities. This is mainly attributed to
the layout capability of the LLM.

###  4.3 Human evaluation

Figure 4: Ablation study on the effects of lineart and latent guidance.

To demonstrate the effectiveness of our evaluation process and metrics, as
well as to provide an intuitive comparison of model performance, we conduct a
user study. We randomly select 50 dialogues (25 for each task) from CMIGBench,
resulting in 200 images generated per model. 10 volunteers are invited to rate
the semantic consistency and contextual consistency of these images.

From the human evaluation results in Table 2 and Table 3, we can observe that
human ratings of our method are higher than those of other methods. This
further validates the effectiveness of our method.

Besides, we compute Kendall’s correlation coefficient $\tau$, Spearman’s
correlation coefficient $\rho$, and Pearson correlation coefficient $\sigma$
between our evaluation metrics and the results from human evaluation. From
Table 4, it can be observed that all the metrics demonstrate strong
correlations, indicating that the proposed metrics are minimally affected by
the errors introduced by the detection model and can effectively reflect the
quality of multi-turn image generation.

Table 4: Correlation between machine metrics and human evaluations. The
correlation of aFID is expected to be close to -1, while the correlation of
the other metrics is expected to be close to 1.

Metrics | aFID | aCCS | aTIS | Spatial | Attribute | Negative | Numeracy
---|---|---|---|---|---|---|---
$\tau$ | -0.349 | 0.287 | 0.439 | 0.633 | 0.604 | 0.600 | 0.687
$\rho$ | -0.528 | 0.381 | 0.611 | 0.678 | 0.629 | 0.610 | 0.733
$\sigma$ | -0.401 | 0.426 | 0.641 | 0.699 | 0.662 | 0.904 | 0.731

Table 5: Ablation results of two guidance approaches with consistency metrics.

Task | Lineart | Latent | aFID$\downarrow$ | aCCS(%)$\uparrow$ | aTIS(%)$\uparrow$
---|---|---|---|---|---
Story | ✓ | ✓ | 252.31 | 78 | 31.52
✓ |  | 281.17 | 76.96 | 31.33
|  | ✓ | 306.24 | 68.45 | 31.14
Editing | ✓ | ✓ | 240.32 | 84.31 | 29.67
✓ |  | 288.25 | 77.66 | 28.46
|  | ✓ | 300.54 | 78.07 | 28.85

Table 6: Ablation results of two guidance approaches with alignment accuracy
metrics.

Lineart | Latent | Spatial(%) | Attribute(%) | Negative(%) | Numeracy(%)
---|---|---|---|---|---
✓ | ✓ | 87.27 | 88.21 | 89.31 | 47.74
✓ |  | 81.48 | 88.76 | 79.78 | 42.32
| ✓ | 58.36 | 67.91 | 75.52 | 14.93

###  4.4 Ablation experiment

####  4.4.1 Latent guidance

As shown in Table 4.3 and Table 4.3, we conducted ablation experiments on
latent guidance, revealing its importance for multi-turn image generation. In
the comparison between the second and first columns of Figure 4, we can more
intuitively see the role of latent guidance. It can be seen that without
latent guidance, which means that we do not introduce the image features of
the reference images into the diffusion model, the character in the generated
images will be inconsistent with those in the reference images, causing the
model to fail to maintain contextual consistency across multi-turn
generations.

####  4.4.2 Lineart guidance

As shown in Table 4.3 and Table 4.3, we also conducted ablation study on the
role of lineart guidance. It can find that after removing $G_{lineart}$, all
metrics of the model will experience a significant decrease, verifying the
effectiveness of $G_{lineart}$. In Figure 4, we can observe that in the
absence of lineart guidance, the model struggles to handle details such as the
positions and quantities of characters. Additionally, the model fails to
effectively process latent guidance, leading to conflicts between characters
and background. This indicates that our two types of guidance complement each
other.

Figure 5: Visualization results of story generation. Table 7: Ablation results
of different LLMs with consistency metrics.

LLM | Task | $ACC_{id}(\%)$ | $ACC_{box}(\%)$
---|---|---|---
GPT-3.5 | Story | 96.21 | 82.68
Gemini-pro | 95.33 | 95.10
GPT-4 | 99.73 | 92.48
GPT-3.5 | Editing | 100 | 96.97
Gemini-pro | 99.5 | 96.97
GPT-4 | 100 | 99.63

Table 8: Ablation results of different LLMs with alignment accuracy metrics.

LLM | Spatial(%) | Attribute(%) | Negative(%) | Numeracy(%)
---|---|---|---|---
GPT-3.5 | 90 | 84.85 | 95.96 | 96.97
Gemini-pro | 91.92 | 96.97 | 96.97 | 100
GPT-4 | 100 | 100 | 100 | 100

####  4.4.3 LLM Character Designer

We also evaluate the recent LLM models’ ability to generate proper prompt
books. As shown in Table 4.4.2 and Table 4.4.2, their generation capabilities
for editing instructions are slightly different, yet the overall quality is
satisfying. Among them, GPT-4 exhibits the best average performance. We
evaluated the accuracy of LLM in maintaining the corresponding character ID as
$ACC_{id}$ and the accuracy of generating bounding boxes that contain the
required main character as $ACC_{box}$. The results in Table 3 of the prompt
book show that our selected LLMs assign proper character IDs and generate
layouts that are consistent with the captions in most cases.

###  4.5 Visualization

Figure 6: Visualization results of multi-turn editing.

We visualize the generation results of the chosen models. In Figure 5 and
Figure 6, we observe that the images generated by TheaterGen perform well in
both semantic and contextual consistency. This suggest TheaterGen can handle
complex referential issues and maintain discernible features for the main
characters. More results can be seen in Appendix.

##  5 Discussion and Conclusion

Conclusion. In this paper, we propose TheaterGen, a training-free framework
integrating LLMs with T2I models for multi-turn image generation. By
leveraging a large language model as a "Screenwriter", TheaterGenmanages a
prompt book with character prompts and layouts. It then introduces a character
image manager, which customizes the image generation for each character and
arranges the "Rehearsal" for the entire stage scene, generating latent
guidance and lineart guidance. To generate the “Final Performance" image,
TheaterGenutilizes the prompt book and guidance information to incorporate
customized content effectively, and eventually addresses the challenges of
maintaining semantic and contextual consistency in multi-turn image
generation.

Limitations. Despite customizing the generation for each character, the
randomness in existing pre-trained image adapters might result in contextual
inconsistency. Additionally, limited by the generation capability, T2I models
might cause semantic inconsistency. In the future, more power T2I models will
be incorporated into TheaterGen to address the above issues.

## References

  * [1] Betker, J., Goh, G., Jing, L., Brooks, T., Wang, J., Li, L., Ouyang, L., Zhuang, J., Lee, J., Guo, Y., et al.: Improving image generation with better captions. Computer Science. https://cdn. openai. com/papers/dall-e-3. pdf 2(3), 8 (2023)
  * [2] Cho, K., Van Merriënboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., Bengio, Y.: Learning phrase representations using rnn encoder-decoder for statistical machine translation. arXiv preprint arXiv:1406.1078 (2014)
  * [3] Dong, R., Han, C., Peng, Y., Qi, Z., Ge, Z., Yang, J., Zhao, L., Sun, J., Zhou, H., Wei, H., et al.: Dreamllm: Synergistic multimodal comprehension and creation. arXiv preprint arXiv:2309.11499 (2023)
  * [4] El-Nouby, A., Sharma, S., Schulz, H., Hjelm, D., Asri, L.E., Kahou, S.E., Bengio, Y., Taylor, G.W.: Tell, draw, and repeat: Generating and modifying images based on continual linguistic instruction. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 10304–10312 (2019)
  * [5] Feng, Y., Gong, B., Chen, D., Shen, Y., Liu, Y., Zhou, J.: Ranni: Taming text-to-image diffusion for accurate instruction following. arXiv preprint arXiv:2311.17002 (2023)
  * [6] Fu, T.J., Wang, X.E., Grafton, S., Eckstein, M., Wang, W.Y.: Sscr: Iterative language-based image editing via self-supervised counterfactual reasoning. arXiv preprint arXiv:2009.09566 (2020)
  * [7] Gafni, O., Polyak, A., Ashual, O., Sheynin, S., Parikh, D., Taigman, Y.: Make-a-scene: Scene-based text-to-image generation with human priors. In: European Conference on Computer Vision (2022)
  * [8] Ge, Y., Zhao, S., Zeng, Z., Ge, Y., Li, C., Wang, X., Shan, Y.: Making llama see and draw with seed tokenizer. arXiv preprint arXiv:2310.01218 (2023)
  * [9] Gong, Y., Pang, Y., Cun, X., Xia, M., Chen, H., Wang, L., Zhang, Y., Wang, X., Shan, Y., Yang, Y.: Talecrafter: Interactive story visualization with multiple characters. arXiv preprint arXiv:2305.18247 (2023)
  * [10] Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., Bengio, Y.: Generative adversarial networks. Communications of the ACM pp. 139–144 (2020)
  * [11] Gupta, T., Schwenk, D., Farhadi, A., Hoiem, D., Kembhavi, A.: Imagine this! scripts to compositions to videos. In: Proceedings of the European conference on computer vision (ECCV). pp. 598–613 (2018)
  * [12] He, S., Liao, W., Yang, M.Y., Yang, Y., Song, Y.Z., Rosenhahn, B., Xiang, T.: Context-aware layout to image generation with enhanced object appearance. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 15049–15058 (2021)
  * [13] Ho, J., Jain, A., Abbeel, P.: Denoising diffusion probabilistic models. Advances in neural information processing systems 33, 6840–6851 (2020)
  * [14] Huang, M., Long, Y., Deng, X., Chu, R., Xiong, J., Liang, X., Cheng, H., Lu, Q., Liu, W.: Dialoggen: Multi-modal interactive dialogue system for multi-turn text-to-image generation. arXiv preprint arXiv:2403.08857 (2024)
  * [15] Jia, C., Yang, Y., Xia, Y., Chen, Y.T., Parekh, Z., Pham, H., Le, Q., Sung, Y.H., Li, Z., Duerig, T.: Scaling up visual and vision-language representation learning with noisy text supervision. In: International conference on machine learning. pp. 4904–4916 (2021)
  * [16] Kang, M., Zhu, J.Y., Zhang, R., Park, J., Shechtman, E., Paris, S., Park, T.: Scaling up gans for text-to-image synthesis. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 10124–10134 (2023)
  * [17] Karras, T., Laine, S., Aila, T.: A style-based generator architecture for generative adversarial networks. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 4401–4410 (2019)
  * [18] Karras, T., Laine, S., Aittala, M., Hellsten, J., Lehtinen, J., Aila, T.: Analyzing and improving the image quality of stylegan. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 8110–8119 (2020)
  * [19] Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitehead, S., Berg, A.C., Lo, W.Y., et al.: Segment anything. arXiv preprint arXiv:2304.02643 (2023)
  * [20] Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitehead, S., Berg, A.C., Lo, W.Y., et al.: Segment anything. arXiv preprint arXiv:2304.02643 (2023)
  * [21] Koh, J.Y., Fried, D., Salakhutdinov, R.R.: Generating images with multimodal language models. Advances in Neural Information Processing Systems 36 (2024)
  * [22] Li, J., Li, D., Xiong, C., Hoi, S.: Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In: International Conference on Machine Learning. pp. 12888–12900. PMLR (2022)
  * [23] Li, J., Selvaraju, R., Gotmare, A., Joty, S., Xiong, C., Hoi, S.C.H.: Align before fuse: Vision and language representation learning with momentum distillation. Advances in neural information processing systems 34, 9694–9705 (2021)
  * [24] Li, Y., Gan, Z., Shen, Y., Liu, J., Cheng, Y., Wu, Y., Carin, L., Carlson, D., Gao, J.: Storygan: A sequential conditional gan for story visualization. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 6329–6338 (2019)
  * [25] Lian, L., Li, B., Yala, A., Darrell, T.: Llm-grounded diffusion: Enhancing prompt understanding of text-to-image diffusion models with large language models. arXiv preprint arXiv:2305.13655 (2023)
  * [26] Lian, L., Li, B., Yala, A., Darrell, T.: Llm-grounded diffusion: Enhancing prompt understanding of text-to-image diffusion models with large language models. arXiv preprint arXiv:2305.13655 (2023)
  * [27] Liu, S., Zeng, Z., Ren, T., Li, F., Zhang, H., Yang, J., Li, C., Yang, J., Su, H., Zhu, J., et al.: Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv preprint arXiv:2303.05499 (2023)
  * [28] Lu, Y., Huang, B.: Structured output learning with conditional generative flows. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 34, pp. 5005–5012 (2020)
  * [29] Matsumori, S., Abe, Y., Shingyouchi, K., Sugiura, K., Imai, M.: Lattegan: Visually guided language attention for multi-turn text-conditioned image manipulation. IEEE Access 9, 160521–160532 (2021)
  * [30] OpenAI: Dall·e 3 system card (2023)
  * [31] OpenAI: Gpt-4 technical report (2023)
  * [32] Podell, D., English, Z., Lacey, K., Blattmann, A., Dockhorn, T., Müller, J., Penna, J., Rombach, R.: Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.01952 (2023)
  * [33] Qin, J., Wu, J., Chen, W., Ren, Y., Li, H., Wu, H., Xiao, X., Wang, R., Wen, S.: Diffusiongpt: Llm-driven text-to-image generation system. arXiv preprint arXiv:2401.10061 (2024)
  * [34] Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al.: Learning transferable visual models from natural language supervision. In: International conference on machine learning. pp. 8748–8763 (2021)
  * [35] Ramesh, A., Dhariwal, P., Nichol, A., Chu, C., Chen, M.: Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125 (2022)
  * [36] Rombach, R., Blattmann, A., Lorenz, D., Esser, P., Ommer, B.: High-resolution image synthesis with latent diffusion models. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 10684–10695 (2022)
  * [37] Team, G., Anil, R., Borgeaud, S., Wu, Y., Alayrac, J.B., Yu, J., Soricut, R., Schalkwyk, J., Dai, A.M., Hauth, A., et al.: Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805 (2023)
  * [38] Tewel, Y., Kaduri, O., Gal, R., Kasten, Y., Wolf, L., Chechik, G., Atzmon, Y.: Training-free consistent text-to-image generation. arXiv preprint arXiv:2402.03286 (2024)
  * [39] Tibebu, H., Malik, A., De Silva, V.: Text to image synthesis using stacked conditional variational autoencoders and conditional generative adversarial networks. In: Science and Information Conference. pp. 560–580 (2022)
  * [40] Wang, C.: A diffusion-based method for multi-turn compositional image generation. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. pp. 381–391 (2024)
  * [41] Wang, C.: A diffusion-based method for multi-turn compositional image generation. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) Workshops. pp. 381–391 (January 2024)
  * [42] Wang, W., Zhao, C., Chen, H., Chen, Z., Zheng, K., Shen, C.: Autostory: Generating diverse storytelling images with minimal human effort. arXiv preprint arXiv:2311.11243 (2023)
  * [43] Wu, S., Fei, H., Qu, L., Ji, W., Chua, T.S.: Next-gpt: Any-to-any multimodal llm. arXiv preprint arXiv:2309.05519 (2023)
  * [44] Yao, S., Yu, D., Zhao, J., Shafran, I., Griffiths, T., Cao, Y., Narasimhan, K.: Tree of thoughts: Deliberate problem solving with large language models. Advances in Neural Information Processing Systems 36 (2024)
  * [45] Ye, H., Zhang, J., Liu, S., Han, X., Yang, W.: Ip-adapter: Text compatible image prompt adapter for text-to-image diffusion models. arXiv preprint arXiv:2308.06721 (2023)
  * [46] Zeqiang, L., Xizhou, Z., Jifeng, D., Yu, Q., Wenhai, W.: Mini-dalle3: Interactive text to image by prompting large language models. arXiv preprint arXiv:2310.07653 (2023)
  * [47] Zhang, L., Rao, A., Agrawala, M.: Adding conditional control to text-to-image diffusion models. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 3836–3847 (2023)
  * [48] Zheng, G., Zhou, X., Li, X., Qi, Z., Shan, Y., Li, X.: Layoutdiffusion: Controllable diffusion model for layout-to-image generation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 22490–22499 (2023)
  * [49] Zheng, K., He, X., Wang, X.E.: Minigpt-5: Interleaved vision-and-language generation via generative vokens. arXiv preprint arXiv:2310.02239 (2023)

Appendix Materials of TheaterGen

##  Appendix 0.A Prompt book Generation Details

We utilize an LLM to translate human instructions into a prompt book. Our
approach incorporates the user instruction into a template and requests the
LLM to generate the rest of the content, yielding the following prompt:

<Task Description>: Your task is to generate the bounding boxes for the
objects mentioned in the caption and number them, along with a background
prompt describing the scene. The number indicates the number of mentioned
objects. <Supporting details>: The images are of size 512 x 512. The top-left
corner has coordinates [0, 0]. The bottom-right corner has coordinates [512,
512]. Pay attention to ensure the generated layout size is appropriate and not
too small. The bounding boxes should not overlap or go beyond the image
boundaries. Each bounding box should be in the format of ( an object with an
article a or an and a modifier, [ top - left x coordinate, top - left y
coordinate, box width, box height ], object number) and the content in ” must
be the singular form of a or an plus an adjective plus a countable noun and
should not include more than one object. You should completely avoid
situations where the bounding boxes of objects overlap, so you can make the
bounding boxes between different objects have an appropriate distance and need
design reasonable x, y, box width, and box height. Do not put objects that are
already provided in the bounding boxes into the background prompt. When
describing the same picture, different objects should not have the same
numbers. If you think the description refers to the same objects in the
previous conservation, its number should be consistent with the previous
conservation, such as a green apple and a red apple. Do not include non-
existing or excluded objects in the background prompt. Use "a realistic scene"
as the background prompt if no background is given in the prompt. If needed,
you can make reasonable guesses. <Examples>: several examples of multi-turn
layout generation. <History dialogue>: The history dialogue <Current
instruction>: Current prompt by human.

Once the above prompt is obtained, the LLM is expected to generate the prompt
book in the following format:

<Characters>: [(an object with an article a or an and a modifier, [ top-left x
coordinate , top-left y coordinate , box width, box height ], object number)]
<Background prompt>: The instruction of background. <Negative prompt>: The
instruction of which object is not supposed to appear.

##  Appendix 0.B Model Deployment Details

###  0.B.1 Models Input

As our selected baselines have different designs, their inputs vary slightly.
In our model, we utilize the prompt book as input directly. For MiniGPT-5 and
SEED-LLaMA, we construct a combined input that consists of the dialogue
history, the current instruction, and their output w.r.t the current
instruction. As for Mini DALLE·3, it employs an LLM to transform the
instruction into commands for image generation. In this case, we uniformly use
the global prompt $p_{global}$ in our prompt book and incorporate the
"</edit>" tag provided by Mini DALLE·3 as its input. This enables Mini DALLE·3
to consider the historical results during the current generation turn.

###  0.B.2 Inference Time

In our experiments, the diffusion time steps for SD1.5 and SDXL are set to 50
and 30. When applying SD1.5, the baselines take about 10 to 15 seconds to
complete a dialogue (4 turns), while our TheaterGen requires about 35 to 40
seconds. This is reasonable, as we generate on-stage character images in each
turn. And such a strategy is worthwhile, as our experimental results validate
that it improves image quality and consistency significantly. Moreover, with
more powerful diffusion models, the gap in computational costs tends to be
marginal. For example, with SDXL, Mini DALLE·3 takes approximately 36 seconds
for a dialogue, while while TheaterGen requires around 86 seconds.

##  Appendix 0.C Dataset Construction Details

###  0.C.1 Character Pools

We create separate pools from 50 common fruits, 100 objects, 7 humans, and 50
scenes, respectively, to ensure the diversity of characters. Our selected
characters are listed as follows:

fruit pool = [‘apple’, ‘banana’, ‘orange’, ‘pear’, ‘grape’, ‘strawberry’,
‘watermelon’, ‘cantaloupe’, ‘cherry’, ‘peach’, ‘plum’, ‘kiwi’, ‘pineapple’,
‘mango’, ‘lemon’, ‘pomegranate’, ‘lychee’, ‘papaya’, ‘blueberry’, ‘raspberry’,
‘blackberry’, ‘apricot’, ‘nectarine’, ‘grapefruit’, ‘lime’, ‘tangerine’,
‘fig’, ‘guava’, ‘persimmon’, ‘dragon fruit’, ‘passion fruit’, ‘star fruit’,
‘jackfruit’, ‘durian’, ‘avocado’, ‘cranberry’, ‘mulberry’, ‘quince’,
‘kumquat’, ‘date’, ‘coconut’, ‘olive’, ‘honeydew’, ‘currant’, ‘gooseberry’,
‘boysenberry’, ‘elderberry’, ‘pummelo’, ‘cloudberry’, ‘jujube’] object pool =
[‘cup’, ‘book’, ‘pen’, ‘car’, ‘glasses’, ‘chair’, ‘phone’, ‘laptop’, ‘key’,
‘wallet’, ‘watch’, ‘shoe’, ‘sock’, ‘hat’, ‘shirt’, ‘pants’, ‘dress’, ‘coat’,
‘pillow’, ‘towel’, ‘soap’, ‘toothbrush’, ‘broom’, ‘camera’, ‘clock’, ‘lamp’,
‘scissors’, ‘notebook’, ‘backpack’, ‘umbrella’, ‘bowl’, ‘plate’, ‘fork’,
‘knife’, ‘spoon’, ‘toothpaste’, ‘brush’, ‘balloon’, ‘battery’, ‘blanket’,
‘headphones’, ‘paper’, ‘envelope’, ‘stamp’, ‘card’, ‘toy’, ‘plant’, ‘vase’,
‘necklace’, ‘belt’, ‘bicycle’, ‘mirror’, ‘comb’, ‘bed’, ‘cushion’, ‘diary’,
‘teapot’, ‘kettle’, ‘mug’, ‘earphones’, ‘charger’, ‘remote’, ‘skateboard’,
‘surfboard’, ‘ball’, ‘glove’, ‘sunglasses’, ‘tie’, ‘scarf’, ‘calendar’,
‘flashlight’, ‘tent’, ‘grill’, ‘hammock’, ‘bucket’, ‘teacup’, ‘saucer’,
‘colander’, ‘pitcher’, ‘whisk’, ‘spatula’, ‘ladle’, ‘cutting board’,
‘measuring cup’, ‘cookie sheet’, ‘blender’, ‘mixer’, ‘oven mitt’, ‘apron’,
‘trash can’, ‘recycling bin’, ‘suitcase’, ‘map’,‘postcard’, ‘album’, ‘puzzle’,
mosquito, tissue, coin, eraser] animal pool = [‘dog’, ‘cat’, ‘rabbit’,
‘horse’, ‘cow’, ‘sheep’, ‘pig’, ‘goat’, ‘chicken’, ‘duck’, ‘turkey’, ‘deer’,
‘elephant’, ‘tiger’, ‘lion’, ‘bear’, ‘monkey’, ‘kangaroo’, ‘panda’, ‘zebra’,
‘penguin’, ‘frog’, ‘toad’, ‘lizard’, ‘snake’, ‘turtle’, ‘crocodile’, ‘parrot’,
‘sparrow’, ‘eagle’, ‘bat’, ‘snail’, ‘crab’, ‘lobster’, ‘starfish’] human pool
= [‘baby’, ‘boy’, ‘girl’, ‘man’, ‘woman’, ‘old man’, ‘old woman’] background
pool = [‘park’, ‘beach’, ‘zoo’, ‘forest’, ‘school’, ‘playground’, ‘city
street’, ‘suburban neighborhood’, ‘mountain range’, ‘countryside’, ‘farm’,
‘garden’, ‘riverbank’, ‘desert’, ‘ocean’, ‘island’, ‘downtown skyline’,
‘harbor’, ‘lake’, ‘meadow’, ‘campsite’, ‘hiking trail’, ‘waterfall’, ‘bridge’,
‘marketplace’, ‘amusement park’, ‘aquarium’, ‘library’, ‘museum’, ‘art
gallery’, ‘cafe’, ‘restaurant’, ‘bakery’, ‘bookstore’, ‘gym’, ‘stadium’,
‘arena’, ‘theater’, ‘cinema’, ‘concert hall’, ‘classroom’, ‘office’,
‘warehouse’, ‘factory’, ‘hospital’, ‘clinic’, ‘police station’, ‘fire
station’, ‘train station’, ‘airport’]

###  0.C.2 Prompt for Story Generation

After randomly selecting characters from the pools, we use the following
format of prompts as input to GPT-4 for generating story generation dialogues
simulating human conversations, along with the ground truth prompt book.

<characters string>: Randomly selected characters from human and animal pool
<scene string>: Randomly selected background from background pool <prompt>:
"Please give me a story with " + characters string + " as protagonists. The
setting background is " + scene string + ". Each story contains four simple
and human natural language sentences. Each sentence contains less than 4
objects. All objects are countable nouns and are singular, with no
postattributive. If the protagonist has been mentioned in the previous text,
subsequent sentences should use pronouns to refer to the protagonist. The
protagonist must be something with a precise number of nouns. You should
generate the bounding boxes for the objects mentioned in the caption and
number them, along with a background prompt describing the scene. The number
indicates the number of mentioned objects. The images are of size 512 x 512.
The top-left corner has coordinates [0, 0]. The bottom-right corner has
coordinates [512, 512]. Pay attention to ensure the generated layout size is
appropriate and not too small. The bounding boxes should not overlap or go
beyond the image boundaries. Each bounding box should be in the format of ( an
object with an article a or an and a modifier, [ top - left x coordinate, top
- left y coordinate, box width, box height ], object number) and the content
in ” must be the singular form of a or an plus an adjective plus a countable
noun and should not include more than one object. You should completely avoid
situations where the bounding boxes of objects overlap, so you can make the
bounding boxes between different objects have an appropriate distance and need
design reasonable x, y, box width, and box height. Do not put objects that are
already provided in the bounding boxes into the background prompt. When
describing the same picture, different objects should not have the same
numbers. If you think the description refers to the same objects in the
previous conservation, its number should be consistent with the previous
conservation. Do not include non-existing or excluded objects in the
background prompt. Use ’a realistic scene’ as the background prompt if no
background is given in the prompt. If needed, you can make reasonable guesses.
Please refer to the example below for the desired format. The following is an
example of a story and its layout, and you needn’t mimic its sentence
structure, you should focus on the format of the following example:…… Now
generate the story, the box for each character should be as large as
possible."

###  0.C.3 Prompt for Multi-turn Editing

We use the following input to obtain the dataset for multi-turn editing
similar to story generation.

<characters string>: Randomly selected characters from object and fruit pools
<number pick>: Randomly selected from ([‘two’, ‘three’, ‘four’, ‘five’])
<relation pick>: Randomly selected from ([‘to the left of’, ‘to the right of’,
‘to the top of’, ‘to the down of’]) <prompt>: "You need to simulate a series
of multi-round editing instructions. You will receive an object list:[]. The
[Spacial Round] should focus on spatial relations. The [Attribute round]
should focus on attributes. The [Negative Round] should focus on negatives.
The [Numeracy Round] should focus on numeracy. If the object has been
mentioned in the last text, subsequent sentences should use pronouns to refer
to the protagonist. You should generate the bounding boxes for the objects
mentioned in the caption and give them an id, along with a background prompt
(empty). The images are of size 512 x 512. The top-left corner has coordinates
[0, 0]. The bottom-right corner has coordinates [512, 512]. The bounding boxes
should not overlap or go beyond the image boundaries. Each bounding box should
be in the format of ( an object with an article and a modifier, [ top - left x
coordinate, top - left y coordinate, box width, box height ], object number)
and the content in ” must be the singular form of a or an plus an adjective
plus a countable noun and should not include more than one object. Do not put
objects that are already provided in the bounding boxes ito the background
prompt. If you think the description refers to the same objects in the
previous conservation, its id should be consistent with the previous
conservation(even changed attribute or numeracy). Do not include non-existing
or excluded objects in the background prompt. The dialogue must include four
types of edit rounds, and [indicate the type] should be used at the beginning.
Here is an example output…… You must use quantities at [Numeracy Round] as " +
number pick + " and positional relationships in [Spacial Round] as " +
relation pick + ". Now, generate the following object list:" + characters
string.

###  0.C.4 Automated & Human Corrections

We further validate and rectify the dialogues generated by GPT-4 to yield the
final CMIGBench.

First, we streamline the output format by utilizing regular expressions to
identify and rectify format discrepancies across various scenarios, enabling
us to reconstruct dialogues that are imperfect in form (but are correct in
content). This includes differentiating between text and lists, addressing
punctuation inconsistencies across languages, and resolving unenclosed
brackets.

Secondly, we detect characters that are mistakenly placed in the background
and have GPT-4 regenerate them.

Next, we devise a dispersion strategy to refine the overlap of character
layouts within individual turns. For layouts where overlap exceeds a
predefined threshold $T_{o}$, we adjust their positions as follows: We first
calculate the collective bounding rectangle of the overlapping layout
elements. We then disperse the elements from the center of the rectangle with
a random distance. This dispersion step is iterated multiple times and if the
overlap is still larger than $T_{o}$ after that, we utilize GPT-4 to
regenerate the layout. We set $T_{o}$ as $25\%$ in this paper.

Last but not least, we implement a toolkit for manual visualization alongside
drag-and-drop functionalities for layout adjustments. This allows for the
hands-on verification and refinement of dialogues and eventually ensures a
higher level of accuracy and presentation quality of our dataset.

###  0.C.5 Benchmark Examples

Here we provide the first dialogues of our story generation and multi-turn
editing task to show the description format of our benchmark:

Story Generation "dialogue 1": {
"characters": ["sparrow", "lion", "eagle"],
"scene": ["library"],
"turn 1": {
"caption": "In the silent library, a tiny sparrow was fluttering near a
shelf.",
"objects": [
["a tiny sparrow", [115, 170, 89, 59], 1],
["a library shelf", [215, 165, 171, 171], 2]]
"background": "A silent library",
"negative": "None"
},
"turn 2": {
"caption": "An attentive lion in one corner was carefully observing the bird
and holding its breath.",
"objects": [
["an attentive lion", [300, 221, 162, 180], 3],
["a tiny sparrow", [40, 101, 89, 59], 1]]
"background": "A silent library",
"negative": "None"
},

Story Generation (Continued) "turn 3": {
"caption": "Above them, a vigilant eagle watched the suspenseful scene unfold
from the library ceiling.",
"objects": [
["a vigilant eagle", [345, 41, 119, 72], 4],
["an observing lion", [295, 281, 162, 180], 3],
["a sparrow", [45, 171, 89, 59], 1]]
"background": "A silent library",
"negative": "None"
},
"turn 4": {
"caption": "The scenario ended peacefully as the eagle, the lion, and the
sparrow all resumed their own activities in the vast library.",
"objects": [
["an occupying eagle", [335, 41, 119, 72], 4],
["a peaceful lion", [285, 281, 162, 180], 3],
["a sparrow", [55, 181, 89, 59], 1]]
"background": "A vast library",
"negative": "None"
}
}

Multi-Turn Editing "dialogue 1": {
"characters": ["spatula", "pen"],
"scene": "empty background",
"turn 1": {
"caption": "I want a pen down of a spatula",
"objects": [
["a pen", [97, 235, 162, 222], 1],
["a spatula", [217, 55, 198, 232], 2]]
"background": "empty background",
"negative": "None"
},

Multi-Turn Editing (Continued) "turn 2": {
"caption": "Turn the pen into a blue one",
"objects": [
["a blue pen", [97, 235, 162, 222], 1],
["a spatula", [217, 55, 198, 232], 2]]
"background": "empty background",
"negative": "None"
},
"turn 3": {
"caption": "I don’t want this anymore",
"objects": [
["a spatula", [157, 140, 198, 232], 2]]
"background": "empty background",
"negative": "a blue pen"
},
"turn 4": {
"caption": "I want four of the remaining object.",
"objects": [
["a spatula", [4, 20, 198, 232], 2],
["a spatula", [219, 20, 198, 232], 2],
["a spatula", [85, 260, 198, 232], 2],
["a spatula", [310, 260, 198, 232], 2]]
"background": "empty background",
"negative": "None"
}
}

##  Appendix 0.D More Visualization Results

Figure 7 to Figure 11 showcase results of multi-turn editing. Figure 12 to
Figure 16 showcase results of story generation. Figure 17 to Figure 20
showcases high-quality final images and the corresponding on-stage character
images generated by TheaterGen based on SDXL. This demonstrates that
TheaterGen is capable of preserving character features to the maximum extent
while achieving a seamless integration of foreground and background.

Figure 7: Visualization results of multi-turn editing. Figure 8: Visualization
results of multi-turn editing. Figure 9: Visualization results of multi-turn
editing. Figure 10: Visualization results of multi-turn editing. Figure 11:
Visualization results of multi-turn editing. Figure 12: Visualization results
of story generation. Figure 13: Visualization results of story generation.
Figure 14: Visualization results of story generation. Figure 15: Visualization
results of story generation. Figure 16: Visualization results of mstory
generation. Figure 17: Comparison between on-stage character images and the
final image. Figure 18: Comparison between on-stage character images and the
final image. Figure 19: Comparison between on-stage character images and the
final image. Figure 20: Comparison between on-stage character images and the
final image.


* [20] D. I. Kazakov and S. N. Solodukhin, Nucl. Phys. B 429 (1994), 153-176 [arXiv:hep-th/9310150 [hep-th]].
  * [21] T. Damour and S. N. Solodukhin, “Wormholes as black hole foils,” Phys. Rev. D 76 (2007), 024016 [arXiv:0704.2667 [gr-qc]].
  * [22] S. N. Solodukhin, Phys. Rev. D 53 (1996), 824-835 [arXiv:hep-th/9506206 [hep-th]].
  * [23] O. B. Zaslavskii, Mod. Phys. Lett. A 21, 2283-2290 (2006) [arXiv:hep-th/0608183 [hep-th]].
  * [24] J. Boos, V. P. Frolov and A. Zelnikov, “Ghost-free modification of the Polyakov action and Hawking radiation,” Phys. Rev. D 100 (2019) no.10, 104008 [arXiv:1909.01494 [hep-th]].
  * [25] R. C. Myers, “Black hole entropy in two-dimensions,” Phys. Rev. D 50, 6412-6421 (1994) [arXiv:hep-th/9405162 [hep-th]].
  * [26] S. N. Solodukhin, “The Conical singularity and quantum corrections to entropy of black hole,” Phys. Rev. D 51, 609-617 (1995) [arXiv:hep-th/9407001 [hep-th]].
  * [27] B. A. Dubrovin, A. T. Fomenko and S. P. Novikov, “Modern Geometry — Methods and Applications,” Graduate Texts in Mathematics, vol 93. Springer, New York, NY.