# Knowledge Pursuit Prompting for Zero-Shot Multimodal Synthesis

Jinqi Luo Kwan Ho Ryan Chan Dimitris Dimos René Vidal
Center for Innovation in Data Engineering and Science, University of
Pennsylvania
{jinqiluo, ryanckh, dimos<EMAIL_ADDRESS>
###### Abstract

Hallucinations and unfaithful synthesis due to inaccurate prompts with
insufficient semantic details are widely observed in multimodal generative
models. A prevalent strategy to align multiple modalities is to fine-tune the
generator with a large number of annotated text-image pairs. However, such a
procedure is labor-consuming and resource-draining. The key question we ask
is: can we enhance the quality and faithfulness of text-driven generative
models beyond extensive text-image pair annotations? To address this question,
we propose Knowledge Pursuit Prompting (KPP), a zero-shot framework that
iteratively incorporates external knowledge to help generators produce
reliable visual content. Instead of training generators to handle generic
prompts, KPP employs a recursive knowledge query process to gather informative
external facts from the knowledge base, instructs a language model to compress
the acquired knowledge for prompt refinement, and utilizes text-driven
generators for visual synthesis. The entire process is zero-shot, without
accessing the architectures and parameters of generative models. We evaluate
the framework across multiple text-driven generative tasks (image, 3D
rendering, and video) on datasets of different domains. We further demonstrate
the extensibility and adaptability of KPP through varying foundation model
bases and instructions. Our results show that KPP is capable of generating
faithful and semantically rich content across diverse visual domains, offering
a promising solution to improve multimodal generative models.

##  1 Introduction

Figure 1: Given a generic text prompt (e.g., on bird synthesis), black-box
multimodal generative models often produce (I) unsatisfactory synthesis. Our
proposed knowledge pursuit prompting recursively queries facts from external
knowledge and instructs a language model to aggregate the facts to achieve
(II) faithful multimodal synthesis in a zero-shot manner.

Recent years have witnessed remarkable advancements in multimodal generative
models that take text prompts to generate various visual outputs (e.g., images
[34, 30] and 3D rendering [31, 45]). The successful deployment of such models
hinges on their ability to precisely map text prompts to corresponding visual
representations. However, this text-to-vision translation process encounters
significant challenges when handling out-of-domain requests. This includes
rare creatures, technical terms, and ambiguous expressions, all of which can
lead to inaccuracies or irrelevant visual synthesis. To address the alignment
on such use cases, a common approach is to annotate large-scale datasets of
text-image pairs [24, 37, 27] and fine-tune the generator to reduce
hallucination. Yet, such an approach demands extensive manual labor and
computational resources. Furthermore, an increasing trend among powerful
generative foundation models [28, 35, 33] is the inaccessibility of trainable
parameters to external users (i.e., models are more frequently closed-source).
Even if black-box generative models are well-trained and sufficiently
expressive, common prompts often provide insufficient details for high-quality
synthesis, give no interpretation about the key visual elements, and leave
room for hallucination. In light of these challenges, our proposed framework
shown in Figure 1 explores the enhancement of black-box text-driven generators
for factual, explicit, and high-quality synthesis that is more widely
accessible.

Knowledge retrieval [9, 22], which seeks informative references and evidence
from external knowledge bases (e.g., Wikipedia), has been shown to be
effective in assisting natural language tasks. Our work considers
strengthening text-driven generative models with external factual knowledge.
Such a procedure can be motivated by how scientists form reliable arguments:
they cite external references and extract informative evidence from past
literature for credibility. Unlike fine-tuning large models, which store new
knowledge implicitly in their model parameters, storing such knowledge
externally (and retrieving if needed) is more adaptable, explicit, and
efficient since modern knowledge evolves quickly over time. However, existing
retrieval pipelines for generative foundation models adopt static queries
[39], require extensive training [47, 2], or request access to the model’s
posterior (e.g., probability of the next word) [15, 39].

On the other hand, with the advent of emergent abilities (e.g., scene
imagination, numerical operations), Large Language Models (LLMs) can leverage
additional information to assist multimodal synthesis [49, 50, 23, 34, 44].
However, the uninterpretable hallucination phenomenon [51, 3] of LLMs
themselves presents non-trivial challenges for downstream vision tasks. We
hypothesize that the hallucinatory responses from these generative models
arise due to under-informative prompts with insufficient factual details. To
address this, we draw inspirations from Information Pursuit (IP) [5, 4], a
paradigm that sequentially selects informative features for downstream visual
tasks. In this paper, we design a framework that recursively queries
informative facts from a knowledge base in a contextual manner for visual
generative tasks.

The key concept of our framework, Knowledge Pursuit Prompting (KPP), is shown
in Figure 1. Consider a wildlife researcher who wants to generate an image of
Icthyophaga Leucogaster, the scientific name for white-bellied sea eagle.
Although this Latin scientific name can be comprehended by readers who are
familiar with the biological naming system, the name turns out to be a
difficult prompt for text-driven generators (see part (I) in Figure 1).
Instead of training the generator to adapt the request, KPP drives an off-the-
shelf text querier through an iterative querying process to acquire external
informative evidence for this difficult prompt, instructs an LLM to parse the
queried knowledge context for prompt enhancement, and adopts text-driven
generators to synthesize visual contents. After each query to the external
knowledge base, we append the queried fact to the knowledge context. This
context is then fed into the querier for the next iteration of knowledge
search. The final knowledge context is a series of facts that is informative
and faithful for visual synthesis. Given modality-aware instructions that
inform the type of downstream generator, the LLM will read the knowledge
context and follow the instruction rules to compress the context into concise
generative prompts. Through our framework, the enhanced prompts contain
understandable descriptions, semantics, and details summarized in a modality-
aware manner. The user is able to obtain zero-shot visual syntheses (e.g.,
images, 3D models, videos) from the factually-enhanced prompt. Our framework
does not require white-box parameter access to generative foundation models.
Instead, by varying the instructions to the LLM, our framework adapts to
multiple text-driven generative models in a plug-and-play manner.

In summary, our proposed method, KPP, makes three major improvements over past
approaches:

  * •

KPP is zero-shot: it requires neither generator training nor parameter access
to obtain high-quality visual synthesis.

  * •

KPP serves as a unified prompt interface: the framework is plug-and-play for
multiple text-driven generators.

  * •

KPP seeks knowledge in a dynamic and contextual way: queried knowledge is
appended in a knowledge context which is used for the next pursuit iteration.

##  2 Related Work

###  2.1 Language-Conditioned Generative Models

Synthesis from deep generative models, especially variants of StyleGAN [17,
18] and Diffusion Models [12, 34, 42, 8], can be manipulated in the latent
space [46, 21] using explicit multimodal conditioning (e.g., descriptive text,
visual/symbolic template). Earlier efforts adopt CLIP [32] as the text encoder
to guide image sampling [29] or optimize the image generator [34, 20] with the
user text prompt. These zero-shot text-to-image generators can serve to
supervise the text-to-3D task [31, 45] and text-to-video task [26, 19]. More
recent works explore the use of language models to improve visual synthesis:
[44] takes diffusion models as the visual decoder for LLM-centered multimodal
generation, and [50, 23] enhance compositionality of visual synthesis through
conditional templates proposed by language models. However, the faithfulness
of LLM-assisted visual synthesis is vaguely addressed in past work. Our work,
instead, pursues a series of factual knowledge as the context to enhance text-
driven synthesis, and it generalizes to multiple modalities in a zero-shot
manner.

Figure 2: The KPP framework. The user inputs a generic prompt that lacks
details and KPP recursively aggregates knowledge to refine the request. For
each iteration of knowledge pursuit, the querier picks the most informative
fact for the current knowledge context and appends this fact to update the
context. The LLM aggregates the final context to produce a faithfully-enhanced
prompt for text-driven generators.

###  2.2 Knowledge Retrieval and Query

Querying informative textual evidence (e.g., background knowledge, semantic
context) from external knowledge bases is widely adopted for faithful text
synthesis [9, 22, 15], symbolic reasoning [13], and medical guidelines [48].
Retrieving image data has been adopted for training diffusion model [2] and
joint multimodal transformer training [47]. [39] found that seeking external
evidence for a black-box LLM (e.g., GPT [28]) can improve its reasoning.
However, these approaches either focus on the text modality only, request
training the base model, or query all evidence from a single static query. On
the other hand, [5] made insightful discoveries that, by sequentially querying
the data for informative visual semantics, models can reach more interpretable
and confident predictions. Inspired by such a sequential procedure, we propose
to recursively seek factual knowledge for zero-shot multimodal synthesis,
boosting information expansion and contextual awareness.

##  3 Method

Our method KPP consists of three parts: We first describe the process of
iteratively curating a set of trusted facts from our knowledge base to form a
knowledge-driven context (Section 3.1). Then, we instruct language models to
aggregate our generated context into a structured prompt (Section 3.2).
Finally, the prompt is fed into a text-to-vision generator to synthesize the
desired visual output (Section 3.3). Figure 2 shows the overview of our
framework.

###  3.1 Knowledge Pursuit

We assume access to a knowledge base that contains a large set of user-trusted
and useful facts that are sufficient for the given task. Let $D$ denote the
given knowledge base consisting of a set of $m$ facts, where the size of $D$
is assumed to be large, and let $C$ be our knowledge-driven context, a set
that will consist of our given task prompt $x$ and a list of facts from $D$.
In what follows, we explain the iterative process of updating the state of our
knowledge base and context.

Initially, at iteration $k=0$, the knowledge base is the set of all facts,
_i.e_. $D_{0}=D$, and the context consists of only the task prompt, _i.e_.
$C_{0}=\\{x\\}$. At each step $k$, given the context $C_{k}$ and the knowledge
base $D_{k}$, the goal is to find the most relevant fact from $D_{k}$ that is
contextual to $C_{k}$ and informative for the downstream generative task. The
most relevant fact for the current given context is a query $d\in D_{k}$ that
maximizes textual relevance between the text embedding of $C_{k}$ and the text
embedding of $d$, _i.e_.:

$d_{k+1}=\operatorname*{argmax}_{d\in
D_{k}}R(\mathcal{E}_{\psi}(d),\mathcal{E}_{\psi}(C_{k})),$ (1)

where $\mathcal{E}_{\psi}$ is an encoder that maps text tokens to a numeric
embedding vector in $\mathbb{R}^{p}$ and
$R:\mathbb{R}^{p}\times\mathbb{R}^{p}\rightarrow\mathbb{R}^{+}$ outputs a non-
negative score that measures the relevance between the two input text
embeddings. In this work, we choose $R$ to be the inner-product similarity and
$\mathcal{E}_{\psi}$ to be a pre-trained contrastive text encoder [14]. Note
that $R$ and $\mathcal{E}_{\psi}$ can be chosen flexibly depending on the
user’s task. Then, the most relevant fact $d_{k+1}$ is pulled from the current
knowledge base $D_{k}$ and pushed into the current context $C_{k}$:

$\displaystyle D_{k+1}=D_{k}\setminus\\{d_{k+1}\\},$ (2) $\displaystyle
C_{k+1}=C_{k}\cup\\{d_{k+1}\\}.\vspace{-1.5mm}$ (3)

The querying process stops after $n$ steps, where $n$ is either a user-defined
upper bound of knowledge context size or depends on the maximum number of
tokens allowed for the subsequent language model. Alternatively, one can
design a stopping criterion, _e.g_. when the textual relevance score (Equation
1) of any query and the context is below a pre-determined threshold, or when
the embedding of $C_{k}$ does not change much from one iteration to the next.

As validated in Section 4.3.3, the recursive paradigm searches more
informative knowledge compared to the static top-K retrieval. Using
“Icthyophaga Leucogaster” as an example, the database’s directly related
knowledge regarding this string is quite limited. However, “white-bellied sea
eagle” (which is the bird’s common name) is associated with a richer pool of
available knowledge. Our approach establishes an association between them in
the first iteration, and then recursively explores other relevant knowledge
based on both strings. The construction of the knowledge context allows for a
more expansive and rich exploration of the implicit knowledge distribution.

###  3.2 Knowledge Aggregation

Once we have generated a knowledge-driven context $C$, we instruct an LLM
$\mathcal{F}_{\theta}$ to further parse and aggregate the context into a
concise prompt for the downstream text-to-vision generator. Our in-context LLM
instruction consists of three paragraphs, namely Parsing Demonstration,
Enhancement Request, and Knowledge Rejection.

Here is a demonstration:
Knowledge: $d_{1},\dots,d_{n}$
Original Prompt: $x_{\text{demo}}$
Enhanced Prompt: $T_{\text{demo}}$

Now read the following knowledge and enhance the user original prompt for the
<Generator>. You may add more appearance details, semantic attributes, and
fine-grained visual elements.

If you find some piece of knowledge irrelevant or conflicting to the original
prompt, you may ignore the piece. You may also remove meaningless words. You
should make the prompt concise, expressive, and accurate.

In the first paragraph, Parsing Demonstration guides the language model to
digest the curated list of knowledge and enhance the prompt for downstream
text-driven generators by providing one in-context example. Then, Enhancement
Request describes the task that the language model should execute. Note that
different modalities have different styles of enhancements and we include
examples in Appendix C for each task. We associate the generator information
(e.g., <Generator> = “Stable Diffusion XL”) in the instruction for better
generator-aware enhancement. Last but not least, Knowledge Rejection addresses
the issue that the queried knowledge is not always relevant to the user’s
task. There exist rare cases in which the underlying faithfulness and
relevance of the knowledge retrieved for the user’s task are not satisfactory.
The knowledge rejection rule lets the language model override the knowledge
context when it finds conflicts or redundancies.

By concatenating three components as the modality instruction ($I$) with the
query knowledge context (as $C$), the language model will generate the
enhanced prompt $T=\mathcal{F}_{\theta}(I\cup C)$ for the downstream text-
driven generative model.

###  3.3 Multimodal Synthesis

Figure 3: Prompt visualization that demonstrates the detail and faithfulness
of prompts enhanced by KPP compared to directly instructing language model
methods. The direct prompting method offers generic descriptions, while ours
generates a more detailed, precise, and enriched prompt.

Our framework is a unified prompt interface to enhance synthesis in multiple
modalities. Figure 3 shows that our knowledge-driven framework enhances the
text prompt with more faithful and fine-grained details, while direct LLM
prompting generates generic descriptions. Given such an enhanced prompt $T$ by
knowledge aggregation (Section 3.2), we obtain the visual synthesis
$S=\mathcal{G}_{\phi}(T)$, where $\mathcal{G}_{\phi}$ is the pre-trained text-
driven generator.

The enhancement procedure is applicable across text-to-image [34, 30], text-
to-3D [31, 45] and text-to-video models [26, 19] in a plug-and-play manner.
The modality-aware design adapts to different visual paradigms by changing the
string <Generator>. This allows the language model to refine the knowledge
context catered to different modalities. Namely, the prompt styles of 3D
rendering or video synthesis can be different from those of image synthesis.
Detailed examples of full instructions are in Appendix C. For our pre-trained
latent diffusion model, we adopt ancestral sampling in the latent space
conditioned on the enhanced prompt, also described in Appendix C.

Figure 4: Comparison of images generated from original captions and KPP
enhancements on the GBIF (upper two rows), MSCOCO (three captions on the lower
left), and GUIE LAION-5B (two captions on the lower right) datasets. Blue
columns demonstrate improvements in image quality, while green columns
emphasize increased faithfulness of the synthesis. We outline deficits in
bounding boxes (e.g., the missing deer legs, the stuck closed fridge door).
and underline notable concepts that the generator should express sufficiently.

##  4 Experimental Results

We present the empirical validation of the KPP enhancement on multiple text-
driven generative paradigms. Section 4.1 states the necessary setup of the
framework and experiments. Section 4.2 visualizes the multimodal enhancement
results, and Section 4.3 evaluates the visual synthesis by several benchmarks.
Section 4.4 analyzes the design principles and extensibility of our framework.

###  4.1 Implementation Details

Dataset. We assess the effectiveness of our method across image captions from
various datasets including MSCOCO 2017 [25], Global Biodiversity Information
Facility (GBIF) Taxonomy [38], and GUIE LAION-5B [37, 7]. For each dataset, we
sample 2,000 captions and generate 10 images for each caption. We evaluate the
effectiveness of our iterative querying strategy on sampled questions from
MMLU [10]. The knowledge context size is 8 in KPP. The external knowledge base
is Wikipedia up to December 2018 with FAISS indexing [16]. The base consists
of 21 million passages, which are split into disjoint blocks of 100 words.
Appendix B elaborates full details of the knowledge base.

Language Model. The knowledge aggregation backbone of KPP is the GPT-4 [28].
Recent literature reports that its behaviors shift over time since the base
model behind the interface/API could be changing [6]. To ensure minimal
behavior shift over time for better reproducibility, we use the fixed snapshot
gpt-4-0613 completion API. In Section 4.4.1 (multimodal generality), we embed
KPP with the multimodal gpt-4-vision-preview.

Text-Driven Generator. We adopt publicly available generators for different
modalities. We use Stable Diffusion (SD) XL [30] for all text-to-image
synthesis except for Section 4.4.2 (component study) where we start with
SD-1.5 [34]. In Section 4.4.1 (multimodal generality), we also experiment with
dall-e-3 API. The quality evaluation uses the Fréchet Inception Distance (FID)
[11] and Inception Score (IS) [36]. We resize all images to 224 $\times$ 224
for FID and 299 $\times$ 299 for IS. In the text-to-3D domain, we employ
DreamFusion [31] embedded with DeepFloyd IF [40] for experiments considering
its efficient inference, and ProlificDreamer [45] for pipeline demonstrations.
Video synthesis was performed using ZeroScope [43, 26] and Text2Video-Zero
[19] in the manuscript, and Stable Video Diffusion in Appendix A.

Figure 5: Comparison of 3D rendering generated from original captions and KPP
enhancements on the GBIF and MSCOCO datasets. The text-to-3D model is
DreamFusion [31] embedded with DeepFloyd-IF [40]. We can observe that our
framework enhances the color patterns and semantic contour of rare animal
prompts, and improves the synthesis quality with more details on common object
prompts. Figure 6: User evaluation on images generated by different
approaches. The vertical axis represents the fraction of images that are
selected as the most high-quality $\setminus$ faithful $\setminus$ realistic.
We can observe that our framework outperforms in all three aspects.

###  4.2 Synthesis Visualization

This section demonstrates the benefits of knowledge-driven refinement for
visual synthesis across multiple modalities.

####  4.2.1 Image Synthesis

Our framework supports high-quality image synthesis by prompting text-driven
generators in a faithful manner. Figure 4 shows the enhanced synthesis using
captions of creature names (for biological research) from GBIF, scene captions
from MSCOCO, and landmark captions from GUIE LAION-5B. We observe that the
original stable diffusion XL produces hallucinatory contents. We categorize
the enhancement over two kinds of failures: columns with blue captions
demonstrate that KPP improves the synthesis quality (e.g., the legs for deer
and hamster), and the green ones indicate that our knowledge-driven process
corrects the ambiguity of the prompts by augmenting with more facts about
creatures (e.g., Money-Thorn111https://www.feedipedia.org/node/352 is actually
a kind of tree growing in Africa, and Kelung Cat
Snake222https://eol.org/pages/795578 is a species of colubrid snakes). From
the MSCOCO captions, we observe that KPP produces more fine-grained content
such as the reasonable layout of kitchen, the elegance of the decorated
dessert, and the Victorian style of the bed. The KPP synthesis on LAION-5B
captions is more faithful, as the Baker Beach in San Francisco is famous for
the joint scenic view with the Golden Gate Bridge, which the SD-XL caption
synthesis fails to capture. We further show 100 more images synthesized by KPP
from the three datasets in Appendix A.

####  4.2.2 3D and Video Synthesis

KPP can enhance textual descriptions to generate fine-grained 3D rendering.
Figure 5 shows the enhanced 3D rendering from the GBIF and MSCOCO captions.
Specifically, the 3D rendering with generic captions either produces
inaccurate depictions or fails to converge. For example, DreamFusion renders a
toilet bowl with a missing water tank, and the synthesis of the beetle is
ambiguous. KPP results, instead, show clear semantic attributes for captions
of rare species, and valid appearance with correct shapes on compositional
captions.

Figure 7 shows the generated video from the LAION captions by ZeroScope. The
KPP-enhanced videos are more coherent with the captions, and the visual
faithfulness is much improved. For example, the Orange Cranberry
Pancakes333https://www.food.com/recipe/cranberry-orange-pancakes-304319 (a
breakfast dish with pancake, cranberry, and orange fruit) directly synthesized
by ZeroScope is a hallucinatory mixture of an orange and a pancake shape.
After the enhancement by KPP, the pancakes show adequate shape and a
reasonable appearance. The cranberries with butter squares are placed on top
of the pancakes, while the orange is placed adjacent to the pancakes. There is
also a dynamically pouring orange juice in the video by KPP.

Figure 7: Comparison of videos generated with original captions and KPP
enhancement on the GUIE LAION-5B dataset. The text-to-video model is ZeroScope
[43], a watermark-free fine-tuning of VideoFusion [26]. We can observe that
our framework enhances the synthesis quality (e.g., synthesis of street view,
correct patterns of pancakes with decomposed oranges), and improves the
realism and faithfulness for knowledge-intensive captions (e.g., landmarks
like North Bondi Beach in Sydney and Cana Light in Wisconsin).

####  4.2.3 Enhanced Prompts

Figure 3 shows a comparison of three prompting methods, the original caption,
direct prompting, and KPP. The original captions are from LAION-5B and
prompting (the second method) means enhancing the caption by LLM prompting
without any factual knowledge. Such direct approaches miss key visual elements
of the landmarks. For example, the direct prompting method lacks distinctive
facts such as the stone structure of the Rialto Bridge in Venice and the
Golden Gate Bridge in the background of the Baker
Beach444https://presidio.gov/explore/attractions/baker-beach. Our framework,
instead, successfully parses core semantics and concepts for the synthesis
from the recursive knowledge queries. Appendix D provides the full retrieved
knowledge context, and Appendix C shows LLM instructions.

|  FID ($\downarrow$) |  IS ($\uparrow$)
---|---|---
| MSCOCO | LAION | MSCOCO | LAION
Caption | 128.59 | 91.50 | 5.32 | 5.44
Prompting [28] | 86.42 | 52.96 | 5.34 | 5.53
KPP (Ours) | 85.89 | 46.69 | 5.37 | 5.56
Table 1:  Image quality evaluation. Observe that KPP not only improves the
faithfulness of the visualization but also has higher image quality compared
to directly prompting the language model.

###  4.3 Quantitative Evaluation

This section assesses the quality and faithfulness of KPP, and benchmark our
recursive querying strategy against established baselines.

####  4.3.1 Visual Quality

The quality of visual synthesis serves as a direct indicator of the
effectiveness of our knowledge-driven framework. We employ FID and IS to
provide a measure of visual fidelity and diversity in Table 1. Our baseline
comparison involves the synthesis from the caption and the LLM prompting
approach without the integration of any external knowledge contexts (mentioned
in Section 4.2.3). The result demonstrates that KPP boosts for the finest
visual quality as factual knowledge is introduced, indicating its outstanding
capability to enhance textual prompts. On the other hand, the baseline methods
often result in suboptimal synthesis. Note that another major advantage of KPP
is the improved faithfulness, which exhibits a significant reduction for
hallucinatory content. As Section 4.2 visualizes, KPP synthesis is not only
visually compelling but also factually sound.

|  Static Aggregation |  REPLUG [39] |  KPP (Ours)
---|---|---|---
Geography (HS) | 24/30 | 23/30 | 24/30
Global Statistics | 11/30 | 9/30 | 13/30
Biology (HS) | 20/30 | 22/30 | 22/30
World History | 25/30 | 24/30 | 25/30
Total Count ($\uparrow$) | 80/120 | 78/120 | 84/120
Table 2:  The question-answer evaluation on different retrieval paradigms. HS
indicates the High-School level in MMLU. The numbers are presented as correct
answers / total questions. We can observe that KPP has the highest correctness
in all subjects.

####  4.3.2 Synthesis Faithfulness

We conduct a user study to evaluate the visual soundness of our generative
framework. We recruit 10 independent users with at least Bachelor’s degrees to
evaluate the faithfulness, quality, and realism (how realistic the image
looks) of the images by KPP and two baselines. Take the faithfulness
evaluation as an example. Each user reads 100 triplets of images by three
methods corresponding to the same instance of data. The sequences of the
images are randomly shuffled. The user selects the image that is most faithful
among all. Figure 6 shows the frequency of each method being selected. We can
observe that the average rating for KPP is higher than the two baselines. This
indicates that our knowledge pursuit process effectively enhances the
faithfulness and quality of synthesis from the user perspective.

Method |  FID ($\downarrow$) |  IS ($\uparrow$)
---|---|---
KPP (GPT-3.5, SD-1.5) |  55.07 |  5.30
\+ Increased Context Size to 8 |  53.74 |  5.35
\+ GPT-4 |  52.91 |  5.41
\+ Stable Diffusion XL |  50.45 |  5.44
\+ FreeU for Diffusion [41] |  46.69 |  5.56
Table 3:  Procedural addition of each framework component on the synthesis
quality of the GUIE LAION-5B dataset. Starting from the initialization with
GPT-3.5, SD-1.5, and a context with at most two facts, each subsequent
addition leads to better performance.

####  4.3.3 Effectiveness of the Query Strategy

This section emphasizes our sequential knowledge pursuit process which
dynamically queries each new fact based on the state of knowledge context. We
compare our query strategy with REPLUG [39] and Static Aggregation in the
textual question answering task. REPLUG queries the top-K most informative
facts in a static one-time search and makes a prediction by ensembling the LLM
posteriors from each fact. Static Aggregation, the static query version of
KPP, takes all top-K facts in an one-time query as the context and instructs
the LLM to aggregate. Table 2 shows the results using four subjects from MMLU
[10] that are relevant to factual visual synthesis. We sample 30 questions
from each subject. We follow REPLUG’s setup and choose OpenAI’s text-
davinci-003 legacy API as the language model base for all three methods. We
see that our recursive querying strategy is more performant in knowledge-
intensive language tasks, indicating the strong capability to aggregate
knowledge for our multimodal synthesis. Our observations reveal that when
multiple corpora are queried in a single request, the knowledge received tends
to overlap, offering little diversity. Instead, our recursive method
iteratively introduces new terms into the knowledge context for the next
knowledge query. We hypothesize that the recursive construction of this
knowledge context contributes to the superior performance of our paradigm.

Figure 8: The comparisons of real images of the species from the GBIF Taxonomy
Website, synthesis by captions, and synthesis by KPP embedded with dall-e-3
API as the generator $\mathcal{G}_{\phi}$. Full captions of creatures are
sampled from GBIF and shown in Appendix B. We can observe that KPP synthesis
is more faithful and aligned to the species.

###  4.4 Framework Design Analysis

This section discusses the modular design of KPP, explores variants of
foundation model bases, and analyzes how our system is progressively built.

####  4.4.1 Multimodal Generality and Adaptability

Our modular design enables flexible choices of generative models
($\mathcal{G}_{\phi}$), language models ($\mathcal{F}_{\theta}$), and in-
context instructions ($I$). For instance, Figure 8 shows that enhanced prompts
by KPP strengthen the synthesis of DALLE-3 [1], which is specialized at
handling complex captions with descriptive details. Note that dall-e-3 API
automatically rewrites any input prompt for safety and quality reasons. To
ensure a fair comparison and better evaluation of KPP, we follow OpenAI’s
prompting instruction to constrain such post-
processing555https://platform.openai.com/docs/guides/images/prompting. Another
example is that KPP supports diverse synthesis by modifying instructions $I$
to guide the language model to produce semantic variations (Appendix A).

In the case that users have reference images in addition to their initial
prompt as multimodal inputs, we can set $\mathcal{F_{\theta}}$ as a Vision
Language Model (VLM) to accommodate the multimodality. A suitable VLM should
have emergent abilities of (1) handling long in-context instructions to
aggregate knowledge, and (2) understanding intricate images to comprehend
semantics. Figure 9 shows the KPP synthesis embedded with gpt-4-vision-preview
API666https://platform.openai.com/docs/guides/vision on GUIE LAION-5B text-
image pairs. The modified instructions, which attach the reference image after
the knowledge, are shown in Appendix C. By augmenting the KPP with an in-
context VLM that perceives visual grounding, the KPP synthesis not only has
more faithfulness but also has a closer appearance with the original images.
Note that we show in-context VLM as a supplementary to our knowledge pursuit
paradigm, since common user requests (text prompts) do not always come with
reference images.

Figure 9: The comparisons of reference images, synthesis by original captions,
and synthesis by KPP embedded with gpt-4-vision-preview API as the
(multimodal) language model $\mathcal{F}_{\theta}$. Full captions of images
are sampled from LAION-5B and shown in Appendix B. We can observe that KPP
synthesis is outstanding in both faithfulness and similarity to reference
images.

####  4.4.2 Component Study

This section quantifies the contribution of each framework component to the
overall performance. We experiment with the variants of Stable Diffusion (SD)
and a comparison between GPT-3.5 and GPT-4. Table 3 shows a clear improvement
of image quality on the GUIE LAION-5B dataset with each component
progressively added. Our proposed KPP is a cumulative system summarizing all
these improvements.

##  5 Conclusion and Discussion

This paper presents KPP, a knowledge pursuit framework for enhancing zero-shot
multimodal synthesis. We propose a new generative system that conducts
recursive queries of external knowledge to support high-quality faithful
synthesis. Unlike methods that rely on a single retrieval or heavy training,
our knowledge context with customizable foundational components enables a more
comprehensive aggregation and compression of curated facts. While many
multimodal generative models face challenges on user prompts and produce
hallucinatory outputs, KPP shows a zero-shot direction in handling these
intricacies with publicly available neural toolkits. Extensive experiments
show that, as a unified prompt interface, KPP has plug-and-play flexibility
and convincing effectiveness across different modalities such as images, 3D
rendering, and videos.

Our framework has two potential limitations. Firstly, KPP assumes that
indexing operations are scalable when recursively querying the knowledge base.
Engineering innovations in indexing algorithms can further reduce the curation
time of our knowledge-driven context. Secondly, the framework presumes that
the external sources of facts are trustworthy and comprehensive. We explore
robustness against misleading adversaries or inaccurate facts in existing
knowledge bases in future works.

Acknowledgments. This research was supported by the ARO MURI contract
W911NF-17-1-0304, the NSF grant 2031985, the Simons Foundation grant 135615,
and NSF grant DGE2139757. Moreover, the authors thank Aditya Chattopadhyay,
Tianjiao Ding, Ryan Pilgrim, Tianyuan Zhang, and Bowen Li for their insightful
feedback that improves this work.

## References

  * Betker et al. [2023] James Betker, Gabriel Goh, Li Jing, Tim Brooks, Jianfeng Wang, Linjie Li, Long Ouyang, Juntang Zhuang, Joyce Lee, Yufei Guo, Wesam Manassra, Prafulla Dhariwal, Casey Chu, and Yunxin Jiao.  Improving image generation with better captions.  _https://cdn.openai.com/papers/dall-e-3.pdf_ , 2023.
  * Blattmann et al. [2022] Andreas Blattmann, Robin Rombach, Kaan Oktay, Jonas Müller, and Björn Ommer.  Semi-parametric neural image synthesis.  In _NeurIPS_ , 2022.
  * Bommasani et al. [2021] Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S. Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, Erik Brynjolfsson, Shyamal Buch, Dallas Card, Rodrigo Castellon, Niladri Chatterji, Annie Chen, Kathleen Creel, Jared Quincy Davis, Dora Demszky, Chris Donahue, Moussa Doumbouya, Esin Durmus, Stefano Ermon, John Etchemendy, Kawin Ethayarajh, Li Fei-Fei, Chelsea Finn, Trevor Gale, Lauren Gillespie, Karan Goel, Noah Goodman, Shelby Grossman, Neel Guha, Tatsunori Hashimoto, Peter Henderson, John Hewitt, Daniel E. Ho, Jenny Hong, Kyle Hsu, Jing Huang, Thomas Icard, Saahil Jain, Dan Jurafsky, Pratyusha Kalluri, Siddharth Karamcheti, Geoff Keeling, Fereshte Khani, Omar Khattab, Pang Wei Koh, Mark Krass, Ranjay Krishna, Rohith Kuditipudi, Ananya Kumar, Faisal Ladhak, Mina Lee, Tony Lee, Jure Leskovec, Isabelle Levent, Xiang Lisa Li, Xuechen Li, Tengyu Ma, Ali Malik, Christopher D. Manning, Suvir Mirchandani, Eric Mitchell, Zanele Munyikwa, Suraj Nair, Avanika Narayan, Deepak Narayanan, Ben Newman, Allen Nie, Juan Carlos Niebles, Hamed Nilforoshan, Julian Nyarko, Giray Ogut, Laurel Orr, Isabel Papadimitriou, Joon Sung Park, Chris Piech, Eva Portelance, Christopher Potts, Aditi Raghunathan, Rob Reich, Hongyu Ren, Frieda Rong, Yusuf Roohani, Camilo Ruiz, Jack Ryan, Christopher Ré, Dorsa Sadigh, Shiori Sagawa, Keshav Santhanam, Andy Shih, Krishnan Srinivasan, Alex Tamkin, Rohan Taori, Armin W. Thomas, Florian Tramèr, Rose E. Wang, William Wang, Bohan Wu, Jiajun Wu, Yuhuai Wu, Sang Michael Xie, Michihiro Yasunaga, Jiaxuan You, Matei Zaharia, Michael Zhang, Tianyi Zhang, Xikun Zhang, Yuhui Zhang, Lucia Zheng, Kaitlyn Zhou, and Percy Liang.  On the opportunities and risks of foundation models.  _arXiv preprint arXiv:2108.07258_ , 2021.
  * Chattopadhyay et al. [2022] Aditya Chattopadhyay, Stewart Slocum, Benjamin D. Haeffele, Rene Vidal, and Donald Geman.  Interpretable by design: Learning predictors by composing interpretable queries.  In _IEEE TPAMI_ , 2022.
  * Chattopadhyay et al. [2023] Aditya Chattopadhyay, Kwan Ho Ryan Chan, Benjamin D Haeffele, Donald Geman, and René Vidal.  Variational information pursuit for interpretable predictions.  In _ICLR_ , 2023.
  * Chen et al. [2023] Lingjiao Chen, Matei Zaharia, and James Zou.  How is chatgpt’s behavior changing over time?  _arXiv preprint arXiv:2307.09009_ , 2023.
  * Competition [2022] Google Universal Image Embedding Competition.  Guie laion-5b dataset.  _Kaggle_ , 2022.
  * Dhariwal and Nichol [2021] Prafulla Dhariwal and Alexander Quinn Nichol.  Diffusion models beat gans on image synthesis.  In _NeurIPS_ , 2021.
  * Guu et al. [2020] Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang.  Realm: Retrieval-augmented language model pre-training.  In _ICML_ , 2020.
  * Hendrycks et al. [2021] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt.  Measuring massive multitask language understanding.  In _ICLR_ , 2021.
  * Heusel et al. [2017] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter.  Gans trained by a two time-scale update rule converge to a local nash equilibrium.  In _NeurIPS_ , 2017.
  * Ho et al. [2020] Jonathan Ho, Ajay Jain, and Pieter Abbeel.  Denoising diffusion probabilistic models.  In _NeurIPS_ , 2020.
  * Hu et al. [2023] Chenxu Hu, Jie Fu, Chenzhuang Du, Simian Luo, Junbo Jake Zhao, and Hang Zhao.  Chatdb: Augmenting llms with databases as their symbolic memory.  _arXiv preprint arXiv:2306.03901_ , 2023.
  * Izacard et al. [2021] Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave.  Unsupervised dense information retrieval with contrastive learning.  _arXiv preprint arXiv:2112.09118_ , 2021.
  * Jiang et al. [2023] Zhengbao Jiang, Frank F. Xu, Luyu Gao, Zhiqing Sun, Li-Yu Daisy Liu, Jane Dwivedi-Yu, Yiming Yang, Jamie Callan, and Graham Neubig.  Active retrieval augmented generation.  _arXiv preprint arXiv:2305.06983_ , 2023.
  * Johnson et al. [2019] Jeff Johnson, Matthijs Douze, and Hervé Jégou.  Billion-scale similarity search with GPUs.  In _IEEE Transactions on Big Data_ , 2019.
  * Karras et al. [2019] Tero Karras, Samuli Laine, and Timo Aila.  A Style-Based Generator Architecture for Generative Adversarial Networks.  In _CVPR_ , 2019.
  * Karras et al. [2020] Tero Karras, Miika Aittala, Janne Hellsten, Samuli Laine, Jaakko Lehtinen, and Timo Aila.  Training Generative Adversarial Networks with Limited Data.  In _NeurIPS_ , 2020.
  * Khachatryan et al. [2023] Levon Khachatryan, Andranik Movsisyan, Vahram Tadevosyan, Roberto Henschel, Zhangyang Wang, Shant Navasardyan, and Humphrey Shi.  Text2video-zero: Text-to-image diffusion models are zero-shot video generators.  In _ICCV_ , 2023.
  * Kim et al. [2022] Gwanghyun Kim, Taesung Kwon, and Jong Chul Ye.  Diffusionclip: Text-guided diffusion models for robust image manipulation.  In _CVPR_ , 2022.
  * Kwon et al. [2023] Mingi Kwon, Jaeseok Jeong, and Youngjung Uh.  Diffusion models already have a semantic latent space.  In _ICLR_ , 2023.
  * Lewis et al. [2020] Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela.  Retrieval-augmented generation for knowledge-intensive nlp tasks.  In _NeurIPS_ , 2020.
  * Lian et al. [2023] Long Lian, Boyi Li, Adam Yala, and Trevor Darrell.  Llm-grounded diffusion: Enhancing prompt understanding of text-to-image diffusion models with large language models.  _arXiv preprint arXiv:2305.13655_ , 2023.
  * Lin et al. [2014a] Tsung-Yi Lin, Michael Maire, Serge Belongie, Lubomir Bourdev, Ross Girshick, James Hays, Pietro Perona, Deva Ramanan, C. Lawrence Zitnick, and Piotr Dollár.  Microsoft coco: Common objects in context.  _arXiv preprint arXiv:1405.0312_ , 2014a.
  * Lin et al. [2014b] Tsung-Yi Lin, Michael Maire, Serge Belongie, Lubomir Bourdev, Ross Girshick, James Hays, Pietro Perona, Deva Ramanan, C. Lawrence Zitnick, and Piotr Dollár.  Microsoft coco: Common objects in context.  _arXiv preprint arXiv:1405.0312_ , 2014b.
  * Luo et al. [2023] Zhengxiong Luo, Dayou Chen, Yingya Zhang, Yan Huang, Liang Wang, Yujun Shen, Deli Zhao, Jingren Zhou, and Tieniu Tan.  Videofusion: Decomposed diffusion models for high-quality video generation.  In _CVPR_ , 2023.
  * Nguyen et al. [2023] Thao Nguyen, Samir Yitzhak Gadre, Gabriel Ilharco, Sewoong Oh, and Ludwig Schmidt.  Improving multimodal datasets with image captioning.  In _NeurIPS Datasets and Benchmarks Track_ , 2023.
  * OpenAI [2023] OpenAI.  Gpt-4 technical report.  _arXiv preprint arXiv:2303.08774_ , 2023.
  * Patashnik et al. [2021] Or Patashnik, Zongze Wu, Eli Shechtman, Daniel Cohen-Or, and Dani Lischinski.  StyleCLIP: Text-Driven Manipulation of StyleGAN Imagery.  In _ICCV_ , 2021.
  * Podell et al. [2023] Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach.  Sdxl: Improving latent diffusion models for high-resolution image synthesis.  _arXiv preprint arXiv:2307.01952_ , 2023.
  * Poole et al. [2023] Ben Poole, Ajay Jain, Jonathan T. Barron, and Ben Mildenhall.  Dreamfusion: Text-to-3d using 2d diffusion.  In _ICLR_ , 2023.
  * Radford et al. [2021] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever.  Learning transferable visual models from natural language supervision.  In _ICML_ , 2021.
  * Ramesh et al. [2022] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen.  Hierarchical text-conditional image generation with clip latents.  _arXiv preprint arXiv:2204.06125_ , 2022.
  * Rombach et al. [2021] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models.  In _CVPR_ , 2021.
  * Saharia et al. [2022] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Raphael Gontijo-Lopes, Burcu Karagol Ayan, Tim Salimans, Jonathan Ho, David J. Fleet, and Mohammad Norouzi.  Photorealistic text-to-image diffusion models with deep language understanding.  In _NeurIPS_ , 2022.
  * Salimans et al. [2016] Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen.  Improved techniques for training gans.  In _NeurIPS_ , 2016.
  * Schuhmann et al. [2022] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski, Srivatsa Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmarczyk, and Jenia Jitsev.  Laion-5b: An open large-scale dataset for training next generation image-text models.  In _NeurIPS Datasets and Benchmarks Track_ , 2022.
  * Secretariat [2014] GBIF Secretariat.  Global biodiversity information facility backbone taxonomy.  _https://doi.org/10.15468/39omei_ , 2014.
  * Shi et al. [2023] Weijia Shi, Sewon Min, Michihiro Yasunaga, Minjoon Seo, Rich James, Mike Lewis, Luke Zettlemoyer, and Wen tau Yih.  Replug: Retrieval-augmented black-box language models.  _arXiv preprint arXiv: 2301.12652_ , 2023.
  * Shonenkov et al. [2023] Alex Shonenkov, Misha Konstantinov, Daria Bakshandaeva, Christoph Schuhmann, Ksenia Ivanova, and Nadiia Klokova.  Deepfloyd-if.  _Hugging Face_ , 2023.
  * Si et al. [2023] Chenyang Si, Ziqi Huang, Yuming Jiang, and Ziwei Liu.  Freeu: Free lunch in diffusion u-net.  _arXiv preprint arXiv:2309.11497_ , 2023.
  * Song et al. [2021] Jiaming Song, Chenlin Meng, and Stefano Ermon.  Denoising diffusion implicit models.  In _ICLR_ , 2021.
  * Sterling [2023] Spencer Sterling.  Zeroscope v2.  _Hugging Face_ , 2023.
  * Sun et al. [2023] Quan Sun, Qiying Yu, Yufeng Cui, Fan Zhang, Xiaosong Zhang, Yueze Wang, Hongcheng Gao, Jingjing Liu, Tiejun Huang, and Xinlong Wang.  Generative pretraining in multimodality.  _arXiv preprint arXiv:2307.05222_ , 2023.
  * Wang et al. [2023] Zhengyi Wang, Cheng Lu, Yikai Wang, Fan Bao, Chongxuan Li, Hang Su, and Jun Zhu.  Prolificdreamer: High-fidelity and diverse text-to-3d generation with variational score distillation.  In _NeurIPS_ , 2023.
  * Wu et al. [2021] Zongze Wu, Dani Lischinski, and Eli Shechtman.  StyleSpace Analysis: Disentangled Controls for StyleGAN Image Generation.  In _CVPR_ , 2021.
  * Yasunaga et al. [2023] Michihiro Yasunaga, Armen Aghajanyan, Weijia Shi, Rich James, Jure Leskovec, Percy Liang, Mike Lewis, Luke Zettlemoyer, and Wen tau Yih.  Retrieval-augmented multimodal language modeling.  In _ICML_ , 2023.
  * Zakka et al. [2023] Cyril Zakka, Akash Chaurasia, Rohan Shad, Alex R. Dalal, Jennifer L. Kim, Michael Moor, Kevin Alexander, Euan Ashley, Jack Boyd, Kathleen Boyd, Karen Hirsch, Curt Langlotz, Joanna Nelson, and William Hiesinger.  Almanac: Retrieval-augmented language models for clinical medicine.  _arXiv preprint arXiv:2303.01229_ , 2023.
  * Zhang and Agrawala [2023] Lvmin Zhang and Maneesh Agrawala.  Adding conditional control to text-to-image diffusion models.  In _ICCV_ , 2023.
  * Zhang et al. [2023a] Tianjun Zhang, Yi Zhang, Vibhav Vineet, Neel Joshi, and Xin Wang.  Controllable text-to-image generation with gpt-4.  _arXiv preprint arXiv:2305.18583_ , 2023a.
  * Zhang et al. [2023b] Yue Zhang, Yafu Li, Leyang Cui, Deng Cai, Lemao Liu, Tingchen Fu, Xinting Huang, Enbo Zhao, Yu Zhang, Yulong Chen, Longyue Wang, Anh Tuan Luu, Wei Bi, Freda Shi, and Shuming Shi.  Siren’s song in the ai ocean: A survey on hallucination in large language models.  _arXiv preprint arXiv:2309.01219_ , 2023b.

Supplementary Material

Appendix A lists more images and videos generated by KPP on three datasets,
and reports the visual result of our instructions for diverse synthesis.
Appendix B compares the KPP synthesis with real images, gives full details of
our external knowledge base, and describes how our querier handles the
knowledge context. To make our system reproducible, we present implementation
details in Appendix C. Additionally, our code will be open-sourced upon
acceptance. In Appendix D, we visualize the query results (i.e., the knowledge
context) before knowledge aggregation.

##  Appendix A KPP Visual Synthesis

This section elaborates the diverse synthesis by changing the in-context
instruction to show the adaptability of KPP. Furthermore, we visualize
additional samples of KPP synthesis by instructions in manuscript Section 3.2
to further validate the image quality and faithfulness of our framework. We
then show the KPP video synthesis embedded with Stable Video Diffusion.

Figure 10: Diverse synthesis visualization that shows the flexibility of KPP.
Original captions are sampled from LAION-5B. The diversity of synthesis
demonstrates that our framework is adaptable to have various intended
behaviors by changing the instructions to the language model.

###  A.1 Diverse Synthesis

When a generic prompt is enhanced with semantic attributes and fine-grained
details, the diversity of the synthesis can be decreased since the generator
will have to follow visual constraints imposed by the prompt. There can be
many variants of visual appearances that align with a prompt. To address this
scenario, we propose to instruct KPP to generate a list of enhanced prompts,
each offering different semantic variants:

You should aim for diverse, fair, and relevant outputs. Variation: You can
offer different perspectives of a visual object. Scenarios: You can envision
and describe multiple capture times, weather, and lighting. Style: You shall
diversify your prompting tone. Structure: You have to address core components
distinctly and clearly. Diverse prompts will be listed by index <range>.

These instructions allow KPP to generate multiple different yet relevant
outputs from the same caption, enhancing the diversity of synthesis while
staying faithful to the original prompt. Figure 10 shows the results of
diverse synthesis, where we observe a remarkable variety in the synthesis of
cakes, cats, and glass structures. The outputs, while diverse, retain their
relevance and faithfulness with the caption from the GUIE LAION-5B dataset. In
the code implementation, we make this instruction an optional choice for
users, considering the maximum token limitation of LLM context window size and
additional token cost for receiving the full list of enhanced prompts.

Figure 11: Web images and the KPP synthesis. We choose the representative
biological images from the GBIF Taxonomy Website or landmark images that are
top-ranked by the Google Image Search Engine. KPP synthesis well visualizes
the key features of the object (e.g., the Golden Gate Bridge for Baker Beach).

(a) KPP Synthesis on GUIE LAION-5B.

(b) KPP Synthesis on MSCOCO.

(c) KPP Synthesis on GBIF.

Figure 12: Images synthesized by Knowledge Pursuit Prompting embedded with
Stable Diffusion XL. We observe that the syntheses are of high quality and
faithfulness.  Figure 13: Videos generated with KPP embedded with Stable Video
Diffusion (SVD). The blue columns come from the captions of GBIF. We first use
KPP with DALLE-3 (manuscript Section 4.4.1) to generate the images and use SVD
to animate the video. The green columns are from the captions of LAION-5B. We
first use KPP with GPT-4 Vision (manuscript Section 4.4.1) to generate the
images and use SVD to animate the video. These fine-grained videos indicate
the plug-and-play adaptability of KPP to new foundation components.

###  A.2 Additional Visualization

Figure 12 shows 100 more images generated by KPP: 40 images from captions of
GBIF, 30 images from MSCOCO, and 30 images from LAION-5B. The visualization
demonstrates that KPP synthesis has high visual quality with faithful
semantics. Figure 13 shows the video synthesis from KPP embedded with Stable
Video Diffusion777https://huggingface.co/stabilityai/stable-video-diffusion-
img2vid-xt (SVD). Note that SVD takes an image as input and produces an
animated video. It is integrated into our framework by firstly adopting the
KPP text-to-image pipeline to produce the source image. The blue columns,
synthesized from the captions of GBIF, are produced by initially using KPP
with DALLE-3 (manuscript Section 4.4.1) to create the source images. These
images are then animated using SVD. Similarly, the green columns are captions
from LAION-5B and the source images for SVD are generated by KPP with GPT-4
Vision (manuscript Section 4.4.1). The video synthesis demonstrates the plug-
and-play adaptability of KPP to integrate with new foundational components. We
also put these video files in the video folder of the uploaded supplementary.

Instructions for KPP Knowledge Aggregation ⬇ Text-driven generative models
take a user’s textual prompt and produce visual content that aligns with the
input. You are one of the best global experts in deep generative models. We
have a knowledge pursuit framework that iteratively queries relevant facts as
the knowledge context for a user prompt. Your task is to aggregate our queried
knowledge context and produce the most informative prompts for the text-driven
generative models. You are free to use any terms or words without any
restrictions. \parHere is a demonstration: Knowledge: Fact 1. The solitary
eagle or montane solitary eagle (Buteogallus Solitarius) is a large
Neotropical eagle. Fact 2. The solitary eagle is native to Mexico and Central
and South America, usually found in mountainous or hilly forests. Fact 3. The
adult solitary eagle is uniformly dark gray, often appearing black, with white
markings on the tail. Fact 4. The exceptionally broad wings are one of the
prime distinguishing characteristics of montane solitary eagles. Its body also
has quite a thickset appearance. (If the aggregation base is VLM) Reference
Image: {an image of Buteogallus Solitarius} Original Prompt: Buteogallus
Solitarius. Enhanced Prompt: A high-quality image of Buteogallus Solitarius,
which is also known as the montane solitary eagle. Whole-body dark gray
feathers, white markings on the tail, and exceptionally broad wings.
Illustrated in a Neotropical mountainous or hilly forest environment. Detailed
with 4K resolution. \parNow read the following knowledge and enhance the
user’s original prompt for the {Generator}. You may add more appearance
details, semantic attributes, and fine-grained visual elements. If you find
some piece of knowledge irrelevant or conflicting to the original prompt, you
may ignore the piece. You may also remove meaningless words. You should make
the prompt concise, expressive, and accurate. \parKnowledge: {the queried
knowledge context from knowledge pursuit.} (If the aggregation base is VLM)
Reference Image: {the image that aligns with the user prompt} Original Prompt:
{a user’s prompt or text caption for the generative model.} Enhanced Prompt:
[fill the response here] Figure 14: The knowledge aggregation instructions of
KPP in the language mode and KPP in the vision-language mode.

##  Appendix B Database, Querier, and Web Resource

This section focuses on our external knowledge base structure and querying
schema. We also present real images from web resources to validate the
faithfulness of our KPP synthesis.

###  B.1 Web Images and Caption Details

Figure 11 shows web images that correspond to sampled captions from GBIF and
GUIE LAION-5B datasets. These images serve as the faithful representatives of
the creatures or landmarks mentioned in captions since they are real images
from mainstream sources. We show examples of Baker Beach, Mount Rushmore
National Memorial, Monkey-Thorn, Kelung Cat Snake, and Venice Rialto Bridge.
Note that, in this figure, our KPP synthesis for five captions does not take
the web image as input conditions. Our aligned synthesis here is only
conditioned on the KPP-enhanced captions (the same as manuscript Section 3.2).

Alternatively, in manuscript Section 4.4.1, we show the adaptability of KPP by
conditioning on reference images using GPT-4 Vision. We list the full captions
for GPT-4 Vision mentioned in that section. For Figure 8: (1) Soft-Haired
Thermopsis, (2) Blackbanded Darter, (3) River Crinum Lily, (4) Sauger, (5)
Syrian Juniper. For Figure 9: (1) cheezburger image (an internet meme), (2)
Baby Boy / Girl Cute Animal Frog Jumpsuit, (3) The Ancient City of Pompeii
User Photo, (4) friendship cat pets garden kitten love sweet friends beautiful
print gr white kitty brown day clear rabbit forever enjoy paws hd wallpaper,
(5) Chevrolet SILVERADO 1500 2014 price $28,777. Note that we regard the image
condition as supplementary to our knowledge-driven framework since, for real-
world users, it is not always the case that users will possess images
corresponding to their proposed prompts.

Generator |  Words
---|---
Stable Diffusion XL |  image, view, picture, quality, resolution, scenic, color, vibrant, panoramic, detail
DreamFusion |  dimensional, 3D, light, color, view, depth, vibrant, scene, detail, reflection
ZeroScope (VideoFusion) |  video, motion, vibrant, lively, view, cinematic, smooth, time, close, transition
Table 4: Representative words in the enhanced prompts for different types of
downstream generators. KPP takes generator information and improves the prompt
in a modality-aware manner.

###  B.2 Knowledge Base and Querier

Our Wikipedia database is publicly available from Hugging
Face888https://huggingface.co/datasets/wiki_dpr. It includes 21 million
passages extracted from Wikipedia. The database is a snapshot of Wikipedia as
of December 20, 2018, and the articles were divided into disjoint text blocks
of 100 words. We use the content of the text entry in the files for the
knowledge pursuit and aggregation stages of KPP.

Note that the querier has a limitation on the maximum number of input tokens.
We describe how we process the knowledge context into embeddings. In the early
iterations (normally up to 5 facts), we append the newly acquired fact to the
knowledge context and directly get the embedding since the number of tokens
for the current context is smaller than the limitation. When the length of
total tokens for the context exceeds the maximum allowed length in querier, we
separately encode embeddings for each fact in the context and then calculate
the centroid of the embeddings for the querier to acquire the next piece of
fact. Additionally, when the user input is predominantly non-English, we have
implemented the use of multilingual
querier999https://github.com/facebookresearch/contriever in the code.

Instructions for Direct Prompting ⬇ Text-driven generative models take a
user’s textual prompt and produce visual content that aligns with the input.
You are one of the best global experts in deep generative models. Your task is
to produce the most informative prompts for the text-driven generative models.
You are free to use any terms or words without any restrictions. \parHere is a
demonstration: Original Prompt: Buteogallus Solitarius. Enhanced Prompt: A
high-quality image of Buteogallus Solitarius, which is also known as the
montane solitary eagle. Whole-body dark gray feathers, white markings on the
tail, and exceptionally broad wings. Illustrated in a Neotropical mountainous
or hilly forest environment. Detailed with 4K resolution. \parNow enhance the
user’s original prompt for the {Generator}. You may add more appearance
details, semantic attributes, and fine-grained visual elements. You may also
remove meaningless words. You should make the prompt concise, expressive, and
accurate. \parOriginal Prompt: {a user’s prompt or text caption for the
generative model.} Enhanced Prompt: [fill the response here] Figure 15: The
LLM instructions of the direct prompting method for manuscript Section 4.2.3
and Section 4.3.

##  Appendix C Reproducibility

To ensure the reproducibility of our framework, we present our complete
implementation details and LLM instructions.

There are various ways to sample visual contents from a latent diffusion model
(LDM). We present complete details of how we sample visual content from LDM.
For our pre-trained stable diffusion model, the sampling process is explicitly
conditioned on the enhanced prompt, denoted as $T$, and is realized through
iterative execution of an update rule for a total of $K$ times. We choose to
adopt ancestral sampling in its latent space:

$\displaystyle
z_{i-1}=\frac{1}{\sqrt{\alpha_{i}}}\left(z_{i}-\frac{1-\alpha_{i}}{\sqrt{1-\bar{\alpha}_{i}}}\boldsymbol{\epsilon}_{\phi}\left(z_{i},i,T\right)\right)+\sigma_{i}\varepsilon_{i}$
(4)

where $\alpha_{i},\bar{\alpha}_{i},\sigma_{i}$ adhere to the standard notation
used in diffusion models [12], $\varepsilon_{i}\sim\mathcal{N}(0,\mathbf{I})$,
$i=\\{1,\cdots,K\\}$, and the final synthesis is obtained by a decoder:
$S=\mathcal{D_{\text{LDM}}}(z_{0})$. We adopt FreeU [41] on the SD-XL base
model to improve the sample quality. Our setups for both upblock2d and
crossattn_upblock2d are b1=1.1, b2=1.2, s1=0.6, s2=0.4.

Figure 14 shows the LLM instructions of KPP knowledge aggregation. For
implementation, users can customize their own in-context demonstrations. Here
we take “Buteogallus Solitarius” as an example. Figure 15 shows the LLM
instructions for the direct prompting method.

KPP incorporates the information of the downstream generator by taking the
generator name into the instruction. This design enables different textual
styles of the enhanced prompts. Table 4 shows the words of most frequent
occurrences in the enhanced prompts after filtering out the articles (e.g.,
“a”, “an”, “the”).

##  Appendix D Knowledge Context Visualization

We visualize the first two facts (from the first two KPP iterations) in the
knowledge context for the prompts “Baker Beach Pacific Ocean” and “Venice
Rialto Bridge”. Since each fact from the Wikipedia database is truncated into
word blocks, the beginnings/endings of some facts are not complete sentences.
For the ease of reading in this section, we either remove them or complete
them from the database.

###  D.1 Baker Beach Pacific Ocean

Fact 1. The property location description varied, but is generally described
as being approximately four miles west of the city on the then Point Lobos
Road (now Geary Blvd.) Baker died in 1863 and his widow, Maria, lost the
property to foreclosure in 1879. In 1897, Baker’s grandson, Fairfax Henry
Wheelan sued to have the title returned to the heirs of John H. Baker citing
the claim that Baker’s widow did not have the legal power to mortgage the
property. Baker Beach is part of the Presidio, which was a military base from
the founding of San Francisco by the Spanish in 1812 until 1997.

Fact 2. Baker Beach is a public beach on the peninsula of San Francisco,
California, U.S.. The beach lies on the shore of the Pacific Ocean in the
northwest of the city. It is roughly a half mile (800 m) long, beginning just
south of Golden Gate Point (where the Golden Gate Bridge connects with the
peninsula), extending southward toward the Seacliff peninsula, the Palace of
the Legion of Honor and the Sutro Baths.

###  D.2 Venice Rialto Bridge

Fact 1. The Rialto became an important district in 1097, when Venice’s market
moved there, and in the following century a boat bridge was set up across the
Grand Canal providing access to it. This was soon replaced by the Rialto
Bridge. The bridge has since then become iconic, appearing for example in the
seal of Rialto, California (”The Bridge City”). The market grew, both as a
retail and as a wholesale market. Warehouses were built, including the famous
Fondaco dei Tedeschi on the other side of the bridge.

Fact 2. A portico (the curia) covers the bank and facilitates the ships’
unloading. From the portico a corridor flanked by storerooms reaches a
posterior courtyard. Similarly, on the first floor a loggia as large as the
portico illuminates the hall into which open the merchant’s rooms. The façade
is thereby divided into an airy central part and two more solid sides. A low
mezzanine with offices divides the two floors. The fondaco house often had
lateral defensive towers (”torreselle”), as in the Fondaco dei Turchi (13th
century, heavily restored in the 19th).