#  : Supporting Reference Recombination for Graphic Design Ideation with
Generative AI

DaEun Choi<EMAIL_ADDRESS>KAISTDaejeonRepublic of Korea ,  Sumin Hong
<EMAIL_ADDRESS>Seoul National University of Science and
TechnologySeoulRepublic of Korea ,  Jeongeon Park<EMAIL_ADDRESS>DGISTDaeguRepublic of Korea ,  John Joon Young Chung<EMAIL_ADDRESS>MidjourneySan FranciscoCA, USA and  Juho Kim<EMAIL_ADDRESS>KAISTDaejeonRepublic of Korea

(2024)

###### Abstract.

Graphic designers often get inspiration through the recombination of
references. Our formative study (N=6) reveals that graphic designers focus on
conceptual keywords during this process, and want support for discovering the
keywords, expanding them, and exploring diverse recombination options of them,
while still having room for their creativity. We propose , a system with
generative AI pipelines that helps users discover useful elements from the
reference image using keywords, recommends relevant keywords, generates
diverse recombination options with user-selected keywords, and shows
recombinations as sketches with text descriptions. Our user study (N=16)
showed that helped users discover keywords from the reference and generate
multiple ideas based on them, ultimately helping users produce more design
ideas and higher self-reported creativity, compared to the baseline system
without generative pipelines. While was effective in ideation, we discussed
how can be extended to support other types of tasks in creativity support.

Creativity support tool, Graphic Design ideation, Reference recombination,
Machine Learning

††copyright: acmcopyright††journalyear: 2024††doi:
XXXXXXX.XXXXXXX††conference: arXiv; December, 2023; ††booktitle: arXiv
December, 2023††ccs: Human-centered computing Interactive systems and tools

Figure 1. Three main features of supports for enhancing reference
recombination. (Left) Extraction of 4 different types of keywords from the
reference image. (Center) Mood board showing the reference image with the
user-selected keywords and keyword recommendations. (Right) Merging selected
keywords to generate diverse recombination options and showing them as
sketches and descriptions.

This figure briefly summarizes the 3 main features supported by . The left
part depicts the keyword extraction feature, where a reference image is shown
and the system automatically extracts the keywords into 4 aspects of the
image: subject matter, action & poses, theme & mood, and arrangement. For
example, there is Santa, a Christmas tree, and elves included in the subject
matter, and decorating, holding a gift for action for & pose, and festive,
holiday for theme & mood. The center part describes the mood board that shows
those user-selected keywords along with the reference images and recommends
more relevant keywords. The keywords that the user selected are shown next to
the image, while some keywords (e.g. pool party) are recommended based on
selected words. On the right side, the system merges keywords that the user
selected. This includes Santa, turtle, festive, holding a gift, and an
arrangement keyword from the specific reference image. The system merges them
into descriptions (e.g. A group of turtles is wearing Santa hat and holding a
gift.) and line-drawing sketches of them.

##  1\. Introduction

References play a crucial role in creative thinking, serving as valuable
sources for graphic designers to both grasp the landscape of the existing
ideas and ignite novel ones (Herring et al., 2009; Lee et al., 2010; Ritchie
et al., 2011; Müller-Wienbergen et al., 2011). They offer diverse visual,
conceptual, and functional stimuli, allowing individuals to explore various
creative directions and draw lessons from established successful examples
(Bonnardel, 1999). One effective method to generate new ideas with references
is making a combination of existing examples, which is often called
combinatorial creativity (Campbell, 1960; Simonton, 2003; Yu and Nickerson,
2011; Chan et al., 2016). In practice, this is often done through reference
recombination, which is the process of extracting the elements or aspects of
multiple references, considering connections of the extracted elements with
those from other references (Herring et al., 2009), and blending those to
ultimately gain novel design ideas (Boden, 1998).

However, each step of recombination requires significant effort from
designers. To discover sources for recombination, designers need to dissect
the references into individual elements and analyze them to figure out which
combinations of elements are worth mixing. Additionally, to find effective
methods of blending those elements into a new design idea, they must engage in
exploratory efforts by drawing multiple sketches. This takes a long time and
multiple iterations, especially for those who are less experienced in the
design process, as they have difficulty identifying various factors from
references and integrating references from disparate domains compared to
professionals (Bonnardel and Marmèche, 2005).

Previous research has provided support for these individual steps. Several
approaches have been proposed (Kang et al., 2021; Ivanov et al., 2022; Hope et
al., 2022) to decompose the references or show connections between them,
aiding users in identifying the sources for recombination. However, these
approaches do not guide how to incorporate extracted elements into a design.
Also, many approaches have attempted to help users blend different concepts or
images into a novel one (Chilton et al., 2019b, a; Zhao et al., 2020; Wang et
al., 2023). However, these approaches primarily emphasize generating precise
combinations that effectively incorporate all elements harmoniously, rather
than aiming to produce diverse combinations for exploratory purposes. Another
thread of research focuses on searching by genetic recombination (Kim et al.,
2022; Chouchoulas and Day, 2007; Turrin et al., 2011; Zaman et al., 2015), but
these techniques were focused on widening the range of the design exploration,
rather than offering inspiration on how to effectively combine specific design
elements.

Through a formative study with six novice graphic designers and design
students, we aimed to understand the process of reference recombination and
identify challenges that they have during the process. There were two
distinctive stages of ideation, 1) conceptual ideation, aiming to effectively
convey the design topic, and 2) visual ideation, deciding style-wise details
on top of the selected concept. We decided to focus on the conceptual one as
the recombination of references tends to be more prevalent in this stage.
During the conceptual ideation, designers extracted four types of elements
from the reference—subject matter, action & pose, theme & mood, and
compositional aspects (arrangement). Then, participants tried to brainstorm
more elements related to the extracted ones and combine them in several ways.
However, due to the high effort required to manually recombine them, they were
concerned that they couldn’t try out all of the possible methods. They also
mentioned that support to help ideation should not be in an overly completed
form as it can diminish their own input. With these observations, we propose
four design goals for a reference recombination support system: (1) enable
users to effortlessly specify the four types of conceptual elements from the
reference image, (2) recommend relevant elements, (3) provide many
recombinations as much as possible, and (4) intentionally keep the generated
output partially unfinished, fostering a space for user creativity.

Based on the design goals, we propose , a system that supports the ideation of
visual designers by helping them extract elements from the reference images
and generate a wide range of recombinations of those elements. Using , users
can easily discover and select elements from the reference image based on the
four element types and can get recommendations for more relevant keywords.
Once the user has chosen the keywords to combine, they get a variety of
recombination options presented as pairs of sketch images and one-line
descriptions. We introduced novel pipelines with generative models to automate
the extraction of keywords from images, generate recombination options, and
transform them into descriptions and sketches.

We conducted a within-subjects study with 16 design students, and compared
with the baseline, which consisted of a mood board with manual keyword notes,
a layout diffusion model, and ChatGPT. Results showed that could support the
two stages of the reference recombination process, discovering elements and
generating design ideas by recombining them. Participants were also able to
produce more design ideas in a given time, and perceived that helped them come
up with more creative sketches compared to the baseline. They emphasized that
’s features were especially beneficial when they wanted to explore ideas
vastly different from their initial concepts. We compared the creativity
support of with baseline and proposed an opportunity to design a comprehensive
recombination support tool that could support a wide spectrum of design needs
and situations. We also found that the sketch-based output leads users to get
more stimulus for their creativity by letting users further imagine through
the low-fidelity. Finally, we discussed the generalizability of in terms of
the designer’s expertise, collaborative settings, and different domains of
design.

This paper presents the following contributions:

  * •

, a system that aids graphic designers in their ideation by assisting them in
extracting elements from reference images and suggesting a wide range of
recombinations of those elements.

  * •

Computational pipelines with generative models that automate the extraction of
keywords from images and generate recombination options in descriptions and
sketches.

  * •

Findings from a user study (N=16) about how can aid designers in each step of
recombination, leading to the generation of more design ideas and participants
to perceive their ideas as more creative.

##  2\. Related Work

This work aims to support designers in their reference recombination process
for creativity. In this section, we review previous literature on (1) how
references are used in graphic design ideation, (2) how recombination is
employed for creative thinking, and (3) previous generative AI approaches for
creativity.

###  2.1. Reference in Graphic Design Ideation

The creative process begins by collecting relevant inspirational materials
from various sources (Shneiderman, 2000; Eckert and Stacey, 2000). Designers
leverage these collected examples to gain a comprehensive understanding of the
problem space. As the process advances into idea generation, these compiled
examples play a pivotal role in fostering creativity, igniting new ideas
through analogical thinking (Holyoak and Thagard, 1996; Goel, 1997).
Recognized as one of the most challenging phases in the entire design process,
previous research on creativity-supporting tools has extensively concentrated
on enhancing this ideation step (Frich et al., 2019b). Previous research
demonstrated that designers get valuable insights and inspirations in
different ways (Herring et al., 2009), and many studies have delved into the
significance of these references in design thinking, showing their potential
to stimulate creativity and innovation (Bonnardel, 1999; Siangliulue et al.,
2015).

One of the primary approaches to support idea generation with references is to
help designers see diverse references. Exploring diverse ideas is crucial in
terms of preventing fixation (Jansson and Smith, 1991), in which a designer
becomes overly fixated on a single concept, potentially hindering creativity
and innovation. Therefore, Zhang et al. (Zhang and Banovic, 2021) have
utilized a Generative Adversarial Network (GAN) for exploring diverse images,
while Matejka et al. (Matejka et al., 2018) developed the Dream Lens to assist
in exploring generative 3D design solution space.

Another avenue of research is to help designers manage their inspirations
drawn from references, particularly through the use of mood boards (Eckert and
Stacey, 2000). Prior research demonstrated that building a mood board can
enhance the comprehension and interpretation of ephemeral elements in design
(Garner and McDonagh-Philp, 2001), which is beneficial for both defining and
resolving design challenges (Cassidy, 2011), and ultimately leading to a boost
in creativity (Lucero, 2015). Therefore, many computational systems have been
proposed to help designers to build an interactive mood board, such as Funky
Wall (Lucero et al., 2009), SemanticCollage (Koch et al., 2020), and May AI
(Koch et al., 2019).

While this paper primarily focuses on the recombination of references, we have
integrated two significant insights from prior research about design
references. First, we emphasize the importance of offering users a diverse
array of images to support their creative process. Second, we have
incorporated the concept of a mood board as a valuable tool for organizing
references within our system.

###  2.2. Recombination for Creative Thinking

In the creative thinking process, new ideas often come through the combination
of the existing examples (Campbell, 1960; Simonton, 2003). It was shown that
creativity often arises from forging new associations among previously
unrelated frames (Koestler, 1964). There are two crucial components in this
process: recognizing the differences between existing concepts and blending
them (Nagai et al., 2009; Boden, 1998). Also, the diversity of the given
examples is important for building novel associations between them during this
process (Mumford et al., 1991). Observations of designers’ creative processes
showed that designers often maintain multiple small components and keep
employing them to generate new variations through a process akin to
recombination (Frich et al., 2019a). Many computational systems were also
proposed for building recombinations and verified to be effective in tasks
such as chair design (Yu and Nickerson, 2011) or text-based ideation (Chan et
al., 2016).

One practical implementation of this concept in terms of design ideation is
genetic exploration. Genetic exploration involves generating novel solutions
by merging elements from preexisting designs to widen the range of references.
This approach has been applied in diverse domains such as garden design (Kim
et al., 2022), 3D modeling (Chouchoulas and Day, 2007; Pilat and Jacob, 2008),
architecture (Turrin et al., 2011), and 2D graphics (Zaman et al., 2015).
However, the primary aim of this approach is to enrich the reference in the
information-gathering stage by utilizing existing references, rather than
supporting designers to generate their own ideas from those recombinations in
the next stage.

In recombination, it is also critical to decompose the reference and get
elements that are worth combining. Therefore, several tools have been
developed to facilitate this process, especially by automatically decomposing
the original source and showing the fine-grained aspects. CollageMachine
(Kerne, 2000) decomposes websites and makes them into an interactive collage.
MetaMap (Kang et al., 2021) provides a decomposed view of the reference image
into three dimensions (semantic, color, and shape) and lets users explore more
references using it. Hope et al. (Hope et al., 2022) divides the product’s
information into fine-grained functional parts, allowing users to combine the
inspiring part. MoodCubes (Ivanov et al., 2022) offers a new mood board
experience by decomposing multimedia references into constituent elements and
using it to provide suggestions for new inspirational materials. They may not,
however, directly discuss the exact strategies for merging these outputs as a
new design idea. On the other hand, VRicolage (Stemasov et al., 2023) enables
users to decompose objects into different parts, motions, or colors, and mix
them. However, this process was more of utilizing collected assets, rather
than generating a new idea from recombination.

Additionally, there were many previous approaches to support the process of
mixing the reference images or concepts. For example, VisiBlends (Chilton et
al., 2019b) and VisiFit (Chilton et al., 2019a) introduced a novel pipeline to
blend two objects to convey integrated meaning. ICONATE (Zhao et al., 2020)
supports users to generate a new icon by mixing different icons, and PopBlends
(Wang et al., 2023) automatically suggests conceptual blends of reference
images. FashionQ (Jeon et al., 2021) supports this blending in the domain of
fashion design. Artinter (Chung and Adar, 2023a) supports recombining style
elements from the reference to facilitate communication. Nevertheless, these
approaches primarily focus on seamlessly merging entire references rather than
breaking them down to the element level. This approach may not fully align
with the creative recombination process, which often begins by identifying
specific elements to combine within the provided examples. 3DALL-E (Liu et
al., 2023) presents a recombination workflow for generating a new idea, which
suggests diverse low-level keywords and combines them into a prompt for text-
to-image models. This approach, however, differs from our definition of
reference recombination as the keywords are from LLM’s understanding of the
world rather than the design references.

###  2.3. Generative AI Approaches for Creativity

Before looking into AI systems for creativity, it’s important to know how
visual designers perceive AI for supporting their design tasks. Ko et al. (Ko
et al., 2023) looked into how graphic designers use large-scale text-to-image
generation models (LTGMs) to help with their creative works and suggest the
design guidelines for building creative supporting systems using them.

Recently, diffusion-based techniques (Rombach et al., 2022; Ramesh et al.,
2022; Nichol et al., 2022) and CLIP embedding (Radford et al., 2021) enable
people to represent their ideas to visual materials quickly and easily using
text prompting. There were also many previous approaches to incorporate inputs
with additional modalities, such as layout (Li et al., 2023b; Zheng et al.,
2023; Chen et al., 2023) or sound (Lee et al., 2023; Sung-Bin et al., 2023).
Techniques to add extra conditions and styles for more granular control have
been proposed as well (Zhang et al., 2023; Mou et al., 2023). There is also a
thread of research on modifying the generated image to align with user intent
better, such as adding style (Gatys et al., 2016), latent-space manipulation
(Karras et al., 2019, 2020), human-prompt editing (Brooks et al., 2023), and
editing a specific part in generated images (Ruiz et al., 2023; Gal et al.,
2022).

With those novel ML techniques, the creative landscape is also continuously
being reshaped, offering innovative solutions and enriching the artistic
experience. Promptify (Brade et al., 2023) stands out as an iterative prompt
refine tool, letting users get closer to their intended result by clearing
unintended outcomes. PromptPaint (Chung and Adar, 2023b) allows users to go
beyond language to mix prompts to express challenging concepts, supporting for
iterative shaping of the image. On the other aspect, the interplay between
humans and AI is also fast evolving. The concept that Karimi et al. (Karimi et
al., 2020, 2019) proposes is a generative AI system that helps designers by
collaborating during the design phase, instead of taking over the design
process. Oh et al. (Oh et al., 2018) and Framer (Lawton et al., 2023) proposed
a user-AI collaborative interface to allow a co-drawing experience.

While there has been a lot of research on expressing user intention to ML
models accurately to get a better image or collaborate with AI during the
design execution phase, it is less relevant to the ideation task of expanding
the variety of ideas. Especially, it is still unknown how to design
interaction with generative AI models to inspire graphic designers by
recombining the references.

##  3\. Formative Study

To understand how designers recombine design references for ideation and what
challenges they encounter during the process, we conducted a formative study.
As indicated by prior research (Bonnardel and Marmèche, 2005), novice tends to
encounter more challenges in getting inspiration from references and combining
them. Therefore, we recruited novice designers as they are expected to have an
understanding of the overall design process but still struggle with many
challenges in ideation through recombination. Six participants (6 female; age
M=25.3 and SD=3.32) were recruited through an online recruitment posting. Two
were professional UI/UX designers with 1 year of experience each, and one was
a freelance brand designer. Three were students majoring in industrial design,
with two at the graduate level and one in their fourth year of undergraduate
studies. All participants reported that they had experience in at least three
different graphic design projects before.

###  3.1. Study Process

The study included two parts: (1) an observation on reference searching and
idea sketching, and (2) a semi-structured interview. For the first part of the
study, the task was to draw an illustration for one of three different design
topics: ”Tourism service for kids”, ”Pet grooming service”, or ”Eco-friendly
restaurant”. We let participants choose one of these topics. They were first
given 10 minutes to search for reference images that they wanted to utilize in
their ideation process. For each reference they chose, they were asked to
describe what aspects of the references they found appealing. Then,
participants sketched their design ideas using their preferred method for 30
minutes. Three participants used pen and paper to sketch their ideas, while
the other three used a tablet and digital drawing software. They were asked to
generate at least three different design ideas, and they were prompted to
describe how they integrated their references into each sketch. After that, we
conducted a semi-structured interview to ask about their challenges in
generating multiple ideas using references.

Following each study, two of the authors independently coded the recombination
methods employed by participants in their ideation tasks and the semi-
structured interview results. The coded data were then discussed
collaboratively. After conducting six studies, codes were saturated and no
further study sessions were conducted.

###  3.2. Findings

Through the observation of participants’ design processes, we discovered that
the recombination of different references primarily occurs during the initial
stages of design ideation, with a specific focus on the conceptual aspects of
the reference images rather than the visual elements. We identified four
distinct categories of elements employed in this process. We also found
specific challenges associated with it and observed that the system supporting
this process should reserve a degree of incompleteness to encourage
creativity.

####  3.2.1. Early-Stage Design Ideation Focuses on Conceptual Aspects

All six participants said that they refer to the references in two distinct
stages: conceptual ideation and visual development. During the conceptual
ideation stage, designers focus on elements that could effectively convey the
design topic, such as objects or mood. After looking at those elements, they
generated multiple drafts by combining them in several ways. On the other
hand, the visual development stage revolved around adding visual details like
color and texture to complete the sketches derived from the conceptual
ideation. During this stage, designers often had a clear direction in mind and
referred to a specific set of references that aligned well with their chosen
direction, with less emphasis on exploring different recombinations of diverse
references. This aligns with findings from previous research (Holinaty et al.,
2021), which indicates that artists engage in a spectrum of reference usage in
their creative process, ranging from detailed recreation (visual development)
by tracing images to interpretive inspiration for high-level components
(conceptual ideation). In summary, designers recombine references primarily
for conceptual ideation, which is usually the first step of the design
ideation, suggesting that a system supporting the reference recombination
process should focus on how to facilitate this early-stage step.

####  3.2.2. Types of Elements Used for Recombination

During the conceptual ideation phase, participants tried to extract specific
elements from references and incorporate them into their design concepts. They
employed a variety of approaches for this. The simplest approach that was
observed in all participants was to utilize objects in a reference in their
own sketch. For example, for drawing an illustration for ”Tourism service for
kids”, one participant took an image of a paper plane from a reference to
convey the image of playful children and a tour service at the same time. Five
participants extracted the abstract semantic meaning or overall theme conveyed
by references. For example, after looking at an image of a person holding a
pamphlet and deep in thought, one participant said that the keyword
”imagination” could effectively capture the concept of kids. So they came up
with a design concept about children imagining various travel destinations.
Another approach observed in three out of six participants was to draw the
action of a character in a reference. For instance, from a reference depicting
an animal and a person holding hands, a participant got the concept of
children holding hands together. Lastly, five participants extracted the
composition from the references. For example, by looking at a reference where
leaf shapes were arranged together to form a shovel, one participant came up
with the idea of using multiple tree trunk shapes to represent the structure
of a building.

####  3.2.3. Challenges During Finding Elements

We identified some opportunities to support the process of extracting elements
from the references. There were many cases where the elements that designers
initially found appealing in the reference search phase differed from the
elements they eventually utilized in their design concepts. In the interview,
participants said that upon closer examination of the references, they
discovered new elements of interest and incorporated them. This means that
designers couldn’t immediately extract elements upon viewing the reference and
it often required several examinations to uncover such elements, which is
time-consuming.

Another observation was that participants often came up with new keywords
based on what they had already found for further brainstorming at the element
level. For instance, P3 identified ”toy blocks” from one reference and ”train”
from another reference, then came up with the new keyword ”toy train” and
incorporated it into their final idea. However, this process was often more
challenging compared to finding elements directly from the reference images.
P4 highlighted an opportunity for system support for this by mentioning that
“I usually talk with others about my ideas, which leads me to discover new
keywords related to the original one. Just like that, I think it would be nice
if the system could recommend a new keyword to expand the design idea I have
now.”

####  3.2.4. Challenges During Recombining Elements

After finding out the elements from the references that they want to utilize
in their own design ideas, another challenge became apparent. While there can
be numerous ways to combine these elements, participants are often frustrated
as they can’t sketch out all the possibilities to determine if they are
viable. Three out of six participants expressed anxiety about not being able
to consider all possible combinations. P3 stated, “I always feel anxious that
there might be a better way, but I can’t think of it.” P6 also mentioned that
“The more options I explore, the more I become confident about my final design
idea. I want some faster way to explore alternatives as much as possible.”
Four out of six participants said that they rely on their imagination to
envision numerous recombination possibilities within their minds, as it is too
time-consuming and effortful to sketch out all. However, two participants
expressed frustration that, although combinations seemed good in their minds,
they might not come together as effectively in actual sketches.

####  3.2.5. System Support should be Incomplete

Designers tend to deliberately exclude visual details during conceptual
ideation. Participants said that when recombining the references for
conceptual inspiration, they did not pay attention to visual details, and
several participants said that they even needed to intentionally exclude those
details. P2 stated, “When combining different concepts, colors and textures
often become messy, so I deliberately use the same brush for all elements.” P3
agreed this with another viewpoint, by expressing concern about becoming
overly fixated on frequently recurring visual details while exploring
conceptual recombinations. We also asked the participants if they could get
recommendations for different recombination options, what form would be
preferred. Four participants mentioned that they would prefer incomplete
outputs such as a sketch or even a textual description of the idea so they
could focus on the concept itself. The main reason for this was the concern
that the model would compromise their creativity or lead them to perform
unintentional plagiarism.

###  3.3. Design Goals

Based on the findings of the formative study, we identified four design goals
to build a system to support designers’ reference recombination process during
early-stage ideation.

  * DG 1.

Facilitate Element Extraction from References. To help users efficiently find
the elements that would be used for the recombination, the system should help
users discover the overlooked elements. Based on our observation, elements
that users want to extract from references are (1) subject matters (e.g.,
objects, characters, landscapes), (2) action & pose, (3) theme & mood, and (4)
arrangement.

  * DG 2.

Suggest Diverse and Relevant Elements. To help users explore more elements on
top of what they found from the references, the system should provide some
recommendations of relevant elements that users might like.

  * DG 3.

Generate Diverse Recombination Options. To help users explore diverse
recombination possibilities, the system should show users a varied range of
recombination options and reduce their anxiety over not considering all
feasible combinations. This goal highlights the system’s ability to propose
combinations that the user might not have considered independently.

  * DG 4.

Present Recombination in an Incomplete Format. To align with designers’
preference for conceptual sketches over highly detailed artwork during the
initial ideation phase, the system-generated outputs should be intentionally
incomplete, such as sketches. This emphasizes the importance of allowing users
to inject their own creativity into the images.

##  4\.

With derived design goals, we implemented (Figure 2), an AI-powered design
tool that supports graphic designers in coming up with novel design ideas by
recombining reference images in early-stage conceptual ideation. mainly
consists of a mood board where users can import reference images and select
what they like about the reference. When the image is imported, the system
extracts keywords from it according to the four categories defined in the
formative study (Section 3.2.2). This helps users easily discover and select
keywords (DG 1). Selected keywords are then displayed on the mood board along
with the images. offers further keyword recommendations based on the keywords
users have added on the board or their specific selections (DG 2). Also, when
the user chooses a set of keywords that they want to recombine, the system
generates multiple drafts with diverse ways of combining them (DG 3). All of
the system-generated images are produced in line sketches with one-line
descriptions so that users can further reinterpret by themselves (DG 4).

Figure 2. Screenshot of . (a) Keyword Extraction Panel: The system
automatically extracts keywords in four categories (subject matter, action &
pose, theme & mood, and arrangement) from the reference image. Users can
select these keywords, or add keywords manually. (b) Interactive Mood Board
with Keyword Suggestion Panel: Users can organize the reference images on the
mood board, along with the selected keywords. Users can also import system-
recommended keywords shown below, suggested based on all keywords on the board
or the keywords that users selected on this mood board. (c) Keyword Merge
Panel: When users select keywords that they want to recombine on the mood
board, the system generates sketches and their respective descriptions
including all of the selected keywords. Users can view more generated sketches
by clicking the ”More Sketches” button.

This is the figure of the overall system scene. (a) On the left, there is a
keyword extraction panel. When the user uploads a reference image, the system
automatically extracts keywords in four categories (subject matter, action &
pose, theme & mood, and arrangement) and displays them here. Users can select
the extracted keywords or add keywords manually. (b) In the center, there is
an interactive mood board where users can organize the reference images on the
mood board, and the selected keywords are placed near the corresponding image.
At the bottom, there is a keyword recommendation panel, where users can
explore the system-recommended keyword suggestions, which are suggested based
on all keywords on the board or the keywords that the user selected on this
mood board. On the right side, there is a keyword merge panel. When the user
selects keywords that they want to recombine on the mood board, the system
mixes them and generates three pairs of sketches and their respective
descriptions including all of the selected keywords. There is a ”More
Sketches” button for each pair, which the user can view additional sketches
for the descriptions by clicking.

###  4.1. User Scenario

To demonstrate our system, we show how Sarah, a junior illustration designer,
uses to generate ideas for her design project. Sarah recently accepted a new
commission to draw an illustration for the cover of a children’s book titled,
”A Christmas Dinner in the Underwater World”. As the given topic is an unusual
combination of two themes, she felt difficulties in getting inspiration from
the references and mixing them to come up with ideas, so she decided to
explore references with the help of .

####  4.1.1. Getting User Inputs on the Design Reference

Sarah first uploads 10 reference images she got from her client into . Looking
through the references, she is intrigued by the one where two scuba divers
swim with a turtle. When she chooses the image, shows some keywords that can
be found in the image, divided into four categories – subject matter, action &
pose, theme & mood, and arrangement (Figure 2 (a)). As she found the scuba
diver concept interesting, she clicks on the subject matter category. In the
keyword list, she finds ”scuba diver” and clicks it. She also finds ”coral
reef” on the list, which she didn’t recognize before. She looks at the
references again and thinks coral reefs would look great in her illustration,
so she clicks ”coral reef” as well. Similarly, she looks through the list of
the keywords in the ”action & pose” and ”theme & mood” categories and selects
”swimming” and ”adventure” from each list. She also liked the overall
composition of the image, so she clicks its ”arrangement” as well. She also
works on selecting keywords that she likes on other references.

####  4.1.2. Mood Board with the User-selected Keywords & Keyword
Recommendation

As Sarah selects the keywords she finds useful from each image, the canvas of
the offers a dynamic mood board that shows the references with user-selected
keywords, capturing her creative goal and preferences (Figure 2 (b)). As she
freely moves the images to organize them, the selected keywords move along
with the image. By looking at the keywords, Sarah wants to come up with
additional ideas for character actions that align with the adventurous theme,
similar to swimming or scuba diving. Therefore, she selects ”subject matter:
scuba diver”, ”action & pose: swimming”, and ”theme & mood: adventure” to get
system recommendations with these keywords. shows a set of keywords, such as
“action & pose: exploring sunken ship”, and “subject matter: anchor”. Sarah
finds those keywords valuable, so she drags them into the mood board.

####  4.1.3. Recombining Design References using Keywords

From the set of keywords on the mood board, Sarah now selects some keywords
that she wants to include in her design and uses the system to make a first
draft. She selects ”Christmas tree” and ”Santa Claus” for a Christmas dinner
theme, and ”whale”, ”swimming”, ”exploring the sunken ship”, and ”adventure”
for the underwater theme. She also selects the ”arrangement” of one of the
images with an interesting composition.

After clicking the merge button, generates three different drafts, each
showcasing a unique and different way of incorporating these keywords (Figure
2 (c)). Each draft contains a one-line text description of the image concept
and a sketch-style generated image based on the description and the
arrangement that Sarah selected. Sarah appreciates the generated results as
the way each draft combining the keywords would be difficult to think of by
herself and she likes how all three drafts are distinct from each other. Also,
the sketch format allows her to further imagine design concepts rather than
fixating on the concept and details in the generated results.

Among the drafts, Sarah finds one description interesting: ”Santa Claus goes
on an underwater adventure on a sled pulled by a whale.” However, she feels
dissatisfied with the generated sketch, so she presses the ”More Sketches”
button. Then, generates five more sketches with the same description, but in a
slightly different way. She gets some good design ideas from the new sketches
and starts working on her draft.

###  4.2. Technical Details

was built as a web-based system with a ReactJS111https://react.dev/-based
front-end client and a Flask222https://flask.palletsprojects.com/-based back-
end server. We implemented ML pipelines for extracting the keywords from the
references and merging keywords into recombinations. The technical details of
these pipelines are discussed in the following sections. Some examples of
outputs from the pipeline are presented in Figure 9 in the Appendix.

Figure 3. Technical pipeline of . (a) Keyword extraction from image: The
caption generated from the image captioning model goes into the LLM to extract
subject matter, action & pose, and theme & mood. The segmentation model is
used to detect the image’s arrangement. (b) Keyword-based image generation:
the LLM generates descriptions based on the given keywords, and the layout
variator generates similar arrangements. The image generation model generates
the image, and the style transfer model converts this into a sketch.

This figure depicts the technical pipeline of . (a) Keyword extraction from
image: From the input reference image, the image captioning model generates
captions for the image and the LLM is used to extract subject matter, action &
pose, and theme & mood from the captions. In parallel, the segmentation model
is used to detect the arrangement. (b) Keyword-based image generation: the LLM
first generates three descriptions using the given keywords, and the layout
variator transforms the original arrangement into three similar arrangements.
Then, the image generation model uses the description and layout to generate
the image, and the style transfer model is used to convert this into a sketch.

####  4.2.1. Extracting Keywords from Reference Images (Figure 3. (a))

Based on the findings from our formative study, our pipeline is designed to
extract keywords from a provided reference image in four categories: subject
matter, action & pose, theme & mood, and arrangement. To achieve this, we
follow a multi-step process.

To identify the subject matter, action & pose, and theme & mood within the
image, we initially employ an image captioning model BLIP-2 (Li et al., 2023a)
to generate textual descriptions of the image contents. For a comprehensive
understanding of the entire image, we divide it into $3\times 3$ segments and
generate captions for each segment as well as the whole image. These
captioning results are then processed by GPT-4 (OpenAI, 2023), a Large
Language Model (LLM), to extract lists of subject matter, action & pose, and
theme & mood present in the image captions. Prompts used for this are in the
Appendix A.1.

For the arrangement, we utilize the Segment Anything model (Kirillov et al.,
2023) to identify the top ten prominent segments within the image. We then
used the approach from LLM-grounded Diffusion (Lian et al., 2023) to score
these segments. Bounding boxes around these segments provide information about
the image’s overall structure, such as where the items are placed and where
large negative spaces are.

Additionally, for generating the recommendations of the relevant keywords, we
also used GPT-4, and the prompts used for this are shown in the Appendix A.2.

####  4.2.2. Generating Recombinations (Figure 3. (b))

When the user selects a set of keywords to generate a new recombination, our
system generates a range of options to mix those keywords.

The system first generates three textual descriptions that encompass the
selected subject matter, action & pose, and theme & mood keywords. Then, it
extracts the list of the objects that must be drawn on the image for this
description. For this, we used few-shot prompting with GPT-3.5-turbo (OpenAI,
2020). The used prompt is in the Appendix A.3. For the arrangement, we
developed a layout variator to create layouts similar to the selected image’s
arrangement while aligned with the generated text description. The layout
variator first applies an empirically defined random variation of -50 to 50
pixels on each bounding box component (i.e., x, y, w, h) in the original
arrangements. Then, it randomly selects boxes depending on the number of
objects that need to be drawn and sorts highly similar layouts first using the
similarity metric. The similarity is calculated by summing the IoU and the
complement of the min-max normalized centroid distance between the closest
pairs of bounding boxes. Following this similarity, the top five arrangements
are utilized for the recombination generation. The most similar layout is used
for generating the image in the initial iteration, and other layouts are used
when the user requests more sketches. A few shot prompting with GPT-3.5-turbo
is used to map between the arrangements and the objects to create the best
image possible. The full prompt used is shown in the Appendix A.4.

However, when the user does not select any arrangement from the references,
the system generates a broader range of diverse layout options. A few-shot
prompting pipeline using GPT-3.5-turbo generates the three most appropriate
layouts for the given text description and object list. This pipeline is built
based on the previous work (Lian et al., 2023), and the full prompt for this
is in the Appendix A.5.

Given the textual description and the list of the objects mapped with the
generated layout, the system generates images with a layout diffusion model
(Li et al., 2023b). Following our design goal, the system converts the
generated image into a simple line sketch using the U-Net structured style
transfer model (Datasculptor, 2023).

###  4.3. Technical Evaluation

We evaluated ML-based pipelines, especially for keyword extraction, keyword
recommendation, and textual description generation by merging keywords.

####  4.3.1. Keyword Extraction Pipeline

We built a dataset consisting of 100 images and tags of each image categorized
by the subject matter, action & pose, and theme & mood. We asked 20 people
with expertise in design or HCI to annotate five images each. On average,
there were 5.03, 1.87, and 2.29 keywords collected per image in the category
of subject matter, action & pose, and theme & mood, respectively.

Using this dataset as ground truth, we evaluated the prediction result from
the keyword extraction pipeline. Keywords in subject matter and action & pose
categories were matched manually one by one between similar ones. The
precision and recall of our pipeline were 94.2% and 58.2% in subject matter,
and 35.3% and 51.3% in action & pose. Although some salient keywords in the
dataset were missing, the pipeline provided quite accurate keywords in the
subject matter. The predicted action & pose keywords were not perfectly
aligned with the dataset tags, but they were still acceptable in user-side
because they were perceived as similar to users even if they were not
completely accurate (e.g. for an image of a cat standing straight, our
pipeline predicted ”stretching arms”, while the ground-truth is ”dancing”).
For theme & mood keywords, we calculated the cosine similarity of mean
embedding vectors (Wang et al., 2020) of ground-truth and predicted result to
compare the semantic similarity. This was because for theme and mood, even if
words are not exactly the same, there can be many other words that can be
accepted as similar. The similarity of the ground truth and prediction was
0.826, which means the keyword extraction model estimates the theme & mood
words quite closely. Examples of the predictions are presented in the Appendix
(Figure 10).

####  4.3.2. Keyword Recommendation Pipeline

We evaluated the keyword recommendation pipeline based on whether there was a
proper level of similarity between the original keywords and the recommended
keywords. This was because it would be useless if the recommendation is too
similar or too irrelevant to the original keywords.

We randomly sampled three to ten keywords from each image-keyword pair in the
dataset and made 100 sets of keywords. Then, from the pipeline, we got the
recommendations for each set. To verify whether these recommendations have a
proper range of diversity, we generated two comparison groups of keywords, the
irrelevant group and the synonym group. The irrelevant group consists of
random keywords from the dataset, and the synonym group is generated by
paraphrasing the keyword in each set. NLTK (Bird et al., 2009) and GPT-3.5
were employed to find synonyms. Then, we used the text embedding (Wang et al.,
2020) to calculate the cosine similarity of each group with the original
keywords.

The similarity of the irrelevant and synonym groups to the original keywords
was 0.624 and 0.774, respectively, and the recommended keywords had a
similarity of 0.696, which is in the middle. This shows that our
recommendations are less similar to original keywords than synonyms, but more
similar compared to the irrelevant phrases.

####  4.3.3. Recombination Generation Pipeline

The recombination generation pipeline gets a user selection of a set of
keywords and generates three different descriptions of the possible image that
includes those keywords. As the pipeline aims to provide diverse options, we
evaluated the diversity of the description generation model.

Similar to section 4.3.2, we built 100 sets of keywords randomly extracted
from the dataset. For each set, we generated three descriptions using our
pipeline, calculated the cosine similarities between those three, and averaged
them. Here, we calculated diversity as $1-similarity$. To validate our
description generator, we prepared two more description sets, one consisting
of explicitly unrelated descriptions that were randomly acquired from the
dataset, and the other one consisting of descriptions that merely paraphrase
one of the generated descriptions using paraphraser with T5-based model
(Vladimir Vorobev, 2023). The diversity within the random and paraphrased
groups were 0.801 and 0.209 respectively, while the generated descriptions
from our pipeline show a diversity of 0.395. This shows that generated output
is more diverse than just paraphrasing and less diverse than random ones,
which means that the pipeline generates descriptions of a reasonable amount of
diversity.

We didn’t evaluate the later part of this pipeline which is about generating
images and transforming them into sketches, as we used models from previous
research (Lian et al., 2023) without any customization or adaptation.

##  5\. Evaluation

We conducted a within-subjects comparative study with 16 participants. As our
design goals encompassed two steps of the reference recombination – (1)
Finding elements (DG 1 and DG 2) and (2) Recombining elements (DG 3 and DG 4),
we first observed how supported each of these steps. We also evaluated whether
eventually improves designers’ idea generation results and how it supports the
creative process.

  * •

RQ1. How does support the two steps of the recombination process—finding
elements from the references and recombining elements?

  * •

RQ2. Can help users generate better quality and quantity of design ideas?

  * •

RQ3. How do users utilize ’s outputs in their ideation process?

The baseline system shared a similar interface with but without the key
features of —extracting keywords from the reference, suggesting relevant
keywords, and generating recombination options. In this baseline system, users
could manually leave keyword notes on each reference image, create sketches by
specifying layouts and prompts to the image generation model, and use
ChatGPT333OpenAI. (2023). ChatGPT (August 3 Version).
https://chat.openai.com/. To assess the efficacy of the design of ’s features
and pipelines rather than the effect of AI functionalities, the same AI
functionalities are also included in the baseline system. After observing
prevalent use cases of AI in design processes through recent survey (Palmer et
al., 2023) and videos (Zhu, 2023; Design with Canva, 2023; Satori Graphics,
2023), we included both the language model and the image generation model in
the baseline system to simulate real-world scenarios of designers with AI
tools. To prevent the model performance from affecting the study results, the
baseline included a model closely aligned with the pipeline. Instead of the
GPT models, we provided GPT-3.5-based ChatGPT, and for image generation, we
provided the same layout diffusion model as . The screenshot of the baseline
interface is presented in Figure 11 in the Appendix.

###  5.1. Participants

We recruited 16 participants (10 female, 6 male; age M=24.81 and SD=3.78)
through an online recruitment posting. To figure out whether the can handle
the challenges found in the formative study with novice designers, our
participants were set as a similar group to the formative study. We required
participants to have a degree in design or art and have participated in at
least 3 different design projects. 11 participants were students majoring in
design—5 participants at the graduate level and 6 participants at or above the
third-year undergraduate level. The other 5 participants have graduated—2
majored in design, 1 minored in design, while others pursued majors in media
arts and painting.

All of the participants also reported themselves as having enough sketching
skills to express their design ideas, since we asked participants to draw
their ideas during the task. The study was conducted for 2 hours, and we
compensated participants with 70,000 KRW (approximately 53 USD).

###  5.2. Study Procedure

Figure 4. User study process. The 2-hour user study consists of two sessions
with different tools, each including a 30-minute ideation phase utilizing the
given tool. The order of the tool and the design tasks are counterbalanced.
After the two sessions, they had 20-minute semi-structured interviews about
their experiences.

User study process. Participants engaged in a 2-hour user study, beginning
with a 5-minute introduction outlining the overall process of the study. Then,
they underwent two sessions, each including an 8-minute tutorial on the
provided tools and task briefing, followed by a 30-minute ideation phase
utilizing the designated system. A 5-minute post-survey concluded each
session. Participants could have a 10-minute break between the two sessions,
and the order of the given tool and the design tasks are counterbalanced.
After the two sessions, they had 20-minute semi-structured interviews about
their experiences.

The whole process of the user study is shown in Figure 4. Participants were
asked to perform design ideation tasks twice in two settings: and baseline.
The task was to draw an illustration for the cover of a children’s book named
”Starry Safari: Exploring Alien Jungles” or ”A Christmas Dinner in the
Underwater World”. They were also provided with 10 reference images for each
topic. The order of topics and tools was counterbalanced for each participant.

For the first five minutes of each round, participants were provided a
tutorial on the given system interface and tried out the system with sample
images to get used to the system. They were then given the topic and the
reference images, and started ideation using the given tool for 30 minutes. If
the participants came up with a design idea that they wanted to further
develop, they were asked to sketch it on the paper using a pen. After each
round, they completed the post-task survey. Between the two rounds, they could
get a 10-minute break if they want. After both rounds, we conducted a
20-minute semi-structured interview to ask about the difference between the
two conditions and the effect of the tools on their ideation process. The
interview questions are presented in Appendix B.2.

###  5.3. Measures

The post-survey after each round included questions about the usefulness of
the given system for the different steps of the ideation, which are (1)
organizing the references, (2) discovering useful elements from the reference,
(3) exploring multiple ideas, (4) discovering new ideas, and (5) exploring
multiple ideas. The survey also included five questions about satisfaction
with the overall outcome, quantity, quality, diversity, and creativity of
participants’ own sketch results. We also included five questions from (Wu et
al., 2022) to assess participants’ self-perceived experience of using the AI
system. Participants answered these questions only for the image generation
feature and ChatGPT after the baseline session, and for the keyword
extraction, keyword recommendation, and image/description generating features
after the session. Also, the survey included the Creativity Support Index
(Cherry and Latulipe, 2014) and NASA-TLX questionnaire (Hart and Staveland,
1988).

We also gathered the usage logs (i.e., participant actions with timestamps) to
get quantitative metrics for user behaviors. We used this data to calculate
the time taken for each sketch, the number of images generated, the number of
inputs provided to the image-generating model, etc. Also, every time the
participants completed the sketch, the system prompted participants to rate
how well the given tool assisted them in producing the idea.

Additionally, we conducted an expert evaluation of the participant’s sketches.
We recruited two experts, both graduating with a bachelor’s degree in art and
having 6 and 1.5 years of art education each. We asked them to evaluate two
factors in the 7-point Likert scale: (1) the creativity of each sketch and (2)
the diversity of ideas within a set of sketches. We randomly chose three
sketches drawn by each participant on each design topic, and a total of 96
sketches (3 sketches x 16 participants x 2 conditions) were evaluated. The
evaluators rated the sketches individually, and for cases of significant score
differences (more than 3 points), we asked evaluators to re-evaluate them.
While re-evaluating, they were given each other’s comments and scores and
could choose to change their original score or leave it. They also had to
leave comments about their decision as well. There were 9 sketches that
required re-evaluation, and all of the conflicts were resolved after one round
of re-evaluation. After that, we used the average score for the two
evaluators’ scores for the result analysis.

##  6\. Results

Results showed that helped participants with both finding and recombining
elements for reference recombination. Also, it was shown that users with were
able to generate more design ideas in a given time and perceived their ideas
as more creative compared to the baseline. We also found some differences
between and baseline in terms of how users utilize the tool for their creative
process.

Figure 5. Survey result on the user-perceived efficiency during each step of
the recombination step with 95% confidence interval. is significantly useful
in discovering elements from the reference image, and also in generating
multiple ideas.

Survey result on the user-perceived efficiency during each step of the
recombination step with 95% confidence interval. There are 5 parts in the
survey (i.e. organizing reference, discovering useful elements from the
reference, exploring multiple ideas, discovering new ideas, and generating
multiple ideas), which can be divided into two recombination steps. is
significantly useful in discovering elements from the reference image, and
also in generating multiple ideas.

###  6.1. Support for Different Recombination Steps

To answer RQ1, we examined survey questions and log analysis results divided
into three steps of reference recombination: (1) discovering keywords from the
reference, (2) exploring different recombinations, and (3) generating new
concepts. For all survey questions, we used a Wilcoxon signed-rank test as all
of them were ordinal data on a 7-point Likert scale. For the usage log
analysis, we conducted a two-sample t-test or two-sample paired t-test to
compare between and baseline.

####  6.1.1. Finding Keywords from the Reference

Participants perceived that helped discover valuable keywords from the given
reference images. As shown in Figure 5, participants found out that is
significantly more useful for discovering valuable elements from the reference
that can be used for their ideation (M=6.13, SD=1.31) compared to the baseline
system (M=3.75, SD=1.98 / p=0.001, W=0.0). Regarding how and baseline helped
with organizing references, the rating was not significantly different, but
with a slightly higher average rating for (Baseline: M=4.38, SD=1.96 / :
M=5.31, SD=1.58 / p=0.121, W=23.5).

Usage logs also showed that effectively encouraged participants to explore and
extract different keywords. In comparing the numbers of the keyword notes that
participants left in both conditions using a two-sampled paired t-test,
participants with added more keyword notes (M=34.69, SD=10.74) compared to the
baseline system (M=13.19, SD=10.53 / p¡0.0001, t=5.52). Also, as shown in
Figure 6, participants with baseline typically extracted keywords exclusively
during their initial sketch, thereafter relying solely on the previously
extracted keywords without actively discovering additional keywords. In
contrast, participants using consistently added more keywords throughout the
whole process. While they also extracted the most keywords at the beginning,
they continued to extract new keywords from references for every new sketch.
One participant (P15) drew out all the sketches at the end after developing
multiple design ideas at once, instead of drawing a sketch immediately after
developing each design idea. As it cannot be clearly identified which keyword
notes correspond to each specific sketch in P15’s usage log, this data was
excluded only from this analysis, which is about the actions related to each
sketching turn. The raw usage log data for all participants, including P15, is
provided in Appendix B.3.

Figure 6. Comparison of the count of two different actions (adding keywords,
generating image through generation model) taken to generate each sketch (from
the first sketch to the fifth sketch) in and baseline. The results show that
users use the add keyword action more in compared to the baseline where users
only add keywords for the initial sketches. There is no significant difference
in the count of generated image action.

The graph shows the comparison of the count of the action taken to generate
each sketch in and baseline. Add keyword action refers to the action where
users add a keyword note. Generate image action refers to the point when the
user gives input to the image generation model. The results show that users
use the add keyword action more in compared to the baseline where users only
add keywords for the initial sketches. There is no significant difference in
the count of generated image action.

####  6.1.2. Recombining elements

The survey’s findings indicated that the can be useful for recombining
different elements into new design ideas. Participants said that is
significantly more helpful (M=5.94, SD=1.34) than the baseline system (M=4.88,
SD=1.89 / p=0.023, W=10.5) for them to generate multiple ideas from the
collected elements (Figure 5). However, participants’ perception of how much
the system helped explore multiple ideas was not significantly different in
both conditions, although had a slightly higher average rate (Baseline:
M=4.88, SD=1.93 / : M=5.69, SD=1.35 / p=0.178, W=22.0). Also, it was not
significant in terms of discovering novel ideas, but the average was slightly
higher in (baseline: M=5.00, SD=1.75 / : M=5.75, SD=1.48 / p=0.110, W=19.0).

We also looked into how participants used the given image generation model for
recombining elements into a design idea. As shown in Table 1, there was also
no significant difference between the two conditions in the number of
generated images. However, in terms of exploring diverse recombinations using
the model, showed distinct advantages, as evident from the unique patterns
observed when users interacted with the generation model under two conditions.
Using the baseline system, users could provide separate inputs for overall
image descriptions and for each object. allowed users to select multiple
keywords to merge. In both conditions, participants had the option to input
multiple phrases together to combine them. We conducted an analysis to
evaluate how diverse phrases inputted together into the model. Out of a total
of 347 input sets (202 from the baseline, 145 from ), 14 sets (11 from the
baseline, 3 from ) consisted of only one input, and they were excluded from
the analysis since our objective was to compare the semantic similarity
between phrases that were provided to the model together. For the remaining
333 input sets (191 from baseline, 142 from ), we computed the semantic
similarity between all pairs of phrases within each input set and calculated
the mean and minimum similarity. The mean similarity represents the overall
similarities between phrases provided as input together, while the minimum
similarity represents the most diverse pairs within the set. Finally, we
conducted a two-sample t-test for each metric.

As shown in Table 1, the input sets created within showed significantly lower
similarity between the keywords (M=0.222, SD=0.094) compared to the sets made
within the baseline (M=0.263, SD=0.166 / p=0.008, t=2.66) when they are
calculated based on the minimum similarity. This difference is also similar
when they are calculated based on the mean similarity, but it was slightly not
significant (Baseline: M=0.356, SD=0.148 / : M=0.330, SD=0.075 / p=0.051,
t=1.95). This means that participants with actively sought to create unique
recombinations with greater semantic similarity, ultimately leading to the
exploration of diverse and distinct recombinations compared to the baseline
condition.

|  |  | Baseline | Statistics
---|---|---|---|---
|  | mean | std | mean | std | p | Sig.
Image Generation Model Usage (Per session) | # of generated image | 57.06 | 17.91 | 46.69 | 23.52 | 0.119 | -
# of user inputs to the model | 9.31 | 4.57 | 10.56 | 4.76 | 0.468 | -
Semantic Similarity within Input Sets | Semantic Similarity (Mean) | 0.330 | 0.075 | 0.356 | 0.148 | 0.051 | -
Semantic Similarity (Min) | 0.222 | 0.094 | 0.263 | 0.166 | 0.008 | **

Table 1. Number of image generation model usage and the semantic similarity
between user inputs in and baseline. (-: p ¿ .05, *: p ¡ .050, **: p ¡ .010,
***: p ¡ .001)

Number of image generation model usage and the semantic similarity between
user inputs in and baseline. For image generation model usage tendency, the
number of generated images and the number of user inputs to the model are
compared. Semantic similarity of calculated by minimum value is significantly
lower than baseline with a p-value of 0.008. Semantic similarity calculated by
mean is also lower in but not significant with a p-value of 0.051.

###  6.2. Ideation Results

To answer RQ2, we analyzed the design idea sketches that participants drew
during the study session through expert evaluation, usage log, and survey
results. Same as the RQ1, the Wilcoxon signed-rank test was used for survey
questions. For expert evaluation results and the log analysis results, we used
a two-sample t-test. For pairwise data, such as a comparison between the
number of sketches drawn in each condition by each participant, we conducted a
two-sample paired t-test.

####  6.2.1. Creativity & Diversity of the Final Sketches

Figure 7. Evaluation results of user-drawn sketch with 95% confidence
interval. (a) Expert evaluation on the diversity and creativity for and
baseline condition. (b) Self-reported satisfaction on sketch result in terms
of quantity, quality, diversity, creativity, and overall for and baseline
condition.

This is a bar graph about two evaluation results of the user-drawn sketch with
a 95% confidence interval. (a) Expert evaluation: Statistics (i.e. mean and
p-value) of expert ratings of diversity and creativity for and baseline. The
diversity of and baseline has few differences in mean, while the creativity of
is slightly better but has no significance. (b) Self-reported satisfaction on
sketch result: Statistics of self-reported satisfaction ratings in terms of
quantity, quality, diversity, creativity, and overall for , baseline. The
score of in creativity is significantly higher than the baseline.

As shown in Figure 7 (b), the survey results showed that participants
perceived their sketch as more creative when they were using (M=5.38, SD=1.09)
compared to the baseline (M=4.19, SD=1.64 / p=0.004, W=0.0). During the
interview, 12 out of 16 participants said that they felt they could be more
creative with the support of rather than the baseline, especially when they’re
having a hard time coming up with a new idea in the early ideation stage. In
terms of other factors including overall satisfaction, quantity, quality, and
diversity of the sketches, there were no significant statistical differences
between the two conditions.

As shown in Figure 7 (a), however, the expert evaluation does not show a
significant difference between the two conditions. The creativity score of the
expert evaluation were slightly better in (M=4.854, SD=1.418) compared to the
baseline (M=4.344, SD=1.708 / p=0.114, t=1.59), but it was not significant
according to the two-sample t-test results. In terms of diversity, there was
also no significant difference between them (Baseline: M=4.625, SD=1.607 / :
M=4.75, SD=1.418 / p=0.833, t=0.23). There were possible reasons that expert
evaluation was different from the survey results. First, even though the
experts were asked to focus on the idea as much as possible, the participants’
sketch skills were inevitably reflected in the evaluation, and some of the
comments left by the evaluators were actually about the sketch skills. There
is also a possibility that deviations according to the design topic may have
been affected. In fact, sketches about the topic of underwater Christmas were
rated higher on average.

####  6.2.2. Efficiency of the Ideation Process

|  | Baseline | Statistics
---|---|---|---
| mean | std | mean | std | p | Sig.
# of sketch per session | 5.56 | 1.63 | 5.06 | 1.73 | 0.041 | *
time per sketch (min) | 5.01 | 2.87 | 5.39 | 3.03 | 0.403 | -

Table 2. Number of sketches drawn by the participants per session and the
average time taken for sketches (-: p ¿ .05, *: p ¡ .050, **: p ¡ .010, ***: p
¡ .001)

This table shows the number of sketches drawn by the participants per session
and the average time taken for sketches. The per session average number of
sketches from has a significant difference from than baseline.

As shown in Table 2, the two-sample pairwise t-test result showed that
participants came up with more sketches in the same 30-minute ideation session
with the support of (M=5.56, SD=1.63) than with the baseline (M=5.06, SD=1.73
/ p=0.041, t=2.24). This result indicates that can be useful for efficient
ideation. The interview results also demonstrated that could be useful when
they have to come up with a lot of ideas in a limited set of references and
time, which is a common scenario in professional design tasks where clients
provide references and designers must provide drafts with them.

####  6.2.3. Perceived Workload

As shown in Table 4, there was no difference between the two conditions in
terms of the perceived workload. While has additional complications such as
requiring users to specify keywords to give inputs to the image generation
model, this does not cause users to feel overwhelmed while performing the
task.

###  6.3. Impact on User’s Creative Process

####  6.3.1. Source of the Inspiration

|  | Baseline
---|---|---
Source of Inspiration |  Within the tool | Generated image/description | 9 | 5
Recommended keywords | 1 | -
ChatGPT answers | - | 2
Outside of the tool | Own creativity | 1 | 1
Reference images | 5 | 8
Avg. tool assistance rating | 5.625 | 4.563

Table 3. Count of the inspiration sources for the most creative sketches
chosen by the participants and the average rating of the efficiency of the
tool assistance for drawing those sketches. Figure 8 illustrates the example
use cases of inspiration within the tool.

This table shows the counts of the inspiration sources for the most creative
sketches chosen by the participants and the average rating of the efficiency
of the tool assistance for drawing those sketches. In terms of source of
inspiration, the users had more within the tool, especially from the generated
images and descriptions. Compared to that, baseline users got more
inspirations outside of the tool, such as reference images. As a result, the
average rating on the efficiency of the tool’s assistance was higher in .
Figure 8 illustrates the example use cases of inspiration within the tool.

To investigate how users use the output from and baseline system for
generating new design ideas differently, we asked participants to pick one
sketch that they think is the most creative for each study session and explain
how they got the inspiration for that.

There are five different inspirational sources found in two conditions. In
both conditions, many participants got their ideas from the generated images
or text descriptions. In condition, more than half of the participants said
that their best ideas are inspired by these generated images or text (Table
3). As illustrated in Figure 8 (a), participants utilized keywords from both
reference images and recommendations and merged them using the system. What
was notable is that they got the generated images and tried to reinterpret
them in their own way, rather than accepting what is drawn there. Participants
with the baseline system were also influenced by the images generated, but the
number was slightly less (Table 3), and the way they were influenced was
slightly different. They tend to refer to the visual compositions or details
of the shapes and apply them into their own sketch. P7 mentioned the reason
for this, “While putting prompts into the image generation model (in the
baseline), I already had the concept that I wanted. Therefore, I refer to the
expression method of it, rather than trying to find something new out of it.”

One thing noticeable is that participants were influenced more by the given
reference images when they were using the baseline. This shows that can
potentially make users less directly influenced by reference images,
ultimately preventing them from fixating on them. P16 explicitly pointed out
this by saying, “When using , I gave less focus to given images, and as I can
expand to a lot of ideas only with a small number of references, I didn’t even
use all of them.” P14 mentioned, “This (baseline) tool feels like a notepad
that manages references, so I kept referring to the reference images
themselves.”

As shown in Figure 8 (d), there was also a participant who got an idea from ’s
recommended keywords. In baseline, instead of this keyword recommendation
feature, they could use ChatGPT, and 2 participants said that they got their
inspiration from this. However, the number of this usage was relatively small
(Table 3), mainly because of the challenges of using it for visual tasks.
Participants mentioned difficulties in both formulating prompts and leveraging
the language-based output for their design during the interview.

This difference in sources of inspiration affected the results of the user’s
rating of how effective the assistance of the tool was. We conducted a two-
sample pairwise t-test to compare participants’ ratings on the tool’s
usefulness for generating their favorite ideas, and it was shown to be higher
in (M=5.63, SD=1.41) compared to the baseline (M=4.56, SD=1.89 / p=0.045,
t=2.18) (Table 3), indicating that users perceived the features of are more
useful for coming up with their best ideas, compared to the baseline system.

Figure 8. Examples of user input, system-generated output, user-drawn sketches
inspired by the system output, and corresponding user quotes. In both
conditions, users were inspired by the generated images. However, users were
more inspired by the overall concept of the image, while baseline users
referred to a specific composition or the detail of the object from the image.
Also, users often got inspired by the recommended keywords, while baseline
users had ChatGPT support instead of it.

This figure shows examples of user input, system output usage to user-drawn
sketches. The corresponding user quote is located under with blue color. (a),
(b) Inspired by generated Image ( and baseline): system-generated image and
inspired sketch pair. Each section consists of two pairs featuring a system-
generated image, a user sketch, and the corresponding user quotes. (e.g. (a):
”I only thought about the safari tour, but after seeing at this image, I got
an idea of exploring with safari animals by stacking them.” (b): ”I referred
to the composition of looking from the top to bottom and the shape of the
spacecraft.”) (c) Inspired by recommended keywords (): a set of recommended
keywords and sketches that express the new concept containing the keywords.
Commented to the sketch, ”The system suggested many energetic actions,
inspiring me to create a drawing with characters in diverse active poses.” (d)
Inspired by ChatGPT answers (baseline): ChatGPT answers and sketch inspired by
them. Commented to the sketch, ”ChatGPT’s response made me realize the
existence of the title on the book cover, sparkling ideas to make something
with this title.”

The survey results about the perceived experience of using the AI-based system
also showed a more specific reason for this helpfulness. As shown in Table 4,
is shown to be significantly better for thinking through what kind of outputs
users want to complete for the given task (baseline: M=5.00, SD=1.97 / :
M=6.13, SD=1.02 / p=0.045, W=14.5). This shows that participants don’t think
of the results of the ’s image generation model as their final results, but
more as a guide to think about what they want. It leads users to think in
diverse ways. P9 mentioned that “In baseline, the result came out exactly what
I thought, so I replicated the output. However, shows me various high-level
ways to combine things, so I could explore those methods and expand those
processes on my own.”

|  |  | Baseline | Statistics
---|---|---|---|---
|  | mean | std | mean | std | p | Sig.
Self-perceived experience on ML model | Match goal | 5.00 | 1.63 | 4.63 | 1.96 | 0.5805 | -
Think through | 6.13 | 1.02 | 5.00 | 1.97 | 0.0454 | *
Transparent | 4.81 | 1.80 | 4.38 | 1.67 | 0.4488 | -
Controllable | 4.75 | 1.95 | 4.06 | 1.84 | 0.2976 | -
Collaborative | 5.38 | 1.59 | 4.94 | 2.08 | 0.4809 | -
NASA-TLX | Mental | 3.69 | 1.82 | 4.19 | 1.94 | 0.39 | -
Physical | 1.81 | 1.22 | 2.50 | 2.10 | 0.10 | -
Temporal | 2.81 | 1.83 | 3.50 | 2.28 | 0.23 | -
Effort | 3.63 | 1.82 | 3.94 | 2.05 | 0.63 | -
Performance | 5.31 | 1.08 | 5.06 | 1.39 | 0.78 | -
Frustration | 2.63 | 1.93 | 3.50 | 1.75 | 0.14 | -
Creativity Support Index | Enjoyment | 5.91 | 1.00 | 5.09 | 1.78 | 0.077 | -
Exploration | 5.38 | 1.54 | 4.81 | 1.56 | 0.211 | -
Expressiveness | 5.44 | 1.18 | 4.53 | 1.75 | 0.032 | *
Immersion | 4.69 | 1.99 | 4.69 | 1.82 | 1 | -
Results Worth Effort | 5.47 | 1.27 | 5.25 | 1.71 | 0.591 | -
Collaboration | 5.19 | 1.25 | 4.41 | 1.71 | 0.016 | *

Table 4. Survey results of self-perceived experience on ML features, NASA-TLX
questionnaire, and Creativity Support Index. (-: p ¿ .05, *: p ¡ .050, **: p ¡
.010, ***: p ¡ .001)

Survey results of self-perceived experience on ML features, NASA-TLX
questionnaire, and Creativity Support Index. The experience on the ML model is
investigated into 5 parts (i.e., match goal, think through, transparent,
controllable, and collaborative). is significantly helpful for thinking
through. Through the survey on NASA-TLX, and baseline have no significant
difference in workload. In the Creativity Support Index, was shown to be
significantly preferred in terms of expressiveness with a p-value of 0.032 and
collaboration with a p-value of 0.016. There were no significant differences
between the two for other criteria.

####  6.3.2. Creativity Support Index

According to Table 4, users tend to prefer significantly more than the
baseline in terms of expressiveness and collaboration, but in the other
criteria, there was no significant difference between the two systems. Through
the post-interview, we found out that participants felt different types of
creativity support in each system. Participants said that the baseline was
helpful when they had an overall idea in their mind and wanted to get support
for expressing it in the sketch. On the other hand, participants said that the
is helpful for their creativity when they have no idea in their minds. These
differences will be explained in more detail in section 7.1.

##  7\. Discussion

We propose a novel AI-infused creativity support tool , which assists graphic
designers in generating their design ideas by recombining reference images.
Based on our findings, we suggest some design implications for future
creativity support tools.

###  7.1.  vs. baseline - Two Different Types of Creativity Support

The results show that supports the early-stage conceptual ideation with
reference recombination process successfully, by aligning well with the four
design goals that we derived from the formative study. Participants could
easily extract keywords (DG 1) and utilize keyword recommendations as a source
of new inspirations (DG 2) which led them to make more keyword notes. Also,
they explored diverse keyword recombinations (DG 3) which led them to make
more design ideas in a given time. Additionally, they perceived their idea as
more creative as provided the output as an incomplete sketch and let
participants inject their creativity into it (DG 4). However, participants
couldn’t feel the difference in the overall degree of creativity support
between the two tools. The interviews revealed that this was because and
baseline both provided valid creativity support, but in a distinct way based
on users’ current needs.

In the baseline system, users should specify all the details of the generated
image, so they appreciated the transparency and control. The system faithfully
reproduced user input by that control, resulting in a final output that
closely mirrors the concept in their mind. These generated outputs helped
users actualize their existing ideas, more supporting implementation (Chung et
al., 2022, 2021). Dynamic Brushes (Jacobs et al., 2018) or Framer (Lawton et
al., 2023) had a similar approach to creativity in terms of this.

Conversely, stimulates creativity by providing inspiration (Chung et al.,
2022, 2021). Instead of requiring users to provide detailed input, accepts
keywords and deliberately refrains from exact expression, generating a wider
range of outcomes, potentially with serendipity. How can provide participants
with this creative leap can be explained by Cross’ descriptive model of
creative design (Cross, 1997). The keyword extraction feature actively
supports emergence, allowing designers to find unrecognized properties of the
existing design. The keyword recommendation also supports mutation, helping
designers to generate new ideas by modifying existing designs partially. P15
metaphorically likened this process to having someone nearby constantly
talking with them with fresh variations in the form of keywords. Furthermore,
the keyword merging feature enhances combination, where new ideas are
generated by combining features from existing designs. Therefore, could be
potentially useful for addressing a common challenge known as “artist’s block”
or “creative block”, similar to the “writer’s block” experienced by writers
(Hirst, 1992). could provide proper support when designers find themselves
creatively stuck, breaking creative inertia by sparking novel ideas and
opening new creative avenues.

These differences can be valuable design implications for future creativity
support tools as designers require different types of creativity support in
different stages of the ideation process. By dynamically adjusting the type of
support based on the user’s context, such a tool can offer a more personalized
and effective creative experience. For instance, when the system detects that
a user is in the exploration phase, it can employ an approach similar to ,
encouraging the generation of diverse and abstract ideas. Conversely, when the
user demonstrates a desire to refine and develop a particular concept, the
tool can provide baseline-like features to ensure greater control and fidelity
in the generated output. This adaptable approach acknowledges the multifaceted
nature of the creative process and supports users with the right tools at the
right moment, ultimately enhancing their creativity. Also, by integrating
those inspiration and implementation support into a single tool, it can enable
a seamless transition between generating diverse ideas and refining specific
concepts, fostering a more iterative and efficient creative workflow.

###  7.2. The Role of Low-fidelity Output for Creativity Support

The post-interview showed that adopting low-fidelity output can facilitate
further imagination beyond what the system provided. In both and the baseline,
we deliberately employ a low-fidelity sketch output. During the interview, 12
out of 16 participants said that they preferred the sketch output rather than
a complete image, as it allowed users room for imagination and interpretation.
The image converted into a sketch omits small details and retains only the
larger forms, which in turn generates large empty space. This emptiness
encourages users not just to perceive the generated image but to see it as
room for further development and makes users deeply engaged in further
ideation. Some participants even expressed opposition to completed images for
the ideation stage, as they believed that an abundance of details in reference
images makes them fixated on that specific design idea and hinders them from
utilizing the images in their own ideas. P2 said, “I usually get completed
artworks from Pinterest 444https://www.pinterest.com/ as a reference, and I
found myself unavoidably looking at the unique style of that designer, wanting
to replicate it. This time, I liked that I could maintain my own style while
exploring different references of concepts.” From our findings, we argue that
adopting low-fidelity output should be an option to consider when designing
creativity support systems to prevent fixation and facilitate the user’s
creativity in ideation. For example, a design reference tool can dynamically
adjust the levels of details of the provided images based on the user’s
current design stage. When the user wants references for overall concepts, the
system can convert reference images to a simple black line drawing or even
present it solely as a textual description. Conversely, when the user has
determined a specific concept and is exploring different visual details, the
system can offer the original images with full details.

###  7.3. Generalizability of in Different Context

is designed to support novice designers, such as design students with a
general understanding of the design process but struggle with reference
recombination. However, our user study revealed some insights applicable to
different expertise levels. We observed that participants with limited
sketching skills were satisfied more with the baseline system, as it was more
aligned with their intentions and suitable for the aid for the actual
sketching. Therefore, for users less familiar with artistic expression, an AI
tool’s output should prioritize alignment with the users’ original intent,
rather than abstraction. Conversely, for experts accustomed to extracting
inspiration from references and combining them into their original idea
(Bonnardel and Marmèche, 2005), could serve as a tool for serendipity rather
than helping them with the process of keyword extraction and recombination.
For example, P16 said that suggested keywords and merged images acted as
prompts to remind them of some aspects initially overlooked. Therefore,
features should be redesigned to encourage reflection and creative
exploration, such as highlighting the part of the generated images that were
not present in existing references but emerged through our system features.

The user study results showed that could be also applied for other design
contexts such as collaborative projects. According to the CSI survey results
(Section 6.3.2), participants indicated that would be significantly helpful
for collaborating with other designers. This was because is designed to follow
the sequential steps of leaving keyword notes and merging them, and it keeps
track of these processes on the mood board and the merging panel. Therefore,
participants said that simply showing screen could be used to share their
creative processes with other designers, making it easier for them to
understand each other’s thought processes and quickly reach an agreement on
the design direction. One future work direction can be incorporating features
of to collaborative mood board tools (Chung and Adar, 2023a; Koch et al.,
2020, 2019) and studying the benefits of keyword-based recombination features.

could also be applied to other design domains. Our design goals and the
feature design of are primarily tailored to the illustration design task.
Illustrations predominantly about conveying design topics through visual
subject matters and do not usually include other modalities such as text
(common in poster or publication design) or motion & interaction (common in
UI/UX and motion graphic design). However, even in other design domains, the
recombination process of extracting elements from the reference and
recombining them would be an effective strategy. To apply the recombination
approach to another design domain, we have to first identify what elements
designers in that domain focus on when looking at references and use those
different categories of elements as keywords in the pipeline of .

##  8\. Limitations and Future Work

Our work has several limitations which can be potentially addressed in future
work.

  * •

In our user study, the ideation tasks were conducted for 30 minutes in each
condition, which was shorter than the actual design process. Therefore, it was
difficult to observe how the behavior changed over a long time. Future work
can be done to incorporate with real-world design projects and see how their
behavior patterns differ from lab studies.

  * •

Our pipeline generates an image description containing all of the keywords
selected by the user as a method of recombination. However, there can be
various ways of recombination other than this, such as blending objects or
indirectly expressing some keywords through visual details such as colors.
Further work can be done on these various recombination methods and how to
support them.

  * •

As and baseline both leverage generative AI including LLM and layout diffusion
model, the result may be influenced based on users’ familiarity with AI. Since
this study did not explore those dimensions, future research can be conducted
to find how creativity supporting tools with AI features may have varying
effects depending on the user’s knowledge level of AI or prior experiences of
using AI.

##  9\. Conclusion

This paper proposed , a system designed to support graphic designers in the
reference recombination process, allowing them to generate novel design ideas.
Building on our formative study observations, offers users assistance in
identifying key elements within reference images. It also provides a diverse
range of recommendations for relevant keywords and recombination options.
Notably, the low-fidelity sketch-based output of was shown to encourage
creativity by enabling further imaginative exploration. Our user study
demonstrated that efficiently supported both steps of finding and recombining
elements and helped participants to come up with more design ideas and
perceive their ideas as more creative compared to the baseline. While
represents a promising step towards comprehensive recombination support tools
for designers, we also suggested an opportunity to expand such systems to
cater to address a wider spectrum of design needs and situations.

## References

  * (1)
  * Bird et al. (2009) Steven Bird, Ewan Klein, and Edward Loper. 2009.  _Natural language processing with Python: analyzing text with the natural language toolkit_.  ” O’Reilly Media, Inc.”.
  * Boden (1998) Margaret A. Boden. 1998.  Creativity and artificial intelligence.  _Artificial Intelligence_ 103, 1 (1998), 347–356.  https://doi.org/10.1016/S0004-3702(98)00055-1 Artificial Intelligence 40 years later.
  * Bonnardel (1999) Nathalie Bonnardel. 1999.  Creativity in design activities: The role of analogies in a constrained cognitive environment. In _Proceedings of the 3rd conference on Creativity & cognition_. 158–165.
  * Bonnardel and Marmèche (2005) Nathalie Bonnardel and Evelyne Marmèche. 2005.  Towards supporting evocation processes in creative design: A cognitive approach.  _International Journal of Human-Computer Studies_ 63, 4 (2005), 422–435.  https://doi.org/10.1016/j.ijhcs.2005.04.006 Computer support for creativity.
  * Brade et al. (2023) Stephen Brade, Bryan Wang, Mauricio Sousa, Sageev Oore, and Tovi Grossman. 2023.  Promptify: Text-to-Image Generation through Interactive Prompt Exploration with Large Language Models. In _Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology_ (San Francisco, CA, USA) _(UIST ’23)_. Association for Computing Machinery, New York, NY, USA, Article 96, 14 pages.  https://doi.org/10.1145/3586183.3606725
  * Brooks et al. (2023) Tim Brooks, Aleksander Holynski, and Alexei A. Efros. 2023.  InstructPix2Pix: Learning to Follow Image Editing Instructions. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 18392–18402.
  * Campbell (1960) Donald T Campbell. 1960.  Blind variation and selective retentions in creative thought as in other knowledge processes.  _Psychological review_ 67, 6 (1960), 380.
  * Cassidy (2011) Tracy Cassidy. 2011.  The Mood Board Process Modeled and Understood as a Qualitative Design Research Tool.  _Fashion Practice_ 3, 2 (2011), 225–251.  https://doi.org/10.2752/175693811X13080607764854 arXiv:https://doi.org/10.2752/175693811X13080607764854
  * Chan et al. (2016) Joel Chan, Steven Dang, and Steven P. Dow. 2016.  Comparing Different Sensemaking Approaches for Large-Scale Ideation. In _Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems_ (San Jose, California, USA) _(CHI ’16)_. Association for Computing Machinery, New York, NY, USA, 2717–2728.  https://doi.org/10.1145/2858036.2858178
  * Chen et al. (2023) Minghao Chen, Iro Laina, and Andrea Vedaldi. 2023.  Training-Free Layout Control with Cross-Attention Guidance.  arXiv:2304.03373 [cs.CV]
  * Cherry and Latulipe (2014) Erin Cherry and Celine Latulipe. 2014.  Quantifying the Creativity Support of Digital Tools through the Creativity Support Index.  _ACM Trans. Comput.-Hum. Interact._ 21, 4, Article 21 (jun 2014), 25 pages.  https://doi.org/10.1145/2617588
  * Chilton et al. (2019b) Lydia B. Chilton, Savvas Petridis, and Maneesh Agrawala. 2019b.  VisiBlends: A Flexible Workflow for Visual Blends. In _Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems_ (Glasgow, Scotland Uk) _(CHI ’19)_. Association for Computing Machinery, New York, NY, USA, 1–14.  https://doi.org/10.1145/3290605.3300402
  * Chilton et al. (2019a) Sam Ross Lydia B Chilton, Ecenaz Jen Ozmen, and Sam Ross. 2019a.  VisiFit: AI Tools to Iteratively Improve Visual Blends.  (2019).
  * Chouchoulas and Day (2007) Orestes Chouchoulas and A.K. Day. 2007.  Design Exploration Using A Shape Grammar With A Genetic Algorithm.  _Open House International_ 32 (06 2007), 26–35.  https://doi.org/10.1108/OHI-02-2007-B0004
  * Chung and Adar (2023a) John Joon Young Chung and Eytan Adar. 2023a.  Artinter: AI-Powered Boundary Objects for Commissioning Visual Arts. In _Proceedings of the 2023 ACM Designing Interactive Systems Conference_ (Pittsburgh, PA, USA) _(DIS ’23)_. Association for Computing Machinery, New York, NY, USA, 1997–2018.  https://doi.org/10.1145/3563657.3595961
  * Chung and Adar (2023b) John Joon Young Chung and Eytan Adar. 2023b.  PromptPaint: Steering Text-to-Image Generation Through Paint Medium-like Interactions. In _Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology_ (, San Francisco, CA, USA,) _(UIST ’23)_. Association for Computing Machinery, New York, NY, USA, Article 6, 17 pages.  https://doi.org/10.1145/3586183.3606777
  * Chung et al. (2021) John Joon Young Chung, Shiqing He, and Eytan Adar. 2021.  The Intersection of Users, Roles, Interactions, and Technologies in Creativity Support Tools. In _Proceedings of the 2021 ACM Designing Interactive Systems Conference_ (Virtual Event, USA) _(DIS ’21)_. Association for Computing Machinery, New York, NY, USA, 1817–1833.  https://doi.org/10.1145/3461778.3462050
  * Chung et al. (2022) John Joon Young Chung, Shiqing He, and Eytan Adar. 2022.  Artist Support Networks: Implications for Future Creativity Support Tools. In _Proceedings of the 2022 ACM Designing Interactive Systems Conference_ (Virtual Event, Australia) _(DIS ’22)_. Association for Computing Machinery, New York, NY, USA, 232–246.  https://doi.org/10.1145/3532106.3533505
  * Cross (1997) Nigel Cross. 1997.  Descriptive models of creative design: application to an example.  _Design Studies_ 18, 4 (1997), 427–440.  https://doi.org/10.1016/S0142-694X(97)00010-0 Descriptive models of design.
  * Datasculptor (2023) Datasculptor. 2023.  Image2LineDrawing.  https://huggingface.co/spaces/Datasculptor/Image2LineDrawing Hugging Face Spaces.
  * Design with Canva (2023) Design with Canva. 2023.  How to Use ChatGPT to Design Like a Pro.  https://www.youtube.com/watch?v=VmBLuvBf0xE.  [Online; accessed 2023-12-04].
  * Eckert and Stacey (2000) Claudia Eckert and Martin Stacey. 2000.  Sources of inspiration: a language of design.  _Design studies_ 21, 5 (2000), 523–538.
  * Frich et al. (2019a) Jonas Frich, Michael Mose Biskjaer, Lindsay MacDonald Vermeulen, Christian Remy, and Peter Dalsgaard. 2019a.  Strategies in Creative Professionals’ Use of Digital Tools Across Domains. In _Proceedings of the 2019 Conference on Creativity and Cognition_ (San Diego, CA, USA) _(C &C ’19)_. Association for Computing Machinery, New York, NY, USA, 210–221.  https://doi.org/10.1145/3325480.3325494
  * Frich et al. (2019b) Jonas Frich, Lindsay MacDonald Vermeulen, Christian Remy, Michael Mose Biskjaer, and Peter Dalsgaard. 2019b.  Mapping the Landscape of Creativity Support Tools in HCI. In _Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems_ (Glasgow, Scotland Uk) _(CHI ’19)_. Association for Computing Machinery, New York, NY, USA, 1–18.  https://doi.org/10.1145/3290605.3300619
  * Gal et al. (2022) Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H. Bermano, Gal Chechik, and Daniel Cohen-Or. 2022.  An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion.  arXiv:2208.01618 [cs.CV]
  * Garner and McDonagh-Philp (2001) Steve Garner and Deana McDonagh-Philp. 2001.  Problem interpretation and resolution via visual stimuli: the use of ‘mood boards’ in design education.  _Journal of Art & Design Education_ 20, 1 (2001), 57–64.
  * Gatys et al. (2016) Leon A. Gatys, Alexander S. Ecker, and Matthias Bethge. 2016.  Image Style Transfer Using Convolutional Neural Networks. In _2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_. 2414–2423.  https://doi.org/10.1109/CVPR.2016.265
  * Goel (1997) A.K. Goel. 1997.  Design, analogy, and creativity.  _IEEE Expert_ 12, 3 (1997), 62–70.  https://doi.org/10.1109/64.590078
  * Hart and Staveland (1988) Sandra G. Hart and Lowell E. Staveland. 1988.  Development of NASA-TLX (Task Load Index): Results of Empirical and Theoretical Research.  In _Human Mental Workload_ , Peter A. Hancock and Najmedin Meshkati (Eds.). Advances in Psychology, Vol. 52. North-Holland, 139–183.  https://doi.org/10.1016/S0166-4115(08)62386-9
  * Herring et al. (2009) Scarlett R. Herring, Chia-Chen Chang, Jesse Krantzler, and Brian P. Bailey. 2009.  Getting Inspired! Understanding How and Why Examples Are Used in Creative Design Practice. In _Proceedings of the SIGCHI Conference on Human Factors in Computing Systems_ (Boston, MA, USA) _(CHI ’09)_. Association for Computing Machinery, New York, NY, USA, 87–96.  https://doi.org/10.1145/1518701.1518717
  * Hirst (1992) Barbara Hirst. 1992.  How artists overcome creative blocks.  _The Journal of Creative Behavior_ (1992).
  * Holinaty et al. (2021) Josh Holinaty, Alec Jacobson, and Fanny Chevalier. 2021.  Supporting Reference Imagery for Digital Drawing. In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) Workshops_. 2434–2442.
  * Holyoak and Thagard (1996) Keith J Holyoak and Paul Thagard. 1996.  _Mental leaps: Analogy in creative thought_.  MIT press.
  * Hope et al. (2022) Tom Hope, Ronen Tamari, Daniel Hershcovich, Hyeonsu B Kang, Joel Chan, Aniket Kittur, and Dafna Shahaf. 2022.  Scaling Creative Inspiration with Fine-Grained Functional Aspects of Ideas. In _Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems_ (New Orleans, LA, USA) _(CHI ’22)_. Association for Computing Machinery, New York, NY, USA, Article 12, 15 pages.  https://doi.org/10.1145/3491102.3517434
  * Ivanov et al. (2022) Alexander Ivanov, David Ledo, Tovi Grossman, George Fitzmaurice, and Fraser Anderson. 2022.  MoodCubes: Immersive Spaces for Collecting, Discovering and Envisioning Inspiration Materials. In _Proceedings of the 2022 ACM Designing Interactive Systems Conference_ (Virtual Event, Australia) _(DIS ’22)_. Association for Computing Machinery, New York, NY, USA, 189–203.  https://doi.org/10.1145/3532106.3533565
  * Jacobs et al. (2018) Jennifer Jacobs, Joel Brandt, Radomír Mech, and Mitchel Resnick. 2018.  Extending Manual Drawing Practices with Artist-Centric Programming Tools. In _Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems_ (Montreal QC, Canada) _(CHI ’18)_. Association for Computing Machinery, New York, NY, USA, 1–13.  https://doi.org/10.1145/3173574.3174164
  * Jansson and Smith (1991) David G Jansson and Steven M Smith. 1991.  Design fixation.  _Design studies_ 12, 1 (1991), 3–11.
  * Jeon et al. (2021) Youngseung Jeon, Seungwan Jin, Patrick C. Shih, and Kyungsik Han. 2021.  FashionQ: An AI-Driven Creativity Support Tool for Facilitating Ideation in Fashion Design. In _Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems_ (Yokohama, Japan) _(CHI ’21)_. Association for Computing Machinery, New York, NY, USA, Article 576, 18 pages.  https://doi.org/10.1145/3411764.3445093
  * Kang et al. (2021) Youwen Kang, Zhida Sun, Sitong Wang, Zeyu Huang, Ziming Wu, and Xiaojuan Ma. 2021.  MetaMap: Supporting Visual Metaphor Ideation through Multi-Dimensional Example-Based Exploration. In _Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems_ (Yokohama, Japan) _(CHI ’21)_. Association for Computing Machinery, New York, NY, USA, Article 427, 15 pages.  https://doi.org/10.1145/3411764.3445325
  * Karimi et al. (2019) Pegah Karimi, Nicholas Davis, Mary Lou Maher, Kazjon Grace, and Lina Lee. 2019.  Relating Cognitive Models of Design Creativity to the Similarity of Sketches Generated by an AI Partner. In _Proceedings of the 2019 Conference on Creativity and Cognition_ (San Diego, CA, USA) _(C &C ’19)_. Association for Computing Machinery, New York, NY, USA, 259–270.  https://doi.org/10.1145/3325480.3325488
  * Karimi et al. (2020) Pegah Karimi, Jeba Rezwana, Safat Siddiqui, Mary Lou Maher, and Nasrin Dehbozorgi. 2020.  Creative Sketching Partner: An Analysis of Human-AI Co-Creativity. In _Proceedings of the 25th International Conference on Intelligent User Interfaces_ (Cagliari, Italy) _(IUI ’20)_. Association for Computing Machinery, New York, NY, USA, 221–230.  https://doi.org/10.1145/3377325.3377522
  * Karras et al. (2019) Tero Karras, Samuli Laine, and Timo Aila. 2019.  A Style-Based Generator Architecture for Generative Adversarial Networks. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * Karras et al. (2020) Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. 2020.  Analyzing and Improving the Image Quality of StyleGAN. In _IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * Kerne (2000) Andruid Kerne. 2000.  CollageMachine: An Interactive Agent of Web Recombination.  _Leonardo_ 33, 5 (10 2000), 347–350.  https://doi.org/10.1162/002409400552801 arXiv:https://direct.mit.edu/leon/article-pdf/33/5/347/1570572/002409400552801.pdf
  * Kim et al. (2022) Kevin Gonyop Kim, Richard Lee Davis, Alessia Eletta Coppi, Alberto Cattaneo, and Pierre Dillenbourg. 2022.  Mixplorer: Scaffolding Design Space Exploration through Genetic Recombination of Multiple Peoples’ Designs to Support Novices’ Creativity. In _Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems_ (New Orleans, LA, USA) _(CHI ’22)_. Association for Computing Machinery, New York, NY, USA, Article 308, 13 pages.  https://doi.org/10.1145/3491102.3501854
  * Kirillov et al. (2023) Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Dollár, and Ross Girshick. 2023.  Segment Anything.  arXiv:2304.02643 [cs.CV]
  * Ko et al. (2023) Hyung-Kwon Ko, Gwanmo Park, Hyeon Jeon, Jaemin Jo, Juho Kim, and Jinwook Seo. 2023.  Large-Scale Text-to-Image Generation Models for Visual Artists’ Creative Works. In _Proceedings of the 28th International Conference on Intelligent User Interfaces_ (Sydney, NSW, Australia) _(IUI ’23)_. Association for Computing Machinery, New York, NY, USA, 919–933.  https://doi.org/10.1145/3581641.3584078
  * Koch et al. (2019) Janin Koch, Andrés Lucero, Lena Hegemann, and Antti Oulasvirta. 2019.  May AI? Design Ideation with Cooperative Contextual Bandits. In _Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems_ (Glasgow, Scotland Uk) _(CHI ’19)_. Association for Computing Machinery, New York, NY, USA, 1–12.  https://doi.org/10.1145/3290605.3300863
  * Koch et al. (2020) Janin Koch, Nicolas Taffin, Andrés Lucero, and Wendy E. Mackay. 2020.  SemanticCollage: Enriching Digital Mood Board Design with Semantic Labels. In _Proceedings of the 2020 ACM Designing Interactive Systems Conference_ (Eindhoven, Netherlands) _(DIS ’20)_. Association for Computing Machinery, New York, NY, USA, 407–418.  https://doi.org/10.1145/3357236.3395494
  * Koestler (1964) A Koestler. 1964.  The Act of Creation: A study of the conscious and unconscious in art.
  * Lawton et al. (2023) Tomas Lawton, Kazjon Grace, and Francisco J Ibarrola. 2023.  When is a Tool a Tool? User Perceptions of System Agency in Human–AI Co-Creative Drawing. In _Proceedings of the 2023 ACM Designing Interactive Systems Conference_ (Pittsburgh, PA, USA) _(DIS ’23)_. Association for Computing Machinery, New York, NY, USA, 1978–1996.  https://doi.org/10.1145/3563657.3595977
  * Lee et al. (2010) Brian Lee, Savil Srivastava, Ranjitha Kumar, Ronen Brafman, and Scott R. Klemmer. 2010.  Designing with Interactive Example Galleries. In _Proceedings of the SIGCHI Conference on Human Factors in Computing Systems_ (Atlanta, Georgia, USA) _(CHI ’10)_. Association for Computing Machinery, New York, NY, USA, 2257–2266.  https://doi.org/10.1145/1753326.1753667
  * Lee et al. (2023) Seungwoo Lee, Chaerin Kong, Donghyeon Jeon, and Nojun Kwak. 2023.  AADiff: Audio-Aligned Video Synthesis with Text-to-Image Diffusion.  arXiv:2305.04001 [cs.CV]
  * Li et al. (2023a) Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023a.  BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models.  arXiv:2301.12597 [cs.CV]
  * Li et al. (2023b) Yuheng Li, Haotian Liu, Qingyang Wu, Fangzhou Mu, Jianwei Yang, Jianfeng Gao, Chunyuan Li, and Yong Jae Lee. 2023b.  GLIGEN: Open-Set Grounded Text-to-Image Generation.  _CVPR_ (2023).
  * Lian et al. (2023) Long Lian, Boyi Li, Adam Yala, and Trevor Darrell. 2023.  LLM-grounded Diffusion: Enhancing Prompt Understanding of Text-to-Image Diffusion Models with Large Language Models.  arXiv:2305.13655 [cs.CV]
  * Liu et al. (2023) Vivian Liu, Jo Vermeulen, George Fitzmaurice, and Justin Matejka. 2023.  3DALL-E: Integrating Text-to-Image AI in 3D Design Workflows. In _Proceedings of the 2023 ACM Designing Interactive Systems Conference_ (Pittsburgh, PA, USA) _(DIS ’23)_. Association for Computing Machinery, New York, NY, USA, 1955–1977.  https://doi.org/10.1145/3563657.3596098
  * Lucero (2015) Andrés Lucero. 2015.  Funky-Design-Spaces: Interactive Environments for Creativity Inspired by Observing Designers Making Mood Boards. In _Human-Computer Interaction – INTERACT 2015_ , Julio Abascal, Simone Barbosa, Mirko Fetter, Tom Gross, Philippe Palanque, and Marco Winckler (Eds.). Springer International Publishing, Cham, 474–492.
  * Lucero et al. (2009) Andrés Lucero, Dzmitry Aliakseyeu, Kees Overbeeke, and Jean-Bernard Martens. 2009.  An Interactive Support Tool to Convey the Intended Message in Asynchronous Presentations. In _Proceedings of the International Conference on Advances in Computer Entertainment Technology_ (Athens, Greece) _(ACE ’09)_. Association for Computing Machinery, New York, NY, USA, 11–18.  https://doi.org/10.1145/1690388.1690391
  * Matejka et al. (2018) Justin Matejka, Michael Glueck, Erin Bradner, Ali Hashemi, Tovi Grossman, and George Fitzmaurice. 2018.  Dream Lens: Exploration and Visualization of Large-Scale Generative Design Datasets. In _Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems_ (Montreal QC, Canada) _(CHI ’18)_. Association for Computing Machinery, New York, NY, USA, 1–12.  https://doi.org/10.1145/3173574.3173943
  * Mou et al. (2023) Chong Mou, Xintao Wang, Liangbin Xie, Yanze Wu, Jian Zhang, Zhongang Qi, Ying Shan, and Xiaohu Qie. 2023.  T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Models.  arXiv:2302.08453 [cs.CV]
  * Müller-Wienbergen et al. (2011) Felix Müller-Wienbergen, Oliver Müller, Stefan Seidel, and Jörg Becker. 2011.  Leaving the beaten tracks in creative work–A design theory for systems that support convergent and divergent thinking.  _Journal of the Association for Information Systems_ 12, 11 (2011), 2.
  * Mumford et al. (1991) Michael D Mumford, Michele I Mobley, Roni Reiter-Palmon, Charles E Uhlman, and Lesli M Doares. 1991.  Process analytic models of creative capacities.  _Creativity research journal_ 4, 2 (1991), 91–122.  https://doi.org/10.1080/10400419209534428 arXiv:https://doi.org/10.1080/10400419209534428
  * Nagai et al. (2009) Yukari Nagai, Toshiharu Taura, and Futoshi Mukai. 2009.  Concept blending and dissimilarity: factors for creative concept generation process.  _Design Studies_ 30, 6 (2009), 648–675.  https://doi.org/10.1016/j.destud.2009.05.004
  * Nichol et al. (2022) Alexander Quinn Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob Mcgrew, Ilya Sutskever, and Mark Chen. 2022.  GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models. In _Proceedings of the 39th International Conference on Machine Learning_ _(Proceedings of Machine Learning Research, Vol. 162)_ , Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (Eds.). PMLR, 16784–16804.  https://proceedings.mlr.press/v162/nichol22a.html
  * Oh et al. (2018) Changhoon Oh, Jungwoo Song, Jinhan Choi, Seonghyeon Kim, Sungwoo Lee, and Bongwon Suh. 2018.  I Lead, You Help but Only with Enough Details: Understanding User Experience of Co-Creation with Artificial Intelligence. In _Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems_ (Montreal QC, Canada) _(CHI ’18)_. Association for Computing Machinery, New York, NY, USA, 1–13.  https://doi.org/10.1145/3173574.3174223
  * OpenAI (2020) OpenAI. 2020.  Language Models are Few-Shot Learners.  arXiv:2005.14165 [cs.CL]
  * OpenAI (2023) OpenAI. 2023.  GPT-4 Technical Report.  arXiv:2303.08774 [cs.CL]
  * Palmer et al. (2023) Taylor Palmer, Jordan Bowman, and Tommy Geoco. 2023.  2023 Design Tools Survey - AI.  https://uxtools.co/survey/2023/ai
  * Pilat and Jacob (2008) Marcin L. Pilat and Christian Jacob. 2008.  Creature Academy: A system for virtual creature evolution. In _2008 IEEE Congress on Evolutionary Computation (IEEE World Congress on Computational Intelligence)_. 3289–3297.  https://doi.org/10.1109/CEC.2008.4631243
  * Radford et al. (2021) Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021.  Learning Transferable Visual Models From Natural Language Supervision. In _Proceedings of the 38th International Conference on Machine Learning_ _(Proceedings of Machine Learning Research, Vol. 139)_ , Marina Meila and Tong Zhang (Eds.). PMLR, 8748–8763.  https://proceedings.mlr.press/v139/radford21a.html
  * Ramesh et al. (2022) Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. 2022.  Hierarchical Text-Conditional Image Generation with CLIP Latents.  arXiv:2204.06125 [cs.CV]
  * Ritchie et al. (2011) Daniel Ritchie, Ankita Arvind Kejriwal, and Scott R. Klemmer. 2011.  D.Tour: Style-Based Exploration of Design Example Galleries. In _Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology_ (Santa Barbara, California, USA) _(UIST ’11)_. Association for Computing Machinery, New York, NY, USA, 165–174.  https://doi.org/10.1145/2047196.2047216
  * Rombach et al. (2022) Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. 2022.  High-Resolution Image Synthesis With Latent Diffusion Models. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_. 10684–10695.
  * Ruiz et al. (2023) Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. 2023.  DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_. 22500–22510.
  * Satori Graphics (2023) Satori Graphics. 2023.  Ai Tools That TOTALLY Innovate A Designer’s Life!  https://www.youtube.com/watch?v=eTffPm_e1ko.  [Online; accessed 2023-12-04].
  * Shneiderman (2000) Ben Shneiderman. 2000.  Creating Creativity: User Interfaces for Supporting Innovation.  _ACM Trans. Comput.-Hum. Interact._ 7, 1 (mar 2000), 114–138.  https://doi.org/10.1145/344949.345077
  * Siangliulue et al. (2015) Pao Siangliulue, Joel Chan, Krzysztof Z. Gajos, and Steven P. Dow. 2015.  Providing Timely Examples Improves the Quantity and Quality of Generated Ideas. In _Proceedings of the 2015 ACM SIGCHI Conference on Creativity and Cognition_ (Glasgow, United Kingdom) _(C &C ’15)_. Association for Computing Machinery, New York, NY, USA, 83–92.  https://doi.org/10.1145/2757226.2757230
  * Simonton (2003) Dean Keith Simonton. 2003.  Scientific creativity as constrained stochastic behavior: the integration of product, person, and process perspectives.  _Psychological bulletin_ 129, 4 (2003), 475.
  * Stemasov et al. (2023) Evgeny Stemasov, David Ledo, George Fitzmaurice, and Fraser Anderson. 2023.  Immersive Sampling: Exploring Sampling for Future Creative Practices in Media-Rich, Immersive Spaces. In _Proceedings of the 2023 ACM Designing Interactive Systems Conference_ (Pittsburgh, PA, USA) _(DIS ’23)_. Association for Computing Machinery, New York, NY, USA, 212–229.  https://doi.org/10.1145/3563657.3596131
  * Sung-Bin et al. (2023) Kim Sung-Bin, Arda Senocak, Hyunwoo Ha, Andrew Owens, and Tae-Hyun Oh. 2023.  Sound to Visual Scene Generation by Audio-to-Visual Latent Alignment. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_. 6430–6440.
  * Turrin et al. (2011) Michela Turrin, Peter von Buelow, and Rudi Stouffs. 2011.  Design explorations of performance driven geometry in architectural design using parametric modeling and genetic algorithms.  _Advanced Engineering Informatics_ 25, 4 (2011), 656–675.  https://doi.org/10.1016/j.aei.2011.07.009 Special Section: Advances and Challenges in Computing in Civil and Building Engineering.
  * Vladimir Vorobev (2023) Maxim Kuznetsov Vladimir Vorobev. 2023.  A paraphrasing model based on ChatGPT paraphrases.
  * Wang et al. (2023) Sitong Wang, Savvas Petridis, Taeahn Kwon, Xiaojuan Ma, and Lydia B Chilton. 2023.  PopBlends: Strategies for Conceptual Blending with Large Language Models. In _Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems_ (Hamburg, Germany) _(CHI ’23)_. Association for Computing Machinery, New York, NY, USA, Article 435, 19 pages.  https://doi.org/10.1145/3544548.3580948
  * Wang et al. (2020) Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou. 2020.  MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers. In _Advances in Neural Information Processing Systems_ , H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (Eds.), Vol. 33. Curran Associates, Inc., 5776–5788.  https://proceedings.neurips.cc/paper_files/paper/2020/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf
  * Wu et al. (2022) Tongshuang Wu, Michael Terry, and Carrie Jun Cai. 2022.  AI Chains: Transparent and Controllable Human-AI Interaction by Chaining Large Language Model Prompts. In _Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems_ (New Orleans, LA, USA) _(CHI ’22)_. Association for Computing Machinery, New York, NY, USA, Article 385, 22 pages.  https://doi.org/10.1145/3491102.3517582
  * Yu and Nickerson (2011) Lixiu Yu and Jeffrey V. Nickerson. 2011.  Cooks or Cobblers? Crowd Creativity through Combination. In _Proceedings of the SIGCHI Conference on Human Factors in Computing Systems_ (Vancouver, BC, Canada) _(CHI ’11)_. Association for Computing Machinery, New York, NY, USA, 1393–1402.  https://doi.org/10.1145/1978942.1979147
  * Zaman et al. (2015) Loutfouz Zaman, Wolfgang Stuerzlinger, Christian Neugebauer, Rob Woodbury, Maher Elkhaldi, Naghmi Shireen, and Michael Terry. 2015.  GEM-NI: A System for Creating and Managing Alternatives In Generative Design. In _Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems_ (Seoul, Republic of Korea) _(CHI ’15)_. Association for Computing Machinery, New York, NY, USA, 1201–1210.  https://doi.org/10.1145/2702123.2702398
  * Zhang and Banovic (2021) Enhao Zhang and Nikola Banovic. 2021.  Method for Exploring Generative Adversarial Networks (GANs) via Automatically Generated Image Galleries. In _Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems_ (Yokohama, Japan) _(CHI ’21)_. Association for Computing Machinery, New York, NY, USA, Article 76, 15 pages.  https://doi.org/10.1145/3411764.3445714
  * Zhang et al. (2023) Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. 2023.  Adding Conditional Control to Text-to-Image Diffusion Models.  , 3836-3847 pages.
  * Zhao et al. (2020) Nanxuan Zhao, Nam Wook Kim, Laura Mariah Herman, Hanspeter Pfister, Rynson W.H. Lau, Jose Echevarria, and Zoya Bylinskii. 2020.  ICONATE: Automatic Compound Icon Generation and Ideation. In _Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems_ (Honolulu, HI, USA) _(CHI ’20)_. Association for Computing Machinery, New York, NY, USA, 1–13.  https://doi.org/10.1145/3313831.3376618
  * Zheng et al. (2023) Guangcong Zheng, Xianpan Zhou, Xuewei Li, Zhongang Qi, Ying Shan, and Xi Li. 2023.  LayoutDiffusion: Controllable Diffusion Model for Layout-to-Image Generation. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_. 22490–22499.
  * Zhu (2023) Xinyi Zhu. 2023.  Research Guides: Machines and Society: ChatGPT for Visual Design.  https://guides.nyu.edu/data/chatgpt-visual-design

##  Appendix A Technical Details

###  A.1. Prompt: Extracting Keywords from Image Captions

System Prompt:
You will be provided with multiple sentences to describe an illustration. Your
task is to extract a list of Subject matter, Action & pose, and Theme & mood.
Subject matters are one-word, describing the specific physical objects,
characters, or landscape that the user wants to include in their illustration.
Example subject matters include pencil, children, or wave. For subject
matters, no adjectives should be included. They should always be a noun.
Actions & poses are word-level or phrase-level actions that the character or
the object in the illustration performs. Example actions & poses include
riding a bus, standing still, or traveling.
Themes & moods are words not directly present in the illustration, but those
that can potentially convey the overall theme or mood of the illustration.
Example themes & moods include imaginative, eco-friendly, or sad. They should
be adverbs, preferably one or two words.
If you are provided sentences including some style such as cartoon,
illustration, image, or photo, exclude it. For other examples, ’an
illustration of a woman sitting at a table’ caption is extracted to ’woman’,
’table’, ’sitting at a table’, ’cozy’. The ’illustration’ is not contained.
Eliminate the changed forms of the same word, such as plurals. Only include
roots. For example of ’trees’ and ’tree’, only include ’tree’."

Examples:
User:
a card with chinese writing with colorful objects on it
a red and orange background with a blank paper, chinese, pencils, stationery
and more
an image of a classroom scene with various supplies
Assistant:
Subject matter: card, Chinese writing, colorful objects, red and orange
background, blank paper, Chinese, pencils, stationery, classroom, supplies.
Action & pose:
Theme & mood: education, learning, multiculturalism
User:
a man and a woman sitting at a table eating food
a woman sitting in a chair in a room with potted plants
a man sitting at a table eating a hot dog
a illustration of a woman sitting at a table in a kitchen with potted plants
an illustration of a woman sitting at a table
an illustration of a refrigerator next to a chair and a table
an illustration of two people sitting at a table with their feet on a chair
a cartoon of a person sitting in a chair in a shower
Assistant:
Subject matter: man, woman, table, food, chair, potted plants, hot dog,
kitchen, refrigerator, feet
Action & pose: sitting at a table, eating food
Theme & mood: cozy, heartwarming
User:
a painting of a white barn in a field of flowers
a painting of a blue house with a tree next to it
a painting of a blue house with a roof and trees
a painting of a window of a house with flowers
a painting of a field of flowers in front of a house
Assistant:
Subject matter: painting, white barn, field, flowers, blue house, tree, roof,
window
Action & pose:
Theme & mood: rural, peaceful, nature
User:
the album cover of the beatles abbey road
a man and a woman standing in front of a car
a man and a woman walking down a street
a group of people walking across a crosswalk
Assistant:
Subject matter: Beatles, Abbey Road, man, woman, car, street, group, people,
crosswalk
Action & pose: standing in front of a car, walking down a street, walking
across a crosswalk
Theme & mood: urban, nostalgia

###  A.2. Prompt: Recommending Relevant Keywords

System Prompt:
We are trying to support novice designers’ ideation process by semantically
combining different parts of illustration references. You will be provided
with the topic of the ideation, and multiple keywords users like in the
illustrations they found as references. There are three types of keywords:
Subject matter, Action & Pose, and Theme & Mood.
Subject matters are one-word, describing the specific physical objects,
characters, or landscape that the user wants to include in their illustration.
Example subject matters include pencil, children, or wave. For subject
matters, no adjectives should be included. They should always be a noun. Come
up with more than four new keywords for Subject matter.
Actions & poses are word-level or phrase-level actions that the character or
the object in the illustration performs. Example actions & poses include
riding a bus, standing still, or traveling.
Themes & moods are words not directly present in the illustration, but those
that can potentially convey the overall theme or mood of the illustration.
Example themes & moods include imaginative, eco-friendly, or sad. They should
be adverbs, preferably one word.
Your task is to expand on the keywords being given, by combining multiple
keywords or looking for synonyms that can inspire new creations or ideas. For
example, the subject matter "pencil" can be combined with the action & pose
"traveling" to inspire a new action & pose "writing a diary". You can combine
as many keywords at once. Another example is to generate "hair salon" from
"hair dryer", "comb", and "scissors". For combinations that result in theme &
mood, make them as abstract as possible. An example is to make "adventurous",
"gusty" from "riding on ship" and "tent".
Come up with new keywords for each keyword type with creative combinations.
Only use the original keywords provided for creating new keywords. Do not just
paraphrase original keywords. Do not suggest similar keywords to the original
ones.
Important: Include at least one subject matter for each combination. Subject
matter and theme & mood should be a SINGLE WORD. Combinations among subject
matters are highly recommended. New keywords should be śurprisingćompared to
original ones. It means the character of your suggested word should have low
similarity.’

Examples:
User:
Subject matter: camping, tent, tree, animals, Eiffel tower, family
Action & pose: riding on a bus, riding on a ship
Theme & mood: playful, imaginative
Assistant:
Subject matter: bear, sleeping person, safari, cruise
Action & pose: traveling, setting up camp, dancing jazz
Theme & mood: adventurous, serene, joyful, romantic
User:
Subject matter: boy, dinosaur, flower
Action & pose: watching television
Theme & mood: fantasy, playful
Assistant:
Subject matter: wind mill, volcano, movie screen
Action & pose: exploding strongly, riding a dinosaur, flying away to the sky
Theme & mood: vast, whimsical, rustic, frenetic
User:
Subject matter: dreamy scene, boy
Action & pose: playing with dino toys
Assistant:
Subject matter: universe, Saturn, astronauts
Action & pose: imagining adventures, floating on the space, role-playing,
daydreaming
Theme & mood: jolly, imaginative, impactful
User:
Subject matter: Christmas tree
Action & pose: dancing around the Christmas tree
Theme & mood: family-bonding
Assistant:
Subject matter: Fireplace, wooden sled, Snowman, jazz, piano
Action & pose: melting, giving present, body-warming
Theme & mood: jubilant, sparkling, heartwarming
User:
Subject matter: sea turtles, Christmas tree, marine life
Action & pose: swimming, dancing around the Christmas tree
Theme & mood: fantasy, underwater, family-bonding
Assistant:
Subject matter: Sea horse, Christmas lights, coral, mermaid
Action & pose: floating on the wave, blinking eye, singing under the sea
Theme & mood: ethereal, aquatic, charming, panoramic
User:
Subject matter: kid, cat
Action & pose: laying on top of a suitcase, playing hide and seek
Theme & mood: Rustic, vivid, exhilarating
Assistant:
Subject matter: Birdcage, attic, trunk, blue bird
Action & pose: jumping on boxes, chasing birds, hiding in a suitcase
Theme & mood: quaint, mischievous, lively, nostalgic

###  A.3. Prompt: Generating Recombinations in Text Descriptions

System Prompt:
The user wants to draw an illustration, with the assistance of you. You will
be provided with multiple keywords users want to include in their
illustrations. There are three types of keywords: Subject matter, Action &
pose, and Theme & mood.
Subject matters are one-word, describing the specific physical objects,
characters, or landscape that the user wants to include in their illustration.
Example subject matters include pencil, children, or wave. For subject