Figure 1. Overview of the Flowy system’s approach of UX design support. Flowy
system uses a pipeline enabled by multimodal AI to annotate design patterns in
high-quality user flow examples from existing apps (A). It provides designers
with an analysis of the underlying design patterns in the examples (B) and
surfaces other high-quality design examples with related patterns (C). This
scaffolds UX designers’ decision making processes and accelerates their
information foraging process in design ideation, facilitating a holistic
understanding of the design space.

The figure consists of multiple smartphone screenshots and user interface
elements arranged in a collage layout. The central part of the figure (labeled
A) shows the system’s multimodal AI-enabled design pattern annotation feature.
It displays a smartphone screenshot with various UI components, such as
buttons and text fields, along with a panel on the right side showing the
purpose, advantages, disadvantages, and considerations of a specific design
pattern. The left side of the figure showcases user flow examples from
existing apps on smartphone screens. The right side of the figure (labeled B)
presents the analysis of the underlying design patterns. It includes three
smartphone screenshots displaying different design patterns, such as a filter
screen, a cart screen, and a bottom sheet with buttons and sliders. Arrows
connect the user flow examples to the central AI annotation feature and then
to the design pattern analysis screens, illustrating the system’s workflow in
supporting designers’ UX inspiration foraging and informed decision-making
process.

# Flowy: Supporting UX Design Decisions Through AI-Driven Pattern Annotation
in Multi-Screen User Flows

Yuwen Lu  University of Notre DameNotre DameINUSA<EMAIL_ADDRESS>,  Ziang Tong
University of Notre DameNotre DameINUSA<EMAIL_ADDRESS>,  Qinyi Zhao
University of WashingtonSeattleWAUSA<EMAIL_ADDRESS>,  Yewon Oh  University of
Notre DameNotre DameINUSA<EMAIL_ADDRESS>,  Bryan Wang  University of
TorontoTorontoONCanada<EMAIL_ADDRESS>and  Toby Jia-Jun Li  University
of Notre DameNotre DameINUSA<EMAIL_ADDRESS>
(2024)

###### Abstract.

Many recent AI-powered UX design tools focus on generating individual static
UI screens from natural language. However, they overlook the crucial aspect of
interactions and user experiences across multiple screens. Through formative
studies with UX professionals, we identified limitations of these tools in
supporting realistic UX design workflows. In response, we designed and
developed Flowy, an app that augments designers’ information foraging process
in ideation by supplementing specific user flow examples with distilled design
pattern knowledge. Flowy utilizes large multimodal AI models and a high-
quality user flow dataset to help designers identify and understand relevant
abstract design patterns in the design space for multi-screen user flows. Our
user study with professional UX designers demonstrates how Flowy supports
realistic UX tasks. Our design considerations in Flowy, such as
representations with appropriate levels of abstraction and assisted navigation
through the solution space, are generalizable to other creative tasks and
embody a human-centered, intelligence augmentation approach to using AI in UX
design.

††journalyear: 2024

##  1\. Introduction

Recent advances in artificial intelligence (AI) have spurred the development
of research prototypes and commercial tools designed to aid in User Interface
(UI) and User Experience (UX) design. A prominent group of these UI/UX design
support tools automatically generate mid-fidelity UI wireframes (Lu et al.,
2023; Feng et al., 2023b; Huang et al., 2021) or high-fidelity UI mockups111A
non-exclusive list of non-academic tools include Uizard, Galileo, V0 by
Vercel, Magic Patterns. (Zhao et al., 2021; Wei et al., 2023; Mozaffari et
al., 2022). Most of these tools follow a “prompt-to-UI” interaction paradigm,
similar to the prompt-based image generation paradigm in diffusion models
(Dhariwal and Nichol, 2021; Ho et al., 2020), where a user provides the
description of a UI and the system returns a high-fidelity mockup, aiming to
reduce designers’ manual efforts in prototyping.

However, existing prompt-to-UI tools predominantly focus on individual static
UI design, thereby not fully addressing the broader and crucial aspect of
experience design and interactions in UX (Norman and Nielsen, 1998; Norman,
2013), which is oftern embodied in user flows that extend beyond the scope of
individual UI screens. This observation led us to hypothesize that the current
prompt-to-UI tools might not adequately support the workflows of professional
UX designers.

To validate this hypothesis, we conducted formative studies with professional
UX designers to understand the limitations of existing prompt-to-UI tools in
real-world UX design workflows. We identified significant gaps between these
tools’ capabilities and the needs of designers, including: (1) the inability
to generate designs based on corporate design systems (Frost, 2016; Churchill,
2019), (2) the lack of design rationale or justifications, and (3) the narrow
focus on individual UI screens, overlooking critical UX details and contexts.
We also learned about the important role of design patterns in helping UX
designers understand the design considerations and potential design space for
different product features. Furthermore, we discovered that the UX design
process, particularly the ideation phase, resembles an information foraging
process, where individuals use strategies to maximize the rate of acquiring
valuable information while minimizing search costs (Pirolli and Card, 1999).
For example, UX designers often need to explore a vast design space of user
flows and interactions, implicitly evaluating and selecting examples most
relevant to their design context. Motivated by these findings, we aim to
develop tools that enhance the designers’ inspiration foraging process during
ideation, rather than building fully automatic UI generation tools, which have
fallen short in supporting the UX design process.

We present Flowy, an interactive system that enhances and streamlines UX
ideation through the automated analysis of UX design patterns. Flowy
automatically identifies three types of patterns: component layout, user
interaction, and product feature-specific patterns, aiding designers in
decision-making across diverse scenarios. To accomplish this, Flowy leverages
a novel pipeline based on multimodal generative AI models, including vision-
language models, and utilizes a high-quality commercial dataset, featuring
comprehensive user flow exmaples of mobile UIs from
Mobbin222https://mobbin.com/. With Flowy, designers can extend their vision
beyond the design of individual UI screen by accessing information on design
patterns across multiple screens in user flows.

We conducted a user study with eight professional UX designers to understand
the effectiveness of Flowy in supporting realistic UX design tasks. The study
results are promising, with participants reporting that Flowy’s design pattern
analysis and visual comparison of similar examples help guide their attention
and facilitate the learning of design patterns. Moreover, through the study,
we identified important areas for future work in assisting real-world UI/UX
design, such as further integrating product goals or business objectives and
prioritizing visuals over textual descriptions in similar tools. Based on
these findings, we derive three key design implications that shed light on
future research in UX design ideation support tools. More broadly, Flowy
exemplifies a intelligence-augmentation approach to human-AI collaboration,
with the goal of supporting the cognitive processes of target users to improve
task efficiency and effectiveness.

Altogether, Flow provides the following contributions:

  1. (1)

The design and implementation of Flowy, a novel system that supports UX design
ideation as an information foraging process;

  2. (2)

A technical pipeline leveraging state-of-the-art multimodal AI to
automatically annotate design pattern information on user flow design
examples;

  3. (3)

Study results demonstrating the effectiveness of Flowy, along with three
design implications for UX ideation support tools proposed based on the study
findings.

##  2\. Background and Related Work

###  2.1. Data-Driven UI/UX Design Tools

There have been long-lasting efforts to build tools that support the UI design
process. For example, non-AI tools have attempted to help designers with
automatic exploration of design alternatives (e.g. Scout (Swearngin et al.,
2020), SpaceWalker (Zhong et al., 2021)) and adapting screenshot examples into
designs (e.g. Rewire (Swearngin et al., 2018)). Since UI design is inherently
diverse and challenging to capture with rule-based methods, more recently,
studies have leveraged data-driven approaches to learn design semantics or
patterns from large-scale UI datasets. For instance, the Rico (Deka et al.,
2017) dataset is one of the most prominent open-source UI datasets for
Android. ERICA (Deka et al., 2016) offers a collection of user interaction
data for mobile UIs, captured during app usage. WebUI (Wu et al., 2023) is a
dataset for enhancing visual UI understanding with web semantics. Widget
Captioning (Li et al., 2020) provides language descriptions of individual UI
elements, while Screen2Words (Wang et al., 2021) delivers summaries of entire
UI screens. Building on these datasets, prior research has explored machine
learning approaches to model UI semantics, achieve computational understanding
of UIs, and enable novel interactions with UIs. For example, Screen2Vec (Li et
al., 2021) employs a self-supervised technique for generating representations
of GUI screens in embedding vectors. More recently, there has been significant
interest in modeling UIs using foundation models. For instance, Wang et al.
(Wang et al., 2023) utilized a large language model to process mobile UIs,
enabling diverse conversational interactions with the UI. Conversely,
Spotlight (Li and Li, 2023) proposed a vision-only approach for understanding
mobile UIs. In this study, Flowy built upon this prior work and proposed a
computational pipeline that integrates state-of-the-art multimodal AI models
to articulate design pattern information on UX prototypes and provide support
for UX designers.

It is noteworthy that the prevalent adoption of design systems in the UX
industry has significantly re-shaped the landscape of UX design research.
Design systems are centralized collections of pre-defined UI components and
design guidelines that ensure design consistency across products and teams
(Frost, 2016; Churchill, 2019). Examples of established design systems in the
industry nowadays include Apple’s Human Interface
Guidelines333https://developer.apple.com/design/human-interface-guidelines and
Google’s Material Design444https://material.io/. As a result of the wide
adoption of design systems, UX designers focus less on crafting pixel-perfect
visual details. Instead, they spend more time creating holistic user
experiences that support user goals and product objectives using pre-built UI
components (Lu et al., 2022). This change in UX practice is often
underestimated in academia.

###  2.2. Ideation with Generative AI

Generative AI has been utilized to enhance user creativity by quickly
providing a wide array of ideas in various formats, including text and
visuals, aiding users in overcoming creative blocks and explore new concepts.
The ideation process may involve the generation of text that directly
articulates ideas (Di Fede et al., 2022; YIN et al., 2024; Gonzalez et al.,
2024; Girotra et al., 2023; Shaer et al., 2024). Moreover, ideation can also
be based on examples, such as when users generate various images using a text-
to-image generation model to explore design examples they prefer (Paananen et
al., 2023; Lamac, 2023; Oppenlaender, 2022; Ko et al., 2023; Brade et al.,
2023; Qu et al., 2023). While straightforward prompting has been widely used
to harness generative AI for idea generation, recent research has explored
more systematic approaches to further enhance the ideation process, such as
incorporating object-oriented principles into LLM prompting (Kim et al., 2023)
and enabling structured exploration and expansion of the design space by
leveraging LLM outputs (Suh et al., 2023). In this work, Flowy takes a less
common approach to inspiration. Instead of generating example results to help
designers find new inspiration, Flowy focuses on augmenting designers’ natural
information foraging process for UX inspirations through design pattern
analysis.

###  2.3. Supporting the Information Foraging Process

Since its proposal in 2016, the information foraging theory has become a
seminal in understanding how humans seek and gather information (Pirolli and
Card, 1999). Information foraging theory draws an analogy between how humans
search for information and how animals forage for food in the wild. It
provides a powerful framework for analyzing and predicting human information-
seeking behavior. Studies have been done to extend the theory and try to
understand people’s information foraging behaviors in different contexts
(Sandstrom, 1994; Lawrance et al., 2010; Dwairy et al., 2011; Vigo and Harper,
2013). However, few studies have investigated ways to explicitly support
people’s information foraging processes. Vattam et al. (Vattam and Goel, 2011)
developed an information-processing model of seeking bio-inspiration in online
information environments and proposed an approach for enhancing these
environments by augmenting individual information resources with conceptual
models, to make finding relevant information more efficient. They divided the
inspiration foraging process into two main stages: within-patch foraging and
between-patch foraging, and supported each stage differently to cater to
foragers’ informational needs. Taking inspiration from these works, Flowy took
a novel approach to applying information foraging theory to the UX design
inspiration domain. The design considerations in Flowy reflect a human-
centered, intelligence augmentation approach of AI adoption for supporting UX
designers’ information foraging processes of inspirations in early design
stages.

##  3\. Formative Studies

The concept of user flow, while long-standing in design practices (Kaplan,
2023; Foundation, [n. d.]; Deaton, 2003), has been understudied in academic
research. User flows are often represented using simple flow diagrams and are
reflected in various design artifacts, including user journey maps, UI widget
flows, and UI prototypes of various modalities (Kaplan, 2023; Foundation, [n.
d.]). Despite their widespread use in practice, little empirical research has
been done to understand UX designers’ practices regarding user flows.
Additionally, we observed that most AI tools designed to support UX work
primarily focus on generating individual UI screens, neglecting the broader
context of user flows in the design process. Consequently, our formative study
aimed to explore the following research questions to fill this gap:

  1. (1)

RQ1: What feedback do designers have towards existing prompt-to-UI AI design
support tools?

  2. (2)

RQ2: How do UX designers manage user flows using different abstractions and
design artifacts such as user journey diagrams, UI widget flows, and high-
fidelity UI screens?

Our goal is to understand user flow as a unique aspect of UX design beyond
static UI design, and further identify opportunities for naturally integrating
AI support into UX design workflows. We recruited five professional UX
designers through online advertising and word of mouth. A description of the
participant demographics is shown in Appendix 1. We conducted a semi-
structured, one-hour interview with each participant. In our interviews, we
used retrospective analysis, a common method to reconstruct the behaviors,
rationales, and emotions of the study participants for the recorded events
(Russell and Chi, 2014). We asked participants to refer to digital design and
research files from a previous design project while explaining the behavior,
rationale, and ideas in the design processes. The interview questions were
organized into three sections:

  1. (1)

Understanding the participant’s previous UX design project;

  2. (2)

Practices to manage user journeys, user flows, UI widgets, and high-fidelity
prototypes in the design project;

  3. (3)

Testing out a representative existing AI-powered prompt-to-UI generation tool,
Uizard555We selected Uizard (https://uizard.io/) as one of the earliest tool
in this space. It shares the same core features and very similar experiences
with alternatives such as V0 by Vercel (https://v0.dev/) and Galileo
(https://www.usegalileo.ai/, not yet available during the time of our study)..

Each participant was compensated $15 for their time666The study protocol was
reviewed and approved by the Institutional Review Board (IRB) at our
institution.. The interview notes were analyzed using affinity diagramming
(Pernice, 2018) by four of the authors to identify common themes in the
responses of the participants. The subsequent sections detail the insights
derived from our affinity diagramming analysis.

###  3.1. Analysis Findings

####  3.1.1. Finding 1: Existing prompt-to-UI tools provide limited utility
for UX design

During the interviews, all designers confirmed our hypothesis that existing
prompt-to-UI tools are too focused on individual UI screens. Thereby, these
tools miss the broader context of user flows vital for UX design, as
unanimously noted by the participants (P1–5) (Norman and Nielsen, 1998). Most
participants did not continue to improve on the prompts and generate UI
screens after the first round of interaction with Uizard, finding the
generated results of little help. P4 reported with frustration that they are
“not sure about the tool’s (Uizard’s) jobs to be done… it is very far from
current UX designers’ practices”. P3 emphasized that in UX design “you are not
designing for aesthetics, but for functionality”, explaining that Uizard
cannot support realistic UX design tasks beyond the visual aspects of
individual UI screens. With the proliferation of design systems (Churchill,
2019) that contains pre-defined UI components for UX designers to use,
designing high-quality visual UI screens is no longer a major challenge for UX
designers (Lu et al., 2022), further limiting existing AI tools’ value for UX
design.

Designers identified three main gaps between prompt-to-UI AI tools and
realistic UX workflows. First, prompt-to-UI AI tools generate overly generic
and simplistic results that cannot meet the requirements of real-world
projects. P2 felt that the AI model “does not understand the complexity that a
screen holds”, while P3 mentioned that the AI generated results “can be
helpful only in generic industries”. The second gap is that no rationale of
design decisions are provided with the generated UI results, leaving designers
without the insights needed to leverage these outputs as meaningful
inspiration. P1 highlighted that in UX inspirations, “it’s really important to
understand the rationale behind the design scenarios and the selection of
design patterns”. Such rationales contain practical value in communications
when designers need to justify their design decisions to teammates and
stakeholders (P1, P3, P4). In contrast, examples from existing products, like
the ones featured on Mobbin, have been tested with real users, and thus serve
as better, more trustworthy inspirations (P1, P4). Lastly, integration with
organizational design systems presents a challenge. Due to the widespread
expectation that professional UX designers need to work with their
organizations’ design systems, the inability of current AI tools to
accommodate these systems effectively forces designers to engage in almost as
much effort as would be required to develop a new prototype from scratch (P1,
P3, P4).

####  3.1.2. Finding 2: User flow management is highly adaptive and project-
specific

Regarding RQ2, designers reported that they adapt or invent many forms of
representations and artifacts based on the needs and requirements of
individual projects to visualize, understand, and manage user flows (P1, P3,
P4, P5). Basic flow diagrams are versatile and functional enough in most cases
(P2, P3, P4). UX designers often also need to manage other aspects of the
system associated with user flows, such as the features that support each step
of the user flow (P4) and the accompanying data flow (P5). P4 mentioned a
specific intermediary design artifact named “breadboard” (Singer, [n. d.])
they found very useful to manage features in user flows, while recognizing
that it is not yet well-known in the UX community. P5’s team specifically
“invented” an artifact called “receipe” to list and manage the changing data
flow across multiple UI screens in a previous project. Meanwhile, P3 pointed
out that deliberate management of user flows is only necessary when they are
complex and not self-explanatory. In many cases, “user flows are often
directly encapsulated in high-fidelity prototypes”, with “no specific
abstractions or artifacts for them” (P3). In general, designers’ adaptive
management of user flows echos previous research findings that UX
methodologies are “more like mindsets” than formal procedures to follow (Gray,
2016). It helps designers to make sense of the problem domain, which is
critical in practical UX design, but unsupported by existing prompt-to-UI
tools.

####  3.1.3. Finding 3: UX designers use design patterns to understand the
design space and scaffold decision-making

An unexpected finding is the value of design patterns in the creation of UX
prototypes (P1, P3, P4, P5). The term “design pattern” is loosely-defined in
UX. Based on our interviews and analysis of previous work (Tankala and Joyce,
[n. d.]; Nguyen et al., 2018; Tidwell, 2010; Neil, 2014), we refer to design
pattern as “common ways” of designing static layout and dynamic interaction
across screens. For professional UX designers, patterns are exemplified
through individual UX examples. Design patterns help designers comprehensively
understand the potential design space. They scaffold the UX designers’
decision-making process by providing rationales for different design choices
and connecting design decisions to effects on user experiences. P3 mentioned
that “it’s really important to understand the rationale behind the design
scenarios and the design patterns to adopt others’ designs”.

We observed that UX designers’ search of user flow inspirations greatly
resembles the information foraging process (Pirolli and Card, 1999). Designers
look through the user flow prototypes of existing products, (find), implicitly
reason about the design scenario and the rationales behind selecting specific
design patterns (evaluate), and collect those that best fit the design problem
at hand (aggregate). During our interviews, all participants showed an
instinct to reason about the rationales behind the design decisions in example
user flow prototypes. Carrying these rationales forward, designers also use
them to justify their design decisions to collaborators and stakeholders.

A major challenge is that design pattern knowledge is acquired primarily
through experience and deliberate learning over time. For example, when
creating prototypes in tools like Figma, designers are often constrained in
their ability to explore a broad array of user flow examples. This makes it
difficult for them to comprehensively understand the design space and the
potential solutions offered by various design patterns. This constraint can
lead to locally optimal design decisions based on the narrow range of examples
they happen to choose, potentially resulting in design fixation . To account
for this, UX designers need to consult design pattern analysis articles on
websites like Pencil & Papers777https://pencilandpaper.io/articles/user-
experience/user-experience-pattern-analysis/ and Built for
Mars888https://builtformars.com/ to learn to design specific UX features (P1,
P4). P1 also mentioned that “as a grew more senior, I accumulate more examples
in their head, so I see alternative pattern whenever looking a specific
example”.

##  4\. Design Goals

Our formative study revealed a key insight about current UX design processes:
with the widespread use of pre-defined components in companies’ design systems
(Frost, 2016; Churchill, 2019), the bulk of UX design work has shifted to
adopting the most suitable design patterns to the design scenario at hand.

It became evident to us that designers can benefit more directly from
intelligent support in understanding the design space, rather than being
provided with AI-generated UI screens (Finding 3). Thus, we designed our
prototype, Flowy, with the following design goals:

DG1: Support scaffolding design decision making and accelerating information
foraging of UX inspirations. Instead of adopting an approach to generate user
flows and using a similar paradigm to prompt-to-UI generation, we designed
Flowy to explicitly support designers’ existing cognitive processes in
inspiration foraging. We observed that UX design processes and decisions are
often very complex and context-specific (Finding 2). It is almost impractical
for designers to prompt an AI model with sufficient design scenario context to
generate useful UX flows that matches their specific project context (Finding
1). Based on our formative study (Finding 3), we designed Flowy to support
designers’ current practices of learning design patterns through user flow
examples. Flowy aims to scaffold UX designers’ foraging process by helping
them identify design patterns they can potentially adopt and the design
considerations to make when deciding whether to adopt a pattern in their
specific scenario. This approach allows designers to make more informed design
decisions supported by evidence, while also accelerating the decision-making
process.

DG2: Prioritize design patterns regarding user flows and user interactions
over individual, static UI screens. Static UI screens alone provide limited
information about the overall user experience and the flow of interactions
within a product. In reality, designers work with dynamic, interconnected user
flows that define how users navigate through a product and complete tasks
(Finding 1). By providing information on design patterns for user interactions
and user flows, we can more closely align our support with the challenges
designers face in their everyday work. Providing guidance and inspiration at
the user flow level helps designers understand how different screens and
interactions fit together to create a cohesive user experience.

DG3: Provide high-quality design pattern insights connected to individual
examples. In designing AI-powered tools to support UX designers, providing
high-quality design pattern information and avoid hallucination is crucial.
Only by providing high-quality, professional-grade design pattern information,
can we ensure that designers receive reliable guidance and inspiration
(Finding 3). Furthermore, it is important to recognize that design patterns
and specific user flow examples serve different but complementary purposes.
Design patterns represent aggregated knowledge and best practices, offering a
high-level understanding of common solutions (Tankala and Joyce, [n. d.];
Nguyen et al., 2018; Tidwell, 2010; Neil, 2014). On the other hand, individual
user flow screenshots showcase specific design implementation details, such as
component usage, placement, and interaction details. We design Flowy to
provide both pattern-level information and concrete examples, so that
designers can gain a comprehensive understanding of the design space and make
informed decisions tailored to their specific context.

DG4: Provide easy-to-consume design pattern information to accelerate design
decision making. It is essential to provide UX designers with design pattern
information that is easy to understand and apply. Overwhelming designers with
complex or overly detailed information can hinder their ability to quickly
grasp key insights and make informed decisions (Koch et al., 2019; Kim and
Maher, 2023). This is particularly important in design ideation, where
designers often need to explore and evaluate a wide range of examples. By
presenting design patterns in a clear, concise, and easily digestible format,
we can help designers rapidly process and integrate this knowledge into their
work. This can also enable designers to more flexibly manage user flows and
adapt their design decisions based on the specific project context (Finding
2).

##  5\. Flowy Design & Implementation

In our Flowy prototype, we implemented a design pattern annotation pipeline
that utilizes the latest multimodal AI with the high-quality user flow
prototype dataset Mobbin. We take design pattern analysis for each user flow
prototype, then propagated the annotation results to our web app’s frontend.

Figure 2. Flowy’s backend design pattern annotation pipeline. The pipeline
utilizes Set-of-Mark prompting (1) and Retrival-Augmented-Generation (2). It
takes original screenshots of user flow examples (1), which are passed through
a Segment Anything Model (SAM) and Optical Character Recognition (OCR) to
identify relevant UI elements. Relevant design pattern analysis articles are
retrieved from our curated knowledge base, stored in a vector database (2).
The screenshots and retrieved articles are input to a multimodal AI model to
generate design pattern annotations, including the pattern name, purpose,
design considerations, etc. (3). Prompts are simplified to improve readability
yet still highlight the core utilities.

The figure depicts an annotation pipeline for generating design pattern
information from app screenshots. The pipeline consists of three main stages:
(1) Inputting original screenshots from user flow examples, which are
processed to identify relevant UI elements, (2) Retrieving relevant design
guideline articles from a curated knowledge database, and (3) Inputting the
screenshots and articles into a language model to generate detailed design
pattern annotations. The generated annotations include key information such as
pattern name, purpose, advantages and disadvantages, and design
considerations.

###  5.1. Design Pattern Annotation Pipeline

Our annotation pipeline aims to provide high-quality, professional-grade
design pattern information that is easy to consume (DG3, DG4). Thus, we
designed a Set-of-Mark (SoM) prompting module to anchor annotations onto
specific visual UI elements on screens, together with a Retrieval-Augmented
Generation (RAG) module to ground our annotation in existing high-quality UX
design pattern analysis articles. Generally, we used state-of-the-art large
multimodal models to annotate design pattern information on user flow
prototypes.

####  5.1.1. Visually-Anchored Annotation Through Set-of-Mark Prompting

We employed Set-of-Mark (SoM) prompting, a method for improving the visual
grounding abilities of large multimodal models (Yang et al., 2023), to
associate design pattern annotations with visual UI elements on the
screenshots (Section 1 in Figure 2). SoM is typically done by marking images
with main object region masks from an object detection model and associating
each mask with a numeric “tip”. The original and marked images are then fed
into a multimodal AI model. In our project, using SoM enables us to relate
design pattern annotations with associated UI elements in user flow examples,
improving the ease of consumption for designers (DG3, DG4).

####  5.1.2. High-Quality Design Pattern Information With Retrieval Augmented
Generation

To ensure the quality of our annotation results, the research team manually
curated 54 high-quality articles on UX design pattern analysis through
collective discussion and suggestions from professional UX designers, as our
design pattern knowledge base (DG2, DG3). We used Retrieval-Augmented
Generation (RAG) (Lewis et al., 2020), a common technique in knowledge-
intensive tasks to supplement large multimodal models with domain-specific
information. We employed RAG to retrieve relevant articles from our curated
knowledge base based on the similarity between the article content and the
input user flow example (Section 2 in Figure 2). The retrieved articles were
then used as additional context in our prompt to guide the model in generating
accurate and informative design pattern annotations.

####  5.1.3. Specific Features for Efficient UX Inspiration Foraging

We took inspiration from past research on inspiration foraging supporting
tools (Vattam and Goel, 2011; Goel et al., 2012) and specifically designed our
annotation task around the within-patch foraging and between-patch foraging
state of designers (DG1). In within-patch foraging, the forager gains net
information by consuming the content of the information patch. To support
designers’ net information gain of design patterns in user flow examples
(DG4), on top of design pattern definitions and descriptions, we also defined
key aspects of rationale behind the adoption of design patterns based on
previous work on design patterns (Tidwell, 2010; Neil, 2014), namely purposes,
advantages, disadvantages, and design considerations and generated such
information based on the RAG-retrieved design guideline articles. Such
information adds values to designers by providing them more granular details
in the context, trade-offs, and considerations behind the use of specific
design patterns. During between-patch foraging, the forager tries to find the
next best information patches to consume. Thus, we provide screenshots of
related designs adopting similar design patterns to help designers identify
the other patches of interest (see visualized illustration in Section 5.2).

####  5.1.4. Prompt Structure

The general prompt structure of our design pattern annotation pipeline
consists of three main prompts (Figure 2). In the first prompt, we ask the
large multimodal model to visually identify and explain UI elements that are
relevant to the design pattern of interest. This prompt was designed to ground
the subsequent annotations in accurate understandings of each UI element’s
function and reduce the model’s hallucination. Then, the explanation of
relevant UI elements were combined with retrieved professional UX pattern
analysis articles (Section 5.1.2), to prompt the model to annotate the
purposes, considerations, and advantages/disadvantages of a design pattern
example. In the last prompt, we asked the model to reflectively select the
most relevant sections of UX pattern analysis articles for each annotation, to
provide users with accurate source information as references.

####  5.1.5. Annotation Pipeline Implementation Details

To implement SoM prompting, we first performed UI element detection using the
Segment Anything Model (SAM) (Kirillov et al., 2023), which generates fine-
grained segmentation masks over UI elements on screenshots. To mitigate over-
segmentation on text content, where texts on screenshots are divided into
overly small pieces, we used Optical Character Recognition (OCR) to identify
text regions and filtered out SAM segmentations where the Intersection over
Union (IoU) with any text region exceeded 0.55, a threshold empirically set
through experimentation. Additionally, tiny segmentations constituting less
than 0.5% of the screen area were filtered out. The resulting segmentation
masks, their numerical identifiers, and the original screenshots were then
input into the large multimodal model (Figure 2), following the approach
described in the original SoM paper.

To implement our RAG component, we used cosine similarity between the article
content embeddings and the current product’s feature name to retrieve design
pattern analysis for the specific feature999In our curated knowledge base,
design pattern analysis articles are mostly written around specific product
features.. Specifically, we used LangChain and ChromaDB to implement this
component. Then, we added the retrieved article’s original text, together with
the original and marked screenshots from our SoM prompting module, into the
multimodal AI model to annotate for the design pattern information on UI
screenshots.

We selected GPT-4V as the multimodal AI model for Prompt 1 and Prompt 2
(Figure 2). Prompt 2 and 3 were initially a same prompt. However, in our
experimentation, we empirically observed that GPT-4V can seldom correctly
retrieve the relevant paragraphs from source pattern analysis documents, yet
Claude-3-Opus was able to reliably perform this task. As a result, we
separated Prompt 3 from 2 and specifically used Claude-3-Opus for Prompt 3.
Our prompt structure adopts the Chain-of-Thought (Wei et al., 2022) and prompt
chaining (Wu et al., 2022) techniques to enhance the generation pipeline’s
performance. We annotate all user flow prototypes in advance and statistically
store them in our web app, to ensure information consistency and avoid long
inference time.

Figure 3. Flowy’s frontend design offers a multi-layered, flexible way to
explore design examples and associated design patterns at different
granularity. The homepage (A) showcases user flows from existing products.
Selecting a flow opens an interactive annotated view (B), with key elements
highlighted and briefly explained on hover. Clicking an annotation reveals a
dedicated page (C1) with detailed pattern analysis and related design
screenshot examples for side-by-side comparison (C2).

The image shows 4 screenshots demonstrating the user interface and flow of a
tool called Flowy. Image A shows the homepage with examples of user flows from
different apps. Image B shows a user flow page with design pattern
annotations. Image C1 shows a detailed design pattern page, and image C2 shows
that same page scrolled down to reveal additional information and example
screenshots related to the selected design pattern. The overall figure
illustrates how Flowy enables exploring UI/UX design examples at varying
levels of detail, from high-level flows down to specific design patterns.

###  5.2. Frontend Web Design & Implementation

To support the efficient consumption of design pattern information through
specific design examples (DG4), Flowy’s frontend adopts the Visual
Information-Seeking Mantra: overview first, zoom and filter, then details-on-
demand (North and Shneiderman, 2000). As illustrated in Figure 3, after a user
selects a user flow example on Flowy’s homepage (Screen A), they enter the
page displaying annotations for this user flow (Screen B). The icon tags (1)
on each UI screenshot, indicating user pattern annotations and their
associated UI elements, together with the list of design pattern annotations
(2) on the right, provide an overview of the available annotations. The user
can hover over annotations they are interested in and see a popup window with
concise, bite-sized summaries regarding this design pattern in a bullet list
(zoom and filter). If the user is interested in learning further, they can
click on the annotation popup to go to the annotation detail page (Screen C1,
C2), where related designs (3) and more details regarding this design pattern
(4) are provided (details-on-demand). This multi-level, interactive design
allows users to quickly grasp the overall design patterns present in a user
flow example while enabling them to selectively dive deeper into patterns of
interest, facilitating efficient design inspiration foraging.

As discussed in Section 5.1.3, we support UX designers’ inspiration foraging
process by specifically catering to the within-patch foraging and between-
patch foraging stages. The detailed information (4) regarding a design pattern
increases a designer’s net information gain during within-patch foraging,
while the “Related Design” section (3) helps a designers to identify potential
patches to look through next.

The frontend of Flowy was implemented using Next.js and Tailwind CSS. As a
prototype, in Flowy, we included 31 unique, high-quality user flow examples
for three product features, namely Filter & Sort, Search & Find, and
Subscribing. On Screen C1, the “Related Design” section (3) was implemented by
ranking the cosine similarities between pattern definitions in user flows for
the same product feature.

##  6\. User Study

We conducted a user study with eight UX designers of varying experience to
evaluate the Flowy prototype101010The study protocol was reviewed and approved
by the Institutional Review Board (IRB) at our institution.. Among the eight
participants, one participant is a student in UX while the rest are all
professional UX designers. A table describing the participant demographics is
shown in Appendix 2. Each study session lasted one hour, and participants
received $15 USD as compensation for their time. Our study aimed to answer the
following research questions:

  * •

RQ3111111We continued the RQ numbering to differentiate from research
questions in our formative study.: How might Flowy support UX designers’
ideation process in creating UX prototypes?

  * •

RQ4: How might Flowy fit into existing UX design workflows and provide
support?

  * •

RQ5: What challenges do UX designers encounter when using Flowy for UX
ideation support?

###  6.1. Study Design

Each study session was conducted online and lasted approximately one hour. The
sessions consisted of two main activities: system feedback (20-25 minutes) and
a mock design task (20-25 minutes). First, after obtaining informed consent
from the participants, the researchers introduced them to Flowy and provided a
high-level explanation of its key features. Then, each participant
independently navigated through the interfaces of Flowy as a potential user of
the system. The participants were asked to provide feedback regarding the
design pattern annotations, focusing on how the annotation information might
help or hinder their everyday design tasks. Collecting feedback on the content
of design pattern annotations was crucial to ensure that the information
provided by Flowy was relevant, accurate, and helpful for UX designers in
their real-world projects.

Afterwards, each participant was tasked with designing a filter feature inside
an existing Figma prototype for a mobile electronics shopping app. This mock
design activity aimed to simulate a realistic UX design scenario and assess
the effectiveness of Flowy in supporting practical design challenges within
participants’ actual workflows. The researchers encouraged the participants to
refer to Flowy during the design task while also inviting them to use other
tools they would typically employ in their natural workflows. During the
design task, participants were instructed to think aloud, enabling researchers
to gain a deeper understanding of their thought processes (McDonald et al.,
2012). Researchers asked follow-up questions throughout the study for
clarification. At the end of the session, participants completed a post-study
questionnaire, which gathered feedback regarding the usability and usefulness
of the Flowy prototype, as well as its potential integration into their
existing UX design workflows.

Figure 4. Questionnaire results from our user study.

Figure 4 shows questionnaire results from the user study, with various
statements about the Flowy system on the y-axis and the number of participants
who strongly disagree, disagree, are neutral, agree, or strongly agree with
each statement represented by colored bars. The results indicate that most
participants would like to use a system like Flowy frequently, found it easy
to use, and thought the design pattern information provided was easy to
comprehend and helped them understand design options and the design space. The
majority also felt the system functions were well integrated, they learned to
use it very quickly, and agreed it integrates into their design workflows
naturally. Overall, the questionnaire responses suggest participants had a
positive experience with Flowy and found it helpful for understanding relevant
design patterns and options to support their UX design work.

###  6.2. Key Insights

Four of the researchers collaboratively analyzed the notes take during the
user studies as a group qualitatively using affinity diagramming (Pernice,
2018). The goal is to identify common themes in the responses of the user
study participants. The questionnaire responses collected at the end of user
study sessions are also visualized in Figure 4. In this section, we present
the key insights from our annotations with reference to the quantitative
measures in the questionnaires.

####  6.2.1. Insight 1: Flowy can accelerate inspiration foraging through more
informed, evidence-based UX design decisions.

Our participants generally found value in Flowy’s features for scaffolding
their considerations in UX design decision making and accelerating their
inspiration foraging in the mock design activity. This is reflected in
questionnaire responses (Figure 4) regarding Flowy’s high value on
understanding the design space and good potential to naturally integrate into
designers’ workflows. Confirming our design goal (DG2), our participants
highlighted that design decisions at the user flow level as a critical yet
often overlooked aspect of UX design (P1, P4, P7). P1 highly appreciated the
“Related Design” section’s examples (Figure 3, element 3), which helped her to
directly compare similar design patterns and identify more promising
inspirational examples. P7 specifically said that for the design pattern
analysis, “if you do not do it for me, I will have to do it myself”.

Participants found Flowy’s annotations on the flow example page (Figure 3,
element 1) helpful in guiding their attention onto important UI elements. P4
and P5 noted that Flowy’s annotations help prevent overlooking relevant UI
elements in user flow examples, a common issue when looking through many user
flow examples. In addition, P2, P4, P5, and P7 mentioned that Flowy’s design
pattern annotations can serve as positive examples of design rationales, which
can greatly help UX designers in justifying their design decisions in intra-
and inter-team communications. This echos findings in previous empirical
research of UX practices (Lu et al., 2022). P8 said that “(Flowy) can be
particularly helpful for UX beginners when they work on new features they are
not familiar with”, a sentiment also shared by P1 and P5. In general, our
participants would want to use Flowy’s features in their everyday work and
mostly think it will easily integrate into their natural workflows, as
reflected in the post-study questionnaire responses (Figure 4).

####  6.2.2. Insight 2: Designers desire design pattern annotations beyond
user flows, including product, business, and development considerations.

Our UX design pattern annotations primarily focused on user flow and
interaction patterns as a result of our knowledge base articles’ focus.
However, our participants also highlighted that in practice, their design
decisions often also involve considerations for their products’ goals,
business objectives, and technical development limitations. These further
considerations often cannot be directly inferred from the visible UI screens
and UX prototypes. However, they can significantly help designers in
evaluating design patterns’ adaptability to their own design scenarios. P4
analyzed that “intuitively I have a gut instinct of why the designer did this,
and it’s not really mind-blowing to read the advantages and disadvantages (of
a design pattern)… what would be really interesting that I am curious on, is
more product-specific context, … why this design decision relates back to the
app store design specifically”. Similar opinions are also shared by P2 and P6
during the sessions.

This uncovered the fact that UX design decisions often are numerous determined
by factors beyond user flows and experiences. We summarized the factors
mentioned by designers into three main categories: target user groups, (P1,
P2, P5, P7), product decisions (P2, P5, P7), and development constraints (P2,
P5, P6). UX designers often have to base their design decisions of user flows
and multi-screen component flows on careful trade-offs between these main
factors. As a result, the UX ideation process can be very cognitively
demanding. We observed that during our study, P2, P4, P5, and P7 all
implicitly quickly ignored overly complex user flow examples and information
that is not straightforward on screen to conserve cognitive resources. This
inspired us to create more concise, straightforward, and specific design
pattern annotations in future iterations of Flowy.

####  6.2.3. Insight 3: Designers prioritize visual comparative learning of
design patterns.

Interestingly, during the study sessions, the majority of participants spent
the most time trying to understand design patterns by using the “Related
Design” section in Flowy, where design examples adopting similar design
patterns are displayed side-by-side (P1-4, P6, P8). Six out of eight
participants (P2-4, P6-8) explicitly mentioned that they preferred comparing
different user flow screenshots to reading textual design pattern analysis,
although they acknowledge the latter can provide design rationale references.
More specifically, P4 mentioned that they “do not want to read the text”,
despite our efforts to make textual information short and concise. We assume
that this, combined with the fact that Flowy’s pattern analysis is mostly
represented as text (Figure 3, element 4), is the main reason behind some
participants perceiving Flowy as not easy to use in the questionnaire
response. Yet, when P4 was asked what information regarding the design
patterns they would find helpful, they describe information with the content
and representation very similar to what Flowy has. P4 articulated: “I don’t
know, I feel like something just makes me not want to read this… but to be
honest, that’s also where I am confused in myself that, I understand you guys
have these (information).”

We believe that this prioritization reflects the Structural Alignment Theory,
which proposes that when comparing two items, people align their common
relational structures to identify similarities and differences (Gentner and
Markman, 1997). This process has been applied to the abstraction of common
principles and design ideation (Dahl and Moreau, 2002; Chan et al., 2018;
Gassmann and Zeschky, 2008; Ozkan and Dogan, 2013). For our participants,
learning design patterns through visual comparisons of similar design examples
was more effective than reading textual explanations, even though the latter
were also intentionally designed to be concise and easy-to-consume. P6
explained that they only used textual information as a fallback when they
cannot intuitively understand visceral information. This key insight provides
us with a future design implication to prioritize visual comparative
annotation of design patterns, over textual explanations.

###  6.3. Design Implications

We generalized our study findings to derive key design implications for AI-
powered UX ideation tools like Flowy. These implications highlight the needs
and preferences of UX designers when using such tools in their design
workflows.

  1. (1)

Provide concise, straightforward, and precise design pattern annotations to
reduce designers’ cognitive load;

  2. (2)

Expand design pattern analysis beyond user flows and interactions, to include
relevant product goals, business objectives, and technical development
considerations.

  3. (3)

Prioritize visual, comparative learning of design patterns between screenshot
examples over providing textual explanations.

Based on these design implications, we plan to iterate on the Flowy prototype
to better align with the needs and expectations of UX designers. We believe
that these implications can also serve as valuable guidelines for future
research in creativity support for UX design.

##  7\. Discussion

###  7.1. Designerly Understanding of UX Protoypes with Multimodal AI

While some previous research in AI has tried to integrate professional UX
designers’ considerations into computational deep learning models (Schoop et
al., 2022; Li et al., 2021; Huang et al., 2019), but most existing work in AI
takes end users as the target group for UI/UX related tasks (Wang et al.,
2023, 2021; Yan et al., 2023). Much of supposedly designer-centric AI research
seems to be disconnected with designers’ latest practices, focusing on
concepts such as aesthetics that are not grounded in design practitioners’
main considerations. During our design of Flowy, we empirically observed that
current large multimodal AI models can only recognize basic UI elements and
their purposes (end-user-centric), but are limited in understanding the design
rationale and considerations behind design decisions (designer-centric).

We assume that these phenomena reveal the current gap and many potential
directions in AI support for UX design. We call for the empirical
understanding of two important aspects of UX design for future AI research: UX
is much more than UI, and UI screens often embed more design considerations
than a good composition of visual UI components (Lu et al., 2024). Coming up
with good composition of UI layouts is considered as UI design and only part
of the goals for UX designers (Norman and Nielsen, 1998). The more important
end goal is to construct user flows and interactions based on established best
practices and design patterns, fitting the UI layouts, interactions, and flows
to users’ goals, business values, and development resources.

###  7.2. Transforming UX Design Practices: Supporting Evidence-Based
Decisions, Automating Implementations

In our formative studies, we found that the widespread use of design systems
and pre-defined design component libraries shifted the essence of UX design
tasks to adopting the most suitable design patterns to the design scenario at
hand. Many existing generative AI tools for UX aim to automate the creation of
UI screens through a prompt-to-UI paradigm. Our effort to prototyping Flowy
represents a complementary approach to support UX designers in making
evidence-based design decisions through design pattern annotations on high-
quality user flow prototype examples. We argue that by combining these two
approaches and carefully design the augmentation and automation roles of AI in
different stages of UI/UX design, great future directions can be pursued to
transform the current UX design practices. Additionally, by considering latest
advancements in streamlining the designer-developer hand-off process (Feng et
al., 2021; Feng and Zhang, 2022; Feng et al., 2023a), further potential lies
in the general pipeline and paradigm of software design and implementation.

###  7.3. Understanding Task Complexity to Design AI Support for Professional
Users

In our formative study, we confirmed our assumption of UX designers’ heavy
focus on user flows across multiple screens than individual UI screens. We
also uncovered the key insight that current UX design processes is more about
adopting the most suitable design patterns to the current design scenario,
than coming up with visually appealing UI screens. From our user study of
Flowy, we discovered that UX designers also work with high complexity in
target user groups, business goals, product decisions, and development
resources. These findings revealed that UX design processes are much more
complex than visible UI screens.

While grounded in the unique characteristics of UX, the principles underlying
Flowy’s design are generalizable to other creative tasks. By choosing
representations with appropriate levels of abstraction (e.g. user flow
examples, design pattern knowledge), Flowy allows users to make sense of the
solution space at various conceptual levels. Additionally, by helping users
navigate through connections and affinities during the foraging process, Flowy
facilitates the exploration and sensemaking of the solution space. These
design considerations are particularly relevant to many open-ended tasks
characterized by the absence of a single optimal solution, where decision-
making requires careful reasoning over a comprehensive understanding of the
solution space. Flowy’s approach embodies a human-centered, intelligence
augmentation approach (Hassani et al., 2020) of human-AI collaboration, with
the end goal of supporting target users’ cognitive processes to improve task
efficiency and effectiveness.

Human-centered adoption of AI requires deep knowledge and understanding of the
complexity of tasks of professionals and goes beyond automating the creation
of visible artifacts (Shneiderman, 2022). Our design considerations for Flowy
suit many tasks with no single optimal solution, where decision making
requires careful reasoning over the solution space. More strategies should be
devised for future human-centered AI research as design principles, such as
choosing mediums with appropriate levels of abstraction and assisting users in
discovering connections and affinities in the foraging process. By focusing on
similar key design principles, future AI systems can effectively support
professionals in navigating the complex decision-making processes involved in
their work, while ensuring that users retain control and maintain a sense of
ownership over the creative process.

##  8\. Limitaiton

Despite our best efforts to mitigate large-scale multimodal AI’s hallucination
using Retrieval-Augmented Generation and Chain-of-Thought prompting, we cannot
fully avoid design patterns that deviate from the user flow examples. However,
we expect this limitation to gradually improve as future multimodal AI model
capabilities advance. It is also important to acknowledge Flowy’s current
limitations in scope and the product features it supports as a prototype. Our
evaluation study also has several threats to validity: the representativeness
of our participants and tasks may not fully capture the diversity of real-
world UX design scenarios. Additionally, the novelty bias associated with
using a new tool like Flowy could have influenced the participants’ behaviors
and attitudes during the study. To address these limitations and gain deeper
insights into Flowy’s long-term impact on UX design practices, we plan to
conduct longitudinal deployments in real-world design settings for future
research.

##  9\. Conclusion

In this work, we presented Flowy, an AI-powered tool that supports UX
designers’ inspiration foraging process by supplementing user flow examples
with distilled, high-quality design pattern knowledge. Our user study
demonstrated Flowy’s effectiveness in supporting realistic UX design tasks,
with participants finding the visually anchored design pattern annotations
helpful for efficiently exploring and understanding relevant design patterns.
The design considerations in Flowy, such as multi-level representations and
interactive exploration, reflect a human-centered, intelligence augmentation
approach to leveraging AI in creative workflows. Our design implications
summarized from our user studies point out meaningful considerations for
future UX design support tools.

## References

  * (1)
  * Brade et al. (2023) Stephen Brade, Bryan Wang, Mauricio Sousa, Sageev Oore, and Tovi Grossman. 2023.  Promptify: Text-to-image generation through interactive prompt exploration with large language models. In _Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology_. 1–14.
  * Chan et al. (2018) Joel Chan, Steven P Dow, and Christian D Schunn. 2018.  Do the best design ideas (really) come from conceptually distant sources of inspiration?  _Engineering a Better Future: Interplay between Engineering, Social Sciences, and Innovation_ (2018), 111–139.
  * Churchill (2019) Elizabeth F Churchill. 2019.  Scaling UX with design systems.  _Interactions_ 26, 5 (2019), 22–23.
  * Dahl and Moreau (2002) Darren W Dahl and Page Moreau. 2002.  The influence and value of analogical thinking during new product ideation.  _Journal of marketing research_ 39, 1 (2002), 47–60.
  * Deaton (2003) Mary Deaton. 2003.  The elements of user experience: user-centered design for the Web.  _interactions_ 10, 5 (2003), 49–51.
  * Deka et al. (2017) Biplab Deka, Zifeng Huang, Chad Franzen, Joshua Hibschman, Daniel Afergan, Yang Li, Jeffrey Nichols, and Ranjitha Kumar. 2017.  Rico: A Mobile App Dataset for Building Data-Driven Design Applications. In _Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology_. ACM, Québec City QC Canada, 845–854.  https://doi.org/10.1145/3126594.3126651
  * Deka et al. (2016) Biplab Deka, Zifeng Huang, and Ranjitha Kumar. 2016.  ERICA: Interaction Mining Mobile Apps. In _Proceedings of the 29th Annual Symposium on User Interface Software and Technology_ _(UIST ’16)_. Association for Computing Machinery, New York, NY, USA, 767–776.  https://doi.org/10.1145/2984511.2984581
  * Dhariwal and Nichol (2021) Prafulla Dhariwal and Alexander Nichol. 2021.  Diffusion models beat gans on image synthesis.  _Advances in neural information processing systems_ 34 (2021), 8780–8794.
  * Di Fede et al. (2022) Giulia Di Fede, Davide Rocchesso, Steven P Dow, and Salvatore Andolina. 2022.  The idea machine: LLM-based expansion, rewriting, combination, and suggestion of ideas. In _Proceedings of the 14th Conference on Creativity and Cognition_. 623–627.
  * Dwairy et al. (2011) Mai Dwairy, Anthony C Dowell, and Jean-Claude Stahl. 2011.  The application of foraging theory to the information searching behaviour of general practitioners.  _BMC family practice_ 12 (2011), 1–8.
  * Feng et al. (2023a) K. J. Kevin Feng, Tony W Li, and Amy X. Zhang. 2023a.  Understanding Collaborative Practices and Tools of Professional UX Practitioners in Software Organizations. In _Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems_ _(CHI ’23)_. Association for Computing Machinery, New York, NY, USA, 1–20.  https://doi.org/10.1145/3544548.3581273
  * Feng and Zhang (2022) K J Kevin Feng and Amy X Zhang. 2022.  From Handoffs to Co-Creation: Deepening Collaboration between Designers, Developers, and Data Science Workers in UX Design.  (2022).
  * Feng et al. (2023b) Weixi Feng, Wanrong Zhu, Tsu-jui Fu, Varun Jampani, Arjun Akula, Xuehai He, Sugato Basu, Xin Eric Wang, and William Yang Wang. 2023b.  LayoutGPT: Compositional Visual Planning and Generation with Large Language Models.  https://doi.org/10.48550/arXiv.2305.15393 arXiv:2305.15393 [cs].
  * Feng et al. (2021) Zhen Feng, Jiaqi Fang, Bo Cai, and Yingtao Zhang. 2021.  GUIS2Code: A Computer Vision Tool to Generate Code Automatically from Graphical User Interface Sketches. In _Artificial Neural Networks and Machine Learning – ICANN 2021_ _(Lecture Notes in Computer Science)_ , Igor Farkaš, Paolo Masulli, Sebastian Otte, and Stefan Wermter (Eds.). Springer International Publishing, Cham, 53–65.  https://doi.org/10.1007/978-3-030-86365-4_5
  * Foundation ([n. d.]) Interaction Design Foundation. [n. d.].  What Are User Flows?  https://www.interaction-design.org/literature/topics/user-flows
  * Frost (2016) Brad Frost. 2016.  _Atomic Design_.  http://atomicdesign.bradfrost.com/
  * Gassmann and Zeschky (2008) Oliver Gassmann and Marco Zeschky. 2008.  Opening up the solution space: The role of analogical thinking for breakthrough product innovation.  _Creativity and Innovation Management_ 17, 2 (2008), 97–106.
  * Gentner and Markman (1997) Dedre Gentner and Arthur B Markman. 1997.  Structure mapping in analogy and similarity.  _American psychologist_ 52, 1 (1997), 45.
  * Girotra et al. (2023) Karan Girotra, Lennart Meincke, Christian Terwiesch, and Karl T Ulrich. 2023.  Ideas are dimes a dozen: Large language models for idea generation in innovation.  _Available at SSRN 4526071_ (2023).
  * Goel et al. (2012) Ashok K Goel, Swaroop Vattam, Bryan Wiltgen, and Michael Helms. 2012.  Cognitive, collaborative, conceptual and creative—four characteristics of the next generation of knowledge-based CAD systems: a study in biologically inspired design.  _Computer-Aided Design_ 44, 10 (2012), 879–900.
  * Gonzalez et al. (2024) Gabriel Enrique Gonzalez, Dario Andres Silva Moran, Stephanie Houde, Jessica He, Steven Ross, Michael Muller, Siya Kunde, and Justin Weisz. 2024.  Collaborative Canvas: A Tool for Exploring LLM Use in Group Ideation Tasks. In _ACM International Conference on Intelligent User Interfaces_.
  * Gray (2016) Colin M. Gray. 2016.  ”It’s More of a Mindset Than a Method”: UX Practitioners’ Conception of Design Methods. In _Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems_ _(CHI ’16)_. Association for Computing Machinery, New York, NY, USA, 4044–4055.  https://doi.org/10.1145/2858036.2858410
  * Hassani et al. (2020) Hossein Hassani, Emmanuel Sirimal Silva, Stephane Unger, Maedeh TajMazinani, and Stephen Mac Feely. 2020.  Artificial intelligence (AI) or intelligence augmentation (IA): what is the future?  _Ai_ 1, 2 (2020), 8.
  * Ho et al. (2020) Jonathan Ho, Ajay Jain, and Pieter Abbeel. 2020.  Denoising diffusion probabilistic models.  _Advances in neural information processing systems_ 33 (2020), 6840–6851.
  * Huang et al. (2019) Forrest Huang, John F. Canny, and Jeffrey Nichols. 2019.  Swire: Sketch-based User Interface Retrieval. In _Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems_ _(CHI ’19)_. Association for Computing Machinery, New York, NY, USA, 1–10.  https://doi.org/10.1145/3290605.3300334
  * Huang et al. (2021) Forrest Huang, Gang Li, Xin Zhou, John F. Canny, and Yang Li. 2021.  Creating User Interface Mock-ups from High-Level Text Descriptions with Deep-Learning Models.  https://doi.org/10.48550/arXiv.2110.07775 arXiv:2110.07775 [cs].
  * Kaplan (2023) Kate Kaplan. 2023.  User Journeys vs. User Flows.  https://www.nngroup.com/articles/user-journeys-vs-user-flows/
  * Kim and Maher (2023) Jingoog Kim and Mary Lou Maher. 2023.  The effect of AI-based inspiration on human design ideation.  _International Journal of Design Creativity and Innovation_ 11, 2 (April 2023), 81–98.  https://doi.org/10.1080/21650349.2023.2167124 Publisher: Taylor & Francis _eprint: https://doi.org/10.1080/21650349.2023.2167124.
  * Kim et al. (2023) Tae Soo Kim, Yoonjoo Lee, Minsuk Chang, and Juho Kim. 2023.  Cells, generators, and lenses: Design framework for object-oriented interaction with large language models. In _Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology_. 1–18.
  * Kirillov et al. (2023) Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. 2023\.  Segment anything. In _Proceedings of the IEEE/CVF International Conference on Computer Vision_. 4015–4026.
  * Ko et al. (2023) Hyung-Kwon Ko, Gwanmo Park, Hyeon Jeon, Jaemin Jo, Juho Kim, and Jinwook Seo. 2023.  Large-scale text-to-image generation models for visual artists’ creative works. In _Proceedings of the 28th international conference on intelligent user interfaces_. 919–933.
  * Koch et al. (2019) Janin Koch, Andrés Lucero, Lena Hegemann, and Antti Oulasvirta. 2019.  May AI? Design Ideation with Cooperative Contextual Bandits. In _Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems_ _(CHI ’19)_. Association for Computing Machinery, New York, NY, USA, 1–12.  https://doi.org/10.1145/3290605.3300863
  * Lamac (2023) Radovan Lamac. 2023.  Text-to-image AI as a tool for the designer’s ideation process.  (2023).
  * Lawrance et al. (2010) Joseph Lawrance, Christopher Bogart, Margaret Burnett, Rachel Bellamy, Kyle Rector, and Scott D Fleming. 2010.  How programmers debug, revisited: An information foraging theory perspective.  _IEEE Transactions on Software Engineering_ 39, 2 (2010), 197–215.
  * Lewis et al. (2020) Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. 2020\.  Retrieval-augmented generation for knowledge-intensive nlp tasks.  _Advances in Neural Information Processing Systems_ 33 (2020), 9459–9474.
  * Li and Li (2023) Gang Li and Yang Li. 2023.  Spotlight: Mobile UI Understanding using Vision-Language Models with a Focus.  https://doi.org/10.48550/arXiv.2209.14927 arXiv:2209.14927 [cs].
  * Li et al. (2021) Toby Jia-Jun Li, Lindsay Popowski, Tom Mitchell, and Brad A Myers. 2021.  Screen2Vec: Semantic Embedding of GUI Screens and GUI Components. In _Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems_. ACM, Yokohama Japan, 1–15.  https://doi.org/10.1145/3411764.3445049
  * Li et al. (2020) Yang Li, Gang Li, Luheng He, Jingjie Zheng, Hong Li, and Zhiwei Guan. 2020.  Widget Captioning: Generating Natural Language Description for Mobile User Interface Elements.  http://arxiv.org/abs/2010.04295 arXiv:2010.04295 [cs].
  * Lu et al. (2023) Yuwen Lu, Ziang Tong, Qinyi Zhao, Chengzhi Zhang, and Toby Jia-Jun Li. 2023.  UI Layout Generation with LLMs Guided by UI Grammar.  _arXiv preprint arXiv:2310.15455_ (2023).
  * Lu et al. (2024) Yuwen Lu, Yuewen Yang, Qinyi Zhao, Chengzhi Zhang, and Toby Jia-Jun Li. 2024.  AI Assistance for UX: A Literature Review Through Human-Centered AI.  _arXiv preprint arXiv:2402.06089_ (2024).
  * Lu et al. (2022) Yuwen Lu, Chengzhi Zhang, Iris Zhang, and Toby Jia-Jun Li. 2022.  Bridging the Gap Between UX Practitioners’ Work Practices and AI-Enabled Design Support Tools. In _Extended Abstracts of the 2022 CHI Conference on Human Factors in Computing Systems_ _(CHI EA ’22)_. Association for Computing Machinery, New York, NY, USA, 1–7.  https://doi.org/10.1145/3491101.3519809
  * McDonald et al. (2012) Sharon McDonald, Helen M Edwards, and Tingting Zhao. 2012.  Exploring think-alouds in usability testing: An international survey.  _IEEE Transactions on Professional Communication_ 55, 1 (2012), 2–19.
  * Mozaffari et al. (2022) Mohammad Amin Mozaffari, Xinyuan Zhang, Jinghui Cheng, and Jin L. C. Guo. 2022.  GANSpiration: Balancing Targeted and Serendipitous Inspiration in User Interface Design with Style-Based Generative Adversarial Network. In _CHI Conference on Human Factors in Computing Systems_. 1–15.  https://doi.org/10.1145/3491102.3517511 arXiv:2203.03827 [cs].
  * Neil (2014) Theresa Neil. 2014.  _Mobile design pattern gallery: UI patterns for smartphone apps_.  ” O’Reilly Media, Inc.”.
  * Nguyen et al. (2018) Tam The Nguyen, Phong Minh Vu, Hung Viet Pham, and Tung Thanh Nguyen. 2018.  Deep learning UI design patterns of mobile apps. In _Proceedings of the 40th International Conference on Software Engineering: New Ideas and Emerging Results_. 65–68.
  * Norman (2013) Don Norman. 2013.  _The design of everyday things: Revised and expanded edition_.  Basic books.
  * Norman and Nielsen (1998) Don Norman and Jakob Nielsen. 1998.  The Definition of User Experience.  _Nielsen Norman Group_ (Aug 1998).  https://www.nngroup.com/articles/definition-user-experience/
  * North and Shneiderman (2000) Chris North and Ben Shneiderman. 2000.  Snap-together visualization: a user interface for coordinating visualizations via relational schemata. In _Proceedings of the working conference on Advanced visual interfaces_. 128–135.
  * Oppenlaender (2022) Jonas Oppenlaender. 2022.  The creativity of text-to-image generation. In _Proceedings of the 25th International Academic Mindtrek Conference_. 192–202.
  * Ozkan and Dogan (2013) Ozgu Ozkan and Fehmi Dogan. 2013.  Cognitive strategies of analogical reasoning in design: Differences between expert and novice designers.  _Design Studies_ 34, 2 (2013), 161–192.
  * Paananen et al. (2023) Ville Paananen, Jonas Oppenlaender, and Aku Visuri. 2023.  Using text-to-image generation for architectural design ideation.  _International Journal of Architectural Computing_ (2023), 14780771231222783.
  * Pernice (2018) Kara Pernice. 2018.  Affinity Diagramming: Collaboratively Sort UX Findings & Design Ideas.  https://www.nngroup.com/articles/affinity-diagram/
  * Pirolli and Card (1999) Peter Pirolli and Stuart Card. 1999.  Information foraging.  _Psychological review_ 106, 4 (1999), 643.
  * Qu et al. (2023) Zhiyu Qu, Tao Xiang, and Yi-Zhe Song. 2023.  SketchDreamer: Interactive Text-Augmented Creative Sketch Ideation.  _arXiv preprint arXiv:2308.14191_ (2023).
  * Russell and Chi (2014) Daniel M Russell and Ed H Chi. 2014.  Looking back: Retrospective study methods for HCI.  In _Ways of Knowing in HCI_. Springer, 373–393.
  * Sandstrom (1994) Pamela Effrein Sandstrom. 1994.  An optimal foraging approach to information seeking and use.  _The library quarterly_ 64, 4 (1994), 414–449.
  * Schoop et al. (2022) Eldon Schoop, Xin Zhou, Gang Li, Zhourong Chen, Bjoern Hartmann, and Yang Li. 2022.  Predicting and Explaining Mobile UI Tappability with Vision Modeling and Saliency Analysis. In _CHI Conference on Human Factors in Computing Systems_. ACM, New Orleans LA USA, 1–21.  https://doi.org/10.1145/3491102.3517497
  * Shaer et al. (2024) Orit Shaer, Angelora Cooper, Osnat Mokryn, Andrew L Kun, and Hagit Ben Shoshan. 2024.  AI-Augmented Brainwriting: Investigating the use of LLMs in group ideation.  _arXiv preprint arXiv:2402.14978_ (2024).
  * Shneiderman (2022) Ben Shneiderman. 2022.  _Human-Centered AI_.  Oxford University Press.  Google-Books-ID: mSRXEAAAQBAJ.
  * Singer ([n. d.]) Ryan Singer. [n. d.].  Find the Elements — Shape Up.  https://basecamp.com/shapeup/1.3-chapter-04
  * Suh et al. (2023) Sangho Suh, Meng Chen, Bryan Min, Toby Jia-Jun Li, and Haijun Xia. 2023.  Structured Generation and Exploration of Design Space with Large Language Models for Human-AI Co-Creation.  _arXiv preprint arXiv:2310.12953_ (2023).
  * Swearngin et al. (2018) Amanda Swearngin, Mira Dontcheva, Wilmot Li, Joel Brandt, Morgan Dixon, and Amy J. Ko. 2018.  Rewire: Interface Design Assistance from Examples. In _Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems_ _(CHI ’18)_. Association for Computing Machinery, 1–12.  https://doi.org/10.1145/3173574.3174078 Place: New York, NY, USA.
  * Swearngin et al. (2020) Amanda Swearngin, Chenglong Wang, Alannah Oleson, James Fogarty, and Amy J. Ko. 2020.  Scout: Rapid Exploration of Interface Layout Alternatives through High-Level Design Constraints. In _Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems_. 1–13.  https://doi.org/10.1145/3313831.3376593 arXiv:2001.05424 [cs].
  * Tankala and Joyce ([n. d.]) Samiha Tankala and Alita Joyce. [n. d.].  Design-Pattern Guidelines: Study Guide.  https://www.nngroup.com/articles/design-pattern-guidelines/
  * Tidwell (2010) Jenifer Tidwell. 2010.  _Designing interfaces: Patterns for effective interaction design_.  ” O’Reilly Media, Inc.”.
  * Vattam and Goel (2011) Swaroop S Vattam and Ashok K Goel. 2011.  Foraging for inspiration: understanding and supporting the online information seeking practices of biologically inspired designers. In _International Design Engineering Technical Conferences and Computers and Information in Engineering Conference_ , Vol. 54860. 177–186.
  * Vigo and Harper (2013) Markel Vigo and Simon Harper. 2013.  Challenging information foraging theory: Screen reader users are not always driven by information scent. In _Proceedings of the 24th acm conference on hypertext and social media_. 60–68.
  * Wang et al. (2023) Bryan Wang, Gang Li, and Yang Li. 2023.  Enabling Conversational Interaction with Mobile UI using Large Language Models.  http://arxiv.org/abs/2209.08655 arXiv:2209.08655 [cs].
  * Wang et al. (2021) Bryan Wang, Gang Li, Xin Zhou, Zhourong Chen, Tovi Grossman, and Yang Li. 2021.  Screen2Words: Automatic Mobile UI Summarization with Multimodal Learning.  http://arxiv.org/abs/2108.03353 arXiv:2108.03353 [cs].
  * Wei et al. (2023) Jialiang Wei, Anne-Lise Courbis, Thomas Lambolais, Binbin Xu, Pierre Louis Bernard, and Gérard Dray. 2023.  Boosting gui prototyping with diffusion models. In _2023 IEEE 31st International Requirements Engineering Conference (RE)_. IEEE, 275–280.
  * Wei et al. (2022) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022\.  Chain-of-thought prompting elicits reasoning in large language models.  _Advances in neural information processing systems_ 35 (2022), 24824–24837.
  * Wu et al. (2023) Jason Wu, Siyan Wang, Siman Shen, Yi-Hao Peng, Jeffrey Nichols, and Jeffrey P. Bigham. 2023.  WebUI: A Dataset for Enhancing Visual UI Understanding with Web Semantics.  https://doi.org/10.48550/arXiv.2301.13280 arXiv:2301.13280 [cs].
  * Wu et al. (2022) Tongshuang Wu, Michael Terry, and Carrie Jun Cai. 2022.  Ai chains: Transparent and controllable human-ai interaction by chaining large language model prompts. In _Proceedings of the 2022 CHI conference on human factors in computing systems_. 1–22.
  * Yan et al. (2023) An Yan, Zhengyuan Yang, Wanrong Zhu, Kevin Lin, Linjie Li, Jianfeng Wang, Jianwei Yang, Yiwu Zhong, Julian McAuley, Jianfeng Gao, et al. 2023\.  Gpt-4v in wonderland: Large multimodal models for zero-shot smartphone gui navigation.  _arXiv preprint arXiv:2311.07562_ (2023).
  * Yang et al. (2023) Jianwei Yang, Hao Zhang, Feng Li, Xueyan Zou, Chunyuan Li, and Jianfeng Gao. 2023.  Set-of-Mark Prompting Unleashes Extraordinary Visual Grounding in GPT-4V.  arXiv:2310.11441 (Nov. 2023).  https://doi.org/10.48550/arXiv.2310.11441 arXiv:2310.11441 [cs].
  * YIN et al. (2024) JIAYU YIN, CATHERINE GU, JENNY MAR, SYDNEY ZHANG, and STEVEN P DOW. 2024.  Jamplate: Exploring LLM-Enhanced Templates for Idea Reflection.  (2024).
  * Zhao et al. (2021) Tianming Zhao, Chunyang Chen, Yuanning Liu, and Xiaodong Zhu. 2021.  GUIGAN: Learning to Generate GUI Designs Using Generative Adversarial Networks.  https://doi.org/10.48550/arXiv.2101.09978 arXiv:2101.09978 [cs].
  * Zhong et al. (2021) Mingyuan Zhong, Gang Li, and Yang Li. 2021.  Spacewalker: Rapid UI design exploration using lightweight markup enhancement and crowd genetic programming. In _Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems_. 1–11.

##  Appendix A Participant Demographics

###  A.1. Formative Study

Gender |  Education |  Industry | Job Title |  Years in Industry |  Company Size
---|---|---|---|---|---
Female |  Master’s degree |  Software | Product Designer |  1 to 3 years |  1,000 to 10,000
Female |  Master’s degree |  Software | Senior Product Designer |  3 to 5 years |  100 to 1,000
N/A |  Master’s degree |  Software | UX Designer |  1 to 3 years |  More than 10,000
Female |  Master’s degree |  Finance and Insurance | Senior UX Designer |  Less than 1 year |  More than 10,000
N/A |  Master’s degree |  Software | Senior Product Designer |  3 to 5 years |  10 to 100
Table 1. The demographics of our formative study participants. We used
“Software” in short for “Software, Information Services and Data Processing”
for the ease of display.

###  A.2. Flowy User Study

Gender |  Education |  Industry |  Job Title |  Industry Experience |  Company Size |  UX Experience | Used Mobbin
---|---|---|---|---|---|---|---
N/A |  Master’s degree |  Software |  Product designer |  Less than 1 year |  100 to 1,000 |  Less than 1 year | Yes
N/A |  Bachelor’s degree |  Unemployed |  interaction design student |  Less than 1 year |  1 to 10 |  Less than 1 year | Yes
N/A |  Master’s degree |  design |  product designer |  1 to 3 years |  1,000 to 10,000 |  1 to 3 years | Yes
Female |  Master’s degree |  Finance and Insurance |  Senior UX designer |  1 to 3 years |  More than 10,000 |  1 to 3 years | Maybe
Male |  Master’s degree |  Software |  Associate UX Designer |  3 to 5 years |  1,000 to 10,000 |  3 to 5 years | Yes
Female |  Master’s degree |  Software |  Product designer |  3 to 5 years |  More than 10,000 |  3 to 5 years | Yes
Female |  Bachelor’s degree |  Unemployed |  Product Designer |  1 to 3 years |  1 to 10 |  1 to 3 years | Maybe
N/A |  Master’s degree |  Scientific or Technical Services |  UX Researcher |  Less than 1 year |  1 to 10 |  Less than 1 year | No
Table 2. Demographics of our user study’s participants, including gender,
educational background, industry, job title, years of experience in the
industry, company size, years of experience as a UX designer, and whether they
have used Mobbin.com.