# Aggregating Macroeconomic Forecasts

Hurwitz  Arca Blanca Ltd.
Manfield House, One Southampton St, London WC2R 0LR, United Kingdom  Evan
<EMAIL_ADDRESS>Arca Blanca Ltd.
Manfield House, One Southampton St, London WC2R 0LR, United Kingdom  Peace
Arca Blanca Ltd.
Manfield House, One Southampton St, London WC2R 0LR, United Kingdom  Nelson
<EMAIL_ADDRESS>Arca Blanca Ltd.
Manfield House, One Southampton St, London WC2R 0LR, United Kingdom  Čevora
Arca Blanca Ltd.
Manfield House, One Southampton St, London WC2R 0LR, United Kingdom  George
<EMAIL_ADDRESS>Arca Blanca Ltd.
Manfield House, One Southampton St, London WC2R 0LR, United Kingdom

#  Achieving Stable Training of Reinforcement Learning Agents in Bimodal
Environments through Batch Learning

Hurwitz  Arca Blanca Ltd.
Manfield House, One Southampton St, London WC2R 0LR, United Kingdom  Evan
<EMAIL_ADDRESS>Arca Blanca Ltd.
Manfield House, One Southampton St, London WC2R 0LR, United Kingdom  Peace
Arca Blanca Ltd.
Manfield House, One Southampton St, London WC2R 0LR, United Kingdom  Nelson
<EMAIL_ADDRESS>Arca Blanca Ltd.
Manfield House, One Southampton St, London WC2R 0LR, United Kingdom  Čevora
Arca Blanca Ltd.
Manfield House, One Southampton St, London WC2R 0LR, United Kingdom  George
<EMAIL_ADDRESS>Arca Blanca Ltd.
Manfield House, One Southampton St, London WC2R 0LR, United Kingdom

###### Abstract

Bimodal, stochastic environments present a challenge to typical Reinforcement
Learning problems. This problem is one that is surprisingly common in real
world applications, being particularly applicable to pricing problems. In this
paper we present a novel learning approach to the tabular Q-learning
algorithm, tailored to tackling these specific challenges by using batch
updates. A simulation of pricing problem is used as a testbed to compare a
typically updated agent with a batch learning agent. The batch learning agents
are shown to be both more effective than the typically-trained agents, and to
be more resilient to the fluctuations in a large stochastic environment. This
work has a significant potential to enable practical, industrial deployment of
Reinforcement Learning in the context of pricing and others.

Much of the work and literature within the field of Reinforcement Learning
[RL] focuses on problem domains with well-regulated rewards [6] [4]. These
rewards fall into expected ranges with similar orders of magnitude. This
ignores a common problem of failure states, in which a very different reward,
typically zero, would be received that is very dissimilar to the reward
received in an episode resulting in a success state [2]. This results in a
bimodal distribution of rewards, as shown in Figure (0). While typical methods
of applying RL will have some degree of success, the bimodal nature of the
reward distribution disrupts the learning capability of the agent
considerably. This paper explores a novel batch-learning approach to the
problem in order to circumvent this disrupted learning pattern, smoothing out
the learning process and thus facilitating better performance.

Figure 1: Bimodal reward signals resulting from a theoretical pricing problem.
Some customers do not purchase and produce reward signal of 0$, others
purcahse at a varying level of discount.

A commercial pricing problem is an apt example to support our discussion:
decision makers are commonly presented with difficult decisions surrounding
what discount if any to their products will result in optimal revenue
generation [3]. The bimodal distribution of rewards emerges as a result of
binary consumer behaviors regardless of what discount is applied: consumers
will either purchase or not. The resulting reward distribution therefore has
two modes 1) for the purchases at varying discount levels, 2) zero when the
customer doesn’t purchase. Using conventional RL techniques this results in an
unstable and unpredictable training process.

Our investigation was performed in an experimental manner, by first creating a
parameterised model environment that produces the problematic behaviour and
then creating agents that use RL to solve the environment across its
parameterisations, both with and without the proposed batch learning
innovation. These results are then contrasted in order to arrive at robust
conclusions as to the effectiveness of the batch learning innovation and the
conditions under which it is most impactful.

##  1 Methods

In line with the commercial pricing example we construct a toy-problem to
demonstrate the efficiency of batch learning. The experiments will allow an
agent offering discounts and learning in a RL fashion to interact with
simulated environment. A number of variants on the environment and agent will
be explored.

###  1.1 Environment

Pricing problems are a simple yet relevant context to investigate bimodal
rewards by virtue of only one decision stage occurring in the environment; a
purchase or not. The environment is set up such that it contains a customer
who has a base probability of purchasing a given product, and that the
customer’s probability of purchase will increase or decrease in a nonlinear
manner dependant on the discount offered.

The bimodal distribution captures the probability of no purchase, or failure,
corresponding to a return of $0, while also providing approximately normally
distributed values, representing the returns in the instance of a purchase, or
success as shown in Figure 1. Two distinct problems are captured in this
broader RL problem domain. The first problem is to determine whether the
result will be a success or a failure. The second problem is to map the degree
of success if success occurs. These two problems can result in significant
challenges for a conventional RL solution, as the failure modes skew the
q-values recorded in the q-table, particularly in instances where early
training results in an outlier volume of sequential failures.

The agent interacts with the environment through offering discounts $d$ from
the base price $\pi$ to individual customers thus providing reward
$R=\pi(1-d)$ to the agent. Individual customers $c$ vary in their base
probability of considering a purchase at all $P(\beta_{c})$ but if they do
consider a purchase, the probability of purchase $\gamma$ increases as a
function of discount $d$ according to $P(\gamma|\beta_{c})=1-e^{\zeta d}$
where $\zeta$ is an arbitrary constant. This results in an overall probability
of purchase $\tau$ for a given customer of
$P{\tau_{c}}=P{\beta_{c}}-P(\neg\gamma|\beta_{c},d)$ and the expected reward
from offering a discount to customer
$\mathbb{E}(R_{c})=P({\tau_{c}})R(\pi,d)$. The essential quantities are
summarised in figure 2 for the sake of clarity.

|  meaning | properties
---|---|---
$c$ |  customer identity |
$\pi$ |  base (undiscounted) price | $R\in\mathcal{R^{+}}$
$d$ |  discount | $d\in\mathcal{R}:\langle 0,1\rangle$
$R$ |  reward / revenue, objective of maximisation through RL | $R\in\mathcal{R^{+}}$
$\beta$ |  considering a purchase at all irrespective of discount | binary event
$\gamma$ |  purchase event, given a discount $d$ and an initial consideration $\beta$ | binary event
$\tau$ |  purchase | binary event
Figure 2: The fundamental quantities describing the environment of our RL
task.

While $P(\gamma)$ increases as the discount increases, the revenue resulting
from the purchase decreases with the discount increasing. This results in a
non-monotonic relationship between discount and expected revenue
$\mathbb{E}(R_{c}|d)$ shown in Figure (0). The optimum discount is therefore
the value which maximises $\mathbb{E}(R_{c})$. This value can be found in an
analytical fashion for the environment defined here, but is unknown in the
real world. The task of RL agents tested in this environment is to maximise
the function without knowledge of the environment.

Figure 3: Expected revenue $\mathbb{E}(R_{c}|d,\beta)$ as a function of
discount and different probabilities of initial consideration of the purchase
by customers.

While the $\mathbb{E}(R_{c}|d,\beta)$ function shown in Figure 3 is smooth,
and with no local minima, the optimisation task is not trivial due to the
bimodal nature of the rewards. To explore the characteristics of the task we
set up two different versions of the environment:

  * •

The initial consideration of purchase $\beta$ varying in probability between
0.2 and 0.8 in 0.1 steps, called the sparse state-space. Resulting in
$P(\beta)\in[0.2;0.3;0.4;0.5;0.6;0.7;0.8]$.

  * •

The initial consideration of purchase $\beta$ varying in probability between
0.2 and 0.785 in 0.015 steps, called the granular state-space. Resulting in 40
different values of $P(\beta)$.

###  1.2 Agent

The agent is required to make a single decision what discount to offer the
customer at a single time-point. To accommodate a standard, tabular q-learning
approach, this action-space is discretised. Two versions of the action-space
discretisation are explored, each representing a different level of
granularity:

  * •

The agent may choose from 10 discount levels, the sparse action-space:
$d\in[0;0.1;0.2;0.3;0.4;0.5;0.6;0.7;0.8;0.9]$.

  * •

The agent may choose from 81 discount levels in 0.01 intervals between 0 and
0.79, the granular action-space.

The agent learns using standard tabular Q-learning with Bellman equation [1],
making decisions according to an an $\epsilon$-greedy policy [5]. The learning
rate used is 0.1, and the $\epsilon=0.9$.

###  1.3 Experimental setup

To demonstrate the benefit of batch learning for the task defined here the
agent will either update the Q-table after each observation as is common in
online RL or alternatively store a user-defined number (1000 in the
experiments presented here) of observations in memory before updating the
Q-table with average rewards over the whole batch. Factorial design is used to
explore all four combinations of state- and action- spaces.

All experiments are run for 100,000 iterations allowing for learning until
convergence. Convergence is defined as the agent achieving (on a rolling mean
basis) 95% of the optimum performance under perfect knowledge of the
environment. The optimum was calculated analytically for sparse state-space,
but approximated using MCMC for the granular state-space.

###  1.4 Illustrative results from a simplified experiment

In order to demonstrate the shortfalls and merits of single-update and batch-
update visually to the reader an additional experiment is run in a more
simplistic environment. For this purpose the base purchase probability is
fixed at 0.6 making 0.1 discount the optimum action for the agent.

(a) Standard Q-learning

(b) Batch Q-learning

Figure 4: Learning dynamics in a simplified experimental setup to facilitate
understanding of agent’s performance. The agent in both traditional Q-learning
and in the batched Q-learning cases have the full range of 80 actions to
choose from, and their Q-states and rewards are tracked at each step. The
illustrative results are then shown for a single action $d=0.17$, with events
from all other actions filtered out. This allows us to observe the change in
Q-value expectations as well as the reward signals received by the agent in
sequential order.

As can be seen in Figure 4 both standard and batch Q-learning results in a
gradual convergence towards the analytically calculated expectation, which is
inaccessible to the agent, then oscillate around that value in response to the
stochasticity of the rewards. Compared to the standard approach, batch
learning offers slower convergence, but is less sensitive to the stochastic
oscillations, with higher overall reward achieved, making it much more stable
and usable tool to be deployed. Convergence is clearly slower, but this is
more than made up for in better converged performance which comes out to a
2.17% overall improvement in reward over the time period, as well as more
stable converged performance.

##  2 Results

Four unique experimental setups shown in figure 5 were tested in eight
experiments, with batch learning outperforming the standard approach in every
single case as measured by the total reward accumulated over the experiments.
The final reward at fully-trained state was improved by batch learning in all
cases except for the largest state-space.

Action-space size |  State-space size |  Update method |  Final reward [$] |  Total reward [$10^{6}$$]
---|---|---|---|---
10 |  10 |  single |  38.18 |  3.35
10 |  10 |  batch |  38.73 |  3.46
10 |  40 |  single |  37.77 |  3.31
10 |  40 |  batch |  38.58 |  3.35
80 |  10 |  single |  38.66 |  3.31
80 |  10 |  batch |  38.93 |  3.42
80 |  40 |  single |  37.97 |  3.11
80 |  40 |  batch |  37.41 |  3.17
Figure 5: Benchmark rewards vs RL agent learned rewards for both q-learning
methodologies

##  3 Conclusions

This paper proposes a batch training method for Reinforcement Learning [RL]
agents. This update method has been shown to be superior compared to standard
methods in environments with bimodal rewards such as offering discounts to
customers with differing price sensitivity. The batch training method not only
produces better performance in these challenging environments, but also
produces more decisive agents acting on learned information. Convergence time
is adversely affected, as is intuitive, which becomes the cost for a more
stable and optimal converged state when operating in a bimodal manner. Stable
performance and better convergence in real-world scenarios like this is an
important step on the path to industrialize RL.

## References

  * [1] EN Barron and H Ishii.  The bellman equation for minimizing the maximum cost.  NONLINEAR ANAL. THEORY METHODS APPLIC., 13(9):1067–1090, 1989.
  * [2] J.C. Hoskins; D.M. Himmelblau.  Process control via artificial neural networks and reinforcement learning.  Computers & Chemical Engineering, 16:241–251, 1992.
  * [3] HE Horden and C Bisset.  Optimisation of promotions in the retail industry.  South African Journal of Industrial Engineering, 33(3):40–54, 2022\.
  * [4] R.T. Icarte, Klassen T. Q., Valenzano R., and McIlraith S.A.  Reward machines: Exploiting reward function structure in reinforcement learning.  Journal of Artificial Intelligence Research, 73(1):173–208, 2022\.
  * [5] Richard S Sutton and Andrew G Barto.  Reinforcement learning: An introduction.  MIT press, 2018.
  * [6] Jane X Wang, Zeb Kurth-Nelson, Dhruva Tirumala, Hubert Soyer, Joel Z Leibo, Remi Munos, Charles Blundell, Dharshan Kumaran, and Matt Botvinick.  Learning to reinforcement learn, 2017.


# Psychologically-Inspired, Unsupervised Inference of Perceptual Groups of GUI
Widgets from GUI Images

Mulong Xie<EMAIL_ADDRESS>Australian National UniversityAustralia ,
Zhenchang Xing<EMAIL_ADDRESS>CSIRO’s Data61 & ANUAustralia ,
Sidong Feng<EMAIL_ADDRESS>Monash UniversityAustralia ,  Xiwei Xu
<EMAIL_ADDRESS>CSIRO’s Data61Australia ,  Liming Zhu
<EMAIL_ADDRESS>CSIRO’s Data61 & UNSWAustralia and  Chunyang Chen
<EMAIL_ADDRESS>Monash UniversityAustralia

(2022)

###### Abstract.

Graphical User Interface (GUI) is not merely a collection of individual and
unrelated widgets, but rather partitions discrete widgets into groups by
various visual cues, thus forming higher-order perceptual units such as tab,
menu, card or list. The ability to automatically segment a GUI into perceptual
groups of widgets constitutes a fundamental component of visual intelligence
to automate GUI design, implementation and automation tasks. Although humans
can partition a GUI into meaningful perceptual groups of widgets in a highly
reliable way, perceptual grouping is still an open challenge for computational
approaches. Existing methods rely on ad-hoc heuristics or supervised machine
learning that is dependent on specific GUI implementations and runtime
information. Research in psychology and biological vision has formulated a set
of principles (i.e., Gestalt theory of perception) that describe how humans
group elements in visual scenes based on visual cues like connectivity,
similarity, proximity and continuity. These principles are domain-independent
and have been widely adopted by practitioners to structure content on GUIs to
improve aesthetic pleasantness and usability. Inspired by these principles, we
present a novel unsupervised image-based method for inferring perceptual
groups of GUI widgets. Our method requires only GUI pixel images, is
independent of GUI implementation, and does not require any training data. The
evaluation on a dataset of 1,091 GUIs collected from 772 mobile apps and 20 UI
design mockups shows that our method significantly outperforms the state-of-
the-art ad-hoc heuristics-based baseline. Our perceptual grouping method
creates opportunities for improving UI-related software engineering tasks.

Graphical User Interface, Widget Grouping, Perceptual Grouping

††copyright: rightsretained††doi: 10.1145/3540250.3549138††journalyear:
2022††submissionid: fse22main-p648-p††isbn:
978-1-4503-9413-0/22/11††conference: Proceedings of the 30th ACM Joint
European Software Engineering Conference and Symposium on the Foundations of
Software Engineering; November 14–18, 2022; Singapore, Singapore††booktitle:
Proceedings of the 30th ACM Joint European Software Engineering Conference and
Symposium on the Foundations of Software Engineering (ESEC/FSE ’22), November
14–18, 2022, Singapore, Singapore††ccs: Software and its engineering

##  1\. introduction

Figure 1. Examples of perceptual groups of GUI widgets (perceptual groups are
highlighted in pink box in this paper)

We do not just see a collection of separated texts, images, buttons, etc., on
GUIs. Instead, we see perceptual groups of GUI widgets, such as card, list,
tab and menu shown in Figure 1. Forming perceptual groups is an essential step
towards visual intelligence. For example, it helps us decide which actions are
most applicable to certain GUI parts, such as, clicking a navigation tab,
expanding a card, scroll the list. This would enable more efficient automatic
GUI testing (Li et al., 2019; Degott et al., 2019). As another example, screen
readers (app, 2021; goo, [n.d.]a) help visually impaired users access
applications by reading out content on GUI. Recognizing perceptual groups
would allow screen readers to navigate the GUI at higher-order perceptual
units (e.g., sections) efficiently (Zhang et al., 2021). Last but not least,
GUI requirements, designs and implementations are much more volatile than
business logic and functional algorithms. With perceptual grouping, modular,
reusable GUI code can be automatically generated from GUI design images, which
would expedite rapid GUI prototyping and evolution (Nguyen and Csallner, 2015;
Moran et al., 2020).

Although humans can intuitively see perceptual groups of GUI widgets, current
computational approaches are limited in partitioning a GUI into meaningful
groups of widgets. Some recent work (Beltramelli, 2018; Chen et al., 2018)
relies on supervised deep learning methods (e.g., image captioning (Vinyals et
al., 2015; Lecun et al., 2015)) to generate a view hierarchy for a GUI image.
This type of method is heavily dependent on GUI data availability and quality.
To obtain sufficient GUI data for model training, they use GUI screenshots and
view hierarchies obtained at application runtime. A critical quality issue of
such runtime GUI data is that runtime view hierarchies often do not correspond
to intuitive perceptual groups due to many implementation-level tricks. For
example, in the left GUI in Figure 2, the two ListItems in a ListView has no
visual similarity (a large image versus some texts), so they do not form a
perceptual group. In the right GUI, a grid of cards form a perceptual group
but is implemented as individual FrameLayouts. Such inconsistencies between
the implemented widget groups and the human’s perceptual groups make the
trained models unreliable to detect perceptual groups of GUI widgets.

Decades of psychology and biological vision research have formulated the
Gestalt theory of perception that explains how humans see the whole rather
than individual and unrelated parts. It includes a set of principles of
grouping, among which connectedness, similarity, proximity and continuity are
the most essential ones (Sternberg and Robert, 2003; ges, 2006). Although
these principles and other related UI design principles such as CRAP (S1T2,
2021) greatly influence how designers and developers structure GUI widgets
(Thalion, 2020), they have never been systematically used to automatically
infer perceptual groups from GUI images. Rather, current approaches (Nguyen
and Csallner, 2015; Zhang et al., 2021) rely on ad-hoc and case-specific rules
and thus are hard to generalize on diverse GUI designs.

In this work, we systematically explore the Gestalt principles of grouping and
design the first psychologically-inspired method for visual inference of
perceptual groups of GUI widgets. Our method requires only GUI pixel images
and is independent of GUI implementations. Our method is unsupervised, thus
removing the dependence on problematic GUI runtime data. As shwon in Figure 3,
our method enhances the state-of-the-art GUI widget detection method (UIED
(Chen et al., 2020d; Xie et al., 2020)) to detect elementary GUI widgets.
Following the Gestalt principles, the method first detects containers (e.g.,
card, list item) with complex widgets by the connectedness principle. It then
clusters visually similar texts (or non-text widgets) by the similarity
principle and further groups clusters of widgets by the proximity principles.
Finally, based on the widget clusters, our method corrects erroneously
detected or missing GUI widgets by the continuity principle (not illustrated
in Figure 3, but can be seen in Figure 5).

At the right end of Figure 3, we show the grouping result by the state-of-the-
art heuristic-based method Screen Recognition (Zhang et al., 2021). Screen
Recognition incorrectly partitions many widgets into groups, such as the
bottom navigation bar and the four widgets above the bar, the card on the left
and the text above the card. It also fails to detect higher-order perceptual
groups, such as groups of cards. In contrast, our approach correctly
recognizes the bottom navigation bar and the top and middle row of cards.
Although the text label above the left card is very close to the card, our
approach correctly recognizes the text labels as separate widgets rather than
as a part of the left card. Our approach does not recognize the two cards just
above the bottom navigation bar because these two cards are partially occluded
by the bottom bar. However, it correctly recognizes the two blocks of image
and text and detects them as a group. Clearly, the grouping results by our
approach correspond more intuitively to human perception than those by Screen
Recognition.

For the evaluation, we construct two datasets: one contains 1,091 GUI
screenshots from 772 Android apps, and the other contains 20 UI prototypes
from a popular design tool Figma (fig, [n.d.]). To ensure the validity of
ground-truth widget groups, we manually check all these GUIs and confirm that
none of these GUIs has the perception-implementation misalignment issues shown
in Figure 2. We first examine our enhanced version of UIED and observe that
the enhanced version reaches a 0.626 F1 score for GUI widget detection, which
is much higher than the original version (0.524 F1). With the detected GUI
widgets, our perceptual grouping approach achieves the F1-score of 0.593 on
the 1,091 app GUI screenshots and 0.783 F1-score on the 20 UI design
prototypes. To understand the impact of GUI widget misdetections on perceptual
grouping, we extract the GUI widgets directly from the Android app’s runtime
metadata (i.e., ground-truth widgets) and use the ground-truth widgets as the
inputs to perceptual grouping. With such “perfectly-detected” GUI widgets, our
grouping approach achieves a 0.672 F1-score on app GUIs. In contrast, Screen
Recognition (Zhang et al., 2021) performs very poorly: 0.123 F1 on the ground-
truth widgets and 0.092 F1 on the detected widgets for app screenshots, and
0.232 F1 on the detected widgets for UI design prototypes. Although our
grouping results sometimes do not exactly match the ground-truth groups, our
analysis shows that some of our grouping results still comply with how humans
perceive the widget groups because there can be diverse ways to structure GUI
widgets in some cases.

Figure 2. Implemented view hierarchy does not necessarily correspond to
perceptual groups Figure 3. Left: Our approach overview: (1) Enhanced UIED
(Chen et al., 2020d) for GUI widget detection; (2) Gestalt-principles inspired
perceptual grouping. Right: Grouping result of the state of of the art
heuristic-based approach ScreenRecognition (Zhang et al., 2021)

To summarize, this paper makes the following contributions:

  * •

A robust, psychologically-inspired, unsupervised visual inference method for
detecting perceptual groups of GUI widgets on GUI images, the code is released
on GitHub111https://github.com/MulongXie/GUI-Perceptual-Grouping.

  * •

A comprehensive evaluation of the proposed approach and the in-depth analysis
of the performance with examples.

  * •

An analysis of how our perceptual grouping method can improve UI-related SE
tasks, such as UI design, implementation and automation.

##  2\. GUI Widget Detection

Our approach is a pixel-only approach. It does not assume any GUI metadata or
GUI implementation about GUI widgets. Instead, our approach detects text and
non-text GUI widgets directly from GUI images. To obtain the widget
information from pixels, it enhances the state-of-the-art GUI widget detection
technique UIED (Xie et al., 2020). In order to fit with subsequent perceptual
grouping, our approach mitigates the incorrect merging of GUI widgets in the
containers by UIED and simplifies the widget classification of UIED.

###  2.1. UIED-Based GUI Widget Detection

UIED comprises three steps: (1) GUI text detection, (2) non-text GUI widget
detection and (3) merging of text and non-text widgets. UIED uses an off-the-
shelf scene text detector EAST (Zhou et al., 2017) to identify text regions in
the GUI images. EAST is designed for handling nature scene images that differ
from GUI images, such as figure-background complexity and lighting effects.
Although EAST outperforms traditional OCR tool Tesseract (Tesseract-Ocr,
[n.d.]), we find the latest OCR tool developed by Google (goo, [n.d.]b)
achieves the highest accuracy of GUI text recognition (see Section 4.1).
Therefore, in our use of UIED, we replace EAST with the Google OCR tool.

For locating non-text widgets, our approach adopts the design of UIED that
uses a series of traditional, unsupervised image processing algorithms rather
than deep-learning models (e.g., Faster RCNN (Girshick, 2015) or YOLO (Redmon
and Farhadi, 2018)). This design removes the data dependence on GUI
implementation or runtime information while accurately detecting GUI widgets.
UIED then merges the text and non-text detection results. The purpose of this
merging step is not only to integrate the identified GUI widgets but also to
cross-check the results. Because non-text widget detection inevitably extracts
some text regions, UIED counts on the OCR results to remove these false-
positive non-text widgets. Specifically, this step checks the bounding box of
all candidate non-text widget regions and removes those intersected with text
regions resulting from the OCR.

###  2.2. Improvement and Simplification of UIED

We find that the UIED detection results often miss some widgets in a container
(e.g. card). The reason is that, in order to filter out invalid non-text
regions and mitigate over-segmentation that wrongly segments a GUI widget into
several parts, UIED checks the widgets’ bounding boxes and merges all
intersected widgets regions into a big widget region. This operation may cause
the wrong neglection of valid GUI widgets that are enclosed in some
containers. Therefore, we equip our GUI widget detector with a container
recognition algorithm (see Section 3.2) to mitigate the issue. If a widget is
recognized as a container, then all its contained widgets are kept and
regarded as proper GUI widgets rather than noises.

The original UIED classifies non-text GUI widgets as specific widget
categories (e.g., image, button, checkbox). In contrast, our GUI widget
detector only distinguishes text from non-text widgets. Although GUI involves
many types of non-text widgets, there is no need of distinguishing actual
classes of non-text widgets for perceptual grouping. GUI widget classes
indicate different user interactions and GUI functionalities, but widgets with
different classes can form a perceptual group as long as they have similar
visual properties, such as size, shape, relative position and alignment with
other widgets. Therefore, we do not distinguish different classes of non-text
widgets. However, we need to distinguish non-text widgets from text widgets,
as they have very different visual properties and need to be clustered by
different strategies (see Section 3).

##  3\. GUI Widget Perceptual Grouping

After obtaining text and non-text widgets on a GUI image, the next step is to
partition GUI widgets into perceptual groups (or blocks of items) according to
their visual and perceptual properties.

###  3.1. Gestalt Laws and Approach Overview

Our approach is inspired by psychology and biological-vision research.
Perceptual grouping is a cognitive process in which our minds leap from
comprehending all of the objects as individuals to recognizing visual patterns
through grouping visually related elements as a whole (Eysenck and Brysbaert,
2018). This process affects the way we design GUI layouts (S1T2, 2021) from
alignment, spacing and grouping tool support (fig, [n.d.]; Sketch, [n.d.]) to
UI design templates (Dribbble, [n.d.]) and GUI frameworks (wix, [n.d.]) It
also explains how we perceive GUI layouts. For instance, in the examples in
Figure 1, we subconsciously observe that some visually similar widgets are
placed in a spatially similar way and identify them as in a group (e.g. a
card, list, multitab or menu).

Previous studies rely on ad-hoc, rigid heuristics to infer UI structure
without a systematic theoretical foundation. Our approach is the first attempt
to tackle the perceptual grouping of GUI widgets guided by an influential
psychological theory (named Gestalt psychology (ges, 2006)) that explains how
the human brain perceives objects and patterns. Gestalt psychology’s core
proposition is that human understands external stimuli as wholes rather than
as the sums of their parts (Stevenson, [n.d.]). Based on the proposition, the
Gestaltists studied perceptual grouping (Eysenck and Brysbaert, 2018)
systematically and summarized a set of “gestalt laws of grouping” (Sternberg
and Robert, 2003). In our work, we adopt the four most effective principles
which greatly influence GUI design (Thalion, 2020) in practice as the
guideline for our approach design:  (1) connectedness  (2) similarity  (3)
proximity and  (4) continuity.

We define a group of related GUI widgets as a layout block of items. A typical
example is a list of list items in the GUI, or a card displaying an image and
some texts. The fundamental intuition is: if a set of widgets have similar
visual properties and are placed in alignment with similar space between each
other, they will be “perceived” as in the same layout block by our approach
according to the Gestalt principles. In detail, our approach consists of four
grouping steps in accordance with four Gestalt principles. First, it
identifies containers along with their contained widgets that fulfil the
connectedness law. Second, it uses an unsupervised clustering method DBSCAN
(Ester et al., 1996) to cluster text or non-text GUI widgets based on their
spatial and size similarities. Next, it groups proximate and spatially aligned
clusters to form a larger layout block following the proximity law. Finally,
in line with the continuity principle, our approach corrects some mistakes of
GUI widget detection by checking the consistency of the groups’ compositions.

###  3.2. Connectedness - Container Recognition

In Gestalt psychology, the principle of uniform connectedness is the strongest
principle concerned with relatedness (Rutledge, 2009). It implies that we
perceive elements connected by uniform visual properties as being more related
than those not connected. The forms of the connection can be either a line
connecting several elements or a shape boundary that encloses a group of
related elements. In the GUI, the presentation of the connectedness is usually
a box container that contains multiple widgets within it, and all the enclosed
widgets are perceived as in the same group. Thus, the first step of our
grouping approach is to recognize the containers in a GUI.

In particular, we observe that a container is visually a (round) rectangular
wireframe enclosing several children widgets. The card is a typical example of
such containers, as shown in Figure 1(a). Therefore, with the detected non-
text widgets, the algorithm first checks if a widget is of rectangle shape by
counting how many straight lines its boundary comprises and how they compose.
Specifically, we apply the geometrical rule that a rectangle’s sides are made
of 4 straight lines perpendicular to each other. Subsequently, our approach
determines if the widget’s boundary is a wireframe border by checking if it is
connected with any widgets inside its boundary. If a widget satisfies the
above criteria, it will be identified as a container, and all widgets
contained within it are partitioned into the same perceptual group.

###  3.3. Similarity - Widget Clustering

The principle of similarity suggests that elements are perceptually grouped
together if they are similar to each other (wik, 2021). Generally, similarity
can be observed in aspects of various visual cues, such as size, color, shape
or position. For example, in the second GUI of Figure 1, the image widgets are
of the same size and aligned with each other in the same way (i.e., same
direction and spacing), so we visually perceive them as a group. Likewise, the
text pieces on the right of the image widgets are perceptually similar even
though they have different font styles and lengths because they have the same
alignment with neighbouring texts.

####  3.3.1. Initial Widget Clustering

Our approach identifies similar GUI widgets by their spatial and visual
properties and aggregates similar GUI widgets into blocks by similarity-based
clustering. It clusters texts and non-text widgets through different
strategies. In general, similar non-text widgets in the same block (e.g. a
list) usually have similar sizes and align to one another vertically or
horizontally with the same spacing. Texts in the same block are always left-
justified or top-justified (assume left-to-right text orientation), but their
sizes and shapes can vary significantly because of different lengths of text
contents. Thus, the approach clusters the non-text widgets by their center
points $(Center_{X},Center_{Y})$ and areas, and it clusters texts by their
top-left corner $(Top,Left)$

Figure 4. Widget clustering, cluster conflict resolving and final resulting
groups in which we use the same color to paint the widgets in the same
subgroup and highlight higher-order groups in pink boxes

Our approach uses the DBSCAN (Density-Based Spatial Clustering of Applications
with Noise) algorithm (Ester et al., 1996) to implement the clustering.
Intuitively, DBSCAN groups the points closely packed together (points with
many nearby neighbors), while marking points whose distance from the nearest
neighbor is greater than the maximum threshold as outliers. In the GUI widget
clustering context, the point is the GUI widget, and the distance is the
difference between the values of the widgets’ attribute that the clustering is
based on. Figure 4 illustrates the clustering process. For non-text widgets,
our approach performs the clustering three times based on three attributes
respectively. It first clusters the widgets by $Center_{x}$ for the horizontal
alignment, then by $Center_{Y}$ for the vertical alignment and finally by
$area$. These operations produce three clusters: $Cluster_{horizontal}^{non-
text}$, $Cluster_{vertical}^{non-text}$ and $Cluster_{area}^{non-text}$. Our
approach then clusters the text widgets twice based on their top left corner
point $(Top,Left)$ for left-justified (vertical) and top-justified
(horizontal) alignment. It produces the $Cluster_{horizontal}^{text}$ based on
the texts’ $Top$, and the $Cluster_{vertical}^{text}$ based on the texts’
$Left$. The resulting clusters are highlighted by different colors and numbers
in Figure 4. We only keep the clusters with at least two widgets and discard
those with only one widget.

####  3.3.2. Cluster Conflicts Resolving

It is common that some widgets can be clustered into different clusters by
different attributes, which causes cluster conflicts. For example, as
illustrated in Figure 4, several non-text widgets (e.g., the bottom-left
image) are both in a vertical cluster (marked in blue) and a horizontal
cluster (marked in red). The intersection of clusters illustrates the
conflict. The approach shall resolve such cluster conflicts to determine to
which group the widget belongs. This conflict-resolving step also complies
with the similarity principle that suggests the widgets in the same perceptual
group should share more similar properties.

The conflict resolving step first calculates the average widget areas of the
groups to which the conflicting widget has been assigned. In accordance with
the similarity principle, the widget is more likely to be in a group whose
average widget area is similar to the conflicting widget’s area. In addition,
another observation is that repetitive widgets in a group have similar spacing
between each other. So for a widget that is clustered into multiple candidate
groups, the approach checks the difference between the spacing of this widget
and its neighboring widgets in a group and the average spacing between other
widgets in that group. It keeps the widget in the group where the conflicting
widget has the largest widget-area similarity and the smallest spacing
difference compared with other widgets in the group. For example, the bottom-
left image widget will be assigned to the vertical cluster rather than the
horizontal one according to our conflict resolving criteria. After conflict
resolving, our approach produces the final widget clustering results as shown
in the right part of Figure 4. We use different colors and indices to
illustrate the resulting non-text (nt) and text (t) clusters.

Figure 5. Examples of widget detection error correction. (1st column - green
box: non-text; red-box: text; 2nd column - same color: higher-order perceptual
group; 3rd column - same color: subgroup of widgets)

###  3.4. Proximity - Iterative Group Pairing

So far, GUI widgets are aggregated into groups as per the connectedness and
similarity principles. Some groups are close to each other and similar in
terms of the number and layout of the contained widgets, which may further
form a larger perceptual group even though these groups may contain different
types of widgets. For example, in the clustering result of Figure 4, we can
observe that the clusters nt-0, t-0, t-0-1 and nt-2 are proximate and have the
same or similar number of widgets aligned in the same way. We can see this
feature intentionally or subconsciously and perceive them as in the same large
group as a whole. Gestalt psychology states that when people see an assortment
of objects, they tend to perceive objects that are close to each other as a
group (wik, 2021). The close distance, also known as proximity, of elements is
so powerful that it can override widget similarity and other factors that
might differentiate a group of objects (UserTesting, 2019). Thus, the next
step is based on widget clusters’ proximity and composition similarity to pair
the clusters into a larger group (i.e., layout block).

If two groups $Group_{a}$ and $Group_{b}$ are next (proximate) to each other
(i.e., no other groups in between), and they contain the same number of
widgets and the widgets in the $Group_{a}$ and the $Group_{b}$ share the same
orientation (vertical or horizontal), our approach combines $Group_{a}$ and
$Group_{b}$ into a larger block. A widget in $Group_{a}$ and its closet widget
in $Group_{b}$ will be paired and form a subgroup of widgets. Our approach
first combines the two proximate groups containing the same type of widgets,
and then the groups containing different types of widgets. The formed larger
block can be iteratively combined with the proximate groups until no more
proximate groups are available.

Sometimes there are different numbers of widgets in the two proximate groups
but the two groups may still form one larger perceptual block. For example,
the cluster nt-2 in Figure 4 has one less widget compared to nt-0, t-0 and
t-0-1 because the bottom widget in the right column is occluded by the float
action button and thus missed by the detector. Another common reason for the
widget number difference is that widgets in a group may be set as invisible in
some situations, and thus they do not appear visually. Therefore, if the
difference between the number of widgets in the two proximate groups is less
than 4 (empirically determined from the ground-truth groups in our dataset),
our approach also combines the two groups into a larger block.

As shown in the final groups in Figure 4, our approach identifies a set of
perceptual groups (blocks), including the multitab at the top and the list in
the main area. Each list item is a combined widget of some non-text and text
widgets (highlighted in the same color). These perceptual groups encode the
comprehension of the GUI structure into higher-order layout blocks that can be
used in further processing and applications.

###  3.5. Continuity - Detection Error Correction

Figure 6. Examples of GUI widget detection and perceptual grouping results
(red box - text widget, green box - non-text widget, pink box - perceptual
group). Metadata-based means grouping the ground-truth widgets directly from
GUI metadata.

The GUI widget detector may make two types of detection errors - missed
widgets and misclassified widgets. Missed widgets means that the detector
fails to detect some GUI elements on the GUI (e.g., the bottom-right icon in
Figure 5(a)). Misclassified widgets refer to the widgets that the detector
reports the wrong type, for example, the top-right small icon (i.e., a non-
text widget) in the middle card in Figure 5(b) is misclassified as a text
widget due to an OCR error.

It is hard to recognize and correct these detection errors from the individual
widget perspective, but applying the Gestalt continuity principle to expose
such widget detection errors by contrasting widgets in the same perceptual
groups can mitigate the issue. The continuity principle states that elements
arranged in a line or curve are perceived to be more related than elements not
in a line or curve (Rutledge, 2009). Thus, some detection errors are likely to
be spotted if a GUI area or a widget aligns with all the widgets in a
perceptual group in a line but is not gathered into that group.

Our approach tries to identify and fix missed widgets as follows. It first
inspects the subgroups of widgets in a perceptual group and checks if the
widgets in the subgroups are consistent in terms of the number and relative
position of the contained widgets. If a subgroup contains fewer widgets than
its counterparts, then the approach locates the inconsistent regions by
checking the relative positions and areas of other subgroups’ widgets. Next,
the approach crops the located UI regions and uses the widget detector upon
the cropped regions with relaxed parameters (i.e. double of the minimum area
threshold for valid widgets) to try to identify the missed widget, if any. For
example, the tiny icon at the bottom right in Figure 5(a) is missed because
its area is so small that the detector regards it as a noisy region and hence
discards it in the initial detection. By analyzing the resulting perceptual
group and its composition, our approach finds that seven of the eight
subgroups have two widgets (marked in the same color), while the subgroup at
the bottom right has only one widget. It crops the area that may contain the
missed widget according to the average sizes and average relative positions of
the two widgets in the other seven subgroups. The missed tiny icon can be
recovered by detecting the widget with the relaxed valid-widget minimum area
threshold in the missing area.

Our approach uses the exact mechanism that contrasts the subgroups to identify
the misclassified widgets, but here it focuses on widget type consistency. As
shown in Figure 5(b), our approach groups the three cards in a perceptual
group. By contrasting the widgets in the three cards, it detects that the
middle card has text widgets at the top right corner, while the other two
cards have a non-text widget at the same relative positions. Based on the
continuity principle, our approach re-classifies the top-right widget in the
middle card as non-text with a majority-win strategy.

##  4\. Evaluation

We evaluate our approach in two steps: (1) examine the accuracy of our
enhanced version of UIED and compare it with the original UIED (Chen et al.,
2020d); (2) examine the accuracy of our widget perceptual grouping approach
and compare it with the state-of-the-art heuristic-based method Screen
Recognition (Zhang et al., 2021).

Table 1. Overall results of widget detection (IoU $>$ 0.9)

| Our Enhanced Revision | Original UIED
---|---|---
Type | Precision | Recall | F1 | Precision | Recall | F1
Non-Text | 0.589 | 0.823 | 0.687 | 0.431 | 0.469 | 0.449
Text | 0.678 | 0.693 | 0.686 | 0.402 | 0.720 | 0.516
All Widgets | 0.580 | 0.680 | 0.626 | 0.490 | 0.557 | 0.524

###  4.1. Accuracy of GUI Widget Detection

Compared with the original UIED (Chen et al., 2020d), our GUI widget detector
uses the latest Google OCR tool and improve the text and non-text widget
merging by container analysis. We evaluate GUI widget detection from the three
perspectives: text widget detection, non-text widget detection and the final
widget results after merging. To be consistent with the evaluation setting in
the UIED paper (Chen et al., 2020d), we run experiments on the same Rico
dataset of Android app GUIs (Liu et al., 2018) and regard the detected widgets
whose intersection over union (IoU) with the ground truth widget is over 0.9
as true positive. The ground-truth widgets are the leaf widgets (i.e., non-
layout classes) extracted from the GUI’s runtime view hierarchy.

Table 1 shows the widget detection performance of the enhanced and the
original UIED. Our enhanced version achieves a much higher recall (0.823) for
non-text widgets than the original UIED (0.469), and meanwhile, it also
improves the precision (0.589 over 0.431). This significant improvement is due
to the more intelligent container-aware merging of text and non-text widgets
by our enhanced version. As the original UIED is container-agnostic, it
erroneously discards many valid widgets contained in other widgets as noise.
For GUI text, the Google OCR tool used in the enhanced version achieves much
higher precision (0.678) than the EAST model used in the original UIED
(0.402), with a slight decrease in recall (0.693 versus 0.720). The
improvements in both text and non-text widgets result in a much better overall
performance (0.626 F1 by the enhanced version versus 0.524 by the original
UIED).

###  4.2. Perceptual Grouping Performance

We evaluate our perceptual grouping approach on both Android app GUIs and UI
design prototypes. Figure 6 shows some perceptual grouping results by our
approach. These results show our approach can reliably detect GUI widgets and
infer perceptual groups for diverse visual and layout designs.

####  4.2.1. Datasets

Our approach simulates how humans perceive the GUI structure and segment a GUI
into blocks of widgets according to the Gestalt principles of grouping. To
validate the recognized blocks, we build the ground-truth dataset from two
sources: Android apps and UI design prototypes. The ground truth annotates the
widget groups according to the GUI layout and the widget styles and properties
as shown in Figure 7.

Figure 7. Examples of Android app GUI and UI design prototype, view hierarchy
and ground truth

Android App GUI Dataset The ground truth of widget groups can be obtained by
examining the layout classes used to group other widgets in the
implementations. However, as shown in Figure 2, the layout classes do not
always correspond to the perceptual groups of GUI widgets. Therefore, we
cannot use the GUI layout classes directly as the ground truth. Instead, we
first search the GUIs in the Rico dataset of Android app GUIs (Liu et al.,
2018) that use certain Android layout classes that may contain a group of
widgets (e.g., ListView, FrameLayout, Card, TabLayout). Then we manually
examine the candidate GUIs to filter out those whose use of layout classes has
obvious violations against the Gestalt principles. Furthermore, the Rico
dataset contains many highly-similar GUI screenshots for an application. To
increase the visual and layout diversity of GUIs in our dataset, we limit up
to three distinct GUI screenshots per application. Distinction is determined
by the number and type of GUI widgets and the GUI structure. We obtain 1091
GUI screenshots from 772 Android applications. Using this dataset, we evaluate
both detection-based and metadata-based grouping. Detection-based grouping
processes the detected widgets, while metadata-based grouping uses the widgets
obtained from the GUI metadata (i.e., assumes the perfect widget detection).

UI Design Prototypes We collect 20 UI design prototypes shared on a popular UI
design website (Figma (fig, [n.d.])). These UI design prototypes are created
by professional designers for various kinds of apps and receive more than 200
likes. This small set of UI design prototypes demonstrates how professional
designers structure the GUIs and group GUI widgets from the design rather than
the implementation perspective. As a domain-independent tool, Figma supports
only elementary visual elements (i.e., text, image and shape). Designers can
create any widgets using these elementary visual elements. Due to the lack of
explicit and uniform widget metadata in the Figma designs, we evaluate only
the detection-based grouping on these UI design prototypes.

####  4.2.2. Metrics

The left part of Figure 7 shows an example in our Android app GUI dataset. We
see that the layout classes (e.g., ListView, TabLayout) in the view hierarchy
map to the corresponding perceptual groups. In our dataset, specific layout
classes are generalized to blocks, as we only care about generic perceptual
groups in this work. Following the work (Chen et al., 2018) for generating GUI
component hierarchy from UI image, we adopt the sequence representation of a
GUI component hierarchy. Through depth-first traversal, a view hierarchy can
be transformed into a string of GUI widget names and brackets (“[]” and “()”
corresponding to the blocks). This string represents the ground-truth
perceptual groups of an app GUI. As discussed in Section 2.1, perceptual
grouping is based on the widgets’ positional and visual properties while the
actual classes of non-text widgets are not necessary. Thus, the ground-truth
string only has two widget types: Text and Non-text. Specifically, it converts
TextView in the view hierarchy to text and all other classes as Non-text.
Similarly, as shown in the right part of Figure 7, the designers organize
texts and non-text widgets (images or shape compositions referred to as
frames) into a hierarchy of groups. Based on the design’s group hierarchy, we
output the ground-truth string of perceptual groups. The perceptual groups
“perceived” by our approach are output in the same format for comparison.

We compute the Levenshtein edit distance between the two strings of a ground-
truth block and a perceived block. The Levenshtein edits inform us of the
specific mismatches between the two blocks, which is important to understand
and analyze grouping mistakes. If the edit distance between the ground-truth
block and the perceived block is less than a threshold, we regard the two
blocks as a candidate match. We determine the optimal matching between the
string of ground-truth blocks and the string of the perceived blocks by
minimizing the overall edit distance among all candidate matches. If a
perceived group matches a ground-truth group, it is a true positive (TP),
otherwise a false positive (FP). If a ground-truth group does not match any
perceived group, it is a false negative (FN). Based on the matching results,
we compute: (1) precision (TP/(TP+FP)) (2) recall (TP/(TP+FN)), and (3)
F1-score ((2*precision*recall) / (precision+recall))

####  4.2.3. Performance on Android App GUIs

Figure 8. Performance at different edit distance thresholds

We experiment with five edit distance thresholds (0-4). The distance 0 means
the two blocks have the perfect match, and the distance 4 means as long as the
unmatched widgets in the two blocks are no more than 4, the two blocks can be
regarded as a candidate match. As shown in Figure 8, for detection-based
grouping, the precision, recall and F1-score is 0.437, 0.520 and 0.475 at the
distance 0. As the distance threshold increases (i.e., the matching criterion
relaxes), the precision, recall and F1-score keeps increasing to 0.652, 0.776
and 0.709 at the distance threshold 4. As shown in Figure 8 and Table 2, the
metadata-based grouping with the ground-truth GUI widgets achieves a
noticeable improvement over the detection-based grouping with the detected
widgets, in terms of all three metrics, especially for recall. This suggests
that improving GUI widget detection will positively improve the subsequent
perceptual grouping.

As the examples in Figure 6 show, our approach can not only accurately process
GUIs with clear structures (e.g., the first row), but it can also process GUIs
with large numbers of widgets that are placed in a packed way (e.g., the
second and third rows). Furthermore, our approach is fault-tolerant to GUI
widget detection errors to a certain extent, for example, the second row of
detection-based grouping for screenshot and design. The map and the pushed-
aside partial GUI result in many inaccurately detected GUI widgets in these
two cases. However, our approach still robustly recognizes the proper
perceptual groups.

We compare our approach with the heuristic-based grouping method (Screen
Recognition) proposed in Zhang et.al. (Zhang et al., 2021) (which received the
distinguished paper award at CHI2021). The results in Table 2 shows that
Screen Recognition can hardly handle visually and structurally complicated
GUIs based on a few ad-hoc and rigid heuristics. Its F1 score is only 0.092 on
the detected widgets and 0.123 on the ground-truth widgets. This is because
its heuristics are designed for only some fixed grouping styles such as cards
and multi-tabs. In contrast, our approach is designed to fulfil generic
Gestalt principles of grouping.

Table 2. Performance comparison (edit distance$\leq$1)

Widgets | Approach | #Bock | Precision | Recall | F1
---|---|---|---|---|---
Metadata | Our Approach | 1,465 | 0.607 | 0.754 | 0.672
Screen Recog | 1,038 | 0.131 | 0.116 | 0.123
Detection | Our Approach | 1,260 | 0.546 | 0.650 | 0.593
Screen Recog | 992 | 0.103 | 0.083 | 0.092

Figure 9. Typical causes of grouping mistakes (red box - text widget, green
box - non-text widget, pink box - perceptual group, red dashed box - unmatched
ground-truth widget)

We manually inspect the grouping results by our approach against the ground-
truth groups to identify the potential improvements. Figure 9 presents four
typical cases that cause the perceived groups to be regarded as incorrect. For
the detection-based grouping, the major issue is GUI widget over-segmentation
(a widget is detected as several widgets) or under-segmentation (several
widgets are detected as one widget). In the first row, the detector segments
the texts on the right side of the GUI into several text and non-text widgets.
As indicated by the same color in the Grouping Result column, our approach
still successfully partitions the widgets on the same row into a block, and
recognizes the large group containing these row blocks. But as shown in the
Group Comparison column, one widget in the second, third and fourth detected
blocks do not match those in the corresponding ground-truth blocks. In the
second row, the GUI widget detector merges close-by multi-line texts as a
single text widget, while these text widgets are separate widgets in the
ground truth. Again, our approach recognizes the overall perceptual groups
correctly, but the widgets in the corresponding blocks do not completely
match.

While using the ground-truth widgets from the GUI metadata to mitigate the GUI
widget misdetection, the grouping results see the improvement but suffer from
two other problems. First, the widgets in the metadata contain some widgets
that are visually occluded or hidden. The third row in Figure 9 illustrates
this problem, where some widgets are actually occluded behind the menu on the
left, but they are still available in the runtime metadata and are extracted
as the ground-truth widgets. This results in a completely incorrect grouping.
The issue of widget occlusion or modal window could be mitigated as follows:
train an image classifier to predict the presence of widget occlusion or modal
window, then follow the figure-ground principle (gel, [n.d.]) to separate
foreground modal window from the background, and finally detect the perceptual
groups on the separated model window. Second, alternative ways exist to
partition the widgets into groups. For example, for the GUI in the fourth row,
the ground truth contains eight blocks, each of which has one image and one
text while our grouping approach partitions these blocks into four rows of a
large group, and each row contains two blocks (as indicated by the same color
in Grouping Result). Perceptually, both ways are acceptable but the group
differences cause the grouping result by our approach to be regarded as
incorrect.

####  4.2.4. Performance on UI Design Prototypes

Tested on the 20 UI design prototypes, our approach achieves the precision of
0.750, the recall 0.818 and the F1-score 0.783. The third column in Figure 6
shows some results of our grouping approach for the UI design prototypes,
where we see it is able to infer the widget groups well for different styles
of GUI designs. GUI widget detection is more robust on UI design prototypes
due to the more accurate GUI widget detection, which leads to the improvement
of the subsequent grouping of detected widgets. As shown in Figure 6, the
widgets in a UI design prototype is usually scattered, while the real app GUIs
are packed. Both GUI widget detection and perceptual grouping become
relatively easier on less packed GUIs.

####  4.2.5. Processing Time

As the GUI widget grouping can be used as a part of various automation tasks
such as automated testing, the runtime performance can be a concern. We record
the processing time while running our approach over the dataset to get a sense
of its efficiency. Our experiments run on a machine with Windows 10 OS, Intel
i7-7700HQ CPU, and 8GB memory. Our approach comprises two major steps: widget
detection and perceptual grouping. We improved and refactored the original
UIED to boost the run performance of the widget detection, and now it takes an
average of 1.1s to detect the widgets in a GUI, which significantly exceeds
the original UIED that takes on average 9s per GUI. The grouping process is
also efficient, which takes an average of 0.6s to process a GUI. In total, the
average processing time of the entire approach is 1.7s per GUI image.
Furthermore, as our approach does not involve any deep learning techniques, it
does not require advanced computing support such as GPU.

##  5\. Related Work

Our work falls into the area of reverse-engineering the hidden attributes of
GUIs from pixels. There are two lines of closely related work: GUI widget
detection and GUI-to-text/code generation.

GUI widget detection is a special case of object detection (Uijlings et al.,
2013; Girshick, 2015; Redmon and Farhadi, 2018). Earlier work (Nguyen and
Csallner, 2015) uses classic computer vision (CV) algorithms (e.g., Canny edge
and contour analysis) to detect GUI widgets. Recently, White et al. (White et
al., 2019) apply a popular object detection model YOLOv2 (Redmon and Farhadi,
2018) to detect GUI widgets in GUI images for random GUI testing. Feng et al.
(Chen et al., 2019) apply Faster RCNN (Girshick, 2015) to obtain GUI widgets
from app screenshots and construct a searchable GUI widget gallery. Chen et
al. (Chen et al., 2022b) proposed an approach to complete icon labeling in
mobile applications. A recent study by Xie et al. (Chen et al., 2020d) shows
that both classic CV algorithms and recent deep learning models have
limitations when applied to GUI widget detection, which has different visual
characteristics and detection goals from natural scene object detection. They
design a hybrid method UIED inspired by the figure-ground (gel, [n.d.])
characteristic of GUI, which achieves the start-of-the-art performance for GUI
widget detection.

GUI-to-text/code generation also receives much attention. To improve GUI
accessibility, Chen et al. (Chen et al., 2020b) propose a transformer-based
image captioning model for producing labels for icons. To implement GUI view
hierarchy, REMAUI (Nguyen and Csallner, 2015) infers three Android-specific
layouts (LinearLayout, FrameLayout and ListView) based on hand-craft rules to
group widgets. Recently, Screen Recognition (Zhang et al., 2021) develops some
heuristics for inferring tabs and bars. However, these heuristic-based widget
grouping methods cannot handle visually and structurally complicated GUI
designs (e.g., nested perceptual groups like a grid of cards). Alternatively,
image captioning models (Vinyals et al., 2015; Lecun et al., 2015) have been
used to generate GUI view hierarchy from GUI images (Beltramelli, 2018; Chen
et al., 2018). Although these image-captioning based methods get rid of hard-
coded heuristics, they suffer from GUI data availability and quality issues
(as discussed in Introduction and illustrated in Figure 2). These methods also
suffer from code redundancy and no explicit image-code traceability issues
(see Section 6.2). The perceptual groups recognized by our approach could help
to address these issues.

None of the existing GUI widget detection and GUI-to-code approaches solve the
perceptual grouping problem in a systematic way as our approach does. ReDraw
(Moran et al., 2020) and FaceOff (Zheng et al., 2019) solves the layout
problem by finding in the codebase the layouts containing similar GUI widgets.
Some other methods rely on source code or specific layout algorithm (e.g.,
Android RelativeLayout) to synthesize modular GUI code or layout (Bielik et
al., 2018; Bajammal et al., 2018) or infer GUI duplication (Yandrapally et
al., 2020). All these methods are GUI implementation-oriented, and hard to
generalize for other application scenarios such as UI design search, UI
automation, robotic GUI testing or accessibility enhancement. In contrast, our
approach is based on domain-independent Gestalt principles and is application-
independent, so it can support different downstream SE tasks (see Section 6).

In the computer vision community, some machine learning techniques (Zellers et
al., 2018; Knyazev et al., 2020; Yang et al., 2018) have been proposed to
predict structure in the visual scene, i.e., so-called scene graphs. These
techniques can infer the relationships between objects detected in an image
and describe these relationships by triplets (¡subject, relation, object¿).
However, such relationship triplets cannot represent complex GUI widget
relations in perceptual groups. Furthermore, these techniques also require
sufficient high-quality data for model training, which is a challenging issue
for GUIs.

##  6\. Perceptual Grouping Applications

Our perceptual grouping method fills in an important gap for automatic UI
understanding. Perceptual groups, together with elementary widget information,
would open the door to some innovative applications in software engineering
domain.

###  6.1. UI Design Search

UI design is a highly creative activity. The proliferation of UI design data
on the Internet enables data-driven methods to learn UI designs and obtain
design inspirations (Deka et al., 2017; Liu et al., 2018; Chen et al., 2020c).
However, this demands effective UI design search engines. Existing methods
often rely on the GUI metadata, which limits their applicability as most GUI
designs exist in only pixel format. GalleryDC (Chen et al., 2019) builds a
gallery of GUI widgets and infer elementary widget information (e.g., size,
primary color) to help widget search. Unfortunately, this solution does not
apply to the whole and complex UIs. Chen et al. (Chen et al., 2020a) and Rico
(Liu et al., 2018) use image auto-encoder to extract image features through
self-supervised learning, which can be used to find visually similar GUI
images. However, the image auto-encoder encodes only pixel-level features, but
is unaware of GUI structure, which is very critical to model and understand
GUIs. As such, given the GUI in the left of Figure 2, these auto-encoder based
methods may return a GUI like the one on the right of Figure 2, because both
GUIs have rich graphic features and some textural features. Unfortunately,
such search results are meaningless, because they bear no similarity in terms
of GUI structure and perceptual groups of GUI widgets. Our approach can
accurately infer perceptual groups of GUI widgets from pixels. Based on its
perceptual grouping results, a UI design search would become structure-aware,
and finds not only visually but also structurally similar GUIs. For example, a
structure-aware UI design search would return a GUI like the one in 2nd-
row-1st-column of Figure 6 for the left GUI in Figure 2.

###  6.2. Modular GUI-to-Code Generation

Existing methods for GUI-to-code generation either use hand-craft rules or
specific layout algorithms to infer some specific implementation layout
(Nguyen and Csallner, 2015; Bielik et al., 2018), or assume the availability
of a codebase to search layout implementations (Moran et al., 2020; Zheng et
al., 2019). Image-captioning based GUI-to-Code methods (Beltramelli, 2018;
Chen et al., 2018; Feng et al., 2021, 2022) are more flexible as they learn
how to generate GUI view hierarchy from GUI metadata (if available). However,
the nature of image captioning is to just describe the image content, but it
is completely unaware of GUI structure during the code generation. As such,
the generated GUI code is highly redundant for repetitive GUI blocks. For
example, for the card-based GUI design in Figure 1(a), it will generate eight
pieces of repetitive code, one for each of the eight cards. This type of
generated code is nothing like the modular GUI code developers write. So it
has little practicality. Another significant limitation of image captioning is
that the generated GUI layouts and widgets have no connection to the
corresponding parts in the GUI image. For a GUI with many widgets (e.g., those
in the 2nd and 3rd rows in Figure 6), it would be hard to understand how the
generated code implements the GUI. With the support of our perceptual
grouping, GUI-to-code generation can encapsulate the widget grouping
information into the code generation process and produce much less redundant
and more modular, reusable GUI code (e.g., extensible card component).

###  6.3. UI Automation

Automating UI understanding from pixels can support many UI automation tasks.
A particular application of UI automation in software engineering is automatic
GUI testing. Most existing methods for automatic GUI testing rely on OS or
debugging infrastructure (Guo et al., 2019; Sahin et al., 2019; and, [n.d.];
UIa, 2021). In recent years, computer vision methods have also been used to
support non-intrusive GUI testing (White et al., 2019; Qian et al., 2020; Feng
and Chen, 2022b, a). However, these methods only work at the GUI widget level
through either traditional widget detection (Nguyen and Csallner, 2015) or
deep learning models like Yolo (Redmon and Farhadi, 2018). Furthermore, they
only support random testing, i.e., random interactions with some widgets. Some
studies (Degott et al., 2019; YazdaniBanafsheDaragh, [n.d.]; Chen et al.,
2022a) show that GUI testing would be more effective if the testing methods
were aware of more likely interactions. They propose deep learning methods to
predict such likely interactions. However, the learning is a completely black
box. That is, they can predict where on the GUI some actions could be applied,
but they do not know what will be operated and why so. Our approach can inform
the learning with higher-order perceptual groups of GUI widgets so that the
model could make an explainable prediction, for example, scrolling is
appropriate because this part of GUI displays a list of repetitive blocks. It
may also guide the testing methods to interact with the blocks in a perceptual
group in an orderly manner, and ensure all blocks are tested without
unnecessary repetitions. Such support for UI automation would also enhance the
effectiveness of screen readers which currently heavily rely on accessibility
metadata and use mostly elementary widget information.

##  7\. Conclusion and Future Work

This paper presents a novel approach for recognizing perceptual groups of GUI
widgets in GUI images. The approach is designed around the four psychological
principles of grouping - connectedness, similarity, proximity and continuity.
To the best of our knowledge, this is the first unsupervised, automatic UI
understanding approach with a systematic theoretical foundation, rather than
relying on ad-hoc heuristics or model training with GUI metadata. Through the
evaluation of both mobile app GUIs and UI design prototypes, we confirm the
high accuracy of our perceptual grouping method for visually and structurally
diverse GUIs. Our approach fills the gap of visual intelligence between the
current widget-level detection and the whole-UI level GUI-to-code generation.
As a pixel-only and application-independent approach, we envision our approach
could enhance many downstream software engineering tasks with the visual
understanding of GUI structure and perceptual groups, such as structure-aware
UI design search, modular and reusable GUI-to-code generation, and layout-
sensitive UI automation for GUI testing and screen reader. Although our
current approach achieves very promising performance, it can be further
improved by dealing with widget occlusion or modal window. Moreover, we will
investigate semantic grouping that aims to recognize both interaction and
content semantics of perceptual groups.

## Acknowledgements

This research was partially funded by Data61-ANU Collaborative Research
Project. Chunyang is partially supported by Facebook/Meta Gift grant.

## References

  * (1)
  * gel ([n.d.]) [n.d.].  7 Gestalt Principles of Visual Perception: Cognitive Psychology for UX: UserTesting Blog.  https://www.usertesting.com/blog/gestalt-principles#figure
  * wix ([n.d.]) [n.d.].  Free website builder: Create a free website.  http://www.wix.com/
  * goo ([n.d.]a) [n.d.]a.  Get started on Android with TalkBack - Android Accessibility Help.  https://support.google.com/accessibility/android/answer/6283677?hl=en
  * fig ([n.d.]) [n.d.].  the collaborative interface design tool.  https://www.figma.com/
  * and ([n.d.]) [n.d.].  UI/Application Exerciser Monkey : Android Developers.  https://developer.android.com/studio/test/monkey#:~:text=TheMonkeyisaprogram,arandomyetrepeatablemanner.
  * goo ([n.d.]b) [n.d.]b.  Vision AI — Derive Image Insights via ML — Cloud Vision API.  https://cloud.google.com/vision/
  * ges (2006) 2006\.  _”Gestalt psychology”. Britannica concise encyclopedia_.  Britannica Digital Learning.
  * app (2021) 2021\.  Accessibility - Vision.  https://www.apple.com/accessibility/vision/
  * wik (2021) 2021\.  Gestalt psychology.  https://en.wikipedia.org/wiki/Gestalt_psychology#cite_note-1
  * UIa (2021) UIAutomator, 2021.  https://developer.android.com/training/testing/ui-automator.
  * Bajammal et al. (2018) Mohammad Bajammal, Davood Mazinanian, and Ali Mesbah. 2018\.  Generating Reusable Web Components from Mockups. In _Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering_ (Montpellier, France) _(ASE 2018)_. New York, NY, USA, 601–611.  https://doi.org/10.1145/3238147.3238194
  * Beltramelli (2018) Tony Beltramelli. 2018\.  pix2code: Generating code from a graphical user interface screenshot. In _Proceedings of the ACM SIGCHI Symposium on Engineering Interactive Computing Systems_. 1–6.
  * Bielik et al. (2018) Pavol Bielik, Marc Fischer, and Martin Vechev. 2018\.  Robust Relational Layout Synthesis from Examples for Android.  _Proc. ACM Program. Lang._ 2, OOPSLA, Article 156 (Oct. 2018), 29 pages.  https://doi.org/10.1145/3276526
  * Chen et al. (2020c) Chunyang Chen, Sidong Feng, Zhengyang Liu, Zhenchang Xing, and Shengdong Zhao. 2020c.  From lost to found: Discover missing ui design semantics through recovering missing tags.  _Proceedings of the ACM on Human-Computer Interaction_ 4, CSCW2 (2020), 1–22.
  * Chen et al. (2019) Chunyang Chen, Sidong Feng, Zhenchang Xing, Linda Liu, Shengdong Zhao, and Jinshui Wang. 2019\.  Gallery D.C.: Design Search and Knowledge Discovery through Auto-Created GUI Component Gallery.  _Proc. ACM Hum.-Comput. Interact._ 3, CSCW, Article 180 (Nov. 2019), 22 pages.  https://doi.org/10.1145/3359282
  * Chen et al. (2018) Chunyang Chen, Ting Su, Guozhu Meng, Zhenchang Xing, and Yang Liu. 2018.  From UI Design Image to GUI Skeleton: A Neural Machine Translator to Bootstrap Mobile GUI Implementation. In _Proceedings of the 40th International Conference on Software Engineering_ (Gothenburg, Sweden) _(ICSE ’18)_. Association for Computing Machinery, New York, NY, USA, 665–676.  https://doi.org/10.1145/3180155.3180240
  * Chen et al. (2020a) Jieshan Chen, Chunyang Chen, Zhenchang Xing, Xin Xia, Liming Zhu, John Grundy, and Jinshui Wang. 2020a.  Wireframe-based UI Design Search through Image Autoencoder.  _ACM Transactions on Software Engineering and Methodology_ 29, 3 (Jul 2020), 1–31.  https://doi.org/10.1145/3391613
  * Chen et al. (2020b) Jieshan Chen, Chunyang Chen, Zhenchang Xing, Xiwei Xu, Liming Zhut, Guoqiang Li, and Jinshui Wang. 2020b.  Unblind Your Apps: Predicting Natural-Language Labels for Mobile GUI Components by Deep Learning. In _2020 IEEE/ACM 42nd International Conference on Software Engineering (ICSE)_. 322–334.
  * Chen et al. (2022a) Jieshan Chen, Amanda Swearngin, Jason Wu, Titus Barik, Jeffrey Nichols, and Xiaoyi Zhang. 2022a.  Extracting Replayable Interactions from Videos of Mobile App Usage.  _arXiv preprint arXiv:2207.04165_ (2022).
  * Chen et al. (2022b) Jieshan Chen, Amanda Swearngin, Jason Wu, Titus Barik, Jeffrey Nichols, and Xiaoyi Zhang. 2022b.  Towards Complete Icon Labeling in Mobile Applications. In _CHI Conference on Human Factors in Computing Systems_. 1–14.
  * Chen et al. (2020d) Jieshan Chen, Mulong Xie, Zhenchang Xing, Chunyang Chen, Xiwei Xu, Liming Zhu, and Guoqiang Li. 2020d.  Object detection for graphical user interface: old fashioned or deep learning or a combination?  _Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering_ (Nov 2020).  https://doi.org/10.1145/3368089.3409691
  * Degott et al. (2019) Christian Degott, Nataniel P. Borges Jr., and Andreas Zeller. 2019.  Learning User Interface Element Interactions. In _Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis_ (Beijing, China) _(ISSTA 2019)_. Association for Computing Machinery, New York, NY, USA, 296–306.  https://doi.org/10.1145/3293882.3330569
  * Deka et al. (2017) Biplab Deka, Zifeng Huang, Chad Franzen, Joshua Hibschman, Daniel Afergan, Yang Li, Jeffrey Nichols, and Ranjitha Kumar. 2017\.  Rico: A Mobile App Dataset for Building Data-Driven Design Applications. In _Proceedings of the 30th Annual Symposium on User Interface Software and Technology_ _(UIST ’17)_.
  * Dribbble ([n.d.]) Dribbble. [n.d.].  Discover the world’s Top Designers & Creatives.  https://dribbble.com/
  * Ester et al. (1996) Martin Ester, Hans-Peter Kriegel, Jörg Sander, and Xiaowei Xu. 1996.  A Density-Based Algorithm for Discovering Clusters in Large Spatial Databases with Noise. In _Proceedings of the Second International Conference on Knowledge Discovery and Data Mining_ (Portland, Oregon) _(KDD’96)_. AAAI Press, 226–231.
  * Eysenck and Brysbaert (2018) Michael W. Eysenck and Marc Brysbaert. 2018.  Fundamentals of Cognition.  (2018).  https://doi.org/10.4324/9781315617633
  * Feng and Chen (2022a) Sidong Feng and Chunyang Chen. 2022a.  GIFdroid: An Automated Light-weight Tool for Replaying Visual Bug Reports.  (2022).
  * Feng and Chen (2022b) Sidong Feng and Chunyang Chen. 2022b.  GIFdroid: automated replay of visual bug reports for Android apps. In _Proceedings of the 44th International Conference on Software Engineering_. 1045–1057.
  * Feng et al. (2022) Sidong Feng, Minmin Jiang, Tingting Zhou, Yankun Zhen, and Chunyang Chen. 2022.  Auto-Icon+: An Automated End-to-End Code Generation Tool for Icon Designs in UI Development.  _ACM Transactions on Interactive Intelligent Systems (TiiS)_ (2022).
  * Feng et al. (2021) Sidong Feng, Suyu Ma, Jinzhong Yu, Chunyang Chen, TingTing Zhou, and Yankun Zhen. 2021\.  Auto-icon: An automated code generation tool for icon designs assisting in ui development. In _26th International Conference on Intelligent User Interfaces_. 59–69.
  * Girshick (2015) Ross Girshick. 2015\.  Fast R-CNN. In _The IEEE International Conference on Computer Vision (ICCV)_.
  * Guo et al. (2019) Jiaqi Guo, Shuyue Li, Jian-Guang Lou, Zijiang Yang, and Ting Liu. 2019.  SARA: self-replay augmented record and replay for Android in industrial cases. In _Proceedings of the 28th acm sigsoft international symposium on software testing and analysis_. 90–100.
  * Knyazev et al. (2020) Boris Knyazev, Harm de Vries, Cătălina Cangea, Graham W. Taylor, Aaron Courville, and Eugene Belilovsky. 2020.  Graph Density-Aware Losses for Novel Compositions in Scene Graph Generation.  arXiv:2005.08230 [cs.CV]
  * Lecun et al. (2015) Yann Lecun, Yoshua Bengio, and Geoffrey Hinton. 2015\.  Deep learning.  _Nature_ 521, 7553 (2015), 436–444.  https://doi.org/10.1038/nature14539
  * Li et al. (2019) Yuanchun Li, Ziyue Yang, Yao Guo, and Xiangqun Chen. 2019\.  Humanoid: A Deep Learning-Based Approach to Automated Black-box Android App Testing. In _2019 34th IEEE/ACM International Conference on Automated Software Engineering (ASE)_. 1070–1073.  https://doi.org/10.1109/ASE.2019.00104
  * Liu et al. (2018) Thomas F. Liu, Mark Craft, Jason Situ, Ersin Yumer, Radomir Mech, and Ranjitha Kumar. 2018\.  Learning Design Semantics for Mobile Apps. In _The 31st Annual ACM Symposium on User Interface Software and Technology_ (Berlin, Germany) _(UIST ’18)_. ACM, New York, NY, USA, 569–579.  https://doi.org/10.1145/3242587.3242650
  * Moran et al. (2020) Kevin Moran, Carlos Bernal-Cárdenas, Michael Curcio, Richard Bonett, and Denys Poshyvanyk. 2020\.  Machine Learning-Based Prototyping of Graphical User Interfaces for Mobile Apps.  _IEEE Transactions on Software Engineering_ 46, 2 (2020), 196–221.  https://doi.org/10.1109/TSE.2018.2844788
  * Nguyen and Csallner (2015) Tuan Anh Nguyen and Christoph Csallner. 2015.  Reverse Engineering Mobile Application User Interfaces with REMAUI. In _Proceedings of the 30th IEEE/ACM International Conference on Automated Software Engineering_ (Lincoln, Nebraska) _(ASE ’15)_. IEEE Press, 248–259.  https://doi.org/10.1109/ASE.2015.32
  * Qian et al. (2020) Ju Qian, Zhengyu Shang, Shuoyan Yan, Yan Wang, and Lin Chen. 2020.  RoScript: A Visual Script Driven Truly Non-Intrusive Robotic Testing System for Touch Screen Applications. In _Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering_ (Seoul, South Korea) _(ICSE ’20)_. Association for Computing Machinery, New York, NY, USA, 297–308.  https://doi.org/10.1145/3377811.3380431
  * Redmon and Farhadi (2018) Joseph Redmon and Ali Farhadi. 2018.  YOLOv3: An Incremental Improvement.  _CoRR_ abs/1804.02767 (2018).  arXiv:1804.02767 http://arxiv.org/abs/1804.02767
  * Rutledge (2009) Andy Rutledge. 2009\.  Gestalt Principles of Perception - 3: Proximity, Uniform Connectedness, and Good Continuation.  http://andyrutledge.com/gestalt-principles-3.html
  * S1T2 (2021) S1T2. 2021.  Apply crap to design: S1T2 blog.  https://s1t2.com/blog/step-1-generously-apply-crap-to-design
  * Sahin et al. (2019) Onur Sahin, Assel Aliyeva, Hariharan Mathavan, Ayse Coskun, and Manuel Egele. 2019\.  Randr: Record and replay for android applications via targeted runtime instrumentation. In _2019 34th IEEE/ACM International Conference on Automated Software Engineering (ASE)_. IEEE, 128–138.
  * Sketch ([n.d.]) Sketch. [n.d.].  https://www.sketch.com/
  * Sternberg and Robert (2003) Sternberg and Robert. 2003\.  _Cognitive Psychology Third Edition_.  Thomson Wadsworth.
  * Stevenson ([n.d.]) Herb Stevenson. [n.d.].  Emergence: The Gestalt Approach to Change.  http://www.clevelandconsultinggroup.com/articles/emergence-gestalt-approach-to-change.php
  * Tesseract-Ocr ([n.d.]) Tesseract-Ocr. [n.d.].  tesseract-ocr/tesseract: Tesseract Open Source OCR Engine (main repository).  https://github.com/tesseract-ocr/tesseract
  * Thalion (2020) Thalion. 2020.  Ui Design in practice: Gestalt principles.  https://uxmisfit.com/2019/04/23/ui-design-in-practice-gestalt-principles/
  * Uijlings et al. (2013) J. R. R. Uijlings, K. E. A. van de Sande, T. Gevers, and A. W. M. Smeulders. 2013.  Selective Search for Object Recognition.  _International Journal of Computer Vision_ 104, 2 (01 Sep 2013), 154–171.  https://doi.org/10.1007/s11263-013-0620-5
  * UserTesting (2019) UserTesting. 2019\.  7 Gestalt Principles of Visual Perception: Cognitive Psychology for UX: UserTesting Blog.  https://www.usertesting.com/blog/gestalt-principles#proximity
  * Vinyals et al. (2015) Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan. 2015.  Show and Tell: A Neural Image Caption Generator.  arXiv:1411.4555 [cs.CV]
  * White et al. (2019) Thomas D. White, Gordon Fraser, and Guy J. Brown. 2019\.  Improving Random GUI Testing with Image-based Widget Detection. In _Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis_ (Beijing, China) _(ISSTA 2019)_. ACM, New York, NY, USA, 307–317.  https://doi.org/10.1145/3293882.3330551
  * Xie et al. (2020) Mulong Xie, Sidong Feng, Zhenchang Xing, Jieshan Chen, and Chunyang Chen. 2020.  UIED: a hybrid tool for GUI element detection. In _Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering_. 1655–1659.
  * Yandrapally et al. (2020) Rahulkrishna Yandrapally, Andrea Stocco, and Ali Mesbah. 2020\.  Near-Duplicate Detection in Web App Model Inference. In _Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering_ (Seoul, South Korea) _(ICSE ’20)_. Association for Computing Machinery, New York, NY, USA, 186–197.  https://doi.org/10.1145/3377811.3380416
  * Yang et al. (2018) Jianwei Yang, Jiasen Lu, Stefan Lee, Dhruv Batra, and Devi Parikh. 2018.  Graph R-CNN for Scene Graph Generation.  arXiv:1808.00191 [cs.CV]
  * YazdaniBanafsheDaragh ([n.d.]) YazdaniBanafsheDaragh. [n.d.].  Deep-GUI: Towards Platform-Independent UI Input Generation with Deep Reinforcement Learning.  _UC Irvine_ ([n. d.]).  https://escholarship.org/uc/item/3kv1n3qk
  * Zellers et al. (2018) Rowan Zellers, Mark Yatskar, Sam Thomson, and Yejin Choi. 2018\.  Neural Motifs: Scene Graph Parsing with Global Context.  arXiv:1711.06640 [cs.CV]
  * Zhang et al. (2021) Xiaoyi Zhang, Lilian de Greef, Amanda Swearngin, Samuel White, Kyle Murray, Lisa Yu, Qi Shan, Jeffrey Nichols, Jason Wu, Chris Fleizach, Aaron Everitt, and Jeffrey P. Bigham. 2021.  Screen Recognition: Creating Accessibility Metadata for Mobile Applications from Pixels.  arXiv:2101.04893 [cs.HC]
  * Zheng et al. (2019) Shuyu Zheng, Ziniu Hu, and Yun Ma. 2019.  FaceOff: Assisting the Manifestation Design of Web Graphical User Interface. In _Proceedings of the Twelfth ACM International Conference on Web Search and Data Mining_ (Melbourne VIC, Australia) _(WSDM ’19)_. Association for Computing Machinery, New York, NY, USA, 774–777.  https://doi.org/10.1145/3289600.3290610
  * Zhou et al. (2017) Xinyu Zhou, Cong Yao, He Wen, Yuzhi Wang, Shuchang Zhou, Weiran He, and Jiajun Liang. 2017.  EAST: an efficient and accurate scene text detector. 5551–5560.