#  Auto-Icon+: An Automated End-to-End Code Generation Tool for Icon Designs
in UI Development

Sidong Feng<EMAIL_ADDRESS>Faculty of Information TechnologyMonash
UniversityMelbourneAustralia ,  Minmin Jiang  Alibaba GroupHangzhouChina
<EMAIL_ADDRESS>,  Tingting Zhou  Alibaba GroupHangzhouChina
<EMAIL_ADDRESS>,  Yankun Zhen  Alibaba GroupHangzhouChina
<EMAIL_ADDRESS>and  Chunyang Chen  Faculty of Information
TechnologyMonash UniversityMelbourneAustralia<EMAIL_ADDRESS>
###### Abstract.

Approximately 50% of development resources are devoted to UI development tasks
(Beaudouin-Lafon, 2004). Occupying a large proportion of development
resources, developing icons can be a time-consuming task, because developers
need to consider not only effective implementation methods but also easy-to-
understand descriptions. In this paper, we present Auto-Icon+, an approach for
automatically generating readable and efficient code for icons from design
artifacts. According to our interviews to understand the gap between designers
(icons are assembled from multiple components) and developers (icons as single
images), we apply a heuristic clustering algorithm to compose the components
into an icon image. We then propose an approach based on a deep learning model
and computer vision methods to convert the composed icon image to fonts with
descriptive labels, thereby reducing the laborious manual effort for
developers and facilitating UI development. We quantitatively evaluate the
quality of our method in the real world UI development environment and
demonstrate that our method offers developers accurate, efficient, readable,
and usable code for icon designs, in terms of saving 65.2% implementing time.

code accessibility, icon implementation, neural networks

††copyright: acmcopyright††journal: TIIS††journalyear: 2022††journalvolume:
1††journalnumber: 1††article: 1††publicationmonth: 1††price: 15.00††doi:
10.1145/3531065††ccs: Human-centered computing Empirical studies in
accessibility

##  1\. Introduction

A user interface (UI) consists of series of elements, such as text, colors,
images, widgets, etc. Designers are constantly focusing on icons as they are
highly functional in a user interface (Koutsourelakis and Chorianopoulos,
2010; Black, 2017; Islam, 2015; Malik et al., 2017). One of the biggest
benefits of icons is that they can be universal. For instance, by adding a red
“X” icon to your user interface design, users are informed that clicking this
icon leads to the closure of a component. Furthermore, icons can make UIs look
more engaging. For example, instead of using basic bullets or drop-downs
filled with words, a themed group of icons can capture instant attention from
users. Consequently, icons become an elegant yet efficient way to communicate
with and help guide user through experience.

Despite of all these benefits, icons have three fundamental limitations in the
day-to-day development environment, in terms of transition gap, rendering
speed and code accessibility. First, it is a challenging task for developers
to implement icons from the design artifacts as many designers follow their
own design styles which are highly differ from each other. For example, some
of them design an icon with many small components, though just one combined
image in the icon implementation for optimizing the network traffic and
caching. Such gap adds the complexity and effort for developers to preprocess
the design draft before icon implementation. Second, to ensure a smooth user
interaction, UI should be rendered in under 16ms (Gómez et al., 2016;
Developers, 2020a, b), while icon implemented as an image faces the slow
rendering problem, due to image download speed, image loading efficiency, etc.
These issues will directly affect the quality of the product and user
experience, requiring more effort from developers to develop an advanced
method to overcome the problem. Third, in the process of UI implementation,
many developers directly import the icon resources from the design artifacts
without considering the meaning of the content, resulting in poor
description/comment during coding. Different codes render the same visual
effect to users, while it is different for developers to develop and maintain.
A non-descriptive code increases the complexity and effort required to
developers as they need to look at the associated location of the UI to
understand the meaning of the code.

This challenge motivates us to develop a proactive tool to address the
existing UI development limitations and improve the efficiency and
accessibility of code. Our tool, Auto-Icon+ involves four main features.
First, to bridge the conceptual gap between icon design and icon
implementation, we propose a heuristic machine learning technique to automated
agglomerate the scattered icon components into clusters. The component
clusters can then be composed to icon images, which can be applied in various
downstream tasks related to improving the rendering efficiency (i.e., icon
font, css clipping) and code accessibility (i.e., description, color). Second,
to meet the requirement of efficient rendering, we develop an automated
technique to convert icon image to icon font, which is a typeface font. Once
the font is loaded, the icon will be rendered immediately without downloading
the image resources, thereby reducing HTTP requests and improving the
rendering speed. Icon font can further optimize the performance of rendering
by adopting HTML5 offline storage. Besides, icon font has other potential
attributes that can facilitate UI development, such as easy to use (i.e., use
the CSS’s @fontface attribute to load the font), flexible (i.e., capable to
change color, lossless scale), etc. Third, understanding the meaning of icons
is a challenging problem. There are numerous types of icons in the UIs. Icons
representing the same meaning can have different styles and can be presented
in different scales as shown in Table 1. Also, icons are often not co-located
with texts explaining their meaning, making it difficult to understand from
the context. In order to offer an easy access for developers to develop
through understanding the meaning of icons, we collect 100k icons from
existing icon sharing website Alibaba Iconfont (Alibaba, 2020a) \- each
associating with a label described by designer. By analyzing the icons and
labels, we construct 100 categories, such as ”left”, ”pay”, ”calendar”,
”house”, etc. We then train a deep learning classification model to predict
the category of the icon as its description. The experiments demonstrate that
our model with the average accuracy as 0.87 in an efficient classification
speed as 17.48ms, outperforms the other deep learning based models and
computer vision based methods. Third, to provide more accessibility to
developers on the description of icon images, we also detect the primary color
of icons by adopting HSV color space (Sural et al., 2002). We refer to our
mechanism tool Auto-Icon+ to build an intelligent support for developers in
the real context of UI development, assisting developing standardized and
efficient code.

To demonstrate the usefulness of Auto-Icon+, we carry out an user study to
show if our tool for automatically converting an icon to a font with label
descriptions can help provide more knowledge on code accessibility and
accelerate UI development for developers. After analyzing ten professional
developers’ feedback with all positive responses on our mechanism tool and we
find that the code for icon generated by our tool can achieve better
readability compared with the code manually written by professional
developers. Besides, Auto-Icon+ has been implemented and deployed in Alibaba
Imgcook platform. The results demonstrates that our tool provides 84% usable
code for icon designs in a realistic development situations. Our contributions
can be summarized below:

  * •

We identify the fundamental limitations of existing UI development of icon
images. The informal interviews with professional developers also confirm
these issues qualitatively.

  * •

To bridge the gap between icon design and icon implementation, we develop a
machine-learning based technique to compose the components of icon in the
design artifact.

  * •

Based on the emerging 100 icon categories, we develop deep-learning and
computer-vision based techniques for specifically converting icon to font with
label describing its meaning and color to provide developers understand
knowledge of code.

  * •

We conduct large-scale experiments to evaluate the performance of our tool
Auto-Icon+ and shows that our tool achieves good accuracy compared with
baselines. The evaluation conducted with developers and tested on the real-
world development platform demonstrates the usefulness of our tool.

  * •

We contribute to the community by offering intelligent support for developers
to efficiently implement icon designs comply with code standardization.

##  2\. Related Works

###  2.1. Transition Gap

UI design and implementation require different mindset and expertise. The
former is performed by user experience designers and architects via design
tools (e.g., Sketch (Sketch, 2021b), Photoshop (Photoshop, 2021)), while the
latter performed by developers via development tools (e.g., Android Studio
(Studio, 2021), Xcode (Apple, 2021)). Existing researches well support these
two phases respectively (Lee et al., 2020; Yang et al., 2021; Chen et al.,
2021; Todi et al., 2021; Bunian et al., 2021; Zhang et al., 2021; Ang and Lim,
2021; Feng et al., 2021). For example, Li et al. (Li et al., 2021) provide a
holistic representation of UI to help designers quickly build up a realistic
understanding of the design space for an app feature and get design
inspirations from existing UI designs. But few of them support effective
transition from UI design artifacts to UI implementation.

Supporting this transition is challenging due to the gap between designers and
developers and the complexity of design artifacts (see Fig 1). Some researches
lower the transition gap by translating UI screenshots into UI implementation
(Chen et al., 2018a; Moran et al., 2018a; Microsoft, 2021; Airbnb, 2021; Chen
et al., 2019a; Wu et al., 2021). Nguyn et al. (Nguyen and Csallner, 2015)
follow the mobile specific heuristics and adopt hybrid method based on Optical
Character Recognition (OCR) and image processing to generate a static code
from UI screenshots. Pix2Code (Beltramelli, 2018) adopts an iterative
encoder/decoder deep learning model consisting of a CNN for extracting UI
screenshots features, and a LSTM decoder for generating the UI code tokens.
However, these works are unclear if the UI implementations are realistic or
useful from a developer’s point of view, as the implementation of the
approaches are only validated on a small set of synthetically UI screenshots.
It is difficult to judge how well the approaches would perform on real UI
screenshots. In contrast, Auto-Icon+ is tailored for the design artifact
driven development practice, and is aimed at solving real-world UI development
challenges, such as developers requiring manual effort to compose the icon
from scattered components in the design artifact. There has also been both
commercial and academic work related to design artifact driven development for
creating high-fidelity design artifacts (FluidUI, 2021; Mockup, 2021; Proto,
2021), deploying real-device prototype environments (Meskens et al., 2009a,
b), and verifying design violations (Moran et al., 2018b). However, such tools
and approaches tend to either impose too many restrictions on designers or do
not allow for direct creation of code, thus the icon composition problem still
persists in practice. To address this, we adopt a machine-learning approach to
automatically agglomerate the scattered icon components into clusters through
heuristic correlation coefficient to bridge the gap between icon design and
icon implementation.

###  2.2. UI Rendering

Ensuring fast rendering speed is an essential part in UI development, since
slow rendering creates poor user experience. Many studies focus on improving
rendering speed via reducing bugs (Carette et al., 2017; Li et al., 2019;
Nejati and Balasubramanian, 2016; Huang et al., 2017; Rosen et al., 2017). In
contrast, we focus on analyzing image displaying performance in UI rendering.
There are a few related works in this domain. For example, Systrace (Android,
2019) is a tool that allows developers to collect precise timing information
about UI rendering on devices. However, it does not provide any suggestions
for improvement. To address this problem, many studies introduce reliable
approaches to improve rendering efficiency such as image resizing based on
pattern-based analysis (Liu et al., 2014), a manual image resource management
based on resource leakage analysis (Wu et al., 2016). Gao et al. (Gao et al.,
2017) implement a system called DRAW which aims to reveal UI performance
problems in an application such as excessive overdraw and slow image
components detection. With the suggestion of the image displaying performance
analysis by DRAW, developers can manually improve the rendering performance of
slow image displaying. While these works offer image management suggestions to
developers to achieve better rendering performance, they still need to be
improved manually. In contrast, we propose an image conversion technology
based on computer vision and graphic algorithms to convert icons into font
types for achieving faster UI rendering.

###  2.3. Code Accessibility

Digital devices such as computer, mobile phone and tablets are widely used. To
ensure the quality of software, many research works have been conducted
(Butler, 2010; Fu et al., 2013; Liu et al., 2022). Most of these works focus
on the functionality and usability of apps such as GUI design (Chen et al.,
2019c; Feng et al., 2022; Chen et al., 2019a, 2020c; Xie et al., 2020; Liu et
al., 2020; Zhao et al., 2021), GUI animation linting (Zhao et al., 2020,
2019), localization (Wang et al., 2019; Feng and Chen, 2021; Xie et al.,
2020), privacy and security (Chen et al., 2019b, 2018b; Dehling et al., 2015;
Feng et al., 2019), and performance (Linares-Vasquez et al., 2015; Zhao et
al., 2016). Few research works are related to accessibility issues. Some works
in Human-Computer Interaction area have explored the accessibility issues of
mobile apps (Yan and Ramachandran, 2019; Wang, 2012; Mora et al., 2017; King
and Youngblood, 2016; Chen et al., 2020a). In these work, the lack of
description in image-based components in UI is commonly regarded as an
important accessibility issue. For instance, Harrison et al. (Harrison et al.,
2011) establish an initial ‘kineticon vocabulary’ containing a set of 39
kinetic behaviors for icon images, such as spin, bounce, running, etc. Ross et
al. (Ross et al., 2018) identify some common labeling issues in Android apps
via analyzing the icon image labeling. With crowd source method, Zhang et al
(Zhang et al., 2018) annotate GUI elements without content description.
However, these works still require support from developers. Due to the
increasingly developed Convolutional Neural Networks (CNNs) technologies,
dramatic advances appears in the field of image classification which is
applied to automatically annotate tags for images. Chen et al. (Chen et al.,
2020b) analyze the tags associated with the whole GUI artwork collected from
Dribbble, and emerge an vocabulary that summarizes the relationship between
the tags. Based on the vocabulary, they adopt a classification model to
recommend the general tags in the GUI, such as ”sport”, ”food”, etc. Different
from their work, we predict more fine-grained categories, such as ”football”,
”burger”, etc. And also, they focus on predicting the categories of the whole
UI which is subjective to human perception, but the categories of small icons
are usually more intuitive. A similar work to ours is the icon sensitive
classification by Xiao et al (Xiao et al., 2019). They utilize traditional
computer vision techniques like SIFT and FAST to extract the features of icons
and classify icons into 8 categories through calculating their similarity.
After the systematically investigation of icons, we discover the fundamental
limitations in icons discussed in Section 4.1, in terms of high cross-class
similarity and small, transparent and low contrast. These findings conflict
with methods applied in their paper such as applying rotation to augment the
dataset. Moreover, we show that deep learning model is fruitful for the icon
classification problem than the tradition computer vision technique in Section
6.2.3. In our work, according to the characteristic of icons, we propose a
deep learning model to automatically classify icons in a more fine-grained
(100) category and also adopt a computer vision technique to detect its
primary color.

##  3\. Preliminary Study

To better understand the challenges in the real-world UI development
environment, we conducted an interview with 12 front-end developers from the
big companies. Two authors first developed the interview protocol, and
conducted pilot studies with two participants. Based on the pilot studies, we
refined the interview protocol and conducted 10 interviews formally. The
average length of these interviews is 20 minutes. We started with general
questions, such as questions about working years, workload of development, and
number of projects developed. Then, we asked the interviewees how they
developed the code for icon images. We particularly asked what motivated them
to adopt the approach, whether they revised the implementation, what
approaches could achieve the same effect, what they perceived as the impact of
the implementation, how the implementation behaved in the process of
development and is there any difference on UI development between personal
projects and company tasks.

Figure 1. An illustration of icon design in the design artifact given by
designers. A ”gift” icon is composed by multiple components such as bitmap,
lines, and shapes (highlighted).

###  3.1. Research Question 1: do developers implement icons directly from the
design artifacts?

Once the design artifacts are completed by designers, they are handed off to
development teams who are responsible for implementing the designs in code.
Fig. 1 shows an example of the design artifact handed off to developers. The
design artifacts are stored as archives containing JSON encoded layout and a
number of binary assets such as bitmap images, curves, shapes (Sketch, 2021a).
The layout is structured as a cumulative hierarchy comprising a tree
structure, starting with a single root node, where the spatial layout of
parent always encompasses contained child components. A discrete component
with a corresponding set of attributes can be represented as a five-tuple in
the form ($<name>,<x-position,y-position>,<width,height>,<text>,<image>$).
Here the first element of the tuple describes the name of the component. The
next four elements of the tuple describe the location of the top left point
for the bounding box of the component, and the height and width attributes
describe the size of the bounding box. The text attribute corresponds to text
displayed (if any) by the component and the image attribute represents an
image (if any) of the component with bounds adhering to the position
attributes.

All of the developers demonstrated that they always preprocess the icon
designs before implementing, due to the gap to designers. As D9 said that ”We
are all fulfilling of our goals, but in different directions.” At the core of
it, designers are focused on the graphical/visual representations of icons,
while the developers take care of the functionality of icons, such as
efficient rendering. However, this gap points to a more prominent problem for
icon implementation in practice as D10 showcased an icon design in the design
artifact (as shown in Fig 1) and said:

In most big IT or software companies, the visual designers design the UI with
professional tools like Sketch (Sketch, 2021b) or Adobe Photoshop (Photoshop,
2021). These tools provide build-in curvature pen/pencil tool allow designers
to create some complicated components by drawing smooth curves and straight
line segments with equal ease in order to sketch out the desired shape. Well-
designed icons are usually composed of images, curves, lines, shapes, etc. For
example, in Fig 1, the ”gift” icon is composed by lines, shapes, and bitmaps.
Developers need to spend extra effort to compose all the small pieces into an
icon to gain icon rendering efficiency and code accessibility.

They also mentioned that finding all the components of an icon is surprisingly
non-trivial task. This is because that as designers only concern with the
visual representation of icon, they may create one icon with many components
scattered all over the design artifact. For example, the bitmap (”knot-bow”)
is a component of the ”gift” icon but not in the same hierarchy of other
components in Fig 1. D9 supported this claim that:

Although there is much information (i.e., name, attribute, hierarchical
relationship, etc.) in the design artifact, that information does not align
well with that required in the icon composition and implementation. In
practice, I potentially consider three characteristics to compose an icon.
First, the rendering of the components of icon should be close/overlap to each
other. Second, some designers may organize the resources of an icon with some
layout hierarchies to group them, for example ”group6” in Fig 1. Third, some
designers may use the same format to name icon components, for example ”gift-
piece-x”. Therefore, it takes time to search all the components with reference
to their rendering, hierarchy, and attribute.

Due to the gap between designers and developers, developers can hardly
implement icons from design artifacts directly. Given a design artifact,
developers require to manually compose the icon image by searching all the
related components based on their attributes, hierarchies and renderings,
adding to the complexity and effort required to icon implementation in
practice.

###  3.2. Research Question 2: do developers implement icons in font or
images?

By summarising the approaches, we collected 4 ways of rendering icons, i.e.,
image tag ¡img¿ or ¡svg¿, icon tag ¡i¿, css background image, and custom tag
¡SvgIcon¿ as shown in Table 4. One third of our developers listed all
approaches, 80% developers knew the way of using image and font. There are 2
developers who have never heard of or used the fonts to render icons, D2 said:

Making front-end development is fun, although sometimes it hurts because I do
not have adequate learning experience. There are few front-end courses in
universities, and these courses usually contain relatively simple knowledge,
such as what is ¡div¿ block, how to connect HTML and CSS together, etc. They
do not teach the usage of font, especially they do not distinguish the
difference between fonts and images in rendering icons.

70% developers implemented icons as image when developing front-end codes
based on UI design draft files because they found that converting icon to font
is a complicated and laborious process. For example, D7 mentioned:

To implement the approach of icon font, I first need to upload the image to
the existing conversion websites such as icomoon (IcoMoon, 2020) and Fontello
(Fontello, 2020). Then, I need to download the generated icon font to my local
device. Last but not least, I need to copy the generated CSS code to CSS
files. This entire process requires a lot of time and effort, but due to time
constraints, the process is not compatible in industry.

One developer D2 from Alibaba described how limit the time in their UI
development:

Every year, Alibaba has more than 240 events which stores offer special
discount, such as Double 11 Global Delight Event, Tmall Thanksgiving Day
Event, 1212 Global Discount Event, etc. Due to the high demand for the UI
development in the duration of events, we are required to implement UIs in 3
or 4 days.

Developers also considered the trade-off between UI performance and its value.
Since the usage of icon font does not provide business value, it is often in a
low priority in industry. Even if they knew the benefit of using font, they
would not put effort in doing this. For instance, D9 explained:

Although I know the icon font is better compared to icon image, I will not
apply this approach in development. I usually have 3 tasks in a week, such as
UI implementation, bug testing, algorithm implementation, etc. I agree that
icon font can improve UI rendering performance and provide better user
experience. But, the overall functionality will still work without icon font.
In contrast, without bug testing, the front-end codes may not work, resulting
in significantly impact on the company business. And if I do not implement the
algorithm, other developers will not be able to apply the API in their
development, which will slow down the development speed and delay the product
release time.

Another example shows the potential gap between industry and individual is
that 50% developers mentioned that they use font to render icons when
developing their personal projects, such as homepage, blog, tutorial, etc. For
example, D2 said,

When I developed my first personal website, I discovered Font Awesome
(Fonticons, 2020), a font toolkit to render icons by simply adding class
description. Since this is my website, I can design freely according to my
preferences. To quickly develop my website, I used the font in Font Awesome to
implement all the icons in my website. However, it is not applicable in
industry. In industry, every icon is well designed according to the company
culture and design specifications. Therefore, it is not suitable to apply
widely used icon font resources from online platforms. In addition, using
online icon fonts involves intellectual property (IP) issues which must be
avoided in the industry.

Despite most of the developers know the benefits of using font to render
icons, few of them implement font in practice. The icon they used is distinct
to the online resources as it comprises company culture and design guidelines.
Therefore, rather than directly using the online resources, developers have to
spend extra effort in converting icons to font, which is time-consuming and
laborious.

###  3.3. Research Question 3: do developers write descriptions for icons?

All of them mentioned that they did write descriptions/comments in their
personal projects, such as assignments, homepage, etc. However, half of
developers did not write descriptions in practice due to the following
practical reasons. First, since the readability of code is not a mandatory
requirement, many developers did not write well-formatted descriptions for
code. For example, the code in the industry cannot be released as open-source.
As D9 said that ”Since our code can not be released to public, I would not
spend too much time on writing comments in code because only a few internal
developers would collaborate on my tasks.” In addition, since updating
iteration in the industry is fast, it is not worth to put too much effort in
commenting, especially for icons. For example, D10 said, In the year of 2019,
our company developed over 1 million UIs. Due to the diversity of UIs, few
designs are re-implemented and few code are reviewed. Because of the fast
updating iteration and low reusing rate, I did not write well-formatted
comment, particularly for the images. I was developing a shopping application
which images cover more than half of the UIs. To develop the large amount of
images quickly, I prefer using ¡img¿ tag without any alternative description.

Second, 80% developers mentioned that writing a well understood description is
a challenging task. It requires developers to understand the intention of the
icons, while few developers pay attention to the content of the UIs. For
example, D7 explained,

I agree that the clear descriptions in the code can keep the code readable and
“save lives”, while unreasonable descriptions “kill lives”. However, it is
hard to write a good description. Here is the process of how I write the
descriptions: Firstly, I design a comment for every component, image, …, based
on its characteristic. Secondly, I rename and simplify the comments according
to practical requirement. Thirdly, I check if the comments match the content
of UIs or not. Then I repeat this process until the deadline. And obviously,
the process is time-consuming and not applicable in the industry.

Despite the insufficient descriptions in the code may not impede professional
developers, it creates a significant cognitive burden for interns and new
developers. For example, D3 said,

I am a junior student who came to the company for internship. The first task
assigned to me by my leader was to understand the code. However, I found that
most of the code is uncommented, which makes it very difficult for me to
understand. To understand this part of code, I asked more than 5 developers
who participated this project. These uncommented codes negatively influenced
my work.

Developers rarely write descriptions for images, especially for icons, because
the loose restriction on code readability makes developers less cared about
code descriptions. Most of developers agree with difficulty on designing
simple, concise and easy-understood descriptions. The lack of description can
adversely affect novice employees and lead to inadequate understanding of the
code.

##  4\. Icon Characteristics

Table 1. The 40 icon classes identified through an iterative open coding of 100k icons from the Iconfont (Alibaba, 2020a). CLASS |  ASSOCIATED LABEL |  EXAMPLES | NUMBER
---|---|---|---
add |  plus, addition, increase, expand, create |  | 357
calendar |  date, event, time, planning |  | 324
camera |  photo, take-photo |  | 355
chat |  chat-bubble, message, request, comment |  | 372
complete |  finish, confirm, tick, check, ok, done |  | 432
computer |  laptop, device, computer-response, desktop |  | 521
crop |  prune, crop-tool, shear, clipper crop-portrait |  | 436
download |  file-download, save, import, cloud |  | 444
edit |  editing, handwriting, pencil, pen, edit-fill, modify |  | 546
emoji |  amojee, sad, happy, emotion |  | 374
envelope |  letter, email, mail, inbox |  | 332
exit |  quit, close, switch-off, logout |  | 404
flower |  flowers, flower pot, sunflower, valentine-flower |  | 377
gift |  present, reward, surprise |  | 340
house |  home, rent, house-area, house asset, building, mall |  | 378
left |  return, back, prev, backwards |  | 531
like |  thumb-up, heart, vote, hand-like, upvote, dislike, favourite |  | 386
location |  gps, direction, compass, navigation |  | 543
menu |  menu file, card, menufold, menu-line, more, dashboard |  | 351
minus |  remove, minus (with circle), minus-sign |  | 556
music |  music-note, music-library, musical-instrument |  | 375
news |  newspaper, info, announcement |  | 423
package |  package-up, package-sent, handpackage, personal package |  | 362
pay |  money, wallet, dollar, commerce |  | 364
person |  user, avatar, account, customer |  | 562
photo |  image, picture, camera |  | 481
play |  playicon, broadcast, play voice, play button, play arrow |  | 498
question |  ask, faq, information, help, info, support |  | 350
refresh |  reload, sync, reset, recreate |  | 321
right |  forward, next, go, arrow-forward |  | 412
safe |  safe box, safety, safety certificate, lock, secure |  | 476
search |  investigate, search-engine, magnifier, find, glass |  | 377
send |  send-arrow, paper-plane, message |  | 318
settings |  toolbox, gear, preferences, options |  | 317
shopping |  cart, shopping-bag, checkout |  | 472
signal |  signal-tower, wave, radio, broadcast |  | 548
sound |  speaker, sound volume, player |  | 415
star |  collection, rate, favourite |  | 424
switch |  switch-on/off, switcher, open, close |  | 319
text |  word, textbox, font, size |  | 446
visibility |  visible, show, hide, visibility-off, in-sight |  | 339
warn |  alarm, warning, error, report, alert |  | 369
wifi |  wi-fi, wireless, network, signal |  | 429
zoom-in |  fullscreen, expand, adjust, magnifier |  | 384

In this section, we carry out an empirical study of collaborative icon
labelling in online icon design sharing websites to understand its
characteristics for motivating the required tool support. There are numerous
online icon design sharing websites such as Font Awesome (Fonticons, 2020),
Google Material Design Icons (Google, 2020), provide comprehensive icon
library to assist designers and developers in designing and coding. In these
online icon websites, each label matches only one icon design. While in real
case, one label may have several different designs, revealing the limited
diversity of these websites. In this work, we select Alibaba Iconfont website
(Alibaba, 2020a) as our study subject - not only because it has gained
significant popularity among designers’ community, but also due to it has
became repositories of knowledge with millions of diverse icon designs created
by designers. To collect icons and associated labels, we built a web crawler
based on the Breadth-First search strategy (Najork and Wiener, 2001) i.e.,
collecting a queue of URLs from a seed list, and putting the queue as the seed
in the second stage while iterating. The crawling process continued from
December 12, 2019 to July 1, 2020 with a collection of 100k graphical icons.

###  4.1. Overview

During the process of open coding the categories of icons semantic, we find
that one label can be written in different styles. For example, the label
”crop” can be written in not only its standard format, but also its
derivations synonyms like ”prune”, ”clip”, ”crop-tool”. Moreover, due to the
icon labelling process in Iconfont is informal and icon designs are
contributed by thousands of designers with very diverse technical and
linguistic backgrounds the same concept may labeled in many user defined terms
such as ”crop-fill”, ”crop-portrait”, ”icon-crop-solid-24px”. The wide
presence of forms poses a serious challenge to icon classification task. For
example, the icon can be described to the class of ”crop” or ”clip”, which
makes sense in both classes.

To address the problem, we adopted association rule mining (Agrawal et al.,
1994) to discover label correlations from label co-occurrences in icons. We
leveraged the visual information from the icons and textual information from
the labels to group a pair-wise correlation of labels. For measuring the
visual similarity, we adopted the image similarity score MSE (Wang et al.,
2004) $sim_{vis(x,y)}$ to calculate the likelihood if two icons are the same.
For measuring the textual information, we first trained a word embedding
(Mikolov et al., 2013) model to convert each label into a vector that encodes
its semantic. Then we defined a lexical similarity threshold based on the
string edit distance (Levenshtein, 1966) $sim_{text(x,y)}$ to check if two
labels are similar enough in the form. The labels are grouped as a pair-wise
correlation if $sim_{vis(x,y)}\geq 0.9$ or $sim_{text(x,y)}\geq 0.9$. As we
wanted to discover the semantics and construct a lexicon of categories, we
found frequent pairs of labels. A pair of labels is frequent if the percentage
of how many icons are labelled with this pair of tags compared with all the
icons are above the minimum support threshold $t_{sup}\geq 0.001$. Given a
frequent pair of labels $\\{t_{1},t_{2}\\}$, association rule mining generated
an association rule $t_{1}\Rightarrow t_{2}$ if the confidence of the rule
$t_{conf}\geq 0.2$. Given the mined association rules, we constructed an
undirected graph $G(V,E)$, where the node set $V$ contains the labels
appearing in the association rules, and the edge set $E$ contains undirected
edges $<t_{1},t_{2}>$ (i.e., pair of label associations) if the two labels
have the association $t_{1}\Rightarrow t_{2}$ or $t_{2}\Rightarrow t_{1}$.
Note that the graph is undirected because association rules indicate only the
correlations between antecedent and consequent. All threshold values were
carefully selected through manually check, considering the balance between the
information coverage and overload.

To identify the set of frequently occurred icon label categories, we performed
an iterative open coding of most frequent co-occurring labels (or
approximately 9.2% of the dataset 542,334 in total) with existing expert
lexicon of categories in books and websites such as Google’s Material icon set
(Google, 2020), IBM’s Design Language of Iconography (IBM, 2020) and Design
Pattern Gallery (Neil, 2014). Two researchers from our team independently
coded the categories of these labels, noting any part of the initial
vocabulary. Note that both researchers have design experiences in both icons
and UI development. After the initial coding, the researchers met and
discussed the discrepancies and the set of new label categories until
consensus was reached. A semantic icon categories can be seen in Table 1. We
observed two distinct characteristics in icons compared to the physical-world
objects.

High cross-class similarity: Icons of different classes often have similar
size, shape and visual features. The visual differences to distinguish
different classes of icons can be subtle, particularly small widgets are
differentiated by small visual cues. For example, the difference between
”newspaper” and “file” lies in a text of news at the top/bottom side of
”newspaper”, while a plus/minus symbol distinguishes ”zoom in”/”zoom out” from
”search”. In addition, direction is also an important aspect to distinguish
classes. For example, the inclined waves represent ”signal” and the upward
waves represent ”wifi”. Existing object classification tasks usually deal with
physical objects with distinct features across classes, for example, fishes,
flowers, hockey and people in the popular ImageNet dataset (Deng et al.,
2009). High cross-class similarity affects classification as the class can be
not easily distinguished.

Small, transparent and low contrast: To make UI unique and stylish in the
screen, icons are usually small and partially transparent, such as the last
icon in the ”minus” class shown in Table 1. The transparent icons in the UIs
do not cause vision conflict, while they are less visible when separated from
the background context. For example, the first icon in the ”text” class in
Table 1 is an icon with low color contrast and uses transparency and shadow to
stress contrast. While the contrast of the object is obvious in the current
dataset, especially apparent in the greyscale format such as MNIST dataset
(LeCun et al., 1998).

Existing icon sharing sites contain a wide presence of forms of labeling.
Based on different background knowledge, designers use different same-meaning
labels to annotate the same icon. Such limitation not only confirms our
finding of difficulty of commenting in Section 3.3, but also hinders the
potential challenge in classification task. Therefore, a data mining approach
capturing visual and textual information is applied to construct a lexicon in
icons. By observing the lexicon, we find that two distinct characteristics of
icon different from the existing physical object orientated dataset.

Figure 2. The approach of our tool, Auto-Icon+. Given a design artifact as
input, we first compose the components into icons by clustering, for example,
the ”eye”, ”mouth”, ”face” components are composed into a smiling face icon.
We then apply three major functions, involving font conversion, prediction and
color detection, to generate the output code for the icon, which consists of a
font file (.ttf) and a descriptive tag, for example ”icon-emoji” and ”blue”.

##  5\. Approach

Based on the interview study in Section 3, we summarized three design
considerations for our approach: (1) icons are assembled from multiple
components in the design artifact. (2) icons implemented as fonts can expedite
rendering. (3) adding descriptions to icons can increase readability and
accessibility of code.

To solve (1), we proposed a machine learning technique to cluster the
scattered components in the design artifact through heuristic correlation
coefficient and then composed the clusters of components into icons (Section
5.1). To solve (2), we proposed an automated conversion technique, taking an
icon as the input, and outputting a vector graphics font(Section 5.2). To
address (3), we got inspired from the findings in our study on icon
characteristic in Section 4 to develop a deep learning model to automatically
assign the classes of icons in order to reduce the effort of manually
designing the description of icons (Section 5.3). Additionally, we applied a
primary color detection method based on computer vision to keep track of the
primary color of the icon in order to support more detailed description in
code (Section 5.4). The overview of our approach is shown in Fig. 2.

###  5.1. Icon Composition

Input : UI design artifact $X$ with $n$ elements
$\\{x_{1},x_{2},x_{3},...,x_{n}\\}$

Output : Hierarchical Dendrogram $HD$

1 construct $n\times n$ matrix $HD$ with correlation metric $cor(i,j)$ between
the elements ;

2 while  _len( $X$) ¿ 1_ do

3 Select the pair $(x_{i},x_{j})$ with the largest correlation value, such as
$x_{i},x_{j}\in X$ ;

4 Merge the pair $(x_{i},x_{j})$ into a new cluster $x_{merge}=x_{i}\cup
x_{j}$, let $x_{i},x_{j}$ be the sub clusters of cluster $x_{merge}$. ;

5 Update $X\leftarrow X\cup\\{x_{merge}\\}-\\{x_{i},x_{j}\\}$ ;

6 foreach  _$x\in X$_ do

7 Update the matrix $HD$ with correlation metric $cor(x,x_{merge})$ ;

8

9 end foreach

10

11 end while

return $HD$

Algorithm 1 Hierarchical Agglomerative Clustering (HAC)

The most common way to group related pieces is through clustering. Normally,
the assumed number of clusters may be unreliable since the number of the icons
among the design artifact is unknown and thus the top-down partitioning
methods (i.e., K-Means (Likas et al., 2003), Expectation-Maximization
(Dempster et al., 1977), etc.) will not applicable. To provide clustering
without requiring the knowledge of clusters, we adopted a bottom-up approach
Hierarchical Agglomerative Clustering (HAC) (Murtagh and Legendre, 2014). The
details of HAC is shown in Algorithm 1, where each component in the design
artifact is treated as a singleton cluster to start with and then they are
successively merged into pairs of clusters until all components have merged
into one single large cluster. The main parameters in this algorithms are the
metric used to compute the correlation value of components, which determines
the pair of clusters to be merged at each step. According to our observations
in Section 3.1, we defined a heuristic correlation metric that takes into
account component’s attribute, hierarchy, and rendering:

(1) $Correlation(x,y)=\alpha\times ATTR(x,y)+\beta\times
HRCHY(x,y)+\gamma\times IOU(x,y)$

where $ATTR(x,y)$ measures the attribute type of two components $x,y$, that is
if they have same type (i.e., image, curve, shape, etc.), it assigns the value
to 1 else 0. $HRCHY(x,y)$ measures the hierarchy between components, the value
is assigned to 1 if they are under the same group, 0 otherwise. $IOU(x,y)$
measures the overlap between components, taking a value between 0 to 1. And
$\alpha,\beta,\gamma$ is the user-defined weights for each measurements. The
higher the correlation value, the more likely the cluster can composed to an
icon.

Figure 3. The HAC clustering algorithm to compose components into icons in the
design artifacts, that repeatedly selecting and merging pairs of clusters
based on a heuristic correlation metric, until a single all-inclusive cluster
(UI). By empirically setting a correlation threshold to 0.6 and distinguishing
icon cluster by its feature as discussed in Section 4.1 (small, transparent
and low contrast), the blue cluster is determined as an icon.

The agglomeration of clusters results in a tree-like structure called the
dendrogram as shown in Fig 3. The value is highest at the lowest level of the
dendrogram and it decreases as the clusters merge into the final single
cluster. By cutting the dendrogram at an empirically setting of value
threshold (e.g., 0.6), we may retrieved several clusters, for example, the
icon (blue cluster), text (red cluster), and background (green cluster) in Fig
3. To further identify if the aggregated cluster is an icon (blue cluster), we
followed the feature of icon discussed in Section 4.1, that the icon should be
small and low contrast.

###  5.2. Font Conversion from Icon

Unlike converting font to image, transcribing image to font, which is also
known as image tracing problem, is a difficult task. In this work, we adopted
the state-of-the-art Potrace (Selinger, 2003) in Fig 2A. We first applied a
pre-processing method for converting color to binary (i.e., black and white)
image by setting a threshold to control the bit of each pixel after
calculating the average value in three channels $(R+G+B)/3$. We regarded the
pixel is of white if the average value is larger than 128, while black if the
value is equal to or smaller than 128. Then, we detected the edge in the
black-white image. An edge is defined to be a border between a white pixel and
a black pixel, which indicate which pixels from the original image constitute
the borders of region. Note that the edge is assigned a direction so that when
moving from the first endpoint to the second, the black pixel is on the left
(as shown in Fig 2A edge detection). This process was repeated until we
reached the starting point, at which point we have found a closed path which
encloses a black region. Once the border was found, we approximated/optimized
the border with a polygon to figure out which border pixels is possible to
connect with a straight line such that the line passes through all the border
pixels between its endpoints. To detect the optimal polygon, we computed a
penalty value to measure the average distance from the edge to the pixels it
approximates. The polygon with the smallest penalty (equivalent to the polygon
with the fewest pixels) is the optimal one. Finally, we used a cubic curve
defined by four control points (also known as Bezier curve (Sederberg and
Farouki, 1992)) to smooth the corners. The first and fourth control points
(i.e., midpoints of the edges of the polygon) give the locations of the two
endpoints of the curve, while the second and third (i.e., chosen on the
polygon edges through the endpoints) indicate the direction and magnitude of
the derivative of the curve at each endpoint.

###  5.3. Prediction for Icon

Traditional Convolutional Nerual Network (CNN) (LeCun et al., 1998; Krizhevsky
et al., 2012) has shown great potential as a solution for difficult vision
problems. MobileNetV2 (Sandler et al., 2018) distills the best practices in
convolutional network design into a simple architecture that can serve as
competitive performance but keep low parameters and mathematical operations to
reduce computational power. The architecture of the network is shown in Fig
2B.

Instead of using regular convolutional layers widely used in traditional CNN
architectures to capture essential information from images but are expensive
to compute, MobileNetV2 adopted a more advanced one, depthwise separable
convolutions. Depthwise separable convolution combined a $3*3$ convolution
layer and two $1*1$ convolution layers. The $1*1$ convolution layer (also
named as pointwise convolution layer) was used to combine the filter values
into new features, while the $3*3$ convolution (also called as depthwise
convolution layer) was used to filter the input feature map. Inspired from the
dimension augmentation in the work of (Lin et al., 2013), MobileNetV2 used a
$1*1$ pointwise convolution layer to expand the number of channels in the
input feature map. Then it used a $3*3$ depthwise convolution layer to filter
the input feature map and a $1*1$ convolution layer to reduce the number of
channels of feature map. The network borrowed the idea of residual connection
in ResNet (He et al., 2016) to help with the flow of gradients. In addition,
batch normalization and activation layer were added between each depthwise
convolution layer and pointwise convolution layer to make the network more
stable during training. For detailed implementation, we adopted the stride of
2 in the depthwise convolution layer to downsample the feature map. For the
first two activation layers, the network used ReLU6 defined as
$y=min(max(0,x),6)$ because of its robustness in low-precision computation
(Howard et al., 2017), and a linear transformation (also known as Linear
Bottleneck Layer) was applied to the last activation layer to prevent ReLU
from destroying features.

###  5.4. Color Detection of Icon

Since the conversion between icon and font sacrifices the color identity, we
added an attribute to keep track of the primary color of the icon. To that
end, we adopted HSV colorspace for color detection. We first removed the
fourth alpha channel as transparent and made a conversion from RGB color to
HSV colorspace. Each RGB color has a range of HSV vale. The lower range is the
minimum shade of the color that will be detected, and the upper range is the
maximum shade. For example, blue is in the range of $\langle
100,43,46\rangle$–$\langle 124,255,255\rangle$. Then, we created a mask for
each color (black, blue, cyan, green, lime, megenta, red, white) as shown in
Fig. 2C. The mask is the areas that HSV value on pixels match the color
between the lower range and upper range. Finally, we calculated the area of
the mask in each color and the corresponding image occupancy ratio. The color
with the maximum ratio was identified as the primary color of the icon (the
blue in the example in Fig. 2C).

##  6\. Experiments

In this section, we first set up an experiment to analyze the performance of
our tool. Then we conduct a pilot user study to evaluate the usefulness of our
tool. Furthermore, we demonstrate its usefulness on a large-scale industrial
benchmark. The goal of our experiments is to answer the following research
questions, in terms of accuracy, efficiency and applicability. RQ 1: how
accurate is our heuristic algorithm in composing icons from design artifacts?
RQ 2: how accurate is our model in predicting labels for icon images? RQ 3:
how much do our tool increase the efficiency of UI development? RQ 4: what are
the developers’ opinions on the usability of our tool?

###  6.1. Icon Composition

####  6.1.1. Dataset:

To evaluate the performance of our heuristic composition algorithm, we compare
our automatically generated icon to manually generated ground truth. To
generate the ground truth, we had one author and 10 non-author paid annotators
independently label the components in the design artifacts (e.g., the
component belongs to not-icon or $icon_{x}$, as one artifact may contains
multiple icons, we apply $x$ to annotate icon separately). Each design
artifact was labeled by 3 different annotators and the ground truth was
generated until an agreement was reached. All annotators have previous
experience designing and implementing icons. In total, annotators labeled
1,012 real-world design artifacts with 9,883 icons.

####  6.1.2. Baselines & Metrics:

Since the number of icons in the design artifact is unknown, we set up two
widely-used clustering algorithms that do not need to pre-set the number of
clusters in advance as baselines, Mean-Shift Clustering (Cheng, 1995) and
Density-Based Spatial Clustering of Applications with Noise (DBSCAN) (Ester et
al., 1996). To further gauge the advantages of our heuristic icon composition
algorithm, we compare the icon produced by our correlation metric to each
metric individually.

Mean-Shift: Mean-shift is a centroid-based clustering algorithm to detect the
close components in the design artifact. It works by updating the candidates
for the center points as the mean of the points within the sliding window.

DBSCAN: It is an improvement over the Mean-Shift clustering as it involves a
transitivity based chaining-approach to determine whether components are
located in a particular cluster, in order to separate clusters of noise and
outliers.

ATTR only: It composes the icon based on the component’s attribute in the
artifacts, such as the curve components are merged together as an icon.

HRCHY only: As designers may manage all the resources of the icon under a
layout folder, we rely on this hierarchical structure information to assemble
the icon.

IOU only: The rendering of components in the icon are likely intersecting and
overlapping to each other, therefore we compose the icon based on measuring
the IOU of rendering.

To evaluate the performance, we set up three evaluation metrics, e.g.,
precision, recall, F1-score. Precision is the proportion of icons that are
correctly composed among all icons in the design artifacts. Recall is the
proportion of icons that are correctly composed among all composed components
(e.g., $icon_{x}$ and not-icon). F1-score is the harmonic mean of precision
and recall, which combine both of the two metrics above.

Method | Mean-Shift | DBSCAN | ATTR only | HRCHY only | IOU only | Our
---|---|---|---|---|---|---
Precision | 40.16% | 55.54% | 27.10% | 3.23% | 45.40% | 76.50%
Recall | 51.91% | 73.34% | 35.20% | 6.25% | 62.51% | 81.25%
F1-score | 45.28% | 63.21% | 30.62% | 4.26% | 52.59% | 78.80%
Table 2. Performance comparison for icon composition.

(a) Compared to ATTR only, our algorithm can compose icon with different
attributes (bitmap and path).

(b) Compared to HRCHY only, our algorithm can compose icon with different
hierarchies (”Oval” not in the layout folder).

(c) Compared to IOU only, our algorithm can compose icon with no intersection
(shapes are widely spaced).

Figure 4. The results of our icon composition algorithm. The top represents
the design artifact, and the bottom represents the clustering steps.

####  6.1.3. Results:

Table 2 shows the performance of all methods. The centroid-based clustering
algorithm can only achieve 45.28% and 63.21% F1-score, for Mean-Shift and
DBSCAN respectively. The issues with these baselines are that they are
designed for large and dense data, such as natural disasters, stations, etc.
However, different from those data, the number of components in design
artifacts is relatively small. The performance of our algorithm is much better
than that of other correlation baselines, e.g., 31.1%, 18.74%, 26.21 boost in
precision, recall, and F1-score compared with the best correlation baseline
(IOU only). Relying only on hierarchical structure information (HRCHY only)
yields poor performance in all metrics, 3.23%, 6.25%, and 4.26% respectively
compared to the ground truth. This is because that many designers follow their
own design styles which highly differ from each other, and some of their
design miss the rigid layout information. ATTR only yields an average F1-score
of 30.62%, which is better than the F1-score of HRCHY only, indicating that
the attribute of components provides better information for icon composition
as designers tend to use consistent attributes for icon designs. IOU only
yields the best performance among all baselines with the average F1-score to
52.59%, indicating that the icons are normally composed by several
intersecting components. Fig 4 shows some results of icon composition by our
heuristic algorithm. It shows that our algorithm can help cluster the
components with different attributes (e.g., bitmaps and path for the first
icon), different hierarchies (e.g, ”Oval” component is not in the layout for
the second icon), and no intersection (e.g., the components of icon have
spacing for the third icon).

To identify the common causes of errors, we further manually check the wrong
composition cases in the test dataset. According to our observation, we found
two main reasons. First, the components of icons are widely spaced, such as,
three separated waves for ”wifi”, three horizontal lines for ”menu”, several
lines from low to high for ”signal”, etc. In addition, designers may depict
some effects around the icon, such as, the ripple effect on a hand for ”tap”,
the shiny effect on a bulb for ”light”, etc. The wide space between components
decreases the correlation value of IOU, resulting in poor clustering. Second,
some designers place icon and text in the same hierarchy to depict an image
button, for example, a folder consists of icon components, text components,
button border, etc., causing our algorithm to aggregate an image button,
rather than an icon.

###  6.2. Icon Prediction

####  6.2.1. Dataset:

We leveraged the categorization during the creation of the semantic vocabulary
(in Table 1), and corresponding icons and attached labels as the training
data. The foundation of the deep learning model is the big data, so we only
selected categories with frequency larger than 300 for training the model.
Therefore, there are 100 categories left with the number of icons ranging from
311 to 589. Given all the data assigned to each label, we randomly split these
41k icons into train/validation/test dataset with a ratio of 8:1:1
(33K:4K:4k). To avoid the bias of of “seen samples” across training,
validation and testing, we performed 5-fold cross-validation.

####  6.2.2. Baselines:

We set up several basic machine-learning baselines including the feature
extraction (e.g., color histogram (Wang et al., 2010), scale-invariant feature
transform (Lowe et al., 1999)) with machine-learning classifiers (e.g.,
decision tree (Quinlan, 1983), SVM (Cortes and Vapnik, 1995)). Apart from
these conventional machine learning based baselines, we also set up several
derivations of state-of-the-art deep learning models as baselines to test the
importance of different inputs of our approach including backbones (ResNet (He
et al., 2016), VGG (Simonyan and Zisserman, 2014), MobileNet (Sandler et al.,
2018)), different input channels (RGB, RGBA). The training and testing
configurations for these baselines were the same.

Method |  | Histo
---
+SVM
| Histo
---
+DT
| SIFT
---
+SVM
| SIFT
---
+DT
ResNet-50 | VGG-16 |  | MobileNetV2
---
(RGBA)
| MobileNetV2
---
(RGB)
Accuracy | 0.5657 | 0.3267 | 0.5806 | 0.4686 | 0.8839 | 0.8764 | 0.8348 | 0.8772
Time (ms) | 0.103 | 0.152 | 1.702 | 1.941 | 26.535 | 27.282 | 17.567 | 17.485
Table 3. Label classification accuracy and time estimation in different
methods.

####  6.2.3. Results:

As we trained a classifier to predict label for icon, we adopted the accuracy
as the evaluation metric for the model, illustrated in Table 3. The
traditional machine learning method based on the human-crafted features can
only achieve about 0.6 average accuracy. Deep learning models perform much
better than the best old fashioned methods, i.e., with the 0.3033, 0.29712,
0.2958 increase for ResNet-50, VGG-16, and MobileNetV2 respectively. Although
ResNet model performs the best in icon classification task, it requires
relatively long time for prediction (26.535ms per icon) which strongly
violates the performance of UI rendering (16ms) as we aim to deploy into an
online platform (Imgcook). In contrast, our model MobileNet is nearly as
accurate as ResNet with a performance lag of 0.67%, while being 34.1% faster.
And also, we find that the increase of a fourth alpha channel (RGBA) decreases
the accuracy from 0.8772 to 0.8348, due to two main reasons. First, the result
shows that the model with RGB input has a loss value of 0.7844 at epoch 200,
which is better than the model with RGBA (0.9231). This is because the
supplemented channel greatly increases the parameters of the model, which
leads to a decline in the ability of gradient training at the same epochs.
Second, based on the principle of optics, the fourth alpha channel does not
reflect the morphological characteristics of the image. It is used to reduce
information of other three channels by adjusting their color/degree, causing
less information captured through training process.

###  6.3. User Study

Table 4. Examples of development for icons in Experimental and Control groups. |  E |  ¡i class=”icon-left red”¿¡/i¿
---|---|---
C1 |  ¡img src=”8E431911-61BB-4A19-8C01.svg”/¿
C2 |  ¡img src=”next-icon-design.svg” alt=”next” width=”100%”/¿
C3 |  ¡div style=”background-image: url(’8E431911-61BB-4A19-8C01.svg’);”¿¡/div¿
|  E |  ¡i class=”icon-information white”¿¡/i¿
C1 |  <svg class=”icon” aria-hidden=”true”> <use xlink:url=”icon-information”></use> </svg>
C2 |  ¡a style=”background-image: url(’q&a.svg’); width: 100%; height:100%;” class=”help-icon”¿¡/a¿
C3 |  ¡a class=”svg-icon-info” href=”#”¿¡SvgIcon name=”white”¿¡/a¿

####  6.3.1. Procedures for User Study

10 developers (generators), all proficient in UI development and have at least
1-year experience, were recruited for this study. We randomly selected five
icons from the real-life UI designs and asked each generator to develop them.
To guarantee the generators can objectively develop the icons, we asked
whether they have prior knowledge on the icons (such as development
experience, design experience, etc.). The time of the development were
recorded. To be fair, generators did not know we were recording the time as
the time pressure may affect their development (in quality, speed, etc.)
(Austin, 2001; Mäntylä et al., 2014). We set the manual development as the
control group. Then, we also asked them to develop five other icons with the
help of our tool which not only automatically convert the image to font, but
also provide the description (predicted label and color). We called this the
experimental group. The detailed developments of two groups for icons are
shown in Table 4.

We then recruited another 10 developers (evaluators), and each of them was
assigned the developments from two control groups and one experimental group.
Note that they did not know which one if from the experimental or control
group, and for each icon, we randomly shuffled the order of candidates to
avoid potential bias. Given each development, they individually marked it as
readable or not in five-point likert scale (1:not readable at all and
5:strongly readable). To evaluate the performance of usability, we also asked
evaluators to rate how likely they would like to use the development in
practice (Acceptable). The measurement is also in five-point scale.

Figure 5. The comparison of Experimental and Control groups. ∗denotes $p<$
0.01, ∗∗denotes $p<$ 0.05.

Figure 6. The icon description varies by context.

####  6.3.2. Results:

Box plot in Fig 6 shows that the time spent on the development of icons in the
experimental group is much shorter than that in the control group (with an
average of 6.05s versus 17.39s, saving 65.2% of time). That is the biggest
strength of our tool i.e., developers can quickly develop an icon by providing
descriptions and a font pattern. On average, the overall readability ratings
for the experiment group is 4.16, which are significantly higher (48.5%) than
the control group (2.8) in Fig 6. Most developers admit that our tool can
provide more acceptable results for them. In other words, 94% (4.7/5.0) of
developers hope to develop the icons with the help of our tool in their real
development environment compared to 3.96 in the control group. To understand
the significance of the differences, we carry out the Mann-Whitney U test (Fay
and Proschan, 2010) (specifically designed for small samples) on the
readability and acceptability ratings between the experiment and the control
group respectively. The test results suggest that our tool does significantly
outperform the baseline in term of these metrics with $p<0.01$ or $p<0.05$.

For some icons, the developer gives very low acceptability score to the
labels. According to our observation and further survey, we summarise two
reasons accounting for those bad cases. (1) Albeit the good performance of our
model, we still make wrong predictions for serendipitous icons. Based on the
context of icon, the same icon can have different meaning. For example, the
icon in Fig 6 represents the meaning of ”information” in the common case, but
consider the text on the right, the meaning of the icon should be
”glossary”/”dictionary”. (2) Developers admit the usefulness of converting
images to font for providing faster rendering speed. However, they also point
out the limitation of replacing image with font. Font is not fully compatible
in all browsers and devices. One developer mentioned that they need to make
sure that the development works on old devices, in which they usually need to
give up latest efficient methods, such as iconfont.

###  6.4. Industrial Usage

We cooperate our tool with the Imgcook platform(Alibaba, 2020b) developed by
Alibaba, an intelligent tool to automatically generate front-end codes from UI
design files. Imgcook has attracted a lot of attention in the community which
has a large user base (15k) and generates over 40k UIs. Auto-Icon+ is
integrated with the internal automated code generation process and is
triggered whenever the design files contain an icon.

In order to evaluate the usability of our tool, we set up a code review metric
for measuring the code modification for icons. Note that the code modification
contains multiple contents, such as text, button, etc, we only measure the
modification if the object is icon to reduce the potential bias. We adopt a
case-insensitive BLEU (BiLingual Evaluation Understudy) (Papineni et al.,
2002) as the metric to evaluate the preservation of code. BLEU is an automatic
evaluation metric widely used in code difference studies. It calculates the
similarity of machine-generated code and human-modified reference code (i.e.,
ground truth) as $BLEU=BP*exp(\sum_{N}^{n=1}w_{n}logp_{n})$ where $p_{n}$
denotes the precision, i.e., the ratio of length n token sequences generated
by our method are also present in the ground-truth; $w_{n}$ denotes the weight
of different length of n-gram summing to one; $BP$ is 1 if c ¿ r, otherwise
$e^{(1-r/c)}$ where c is the length of machine-generated sequences and r is
the length of ground-truth. A high BLEU score indicates less modification in
the code review.

We run the experiment in Imgcook with 6,844 icons in 2,031 UIs from August 20,
2020 to September 20, 2020. Among all the testing UI developments, the
generated code for icon reaches 84% BLEU score, which means that most of the
code is used directly without any modification. It demonstrates the high
usability of Auto-Icon+ in practice. Based on the inspection results, we
categorize the modification into four categories. Two reasons are discussed in
the Section 6.3.2, in terms of wrong prediction and compatibility concern.
There are another two modifications mainly due to industrial practice. First,
in order to maintain the consistency of company’s coding experience, some
developers modify to a prescribed naming/rendering method, for example,
packing the icon of ”icon-camera” to a ¡Icon-Camera¿ tag. Second, UI
dynamically changes in practice. Once an element in the UI is changed, the
attribute of icon may change, such as color and font size.

Overall, our method achieves 78.8% F1-score in the icon composition from
design artifacts (RQ1), and 87.7% accuracy in the label prediction for icons
(RQ2). In the survey of 10 developers, we improve the efficiency of developing
time and code readability by 65.2% and 48.5%, respectively (RQ3). The majority
(4.7/5.0) of the interviewed developers acknowledges the usability of the
generated code for icon by our method, and it is further confirmed in the
practice of Imgcook with 84% BLEU score (RQ4).

##  7\. Discussion

On developers:

Due to the conceptual gap between designers and developers, there is a
transition barrier from design artifact to implementation. Developers require
extra effort to compose design components into icons for optimizing the
network traffic and caching. To bridge this transition gap, our work proposes
a heuristic machine learning technique to cluster components in the design
artifacts to automated agglomerate the scattered icon components into
clusters, helping to ease the burden of manual search for developers. The
clusters can be further applied to various downstream tasks such as icon
implementation, UI testing (Moran et al., 2018a), css clipping (Larsen, 2018),
etc. Icon implementation is a challenging and time-consuming task, even for
professional developers. On the one hand, UI developers must enhance
performance. Poor development has an adverse effect on the performance of the
site. The performance issues comprise a multitude of factors like rendering
speed, reusability & flow of the code, etc. On the other hand, UI developers
must write a clean, high quality code which can be easily understood and
maintained. Inspired by the high performance of font rendering, our work
designs an automated method to convert icon to font using computer vision
techniques to trace the edge of icon and using graphic algorithm to optimize
the edge. In addition, compared with the missing descriptions in the
development or brainstorming suitable names which is limited to several
developers in the physical world, our deep learning and computer vision
techniques based method can quickly identify the label and the color of icon.
Our method once made accessible to developers, can very well help developers
achieve efficient icon coding.

On the generalization of our method: We report and analyze the performance of
our CNN-based model for predicting icon labels in Section 6.2.3. One
limitation with our model is that we currently only test our model on 100
labels with enough corresponding icons. With the cooperation with Imgcook
platform, the icons in the UI images are a gold resource. First, the icons are
relatively unique, otherwise, developers can reuse the online resources
directly. These unique icons can significantly increase the amount of data,
consequently improving the accuracy of our model. Second, developers may
modify the description to a serendipitous label which can augment the labels
and generalize a broader range of icon descriptions. Due to the time limit, we
only collect a small amount of icons from Imgcook. However, we have seen some
interesting icons that do not exists on online sharing platforms and they may
improve the generalization of our method.

Area of improvements: Currently, we only predict the label based on icon
itself. As discussed in Section 6.3.2, the meaning of icon varies in different
context. To address this problem, we can consider the entire UI, capturing all
the related information to make the final prediction. Developers praise the
idea of adding descriptions to the code which is a tedious task for them. They
wonder whether our model can extend to other elements. An developer hope us to
support description for buttons as he finds many buttons do not have
descriptive texts to explain its intention, resulting in a bad user
experience. We believe our model could help developers in this case as it will
not be difficult to extend to other elements once we obtain enough data for
the training. Moreover, developers envision the high potential in being able
to add icon size descriptions as one of the biggest strength of font is
lossless scalability. To that end, we can measure the height of the icon and
map it to the corresponding font size.

##  8\. Conclusion

In this paper, we present a novel approach, Auto-Icon+, that can provide
developers with intelligent support to reduce the development time of icon
design in the UI. Our approach consists of two integral parts: a heuristic
machine learning methods for icon composition, and a deep learning and
computer vision method for icon implementation. Our work possesses several
distinctive advantages: 1) to bridge the conceptual gap between designers and
developers in icon, we propose a heuristic-based clustering method to compose
the scattered icon pieces in the design artifact into an icon. 2) we develop
an automated image conversion method to turn an icon into a font in which
improving icon rendering speed. 3) to assist developers with better code
accessibility, we adopt a deep learning model to automatically predict the
descriptive label that convey the semantics of the icon. 4) base on the
colorspace of the image, we detect the primary color of the icon to provide
developers more knowledge on the icon. Our method is incorporated into
existing automated code generation platform to extend them beyond effective
and descriptive coding.

###### Acknowledgements.

We appreciate Yanfang Chang, Zixiao Zhao, and Jiawen Huang for designing and
conducting a survey experiment, and all the participants from Imgcook team in
Alibaba Group for taking surveys.

## References

  * (1)
  * Agrawal et al. (1994) Rakesh Agrawal, Ramakrishnan Srikant, et al. 1994\.  Fast algorithms for mining association rules. In _Proc. 20th int. conf. very large data bases, VLDB_ , Vol. 1215. 487–499.
  * Airbnb (2021) Airbnb. 2021.  Sketching Interfaces: Generating code from low fidelity wireframes.  https://airbnb.design/sketching-interfaces/.
  * Alibaba (2020a) Alibaba. 2020a.  Iconfont+.  https://www.iconfont.cn/.  Accessed: 2020-09-28.
  * Alibaba (2020b) Alibaba. 2020b.  Imgcook.  https://www.imgcook.com/.  Accessed: 2020-09-29.
  * Android (2019) Android. 2019.  Systrace.  https://developer.android.com/studio/profile/systrace/navigate-report.  Accessed: 2020-09-01.
  * Ang and Lim (2021) Gary Ang and Ee Peng Lim. 2021.  Learning Network-Based Multi-Modal Mobile User Interface Embeddings. In _26th International Conference on Intelligent User Interfaces_. 366–376.
  * Apple (2021) Apple. 2021.  Xcode Overview - Apple Developer.  https://developer.apple.com/xcode/.
  * Austin (2001) Robert D Austin. 2001\.  The effects of time pressure on quality in software development: An agency model.  _Information systems research_ 12, 2 (2001), 195–207.
  * Beaudouin-Lafon (2004) Michel Beaudouin-Lafon. 2004\.  Designing interaction, not interfaces. In _Proceedings of the working conference on Advanced visual interfaces_. 15–22.
  * Beltramelli (2018) Tony Beltramelli. 2018\.  pix2code: Generating code from a graphical user interface screenshot. In _Proceedings of the ACM SIGCHI Symposium on Engineering Interactive Computing Systems_. 1–6.
  * Black (2017) Alison Black. 2017\.  Icons as carriers of information.  _Information design: Research and practice_ (2017), 315–329.
  * Bunian et al. (2021) Sara Bunian, Kai Li, Chaima Jemmali, Casper Harteveld, Yun Fu, and Magy Seif Seif El-Nasr. 2021\.  VINS: Visual Search for Mobile User Interface Design. In _Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems_. 1–14.
  * Butler (2010) Margaret Butler. 2010\.  Android: Changing the mobile landscape.  _IEEE pervasive Computing_ 10, 1 (2010), 4–7.
  * Carette et al. (2017) Antonin Carette, Mehdi Adel Ait Younes, Geoffrey Hecht, Naouel Moha, and Romain Rouvoy. 2017\.  Investigating the energy impact of android smells. In _2017 IEEE 24th International Conference on Software Analysis, Evolution and Reengineering (SANER)_. IEEE, 115–126.
  * Chen et al. (2020b) Chunyang Chen, Sidong Feng, Zhengyang Liu, Zhenchang Xing, and Shengdong Zhao. 2020b.  From lost to found: Discover missing ui design semantics through recovering missing tags.  _Proceedings of the ACM on Human-Computer Interaction_ 4, CSCW2 (2020), 1–22.
  * Chen et al. (2019c) Chunyang Chen, Sidong Feng, Zhenchang Xing, Linda Liu, Shengdong Zhao, and Jinshui Wang. 2019c.  Gallery DC: Design Search and Knowledge Discovery through Auto-created GUI Component Gallery.  _Proceedings of the ACM on Human-Computer Interaction_ 3, CSCW (2019), 1–22.
  * Chen et al. (2018a) Chunyang Chen, Ting Su, Guozhu Meng, Zhenchang Xing, and Yang Liu. 2018a.  From ui design image to gui skeleton: a neural machine translator to bootstrap mobile gui implementation. In _Proceedings of the 40th International Conference on Software Engineering_. 665–676.
  * Chen et al. (2020a) Jieshan Chen, Chunyang Chen, Zhenchang Xing, Xiwei Xu, Liming Zhut, Guoqiang Li, and Jinshui Wang. 2020a.  Unblind your apps: Predicting natural-language labels for mobile GUI components by deep learning. In _2020 IEEE/ACM 42nd International Conference on Software Engineering (ICSE)_. IEEE, 322–334.
  * Chen et al. (2020c) Jieshan Chen, Mulong Xie, Zhenchang Xing, Chunyang Chen, Xiwei Xu, and Liming Zhu. 2020c.  Object Detection for Graphical User Interface: Old Fashioned or Deep Learning or a Combination?  _arXiv preprint arXiv:2008.05132_ (2020).
  * Chen et al. (2021) Qiuyuan Chen, Chunyang Chen, Safwat Hassan, Zhengchang Xing, Xin Xia, and Ahmed E Hassan. 2021\.  How Should I Improve the UI of My App? A Study of User Reviews of Popular Apps in the Google Play.  _ACM Transactions on Software Engineering and Methodology (TOSEM)_ 30, 3 (2021), 1–38.
  * Chen et al. (2019a) Sen Chen, Lingling Fan, Chunyang Chen, Ting Su, Wenhe Li, Yang Liu, and Lihua Xu. 2019a.  Storydroid: Automated generation of storyboard for Android apps. In _2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE)_. IEEE, 596–607.
  * Chen et al. (2019b) Sen Chen, Lingling Fan, Chunyang Chen, Minhui Xue, Yang Liu, and Lihua Xu. 2019b.  GUI-Squatting Attack: Automated Generation of Android Phishing Apps.  _IEEE Transactions on Dependable and Secure Computing_ (2019).
  * Chen et al. (2018b) Sen Chen, Minhui Xue, Lingling Fan, Shuang Hao, Lihua Xu, Haojin Zhu, and Bo Li. 2018b.  Automated poisoning attacks and defenses in malware detection systems: An adversarial machine learning approach.  _computers & security_ 73 (2018), 326–344.
  * Cheng (1995) Yizong Cheng. 1995\.  Mean shift, mode seeking, and clustering.  _IEEE transactions on pattern analysis and machine intelligence_ 17, 8 (1995), 790–799.
  * Cortes and Vapnik (1995) Corinna Cortes and Vladimir Vapnik. 1995.  Support-vector networks.  _Machine learning_ 20, 3 (1995), 273–297.
  * Dehling et al. (2015) Tobias Dehling, Fangjian Gao, Stephan Schneider, and Ali Sunyaev. 2015.  Exploring the far side of mobile health: information security and privacy of mobile health apps on iOS and Android.  _JMIR mHealth and uHealth_ 3, 1 (2015), e8.
  * Dempster et al. (1977) Arthur P Dempster, Nan M Laird, and Donald B Rubin. 1977\.  Maximum likelihood from incomplete data via the EM algorithm.  _Journal of the Royal Statistical Society: Series B (Methodological)_ 39, 1 (1977), 1–22.
  * Deng et al. (2009) Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009\.  Imagenet: A large-scale hierarchical image database. In _2009 IEEE conference on computer vision and pattern recognition_. Ieee, 248–255.
  * Developers (2020a) Android Developers. 2020a.  Inspect GPU rendering speed and overdraw.  https://developer.android.com/topic/performance/rendering/inspect-gpu-rendering.  Accessed: 2020-09-28.
  * Developers (2020b) Android Developers. 2020b.  Test UI performance.  https://developer.android.com/training/testing/performance.  Accessed: 2020-09-28.
  * Ester et al. (1996) Martin Ester, Hans-Peter Kriegel, Jörg Sander, Xiaowei Xu, et al. 1996\.  A density-based algorithm for discovering clusters in large spatial databases with noise.. In _kdd_ , Vol. 96. 226–231.
  * Fay and Proschan (2010) Michael P Fay and Michael A Proschan. 2010.  Wilcoxon-Mann-Whitney or t-test? On assumptions for hypothesis tests and multiple interpretations of decision rules.  _Statistics surveys_ 4 (2010), 1.
  * Feng et al. (2019) Ruitao Feng, Sen Chen, Xiaofei Xie, Lei Ma, Guozhu Meng, Yang Liu, and Shang-Wei Lin. 2019.  Mobidroid: A performance-sensitive malware detection system on mobile platform. In _2019 24th International Conference on Engineering of Complex Computer Systems (ICECCS)_. IEEE, 61–70.
  * Feng and Chen (2021) Sidong Feng and Chunyang Chen. 2021.  GIFdroid: Automated Replay of Visual Bug Reports for Android Apps.  _arXiv preprint arXiv:2112.04128_ (2021).
  * Feng et al. (2022) Sidong Feng, Chunyang Chen, and Zhenchang Xing. 2022\.  Gallery DC: Auto-created GUI Component Gallery for Design Search and Knowledge Discovery.  _arXiv preprint arXiv:2204.06700_ (2022).
  * Feng et al. (2021) Sidong Feng, Suyu Ma, Jinzhong Yu, Chunyang Chen, Tingting Zhou, and Yankun Zhen. 2021\.  Auto-icon: An automated code generation tool for icon designs assisting in ui development. In _26th International Conference on Intelligent User Interfaces_. 59–69.
  * FluidUI (2021) FluidUI. 2021.  FluidUI.com - Create Web and Mobile Prototypes in Minutes.  https://www.fluidui.com/.
  * Fontello (2020) Fontello. 2020.  icon fonts generator.  http://fontello.com/.  Accessed: 2020-09-07.
  * Fonticons (2020) Inc. Fonticons. 2020\.  Font Awesome.  https://fontawesome.com/.  Accessed: 2020-09-28.
  * Fu et al. (2013) Bin Fu, Jialiu Lin, Lei Li, Christos Faloutsos, Jason Hong, and Norman Sadeh. 2013\.  Why people hate your app: Making sense of user feedback in a mobile app store. In _Proceedings of the 19th ACM SIGKDD international conference on Knowledge discovery and data mining_. 1276–1284.
  * Gao et al. (2017) Yi Gao, Yang Luo, Daqing Chen, Haocheng Huang, Wei Dong, Mingyuan Xia, Xue Liu, and Jiajun Bu. 2017\.  Every pixel counts: Fine-grained UI rendering analysis for mobile applications. In _IEEE INFOCOM 2017-IEEE Conference on Computer Communications_. IEEE, 1–9.
  * Gómez et al. (2016) María Gómez, Romain Rouvoy, Bram Adams, and Lionel Seinturier. 2016.  Mining test repositories for automatic detection of UI performance regressions in Android apps. In _Proceedings of the 13th International Conference on Mining Software Repositories_. 13–24.
  * Google (2020) Google. 2020.  Material Icons.  https://material.io/resources/icons/.  Accessed: 2020-09-09.
  * Harrison et al. (2011) Chris Harrison, Gary Hsieh, Karl DD Willis, Jodi Forlizzi, and Scott E Hudson. 2011.  Kineticons: using iconographic motion in graphical user interface design. In _Proceedings of the SIGCHI Conference on Human Factors in Computing Systems_. 1999–2008.
  * He et al. (2016) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016\.  Deep residual learning for image recognition. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 770–778.
  * Howard et al. (2017) Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. 2017\.  Mobilenets: Efficient convolutional neural networks for mobile vision applications.  _arXiv preprint arXiv:1704.04861_ (2017).
  * Huang et al. (2017) Gang Huang, Mengwei Xu, Felix Xiaozhu Lin, Yunxin Liu, Yun Ma, Saumay Pushp, and Xuanzhe Liu. 2017.  Shuffledog: Characterizing and adapting user-perceived latency of android apps.  _IEEE Transactions on Mobile Computing_ 16, 10 (2017), 2913–2926.
  * IBM (2020) IBM. 2020.  Design Language.  https://www.ibm.com/design/language/iconography/ui-icons/library.  Accessed: 2020-09-09.
  * IcoMoon (2020) IcoMoon. 2020.  Icon Font & SVG Icon Sets.  https://icomoon.io/.  Accessed: 2020-09-07.
  * Islam (2015) Muhammad Nazrul Islam. 2015\.  Exploring the intuitiveness of iconic, textual and icon with texts signs for designing user-intuitive web interfaces. In _2015 18th International Conference on Computer and Information Technology (ICCIT)_. IEEE, 450–455.
  * King and Youngblood (2016) Bridgett A King and Norman E Youngblood. 2016.  E-government in Alabama: An analysis of county voting and election website content, usability, accessibility, and mobile readiness.  _Government Information Quarterly_ 33, 4 (2016), 715–726.
  * Koutsourelakis and Chorianopoulos (2010) Charalambos Koutsourelakis and Konstantinos Chorianopoulos. 2010\.  Icons in mobile phones: Comprehensibility differences between older and younger users.  _Information Design Journal_ 18, 1 (2010), 22–35.
  * Krizhevsky et al. (2012) Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. 2012\.  Imagenet classification with deep convolutional neural networks. In _Advances in neural information processing systems_. 1097–1105.
  * Larsen (2018) Rob Larsen. 2018\.  _Mastering SVG: Ace web animations, visualizations, and vector graphics with HTML, CSS, and JavaScript_.  Packt Publishing Ltd.
  * LeCun et al. (1998) Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. 1998.  Gradient-based learning applied to document recognition.  _Proc. IEEE_ 86, 11 (1998), 2278–2324.
  * Lee et al. (2020) Chunggi Lee, Sanghoon Kim, Dongyun Han, Hongjun Yang, Young-Woo Park, Bum Chul Kwon, and Sungahn Ko. 2020.  GUIComp: A GUI design assistant with real-time, multi-faceted feedback. In _Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems_. 1–13.
  * Levenshtein (1966) Vladimir I Levenshtein. 1966\.  Binary codes capable of correcting deletions, insertions, and reversals. In _Soviet physics doklady_ , Vol. 10. 707–710.
  * Li et al. (2021) Toby Jia-Jun Li, Lindsay Popowski, Tom Mitchell, and Brad A Myers. 2021.  Screen2Vec: Semantic Embedding of GUI Screens and GUI Components. In _Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems_. 1–15.
  * Li et al. (2019) Wenjie Li, Yanyan Jiang, Chang Xu, Yepang Liu, Xiaoxing Ma, and Jian Lü. 2019\.  Characterizing and detecting inefficient image displaying issues in Android apps. In _2019 IEEE 26th International Conference on Software Analysis, Evolution and Reengineering (SANER)_. IEEE, 355–365.
  * Likas et al. (2003) Aristidis Likas, Nikos Vlassis, and Jakob J Verbeek. 2003\.  The global k-means clustering algorithm.  _Pattern recognition_ 36, 2 (2003), 451–461.
  * Lin et al. (2013) Min Lin, Qiang Chen, and Shuicheng Yan. 2013.  Network in network.  _arXiv preprint arXiv:1312.4400_ (2013).
  * Linares-Vasquez et al. (2015) Mario Linares-Vasquez, Christopher Vendome, Qi Luo, and Denys Poshyvanyk. 2015.  How developers detect and fix performance bottlenecks in android apps. In _2015 IEEE international conference on software maintenance and evolution (ICSME)_. IEEE, 352–361.
  * Liu et al. (2014) Yepang Liu, Chang Xu, and Shing-Chi Cheung. 2014.  Characterizing and detecting performance bugs for smartphone applications. In _Proceedings of the 36th international conference on software engineering_. 1013–1024.
  * Liu et al. (2020) Zhe Liu, Chunyang Chen, Junjie Wang, Yuekai Huang, Jun Hu, and Qing Wang. 2020\.  Owl eyes: Spotting ui display issues via visual understanding. In _2020 35th IEEE/ACM International Conference on Automated Software Engineering (ASE)_. IEEE, 398–409.
  * Liu et al. (2022) Zhe Liu, Chunyang Chen, Junjie Wang, Yuekai Huang, Jun Hu, and Qing Wang. 2022\.  Guided bug crush: Assist manual gui testing of android apps via hint moves.  _arXiv preprint arXiv:2201.12085_ (2022).
  * Lowe et al. (1999) David G Lowe et al. 1999\.  Object recognition from local scale-invariant features.. In _iccv_ , Vol. 99. 1150–1157.
  * Malik et al. (2017) Muhammad Noman Malik, Huma Hayat Khan, and Fazli Subhan. 2017\.  Sustainable Design of Mobile Icons: Investigating Effect on Mentally Retarded User’s.  _Journal of Medical Imaging and Health Informatics_ 7, 6 (2017), 1419–1428.
  * Mäntylä et al. (2014) Mika V Mäntylä, Kai Petersen, Timo OA Lehtinen, and Casper Lassenius. 2014.  Time pressure: a controlled experiment of test case development and requirements review. In _Proceedings of the 36th International Conference on Software Engineering_. 83–94.
  * Meskens et al. (2009a) Jan Meskens, Kris Luyten, and Karin Coninx. 2009a.  Plug-and-design: Embracing mobile devices as part of the design environment. In _Proceedings of the 1st ACM SIGCHI symposium on Engineering interactive computing systems_. 149–154.
  * Meskens et al. (2009b) Jan Meskens, Kris Luyten, and Karin Coninx. 2009b.  Shortening user interface design iterations through realtime visualisation of design actions on the target device. In _2009 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC)_. IEEE, 132–135.
  * Microsoft (2021) Microsoft. 2021\.  Microsoft: Sketch2Code.  https://sketch2code.azurewebsites.net/.
  * Mikolov et al. (2013) Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013.  Distributed representations of words and phrases and their compositionality. In _Advances in neural information processing systems_. 3111–3119.
  * Mockup (2021) Mockup. 2021.  Mockup.io – Present and Manage iOS Mockups.  https://mockup.io/about/.
  * Mora et al. (2017) Higinio Mora, Virgilio Gilart-Iglesias, Raquel Pérez-del Hoyo, and María Dolores Andújar-Montoya. 2017\.  A comprehensive system for monitoring urban accessibility in smart cities.  _Sensors_ 17, 8 (2017), 1834.
  * Moran et al. (2018b) Kevin Moran, Boyang Li, Carlos Bernal-Cárdenas, Dan Jelf, and Denys Poshyvanyk. 2018b.  Automated reporting of GUI design violations for mobile apps. In _Proceedings of the 40th International Conference on Software Engineering_. 165–175.
  * Moran et al. (2018a) Kevin Patrick Moran, Carlos Bernal-Cárdenas, Michael Curcio, Richard Bonett, and Denys Poshyvanyk. 2018a.  Machine learning-based prototyping of graphical user interfaces for mobile apps.  _IEEE Transactions on Software Engineering_ (2018).
  * Murtagh and Legendre (2014) Fionn Murtagh and Pierre Legendre. 2014.  Ward’s hierarchical agglomerative clustering method: which algorithms implement Ward’s criterion?  _Journal of classification_ 31, 3 (2014), 274–295.
  * Najork and Wiener (2001) Marc Najork and Janet L Wiener. 2001.  Breadth-first crawling yields high-quality pages. In _Proceedings of the 10th international conference on World Wide Web_. 114–118.
  * Neil (2014) Theresa Neil. 2014\.  _Mobile design pattern gallery: UI patterns for smartphone apps_.  ” O’Reilly Media, Inc.”.
  * Nejati and Balasubramanian (2016) Javad Nejati and Aruna Balasubramanian. 2016.  An in-depth study of mobile browser performance. In _Proceedings of the 25th International Conference on World Wide Web_. 1305–1315.
  * Nguyen and Csallner (2015) Tuan Anh Nguyen and Christoph Csallner. 2015.  Reverse engineering mobile application user interfaces with remaui (t). In _2015 30th IEEE/ACM International Conference on Automated Software Engineering (ASE)_. IEEE, 248–259.
  * Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002\.  BLEU: a method for automatic evaluation of machine translation. In _Proceedings of the 40th annual meeting of the Association for Computational Linguistics_. 311–318.
  * Photoshop (2021) Photoshop. 2021\.  Photoshop apps - desktop, mobile, and tablet.  https://www.photoshop.com/en.
  * Proto (2021) Proto. 2021.  Proto.io Prototyping Tool - Prototyping for all.  https://proto.io/.
  * Quinlan (1983) J Ross Quinlan. 1983\.  Learning efficient classification procedures and their application to chess end games.  In _Machine learning_. Springer, 463–482.
  * Rosen et al. (2017) Sanae Rosen, Bo Han, Shuai Hao, Z Morley Mao, and Feng Qian. 2017.  Push or request: An investigation of HTTP/2 server push for improving mobile performance. In _Proceedings of the 26th International Conference on World Wide Web_. 459–468.
  * Ross et al. (2018) Anne Spencer Ross, Xiaoyi Zhang, James Fogarty, and Jacob O Wobbrock. 2018.  Examining image-based button labeling for accessibility in Android apps through large-scale analysis. In _Proceedings of the 20th International ACM SIGACCESS Conference on Computers and Accessibility_. 119–130.
  * Sandler et al. (2018) Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen. 2018.  Mobilenetv2: Inverted residuals and linear bottlenecks. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 4510–4520.
  * Sederberg and Farouki (1992) Thomas W Sederberg and Rida T Farouki. 1992.  Approximation by interval Bézier curves.  _IEEE Computer Graphics and Applications_ 5 (1992), 87–88.
  * Selinger (2003) Peter Selinger. 2003\.  Potrace: a polygon-based tracing algorithm.  _Potrace (online), http://potrace. sourceforge. net/potrace. pdf (2009-07-01)_ (2003).
  * Simonyan and Zisserman (2014) Karen Simonyan and Andrew Zisserman. 2014.  Very deep convolutional networks for large-scale image recognition.  _arXiv preprint arXiv:1409.1556_ (2014).
  * Sketch (2021a) Sketch. 2021a.  Sketch Developer.  https://developer.sketch.com/file-format/.
  * Sketch (2021b) Sketch. 2021b.  Sketch — The digital design toolkit.  https://www.sketch.com.
  * Studio (2021) Android Studio. 2021\.  Android Studio provides the fastest tools for building apps on every type of Android device.  https://developer.android.com/studio.
  * Sural et al. (2002) Shamik Sural, Gang Qian, and Sakti Pramanik. 2002.  Segmentation and histogram generation using the HSV color space for image retrieval. In _Proceedings. International Conference on Image Processing_ , Vol. 2. IEEE, II–II.
  * Todi et al. (2021) Kashyap Todi, Luis A Leiva, Daniel Buschek, Pin Tian, and Antti Oulasvirta. 2021.  Conversations with GUIs. In _Designing Interactive Systems Conference 2021_. 1447–1457.
  * Wang (2012) Fahui Wang. 2012\.  Measurement, optimization, and impact of health care accessibility: a methodological review.  _Annals of the Association of American Geographers_ 102, 5 (2012), 1104–1112.
  * Wang et al. (2019) Xu Wang, Chunyang Chen, and Zhenchang Xing. 2019.  Domain-specific machine translation with recurrent neural network for software localization.  _Empirical Software Engineering_ 24, 6 (2019), 3514–3545.
  * Wang et al. (2010) Xiang-Yang Wang, Jun-Feng Wu, and Hong-Ying Yang. 2010\.  Robust image retrieval based on color histogram of local feature regions.  _Multimedia Tools and Applications_ 49, 2 (2010), 323–345.
  * Wang et al. (2004) Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. 2004\.  Image quality assessment: from error visibility to structural similarity.  _IEEE transactions on image processing_ 13, 4 (2004), 600–612.
  * Wu et al. (2021) Jason Wu, Xiaoyi Zhang, Jeff Nichols, and Jeffrey P Bigham. 2021\.  Screen Parsing: Towards Reverse Engineering of UI Models from Screenshots. In _The 34th Annual ACM Symposium on User Interface Software and Technology_. 470–483.
  * Wu et al. (2016) Tianyong Wu, Jierui Liu, Zhenbo Xu, Chaorong Guo, Yanli Zhang, Jun Yan, and Jian Zhang. 2016.  Light-weight, inter-procedural and callback-aware resource leak detection for android apps.  _IEEE Transactions on Software Engineering_ 42, 11 (2016), 1054–1076.
  * Xiao et al. (2019) Xusheng Xiao, Xiaoyin Wang, Zhihao Cao, Hanlin Wang, and Peng Gao. 2019.  Iconintent: automatic identification of sensitive ui widgets based on icon classification for android apps. In _2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE)_. IEEE, 257–268.
  * Xie et al. (2020) Mulong Xie, Sidong Feng, Zhenchang Xing, Jieshan Chen, and Chunyang Chen. 2020.  UIED: a hybrid tool for GUI element detection. In _Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering_. 1655–1659.
  * Yan and Ramachandran (2019) Shunguo Yan and PG Ramachandran. 2019.  The current status of accessibility in mobile apps.  _ACM Transactions on Accessible Computing (TACCESS)_ 12, 1 (2019), 1–31.
  * Yang et al. (2021) Bo Yang, Zhenchang Xing, Xin Xia, Chunyang Chen, Deheng Ye, and Shanping Li. 2021\.  UIS-Hunter: Detecting UI Design Smells in Android Apps. In _2021 IEEE/ACM 43rd International Conference on Software Engineering: Companion Proceedings (ICSE-Companion)_. IEEE, 89–92.
  * Zhang et al. (2021) Xiaoyi Zhang, Lilian de Greef, Amanda Swearngin, Samuel White, Kyle Murray, Lisa Yu, Qi Shan, Jeffrey Nichols, Jason Wu, Chris Fleizach, et al. 2021\.  Screen Recognition: Creating Accessibility Metadata for Mobile Applications from Pixels. In _Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems_. 1–15.
  * Zhang et al. (2018) Xiaoyi Zhang, Anne Spencer Ross, and James Fogarty. 2018\.  Robust Annotation of Mobile Application Interfaces in Methods for Accessibility Repair and Enhancement. In _Proceedings of the 31st Annual ACM Symposium on User Interface Software and Technology_. 609–621.
  * Zhao et al. (2019) Dehai Zhao, Zhenchang Xing, Chunyang Chen, Xin Xia, and Guoqiang Li. 2019.  ActionNet: Vision-based workflow action recognition from programming screencasts. In _2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE)_. IEEE, 350–361.
  * Zhao et al. (2020) Dehai Zhao, Zhenchang Xing, Chunyang Chen, Xiwei Xu, Liming Zhu, Guoqiang Li, and Jinshui Wang. 2020.  Seenomaly: Vision-Based Linting of GUI Animation Effects Against Design-Don’t Guidelines. In _42nd International Conference on Software Engineering (ICSE’20). ACM, New York, NY_.
  * Zhao et al. (2016) Hui Zhao, Min Chen, Meikang Qiu, Keke Gai, and Meiqin Liu. 2016.  A novel pre-cache schema for high performance Android system.  _Future Generation Computer Systems_ 56 (2016), 766–772.
  * Zhao et al. (2021) Tianming Zhao, Chunyang Chen, Yuanning Liu, and Xiaodong Zhu. 2021\.  GUIGAN: Learning to Generate GUI Designs Using Generative Adversarial Networks. In _2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE)_. IEEE, 748–760.